3 From: "Glenn C. Everhart" <EVERHART@arisia.gce.com>  To:   info-vax@crvax.sri.com Subject: compressing disk   A I have thought about a compressing virtual disk for some time but E never got one working, though I did make a start way back. My initial @ concept was to use an isam file to hold the storage, compressingA records. Others use something like this approach, doing their own 9 space management and doing a better job than normal ISAM. E    On reflection, I don't think that's a terribly good approach since D the isam file would need to be frequently reorganized to gain space.  E What I'd do now would be to have the host program manage a whole slew D of files, each file comprising the contents of perhaps 100 blocks of@ the virtual disk (so it's long enough for adaptive algorithms toD do some good). A good host process for fddriver/fqdriver should also have some caching in itself.H  On reading a block, the host would read the file and decompress it intoB its cache someplace. Since each cache "cell" would be for 100 didkG oops, disk blocks or so, storage would be easy to manage. Writing would C write into this cache area. When the space was needed, the relevant F cache chunk would be compressed and written over the old file that hadE held that range of blocks. The host file system would then manage the B actual space allocation. This scheme would work on unix too, usingE sunfddvr.c (also on sig tapes). I believe that having a "fence" block B below which the disk storage would be just kept on A file is stillK a good idea; this would allow the disk to be INITed /INDEX=BEG/HEADERS=nnnn K so the index file would not be compressed, speeding many access operations. G I'd keep at least 10 and possibly more of these cache blocks, depending B on how much virtual address space I felt could be used for it; 100E would be better still; I'd treat them as LRU cache, but would provide F a periodic flush to disk every 10-20 minutes (guess) and would provideF a way to tell the host process to flush on command so a shutdown wouldH be able to ensure all data had flushed to disk. The overhead of flushing@ to disk for EVERY write would be a lot; a write to 1 block mightF result in compressing 100 and probably doing file open/close/truncate.D    I would also advise that if one plays this game (and I'd be happyG to see someone do it and put the result on the sig tapes for next time) I that the files containing all the actual storage live on a VD: or similar 4 virtual disk, because it's gonna fragment REAL fast.E    It should be added that a compressing disk is vulnerable to system F crashes, so that maintaining journals of written material and removing@ them once compressed versions are safely written is a worthwhileD thing. A journal can be roughly the size of the in memory cache, andH might be organized to be in the same order, with maybe one or two blocksE extra off to the side that had the index of cache block <-> disk LBN. D Periodic flushes of a cache are a kludge that might be tried insteadA (Unix uses such a scheme) but it is inherently more vulnerable to C corruption. By keeping the index file NOT compressed, of course, we D reduce the liklihood of total disk corruption; this is a good thing.  C    Incidentally, I have working code that catches FDT time activity O and lets me compress files onto disk when space is filling up and automatically H decompresses them if anyone opens them. The effect is that basically anyF files you like can be compressed on the disk (even to all the ones youH ever look at) and automatically get decompressed and pulled out when youF try to open them. It's very flexible, since it can store them anywhereH you like also (e.g., on tape, other disks, across any net, and so on...)@ compressed or not as pleases you. However, it's non-free and mayF become a product. It has a ton of other functions also. The nice thingG about that method of operation is that you can have a site policy about G what gets compressed first or preferentially, so you can play this game I with seldom-accessed files mostly, and take less of a hit in performance. G You can use the network file system hooks in unix to get a similar kind I of effect, but the performance hit may be worse; my code on vms is pretty . efficient at deciding what needs to be done...  J    I should add that the concept of leaving file headers out there as tagsE for files stored "somewhere else" is a dirty approach if there is any H way to store your index some other way; I plan to clean that up and have" most of the support for such done.  D    If anyone cares to build a compressing host for fddriver/fqdriverH though, please feel free to contact me if you have questions and/or wantI to ensure you are starting with the latest & greatest code (the F92 tapes  are actually fine).  Glenn C. Everhart (raxco, inc) Everhart@Arisia.GCE.Com  215 358 5875  