H From:	MERC::"uunet!VM1.NODAK.EDU!owner-anu-news" 21-APR-1993 17:33:06.33A To:	Multiple recipients of list ANU-NEWS <ANU-NEWS@VM1.NoDak.EDU>  CC:	, Subj:	Splitting ANU files over several disks  / In article <1993Apr18.220744.1@alien.gici.com>, ' laut@alien.gici.com (Bill Laut) writes:   O > A nice option would be to allow us to explicitly specify the device/directory F > of a given newsgroup and/or hierarchy, so that we can split the diskI > consumption across multiple spindles without resorting to bound volumes   M This is already provided for you by VMS - use search list in logical name(s).   9 Let me elaborate on this and broaden up a theme somewhat. 4 I will also touch on the issue of disk fragmentation? since this is a closely related theme - so please bear with me. A I speak of ANU-NEWS V6.1 (beta versions 5,6,7) but I believe that   most of it applies also to V6.0.  B The purpose of logical names NEWS_ROOT and NEWS_DEVICE is somewhat3 blurred as described in NEWS_STARTUP.COM_TEMPLATE :   I $   ! NEWS_DEVICE points to the same directory as NEWS_ROOT, but is setup  $   ! as a concealed device.  > This implies that NEWS_ROOT and NEWS_DEVICE differ only in theD /TRANSLATION=CONCEALED attribute of a logical name and that ANU-NEWSH expects them to point to the same top directory  -  which is fortunatelyH *not true*.  There are two completely unrelated set of files/directoriesB which *happen* to reside in the same directory *if* you follow theG suggestion in NEWS_STARTUP.COM_TEMPLATE (which may *not* be a good idea ; if you have more than one disk drive - more on that later).    The thruth is:  H 1. that NEWS_ROOT points to a directory which must contain the followingA    files:  NEWS.ITEMS, NEWS.GROUPS, HISTORY.V60, NNTP_ACCESS.NEWS   > 2. that NEWS_DEVICE (a concealed logical name) must point to aE    directory containing top level hierarchy directories of news items $    (such as COMP.DIR, REC.DIR, ...).  ? I suggest to the ANU-NEWS maintainers to change the comments in I NEWS_STARTUP.COM_TEMPLATE to distinguish the purpose of the two logicals. > The moving of index-seq files to another disk has already beenE mentioned in this group a couple of times, once even done by a patch, ; which clearly indicates the confusion / lack of information " in this area in the documentation.    E The NEWS_DEVICE file system hiararchy (containing text of news items) A lives a very dynamic life and tends to fragment disk in no time - A - which is not that bad for item files themselves (since they are B usually short and do not live long), but it sure gives hell to any@ *other* files living on that disk.  The only thing that is worthA considering for this disk (to reduce fragmentation) is specifying F CLUSTER SIZE of 3 or even 4 during disk INIT.  I believe it is a waste. of time to run disk defragmenter in this disk.    > The index-sequential files in NEWS_ROOT (specially NEWS.ITEMS)0 also live a dynamic life but in a different way.  E - If you put these index-seq files (by defining NEWS_ROOT) on another @   much less fragmented disk (make sure the initial copy makes itB   in one or very few pieces - use DUMP/HEAD/BLOCK=END:0 to check). and > - if you optimize FDL for NEWS.ITEMS and specify a large chunkE   for file allocation extensions (here is my FDL setup as an example: G   AREA 0 EXTENSION 13216; AREA 1 EXTENSION 248; AREA 2 EXTENSION 3900 )  and @ - if you perform CONVERT/RECLAIM on these index-sequential filesD   (the /RECLAIM qualifier is important!) regularly (say once a week)>   which reclaims space in the file occupied by deleted records)   and makes it available for new records,   7 than the disk fragmentation becomes a non-issue, since: I - you end up with one badly fragmented disk with news items (NEWS_DEVICE)    and you do not worry about it B - you end up with another disk holding NEWS_ROOT which gets almostC   no fragmentation from ANU-NEWS.  Having file NEWS.ITEMS not badly 7   fragmented brings a *significant* performance boost !   A I have been running our ANU-NEWS with this configuration of disks @ for more than a month now and I am very pleased with the result.@ I also made sure the *.BATCH files on NEWS_MANAGER (which should@ *not* be on the same disk as item files) get allocation in largeF chunks changing #define ALQ_NONDEF to 1 and changing #define OPEN_MODE7 to: #define OPEN_MODE "w","alq=256","deq=128","fop=cbt"  in NNTP_SERVER.C .    " Now back to the original question:  O > A nice option would be to allow us to explicitly specify the device/directory F > of a given newsgroup and/or hierarchy, so that we can split the diskI > consumption across multiple spindles without resorting to bound volumes   : If after moving the index-sequential files to another disk5 you still feel a need to split the news items hierchy < to more than one disk, you may copy part of a news hierarchy* to another disk (at least the directories)? and put a search list as a value of a logical name NEWS_DEVICE.    Here is an example:   / DISK1:[NEWS] contains the full hierarchy, e.g.: 4   ALT.DIR, COMP.DIR, CONTROL.DIR, GNU.DIR, JUNK.DIR,'   MISC.DIR, NEWS.DIR, REC.DIR, SCI.DIR, ?   SOC.DIR, TALK.DIR, VMSNET.DIR   (with all the subdirectories)   9 DISK2:[NEWS] contains only a part of the hierarchy, e.g.: 2   ALT.DIR, GNU.DIR   (with all the subdirectories)  7 Now you set up the NEWS_DEVICE logical name as follows: K   $ DEFINE/SYSTEM/TRANS=(conc)/EXEC NEWS_DEVICE disk2:[NEWS.],disk1:[NEWS.] - (NOTE: specify the partial hierarchy *first*)   ; This will tell RMS to look in the partial hierarchy *first* = and if the required directory/file is not there, than it will ; look into the next directory in the search list, eventually = finding the required file.  The newly created item files will < be created in the first directory in the list which contains the specified (sub)directory.   D You may remove ALT.DIR and GNU.DIR (in our example) later from DISK1< by hand or you may leave to the skim mechanism to eventually3 remove all old items from the original directories.   B All this is done by RMS alone and ANU-NEWS knows and cares nothing about the trick.  F Taking the idea a bit further, it also gives an answer to the questionF posted to this group a couple of weeks ago on how would it be possible? to store a particular (generally uninteresting) newsgroup under > a particular user's directory / disk quota.  The concept wouldA even make it possible to archive e.g. long news items with a long B expiration date on another disk and still leave them accessible in a usual manner.   ; Sorry for a rather long text but I hope this will help some < sites to forget the nightmare of CONVERTs (without /RECLAIM)A producing even more fragmented NEWS.INDEX than the original file, I unsuccesful disk defragmenters, poor performance (specially during SKIM), F running out of file headers in INDEXF.SYS without ANU-NEWS telling youH about the real cause of trouble and so on.  For me it works like charm !   --E   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! E   !!  Mark Martinec, System Manager            tel  +386 1 159-199 !! E   !!  J. Stefan Institute, Jamova 39           fax  +386 1 161-029 !! E   !!  61000 Ljubljana, Slovenia       e-mail: Mark.Martinec@ijs.si !! E   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 