[ From:	CRDGW2::CRDGW2::MRGATE::"SMTP::CRVAX.SRI.COM::RELAY-INFO-VAX" 30-OCT-1990 15:34:32.95  To:	MRGATE::"ARISIA::EVERHART" CC:	3 Subj:	Re: System disk performance on large clusters   * Received:  by crdgw1.ge.com (5.57/GE 1.76)) 	 id AA17214; Tue, 30 Oct 90 15:14:52 EST Y Received: From UCBVAX.BERKELEY.EDU by CRVAX.SRI.COM with TCP; Tue, 30 OCT 90 10:45:45 PDT , Received: by ucbvax.Berkeley.EDU (5.63/1.42)* 	id AA09052; Tue, 30 Oct 90 10:36:57 -08009 Received: from USENET by ucbvax.Berkeley.EDU with netnews . 	for info-vax@kl.sri.com (info-vax@kl.sri.com); 	(contact usenet@ucbvax.Berkeley.EDU if you have questions)  Date: 30 Oct 90 15:43:57 GMT\ From: usc!wuarchive!rex!uflorida!mlb.semi.harris.com!rtpark.rtp.semi.harris.com!rlb@ucsd.edu; Organization: Harris Semiconductor, Microelectronics Center 6 Subject: Re: System disk performance on large clusters= Message-Id: <1990Oct30.114357.216@rtpark.rtp.semi.harris.com>  References: <5746@mwk.uucp> # Sender: info-vax-request@kl.sri.com  To: info-vax@kl.sri.com   \ In article <5746@mwk.uucp>, gleason@mwk.uucp (Lee K. Gleason, Control-G Consultants) writes: > M >   I would like comments from other people managing large VMS CI clusters...  > ( >   I have a CI cluster consisting of... ..H >   We have,  of course, already moved everything off of the system diskH > that we can - SYSUAF,RIGHTSLIST, JBCSYSQUE, ACCOUNTNG, etc - it's downH > to just images and libraries, almost 100% read accesses. In like wise,A > the XQP caches are sized to take maximum advantage of caching.   > I >   We invite comment from any other large configuration sites that have  H > experienced IO bandwidth problems to the system disk - what have your C > experiences been? How did you attack the problem? What were your  
 > results? --  I You didn't mention if you had moved the operator log files, network event I log files, accounting data files, audit log files, default decnet account H to other disks.  These are all candidates.  We've done all of these hereE and I can point you to code we use to manage that.  The bulk of it is E available from acfcluster.nyu.edu as AUDIT_LOG_KIT via anonymous ftp   or MAILSERV.    9 We follow a procedure which creates a new accounting file D each midnight.  The files are named by node and date, which makes itI easy to track down old accounting info.  We are also running ARSAP, which J requires some trickery for managing this -- it does work just fine though.F If you're not running one of the accounting packages that does active E collection by placing itself between the JOB_CONTROL process and the  ? accounting data file, then you can do this with a DEFINE/SYSTEM H ACCOUNTNG <wherever your file will go> and then do a SET ACCOUNTING/NEW.G If you're interested in the procedure we use to do this, I will be glad  to make a copy available.   K You can also move the NET*.dat files from the system directory to somewhere H else by defining the logical names NETCIRC, NETCONF, NETLINE, NETLOGING,/ NETNODE_LOCAL, NETNODE_REMOTE, and NETOBJECT.     G There is another option that we have not resorted to since we have been G able to balance our I/O load short of doing this.  This option has been , used by some system managers at other sites.  E The basic idea is to create a collection of alternative locations for K some of the most often accessed executables, help libraries, message files, H and anything else that isn't just totally bolted down as having to be on the system disk.   The basic approach is this:   F Create alternate directories on other, more moderately loaded disk(s).E Copy the most heavily used images/files -- these should be files that J do not change except when updating software -- to these other directories.  K Modify the search list(s) for SYS$SYSTEM, SYS$SHARE, SYS$LIBRARY, SYS$HELP, F SYS$MESSAGE, SYS$MANAGER, etc... so that the new definition includes a< reference to the "copy" directory before the original value.  F It is incredibly important in any large configuration of computers to D document a comprehensive software update checklist/procedure that isD available for hardcopy.  During any software update a fresh copy is F printed and each step is checked off before proceeding with the actualI update.  I've developed a checklist like this in my head, but have seldom I written it down.  Unfortunately, sometimes this has bitten me in the rear . because I would forget 1 or 2 little things.    N If someone should decide they want to try out this idea, there are some things to watch when doing this:   K 1.  Software updates must only be done with the original logicals in place. H     It is probably best that during any updates that the logicals be putH     back to their original values on all members of the cluster to avoidG     problems.  I would suggest creating a command procedure which has 2 H     modes of operation.  The 1st mode pre-pends the copy directory specsF     to the appropriate system logical names translation value, turningJ     them into search lists.  The 2nd mode removes the extra value from theG     front of the search lists, returning them to their original values.   J     I have a command procedure which does these 2 operations for a generalF     logical name search list which I'm appending to this posting.  TheH     comments explain how to use it.  If you have any questions about it,,     please send me a message/give me a call.     L 2.  After doing any software updates, fresh copies of all the relevant filesJ     should be placed in the copy directories.  Because of this, it is bestL     to manage this by creating a command procedure and a data file to manage*     what gets copied and where it goes to.  F 3.  It might be a good idea to delete the copies before initiating anyM     updates so that users on other nodes in the cluster will not accidentally !     get an old version of a file.   L 4.  Document any and everything you do in your site procedures documentationG     so that in the event of catastrophe to you and/or your other system H     support staff, there will be a record of what you decided and how itF     works:  which command procedures do what and when/how to use them.  D 5.  If images are installed before these logical names are modified J     in the system startup, you should include code to REINSTALL the imagesG     so that the originals on the system disk are replaced by the copies G     as the KNOWN images.  I suggest that the code for copying the files K     also include a section that does the REINSTALL as an optional function. E     That way you can include the necessary invocation in your startup 9     procedures without having to add a lot of code there.    E NOTE:  I doubt very seriously that DEC will condone this idea.  It is J        important to be very careful when/if you try this.  I would suggestG        that if you have a workstation or some other form of test system G        that you try this out where it can't hurt too badly if something D        goes a little haywire with the trial.  Unless you like to fixL        things in a panic, I wouldn't try this with your users on the system.E        I would suggest trying this out 1 file at a time until you are G        confident it's working ok before doing a mass transfer of stuff.   - Here's the LNMINSREM.COM in VMS_SHARE format:   6 $! ------------------ CUT HERE -----------------------( $ v='f$verify(f$trnlnm("SHARE_VERIFY"))' $!A $! This archive created by VMS_SHARE Version 7.2-007  22-FEB-1990 . $!   On 30-OCT-1990 11:37:58.97   By user RLB  $! $! This VMS_SHARE Written by: * $!    Andy Harper, Kings College London UK $! $! Acknowledgements to: + $!    James Gray       - Original VMS_SHARE < $!    Michael Bednarek - Original Concept and implementation $!< $! TO UNPACK THIS SHARE FILE, CONCATENATE ALL PARTS IN ORDER2 $! AND EXECUTE AS A COMMAND PROCEDURE  (  @name  ) $!9 $! THE FOLLOWING FILE(S) WILL BE CREATED AFTER UNPACKING:  $!       1. LNMINSREM.COM;16 $!
 $set="set"$ $set symbol/scope=(nolocal,noglobal)? $f=f$parse("SHARE_TEMP","SYS$SCRATCH:.TMP_"+f$getjpi("","PID")) # $e="write sys$error  ""%UNPACK"", " # $w="write sys$output ""%UNPACK"", " ) $ if f$trnlnm("SHARE_LOG") then $ w = "!"V $ ve=f$getsyi("version")7 $ if ve-f$extract(0,1,ve) .ges. "4.4" then $ goto START"* $ e "-E-OLDVER, Must run at least VMS 4.4" $ v=f$verify(v) 	 $ exit 44 . $UNPACK: SUBROUTINE ! P1=filename, P2=checksum2 $ if f$search(P1) .eqs. "" then $ goto file_absent, $ e "-W-EXISTS, File ''P1' exists. Skipped."
 $ delete 'f'*4 $ exit
 $file_absent:b+ $ if f$parse(P1) .nes. "" then $ goto diroke $ dn=f$parse(P1,,,"DIRECTORY")* $ w "-I-CREDIR, Creating directory ''dn'." $ create/dir 'dn'f $ if $status then $ goto dirok: $ e "-E-CREDIRFAIL, Unable to create ''dn'. File skipped."
 $ delete 'f'*  $ exit $dirok:7( $ w "-I-PROCESS, Processing file ''P1'."7 $ if .not. f$verify() then $ define/user sys$output nl: 1 $ EDIT/TPU/NOSEC/NODIS/COM=SYS$INPUT 'f'/OUT='P1'sH PROCEDURE Unpacker ON_ERROR ENDON_ERROR;SET(FACILITY_NAME,"UNPACK");SET(M SUCCESS,OFF);SET(INFORMATIONAL,OFF);f:=GET_INFO(COMMAND_LINE,"file_name");b:=uE CREATE_BUFFER(f,f);p:=SPAN(" ")@r&LINE_END;POSITION(BEGINNING_OF(b)); F LOOP EXITIF SEARCH(p,FORWARD)=0;POSITION(r);ERASE(r);ENDLOOP;POSITION(M BEGINNING_OF(b));g:=0;LOOP EXITIF MARK(NONE)=END_OF(b);x:=ERASE_CHARACTER(1);lK IF g=0 THEN IF x="X" THEN MOVE_VERTICAL(1);ENDIF;IF x="V" THEN APPEND_LINE; K MOVE_HORIZONTAL(-CURRENT_OFFSET);MOVE_VERTICAL(1);ENDIF;IF x="+" THEN g:=1;tM ERASE_LINE;ENDIF;ELSE IF x="-" THEN IF INDEX(CURRENT_LINE,"+-+-+-+-+-+-+-+")=mG 1 THEN g:=0;ENDIF;ENDIF;ERASE_LINE;ENDIF;ENDLOOP;t:="0123456789ABCDEF";aM POSITION(BEGINNING_OF(b));LOOP r:=SEARCH("`",FORWARD);EXITIF r=0;POSITION(r);nM ERASE(r);x1:=INDEX(t,ERASE_CHARACTER(1))-1;x2:=INDEX(t,ERASE_CHARACTER(1))-1;dF COPY_TEXT(ASCII(16*x1+x2));ENDLOOP;WRITE_FILE(b,GET_INFO(COMMAND_LINE,+ "output_file"));ENDPROCEDURE;Unpacker;QUIT;  $ delete/nolog 'f'*o $ CHECKSUM 'P1'i+ $ IF CHECKSUM$CHECKSUM .eqs. P2 THEN $ EXITr- $ e "-E-CHKSMFAIL, Checksum of ''P1' failed."i $ ENDSUBROUTINEu $START:  $ create 'f'* X$ vfl = f$VER(0.or.f$TRNLNM("debug$dcl")) X$! modify logical name list# X$! p1 -- function INSERT or REMOVEa+ X$! p2 -- equivalence name to insert/removea/ X$! p3 -- Logical name to affect ( no default )f+ X$! p4 -- table ( defaults to LNM$PROCESS ) C X$! p5 -- if function = REMOVE logical value determines if the lasta/ X$!       member of list may be removed or not.cF X$!       if function = INSERT determines relative position for INSERT) X$! p6 -- mode ( defaults to SUPERVISOR )  X$! p7 -- other qualifiers, X$!-----------------------------------------& X$! author:  R.L. Boyd, September 1985, X$!-----------------------------------------< X$!  insert a new name at the head or remove from the middle( X$   if p3.eqs."" then $ goto ERR_NONAME& X$   if p1.eqs."" then $ p1 = "INSERT"+ X$   if p4.eqs."" then $ p4 = "LNM$PROCESS"uE X$   if p1.eqs."REMOVE" .and. f$TRNLNM(p3,p4).eqs."" then $ goto EXITl* X$   if p6.eqs."" then $ p6 = "SUPERVISOR" X$ X$   mode = "/"+p6 X$! initialize X$   com = "," X$   num = "#" X$   null = ""`20G X$   val_list = null X$   in_list = 0 X$   val_eq = null X$   on error then $ goto EXIT' X$! see if it exists and how long it isb( X$   exists = f$TRNLNM(p3,p4,,p6).nes.""1 X$   max_index = f$TRNLNM(p3,p4,,p6,,"max_index")a' X$   if .not.exists then $ goto COMMANDi% X$! let's get the existing list first  X$   name_attributes == "". X$   if f$TRNLNM(p3,p4,,p6,,"no_alias") then -+ X$`09call ADD_LIST name_attributes no_aliast- X$   if f$TRNLNM(p3,p4,,p6,,"confine") then -h* X$`09call ADD_LIST name_attributes confine% X$   if name_attributes.nes."" then -t> X$`09name_attributes = "/name_attribute=("+name_attributes+")" X$   cnt = 0
 X$VAL_GET:& X$   next_val = f$TRNLNM(p3,p4,cnt,p6) X$   add_val = next_val-! X$   translation_attributes == ""i2 X$   if f$TRNLNM(p3,p4,cnt,p6,,"concealed") then -3 X$`09call ADD_LIST translation_attributes concealed.1 X$   if f$TRNLNM(p3,p4,cnt,p6,,"terminal") then -o2 X$`09call ADD_LIST translation_attributes terminal, X$   if translation_attributes.nes."" then -J X$`09add_val=add_val+"/translation_attribute=("+translation_attributes+")"- X$   if cnt.gt.0 then $ add_val = num+add_valr< X$   if next_val.nes.null then $ val_list = val_list+add_val X$   if next_val.eqs.p2`20 X$   then`20 X$     val_eq = cntv X$     val_match = add_val-num X$     in_list = 1`20s
 X$   endif X$   cnt = cnt+1, X$   if cnt.le.max_index then $ goto VAL_GET
 X$COMMAND: X$   goto 'p1'	 X$REMOVE:i% X$   if .not.in_list then $ goto EXITi5 X$   if val_eq.lt.max_index then $ goto REMOVE_MIDDLEs! X$   if .not. p5 then $ goto EXIT  X$   remove_string = val_match? X$   if max_index.gt.0 then $ remove_string = num+remove_string ( X$   val_list = val_list - remove_string X$   goto REMOVE_DONEi X$REMOVE_MIDDLE:) X$   val_list = val_list -(val_match+num)  X$REMOVE_DONE:? X$   if val_list.eqs.null then $ deassign/table='p4''mode' 'p3'f X$   gosub FINAL_DELIMITERS-  X$   if val_list.nes.null then -= X$`09define/table='p4''mode''name_attributes' 'p3' 'val_list'r X$   goto EXIT	 X$INSERT:l! X$   if p5.eqs.null then $ p5 = 0r/ X$   if p5.gt.max_index then $ p5 = max_index+1r: X$   if in_list .and. val_eq.le.p5 then $ goto INSERT_DONE* X$   if .not.in_list then $ goto INSERT_GOD X$! it's in the list but too early -- we'll have to delete it first." X$  val_list = val_list - (p2+num) X$INSERT_GO:C X$! now we have to figure out how to stick it in at the right placee* X$! 3 different ways -- front, end, middle X$! set up for default X$  pre_list = nulle X$  post_list = val_listD X$  if p5.eq.0 then $ goto INSERT_DO  ! see if it is front (default)0 X$  if p5.gt.max_index then $ goto INSERT_END`20H X$! yuk -- have to put it in the middle -- now we have to split the list X$  pre_cnt = 0e X$PRE_LOOP:h. X$  next_val = f$ELEMENT(pre_cnt,num,val_list) X$  pre_cnt = pre_cnt+1i3 X$  if pre_cnt.lt.p5 then $ next_val = next_val+numv  X$  pre_list = pre_list+next_val) X$  if pre_cnt.lt.p5 then $ goto PRE_LOOPc3 X$! we now have all the first p5 values in pre_listaC X$  post_list = val_list - (pre_list+num) ! lop off the front of itl X$  goto INSERT_DO X$INSERT_END:`20 X$  pre_list = val_list  X$  post_list = null X$INSERT_DO: X$  val_ins = p2+p7.5 X$  if pre_list.nes.null then $ val_ins = num+val_inse6 X$  if post_list.nes.null then $ val_ins = val_ins+num' X$! now we stick the list back togethera) X$  val_list = pre_list+val_ins+post_lista X$   gosub FINAL_DELIMITERSe< X$  define/table='p4''mode''name_attributes' 'p3' 'val_list' X$INSERT_DONE: X$EXIT:i* X$  if vfl then $ show log 'p3'/table='p4' X$  exit ! 'f$VER(vfl)'u
 X$ERR_NONAME:e X$  ss$_argreq = "%X00038268"tD X$  write sys$error f$MESSAGE(ss$_argreq)," -- P3 (= logical name) " X$  exit, X$! replace # signs with ,'s in the VAL_LIST X$FINAL_DELIMITERS: 0 X$ if vfl.or.f$VER() then $ show symbol /loc/all X$ out_list == ""t X$ pre_cnt = 0
 X$FINAL_LOOP:k. X$  next_val = f$ELEMENT(pre_cnt,num,val_list) X$  if next_val.nes.num`20 X$  then X$ `09pre_cnt = pre_cnt+1n- X$       call ADD_LIST out_list "''next_val'". X$`09goto FINAL_LOOP X$  endif`20 X$  val_list = out_listt1 X$  if vfl.or.f$VER() then $ show symbol val_listw	 X$ returnt X$!nA X$! ADD_LIST:  add a value to a list contained in a global symbolo- X$! p1 global symbol name containing the list  X$! p2 value to add to the listMC X$! p3 delimiter to use to separate values in the list, default ","- X$ADD_LIST: subroutine
 X$  list = p1I X$  value = p2 X$  delimiter = p3. X$  if delimiter.eqs."" then $ delimiter = "," X$  if f$TYPE('list').eqs.""`20u X$  then`20  X$ `09'list' == valuee X$  else X$`09if 'list'.eqs.""`20 X$`09then`20 X$`09`09'list' == value 	 X$`09else ( X$`09`09'list' == 'list'+delimiter+value
 X$`09endif	 X$  endifO X$  exit X$endsubroutine ! add_list6 X$!Last Modified:  22-MAR-1990 13:04:18.66, By: RLB`20( $ CALL UNPACK LNMINSREM.COM;16 254021580 $ v=f$verify(v)  $ EXIT  
           Bob(A ----------------------------------------------------------------- )  Bob Boyd   Voice:     (919)549-3627     n-  Harris Semiconductor Microelectronics Center.  _4  E-Mail Address:      rlb@rtpark.rtp.semi.harris.com  '  Disclaimer?  Datclaimer?  Reclaimer?     