From:	CRDGW2::CRDGW2::MRGATE::"SMTP::CRVAX.SRI.COM::RELAY-INFO-VAX" 12-JUN-1991 00:19:53.02
To:	ARISIA::EVERHART
CC:	
Subj:	Batch queue monitor/control

From:	RELAY-INFO-VAX@CRVAX.SRI.COM@SMTP@CRDGW2
To:	Everhart@Arisia@MRGATE

Received:  by crdgw1.ge.com (5.57/GE 1.99)
	 id AA22558; Tue, 11 Jun 91 07:32:52 EDT
Received: From ucbvax.Berkeley.EDU ([128.32.133.1]) by CRVAX.SRI.COM with TCP; Tue, 11 JUN 91 04:21:29 PDT
Received: by ucbvax.Berkeley.EDU (5.63/1.42)
	id AA25672; Tue, 11 Jun 91 04:14:14 -0700
Received: from USENET by ucbvax.Berkeley.EDU with netnews
	for info-vax@kl.sri.com (info-vax@kl.sri.com)
	(contact usenet@ucbvax.Berkeley.EDU if you have questions)
Date: 10 Jun 91 15:24:55 GMT
From: cis.ohio-state.edu!zaphod.mps.ohio-state.edu!mips!spool.mu.edu!cs.umn.edu!sialis!orbit!incstar!lhotka@ucbvax.Berkeley.EDU
Organization: INCSTAR Corp, Stillwater MN
Subject: Batch queue monitor/control
Message-Id: <2658.285399c7@incstar.com>
Sender: info-vax-request@kl.sri.com
To: info-vax@kl.sri.com

Following is a VMS_SHARE archive containing the routines we use at 
INCSTAR Corp to achieve a finer degree of control over our user batch
queue than would normally be available under VMS.  While I am certain
there are other (better) approaches to solve this problem, this 
solution took little time to create and solved many of our problems.

As it is just a DCL routine, it should be fairly easy to modify
it to fit into many different situations.

 Rockford Lhotka                                INCSTAR Corp
 Applications Project Leader                    1990 Industrial Blvd
 lhotka@incstar.com                             PO Box 285
 612/779-1701                                   Stillwater, MN 55082


$! ------------------ CUT HERE -----------------------
$ v='f$verify(f$trnlnm("SHARE_VERIFY"))'
$!
$! This archive created by VMS_SHARE Version 7.2-007  22-FEB-1990
$!   On 10-JUN-1991 15:18:46.28   By user LHOTKA 
$!
$! This VMS_SHARE Written by:
$!    Andy Harper, Kings College London UK
$!
$! Acknowledgements to:
$!    James Gray       - Original VMS_SHARE
$!    Michael Bednarek - Original Concept and implementation
$!
$! TO UNPACK THIS SHARE FILE, CONCATENATE ALL PARTS IN ORDER
$! AND EXECUTE AS A COMMAND PROCEDURE  (  @name  )
$!
$! THE FOLLOWING FILE(S) WILL BE CREATED AFTER UNPACKING:
$!       1. [.QMON]QMON_CHECK.COM;1
$!       2. [.QMON]QMON_MONITOR.COM;1
$!       3. [.QMON]QMON_PROCESS.COM;1
$!       4. [.QMON]README.;1
$!
$set="set"
$set symbol/scope=(nolocal,noglobal)
$f=f$parse("SHARE_TEMP","SYS$SCRATCH:.TMP_"+f$getjpi("","PID"))
$e="write sys$error  ""%UNPACK"", "
$w="write sys$output ""%UNPACK"", "
$ if f$trnlnm("SHARE_LOG") then $ w = "!"
$ ve=f$getsyi("version")
$ if ve-f$extract(0,1,ve) .ges. "4.4" then $ goto START
$ e "-E-OLDVER, Must run at least VMS 4.4"
$ v=f$verify(v)
$ exit 44
$UNPACK: SUBROUTINE ! P1=filename, P2=checksum
$ if f$search(P1) .eqs. "" then $ goto file_absent
$ e "-W-EXISTS, File ''P1' exists. Skipped."
$ delete 'f'*
$ exit
$file_absent:
$ if f$parse(P1) .nes. "" then $ goto dirok
$ dn=f$parse(P1,,,"DIRECTORY")
$ w "-I-CREDIR, Creating directory ''dn'."
$ create/dir 'dn'
$ if $status then $ goto dirok
$ e "-E-CREDIRFAIL, Unable to create ''dn'. File skipped."
$ delete 'f'*
$ exit
$dirok:
$ w "-I-PROCESS, Processing file ''P1'."
$ if .not. f$verify() then $ define/user sys$output nl:
$ EDIT/TPU/NOSEC/NODIS/COM=SYS$INPUT 'f'/OUT='P1'
PROCEDURE Unpacker ON_ERROR ENDON_ERROR;SET(FACILITY_NAME,"UNPACK");SET(
SUCCESS,OFF);SET(INFORMATIONAL,OFF);f:=GET_INFO(COMMAND_LINE,"file_name");b:=
CREATE_BUFFER(f,f);p:=SPAN(" ")@r&LINE_END;POSITION(BEGINNING_OF(b));
LOOP EXITIF SEARCH(p,FORWARD)=0;POSITION(r);ERASE(r);ENDLOOP;POSITION(
BEGINNING_OF(b));g:=0;LOOP EXITIF MARK(NONE)=END_OF(b);x:=ERASE_CHARACTER(1);
IF g=0 THEN IF x="X" THEN MOVE_VERTICAL(1);ENDIF;IF x="V" THEN APPEND_LINE;
MOVE_HORIZONTAL(-CURRENT_OFFSET);MOVE_VERTICAL(1);ENDIF;IF x="+" THEN g:=1;
ERASE_LINE;ENDIF;ELSE IF x="-" THEN IF INDEX(CURRENT_LINE,"+-+-+-+-+-+-+-+")=
1 THEN g:=0;ENDIF;ENDIF;ERASE_LINE;ENDIF;ENDLOOP;t:="0123456789ABCDEF";
POSITION(BEGINNING_OF(b));LOOP r:=SEARCH("`",FORWARD);EXITIF r=0;POSITION(r);
ERASE(r);x1:=INDEX(t,ERASE_CHARACTER(1))-1;x2:=INDEX(t,ERASE_CHARACTER(1))-1;
COPY_TEXT(ASCII(16*x1+x2));ENDLOOP;WRITE_FILE(b,GET_INFO(COMMAND_LINE,
"output_file"));ENDPROCEDURE;Unpacker;QUIT;
$ delete/nolog 'f'*
$ CHECKSUM 'P1'
$ IF CHECKSUM$CHECKSUM .eqs. P2 THEN $ EXIT
$ e "-E-CHKSMFAIL, Checksum of ''P1' failed."
$ ENDSUBROUTINE
$START:
$ create 'f'
X$ Set Noverify
X$!+++
X$!`09This routine runs every morning and checks the
X$!`09night processing batch queues for jobs which are
X$!`09retained.  The list of retained jobs is mailed
X$!`09to the system administrator for further action.
X$!
X$!`09 5JUN91 RDL  Modify so it doesn't run over the weekend
X$!`0922APR91 RDL  Created
X$!---
X$ On Severe_Error then Continue
X$ Stop/que/next MAN$BATCH
X$ Set Proc/name="Que Check"
X$ Open/write F1 SYS$SCRATCH:QMON_CHECK_TMP.COM
X$  Write F1 "$ mail"
X$  Write F1 "send"
X$  Write F1 "lhotka"
X$  Write F1 "Daily queue report "+F$CVTIME("TODAY","ABSOLUTE","DATE")
X$!
X$RESTART:
X$  TEMP = F$GETQUI("")
X$!
X$QLOOP:
X$  QNAME = F$GETQUI("DISPLAY_QUEUE","QUEUE_NAME","*$BATCH")
X$  If QNAME.eqs."MAN$BATCH" then Goto PROC_QUEUE
X$  If QNAME.eqs."MFG$BATCH" then Goto PROC_QUEUE
X$  If QNAME.eqs."OMR$BATCH" then Goto PROC_QUEUE
X$  If QNAME.eqs."" then Goto SUBMIT
X$ Goto QLOOP
X$!
X$PROC_QUEUE:
X$  NOW = F$CVTIME("","ABSOLUTE","TIME")
X$  JOB_COUNT = 0
X$  JOB_ACTIVE = 0
X$  JOB_RETAINED = 0
X$  QJOBS = F$GETQUI("DISPLAY_QUEUE","JOB_LIMIT","''QNAME'*","FREEZE_CONTEXT"
V)
X$  Write F1 "Summary information for queue ''QNAME'"
X$  Write F1 ""
X$  Write F1 "Jobs retained:"
X$!
X$JLOOP:
X$  NOACCESS = F$GETQUI("DISPLAY_JOB","JOB_INACCESSIBLE",,"ALL_JOBS")
X$  IF NOACCESS .EQS. "TRUE" THEN GOTO JLOOP
X$  IF NOACCESS .EQS. "" THEN GOTO ENDRUN
X$  USERNAME = F$GETQUI("DISPLAY_JOB","USERNAME",,"FREEZE_CONTEXT,ALL_JOBS")
X$  JOBNAME = F$GETQUI("DISPLAY_JOB","JOB_NAME",,"FREEZE_CONTEXT,ALL_JOBS")
X$  ENTRY = F$GETQUI("DISPLAY_JOB","ENTRY_NUMBER",,"FREEZE_CONTEXT,ALL_JOBS")
X$  STATUS = F$GETQUI("DISPLAY_JOB","JOB_STATUS",,"FREEZE_CONTEXT,ALL_JOBS")
X$  If STATUS.eq.192  then Write F1 "`09User ''USERNAME' job ''JOBNAME' (entr
Vy ''ENTRY')"
X$  If STATUS.eq.192 then JOB_RETAINED = JOB_RETAINED + 1
X$  If STATUS.eq.2 then JOB_ACTIVE = JOB_ACTIVE + 1
X$  If STATUS.eq.2048 then JOB_COUNT = JOB_COUNT + 1
X$ GOTO JLOOP
X$!
X$ENDRUN:
X$  If JOB_RETAINED.eq.0 then Write F1 "`09<none>"
X$  Write F1 "Number of jobs pending: ''JOB_COUNT'"
X$  Write F1 ""
X$ Goto QLOOP
X$!
X$SUBMIT:
X$ Close F1
X$ @SYS$SCRATCH:QMON_CHECK_TMP
X$ If F$CVTIME("TODAY","ABSOLUTE","WEEKDAY").eqs."Friday" then TOM = F$CVTIME
V("TODAY+3-00:00","ABSOLUTE","DATE")+":07:30"
X$ If F$CVTIME("TODAY","ABSOLUTE","WEEKDAY").nes."Friday" then TOM = F$CVTIME
V("TOMORROW","ABSOLUTE","DATE")+":07:30"
X$ Submit/que=TIM$BATCH/AFTER='TOM'/noprint QMON_CHECK
X$ Dele/nolog SYS$SCRATCH:QMON_CHECK_TMP.COM;
X$ Exit
$ CALL UNPACK [.QMON]QMON_CHECK.COM;1 725693933
$ create 'f'
X$ Set Noverify
X$!+++
X$!`09This routine runs in TIM$BATCH every day throughout the day.  It
X$!`09keeps MAN$BATCH in a stopped or stop pending state.  Each minute
X$!`09it wakes up and scans MAN$BATCH looking for pending jobs.
X$!
X$!`09Jobs which are found to be pending are passed to QMON_PROCESS
X$!`09where they may either have their priorities adjusted or may be
X$!`09put on hold to run at night, etc.
X$!
X$!`0919APR91 RDL  Set queue/job=3 if # of pending jobs > 8 (VAX 6410 after a
Vll)
X$!`0919APR91 RDL  Automatically remove retained APR415's
X$!`09 4DEC90 RDL  Set queue/job=2 if # of pending jobs > 8
X$!`0919NOV90 RDL  Created
X$!---
X$ On Severe_Error then Continue
X$ Stop/que/next MAN$BATCH
X$ Set Proc/name="Que Monitor"
X$!
X$RESTART:
X$  TEMP = F$GETQUI("")
X$!
X$QLOOP:
X$  QNAME = F$GETQUI("DISPLAY_QUEUE","QUEUE_NAME","MAN$BATCH*")
X$  If QNAME.eqs."MAN$BATCH" then Goto PROC_MAN
X$  If QNAME.eqs."" then Goto ENDRUN
X$ Goto QLOOP
X$!
X$PROC_MAN:
X$  NOW = F$CVTIME("","ABSOLUTE","TIME")
X$  JOB_COUNT = 0
X$  JOB_ACTIVE = 0
X$  QJOBS = F$GETQUI("DISPLAY_QUEUE","JOB_LIMIT","MAN$BATCH*","FREEZE_CONTEXT
V")
X$  If NOW.gts."07:00" .and. NOW.lts."07:05" .and. QJOBS.lt.2 then Set Que/jo
Vb=2 MAN$BATCH
X$  If NOW.gts."16:00" .and. NOW.lts."16:05" .and. QJOBS.ge.2 then Set Que/jo
Vb=1 MAN$BATCH
X$!
X$JLOOP:
X$  NOACCESS = F$GETQUI("DISPLAY_JOB","JOB_INACCESSIBLE",,"ALL_JOBS")
X$  IF NOACCESS .EQS. "TRUE" THEN GOTO JLOOP
X$  IF NOACCESS .EQS. "" THEN GOTO ENDRUN
X$  USERNAME = F$GETQUI("DISPLAY_JOB","USERNAME",,"FREEZE_CONTEXT,ALL_JOBS")
X$  JOBNAME = F$GETQUI("DISPLAY_JOB","JOB_NAME",,"FREEZE_CONTEXT,ALL_JOBS")
X$  ENTRY = F$GETQUI("DISPLAY_JOB","ENTRY_NUMBER",,"FREEZE_CONTEXT,ALL_JOBS")
X$  STATUS = F$GETQUI("DISPLAY_JOB","JOB_STATUS",,"FREEZE_CONTEXT,ALL_JOBS")
X$  If STATUS.eq.2048 then @QMON_PROCESS 'USERNAME' 'JOBNAME' 'ENTRY' "PEND"
X$  If STATUS.eq.192  then @QMON_PROCESS 'USERNAME' 'JOBNAME' 'ENTRY' "RET"
X$  If STATUS.eq.2 then JOB_ACTIVE = JOB_ACTIVE + 1
X$  If STATUS.eq.2048 then JOB_COUNT = JOB_COUNT + 1
X$ GOTO JLOOP
X$!
X$ENDRUN:
X$ If JOB_COUNT.eq.0 then Goto WAIT
X$ If JOB_COUNT.gt.12 .and. QJOBS.le.2 .and. NOW.gts."08:00" then Set Queue/j
Vob=3 MAN$BATCH
X$ If JOB_COUNT.gt.12 .and. QJOBS.le.2 .and. NOW.gts."08:00" then QJOBS = 3
X$ If JOB_COUNT.gt.8 .and. QJOBS.le.1 .and. NOW.gts."08:00" then Set Queue/jo
Vb=2 MAN$BATCH
X$ If JOB_COUNT.gt.8 .and. QJOBS.le.1 .and. NOW.gts."08:00" then QJOBS = 2
X$!
X$ If JOB_COUNT.lt.8 .and. QJOBS.ge.3 then Set Queue/job=2 MAN$BATCH
X$ If JOB_COUNT.lt.8 .and. QJOBS.ge.3 then QJOBS = 2
X$ If JOB_COUNT.lt.5 .and. QJOBS.ge.2 then Set Queue/job=1 MAN$BATCH
X$ If JOB_COUNT.lt.5 .and. QJOBS.ge.2 then QJOBS = 1
X$!
X$ If QJOBS.le.JOB_ACTIVE then Goto WAIT
X$ Star/que MAN$BATCH
X$ Stop/que/next MAN$BATCH
X$ JOB_COUNT = JOB_COUNT - 1
X$ JOB_ACTIVE = JOB_ACTIVE + 1
X$ If JOB_COUNT.gt.0 .and. JOB_ACTIVE.lt.QJOBS then Goto RESTART
X$!
X$WAIT:
X$  If NOW.gts."17:00" then Goto SUBMIT
X$  WAIT 00:02:00
X$  GOTO RESTART
X$!
X$SUBMIT:
X$ If F$CVTIME("","ABSOLUTE","WEEKDAY").eqs."Friday" then TOM = F$CVTIME("TOM
VORROW","ABSOLUTE","DATE")+":12:00"
X$ If F$CVTIME("","ABSOLUTE","WEEKDAY").nes."Friday" then TOM = F$CVTIME("TOM
VORROW","ABSOLUTE","DATE")+":06:00"
X$ Submit/que=TIM$BATCH/AFTER='TOM'/noprint QMON_MONITOR
X$ Set Que/job=1 MAN$BATCH
X$ Exit
$ CALL UNPACK [.QMON]QMON_MONITOR.COM;1 925457181
$ create 'f'
X$!+++
X$!`09This routine is passed batch job information and it may
X$!`09modify that job's priority or put it on hold to run later.
X$!
X$!`09This routine is called by QMON_MONITOR.  The reason this
X$!`09code is in a seperate file is so it can be modified while
X$!`09QMON_MONITOR is running without restarting the monitor,
X$!`09allowing dynamic control of jobs throughout the day.
X$!
X$!`09P1 = USERNAME
X$!`09P2 = JOBNAME
X$!`09P3 = ENTRY NUMBER
X$!`09P4 = ACTION MODE (PEND = Job pending,
X$!`09`09`09  RET  = Job retained)
X$!---
X$ If P4.eqs."RET" then Goto RETAINED
X$ If F$CVTIME("","ABSOLUTE","WEEKDAY").eqs."Saturday" .or. F$CVTIME("","ABSO
VLUTE","WEEKDAY").eqs."Sunday" then Exit
X$ If F$CVTIME("","ABSOLUTE","TIME").ges."16:55" then Goto END_DAY
X$!
X$ If P2.eqs."MAU111" then Goto HIGH_PRIORITY
X$ If P2.eqs."GLR124" then Goto HIGH_PRIORITY
X$ If P2.eqs."GLR131" then Goto HIGH_PRIORITY
X$ IF P2.eqs."APR620" then Goto HIGH_PRIORITY
X$ If P2.eqs."MGR149" then Goto HIGH_PRIORITY
X$!
X$ If P2.eqs."APR640" then Goto SHORT_JOB
X$ If P2.eqs."MGR830" then Goto SHORT_JOB
X$ If P2.eqs."MGR831" then Goto SHORT_JOB
X$!
X$ If P2.eqs."MGR150" then Goto LONG_JOB
X$ If P2.eqs."MGR151" then Goto LONG_JOB
X$!
X$ If P2.eqs."MGR330" then Goto NIGHT_ONLY
X$ If P2.eqs."MGR331" then Goto NIGHT_ONLY
X$ If P2.eqs."MGR350" then Goto NIGHT_ONLY
X$ Exit
X$!
X$SHORT_JOB:
X$ Set Entry/prio=11 'P3'
X$ Exit
X$!
X$HIGH_PRIORITY:
X$ Set Entry/prio=12 'P3'
X$ Exit
X$!
X$LONG_JOB:
X$ If F$CVTIME("","ABSOLUTE","TIME").ges."14:00" then Goto NIGHT_ONLY
X$ Set Entry/prio=9 'P3'
X$ Exit
X$!
X$LOW_PRIORITY:
X$ If F$CVTIME("","ABSOLUTE","TIME").ges."14:30" then Goto NIGHT_ONLY
X$ Set Entry/prio=8 'P3'
X$ Exit
X$!
X$NIGHT_ONLY:
X$ Set Entry/after=TOM 'P3'
X$ Exit
X$!
X$HOLD_JOB:
X$ Set Entry/hold 'P3'
X$ Open/write F1 SYS$SCRATCH:QMON_HOLD_TMP.COM
X$ Write F1 "$ mail"
X$ Write F1 "send"
X$ Write F1 "lhotka"
X$ Write F1 "ADAM::MAN$BATCH (Bad batch job)"
X$ Write F1 "Message from ADAM::MAN$BATCH"
X$ Write F1 ""
X$ Write F1 "Job ''P2' (entry ''P3') user ''P1'"
X$ Write F1 "This job has been held until approved for processing."
X$ Close F1
X$ @SYS$SCRATCH:QMON_HOLD_TMP.COM
X$ Dele/nolog/noconf SYS$SCRATCH:QMON_HOLD_TMP.COM;
X$ Exit
X$!
X$END_DAY:
X$ Set Entry/after=TOM 'P3'
X$ Exit
X$!
X$RETAINED:
X$ If P2.eqs."APR415" then Del/ent='P3'
X$ If P2.eqs."APR570" then Del/ent='P3'
X$ Exit
$ CALL UNPACK [.QMON]QMON_PROCESS.COM;1 527577895
$ create 'f'
XThe files in this archive were created in response to a need we had at
XINCSTAR Corp to achieve a finer degree of control over the batch jobs which
Xwere being run by users.  Due to the nature of the database system we are
Xusing, various reports and utilities can cause many of the users to become
Xlocked up.  In addition, some jobs are lengthy and tend to get in the way
Xof short or high priority jobs.
X
XThese routines raise the priority of fast or important jobs, lower the
Xpriority of longer or less important jobs and actually prohibit the running
Xof jobs which lock up users during the daytime.
X
XThis archive contains four files:
X
X`09README`09`09- This file
X
X`09QMON_CHECK.COM`09- A utility to report jobs which
X`09`09`09  have been retained on error on
X`09`09`09  a daily basis
X
X`09QMON_MONITOR.COM- The queue monitor routine which
X`09`09`09  runs continually in a special
X`09`09`09  batch queue throughout the day
X
X`09QMON_PROCESS.COM- The routine called by QMON_MONITOR
X`09`09`09  which screens all pending jobs before
X`09`09`09  allowing them to run
X
XI run QMON_CHECK at 7:30 each morning so when I come in to work I have a
Xnice report in mail which tells me if any jobs failed during the night and
Xif there are any jobs pending at that point.  While I could just look at
Xthe queues to find this information, I find it to be more convenient to get
Xmail.
X
XQMON_MONITOR runs in a separate batch queue (TIM$BATCH) from the user batch
Xqueue.  In our case, TIM$BATCH is the queue where any 'timed' jobs or
Xregularly scheduled control jobs, etc are run.  This routine runs from 7:00
Xin the morning until 5:00 in the afternoon.  It leaves the user batch queue
Xin a stopped state at the end of the day so the system is pretty much
Xguaranteed to be free for backups.
X
XQMON_PROCESS is only called by QMON_MONITOR - it is not to be run directly
Xor anything like that.  It contains the code to change job priorities and
Xto push jobs off until night processing, etc.  The reason these functions
Xare split into a seperate file is so that the operators can change how the
Xqueue monitor treats certain jobs without having to actually restart the
Xmonitor.
X
XSome important things to note:
X      - The user batch queue is essentially always stopped.  The monitor
X`09will start it just long enough to let the specified number of jobs
X`09start and will stop the queue again.  This can be disconcerting!!
X      - The monitor watches the number of jobs pending.  Normally it will
X`09only allow one job to run at a time, but if a lot of jobs are
X`09pending it will allow two or even three jobs to run at once.
X      - The user batch queue is left in a stopped state at the end of the
X`09day.  This is because we need to have exclusive access to our
X`09database for backups each night, and this pretty much ensures there
X`09won't be any stray batch jobs around...
X      - Just before 5:00 pm the monitor sets any pending jobs to run at
X`09midnight.  For us this effectively pushes them into the night
X`09processing schedule and gets them out of the way.
X
X
XThese routines are offered as an example of one way to acquire a finer
Xdegree of control over your batch processing.  They also provide an example
Xof sorts on how to use the F$GETQUI lexical, hence VMS 5.x is required.
X
XNO WARRANTY OF ANY SORT IS IMPLIED ON THIS SOFTWARE, IT IS STRICTLY USE AT
XYOUR OWN RISK.
X
XIf you have any questions, comments or (better yet) suggestions please send
Xthem to:
X
X`09lhotka@incstar.com
X
X`09Rockford Lhotka`09`09`09`09612/779-1701
X`09Applications Project Leader
X
X`09INCSTAR Corp
X`091990 Industrial Blvd
X`09PO Box 285
X`09Stillwater, MN 55082
$ CALL UNPACK [.QMON]README.;1 745684562
$ v=f$verify(v)
$ EXIT

