From:	CSBVAX::MRGATE!steinberger-RELAY@KL.SRI.COM@SMTP 21-JUN-1988 14:13
To:	ARISIA::EVERHART
Subj:	Re: MPW_HILIMIT


Received: from Warbucks.AI.SRI.COM by KL.SRI.COM with TCP; Tue, 21 Jun 88 09:12:33 PDT
Date:     Tue, 21 Jun 88 09:12:53 PDT
From:     adelman@Warbucks.AI.SRI.COM (Kenneth Adelman)
Reply-To: Adelman@Warbucks.AI.SRI.COM (Kenneth Adelman)
Message-Id: <880621091140.35f@Warbucks.AI.SRI.COM>
Subject:  Re: MPW_HILIMIT
To:       steinberger@kl.sri.com
cc:       info-vax@kl.sri.com

> I have been reading the VMS Tuning Guide provided (free) from Touch Technol-
> ogies (they market the Dynamic Load Balancer).  The guide advises setting
> MPW_HILIMIT (limit at which VMS starts writing pages from the modified
> page list to disk.  Sets a limit on the MP list size) at 15 percent of the
> size of physical memory.  The claim is made that the loss of physical
> memory is more than compensated by a reduced paged I/O activity.
>     This seems like a rather large number for this sysgen parameter,
> as DEC set the default at 500 pages.	Has anyone followed Touch's advise?
> Have you noticed a significant performance improvement?  We run large
> images, use large pagefiles and often encounter applications where much
> paging is required even with large WSEXT values.  I am thus trying to
> evaluate whether increasing MPW_HILIMIT (and MPW_WAITLIMIT) is a
> reasonable thing to do (our current setting is 655, physical memory
> is 32768 pages and VMS already uses 4400+ of these).
>     Any comments or other rules of thumb?

    Think of the modified list as a page cache. Raising HILIMIT creates
a cache which reduces disk I/O caused by pagefaults by turning some 'hard'
faults into 'soft' faults. This used to be a good performance tip under
VMS V2 (before working set extents and cheap memory) because it made it
easier for processes to pagefault, and VMS was able to recover that memory
when the freelist became too small.

    I have yet to see a system which wouldn't benefit better by
putting that extra memory into WSEXTENTs than into the free or
modified list size.  The advantage of using WSEXTENTs is that no
pagefault exceptions occur at all to use this 'cache'. The only hitch
is that the WSEXTENT is per-processes and the modified list size is
per-machine, so unless you have a good feeling for how many big
processes will be running, where might be no one good number for the
WSEXTENT.

    There are also some disadvantages to having a large modified list.
When the list grows to size MPW_HILIMIT is written down to size MPW_LOLIMIT.
When the freelist gets to small, the modified list is written down to zero.
With a very large list both of these actions can cause a sudden burst
of activity directed at your paging disks, and sometimes you machine might
appear to 'hang' for seconds.

    This isn't to say that you can't tune your modified page writer.
Doubling the size of your modified list probably isn't unreasonable.
The parameter MPW_WRTCLUSTER controls the size of the I/O to write the
list, and hence the number of I/Os required to write the list. Raising
it reduces the load on the disk, but increases the CPU overhead of
building the lists. I don't know where the break-even is, but on a
large machine, or in a big cluster where the disk I/O is a problem,
raising it would probably help. If the size of the modified list is
greater than MPW_WAITLIMIT a processes which faults out a modified page is
places into the RWMPB wait state until the list shrinks. Autogen
sets WAITLIMIT to be the same as HILIMIT, meaning that the process
which faults the page which starts the modified list writing must wait
until some of the writing completes. This is usually an unnecessary
'throttle' which penalizes processes generating lots of modified
pages.	If you raise MPW_WAITLIMIT to two or three times HILIMIT these
processes will be allowed to continue.	The parameter MPW_WAITLIMIT
must be greater than MPW_HILIMIT or a system deadlock will occur.

    And now, for an added bonus, a program which lets you change the
working set of another process. Perhaps someone with time will write a
SET WORK/ID= CLD interface to this.

to compile:
$ cc setwork
$ macro setwork-kernel
$ link setwork,setwork-kernel,sys$system:sys.stb/sel,sys$library:vaxcrtl/lib

----cut for setwork.c----
/*  Program to poke the working set of another process

    This program queues a special kernel AST to the target process
    to change its working set limit, quota, and extent. Note that the
    values printed below are not a reliable indication of the working
    set because the AST may not yet have completed.

    The actual change in the size of the working set is subject to the
    VMS scheduler.  If you raise the working set limits, VMS may not
    raise the size of the working set unless the processes is
    faulting, etc.

    The purpose of the program is to allow you to raise the working set
    of a heavily faulting program.
*/

#include <stdio.h>

main(argc,argv)
int argc;
char *argv[];
{

	long status, sys$cmkrnl(), arglst[5];
	extern long pokewset();

	if (argc != 5) {
		fprintf(stderr,"Usage: %s pid wslimit wsquota wsextent\n",argv[0]);
		exit(0x10000000);
	}

	arglst[0] = 1;

	if(!sscanf(argv[1],"%x",&arglst[1])) {
		fprintf(stderr,"%s: error parsing pid\n",argv[0]);
		exit(0x10000000);
	}

	if(!sscanf(argv[2],"%d",&arglst[2])) {
		fprintf(stderr,"%s: error parsing wslimit\n",argv[0]);
		exit(0x10000000);
	}

	if(!sscanf(argv[3],"%d",&arglst[3])) {
		fprintf(stderr,"%s: error parsing wsquota\n",argv[0]);
		exit(0x10000000);
	}

	if(!sscanf(argv[4],"%d",&arglst[4])) {
		fprintf(stderr,"%s: error parsing wsextent\n",argv[0]);
		exit(0x10000000);
	}

	status = sys$cmkrnl(pokewset,arglst);
	if (!(status&1)) return(status);
}
----cut for setwork-kernel.mar----
	.title setwork-kernel
;
;   SETWORK - short routine to queue a special kernel AST to another
;	      process to change its working set limit, quota, and extent.
;
;   Kenneth Adelman;	Nov 1987
;
;	4(AP) = PID
;	8(AP) = LIMIT
;	12(AP) = QUOTA
;	16(AP) = EXTENT
;
	.library /SYS$LIBRARY:LIB/
	$acbdef

spec_limit  = acb$c_length
spec_quota  = acb$c_length+4
spec_extent = acb$c_length+8
code_off    = acb$c_length+12

	$dyndef
	$ipldef
	$ssdef
	$pcbdef
	$phddef
	$pridef

	.macro	minw2,x,y,?l
	cmpw	x,y
	bgtru	l
	movw	x,y
l:
	.endm

	.macro	maxw2,x,y,?l
	cmpw	x,y
	blssu	l
	movw	x,y
l:
	.endm

	.entry pokewset,^m<r2,r3,r4,r5,r6>
;
;	Get the internal PID of the process to queue the nASTy by
;

	movl	4(ap),r0
	jsb	g^exe$epid_to_ipid
	tstl	r0
	beql	nonproc
	movl	r0,r6		;save ipid

;
;	Allocate a buffer to hold the ACB, three longwords of data, and the
;	AST code.
;

	movl	#codelen+code_off,r1
	jsb	g^exe$allocbuf
	blbs	r0,10$
	ret
10$:	movl	r2,r5		;save address

;
;	Fill in the ACB
;

	movl	r6,acb$l_pid(r5)
	movb	#dyn$c_acb,acb$b_type(r5)
	movw	r1,acb$w_size(r5)
	movb	#1@acb$v_kast,acb$b_rmod(r5)
	movab	code_off(r5),acb$l_kast(r5)

;
;	Copy the working set sizes and the code.
;

	movl	8(ap),spec_limit(r5)
	movl	12(ap),spec_quota(r5)
	movl	16(ap),spec_extent(r5)
	pushl	r5
	movc3	#codelen,w^code,@acb$l_kast(r5)
	popl	r5

;
;	Queue the nASTy
;

	movzbl	#pri$_ticom,r2	;set priority increment class
	movzwl	acb$l_pid(r5),r0   ;destination pid
begin_lock:
	dsbint	lock_ipl
	jsb	g^sch$qast
	enbint
	ret
lock_ipl:
	.long	ipl$_synch
end_lock:
	assume <end_lock-begin_lock> le 512
nonproc:
	movl	#ss$_nonexpr,r0
	ret

;
;	This *PIC* code executes in the context of the target.
;

code:
	movl	g^ctl$gl_phd,r1
	pushr	#^m<r8,r9,r10,r11>

;	min_wset = phd->phd$w_wsdyn - phd->phd$w_wslist +
;		   2 * phd->phd$w_wsfluid + 3;

	subw3	#1,phd$w_wslist(r1),r0
	mulw3	#2,phd$w_wsfluid(r1),r8
	addw2	#2,r8
	addw2	phd$w_wsdyn(r1),r8
	subw2	r0,r8

;	if (spec_limit) {
;	    limit = min(auth_limit,max(min_wset,spec_limit));
;	} else {
;	    limit = phd->phd$w_dfwscnt - phd->phd$w_wslist + 1;
;	}

	movw	spec_limit(r5),r9
	beql	10$
	maxw2	r8,r9
	brb	11$
10$:	subw3	r0,phd$w_dfwscnt(r1),r9
11$:

;	if (spec_quota) {
;	    quota = min(auth_limit,max(min_wset,spec_quota));
;	} else {
;	    quota = phd->phd$w_wsquota - phd->phd$w_wslist + 1;
;	}

	movw	spec_quota(r5),r10
	beql	20$
	maxw2	r8,r10
	brb	21$
20$:	subw3	r0,phd$w_wsquota(r1),r10
21$:

;	if (spec_extent) {
;	    extent = min(auth_extent,max(min_wset,spec_extent));
;	} else {
;	    extent = phd->phd$w_wsext - phd->phd$w_wslist + 1;
;	}

	movw	spec_extent(r5),r11
	beql	30$
	maxw2	r8,r11
	brb	31$
30$:	subw3	r0,phd$w_wsextent(r1),r11
31$:

;	    extent = max(limit, extent);
;	    quota = max(limit, quota);

	maxw2	r9,r10
	maxw2	r10,r11

;	phd->phd$w_dfwscnt = phd->w_wslist - 1 + limit;
;	phd->phd$w_wsquota = phd->w_wslist - 1 + quota;
;	phd->phd$w_wsextent = phd->w_wslist - 1 + extent;

	addw3	r0,r9,phd$w_dfwscnt(r1)
	addw3	r0,r10,phd$w_wsquota(r1)
	addw3	r0,r11,phd$w_wsextent(r1)
	maxw2	phd$w_wsquota(r1),phd$w_wsauth(r1)
	maxw2	phd$w_wsextent(r1),phd$w_wsauthext(r1)

;
;	After changing the working set limits, check to see if this
;	process is over its limit. If it is, $ADJWSL it down.
;
	addw3	phd$w_wssize(r1),phd$w_wslist(r1),r8
	movw	phd$w_wsquota(r1),r9
	cmpl	g^sch$gl_borrowlim,g^sch$gl_freecnt
	bgtru	40$
	movw	phd$w_wsextent(r1),r9
40$:	subw	r8,r9			; r8 contains the number of pages to adjust by
	bgequ	50$			; if the overdraft < 0, WS is ok
	clrl	-(sp)			; return ws
	cvtwl	r9,-(sp)
	calls	#2,g^sys$adjwsl
50$:
	popr	#^m<r8,r9,r10,r11>

	setipl	#ipl$_astdel
	movl	r5,r0
	jmp	@#exe$deanonpaged
codelen = .-code
	.end
