From:	SMTP%"RELAY-INFO-VAX@CRVAX.SRI.COM"  6-JAN-1994 12:52:21.64
To:	EVERHART
CC:	
Subj:	Re: Q: Finding memory leak in VMS/Motif

X-Newsgroups: comp.os.vms
Subject: Re: Q: Finding memory leak in VMS/Motif
Message-Id: <1994Jan5.092205.317@ondec.lonestar.org>
From: d_north@ondec.lonestar.org (David G. North, CCP)
Date: 5 Jan 94 09:22:05 CDT
Followup-To: comp.os.vms
Organization: onDEC systems, Dallas, TX
Lines: 48
To: Info-VAX@CRVAX.SRI.COM
X-Gateway-Source-Info: USENET

In article <1994Jan4.190533.12932@hns.com>,dharter@hns.com (Derek Harter) wrote:
> I have what appears to be a memory leak problem on a program running under
> VMS and using DEC motif and X window environments.  The application runs under
> various versions of VMS (mostly V5.4 and V5.5) and uses DEC motif (version 1.1
                            ^^^^^^^^^^^^^^^^^^^I assume VAX/VMS then...
> I believe). My questions are the following:
> 
> 1) In general what are the best methods of detecting a memory leak in a
> program running under VMS.
If you're on a VAX... write code to intercept calls into malloc & free, and
even possibly LIB$GET_VM & LIB$FREE_VM.  Then you can track every single
call & every single block if you have to.

General outline of how to do this is:
  Locate the entrypoint into the routine to intercept [i.e. it's address]
  Make a copy of that page of memory
  Delete that page of memory ($DELTVA)
  Recreate that page of memory ($CRETVA)
  Restore original contents
  Save the contents of JMP L^xxx instruction in the entry vector
  Redirect the JMP L^xxx instruction to your own routine
  Call back into original function as needed
  Don't try to do all this in C... you'll need some macro too

Hints:
  1) put a header on the alloc'd blocks that describes the call stack that
     requested the block... that way if you end up with 55 guzzillion
     48-byte blocks for no aparrent reason, you'll at least know what
     call-chain asked for them.
  2) keep a list of the last 256 malloc'd blocks so you can suspend the
     process & backtrack over what it's been doing with its memory by
     looking at the blocks with SDA.
  3) If you suspect incorrect usage of blocks (i.e. useage after free,
     or usage after malloc w/o proper initialization), you can also write
     trash (0xfefefefe etc.) into the block upon first allocation, and
     prior to final deallocation.  Now if it gets [re]used at the wrong time,
     the program will probably choke.

> 2) We long ago looked at all our malloc/free interactions and believe them
> to be correct.
Many other things use malloc & free, and the LIB$xxx_VM routines.
Look for instances where you call getenv() repeatedly for a name that
exists.  getenv() has a permanent memory leak {designed into it!}.
The leakage may be in the X routines.  The only thing you can do here is
prove whodunnit, then "inform" the appropos vendor.
-- 
David G. North, CCP, D_North@Ondec.Lonestar.Org {...!letni!tachyon!ondec...}
"PI is approximately 3.14159265358979323846264338327950288419716939937510582097"