To: Len Szubowicz
From: Glenn Everhart
Re: REQCOM fix
1/28/1997

Where REQCOM finishes with no delays, it calls the driver start-io again
and on alpha this pushes kernel stack down, possibly causing a system
crash if kernel stack becomes invalid.

A variant of this existed in dkdriver where it was trying to initiate
many I/Os and its calls to driver start-io also overflowed knl stack.

The DKdriver problem was worked around by looping inside start_io and
not calling driver start-io. This broke intercepts (e.g. every 3rd party
cacher) that relied on seeing every startio, and prevents a simple
switch driver if not fixed.

The REQCOM problem itself is akin but not fixed.


Now, the dkdriver problem can be addressed with a callout for the benefit
of intercepts, but the reqcom issues still will exist there even so.


I see maybe two ways around this.

1. Try and work some Macro-64 logic that will have the effect of
jumping via the driver start-io entry instead of calling it, making
the requirement that the lock and register state be the same as
at the initial call (as on VAX) and therefore NOT pushing the stack.
DKdriver and REQcom could both use such.

Major disadvantage is that I don't know if this can even be done
from C. Still, we are at the point of call ready to do a RET
to return from the start-io routine and the state of the machine
should be well determined. A jacket for the benefit of C would
also be well defined, though the manipulations would not be
within the call standard exactly. Under the circumstances I think
that may not be a problem.

Another problem is that someone trying to gain control at start_io
needs then to have a way to transfer control again to the original
start_io entry. Thus there needs to be something that will do
what amounts to a jump both for use by REQCOM and friends, and for
anyone's intercept, so that control is transferred cleanly.


2. You can have REQCOM and whatever else (ie, the dkdriver loop code)
issue a fork and then call start_io from the fork routine (with the
stack clean). If the stack weren't "too" near the bottom of a page
you might just avoid the fork, but there is no simple way to detect
the bottom of the stack; nor is it obvious how near is "too" near,
particularly if intercepts or drivers like shdriver or striping
driver might be redispatching anything.

This is of course consistent with call standard, but adds runtime
load of extra forking. 

3. A third way might be to define some other address that must be
branched to and just add it to the driver interface and the DDTAB.
This would function like the branch in DKDRIVER, but by convention
might call out to a routine in ddtab. Using a callout would at least
make calling understandable. The return from such a callout would
need to be tested and the IRP either junked (and the routine just
RET'd from) or processed, to allow external control and moving the
IRP around. (It would be nice to be able to just jump to a point
past the start_io entry point, but this jumping between routines
could pose some addressing problems. Also and again a jump would be
hard to use from C, I would suspect. A call on the other hand would
present no problems. Since it is a call-OUT, the stack would come
back afterwards, and the default could be just to call a RET. I'd
add a DPTAB bit to say if this were done and a DDTAB field to hold
the callout address, so only drivers that were modified would have
the bit or the extra DDTAB field. There are plenty of DPTAB status
bits left. Doing what amounts to a "jump" for REQCOM would mean
probably some Macro64 routine to do the actual work, but avoiding
kernel stack invalid would seem not an impossible justification.
Also, note, if you do this, you might get past the register saves
and so a few instructions dispatching might be offset by some
instructions saved by not having to re-save registers. REQCOM
has access to the UCB, hence the DDT, though I'd add an ENVIRONMENT
or some such arg to the macro for altered drivers so the expansion
would be as before for unaltered devices, but call the "jump-fake"
routine for new ones, and let the DPTAB macro set up a default
of some kind so drivers might not even have to know that some
macros expand a little differently.

Anyhow, the foregoing are my thoughts. Not really an IR; this isn't
a separate project at the moment, but stuff I need to get decided
and might conceivably need to do if nobody else does it.

I can concoct dkdriver that will just fork before it loops and work
around the problem for a while, but this does not solve the general
reqcom problem. I would VERY much like not to bother with catching
IRPs at either pending_io OR start_io OR both and do NOT want to
mess with a whole separate queue (to be manipulated by hand) if
it can be avoided. I got it working, but the code is less clear than
I want it to be.

Glenn Everhart

The foregoing are some thoughts on the matter. I'm inclined to
favor the last, though it would mean REQCOM would differ