From - Fri Sep 12 13:54:30 1997 Path: news.mitre.org!blanket.mitre.org!nntprelay.mathworks.com!infeed1.internetmci.com!newsfeed.internetmci.com!205.252.116.205!howland.erols.net!winter.news.erols.com!mellon+usenet From: mellon+usenet@pobox.com (Anatoly Vorobey) Newsgroups: comp.os.ms-windows.programmer.nt.kernel-mode Subject: Re: Kernel Mode APC's Date: 27 Aug 1997 08:19:19 GMT Organization: Erol's Internet Services Lines: 234 Message-ID: References: <340308E7.167E@sgi.com> <1997Aug26.141508.8103@cmkrnl> Reply-To: mellon+usenet@pobox.com NNTP-Posting-Host: sasami.jurai.net X-Received-On: 27 Aug 1997 08:19:19 GMT X-Newsreader: slrn (0.9.2.1 BETA UNIX) On 26 Aug 97 14:15:08 PST, Jamie Hanrahan wrote: >In article <340308E7.167E@sgi.com>, Vimal Parikh writes: >> Is there any way for a kernel mode driver to queue an APC (i.e. >> asynchronous procedure call) to be executed in the context of a >> user mode thread ? Yes. But it's _highly_ undocumented. >> I have seen references in the Helen Custer's book about I/O system >> using APC's to signal I/O completion events. But I haven't been >> able to find kernel mode functions to queue an APC. >> >> Is there an equivalent of "QueueUserAPC" for use by the drivers ? > >There is, but it isn't documented. The call you want is >KeInsertQueueApc. The APC data structure also appears in NTDDK.H. It >would appear it's a matter of just filling it in and calling the >routine. Unfortunately not. Inserting the APC is one thing; getting the thread to notice it is more difficult. The design is that the thread must indicate it wants to receive APCs in order to be able to receieve APCs. This the thread does by using extended versions of waiting/sleeping/etc. functions with notification flag on. See SDK documentation on QueueUserApc() and around. If that flag is on (the flag is just a byte offset inside the KTHREAD structure for the thread), the APC will get dispatched first time IRQL drops to APC_LEVEL or below. The thread does _not_ have to actually sleep/wait at that time in order to receive the APC, but unfortunately, only these waiting/sleeping routines will set the notification flag on (an APC call is also used to "boot" a newly created thread; in that case, the notification flag for the new thread is set artificially by the creation routine). I think this is overly restrictive design. It's probably done this way so that a programmer who uses IO completion with a thread would be forced to wait for it in that thread, to prevent possible synchronization problems between usual code and completion routine. However, the user-mode APC routine does _not_ have to conflict with the usual code, and synchronization can be effectively used to prevent conflicts when they may arise; so the design prevents a programmer who knows what he/she's doing from the most efficient way of kernel-user mode notification. >Of course this will be unsupported, may change in a future >release, comments may have settled during shipment, etc., etc. > >If you get it working, please post the details here. I have got it to work, but in a very undocumented manner. I don't recommend its to use to anyone except those who really need this (APC notification is much, MUCH faster than event or IOCTL notification when delivered to the calling thread, and faster, but not much, when delivered across threads) and understand the portability implications. In order to make it work, I have to set the notification flag for the target thread using hard-coded byte offset inside its thread structure. Although this appears to work fine in NT 3.51 as well as all versions and service packs of 4.0, nobody knows what will happen in the future. I've posted my article detailing this before; here's a repost. If anybody needs more details or help with this, contact me. ------------------------- Hi there, The issue of notifying user-mode thread from a kernel-mode driver, or calling a user-mode routine from same, is very often discussed here. The usual solutions offered are: 1) a dedicated thread sending an IRP to the driver which the driver leaves uncompleted and completes when it needs to communicate information to the thread; and 2) signaling an event which can be accessed in both user and kernel mode, with user-mode thread waiting on it. Both these approaches have their drawbacks; in particular, they need a dedicated thread waiting, and they're relatively slow - sometimes you _know_ you're in the right context, and you just need to tell something to the user-mode thread or pass some information to it as soon as possible. I've tried to find other ways of doing the same, being driven mostly by curiosity, spirit of exploration and stubborness. I've found two other mechanisms by which it is possible to make a thread call some specific user-mode function: 1. User-mode APC (Asynchronous Procedure Call). The whole issue of APCs is quite undocumented in the DDK. Actually, more informatiion about APCs can be found in Win32 SDK help than in NT DDK help! That's because the whole mechanism of completion routine-based I/O routines (like ReadFileEx(), etc.) is quite transparently based on APC, and the SDK help says a few basic things about APCs. They're also discussed briefly in the classic Helen Custer's "Inside Windows NT". The main problem with this approach is that according to the NT design principles, a thread can receive a user-mode APC _only_ if it declares itself alertable: either by waiting on a synchronization object with alertable flag set on, or explicitly calling a certain function in ntdll.dll to check whether it should be alerted. Since we _don't_ want our thread to wait on anything (we may just as well use named event then) we're stuck. In the code I give below, I overcome this problem in an undocumented and somewhat dirty way; I'm still searching for more "clean" ways to do it. When a user-mode APC is passed to a thread in this way, its routine be called next time the thread runs in user-mode. If the thread already runs in user-mode (which is only an issue on multiprocessor machines), the routine won't be invoked immediately, however; it'll be called next time the thread _returns_ to user-mode from some kernel-mode service. Usually that happens almost immediately since an active thread is calling kernel-mode services all the time; in the worst case it'll happen after the next clock tick, when clock tick procedure returns to user-mode. The user-mode APC will not, however, interrupt any kernel-mode activity; i.e. if the thread is waiting on an object, it won't be woken up; when it wakes up by itself, however, it will receive the pending APC immediately. 2. KeUserModeCallback(). This is an undocumented function used by Win32 subsystem running in the kernel in NT 4.0 (win32k.sys). It's used when the subsystem either needs to know some information stored in user-mode (for example, in user32.dll's data), or needs to call a thread's window procedure (for example, when you move your mouse, the subsystem eventually receives notification of it in kernel mode, and it calls your window procedure with WM_MOUSEMOVE message using this function). A catch here is that this mechanism is predefined to call only some specified functions: one of parameters to KeUserModeCallback() is an index to a special table from which later in user mode an address to call is fetched. Still it's possible to exploit this mechanism to call, _very_ quickly (much quicker than the APC, named event or pending IRP mechanisms allow) an arbitrary routine of your code. The catch here is that you _must_ be in your thread's context for this to work; in this respect user-mode APC is better since you can freely send it to any thread in the system. Below is sample code that will send user-mode APC to the current thread, calling an arbitrary user-mode routine and passing it three arbitrary parameters. If you want to send an APC to your thread from arbitrary context, you should capture its KTHREAD pointer by calling KeGetCurrentThread() in _its_ context (say, when receiving a custom control request from it and being top-level), and use it in a call to KeInitializeApc later at any time. Unfortunately, it will only work on x86 architecture due to one machine-dependent line which modifies directly the KTHREAD structure. It should, however, work on free/checked builds and SMP/uniprocessor machines alike. It's taken right out of an article I'm completing which discusses in detail both kernel-mode and user-mode APCs; I'm still uncertain whether I should simply put it on the web or try to offer it to some paper magazines. If you're interested, please try out this code and report to me whether it works/ doesn't work for you and what happens if it doesn't. Note that I of course disclaim any responsibility; your system might very well crash/bugcheck, though I've tested this code for some time now and it's always been working fine for me. /* The APC structure is defined in ntddk.h */ /* this is KERNEL_ROUTINE for our APC; in particular, it gets called when the APC is being delivered. Usually one of predefined useful routines in the kernel are used for this purpose, but we can't use any of them as none of them are exported. */ void MyRoutine(struct _KAPC *Apc, PKNORMAL_ROUTINE norm_routine, void *context, void *SysArg1, void *SysArg2) { ExFreePool(Apc); return; } /* pointer to the APC we will create */ static struct _KAPC *apc; /* KeInitializeApc() and KeInsertQueueApc() are the two functions needed to send an APC; they're both exported but not prototyped in the DDK, so we prototype them here. */ void KeInitializeApc(struct _KAPC *Apc, PKTHREAD thread, unsigned char state_index, PKKERNEL_ROUTINE ker_routine, PKRUNDOWN_ROUTINE rd_routine, PKNORMAL_ROUTINE nor_routine, unsigned char mode, void *context); void KeInsertQueueApc(struct _KAPC *APC, void *SysArg1, void *SysArg2, unsigned char arg4); /* call this function when you need to send a user-mode APC to the current thread. addr must be linear address of your user-mode function to call: void MyApcRoutine(ULONG arg1, ULONG arg2, ULONG arg3); ... SendAddrToTheDriverUsingIoctl((ULONG)MyApcRoutine); you should send it to the driver using your custom IOCTL. arg1, arg2, arg3 are arbitrary ulong's which are passed to the function residing at addr; this function should be prototyped as receiving three parameters and returning void. */ void SendAPC(ULONG addr, ULONG arg1, ULONG arg2, ULONG arg3) { PKTHREAD thread=KeGetCurrentThread(); /* this is self-explanatory */ apc=ExAllocatePool(NonPagedPool, sizeof(struct _KAPC)); /* Initialize the user-mode APC */ KeInitializeApc(apc, thread, 0, (PKKERNEL_ROUTINE)&MyRoutine, 0, (PKNORMAL_ROUTINE)addr, 1, (PVOID)arg1); /* Insert it to the queue of the target thread */ KeInsertQueueApc(apc, (PVOID)arg2, (PVOID)arg3, 0); /* Mark the current thread as alertable to force it to deliver the APC on the next return to the user-mode. NOTE: severely undocumented code here! */ *((unsigned char *)thread+0x4a)=1; } That's about it. I'll be very grateful to hear comments, corrections, additions or flames from you. Questions are always welcome. Yours, Anatoly. -- Anatoly Vorobey, mellon@pobox.com http://pobox.com/~mellon/ "Angels can fly because they take themselves lightly" - G.K.Chesterton