Using the NT API for file I/O
Ó 1997 OSR Open Systems Resources, Inc.
Did you know there’s a complete set of system services that are supported
by Windows NT but are not documented? Come with us while we describe the
Native Windows NT API, and even demonstrate how you can use it in your own
programs.
If you asked a random collection of techno-dweebs to name the system
service API that is native to Windows NT chances are the vast majority
would say something that eventually translated to "The Win32 API". Those of
you properly schooled in NT systems internals will know that this just is
not the case. The Win32 API is implemented by a "Client-Side DLL" that is
specific to the Win32 Subsystem. The Win32 Subsystem is just one of Windows
NT’s Operating System (OS) Emulation Subsystems. All of NT’s OS Environment
Subsystems (Win32, POSIX, OS/2, and DOS/WoW) utilize services provided by
the Windows NT Executive. These services are accessed by the OS Environment
Subsystems via Windows NT’s actual native system service API, which is
called "the NT API".
A Special Purpose
From the ground up, Windows NT was designed to be an operating system that
facilitates the emulation of other operating systems and their APIs. The NT
OS itself works hard at providing a robust infrastructure, while not
imposing undo constraints on its OS Environment Subsystem clients and their
applications. For example, there is no uniform set of wildcards imposed on
NT file systems. This allows one to build a file system that includes "*"
or "?" in the names of files.
The native NT API was designed for the use of OS Environment Subsystems to
provide services to their clients. Thus, when a program running under
control of the Win32 OS Environment Subsystem wants to create a new process
it uses the Win32 function call CreateProcess(). The parameters to this
function are designed to be easy to use and make sense within the Win32 OS
Environment Subsystem’s framework. When the Win32 OS Environment Subsystem
receives the call, it can check its own information on the calling process
to see if (for example) the caller has the resources and privileges
necessary to have the request granted. If all his internal requirements are
met, the Win32 OS Environment Subsystem then issues a function call to the
native NT API function NtCreateProcess() to request Windows NT to create a
process on behalf of the user.
Not all client function calls are processed or even seen directly by the OS
Environment Subsystem however. If NT’s native handling of a function is
close enough to meet the OS Environment Subsystem’s requirements, and no
additional protection or resource checks are required by the OS Environment
Subsystem, the mapping between the subsystem’s function call and the native
NT function call can be performed right in the client-side DLL. This is the
case, for example, with file I/O requests to the Win32 subsystem. Win32
appears to rely for the most part on Windows NT’s native protection, quota,
synchronization, and handle management schemes. Thus, a Win32 application’s
ReadFile() call is translated within Win32’s client-side DLL to a call to
the native NT function NtReadFile(). The overhead of a call to the Win32 OS
Environment Subsystem is thus saved for this very common, and performance
critical, operation.
One of the most interesting things about the NT API is that it has never
been comprehensively documented by Microsoft. This must make NT the world’s
only commercially available operating system with an undocumented set of
native system services! In one way, the existence of the NT API doesn’t
really matter to users: Programs can do anything they legitimately need to
by using the interface provided by their OS Environment Subsystem. On the
other hand, it is through the native interfaces provided by an operating
system that we get a feel for how the operating system actually works. Not
to mention that in order to build your own reasonably efficient OS
Environment Subsystem, you would have to know the native NT API.
Native NT File I/O
So, what does the NT API look like? Well, let’s take a look at a few of its
most basic function calls.
First, NT’s native function to either access an existing file on disk or
create a new such file is the NtCreateFile()function. The prototype for
this function appears in Figure 1.
NTSYSAPI NTSTATUS NTAPI NtCreateFile(PHANDLE
FileHandle,
ACCESS_MASK DesiredAccess,
POBJECT_ATTRIBUTES ObjectAttributes,
PIO_STATUS_BLOCK IoStatusBlock,
PLARGE_INTEGER AllocationSize,
ULONG FileAttributes,
ULONG ShareAccess,
ULONG CreateDisposition,
ULONG CreateOptions,
PVOID EaBuffer,
ULONG EaLength);
Figure 1 -- NtCreateFile()
Notice that the parameters for NtCreateFile() are identical to those for
the ZwCreateFile() function, which is extensively and clearly documented in
the Windows NT DDK. This is more than mere coincidence. Each of the native
NT System Services comes in a ZwXxxx and NtXxxx variant.
According to NT’s Kernel-Mode Glossary (which contains a wealth of
fascinating trivia, by the way) the NtXxxx functions check the supplied
parameters and access modes for validity and explicitly set the previous
mode to USER mode. The ZwXxxx function variants do not. Hence, NT Drivers
call ZwCreateFile()when they are opening a file on their own behalf. OS
Environment Subsystems (or, indeed native NT applications) would call
NtXxxxx since they are calling from user mode.
Since the NT DDK is pretty clear on the meaning of the ZwCreateFile()
function parameters, we’ll avoid describing each of them here. However,
recall that this function is used to access lots of things other than files
on disk. In addition to disk files, NtCreateFile() may be used to access a
device, partition, directory, or even a socket, a pipe, or a mailslot.
One thing that might at first appear unusual about the NtCreateFile()
function is that the name of the file to open is not one of the immediate
parameters. Rather, the UNICODE_STRING specification for the file appears
in the OBJECT_ATTRIBUTES structure supplied in the ObjectAttributes
argument. This OBJECT_ATTRIBUTES structure is initialized with the function
call InitializeObjectAttributes(), which is also documented in the NT DDK.
The prototype for InitializeAttributes is shown in Figure 2 .
VOID InitializeObjectAttributes(POBJECT_ATTRIBUTES
InitializedAttributes,
PUNICODE_STRING ObjectName,
ULONG Attributes,
HANDLE RootDirectory,
PSECURITY_DESCRIPTOR SecurityDescriptor);
Figure 2 -- InitializeAttributes()
The ObjectName parameter takes a pointer to a UNICODE_STRING which contains
either a fully qualified path specification for the file to be opened, or a
partial file specification relative to a previously opened directory. If
the latter, the RootDirectory contains the handle to that previously opened
directory. Note that there is no default directory for the file being
opened as there is in Win32.
Using the OBJECT_ATTRIBUTES structure, there are two possible ways to open
the file "fred.txt" in the directory C:\temp. Either you build a
UNICODE_STRING for ObjectName that contains the fully qualified file path
specification, such as "\DosDevices\C:\Temp\fred.txt" or else you first
open the directory "\DosDevices\C:\Temp\" via a CreateFile() request,
obtaining a handle to this open (directory) file. You then build a
UNICODE_STRING for ObjectName that contains just the file name portion
"fred.txt" and pass it, along with the handle to the open directory to
InitializeObjectAttributes().
Here, once again, NT shows it’s true colors as an operating system designed
to facilitate the emulation of other operating systems. By not providing a
default path specification, Windows NT allows it’s OS Environment
Subsystems to the path for a file without intrusion by the underlying
system. In addition, whether or not the file specification is case
sensitive is determined by the Attributes parameter to the
InitializeObjectAttributes() function. This allows the OS Environment
Subsystem to supply its default behavior for that of Windows NT.
By the way, there is also a "short cut" API that can be used to access an
already existing file, and hence dispenses with some of the lesser-used
parameters. This is the NtOpenFile() function, which appears in Figure 3.
NTSYSAPI NTSTATUS NTAPI NtOpenFile(PHANDLE
FileHandle,
ACCESS_MASK DesiredAccess,
POBJECT_ATTRIBUTES ObjectAttributes,
PIO_STATUS_BLOCK IoStatusBlock,
ULONG ShareAccess,
ULONG OpenOptions);
Figure 3 -- NtOpenFile()
Note that things like a buffer for the ExtendedAttributes, and the
CreateDisposition are absent from this call, making it quick and easy to
code.
With the file opened, and the FileHandle returned, we can now proceed to
issue read and write requests to the file.
NT’s native API to read from a file is the NtReadFile() function. The
function to write to a file is the NtWriteFile() function. The prototypes
for these two are functions are identical, differing only in name. The
prototype for NtReadFile()is shown in Figure 4.
NTSYSAPI NTSTATUS NTAPI NtReadFile(HANDLE
FileHandle,
HANDLE Event,
PIO_APC_ROUTINE ApcRoutine,
PVOID ApcContext,
PIO_STATUS_BLOCK IoStatusBlock,
PVOID Buffer,
ULONG Length,
PLARGE_INTEGER ByteOffset,
PULONG Key);
Figure 4 -- NtReadFile()
This function is also documented in the NT DDK in it’s ZwReadFile()
variant. However, the documentation is not quite as extensive as it might
be. Some notes on each of the parameters are shown in List 1 .
Arguments to NtWriteFile()
FileHandle
The file handle returned from the CreateFile() call. If you have to ask,
you’re reading the wrong publication;
Event
Handle to a previously created event, to use for synchronization. NTOS sets
the state of this event to Signalled when the request is complete.
ApcRoutine
An optional pointer to a user Asynchronous Procedure Call (APC) function to
be called when the request completes. This is what Win32 refers to as a
FileIoCompletionRoutine(). This function will only be called if the calling
thread is in an alertable wait state. A wait is "alertable" if the
Alertable argument to the wait function (such as KeWaitForSingleObject())
is set toTRUE.
ApcContext
If ApcRoutine was supplied, above, this is a context argument passed to
that function when it is called.
IoStatusBlock
A pointer to an IO_STATUS_BLOCK to receive the result for the operation.
The returned data is not valid until the request completes.
Buffer
Pointer to the user buffer for the operation;
Length
Length in bytes of Buffer;
ByteOffset
An optional pointer to a LARGE_INTEGER containing the byte offset into the
file at which this operation is to begin.
Key
An optional value for a key, corresponding to a previously granted lock
taken out on the file.
List 1 -- Parameters
Beyond obviously writing to or reading from the file handle provided, the
precise behavior of the NtReadFile() and NtWriteFile() call depends on the
values supplied during the NtCreateFile(). For example, if during the
NtCreateFile() operation the "Synchronize" flag set in the DesiredAccess
parameter, and one of the FILE_SYNCHRONOUS_IO_xxxx flags has been specified
in the CreateOptions, calls to NtReadFile() and NtWriteFile() will complete
synchronously and the I/O system will track the current read/write offset
in the file.
There are two small differences between the NtReadFile()function and
Win32’s ReadFile() and ReadFileEx() functions. While these differences are
far from earth shattering, they still serve as an interesting example of
how an OS Environment Subsystem customizes its interface to meet its users
expectations or needs.
In NtReadFile() there is an explicit Key parameter. This Key is a value
supplied by the user during a previously successful NtLockFile() function
call. Supplying this Key allows the user to bypass a lock previously taken
out on a specific region of the file. In the Win32 ReadFile() function, no
Key value can be supplied. If a Win32 process has taken out a lock on a
file, the Key for that lock is apparently automatically supplied by the
Win32 Subsystem’s Client-Side DLL. Similarly, Win32 does not allow the user
to specify the ApcContext, choosing to uniformly supply a pointer to its
OVERLAPPED structure automatically instead.
A neat facility available through the native NT API but not through Win32
is the ability to cancel all I/O requests that you have outstanding on a
particular file handle. The function to accomplish this is in Figure 5.
NTSYSAPI
NTSTATUS NTAPI NtCancelIoFile(HANDLE FileHandle,PIO_STATUS_BLOCK
IoStatusBlock);
Figure 5 -- NtCancelIoFile()
If you’ve ever wondered how you accomplish an I/O cancel operation, now you
know!
Finally, to close a previously opened file handle, the NtClose() function
is used, also shown in Figure 5.
NTSYSAPI NTSTATUS NTAPI NtClose(HANDLE
HandleToClose);
NtClose()
The operation of this call is similar to that of the Win32 function
CloseHandle() function. Again, the differences between the native NT call
and the Win32 function can be seen: The Win32 function returns a BOOLEAN
and raises an exception (ugh!) if an invalid handle value is provided. The
native NT function simply returns an NTSTATUS value that indicates the
outcome of the call.
Building Native NT Programs
While all this new-found knowledge might be interesting, it would all be
totally academic if you couldn’t actually USE it. So how do you actually
build a native NT program? Well, the list below shows how we do it at OSR:
* You’ll need to define the function prototypes. Each of the NT API
functions needs to be defined as shown above.
* When you compile, you’ll need to include ntddk.h, or else redefine the
many types and structures that are found only there and are required
for these functions. For example, the structure OBJECT_ATTRIBUTES
appears to be defined in ntddk.h and nowhere else.
* You’ll need to link against ntdll.lib, which resolves your function
references to calls into ntdll.dll.
* Want to free yourself completely of subsystem control? When you link
your program, define the subsystem under which it runs to be "native"
(using the /Subsystem:native linker option).
And that’s all there is to it! If you’d like to grab a very simple sample
application that writes "hello world" to a file using the native NT API,
you can download it from our web site (native.zip).
Just Because You Can
So, is this the new way we should all write our user-mode code from now on?
What does one achieve by "going native" and directly using the Windows NT
system service API?
Well, you could certainly achieve a lot of hassles. Since the NT API is
neither documented nor supported, there’s nobody at Microsoft you can
complain to if the interface changes arbitrarily or doesn’t work as (not)
advertised. Plus, with the absence of a default directory for your file
opens you get to manually specify the path name to each of the files you
want to open. Remember, the NT API really wasn’t designed to be an end-user
interface.
Is there any good reason at all, then, to use the NT API directly? Well, of
course. For one, there’s the ability to cancel I/O requests, which you
can’t even get at from Win32. Playing with the NT API also gives you a
window into the handiwork of the NT developers. So much of NT is buried
underneath other stuff, much of which might be considered pretty ugly by
comparison. Using the NT API helps you get a better "feeling" for what NT
itself does as an operating system, and how it works. Also, we have in our
travels found at least one real application that required the use of the NT
API.
Return to Previously Published Articles
Return to OSR's Home Page
[Image]consulting& developing/ kits / seminars / NT insider/ resources /
client area / about OSR