HP OpenVMS systems
The Create Mailbox and Assign Channel (SYS$CREMBX), Get Device/Volume Information (SYS$GETDVI), and Queue I/O Request (SYS$QIO) system services are described in greater detail in Chapter 23.
OpenVMS Alpha and OpenVMS I64 support tightly coupled symmetric multiprocessing (SMP). This chapter presents a brief overview of symmetric multiprocessing terms and characteristics. For more information about SMP concepts and hardware configurations, refer to VMS for Alpha Platforms Internals and Data Structures.
A multiprocessing system consists of two or more CPUs that address common memory and that can execute instructions simultaneously. If all CPUs in the system execute the same copy of the operating system, the multiprocessing system is said to be tightly coupled. If all CPUs have equal access to memory, interrupts, and I/O devices, the system is said to be symmetric.
In most respects the members of an OpenVMS SMP system are symmetric. Each member can perform the following tasks:
The members of an SMP system are characterized in several ways. One
important characteristic is that of primary CPU. During system
operation the primary CPU has several unique responsibilities for
system timekeeping, writing messages to the console terminal, and
accessing any other I/O devices that are not accessible to all members.
Although the hardware and software permit device interrupts to be
serviced by any processor, in practice all device interrupts are
serviced on the primary CPU. An SMP configuration may include some
devices that are not accessible from all SMP members. The console
terminal, for example, is accessible only from the primary processor.
5.2.1 Booting an SMP System
Booting the system is initiated on a CPU with full access to the console subsystem and terminal, called the BOOT CPU. The BOOT CPU controls the bootstrap sequence and boots the other available CPUs. On OpenVMS Alpha and OpenVMS I64 systems, the BOOT CPU and the primary CPU are always the same; the others are called secondary processors.
The booted primary and all currently booted secondary processors are
called members of the active set. These processors actively participate
in system operations and respond to interprocessor interrupts, which
coordinate systemwide events.
5.2.2 Interrupt Requests on SMP System
In an SMP system, each processor services its own software interrupt requests, of which the most significant are the following:
SMP supports the following goals:
This chapter describes the operating system's synchronization features. It focuses on referencing memory and the techniques used to synchronize memory access. These techniques are the basis for mechanisms OpenVMS itself uses and for mechanisms OpenVMS provides for applications to use.
This chapter contains the following sections:
Section 6.1 describes synchronization, execution of threads, and atomicity.
Section 6.2 describes alignment, granularity, ordering of read and write operations, and performance of memory read and write operations on VAX and Alpha systems in uniprocessor and multiprocessor environments.
Section 6.3 describes how alignment and granularity affect the access of shared data on I64 systems. It also discusses the importance of the order of reads and writes completed on I64 systems, and how I64 systems perform memory reads and writes.
Section 6.4 describes memory read-modify-write operations on VAX and Alpha systems in uniprocessor and multiprocessor environments.
Section 6.5 describes memory read-modify-write operations on I64 systems.
Section 6.6 describes hardware-level synchronization methods, such as interrupt priority level, load-locked/store-conditional and interlocked instructions, memory barriers, and PALcode routines.
Section 6.7 describes software-level synchronization methods, such as process-private synchronization techniques, process priority, and spin locks. It also describes how to write applications for a multiprocessor environment using higher-level synchronization methods and how to write to global sections.
Section 6.8 describes how to use local and common event flags for synchronization.
Software synchronization refers to the coordination of events in such a way that only one event happens at a time. This kind of synchronization is a serialization or sequencing of events. Serialized events are assigned an order and processed one at a time in that order. While a serialized event is being processed, no other event in the series is allowed to disrupt it.
By imposing order on events, synchronization allows reading and writing
of several data items indivisibly, or atomically, in order to obtain a
consistent set of data. For example, all of process A's writes to
shared data must happen before or after process B's writes or reads,
but not during process B's writes or reads. In this case, all of
process A's writes must happen indivisibly for the operation to be
correct. This includes process A's updates---reading of a data item,
modifying it, and writing it back (read-modify-write sequence). Other
synchronization techniques are used to ensure the completion of an
asynchronous system service before the caller tries to use the results
of the service.
6.1.1 Threads of Execution
Code threads that can execute within a process include the following:
Process-based threads of execution can share any data in the per-process address space and must synchronize access to any data they share. A thread of execution can incur an exception, which results in passing of control to a condition handler. Alternatively, the thread can receive an AST, which results in passing of control to an AST procedure. Further, an AST procedure can incur an exception, and a condition handler's execution can be interrupted by an AST delivery. If a thread of execution requests a system service or RMS service, control passes to an inner access-mode thread of execution. Code that executes in the inner mode can also incur exceptions, receive ASTs, and request services.
Multiple processes, each with its own set of threads of execution, can execute concurrently. Although each process has private address space, processes can share data in a global section mapped into each process's address spaces. You need to synchronize access to global section data because a thread of execution accessing the data in one process can be rescheduled, allowing a thread of execution in another process to access the same data before the first process completes its work. Although processes access the same system address space, the protection on system space pages usually prevents outer mode access. However, process-based code threads running in inner access modes can access data concurrently in system space and must synchronize access to it.
Interrupt service routines access only system space. They must synchronize access to shared system space data among themselves and with process-based threads of execution.
A CPU-based thread of execution and an I/O processor must synchronize access to shared data structures, such as structures that contain descriptions of I/O operations to be performed.
Multiprocessor execution increases synchronization requirements when the threads that must synchronize can run concurrently on different processors. Because a process with only one kernel thread executes on only one processor at a time, synchronization of threads of execution within such a process is unaffected by whether the process runs on a uniprocessor or on an SMP system. However, a process with multiple kernel threads can be executing on multiple processors at the same time on an SMP system. The threads of such a process must synchronize their access to writable per-process address space.
Also, multiple processes execute simultaneously on different
processors. Because of this, processes sharing data in a global section
can require additional synchronization for SMP system execution.
Further, process-based inner mode and interrupt-based threads can
execute simultaneously on different processors and can require
synchronization of access to system space beyond what is sufficient on
Atomicity is a type of serialization that refers to the indivisibility of a small number of actions, such as those occurring during the execution of a single instruction or a small number of instructions. With more than one action, no single action can occur by itself. If one action occurs, then all the actions occur. Atomicity must be qualified by the viewpoint from which the actions appear indivisible: an operation that is atomic for threads running on the same processor can appear as multiple actions to a thread of execution running on a different processor.
An atomic memory reference results in one indivisible read or write of
a data item in memory. No other access to any part of that data can
occur during the course of the atomic reference. Atomic memory
references are important for synchronizing access to a data item that
is shared by multiple writers or by one writer and multiple readers.
References need not be atomic to a data item that is not shared or to
one that is shared but is only read.
6.2 Memory Read and Memory Write Operations for VAX and Alpha
This section presents the important concepts of
alignment and granularity and how
they affect the access of shared data on VAX and Alpha systems. It also
discusses the importance of the order of reads and writes completed on
VAX and Alpha systems, and how VAX and Alpha systems perform memory
reads and writes.
6.2.1 Accessing Memory
The term alignment refers to the placement of a data item in memory. For a data item to be naturally aligned, its lowest-addressed byte must reside at an address that is a multiple of the size of the data item in bytes. For example, a naturally aligned longword has an address that is a multiple of 4. The term naturally aligned is usually shortened to "aligned."
On VAX systems, a thread on a VAX uniprocessor or multiprocessor can read and write aligned byte, word, and longword data atomically with respect to other threads of execution accessing the same data.
In contrast to the variety of memory accesses allowed on VAX systems, an Alpha processor may allow atomic access only to an aligned longword or an aligned quadword. Reading or writing an aligned longword or quadword of memory is atomic with respect to any other thread of execution on the same processor or on other processors. Newer Alpha processors with the byte-word extension also provide atomic access to bytes and aligned words.
VAX and Alpha systems differ in granularity of data access. The phrase granularity of data access refers to the size of neighboring units of memory that can be written independently and atomically by multiple processors. Regardless of the order in which the two units are written, the results must be identical.
VAX systems have byte granularity: individual adjacent or neighboring bytes within the same longword can be written by multiple threads of execution on one or more processors, as can aligned words and longwords.
VAX systems provide instructions that can manipulate byte-sized and aligned word-sized memory data in a single, noninterruptible operation. On VAX systems, a byte-sized or word-sized data item that is shared can be manipulated individually.
Alpha systems guarantee longword and quadword granularity. That is, adjacent aligned longwords or quadwords can be written independently. Because earlier Alpha systems support instructions that load or store only longword-sized and quadword-sized memory data, the manipulation of byte-sized and word-sized data on such Alpha systems may require that the entire longword or quadword containing the byte- or word-sized item be manipulated. Thus, simply because of its proximity to an explicitly shared data item, neighboring data might become shared unintentionally.
Manipulation of byte-sized and word-sized data on such Alpha systems requires multiple instructions that:
On such Alpha systems, because this sequence is interruptible, operations on byte and word data are not atomic. Also, this change in the granularity of memory access can affect the determination of which data is actually shared when a byte or word is accessed.
On such Alpha systems, the absence of byte and word granularity has important implications for access to shared data. In effect, any memory write of a data item other than an aligned longword or quadword must be done as a multiple-instruction read-modify-write sequence. Also, because the amount of data read and written is an entire longword or quadword, programmers must ensure that all accesses to fields within the longword or quadword are synchronized with each other.
Alpha systems with the byte-word extension provide instructions that
can read or write byte-size and aligned word-sized memory data in a
single noninterruptible operation.
6.2.2 Ordering of Read and Write Operations
On VAX uniprocessor and multiprocessor systems, write operations and read operations appear to occur in the same order in which you specify them from the viewpoint of all types of external threads of execution. Alpha uniprocessor systems also guarantee that read and write operations appear ordered for multiple threads of execution running within a single process or within multiple processes running on a uniprocessor.
On Alpha multiprocessor systems, you must order reads and writes
explicitly to ensure that they occur in a specific order from the
viewpoint of threads of execution on other processors. To provide the
necessary operating system primitives and compatibility with VAX
systems, Alpha systems provide instructions that impose an order on
read and write operations.
6.2.3 Memory Reads and Memory Writes
On VAX systems, most instructions that read or write memory are noninterruptible. A memory write done with a noninterruptible instruction is atomic from the viewpoint of other threads on the same CPU.
On VAX systems, on a uniprocessor system, reads and writes of bytes, words, longwords, and quadwords are atomic with respect to any thread on the processor. On a multiprocessor, not all of those accesses are atomic with respect to any thread on any processor; only reads and writes of bytes, aligned words, and aligned longwords are atomic. Accessing unaligned data can require multiple operations. As a result, even though an unaligned longword is written with a noninterruptible instruction, if it requires multiple memory accesses, a thread on another CPU might see memory in an intermediate state. VAX systems do not guarantee multiprocessor atomic access to quadwords.
On Alpha systems, there is no instruction that performs multiple memory accesses. Each load or store instruction performs a maximum of one load from or one store to memory. On an Alpha processor without the byte-word extension, a load can occur only from an aligned longword or quadword; a store can occur only to an aligned longword or quadword. On an Alpha processor with the byte-word extension, a load can also occur from a byte or an aligned word; a store can also occur to a byte or an aligned word.
On Alpha systems, although reads and writes from one thread appear to occur ordered from the viewpoint of other threads on the same processor, there is no implicit ordering of reads and writes as seen by threads on other processors.