HP OpenVMS Guide to System Security > Chapter 2 OpenVMS Security Model

Implementation of the Reference Monitor

  Table of Contents



While the OpenVMS operating system does not implement the reference monitor as a security-related subset, or security kernel, its interface to users and system managers does mirror the basic structure dictated by the reference monitor concept. Experience shows that incorporating such a structure is the best way to build a system resistant to probing and to most attempts at penetration.

The following sections describe the OpenVMS operating system's implementation of the reference monitor model.


Subjects are the users or user agents (the user processes) that access information and, in some cases, may be prevented from accessing information. Subjects can be interactive processes, network processes, or batch jobs, and:

  • Must pass security controls

    During process creation
    During information access
  • Require identification

    User names
    User identification codes
    Rights identifiers

When a user logs in to use the operating system interactively or when a batch or network job starts, the operating system creates a process that includes the identity of the user. That process gains access to information as the agent for the user, as described in Chapter 4 “Protecting Data”.

Processes are vulnerable to security breaches while they are being created and while they are accessing information. The system manages process access to information by using its authorization data and internal mechanisms, such as hardware controls. Because process creation has many areas of security vulnerability, many operating security features concentrate on the area of process (or subject) creation.

When a user attempts to log in to a system, the user provides a user name (a name that will be given to the resulting process) and a password. The password serves as an authenticator that should be known only to the user and to the operating system.

Because a short or obvious password is likely to fail this requirement, the system incorporates many password protection mechanisms that can be invoked by the user or required by the security administrator (see Chapter 7 “Managing System Access”). The operating system is also capable of limiting the number of attempts that an intruder can make to guess a password.

The file of users' passwords is part of the security database that must be protected from unauthorized observation and modification. The system meets this requirement by storing the passwords in a file protected from general access, the system user authorization file (SYSUAF.DAT). The system takes the additional precaution of storing passwords in an encoded form that is hard to use if stolen.

Once the operating system creates a process for a user, it assigns a user identification code or UIC from the user authorization record to that process. The UIC corresponds to the name of the user who created the process (as authenticated by the user's password). In addition, the UIC indicates the user's membership in a group that can correspond to the user's department, project, or function. The system can also attach additional information to the process regarding the creation of the process and the affiliation of the process owner with other groups. This additional information plays a part in the application of the authorization database. (Both Chapter 4 “Protecting Data” and Chapter 8 “Controlling Access to System Data and Resources” discuss UICs.)


In the reference monitor concept, objects are passive repositories of information. In the OpenVMS system, there are many objects, such as files and devices, that are subject to protection, as described in Table 2-1 “Objects Protected by Security Controls”. The operating system protects objects from unauthorized access and provides a variety of mechanisms (described in Chapter 4 “Protecting Data” and Chapter 5 “Descriptions of Object Classes” ) for sharing them in a controlled manner. These mechanisms are also used to control access to system resources.

Table 2-1 Objects Protected by Security Controls

Class Name Definition


A resource to which the system controls access; currently, the only defined capability is the vector processor.

Common event flag cluster

A set of 32 event flags that enable cooperating processes to post event notifications to each other.


A class of peripherals connected to a processor that are capable of receiving, storing, or transmitting data.


Files-11 On-Disk Structure Level 2 or 5 (ODS-2 or ODS-5) files and directories.

Group global section

A shareable memory section potentially available to all processes in the same group.

Logical name table

A shareable table of logical names and their equivalence names for the system or a particular group.


A set of jobs to be processed in a batch, terminal, server, or print job queue.

Resource domain

A namespace controlling access to the lock manager's resources.

Security class

A data structure containing the elements and management routines for all members of the security class.

System global section

A shareable memory section potentially available to all processes in the system.


A mass storage medium, such as a disk or tape, that is in ODS-2 or ODS-5 format. Volumes contain files and may be mounted on devices.


Authorization Database

According to the reference monitor model, each subject's authorization to gain access to an object is based on an abstract authorization database. This database is a set of dynamic security attributes that govern a subject's access to an object at any given time. In the OpenVMS system, the database is distributed and stored in association with the objects that must be protected. For example, the authorization data for a file or directory is stored in the file header for that file or directory. The following table summarizes the information stored in the authorization database.

File Contents Data Used to Interpret


User names







Access control checks


User names



User names



Rights identifiers

Access control checks



Access control checks


Protection codes

Access control checks


Access control lists

Access control checks


Auditable events

Reporting of events

As “Objects” suggests, different objects in the OpenVMS system can be shared with differing levels of flexibility. Protected objects are subject to a protection code. This code specifies whether access is allowed or denied to processes run on behalf of system users, the user who is owner of the object, other members of the UIC group of the owner, and all other users.

In addition to the protection code, objects can be shared under control of access control lists (ACLs). ACLs provide a finer granularity of access control than UIC-based protection, especially for user groups or subsets of groups. ACLs list individual users or groups of users who are to be allowed or denied particular types of access to the object. ACLs specify sharing on the basis of UIC identification as well as other groupings or identifiers that can be associated with a process. For example, it is possible to specify that a file should never be read by a process connected to a terminal on a dialup line. “Authorization Database Represented as an Access Matrix” uses an access matrix to explain the concept of an ACL. “Controlling Access with ACLs” gives a general discussion of ACLs and identifiers, and Chapter 8 “Controlling Access to System Data and Resources” explains how you, as security administrators, can create identifiers and construct ACLs for system resources.

Audit Trail

All security-relevant events can be recorded in an audit log file, sent to an operator terminal, or both. A terminal can be designated as a security operator terminal where all auditable events can be displayed. An audit log file provides a permanent record of security events. Many times a security administrator can find a pattern of activity, called an audit trail, by studying the log file.

The operating system audits the classes of security events shown in Table 2-2 “Security Auditing Overview” by default. You can select other events for auditing, such as volume mounts or changes to system time.

Table 2-2 Security Auditing Overview

Destination Events Audited by Default

Log file or terminal display

Authorization database changes


Intrusion attempts


Login failures

Use of DCL command SET AUDIT

Events triggered by Audit or Alarm ACEs


The audit log allows users and security administrators to record many events. Because it is time-consuming to examine every event, it is most efficient to audit events that will contribute the most information to your security picture. See Chapter 9 “Security Auditing” for a description of security auditing.

Reference Monitor

In the OpenVMS operating system, the executive performs the role of the reference monitor. All system programs that run in kernel and executive mode help implement the reference monitor, as do the command line interpreter and certain user-mode images that run with privilege. While the volume of code comprising the executive is large, HP attempts to ensure that none of the code can be used to bypass system security.

Some privileges can grant a user the authority to modify or subvert the reference monitor. For example, a process with the CMKRNL privilege can execute code of its own within the system kernel, gaining access to the reference monitor's internal data and the internal representation of protected objects. Clearly, granting such critical privileges should be severely limited.

Similarly, give privileges such as SYSPRV and SECURITY only to users whose processes help maintain the reference monitor and authorization database.

Authorization Database Represented as an Access Matrix

The reference monitor model specifies an authorization database, which describes all access authorizations in the system for all subjects and all objects. This database is often represented as an access matrix, which lists subjects on one axis and objects on the other (see Figure 2-2 “Authorization Access Matrix”). Each crosspoint in the matrix thus represents the access that one subject has to one object.

Figure 2-2 Authorization Access Matrix

Authorization Access Matrix

In this access matrix, an asterisk (*) denotes that the subject has access to that object. (Different types of access, such as read and write, are omitted from this example for simplicity.) Thus, subjects B, C, and D all have access to objects W, X, and Y. In addition, subject A has access to objects W and Z, subject D to object V, and subject E to object V.

Breaking up the access matrix by rows yields a capability-based model, in which each subject carries a list of the objects that it can access. Thus, a capability representation of this access matrix would appear as follows:

A: W, Z B: W, X, Y C: W, X, Y D: V, W, X, Y E: V

It is also possible to break up the access matrix by columns, listing for each object the subjects that have access to it. This results in an authority-based model, implemented in the OpenVMS system by ACLs (see Chapter 4 “Protecting Data”). The ACL representation appears as follows:

V: D, E W: A, B, C, D X: B, C, D Y: B, C, D Z: A

The ACL and identifier controls used by the operating system combine the properties of both the capability- and authority-based systems. In OpenVMS systems, both subjects and objects carry identifiers. Subjects can access objects if they have matching identifiers and if the objects' access statements grant the requested access.

The result of combining properties of the capability- and authority-based systems is an extremely powerful and flexible system capable of representing complex access matrixes in a compact and convenient manner. Consider what happens to the previous example of an access matrix when some of the cross-points have labels, as shown in Figure 2-3 “Authorization Access Matrix with Labeled Cross-Points”.

Figure 2-3 Authorization Access Matrix with Labeled Cross-Points

Authorization Access Matrix with Labeled Cross-Points

Some labeled cross-points can be grouped and treated as a single entity. Thus, the points that are labeled Q in Figure 2-3 “Authorization Access Matrix with Labeled Cross-Points” represent the access that subjects B, C, and D have to objects W, X, and Y. All the Q points can be considered as a single area of interest. The system provides the concept of identifiers to take practical advantage of this grouping of areas of interest.

You can define identifiers to represent the two groups of access, P and Q, in Figure 2-3 “Authorization Access Matrix with Labeled Cross-Points”. Note that two of the cross-points in the matrix remain unlabeled. Identifiers can also represent individual subjects and thus allow the traditional ACL facility.

To represent the access matrix, the OpenVMS operating system uses two structures, one for each dimension:

  • The rights list (RIGHTSLIST.DAT) represents the rows of the access matrix and thus corresponds to the capability-based model. For the matrix in Figure 2-3 “Authorization Access Matrix with Labeled Cross-Points”, you would need the following rights list:

    B: Q C: Q D: P, Q E: P

  • ACLs for the protected objects represent the columns of the access matrix. For this example, you would need the following ACLs:

    V: P W: A, Q X: Q Y: Q Z: A

Note that the system structures required to represent the access matrix are simpler than either the traditional capability- or authority-based model and require fewer terms in total. In the example, the difference is slight. However, complexity of the access matrix increases with the square of its size.