13    Managing CPU Performance

You can improve system performance by optimizing CPU resources. This chapter describes how to perform the following tasks:

• Obtain information about CPU performance by using the CPU monitoring tools (Section 13.1)

• Improve CPU performance by adding processors or running the class scheduler to control the execution of tasks and applications. (Section 13.2)

13.1    Monitoring CPU Performance Information

Table 13-1 describes the tools you can use to gather information about CPU usage.

Table 13-1:  CPU Monitoring Tools

The following sections describe the ps, vmstat, uptime, cpustat, and lockstats commands in detail.

13.1.1    Monitoring the Load Average by Using the uptime Command

The uptime command shows how long a system has been running and the load average. The load average counts the jobs that are waiting for disk I/O, and applications whose priorities have been changed with either the nice or the renice command. The load average numbers give the average number of jobs in the run queue for the last 5 seconds, the last 30 seconds, and the last 60 seconds.

An example of the uptime command is as follows:

# /usr/ucb/uptime
1:48pm  up 7 days,  1:07,  35 users,  load average: 7.12, 10.33, 10.31

The command output displays the current time, the amount of time since the system was last started, the number of users logged into the system, and the load averages for the last 5 seconds, the last 30 seconds, and the last 60 seconds.

From the command output, you can determine whether the load is increasing or decreasing. An acceptable load average depends on your type of system and how it is being used. In general, for a large system, a load of 10 is high, and a load of 3 is low. Workstations should have a load of 1 or 2.

If the load is high, look at what processes are running with the ps command. You may want to run some applications during offpeak hours. See Section 12.3.2 for information about the ps command.

You can also lower the priority of applications with the nice or renice command to conserve CPU cycles. See nice(1) and renice(8) for more information.

13.1.2    Checking CPU Usage by Using the kdbx Debugger cpustat Extension

The kdbx debugger cpustat extension displays CPU statistics, including the percentages of time the CPU spends in the following states:

• Running user-level code

• Running system-level code

• Running at a priority set with the nice function

• Idle

• Waiting (idle with input or output pending)

The cpustat extension to the kdbx debugger can help application developers determine how effectively they are achieving parallelism across the system.

By default, the kdbx cpustat extension displays statistics for all CPUs in the system. For example:

# /usr/bin/kdbx -k /vmunix /dev/mem 
(kdbx)cpustat
 Cpu   User (%)    Nice (%) System (%)  Idle (%)   Wait (%)
===== ========== ========== ========== ========== ==========
    0       0.23       0.00       0.08      99.64       0.05
    1       0.21       0.00       0.06      99.68       0.05

See the Kernel Debugging manual and kdbx(8) for more information.

13.1.3    Checking Lock Usage by Using the kdbx Debugger lockstat Extension

The kdbx debugger lockstats extension displays lock statistics for each lock class on each CPU in the system, including the following information:

• Address of the structure

• Class of the lock for which lock statistics are being recorded

• CPU for which the lock statistics are being recorded

• Number of instances of the lock

• Number of times that processes have tried to get the lock

• Number of times that processes have tried to get the lock and missed

• Percentage of time that processes miss the lock

• Total time that processes have spent waiting for the lock

• Maximum amount of time that a single process has waited for the lock

• Minimum amount of time that a single process has waited for the lock

For example:

# /usr/bin/kdbx -k /vmunix /dev/mem 
(kdbx)lockstats

See the Kernel Debugging manual and kdbx(8) for more information.

13.2    Improving CPU Performance

A system must be able to efficiently allocate the available CPU cycles among competing processes to meet the performance needs of users and applications. You can improve performance by optimizing CPU usage.

Table 13-2 describes the guidelines to improve CPU performance.

Table 13-2:  Primary CPU Performance Improvement Guidelines

The following sections describe how to optimize your CPU resources. If optimizing CPU resources does not solve the performance problem, you may have to upgrade your CPU to a faster processor.

13.2.1    Adding Processors

Multiprocessing systems allow you to expand the computing power of a system by adding processors. Workloads that benefit most from multiprocessing have multiple processes or multiple threads of execution that can run concurrently, such as database management system (DBMS) servers, Internet servers, mail servers, and compute servers.

You may be able to improve the performance of a multiprocessing system that has only a small percentage of idle time by adding processors. See Section 12.3.1 for information about checking idle time.

Before you add processors, you must ensure that a performance problem is not caused by the virtual memory or I/O subsystems. For example, increasing the number of processors will not improve performance in a system that lacks sufficient memory resources.

In addition, increasing the number of processors may increase the demands on your I/O and memory subsystems and could cause bottlenecks.

If you add processors and your system is metadata-intensive (that is, it opens large numbers of small files and accesses them repeatedly), you can improve the performance of synchronous write operations by using a RAID controller with a write-back cache (see Section 9.4).

13.2.2    Using the Class Scheduler

The class scheduler provides you with a method of controlling the execution of tasks or applications by restricting the length of time that they can access the processor (CPU). For example, daemons such as the print spooler are given less access time. The CPU will then have more time available to perform other tasks. To do this, you specify that the print daemon /usr/lbin/lpd is allowed to use no more than a certain percentage of the available CPU time. You can group resource user identifiers, such as a user's UID (user identification), into classes and assign the required CPU access time to each class.

This feature can help you to allocate system resources so that the most important work receives the required processing time. For example, you might want to run two versions of a production database on your system. One version is used as part of your business operations, while the other is a test copy, with different tuning parameters. You can assign the test database to a different class to prevent it from affecting your daily operations.

To set up and use the class scheduler, you must complete the following steps:

1. Plan the allocation of CPU resources.

2. Use class_admin to set up and maintain the class database.

3. Create classes and add members to the classes.

4. Verify class entries using the show command.

5. Save the entries to the database.

6. Enable class scheduling to start the daemon.

Use the class scheduler commands to monitor and control scheduling as follows:

• Execute class_admin commands such as stat from the command line or a shell script without running an interactive session

• Use the runclass command to execute a task according to the priorities set for a particular class

The following sections suggest a systematic approach to using class scheduling, although it is not necessary to perform tasks in a specific sequence. There are two methods of accessing the class scheduler:

• Manual

  By executing class_admin commands from the command line to configure a default database, add classes and class members, and enable the class scheduling daemon to create a quick fix to a CPU resource sharing problem.

• Graphical Interface

  By using the graphical user interface available as a SysMan Menu suboption, Class Scheduling, which is available under the Monitoring and Tuning menu option.

  See the System Administration manual for information on running the SysMan Menu. Section 13.2.2.6 describes how you use the graphical interface. See the online help for additional information on valid data entries.

The following reference pages contain detailed information on using the class scheduler commands and options:

• class_scheduling(4)

• class_admin(8)

• runclass(1)

• sysman(8)

The following command displays online help for the class_admin command:

# /usr/sbin/class_admin help
 
 

13.2.2.1    Class Scheduler Overview

To use the class scheduler, you must first create a database file and populate the file with one or more classes. Each class is assigned a CPU value that controls its access to processing time, expressed as a percentage of the total CPU time availability. You can assign one or more applications or groups of applications to a class, identified according to a unique system process identifier such as:

• UID — User identifier, a unique number assigned to each user account (login)

• GID — Group identifier, a number or name assigned to several user accounts to indicate that they belong to the same group

• PID — Process identifier, a system-assigned number that is unique to each process

• PGID — Process group identifier, a system-assigned number that is unique to each process group

• SESS — Session identifier, a system-assigned number that is unique to each session

The PID, PGID, and SESS identifiers are usually temporary and do not persist across a reboot, ceasing to exist when a task is completed. They are not stored in the database and have no effect when the system or task is restarted.

After the database is established, you can enable class scheduling. This operation starts a class scheduling daemon and puts the CPU access restrictions into effect. Other commands enable you to review classes, change contents or scheduling parameters, and delete components or entire classes. When a class scheduling database is configured and enabled, you can:

• Use runclass to execute a task (process) according to the CPU access value set for a specific class. For example, you might set a value for interactive operations that is much higher than background processes such as print daemons. To temporarily use the higher value for a print job, you can execute the lpr command in the same class as interactive operations.

• Use the class_admin command to execute class scheduling commands from within scripts.

13.2.2.1.1    Related Utilities

The following utilities are also available for use when monitoring and tuning processes:

• The nice command

• The Process Tuner (dxproctuner) graphical interface, available from the CDE MonitoringTuning folder in the Application Manager - System_Admin

• You can invoke the iostat and vmstat commands from the SysMan Menu

13.2.2.1.2    Invoking the Class Scheduler

The class scheduler is provided as both a command-line interface and a graphical user interface. You can invoke the class scheduler in several ways, depending on which user environment you are working from:

• From the SysMan Menu, select the Monitoring and Tuning branch, then select the Class Scheduling task.

• From the command line, enter either of the following commands:

  # sysman class_sched
  

  # sysman -menu "Class Scheduling"
  

• From CDE (assuming your system is running a graphics environment with CDE), follow these steps:

  1. Select the Application Manager from the CDE front panel.

  2. Select the System_Admin Software Management Group.

  3. Select the Configuration Software Management Group.

  4. Select the class scheduler icon.

The following sections focus on using the command-line method, and provide a brief introduction to using the graphical interface. See the online help for more information in using the graphical interface.

13.2.2.2    Planning Class Scheduling

How you allocate CPU resources depends on your system environment and what resources and priorities must be considered. A typical scenario is to assign a higher CPU percentage to interactive tasks so that users do not encounter long response times. Most batch or background processes will be assigned a lower CPU percentage, while some specific background processes might require a higher CPU percentage. For example, if a nightly backup is being performed, you might not want it to have such a low CPU percentage that it does not complete in a reasonable time.

If your system is involved with critical real-time tasks that must take precedence over interactive processes, your course of action might be different. In such cases you should design a baseline that assigns processes to classes. You can then monitor processes and gather user feedback to tune the database by moving tasks from class to class or by changing the CPU access time of the classes.

Do not use the root account to create test processes when you configure class scheduling. Root account processes always take precedence over others, even when assigned to an existing restricted class.

13.2.2.3    Configuring Class Scheduling

Use the class_admin command to configure an initial database. This command provides:

• An interactive command (with subcommands) that enables you to create and administer a database of classes. The database is stored in the binary file /etc/class, which cannot be manually edited. Type help at the class> command prompt for a list of options.

• A command mode that allows you to execute class_admin commands at the command prompt, or include commands in shell scripts.

A database must be configured before you can enable class scheduling with the enable command. If a database does not exist when you enter the class_admin command, the command will invoke an interactive session and prompt you to automatically configure a database. If a script invokes the class_admin command, it uses the system defaults to configure the database.

The following example shows an interactive configuration session using class_admin. In the actual output, the lines will be formatted to fit in 80 columns.

# /usr/sbin/class_admin
                    Class Scheduler Administration
 
configure:
 
Shall processes that have not been explicitly
assigned to a defined class be assigned to a
'default' class?  Enter (yes/no) [no]: yes
 
Enforce class scheduling when the CPU is otherwise
idle? (yes/no) [no]: yes
 
How often do you want the system to reset class usage?
Enter number of seconds (1): 2
class>

The configuration values have the following effect:

• To be scheduled, a process must be assigned to a class. If you answer yes to the first prompt, a special class called the default class is created. Any process that is not explicitly assigned to a defined class will be assigned to the default class.

  If you answer no to this prompt, then only those processes that are explicitly assigned to a defined class will be class scheduled.

• If you answer yes to the second prompt, you allow classes to exceed their allotted CPU time percentage when the system is otherwise idle. If you answer no, classes are restricted to their allotted percentage even if the CPU has no other work.

• The third prompt allows you to set the standard reset time for all classes. For example, if you choose the short default time of 1 second, each class will have more frequent, but shorter, opportunities to access the CPU.

  Use a small number (several seconds) if there are interactive jobs subject to class scheduling to give them a quick response time. If only batch jobs are class scheduled, response time is not an issue and you can specify larger values.

In the example, a default class was created and all current processes were assigned to that class. Class scheduling will be enforced even when the CPU is idle and class usage will be reset every five seconds.

To review the current configuration, enter the show command:

class> show
Configuration:
 -Processes not explicitly defined in the database are
  class scheduled.
 -If the processor has some idle time, class scheduled
  processes are not allowed to exceed their cpu percentage.
 -The class scheduler will check class CPU usage every 2
  seconds.
 
Class scheduler status: disabled  current database: /etc/class
 
Classes:
 
 default targeted at 100%:
    class members:
    Every one not listed below
 
 

The next step in the process is to create classes and populate the classes with system processes such as tasks, daemons, or user accounts by using the appropriate identifiers such as UID or SESS.

13.2.2.4    Creating and Managing Classes

When the database is configured, you can administer classes as follows:

• Create a class:

  • Add processes to the class

  • Delete processes from a class

• Change the CPU access value (time percentage) of any class

• Destroy an entire class, whether empty or populated

• Show details of class members and configuration settings

• View statistics of actual CPU use against current priority settings

Some of these options are described briefly in the following sections. For detailed descriptions of command options, see the online help and reference pages.

13.2.2.4.1    Creating a Class

To create a class, either use the command-mode or enter an interactive session as follows:

# class_admin
class> create high_users 50

The command-mode version is entered as follows:

# class_admin create batch_jobs 10
batch_jobs created at 10% cpu usage
 
 
changes saved

The first command creates a class named high_users and assigns a CPU usage restriction of 50 percent. The second command creates a class named batch_jobs and assigns a CPU usage restriction of 10 percent. In command mode, the changes are automatically saved to the database in /etc/class. When making changes to classes interactively, use the save command to commit changes to the database. If you attempt to end the session with the quit command and there are unsaved changes, you will be prompted to save or discard the changes before quitting the interactive session as follows:

class> quit
Class scheduler database modified.
Save changes? (yes/no) [yes]:yes
 
changes saved
 
 

13.2.2.4.2    Managing Identifier Types Within Classes

Unique system-assigned identifiers that the class scheduler recognizes (such as the PID, GID, or UID) identify which processes are members of a specific class. After you have created classes, you can add UIDs and GIDs or processes to one or more classes by using the add command. You must specify the type of identifier (ID) used and enter one or more unique identifiers. UIDs and GIDs can be determined from the /etc/passwd and /etc/group files. Alternatively, you can use the graphical interface Account Manager (dxaccounts) to display UID and Group information.

Process identifiers can be obtained from system files or by using a command such as ps. With the ps command, you can determine the values of PID, PGID, and SESS. Enter the following command to display the PID for every process running on the system:

# /sbin/ps aj
 
 
USER   PID PPID  PGID  SESS JOBC S    TTY          TIME COMMAND
walt  5176 5162  5176  2908   1  S    ttyp1    0:01.30 -sh (csh)
root 12603 5176 12603  2908   1  R  + ttyp1    0:00.05 ps aj

See ps(1) for more information and a definitive list of the process data items displayed when you use this command.

The following identifiers are supported:

gid

A group identification number from the /etc/group file. For example, if you are adding members to a class, using this number will add all the users that are assigned to the group.

uid

A user identification number from the /etc/passwd file. For example, if you are adding members to a class, this number will add only the specific user to which the UID is assigned.

pgrp

A process group identifier. In the output from the ps aj command, see the entries under the PGID table heading in the previous example.

session

A session identifier. In the output from the ps aj command, see the entries under the SESS table heading in the previous example.

pid

The process identifier. In the output from the ps aj command, see the entries under the PID table heading in the previous example.

It is most likely that you will use types uid and gid in your established classes, as these values will persist across a reboot or when class scheduling is stopped and restarted. You can use the account management tools, such as dxaccounts or the Accounts option of the SysMan Menu, to list UIDs and GIDs for users and groups. The identifiers associated with types pgrp, session, and pid are temporary, and will not exist on reboot or when a process terminates.

13.2.2.4.3    Enabling the Class Scheduler

To enable the class scheduler daemon, enter the following command:

# class_admin enable
Class scheduling enabled and daemon \
/usr/sbin/class_daemon started.

To disable the daemon, enter the following command:

# class_admin disable
Class scheduling disabled.

13.2.2.4.4    Adding Members to a Class

To add a process to a class, enter the add command as shown in the following interactive mode example:

class> add batch_jobs uid 234 457 235
 
 

You must use one of the unique identifiers previously specified, and you cannot add the same identifier to a class more than once. The same procedure can be performed in command mode or from a script as follows:

# class_admin add batch_jobs uid 234 457 235
uid 234 457 235 added to high_users

In command mode, additions to a class are automatically saved to the /etc/class database.

13.2.2.4.5    Deleting Members From a Class

To delete one or more processes from a class, enter the delete command in interactive or command mode. For example:

class> delete high_users uid 11
uid 11 deleted from high_users

This example deletes the single UID number 11 from class high_users.

13.2.2.4.6    Other Class Management Options

See class_admin(8) for information on the following options:

• Change the priority of a class. For example:

  class> change batch_jobs 20
  batch_jobs retargeted at 20%
  

• Destroy an entire class, whether empty or full. For example:

  class> destroy high_users
  high_users is not empty.
          to destroy anyway? [yes/no]:yes
          high_users destroyed
   
  

• Loading and saving scheduling databases. For example:

  class> load database_performance
  current database modified and not saved
  load new database anyway (destroys changes)? (yes/no) [yes]: \
  yes
  database database_performance loaded
  

  In this example the presence of unsaved modifications to the current database was detected, and the user was prompted to save the changes.

• View statistics of actual CPU use against current priority settings. For example:

  class> stats
     Class scheduler status: enabled
   
  class name   target percentage  actual percentage
    high_users      50%                40.0%
    batch_jobs      10%                 2.0%
   
  

13.2.2.5    Using the runclass Command

Once you have established scheduler classes and enabled class scheduling, you can use the runclass command to execute a command in a particular class. When you use the runclass command as root user (superuser), your processes have unrestricted access to CPU resources even when assigned to an existing class. By default, root processes are never restricted. This ensures that no user process can lock up resources needed by the root account. If you need to test a class scheduler configuration, ensure that you log in and create processes using a nonprivileged user account. You might want to set up dummy user accounts to perform such testing.

The following command uses the runclass command to open a terminal window and assign it to the previously created high_users class:

# runclass high_users xterm

The following command shows that the pgrp number for the terminal process is now identified as a member of that class:

# class_admin show
.
.
.
class members:
pgrp 24330     pgrp 24351     pgrp 24373

In this example, the identifier for the xterm process is added to the class. You can use the following command to view the running process:

# ps agx | grep xterm

See runclass(1) for more information.

13.2.2.6    Using the Class Scheduling Graphical Interface

The class scheduler can be launched from the SysMan menu by selecting the Class Scheduling option from the Monitoring and Tuning tasks. Alternatively, you can launch it from the Common Desktop Environment (CDE) Application Manager.

As for the command-line method of using the class scheduler described in preceding sections, the steps involved in initial configuration are as follows:

1. Plan your classes and the processes, users, or groups that will be in each class.

2. Configure and name a database by creating classes and adding them to the database.

3. Define the new database as the current database.

4. Start the class scheduling daemon.

You can complete these steps by using the SysMan Menu Class Scheduling main menu option, where the following three suboptions are available:

Configure Class Scheduler

This is the main option that you use to configure and initialize class scheduling. When you select this option, a window is displayed: Configure Class Scheduler on hostname. From here you can select one of the following options:

• Make Current... — Use this option to choose an existing database and make it the current database. When the system is first used, only the default database is available from the option list. This database is a placeholder and contains no classes. You can modify the default or create new databases, adding options to the list.

• New... — Use this option to create a new database and add it to the list of optional databases. A data entry window will be displayed for you to name the database and select or create classes.

• Copy... — Use this option to copy an existing database to a file so that you can use it as a starting point for a new database. You will be prompted to enter a file name and location for the copy.

• Modify... — Use this option to change the configuration of an existing database. If you want to preserve the original database before modifying it, use the Copy... option first.

• Delete — Use this option to remove databases from the option list. You will not be able to recover these databases once removed.

The New... option is the main option and the one most frequently used. It is described in detail in Section 13.2.2.7. The Modify... option provides an identical interface, which allows you to change existing classes and databases.

The remaining menu options require only a confirmation and do not involve extensive data entry. For example, if you want to delete a database, you will only be prompted to confirm that the database is to be destroyed.

[Re]Start Class Scheduler

Use this option to start the class scheduling daemon, or restart it if it was stopped. You will be prompted to confirm your selection.

Stop Class Scheduler

Use this option to stop the class scheduling daemon. You will be prompted to confirm your selection.

See the System Administration manual for information on running the SysMan Menu. Section 13.2.2.6 describes how you use the graphical interface. See the online help for additional information on valid data entries.

13.2.2.7    Creating or Modifying a Database

When you select the New... or Modify... options, a screen is displayed titled Configure Class Scheduler: Create/Modify Scheduling Database. To create a new database, follow these steps:

1. In the Name: field, type the name of the database that you want to create. The name should reflect the function of the database, so that you can easily recognize it when it is displayed in a list of many options. For example, served_applications.

2. From the option list titled Available Scheduled Classes, you can select any existing classes. If you are setting up the first database, no classes will be listed and only the New.. option will be available for selection.

3. To create a new class, press the New... button to display the window titled Create a new class. In this window, you complete the following steps:

   • Enter a name for the class in the Class name field. The name should enable you to easily recognize the members of the class. For example, principal_users.

   • Move the slider bar adjacent to the CPU allocation label to assign a value for the percentage of CPU time allocated to this class.

   • From the pull-down menu in the Member type field, select the type of identifier you will use to allocate processes to this class. Only the Group ID and User ID will persist across reboots. Session, Process group, and Process ID identifiers will not persist.

   • In the member field, enter the name of the user from the /etc/passwd file, a group from the /etc/group file, or a process identifier from the output of the following command:

     # /sbin/ps aj
     

   • Select the OK button to complete the class entry and return to the previous window, or select the Apply button to complete this entry and retain the window to create further classes. Use the Cancel button if you do not want to proceed with the creation of a class.

4. When classes are created, they appear as entries in the optional list of Available Scheduling Classes. Apart from the class name, the CPU time percentage allocation and member and type are also displayed. You can now select classes to add to the database as follows:

   • Click on a class to highlight it

   • Press the Select button to add the class to the database.

5. When all required classes are selected, press the OK button to create the new database. The new database will be added to the list of Available Scheduling Databases.

You also use the Configure Class Scheduler: Create/Modify Scheduling Database window to perform maintenance and administrative operations on classes as follows:

• Use the Copy... option to copy a class and use it as the base for a new class.

• Use the Modify... option to change characteristics of a class.

• Use the Delete... option to destroy a class and remove it permanently from the Available Scheduling Classes.

To begin using the newly created database, follow these steps:

1. If the window titled Configure Class Scheduler on hostname is not already displayed, invoke the SysMan Menu and select the Configure Class Scheduler option.

2. Highlight the required database by clicking on it, then press the Make Current... button. You will be prompted to confirm or cancel your choice.

3. Press the OK button to return to the SysMan Menu, Class Scheduling options, and select the option titled [Re]Start Class Scheduler. You will be prompted to confirm your choice.

On completing these steps, the class scheduling daemon starts and uses the scheduling database that you specified. To verify and monitor that the database is working as anticipated, use the show command at the terminal command line. For example, to view scheduling statistics, enter the following command:

# class_admin stats
 
Class scheduler status: enabled \
current database: /etc/.cl_lab1
 
class name     target percentage  actual percentage
prio-tasks-lab        10                  10

You might need to spend some time monitoring tasks and system performance, tuning your classes to obtain the required results.

13.2.3    Prioritizing Jobs

You can prioritize jobs so that important applications are run first. Use the nice command to specify the priority for a command. Use the renice command to change the priority of a running process.

See nice(1) and renice(8) for more information.

13.2.4    Scheduling Jobs at Offpeak Hours

You can schedule jobs so that they run at offpeak hours (use the at and cron commands) or when the load level permits (use the batch command). This can relieve the load on the CPU and the memory and disk I/O subsystems.

See at(1) and cron(8) for more information.

13.2.5    Stopping the advfsd Daemon

The advfsd daemon allows Simple Network Management Protocol (SNMP) clients such as Netview to request AdvFS file system information. If you are not using the AdvFS graphical user interface (GUI), you can free CPU resources and prevent the advfsd daemon from periodically scanning disks by stopping the advfsd daemon.

To prevent the advfsd daemon from starting at boot time, rename /sbin/rc3.d/S53advfsd to /sbin/rc3.d/T53advfsd.

To immediately stop the daemon, enter the following command:

# /sbin/init.d/advfsd stop

13.2.6    Using Hardware RAID to Relieve the CPU of I/O Overhead

RAID controllers can relieve the CPU of the disk I/O overhead, in addition to providing many disk I/O performance-enhancing features. See Section 9.4 for more information about hardware RAID.