Title and Copyright Information
 
About This Manual
Audience
New and Changed Features
Organization
Related Documents
Reader's Comments
Conventions
 
Part 1 -- Introduction to System Tuning
 
1    Introduction to System Tuning
1.1    Hardware Configuration
1.1.1    Hardware Configuration Overview
1.2    Performance Terminology and Concepts
1.3    Disk Storage Resources
1.3.1    RAID Technology
1.3.2    SCSI Concepts
1.3.2.1    Data Paths
1.3.2.2    SCSI Bus Speeds
1.3.2.3    Transmission Methods
1.3.2.4    Extending UltraSCSI Bus Segments
1.3.2.5    SCSI Bus Length and Termination
1.3.3    Fibre Channel
1.3.3.1    Fibre Channel Topologies
1.3.3.1.1    Point-to-Point Topology
1.3.3.1.2    Fabric Topology
1.3.3.1.3    Arbitrated Loop Topology
1.3.3.2    Fibre Channel Topology Comparison
1.3.3.3    Zoning
1.3.3.3.1    Switch Zoning Versus Selective Storage Presentation
1.3.3.3.2    Types of Zoning
1.3.3.3.3    Zoning Example
1.3.3.4    Cascaded Switches
1.4    Network Resources
1.4.1    Network Subsystem
1.4.2    Using Redundant Networks
1.4.3    NetRAIN
1.4.4    Routing
1.4.5    LAG Interface
1.5    File System Resources
1.5.1    Using AdvFS
1.5.1.1    Using the UBC
1.5.2    Using NFS
1.6    Memory Resources
1.6.1    Paging and Swapping
1.6.2    Caching Data
1.7    CPU Resources
1.8    Identifying a Resource Model for Your Workload
1.9    Most Commonly Tuned Subsystems
 
2    Gathering System and Performance Information
2.1    Methodology Approach to Solving Performance Problems
2.2    Obtaining Information About System Events
2.2.1    Using Event Manager
2.2.2    Using DECevent
2.2.3    Using Compaq Analyze
2.2.4    Using System Accounting and Disk Quotas
2.3    Primary Tools for Gathering Information
2.3.1    Gathering Hardware Information Using the hwmgr Utility
2.3.2    Gathering System Information by Using the collect Utility
2.3.2.1    Configuring collect to Automatically Start on System Reboot
2.3.2.2    Plotting collect Datafiles
2.3.3    Checking the Configuration by Using the sys_check Utility
2.4    Secondary Tools for Gathering Information
2.4.1    Gathering Locking Statistics by Using the lockinfo Utility
2.4.2    Gathering CPU Usage and Process Statistics by Using the sched_stat Utility
2.4.3    Displaying Network and NFS Statistics by Using the nfsstat Utility
2.4.4    Gathering Information by Using the tcpdump Utility
2.4.5    Monitoring Network Statistics by Using the netstat Command
2.4.5.1    Input and Output Errors and Collisions
2.4.5.2    Device Driver Errors
2.4.5.3    Memory Usage
2.4.5.4    Socket Connections
2.4.5.5    Dropped or Lost Packets
2.4.5.6    Retransmissions, Out-of-Order Packets, and Bad Checksums
2.4.5.7    Routing Statistics
2.4.5.8    Protocol Statistics
2.4.6    Gathering NFS Server Side Information Using ps axlmp
2.4.7    Gathering NFS Client Side Information Using nfsiod
2.4.8    Monitoring Incoming Network Traffic to an NFS Server by Using the nfswatch Command
2.5    Additional Tools for Monitoring Performance
2.6    Gathering Profiling and Debugging Information
 
3    Displaying and Modifying Kernel Subsystem Attributes
3.1    Operating System Support for Attributes
3.2    Displaying Attribute Values
3.3    Modifying Attribute Values
3.3.1    Current Value
3.3.2    Permanent Value
 
Part 2 -- Tuning by Application
 
4    Tuning Oracle
4.1    Monitoring Oracle Statistics
4.2    Improving the Performance of the gettimeofday( ) Function
4.3    Choosing and Enabling IPC Communication Protocols
4.4    Tuning Recommendations
4.4.1    Modifying Virtual Memory Attributes
4.4.1.1    Disabling Shared Memory
4.4.1.2    Allocating Shared Memory
4.4.1.2.1    Modifying the rad_gh_regions Attribute
4.4.1.2.2    Modifying the gh_chunks Attribute
4.4.1.3    Modifying the Percentage of Physical Memory the UBC is Using
4.4.1.4    Modifying the Percentage of Memory the UBC is Borrowing
4.4.1.5    Modifying the Percentage of Memory the UBC Can Use For a Single File
4.4.1.6    Modifying the UBC Threshold
4.4.1.7    Modifying the Percentage of Pages that Must be Dirty
4.4.1.8    Modifying the Swap Allocation Mode
4.4.2    Modifying the Advanced File System Attribute
4.4.3    Modifying the Virtual File System Attribute
4.4.4    Modifying Interprocess Communication Attributes
4.4.4.1    Modifying the System V Shared Regions
4.4.4.2    Modifying the System V Maximum Size of Shared Memory Region
4.4.4.3    Modifying the System V Minimum Size of Shared Memory Region
4.4.4.4    Modifying the Shared Memory Regions that Can be Used at One Time
4.4.4.5    Modifying the Shared Memory Regions that Can be Attached at One Time
4.4.5    Modifying Internet Attributes
4.4.5.1    Modifying the Send Buffer Size for the UDP Sockets
4.4.5.2    Modifying the Receive Buffer Size for the UDP Sockets
4.4.5.3    Modifying the Number of Times a System can make Outgoing Connections
4.4.6    Modifying Process Attributes
4.4.6.1    Modifying the Per Process Stack Size
4.4.6.2    Modifying the Maximum Size of the User Process Stack Size
4.4.6.3    Modifying the Per Process Data Size
4.4.6.4    Modifying the Maximum Size of the Per Process Data Size
4.4.6.5    Modifying the Per Process Address Size
4.4.6.6    Modifying the Maximum Per Process Address Size
4.4.6.7    Modifying the Maximum Number of Processes
4.4.6.8    Modifying the Maximum Number of Threads
4.4.6.9    Modifying the Space Allocated to System Tables
4.4.7    Modifying the Real-Time Attribute
4.4.8    Modifying Reliable Datagram Attributes
4.4.8.1    Modifying the Maximum Number of Objects in the RDG
4.4.8.2    Modifying the Maximum Size of the RDG Message
4.4.8.3    Modifying the Maximum Number of Messages in the RDG
4.4.8.4    Modifying the Maximum Number of Sessions within the RDG Table
4.4.8.5    Modifying the Maximum Number of Pages Wired For Message Packets
4.4.9    Modifying the Memory Channel Attribute
 
5    Tuning Network File Systems
5.1    Monitoring NFS Statistics
5.2    Detecting Poor NFS Performance
5.3    Performance Benefits and Tradeoffs
5.4    NFS Configuration
5.4.1    Configuring Server Threads
5.4.2    Configuring Client Threads
5.4.3    Modifying Cache Timeout Limits
5.5    NFS Retransmissions
5.5.1    Decreasing Network Timeouts
5.6    Tuning NFS Servers
5.6.1    Modifying NFS Server Side Attributes
5.6.1.1    Write Gathering
5.6.1.1.1    Improving NFS Server Response Time to Client Write Requests
5.6.1.2    Specifying the Amount of Time in Seconds the Server will Delay the Write
5.6.1.3    Increasing the NFS Send and Receive Buffer Size
5.7    Tuning NFS Clients
5.7.1    Modifying NFS Client Side Attributes
5.7.1.1    Improving Read Performance
5.7.1.2    Controlling How Long Before the Client will Start Transmitting
5.7.1.3    Directory Name Lookup Cache (DNLC)
5.7.1.4    Negative Name Cache Lookups (NNC)
5.7.1.5    Specifying File Consistency Across NFS Clients
5.7.1.6    Changing the NFS Client Behavior When Fetching File Attributes
 
6    Tuning Internet Servers
6.1    Improving Internet Server Performance
6.1.1    Configuring Hardware
6.1.2    Configuring Memory and Swap Space
6.1.3    Logging IP Addresses
6.1.4    Monitoring Network Statistics
6.1.5    Monitoring Socket Statistics
6.1.6    Monitoring Virtual Memory Statistics
6.1.7    Gathering Configuration Information
6.2    Primary Tuning Recommendations
6.2.1    Modifying Internet Attributes
6.2.1.1    Increasing the Size of the TCP Hash Table
6.2.1.2    Disabling PMTU Discovery
6.2.1.3    Increasing the Number of Outgoing Connection Ports
6.2.2    Modifying Process Attributes
6.2.2.1    Increasing the Size of System Tables and Data Structures
6.2.2.2    Increasing the Number of Processes per User
6.2.2.3    Increasing the Number of Threads per User
6.2.2.4    Increasing the User Process Data Segment Size Limits
6.2.2.5    Increasing the User Process Address Space Limits
6.2.3    Modifying Socket Attributes
6.2.3.1    Increasing the Maximum Number of Pending TCP Connections
6.2.3.2    Increasing the Minimum Number of Pending TCP Connections
6.2.3.3    Enabling the mbuf Cluster Compression
6.3    Advanced Tuning Recommendations
6.3.1    Modifying Generic Attributes
6.3.2    Modifying Internet Attributes
6.3.2.1    Increasing the Number of TCP Hash Table
6.3.2.2    Increasing the Number of Hash Buckets
6.3.2.3    Modifying the TCP Partial Connection Timeout Limit
6.3.2.4    Decreasing the Rate of TCP Retransmissions
6.3.2.5    Enabling TCP Keepalive Functionality
6.3.2.6    Increasing the TCP Connection Context Timeout Rate
6.3.2.7    Modifying the Range for Outgoing Connection Ports
6.3.2.8    Increasing the Number of IP Input Queues
6.3.2.9    Increasing the Maximum Length of the IP Input Queue
6.3.3    Modifying Network Attributes
6.3.3.1    Increasing the Number of Output Packets Before Packets are Dropped
6.3.3.2    Reducing Screening Cache Misses
6.3.3.3    Reducing the Screening Buffer Drops
6.3.4    Modifying Socket Attributes
6.3.5    Modifying Virtual Memory Attributes
 
7    Managing Application Performance
7.1    Improving Application Performance
7.1.1    Using the Latest Operating System Patches
7.1.2    Using the Latest Version of the Compiler
7.1.3    Using Parallelism
7.1.4    Optimizing Applications
7.1.5    Using Shared Libraries
7.1.6    Reducing Application Memory Requirements
7.1.7    Controlling Memory Locking
 
Part 3 -- Tuning by Component
 
8    Managing System Resource Allocation
8.1    Tuning Process Limits
8.1.1    Increasing System Tables and Data Structures
8.1.2    Increasing the Maximum Number of Processes
8.1.3    Increasing the Maximum Number of Threads
8.2    Tuning Program Size Limits
8.2.1    Increasing the Size of a User Process Stack
8.2.2    Increasing the Size of a User Process Data Segment
8.3    Tuning Address Space Limits
8.4    Tuning Interprocess Communication Limits
8.4.1    Increasing the Maximum Size of a System V Message
8.4.2    Increasing the Maximum Size of a System V Message Queue
8.4.3    Increasing the Maximum Number of Messages on a System V Queue
8.4.4    Increasing the Maximum Size of a System V Shared Memory Region
8.4.5    Increasing the Maximum Number of Shared Memory Regions Attached to a Process
8.4.6    Modifying Shared Page Table Sharing
8.5    Tuning the Open File Limits
8.5.1    Increasing the Maximum Number of Open Files
8.5.2    Increasing the Maximum Number of Open File Descriptors
8.6    Aurema ARMTech Suite
 
9    Managing Disk Storage Performance
9.1    Guidelines for Distributing the Disk I/O Load
9.2    Monitoring the Distribution of Disk I/O
9.2.1    Displaying Disk Usage by Using the iostat Command
9.3    Managing Storage with LSM
9.3.1    LSM Features
9.4    Managing Hardware RAID Subsystem Performance
9.4.1    Hardware RAID Features
9.4.2    Hardware RAID Products
9.4.3    Hardware RAID Configuration Guidelines
9.4.3.1    Distributing Storage Set Disks Across Buses
9.4.3.2    Using Disks with the Same Data Capacity
9.4.3.3    Choosing the Correct Hardware RAID Stripe Size
9.4.3.4    Mirroring Striped Sets
9.4.3.5    Using a Write-Back Cache
9.4.3.6    Using Dual-Redundant Controllers
9.4.3.7    Using Spare Disks to Replace Failed Disks
9.5    Managing CAM Performance
 
10    Managing Network Performance
10.1    Gathering Network Information
10.1.1    Checking Socket Listen Queue Statistics by Using the sysconfig Command
10.2    Tuning the Network Subsystem
10.2.1    Improving the Lookup Rate for TCP Control Blocks
10.2.2    Increasing the Number of TCP Hash Tables
10.2.3    Tuning the TCP Socket Listen Queue Limits
10.2.4    Increasing the Number of Outgoing Connection Ports
10.2.5    Modifying the Range of Outgoing Connection Ports
10.2.6    Disabling PMTU Discovery
10.2.7    Increasing the Number of IP Input Queues
10.2.8    Enabling mbuf Cluster Compression
10.2.9    Enabling TCP Keepalive Functionality
10.2.10    Improving the Lookup Rate for IP Addresses
10.2.11    Decreasing the TCP Partial-Connection Timeout Limit
10.2.12    Decreasing the TCP Connection Context Timeout Limit
10.2.13    Decreasing the TCP Retransmission Rate
10.2.14    Disabling Delaying the Acknowledgment of TCP Data
10.2.15    Increasing the Maximum TCP Segment Size
10.2.16    Increasing the Transmit and Receive Buffers for a UDP Socket
10.2.17    Increasing the Maximum Size of a Socket Buffer
10.2.18    Preventing Dropped Input Packets
 
11    Managing File System Performance
11.1    Tuning Caches
11.1.1    Monitoring Cache Statistics
11.1.2    Tuning the namei Cache
11.1.3    Tuning the UBC
11.1.4    Tuning the Metadata Buffer Cache
11.1.5    Tuning AdvFS Access Structures
11.2    Tuning AdvFS
11.2.1    AdvFS Configuration Guidelines
11.2.1.1    Storing Data Using RAID1 or RAID5
11.2.1.2    Forcing a Synchronous Write Request or Enabling Persistent Atomic Write Data Logging
11.2.1.3    Enabling Direct I/O
11.2.1.4    Using AdvFS to Distribute Files
11.2.1.5    Striping Data
11.2.1.6    Defragmenting a File Domain
11.2.1.7    Decreasing the I/O Transfer Size
11.2.1.8    Moving the Transaction Log
11.2.2    Monitoring AdvFS Statistics
11.2.2.1    Displaying AdvFS Performance Statistics
11.2.2.2    Displaying Disks in an AdvFS File Domain
11.2.2.3    Displaying AdvFS File Domains
11.2.2.4    Displaying AdvFS File Information
11.2.2.5    Displaying the AdvFS Filesets in a File Domain
11.2.3    Tuning AdvFS Queues
11.3    Tuning UFS
11.3.1    UFS Configuration Guidelines
11.3.1.1    Modifying the File System Fragment and Block Sizes
11.3.1.2    Reducing the Density of inodes
11.3.1.3    Set Rotational Delay
11.3.1.4    Increasing the Number of Blocks Combined for a Cluster
11.3.1.5    Using MFS
11.3.1.6    Using UFS Disk Quotas
11.3.1.7    Increasing the Number of UFS and MFS Mounts
11.3.2    Monitoring UFS Statistics
11.3.2.1    Displaying UFS Information
11.3.2.2    Monitoring UFS Clustering
11.3.2.3    Displaying the Metadata Buffer Cache
11.3.3    Tuning UFS for Performance
11.3.3.1    Adjusting UFS Smooth Sync and I/O Throttling
11.3.3.2    Delaying UFS Cluster Writing
11.3.3.3    Increasing the Number of Blocks in a Cluster
11.3.3.4    Defragmenting a File System
11.4    Tuning NFS
 
12    Managing Memory Performance
12.1    Virtual Memory Operation
12.1.1    Physical Page Tracking
12.1.2    File-System Buffer Cache Memory Allocation
12.1.2.1    Metadata Buffer Cache Memory Allocation
12.1.2.2    Unified Buffer Cache Memory Allocation
12.1.3    Process Memory Allocation
12.1.3.1    Process Virtual Address Space Allocation
12.1.3.2    Virtual Address to Physical Address Translation
12.1.3.3    Page Faults
12.1.4    Page Reclamation
12.1.4.1    Modified Page Prewriting
12.1.4.2    Reclaiming Memory by Paging
12.1.4.3    Reclaiming Memory by Swapping
12.2    Configuring Swap Space for High Performance
12.3    Monitoring Memory Statistics
12.3.1    Displaying Memory by Using the vmstat Command
12.3.2    Displaying Memory by Using the ps Command
12.3.3    Displaying Swap Space Usage by Using the swapon Command
12.3.4    Displaying the UBC by Using the dbx Debugger
12.4    Tuning to Provide More Memory to Processes
12.4.1    Reducing the Number of Processes Running Simultaneously
12.4.2    Reducing the Static Size of the Kernel
12.4.3    Increasing the Memory Reserved for Kernel malloc Allocations
12.5    Modifying Paging and Swapping Operations
12.5.1    Increasing the Paging Threshold
12.5.2    Managing the Rate of Swapping
12.5.3    Enabling Aggressive Task Swapping
12.5.4    Limiting the Resident Set Size to Avoid Swapping
12.5.5    Managing Modified Page Prewriting
12.5.6    Managing Page-In and Page-Out Clusters Sizes
12.5.7    Managing I/O Requests on the Swap Partition
12.6    Reserving Physical Memory for Shared Memory
12.6.1    Tuning the Kernel to Use Granularity Hints
12.6.2    Modifying Applications to Use Granularity Hints
12.7    Improving Performance with Big Pages
12.7.1    Using Big Pages
12.7.2    Determining when a Memory Object uses Big Pages
 
13    Managing CPU Performance
13.1    Monitoring CPU Performance Information
13.1.1    Monitoring the Load Average by Using the uptime Command
13.1.2    Checking CPU Usage by Using the kdbx Debugger cpustat Extension
13.1.3    Checking Lock Usage by Using the kdbx Debugger lockstat Extension
13.2    Improving CPU Performance
13.2.1    Adding Processors
13.2.2    Using the Class Scheduler
13.2.2.1    Class Scheduler Overview
13.2.2.1.1    Related Utilities
13.2.2.1.2    Invoking the Class Scheduler
13.2.2.2    Planning Class Scheduling
13.2.2.3    Configuring Class Scheduling
13.2.2.4    Creating and Managing Classes
13.2.2.4.1    Creating a Class
13.2.2.4.2    Managing Identifier Types Within Classes
13.2.2.4.3    Enabling the Class Scheduler
13.2.2.4.4    Adding Members to a Class
13.2.2.4.5    Deleting Members From a Class
13.2.2.4.6    Other Class Management Options
13.2.2.5    Using the runclass Command
13.2.2.6    Using the Class Scheduling Graphical Interface
13.2.2.7    Creating or Modifying a Database
13.2.3    Prioritizing Jobs
13.2.4    Scheduling Jobs at Offpeak Hours
13.2.5    Stopping the advfsd Daemon
13.2.6    Using Hardware RAID to Relieve the CPU of I/O Overhead
 
Glossary
 
Figures
1-1    Mapping Out Your Hardware Configuration
1-2    Point-to-Point Topology
1-3    Fabric Topology
1-4    Arbitrated Loop Topology
1-5    A Simple Zoned Configuration
1-6    Meshed Resilient Fabric with Four Cascaded Switches
1-7    Single Interface Configuration
1-8    Mulitple Interfaces
1-9    Physical Memory Usage
1-10    Moving Instructions and Data Through the Memory Hardware
11-1    Striping Data
11-2    AdvFS I/O Queues
12-1    UBC Memory Allocation
12-2    Memory Allocation During High File-System Activity and No Paging Activity
12-3    Memory Allocation During Low File-System Activity and High Paging Activity
12-4    Process Virtual Address Space Usage
12-5    Virtual-to-Physical Address Translation
12-6    Paging and Swapping Attributes
12-7    Paging Operation
 
Tables
1-1    RAID Level Performance and Availability Comparison
1-2    SCSI Bus Speeds
1-3    SCSI Bus and Segment Lengths
1-4    Fibre Channel Fabric and Arbitrated Loop Comparison
1-5    Type of Zoning Supported by Switches
1-6    Memory Management Hardware Resources
1-7    Resource Models and Possible Configuration Solutions
2-1    Tools for Continuous Performance Monitoring
2-2    Application Profiling and Debugging Tools
4-1    Tools to Detect Poor Oracle Application Performance
5-1    Tools to Detect Poor NFS Performance
5-2    Potential NFS Problems and Solutions
5-3    NFS Tuning Guidelines
5-4    NFS Server Tuning Guidelines
5-5    Identifying Your Network Card Type
6-1    Tools for Monitoring Network Statistics
7-1    Application Performance Improvement Guidelines
8-1    Default Values for the maxusers Attribute
8-2    IPC Limits Tuning Guidelines
9-1    Disk I/O Distribution Monitoring Tools
9-2    Hardware RAID Subsystem Configuration Guidelines
10-1    Network Monitoring Tools
10-2    Network Tuning Guidelines
11-1    Tools to Display Cache Information
11-2    When to Change the Values of the Namei Cache Related Attributes
11-3    When to Change the Values of the UBC-Related Attributes
11-4    AdvFS Configuration Guidelines
11-5    Tools to Display AdvFS Information
11-6    UFS Configuration Guidelines
11-7    Tools to Display UFS Information
11-8    UFS Tuning Guidelines
12-1    Default Values for vm_page_free_target Attribute
12-2    Tools to Display Virtual Memory and UBC
12-3    Memory Resource Tuning Guidelines
13-1    CPU Monitoring Tools
13-2    Primary CPU Performance Improvement Guidelines
 
Index