To use LSM most effectively, you should plan your LSM configuration before you use LSM volumes for applications or file systems. Your configuration will depend on the following:
The number, size, and performance characteristics of available disks and their connectivity to the host system or TruCluster Server cluster.
The number of disk groups you need and which disks you will configure in each disk group
The number of volumes you need and their availability and performance requirements
This chapter provides information and worksheets to assist you in planning
LSM volumes and disk groups.
You can make copies of the worksheets for future
use.
2.1 Planning LSM Volumes
Planning LSM volumes includes deciding which attributes you want the LSM volumes to have. An LSM volume has the following attributes:
Name (no default).
Size (no default). You can specify the volume size in sectors (the default unit), kilobytes, megabytes, gigabytes, or terabytes.
Plex layout type (concatenated, striped, or RAID5). Default: concatenated.
Number of plexes. Default: One concatenated data plex, no log plex.
Log plex size. Default: LSM calculates the appropriate log size based on the volume size.
For mirrored volumes, the default size for a DRL log plex is approximately 65 blocks per gigabyte of volume size for both standalone systems and clusters, to allow for migration of a disk group or a standalone system running LSM to a TruCluster Server environment.
The minimum DRL size is approximately 2 blocks per gigabyte of volume size. (You can use the minimum size if the LSM configuration will not be used in a cluster.)
For RAID 5 volumes, the default log plex size is large enough to store up through 10 full stripes of data and parity.
Disk group.
Default:
rootdg
disk group.
A volume can use disks from only one disk group.
Disks used in the volume. Default: LSM chooses the disks.
If the volume uses a striped or RAID5 plex, each column must be the same size and be on different disks, preferably on different buses.
If the volume has mirror plexes, each mirror should use disks on different buses, and the DRL plex should be on a disk that is not used for a data plex.
Usage type.
Default:
fsgen
.
Table 2-1
lists the benefits and trade-offs of
various plex layout types and describes situations where one plex type might
provide better performance, or be more cost effective, than a different plex
type.
For optimal performance you might need to tune your system to the work
load.
The layout you choose depends on your specific system configuration,
data availability and reliability needs, and application requirements.
Table 2-1: Plex Layout Type Benefits and Trade-Offs
Plex Type | Benefits and Applicability | Trade-Offs |
Concatenated | Uses space on multiple disks that might otherwise be wasted. Concatenated plex can be mirrored for data redundancy and high availability. Good for volumes containing infrequently used data or data that does not change often or small volumes that can be confined to a single disk. |
Possible uneven performance (hot spots, where one disk is in use by multiple applications). When mirrored, requires 2 through 32 times as much disk space. |
Striped | Distributes data and therefore I/O load evenly across many disks. Good for:
Striped plexes can be mirrored for data redundancy and high availability. |
When mirrored, requires 2 through 32 times as much disk space. |
RAID 5 | Provides the I/O distribution benefit of striping. Provides redundancy through parity, using fewer disks than a volume with mirrored striped plexes. Good for volumes with a high read-to-write ratio. |
Depending on the I/O stripe size, performance might be slower than a volume with striped plexes due to parity calculation. A RAID5 plex requires a minumum of three disks, plus a separate disk for the log plex. The RAID5 plex type is not supported in a cluster. |
When you create an LSM volume with the
volassist
command (the recommended and simplest method), LSM performs all the necessary
calculations and creates a volume and log plexes of the appropriate sizes.
Section 2.1.1
and
Section 2.1.2
discuss
the space requirements for Dirty Region Logs and RAID5 logs.
The worksheets in
Section 2.1.3,
Section 2.1.4,
Section 2.1.5
and
Section 2.2
are provided to help you
calculate the space needed and ensure the disk group has enough space for
the volumes you want.
2.1.1 Planning Dirty-Region Logs
By default, LSM configures a DRL log subdisk of approximately 65 blocks per gigabyte of volume size for both standalone systems and clusters so that the associated mirrored volume can be used in either a cluster or a standalone system. In a cluster, each member that issues I/O to the volume maintains its own private log to keep track of changes to the volume from that member. The volume's DRL log plex is an aggregate of every member's private log and must therefore be larger than on a standalone system.
If you plan to use LSM strictly on a standalone system, you can use the minimum DRL size, which is 2 blocks per gigabyte of volume size. However, if you create mirrored volumes with the smaller DRL size and then make those volumes available to a cluster (by using the standalone sytem to create a cluster or by moving the storage to an existing cluster), the DRL logs on the volumes will be disabled, but the volumes themselves will be usable.
Table 2-2
lists the approximate default DRL sizes
for various volume sizes.
Table 2-2: Default (Approximate) Sizes of DRL Logs
Volume Size (GB) | DRL Size (blocks) |
<=1 | 65 |
2 | 130 |
3 | 130 |
4 | 195 |
60-61 | 2015 |
62-63 | 2080 |
1021 | 33215 |
1022-1023 | 33280 |
1024 | 33345 |
We highly recommend using the default log size. The log is a bitmap that represents regions of the volume (each bit maps to a number of sectors), and the granularity of the bitmap changes if you specify a larger or smaller log size.
Specifying a larger log size results in smaller regions, like finer resolution in a photograph. Each bit represents a smaller number of volume sectors; for example, one bit per MB (2048 sectors). This means more log I/O overhead, but faster resynchronization after a system crash because fewer regions are resynchronized.
Specifying a smaller log size results in larger regions, like a grainy photograph. Each bit represents a larger number of volume sectors; for example, one bit per 100 MB (204800 sectors). This means less log I/O overhead, but slower resynchronization after a system crash, because more regions are resynchronized.
Using no log at all is like using a log that represents the entire volume (a one-to-one bitmap). When one write operation occurs, the entire volume is dirty and will undergo complete plex resynchronization after a system crash.
For best performance, each volume's DRL plex should be on a different disk from the log plexes for any other volumes. Using the same disk for the DRL plex for many mirrored volumes will slow down that disk and degrade volume performance, because the dirty bits are set before the write actually occurs. (If you are using hardware such as an HSG80 RAID Array, the cache manages and reduces much of the disk contention.)
In addition, if that disk fails, the logs for those volumes are lost.
If the system crashes before you replace that disk, those volumes will go
through complete plex resynchronization when the system restarts.
This is
a performance risk, but not a data loss risk.
2.1.2 Planning RAID 5 Logs
LSM calculates the default log size for a RAID5 volume
using a different algorithm than for mirrored volumes.
The RAID 5 log is large
enough to contain a copy of the full stripe width of data and parity for several
writes.
When you create a RAID5 volume using the
volassist
command, the RAID5 log size is calculated
for you and is sized appropriately to the volume.
The formula for the default RAID 5 log size:
[10 × (number of columns × stripe width)]
For example:
For a RAID 5 volume with 3 columns (the minimum) and a default stripe width of 16K bytes, the RAID5 log size is [10 × (3 × 16K bytes)], or 480K bytes.
For a RAID 5 volume with 5 columns and the same stripe width, the RAID5 log size is [10 × (5 × 16K bytes)], or 800K bytes.
Volumes that use a RAID5 plex are not supported in a cluster, so the RAID5 log size applies only to a standalone system.
For best performance, the RAID5 log plex should be
on its own disk.
Using the same disk for several log plexes (for different
volumes) will slow down that disk and degrade the performance of those volumes,
because the log is written before the data plex is written.
In addition, if
that disk fails, the logs for those volumes are lost.
If the system crashes
before you replace that disk, those volumes will go through complete parity
and data resynchronization when the system restarts.
This is both a performance
risk and a data loss risk.
2.1.3 Worksheet for LSM Volumes with Concatenated Plexes
Use the following worksheet to plan LSM volumes that use concatenated
plexes.
Figure 2-1: Worksheet for LSM Volumes with Concatenated Plexes
Attribute | Default Values | Chosen Values |
Volume name | | |
Volume size | | |
Number of data plexes | 1 | |
DRL plex size (for mirrored volumes) | 65 blocks per GB of volume size (default) or 2 blocks per GB of volume size (for standalone systems only) |
|
Number of DRL plexes | 1 | |
Disk group name | rootdg
|
|
Usage type | fsgen |
|
Total space required | (Volume size × number of plexes) + (DRL size × number of logs) |
2.1.4 Worksheet for LSM Volumes with Striped Plexes
Use the following worksheet to plan LSM volumes that use one or more
striped plexes.
Figure 2-2: Worksheet for LSM Volumes with Striped Plexes
Attribute | Default Values | Chosen Values |
Volume name | | |
Volume size | | |
Data unit size (stripe width) | 64K bytes | |
Number of columns | Minimum of two, based on number of disks in disk group and the volume size | |
Number of data plexes | 1 | |
DRL plex size (for mirrored volumes) | 65 blocks per GB of volume size (default) or 2 blocks per GB of volume size (for standalone systems only) |
|
Number of DRL plexes | 1 | |
Disk group name | rootdg
|
|
Usage type | fsgen |
|
Total space required | (Volume size × number of plexes) + (DRL size × number of logs) |
2.1.5 Worksheet for LSM Volumes with a RAID 5 Plex
Use the following worksheet to plan an LSM volume that uses a RAID 5 plex.
Note
RAID 5 volumes are not supported in a cluster.
Figure 2-3: Worksheet for LSM Volumes with a RAID 5 Plex
Attribute | Default Values | Chosen Values |
Volume name | | |
Volume size | | |
Data unit size (stripe width) | 16K bytes | |
Number of columns (NCOL) | Between 3 and 8 based on number of disks in disk group and the volume size | (Minimum of three) |
Log plex size | 10 × (NCOL × data unit size) | |
Number of logs (NLOGS) | 1 | |
Disk group name | rootdg
|
|
Usage type | Must be
raid5 |
raid5 |
Total space required | (Volume size × NCOL ÷ (NCOL-1)) + (log plex size × NLOGS) |
On every system or cluster running LSM, there will be one mandatory,
default disk group called
rootdg
.
You can create additional
disk groups to help you manage your storage and to support moving storage
(LSM volumes) to a different system.
The following sections provide recommendations
and explain restrictions for
rootdg
and other disk groups.
2.2.1 Guidelines for the Number and Usage of Disks
Each disk group (including
rootdg
) should have two
or more
sliced
or
simple
LSM disks to
ensure multiple copies of the disk group's configuration database.
Only
sliced
and
simple
disks have a private region,
which stores a copy of the disk group's comfiguration database.
By default,
LSM maintains a minimum of four active copies of the configuration database
in each disk group, distributed among several disks (and, if applicable, among
buses) to ensure redundancy.
All the copies should be on separate disks.
A
disk group cannot contain only
nopriv
disks.
The
rootdg
disk group stores information about all
disk groups in its configuration database.
Each time an LSM object is created,
modified, or removed, the configuration database for that disk group and for
rootdg
is updated.
Each LSM object (disk, subdisk, plex, and volume)
requires one record; two records fit in one sector (512 bytes).
To improve performance, keep the number of disks in
rootdg
to ten or fewer.
If possible, use
rootdg
only
for the following:
System-related volumes, such as volumes for the boot disk partitions and primary swap space on a standalone system.
Volumes for the clusterwide file system domains and the swap device volumes for cluster members.
Hot-spare disks for the preceding volumes that are mirrored.
Create additional disk groups if you have more than ten disks to place under LSM control or might move a disk group to a different system or cluster.
In each disk group, configure one or more hot-spare disks for each redundant
(mirrored or RAID5) volume in the disk group.
Each hot-spare
disk should be as large as the largest disk serving a volume.
2.2.2 Guidelines for Connectivity and Availability
A disk group should have storage devices on different buses to improve performance and availability.
Ideally, in a cluster, LSM should be used only with fully shared storage. Storage availability in a cluster increases as more cluster members have direct access to all disks in a disk group. Availability is highest when all disks in a disk group have connectivity with all cluster members.
The
drdmgr
and
hwmgr
commands
can give you information about which cluster members serve which disks.
For
a graphical display of the cluster hardware configuration, including active
members, buses, storage devices, and their connections, use the
sms
command to invoke the graphical interface for the SysMan Station,
and then choose Hardware from the Views menu.
In a cluster, the disks in each disk group should be accessible by all cluster members, so that all members have access to the LSM volumes, regardless of the state of any other member. LSM volumes have the same connectivity when either one of the following conditions is true:
All disks in the LSM disk group are on the same shared SCSI bus.
Disks in the LSM disk group are on different shared SCSI buses, but all of those buses are connected to the same cluster members.
Private disk groups (a disk group all of whose disks are connected to the private bus of a single cluster member) are supported, but if that member becomes unavailable, then the cluster loses access to the disk group. Private disk groups are suitable only when a cluster member that the disk group is physically connected to is also the only member that needs access to the disk group.
Avoid configuring a disk group with a combination of local and shared disks or a disk group whose disks are distributed among the private buses of multiple members. Such disk groups are not recommended, because no single member has direct access to all storage in the group.
It is especially important that the disks in
rootdg
are shared.
However, this might not always be possible.
For example, you can
encapsulate cluster members' swap devices to LSM volumes; all swap volumes
must belong to
rootdg
, but those devices might be private
to their respective member.
Avoid creating volumes that use a mix of shared
and private disks or a mix of private disks that are connected to different
members.
Section 2.2.3 contains a blank worksheet you can use to plan your disk groups. You can make copies and fill in the information on the copies rather than in this manual and keep the completed worksheet with each system running LSM, for your reference. Also, because you can change your LSM configuration at any time, you can make a new copy of the blank worksheet to record your changes.
Include in the worksheet any information that will help you keep track
of the purpose of that disk group.
For example, you might create a disk group
named
finance_dg
whose purpose is to contain one or more
volumes that will be used by a financial application.
You might create another
disk group named
db1
, which will contain a volume used
by a database.
Choose a name for the disk group carefully, as it is not easy
to rename it when it contains active volumes.
Under Volume, Plex and Hot-Spare Disk Information, include the names of all volumes in that disk group, their plex type, which disks belong to which plex, and any hot-spare disks. For information on hot-spare disks, see Section 3.5.
In addition to the information you include in the worksheet, you can
print and save the output from the
volprint
command, which
shows detail such as the subdisks used in each volume and the sizes of all
the LSM objects.
For more information, see
Section 5.4.1
and
volprint
(8)
Table 2-3
is an example of what your disk group
planning worksheets might look like when complete.
This example applies only
to a standalone system, not a cluster.
Table 2-3: Completed Disk Group Worksheets for a Standalone System
Disk Group Name:
rootdg
Root file
system and system disks. |
|||
Disks in Group | Bus/LUN | Disk Size | Volume, Plex, and Hot-Spare Disk Information |
dsk0 dsk1 dsk6 |
1 1 2 |
4 GB (all) | volumes:
rootvol-01(dsk0a)
|
dsk7 | 2 | 4 GB | hot-spare disk |
Disk Group Name:
data_dg
Database, must be redundant.
Contains volume with mirrored striped plexes and a DRL. |
|||
Disks in Group | Bus/LUN | Disk Size | Volume, Plex, and Hot-Spare Disk Information |
dsk2 dsk3 dsk8 dsk9 dsk12 |
1 1 2 2 3 |
18 GB (all) | volume:
plex: db_vol-01 (dsk2, dsk3) plex: db_vol-02 (dsk8, dsk9) plex: db_vol-03 (DRL plex, dsk12) |
dsk13 dsk20 |
3 4 |
18 GB (all) | hot-spare disks (each on different bus) |
Disk Group Name:
finance_dg
Financial application,
must be highly available.
Contains volume with a RAID 5 plex (read-only application). |
|||
Disks in Group | Bus/LUN | Disk Size | Volume, Plex, and Hot-Spare Disk Information |
dsk4 dsk10 dsk14 dsk18 dsk24 |
1 2 3 4 5 |
9 GB (all) | volume: fin_vol plex: fin_vol-01 (column 1 dsk4) plex: fin_vol-01 (column 2 dsk10) plex: fin_vol-01 (column 3 dsk14) plex: fin_vol-01 (column 4 dsk18) plex: fin_vol-02 (log plex dsk24) |
dsk5 dsk11 dsk15 dsk19 dsk25 |
1 2 3 4 5 |
9 GB (all) | hot-spare disks (one on each bus) |
2.2.3 Worksheet for Planning Disk Groups
You can make copies of the following worksheet for each disk group on
your system, including the
rootdg
disk group.
Table 2-4: Worksheet for Disk Group
Disk Group Name: | |||
Disks in Group | Bus/LUN Number | Disk Size | Volume, Plex, and Hot-Spare Disk Information |
2.3 Finding Unused Storage Devices
Unused storage devices are unused disks, partitions, and hardware RAID disks that LSM can initialize as LSM disks, as well as unused LSM disks, initialized for LSM use but not assigned to a disk group.
The following sections describe how to find unused disks, partitions, and LSM disks. See your hardware RAID documentation for information on finding unused hardware RAID disks.
To find unused storage devices, you can use:
The Disk Configuration Graphical User Interface (GUI) (Section 2.3.1).
The command-line interpreter interface (Section 2.3.2).
The
voldisk list
command on a system where
LSM is running (Section 2.3.3).
2.3.1 Using the Disk Configuration GUI to Find Unused Disks
To find unused disks using the Disk Configuration GUI, start the Disk Configuration interface using either of the following methods:
From the system prompt, enter:
# /usr/sbin/diskconfig
From the SysMan Applications pop-up menu on the CDE Front Panel:
Choose Configuration.
Double click the Disk icon in the SysMan Configuration folder.
A window titled Disk Configuration on hostname is displayed. The window lists the following information for each disk:
The disk name, such as
dsk10
The device model, such as RZ1CB-CA
The bus number for the device
For more information about a disk, double click the list item (or click Configure when a disk is highlighted). The Disk Configuration: Configure Partitions: window is displayed.
This window contains:
A graphical representation of the disk partitions in a horizontal bar-chart format and disk information such as the disk name, the total size of the disk, and usage information.
A Partition Table button that you can click to display a bar chart of the current partitions in use, their sizes, and the file system in use.
A Disk Attributes button that you can click to display values for disk attributes.
For more information about the Disk Configuration GUI, see its online
help.
2.3.2 Using Operating System Commands to Find Unused Disks
You can use the following operating system commands to find unused disks:
List all the disks on the system:
# ls /dev/disk/dsk*c
/dev/disk/dsk0c /dev/disk/dsk26c /dev/disk/dsk42c /dev/disk/dsk59c /dev/disk/dsk10c /dev/disk/dsk27c /dev/disk/dsk43c /dev/disk/dsk5c /dev/disk/dsk11c /dev/disk/dsk28c /dev/disk/dsk44c /dev/disk/dsk60c /dev/disk/dsk12c /dev/disk/dsk29c /dev/disk/dsk45c /dev/disk/dsk61c /dev/disk/dsk13c /dev/disk/dsk2c /dev/disk/dsk46c /dev/disk/dsk62c /dev/disk/dsk14c /dev/disk/dsk30c /dev/disk/dsk47c /dev/disk/dsk63c /dev/disk/dsk15c /dev/disk/dsk31c /dev/disk/dsk48c /dev/disk/dsk64c /dev/disk/dsk16c /dev/disk/dsk32c /dev/disk/dsk49c /dev/disk/dsk65c /dev/disk/dsk17c /dev/disk/dsk33c /dev/disk/dsk4c /dev/disk/dsk66c /dev/disk/dsk18c /dev/disk/dsk34c /dev/disk/dsk50c /dev/disk/dsk67c /dev/disk/dsk19c /dev/disk/dsk35c /dev/disk/dsk51c /dev/disk/dsk68c /dev/disk/dsk1c /dev/disk/dsk36c /dev/disk/dsk52c /dev/disk/dsk6c /dev/disk/dsk20c /dev/disk/dsk37c /dev/disk/dsk53c /dev/disk/dsk7c /dev/disk/dsk21c /dev/disk/dsk38c /dev/disk/dsk54c /dev/disk/dsk8c /dev/disk/dsk22c /dev/disk/dsk39c /dev/disk/dsk55c /dev/disk/dsk9c /dev/disk/dsk23c /dev/disk/dsk3c /dev/disk/dsk56c /dev/disk/dsk24c /dev/disk/dsk40c /dev/disk/dsk57c /dev/disk/dsk25c /dev/disk/dsk41c /dev/disk/dsk58c
To verify that a disk or partition is unused, choose a disk
from the output of the previous command and enter the
disklabel
command with the name of the disk; for example:
# disklabel dsk20c | grep -p '8 part'
8 partitions: # size offset fstype fsize bsize cpg # ~Cyl values a: 131072 0 unused 0 0 # 0 - 25* b: 262144 131072 unused 0 0 # 25*- 77* c: 35556389 0 unused 0 0 # 0 - 6999* d: 0 0 unused 0 0 # 0 - 0 e: 0 0 unused 0 0 # 0 - 0 f: 0 0 unused 0 0 # 0 - 0 g: 17581586 393216 unused 0 0 # 77*- 3538* h: 17581587 17974802 unused 0 0 # 3538*- 6999*
For more information, see
disklabel
(8)
If you are using AdvFS, display the disks in use by all domains:
# ls /etc/fdmns/*/*
In a cluster, do not use any partitions (even if unused) on a member's boot disk or on the quorum disk.
In a cluster, identify the cluster quorum disk:
# clu_quorum
If you are using UFS, display all mounted file sets:
# mount
The
drdmgr
and
hwmgr
commands
can give you information about which systems serve which disks.
To get a graphical
display of the cluster hardware configuration, including active members, buses,
storage devices, and their connections, use the
sms
command
to invoke the graphical interface for the SysMan Station, and then choose
Hardware from the Views menu.
2.3.3 Using the LSM voldisk Command to Find Unused Disks
When LSM starts, it obtains a list of disk device addresses from the operating system software and examines the disk labels to determine which devices are initialized for LSM use and which are not.
If LSM is running on the system, you can use the
voldisk
command to display a list of all known disks and to display information about
a particular disk:
To view a list of disks:
# voldisk list
DEVICE [1]TYPE [2] DISK [3] GROUP [4] STATUS [5] dsk0 sliced - - unknown dsk1 sliced - - unknown dsk2 sliced dsk2 rootdg online dsk3 sliced dsk3 rootdg online dsk4 sliced dsk4 rootdg online dsk5 sliced dsk5 rootdg online dsk6 sliced dsk6 dg1 online dsk7 sliced - - online dsk8 sliced dsk8 dg1 online dsk9 sliced - - online dsk10 sliced - - online dsk11 sliced - - online dsk12 sliced - - online dsk13 sliced - - unknown dsk14 sliced - - unknown
The following list describes the information in the output.
Specifies the disk access name. [Return to example]
Specifies the LSM disk type (sliced
,
simple
, or
nopriv
).
A dash (-) means the
device is not under LSM control and therefore does not have an LSM disk type.
[Return to example]
Specifies the LSM disk media name. A dash (-) means the device is not assigned to a disk group and therefore does not have an LSM disk media name. [Return to example]
Specifies the disk group to which the device belongs. A dash (-) means the device is not assigned to a disk group. [Return to example]
Specifies the status of the disk according to LSM:
unknown
The disk is not under LSM
control.
online
The disk is under LSM control.
If both the DISK and GROUP columns contain dashes, the disk does not belong
to a disk group and is an unused LSM disk.
offline
The disk is not accessible,
either because it failed or because it was intentionally placed off line.
online aliased
The disk is a hardware
clone disk.
error
The disk is detected but
has experienced I/O errors.
failed was
An LSM disk media name
exists but the disk is not associated with a DEVICE.
Displays the last device
associated with this name.
To display information about a specific LSM disk:
# voldisk list disk
For example:
# voldisk list dsk5
Device: dsk5 devicetag: dsk5 type: sliced [1] hostid: hostname disk: name=dsk5 id=942260116.1188.hostname [2] group: name=dg1 id=951155418.1233.hostname [3] flags: online ready autoimport imported pubpaths: block=/dev/disk/dsk5g char=/dev/rdisk/dsk5g privpaths: block=/dev/disk/dsk5h char=/dev/rdisk/dsk5h version: n.n iosize: min=512 (bytes) max=2048 (blocks) public: slice=6 offset=16 len=2046748 [4] private: slice=7 offset=0 len=4096 update: time=952956192 seqno=0.11 headers: 0 248 configs: count=1 len=2993 logs: count=1 len=453 Defined regions: config priv 17- 247[ 231]: copy=01 offset=000000 enabled config priv 249- 3010[ 2762]: copy=01 offset=000231 enabled log priv 3011- 3463[ 453]: copy=01 offset=000000 enabled
Identifies the LSM disk type. [Return to example]
The
name=
field contains the disk media
name.
[Return to example]
Identifies the disk group. [Return to example]
The
len=
field shows the amount of available
space in blocks on the LSM disk.
On a Tru64 UNIX system, one block is 512
bytes.
[Return to example]
For more information, see
voldisk
(8)