2    Planning LSM Volumes and Disk Groups

To use LSM most effectively, you should plan your LSM configuration before you use LSM volumes for applications or file systems. Your configuration will depend on the following:

This chapter provides information and worksheets to assist you in planning LSM volumes and disk groups. You can make copies of the worksheets for future use.

2.1    Planning LSM Volumes

Planning LSM volumes includes deciding which attributes you want the LSM volumes to have. An LSM volume has the following attributes:

Table 2-1 lists the benefits and trade-offs of various plex layout types and describes situations where one plex type might provide better performance, or be more cost effective, than a different plex type. For optimal performance you might need to tune your system to the work load. The layout you choose depends on your specific system configuration, data availability and reliability needs, and application requirements.

Table 2-1:  Plex Layout Type Benefits and Trade-Offs

Plex Type Benefits and Applicability Trade-Offs
Concatenated

Uses space on multiple disks that might otherwise be wasted.

Concatenated plex can be mirrored for data redundancy and high availability.

Good for volumes containing infrequently used data or data that does not change often or small volumes that can be confined to a single disk.

Possible uneven performance (hot spots, where one disk is in use by multiple applications).

When mirrored, requires 2 through 32 times as much disk space.

Striped

Distributes data and therefore I/O load evenly across many disks.

Good for:

  • Large volumes that cannot be confined to a single disk.

  • Applications with large read-request size.

  • Volumes that contain data that changes often (many writes).

    Striping is preferred over RAID5 in this case, because RAID5 imposes the overhead of calculating and writing parity data along with volume data.

Striped plexes can be mirrored for data redundancy and high availability.

When mirrored, requires 2 through 32 times as much disk space.

RAID 5

Provides the I/O distribution benefit of striping.

Provides redundancy through parity, using fewer disks than a volume with mirrored striped plexes.

Good for volumes with a high read-to-write ratio.

Depending on the I/O stripe size, performance might be slower than a volume with striped plexes due to parity calculation.

A RAID5 plex requires a minumum of three disks, plus a separate disk for the log plex.

The RAID5 plex type is not supported in a cluster.

When you create an LSM volume with the volassist command (the recommended and simplest method), LSM performs all the necessary calculations and creates a volume and log plexes of the appropriate sizes.

Section 2.1.1 and Section 2.1.2 discuss the space requirements for Dirty Region Logs and RAID5 logs. The worksheets in Section 2.1.3, Section 2.1.4, Section 2.1.5 and Section 2.2 are provided to help you calculate the space needed and ensure the disk group has enough space for the volumes you want.

2.1.1    Planning Dirty-Region Logs

By default, LSM configures a DRL log subdisk of approximately 65 blocks per gigabyte of volume size for both standalone systems and clusters so that the associated mirrored volume can be used in either a cluster or a standalone system. In a cluster, each member that issues I/O to the volume maintains its own private log to keep track of changes to the volume from that member. The volume's DRL log plex is an aggregate of every member's private log and must therefore be larger than on a standalone system.

If you plan to use LSM strictly on a standalone system, you can use the minimum DRL size, which is 2 blocks per gigabyte of volume size. However, if you create mirrored volumes with the smaller DRL size and then make those volumes available to a cluster (by using the standalone sytem to create a cluster or by moving the storage to an existing cluster), the DRL logs on the volumes will be disabled, but the volumes themselves will be usable.

Table 2-2 lists the approximate default DRL sizes for various volume sizes.

Table 2-2:  Default (Approximate) Sizes of DRL Logs

Volume Size (GB) DRL Size (blocks)
<=1 65
2 130
3 130
4 195
60-61 2015
62-63 2080
1021 33215
1022-1023 33280
1024 33345

We highly recommend using the default log size. The log is a bitmap that represents regions of the volume (each bit maps to a number of sectors), and the granularity of the bitmap changes if you specify a larger or smaller log size.

For best performance, each volume's DRL plex should be on a different disk from the log plexes for any other volumes. Using the same disk for the DRL plex for many mirrored volumes will slow down that disk and degrade volume performance, because the dirty bits are set before the write actually occurs. (If you are using hardware such as an HSG80 RAID Array, the cache manages and reduces much of the disk contention.)

In addition, if that disk fails, the logs for those volumes are lost. If the system crashes before you replace that disk, those volumes will go through complete plex resynchronization when the system restarts. This is a performance risk, but not a data loss risk.

2.1.2    Planning RAID 5 Logs

LSM calculates the default log size for a RAID5 volume using a different algorithm than for mirrored volumes. The RAID 5 log is large enough to contain a copy of the full stripe width of data and parity for several writes. When you create a RAID5 volume using the volassist command, the RAID5 log size is calculated for you and is sized appropriately to the volume.

The formula for the default RAID 5 log size:

[10 × (number of columns × stripe width)]

For example:

Volumes that use a RAID5 plex are not supported in a cluster, so the RAID5 log size applies only to a standalone system.

For best performance, the RAID5 log plex should be on its own disk. Using the same disk for several log plexes (for different volumes) will slow down that disk and degrade the performance of those volumes, because the log is written before the data plex is written. In addition, if that disk fails, the logs for those volumes are lost. If the system crashes before you replace that disk, those volumes will go through complete parity and data resynchronization when the system restarts. This is both a performance risk and a data loss risk.

2.1.3    Worksheet for LSM Volumes with Concatenated Plexes

Use the following worksheet to plan LSM volumes that use concatenated plexes.

Figure 2-1:  Worksheet for LSM Volumes with Concatenated Plexes

Attribute Default Values Chosen Values
Volume name  
Volume size  
Number of data plexes 1  
DRL plex size (for mirrored volumes)

65 blocks per GB of volume size (default) or 2 blocks per GB of volume size (for standalone systems only)

 
Number of DRL plexes 1  
Disk group name rootdg  
Usage type fsgen  
Total space required (Volume size × number of plexes) + (DRL size × number of logs)  

2.1.4    Worksheet for LSM Volumes with Striped Plexes

Use the following worksheet to plan LSM volumes that use one or more striped plexes.

Figure 2-2:  Worksheet for LSM Volumes with Striped Plexes

Attribute Default Values Chosen Values
Volume name  
Volume size  
Data unit size (stripe width) 64K bytes  
Number of columns Minimum of two, based on number of disks in disk group and the volume size  
Number of data plexes 1  
DRL plex size (for mirrored volumes)

65 blocks per GB of volume size (default) or 2 blocks per GB of volume size (for standalone systems only)

 
Number of DRL plexes 1  
Disk group name rootdg  
Usage type fsgen  
Total space required (Volume size × number of plexes) + (DRL size × number of logs)  

2.1.5    Worksheet for LSM Volumes with a RAID 5 Plex

Use the following worksheet to plan an LSM volume that uses a RAID 5 plex.

Note

RAID 5 volumes are not supported in a cluster.

Figure 2-3:  Worksheet for LSM Volumes with a RAID 5 Plex

Attribute Default Values Chosen Values
Volume name  
Volume size  
Data unit size (stripe width) 16K bytes  
Number of columns (NCOL) Between 3 and 8 based on number of disks in disk group and the volume size (Minimum of three)
Log plex size 10 × (NCOL × data unit size)  
Number of logs (NLOGS) 1  
Disk group name rootdg  
Usage type Must be raid5 raid5
Total space required (Volume size × NCOL ÷ (NCOL-1)) + (log plex size × NLOGS)  

2.2    Planning Disk Groups

On every system or cluster running LSM, there will be one mandatory, default disk group called rootdg. You can create additional disk groups to help you manage your storage and to support moving storage (LSM volumes) to a different system. The following sections provide recommendations and explain restrictions for rootdg and other disk groups.

2.2.1    Guidelines for the Number and Usage of Disks

Each disk group (including rootdg) should have two or more sliced or simple LSM disks to ensure multiple copies of the disk group's configuration database. Only sliced and simple disks have a private region, which stores a copy of the disk group's comfiguration database. By default, LSM maintains a minimum of four active copies of the configuration database in each disk group, distributed among several disks (and, if applicable, among buses) to ensure redundancy. All the copies should be on separate disks. A disk group cannot contain only nopriv disks.

The rootdg disk group stores information about all disk groups in its configuration database. Each time an LSM object is created, modified, or removed, the configuration database for that disk group and for rootdg is updated. Each LSM object (disk, subdisk, plex, and volume) requires one record; two records fit in one sector (512 bytes).

To improve performance, keep the number of disks in rootdg to ten or fewer. If possible, use rootdg only for the following:

Create additional disk groups if you have more than ten disks to place under LSM control or might move a disk group to a different system or cluster.

In each disk group, configure one or more hot-spare disks for each redundant (mirrored or RAID5) volume in the disk group. Each hot-spare disk should be as large as the largest disk serving a volume.

2.2.2    Guidelines for Connectivity and Availability

A disk group should have storage devices on different buses to improve performance and availability.

Ideally, in a cluster, LSM should be used only with fully shared storage. Storage availability in a cluster increases as more cluster members have direct access to all disks in a disk group. Availability is highest when all disks in a disk group have connectivity with all cluster members.

The drdmgr and hwmgr commands can give you information about which cluster members serve which disks. For a graphical display of the cluster hardware configuration, including active members, buses, storage devices, and their connections, use the sms command to invoke the graphical interface for the SysMan Station, and then choose Hardware from the Views menu.

In a cluster, the disks in each disk group should be accessible by all cluster members, so that all members have access to the LSM volumes, regardless of the state of any other member. LSM volumes have the same connectivity when either one of the following conditions is true:

Private disk groups (a disk group all of whose disks are connected to the private bus of a single cluster member) are supported, but if that member becomes unavailable, then the cluster loses access to the disk group. Private disk groups are suitable only when a cluster member that the disk group is physically connected to is also the only member that needs access to the disk group.

Avoid configuring a disk group with a combination of local and shared disks or a disk group whose disks are distributed among the private buses of multiple members. Such disk groups are not recommended, because no single member has direct access to all storage in the group.

It is especially important that the disks in rootdg are shared. However, this might not always be possible. For example, you can encapsulate cluster members' swap devices to LSM volumes; all swap volumes must belong to rootdg, but those devices might be private to their respective member. Avoid creating volumes that use a mix of shared and private disks or a mix of private disks that are connected to different members.

Section 2.2.3 contains a blank worksheet you can use to plan your disk groups. You can make copies and fill in the information on the copies rather than in this manual and keep the completed worksheet with each system running LSM, for your reference. Also, because you can change your LSM configuration at any time, you can make a new copy of the blank worksheet to record your changes.

Include in the worksheet any information that will help you keep track of the purpose of that disk group. For example, you might create a disk group named finance_dg whose purpose is to contain one or more volumes that will be used by a financial application. You might create another disk group named db1, which will contain a volume used by a database. Choose a name for the disk group carefully, as it is not easy to rename it when it contains active volumes.

Under Volume, Plex and Hot-Spare Disk Information, include the names of all volumes in that disk group, their plex type, which disks belong to which plex, and any hot-spare disks. For information on hot-spare disks, see Section 3.5.

In addition to the information you include in the worksheet, you can print and save the output from the volprint command, which shows detail such as the subdisks used in each volume and the sizes of all the LSM objects. For more information, see Section 5.4.1 and volprint(8).

Table 2-3 is an example of what your disk group planning worksheets might look like when complete. This example applies only to a standalone system, not a cluster.

Table 2-3:  Completed Disk Group Worksheets for a Standalone System

Disk Group Name: rootdg — Root file system and system disks.
Disks in Group Bus/LUN Disk Size Volume, Plex, and Hot-Spare Disk Information

dsk0

dsk1

dsk6

1

1

2

4 GB (all)

volumes: rootvol, swapvol, vol-dsk0g (for the /usr and /var file systems on partition dsk0g)

rootvol-01(dsk0a)

rootvol-02 (dsk6a)

swapvol-01 (dsk0b)

swapvol-02 (dsk6b)

vol-dsk0g-01 (dsk0g)

vol-dsk0g-02 (dsk6g)

dsk7 2 4 GB hot-spare disk
Disk Group Name: data_dg — Database, must be redundant. Contains volume with mirrored striped plexes and a DRL.
Disks in Group Bus/LUN Disk Size Volume, Plex, and Hot-Spare Disk Information

dsk2

dsk3

dsk8

dsk9

dsk12

1

1

2

2

3

18 GB (all)

volume: db_vol

plex: db_vol-01 (dsk2, dsk3)

plex: db_vol-02 (dsk8, dsk9)

plex: db_vol-03 (DRL plex, dsk12)

dsk13

dsk20

3

4

18 GB (all) hot-spare disks (each on different bus)
Disk Group Name: finance_dg — Financial application, must be highly available. Contains volume with a RAID 5 plex (read-only application).
Disks in Group Bus/LUN Disk Size Volume, Plex, and Hot-Spare Disk Information

dsk4

dsk10

dsk14

dsk18

dsk24

1

2

3

4

5

9 GB (all)

volume: fin_vol

plex: fin_vol-01 (column 1 — dsk4)

plex: fin_vol-01 (column 2 — dsk10)

plex: fin_vol-01 (column 3 — dsk14)

plex: fin_vol-01 (column 4 — dsk18)

plex: fin_vol-02 (log plex — dsk24)

dsk5

dsk11

dsk15

dsk19

dsk25

1

2

3

4

5

9 GB (all) hot-spare disks (one on each bus)

2.2.3    Worksheet for Planning Disk Groups

You can make copies of the following worksheet for each disk group on your system, including the rootdg disk group.

Table 2-4:  Worksheet for Disk Group

Disk Group Name:
Disks in Group Bus/LUN Number Disk Size Volume, Plex, and Hot-Spare Disk Information
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       

2.3    Finding Unused Storage Devices

Unused storage devices are unused disks, partitions, and hardware RAID disks that LSM can initialize as LSM disks, as well as unused LSM disks, initialized for LSM use but not assigned to a disk group.

The following sections describe how to find unused disks, partitions, and LSM disks. See your hardware RAID documentation for information on finding unused hardware RAID disks.

To find unused storage devices, you can use:

2.3.1    Using the Disk Configuration GUI to Find Unused Disks

To find unused disks using the Disk Configuration GUI, start the Disk Configuration interface using either of the following methods:

For more information about the Disk Configuration GUI, see its online help.

2.3.2    Using Operating System Commands to Find Unused Disks

You can use the following operating system commands to find unused disks:

  1. List all the disks on the system:

    # ls /dev/disk/dsk*c
    

    /dev/disk/dsk0c   /dev/disk/dsk26c  /dev/disk/dsk42c  /dev/disk/dsk59c
    /dev/disk/dsk10c  /dev/disk/dsk27c  /dev/disk/dsk43c  /dev/disk/dsk5c
    /dev/disk/dsk11c  /dev/disk/dsk28c  /dev/disk/dsk44c  /dev/disk/dsk60c
    /dev/disk/dsk12c  /dev/disk/dsk29c  /dev/disk/dsk45c  /dev/disk/dsk61c
    /dev/disk/dsk13c  /dev/disk/dsk2c   /dev/disk/dsk46c  /dev/disk/dsk62c
    /dev/disk/dsk14c  /dev/disk/dsk30c  /dev/disk/dsk47c  /dev/disk/dsk63c
    /dev/disk/dsk15c  /dev/disk/dsk31c  /dev/disk/dsk48c  /dev/disk/dsk64c
    /dev/disk/dsk16c  /dev/disk/dsk32c  /dev/disk/dsk49c  /dev/disk/dsk65c
    /dev/disk/dsk17c  /dev/disk/dsk33c  /dev/disk/dsk4c   /dev/disk/dsk66c
    /dev/disk/dsk18c  /dev/disk/dsk34c  /dev/disk/dsk50c  /dev/disk/dsk67c
    /dev/disk/dsk19c  /dev/disk/dsk35c  /dev/disk/dsk51c  /dev/disk/dsk68c
    /dev/disk/dsk1c   /dev/disk/dsk36c  /dev/disk/dsk52c  /dev/disk/dsk6c
    /dev/disk/dsk20c  /dev/disk/dsk37c  /dev/disk/dsk53c  /dev/disk/dsk7c
    /dev/disk/dsk21c  /dev/disk/dsk38c  /dev/disk/dsk54c  /dev/disk/dsk8c
    /dev/disk/dsk22c  /dev/disk/dsk39c  /dev/disk/dsk55c  /dev/disk/dsk9c
    /dev/disk/dsk23c  /dev/disk/dsk3c   /dev/disk/dsk56c
    /dev/disk/dsk24c  /dev/disk/dsk40c  /dev/disk/dsk57c
    /dev/disk/dsk25c  /dev/disk/dsk41c  /dev/disk/dsk58c
    

  2. To verify that a disk or partition is unused, choose a disk from the output of the previous command and enter the disklabel command with the name of the disk; for example:

    # disklabel dsk20c | grep -p '8 part'
    

    8 partitions:
    #            size       offset    fstype  fsize  bsize   cpg  # ~Cyl values
      a:       131072            0    unused      0      0        #      0 - 25*
      b:       262144       131072    unused      0      0        #     25*- 77*
      c:     35556389            0    unused      0      0        #      0 - 6999*
      d:            0            0    unused      0      0        #      0 - 0
      e:            0            0    unused      0      0        #      0 - 0
      f:            0            0    unused      0      0        #      0 - 0
      g:     17581586       393216    unused      0      0        #     77*- 3538*
      h:     17581587     17974802    unused      0      0        #   3538*- 6999*
     
    

    For more information, see disklabel(8).

  3. If you are using AdvFS, display the disks in use by all domains:

    # ls /etc/fdmns/*/*
    

    In a cluster, do not use any partitions (even if unused) on a member's boot disk or on the quorum disk.

  4. In a cluster, identify the cluster quorum disk:

    # clu_quorum
    

  5. If you are using UFS, display all mounted file sets:

    # mount
    

The drdmgr and hwmgr commands can give you information about which systems serve which disks. To get a graphical display of the cluster hardware configuration, including active members, buses, storage devices, and their connections, use the sms command to invoke the graphical interface for the SysMan Station, and then choose Hardware from the Views menu.

2.3.3    Using the LSM voldisk Command to Find Unused Disks

When LSM starts, it obtains a list of disk device addresses from the operating system software and examines the disk labels to determine which devices are initialized for LSM use and which are not.

If LSM is running on the system, you can use the voldisk command to display a list of all known disks and to display information about a particular disk:

  1. To view a list of disks:

    # voldisk list
    

    DEVICE [1]TYPE [2]  DISK [3] GROUP [4]  STATUS [5] 
    dsk0         sliced     -            -            unknown 
    dsk1         sliced     -            -            unknown 
    dsk2         sliced     dsk2         rootdg       online  
    dsk3         sliced     dsk3         rootdg       online  
    dsk4         sliced     dsk4         rootdg       online  
    dsk5         sliced     dsk5         rootdg       online  
    dsk6         sliced     dsk6         dg1          online  
    dsk7         sliced     -            -            online  
    dsk8         sliced     dsk8         dg1          online  
    dsk9         sliced     -            -            online  
    dsk10        sliced     -            -            online  
    dsk11        sliced     -            -            online  
    dsk12        sliced     -            -            online  
    dsk13        sliced     -            -            unknown 
    dsk14        sliced     -            -            unknown
    

    The following list describes the information in the output.

    1. Specifies the disk access name. [Return to example]

    2. Specifies the LSM disk type (sliced, simple, or nopriv). A dash (-) means the device is not under LSM control and therefore does not have an LSM disk type. [Return to example]

    3. Specifies the LSM disk media name. A dash (-) means the device is not assigned to a disk group and therefore does not have an LSM disk media name. [Return to example]

    4. Specifies the disk group to which the device belongs. A dash (-) means the device is not assigned to a disk group. [Return to example]

    5. Specifies the status of the disk according to LSM:

      • unknown — The disk is not under LSM control.

      • online — The disk is under LSM control. If both the DISK and GROUP columns contain dashes, the disk does not belong to a disk group and is an unused LSM disk.

      • offline — The disk is not accessible, either because it failed or because it was intentionally placed off line.

      • online aliased — The disk is a hardware clone disk.

      • error — The disk is detected but has experienced I/O errors.

      • failed was — An LSM disk media name exists but the disk is not associated with a DEVICE. Displays the last device associated with this name.

      [Return to example]

  2. To display information about a specific LSM disk:

    # voldisk list disk
    

    For example:

    # voldisk list dsk5
    

    Device:    dsk5
    devicetag: dsk5
    type:      sliced  [1]
    hostid:    hostname
    disk:      name=dsk5 id=942260116.1188.hostname  [2] 
    group:     name=dg1 id=951155418.1233.hostname  [3]
    flags:     online ready autoimport imported
    pubpaths:  block=/dev/disk/dsk5g char=/dev/rdisk/dsk5g
    privpaths: block=/dev/disk/dsk5h char=/dev/rdisk/dsk5h
    version:   n.n
    iosize:    min=512 (bytes) max=2048 (blocks)
    public:    slice=6 offset=16 len=2046748   [4]
    private:   slice=7 offset=0 len=4096
    update:    time=952956192 seqno=0.11
    headers:   0 248
    configs:   count=1 len=2993
    logs:      count=1 len=453
    Defined regions:
     config   priv     17-   247[   231]: copy=01 offset=000000 enabled
     config   priv    249-  3010[  2762]: copy=01 offset=000231 enabled
     log      priv   3011-  3463[   453]: copy=01 offset=000000 enabled
    

    1. Identifies the LSM disk type. [Return to example]

    2. The name= field contains the disk media name. [Return to example]

    3. Identifies the disk group. [Return to example]

    4. The len= field shows the amount of available space in blocks on the LSM disk. On a Tru64 UNIX system, one block is 512 bytes. [Return to example]

For more information, see voldisk(8).