2    Preparation

This chapter describes what to read and what to do before installing the cluster software. Table 2-1 summarizes the preparation guidelines, and references sources of information that provide more detail.

Table 2-1:  Preparation Tasks

2.1    Documentation and PAK Requirements

Before beginning the installation, have the following Version 5.1B documents available:

• Cluster Hardware Configuration

• Cluster Release Notes

• Cluster Technical Overview

• Cluster Highly Available Applications

• Cluster Administration

• Tru64 UNIX Installation Guide

• Tru64 UNIX Installation Guide — Advanced Topics

• Tru64 UNIX Release Notes

• Tru64 UNIX System Administration

• Tru64 UNIX Hardware Management

• Tru64 UNIX Network Administration: Connections

• Tru64 UNIX Network Administration: Services

• Tru64 UNIX Security Administration

• For each member, a Tru64 UNIX product authorization key (PAK)

• For each member, a TruCluster Server PAK

2.2    Member IDs

Each cluster member has a unique member ID, whose value is an integer from 1 through 63 inclusive. The cluster software uses member IDs to identify each member of the cluster. When adding a member to the cluster, the installation programs offer the next available member ID as the default value. During the installation, you can accept the offered default value or enter any unused member ID.

For example, consider a two-node cluster consisting of members pepicelli and polishham. If, when installing the cluster software, the default member IDs are accepted, pepicelli is assigned member ID 1 and polishham is assigned member ID 2:

pepicelli      member ID = 1
polishham      member ID = 2
 

When a member is added to the cluster, its member ID is written to its sysconfigtab file as the value of the memberid variable in the generic subsystem.

2.3    IP Names and Addresses

You will need the following IP name and address information when creating a cluster and adding members:

• A cluster name and IP address, which are assigned to the default cluster alias (Section 2.3.1).

• For each cluster member, a host name and IP names and addresses for each external network interface (Section 2.3.2).

• For each cluster member, an IP name and address for that member's virtual cluster interconnect (Section 2.3.3).

  Notes

  A Memory Channel interconnect requires one IP name and address per member. The installation programs provide default IP names and addresses.

  A LAN interconnect requires two IP names and addresses per member. The installation programs provide default IP names and addresses.

  All cluster members must be configured to use either Memory Channel or LAN hardware. You cannot mix interconnect types within a cluster.

The Tru64 UNIX Network Administration: Connections manual provides guidelines on allocating IP addresses. You can also refer to RFC 1918, in which the Internet Assigned Numbers Authority (IANA) reserves the following blocks of IP address space for use by private internets:

10.0.0.0      -   10.255.255.255
172.16.0.0    -   172.31.255.255
192.168.0.0   -   192.168.255.255
 

When the installation software offers default addresses for cluster interconnect networks, these default addresses are in class C networks (subnet mask 255.255.255.0).

For attributes that use host-name-like strings, such as member host names, the cluster name, and the name (or names) associated with each member's cluster interconnect, the installation programs require that you follow the standard host-name conventions defined in RFC 952, and amended in RFC 1123: use only the letters A-Z or a-z, numbers 0-9, '.' (period), and '-' (hyphen) as characters. The first character of the host name must be a letter or (per RFC 1123) a digit.

The examples in the following sections show the names and addresses used to create a sample cluster named deli. This cluster has two members, pepicelli and polishham, and uses Memory Channel hardware for its cluster interconnect. Figure 2-1 shows the network topology for cluster deli.

Figure 2-1:  Cluster Network Interfaces

2.3.1    Cluster Name and IP Address

Each cluster has a cluster name, which is not the host name of one of the cluster members, but a host name that you assign to the entire cluster. This name is the host name associated with the default cluster alias, which is an IP address assigned to the entire cluster. It provides a method for external clients to request services from the cluster rather than from individual cluster members.

The following example shows the fully qualified cluster name and default cluster alias IP address for cluster deli:

16.140.112.209   deli.zk3.dec.com   deli   # default cluster alias IP address
                                           # cluster host name
                                           # (default cluster alias name)
 

The default cluster alias IP address must be a valid address to which clients can route. Several Internet services in the cluster use this address as the source address when connections originate from the cluster. If the address is not one to which clients can respond, the services will not work.

Do not use a cluster alias address as a broadcast address or a multicast address. Do not allow it to reside in the subnet used by the cluster interconnect. In addition, although cluster members can use and advertise IPv6 addresses, IPv6 addresses are not supported by the cluster alias subsystem. Therefore, you cannot assign IPv6 addresses to cluster aliases.

You can assign a cluster alias an IP address that resides in one of the private address spaces defined in RFC 1918:

10.0.0.0      -   10.255.255.255   (10/8 prefix)
172.16.0.0    -   172.31.255.255   (172.16/12 prefix)
192.168.0.0   -   192.168.255.255  (192.168/16 prefix)
 

However, by default, the cluster alias daemon will not advertise an alias address that resides in these private address spaces. If you assign one of these addresses to the default cluster alias, do the following after creating a single-member cluster:

1. Set the value of the CLUAMGR_ROUTE_ARGS variable to resvok in the /etc/rc.config.common file:

   # rcmgr -c set CLUAMGR_ROUTE_ARGS resvok
    
   

2. Use the cluamgr command to notify the cluster alias daemon, aliasd, that it can advertise reserved addresses:

   # cluamgr -r resvok,start
    
    
   

After CLUAMGR_ROUTE_ARGS is set to resvok in /etc/rc.config.common, each cluster member automatically runs the cluamgr -r resvok command at boot time.

2.3.2    Member Host Names and IP Addresses

Each cluster member has a host name. By convention, this host name is usually associated with an IP address assigned to one of the system's network interfaces. Excluding the cluster interconnect interfaces, which are covered in Section 2.3.3, you will need the following names and IP addresses available for use by cluster members:

• For each member, a host name.

  You supply a host name when installing Tru64 UNIX on the system that will become the first cluster member, and when running clu_add_member to add members to the cluster.

• For each external network interface on each member, the IP name and IP address you will use when configuring that interface.

  You configure a network interface when installing Tru64 UNIX on the system that will become the first cluster member. (You can also configure other external network interfaces on that system before running clu_create. However, do not configure the cluster interconnect interface; clu_create will configure that interface.)

  The clu_add_member command configures only the cluster interconnect interface. During the first boot of a new member, you are given an opportunity to configure additional network interfaces. We recommend that you configure at least one interface, whose IP name is the host name of the system.

For example, for the deli cluster members, each system has one external network interface. We use each member's host name as the IP name associated with that member's external interface's IP address:

16.140.112.238  pepicelli  # First  member's tu0 IP name and address.
                           # This is the Tru64 UNIX system.
                           # The host name is pepicelli.
16.140.112.237  polishham  # Second member's tu0 IP name and address.
                           # The host name is polishham.
 

2.3.3    IP Names and Addresses for Each Member's Cluster Interconnect

A cluster must have a dedicated cluster interconnect, which is a separate and distinct communications channel on which only cluster members reside. When the cluster interconnect is a LAN, all members must be on a single private subnet. All cluster members must be connected to the cluster interconnect.

• For clusters that use a Memory Channel cluster interconnect, you need an IP name and an IP address for the virtual cluster interconnect interface on each member system. (Although a cluster member can have redundant physical interfaces for failover purposes, these interfaces share a single virtual network address.) Only one IFCONFIG and NETDEV entry represent a member's Memory Channel cluster interconnect in its /etc/rc.config file, regardless of the number of physical interfaces present in the system.

  By default, the installation programs offer an IP name set to the short form of the member's host name followed by -ics0, and IP addresses on the 10.0.0 subnet with the host portion of the address set to the member ID. For example, pepicelli-ics0 and 10.0.0.1.

  Note

  In previous releases, where Memory Channel was the only supported cluster interconnect device, -mc0 was the identifier appended to the short form of a member's host name.

• For a cluster using a LAN interconnect, where communications between members traverse two TCP/IP layers, you need the following:

  • An IP name and IP address for the virtual cluster interconnect device on each member system.

    By default, the installation programs offer IP addresses on the 10.0.0 subnet for virtual cluster interconnect, with the host portion of the address set to the member ID and the IP name set to the short form of the member's host name followed by -ics0.

  • An IP name and IP address on a different subnet for the physical LAN interface.

    By default, the cluster installation programs offer IP addresses on the 10.1.0 subnet for the physical cluster interconnect, with the host portion of the address set to the member ID. The IP name is set to membermember-ID-icstcp0.

  Notes

  Manufacturers typically associate a default address with an Ethernet switch to facilitate its management. This address may conflict with the default IP address the cluster installation scripts provide for the virtual cluster interconnect device or the physical LAN interface. In this case, you must ensure that the IP addresses selected for the cluster interconnect differ from that used by the switch.

  By default, the installation programs use Class C subnet masks for the IP addresses of both the virtual and physical LAN interconnect interfaces.

  Cluster interconnect IP addresses cannot end with either .0 or .255. Addresses of this type are considered broadcast addresses. A system with this type of address cannot join a cluster.

The following example shows the cluster interconnect IP names and IP addresses for two members of the deli cluster, pepicelli and polishham, which is running on a Memory Channel cluster interconnect:

10.0.0.1 pepicelli-ics0  # first member's  virtual interconnect IP name and address
10.0.0.2 polishham-ics0  # second member's virtual interconnect IP name and address
 

The following example shows the cluster interconnect IP names and IP addresses for two members of the same cluster running on a LAN interconnect:

# first member's cluster interconnect virtual interface IP name and address
10.0.0.1 pepicelli-ics0 
# first member's cluster interconnect physical interface IP name and address
10.1.0.1 member1-icstcp0
# second member's cluster interconnect virtual interface IP name and address
10.0.0.2 polishham-ics0
# second member's cluster interconnect physical interface IP name and address
10.1.0.2 member2-icstcp0
 

The cluster installation scripts mark both the cluster interconnect virtual interface and physical interface with the cluster interface (CLUIF) flag. For example, the following output of the ifconfig -a command shows the cluster interconnect virtual interface (ics0) and the cluster interconnect physical interface (ee0):

# ifconfig -a | grep -p CLUIF
ee0: flags=1000c63<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,SIMPLEX,CLUIF>
     inet 10.1.0.2 netmask ffffff00 broadcast 10.1.0.255 ipmtu 1500
ics0: flags=1100063<UP,BROADCAST,NOTRAILERS,RUNNING,NOCHECKSUM,CLUIF>
     inet 10.0.0.2 netmask ffffff00 broadcast 10.0.0.255 ipmtu 1500
 

The following example shows a cluster interconnect physical interface (nr0) that is a NetRAIN virtual interface:

# ifconfig -a | grep -p CLUIF
ee0: flags=1000c63<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,SIMPLEX,CLUIF>
     NetRAIN Virtual Interface: nr0 
     NetRAIN Attached Interfaces: ( ee1 ee0 ) Active Interface: ( ee1 )
ee1: flags=1000c63<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,SIMPLEX,CLUIF>
     NetRAIN Virtual Interface: nr0 
     NetRAIN Attached Interfaces: ( ee1 ee0 ) Active Interface: ( ee1 )
ics0: flags=11000c63<BROADCAST,NOTRAILERS,NOCHECKSUM,CLUIF>
     inet 10.0.0.2 netmask ffffff00 broadcast 10.0.0.255 ipmtu 1500
nr0: flags=1000c63<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,SIMPLEX,CLUIF>
     NetRAIN Attached Interfaces: ( ee1 ee0 ) Active Interface: ( ee1 )
     inet 10.1.0.2 netmask ffffff00 broadcast 10.1.0.255 ipmtu 1500 

2.4    Hardware

Install the hardware for the cluster, including storage hardware and hardware for the cluster interconnect (Memory Channel or LAN), before installing the Tru64 UNIX operating system.

If you add new hardware (for example, additional network adapters) after you install or update the operating system, remember to boot /genvmunix and build a customized kernel. Otherwise, the system's kernel configuration file will not contain these hardware options, and the kernel you build during TruCluster Server installation will not recognize the new hardware. The Tru64 UNIX System Administration manual provides information on configuring kernels.

2.5    Disks

This section provides the following information:

• Disks needed for installation (Section 2.5.1)

• LSM considerations (Section 2.5.2)

• Minimum disk layout for a two-node cluster (Section 2.5.3)

• Disk space recommendations (Section 2.5.4)

• How to allocate additional disk space in preparation for a rolling upgrade (Section 2.5.5)

Notes

Because the cluster's root (/), /usr, and /var file systems must be highly available, we recommend that you mirror these file systems. You can use either Redundant Array of Independent Disk (RAID) controllers or LSM (software RAID) to mirror these file systems. Because the Logical Storage Manager (LSM) is not supported for member boot partitions or the quorum disk, use RAID controllers (for example, an HSG80) to mirror those file systems.

If you plan to use Fibre Channel storagesets for any of the disks required for the base operating system or the cluster, read the Fibre Channel chapter in the Cluster Hardware Configuration manual.

Configure all storage hardware before installing the Tru64 UNIX operating system.

2.5.1    Disks Needed for Installation

You need to allocate disks, either physical or logical, for the following uses:

• One or more disks to hold the Tru64 UNIX operating system. The disks are either private disks on the system that will become the first cluster member, or disks on a shared bus that the system can access.

• One or more disks on a shared bus to hold the clusterwide AdvFS file systems: root (/), /usr, and, optionally, /var.

  Using separate disks provides better performance (more I/O in parallel). Separate disks also provide more load-balancing options. The Cluster Administration manual discusses balancing disk I/O.

• One boot disk per member on a shared bus. We recommend that member boot disks be on a shared bus for three reasons:

  1. When adding a member, the current member must have access to the new member's boot disk.

  2. If boot disks are on private buses and a member cannot boot, you cannot mount that disk from a running member and attempt to diagnose the problem.

  3. Some cluster administrative commands, like clu_quorum, might need access to a member's boot disk when the member is down.

• Depending on the number of members in the cluster, one disk on a shared bus to act as the quorum disk.

The examples in this section show each bootable disk from the point of view of a system console device (DK), and from the point of view of the operating system device special file (dsk). If you know the console device name for a bootable disk, put that information in the checklists in Appendix A while you go through this section. However, until you install the base operating system, you cannot know which special file is associated with each physical device. After you install the base operating system, use the information in Section 3.7 to map console device names to device special file names.

The following sections provide more information about the disks needed to install a cluster.

2.5.1.1    Tru64 UNIX Disk (Private or Shared)

The base operating system is installed using AdvFS file systems on one or more physical or logical disks. These disks are either private disks on the system that will become the first cluster member, or disks on a shared bus that the system can access. The following example assumes that you install the base operating system on DKA0 and that the operating system maps DKA0 to dsk0:

DKA0       dsk0a       root_domain#root
           dsk0g       usr_domain#usr
           dsk0h       var_domain#var
 

Because the base operating system is still available after you create a cluster, in an emergency you have the option of booting the base operating system and attempting to diagnose and fix problems.

Caution

Because the base operating system has access to data on any shared buses, do not boot the base operating system while the cluster is running. The base operating system has no knowledge of the barriers and locks that the cluster uses to control multisystem access to shared data. You risk data corruption if the Tru64 UNIX system accesses a disk used by the cluster. Boot the base operating system only after shutting down the entire cluster.

Restrictions: A disk used for the Tru64 UNIX operating system cannot be used as a clusterwide disk, a member boot disk, or the quorum disk.

2.5.1.2    Clusterwide root (/), /usr, and /var Disks (Shared)

When you create a cluster, the installation scripts copy the Tru64 UNIX root (/), /usr, and /var file systems from the Tru64 UNIX disks to the disks you specify for the clusterwide file systems.

Locate the physical or logical disks used for the clusterwide file systems on a shared bus so that cluster members have access to the disks regardless of whether an individual cluster member is up or down. (If a file system that must be accessible to all cluster members is on a member's private bus, and if that member crashes, the remaining cluster members cannot access that file system.)

During the installation, you supply the names of the disk partitions that will contain the clusterwide root (/), /usr, and /var file systems. Each AdvFS file system must be in a separate partition; however, the partitions do not have to be on the same disk. Table 2-2 lists the minimum recommended sizes for all cluster file systems.

For example:

dsk1b       cluster_root#root
dsk2c       cluster_usr#usr
dsk3c       cluster_var#var
 

Note

In a cluster, /var cannot be in /usr. If you install Tru64 UNIX with /var in /usr, clu_create will create separate partitions for them.

The cluster_root#root file system is often put on a b partition for these reasons:

• Size: The b partition is usually larger than a.

• Nonbootable: Cluster members do not boot the cluster_root#root file system. (Each member boots from a minimal root (/) file system on the a partition on its boot disk, not from the clusterwide root (/) file system.)

If, after you create and begin to use the cluster, you decide that you need extra space, you can add volumes to cluster_root#root, cluster_usr#usr, or cluster_var#var.

Note

For volumes in AdvFS domains and storage in LSM volumes for the file systems that all cluster members must access, we recommend that the physical devices be located on a bus or buses shared by all members.

Restrictions: If any partition on a disk is used by a clusterwide root (/), /usr, or /var file system, the disk cannot be used as a member boot disk or as the quorum disk.

2.5.1.3    Member Boot Disks (Shared)

Each member has a boot disk, which can either be a physical or a logical disk. This disk must be on a bus that is accessible to the system that is adding the member to the cluster. We recommend that the disk be on a bus shared by other cluster members.

For example:

DKC400     dsk10     first  member's boot disk  [pepicelli]
DKC600     dsk12     second member's boot disk  [polishham]
 

By default, the installation scripts reformat each member's boot disk to contain three partitions:

• An a partition for that member's bootable root file system

• A b partition for swap

• An h partition (fstype of cnx) for cluster status information

No usr or var file systems are on a member's boot disk.

Note

To avoid booting the wrong disk for a new member, you must know both the /dev/disk/dsk special file name for a member's boot disk and the DK device name for that disk from the new member's console.

When adding a member, the clu_add_member command displays information about a member boot disk. This information includes, if known, the disk's serial number (worldwide ID), manufacturer, model number, and physical location (bus/target/logical unit number (LUN)). Remember that the bus/target/LUN information is based on the system where you run clu_add_member. Depending on system and storage configuration, this information may or may not provide the same view of storage as the new member's console.

Use the information provided by clu_add_member and the information in Section 3.7 to map the special file name for a member boot disk to the physical device. Put this information in the Member Attributes table in Appendix A (Table A-3).

A member disk can contain more than the three required partitions. However, do not use the disk for data.

If a disk meets the following requirements, the installation scripts do not relabel the disk:

• The a partition meets the minimum size requirements in Table 2-2 and has an offset of 0.

• The b partition meets the minimum size requirements in Table 2-2, the partition has an offset greater than the size of partition a, and the file system type is swap.

• The h partition is exactly 1 MB in size, has an offset greater than the size of b plus the offset of b, the end of the partition is contiguous with the end of the disk, and the file system type is cnx.

Restrictions: A member boot disk cannot contain any of the clusterwide root (/), /usr, or /var file systems. A member boot disk cannot be used as the quorum disk.

Although it is not recommended, you can add filesets to a member's boot partition (the a partition). However, if the member leaves the cluster, all filesets mounted from the member's boot partition domain are force unmounted. Those filesets are not relocated.

2.5.1.4    Quorum Disk (Shared)

A quorum disk is a physical or logical disk whose h partition contains cluster status and quorum information. For example:

dsk7        quorum disk

A quorum disk helps to prevent cluster partitions, and also increases the availability of clusters. For a two-node cluster, we recommend that you configure a quorum disk, and give the quorum disk 1 vote. If each member and the quorum disk have 1 vote, the cluster can maintain quorum and continue operating as long as 2 votes are present. Therefore, the cluster can service client requests when both members are up, or when one member is up and the quorum disk is available. However, without a quorum disk, if one member fails, the cluster loses quorum and cannot service client requests.

Before deciding whether or not to configure a quorum disk, see Section 2.6 in this manual and the quorum disk information in the Cluster Administration manual. You can configure a quorum disk after creating a cluster.

Note

Because the quorum disk is accessed by cluster members during cluster transitions, no I/O barriers are placed on the quorum disk. For this reason, do not use any remaining partitions on this disk for data. If you specify a quorum disk during the installation or later with the clu_quorum command, do not specify a disk that contains data you want to preserve. A disk is relabeled when it becomes a quorum disk; any existing raw data or file systems are no longer accessible.

Restrictions: A cluster can have only one quorum disk. Locate the quorum disk on a shared bus to which all cluster members are directly connected; otherwise, members that do not have a direct connection to the quorum disk may lose quorum before members that do have a direct connection to it. A member boot disk cannot be used as the quorum disk. A disk containing a clusterwide root (/), /usr, or /var file system cannot be used as the quorum disk.

2.5.2    LSM Considerations

If you plan to use LSM to mirror cluster file systems, be aware that mirroring file systems such as root (/) (root_domain), /usr (usr_domain), or /var (var_domain) on the Tru64 UNIX system has no influence on the cluster's cluster_root, cluster_usr, or cluster_var domains. For example, if you mirror the root_domain on the Tru64 UNIX system and then run clu_create to create a cluster, the cluster_root is not automatically mirrored. Therefore, if you do not plan to use LSM to mirror file systems on the Tru64 UNIX system but you do want to use LSM to mirror file systems on the TruCluster Server cluster, you can set up LSM on the Tru64 UNIX system but defer the mirroring of file systems until you boot the first member of the cluster.

After you create a single-member cluster with clu_create, you can use the volmigrate command to move the cluster_root, cluster_usr, and cluster_var domains to LSM volumes, and optionally mirror the volumes. If you plan to migrate any of these domains to LSM volumes, be aware that the volmigrate command moves the data from the disk partition you specify to clu_create to new target storage that you must specify when you run volmigrate. (The disk partition you specify to clu_create is then no longer used for the volume and is available for other uses.) If you plan on migrating cluster_root, cluster_usr, or cluster_var to LSM volumes, consider the corresponding disk partitions you specify to clu_create as temporary storage, and plan for the additional storage required by volmigrate.

Note

To migrate the cluster_root, cluster_usr, or cluster_var domains to LSM volumes, you must use the volmigrate command. You cannot use the volencap command.

See the Cluster Administration manual for more information about using LSM in a cluster. See volmigrate(8) for information on using that command in a cluster.

2.5.3    Minimum Disk Layout for a Two-Node Cluster

For a two-node cluster, the minimum disk layout uses at least four disks (although five is recommended):

• One disk (private or shared) for the system that will become the first cluster member. This disk will hold the Tru64 UNIX Version 5.1B operating system.

• One disk on a shared bus for the clusterwide root (/), /usr, and /var file systems.

• Two disks on a shared bus for member boot partitions.

• Recommended: One disk on a shared bus for the quorum disk.

This minimum disk layout is not a no-single-point-of-failure (NSPOF) configuration. Although it puts all member boot disks on a shared bus, it does not mirror the clusterwide root (/), /usr, and /var file systems. We recommend that you mirror these critical file systems.

The Cluster Hardware Configuration manual describes NSPOF hardware configurations.

Figure 2-2 shows the relationship between the file systems these disks contain and the resulting cluster directory structure. The quorum disk is not shown because it does not contain a file system.

Figure 2-2:  Clusterwide Files and Member Boot Partitions

2.5.4    Disk Space Recommendations

Table 2-2 provides the minimum recommended size requirements for disk partitions. These size requirements are probably adequate for a cluster containing up to four members. Section 2.5.5 provides information on the disk space required to perform a rolling upgrade. Read that section as part of your disk space planning.

Table 2-2:  Disk Space Recommendations

2.5.5    Allocating Additional Disk Space in Preparation for a Rolling Upgrade

A rolling upgrade, described in Chapter 7, has the following disk space requirements:

• At least 50 percent free space in root (/), cluster_root#root.

• At least 50 percent free space in /usr, cluster_usr#usr.

• At least 50 percent free space in /var, cluster_var#var, plus, if you are updating the operating system, an additional 425 MB to hold the subsets for the new version of the base operating system.

  During a rolling upgrade, the contents of the following member-specific directories are backed up to the /var file system:

  • /cluster/members/membermemberID

  • /usr/cluster/members/membermemberID

  • /var/cluster/members/membermemberID

  These backup files are removed before the rolling upgrade completes.

  Unless /var has sufficient room for the files in these directories on all cluster members, the setup stage of the rolling upgrade will fail.

  You can minimize the amount of space needed for backup files during a rolling upgrade by doing the following:

  • Before you begin a rolling upgrade, archive the logging and accounting files in the /var/cluster/members/membermemberID/adm directories, and then remove them from the member-specific directories.

  • Do not place data in the member boot partitions, /cluster/members/membermemberID/boot_partition.

• If a separate i18n domain exists for the Worldwide Language Support (WLS) subsets, at least 50 percent free space in that domain.

• No tagged files are placed on member boot partitions. However, programs might need free space when moving kernels to boot partitions. We recommend that you reserve at least 50 MB free space on each member's boot partition.

  Note

  The minimum disk size for a member root domain is 256 MB.

You have two options:

• Increase the size of the root (/), /usr, /var, and, if using a separate domain for the WLS subsets, i18n domains when creating the cluster.

• Use AdvFS management utilities like addvol or dtadvfs to add volumes to the existing domains before starting a rolling upgrade. (The AdvFS Utilities require a separate license.)

  Note

  You cannot use the addvol command to add volumes to a member's root domain (the a partition on the member's boot disk). Instead, you must delete the member from the cluster, use diskconfig or SysMan to configure the disk appropriately, and then add the member back into the cluster.

2.6    Quorum: Member Votes and Expected Votes

Each system that has the potential to join a cluster has 0 or 1 votes. The number of votes assigned to a system is stored in its member-specific /etc/sysconfigtab file as the value of cluster_node_votes. Each system also has a value called expected votes stored in its /etc/sysconfigtab file as the value of cluster_expected_votes. In general, the value of cluster_expected_votes is the sum of the cluster_node_votes values for all potential members plus, if a quorum disk is configured, the votes assigned to the quorum disk.

The subject of member votes, quorum, and the forming and maintaining of a viable cluster is a complex topic that cannot be covered in a few paragraphs. The Cluster Administration manual describes how quorum is calculated and how cluster membership is maintained. Read that quorum information before deciding how many votes to assign to each potential cluster member during installation.

The following list describes the default node vote choices offered during installation by clu_create and clu_add_member:

Note

If you are not sure what to do for node votes during the installation, accept the default choices to ensure that the systems can form a cluster. You can always modify votes later.

• The first member always gets 1 node vote.

  Only members with votes can form a cluster; non-voting members can only join a cluster. Therefore, the first member must have a vote to form a single-member cluster.

• The default for the quorum disk is 1 vote. Legal values are 0 and 1. If you configure a quorum disk, give it 1 vote. (Do this unless you are creating a single-member cluster and do not plan to add another member for a while. A single-member cluster with a quorum disk can lose quorum if the quorum disk has a vote and the disk fails.) If your cluster will have two members, configure a quorum disk and give it 1 vote. Add the second member as soon as possible.

• The default node vote values offered for additional members are derived from the expected votes setting on the member where you run clu_add_member. If the value of expected votes is 1, the default offered by clu_add_member is 0. If the value of expected votes is greater than 1, the default offered by clu_add_member is 1. Legal values are 0 and 1.

  If you plan to create a two-member cluster with a quorum disk, give 1 vote to each member and 1 vote to the quorum disk.

In the sample deli cluster, described in Section 2.3, the first member (pepicelli) was automatically assigned 1 vote, a quorum disk was configured (1 vote), and the second member (polishham) was assigned 1 vote during installation. Therefore, the value of cluster_expected_votes is 3, and 2 votes are required for quorum.

The default vote values offered by clu_create and clu_add_member are biased towards the optimal availability of a two-member cluster (two voting members and a quorum disk). To foster availability in other cluster configurations, take into account the suggestions in Table 2-3 and Table 2-4. We strongly recommend the use of a quorum disk to increase availability for clusters with an even number of members (2, 4, 6, 8). This is especially true for small clusters. Nevertheless, to help you in those situations in which a quorum disk is not an option, Table 2-4 lists the optimal vote assignments for clusters in which a quorum disk is not configured.

Table 2-3:  Recommended Vote Assignments for a Cluster with a Quorum Disk

Table 2-4:  Recommended Vote Assignments for a Cluster Without a Quorum Disk

2.7    Console Variables

The following list contains console variables whose settings are relevant to systems in a cluster. The list indicates whether the clu_create or clu_add_member attempts to set the variable, or whether you have to set it manually.

In some cases, an attempt by clu_create or clu_add_member to set a console variable might fail. If this occurs, you must set the variable manually.

Not all console variables apply to all systems.

boot_osflags

set boot_osflags A
 

If boot_osflags is set to A, the system automatically boots to multiuser mode. The base operating system automatically sets this variable when you install the system that will become the first member of the cluster. When adding members, set this variable before booting the member for the first time.

boot_reset

set boot_reset on
 

If the value boot_reset is off, only a warm boot is performed on a system halt or boot command; if the value is on, a full reset (cold boot) is performed. Setting the value to on brings the system to a known state at each boot.

The clu_create command sets boot_reset to on for the first member during cluster creation. The clu_add_member command creates a one-time-only script that sets boot_reset to on during the first boot of the new member.

You must manually set this variable for the AlphaServer 8200 and 8400 systems.

Systems that do not support the boot_reset console variable perform the necessary initialization while shutting down. On these systems, you do not need to take any action.

bootdef_dev

set bootdef_dev boot_disk
 

The bootdef_dev variable specifies an ordered list of devices (separated by commas with no spaces) from which the system tries to boot if the boot command is entered without parameters, or if an AUTO_ACTION boot is in progress.

The clu_create command attempts to set the boot device for the first member during cluster creation. The clu_add_member command creates a one-time-only script that sets the boot device during the first boot of the new member.

You must manually set this variable for the AlphaServer 8200 and 8400 systems.

If your hardware configuration includes HS controllers that are connected to dual SCSI or Fibre Channel buses and configured for multiple-bus failover, clu_create and clu_add_member can set only one bus path. You must halt the system and manually set both values at the console before booting. For example:

>>> set bootdef_dev device1,device2
 

Separate each device with a comma; do not use any spaces. If the console is unable to boot from the first device, it will try the next device, and so on. If all paths to the device are not specified, you might see the console error message not connected when attempting to boot the member.

Note

If a system is using a Fibre Channel disk as its cluster member boot disk, neither clu_create nor clu_add_member can set the bootdef_dev variable unless you used either the wwidmgr set or wwidmgr quickset console command to configure device and port path information. Even so, clu_create and clu_add_member will set only one path. See the Fibre Channel chapter in the Cluster Hardware Configuration manual for a procedure to set multiple paths for the bootdef_dev variable when a member boot disk is accessed via Fibre Channel.

boot_dev

set boot_dev boot_disk
 

The boot_dev variable determines which disk is used for a reboot.

The clu_create command automatically creates and sets this variable for the first member during cluster creation. The clu_add_member command creates a one-time-only script that creates and sets this variable during the first boot of a new member.

You must manually set this variable for the AlphaServer 8200 and 8400 systems.

bus_probe_algorithm

set bus_probe_algorithm new
 

For systems that support the bus_probe_algorithm console variable, you must set this variable to new on the first system before installing Tru64 UNIX, and on each additional member before booting it for the first time. Setting the value of bus_probe_algorithm to new ensures that peripheral component interconnect (PCI) devices are consistently probed on all member systems.

The following AlphaServer systems support the bus_probe_algorithm console variable: 800, 1000, 1000A, 2000, 2100, and 2100A. Newer systems do not use this variable.

To find and set the bus_probe_algorithm console variable, follow these steps:

1. Shut the system down to console mode and power cycle the system. This clears any transient console variables.

2. Enter the show command:

   >>> show bus_probe_algorithm
   

   If the variable is supported, the console will display its name and its current value (either old or new).

3. If the firmware supports the bus_probe_algorithm console variable and its current value is old, set the value to new:

   >>> set bus_probe_algorithm new
    
   

2.8    Sample Cluster Configuration Checklists

Table 2-5, Table 2-6, and Table 2-7 use the checklists in Appendix A to summarize the information used for the sample deli cluster configuration.

Table 2-5:  Sample Tru64 UNIX System Attributes

Table 2-6:  Sample Cluster Attributes

Table 2-7:  Sample Member Attributes