As cluster administrator, you control the number of aliases, the
membership of each alias, and the attributes specified by each member
of an alias.
You also control the alias-related attributes assigned to
ports in the
/etc/clua_services
file.
This chapter discusses the following topics:
Summary of cluster alias features (Section 3.1)
Files that control alias configuration and behavior (Section 3.2)
General planning considerations (Section 3.3)
Things to do before you create an alias (Section 3.4)
Specifying and joining a cluster alias (Section 3.5)
Modifying cluster alias and service port attributes (Section 3.6)
Monitoring cluster aliases (Section 3.7)
Leaving a cluster alias (Section 3.8)
Deleting a cluster alias (Section 3.9)
Application load balancing (Section 3.10)
Extending clusterwide port space (Section 3.11)
Enabling cluster alias vMAC support (Section 3.12)
Routing configuration guidelines (Section 3.13)
Routing daemon and static routing options (Section 3.14)
Cluster alias and NFS (Section 3.15)
Cluster alias and cluster application availability (CAA) (Section 3.16)
You can use both the
cluamgr
command and the
SysMan Menu to configure cluster aliases:
The
cluamgr
command-line interface configures
parameters for aliases on the cluster member where you run the
command.
The parameters take effect immediately; however, they do not
survive a reboot unless you also add the command lines to the
clu_alias.config
file for that member.
The SysMan Menu graphical user interface (GUI) configures static
parameters for all cluster members.
Static parameters are written to
the member's
clu_alias.config
file but do not
take effect until the next boot.
(If you want the new parameters to
take effect immediately, manually run the command lines that SysMan Menu
wrote to each member's
clu_alias.config
file.)
3.1 Summary of Cluster Alias Features
The chapter on cluster alias in the TruCluster Server Cluster Technical Overview manual describes cluster alias concepts. Read that chapter before modifying any alias or service attributes.
The following list summarizes important facts about the cluster alias subsystem:
A cluster can have multiple cluster aliases with different sets of members.
A cluster has one default cluster alias. The name of the default cluster alias is the name of the cluster.
An alias is defined by an IP address, not by a Domain Name System (DNS) name. An alias IP address can reside in either a common subnet or a virtual subnet.
A cluster member must specify an alias in order to advertise a route to it. A cluster member must join an alias to receive connection requests or packets addressed to that alias.
Each cluster member manages its own set of
aliases.
Entering a
cluamgr
command on one member
affects only that member.
Only cluster members can join cluster aliases. There is no one-to-one correspondence between a cluster alias and a specific network interface card on a cluster member. Neither is there a one-to-one correspondence between a cluster alias and a specific application on one or more cluster members. Cluster aliasing is a general-purpose mechanism to allow one or more members of a cluster to respond to a common IP address. See Section 3.16 for a description of the differences between the cluster alias and cluster application availability (CAA) subsystems.
The
/etc/clu_alias.config
file is a
context-dependent symbolic link (CDSL) that points to member-specific
cluster alias configuration files.
The commands in the file are run at
boot time.
Each member's file contains
cluamgr
command lines that:
Specify and join the default cluster alias.
The
clu_create
and
clu_add_member
commands
add the following line to a new member's
clu_alias.config
file:
/usr/sbin/cluamgr -a selw=3,selp=1,join,alias=DEFAULTALIAS
(The cluster alias subsystem automatically associates the keyword
DEFAULTALIAS
with a cluster's default cluster alias.)
Specify any other aliases that this member will either advertise a route for or join.
Set options for aliases; for example, the selection weight and routing priority.
Because each cluster member reads its copy of
clu_alias.config
at boot time, alias definitions
and membership survive reboots.
Although you can manually edit the
file, the preferred method is through the SysMan Menu.
Edits made
by SysMan do not take effect until the next boot.
If you want the new values to take effect immediately,
manually run the new
cluamgr
command line.
Members of cluster aliases whose names are in the
/etc/exports.aliases
file can accept Network
File System (NFS)
requests addressed to those aliases.
This lets you use aliases other
than the default cluster alias as NFS servers.
See
Section 3.15
for more information.
Releases prior to TruCluster Server Version 5.1B required that you run
the
gated
routing daemon in a cluster.
You could
not use
routed,
ogated, or
set up static routing and use no routing daemon at all.
Starting with
Version 5.1B you can use
gated,
routed, or static routing.
You still cannot use
ogated.
Section 3.14
describes routing options.
By default, the cluster alias subsystem uses
gated
and the Routing Information Protocol (RIP) to advertise host routes
for alias addresses.
You can disable this behavior by specifying the
nogated
option to
cluamgr.
For
example, you can use
routed -q,
gated
with a site-customized configuration file, or
static routing.
When
gated
is used as the routing daemon, the alias
daemon,
aliasd, adds host route entries to a
cluster member's
/etc/gated.conf.membern
file as needed.
The alias daemon does not modify any member's
gated.conf
file.
The network ports that are used by services that are accessed through
a cluster alias are defined as either
in_single
or
in_multi.
These definitions have nothing to do with
whether the service can or cannot run on more than one cluster member
at the same time.
From the point of view of the cluster alias
subsystem:
When a
service is designated as
in_single, only
one alias member will receive connection requests or packets that are
addressed to the service.
If that member becomes unavailable, the cluster alias
subsystem selects another member of the alias as the recipient for all
requests and packets addressed to the service.
When a service is designated as
in_multi,
the cluster alias subsystem routes connection requests and packets for
that service to all eligible members of the alias.
By default, the cluster alias subsystem treats all service ports as
in_single.
In order for the cluster alias subsystem to
treat a service as
in_multi, the service must
either be registered as an
in_multi
service
in the
/etc/clua_services
file, through a call to the
clua_registerservice()
function, or through a call
to the
clusvc_getcommport()
or
clusvc_getresvcommport()functions.
The following attributes identify each cluster alias:
Clusterwide attributes:
IP address and subnet mask: Identifies an alias
Per-member attributes:
Router priority: Determines which member of a cluster alias responds to proxy Address Resolution Protocol (ARP) requests for this alias on a common subnet.
Selection priority: Creates logical subsets of aliases within an alias. You can use selection priority to control which members of an alias normally service requests. As long as those members with the highest selection priority are up, members with a lower selection priority are not given any requests. You can think of selection priority as a way to establish a failover order for the members of an alias.
Selection weight: For
in_multi
services, provides
static load balancing among members of an alias.
It provides a simple
method for controlling which members of an alias get the most
connections.
The selection weight indicates the number of connections,
on average, that a member is given before connections are given to the
next alias member with the same selection priority.
The following configuration files manage cluster aliases and services:
/sbin/init.d/clu_aliasThe boot-time startup script for the cluster alias subsystem.
/etc/clu_alias.configA CDSL pointing to a member-specific
clu_alias.config
file, which is called from the
/sbin/init.d/clu_alias
script.
Each member's
clu_alias.config
file contains the
cluamgr
commands that are run at boot time to
configure and join aliases, including the default cluster alias, for
that member.
/etc/clua_servicesDefines port numbers, protocols, and connection attributes for
Internet services that are provided by the cluster and accessed via
cluster aliases.
The
cluamgr
command reads this
file at boot time and registers each service that has one or more
service attributes assigned to it.
If you modify the
clua_services
file, run
cluamgr -f
on each cluster member.
For more
information, see
clua_services(4)cluamgr(8)
/etc/exports.aliasesContains the names of cluster aliases, one alias per line, whose
members can accept NFS requests.
By default, the default cluster
alias is the only cluster alias that accepts NFS requests.
Use the
/etc/exports.aliases
file to specify
additional aliases as NFS servers.
/etc/gated.conf.membernBy default, each cluster member's cluster alias daemon,
aliasd, creates a member-specific
/etc/gated.conf.membern
file.
The daemon then starts
gated
using this file
as
gated's configuration file rather than the
member's
/etc/gated.conf
file.
If you stop alias routing on a cluster member with
cluamgr -r stop, the alias daemon
restarts
gated
with that member's
gated.conf
as
gated's configuration file.
3.3 General Planning Considerations
Consider the following things when planning how to configure cluster aliases on your cluster:
What services will the cluster provide to clients (for example, mail hub, applications server, NFS server, and so on)?
How many aliases are needed to support client requests effectively?
The default cluster alias might be all that you need. One approach is to use just the default cluster alias for a while, and then decide whether more aliases make sense for your configuration.
If your cluster is providing just the stock set of Internet
services that are listed in
/etc/services, the default
cluster alias should be sufficient.
By default, when a cluster is configured as a Network File System
(NFS) server, external clients must use the default cluster alias as
the name of the NFS server when mounting file systems that are exported by the
cluster.
However, you can create additional cluster aliases and use
them as NFS servers.
This feature is described in
Section 3.15
of this manual, in the
Cluster Technical Overview
manual, and in
exports.aliases(4)
If your cluster will run a third-party application that uses many system resources, you might want to use additional cluster aliases to control access to, and load balancing for, that application.
Section 3.10 provides an example that uses two cluster aliases to provide control and redundancy for a cluster that is used as a Web server.
Decide on a routing strategy based on the cluster's external network topology. Section 3.14 provides information about available routing daemon options.
Which cluster members will belong to which aliases?
If you create aliases that not all cluster members join, make sure that services that are accessed through those aliases are available on the members of the alias. For example:
If you create an alias for use as an NFS server, make sure that its members are all directly connected to the storage containing the exported file systems.
If a CAA-controlled application is accessed through a cluster alias, make sure that the CAA placement policy does not start the service on a cluster member that is not a member of that cluster alias.
Which alias attributes will each member assign to each alias it specifies or joins?
You can start by accepting the default set of attributes for an alias,
rpri=1,selw=1,selp=1, and modify attributes later.
What, if any, additional service attributes do you want to associate
with the Internet service entries in
/etc/clua_services? (The file provides default
alias port settings for a stock set of Internet services.)
Will alias addresses reside on an existing common subnet, on a virtual subnet, or on both?
On a common subnet: Select alias addresses from existing subnets to which the cluster is connected.
Note
Because proxy Address Resolution Protocol (ARP) is used for common subnet cluster aliases, if a local area network (LAN) uses routers or switches that block proxy ARP, the alias will be invisible on nonlocal segments. Therefore, in a common subnet configuration, do not configure routers or switches connecting potential clients of cluster aliases to block proxy ARP.
On a virtual subnet: The cluster alias software automatically
configures the host routes for aliases on a virtual subnet.
If
a cluster member adds the
virtual
attribute when
specifying or joining a member, that member also advertises a
network route to the virtual subnet.
Notes
A virtual subnet must not have any real systems in it.
If you specify the
nogatedoption, there is no advertising of routes to aliases on virtual subnets. These aliases will not be reachable unless client systems configure static routes to them.
The choice of subnet type depends mainly on whether the existing subnet (that is, the common subnet) has enough addresses available for cluster aliases. If addresses are not easily available on an existing subnet, consider creating a virtual subnet. A lesser consideration is that if a cluster is connected to multiple subnets, configuring a virtual subnet has the advantage of being uniformly reachable from all of the connected subnets. However, this advantage is more a matter of style than of substance. It does not make much practical difference which type of subnet you use for cluster alias addresses; do whatever makes the most sense at your site.
3.4 Things to Do Before You Create an Alias
Before specifying and joining a cluster alias with the
cluamgr
command, do the following:
For each alias, make sure that its IP address is associated with a
host name in whatever hosts table your site uses; for example,
/etc/hosts, Berkeley Internet Name Domain (BIND),
or Network Information Service (NIS).
Note
If you modify a
.rhostsfile on a client to allow nonpassword-protected logins and remote shells from the cluster, use the default cluster alias as the host name, not the host names of individual cluster members. Login requests originating from the cluster use the default cluster alias as the source address.
If any alias addresses are on virtual subnets, register the subnet with local routers. (Remember that a virtual subnet cannot have any real systems in it.)
If you want an alias, other than the default cluster alias, to respond
to NFS requests from clients, add its name to the
/etc/exports.aliases
file.
For Internet services with fixed port assignments, examine the entries
in
/etc/clua_services.
Add entries for any
additional services that will be accessed through the new alias or
aliases.
3.5 Specifying and Joining a Cluster Alias
The
cluamgr
command is the command-line interface
for specifying, joining, and managing aliases.
When you specify an
alias on a cluster member, that member is aware of the alias and can
advertise a route to the alias.
When you join a member to an alias,
that member can also receive connection requests or packets addressed
to that alias.
The simplest command that specifies, but does not join, an alias using the default values for all alias attributes is:
# cluamgr -a alias=alias
After alias routing is restarted on this cluster member with
cluamgr
-r start, the system advertises a route to the
alias and can take packets off the wire and forward them to
members of the alias that have joined the alias.
Running the
cluamgr -s
alias
command
on this member displays a status for this alias of
<ENABLED>.
The simplest command that both specifies and joins an alias using the default values for all attributes is:
# cluamgr -a alias=alias,join
After alias routing is restarted on this cluster member with
cluamgr -r start, the system advertises a route to
the alias, can take packets off the wire and forward them to members
of the alias that have joined the alias, and can receive connection
requests or packets addressed to the alias.
Running the
cluamgr -s
alias
command
on this member displays a status for this alias of
<JOINED,ENABLED>.
To specify and join a cluster alias, follow these steps:
Get a host name and IP address for the alias.
Using the SysMan Menu, add the alias.
Specify alias attributes
when you do not want to use the default values for the alias; for
example, to change the value of
selp
or
selw.
SysMan Menu only writes the command lines to a member's
clu_alias.config
file.
Putting the aliases in a
member's
clu_alias.config
file means that the
aliases will be started at the next boot, but it does not start them
immediately.
The following are sample
cluamgr
command lines from
one cluster member's
clu_alias.config
file.
All
alias IP addresses are on a common subnet.
/usr/sbin/cluamgr -a alias=DEFAULTALIAS,rpri=1,selw=3,selp=1,join /usr/sbin/cluamgr -a alias=clua_ftp,join,selw=1,selp=1,rpri=1,virtual=f /usr/sbin/cluamgr -a alias=printall,selw=1,selp=1,rpri=1,virtual=f
Manually run the appropriate
cluamgr
commands on
those members to specify or join the aliases, and to restart alias
routing.
For example:
# cluamgr -a alias=clua_ftp,join,selw=1,selp=1,rpri=1 # cluamgr -a alias=printall,selw=1,selp=1,rpri=1 # cluamgr -r start
The previous example does not explicitly specify
virtual=f
for the two aliases because
f
is the default value for the
virtual
attribute.
The
cluamgr -r
start
command updates kernel routing information so that the
cluster alias subsystem on this member will route for the new aliases.
As mentioned earlier, to join an alias and accept the default values for the alias attributes, the following command suffices:
# cluamgr -a alias=alias,join
The following example shows how to specify and join an alias on a virtual network; it is not much different from configuring an alias on a common subnet.
# cluamgr -a alias=virtestalias,join,virtual,mask=255.255.255.0
The elements that make up this
cluamgr
command do
the following:
cluamgr -a alias=virtestaliasSpecifies the alias.
This cluster member starts advertising a host
route to this alias after you run the
cluamgr -r
start
command.
joinJoins the alias. The member is eligible to receive packets addressed to the alias.
virtualAdvertises a network route to the virtual network where the
virtestalias
address resides.
mask=255.255.255.0
Explicitly defines a subnet mask. If you do not specify a subnet mask, the alias daemon on this member uses the network mask of the first interface on this system through which the virtual subnet is advertised.
If you do not want a cluster member to advertise a network route for a
virtual subnet, do not specify
virtual
or
virtual=t
for an alias in a virtual subnet.
For
example, the cluster member on which the following command is run will
specify and join the
virtestalias
alias, but will
not advertise a network route for that alias:
# cluamgr -a alias=virtestalias,join
See
cluamgr(8)3.6 Modifying Cluster Alias and Service Port Attributes
You can run the
cluamgr
command on any cluster
member at any time to modify alias attributes.
For example, to change
the selection weight of the
clua_ftp
alias for a member of that alias, enter the following command on that
member:
# cluamgr -a alias=clua_ftp,selw=2
To make this change survive a reboot, modify the entry for this alias
in this member's
clu_alias.config
file.
To modify the service port attributes associated with an Internet
service in the clusterwide
/etc/clua_services
file, follow these steps:
Using the information in
clua_services(4)/etc/clua_services.
On each cluster member, enter the following command to force
cluamgr
to reread the file:
# cluamgr -f
Note
Reloading the
clua_servicesfile does not affect currently running services. After reloading the configuration file, you must stop and restart the service.For example, the
telnetservice is started byinetdfrom/etc/inetd.conf. If you modify the service attributes fortelnetinclua_services, you have to runcluamgr -f, and then stop and restartinetdin order for the changes to take effect. Otherwise the changes take effect at the next reboot.
3.7 Monitoring Cluster Aliases
Use the
cluamgr -s all
command to display the status
of cluster aliases known to this cluster member.
For example:
# cluamgr -s all Status of Cluster Alias: deli.zk3.dec.com netmask: 0 aliasid: 1 flags: 7<ENABLED,DEFAULT,IP_V4> connections rcvd from net: 72 connections forwarded: 14 connections rcvd within cluster: 52 data packets received from network: 4083 data packets forwarded within cluster: 2439 datagrams received from network: 28 datagrams forwarded within cluster: 0 datagrams received within cluster: 28 fragments received from network: 0 fragments forwarded within cluster: 0 fragments received within cluster: 0 Member Attributes: memberid: 1, selw=3, selp=1, rpri=1 flags=11<JOINED,ENABLED> memberid: 2, selw=2, selp=1, rpri=1 flags=11<JOINED,ENABLED>
Note
The
netstat -icommand does not display cluster aliases.
For aliases on a common subnet, you can run
arp -a
on each member to determine which member is routing for an
alias.
Look for the alias name and
permanent
published.
For example:
# arp -a | grep permanent deli (16.140.112.209) at 00-00-f8-24-a9-30 permanent published
Enter the following command on each cluster member that you want to leave a cluster alias that it has joined:
# cluamgr -a alias=alias,leave
If configured to advertise a route to the alias, the member will still
advertise a route to this alias but will not be a destination for any
connections or packets that are addressed to this alias.
However, after
a cluster alias is configured, the cluster will not reject packets
addressed to that alias even after the last member of that alias
leaves the alias.
See
Section 3.9
for
information on deleting a cluster alias.
3.9 Deleting a Cluster Alias
No
cluamgr
command deletes a cluster
alias from a member's kernel list of aliases.
Even if all cluster
members that are aware of the alias leave the alias (cluamgr
-a alias=alias,leave), the
cluster will still accept connection requests addressed to the alias.
When a cluster member enables (cluamgr -a
alias=alias) or joins
(cluamgr -a
alias=alias,join), and alias
routing is restarted (cluamgr -r start),
aliasd
adds an entry for that alias to the member's
gated.conf.membern
file, and the alias is
added to that member's kernel list of aliases.
When a cluster member boots, the kernel builds a list of cluster
aliases from the information in that member's
/etc/clu_alias.config
file.
The
aliasd
daemon gets its information from the alias
list in the kernel, not from the
clu_alias.config
file.
Therefore removing an alias entry from the file and restarting
either the alias daemon or restarting alias routing does not delete
the alias.
To delete a cluster alias, do the following:
Note
Do not attempt to delete the default cluster alias. If you must change the name of the cluster, use the procedure in this manual.
Remove the entry for the cluster alias from the
/etc/hosts
file.
(Although not technically
required, removing or commenting out the entry is good housekeeping.)
On each cluster member, run the
cluamgr -s
alias
command to determine which
cluster members are aware of this alias.
On each cluster member that is aware of the alias, delete any
entry associated with that alias from the member's
/etc/clu_alias.config
file.
(Removing the entry
ensures that the alias will not be reconfigured when the system reboots.)
One at a time, reboot each cluster member that was a member of the alias you are deleting. When all cluster members that were aware of the alias are rebooted, the alias is deleted.
3.10 Application Load Balancing
This section discusses application load balancing among members of a cluster alias. For information about routing daemon configuration options that may be useful for network interface load balancing, see Section 3.14.
Note
The concept of load balancing applies only to services that are assigned the
in_multiservice attribute in the/etc/clua_servicesfile. The cluster alias subsystem treats a service asin_singleunless that service is explicitly defined asin_multi. All packets and requests for anin_singleservice go to only one member of the alias at a time.
The cluster alias subsystem does not monitor the performance of
individual cluster members and perform automatic load balancing for
in_multi
services.
You control the distribution of
connection requests by assigning a selection priority and a
selection weight for each member of an alias.
You can manually modify
these values at any time.
The selection priority attribute,
selp=n, determines the
order in which members of an alias receive new connection
requests.
The value of
n
must be an integer
from 1 to 100, inclusive.
You can use an alias's selection priority to
create a hierarchy within an alias.
For example, assume that four cluster members have joined a cluster
alias, and you have set
selp
to the following
values for that alias:
Members A and B have
selp=5.
Members C and D have
selp=4.
As long as any
selp=5
member can respond to
requests, no requests are directed to any
selp=4
members.
Therefore, as long as either A or B is capable of serving
requests, members C and D will not receive any packets or requests
addressed to this alias.
In the example, selection priority is used to create a failover hierarchy among members of a cluster alias. One reason for creating this type of hierarchy might be because members A and B are directly connected to the storage that clients will access through this cluster alias. Therefore, you want A and B to service client requests addressed to this alias. In most cases, C and D will never receive a client request, but they are capable of taking over if neither A nor B can respond.
The selection weight,
selw=n, that a member
assigns to an alias translates, on average, to the number of requests
(per application) that are directed to this member before requests are
directed to the next member of the alias with the same selection
priority.
The value of
n
must be an
integer from 1 to 100 inclusive.
(The
selp
value determines the order in which members are
eligible to receive requests or messages; the
selw
value determines how many requests or messages a member
gets after it is eligible.)
For example, assume that four cluster members
have joined a cluster alias, that all four members have the same
selp
value for the alias, and you have set
selw
to the following values:
Members A and B have
selw=3.
Members C and D have
selw=2.
Because all members of the alias have the same selection priority, the
round-robin algorithm will walk through the list of alias members,
distributing
selw
requests to each member before
moving to the next.
(The cluster alias subsystem keeps track of which
members of an alias are listening on which service ports, and uses
this information when deciding how to distribute connection requests
and packets.) For client requests addressed to this alias for a
particular service (for example,
telnet), Member A
gets three requests, then member B gets three requests, then member C
gets two requests, and so on.
When assigning selection weights to members of an alias, assign higher weights to members whose resources best match those of the application that is accessed through the alias.
Note
The default cluster alias has a
selwof3. Some applications, for example those that use the Secure Shell (SSH), use two connections to establish a single connection. For this reason, connection requests to the default cluster alias that use SSH are distributed such that some systems get two connections while some systems get one connection. There is no real distribution imbalance; connection requests are still distributed according to the selection weight assigned to each cluster member.To make connections via SSH balance among members of the default cluster alias, you can set
selwto any even number on all cluster members for the default cluster alias. To make this change persistent, add the following line to each member's/etc/clu_alias.configfile:/usr/sbin/cluamgr -a selw=4,selp=1,join,alias=DEFAULTALIAS
An administrator with shell script experience can write a script to monitor the performance of cluster members and use this information as a basis for raising or lowering alias selection weights. In this case, performance is determined by whatever is relevant to the applications in question.
As an example, assume you have a four-member cluster that you want to
configure as a Web site whose primary purpose is a file archive.
Users
will connect to the site and download large files.
The cluster
consists of four members that are connected to a common
network.
Within the cluster, members A and B share one set of disks
while members C and D share another set of disks.
The network
interfaces for members A and B are tuned for bulk data transfer (for
example,
ftp
transfers); the network interfaces for
members C and D are tuned for short timeouts and low latency (for
example, connections from the Web).
You define two cluster aliases:
clua_ftp
and
clua_http.
All four cluster members join both
aliases, but with different values.
A and B have the following lines in their
/etc/clu_alias.config
files:
/usr/sbin/cluamgr -a alias=clu_ftp,selw=2,selp=10,join /usr/sbin/cluamgr -a alias=clu_http,selw=2,selp=5,join
C and D have the following lines in their
/etc/clu_alias.config
files:
/usr/sbin/cluamgr -a alias=clu_ftp,selw=2,selp=5,join /usr/sbin/cluamgr -a alias=clu_http,selw=2,selp=10,join
The result is that as long as either A or B is up, they will handle
all
ftp
requests; as long as either C or D is up,
they will handle all
http
requests.
However,
because all four members belong to both aliases, if the two
primary servers for either alias go down, the remaining alias members
(assuming that quorum is maintained) will continue to service client
requests.
3.11 Extending Clusterwide Port Space
The number of ephemeral (dynamic) ports that are available clusterwide
for services is
determined by the
inet
subsystem attributes
ipport_userreserved_min
(default: 1024) and
ipport_userreserved
(default: 5000).
Because port space is shared among all cluster members, clusters with
several members might experience contention for available ports.
If
a cluster has more than two members, we recommend that you set the
value of
ipport_userreserved
to its maximum
allowable value (65535).
(Setting
ipport_userreserved =
65535
has no adverse side effects.)
To set
ipport_userreserved
to its maximum value,
follow these steps:
On one member of the cluster, add the following lines to the clusterwide
/etc/sysconfigtab.cluster
file to configure
members to set
ipport_userreserved
to 65535 when
they next reboot:
inet: ipport_userreserved=65535
On each member of the cluster run the
sysconfig
command to modify the current value of
ipport_userreserved:
# sysconfig -r inet ipport_userreserved=65535
3.12 Enabling Cluster Alias vMAC Support
When vMAC support is enabled, if a cluster member becomes the proxy
ARP master for a cluster alias, it creates a virtual MAC address for
use with that cluster alias.
A virtual MAC address consists of a
prefix (the default is
AA:01) followed by the IP
address of the alias in hexadecimal format.
For example, the default
vMAC address for an alias whose IP address is 16.140.112.209 is
AA:01:10:8C:70:D1:
Default vMAC prefix: AA:01
Cluster Alias IP Address: 16.140.112.209
IP address in hex. format: 10.8C.70.D1
vMAC for this alias: AA:01:10:8C:70:D1
When another cluster member becomes the proxy ARP master for this alias, the virtual MAC address moves with the alias so that a consistent MAC address is presented within the common subnet for each cluster alias.
When configuring vMAC support, configure all cluster members
identically.
For this reason, set vMAC configuration variables in
/etc/rc.config.common.
By default, vMAC support is disabled.
To enable vMAC support, use
rcmgr
to put the appropriate entry in
/etc/rc.config.common:
# rcmgr -c set VMAC_ENABLED yes
Conversely, to disable vMAC support, enter:
# rcmgr -c set VMAC_ENABLED no
To modify the default AA:01 vMAC prefix, enter:
# rcmgr -c set VMAC_PREFIX xx:xx
To manually enable or disable vMAC support on an individual cluster
member, specify the
cluamgr
vmac
or
novmac
routing option.
For example, to enable
vMAC support on a cluster member, enter:
# cluamgr -r vmac
To manually disable vMAC support on an individual cluster member, enter:
# cluamgr -r novmac
Because all cluster members should have the same vMAC settings, the recommended sequence when enabling vMAC support is as follows:
On any cluster member, enter:
# rcmgr -c set VMAC_ENABLED yes
This ensures that vMAC support is automatically enabled at boot time. However, because setting this variable only affects a member when it reboots, the currently running cluster does not have vMAC support enabled.
To manually enable vMAC support for the currently running cluster, enter the following command on each cluster member:
# cluamgr -r vmac
You do not have to add the
cluamgr -r vmac
command
to each cluster member's
/etc/clu_alias.config
file.
Running the
cluamgr -r vmac
command manually
on each member enables vMAC support now; setting
VMAC_ENABLED
to
yes
in the
shared
/etc/rc.config.common
file automatically
enables vMAC support at boot time for all cluster members.
If a cluster member is connected to the external network via a DEGPA
Gigabit Ethernet adapter whose alt driver version is less than V2.07,
and if vMAC is enabled, the member will not respond to external packets
addressed to the cluster alias unless the network interface is set to
promiscuous (promisc) mode.
(DEGPA adapters with
alt driver V2.07 and higher automatically enable promiscuous mode when
a vMAC address is configured.)
Set promiscuous mode with the following command:
# ifconfig alt0 promisc
To automatically enable promiscuous mode at boot time for all DEGPA
adapters, add the following lines to the
/etc/inet.local
file on each cluster member:
for i in `/usr/sbin/ifconfig -l` ; do
case $i in
alt*) ifconfig $i promisc ;;
esac
done
The
Cluster Technical Overview
manual provides a vMAC overview.
3.13 Routing Configuration Guidelines
Cluster alias operations require that all subnets that are connected to the cluster include a functioning router. This allows cluster alias connectivity to work without any manual routing configuration. For a connected subnet with no router, some manual routing configuration is required because the cluster alias daemons on cluster members cannot unambiguously determine and verify routes that act correctly for all possible routing topologies.
If you cannot configure a router in a subnet that is connected to the cluster (for example, one cluster member is connected to an isolated LAN containing only nonrouters), you must manually configure a network route to that subnet on each cluster member that is not connected to that subnet. For each member that is connected to a routerless subnet, add a network route to that subnet to that member's cluster interconnect interface.
Note
Multiple clusters on the same LAN can use the same virtual subnet.
This works because of host routes; any router on the LAN will see each cluster alias's individual host route, and will therefore direct packets to the correct cluster. Off of the LAN, advertisements to the virtual subnet will be propagated using the advertised network routes, and packets to cluster alias addresses in the virtual subnet will find their way to a router on the LAN. In summary, you should not need to use a separate virtual subnet for each cluster as long as (1) host routes are being generated and (2) the clusters share the same LAN.
However, using the same virtual subnet for multiple clusters is more complicated when the clusters are multi-homed. For instance, if two clusters both connect to LAN 1 but are separately connected to LAN 2 and LAN 3, using the same virtual subnet for both clusters does not work for packets that are coming into LAN 2 and LAN 3. A homogeneous LAN connection is required.
3.14 Routing Daemon and Static Routing Options
Prior to Version 5.1B, we required that you use
gated
as a routing daemon for correct operation of cluster alias routing
because the cluster alias subsystem did not coexist gracefully with
either
routed
or static routes.
This release
provides an
aliasd
daemon that does not depend on
having
gated
running in order to function correctly.
The following list describes supported routing daemon options:
The
gated
and
routed
routing
daemons are supported in a cluster.
In addition, static routing is
supported (no routing daemons required).
Because
aliasd
is optimized for
gated, using
gated
remains the
default, preferred routing daemon.
However, it is no longer
mandatory, nor is it the only way to configure routing for a cluster
member.
For example, you can configure a cluster where all members use
static routing, or some members run
routed, or use
a combination of routing daemons and static routes.
However, the existing restriction against using
ogated
still applies; do not use
ogated
as a routing daemon in a cluster.
Note
Cluster members do not have to have identical routing configurations. In general, it is simpler to configure all cluster members identically, but in some instances, an experienced cluster administrator might choose to configure one or more members to perform different routing tasks. For example, one member might have
CLUAMGR_ROUTE_ARGS="nogated"in its/etc/rc.configfile and have a fully populated/etc/routesfile. Or a member might run withnogatedandrouted -q.
The alias daemon will handle the failover of cluster alias IP
addresses via the cluster interconnect for either dynamic routing or
static routing.
If an interface fails,
aliasd
reroutes alias traffic to another member of the cluster.
As long as
the cluster interconnect is working, cluster
alias traffic can get in or out of the cluster.
Multiple interfaces per subnet (for network load balancing).
Although
gated
does not support this
configuration, because static routing is supported, an administrator
can use static (nogated) routing for network load
balancing.
By default, the cluster alias subsystem uses
gated,
customized
gated
configuration files
(/etc/gated.conf.membern),
and RIP to advertise host routes for alias addresses.
You can disable
this behavior by specifying the
nogated
option to
cluamgr, either by running the
cluamgr -r
nogated
command on a member or by setting
CLUAMGR_ROUTE_ARGS="nogated"
in that members
/etc/rc.config
file.
For a cluster, there are three general routing configuration scenarios:
The default configuration: dynamic routing where
aliasd
controls
gated:
Each member has the following line in its
/etc/rc.config
file:
GATED="yes"
If the
CLUAMGR_ROUTE_ARGS
is present in a member's
/etc/rc.config
file, it should either be
commented out, or set to a null string.
If needed, define static routes in that member's
/etc/routes
file.
Note
Static routes in
/etc/routesfiles are installed before routing daemons are started, and are honored by routing daemons.
In this configuration,
aliasd
controls the operation
of
gated.
Host routes are propagated via RIP and
aliasd
can leverage
gated
to
learn paths out of a cluster on behalf of a node that becomes isolated
because of network interface card (NIC) failures.
The
aliasd
daemon can stop and restart
gated, and change the routing table without any
interference from
gated.
Dynamic routing where
gated
and
aliasd
operate independently:
Members run
gated, but the cluster alias subsystem and
aliasd
are independent of it.
The administrator has
total control over
gated
and each member's
gated
configuration file,
/etc/gated.conf.
This approach is useful for an
administrator who wants to enable IP forwarding and configure a
cluster member
as a full-fledged router.
Each member that will follow this policy has the following line in its
/etc/rc.config
file:
GATED="yes" CLUAMGR_ROUTE_ARGS="nogated" ROUTER="yes" #if this member will be a full-fledged router
If needed, define static routes in that member's
/etc/routes
file.
In this scenario,
aliasd
cannot use
gated
to propagate host routes, and cannot leverage
gated
to help cluster nodes learn paths out of the
cluster.
The
gated
daemon owns the routing table.
Static routing: one or more cluster members do not run a routing daemon.
Each member that will use static routing has the following line in its
/etc/rc.config
file:
GATED="no" CLUAMGR_ROUTE_ARGS="nogated" ROUTED="no" ROUTED_FLAGS=""
Define static routes in that member's
/etc/routes
file.
This configuration is useful for sites that do not want to run a
routing daemon.
This may be due to local administrative policies
prohibiting RIP; it may be due to the desire to run multiple NICs in a
subnet (which
gated
does not support.) You must
set up at least one default route;
aliasd
controls
the routing table and has the capacity to reconfigure this default
route in the event of a NIC failure that would otherwise leave the
node isolated.
When a cluster is configured as an NFS server, NFS client requests
must be directed either to the default cluster alias or to an alias
listed in
/etc/exports.aliases.
NFS mount requests
directed at individual cluster members are rejected.
As shipped, the default cluster alias is the only alias that NFS
clients can use.
However, you can create additional cluster
aliases.
If you put the name of a cluster alias in the
/etc/exports.aliases
file, members of that alias
accept NFS requests.
This feature is useful when some members of a
cluster are not directly connected to the storage that contains
exported file systems.
In this case, creating an alias whose members
are directly connected to this storage can reduce the number of
internal hops that are required to service an NFS request.
As described in the
Cluster Technical Overview
manual, you must make sure that
the members of a cluster alias serving NFS requests are directly connected to
the storage containing the exported file systems.
In addition, if any
other cluster members are directly connected to this storage but are
not members of the alias, you must make sure that these systems do
not serve these exported file
systems.
Only members of the alias used to access these file systems
should serve these file systems.
One approach is to use
cfgmgr
to manually relocate these file systems to
members of the alias.
Another option is to create boot-time scripts
that automatically learn which members are serving these file systems
and, if necessary, relocate them to members of the alias.
Before configuring additional aliases for use as NFS servers, read the
sections in the
Cluster Technical Overview
manual that describe how NFS
and the cluster alias subsystem handles NFS-related TCP and UDP
traffic.
Also see
exports.aliases(4)/etc/exports.aliases
file.
3.16 Cluster Alias and Cluster Application Availability
This section provides a general discussion of the differences between the cluster alias subsystem and cluster application availability (CAA).
There is no obvious interaction between the two subsystems. They are independent of each other. CAA is an application-control tool that starts applications, monitors resources, and handles failover. Cluster alias is a routing tool that handles the routing of connection requests and packets addressed to cluster aliases. They provide complementary functions: CAA decides where an application will run; cluster alias decides how to get there, as described in the following:
CAA is designed to work with applications that run on one cluster member at a time. CAA provides the ability to associate a group of required resources with an application, and make sure that those resources are available before starting the application. CAA also handles application failover, automatically restarting an application on another cluster member.
Because cluster alias can distribute incoming requests and packets among multiple cluster members, it is most useful for applications that run on more than one cluster member. Cluster alias advertises routes to aliases, and sends requests and packets to members of aliases.
One potential cause for confusion is the term
single-instance
application.
CAA uses this term
to refer to an application that runs on only one cluster member at a
time.
However, for cluster alias, when an application is designated
in_single, the alias subsystem sends requests and
packets to only one instance of the application, no matter how many
members of the alias are listening on the port that is associated with
the application.
Whether the application is running on all cluster
members or on one cluster member, the alias subsystem arbitrarily
selects one alias member from those listening on the port and directs
all requests to that member.
If that member stops responding, the
alias subsystem directs requests to one of the remaining members.
In the
/etc/clua_services
file, you can designate a
service as either
in_single
or
in_multi.
In general, if a service is in
/etc/clua_services
and is under CAA control,
designate it as an
in_single
service.
However, even if the service is designated as
in_multi, the service operates properly for
the following reasons:
CAA makes sure that the application is running on only one cluster member at a time. Therefore, only one active listener is on the port.
When a request or packet arrives, the alias subsystem checks all members of the alias, but will find that only one member is listening. The alias subsystem then directs all requests and packets to this member.
If the member can no longer respond, the alias subsystem does not find any listeners, and either drops packets or returns errors until CAA starts the application on another cluster member. When the alias subsystem becomes aware that another member is listening, it directs incoming packets to the new port.
All cluster members are members of the default cluster alias. However, you can create a cluster alias whose members are a subset of the entire cluster. You can also restrict which cluster members CAA uses when starting or restarting an application (favored or restricted placement policy).
If you create an alias and tell users to access a CAA-controlled application through this alias, make sure that the CAA placement policy for the application matches the members of the alias. Otherwise, you can create a situation where the application is running on a cluster member that is not a member of the alias. The cluster alias subsystem cannot send packets to the cluster member that is running the application.
The following four scenarios illustrate the interaction of cluster alias and
service attributes with CAA.
Scenario 1
The application is not under CAA control. For each alias, the cluster alias subsystem recognizes which cluster members have joined that alias. When a client request uses that alias as the target host name, the alias subsystem sends the request to one of its members based on the following criteria:
If the requested service has an entry in the
clua_services
file, the port attribute values that are
specified there.
For example,
in_single
versus
in_multi, or
in_nolocal
versus
in_noalias.
Assume that the example service is
designated as
in_multi.
The selection priority (selp) that each member has
assigned to the alias.
The selection weight (selw) that each member has
assigned to the alias.
The alias subsystem uses
selp
and
selw
to determine which
members of an alias are eligible to receive packets and connection
requests.
Whether this eligible member is listening on the port associated with the application.
If so, forward the connection request or packet to that member.
If not, look at the next member of the alias that meets the
selp
and
selw
requirements.
Scenario 2
The application is under CAA control, but someone has mistakenly
designated the application as a
in_multi
service in
clua_services.
The cluster alias subsystem receives a connection request or packet.
Of all eligible alias members, only one is listening (because CAA runs the application on only one cluster member).
The cluster alias subsystem determines that it has only one
place to send the connection request or packet, and sends it to the
member where CAA is running the application (the
in_multi
is, in essence, ignored).
Scenario 3
The application is not under CAA control and is running on several
cluster members.
All instances bind and listen on the same well-known
port.
However, the entry in
clua_services
for this
service is not designated
in_multi; therefore,
the cluster alias subsystem treats the port as
in_single:
The cluster alias subsystem receives a connection request or packet.
The port is
in_single.
The cluster alias subsystem arbitrarily picks an eligible member of the alias to receive the connection request or packet.
The cluster alias subsystem sends connection requests or packets only to that member until the member goes down or the application crashes, or for some reason there is no longer an active listener on that member.
Scenario 4 (not recommended)
This scenario demonstrates how not to combine CAA and cluster alias:
Cluster members A and B join a cluster alias.
CAA controls an application that has a restricted host policy and can run on either node A or node C. Users access the service through the cluster alias.
The application is running on node A. Node A fails. CAA relocates the application to node C.
Users cannot access the service through the alias, even though the application is running on node C.