Monitoring Disk Paths
This section describes two methods for monitoring disk paths in your cluster. The first method is provided by the scdpm command. Use this command to monitor, unmonitor, or display the status of disk paths in your cluster. This command is also useful for printing the list of faulted disks and monitoring disk paths from a file.
The second method for monitoring disk paths in your cluster is provided by the SunPlex Manager graphical user interface (GUI). SunPlex Manager provides a topological view of the monitored disk paths in your cluster. The view is updated every 10 minutes to provide information about the number of failed pings. Use the information that is provided by the SunPlex Manager GUI in conjunction with the scdpm(1M) command to administer disk paths. Refer to "Administering Sun Cluster With the Graphical User Interfaces" in Sun Cluster 3.1 9/03 System Administration Guide for information about SunPlex Manager.
Using the scdpm Command to Monitor Disk Paths
The scdpm(1M) command provides DPM administration commands that enable you to perform the following tasks:
Monitoring a new disk path
Unmonitoring a disk path
Rereading the configuration data from the CCR database
Reading the disks to monitor or unmonitor from a specified file
Reporting the status of a disk path or all disk paths in the cluster
Printing all the disk paths that are accessible from a node
Issue the scdpm(1M) command with the disk-path argument from any active node to perform DPM administration tasks on the cluster. The disk-path argument is always constituted of a node name and a disk name. The node name is not required and defaults to all if none is specified. The following table describes naming conventions for the disk path.
Note - Use of the global disk-path name is strongly recommended, because the global disk-path name is consistent throughout the cluster. The UNIX disk-path name is not consistent throughout the cluster. The UNIX disk path for one disk can differ from cluster node to cluster node. The disk path could be c1t0d0 on one node and c2t0d0 on another node. If you use UNIX disk-path names, use the scdidadm -L command to map the UNIX disk-path name to the global disk-path name before issuing DPM commands. See the scdidadm(1M) man page.
Table 3-3 Sample Disk-Path Names
Name Type | Sample Disk Path Name | Description |
|---|---|---|
Global disk path | schost-1:/dev/did/dsk/d1 | Disk path d1 on the schost-1 node |
all:d1 | Disk path d1 on all nodes in the cluster | |
UNIX disk path | schost-1:/dev/rdsk/c0t0d0s0 | Disk path c0t0d0s0 on the schost-1 node |
schost-1:all | All disk paths on the schost-1 node | |
All disk paths | all:all | All disk paths on all nodes of the cluster |
Using SunPlex Manager to Monitor Disk Paths
SunPlex Manager enables you to perform the following basic DPM administration tasks:
Monitoring a disk path
Unmonitoring a disk path
Viewing the status of all disk paths in the cluster.
Refer to the SunPlex Manager online help for procedural information about how to perform disk-path administration by using SunPlex Manager.
Quorum and Quorum Devices
Because cluster nodes share data and resources, it is important that a cluster never splits into separate partitions that are active at the same time. The CMM guarantees that at most one cluster is operational at any time, even if the cluster interconnect is partitioned.
There are two types of problems that arise from cluster partitions: split brain and amnesia. Split brain occurs when the cluster interconnect between nodes is lost and the cluster becomes partitioned into sub-clusters, each of which believes that it is the only partition. This occurs due to communication problems between cluster nodes. Amnesia occurs when the cluster restarts after a shutdown with cluster data older than at the time of the shutdown. This can happen if multiple versions of the framework data are stored on disk and a new incarnation of the cluster is started when the latest version is not available.
Split brain and amnesia can be avoided by giving each node one vote and mandating a majority of votes for an operational cluster. A partition with the majority of votes has a quorum and is allowed to operate. This majority vote mechanism works fine as long as there are more than two nodes in the cluster. In a two-node cluster, a majority is two. If such a cluster becomes partitioned, an external vote is needed for either partition to gain quorum. This external vote is provided by a quorum device. A quorum device can be any disk that is shared between the two nodes. Disks used as quorum devices can contain user data.
Table 3-4 describes how Sun Cluster software uses quorum to avoid split brain and amnesia.
Table 3-4 Cluster Quorum, and Split-Brain and Amnesia Problems
Partition Type | Quorum Solution |
|---|---|
Split brain | Allows only the partition (sub-cluster) with a majority of votes to run as the cluster (where at most one partition can exist with such a majority); once a node loses the race for quorum, that node panics |
Amnesia | Guarantees that when a cluster is booted, it has at least one node that was a member of the most recent cluster membership (and thus has the latest configuration data) |
The quorum algorithm operates dynamically: as cluster events trigger its calculations, the results of calculations can change over the lifetime of a cluster.
Quorum Vote Counts
Both cluster nodes and quorum devices vote to form quorum. By default, cluster nodes acquire a quorum vote count of one when they boot and become cluster members. Nodes can also have a vote count of zero, for example, when the node is being installed, or when an administrator has placed a node into maintenance state.
Quorum devices acquire quorum vote counts based on the number of node connections to the device. When a quorum device is set up, it acquires a maximum vote count of N-1 where N is the number of connected votes to the quorum device. For example, a quorum device connected to two nodes with non zero vote counts has a quorum count of one (two minus one).
You configure quorum devices during the cluster installation, or later by using the procedures described in the Sun Cluster 3.1 System Administration Guide.
Note - A quorum device contributes to the vote count only if at least one of the nodes to which it is currently attached is a cluster member. Also, during cluster boot, a quorum device contributes to the count only if at least one of the nodes to which it is currently attached is booting and was a member of the most recently booted cluster when it was shut down.
Quorum Configurations
Quorum configurations depend on the number of nodes in the cluster:
Two-Node Clusters - Two quorum votes are required for a two-node cluster to form. These two votes can come from the two cluster nodes, or from just one node and a quorum device. Nevertheless, a quorum device must be configured in a two-node cluster to ensure that a single node can continue if the other node fails.
More Than Two-Node Clusters - You should specify a quorum device between every pair of nodes that shares access to a disk storage enclosure. For example, suppose you have a three-node cluster similar to the one shown in Figure 3-3. In this figure, nodeA and nodeB share access to the same disk enclosure and nodeB and nodeC share access to another disk enclosure. There would be a total of five quorum votes, three from the nodes and two from the quorum devices shared between the nodes. A cluster needs a majority of the quorum votes to form.
Specifying a quorum device between every pair of nodes that shares access to a disk storage enclosure is not required or enforced by Sun Cluster software. However, it can provide needed quorum votes for the case where an N+1 configuration degenerates into a two-node cluster and then the node with access to both disk enclosures also fails. If you configured quorum devices between all pairs, the remaining node could still operate as a cluster.
See Figure 3-3 for examples of these configurations.
Figure 3-3 Quorum Device Configuration Examples
Quorum Guidelines
Use the following guidelines when setting up quorum devices:
Establish a quorum device between all nodes that are attached to the same shared disk storage enclosure. Add one disk within the shared enclosure as a quorum device to ensure that if any node fails, the other nodes can maintain quorum and master the disk device groups on the shared enclosure.
You must connect the quorum device to at least two nodes.
A quorum device can be any SCSI-2 or SCSI-3 disk used as a dual-ported quorum device. Disks connected to more than two nodes must support SCSI-3 Persistent Group Reservation (PGR) regardless of whether the disk is used as a quorum device. See the chapter on planning in the Sun Cluster 3.1 Software Installation Guide for more information.
You can use a disk that contains user data as a quorum device.