Maintaining Storage Arrays
This section contains the procedures about how to maintain a storage array in a Sun Cluster environment.
Some maintenance procedures in Table 1-2 are performed the same as in a noncluster environment. This section references these procedures, but this section does not contain these procedures. Table 1-2 lists the procedures about how to maintain the storage array.
Note - When you upgrade firmware on a storage device or on an enclosure, redefine the stripe size of a LUN, or perform other LUN operations, a device ID (DID) might change unexpectedly. When you perform a check of the DID configuration by running the scdidadm -c command, the following error message appears on your console if the DID changed unexpectedly.
device id for nodename:/dev/rdsk/cXtYdZsN does not match physical device's id for ddecimalnumber, device may have been replaced. |
To fix DIDs that report this error, run the scdidadm -R command for each affected device.
Table 1-2 Tasks. Maintaining a Storage Array
Task | Information |
|---|---|
storage array procedures. | |
Add a storage array to a running cluster. | |
Remove a storage array from a running cluster. | |
Replace a failed storage array. To replace a failed storage array, remove the failed storage array. Add a new storage array to the configuration. | |
Add a disk drive. | |
Replace a disk drive. | |
Remove a disk drive. | |
Upgrade disk drive firmware. | |
Replace a failed controller or restore an offline controller. | How to Replace a Failed Controller or Restore an Offline Controller |
Replace a node's host adapter. | |
The following is a list of administrative tasks that require no cluster-specific procedures. See the Sun StorEdge A1000 and D1000 Installation, Operations, and Service Manual, the Netra st A1000/D1000 Installation and Maintenance Manual and the Netra st A1000/D1000 Installation and Maintenance Manual for the following procedures.
The following is a list of administrative tasks that require no cluster-specific procedures. See the Sun StorEdge RAID Manager User's Guide and the Sun StorEdge RAID Manager Release Notes for the following procedures.
How to Add a Pair of Storage Arrays
Use this procedure to add a pair of storage arrays to a running cluster.
This procedure uses an updated method for setting the scsi-initiator-id. For this storage device, the method published in earlier documentation is still applicable. However, if your cluster configuration uses a Sun StorEdge PCI Dual Ultra3 SCSI host adapter to connect to any other shared storage, you will need to update your nvramrc script and set the scsi-initiator-id by using this procedure.
Install the RAID Manager software on nodes.
For the procedure about how to install RAID Manager software, see the Sun StorEdge RAID Manager Installation and Support Guide.
Note - RAID Manager 6.22 or a compatible version is required for clustering with Sun Cluster 3.0.
Install any storage array patches on nodes.
Note - For the most current list of software, firmware, and patches that are required for the storage array, refer to Info Docs 20029, A1000/A3x00/A1000FC Software/Firmware Configuration Matrix. This document is available online to Sun service providers and to customers with SunSolve service contracts at the SunSolve site. http://sunsolve.sun.com.
Set the Rdac parameters in the /etc/osa/rmparams file on both nodes.
Rdac_RetryCount=1 Rdac_NoAltOffline=TRUE
Power on the storage array.
To power on the storage array, push the power switch to the momentary on position, which is the right side. Release the power switch.
Shut down the Node 1.
For the procedure about how to shut down and power off a node, see your Sun Cluster system administration documentation.
If you are installing new host adapters, power off Node 1.
For the procedure about how to shut down and power off a node, see your Sun Cluster system administration documentation.
Install the host adapters in Node 1.
For the procedure about how to install host adapters, see the documentation that shipped with your host adapters and nodes.
Cable the storage array to Node 1, as shown in Figure 1-3.
If you are adding a storage array, connect the differential SCSI cable between Node 1 and the storage array. Verify that the entire SCSI bus length in each SCSI chain is less than 25m. This measurement includes the cables to both nodes, as well as the bus length that is internal to each enclosure, node, and host adapter.
Since the storage array is not an auto-terminating device, ensure that you place differential SCSI terminators as shown in the figures below.
Figure 1-3 Storage Array Cabling
The following figure illustrates how to cable daisy-chained storage arrays to one node in the cluster. If you place two storage arrays on a SCSI chain, ensure that each storage array's ID dial is set to a unique number.
Figure 1-4 Storage Array Cabling with Daisy Chaining
Ensure that each device in the SCSI chain has a unique SCSI address.
To avoid SCSI-chain conflicts, this procedure instructs you to reserve SCSI address 7 for one host adapter in the SCSI chain and change the other host adapter's global scsi-initiator-id to an available SCSI address. Then the procedure instructs you to change the scsi-initiator-id for local devices back to 7.
Note - A slot in the storage array might not be in use. However, do not set the scsi-initiator-id to a SCSI address for that disk slot. This precaution minimizes future complications if you install additional disk drives.
If necessary, power on the storage devices.
For the procedure on powering on a storage device, see the service manual that shipped with your storage device.
If necessary, power on a node, but do not allow it to boot. If necessary, halt the system to continue with OBP Monitor tasks.
Set the scsi-initiator-id for one node to 6.
{1} ok setenv scsi-initiator-id 6 scsi-initiator-id = 6Find the paths to the host adapters that connect to the local disk drives.
{0} ok show-disksUse this information to change the SCSI addresses in the nvramrc script. Do not include the /sd directories in the device paths.
Edit the nvramrc script to set the scsi-initiator-id for the local devices on the first node to 7.
For a full list of commands, see the OpenBoot 2.x Command Reference Manual.
Caution - Insert exactly one space after the first double quote and before scsi-initiator-id.{0} ok nvedit 0: probe-all 1: cd /pci@1f,4000/scsi@2 2: 7 " scsi-initiator-id" integer-property 3: device-end 4: cd /pci@1f,4000/scsi@3 5: 7 " scsi-initiator-id" integer-property 6: device-end 7: install-console 8: banner[Control C] {0} okStore the changes.
The changes you make through the nvedit command are recorded on a temporary copy of the nvramrc script. You can continue to edit this copy without risk. After you complete your edits, save the changes. If you are not sure about the changes, discard them.
To store the changes, type:
{0} ok nvstore {1} ok
To discard the changes, type:
{0} ok nvquit {1} ok
Verify the contents of the nvramrc script that you created, as shown in the following example.
If the contents of the nvramrc script are incorrect, use the nvedit command to make corrections.
{0} ok printenv nvramrc nvramrc = probe-all cd /pci@1f,4000/scsi@2 7 " scsi-initiator-id" integer-property device-end cd /pci@1f,4000/scsi@3 7 " scsi-initiator-id" integer-property device-end install-console banner {1} okInstruct the OpenBoot PROM (OBP) Monitor to use the nvramrc script, as shown in the following example.
{0} ok setenv use-nvramrc? true use-nvramrc? = true {1} ok
Perform a reconfiguration boot on Node 1 to create the new Solaris device files and links.
Check the storage array NVSRAM file and firmware revisions. If necessary, install the most recent revision.
To verify that you have the current revision, see the Sun StorEdge RAID Manager Release Notes. For the procedure about how to upgrade the NVSRAM file and firmware, see the Sun StorEdge RAID Manager User's Guide.
Shut down Node 2.
For the procedure about how to shut down and power off a node, see your Sun Cluster system administration documentation.
If you are installing new host adapters, power off Node 2.
For the procedure about how to shut down and power off a node, see your Sun Cluster system administration documentation.
Install the host adapters in Node 2.
For the procedure about how to install host adapters, see the documentation that shipped with your nodes.
Cable the storage array to your node, as shown in Figure 1-5.
Connect the differential SCSI cable between the node and the storage array. Make sure that the entire SCSI bus length in each SCSI chain is less than 25m. This measurement includes the cables to both nodes, as well as the bus length that is internal to each enclosure, node, host adapter.
Figure 1-5 Storage Array Cabling
The following figure illustrates how to cable daisy-chained storage arrays to the remaining node in the cluster.
Figure 1-6 Storage Array With Daisy Chaining
Verify that the scsi-initiator-id is set correctly on the second node.
If necessary, power on the second node, but do not allow it to boot. If necessary, halt the system to continue with OBP Monitor tasks.
Verify that the scsi-initiator-id for each host adapter on the second node is set to 7
Use the show-disks command to find the paths to the host adapters that are connected to these enclosures. Select each host adapter's device tree node, and display the node's properties to confirm that the scsi-initiator-id for each host adapter is set to 7.
{0} ok cd /pci@6,4000/pci@3/scsi@5 {0} ok .properties scsi-initiator-id 00000007 ...
Perform a reconfiguration boot on Node 2 to create the new Solaris device files and links.
On one node, verify that the DIDs have been assigned to the storage array LUNs for all nodes. These nodes are attached to the storage array.
# scdidadm -L