Maintaining Storage Systems

This section contains the procedures about how to maintain a storage system in a Sun Cluster environment.

Some maintenance procedures in Table 1-2 are performed the same as in a noncluster environment. This section references these procedures, but this section does not contain these procedures. Table 1-2 lists the procedures about how to maintain a storage system.

Note - When you upgrade firmware on a storage device or on an enclosure, redefine the stripe size of a LUN, or perform other LUN operations, a device ID (DID) might change unexpectedly. When you perform a check of the DID configuration by running the scdidadm -c command, the following error message appears on your console if the DID changed unexpectedly.

 device id for nodename:/dev/rdsk/cXtYdZsN does not match physical 
device's id for ddecimalnumber, device may have been replaced.

To fix DIDs that report this error, run the scdidadm -R command for each affected device.

Table 1-2 Tasks: Maintaining a Storage System

Task	Information
Add a storage system.	How to Add a Storage System
Remove a storage system.	How to Remove a Storage System
Replace a failed controller module. Restore an offline controller module.	How to Replace a Failed Controller or Restore an Offline Controller
Upgrade controller module firmware and NVSRAM file.	How to Upgrade Controller Module Firmware in a Running Cluster
Add a disk drive.	How to Add a Disk Drive in a Running Cluster
Replace a disk drive.	How to Replace a Failed Disk Drive in a Running Cluster
Remove a disk drive.	How to Remove a Disk Drive From a Running Cluster
Upgrade disk drive firmware.	How to Upgrade Disk Drive Firmware in a Running Cluster
Replace a host adapter in a node.	How to Replace a Host Adapter in a Node (StorEdge A3500 Storage System Only) How to Replace a Host Adapter in a Node (StorEdge A3500FC Storage System Only)

With the exception of one step, the following is a list of administrative tasks that require no cluster-specific procedures. Shut down the cluster, and then see the Sun StorEdge A3500/A3500FC Controller Module Guide, the Sun StorEdge A1000 and D1000 Installation, Operations, and Service Manual, and the Sun StorEdge Expansion Cabinet Installation and Service Manual for the following procedures. See the Sun Cluster system administration documentation for procedures about how to shut down a cluster.

Replacing a power cord that connects to the cabinet power distribution unit (see the Sun StorEdge Expansion Cabinet Installation and Service Manual).
Replacing a power cord to a storage array (see the Sun StorEdge A1000 and D1000 Installation, Operations, and Service Manual).
Replacing the power supply housing.
Replacing the battery unit.
Replacing a DC-power or battery harness.
Replacing a controller module assembly.
Replacing a controller module power cord.
Replacing a controller module card cage.

The following is a list of administrative tasks that require no cluster-specific procedures. See the Sun StorEdge A3500/A3500FC Controller Module Guide, the Sun StorEdge RAID Manager User's Guide, the Sun StorEdge RAID Manager Release Notes, the Sun StorEdge FC-100 Hub Installation and Service Manual, and the documentation that shipped with your FC hub or FC switch for the following procedures.

Replacing a storage array-to-host SCSI cable.
Replacing a controller module fan canister.
Replacing the power supply fan canister.
Replacing a SCSI cable from the controller module to the storage array.
Replacing a storage array-to-host or storage array-to-hub fiber-optic cable.
Replacing an FC hub (see the Sun StorEdge FC-100 Hub Installation and Service Manual).
Replacing an FC hub gigabit interface converter (GBIC) or Small Form Factor Pluggable (SFP) that connects cables to the host or hub.
Replacing a GBIC or an SFP on a node.
Replacing an FC switch.

How to Add a Storage System

Use this procedure to add a storage system to a running cluster.

This procedure uses an updated method for setting the scsi-initiator-id. For this storage array, the method published in earlier documentation is still applicable. However, if your cluster configuration uses a Sun StorEdge PCI Dual Ultra3 SCSI host adapter to connect to any other shared storage, you will need to update your nvramrc script and set the scsi-initiator-id by using this procedure.

Install the RAID Manager software.
For the procedure about how to install RAID Manager software, see the Sun StorEdge RAID Manager Installation and Support Guide.

Note - RAID Manager 6.22 or a compatible version is required for clustering with Sun Cluster software.

Note - For the most current list of software, firmware, and patches that are required for the storage system, refer to EarlyNotifier 20029, A1000/A3x00/A3500FC Software/Firmware Configuration Matrix. This document is available online to Sun service providers and to customers with SunSolve service contracts at the SunSolve site: http://sunsolve.sun.com.

Install the storage system patches.
For the location of patches and installation instructions, see your Sun Cluster release notes documentation.

Set the Rdac parameters in the /etc/osa/rmparams file.
Rdac_RetryCount=1 Rdac_NoAltOffline=TRUE

Power on the storage system.
For the procedure about how to power on the storage system, see the Sun StorEdge A3500/A3500FC Controller Module Guide.

Which type of system you are adding?
- If you are adding a StorEdge A3500 system, proceed to Step 6.
- If you are adding a StorEdge A3500FC system, set the loop ID of the controller module. Set the loop ID by installing jumpers to the appropriate pins on the rear of the controller module.
  
  For diagrams and information about how to set FC-AL ID settings, see the Sun StorEdge A3500/A3500FC Controller Module Guide.

Are you installing new host adapters in your nodes?
- If no, skip to Step 8.
- If yes, shut down and power off Node 1.
  
  For the procedure about how to shut down and power off a node, see your Sun Cluster system administration documentation.

Install the host adapters in Node 1.
For the procedure about how to install host adapters, see the documentation that shipped with your host adapters and nodes.

Cable the storage system to Node 1. The method you choose to cable your system depends on the type of system you are adding.
- If you are adding a StorEdge A3500 system, connect the differential SCSI cable between the node and the controller module, as shown in Figure 1-3. Make sure that the entire SCSI bus length to each enclosure is less than 25 m. This measurement includes the cables to both nodes, as well as the bus length that is internal to each enclosure, node, and host adapter.
- If you are installing a StorEdge A3500FC storage system, see Figure 1-4 for an example about how to cable a StorEdge A3500FC storage system. In this example, Node 1 connects a StorEdge A3500FC controller module.
  
  For more sample configurations, see the Sun StorEdge A3500/A3500FC Hardware Configuration Guide.
  
  For the procedure about how to install the cables, see the Sun StorEdge A3500/A3500FC Controller Module Guide.
Note - If you want to create a SAN by using two FC switches and Sun SAN software, see "SAN Solutions in a Sun Cluster Environment" in Sun Cluster 3.x Hardware Administration Manual for more information. Sun SAN software does not support storage array.

Figure 1-3 Sample Storage System Cabling

Figure 1-4 Sample Storage System Cabling (1st Node Attached)

Did you power off the Node 1 to install a host adapter?
- If no, proceed to Step 10.
- If yes, power on Node 1 and the storage system. Do not enable the node to boot. If necessary, halt the system to continue with OpenBoot PROM (OBP) Monitor tasks.

Which type of controller module are you adding?
- If a StorEdge A3500FC controller module, skip to Step 12.
- If a StorEdge A3500 controller module, proceed to Step 11.

Ensure that each device in the SCSI chain has a unique SCSI address.

To avoid SCSI-chain conflicts, this procedure instructs you to reserve SCSI address 7 for one host adapter in the SCSI chain and change the other host adapter's global scsi-initiator-id to an available SCSI address. Then the procedure instructs you to change the scsi-initiator-id for local devices back to 7.

Note - A slot in the storage array might not be in use. However, do not set the scsi-initiator-id to a SCSI address for that disk slot. This precaution minimizes future complications if you install additional disk drives.

If necessary, power on the storage devices.
For the procedure on powering on a storage device, see the service manual that shipped with your storage device.
If necessary, power on a node, but do not allow it to boot. If necessary, halt the system to continue with OBP Monitor tasks.

Set the scsi-initiator-id for one node to 6.

{1} ok setenv scsi-initiator-id 6
scsi-initiator-id =   6

Find the paths to the host adapters that connect to the local disk drives.
{0} ok show-disks
Use this information to change the SCSI addresses in the nvramrc script. Do not include the /sd directories in the device paths.

Edit the nvramrc script to set the scsi-initiator-id for the local devices on the first node to 7.

For a full list of commands, see the OpenBoot 2.x Command Reference Manual.

Caution! Caution - Insert exactly one space after the first double quote and before scsi-initiator-id.

{0} ok nvedit 
 0: probe-all 
 1: cd /pci@1f,4000/scsi@2 
 2: 7 " scsi-initiator-id" integer-property
 3: device-end 
 4: cd /pci@1f,4000/scsi@3 
 5: 7 " scsi-initiator-id" integer-property 
 6: device-end
 7: install-console
 8: banner[Control C] 
{0} ok

Store the changes.
The changes you make through the nvedit command are recorded on a temporary copy of the nvramrc script. You can continue to edit this copy without risk. After you complete your edits, save the changes. If you are not sure about the changes, discard them.
- To store the changes, type:
  {0} ok nvstore {1} ok
- To discard the changes, type:
  {0} ok nvquit {1} ok

Verify the contents of the nvramrc script that you created, as shown in the following example.

If the contents of the nvramrc script are incorrect, use the nvedit command to make corrections.

{0} ok printenv nvramrc
nvramrc =             probe-all
                      cd /pci@1f,4000/scsi@2 
                      7 " scsi-initiator-id" integer-property
                      device-end 
                      cd /pci@1f,4000/scsi@3
                      7 " scsi-initiator-id" integer-property
                      device-end 
                      install-console
                      banner
{1} ok

Instruct the OpenBoot PROM (OBP) Monitor to use the nvramrc script, as shown in the following example.
{0} ok setenv use-nvramrc? true use-nvramrc? = true {1} ok

Did you power off the Node 1 to install a host adapter?
- If no, skip to Step 18.
- If yes, perform a reconfiguration boot to create the new Solaris device files and links.

Are you installing new host adapters in Node 2 to connect Node 2 to the storage system?
- If no, skip to Step 18.
- If yes, shut down and power off the node.
  
  For the procedure about how to shut down and power off a node, see your Sun Cluster system administration documentation.

Install the host adapters in Node 2.
For the procedure about how to install host adapters, see the documentation that shipped with your nodes.

Cable the storage system to your node. The method that you choose to cable your storage system depends on the type of controller module you are adding.
- If you are adding a StorEdge A3500 controller module, connect the differential SCSI cable between the node and the controller module, as shown in Figure 1-3. Make sure that the entire SCSI bus length to each enclosure is less than 25 m. This measurement includes the cables to both nodes, as well as the bus length that is internal to each enclosure, node, and host adapter.
- If you are installing a StorEdge A3500FC controller module, see Figure 1-5 for an example about how to cable a StorEdge A3500FC storage system. In this example, two nodes connect to a StorEdge A3500FC controller module.
  
  For more sample configurations, see the Sun StorEdge A3500/A3500FC Hardware Configuration Guide.
  
  For the procedure about how to install the cables, see the Sun StorEdge A3500/A3500FC Controller Module Guide.
Note - If you want to create a SAN by using two FC switches and Sun SAN software, see "SAN Solutions in a Sun Cluster Environment" in Sun Cluster 3.x Hardware Administration Manual for more information. Sun SAN software does not support storage array.

Figure 1-5 Cabling a Storage System: An Example

Did you power off Node 2 to install a host adapter?
- If no, skip to Step 18.
- If yes, power on the node and the storage system. Do not enable the node to boot. If necessary, halt the system to continue with OpenBoot PROM (OBP) Monitor tasks.

Verify that Node 2 recognizes the new host adapters and disk drives.
If the node does not recognize the new hardware, check all hardware connections and repeat installation steps you performed in Step 14.
{0} ok show-disks ... b) /sbus@6,0/QLGC,isp@2,10000/sd... d) /sbus@2,0/QLGC,isp@2,10000/sd...{0} ok

Which type of controller module are you adding?
- If a StorEdge A3500FC controller module, skip to Step 24.
- If a StorEdge A3500 controller module, Step 19.

Verify that the scsi-initiator-id is set correctly on the second node.
1. If necessary, power on the second node, but do not allow it to boot. If necessary, halt the system to continue with OBP Monitor tasks.
2. Verify that the scsi-initiator-id for each host adapter on the second node is set to 7
  Use the show-disks command to find the paths to the host adapters that are connected to these enclosures. Select each host adapter's device tree node, and display the node's properties to confirm that the scsi-initiator-id for each host adapter is set to 7.
  {0} ok cd /pci@6,4000/pci@3/scsi@5 {0} ok .properties scsi-initiator-id 00000007 ...

Did you power off Node 2 to install a host adapter?
- If no, skip to Step 24.
- If yes, perform a reconfiguration boot to create the new Solaris device files and links.

Check the controller module NVSRAM file revision. If necessary, install the most recent revision.
For the NVSRAM file revision number and boot level, see the Sun StorEdge RAID Manager Release Notes. For the procedure about how to upgrade the NVSRAM file, see the Sun StorEdge RAID Manager User's Guide.

Verify the controller module firmware revision. If necessary, install the most recent firmware revision.
For the revision number and boot level of the controller module firmware, see the Sun StorEdge RAID Manager Release Notes. For the procedure about how to upgrade the controller firmware, see How to Upgrade Controller Module Firmware in a Running Cluster.

One node at a time, boot each node into cluster mode.
# reboot

On one node, verify that the DIDs have been assigned to the LUNs for all nodes. These nodes are attached to the storage system.
# scdidadm -L

Verify that the controller module is set to active/active mode.
For more information on controller modes, see the Sun StorEdge RAID Manager Installation and Support Guide and the Sun StorEdge RAID Manager User's Guide.


1. Installing and Maintaining a Sun StorEdge A3500 System or a Sun StorEdge A3500FC System Configuring Storage Systems How to Correct Mismatched DID Numbers