An interesting characteristic of the SAN is the way it “pumps” data across the interconnection. Because the SAN bus is asynchronous, there is no clocking associated with the data transfer. A synchronous interconnection, in contrast, includes a clock in the control lines and a fixed protocol for communicating that is relative to the clock. A synchronous interconnection requires every attachment to run at the same clock rate, which usually means all attachments must be in close proximity. Asynchronous interconnections work well over longer distances, but they usually require some handshaking protocol where the sender and receiver proceed to the next step only after both agree. This handshaking protocol can reduce the throughput on a given interconnection while the sender waits for confirmation from the receiver that a transmission was received.

When used as a loop, the SAN can bypass this protocol, letting the sender continuously pump packets of data onto the interconnection. The receiver adds the verification to the packet that it was received, and this verification is returned to the sender when the packet completes the loop and is removed by the sender. This approach allows multiple units connected to the SAN to more efficiently use the available bandwidth.

Still another use for the SAN is to provide a connection between systems. Figure 10.1 shows two additional SAN ports coming out of the system. Note that if, instead of using a loop topology, we use these ports to directly connect two systems, the total bandwidth per port is 500 MB/sec (250 MB/sec in each direction). To provide higher availability for this system connection, two ports are usually operated in parallel to provide redundant path routing between systems. If one connection fails for any reason, the systems can still communicate using the redundant connection. Later implementations of SAN, even with redundancy, operate at 1 GB/sec, which, as we will see in Chapter 11, will be ideal for connecting systems in a cluster.

The I/O cards shown in Figure 10.1 contain a PowerPC-based IOP, its memory, and the support hardware necessary to present either an SPD bus interface or a PCI bus interface. The I/O adapter cards (not shown in this figure) then are plugged into either an SPD card cage or a PCI card board that connects to its respective bus, either SPD or PCI. Note that the SPD I/O adapters still contain their own IOPs. In this case, we have two levels of IOPs, one between the SAN interconnection and the SPD bus and another between the SPD bus and the device.

The PCI I/O card packaging used in the AS/400e series is also designed to be shared with the RS/6000 and possibly with other IBM systems. Because the RS/6000 does not use IOPs for its I/O, but rather does all I/O processing in the main processors, similar to a PC, we had to find a way to eliminate the IOP from the PCI I/O card. For that, the IOP can be removed and replaced with a “bridge” chip, which essentially connects the PCI bus directly to the SAN loop and thus to the 6xx buses, allowing the main processors to control the PCI bus directly.

The SPD Bus

We have just seen that an AS/400e system can support SPD, PCI, or both buses on the same system. Being an industry standard, the PCI bus is better known than the SPD bus, so we will not spend much more time discussing its details. Because the SPD bus is still the primary means for most customers to attach their I/O to the system, especially with the SPD-only support for high-end systems, we need to look a little more closely at the SPD bus before I go on to describe the I/O operations for the AS/400.

The SPD I/O cards shown in Figure 10.1 provide the SPD bus to which the SPD I/O adapters are connected. In the terminology from the earlier models of the AS/400, each SPD I/O card provides the equivalent of a bus control unit (BCU) for one SPD bus on the system. Up to 32 SPD I/O adapter cards, which are not shown in the figure, can attach to each SPD bus. The SPD I/O adapter card contains an IOP and the associated hardware to attach one or more devices to the system.

Each IOP on an SPD I/O adapter card has its own memory and contains its own operating system. These special-purpose, real-time operating systems are designed primarily for I/O device control. The application running in the IOP is obviously tuned for the specific device interface, communications interface, or LAN interface that the IOP supports. From an I/O operational viewpoint, the IOP, including all its software, is called an I/O bus unit (IOBU). Thus, attached to the SPD buses are the IOBUs, and these IOBUs provide the interface to the I/O devices.

Recall from Chapter 8 that the processor uses a direct-store address to uniquely identify the bus, the IOBU, and the device. The IOBU receives the command from the system processor, causes the device to perform the requested operation, and controls the data transfers to and from the main memory. When the I/O operation is completed, the IOBU responds to the system processor with status information to tell whether the operation was successful. We examine these operations in more detail in the following sections. First, let’s look at the way these components are physically connected and some of their characteristics.

SPD I/O Hardware Connections

The SPD bus operates asynchronously. To understand why an asynchronous design was chosen for the SPD bus, remember our discussion on the SAN loop. As we noted earlier, a synchronous bus requires every device on the bus to run at the same clock rate, which means the devices must be in close proximity. The PCI bus is a synchronous bus with a clock rate of 33 MHz. This type of bus is fast and its adapters are usually less expensive than an asynchronous bus, whose adapters have to supply their own clocking. However, because an asynchronous bus is not clocked, it can accommodate a wide variety of devices and can be lengthened without worry about clock skew or synchronization. This added flexibility is why the asynchronous SPD bus was originally selected for the AS/400.

Unlike a SAN loop, the SPD bus is not a loop and has to use a handshaking protocol. A handshaking protocol requires that the sender and receiver proceed to the next step only after both agree. The separate control lines with the SPD bus implement this protocol. For example, the sender may put a read request on the control lines and an address on the data lines. These control signals are asserted until the receiver indicates the control lines have been seen by putting a signal on the acknowledge control line.