Serial Attached SCSI Models

Leveraging existing facilities through protocol strategies

By Sam Barnett and David Allen

The proliferation of serial I/O technology has helped catapult storage to the forefront of new specification development. The Serial Attached SCSI (SAS) approach is one that is breathing new life into the enterprise server and storage enclosure community, making efficient use of existing software facilities by using simple protocol mechanisms. With the right deployment model, SAS can meet many different application needs.

OEMs are increasingly adopting high-speed serial I/O connections using Serial Attached SCSI (SAS) for mainstream enterprise storage applications. The approach serves as a replacement for parallel SCSI architectures, providing a flexible, low-cost storage interconnect for the mid-range and enterprise server markets. The flexibility is apparent in the three transport protocols that SAS supports:

• Serial SCSI Protocol (SSP), which provides SCSI command communication for SAS disk drives, tape drives and other SCSI targets.

• Serial ATA Tunneling Protocol (STP), which provides communications to SATA compatible disk drives and other SATA targets.

• Serial Management Protocol (SMP), which carries commands that provide a management mechanism for SAS expanders. The protocol allows the definition of custom SMP commands to support vendor-specific configuration and status messaging.

By supporting both SCSI and Serial ATA command sets in its protocol, the SAS architecture can offer two distinct economic models for the same design. Using SAS hard disk drives (HDDs) allows creation of an enterprise-level system, with the end user receiving all the performance and reliability of today’s parallel SCSI systems. A lower-cost, but less robust system can be achieved using Serial ATA HDDs in the same design configuration.

Flexibility also applies to the connection scheme a SAS system employs. Varying connections allow tradeoffs among bandwidth, scalability, and cost. Bandwidth is relatively easy to adjust in SAS systems. Under the current specification, a given SAS link can provide as much as a 3-Gbit/sec data rate with an effective 300-MByte/sec bandwidth at full utilization in a half-duplex configuration. By taking advantage of SAS’s full duplex nature, a link can provide up to 600 MByte/sec of raw performance.

SAS achieves even greater throughput by allowing the aggregation of links, called “trunking,” into “wide ports.” The wide-port configuration logically bonds multiple links to provide high bandwidth capabilities between two SAS physical layer interfaces (PHYs), typically an initiator and an expander. One of the most common wide-port configurations is a four-wide (x4) link, which can provide up to 2.4 GByte/sec of interconnect throughput.

Attachment methods allow tradeoffs

Bandwidth can be traded for system cost by choosing the method of attachment. The simplest model for attaching SAS hard disk drives (HDD) and other targets to the host processor allows for “direct” connection to the host via a host controller. In this approach, however, scalability and physical connection capacity are strictly limited to the number of PHY ports on the host controller. Additionally, this approach does not allow for aggregation of bandwidth, and therefore much of the inherent performance provided in the SAS controller goes unused. (see Figure 1)

Figure 1: A direct connection between the host controller and the storage array is the simplest configuration for Serial Attached SCSI (SAS), but not the highest performance or capacity.

Where top performance is essential, the use of expander devices allows for a more flexible and scalable design that can take advantage of the bandwidth available through the SAS host controller. Expander devices provide a switching function that links multiple targets to a host connection, but they differ in their cost, capacity, and flexibility, so that the cost and scalability of an expanded SAS implementation depend on the choice of expander device.

There are two types of SAS expanders: the edge expander and the fan-out expander.

An edge expander supports up to 128 SAS PHYs and can be used in “Edge Expander Device Sets” constructed from multiple edge expanders to connect a large number of devices. (see Figure 2)

Figure 2: The use of edge expanders increases SAS system capacity and allows �wide-port� configurations for increased reliability.

Edge expanders can also be used to create a storage subsystem architecture. The communication mechanisms between the individual expanders can be direct connections, subtractive connections, or table routers. Whichever mechanism is in use, however, there is a limit to the number of storage devices that an edge expander set can support. In general, then, edge expanders are best suited to those designs where cost is a significant consideration and storage scalability requirements are limited. (see Figure 3)

Figure 3: Another application of the edge expander is to configure the system in a modular, storage subsystem architecture.

The more flexible expander device, the fan-out expander, provides multiple tiers of expansion capability for increased system scalability and is often used in large hierarchal external storage subsystems. The fan-out expander typically employs a large internal routing table, which provides it with the capability to address multiple expanders and targets in excess of its physical connection limits. The number of expanders and targets that a fan-out device can support is limited only by its inherent table routing capacity (often set by the size of integrated memory) or by the SAS specification limit: 16,384 addresses.

Fan-out expanders also have the capability of performing self-discovery. This function provides the expander with a logical map of what is physically connected, as well as a topological understanding of all devices downstream. The function also allows the expander to have a total picture of the network topology in which it participates.

Adding more than capacity

Expanders do more in an SAS system than provide data pathways. They can employ an integrated Enclosure Management (EM) processor for remote system monitoring. EM mechanisms perform diagnostics, and determine status of the expander and related external devices. In addition, expanders often include fan-speed monitoring and PWM controls along with general purpose I/O lines to handle LEDs and system control signals. By utilizing these additional capabilities of the EM processor, system architects can differentiate their products from competitive offerings with system management software. In most cases, this management function can be handled entirely within the expander and will not affect system performance or data flow priority.

Another advantage of expanded designs is the ability to provide fault tolerant mechanisms within a single host or among multiple hosts. Failover can be achieved within a single-host system through the use of dual port drives and a redundant host controller/expander channel. This approach provides maximum data availability and protection for the system. (see Figure 4)

Figure 4: Expanded SAS architectures allow for fail-over configurations to enhance system reliability through redundancy.

The wide-port configuration also adds to system reliability. Unlike Fibre Channel, the SAS protocol does not stripe the data across individual links in a wide port. Instead, it uses a transaction mapping structure to best utilize link capacity for a given data transaction. This also helps maintain survivability in the event of a link failure; if a connection or PHY in a wide port fails, the system re-routes traffic across the remaining links. Failures thus reduce the effective bandwidth but do not sever the connection. This fault-tolerant capability is needed in applications where system integrity is key.

Large storage subsystems, using high-port-count expanders, can provide an array of other robust fault tolerance alternatives depending on the specifics of the implementation. In many cases expanders include external connections to allow for storage subsystem scalability. As the storage system grows, root expanders provide the addressing capability needed to connect all devices down stream. (see Figure 5)

Figure 5: Many fault-tolerant architectures are possible in expanded SAS systems, allowing redundant channels and the ability to add storage subsystems as needed.

Predicting the future

Yet, with all the advantages SAS offers, technologies like ATA, ESCON, Fibre Channel, FICON, and SCSI are apparently not in danger of imminent demise. As with any technology, the future of SAS will be determined by the needs of the end user. Speed grade increases and additional features will be most prevalent advances within the SAS community itself. SAS is not destined, however, to become the dominant storage architecture.

Instead, the manner in which storage systems are designed is likely to change. It is reasonable to assume that large chipset vendors will ultimately adopt a universal controller architecture. This will allow systems deploying their technology to take advantage of any storage technology by simply attaching the expansion facility and drive of choice to the interface. If the past is any predictor of the future, SAS will find itself integrated along side its brethren-Serial ATA, Fibre Channel, and Gigabit Ethernet-in highly integrated controllers and chipsets. Such integration will provide the most cost effective connections for both internal and external storage subsystems and the highest level of design flexibility.

To learn more

Detailed specification information on SAS can be found on the T10 website under www.t10.org (click “Drafts” then “Serial Attached SCSI (SAS)” under “Protocols and Physical Layers”).

The SCSI Trade Association (STA) also offers a robust set of tutorials on SAS at www.scsita.org.

---

Sam Barnett is a senior product line manager and David Allen is the director of strategic marketing at Vitesse Semiconductor Corp. The company is located in Camarillo, CA, and is an Associate Member of the Intel® Communications Alliance.