FIBRE CHANNEL: Standards, Applications, and Products

Walter Goralski and Gary Kessler
December 1995

An edited version of this paper appeared with the title
"Changing Channels" in Interoperability Report, a supplement to
Network VAR, April 1996, and LAN Magazine, May 1996.

In 1988, the American National Standards Institute (ANSI) formed the X3T9.3 Task Group (now known as X3T11) to develop a high-performance serial link for data transfer between mainframes, supercomputers, workstations, and intelligent peripheral devices. The result of this effort has been Fibre Channel (FC), a family of standards that define a communications interface for the transfer of large amounts of data between a variety of hardware systems. Fibre Channel is an enabling technology that offers many advantages because of its high-speed and low-latency capabilities, and is becoming the network protocol of choice for many high-bandwidth applications.

Fibre Channel combines the concepts of computer channels and data networks to provide an interconnection strategy that is different from both traditional channel and network architectures. Traditional computer channels provide a high-speed connection over relatively short point-to-point links, usually operate at hardware speeds without a dependence on software control, and interconnect a relatively small number of stations. Traditional networks provide a low- to moderate-speed connection using some form of switching over relatively large distances, experience wasted bandwidth due to software processing delays and protocol overhead, and are able to interconnect any pair of devices that are attached to the network. Unfortunately, typical channels rely too heavily on a central processor for anything other than the most basic error control and typical networks rely too heavily on end-user stations for error detection and correction.

Fibre Channel moves the complexity of device interconnection and switching to a Fabric. A Fibre Channel end-station, or Node, is responsible only for managing a simple point-to-point connection between itself and the Fabric; the Fabric, in turn, may be a switch or series of switches, and is responsible for routing between Nodes, error detection and correction, and flow control. Within the Fabric itself, additional parallel paths may be added to a particular connection if additional bandwidth is required, potentially overcoming any limitations of the physical connection between the Node and the Fabric. The operation of the Fabric is independent of the higher layer communications protocols, largely distance-insensitive, and may be based on any technology.

Fibre Channel, then, could relieve system manufacturers from the burden of supporting the variety of computer channels and networks currently in place, as it provides one standard for networking, storage and data transfer.

Topology

Fibre Channel provides a logically point-to-point serial channel for the transfer of data between a buffer at a source device and a buffer at a destination device. In this way, Fibre Channel avoids the problems of handling different network communications protocols; it merely moves buffer contents from one port to another without regard to the format or meaning of the data. Fibre Channel only provides control of the transfer and simple error detection. Fibre Channel supports a number of possible topologies (Figure 1):

• Point-to-point: The simplest topology utilizes bidirectional point-to-point links that interconnect the N_Ports of a pair of Nodes.
• Fabric: A switching fabric may be present, in which case the links provide a bidirectional connection between a Node (N_Port) and the Fabric (F_Port). Since FC relies on port logging between Nodes and the Fabric, it is actually irrelevant whether the Fabric is a switch, a hub, or some other technology. Generic Fabric requirements and characteristics of the switch fabric are defined in the FC-FG and FC-SW standards, respectively.
• Loop: The newest topology type is an arbitrated loop, defined in the FC-AL standard (X3.272). The Loop topology interconnects L_Ports at the Nodes and/or the Fabric via unidirectional links; the nomenclature NL_port and FL_Port refers to an N_Port or F_Port, respectively, that can support arbitrated loop functions in addition to the basic point-to-point functions. The loop protocol permits an L_Port to continuously arbitrate to access the medium to transmit to another L_Port; a fairness algorithm ensures that no L_Port gets blocked from accessing the loop.

   NODE                  NODE
----------            ----------
|        |----------->|        |
| N_Port |            | N_Port |      Point-to-point configuration
|        |<-----------|        |
----------            ----------


   NODE                  NODE
----------            ----------
| L_Port |----------->| L_Port |
----------            ----------
    A                      |
    |                      |          Loop configuration
    |                      V
----------            ----------
| L_Port |<-----------| L_Port |
----------            ----------
   NODE                  NODE


N_Port        N_Port        N_Port
-----         -----         -----
|   |         |   |         |   |
-----         -----         -----
  |             |             |
  |             |             |
  |        -----+-----        |
  ---------+-o  o  o-+---------
           |         |        
N_Port     | Fabric  |      N_Port
-----      |         |      -----
|   |------+-o     o-+------|   |      Fabric
-----      | F_Ports |      -----
           |         |         
  ---------+-o  o  o-+---------
  |        -----+-----        |
  |             |             |
  |             |             |   
-----         -----         -----
|   |         |   |         |   |
-----         -----         -----
N_Port        N_Port        N_Port

FIGURE 1. Fibre Channel topologies.

Each of these topologies has its advantages and disadvantages. The point-to-point topology is non-blocking since each N_Port can transmit to another N_Port at any time, within the limits of the higher-layer protocols. Although this provides instantaneous access to other N_Ports, it also usually underutilizes the bandwidth of the communications link. The Fabric topology can be configured to be non-blocking by providing multiple paths between any two F_Ports. This topology also provides efficient sharing of the available bandwidth between all of the N_Ports by taking advantage of the bursty nature of the communication between high-speed peripheral devices, although this sharing adds contention. The Loop topology provides one communications channel bandwidth to share among all L_Ports, resulting in a configuration with a high level of connectivity. On the other hand, there is also the possibility of blocking since there can be only one active L_Port-L_Port connection at one time. In addition, should any link in a Loop fail, communication between all L_Ports is terminated. The Loop is essentially the simplest form of a Fabric topology, and these two together provide a compromise between connectivity and performance.

Protocol Architecture

The Fibre Channel architecture is structured as a hierarchical set of protocol layers, as shown in Figure 2. The bottom-most layers comprising the Fibre Channel Physical and Signaling Interface (FC-PH) are described in ANS X3.230 and comprise three protocol layers known as FC-0, FC-1, and FC-2.

        ==========Channels========== ====Networks====                       
        ----- ------ ------- ------- ----- ---- -----
FC-4    |IPI| |SCSI| |HIPPI| |SBCCS| |LLC| |IP| |ATM|
        ----- ------ ------- ------- ----- ---- -----
        ---------------------------------------------
FC-3    |             Common services               |
        ---------------------------------------------
        ---------------------------------------------  --
FC-2    |         Framing and flow control          |   |
        ---------------------------------------------   |
        ---------------------------------------------   |
FC-1    |      Transmission, encoding/decoding      |   |- FC-PH
        ---------------------------------------------   |
        ---------------------------------------------   |
FC-0    |  Media, transmitter, link length, speed   |   |
        ---------------------------------------------  --

FIGURE 2. Fibre Channel protocol architecture.

FC-0 is the lowest functional layer of the Fibre Channel architecture and describes the physical characteristics of the link connections. FC-0 options include:

• Transmission speeds of 12.5, 25, 50, and 100 megabytes per second (MBps)
• Single-mode or multimode fiber, coaxial cable, or twisted pair media
• Point-to-point link lengths up to 10 km.

FC-1 defines the transmission protocol, including the serial encoding and decoding rules, special characters, and error control. FC-1 uses an 8B/10B block code, where every 8-bit byte is coded using 10 bits; the 12.5, 25, 50, and 100 MBps data rates, then, correspond to transmission rates of 133, 266, 531, and 1060 Mbps, respectively. This coding scheme guarantees an adequate number of signal transitions to maintain line synchronization as well as the ability to send special control characters.

FC-2 describes how data is transferred between Nodes and includes the definition of the frame format, frame sequences, communications protocols, and service classes. The basic unit of data transmission in Fibre Channel is a variable-sized Frame. Frames can be up to 2,148 bytes in length and can carry up to 2,048 bytes of user data; every Frame contains 36 bytes of overhead that provides framing, source and destination port addressing, service type, and error detection information and up to 64 bytes of additional optional overhead for other miscellaneous information about the user data. A single higher layer protocol message may be larger than a Frame's payload capacity; in that case, the message will be fragmented into a series of Frames called a Sequence.

FC-2 defines three classes of service. Class 1 is a connection-oriented (virtual circuit) service, where two Nodes must establish a logical connection prior to any transfer of data. This type of service guarantees a maximum bandwidth between the two communicating Nodes, as well as sequential ordering of Frames. Class 1 service is best provided by the Fabric and is intended for sustained, high-throughput applications.

Class 2 is an acknowledged connectionless service that allows a channel's bandwidth to be shared amongst several different sources simultaneously. Although neither Frame delivery nor sequentiality is guaranteed, acknowledgments are used to notify the sender of the receipt of data Frames. This type of service might be used for applications where the connection setup time would be greater than the duration of short messages, but where notification of delivery is desired.

Class 3 is a pure connectionless, or datagram, service. It is similar to Class 2, except that there is no verification of Frame delivery.

An Intermix service is also defined as an option of Class 1. With Intermix, Class 1 Frames are guaranteed a certain amount of bandwidth on the channel, while Class 2 and 3 Frames are multiplexed on the channel only when sufficient bandwidth is available.

FC-3 provides a common set of communication services for higher layer protocols above the FC-PH layer. These additional services might include mechanisms for multicast and broadcast data delivery, hunt groups so more than one N_Port can respond to a given address, and multiplexing multiple higher layer protocols and the FC-PH.

FC-4 is the top layer of the Fibre Channel protocol architecture and defines the higher layer applications that can operate over a FC infrastructure. The FC-4 provides a way to utilize existing protocols over Fibre Channel without modifying those protocols. FC-4, then, acts like a protocol convergence layer so that the FC Node appears to provide the exact lower-layer transport services that the higher-layer protocol requires. This convergence function may require that the FC-4 provide additional services such as buffering, synchronization, or prioritization of data. FC-4 mappings have been specified or proposed for a number of higher layer channel and network protocols, including:

• Intelligent Peripheral Interface (IPI-3)
• Small Computer System Interface (SCSI)
• High-Performance Parallel Interface (HIPPI) (X3.254)
• IBM's Block Multiplexer Channel Single Byte Command Code Set (SBCCS) (X3.271)
• IEEE 802.2 Logical Link Control (LLC)
• Internet Protocol (IP)
• Asynchronous Transfer Model (ATM) Adaptation Layer 5 (AAL5)

A Fibre Channel Node contains the functions of FC-0 through FC-4; any resident higher-layer protocols are beyond the scope of the Fibre Channel specifications. Fibre Channel provides a range of implementation possibilities and purposely isolates the transmission medium from the control protocol so that each implementation may use a technology best suited to the application environment.

Applications

The suitable applications for Fibre Channel closely mirror those that are candidates for ATM, although Fibre Channel may have several advantages over the much-hyped ATM technology. Fibre Channel networks can take advantage of the fact that it is a channel technology amd, therefore, offers even lower delays than ATM networks running at similar speeds. Fibre Channel switches remain unaffected by traffic loads, also as befitting a channel running from a CPU to a peripheral.

ATM reacts to congestion by discarding traffic. This is clearly not an option for Fibre Channel. It would hardly be acceptable to discard virtual memory pages on their way to a Fibre Channel attached disk drive. An added plus is the fact that Fibre Channel easily handles multivendor and multiprotocol environments, which ATM still struggles with in most cases.

So the best fit applications for Fibre Channel are those application requiring even lower and more stable delays than ATM, yet need the high speeds that ATM excels at providing. Also, the application cannot be tied to one vendor's equipment and cannot assume that one protocol would be acceptable to all devices accessing the Fibre Channel network.

Fortunately for the potential vendors and users of Fibre Channel equipment, applications with such stringent requirements abound. Medical facilities have used Fibre Channel to transfer medical X-ray images with sizes in excess of 100 MB from scanner to supercomputer to screen. In fact, Fibre Channel medical applications make it possible to not only relay a remote patient's heart monitor and vital sign output from emergency room to doctor's office in real-time, but can also send the patient's latest CAT scans within a few minutes.

The field of electronic publishing is also a natural one for Fibre Channel. Glossy magazine ads originate on computers just like the magazine's articles text. These images are not 600 dots per inch (dpi), but 2400 dpi. They do not employ 256 colors, but frequently 16 million colors. The files are often on the order of 100 MB for a single page, even after compression. Fibre Channel can move them from ad agency to customer to publisher in minutes.

As more and more movies like Jurassic Park, Toy Story, and Jumanji are more fully created entirely within a computer (actually, on a LAN), Fibre Channel may become the only reasonable way to produce animation or the animated content of motion pictures. In contrast, the "studio" that made the famous Coca-Cola polar bear commercials had an output with a 10 Mbps Ethernet of ten minutes of film -- for the entire year!

Data applications will not be neglected by any means. Fibre Channel can easily mix large file transfers and delay-sensitive traffic through a switch. Servers can be backed up even while the on-line transactions are busily committing without delay.

The application opportunities for Fibre Channel will only increase as time goes on.

Products

Fibre Channel products were one of the success stories at NetWorld+Interop in Atlanta. A number of vendors contributed to a display of an impressive array of products from adapter cards to switching fabrics. But as impressive as the products were, there was still a prototype feel to many of the offerings.

The fact remains that Fibre Channel products have not yet attained the stability or user popularity (or even the visibility) of other standard, high-speed networking products like switched Ethernet. This is not to say that Fibre Channel products are somehow unreliable or even rare. They are neither. But these products have tended to have somewhat annoying quirks that are undocumented, driving would be implementers to distraction in some cases.

For instance, the Ancor Fibre Channel adapter for the SGI EISA bus only works in the top-most SGI slot, a notable undocumented "feature". The Solaris OS needs a patch to allow for large TCP/IP windows (RFC 1323) that is not generally available. And the IBM Microchannel version requires an upgrade to the firmware for acceptable performance. The disheartening aspect of all this is that Ancor is generally acknowledged to be the industry leader in Fibre Channel products.

The trouble may be well worth it. A 266 Mbps (25 MBps) Fibre Channel network should cost about the same per port as a 155 Mbps ATM network. Much of this price differential is due to the relatively modest aims of the Fibre Channel technology compared to higher ambitions of ATM.

At least 28 vendors have Fibre Channel products available. Many more vendors have products planned and these are appearing every month. The products tend to fall into one of four categories: fabrics, adapter boards, disk storage, and test equipment. Other vendors make Fibre Channel support products such as cables, optical and electrical drivers, chipsets, software, and even offer education courses. Some vendors, of course, intend to deliver the whole range of products, while others will only market products in one or two categories.

In addition to falling into one of these major component categories, Fibre Channel products also naturally group themselves based on the supported topology configuration(s). Most vendors have entered the Fibre Channel market in the arbitrated loop category, since the use of L_Ports does not require the presence of a switch or hub. A handful of vendors make Fibre Channel switching fabrics and point-to-point products will appear on the market soon.

There are an impressive number of vendors of arbitrated loop Fibre Channel adapter cards. The most active of these has been Ancor Communications, which also makes Fibre Channel-based routers and a HIPPI converter. But Fibre Channel adapters are also available from Adaptec, Bus Logic, Emulex, Genroco, Hewlett Packard, IBM, Interphase, Jaycor, Network Systems, Sun, Symbios, Systran, and Western Digital.

As long as the implementer has the fortune to be building Fibre Channel on a PCI or S-Bus architecture, there will be no shortage of vendors to choose from. At least six different vendors make adapter cards for each the PCI bus (Ancor, Emulex, Genroco, Interphase, Systran, and Western Digital) and the S-Bus architecture (Ancor, Emulex, Genroco, Jaycor, Sun, and Systran). With any other architecture, the choices fall off to two or three; only Ancor and IBM, for example, support the MCA bus with their Fibre Channel interface products, while Ancor and Systran support the VME bus. The EISA bus is not so bad off, with Ancor, HP, and Systran providing adapters. As should be obvious, Ancor and Systran are the two companies that have been the most committed to bringing Fibre Channel to almost any hardware architecture.

When it comes to Fibre Channel switching fabrics, only three vendors' products are really ready for production networks. Ancor, Hewlett Packard, and IBM all make switching fabrics which support 622 Mbps port speeds and very low-latency switching times (on the order of 10 microseconds).

All Fibre Channel fabrics are non-blocking, which allows for full connectivity between all ports. These fabrics allow for the implementation of the different classes of services defined in Fibre Channel. For example, Class 1 dedicated connections can be used for large file transfers while Class 2 or 3 can provide "datagram" services for smaller data transfers. The advantage of employing a switching fabric in a Fibre Channel network is that the fabric provides a higher total throughput than the arbitrated loop, but at a higher cost.

The Ancor switch is called the FCS 266/1062. This Fibre Channel switch is available in three models with 8 to 64 port configurations, and has been the workhorse of most switching fabric Fibre Channel networks to date. A new model has a multistage option intended to accommodate a fabric containing 3,000 nonblocking ports.

The Hewlett Packard switch offering is the OpenSwitch Series I. This switch comes in a basic 16-port size. HP intends to upgrade the port speed to 1 Gbps, but at this point it is hard to see just what would be hooked up to a 1 Gbps switch port.

The IBM product is the 7319 Model 100. The basic configuration of 8 ports can be expanded to 16 ports.

As mentioned above, most Fibre Channel networks built to date have employed basic arbitrated loop or switch fabric configurations. Products based on the Fibre Channel point-to-point configuration have yet to appear in quantity, but should be available soon.

One of the issues holding up the development of standard Fibre Channel products has been the moving target presented by the interface for converting the Fibre Channel 8B/10B parallel data into a serial bit stream on fiber. These interfaces provide complete Fibre Channel FC-0 functionality on a simple card. Also included are the transmit and receive optics, drivers, clock and data recovery, laser safety features, and so forth.

Clearly, the development of a standard set of features and a common architecture for the Fibre Channel media interface is a key to making the production of Fibre Channel products routine, producing the ever-falling prices that any new technology needs for mass acceptance. The trouble is that no fewer than three different schemes exist, which can loosely be called old, new, and newer.

The older method is known as Optical Link Cards (OLC). Most current Fibre Channel products are based on OLC. This daughter board transfers 10 bits at a time and uses a 48-pin connector. Speeds are from 266 to 531 Mbps.

The new method is known as Gigabaud Link Module (GLM). Many new Fibre Channel products are based on this method and GLM will probably supersede OLC products soon. This daughter board transfers 20 bits at a time and uses an 80-pin connector. Speeds range from 266 Mbps to 1 Gbps.

The newer method is known as the 10-bit interface specification. This is primarily intended for electrical (not optical) connections in arbitrated loop configurations, but is not limited to this. Many disk drive vendors are of course interested in this particular method, since linking disk drives with fiber optic cable is not a common practice.

Speaking of disk drive vendors, Compaq, Maximum Strategy, Quantum, Seagate, Storage Concepts, Storage Dimensions, Sun, UNISYS, and Western Digital have all planned support for Fibre Channel disk storage in one form or another. This list will also expand over time.

Additional Information

A number of Internet sites provide information about Fibre Channel standards, products, and applications. Perhaps the best starting point is CERN's Fibre Channel Standard page on the World-Wide Web at http://www.cern.ch/HSI/fcs. This page provides links to several documents (including a FC pocket reference guide) and other FC sites on the net.

Another source of information is the Fibre Channel Association (FCA), an industry consortium formed in January 1993 to encourage use of Fibre Channel and to complement ANSI's standardization activities. The primary objective of the FCA is to provide an industry-wide support structure for Fibre Channel product manufacturers, system integrators, hardware and software suppliers, consultants, and service providers. Interested readers can join the FCA's Internet discussion list by sending e-mail to fca-request@amcc.com or calling them at 1-800-272-4618. Alternatively, the FCA's WWW site provides pointers to many sites with information about products, specifications, meeting minutes, education sources, and other FC-related organizations (http://www.amdahl.com/ext/CARP/FCA/FCA.html).

Walter Goralski and Gary Kessler are both Senior Members of Technical Staff at Hill Associates, a data communications education and consulting firm located in Colchester, Vermont. Walter's latest book, Exploring Internet Virtual Worlds with VRML (co-authored with Matthew Poli and Peter Vogel), and the third edition of Gary's ISDN book (co-authored with P. Southwick) are both due in the summer. Their e-mail addresses are w.goralski@hill.com and kumquat@hill.com, respectively.