A. DESCRIPTION On Solaris, userland applications wishing to interact with SES devices currently use a combination of uscsi(7I) and ses(7D). Both of these interfaces are difficult to use, and require significant infrastructure in each application. As part of the Sun storage strategy, encompassing both work in Solaris and in the Systems group, it is necessary to create libraries that hide such implementation details and provide clean interfaces on which higher level applications can be developed. The initial consumers of this will be the SES libtopo FMA enumerator and the unbundled CAM storage management utility, but the libraries are being delivered separately as part of this case to enable future work (both in management software and hardware development) to occur in parallel. These interfaces will initially be delivered as consolidation private, with the expectation that they will be made public after we have more experience using them in a wider variety of applications. SES is an industry standard protocol for describing SCSI device enclosures and retrieving status information from them. Refer to t10.org for background and copies of the specifications. SES operates by encoding operations inside of SCSI packets directed as particular SCSI targets that offer SES services. All new disk enclosures being produced by Sun, and many by other vendors, use the SAS protocol as the underlying transport, which in turn escapsulates SCSI, and therefore encapsulates SES. Thus SES is used to manage a SAS JBOD. SES was first introduced into Solaris for use with Tabasco back in 1996 as part of the "envsen" driver, which was renamed to "ses" by a later update. (Score 1 on the crusty old-timer list if you remember Tabasco. Score 2 if somewhere in your office right now a Tabasco is collecting dust.) The ses driver fundamentally provided the ability to statically configure the driver for a given parent and set of nodes in ses.conf so that it would bind to these nodes, and then one can use the ioctl API to read and write SES using some simple C abstractions for the SES data. The binding is described in ses(7D) and the complete history of this stuff is found in: PSARC/1996/289 envsen driver PSARC 1996/421 SES device naming PSARC/1997/195 Common SCSI Enclosure Services Driver PSARC 2001/442 Correcting Device Node Type for SES Later on, Dan "i/o Dan" Price introduced a much more generic mechanism for this kind of software: the sgen driver. Rather than having to rewrite special kernel drivers like ses for these operations, sgen is a generic SCSI pass-through driver that permits userland code to read and write the underlying SCSI protocol and anything that travels in it, like SES. Notice that if "sgen" had existed first, no "ses" driver ever would have been needed in Solaris. sgen exports a generic ioctl API described in uscsi(7I). sgen is described in: PSARC 1999/525 Generic SCSI Pass Through for Solaris PSARC 1999/561 Generic SCSI Pass Through -- Corrections and Update Therefore, the strategy for libses and libscsi, described by this case, is to provide generic services that can sit on top of uscsi and sgen, entirely ignoring the legacy "ses" driver. Future work can EOL that driver, but that is an issue entirely orthogonal to the new software in this case. The project team requests that new work build on this case, rather than the "ses" driver. B. LIBSCSI This library is a thin layer on top of uscsi(7I). It provides an abstraction for managing actions, consisting of CDB, payload, and sense data. The library interfaces with devices through a set of engines that are delivered as separate plugins. While the primary engine for Solaris is uscsi, the plan is to eventually deliver an emulation engine that allows for simplified testing or hardware prototyping. This library also allows cross-platform consumers to be written generically, by porting the basic libscsi interface but providing a different engine on different platforms. The current library header file will be provided in the case materials as an example of the interfaces. C. LIBSES The current ses(7D) driver has several major drawbacks. It is hard to use, requires specialized kernel support, and is difficult to extend. Rather than introducing further complexity into the kernel, it is preferable to construct a library that presents a higher level interface built on uscsi(7I). This library will allow for future changes to the spec as well as vendor-specific functionality. The SES-2 specification presents a straightforward topology of the form: TARGET -> ENCLOSURE -> AGGREGATE -> ELEMENT There is one implicit target at the root of the tree, but each subsequent level can have multiple children. The primary responsibility of the core libses library is to take snapshots of the device state and parse the config page into a tree of nodes, where each node has a set of properties exposed as an nvlist. Control operations are modeled as a per-node function that takes a string indicating the control operation and an nvlist of parameters. Apart from reading the supported pages and parsing the configuration page, libses itself doesn't know anything about parsing per-node properties. Instead, these properties are added by plugins, which can be either framework plugins (SES spec and libses extensions), or vendor-specific (indexed by vendor/product/revision). Each plugin can provide the following functionality: - Descriptions of how to retrieve and parse supported pages. - Method to fill in properties for a node. - Method to handle control operations on a node. This allows plugins to reinterpret existing data (such as the STRING IN page or vendor-specific enclosure data), parse data from vendor-specific pages, or define vendor-specific control operations. The ses2 plugin does the the work of interpreting the SES-2 specification. These properties are prefixed by SES_ (or "ses-" in the nvlist representation). It provides control operations for setting properties (a generic control op for a variety of features) and downloading microcode. There are also several properties that are not defined by the spec, but are common across multiple vendors and products. These properties begin with the prefix LIBSES_ (or "libses-"). The most important one is LIBSES_EN_PROP_CSN, which is the chassis serial number. The SES spec doesn't provide a way to correlate multiple SES processes within in a single logical chassis. In order for SES enumeration of enclosures to work, we need to know when two different SES targets physically reside in the same chassis (either accessing the same elements or disparate elements). Other standard properties include part number and serial number that can be attached to any element. The current header files for the library and plugins are provided in the case materials directory. D. SESTOPO While this case does not deliver any public consumers of libscsi or libses, it does introduce a Private utility, sestopo, which simply iterates over the SES topology and dumps all properties for all nodes. This provides a simple test of libses functionality, and a debugging tool for both enclosure firmware and higher level software. Some example output: Node Type: 2 nvlist version: 0 ses-enclosure-id = 0x0 ses-enclosure-service-proc-id = 0x0 ses-enclosure-service-proc-count = 0x0 ses-logical-id = (embedded nvlist) nvlist version: 0 naa-id-integer = 0x500163600004ddbd naa-id-type = 0x5 naa-company-id = 0x1636 naa-vendor-specific-a = 0x4ddbd (end ses-logical-id) ses-vendor-id = SUN ses-product-id = StorageTek J4400 ses-product-revision = 1R20 ses-element-index = 0x31 ses-element-type = 0xe ses-class-description = Enclosure ... E. DELIVERABLES AND INTERFACES All the interfaces are Contracted Consolidation Private. While it would be possible to deliver these objects in a separate package, the core FMA infrastructure will depend on these libraries, so it seems of little use to deliver them separately. These devlierables will be part of the core Solaris packages (SUNWcsl, SUNWhea, SUNWarc). The exception will be vendor specific plugins, which will be delivered as part of a new SUNWscsip package in order to facilitate creation of minimized systems (such as the miniroot) and easier patching. Summary of Exported Interfaces, all Contracted Consolidation Private: --------------------------------------------------------------------- /usr/lib/scsi SCSI libraries and private commands /usr/lib/scsi/libscsi.so.1 /usr/lib/scsi/{amd64,sparcv9}/libscsi.so.1 SCSI library /usr/lib/scsi/libses.so.1 /usr/lib/scsi/$MACH64/libses.so.1 SES library /usr/lib/scsi/plugins/scsi/engines /usr/lib/scsi/plugins/scsi/engines/uscsi.so /usr/lib/scsi/plugins/scsi/engines/$MACH64/uscsi.so SCSI engines and default system engine (uscsi). /usr/lib/scsi/plugins/ses/framework /usr/lib/scsi/plugins/ses/framework/libses.so /usr/lib/scsi/plugins/ses/framework/ses2.so /usr/lib/scsi/plugins/ses/framework/$MACH64/libses.so /usr/lib/scsi/plugins/ses/framework/$MACH64/ses2.so /usr/lib/scsi/plugins/ses/vendor /usr/lib/scsi/plugins/ses/vendor/Sun-StorageTek-J4400.so /usr/lib/scsi/plugins/ses/vendor/$MACH64/Sun-StorageTek-J4400.so SES plugins, including standard framework plugins and a sample vendor-specific plugin for the J4400. /usr/lib/scsi/sestopo Sample SES program for debugging. /usr/include/scsi /usr/include/scsi/libscsi.h /usr/include/scsi/libses.h SCSI/SES include files /usr/include/scsi/plugins/ses /usr/include/scsi/plugins/ses/framework /usr/include/scsi/plugins/ses/framework/ses2.h /usr/include/scsi/plugins/ses/framework/libses.h /usr/include/scsi/plugins/ses/vendor Plugin-specific headers. The primary purpose is to define available properties that are exported by a plugin.