Fasttrack: PSARC/2007/595 Name: Serial Attached SCSI (SAS) Management Protocol (SMP) support for Solaris Submitters: Javen Wu, River Li, Dolpher Du, James Mcpherson 1) Background ------------- The Serial Attached SCSI (SAS) is the next generation of the SCSI protocol. The Serial Management Protocol (SMP) is a component protocol of SAS which defines the management standard used by SMP initiator to communicate with SMP target. SAS based Host Bus Adapters (HBAs) usually function as a SMP initiator to transport SMP requests and responses. SAS expanders are devices similar to switches in the Fibre Channel and Ethernet protocols. And in the context of SAS, they become the proxies of SMP targets that respond SMP requests. The primary role of SMP is to allow the host or the initiator to monitor and control SMP target devices. SMP defines a suite of management functions and their invocations between initiators/targets are through request/response frame format messages. The document proposes a new driver smp(7D) in Solaris to interact with SAS expanders. At present there is no equivalent to the uscsi(7I) passthru interface which allows Solaris to manage SAS expander devices and SAS domains from userland applications. We propose to add an SMP analog to uscsi(7I), called usmp(7I) which will enable userland applications and utilities to perform SMP management tasks. The LSI SAS chips driven by the mpt(7D) has the ability to communicate with SAS expander through SMP. The first phase of the proposal will also enhance the mpt(7D) driver to support SMP as a proof of the proposed interface. 2) Proposal Summary ------------------- To provide SMP target device access, SMP target devices attach via SAS HBA instances (such as mpt(7D)) and are operable via usmp(7I). To achieve this we extend the SCSA framework and request that SAS HBA drivers implement SMP management functions. Please refer to the figure below figure 1: Basic Achitecture Model Diagram. +--------------------------------------+ | SMP specific library or application | +--------------------------------------+ | V usmp(7I) / ioctl() | | USER =========================|============================ | KERNEL v +===================+ | SMP Target Driver | +===================+ ^ | sas_*() kernel functions v +----------------------+ | SCSA Frameworks with | | SAS extension | +----------------------+ ^ | SAS HBA driver transport | extension vector. | (sas_hba_tran_ext) v +---------------------+ | SAS HBA Driver with | | the extension for | | sas_hba_tran_ext. | | eg. mpt(7D) | +---------------------+ Figure 1: Basic Architecture Model Diagram 3) Proposal Details ------------------- 3.1) smp(7D) and usmp(7I) The smp(7D) target driver supports SMP targets such as expanders and SAS switches. The smp(7D) driver supports the ioctl(2) system call to provide an interface for userlevel applications which wish to manage an individual SMP target device. The usmp(7I) interface will be exported by smp(7D). #include ioctl(int fildes, int request, struct usmp_cmd *cmd); usmp_cmd structure is defined in and includes the following members: caddr_t usmp_req; /* The address of SMP request frame */ caddr_t usmp_rsp; /* The address of SMP response frame */ size_t usmp_reqsize; /* The length of request frame */ size_t usmp_rspsize; /* The length of response frame */ int usmp_timeout; /* Request timeout */ The members of the usmp_cmd structure have the following meanings: usmp_req and usmp_rsp are the pointers to the SMP request frame and SMP response frame sturctures. usmp_timeout is time in seconds to allow for completion of the SMP function. The default value is 60 seconds. The structure for SMP request frame and SMP response frame are defined in The SMP request frame structure names usmp_req and includes following members: uint8_t smpo_frametype; /* 40h is SMP request frame */ uint8_t smpo_function; /* SMP function */ uint8_t smpo_reserved; uint8_t smpo_reqsize; /* request frame length */ uint8_t smpo_msgframe[1]; The SMP response frame structure names usmp_rsp and includes following members: uint8_t smpi_frametype; /* 41h is SMP response frame */ uint8_t smpi_function; /* SMP function */ uint8_t smpi_result; uint8_t smpi_rspsize; /* response frame length */ uint8_t smpi_msgframe[1]; The usmp command is very powerful, but somewhat dangerous, and so its use is restricted to processes running as account with PRIV_SYS_DEVICES privilege, regardless of the file permissions on the device node. The details please refer to manpages in case directory. 3.2) /dev and /device naming smp(7D) creates device nodes in /devices with node names of the format "smp@wSASADDR" and minor name of "smp". The SASADDR is a 16 character (64bit hex) SAS World Wide Address. An expander might appear as: /devices/pci@0,0/pci10de,5d@d/pci1000,3150@0/smp@w5001636000005aff:smp According to the SAS specification, SASADDR uniquely identifies an SMP device. This project delivers a devfsadm(1M) link module which creates appropriate links in /dev that point to the device node under /devices. The link module creates an entry under /dev/smp/expd? for each SMP target device node which is detected by Solaris. To continue the above example, devfsadm will create the link /dev/smp/expd0 -> ../../devices/pci@0,0/pci10de,5d@d/pci1000,3150@0 \ /smp@w5001636000005aff:smp 3.3) SCSA extension for SMP transport The proposal enhances the SCSA framework to support the SMP transport service. The kernel structures and functions detailed below will be found in sas_addr - defines the address component of a SMP target device uint8_t a_wwn[8]; /* expander wwn */ struct scsi_hba_tran *a_hba_tran; /* transport vectors */ smp_pkt - defines the SMP function transport packet structure caddr_t pkt_req; /* the address of SMP request frame */ caddr_t pkt_rsp; /* the address of SMP response frame */ size_t pkt_reqsize; /* the length of request frame */ size_t pkt_rspsize; /* the length of response frame */ int pkt_timeout; /* the time allocated to complete SMP request */ uchar_t pkt_reason; /* reson completion called */ struct sas_addr *pkt_address; /* packet destination */ smp_device - kernel struct, which stores common information about each SMP target device. dev_info_t *dip; /* Cross-reference to dev_info_t */ struct sas_addr smp_addr; /* the sas address of the SMP target device */ 3.3.1) sas_smp_transport - request by SMP target driver to start a SMP request. int sas_smp_transport(struct smp_pkt *pktp); The SMP target driver uses sas_smp_transport() to request that the HBA driver transfer an SMP request packet to a specific SMP target device. sas_smp_transport() waits for the SMP request to complete and for the SMP response to return from the specific SMP target. 3.3.2) sas_smp_probe - default SAS HBA probe function for SMP target device only. int sas_smp_probe(struct smp_device *smp_devp); sas_smp_probe() is a kernel function for probing the existence of a specific SMP target device described in the smp_device structure. sas_smp_probe() sends the SMP function REPORT GENERAL to a specific sas address to detect the presence of a SMP target device. 3.3.3) sas_ifgetcap - get SMP transport capability. int sas_ifgetcap(struct sas_addr *ap, char *capstr); sas_ifgetcap() is a kernel function used by SMP target driver to get the capability of the SAS HBA driver. so far only one capability is defined: smp-crc CRC generating capability that is supported by HBA: 0 means HBA has no capability to generate CRC by itself. 1 means HBA handle CRC by itself. if HBA driver return 0, SMP target driver will generate CRC bytes for user's SMP request frame. 3.3.4) sas_hba_lookup_capstr - return index matching capability string int sas_hba_lookup_capstr(char *capstr); sas_hba_lookup_capstr() function attempts to match capstr against a known set of capability strings. If found, the defined index for the matched capability is returned. 3.4) scsi_hba_tran(7S) extension tran_extension is a void type pointer which is added to scsi_hba_tran(7S) as extension for specific HBA operations. In this proposal, we point tran_extension to sas_hba_tran_ext in order to support SMP. 3.5) struct sas_hba_tran_ext sas_hba_tran_ext structure is a kernel structure providing the semantics of tran_extension of scsi_hba_tran(7S). sas_hba_tran_ext defines vectors that an SAS HBA driver exports to SCSA interfaces for SMP transport so that SAS HBA specific function can be executed. In fact, sas_hba_tran_ext is a extension of scsi_hba_tran(9S). The associated struct sas_hba_tran_ext includes the following two members: int (*tran_sas_getcap)( struct sas_addr *ap, char *cap); int (*tran_smp_start)( struct smp_pkt *pkt); tran_sas_getcap Function entry that retrieves a SAS capability. The entry point is similar to tran_getcap() function of scsi_hba_tran(9S) tran_smp_start Function entry that starts a SMP request execution on the SAS HBA hardware. The entry point is similar to the tran_start() function of scsi_hba_tran(9S). 3.6) SAS HBA driver extension To support SMP functions, each SAS HBA driver must provide functions in addition to those required for traditional SCSI device I/O. Phase 1 of this project enhances the mpt(7D) driver to support SMP functions as detailed below: 3.6.1) implementation for sas_hba_tran_ext vector The tran_sas_getcap() and tran_smp_start() interfaces have to be implemented by SAS HBA driver. In this proposal, mpt(7D) will be enhanced to implement the tran_sas_getcap() and tran_smp_start() functions. 3.6.2) SMP targets automatic enumeration When a SAS HBA target driver supports SMP functions, the driver will enumerate all attached SMP targets automatically. Phase 1 of this project enhances mpt(7D) to automatically detect and enumerate SMP target devices in Solaris. 3.6.3) SMP targets dynamic reconfiguration The changes to mpt(7D) for Phase 1 (and other SAS HBAs for later phases) require the driver to be aware of status changes between the host and the attached SMP target device. When there is a change the HBA driver will online or offline the SMP target instances automatically without the needs of reboot or manually invoke devfsadm(1M). 3.7) We also define two new common functions for converting the string representation of a devid to a 64bit SAS address: int devid_str_to_wwn(const char *string, uint64_t *wwn) int ddi_devid_str_to_wwn(const char *string, uint64_t *wwn); 4) Interface Summary and Commitment Levels ------------------------------------------ |-----------------------------------------------------------------------------| | Interface Name | Commitment | Comments | +-------------------------+--------------+------------------------------------+ | | | | | smp(7D) | Cons. Priv. | Driver name for SMP target | | | | devices.(Typically, SAS expander | | | | devices) | | | | | | usmp(7I | Cons. Priv. | Driver exports this interface | | | | allowing user passthrough SMP | | | | function request to SMP target | | | | Devices from user-level. | | | | | | /usr/lib/devfsadm/ | | | | linkmod/SUNW_smp_link | Cons. Priv. | devfsadm link generation module | | | | | | /dev/smp/expd? | Cons. Priv. | Repository for link to SMP devices | | | | | | ddi_sas_smp | Cons. Priv. | minor nodetype for device minor | | | | nodes created by smp(7D) | | | | | | ddi_devid_str_to_wwn() | Cons. Priv. | DDI function for convert a string | | | | to 64 bits SAS address | | | | | | devid_str_to_wwn() | Cons. Priv. | function of libdevid library | | | | | | sas_*() | Proj. Priv | kernel functions for smp transport| | | | service in SCSA | | | | | | tran_extension | Cons. Priv. | scsi_hba_tran_t struct member for | | | | extending scsi_hba_tran(9S) | | | | | | sas_hba_tran_ext | Proj. Priv. | SAS HBA driver transport extension| | | | vector struct. sas_hba_tran_ext | | | | instantiate tran_extension of | | | | scsi_hba_tran_t. | | | | | +-----------------------------------------------------------------------------+ 5) References ------------- PSARC 2001/401 MPT driver (message passing technology) PSARC 2006/703 MPxIO extension for Serial Attached SCSI (SAS) on mpt(7D) PSARC/2004/504 Common Devid/GUID Encode Interfaces for SCSI Devices PSARC/1999/525 GENERIC SCSI PASS THROUGH FOR SOLARIS PSARC/1993/259 SCSI HBA interface LSI Fusion-MPT Message Passing Interface Specification v1.5.2 Serial Attached SCSI (SAS) (http://www.t10.org/ftp/t10/drafts/sas/sas-r05.pdf) Serial Attached SCSI - 1.1 (SAS-1.1) (http://www.t10.org/ftp/t10/drafts/sas1/sas1r10.pdf) Serial Attached SCSI - 2 (SAS-2) (http://www.t10.org/ftp/t10/drafts/sas2/sas2r11.pdf