This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: T11 Storage Management HBA API(SM-HBA) 1.2. Name of Document Author/Supplier: Author: Hyon Kim 1.3 Date of This Document: 22 January, 2009 4. Technical Description - Problem Area: The currently released Serial Attached SCSI(SAS) HBAs are supported through HBA vendor specific utilities. As Sun is working on release of next generation SAS HBAs the administrator may have to deal with additional vendor specific utilities without common Solaris management support. This project is to deliver T11 Storage Management HBA API(SM-HBA) support and provide standard based SAS management utility for upcoming Sun's SAS-2 compliant (SAS Gen-2) HBAs. Note that a draft of SM-HBA-2 is published but it is not mature enough to guide product implementation according to T11. It is currently put on hold. The project team is requesting micro/patch binding. - Technical description 1. Deliverables: The SM-HBA support comprised of SM-HBA common library, a vendor specific library(VSL) for SAS Gen-2 HBAs and CLI which consumes SM-HBA interfaces and reports SAS HBAs and target configuration. libsmhba.so : SM-HBA common library Exports standard defined interfaces for use by utilities, SMI-S agents and any other 3rd party vendor application. libsun_sas.so : Vendor Specific Library(VSL) for SAS Gen-2 HBAs sasinfo.1M : SM-HBA based CLI for SAS HBA configuration discovery. +===============+ +---------------+ || sasinfo.1M || | other clients | +===============+ +---------------+ | | +==================================+ || libsmhba.so(SM-HBA standards) || +==================================+ | +==========================+ || libsun_sas.so(SAS VSL) || +==========================+ | | ______________|______|_____________________________________________ | | For SCSI passthru | +-----------------------------+ private ioctl | | libdevinfo(PSARC/2004/169) | | | libkstat(PSARC/1999/495) | | | libsysevent(PSARC/2001/076) | | +-----------------------------+ | | | | Generating devinfo nodes/properties, | | kstat counter and sysevent. | | +--------------------------------------+ | PSARC/2008/672 thebe SAS/SATA driver | | PSARC/2008/448 LSI MPT 2.0 support | | SAS Gen-2 HBA driver | +--------------------------------------+ 2. In Scope: The SM-HBA standard has been evolved from its predecessor FC-HBA to allow an application to manage both SAS and Fibre Channel HBAs through common interfaces. This project is to take advantage of those newly defined interfaces for SAS HBA management and will be providing VSL and CLI for Sun's SAS Gen-2 HBA support. Even though the SM-HBA standard includes the functionality of FC-HBA for FC HBA support, interface-wise it doesn't provide backward compatibility for FC-HBA clients. Thus the SM-HBA library will be handled as a major new release over the existing FC-HBA library. 3. Out of Scope: The FC HBA support through SM-HBA on existing FC HBAs not part of this project. They will be supported through existing libhbaapi.3LIB and fcinfo.1M. Since the functionality for FC support is same as FC-HBA there is no real benefit of migrating current FC-HBA support to SM-HBA. The current FC-HBA clients can continue to use FC-HBA library and if there is any client which needs to manage both FC and SAS through common interfaces in the future, the migration for the current FC-HBA support will be considered. 4. Interfaces: Interfaces Imported -------------+--------------------+------------------------- Interface | Classification | Comments -------------+--------------------+------------------------- | | Libraries listed | | section 4.8 | Committed | | | Events listed in | | section 4.8 | Consolidation | | private | Interfaces Exported -------------+--------------------+------------------------- Interface | Classification | Comments -------------+--------------------+------------------------- | | libsmhba interfaces | Committed | Implements SM-HBA standards | | Section 4.1 | | /etc/smhba.conf | Committed | To store library name and location | | at Vendor Library installation | | time. Section 4.2 | | sasinfo.1M command | Committed | Consumes libsmhba interfaces. | | section 4.3 | | sasinfo output | Not an interface | | | SAS related event | project private | Section 4.4 | | SEND_SCSI_CMD ioctl | project private | Section 4.5 | | kstat class name/ | project private | Section 4.6 phy stat structure | | | | SAS devinfo prop | project private | Section 4.7 | | 4.1 SM-HBA Interfaces The following list shows the interfaces that are defined in SM-HBA standards. (Section 8.1) The common library(libsmhba.so) will implement all these interfaces and the Support column indicates whether Sun VSL for SAS HBA(libsun_sas.so) would support the particular interface or not. In general, all mandatory interfaces and attributes are supported and a subset of optional ones are also supported based on the need of Sun SAS management. All supported interfaces and attributes below are committed. API Support Note --- ------- ---- - GetVersion Yes Mandatory - LoadLibrary Yes Mandatory - FreeLibrary Yes Mandatory - RegisterLibrary Yes Mandatory - GetNumberofAdapters Yes Mandatory - RefreshInformation Yes Mandatory - RefreshAdapterConfiguration Yes Mandatory - GetWrapperLibraryAttributes Yes Mandatory - GetVendorLibraryAttributes Yes Mandatory - GetAdapterName Yes Mandatory - OpenAdapter Yes Mandatory - CloseAdapter Yes Mandatory - GetAdapterAttributes Yes Mandatory - SMHBA_ADAPTERATTRIBUTES Partial (*) - Manufacturer Yes Mandatory - SerialNumber Yes Mandatory - Model Yes Mandatory - ModelDescription No - HardwareVersion Yes - DriverVersion Yes - OptionROMVersion Yes - FirmwareVersion Yes - VendorSpecificID No - HBASymbolicName No - DriverName Yes - RedundantOptionROMVersion No - RedundantFirmwareVersion No - GetNumberofPorts Yes Mandatory - GetPortType Yes Mandatory - GetAdapterPortAttributes Yes Mandatory - SMHBA_PORTATTRIBUTES Yes - SMHBA_SAS_PORT Yes - GetDiscoveredPortAttributes Yes Mandatory - SMHBA_PORTATTRIBUTES Yes - All the attributes in SMHBA_PORTATTRIBUTES and SMHBA_SAS_PORT are supported except NumberofDiscoveredPorts and NumberofPhys which are not applicable for a discovered port. - GetPortAttributesByWWN Yes - GetPhyAttributtes Yes Mandatory - SMHBA_SAS_PHY attributes Yes - SMHBA_FC_PHY No Only for FC support - GetProtocolStatistics No No driver/firmware support - GetphyStatistics Yes Mandatory - SMHBA_SASPHYSTATISTICS Partial(*) - SecondsSinceLastReset Yes - TxFrames No No driver/firmware support - TxWords No No driver/firmware support - RxFrames No No driver/firmware support - RxWords No No driver/firmware support - InvalidDwordCount Yes Mandatory - RunningDisparityErrorCount Yes Mandatory - LossofDwordSyncCount Yes Mandatory - PhyResetProblemCount Yes Mandatory - GetPortAttributesbyWWN Yes - GetBindingCapability No No administrative binding support - GetBindingSupport No No administrative binding support - SetBindingSupport No No administrative binding support - GetTargetMapping Yes - GetPersistentBinding No No administrative binding support - SetPersistentBinding No No administrative binding support - RemovePersistentBindings No No administrative binding support - RemoveAllPersistentBindings No No administrative binding support - GetLUNStatistics No No driver/firmware support - SendScsiInquiry Yes - SendReportLuns Yes - SendReadCapacity Yes - SendSMPPassThru Yes - SendCTPassThruV2 No Only for FC support - SetRNIDMgmtInfo No Only for FC support - GetRNIDMgmtInfo No Only for FC support - SendRNIDV2 No Only for FC support - SendRPL No Only for FC support - SendRPS No Only for FC support - SendSRL No Only for FC support - SendLIRR No Only for FC support - SendRLS No Only for FC support - SendTEST No Only for FC support - SendECHO No Only for FC support - RegisterForAdapterAddEvents No** - RegisterForAdapterEvents No** - RegisterForAdapterPortEvents No** - RegisterForAdapterPortStatEvents No No driver/FC support - RegisterForAdapterPhyStatEvents No No driver/FC support - RegisterForTargetEvents No** - RegisterForLinkEvents No** - RemoveCallback No** * Interface is implemented, but not all attributes are supported. ** Interfaces will be implemented when a client requests the support in the future. 4.2 Repository for SM-HBA Vendor Specific Library(VSL) The libsmhba.so loads the VSLs from the repository. - /etc/smhba.conf committed 4.3 CLI interfaces - sasinfo hba-port committed - sasinfo remote-port committed - sasinfo logical-unit committed The output of sasinfo is not an interface. 4.4 The SAS related event The following phy event is project private. Event class: EC_HBA Event subclass: ESC_sas_phy_event payload: port driver name, port SAS address, instance number, devfs path, phy id, domain Port Address, event type(one of events below) "phy_online" : Operational use of a phy device supported by the HBA API has been restored. "phy_offline" : Operational use of a phy device supported by the HBA API has become impossible. "phy_remove" : A phy device supported by the HBA API has been removed from the fabric. 4.5 SCSI passthru icotl ioctl: SEND_SCSI_CMD struct sas_scsi_cmd { uint64_t target_sas_addr; /* SAS address of target device */ uint64_t domain_port_sas_addr; /* SAS address for SAS domain. */ uint64_t scsi_lun; /* 8 byte SCSI LUN */ uint32_t scsi_sas_status; /* SCSI status */ uint32_t scsi_sas_rspcode; /* SAS response code */ uchar_t scsi_pkt_state; /* state of the request */ uchar_t scsi_pkt_action; /* corrective action */ uint32_t scsi_pkt_reason; /* reason for the state */ uint32_t scsi_flags; /* data flow direction */ uint32_t scsi_timeout; /* command timeout length */ unit64_t scsi_cdbaddr; /* SCSI CDB */ uint32_t scsi_cdblen; /* length of the CDB */ unint64_t scsi_bufaddr; /* data buffer for the request */ uint32_t scsi_buflen; /* buffer length */ int32_t scsi_bufresid; /* buffer residual */ uint32_t scsi_status; /* buffer status */ uint64_t scsi_rqbufaddr; /* sense data buffer */ uint32_t scsi_rqlen; /* sense data length */ int32_t scsi_rqresid; /* sense data residual */ } sas_scsi_cmd_t; 4.6 kstat related definition #define KSTAT_SAS_PHY_CLASS "SAS_phy_stat" /* class for kstat_t(kstat.3KSTAT)*/ /* PHY - kstat structure definition */ typedef struct sas_phy_stats { kstat_named_t seconds_since_last_reset; kstat_named_t tx_frames; kstat_named_t tx_words; kstat_named_t rx_frames; kstat_named_t rx_words; kstat_named_t invalid_dword_count; kstat_named_t running_disparity_error_count; kstat_named_t loss_of_dword_sync_count; kstat_named_t phy_reset_problem_count; } sas_phy_stats_t; 4.7 Devinfo node property SAS HBA devinfo node: initiator-port uint8 array /* SAS address of SAS HBA */ sm-hba-supported boolean /* indicate if a sm-hba is supported or not */ supported-protocol int /* indicates SSP, STP, SMP or SATA support */ NumberOfPhys int /* number of phys on HBA node */ AttachedSASAddress uint8 array /* SAS address of the other end of link */ Phy related: PhyIdentifier uint8 /* Phy id */ PhyNegotiatedLinkRate int /* negotiated phy speed */ PhyProgrammedMinLinkRate int /* programmed minimum phy speed */ PhyHardwareMinLinkRate int /* hardware minimum speed */ PhyProgrammedMaxLinkRate int /* programmed maximum phy speed */ PhyHardwareMaxLinkRate int /* hardware maximum phy speed */ PhyDomainPortWWN uint8 array /* SAS address for SAS domain */ SAS Target device devinfo node: SATA-target boolean /* indicate if SATA target or not *. AttachedSASAddress uint8 array /* SAS address of the other end of link */ target-port uint8 array /* SAS address of SAS target */ sam-lun uint8 array /* 8 bytes SCSI LUN per SAM-4 */ 4.8 Imported interface The project uses libc libpthreads libsysevent libdevinfo libkstat 5. Doc Impact: Man pages for the common library and CLI will be provided. 6. Admin/Config Impact: A new admin CLI will be provided. No changes on existing product. 7. Packaging & Delivery: Two new packages will be delivered. SUNWsmhba: contains userland libraries and CLI SUNWsmhbar: contains a config file(in /etc) to store VSL list. 8. Security Impact: The project will add the following line to /etc/security/exec_attr to allow a privileged user or process to run the command. File System Management:solaris:cmd:::/usr/sbin/sasinfo:privs=sys_devices - Reference Documents: T11 Storage Management HBA API (SM-HBA) http://www.t11.org/ftp/t11/pub/sm/hba/06-382v1.pdf) T10 Serial Attached SCSI - 2 (SAS-2) http://www.t10.org/cgi-bin/ac.pl?t=f&f=sas2r15.pdf FC HBA-API cases SHARC/2004/054 Enhanced support for T11 FC-HBA PSARC/2002/644 FibreChannel HBA API V2 PSARC/2001/481 Sun SNIA HBA API library