From sac-owner Mon Feb 11 13:28:15 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m1BLSFpS004183 for ; Mon, 11 Feb 2008 13:28:15 -0800 (PST) Received: from sunmail2sca.sfbay.sun.com (localhost [127.0.0.1]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m1BLSENK017866 for ; Mon, 11 Feb 2008 13:28:14 -0800 (PST) Received: (from noaccess@localhost) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/Submit) id m1BLSEQ1017862 for one-pager-not-2b-used-directly; Mon, 11 Feb 2008 13:28:14 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m1BLS9hP017759 for <@sunmail2sca.sfbay.sun.com:one-pager@sun.com>; Mon, 11 Feb 2008 13:28:14 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JW300909FN1K600@brm-avmta-1.central.sun.com> for one-pager@sun.com (ORCPT one-pager@sun.com); Mon, 11 Feb 2008 14:28:13 -0700 (MST) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JW30006LFN0V4B0@brm-avmta-1.central.sun.com> for one-pager@sun.com (ORCPT one-pager@sun.com); Mon, 11 Feb 2008 14:28:12 -0700 (MST) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m1BLSCEB027305 for ; Mon, 11 Feb 2008 21:28:12 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JW300L01FGO2R00@mail-amer.sun.com> (original mail from Ramana.Srikanth@Sun.COM) for one-pager@sun.com (ORCPT one-pager@sun.com); Mon, 11 Feb 2008 14:28:12 -0700 (MST) Received: from [129.148.168.87] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JW3009KWFMTLG60@mail-amer.sun.com> for one-pager@sun.com (ORCPT one-pager@sun.com); Mon, 11 Feb 2008 14:28:06 -0700 (MST) Date: Mon, 11 Feb 2008 16:30:14 -0500 From: "Srikanth, Ramana" Subject: [2008/097]sosd: SCSI Object-based Storage Device driver Sender: Ramana.Srikanth@sun.com To: one-pager@sun.com Reply-to: Ramana.Srikanth@sun.com Message-id: <47B0BE66.9040605@Sun.COM> Organization: SUN Microsystems, Inc MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 User-Agent: Thunderbird 2.0.0.6 (X11/20071119) Status: RO Content-Length: 14073 1. Introduction 1.1. Project/Component Working Name: sosd: SCSI Object-based Storage Device driver 1.2. Name of Document Author/Supplier: Ramana Srikanth (Ramana.Srikanth@Sun.COM) 1.3. Date of This Document: 02/11/2008 1.4. Name of Major Document Customer(s)/Consumer(s): 1.4.1. The PAC or CPT you expect to review your project: Solaris PAC 1.4.2. The ARC(s) you expect to review your project: PSARC 1.4.3. The Director/VP who is "Sponsoring" this project: Scott.Tracy@Sun.COM/Bob.Porras@Sun.COM 1.4.4. The name of your business unit: Software 1.5. Email Aliases: 1.5.1. Responsible Manager: Daniel.Maslowski@Sun.COM 1.5.2. Responsible Engineer: Ramana.Srikanth@Sun.COM 1.5.3. Marketing Manager: Margaret.Hamburger@Sun.COM 1.5.4. Interest List: osd-discuss@opensolaris.org 2. Project Summary 2.1. Project Description: The project implements a new SCSI target driver (sosd) that binds to object-based SCSI devices conforming to the ANSI T10 OSD-2 standard. The new target driver will allow filesystems and applications to interact with the OSD devices using object semantics instead of block semantics. This driver will be a peer of the existing SCSI target drivers sd, st, ses and sgen. 2.2. Risks and Assumptions: Bi-directional transfer is not a requirement of this project, though one of the features of an OSD device is support for SCSI commands with bi-directional transfers (see technical description for more info). Solaris does not currently provide facilites in the I/O framework to support a bi-directional SCSI transfer. The net result however could be achieved if an OSD client sends multiple commmands in succession. 3. Business Summary Object-based storage allows the overhead of mapping and allocation of block storage to objects (such as files and directories) to be pushed down into the device. This can greatly increase the ability of filesystems to scale to large numbers of nodes. Lustre uses object-based storage to achieve its scalability. 3.1. Problem Area: Block allocation in distributed filesystem metadata servers prevents configurations with a large number of clients. These filesystems can address this bottleneck using object-base storage and Solaris does not currently have a driver for object-storage devices. 3.2. Market/Requester: From a technical perspective, a consumer of the sosd driver and API would be OSD enabled filesystems, object clients and OSD enabled XAM implementations. 3.3. Business Justification: Standards complient, object-based storage will enable Solaris to address the scalable parallel filesystem market and increase adoption. 3.4. Competitive Analysis: IBM has a parallel filesystem (GPFS) that uses OSD and supports 8192 stroage nodes. The IBM OSD initiator is Linux-based and open-source. Lustre supports 25000 clients and uses a proprietary object-based protocol to communicate with its object storage servers. Although Sun recently acquired the Lustre filesystem, in the near term Lustre client is only available on Linux. Panasas Inc produces PanFS, a closed source file system which supports T10 OSD. 3.5. Opportunity Window/Exposure: Sun needs a solution within the next year. 3.6. How will you know when you are done?: The sosd driver will be able to accept and process object commands submitted by filesystems and applications. iSCSI and MPxIO will be extended to support the large (256 byte) CDBs that OSD uses. Interoperability will be demonstrated with at least one non-Sun object storage device. 4. Technical Description: 4.1. Details: The project introduces a new SCSI target driver for object-based storage devices which is a peer to traditional block-based drivers like sd and st. In addition to the new target driver the project will provide a kernel module and a user library that export an OSD API to filesystems and user applications. The target driver will also provide uscsi pass-through command support similar to the other SCSI target drivers. There are currently two versions of the ANSI OSD standard. The published standard, "Object-Based Storage Device Commands (OSD)" is an official standard. "Object-Based Storage Device Comands-2 (OSD-2)" defines a second-generation OSD command set and is currently under development. Links to these documents are available in the references section. This project will implement the OSD-2 version of the standard. In order to support the OSD command set, Solaris needs to support 256 byte CDBs. Many Solaris components are currently limited to 16-byte CDBs. SCSA is capable of handling 256 byte CDBs but MPxIO and any HBA drivers that will support OSD must be modified to handle larger CDBs. This project will extend MPxIO and iscsi initiator to support 256 byte CDB's. The OSD command set also allows bi-directional SCSI transfers -- each OSD command can transfer read or write data while getting and/or setting attributes with the same operation. Solaris does not currently allow bi-directional SCSI transfers and a general-purpose solution to enable such transfers would impact a large set of Solaris components ranging from SCSA to the bus nexus drivers. Since the immediate consumers of the sosd driver do not require require bi-directional transfers this capability will be deferred to a future project. Additional details on the design are provided in the design document at: http://opensolaris.org/os/project/osd/ 4.2. Bug/RFE Number(s): 6657724 Deliver SCSI Object-based Storage Device (OSD) Target Driver for Solaris. 4.3. In Scope: - Support for standard OSD commands with large CDBs. (provided by sosd target driver and API) 4.4. Out of Scope: - OSD commands with parameters that would require bi-directional data transfers. 4.5. Interfaces: Imported Interfaces ----------------+--------------------+------------------------------- Interface | Classification | Comments ----------------+--------------------+------------------------------- Solaris DDI | | Solaris LDI | | scsi_init_pkt | | scsi_destroy_pkt| | scsi_dmafree | | physio | All are Committed | Interfaces defined in Sections biowait | interfaces | 9F, 9S and 3LIB of solaris biodone | | man pages bioerror | | buf | | bp_mapout | | getrbuf | | kmem_zalloc | | kmem_free | | libpthread | | libc | | libsysevent | | ----------------+--------------------+------------------------------- Exported Interfaces ---------------+--------------------+-------------------------------- Interface | Classification | Comments ---------------+--------------------+-------------------------------- OSD Kernel API | Volatile | Kernel module implementing Filesystem | | the interface between the interface | | filesystem or other kernel | | applications and the sosd driver ...................................................................... osd_setup_inquiry | | osd_setup_list | | osd_setup_format_lun | | osd_setup_create_partition | V | These interfaces are available osd_setup_remove_partition | O | for kernel clients through osd_setup_create_object | L | scsi_osd module and to user mode osd_setup_remove_object | A | clients through libosd. osd_setup_create_and_write | T | osd_setup_write | I | osd_setup_read | L | osd_setup_append | E | osd_setup_set_1page_attr | | osd_setup_set_get_1page_attr | | osd_add_set_page_attr_to_req | | osd_add_get_page_attr_to_req | | osd_setup_set_get_list_attr | | osd_submit_req (Kernel only) | | osd_submit_sync_req | | osd_get_result | | Volatile since they are new APIs osd_free_req | | with T10 spec pending ratification osd_close | | osd_get_max_dma_size | | ..................................................................... | | libosd | Volatile | Library implementing the | | interface between user-mode | | applications and the sosd driver | | osd_get_handle | Volatile | User level i/f Used to obtain _by_name | | a handle for use in the OSD | | API | | OSD ioctls | Project Private | Ioctls used in the implementation except for | | of the OSD API. get-handle | | ---------------+--------------------+-------------------------------- 4.6. Doc Impact: Documentation will need to be updated to describe the new OSD device types and their representation in devfs and /dev. 4.7. Admin/Config Impact: The administration of OSD devices should be identical to normal block devices with a few exceptions: 1. No equivalent to the "format" utility for object devices. Currently the project team does not see a need for a low level OSD device manipulation tool. 2. Only one Device path for OSD devices. (i.e. "/dev/osd/..." instead of "/dev/dsk" and "/dev/rdsk") 4.8. HA Impact: No new requirements 4.9. I18N/L10N Impact: 4.10. Packaging & Delivery: SUNWsosdr SUNWsosdu 4.11. Security Impact: On the initiator node, the sosd device minor nodes would be created with permissions "0640 root sys". The device would be enumerated under MPxIO by default and clients would require write privilege set to "sys_devices" to gain write access. This is enforced inside the driver using drv_priv() for all API entry points. Additionally, for library to accept commands from a userland client, the client's role should have "OSD Management" Rights profile. There are new authorizations assigned to this Rights profile, which are shown below. New authorizations: solaris.device.object.:::OSD Management::help=OSDMgmtHeader.html solaris.device.object.read:::OSD Read Auths::help=OSDMgmtRead.html solaris.device.object.write:::OSD Write Auths::help=OSDMgmtWrite.html New Rights Profile: Object Storage Device Management:::OSD Management:auths= solaris.device.object.*;help=OSDMgmt.html The library API entry points will check if the role has appropriate authorizations before sending the command down to the driver. The initial intended use of the API is over iSCSI transport. So, IP network security mechanism such as IPSEC and authentication mechanism like CHAP and RADIUS automatically provide transport level security. 4.12. Dependencies: 5. Reference Documents: - ANSI T10 OSD Standard http://www.t10.org/ftp/t10/drafts/osd/osd-r10.pdf - ANSI T10 OSD-2 Standard http://www.t10.org/ftp/t10/drafts/osd2/osd2r02.pdf - OpenSolaris web page of the project: (for Design) http://opensolaris.org/os/project/osd/ 6. Resources and Schedule: 6.1. Projected Availability: Q4 FY08 6.2. Cost of Effort: Development, OpenSolaris effort + - 1-2 persons for 6 months. primary docs. Functional test - 1 person for 1 month. Final docs. - 1 person for 2 weeks. 6.3. Cost of Capital Resources: sparc platforms - 2 (T2000) x86 platforms - 2 (x4200) 6.4. Product Approval Committee requested information: 6.4.1. Consolidation or Component Name: ON Consolidation 6.4.3. Type of CPT Review and Approval expected: Standard 6.4.4. Project Boundary Conditions: 6.4.5. Is this a necessary project for OEM agreements: No 6.4.6. Notes: None 6.4.7. Target RTI Date/Release: ONNV, Build 92 6.4.8. Target Code Design Review Date: 05/01/2008 6.4.9. Update approval addition: N/A 6.5. ARC review type: Standard Patch binding requested 6.6. ARC Exposure: Open 6.6.1. Rationale: 7. Prototype Availability: 7.1. Prototype Availability: 02/25/2008 7.2. Prototype Cost: 2 man months.