From cth@sac.sfbay.sun.com Sun Jan 18 13:30:50 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n0ILUnNL000808 for ; Sun, 18 Jan 2009 13:30:49 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n0ILUYC3007866; Mon, 19 Jan 2009 05:30:48 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KDO00L0JRR76E00@brm-avmta-1.central.sun.com>; Sun, 18 Jan 2009 14:30:43 -0700 (MST) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KDO0023ORR7EBD0@brm-avmta-1.central.sun.com>; Sun, 18 Jan 2009 14:30:43 -0700 (MST) Received: from sac.sfbay.sun.com (new-sac.SFBay.Sun.COM [129.146.175.65]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n0ILUhPA023107; Sun, 18 Jan 2009 13:30:43 -0800 (PST) Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n0ILUfFT000803; Sun, 18 Jan 2009 13:30:41 -0800 (PST) Received: (from cth@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id n0ILUfIA000799; Sun, 18 Jan 2009 13:30:41 -0800 (PST) Date: Sun, 18 Jan 2009 13:30:41 -0800 (PST) From: Christopher Horne Subject: scsi_hba_pkt_comp(9F) [PSARC/2009/033 FastTrack timeout 01/28/2009] To: PSARC-ext@sun.com Cc: scsav3-eng@sun.com Message-id: <200901182130.n0ILUfIA000799@sac.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 10620 I am sponsoring the following fasttrack for Srivijitha Dugganapalli set to 01/28/2009. Micro/patch binding is requested 1. Introduction 1.1. Project/Component Working Name: scsi_hba_pkt_comp(9F) 1.2. Name of Document Author/Supplier: Author: Srivijitha Dugganapalli 1.3. Date of This Document: 18 January, 2009 4.0 Technical Description 4.1 Background The scsi_pkt(9S) 'pkt_comp' field [1] is the completion callback routine established by a SCSA target driver (sd disk driver) before calling scsi_transport(9F) to execute a scsi_pkt(9S). Currently, the 'pkt_comp' completion callback is invoked via (*pkt->pkt_comp)(pkt); by the SCSA HBA driver - transferring control directly from the HBA driver back to the target driver. 4.2 Problem We currently have two issues related to completion of a scsi_pkt(9S): 1. With SCSI, changes in lun configuration are communicated in-band via special status information [2]. Solaris needs to observe completed commands, and detect the in-band lun level hotplug events to trigger dynamic lun reconfiguration. 2. Having an interrupt thread completely process the entire 'pkt_comp' code path serially can cause scalability problems. Implementing asynchronous 'pkt_comp' interrupt processing with fanout to threads running on other cpus has been shown to help performance significantly under heavy load. 4.3 Proposal Both of these issues can be effectively worked from common SCSA framework code by introducing a new public scsi_hba_pkt_comp(9F) function. To use scsi_hba_pkt_comp(9F), the changes to an HBA driver are minimal: instead of (*pkt->pkt_comp)(pkt); the HBA driver does scsi_hba_pkt_comp(pkt); In addition to the the problems mentioned above, the SCSA framework support for the tran_setup_pkt(9E) interfaces introduced by PSARC/2006/240 [3] can be simplified by requiring HBA drivers that use tran_setup_pkt(9E) to also use the new scsi_hba_pkt_comp(9F) function. This is '3' below. We currently have a small number of HBA drivers in ON (adpu320, fcp, mpt, blk2scsa, and pmcs(under development)) that must be modified to comply with this new requirement. While tran_setup_pkt(9E) is public, it has not been backported to S10. Since nevada has not been 'released', adding new requirements to tran_setup_pkt(9E) is permissible. The proposed scsi_hba_pkt_comp(9F) function will allow the SCSA framework to observe completion and perform completion synchronization and optimization. 1. Hotplug: The SCSA framework can observe lun hotplug events. Lun level hotplug events are reported in-band via scsi check conditions with specific ASC/ASCQ values that indicate a 'rescan' of the lun configuration is required. This information is required for SCSA to perform dynamic lun reconfiguration 2. Fanout: The SCSA framework can implement completion fanout, transparent to the SCSA HBA and target drivers. 3. PSARC/2006/240 implementation: Perform DMA sync prior to 'pkt_comp' inside implementation of new scsi_hba_pkt_comp(9F). This eliminates the 'pcw_orig_comp' interpose currently used in scsi_transport(). The initial putback will deliver the new scsi_hba_pkt_comp(9F) interface, and use the new interface for '3'. The initial putback enables subsequent delivery of hotplug observation and completion fanout inside SCSA. The 'hotplug' use case is related to SCSAv3 lun-level hotplug support - which is actively being worked on. The 'fanout' use case is unplanned, but HBA drivers are already starting to use non-DDI interfaces to solve this type of problem [5] in invasive ways. The performance overhead associated with scsi_hba_pkt_comp(9E) is expected to be minimal, and the future delivery of completion fanout is expected to result in improved performance. A call to the scsi_hba_pkt_comp() function will always result in a call to the 'pkt_comp' callback function defined in the scsi_pkt(9S). This 'pkt_comp' callback may, however, occur after return from scsi_hba_pkt_comp() and may occur from a different thread executing on a different CPU. With this case: o HBA drivers that implement tran_setup_pkt(9E) must use scsi_hba_pkt_comp(9F). Failure to comply will result in missed dma_sync operations. Man page changes covering this new requirement are provided below. o HBA driver that use SCSI_HBA_ADDR_COMPLEX [4] must use scsi_hba_pkt_comp(9F). Failure to comply will result in improper lun-level hotplug behavior. Currently SCSI_HBA_ADDR_COMPLEX is private, when it is promoted man page changes describing the scsi_hba_pkt_comp(9F) dependency will be provided. o Other HBA drivers can switch to using scsi_hba_pkt_comp() at any time. Switching early should enable immediate benefit when completion fanout is implemented. During internal discussion, the desire for a vector fanout interface, called something like scsi_hba_pkt_comp_vec(9F), was raised. This interface would take a vector of scsi_pkt(9S)s to complete. While this is a good idea, it is not this case. Such an interface should be investigated by those delivering a real 'fanout' implementation. Also, at that time, careful consideration should also be given to ensure that no semantic completion-order related hazards are being introduced by the fanout implementation. 4.4 Interfaces: ------------------------------------------------------------------ Interface Name Comm.Level Comments ------------------------------------------------------------------ scsi_hba_pkt_comp(9F) Committed use instead of (*pkt->pkt_comp)(pkt); Prototype: void scsi_hba_pkt_comp(struct scsi_pkt *pkt); 4.5 Man page changes: New scsi_hba_pkt_comp(9F) manpage required, see Appendix A. Modified scsi_pkt(9S) and tran_setup_pkt(9E), see Appendix B and C. 4.6 Release Binding Micro/patch binding is requested. 4.7 References [1] scsi_pkt(9S) http://docs.sun.com/app/docs/doc/816-5181/scsi-pkt-9s?a=view [2] http://www.t10.org/lists/asc-num.htm From the above table, following ASC/ASCQ value for lun-hotplug 3F/0E DTLPWROMAE REPORTED LUNS DATA HAS CHANGED [3] PSARC/2006/240 scsa dma enhancement http://sac.eng/PSARC/2006/240 http://www.opensolaris.org/os/community/arc/caselog/2006/240 [4] PSARC/2008/675 SCSI_HBA_ADDR_COMPLEX SCSA data structure linkage http://sac.eng/PSARC/2008/675 http://www.opensolaris.org/os/community/arc/caselog/2008/675 [5] mpt driver needs to deliver competitive IO throughput for SSD 6784459 mpt driver needs to deliver competitive IO throughput for SSD http://onnv.sfbay/log/nv/2009/01/14.brian.xu.560cd237cd72/webrev/ http://monaco.sfbay.sun.com/detail.jsf?cr=6784459 Appendix A: scsi_hba_pkt_comp(9F) man page Kernel Functions for Drivers scsi_hba_pkt_comp(9F) NAME scsi_hba_pkt_comp - scsi_pkt(9S) completion routine SYNOPSIS #include void scsi_hba_pkt_comp(struct scsi_pkt *pkt); INTERFACE LEVEL Solaris DDI specific (Solaris DDI). PARAMETERS pkt Pointer to a scsi_pkt(9S) structure. DESCRIPTION After filling in scsi_pkt(9S) fields with packet completion information, an HBA driver should call the scsi_hba_pkt_comp() function. The scsi_hba_pkt_comp() function is the recommended way for an HBA driver to signal completion of a scsi_pkt(9S). Use is mandatory for HBA drivers that use tran_setup_pkt(9E). Calling the scsi_hba_pkt_comp() function allows SCSA to observe, and possibly react to the completion of a scsi_pkt(9S) request. A call to the scsi_hba_pkt_comp() function will always result in a call to the 'pkt_comp' callback function defined in the scsi_pkt(9S). This 'pkt_comp' callback may, however, occur after return from scsi_hba_pkt_comp(), and may occur from a different thread executing on a different CPU. CONTEXT The scsi_hba_pkt_comp() function can be called from user, interrupt, or kernel context. ATTRIBUTES See attributes(5) for descriptions of the following attributes: ____________________________________________________________ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | |_____________________________|_____________________________| | Interface Stability | Committed | |_____________________________|_____________________________| SEE ALSO scsi_pkt(9S), tran_setup_pkt(9E) NOTE: HBA driver calls scsi_hba_pkt_comp() instead of calling scsi_pkt(9S) 'pkt_comp' directly Appendix B: scsi_pkt(9S) manpage diffs diff -U 2 scsi_pkt.txt scsi_pkt.new.txt --- scsi_pkt.txt Thu Dec 11 14:13:13 2008 +++ scsi_pkt.new.txt Sat Jan 17 21:43:30 2009 @@ -358,5 +358,13 @@ SEE ALSO tran_init_pkt(9E), scsi_arq_status(9S), scsi_init_pkt(9F), - scsi_transport(9F), scsi_status(9S) + scsi_transport(9F), scsi_status(9S), scsi_hba_pkt_comp(9F) +NOTE + HBA drivers should signal scsi_pkt(9S) completion by + calling scsi_hba_pkt_comp(9F). This is mandatory for + HBA drivers that implement tran_setup_pkt(9E), failure + to comply results in undefined behavior. Appendix C: tran_setup_pkt(9E) manpage diffs diff -U 2 tran_setup_pkt.txt tran_setup_pkt.new.txt --- tran_setup_pkt.txt Sat Jan 17 20:53:46 2009 +++ tran_setup_pkt.new.txt Sat Jan 17 21:32:58 2009 @@ -141,4 +141,10 @@ tor. + HBA drivers that implement tran_setup_pkt() must signal + scsi_pkt(9S) completion by calling scsi_hba_pkt_comp(9F) - direct + use of the scsi_pkt(9S) 'pkt_comp' field is not permitted, and + result in undefined behavior. RETURN VALUES tran_setup_pkt() must return zero on success, and -1 on @@ -151,5 +157,6 @@ scsi_hba_attach(9F), scsi_hba_pkt_alloc(9F), scsi_hba_pkt_free(9F), scsi_init_pkt(9F), buf(9S), - scsi_address(9S), scsi_hba_tran(9S), scsi_pkt(9S) + scsi_address(9S), scsi_hba_tran(9S), scsi_pkt(9S), + scsi_hba_pkt_comp(9F) 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From carlsonj@phorcys.east.sun.com Tue Jan 20 05:06:46 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n0KD6kJO025343 for ; Tue, 20 Jan 2009 05:06:46 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n0KD6elJ004966; Tue, 20 Jan 2009 13:06:44 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KDR00K0NTR6WA00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 20 Jan 2009 05:06:42 -0800 (PST) Received: from phorcys.east.sun.com ([129.148.174.143]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KDR00CXPTR5O990@nwk-avmta-1.sfbay.Sun.COM>; Tue, 20 Jan 2009 05:06:41 -0800 (PST) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n0KD6e7t020103; Tue, 20 Jan 2009 08:06:40 -0500 (EST) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n0KD6eMs020100; Tue, 20 Jan 2009 08:06:40 -0500 (EST) Date: Tue, 20 Jan 2009 08:06:40 -0500 From: James Carlson Subject: Re: scsi_hba_pkt_comp(9F) [PSARC/2009/033 FastTrack timeout 01/28/2009] In-reply-to: <200901182130.n0ILUfIA000799@sac.sfbay.sun.com> To: Christopher Horne Cc: PSARC-ext@sun.com, scsav3-eng@sun.com Message-id: <18805.52320.255017.666264@gargle.gargle.HOWL> MIME-version: 1.0 X-Mailer: VM 7.01 under Emacs 21.3.1 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200901182130.n0ILUfIA000799@sac.sfbay.sun.com> Status: RO Content-Length: 913 Christopher Horne writes: > requirement. While tran_setup_pkt(9E) is public, it has not been > backported to S10. Since nevada has not been 'released', adding new > requirements to tran_setup_pkt(9E) is permissible. We have some people claiming loudly that the OpenSolaris releases are indeed real releases, requiring version number bumps, and others insisting just as much that they're not real, and thus incompatible changes to "unreleased" bits are still acceptable. I'm not going to hold *this* case up for it, but it'd be really swell to have something -- anything -- like a coherent plan for the OpenSolaris release mechanism. Otherwise, we're just fooling ourselves. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From Chris.Horne@sun.com Wed Jan 21 10:17:46 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n0LIHkG1018165 for ; Wed, 21 Jan 2009 10:17:46 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n0LIHhO6002869; Wed, 21 Jan 2009 11:17:46 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KDU00E1T2TLEI00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 21 Jan 2009 10:17:45 -0800 (PST) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KDU00ABQ2TJ36C0@nwk-avmta-1.sfbay.Sun.COM>; Wed, 21 Jan 2009 10:17:43 -0800 (PST) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n0LIHhD7029566; Wed, 21 Jan 2009 18:17:43 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KDU003012PVJE00@mail-amer.sun.com> (original mail from Chris.Horne@Sun.COM); Wed, 21 Jan 2009 11:17:43 -0700 (MST) Received: from [172.20.25.50] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KDU001YL2T9W4E0@mail-amer.sun.com>; Wed, 21 Jan 2009 11:17:33 -0700 (MST) Date: Wed, 21 Jan 2009 11:17:16 -0700 From: Chris Horne Subject: Re: scsi_hba_pkt_comp(9F) [PSARC/2009/033 FastTrack timeout 01/28/2009] In-reply-to: <200901182130.n0ILUfIA000799@sac.sfbay.sun.com> Sender: Chris.Horne@sun.com To: Christopher Horne Cc: PSARC-ext@sun.com, scsav3-eng@sun.com Message-id: <497766AC.8000101@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200901182130.n0ILUfIA000799@sac.sfbay.sun.com> User-Agent: Thunderbird 1.5.0.8 (X11/20061204) Status: RO Content-Length: 48 This case was approved at PSARC today. -Chris