From Wed Aug 26 09:53:25 2009 Date: Wed, 26 Aug 2009 09:53:25 -0700 (PDT) From: Postmaster Subject: Message from mail server Content-Length: 95 Mime-Version: 1.0 Status: RO X-IMAP: 1251305605 4 Delete. This is a system message. --END+PSEUDO-- From sacadmin Tue Aug 25 20:12:10 2009 Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n7Q3CAfN022743; Tue, 25 Aug 2009 20:12:10 -0700 (PDT) Received: (from xc149992@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id n7Q3CAjB022726; Tue, 25 Aug 2009 20:12:10 -0700 (PDT) Date: Tue, 25 Aug 2009 20:12:10 -0700 (PDT) From: Xiang-Dong Frank Che Message-Id: <200908260312.n7Q3CAjB022726@sac.sfbay.sun.com> To: PSARC-record@sac.sfbay.sun.com Subject: Transport Layer Retries (TLR) Support [PSARC/2009/461 FastTrack timeout 09/02/2009] Content-Length: 572 Status: RO X-Status: $$$$ X-UID: 0000000001 Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: Transport Layer Retries (TLR) Support 1.2. Name of Document Author/Supplier: Author: Jianfei Wang 1.3 Date of This Document: 25 August, 2009 4. Technical Description See the case directory for more detail 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From Frank.Che@sun.com Tue Aug 25 23:50:06 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n7Q6o6iK004225 for ; Tue, 25 Aug 2009 23:50:06 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n7Q6nweQ015733; Tue, 25 Aug 2009 23:50:05 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KOZ00I011MMRM00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 25 Aug 2009 23:49:34 -0700 (PDT) Received: from sineb-mail-2.sun.com ([192.18.19.7]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KOZ00DSW1MLPW70@nwk-avmta-1.sfbay.Sun.COM>; Tue, 25 Aug 2009 23:49:34 -0700 (PDT) Received: from fe-apac-05.sun.com (fe-apac-05.sun.com [192.18.19.176] (may be forged)) by sineb-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n7Q6nXSx029181; Wed, 26 Aug 2009 06:49:33 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KOZ00A0019J8I00@mail-apac.sun.com>; Wed, 26 Aug 2009 14:49:33 +0800 (SGT) Received: from [129.158.218.60] ([unknown] [129.158.218.60]) by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KOZ00CE71MIIFJ0@mail-apac.sun.com>; Wed, 26 Aug 2009 14:49:32 +0800 (SGT) Date: Wed, 26 Aug 2009 14:45:29 +0800 From: Frank Che Subject: Transport Layer Retries (TLR) Support (PSARC/2009/461 FastTrack Timeout 09/02/20009) Sender: Frank.Che@sun.com To: PSARC-ext@sun.com Cc: Jianfei Wang , Javen Wu , yongfeng du , scsitgt-bj@sun.com Message-id: <4A94DA09.3080306@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 User-Agent: Thunderbird 2.0.0.19 (X11/20090218) Content-Length: 4231 Status: RO X-Status: $$$$ X-UID: 0000000002 I'm sponsoring this case for Jianfei Wang. The timer is set for 09/02/2009. -Frank. 1. Introduction 1.1. Project/Component Working Name: Transport Layer Retries (TLR) Support 1.2. Name of Document Author/Supplier: Author: Jianfei Wang 1.3. Date of This Document: 08/24/2009 4. Technical Description: 4.1. Summary This project is to provide TLR (Transport Layer Retries) support in Solaris so that tape drive devices could be better supported. Requested release binding is Patch/Micro. 4.2. Details To support Transport Layer Retries(TLR), a new SCSI transport capability, SCSI_CAP_TRAN_LAYER_RETRIES, will be added. HBA drivers need TLR support should be updated to know this capability. The storage target driver should also be updated to coordinate the TLR support. When a new tape drive device is detected, the target driver checks the HBA's capability of TLR support with the scsi_ifgetcap(9F) interface in the attach(9E) entry point. The target driver reads Vital Product Data (VPD) page 0x90 to get the target device type. If VPD page 0x90 exists, the target device type is SAS2; otherwise the target device type is SAS1. If the HBA supports TLR and the target device is a SAS1 device, then the target driver uses SCSI command MODE_SENSE and MODE_SELECT to set the mode page (Protocol-Specific Logical Unit mode page)'s TLR flag. If set successfully, Initiator_Target_Lun nexus support TLR property. Because the mode page is not persistent, the TLR flag bit will be lost when the target device do a hard reset. So the target driver needs to set the mode page's TLR flag each time the target device completes hard reset. If the HBA supports TLR and the target device is SAS2 device, then the target driver issues InquiryVPD to get the capability of the tape device to check whether the target device is able to support TLR. If both the HBA driver and the tape device support TLR, the target driver will send a commands packet with a special pkg_flag, FLAG_TLR, to notify the HBA driver to set TLR in control bit before transportation. If the Initiator_Target_Lun nexus does not support TLR, commands from the target driver to the HBA driver would fail and the pkt_reason will be set to CMD_TLR_OFF. When the target driver get this reason, it will turn off the TLR control. 4.3. Exported Interfaces Interface Commitment Comments ============= ========== ===================== SCSI_CAP_TRAN_LAYER_RETRIES Committed SCSI TLR capability tran-layer-retries Committed SCSI_CAP_ASCII content FLAG_TLR Committed pkg_flag indicating TLR is supported CMD_TLR_OFF Committed pkt_reason indicating TLR is not supported 4.4. Bug/RFE Number(s) 6647764 - Solaris storage driver Transport Layer Retries (TLR) support 4.5. References Serial Attached SCSI - 1.1 (SAS-1.1) Serial Attached SCSI - 2 (SAS-2) SCSI Primary Commands - 4 (SPC-4) http://www.t10.org/ 4.6. Manpage changes scsi_hba_lookup_capstr.diff ============================ 156a157 > SCSI_CAP_TRAN_LAYER_RETRIES 157a159,161 > "tran-layer-retries" scsi_ifgetcap.diff =========================== 126a127,128 > tran-layer-retries Transport Layer retries is support by > the host adapter: 0 disables, 1 enables. scsi_pkt.diff =========================== 226a227,230 > FLAG_TLR Run command with Transport > Layer Retries support > 323a328,330 > CMD_TLR_OFF Transport Layer Retries turn off > 6. Resources and Schedule: 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: Fast Track 6.6. ARC Exposure: Open From gdamore@sun.com Wed Aug 26 07:13:21 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n7QEDK5i000466 for ; Wed, 26 Aug 2009 07:13:21 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n7QEDBha007231; Wed, 26 Aug 2009 15:13:20 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KOZ00603M65PC00@nwk-avmta-2.sfbay.sun.com>; Wed, 26 Aug 2009 07:13:17 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KOZ000BSM65XU50@nwk-avmta-2.sfbay.sun.com>; Wed, 26 Aug 2009 07:13:17 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n7QEDH5F016926; Wed, 26 Aug 2009 07:13:17 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KOZ00600M0JYA00@fe-sfbay-09.sun.com>; Wed, 26 Aug 2009 07:13:17 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KOZ007YFM64BE00@fe-sfbay-09.sun.com>; Wed, 26 Aug 2009 07:13:17 -0700 (PDT) Date: Wed, 26 Aug 2009 07:13:16 -0700 From: "Garrett D'Amore" Subject: Re: Transport Layer Retries (TLR) Support (PSARC/2009/461 FastTrack Timeout 09/02/20009) In-reply-to: <4A94DA09.3080306@sun.com> Sender: Garrett.Damore@sun.com To: Frank Che Cc: PSARC-ext@sun.com, Jianfei Wang , Javen Wu , yongfeng du , scsitgt-bj@sun.com Message-id: <4A9542FC.3060703@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <4A94DA09.3080306@sun.com> User-Agent: Thunderbird 2.0.0.18 (X11/20081201) Content-Length: 5295 Status: RO X-Status: $$$$ X-UID: 0000000003 This case looks pretty good, but I've a few questions: 1) Can you describe why tape devices need this? Many reviewers won't be familiar with the motivation behind TLR. A description of how TLR affects the behavior of the devices (beyond just the negotiation phase, which you have adequately described) would be useful. 2) As I'm not one of the people familiar with TLR, I'd like to better understand what this does to failure modes. Specifically, what does TLR do when a SCSI error is encountered? Can it lead to an "interminable" hang-like condition where the transport driver continues retrying a failed SCSI command indefinitely? 3) Does the above have any ramifications for FMA? Sorry if I'm asking for information that seems obvious, but I don't want to have to grunge through the specifications for SAS to find the answers.... -- Garrett Frank Che wrote: > I'm sponsoring this case for Jianfei Wang. The timer is set for > 09/02/2009. > > -Frank. > > 1. Introduction > 1.1. Project/Component Working Name: > Transport Layer Retries (TLR) Support > > 1.2. Name of Document Author/Supplier: > Author: Jianfei Wang > > 1.3. Date of This Document: > 08/24/2009 > > 4. Technical Description: > > 4.1. Summary > > This project is to provide TLR (Transport Layer Retries) support > in Solaris so that tape drive devices could be better supported. > > Requested release binding is Patch/Micro. > > 4.2. Details > > To support Transport Layer Retries(TLR), a new SCSI transport > capability, SCSI_CAP_TRAN_LAYER_RETRIES, will be added. > HBA drivers need TLR support should be updated to know this > capability. > The storage target driver should also be updated to coordinate > the TLR support. > > When a new tape drive device is detected, the target driver checks > the HBA's capability of TLR support with the scsi_ifgetcap(9F) > interface > in the attach(9E) entry point. > > The target driver reads Vital Product Data (VPD) page 0x90 to > get the target device type. If VPD page 0x90 exists, the target > device > type is SAS2; otherwise the target device type is SAS1. > > If the HBA supports TLR and the target device is a SAS1 device, > then the > target driver uses SCSI command MODE_SENSE and MODE_SELECT to set > the > mode page (Protocol-Specific Logical Unit mode page)'s TLR flag. > If set > successfully, Initiator_Target_Lun nexus support TLR property. > Because > the mode page is not persistent, the TLR flag bit will be lost > when the > target device do a hard reset. So the target driver needs to set > the mode > page's TLR flag each time the target device completes hard reset. > > If the HBA supports TLR and the target device is SAS2 device, > then the > target driver issues InquiryVPD to get the capability of the tape > device > to check whether the target device is able to support TLR. If > both the > HBA driver and the tape device support TLR, the target driver > will send a > commands packet with a special pkg_flag, FLAG_TLR, to notify > the HBA driver to set TLR in control bit before transportation. > If the > Initiator_Target_Lun nexus does not support TLR, commands from > the target > driver to the HBA driver would fail and the pkt_reason will be > set to > CMD_TLR_OFF. When the target driver get this reason, it will turn > off > the TLR control. > > 4.3. Exported Interfaces > > Interface Commitment Comments > ============= ========== ===================== > SCSI_CAP_TRAN_LAYER_RETRIES Committed SCSI TLR capability > tran-layer-retries Committed SCSI_CAP_ASCII content > FLAG_TLR Committed pkg_flag indicating > TLR is supported > CMD_TLR_OFF Committed pkt_reason > indicating TLR is not supported > > 4.4. Bug/RFE Number(s) > 6647764 - Solaris storage driver Transport Layer Retries (TLR) > support > > 4.5. References > Serial Attached SCSI - 1.1 (SAS-1.1) > Serial Attached SCSI - 2 (SAS-2) > SCSI Primary Commands - 4 (SPC-4) > http://www.t10.org/ > > 4.6. Manpage changes > scsi_hba_lookup_capstr.diff > ============================ > 156a157 > > SCSI_CAP_TRAN_LAYER_RETRIES > 157a159,161 > > "tran-layer-retries" > > scsi_ifgetcap.diff > =========================== > 126a127,128 > > tran-layer-retries Transport Layer retries is support by > > the host adapter: 0 disables, 1 > enables. > > scsi_pkt.diff > =========================== > 226a227,230 > > FLAG_TLR Run command with Transport > > Layer Retries support > > > 323a328,330 > > CMD_TLR_OFF Transport Layer Retries turn off > > > > 6. Resources and Schedule: > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > ON > 6.5. ARC review type: Fast Track > 6.6. ARC Exposure: Open > > > From psarc-member-list-request@sun.com Wed Aug 26 09:47:53 2009 Return-Path: Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02 [129.146.11.31]) by ivrel.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n7QGlrBq007255 for ; Wed, 26 Aug 2009 09:47:53 -0700 (PDT) Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n7QGlr1o060406; Wed, 26 Aug 2009 09:47:53 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n7QGlkVN014296; Wed, 26 Aug 2009 09:47:46 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KOZ00709TBM1G00@brm-avmta-1.central.sun.com>; Wed, 26 Aug 2009 10:47:46 -0600 (MDT) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KOZ001UNTBL4Y70@brm-avmta-1.central.sun.com>; Wed, 26 Aug 2009 10:47:45 -0600 (MDT) Received: from sac.sfbay.sun.com (sac.SFBay.Sun.COM [129.146.226.132]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n7QGliHs060259; Wed, 26 Aug 2009 09:47:44 -0700 (PDT) Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n7QGkSP2015010 for ; Wed, 26 Aug 2009 09:46:28 -0700 (PDT) Received: from brmea-mail-1.sun.com (brmea-mail-1.Sun.COM [192.18.98.31]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n7QGkSr1030638 for ; Wed, 26 Aug 2009 09:46:28 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n7QGkS8I019478 for ; Wed, 26 Aug 2009 16:46:28 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KOZ00A00R0SHW00@mail-amer.sun.com> for psarc-ext@sac.sfbay.sun.com; Wed, 26 Aug 2009 10:46:28 -0600 (MDT) Received: from [129.147.241.111] ([unknown] [129.147.241.111]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KOZ00JDDT91JFG0@mail-amer.sun.com>; Wed, 26 Aug 2009 10:46:19 -0600 (MDT) Date: Wed, 26 Aug 2009 10:45:39 -0600 From: Randall Ralphs Subject: Re: Please review the one page for TLR(Transport Layer Retries) In-reply-to: <4A826DB8.2010000@Sun.COM> Sender: randall.ralphs@sun.com To: Jianfei Wang , psarc-ext@sac.sfbay.sun.com Cc: ssdg@sun.com, Nikko He , yongfeng du , Javen Wu , Li Ming Message-id: <4A9566B3.4020307@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <4A826DB8.2010000@Sun.COM> User-Agent: Thunderbird 2.0.0.22 (X11/20090720) Content-Length: 3518 Status: RO X-Status: $$$$ X-UID: 0000000004 1) How will this both work with command recovery that is in Nevada and doesn't exsist in previous versions of Solaris where it is not? 2) This looks like it expects the tape driver to be the only source of resets. With MPxIO enabled scsi_vhci can do resets when failing over paths that would not have originated from the target driver. Some HBA's also initiate resets on their own. 3) If MPxIO is enabled, the scsi_ifgetcap(9F) would not get the capability from the HBA and therefor would not do the mode sense/mode select correctly. See bug 6755363 as this would also effect SCSI_CAP_TRAN_LAYER_RETRIES. 4) How will this work with ComStar where the transport may not be as obvious to the target driver. 5) Does TLR give up? If a connection fails in a MPxIO invironment waiting for a connection to reconnect would prevent failing over. Also in a non-MPxIO environment not waiting could cause a failure that could otherwise could be recovered when reconnected. Randy Jianfei Wang wrote: > Hi folks, > Please review the attached file, it's one page for Transport Layer > Retries. Thanks. > > Best Regards > Jianfei Wang > > > Template Version: @(#)onepager.txt 1.35 07/11/07 SMI > Copyright 2007 Sun Microsystems > > xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > > 1. Introduction > 1.1. Project/Component Working Name: > Solaris storage driver Transport Layer Retries (TLR) support > > 1.2. Name of Document Author/Supplier: > Author: Jianfei Wang > > 1.3. Date of This Document: > 08/05/2009 > > 1.5. Email Aliases: > 1.5.1. Responsible Manager:Tzongyu.Lee@Sun.COM > 1.5.2. Responsible Engineer:Jianfei.Wang@Sun.COM > 1.5.3. Marketing Manager: > 1.5.4. Interest List:sas@sun.com > > scsi_ifsetcap > > 4. Technical Description: > Background: > Require support for TLR (Transport Layer Retries) for Sun OEM Tape drives > The Windows and Linux drivers provided by LSI do support TLR. The HP LTO SAS drive > default is TLR disabled. If the HBA driver sees the tape drive supports TLR, the > driver will enable TLR on the HBA and tape driver. > > > 4.1. Details: > For support Transport Layer Retries(TLR), It would add one scsi transport capability. > It could get TLR capability by scsi_ifgetcap(9F).The target driver should check the HBA's > capability by scsi_ifgetcap(9F) in attach(9E) entry point of target driver, if the initiator > supports TLR, then issue InquiryVPD to get the capability of the tape target to check > whether the target device is able to support TLR. If the target and initiator both > support TLR, The commands packet send from target driver to HBA should set a special > pkg_flag to notify HBA driver set TLR in control bit before transportation. > > 4.2. Bug/RFE Number(s): > 6647764 > 4.5. Interfaces: > > INTERFACE STABILITY COMMENTS > SCSI_CAP_TRAN_LAYER_RETRIES evolving SCSI TLR capability > tran-layer-retries evolving SCSI_CAP_ASCII content > FLAG_TLR evolving pkg_flag for TLR > CMD_TLR_OFF evolving pkt_reason for TLR, if the I_T_L nexus does not support the TLR, > command would fail and set this pkt_reason to pkt, target driver > get this reason and turn off the TLR control. > > 6. Resources and Schedule: > > > 6.4. Product Approval Committee requested information: > 6.4.1. Consolidation or Component Name: > 6.4.3. Type of CPT Review and Approval expected: > FastTrack > > > From Frank.Che@sun.com Wed Aug 26 22:31:04 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n7R5V47Q000178 for ; Wed, 26 Aug 2009 22:31:04 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n7R5V342001257; Wed, 26 Aug 2009 22:31:04 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KP000J0FSNSFM00@brm-avmta-1.central.sun.com>; Wed, 26 Aug 2009 23:31:04 -0600 (MDT) Received: from sineb-mail-2.sun.com ([192.18.19.7]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KP00092ASNQG770@brm-avmta-1.central.sun.com>; Wed, 26 Aug 2009 23:31:03 -0600 (MDT) Received: from fe-apac-05.sun.com (fe-apac-05.sun.com [192.18.19.176] (may be forged)) by sineb-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n7R5V1LI011216; Thu, 27 Aug 2009 05:31:01 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KP000900SJKXH00@mail-apac.sun.com>; Thu, 27 Aug 2009 13:31:01 +0800 (SGT) Received: from [129.158.218.60] ([unknown] [129.158.218.60]) by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KP000JLASNOU350@mail-apac.sun.com>; Thu, 27 Aug 2009 13:31:01 +0800 (SGT) Date: Thu, 27 Aug 2009 13:26:56 +0800 From: Frank Che Subject: Re: Transport Layer Retries (TLR) Support (PSARC/2009/461 FastTrack Timeout 09/02/20009) In-reply-to: <4A94DA09.3080306@sun.com> Sender: Frank.Che@sun.com To: PSARC-ext@sun.com, Jianfei Wang , Randall Ralphs Cc: Javen Wu , yongfeng du , scsitgt-bj@sun.com Message-id: <4A961920.7090002@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <4A94DA09.3080306@sun.com> User-Agent: Thunderbird 2.0.0.19 (X11/20090218) Status: RO Content-Length: 5456 Some questions from Randy on this project: 1) How will this both work with command recovery that is in Nevada and doesn't exsist in previous versions of Solaris where it is not? 2) This looks like it expects the tape driver to be the only source of resets. With MPxIO enabled scsi_vhci can do resets when failing over paths that would not have originated from the target driver. Some HBA's also initiate resets on their own. 3) If MPxIO is enabled, the scsi_ifgetcap(9F) would not get the capability from the HBA and therefor would not do the mode sense/mode select correctly. See bug 6755363 as this would also effect SCSI_CAP_TRAN_LAYER_RETRIES. 4) How will this work with ComStar where the transport may not be as obvious to the target driver. 5) Does TLR give up? If a connection fails in a MPxIO invironment waiting for a connection to reconnect would prevent failing over. Also in a non-MPxIO environment not waiting could cause a failure that could otherwise could be recovered when reconnected. Frank. Frank Che wrote: > I'm sponsoring this case for Jianfei Wang. The timer is set for > 09/02/2009. > > -Frank. > > 1. Introduction > 1.1. Project/Component Working Name: > Transport Layer Retries (TLR) Support > > 1.2. Name of Document Author/Supplier: > Author: Jianfei Wang > > 1.3. Date of This Document: > 08/24/2009 > > 4. Technical Description: > > 4.1. Summary > > This project is to provide TLR (Transport Layer Retries) support > in Solaris so that tape drive devices could be better supported. > > Requested release binding is Patch/Micro. > > 4.2. Details > > To support Transport Layer Retries(TLR), a new SCSI transport > capability, SCSI_CAP_TRAN_LAYER_RETRIES, will be added. > HBA drivers need TLR support should be updated to know this > capability. > The storage target driver should also be updated to coordinate > the TLR support. > > When a new tape drive device is detected, the target driver checks > the HBA's capability of TLR support with the scsi_ifgetcap(9F) > interface > in the attach(9E) entry point. > > The target driver reads Vital Product Data (VPD) page 0x90 to > get the target device type. If VPD page 0x90 exists, the target > device > type is SAS2; otherwise the target device type is SAS1. > > If the HBA supports TLR and the target device is a SAS1 device, > then the > target driver uses SCSI command MODE_SENSE and MODE_SELECT to set > the > mode page (Protocol-Specific Logical Unit mode page)'s TLR flag. > If set > successfully, Initiator_Target_Lun nexus support TLR property. > Because > the mode page is not persistent, the TLR flag bit will be lost > when the > target device do a hard reset. So the target driver needs to set > the mode > page's TLR flag each time the target device completes hard reset. > > If the HBA supports TLR and the target device is SAS2 device, > then the > target driver issues InquiryVPD to get the capability of the tape > device > to check whether the target device is able to support TLR. If > both the > HBA driver and the tape device support TLR, the target driver > will send a > commands packet with a special pkg_flag, FLAG_TLR, to notify > the HBA driver to set TLR in control bit before transportation. > If the > Initiator_Target_Lun nexus does not support TLR, commands from > the target > driver to the HBA driver would fail and the pkt_reason will be > set to > CMD_TLR_OFF. When the target driver get this reason, it will turn > off > the TLR control. > > 4.3. Exported Interfaces > > Interface Commitment Comments > ============= ========== ===================== > SCSI_CAP_TRAN_LAYER_RETRIES Committed SCSI TLR capability > tran-layer-retries Committed SCSI_CAP_ASCII content > FLAG_TLR Committed pkg_flag indicating > TLR is supported > CMD_TLR_OFF Committed pkt_reason > indicating TLR is not supported > > 4.4. Bug/RFE Number(s) > 6647764 - Solaris storage driver Transport Layer Retries (TLR) > support > > 4.5. References > Serial Attached SCSI - 1.1 (SAS-1.1) > Serial Attached SCSI - 2 (SAS-2) > SCSI Primary Commands - 4 (SPC-4) > http://www.t10.org/ > > 4.6. Manpage changes > scsi_hba_lookup_capstr.diff > ============================ > 156a157 > > SCSI_CAP_TRAN_LAYER_RETRIES > 157a159,161 > > "tran-layer-retries" > > scsi_ifgetcap.diff > =========================== > 126a127,128 > > tran-layer-retries Transport Layer retries is support by > > the host adapter: 0 disables, 1 > enables. > > scsi_pkt.diff > =========================== > 226a227,230 > > FLAG_TLR Run command with Transport > > Layer Retries support > > > 323a328,330 > > CMD_TLR_OFF Transport Layer Retries turn off > > > > 6. Resources and Schedule: > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > ON > 6.5. ARC review type: Fast Track > 6.6. ARC Exposure: Open > > > From Jianfei.Wang@sun.com Thu Aug 27 03:30:10 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n7RAUAi8013626 for ; Thu, 27 Aug 2009 03:30:10 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n7RAU8WM018976; Thu, 27 Aug 2009 03:30:09 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KP100K216I9W300@nwk-avmta-2.sfbay.sun.com>; Thu, 27 Aug 2009 03:30:09 -0700 (PDT) Received: from sineb-mail-1.sun.com ([192.18.19.6]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KP100MVA6I6H2E0@nwk-avmta-2.sfbay.sun.com>; Thu, 27 Aug 2009 03:30:07 -0700 (PDT) Received: from fe-apac-05.sun.com (fe-apac-05.sun.com [192.18.19.176] (may be forged)) by sineb-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n7RAU6sd021330; Thu, 27 Aug 2009 10:30:06 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KP1002006AKFL00@mail-apac.sun.com>; Thu, 27 Aug 2009 18:30:06 +0800 (SGT) Received: from [129.158.219.20] ([unknown] [129.158.219.20]) by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KP100JMD6I5U3I0@mail-apac.sun.com>; Thu, 27 Aug 2009 18:30:06 +0800 (SGT) Date: Thu, 27 Aug 2009 18:29:22 +0800 From: Jianfei Wang Subject: Re: Transport Layer Retries (TLR) Support (PSARC/2009/461 FastTrack Timeout 09/02/20009) In-reply-to: <4A9542FC.3060703@sun.com> Sender: Jianfei.Wang@sun.com To: "Garrett D'Amore" Cc: Frank Che , PSARC-ext@sun.com, Javen Wu , yongfeng du , scsitgt-bj@sun.com Message-id: <4A966002.8060601@Sun.COM> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <4A94DA09.3080306@sun.com> <4A9542FC.3060703@sun.com> User-Agent: Thunderbird 2.0.0.18 (X11/20081215) Status: RO Content-Length: 6366 Hi Garrett, Please review the answer, thanks. Best Regards Jianfei Wang Garrett D'Amore wrote: > This case looks pretty good, but I've a few questions: > > 1) Can you describe why tape devices need this? Many reviewers won't > be familiar with the motivation behind TLR. A description of how TLR > affects the behavior of the devices (beyond just the negotiation > phase, which you have adequately described) would be useful. Some backup applications will fail the whole backup if any of their frame transmission commands are terminated. TLR introduces a way to retransmit frames. So with TLR support, the backup process is expected to be more smooth, with less failure. > > 2) As I'm not one of the people familiar with TLR, I'd like to better > understand what this does to failure modes. Specifically, what does > TLR do when a SCSI error is encountered? Can it lead to an > "interminable" hang-like condition where the transport driver > continues retrying a failed SCSI command indefinitely? How the retransmission happen are specified in the TLR documents (SAS1R10 and SAS2R16). This transmission is handled in the firmware of the HBA and tape device, according to the SAS specification, not in the HBA driver or the target driver. That's why this document focuses on the negotiation process. Once the negotiation is done, everything else is performed through the firmware. > > 3) Does the above have any ramifications for FMA? Because the firmware would deal with TLR work, it would handle the error, the error is transparent for system, it doesn't report any message to FMA module. > > Sorry if I'm asking for information that seems obvious, but I don't > want to have to grunge through the specifications for SAS to find the > answers.... > > -- Garrett > > Frank Che wrote: >> I'm sponsoring this case for Jianfei Wang. The timer is set for >> 09/02/2009. >> >> -Frank. >> >> 1. Introduction >> 1.1. Project/Component Working Name: >> Transport Layer Retries (TLR) Support >> >> 1.2. Name of Document Author/Supplier: >> Author: Jianfei Wang >> >> 1.3. Date of This Document: >> 08/24/2009 >> >> 4. Technical Description: >> >> 4.1. Summary >> >> This project is to provide TLR (Transport Layer Retries) support >> in Solaris so that tape drive devices could be better supported. >> >> Requested release binding is Patch/Micro. >> >> 4.2. Details >> >> To support Transport Layer Retries(TLR), a new SCSI transport >> capability, SCSI_CAP_TRAN_LAYER_RETRIES, will be added. >> HBA drivers need TLR support should be updated to know this >> capability. >> The storage target driver should also be updated to coordinate >> the TLR support. >> >> When a new tape drive device is detected, the target driver checks >> the HBA's capability of TLR support with the scsi_ifgetcap(9F) >> interface >> in the attach(9E) entry point. >> >> The target driver reads Vital Product Data (VPD) page 0x90 to >> get the target device type. If VPD page 0x90 exists, the target >> device >> type is SAS2; otherwise the target device type is SAS1. >> >> If the HBA supports TLR and the target device is a SAS1 device, >> then the >> target driver uses SCSI command MODE_SENSE and MODE_SELECT to >> set the >> mode page (Protocol-Specific Logical Unit mode page)'s TLR flag. >> If set >> successfully, Initiator_Target_Lun nexus support TLR property. >> Because >> the mode page is not persistent, the TLR flag bit will be lost >> when the >> target device do a hard reset. So the target driver needs to set >> the mode >> page's TLR flag each time the target device completes hard reset. >> >> If the HBA supports TLR and the target device is SAS2 device, >> then the >> target driver issues InquiryVPD to get the capability of the >> tape device >> to check whether the target device is able to support TLR. If >> both the >> HBA driver and the tape device support TLR, the target driver >> will send a >> commands packet with a special pkg_flag, FLAG_TLR, to notify >> the HBA driver to set TLR in control bit before transportation. >> If the >> Initiator_Target_Lun nexus does not support TLR, commands from >> the target >> driver to the HBA driver would fail and the pkt_reason will be >> set to >> CMD_TLR_OFF. When the target driver get this reason, it will >> turn off >> the TLR control. >> >> 4.3. Exported Interfaces >> >> Interface Commitment Comments >> ============= ========== ===================== >> SCSI_CAP_TRAN_LAYER_RETRIES Committed SCSI TLR capability >> tran-layer-retries Committed SCSI_CAP_ASCII >> content >> FLAG_TLR Committed pkg_flag >> indicating TLR is supported >> CMD_TLR_OFF Committed pkt_reason >> indicating TLR is not supported >> >> 4.4. Bug/RFE Number(s) >> 6647764 - Solaris storage driver Transport Layer Retries (TLR) >> support >> >> 4.5. References >> Serial Attached SCSI - 1.1 (SAS-1.1) >> Serial Attached SCSI - 2 (SAS-2) >> SCSI Primary Commands - 4 (SPC-4) >> http://www.t10.org/ >> >> 4.6. Manpage changes >> scsi_hba_lookup_capstr.diff >> ============================ >> 156a157 >> > SCSI_CAP_TRAN_LAYER_RETRIES >> 157a159,161 >> > "tran-layer-retries" >> >> scsi_ifgetcap.diff >> =========================== >> 126a127,128 >> > tran-layer-retries Transport Layer retries is >> support by >> > the host adapter: 0 disables, 1 >> enables. >> >> scsi_pkt.diff >> =========================== >> 226a227,230 >> > FLAG_TLR Run command with Transport >> > Layer Retries support >> > >> 323a328,330 >> > CMD_TLR_OFF Transport Layer Retries turn off >> > >> >> 6. Resources and Schedule: >> 6.4. Steering Committee requested information >> 6.4.1. Consolidation C-team Name: >> ON >> 6.5. ARC review type: Fast Track >> 6.6. ARC Exposure: Open >> >> >> > From gdamore@sun.com Thu Aug 27 07:21:44 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n7RELiGq024892 for ; Thu, 27 Aug 2009 07:21:44 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n7RELia7027837; Thu, 27 Aug 2009 07:21:44 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KP100H01H88B500@nwk-avmta-1.sfbay.Sun.COM>; Thu, 27 Aug 2009 07:21:44 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KP1008H6H87FWB0@nwk-avmta-1.sfbay.Sun.COM>; Thu, 27 Aug 2009 07:21:43 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n7RELhQ4015946; Thu, 27 Aug 2009 07:21:43 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KP100300H3ARW00@fe-sfbay-09.sun.com>; Thu, 27 Aug 2009 07:21:43 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KP1003SSH7M5C90@fe-sfbay-09.sun.com>; Thu, 27 Aug 2009 07:21:25 -0700 (PDT) Date: Thu, 27 Aug 2009 07:21:21 -0700 From: "Garrett D'Amore" Subject: Re: Transport Layer Retries (TLR) Support (PSARC/2009/461 FastTrack Timeout 09/02/20009) In-reply-to: <4A966002.8060601@Sun.COM> Sender: Garrett.Damore@sun.com To: Jianfei Wang Cc: Frank Che , PSARC-ext@sun.com, Javen Wu , yongfeng du , scsitgt-bj@sun.com Message-id: <4A969661.4020006@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <4A94DA09.3080306@sun.com> <4A9542FC.3060703@sun.com> <4A966002.8060601@Sun.COM> User-Agent: Thunderbird 2.0.0.18 (X11/20081201) Status: RO Content-Length: 369 Jianfei Wang wrote: > >> >> 3) Does the above have any ramifications for FMA? > Because the firmware would deal with TLR work, it would handle the > error, the error is transparent for system, it doesn't report any > message to FMA module. Will it eventually give up, or will it keep retrying the command forever? How many retries will it perform? - Garrett From Milan.Jurik@sun.com Thu Aug 27 09:07:35 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n7RG7ZOT004740 for ; Thu, 27 Aug 2009 09:07:35 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n7RG7YJR041414; Thu, 27 Aug 2009 10:07:35 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KP100J01M4LGK00@nwk-avmta-2.sfbay.sun.com>; Thu, 27 Aug 2009 09:07:33 -0700 (PDT) Received: from gmp-eb-inf-2.sun.com ([192.18.6.24]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KP100DDIM4JK370@nwk-avmta-2.sfbay.sun.com>; Thu, 27 Aug 2009 09:07:32 -0700 (PDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-1-fe1.eu.sun.com [192.18.6.7] (may be forged)) by gmp-eb-inf-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n7RG7V5S019526; Thu, 27 Aug 2009 16:07:31 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KP100D00LYXMN00@fe-emea-10.sun.com>; Thu, 27 Aug 2009 17:07:09 +0100 (BST) Received: from [129.157.19.89] ([unknown] [129.157.19.89]) by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KP1002OOM3VCK50@fe-emea-10.sun.com>; Thu, 27 Aug 2009 17:07:08 +0100 (BST) Date: Thu, 27 Aug 2009 18:07:31 +0200 From: Milan Jurik Subject: Re: Transport Layer Retries (TLR) Support (PSARC/2009/461 FastTrack Timeout 09/02/20009) In-reply-to: <4A966002.8060601@Sun.COM> Sender: Milan.Jurik@sun.com To: Jianfei Wang Cc: "Garrett D'Amore" , Frank Che , PSARC-ext@sun.com, Javen Wu , yongfeng du , scsitgt-bj@sun.com Message-id: <1251389251.1501.112.camel@xylabtecra> Organization: Sun Microsystems Inc. MIME-version: 1.0 X-Mailer: Evolution 2.26.3 Content-type: text/plain; charset=UTF-8 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 References: <4A94DA09.3080306@sun.com> <4A9542FC.3060703@sun.com> <4A966002.8060601@Sun.COM> Status: RO Content-Length: 909 Hi, Jianfei Wang píše v čt 27. 08. 2009 v 18:29 +0800: [...] > > > > 2) As I'm not one of the people familiar with TLR, I'd like to better > > understand what this does to failure modes. Specifically, what does > > TLR do when a SCSI error is encountered? Can it lead to an > > "interminable" hang-like condition where the transport driver > > continues retrying a failed SCSI command indefinitely? > How the retransmission happen are specified in the TLR documents > (SAS1R10 and SAS2R16). This transmission is handled in the firmware of > the HBA and tape device, according to the SAS specification, not in the > HBA driver or the target driver. That's why this document focuses on the > negotiation process. Once the negotiation is done, everything else is > performed through the firmware. Any documented way to disable negotiation of this feature by administrator? Best regards, Milan From Jianfei.Wang@sun.com Thu Aug 27 17:49:53 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n7S0nqSG005630 for ; Thu, 27 Aug 2009 17:49:52 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n7S0nlVn018346; Fri, 28 Aug 2009 08:49:51 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KP20070BAB0YC00@nwk-avmta-2.sfbay.sun.com>; Thu, 27 Aug 2009 17:49:48 -0700 (PDT) Received: from sineb-mail-2.sun.com ([192.18.19.7]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KP2005TMAAZSN30@nwk-avmta-2.sfbay.sun.com>; Thu, 27 Aug 2009 17:49:48 -0700 (PDT) Received: from fe-apac-05.sun.com (fe-apac-05.sun.com [192.18.19.176] (may be forged)) by sineb-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n7S0nkwg015195; Fri, 28 Aug 2009 00:49:46 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KP200E009VXMH00@mail-apac.sun.com>; Fri, 28 Aug 2009 08:49:46 +0800 (SGT) Received: from [129.158.219.20] ([unknown] [129.158.219.20]) by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KP200GANAAX4P60@mail-apac.sun.com>; Fri, 28 Aug 2009 08:49:46 +0800 (SGT) Date: Fri, 28 Aug 2009 08:49:02 +0800 From: Jianfei Wang Subject: Re: Transport Layer Retries (TLR) Support (PSARC/2009/461 FastTrack Timeout 09/02/20009) In-reply-to: <4A969661.4020006@sun.com> Sender: Jianfei.Wang@sun.com To: "Garrett D'Amore" Cc: Frank Che , PSARC-ext@sun.com, Javen Wu , yongfeng du , scsitgt-bj@sun.com Message-id: <4A97297E.8020606@Sun.COM> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <4A94DA09.3080306@sun.com> <4A9542FC.3060703@sun.com> <4A966002.8060601@Sun.COM> <4A969661.4020006@sun.com> User-Agent: Thunderbird 2.0.0.18 (X11/20081215) Status: RO Content-Length: 771 Hi, The SAS1and SAS2 Spec don't explicitly tell how many retries it should perform, but it's not a reasonable implementation to retry forever. Because that is a mechanism between Firmware and Tape, I've asked LSI to see if they can provide any numbers. For the worst case, if it retried too many times, the driver has the timeout mechanism to handle that. Garrett D'Amore wrote: > Jianfei Wang wrote: >> >>> >>> 3) Does the above have any ramifications for FMA? >> Because the firmware would deal with TLR work, it would handle the >> error, the error is transparent for system, it doesn't report any >> message to FMA module. > > Will it eventually give up, or will it keep retrying the command > forever? How many retries will it perform? > > - Garrett > From Jianfei.Wang@sun.com Thu Aug 27 18:58:36 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n7S1waRh006886 for ; Thu, 27 Aug 2009 18:58:36 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n7S1wXuI037903; Thu, 27 Aug 2009 19:58:36 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KP200D0VDHM2T00@nwk-avmta-2.sfbay.sun.com>; Thu, 27 Aug 2009 18:58:34 -0700 (PDT) Received: from sineb-mail-2.sun.com ([192.18.19.7]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KP2005DQDHLSS90@nwk-avmta-2.sfbay.sun.com>; Thu, 27 Aug 2009 18:58:34 -0700 (PDT) Received: from fe-apac-06.sun.com (fe-apac-06.sun.com [192.18.19.177] (may be forged)) by sineb-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n7S1wXOj018771; Fri, 28 Aug 2009 01:58:33 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KP200800DA3J900@mail-apac.sun.com>; Fri, 28 Aug 2009 09:58:33 +0800 (SGT) Received: from [129.158.219.20] ([unknown] [129.158.219.20]) by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KP200FMCDHK0S40@mail-apac.sun.com>; Fri, 28 Aug 2009 09:58:33 +0800 (SGT) Date: Fri, 28 Aug 2009 09:57:49 +0800 From: Jianfei Wang Subject: Re: Transport Layer Retries (TLR) Support (PSARC/2009/461 FastTrack Timeout 09/02/20009) In-reply-to: <4A961920.7090002@sun.com> Sender: Jianfei.Wang@sun.com To: Randall Ralphs Cc: Frank Che , PSARC-ext@sun.com, Javen Wu , yongfeng du , scsitgt-bj@sun.com Message-id: <4A97399D.4050507@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=windows-1252 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 References: <4A94DA09.3080306@sun.com> <4A961920.7090002@sun.com> User-Agent: Thunderbird 2.0.0.18 (X11/20081215) Status: RO Content-Length: 5939 Hi Randy, This project focuses on negotiating and setting the TLR control bit through the driver interaction. The TLR operation is done at the firmware level, not the driver level. So question 1, 2, 4 and 5 are not in the scope of this project. For question 3, SAS1 doesn't support TLR when MPXIO is enabled. This is specified in document 07-027r2 (SAS-2 Enabling and disabling Transport Layer Retries) on T10. The SAS1 TLR mode sense/mode select has been proved to be impractical when MPXIO is enabled, and the reason is ' If the mode page is shared, then it's possible that multiple initiators dont agree on the setting interfere with each other. This is less of a problem if the mode page is per I_T nexus, but such implementations are rare.' The protocol author found mode sense/mode select is a bad support way, so SAS1 TLR support is abandoned when MPXIO is enabled. SAS2 inquiries the VPD page to support TLR, it would support MPXIO. Best Regards Jianfei Wang Frank Che wrote: > Some questions from Randy on this project: > > 1) How will this both work with command recovery that is in Nevada and > doesn't exsist in previous versions of Solaris where it is not? > > 2) This looks like it expects the tape driver to be the only source of > resets. With MPxIO enabled scsi_vhci can do resets when failing over > paths that would not have originated from the target driver. Some > HBA's also initiate resets on their own. > > 3) If MPxIO is enabled, the scsi_ifgetcap(9F) would not get the > capability from the HBA and therefor would not do the mode sense/mode > select correctly. See bug 6755363 as this would also effect > SCSI_CAP_TRAN_LAYER_RETRIES. > > 4) How will this work with ComStar where the transport may not be as > obvious to the target driver. > > 5) Does TLR give up? If a connection fails in a MPxIO invironment > waiting for a connection to reconnect would prevent failing over. Also > in a non-MPxIO environment not waiting could cause a failure that > could otherwise could be recovered when reconnected. > > Frank. > > Frank Che wrote: >> I'm sponsoring this case for Jianfei Wang. The timer is set for >> 09/02/2009. >> >> -Frank. >> >> 1. Introduction >> 1.1. Project/Component Working Name: >> Transport Layer Retries (TLR) Support >> >> 1.2. Name of Document Author/Supplier: >> Author: Jianfei Wang >> >> 1.3. Date of This Document: >> 08/24/2009 >> >> 4. Technical Description: >> >> 4.1. Summary >> >> This project is to provide TLR (Transport Layer Retries) support >> in Solaris so that tape drive devices could be better supported. >> >> Requested release binding is Patch/Micro. >> >> 4.2. Details >> >> To support Transport Layer Retries(TLR), a new SCSI transport >> capability, SCSI_CAP_TRAN_LAYER_RETRIES, will be added. >> HBA drivers need TLR support should be updated to know this capability. >> The storage target driver should also be updated to coordinate >> the TLR support. >> >> When a new tape drive device is detected, the target driver checks >> the HBA's capability of TLR support with the scsi_ifgetcap(9F) interface >> in the attach(9E) entry point. >> >> The target driver reads Vital Product Data (VPD) page 0x90 to >> get the target device type. If VPD page 0x90 exists, the target device >> type is SAS2; otherwise the target device type is SAS1. >> >> If the HBA supports TLR and the target device is a SAS1 device, then the >> target driver uses SCSI command MODE_SENSE and MODE_SELECT to set the >> mode page (Protocol-Specific Logical Unit mode page)'s TLR flag. If set >> successfully, Initiator_Target_Lun nexus support TLR property. Because >> the mode page is not persistent, the TLR flag bit will be lost when the >> target device do a hard reset. So the target driver needs to set the >> mode >> page's TLR flag each time the target device completes hard reset. >> >> If the HBA supports TLR and the target device is SAS2 device, then the >> target driver issues InquiryVPD to get the capability of the tape device >> to check whether the target device is able to support TLR. If both the >> HBA driver and the tape device support TLR, the target driver will >> send a >> commands packet with a special pkg_flag, FLAG_TLR, to notify >> the HBA driver to set TLR in control bit before transportation. If the >> Initiator_Target_Lun nexus does not support TLR, commands from the >> target >> driver to the HBA driver would fail and the pkt_reason will be set to >> CMD_TLR_OFF. When the target driver get this reason, it will turn off >> the TLR control. >> >> 4.3. Exported Interfaces >> >> Interface Commitment Comments >> ============= ========== ===================== >> SCSI_CAP_TRAN_LAYER_RETRIES Committed SCSI TLR capability >> tran-layer-retries Committed SCSI_CAP_ASCII content >> FLAG_TLR Committed pkg_flag indicating TLR is supported >> CMD_TLR_OFF Committed pkt_reason indicating TLR is not supported >> >> 4.4. Bug/RFE Number(s) >> 6647764 - Solaris storage driver Transport Layer Retries (TLR) support >> >> 4.5. References >> Serial Attached SCSI - 1.1 (SAS-1.1) >> Serial Attached SCSI - 2 (SAS-2) >> SCSI Primary Commands - 4 (SPC-4) >> http://www.t10.org/ >> >> 4.6. Manpage changes >> scsi_hba_lookup_capstr.diff >> ============================ >> 156a157 >> > SCSI_CAP_TRAN_LAYER_RETRIES >> 157a159,161 >> > "tran-layer-retries" >> >> scsi_ifgetcap.diff >> =========================== >> 126a127,128 >> > tran-layer-retries Transport Layer retries is support by >> > the host adapter: 0 disables, 1 enables. >> >> scsi_pkt.diff >> =========================== >> 226a227,230 >> > FLAG_TLR Run command with Transport >> > Layer Retries support >> > >> 323a328,330 >> > CMD_TLR_OFF Transport Layer Retries turn off >> > >> >> 6. Resources and Schedule: >> 6.4. Steering Committee requested information >> 6.4.1. Consolidation C-team Name: >> ON >> 6.5. ARC review type: Fast Track >> 6.6. ARC Exposure: Open >> >> >> From Jianfei.Wang@sun.com Fri Aug 28 00:11:55 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n7S7Bs8c025697 for ; Fri, 28 Aug 2009 00:11:55 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n7S7BcTI015959; Fri, 28 Aug 2009 08:11:54 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KP200D03RZS8Y00@nwk-avmta-2.sfbay.sun.com>; Fri, 28 Aug 2009 00:11:52 -0700 (PDT) Received: from sineb-mail-1.sun.com ([192.18.19.6]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KP200HT2RZQIXE0@nwk-avmta-2.sfbay.sun.com>; Fri, 28 Aug 2009 00:11:51 -0700 (PDT) Received: from fe-apac-06.sun.com (fe-apac-06.sun.com [192.18.19.177] (may be forged)) by sineb-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n7S7Boll003230; Fri, 28 Aug 2009 07:11:50 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KP200J00RUXUS00@mail-apac.sun.com>; Fri, 28 Aug 2009 15:11:50 +0800 (SGT) Received: from [129.158.219.20] ([unknown] [129.158.219.20]) by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KP200FC2RZO0SH0@mail-apac.sun.com>; Fri, 28 Aug 2009 15:11:49 +0800 (SGT) Date: Fri, 28 Aug 2009 15:11:05 +0800 From: Jianfei Wang Subject: Re: Transport Layer Retries (TLR) Support (PSARC/2009/461 FastTrack Timeout 09/02/20009) In-reply-to: <1251389251.1501.112.camel@xylabtecra> Sender: Jianfei.Wang@sun.com To: Milan Jurik Cc: "Garrett D'Amore" , Frank Che , PSARC-ext@sun.com, Javen Wu , yongfeng du , scsitgt-bj@sun.com Message-id: <4A978309.8080504@Sun.COM> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_4kcVO+TGk3oixZNFgVc2BA)" X-PMX-Version: 5.4.1.325704 References: <4A94DA09.3080306@sun.com> <4A9542FC.3060703@sun.com> <4A966002.8060601@Sun.COM> <1251389251.1501.112.camel@xylabtecra> User-Agent: Thunderbird 2.0.0.18 (X11/20081215) Status: RO Content-Length: 3435 This is a multi-part message in MIME format. --Boundary_(ID_4kcVO+TGk3oixZNFgVc2BA) Content-type: text/plain; format=flowed; charset=UTF-8 Content-transfer-encoding: 8BIT Hi Milan, No, TLR only take effects when there are transport layer errors(lost frame), the protocol will retransmit the lost frame automatically to recover. It has no side effects when there is no error occurs. So that's a value-add feature and there is no reason to disable it if both HBA and Tape support it. Best Regards Jianfei Wang Milan Jurik wrote: > Hi, > > Jianfei Wang píše v čt 27. 08. 2009 v 18:29 +0800: > [...] > >>> 2) As I'm not one of the people familiar with TLR, I'd like to better >>> understand what this does to failure modes. Specifically, what does >>> TLR do when a SCSI error is encountered? Can it lead to an >>> "interminable" hang-like condition where the transport driver >>> continues retrying a failed SCSI command indefinitely? >>> >> How the retransmission happen are specified in the TLR documents >> (SAS1R10 and SAS2R16). This transmission is handled in the firmware of >> the HBA and tape device, according to the SAS specification, not in the >> HBA driver or the target driver. That's why this document focuses on the >> negotiation process. Once the negotiation is done, everything else is >> performed through the firmware. >> > > Any documented way to disable negotiation of this feature by > administrator? > > Best regards, > > Milan > > --Boundary_(ID_4kcVO+TGk3oixZNFgVc2BA) Content-type: text/html; charset=UTF-8 Content-transfer-encoding: 8BIT Hi Milan,
No,  TLR only take effects when there are transport layer errors(lost frame), the protocol will retransmit the lost frame automatically to recover.
It has no side effects when there is no error occurs.

So that's a value-add feature and there is no reason to disable it if both HBA and Tape support it.

Best Regards
Jianfei Wang


Milan Jurik wrote: 
Hi,

Jianfei Wang píše v čt 27. 08. 2009 v 18:29 +0800:
[...]
  
2) As I'm not one of the people familiar with TLR, I'd like to better 
understand what this does to failure modes.  Specifically, what does 
TLR do when a SCSI error is encountered?  Can it lead to an 
"interminable" hang-like condition where the transport driver 
continues retrying a failed SCSI command indefinitely?
      
How the retransmission happen are specified in the TLR documents 
(SAS1R10 and SAS2R16). This transmission is handled in the firmware of 
the HBA and tape device, according to the SAS specification, not in the 
HBA driver or the target driver. That's why this document focuses on the 
negotiation process. Once the negotiation is done, everything else is 
performed through the firmware.
    

Any documented way to disable negotiation of this feature by
administrator?

Best regards,

Milan

--Boundary_(ID_4kcVO+TGk3oixZNFgVc2BA)-- From Randall.Ralphs@sun.com Fri Aug 28 10:37:23 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n7SHbN5Y025192 for ; Fri, 28 Aug 2009 10:37:23 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n7SHbL6P017041; Fri, 28 Aug 2009 10:37:22 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KP300403KYAFK00@nwk-avmta-1.sfbay.Sun.COM>; Fri, 28 Aug 2009 10:37:22 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KP30053FKY97JB0@nwk-avmta-1.sfbay.Sun.COM>; Fri, 28 Aug 2009 10:37:22 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n7SHbL9A007094; Fri, 28 Aug 2009 17:37:21 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KP300E00KQ1P800@mail-amer.sun.com>; Fri, 28 Aug 2009 11:37:21 -0600 (MDT) Received: from [129.147.241.111] ([unknown] [129.147.241.111]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KP300BDYKY94P90@mail-amer.sun.com>; Fri, 28 Aug 2009 11:37:21 -0600 (MDT) Date: Fri, 28 Aug 2009 11:36:48 -0600 From: Randall Ralphs Subject: Re: Transport Layer Retries (TLR) Support (PSARC/2009/461 FastTrack Timeout 09/02/20009) In-reply-to: <4A97399D.4050507@Sun.COM> Sender: Randall.Ralphs@sun.com To: Jianfei Wang Cc: Frank Che , PSARC-ext@sun.com, Javen Wu , yongfeng du , scsitgt-bj@sun.com Message-id: <4A9815B0.3020305@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=windows-1252 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 References: <4A94DA09.3080306@sun.com> <4A961920.7090002@sun.com> <4A97399D.4050507@Sun.COM> User-Agent: Thunderbird 2.0.0.22 (X11/20090720) Status: RO Content-Length: 6632 Jianfei Wang wrote: > Hi Randy, > > This project focuses on negotiating and setting the TLR control bit > through the driver interaction. The TLR operation is done at the > firmware level, not the driver level. So question 1, 2, 4 and 5 are not > in the scope of this project. When MPxIO was implemented it was tested with multi port SAS drives. > > For question 3, SAS1 doesn't support TLR when MPXIO is enabled. This is > specified in document 07-027r2 (SAS-2 Enabling and disabling Transport > Layer Retries) on T10. The SAS1 TLR mode sense/mode select has been > proved to be impractical when MPXIO is enabled, and the reason is ' If > the mode page is shared, then it's possible that multiple initiators > dont agree on the setting interfere with each other. This is less of a > problem if the mode page is per I_T nexus, but such implementations are > rare.' The protocol author found mode sense/mode select is a bad support > way, so SAS1 TLR support is abandoned when MPXIO is enabled. SAS2 > inquiries the VPD page to support TLR, it would support MPXIO. Tape drives are explicitly one user at at time so doing mode sense mode select on open (reservation) and on detected resets should not be an issue. Can the target driver determine if the drive supports TLR by reading the mode sense page? All of this will work if the target driver can detect if TLR is supported without the scsi_getcap() and if the HBA does not attempt to recover a single command forever. > > Best Regards > Jianfei Wang > > > Frank Che wrote: >> Some questions from Randy on this project: >> >> 1) How will this both work with command recovery that is in Nevada and >> doesn't exsist in previous versions of Solaris where it is not? >> >> 2) This looks like it expects the tape driver to be the only source of >> resets. With MPxIO enabled scsi_vhci can do resets when failing over >> paths that would not have originated from the target driver. Some >> HBA's also initiate resets on their own. >> >> 3) If MPxIO is enabled, the scsi_ifgetcap(9F) would not get the >> capability from the HBA and therefor would not do the mode sense/mode >> select correctly. See bug 6755363 as this would also effect >> SCSI_CAP_TRAN_LAYER_RETRIES. >> >> 4) How will this work with ComStar where the transport may not be as >> obvious to the target driver. >> >> 5) Does TLR give up? If a connection fails in a MPxIO invironment >> waiting for a connection to reconnect would prevent failing over. Also >> in a non-MPxIO environment not waiting could cause a failure that >> could otherwise could be recovered when reconnected. >> >> Frank. >> >> Frank Che wrote: >>> I'm sponsoring this case for Jianfei Wang. The timer is set for >>> 09/02/2009. >>> >>> -Frank. >>> >>> 1. Introduction >>> 1.1. Project/Component Working Name: >>> Transport Layer Retries (TLR) Support >>> >>> 1.2. Name of Document Author/Supplier: >>> Author: Jianfei Wang >>> >>> 1.3. Date of This Document: >>> 08/24/2009 >>> >>> 4. Technical Description: >>> >>> 4.1. Summary >>> >>> This project is to provide TLR (Transport Layer Retries) support >>> in Solaris so that tape drive devices could be better supported. >>> >>> Requested release binding is Patch/Micro. >>> >>> 4.2. Details >>> >>> To support Transport Layer Retries(TLR), a new SCSI transport >>> capability, SCSI_CAP_TRAN_LAYER_RETRIES, will be added. >>> HBA drivers need TLR support should be updated to know this capability. >>> The storage target driver should also be updated to coordinate >>> the TLR support. >>> >>> When a new tape drive device is detected, the target driver checks >>> the HBA's capability of TLR support with the scsi_ifgetcap(9F) interface >>> in the attach(9E) entry point. >>> >>> The target driver reads Vital Product Data (VPD) page 0x90 to >>> get the target device type. If VPD page 0x90 exists, the target device >>> type is SAS2; otherwise the target device type is SAS1. >>> >>> If the HBA supports TLR and the target device is a SAS1 device, then the >>> target driver uses SCSI command MODE_SENSE and MODE_SELECT to set the >>> mode page (Protocol-Specific Logical Unit mode page)'s TLR flag. If set >>> successfully, Initiator_Target_Lun nexus support TLR property. Because >>> the mode page is not persistent, the TLR flag bit will be lost when the >>> target device do a hard reset. So the target driver needs to set the >>> mode >>> page's TLR flag each time the target device completes hard reset. >>> >>> If the HBA supports TLR and the target device is SAS2 device, then the >>> target driver issues InquiryVPD to get the capability of the tape device >>> to check whether the target device is able to support TLR. If both the >>> HBA driver and the tape device support TLR, the target driver will >>> send a >>> commands packet with a special pkg_flag, FLAG_TLR, to notify >>> the HBA driver to set TLR in control bit before transportation. If the >>> Initiator_Target_Lun nexus does not support TLR, commands from the >>> target >>> driver to the HBA driver would fail and the pkt_reason will be set to >>> CMD_TLR_OFF. When the target driver get this reason, it will turn off >>> the TLR control. >>> >>> 4.3. Exported Interfaces >>> >>> Interface Commitment Comments >>> ============= ========== ===================== >>> SCSI_CAP_TRAN_LAYER_RETRIES Committed SCSI TLR capability >>> tran-layer-retries Committed SCSI_CAP_ASCII content >>> FLAG_TLR Committed pkg_flag indicating TLR is supported >>> CMD_TLR_OFF Committed pkt_reason indicating TLR is not supported >>> >>> 4.4. Bug/RFE Number(s) >>> 6647764 - Solaris storage driver Transport Layer Retries (TLR) support >>> >>> 4.5. References >>> Serial Attached SCSI - 1.1 (SAS-1.1) >>> Serial Attached SCSI - 2 (SAS-2) >>> SCSI Primary Commands - 4 (SPC-4) >>> http://www.t10.org/ >>> >>> 4.6. Manpage changes >>> scsi_hba_lookup_capstr.diff >>> ============================ >>> 156a157 >>> > SCSI_CAP_TRAN_LAYER_RETRIES >>> 157a159,161 >>> > "tran-layer-retries" >>> >>> scsi_ifgetcap.diff >>> =========================== >>> 126a127,128 >>> > tran-layer-retries Transport Layer retries is support by >>> > the host adapter: 0 disables, 1 enables. >>> >>> scsi_pkt.diff >>> =========================== >>> 226a227,230 >>> > FLAG_TLR Run command with Transport >>> > Layer Retries support >>> > >>> 323a328,330 >>> > CMD_TLR_OFF Transport Layer Retries turn off >>> > >>> >>> 6. Resources and Schedule: >>> 6.4. Steering Committee requested information >>> 6.4.1. Consolidation C-team Name: >>> ON >>> 6.5. ARC review type: Fast Track >>> 6.6. ARC Exposure: Open >>> >>> >>> > From Jianfei.Wang@sun.com Mon Aug 31 01:46:29 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n7V8kS97027378 for ; Mon, 31 Aug 2009 01:46:28 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n7V8kOUG010987; Mon, 31 Aug 2009 16:46:27 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KP80091FGDEH300@brm-avmta-1.central.sun.com>; Mon, 31 Aug 2009 02:46:26 -0600 (MDT) Received: from sineb-mail-1.sun.com ([192.18.19.6]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KP800MOSGDCW450@brm-avmta-1.central.sun.com>; Mon, 31 Aug 2009 02:46:25 -0600 (MDT) Received: from fe-apac-05.sun.com (fe-apac-05.sun.com [192.18.19.176] (may be forged)) by sineb-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n7V8kO93019277; Mon, 31 Aug 2009 08:46:24 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KP800C00G7GX100@mail-apac.sun.com>; Mon, 31 Aug 2009 16:46:24 +0800 (SGT) Received: from [129.158.219.20] ([unknown] [129.158.219.20]) by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KP8007L8GDAJ830@mail-apac.sun.com>; Mon, 31 Aug 2009 16:46:24 +0800 (SGT) Date: Mon, 31 Aug 2009 16:45:38 +0800 From: Jianfei Wang Subject: Re: Transport Layer Retries (TLR) Support (PSARC/2009/461 FastTrack Timeout 09/02/20009) In-reply-to: <4A9815B0.3020305@sun.com> Sender: Jianfei.Wang@sun.com To: Randall Ralphs Cc: Frank Che , PSARC-ext@sun.com, Javen Wu , yongfeng du , scsitgt-bj@sun.com Message-id: <4A9B8DB2.4050808@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=windows-1252 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 References: <4A94DA09.3080306@sun.com> <4A961920.7090002@sun.com> <4A97399D.4050507@Sun.COM> <4A9815B0.3020305@sun.com> User-Agent: Thunderbird 2.0.0.18 (X11/20081215) Status: RO Content-Length: 8401 Hi Randy, > > Jianfei Wang wrote: >> Hi Randy, >> >> This project focuses on negotiating and setting the TLR control bit >> through the driver interaction. The TLR operation is done at the >> firmware level, not the driver level. So question 1, 2, 4 and 5 are >> not in the scope of this project. > > When MPxIO was implemented it was tested with multi port SAS drives. > >> >> For question 3, SAS1 doesn't support TLR when MPXIO is enabled. This >> is specified in document 07-027r2 (SAS-2 Enabling and disabling >> Transport Layer Retries) on T10. The SAS1 TLR mode sense/mode select >> has been proved to be impractical when MPXIO is enabled, and the >> reason is ' If the mode page is shared, then it's possible that >> multiple initiators dont agree on the setting interfere with each >> other. This is less of a problem if the mode page is per I_T nexus, >> but such implementations are rare.' The protocol author found mode >> sense/mode select is a bad support way, so SAS1 TLR support is >> abandoned when MPXIO is enabled. SAS2 inquiries the VPD page to >> support TLR, it would support MPXIO. > > Tape drives are explicitly one user at at time so doing mode sense > mode select on open (reservation) and on detected resets should not be > an issue. > > Can the target driver determine if the drive supports TLR by reading > the mode sense page? All of this will work if the target driver can > detect if TLR is supported without the scsi_getcap() and if the HBA > does not attempt to recover a single command forever. The target driver could not determine the I_T_L support TLR, it should need initiator support TLR. Whether a I-T-L supports TLR not only need target device confirm that it has such capability by Mode Page or Inquiry VPD, but also initiator(HBA) confirm it's capability. That's why we introduce the scsi_getcap/scsi_setcap for HBA TLR capability. I do understand tape is able to support MPxIO, our concern is not whether we can detect Mode page information change after target reset. The problem is MPxIO introduced multiple paths to single target, but no one can guarantee all initiators associated to the paths are TLR-capable. SAS1.x TLR has some defects in protocol perspective, it request initiator enable/disable TLR feature by setting Mode page of the target. Take a example, if a tape connects to two HBA cards, one supports TLR but another doesn't support the feature, it is too hard to manage TLR since it's very hard disable/enable TLR support by mode sense/select frequently across different paths during transport. It's not worth to introduce complex logic in the solaris for the defect protocol implementation since the SAS2.0 correct the defects. So we decide to provide minimum support for SAS1.x tape only when MPxIO is disabled. The SAS2 TLR doesn't have this problem. It uses the inquiryVPD get TLR property, the capability of the LUN is fixed. Enabling the TLR only relies on initiator and we don't need enable/disable any mode page of target per different paths. Best Regards Jianfei Wang > >> >> Best Regards >> Jianfei Wang >> >> >> Frank Che wrote: >>> Some questions from Randy on this project: >>> >>> 1) How will this both work with command recovery that is in Nevada >>> and doesn't exsist in previous versions of Solaris where it is not? >>> >>> 2) This looks like it expects the tape driver to be the only source >>> of resets. With MPxIO enabled scsi_vhci can do resets when failing >>> over paths that would not have originated from the target driver. >>> Some HBA's also initiate resets on their own. >>> >>> 3) If MPxIO is enabled, the scsi_ifgetcap(9F) would not get the >>> capability from the HBA and therefor would not do the mode >>> sense/mode select correctly. See bug 6755363 as this would also >>> effect SCSI_CAP_TRAN_LAYER_RETRIES. >>> >>> 4) How will this work with ComStar where the transport may not be as >>> obvious to the target driver. >>> >>> 5) Does TLR give up? If a connection fails in a MPxIO invironment >>> waiting for a connection to reconnect would prevent failing over. >>> Also in a non-MPxIO environment not waiting could cause a failure >>> that could otherwise could be recovered when reconnected. >>> >>> Frank. >>> >>> Frank Che wrote: >>>> I'm sponsoring this case for Jianfei Wang. The timer is set for >>>> 09/02/2009. >>>> >>>> -Frank. >>>> >>>> 1. Introduction >>>> 1.1. Project/Component Working Name: >>>> Transport Layer Retries (TLR) Support >>>> >>>> 1.2. Name of Document Author/Supplier: >>>> Author: Jianfei Wang >>>> >>>> 1.3. Date of This Document: >>>> 08/24/2009 >>>> >>>> 4. Technical Description: >>>> >>>> 4.1. Summary >>>> >>>> This project is to provide TLR (Transport Layer Retries) support >>>> in Solaris so that tape drive devices could be better supported. >>>> >>>> Requested release binding is Patch/Micro. >>>> >>>> 4.2. Details >>>> >>>> To support Transport Layer Retries(TLR), a new SCSI transport >>>> capability, SCSI_CAP_TRAN_LAYER_RETRIES, will be added. >>>> HBA drivers need TLR support should be updated to know this >>>> capability. >>>> The storage target driver should also be updated to coordinate >>>> the TLR support. >>>> >>>> When a new tape drive device is detected, the target driver checks >>>> the HBA's capability of TLR support with the scsi_ifgetcap(9F) >>>> interface >>>> in the attach(9E) entry point. >>>> >>>> The target driver reads Vital Product Data (VPD) page 0x90 to >>>> get the target device type. If VPD page 0x90 exists, the target device >>>> type is SAS2; otherwise the target device type is SAS1. >>>> >>>> If the HBA supports TLR and the target device is a SAS1 device, >>>> then the >>>> target driver uses SCSI command MODE_SENSE and MODE_SELECT to set the >>>> mode page (Protocol-Specific Logical Unit mode page)'s TLR flag. If >>>> set >>>> successfully, Initiator_Target_Lun nexus support TLR property. Because >>>> the mode page is not persistent, the TLR flag bit will be lost when >>>> the >>>> target device do a hard reset. So the target driver needs to set >>>> the mode >>>> page's TLR flag each time the target device completes hard reset. >>>> >>>> If the HBA supports TLR and the target device is SAS2 device, then the >>>> target driver issues InquiryVPD to get the capability of the tape >>>> device >>>> to check whether the target device is able to support TLR. If both the >>>> HBA driver and the tape device support TLR, the target driver will >>>> send a >>>> commands packet with a special pkg_flag, FLAG_TLR, to notify >>>> the HBA driver to set TLR in control bit before transportation. If the >>>> Initiator_Target_Lun nexus does not support TLR, commands from the >>>> target >>>> driver to the HBA driver would fail and the pkt_reason will be set to >>>> CMD_TLR_OFF. When the target driver get this reason, it will turn off >>>> the TLR control. >>>> >>>> 4.3. Exported Interfaces >>>> >>>> Interface Commitment Comments >>>> ============= ========== ===================== >>>> SCSI_CAP_TRAN_LAYER_RETRIES Committed SCSI TLR capability >>>> tran-layer-retries Committed SCSI_CAP_ASCII content >>>> FLAG_TLR Committed pkg_flag indicating TLR is supported >>>> CMD_TLR_OFF Committed pkt_reason indicating TLR is not supported >>>> >>>> 4.4. Bug/RFE Number(s) >>>> 6647764 - Solaris storage driver Transport Layer Retries (TLR) support >>>> >>>> 4.5. References >>>> Serial Attached SCSI - 1.1 (SAS-1.1) >>>> Serial Attached SCSI - 2 (SAS-2) >>>> SCSI Primary Commands - 4 (SPC-4) >>>> http://www.t10.org/ >>>> >>>> 4.6. Manpage changes >>>> scsi_hba_lookup_capstr.diff >>>> ============================ >>>> 156a157 >>>> > SCSI_CAP_TRAN_LAYER_RETRIES >>>> 157a159,161 >>>> > "tran-layer-retries" >>>> >>>> scsi_ifgetcap.diff >>>> =========================== >>>> 126a127,128 >>>> > tran-layer-retries Transport Layer retries is support by >>>> > the host adapter: 0 disables, 1 enables. >>>> >>>> scsi_pkt.diff >>>> =========================== >>>> 226a227,230 >>>> > FLAG_TLR Run command with Transport >>>> > Layer Retries support >>>> > >>>> 323a328,330 >>>> > CMD_TLR_OFF Transport Layer Retries turn off >>>> > >>>> >>>> 6. Resources and Schedule: >>>> 6.4. Steering Committee requested information >>>> 6.4.1. Consolidation C-team Name: >>>> ON >>>> 6.5. ARC review type: Fast Track >>>> 6.6. ARC Exposure: Open >>>> >>>> >>>> >> From Frank.Che@sun.com Wed Sep 2 17:37:10 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n830b9pK016421 for ; Wed, 2 Sep 2009 17:37:09 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n830b8xB036682; Wed, 2 Sep 2009 18:37:09 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KPD0070LDPVLY00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 02 Sep 2009 17:37:07 -0700 (PDT) Received: from sineb-mail-2.sun.com ([192.18.19.7]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KPD005ZGDPT3C10@nwk-avmta-1.sfbay.Sun.COM>; Wed, 02 Sep 2009 17:37:06 -0700 (PDT) Received: from fe-apac-05.sun.com (fe-apac-05.sun.com [192.18.19.176] (may be forged)) by sineb-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n830b5oR009535; Thu, 03 Sep 2009 00:37:05 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KPD00H00DM36S00@mail-apac.sun.com>; Thu, 03 Sep 2009 08:37:05 +0800 (SGT) Received: from [129.158.218.60] ([unknown] [129.158.218.60]) by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KPD00AV1DPK5QF0@mail-apac.sun.com>; Thu, 03 Sep 2009 08:37:05 +0800 (SGT) Date: Thu, 03 Sep 2009 08:32:50 +0800 From: Frank Che Subject: Re: Transport Layer Retries (TLR) Support (PSARC/2009/461 FastTrack Timeout 09/02/20009) In-reply-to: <4A9B8DB2.4050808@Sun.COM> Sender: Frank.Che@sun.com To: Jianfei Wang , PSARC-ext@sun.com Cc: Javen Wu , yongfeng du , scsitgt-bj@sun.com Message-id: <4A9F0EB2.2090009@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <4A94DA09.3080306@sun.com> <4A961920.7090002@sun.com> <4A97399D.4050507@Sun.COM> <4A9815B0.3020305@sun.com> <4A9B8DB2.4050808@Sun.COM> User-Agent: Thunderbird 2.0.0.19 (X11/20090218) Status: RO Content-Length: 51 This case was approved in the PSARC meeting today.