From John.Forte@sun.com Wed Mar 11 08:59:24 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2BFxN2W027596 for ; Wed, 11 Mar 2009 08:59:23 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n2BFxJ46018918 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 11 Mar 2009 23:59:22 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KGC00CUZN2WPJ00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 08:59:20 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KGC0089YN21JY40@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 08:58:49 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n2BFwnYP016769 for ; Wed, 11 Mar 2009 15:58:49 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23 2008)) id <0KGC00400MAN2I00@mail-amer.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 09:58:49 -0600 (MDT) Received: from dhcp-ubrm05-47-103.Central.Sun.COM ([unknown] [129.147.47.103]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23 2008)) with ESMTPSA id <0KGC00DYDN1Q7170@mail-amer.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 09:58:38 -0600 (MDT) Date: Wed, 11 Mar 2009 09:55:27 -0700 From: John Forte Subject: iSCSI Target PGR directory for ZVOLs [PSARC/2009/168 Self Review] Sender: John.Forte@sun.com To: PSARC-ext@sun.com Cc: jim.dunham@sun.com Message-id: <49B7ECFF.4050202@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=windows-1252 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 User-Agent: Thunderbird 2.0.0.19 (Macintosh/20081209) Status: RO Content-Length: 4387 I'm sponsoring the following fasttrack for Jim Dunham. Requested binding is Patch. I believe this case qualifies for self-review. If anyone disagrees, I'll convert it and set the timer. - John Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: iSCSI Target Daemon's SCSI-3 PGR data needs to follow ZVOLs 1.2. Name of Document Author/Supplier: James Dunham 1.3. Date of This Document: 03/10/09 2. Project Summary 2.1. Project Description: SCSI-3 PGR (Persistent Group Reservation) data must follow the ZVOL that is the backing store for the SCSI-3 LUN. Since ZVOLs are exportable from one host, importable on another host, PGR data must follow. The effort provides a single SCF option to support a PGR_basedir, a directory which has the means to follow a ZFS Storage Pool mount point, as it is exported and imported between nodes. 2.2. Risks and Assumptions: SCSI-3 PGR data in clustered storage nodes, consisting of two or more nodes, need to assured this SCF entry is consistent set between nodes. 3. Business Summary N/A 4. Technical Description: 4.1. Details: The current iSCSI Target Daemon stores SCSI-3 PGR data to the directory /etc/iscsi. This 'pins' the PGR data to a single Solaris host, even though a ZVOL or ZVOLs can be exported from the current node, imported on another node. There is no room in the ZVOL itself, as the iSCSI Client own 100% of the blocks in a ZVOL. There is little remaining space in the 'iscsi-options' ZVOL property, as other iSCSI properties consume ~1000 bytes, of a 2K limit, and PGR data can be 1000s or 10,000s of bytes in size, depending on the number of nodes in a clustering solution. The solution is to provide a PGR base directory, specified in the iSCSI Target Daemon's SCF data. svccfg -s iscsitgt setprop iscsitgt/PGR-basedir = \ astring: "" 4.1.1 Interfaces iscsitgt/PGR-basedir The PGR base directory is a pathname specification that contains one or more ZFS filesystem mount points, a filesystem which contains one or more ZVOLs, ZVOLS which have the following property set: zfs set shareiscsi={on|off} / 4.2. Bug/RFE Number(s): 6794624 SCSI-3 PGR data for LUs whose backing store is a ZVOL, need to persist data in a ZVOL attributes 4.3. In Scope: For OpenSolaris single node, not setting the SCF property will retain current behavior, of storing SCSI-3 PGR in /etc/iscsi For OpenSolaris multinode, setting the SCF property to a "/", or some other ZFS mount point, will provide a default, or alternate means for a ZFS storage pool filesystem mount point to be set. 4.4. Out of Scope: This functionality has no impact on COMSTAR or its support of SCSI-3 PGR 4.5. Interfaces: svccfg(1m) – service configuration facility l libscf(3LIB) – service configuration facility library (man pages ... 4.6. Doc Impact: The iSCSI Target man page will be updataed to document this functionality. 6815596 iscsitgtd(1M) needs to be updated to cover PGR base directory option. 4.7. Admin/Config Impact: Awareness of this setting, and how it relates SCSI-3 PGR data and ZVOLs 4.8. HA Impact: For OpenSolaris, this setting will allow SCSI-3 PGR data to follow ZVOLs and their storage pools 4.9. I18N/L10N Impact: N/A 4.10. Packaging & Delivery: Part of SUNWiscsitgt 4.11. Security Impact: iSCSI Target Daemon is a root privileged Solaris Daemon 4.12. Dependencies: The changes are all project private 5. Reference Documents: 6794624 SCSI-3 PGR data for LUs whose backing store is a ZVOL, need to persist data in a ZVOL attributes 6. Resources and Schedule: 6.1. Projected Availability: ASAP 6.2. Cost of Effort: 1 week 6.3. Cost of Capital Resources: N/A 6.4. Product Approval Committee requested information: 6.4.1. Consolidation or Component Name: ON 6.4.3. Type of CPT Review and Approval expected: BugFix or RFE 6.4.4. Project Boundary Conditions: N/A 6.4.5. Is this a necessary project for OEM agreements: No 6.4.6. Notes: 6.4.7. Target RTI Date/Release: SNV_111 6.4.8. Target Code Design Review Date: 03/04/09 6.4.9. Update approval addition: N/A 6.5. ARC review type: SelfReview 6.6. ARC Exposure: open 6.6.1. Rationale: 7. Prototype Availability: 7.1. Prototype Availability: N/A 7.2. Prototype Cost: N/A From Darren.Moffat@sun.com Wed Mar 11 09:17:18 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2BGHHWP028672 for ; Wed, 11 Mar 2009 09:17:18 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2BGH4qu027652 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 11 Mar 2009 16:17:17 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KGC00G8NNWRLF00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 09:17:15 -0700 (PDT) Received: from gmp-eb-inf-1.sun.com ([192.18.6.21]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KGC0086GNWBK260@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 09:17:00 -0700 (PDT) Received: from fe-emea-09.sun.com (gmp-eb-lb-1-fe3.eu.sun.com [192.18.6.10]) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n2BGGx8E004981 for ; Wed, 11 Mar 2009 16:16:59 +0000 (GMT) Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23 2008)) id <0KGC00N00NBS0K00@fe-emea-09.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 16:16:59 +0000 (GMT) Received: from [129.156.173.21] ([unknown] [129.156.173.21]) by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23 2008)) with ESMTPSA id <0KGC00M90NVXCKA0@fe-emea-09.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 16:16:46 +0000 (GMT) Date: Wed, 11 Mar 2009 16:16:45 +0000 From: Darren J Moffat Subject: Re: iSCSI Target PGR directory for ZVOLs [PSARC/2009/168 Self Review] In-reply-to: <49B7ECFF.4050202@sun.com> Sender: Darren.Moffat@sun.com To: John Forte Cc: PSARC-ext@sun.com, jim.dunham@sun.com Message-id: <49B7E3ED.2050107@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=windows-1252 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <49B7ECFF.4050202@sun.com> User-Agent: Thunderbird 2.0.0.18 (X11/20081201) Status: RO Content-Length: 1575 John Forte wrote: > This 'pins' the PGR data to a single Solaris host, even though a ZVOL or > ZVOLs can be exported from the current node, imported on another node. > > There is no room in the ZVOL itself, as the iSCSI Client own 100% of the > blocks in a ZVOL. There is little remaining space in the 'iscsi-options' > ZVOL property, as other iSCSI properties consume ~1000 bytes, of a 2K > limit, and PGR data can be 1000s or 10,000s of bytes in size, depending > on the number of nodes in a clustering solution. The ZFS dataset property mentioned here is the hidden iscsioptions property right ? So why not use multiple zap objects to store the PGR data (that gives a lot more scope since now you have a list of name=value pairs) like is already done for the ZFS delegation data. Having some of the data inside the ZFS dataset and some of it in plainfiles (even if they are stored in ZFS) seems like a recipe for getting things out of sync to me. Why is it okay for PGR data not to be tightly tide to the ZVOL yet all the other iscsi options are ? In fact the rationale above suggests the desire is that it should be in the same ZFS dataset as the ZVOL. Now having said that I can see how this case can work just fine in a tightly controlled environment. If there is a time to market issue then I can grudgingly approve of this case as specified with one small request that the SCF option be private and a plan is put in place to investigate putting the PGR data with the ZVOL (either a zap object or something else appropriate). -- Darren J Moffat From James.Dunham@sun.com Wed Mar 11 10:09:08 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2BH97l7004362 for ; Wed, 11 Mar 2009 10:09:08 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n2BH95kg001760 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 12 Mar 2009 01:09:06 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KGC00M07QB51900@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 10:09:05 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KGC00GQ6QB38LB0@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 10:09:04 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n2BH93PZ004449 for ; Wed, 11 Mar 2009 17:09:03 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23 2008)) id <0KGC00A00Q28UO00@mail-amer.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 11:09:03 -0600 (MDT) Received: from [10.0.223.239] ([unknown] [129.150.66.69]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23 2008)) with ESMTPSA id <0KGC00EAGQB19OC0@mail-amer.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 11:09:03 -0600 (MDT) Date: Wed, 11 Mar 2009 11:08:58 -0600 From: Jim Dunham Subject: Re: iSCSI Target PGR directory for ZVOLs [PSARC/2009/168 Self Review] In-reply-to: <49B7E3ED.2050107@Sun.COM> Sender: James.Dunham@sun.com To: Darren J Moffat Cc: John Forte , PSARC-ext@sun.com, jim.dunham@sun.com Message-id: <912F74E8-7227-486E-B8ED-1AB48C42483F@sun.com> MIME-version: 1.0 X-Mailer: Apple Mail (2.930.3) Content-type: text/plain; delsp=yes; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <49B7ECFF.4050202@sun.com> <49B7E3ED.2050107@Sun.COM> Status: RO Content-Length: 2889 Darren, > John Forte wrote: >> This 'pins' the PGR data to a single Solaris host, even though a >> ZVOL or >> ZVOLs can be exported from the current node, imported on another >> node. >> There is no room in the ZVOL itself, as the iSCSI Client own 100% >> of the >> blocks in a ZVOL. There is little remaining space in the 'iscsi- >> options' >> ZVOL property, as other iSCSI properties consume ~1000 bytes, of a 2K >> limit, and PGR data can be 1000s or 10,000s of bytes in size, >> depending >> on the number of nodes in a clustering solution. > > The ZFS dataset property mentioned here is the hidden iscsioptions > property right ? Yes. > So why not use multiple zap objects to store the PGR data (that > gives a lot more scope since now you have a list of name=value > pairs) like is already done for the ZFS delegation data. Doing this causes an iSCSI Target Daemon architecture / implementation violation. SCSI-3 PGR operations are SCSI CDB data path operations, invoked by one or more iSCSI Initiators. Interaction between the iSCSI Target Daemon and ZFS, specific to the iscsioptions property is a control path operation. During initIal prototype testing, crossing over from the data path to control path to store or retrieve PGR data caused hangs within daemon. > Having some of the data inside the ZFS dataset and some of it in > plainfiles (even if they are stored in ZFS) seems like a recipe for > getting things out of sync to me. Why is it okay for PGR data not > to be tightly tide to the ZVOL yet all the other iscsi options > are ? In fact the rationale above suggests the desire is that it > should be in the same ZFS dataset as the ZVOL. Given the hang concern outlined above, keeping ZVOL I/O and PGR I/O within the data path side of the iSCSI Target Daemon mitigates risk, is essentially identical to what is done today, where PGR data is now being written into /etc/iscsi/... > Now having said that I can see how this case can work just fine in a > tightly controlled environment. If there is a time to market issue > then I can grudgingly approve of this case as specified with one > small request that the SCF option be private and a plan is put in > place to investigate putting the PGR data with the ZVOL (either a > zap object or something else appropriate). Setting a PGR base directory is a tightly controlled environmental variable. If needed, it would be set once, and not changed. In regards to putting into place a plan to investigate other options, the iSCSI Target Daemon's future is limited due to its replacement with a new iSCSI Target in COMSTAR. That said, there is more development work to be done in COMSTAR before the iSCSI Target Daemon starts its EOL process, and a new PGR implementation is one of these development tasks. > > -- > Darren J Moffat Jim Dunham From Darren.Moffat@sun.com Wed Mar 11 10:23:19 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2BHNIGo005059 for ; Wed, 11 Mar 2009 10:23:18 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n2BHNFlV011144 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 12 Mar 2009 01:23:17 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KGC00605QYR2S00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 10:23:15 -0700 (PDT) Received: from gmp-eb-inf-1.sun.com ([192.18.6.21]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KGC008PFQYOJZC0@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 10:23:13 -0700 (PDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-2-fe2.eu.sun.com [192.18.6.11]) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n2BHNC0w013940 for ; Wed, 11 Mar 2009 17:23:12 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23 2008)) id <0KGC00C00QKD0P00@fe-emea-10.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 17:23:12 +0000 (GMT) Received: from [129.156.173.21] ([unknown] [129.156.173.21]) by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23 2008)) with ESMTPSA id <0KGC00AYDQYNQ4C0@fe-emea-10.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 17:23:12 +0000 (GMT) Date: Wed, 11 Mar 2009 17:23:11 +0000 From: Darren J Moffat Subject: Re: iSCSI Target PGR directory for ZVOLs [PSARC/2009/168 Self Review] In-reply-to: <912F74E8-7227-486E-B8ED-1AB48C42483F@sun.com> Sender: Darren.Moffat@sun.com To: Jim Dunham Cc: John Forte , PSARC-ext@sun.com, jim.dunham@sun.com Message-id: <49B7F37F.1040009@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <49B7ECFF.4050202@sun.com> <49B7E3ED.2050107@Sun.COM> <912F74E8-7227-486E-B8ED-1AB48C42483F@sun.com> User-Agent: Thunderbird 2.0.0.18 (X11/20081201) Status: RO Content-Length: 2360 Jim Dunham wrote: > Darren, > >> John Forte wrote: >>> This 'pins' the PGR data to a single Solaris host, even though a ZVOL or >>> ZVOLs can be exported from the current node, imported on another node. >>> There is no room in the ZVOL itself, as the iSCSI Client own 100% of the >>> blocks in a ZVOL. There is little remaining space in the 'iscsi-options' >>> ZVOL property, as other iSCSI properties consume ~1000 bytes, of a 2K >>> limit, and PGR data can be 1000s or 10,000s of bytes in size, depending >>> on the number of nodes in a clustering solution. >> >> The ZFS dataset property mentioned here is the hidden iscsioptions >> property right ? > > Yes. > >> So why not use multiple zap objects to store the PGR data (that gives >> a lot more scope since now you have a list of name=value pairs) like >> is already done for the ZFS delegation data. > > Doing this causes an iSCSI Target Daemon architecture / implementation > violation. So then I'm confused why the case even mentions that iscsioptions private dataset property has limited size if that was never an option. > SCSI-3 PGR operations are SCSI CDB data path operations, invoked by one > or more iSCSI Initiators. Interaction between the iSCSI Target Daemon > and ZFS, specific to the iscsioptions property is a control path > operation. During initIal prototype testing, crossing over from the data > path to control path to store or retrieve PGR data caused hangs within > daemon. > >> Having some of the data inside the ZFS dataset and some of it in >> plainfiles (even if they are stored in ZFS) seems like a recipe for >> getting things out of sync to me. Why is it okay for PGR data not to >> be tightly tide to the ZVOL yet all the other iscsi options are ? In >> fact the rationale above suggests the desire is that it should be in >> the same ZFS dataset as the ZVOL. > > Given the hang concern outlined above, keeping ZVOL I/O and PGR I/O > within the data path side of the iSCSI Target Daemon mitigates risk, is > essentially identical to what is done today, where PGR data is now being > written into /etc/iscsi/... Okay. As I said I'm okay for the case to go forward as specified I was just trying to offer an alternative to the space restricted iscsioptions that still allowed the PCR config to be stored with the dataset. -- Darren J Moffat From Mark.Carlson@Sun.COM Wed Mar 11 10:28:16 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2BHSFso005310 for ; Wed, 11 Mar 2009 10:28:16 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n2BHSEsw030033 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 11 Mar 2009 11:28:15 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KGC00M37R72V900@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 10:28:14 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KGC00GOUR708LD0@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 10:28:12 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n2BHSCGK006432 for ; Wed, 11 Mar 2009 17:28:12 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23 2008)) id <0KGC00A00Q28UO00@mail-amer.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 11:28:12 -0600 (MDT) Received: from boulder-dhcp-015.Central.Sun.COM ([unknown] [172.20.27.15]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23 2008)) with ESMTPSA id <0KGC00IH5R6KH070@mail-amer.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Mar 2009 11:27:56 -0600 (MDT) Date: Wed, 11 Mar 2009 11:27:55 -0600 From: "Mark A. Carlson" Subject: Re: iSCSI Target PGR directory for ZVOLs [PSARC/2009/168 Self Review] In-reply-to: <49B7F37F.1040009@Sun.COM> Sender: Mark.Carlson@Sun.COM To: Darren J Moffat Cc: Jim Dunham , John Forte , PSARC-ext@Sun.COM, jim.dunham@Sun.COM Message-id: <49B7F49B.7060502@sun.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_tvr1gqH1/C+z2O6q9IOKbg)" X-PMX-Version: 5.4.1.325704 References: <49B7ECFF.4050202@sun.com> <49B7E3ED.2050107@Sun.COM> <912F74E8-7227-486E-B8ED-1AB48C42483F@sun.com> <49B7F37F.1040009@Sun.COM> User-Agent: Thunderbird 2.0.0.19 (Macintosh/20081209) Status: RO Content-Length: 7080 This is a multi-part message in MIME format. --Boundary_(ID_tvr1gqH1/C+z2O6q9IOKbg) Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT I'll add my +1 to this as well. -- mark Darren J Moffat wrote: > Jim Dunham wrote: >> Darren, >> >>> John Forte wrote: >>>> This 'pins' the PGR data to a single Solaris host, even though a >>>> ZVOL or >>>> ZVOLs can be exported from the current node, imported on another node. >>>> There is no room in the ZVOL itself, as the iSCSI Client own 100% >>>> of the >>>> blocks in a ZVOL. There is little remaining space in the >>>> 'iscsi-options' >>>> ZVOL property, as other iSCSI properties consume ~1000 bytes, of a 2K >>>> limit, and PGR data can be 1000s or 10,000s of bytes in size, >>>> depending >>>> on the number of nodes in a clustering solution. >>> >>> The ZFS dataset property mentioned here is the hidden iscsioptions >>> property right ? >> >> Yes. >> >>> So why not use multiple zap objects to store the PGR data (that >>> gives a lot more scope since now you have a list of name=value >>> pairs) like is already done for the ZFS delegation data. >> >> Doing this causes an iSCSI Target Daemon architecture / >> implementation violation. > > So then I'm confused why the case even mentions that iscsioptions > private dataset property has limited size if that was never an option. > >> SCSI-3 PGR operations are SCSI CDB data path operations, invoked by >> one or more iSCSI Initiators. Interaction between the iSCSI Target >> Daemon and ZFS, specific to the iscsioptions property is a control >> path operation. During initIal prototype testing, crossing over from >> the data path to control path to store or retrieve PGR data caused >> hangs within daemon. >> >>> Having some of the data inside the ZFS dataset and some of it in >>> plainfiles (even if they are stored in ZFS) seems like a recipe for >>> getting things out of sync to me. Why is it okay for PGR data not >>> to be tightly tide to the ZVOL yet all the other iscsi options are >>> ? In fact the rationale above suggests the desire is that it should >>> be in the same ZFS dataset as the ZVOL. >> >> Given the hang concern outlined above, keeping ZVOL I/O and PGR I/O >> within the data path side of the iSCSI Target Daemon mitigates risk, >> is essentially identical to what is done today, where PGR data is now >> being written into /etc/iscsi/... > > Okay. > > As I said I'm okay for the case to go forward as specified I was just > trying to offer an alternative to the space restricted iscsioptions > that still allowed the PCR config to be stored with the dataset. > -- * Mark A. Carlson * Sr. Architect *Systems Group* Phone x69559 / 303-223-6139 Email Mark.Carlson@Sun.COM --Boundary_(ID_tvr1gqH1/C+z2O6q9IOKbg) Content-type: text/html; charset=ISO-8859-1 Content-transfer-encoding: 7BIT I'll add my +1 to this as well.

-- mark

Darren J Moffat wrote:
Jim Dunham wrote:
Darren,

John Forte wrote:
This 'pins' the PGR data to a single Solaris host, even though a ZVOL or
ZVOLs can be exported from the current node, imported on another node.
There is no room in the ZVOL itself, as the iSCSI Client own 100% of the
blocks in a ZVOL. There is little remaining space in the 'iscsi-options'
ZVOL property, as other iSCSI properties consume ~1000 bytes, of a 2K
limit, and PGR data can be 1000s or 10,000s of bytes in size, depending
on the number of nodes in a clustering solution.

The ZFS dataset property mentioned here is the hidden iscsioptions property right ?

Yes.

So why not use multiple zap objects to store the PGR data (that gives a lot more scope since now you have a list of name=value pairs) like is already done for the ZFS delegation data.

Doing this causes an iSCSI Target Daemon architecture / implementation violation.

So then I'm confused why the case even mentions that iscsioptions private dataset property has limited size if that was never an option.

SCSI-3 PGR operations are SCSI CDB data path operations, invoked by one or more iSCSI Initiators. Interaction between the iSCSI Target Daemon and ZFS, specific to the iscsioptions property is a control path operation. During initIal prototype testing, crossing over from the data path to control path to store or retrieve PGR data caused hangs within daemon.

Having some of the data inside the ZFS dataset and some of it in plainfiles (even if they are stored in ZFS) seems like a recipe for getting things out of sync to me.  Why is it okay for PGR data not to be tightly tide to the ZVOL yet all the other iscsi options are ?  In fact the rationale above suggests the desire is that it should be in the same ZFS dataset as the ZVOL.

Given the hang concern outlined above, keeping ZVOL I/O and PGR I/O within the data path side of the iSCSI Target Daemon mitigates risk, is essentially identical to what is done today, where PGR data is now being written into /etc/iscsi/...

Okay.

As I said I'm okay for the case to go forward as specified I was just trying to offer an alternative to the space restricted iscsioptions that still allowed the PCR config to be stored with the dataset.


--
Mark A. Carlson
Sr. Architect

Systems Group
Phone x69559 / 303-223-6139
Email Mark.Carlson@Sun.COM

--Boundary_(ID_tvr1gqH1/C+z2O6q9IOKbg)-- From Ellard.Roush@sun.com Thu Mar 12 08:29:21 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CFTKx6007863 for ; Thu, 12 Mar 2009 08:29:21 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2CFTHH6010079; Thu, 12 Mar 2009 15:29:19 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KGE0020VGCTEP00@nwk-avmta-1.sfbay.Sun.COM>; Thu, 12 Mar 2009 08:29:17 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KGE003OAGCOGEE0@nwk-avmta-1.sfbay.Sun.COM>; Thu, 12 Mar 2009 08:29:12 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n2CFTCQT002956; Thu, 12 Mar 2009 08:29:12 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KGE00300G0MJD00@fe-sfbay-10.sun.com>; Thu, 12 Mar 2009 08:29:12 -0700 (PDT) Received: from [129.146.106.191] ([unknown] [129.146.106.191]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KGE00DCHGCEDZ90@fe-sfbay-10.sun.com>; Thu, 12 Mar 2009 08:29:02 -0700 (PDT) Date: Thu, 12 Mar 2009 07:29:01 -0800 From: Ellard Roush Subject: iSCSI Target PGR directory for ZVOLs [PSARC/2009/168 Self Review] Sender: Ellard.Roush@sun.com To: PSARC-ext@sun.com, Jim Dunham Cc: sc-haci , Andrew Hisgen , colorado-iteam@sun.com Reply-to: Ellard.Roush@sun.com Message-id: <49B92A3D.4050101@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 User-Agent: Thunderbird 2.0.0.16 (X11/20080807) Status: RO Content-Length: 1041 To Jim and PSARC From Ellard Roush Subject iSCSI Target PGR directory for ZVOLs [PSARC/2009/168 Self Review] The Sun Cluster product uses SCSI-3 PGR for both Fencing and Quorum operations. Sun Cluster also uses SCSI-2, but that is not a subject of this case. At this point in time, a ZVOL is only accessible from one machine at a time. ZFS enhancements happen regularly. If a ZVOL can be accessed by multiple machines at one time please let me know. If a ZVOL could be accessed by multiple machines concurrently, the ZVOL could be used in place of Solaris Volume Manager. This would be valuable in a high availability cluster. Once a ZVOL can be accessed by multiple machines, we would need the ZVOL to support SCSI-3 PGR for Fencing and Quorum. The one-pager does not mention whether this project will support SCSI-3 PGR operations from multiple nodes concurrently. Please explicitly identify whether this will be supported. Naturally, we would like to see SCSI-3 PGR operations on a ZVOL supported for multiple nodes concurrently. From James.Dunham@sun.com Thu Mar 12 08:52:29 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CFqS3I008292 for ; Thu, 12 Mar 2009 08:52:29 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2CFqFdO026594; Thu, 12 Mar 2009 15:52:28 GMT Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KGE00883HFFCY00@nwk-avmta-2.sfbay.sun.com>; Thu, 12 Mar 2009 08:52:27 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KGE008TMHF13J00@nwk-avmta-2.sfbay.sun.com>; Thu, 12 Mar 2009 08:52:13 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n2CFqDBX014096; Thu, 12 Mar 2009 15:52:13 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KGE00700FPYX600@mail-amer.sun.com>; Thu, 12 Mar 2009 09:52:13 -0600 (MDT) Received: from dhcp-ubrm05-50-13.Central.Sun.COM ([unknown] [129.147.50.13]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KGE008TYHEV7Q40@mail-amer.sun.com>; Thu, 12 Mar 2009 09:52:07 -0600 (MDT) Date: Thu, 12 Mar 2009 09:52:03 -0600 From: Jim Dunham Subject: Re: iSCSI Target PGR directory for ZVOLs [PSARC/2009/168 Self Review] In-reply-to: <49B92A3D.4050101@sun.com> Sender: James.Dunham@sun.com To: Ellard.Roush@sun.com Cc: PSARC-ext@sun.com, sc-haci , Andrew Hisgen , colorado-iteam@sun.com Message-id: <890C829A-A51E-4C24-B5FB-6595FBDF36BC@Sun.COM> MIME-version: 1.0 X-Mailer: Apple Mail (2.930.3) Content-type: text/plain; delsp=yes; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <49B92A3D.4050101@sun.com> Status: RO Content-Length: 2124 Ellard, > To Jim and PSARC > From Ellard Roush > Subject iSCSI Target PGR directory for ZVOLs > [PSARC/2009/168 Self Review] > > The Sun Cluster product uses SCSI-3 PGR for both Fencing > and Quorum operations. Sun Cluster also uses SCSI-2, > but that is not a subject of this case. > > At this point in time, a ZVOL is only accessible from > one machine at a time. ZFS enhancements happen regularly. > If a ZVOL can be accessed by multiple machines at one > time please let me know. At this time a ZVOL can never be accessed from more than a single host at a time. This is an artifact of ZFS, in that a ZFS Storage Pool can only be imported on a single host at a time, and ZVOLs are created out of ZFS filesystems, which are created out of the ZFS Storage Pools. This PSARC case and related change is based on the fact that a ZFS storage pool can be exported from one node, imported on another, and although the ZVOL moves between storage nodes, the ZVOLs associated SCSI-3 PGR data (if present) was left behind. There will be an forthcoming iscsitgt man page describing this change, and how to make the SCSI-3 PGR data follow the ZFS Storage Pool, and thus any ZVOLs created out of this storage pool. > If a ZVOL could be accessed by multiple machines > concurrently, the ZVOL could be used in place > of Solaris Volume Manager. This would be valuable > in a high availability cluster. > > Once a ZVOL can be accessed by multiple machines, > we would need the ZVOL to support SCSI-3 PGR > for Fencing and Quorum. The one-pager does not > mention whether this project will support > SCSI-3 PGR operations from multiple nodes > concurrently. Please explicitly identify > whether this will be supported. > > Naturally, we would like to see SCSI-3 PGR > operations on a ZVOL supported for > multiple nodes concurrently. SCSI-3 PGR command are supported with iSCSI Target Daemon LUNs, and soon all SCSI Targets in COMSTAR. Of course COMSTAR is storage node, and there is no support for a COMSTAR storage node to also be a Sun Cluster node. Jim Dunham Engineering Manager Sun Microsystems, Inc. From Daniel.Orbach@sun.com Thu Mar 12 09:06:14 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CG6DGm008970 for ; Thu, 12 Mar 2009 09:06:14 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2CG6B1D008779; Thu, 12 Mar 2009 16:06:12 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KGE0080LI2AKV00@nwk-avmta-1.sfbay.Sun.COM>; Thu, 12 Mar 2009 09:06:10 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KGE00679I25SF20@nwk-avmta-1.sfbay.Sun.COM>; Thu, 12 Mar 2009 09:06:05 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n2CG65pw023097; Thu, 12 Mar 2009 09:06:05 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KGE00K00EYF4800@fe-sfbay-09.sun.com>; Thu, 12 Mar 2009 09:06:04 -0700 (PDT) Received: from [129.146.107.220] ([unknown] [129.146.107.220]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KGE0078QI21SRB0@fe-sfbay-09.sun.com>; Thu, 12 Mar 2009 09:06:02 -0700 (PDT) Date: Thu, 12 Mar 2009 09:04:09 -0700 From: Dan Orbach Subject: Re: iSCSI Target PGR directory for ZVOLs [PSARC/2009/168 Self Review] In-reply-to: <890C829A-A51E-4C24-B5FB-6595FBDF36BC@Sun.COM> Sender: Daniel.Orbach@sun.com To: Jim Dunham Cc: Ellard.Roush@sun.com, PSARC-ext@sun.com, sc-haci , Andrew Hisgen , colorado-iteam@sun.com, gdd Message-id: <49B93279.60905@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <49B92A3D.4050101@sun.com> <890C829A-A51E-4C24-B5FB-6595FBDF36BC@Sun.COM> User-Agent: Thunderbird 1.5.0.8 (X11/20061127) Status: RO Content-Length: 2505 Hi Jim, way below... Jim Dunham wrote: > Ellard, > >> To Jim and PSARC >> From Ellard Roush >> Subject iSCSI Target PGR directory for ZVOLs >> [PSARC/2009/168 Self Review] >> >> The Sun Cluster product uses SCSI-3 PGR for both Fencing >> and Quorum operations. Sun Cluster also uses SCSI-2, >> but that is not a subject of this case. >> >> At this point in time, a ZVOL is only accessible from >> one machine at a time. ZFS enhancements happen regularly. >> If a ZVOL can be accessed by multiple machines at one >> time please let me know. > > At this time a ZVOL can never be accessed from more than a single host > at a time. This is an artifact of ZFS, in that a ZFS Storage Pool can > only be imported on a single host at a time, and ZVOLs are created out > of ZFS filesystems, which are created out of the ZFS Storage Pools. > > This PSARC case and related change is based on the fact that a ZFS > storage pool can be exported from one node, imported on another, and > although the ZVOL moves between storage nodes, the ZVOLs associated > SCSI-3 PGR data (if present) was left behind. > > There will be an forthcoming iscsitgt man page describing this change, > and how to make the SCSI-3 PGR data follow the ZFS Storage Pool, and > thus any ZVOLs created out of this storage pool. > >> If a ZVOL could be accessed by multiple machines >> concurrently, the ZVOL could be used in place >> of Solaris Volume Manager. This would be valuable >> in a high availability cluster. >> >> Once a ZVOL can be accessed by multiple machines, >> we would need the ZVOL to support SCSI-3 PGR >> for Fencing and Quorum. The one-pager does not >> mention whether this project will support >> SCSI-3 PGR operations from multiple nodes >> concurrently. Please explicitly identify >> whether this will be supported. >> >> Naturally, we would like to see SCSI-3 PGR >> operations on a ZVOL supported for >> multiple nodes concurrently. > > SCSI-3 PGR command are supported with iSCSI Target Daemon LUNs, and > soon all SCSI Targets in COMSTAR. Of course COMSTAR is storage node, > and there is no support for a COMSTAR storage node to also be a Sun > Cluster node. I'm a bit confused by this statement. Is there some issue you see here that would preclude a cluster node from being a COMSTAR storage node? This is, in fact, the config that we use for Solaris iscsi Target currently in our Colorado 2 node shared-nothing config. Thanks, Dan > > Jim Dunham > Engineering Manager > Sun Microsystems, Inc. From Torrey.McMahon@sun.com Thu Mar 12 12:34:48 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CJYmod012436 for ; Thu, 12 Mar 2009 12:34:48 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n2CJYf97038021; Thu, 12 Mar 2009 13:34:47 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KGE006BBRPYOZ00@brm-avmta-1.central.sun.com>; Thu, 12 Mar 2009 13:34:46 -0600 (MDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KGE00I7SRPRYEC0@brm-avmta-1.central.sun.com>; Thu, 12 Mar 2009 13:34:39 -0600 (MDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n2CJYdi2009348; Thu, 12 Mar 2009 19:34:39 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KGE00C00NGFMK00@mail-amer.sun.com>; Thu, 12 Mar 2009 13:34:39 -0600 (MDT) Received: from [192.168.0.135] ([unknown] [69.143.10.149]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KGE005CIRPLO6E0@mail-amer.sun.com>; Thu, 12 Mar 2009 13:34:34 -0600 (MDT) Date: Thu, 12 Mar 2009 16:34:34 -0400 From: Torrey McMahon Subject: Re: iSCSI Target PGR directory for ZVOLs [PSARC/2009/168 Self Review] In-reply-to: <890C829A-A51E-4C24-B5FB-6595FBDF36BC@Sun.COM> Sender: Torrey.McMahon@sun.com To: Jim Dunham Cc: Ellard.Roush@sun.com, PSARC-ext@sun.com, sc-haci , Andrew Hisgen , colorado-iteam@sun.com Message-id: <49B971DA.3030608@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <49B92A3D.4050101@sun.com> <890C829A-A51E-4C24-B5FB-6595FBDF36BC@Sun.COM> User-Agent: Thunderbird 2.0.0.18 (X11/20090203) Status: RO Content-Length: 279 Jim Dunham wrote: > > > There will be an forthcoming iscsitgt man page describing this change, > and how to make the SCSI-3 PGR data follow the ZFS Storage Pool, and > thus any ZVOLs created out of this storage pool. Is there a reason this shouldn't be the default behavior? From James.Dunham@sun.com Thu Mar 12 12:44:49 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CJinGR012612 for ; Thu, 12 Mar 2009 12:44:49 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n2CJinbZ006518; Thu, 12 Mar 2009 12:44:49 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KGE00741S6OSZ00@brm-avmta-1.central.sun.com>; Thu, 12 Mar 2009 13:44:48 -0600 (MDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KGE00I5WS6IYAC0@brm-avmta-1.central.sun.com>; Thu, 12 Mar 2009 13:44:42 -0600 (MDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n2CJig9p014401; Thu, 12 Mar 2009 19:44:42 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KGE00C00NGFMK00@mail-amer.sun.com>; Thu, 12 Mar 2009 13:44:42 -0600 (MDT) Received: from dhcp-ubrm05-50-13.Central.Sun.COM ([unknown] [129.147.50.13]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KGE005BBS5YO6G0@mail-amer.sun.com>; Thu, 12 Mar 2009 13:44:22 -0600 (MDT) Date: Thu, 12 Mar 2009 13:44:11 -0600 From: Jim Dunham Subject: Re: iSCSI Target PGR directory for ZVOLs [PSARC/2009/168 Self Review] In-reply-to: <49B971DA.3030608@sun.com> Sender: James.Dunham@sun.com To: Torrey McMahon Cc: Ellard.Roush@sun.com, PSARC-ext@sun.com, sc-haci , Andrew Hisgen , colorado-iteam@sun.com Message-id: <8AA80BA2-03AE-43B3-BE00-24458C9D71FA@Sun.COM> MIME-version: 1.0 X-Mailer: Apple Mail (2.930.3) Content-type: text/plain; delsp=yes; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <49B92A3D.4050101@sun.com> <890C829A-A51E-4C24-B5FB-6595FBDF36BC@Sun.COM> <49B971DA.3030608@sun.com> Status: RO Content-Length: 753 Torrey, > Jim Dunham wrote: >> >> >> There will be an forthcoming iscsitgt man page describing this >> change, and how to make the SCSI-3 PGR data follow the ZFS Storage >> Pool, and thus any ZVOLs created out of this storage pool. > > Is there a reason this shouldn't be the default behavior? Within a ZFS storage pool, there is no guarantee that there is ZFS filesystem space available, along with the ZVOLs, or that the ZFS filesystem space is even mounted by default. Yes there is some correlation between /dev/zvol/rdsk// and //, but various ZFS mount options can override this default behavior. Jim Dunham Engineering Manager Open Storage Systems Sun Microsystems, Inc. work: 781-442-4042, x24042 From Ellard.Roush@sun.com Thu Mar 12 12:51:47 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CJplaJ012971 for ; Thu, 12 Mar 2009 12:51:47 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n2CJpguM011364; Thu, 12 Mar 2009 12:51:46 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KGE00K0VSIA7C00@nwk-avmta-2.sfbay.sun.com>; Thu, 12 Mar 2009 12:51:46 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KGE00EJ7SI6NUE0@nwk-avmta-2.sfbay.sun.com>; Thu, 12 Mar 2009 12:51:42 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n2CJpg7s010980; Thu, 12 Mar 2009 12:51:42 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KGE00100RKZKE00@fe-sfbay-10.sun.com>; Thu, 12 Mar 2009 12:51:42 -0700 (PDT) Received: from [129.146.106.191] ([unknown] [129.146.106.191]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KGE00N2FSHY50E0@fe-sfbay-10.sun.com>; Thu, 12 Mar 2009 12:51:34 -0700 (PDT) Date: Thu, 12 Mar 2009 11:51:34 -0800 From: Ellard Roush Subject: Re: iSCSI Target PGR directory for ZVOLs [PSARC/2009/168 Self Review] In-reply-to: <49B971DA.3030608@sun.com> Sender: Ellard.Roush@sun.com To: Torrey McMahon Cc: Jim Dunham , PSARC-ext@sun.com, sc-haci , Andrew Hisgen , colorado-iteam@sun.com Reply-to: Ellard.Roush@sun.com Message-id: <49B967C6.6040200@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <49B92A3D.4050101@sun.com> <890C829A-A51E-4C24-B5FB-6595FBDF36BC@Sun.COM> <49B971DA.3030608@sun.com> User-Agent: Thunderbird 2.0.0.16 (X11/20080807) Status: RO Content-Length: 806 Hi, The statement below raises a question. Is the SCSI-3 PGR data associated with the ZFS storage pool or a specific ZVOL in that zpool ? A ZFS storage pool can support multiple ZVOLs. The administrator can only import at the granularity of the zpool. So it is not possible for ZVOLs to be imported to different machines. Still that leaves the design question open as to whether the SCSI-3 PGR data will be associated with a zpool or individual ZVOLs. Regards, Ellard On 03/12/09 12:34, Torrey McMahon wrote: > Jim Dunham wrote: >> >> >> There will be an forthcoming iscsitgt man page describing this change, >> and how to make the SCSI-3 PGR data follow the ZFS Storage Pool, and >> thus any ZVOLs created out of this storage pool. > > Is there a reason this shouldn't be the default behavior? From Torrey.McMahon@sun.com Thu Mar 12 14:08:15 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CL8FDO018877 for ; Thu, 12 Mar 2009 14:08:15 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2CL8Cb1011734; Thu, 12 Mar 2009 21:08:14 GMT Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KGE00013W1PNO00@nwk-avmta-2.sfbay.sun.com>; Thu, 12 Mar 2009 14:08:13 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KGE00KXLW1OMM80@nwk-avmta-2.sfbay.sun.com>; Thu, 12 Mar 2009 14:08:12 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n2CL8CQW005792; Thu, 12 Mar 2009 21:08:12 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KGE00G00VXO2H00@mail-amer.sun.com>; Thu, 12 Mar 2009 15:08:12 -0600 (MDT) Received: from [192.168.0.198] ([unknown] [69.143.10.149]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KGE00BFDW1N1PH0@mail-amer.sun.com>; Thu, 12 Mar 2009 15:08:11 -0600 (MDT) Date: Thu, 12 Mar 2009 17:08:10 -0400 From: Torrey McMahon Subject: Re: iSCSI Target PGR directory for ZVOLs [PSARC/2009/168 Self Review] In-reply-to: <8AA80BA2-03AE-43B3-BE00-24458C9D71FA@Sun.COM> Sender: Torrey.McMahon@sun.com To: Jim Dunham Cc: Ellard.Roush@sun.com, PSARC-ext@sun.com, sc-haci , Andrew Hisgen , colorado-iteam@sun.com Message-id: <49B979BA.6040600@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <49B92A3D.4050101@sun.com> <890C829A-A51E-4C24-B5FB-6595FBDF36BC@Sun.COM> <49B971DA.3030608@sun.com> <8AA80BA2-03AE-43B3-BE00-24458C9D71FA@Sun.COM> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1b4pre) Gecko/20090312 Lightning/1.0pre Shredder/3.0b3pre Status: RO Content-Length: 883 On 3/12/2009 3:44 PM, Jim Dunham wrote: > Torrey, > >> Jim Dunham wrote: >>> >>> >>> There will be an forthcoming iscsitgt man page describing this >>> change, and how to make the SCSI-3 PGR data follow the ZFS Storage >>> Pool, and thus any ZVOLs created out of this storage pool. >> >> Is there a reason this shouldn't be the default behavior? > > Within a ZFS storage pool, there is no guarantee that there is ZFS > filesystem space available, along with the ZVOLs, or that the ZFS > filesystem space is even mounted by default. > > Yes there is some correlation between > /dev/zvol/rdsk// and //, but various > ZFS mount options can override this default behavior. I'm just trying to avoid the calls to Service with "We didn't know we had to do this. Why isn't this automatic like every other storage array on the planet?" type calls. From James.Dunham@sun.com Fri Mar 13 10:57:13 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2DHvCso025254 for ; Fri, 13 Mar 2009 10:57:13 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2DHvBln027211; Fri, 13 Mar 2009 17:57:12 GMT Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KGG00K03HVBWA00@brm-avmta-1.central.sun.com>; Fri, 13 Mar 2009 11:57:11 -0600 (MDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KGG00EG2HVBWK70@brm-avmta-1.central.sun.com>; Fri, 13 Mar 2009 11:57:11 -0600 (MDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n2DHvAKc007745; Fri, 13 Mar 2009 17:57:10 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KGG00400HMP6700@mail-amer.sun.com>; Fri, 13 Mar 2009 11:57:10 -0600 (MDT) Received: from [192.168.1.100] ([unknown] [129.150.67.57]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KGG0098XHV2GKD0@mail-amer.sun.com>; Fri, 13 Mar 2009 11:57:04 -0600 (MDT) Date: Fri, 13 Mar 2009 13:56:57 -0400 From: Jim Dunham Subject: Re: iSCSI Target PGR directory for ZVOLs [PSARC/2009/168 Self Review] In-reply-to: <49B967C6.6040200@sun.com> Sender: James.Dunham@sun.com To: Ellard.Roush@sun.com Cc: Torrey McMahon , PSARC-ext@sun.com, sc-haci , Andrew Hisgen , colorado-iteam@sun.com Message-id: MIME-version: 1.0 X-Mailer: Apple Mail (2.930.3) Content-type: text/plain; delsp=yes; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <49B92A3D.4050101@sun.com> <890C829A-A51E-4C24-B5FB-6595FBDF36BC@Sun.COM> <49B971DA.3030608@sun.com> <49B967C6.6040200@sun.com> Status: RO Content-Length: 1004 Ellard, > The statement below raises a question. > > Is the SCSI-3 PGR data associated with the > ZFS storage pool or a specific ZVOL in that zpool ? SCSI-3 PGR data, if it exists, is associated with a specific ZVOL. > A ZFS storage pool can support multiple ZVOLs. If there are multiple ZVOLs, there are multiple, unique sets of SCSI-3 PGR data. > The administrator can only import at the > granularity of the zpool. So it is not possible > for ZVOLs to be imported to different machines. > > Still that leaves the design question open as to > whether the SCSI-3 PGR data will be associated > with a zpool or individual ZVOLs. > > Regards, > Ellard > > > On 03/12/09 12:34, Torrey McMahon wrote: >> Jim Dunham wrote: >>> >>> >>> There will be an forthcoming iscsitgt man page describing this >>> change, and how to make the SCSI-3 PGR data follow the ZFS Storage >>> Pool, and thus any ZVOLs created out of this storage pool. >> Is there a reason this shouldn't be the default behavior? >