From sacadmin Thu Oct 18 17:52:58 2007 Received: from sfbaymail2sca.sfbay.sun.com (sfbaymail2sca [129.145.155.42]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l9J0qwH8000050 for ; Thu, 18 Oct 2007 17:52:58 -0700 (PDT) Received: from sca-es-mail-1.sun.com (sca-es-mail-1.Sun.COM [192.18.43.132]) by sfbaymail2sca.sfbay.sun.com (8.13.6+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id l9J0nYhE003690 for ; Thu, 18 Oct 2007 17:49:34 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l9J0nTjX016412 for ; Thu, 18 Oct 2007 17:49:29 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JQ400301VLVC500@fe-sfbay-10.sun.com> (original mail from John.Plocher@Sun.COM) for psarc-record@sac.sfbay.sun.com; Thu, 18 Oct 2007 17:49:29 -0700 (PDT) Received: from wp668.local ([208.74.177.212]) by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JQ4002FKVMGEY50@fe-sfbay-10.sun.com> for psarc-record@sac.sfbay.sun.com; Thu, 18 Oct 2007 17:49:28 -0700 (PDT) Date: Thu, 18 Oct 2007 17:49:25 -0700 From: John Plocher Subject: [Fwd: ZFS cachefile property [PSARC/2007/607 FastTrack timeout 10/24/2007]] Sender: John.Plocher@Sun.COM To: psarc-record@sac.sfbay.sun.com Message-id: <4717FF15.7090208@Sun.Com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) Status: RO Content-Length: 17086 -------- Original Message -------- Subject: ZFS cachefile property [PSARC/2007/607 FastTrack timeout 10/24/2007] Date: Thu, 18 Oct 2007 15:57:15 -0700 (PDT) From: Matthew Ahrens To: PSARC-ext@sun.com CC: zfs-team@sun.com Template Version: @(#)sac_nextcase 1.64 07/13/07 SMI This information is Copyright 2007 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: ZFS cachefile property 1.2. Name of Document Author/Supplier: Author: Eric Schrock 1.3 Date of This Document: 18 October, 2007 4. Technical Description This case adds a new pool property to ZFS to control the location of cached configuration data, which also supercedes the previously added 'temporary' property. The interfaces are committed, and the release binding is patch. A. INTRODUCTION When a system running ZFS is booted, the set of available pools and their configuration must be available. While it would be possible to dynamically discover this by iterating over the set of available disks, this is a painfully slow operation, and would incur unacceptable boot time overhead. To avoid this situation, ZFS uses a cache file, /etc/zfs/zpool.cache, to store a list of pools that were active on the system before the last reboot. For normal systems, this is sufficient to allow for rapid booting. In a clustered configuration, however, higher level software needs to be in control of when and how these pools are imported. The 'temporary' pool property (and before that, alternate root pools) is provided as a means to keep pools out of the default cache file, allowing cluster software to import any pools when necessary. Howeer, this software suffers from the same slow discovery process that the original cache file was designed to circumvent. To get around this, cluster software must poll the current pool configuration and keep track of any state in a private configuration file. In addition, install software must set up the /etc/zfs/zpool.cache file in an alternate boot environment. It is possible to copy over the cache file once the pool is configured, but install software also needs to make use of the alternate root property, which prevents the pool from being stored in the cache file. As part of PSARC 2007/342, a new 'temporary' was introduced that allowed the user to specify that the pool should not be cached at all. This provides some flexibility, but not enough to implement a complete clustering solution. It is equivalent to setting the cachefile to 'none'. B. DESCRIPTION This case introduces a new pool property 'cachefile' to control where the configuration for a particular pool is stored. This is a non-persistent property, and is not stored on disk. Multiple pools can share the same cachefile. The cachefile property can be set like any other pool property, changing it will cause the pool to be removed from the previous cache file and added to the new one. Setting it to the empty string (the default) will use the default location (/etc/zfs/zpool.cache). # zpool create -o cachefile=/var/cluster/zpool.cache pool ... # zpool get cachefile pool NAME PROPERTY VALUE SOURCE pool cachefile /var/cluster/zpool.cache local # zpool set cachefile= pool # zpool get cachefile pool NAME PROPERTY VALUE SOURCE pool cachefile - default The system will continue to use only the default cache file for loading pools at boot, so consumers using an alternate cachefile will be responsible importing the pool when appropriate. A new option, '-c', is added to 'zpool import' to read the configuration data from a cache file instead of searching available devices. This option controls only the source of configuration information. To import to a location other than the default cachefile, '-o cachefile' must be used. # zpool import -c /var/cluster/zpool.cache pool: pool id: 15451357997522795478 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: pool ONLINE mirror ONLINE c1t2d0 ONLINE c1t3d0 ONLINE # zpool import -c /var/cluster/zpool.cache pool An enumeration of all combinations of '-c' and '-o' yields the following table: SOURCE CACHEFILE COMMAND device search default zpool import device search file zpool import -o cachefile=file file default zpool import -c file file file2 zpool import -c file -o cachefile=file2 The 'cachefile' property can also be set to the special value 'none' that indicates that the pool should not be cached at all. This is equivalent to the existing 'temporary' property. Since this functionality supercedes that of the 'temporary' property, and to reduce potential conflicts, the 'temporary' property (which only exists in Nevada) is removed as part of this case in favor of 'cachefile=none'. To maintain compatibility with the existing '-R' functionality, 'altroot' will imply 'cachefile=none' unless the user specifies otherwise. The file is written by creating a temporary file in the same directory (.) and doing an atomic rename operation. When the last pool using a particular cachefile is exported or destroyed, the cachefile is removed. C. MANPAGE DIFFS *** zpool.1m Tue Oct 16 13:08:22 2007 --- zpool.new Wed Oct 17 10:41:29 2007 *************** *** 69,84 **** System Administration Commands zpool(1M) ! zpool import [-o mntopts] [-p property=value] ... [-d dir] [-D] [-f] ! [-R root] -a ! zpool import [-o mntopts] [-o property=value] ... [-d dir] [-D] [-f] ! [-R root] pool | id [newpool] zpool export [-f] pool ... --- 69,84 ---- System Administration Commands zpool(1M) ! zpool import [-o mntopts] [-p property=value] ... ! [-d dir | -c cachefile] [-D] [-f] [-R root] -a ! zpool import [-o mntopts] [-o property=value] ... ! [-d dir | -c cachefile] [-D] [-f] [-R root] pool | id [newpool] zpool export [-f] pool ... *************** *** 496,524 **** prepended to any mount points within the pool. This can be used when examining an unknown pool where the mount points cannot be trusted, or in an alternate boot environment, where the typical paths are not valid. altroot is not a persistent property. It is valid only ! while the system is up. Setting altroot implies setting ! temporary=on. - temporary - - By default, all pools are persistent and are automati- - cally opened when the system is rebooted. Setting this - boolean property to "on" causes the pool to exist only - while the system is up. If the system is rebooted, the - pool has to be manually imported by using the "zpool - import" command. Setting this property is often useful - when using pools on removable media, where the devices - may not be present when the system reboots. This pro- - perty can also be referred to by its shortened column - name, "temp". - - - The following properties can be set at creation time and import time, and later changed with the "zpool set" command: --- 496,510 ---- prepended to any mount points within the pool. This can be used when examining an unknown pool where the mount points cannot be trusted, or in an alternate boot environment, where the typical paths are not valid. altroot is not a persistent property. It is valid only ! while the system is up. Setting altroot defaults to ! using cachefile=none, though this may be overridden ! using an explicit setting. The following properties can be set at creation time and import time, and later changed with the "zpool set" command: *************** *** 550,559 **** --- 536,567 ---- Identifies the default bootable dataset for the root pool. This property is expected to be set mainly by the installation and upgrade programs. + cachefile=path | 'none' + + Controls the location of where pool configuration is + cached. Discovering all pools on system startup + requires a cached copy of the configuration data that + is stored on the root filesystem. All pools in this + cache are automatically imported when the system boots. + For some environments (typically install and clustering), + it is desirable to cache this information in a different + location so pools are not automatically imported. + Setting this property will cache the pool configuration + in a different location that can later be imported with + 'zpool import -c'. Setting it to the special value + 'none' will create a temporary pool that is never cached, + and the special value '' will use the default location. + Multiple pools can share the same cache file, and the + file is written by creating a file '.' in + same directory and doing a rename. When the last pool + using a cachefile is exported or destroyed, the file + will be removed. + + delegation=on | off Controls whether a non-privileged user is granted access based on the dataset permissions defined on the dataset. See zfs(1M) for more information on ZFS delegated *************** *** 668,678 **** set. -R root ! Equivalent to "-o temporary=on,altroot=root" -m mountpoint Sets the mount point for the root dataset. The --- 676,686 ---- set. -R root ! Equivalent to "-o cachefile=none,altroot=root" -m mountpoint Sets the mount point for the root dataset. The *************** *** 942,952 **** -s Stop scrubbing. ! zpool import [-d dir] [-D] Lists pools available to import. If the -d option is not specified, this command searches for devices in "/dev/dsk". The -d option can be specified multiple times, and all directories are searched. If the device --- 950,960 ---- -s Stop scrubbing. ! zpool import [-d dir | -c cachefile] [-D] Lists pools available to import. If the -d option is not specified, this command searches for devices in "/dev/dsk". The -d option can be specified multiple times, and all directories are searched. If the device *************** *** 960,979 **** The numeric identifier is unique, and can be used instead of the pool name when multiple exported pools of the same name are available. -d dir Searches for devices or files in dir. The -d ! option can be specified multiple times. -D Lists destroyed pools only. ! zpool import [-o mntopts] [ -o property=value] ... [-d dir] ! [-D] [-f] [-R root] -a Imports all pools found in the search directories. Identical to the previous command, except that all pools with a sufficient number of devices available are imported. Destroyed pools, pools that were previously --- 968,994 ---- The numeric identifier is unique, and can be used instead of the pool name when multiple exported pools of the same name are available. + -c file Reads configuration from the given cache file, + as created by the 'cachefile' pool property. + This cachefile is used instead of searching + for devices. + + -d dir Searches for devices or files in dir. The -d ! option can be specified multiple times, and ! is incompatible with '-c'. -D Lists destroyed pools only. ! zpool import [-o mntopts] [ -o property=value] ... ! [-d dir | -c cachefile] [-D] [-f] [-R root] -a Imports all pools found in the search directories. Identical to the previous command, except that all pools with a sufficient number of devices available are imported. Destroyed pools, pools that were previously *************** *** 1008,1020 **** include. See the "Properties" sec- tion for more information on the available pool properties. -d dir Searches for devices or files in dir. The -d option can be specified ! multiple times. -D Imports destroyed pools only. The -f option is also required. --- 1023,1043 ---- include. See the "Properties" sec- tion for more information on the available pool properties. + -c cachefile Reads configuration from the given + cache file, as created by the + 'cachefile' pool property. This + cachefile is used instead of searching + for devices. + + -d dir Searches for devices or files in dir. The -d option can be specified ! multiple times. This option is ! incompatible with '-c'. -D Imports destroyed pools only. The -f option is also required. *************** *** 1025,1042 **** -a Searches for and imports all pools found. ! -R root Sets the "temporary" property to ! "on" and the "altroot" property to "root". ! zpool import [-o mntopts] [ -o property=value] ... [-d dir] ! [-D] [-f] [-R root] pool | id [newpool] Imports a specific pool. A pool can be identified by its name or the numeric identifier. If newpool is specified, the pool is imported using the name newpool. Otherwise, it is imported with the same name as its exported name. --- 1048,1065 ---- -a Searches for and imports all pools found. ! -R root Sets the "cachefile" property to ! "none" and the "altroot" property to "root". ! zpool import [-o mntopts] [ -o property=value] ... ! [-d dir | -c cachefile] [-D] [-f] [-R root] pool | id [newpool] Imports a specific pool. A pool can be identified by its name or the numeric identifier. If newpool is specified, the pool is imported using the name newpool. Otherwise, it is imported with the same name as its exported name. *************** *** 1074,1087 **** These properties include. See the "Properties" sec- tion for more information on the available pool pro- perties. -d dir Searches for devices or files in dir. The -d option ! can be specified multiple times. -D Imports destroyed pool. The -f option is also --- 1097,1117 ---- These properties include. See the "Properties" sec- tion for more information on the available pool pro- perties. + -c cachefile + + Reads configuration from the given cache file, as + created by the 'cachefile' pool property. This + cachefile is used instead of searching for devices. + -d dir Searches for devices or files in dir. The -d option ! can be specified multiple times. This option is ! incompatible with '-c'. -D Imports destroyed pool. The -f option is also *************** *** 1094,1104 **** tially active. -R root ! Sets the "temporary" property to "on" and the "altroot" property to "root". zpool export [-f] pool ... --- 1124,1134 ---- tially active. -R root ! Sets the "cachefile" property to "none" and the "altroot" property to "root". zpool export [-f] pool ... 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From Darren.Moffat@sun.com Fri Oct 19 03:20:26 2007 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l9JAKPdd008367 for ; Fri, 19 Oct 2007 03:20:26 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l9JAGwPc012821; Fri, 19 Oct 2007 11:17:01 +0100 (BST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JQ500703LWBPW00@brm-avmta-1.central.sun.com>; Fri, 19 Oct 2007 04:16:59 -0600 (MDT) Received: from gmp-eb-mail-1.sun.com ([192.18.6.21]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JQ500KW1LWA90D0@brm-avmta-1.central.sun.com>; Fri, 19 Oct 2007 04:16:59 -0600 (MDT) Received: from fe-emea-09.sun.com (gmp-eb-lb-2-fe3.eu.sun.com [192.18.6.12]) by gmp-eb-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l9JAGvN6017909; Fri, 19 Oct 2007 10:16:57 +0000 (GMT) Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JQ500M01KQCIH00@fe-emea-09.sun.com> (original mail from Darren.Moffat@Sun.COM); Fri, 19 Oct 2007 11:16:55 +0100 (BST) Received: from [129.156.173.21] by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JQ5002ZYLVX9M00@fe-emea-09.sun.com>; Fri, 19 Oct 2007 11:16:46 +0100 (BST) Date: Fri, 19 Oct 2007 11:16:45 +0100 From: Darren J Moffat Subject: Re: ZFS cachefile property [PSARC/2007/607 FastTrack timeout 10/24/2007] In-reply-to: <200710182257.l9IMvFkK028179@localhost.sfbay.sun.com> Sender: Darren.Moffat@sun.com To: Matthew Ahrens Cc: PSARC-ext@sun.com, zfs-team@sun.com Message-id: <4718840D.3010601@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200710182257.l9IMvFkK028179@localhost.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.6 (X11/20070924) Status: RO Content-Length: 317 What is the behaviour of zpool import with -c if the file doesn't exist or isn't a valid cache file (eg zpool import -c /dev/zero) I think the man page should actually say what the default cachefile location actually is, it isn't any old file in the root filesystem but a very specific one. -- Darren J Moffat From eschrock@zion.sfbay.sun.com Fri Oct 19 09:37:41 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l9JGbeE4014095 for ; Fri, 19 Oct 2007 09:37:40 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l9JGY0RT022256; Sat, 20 Oct 2007 00:34:14 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JQ600D093D1M900@nwk-avmta-1.sfbay.Sun.COM>; Fri, 19 Oct 2007 09:34:13 -0700 (PDT) Received: from zion.sfbay.sun.com ([129.146.17.75]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JQ6007IC3D0ZUE0@nwk-avmta-1.sfbay.Sun.COM>; Fri, 19 Oct 2007 09:34:12 -0700 (PDT) Received: from zion.sfbay.sun.com (localhost [127.0.0.1]) by zion.sfbay.sun.com (8.14.1+Sun/8.14.1) with ESMTP id l9JGYC2x307367; Fri, 19 Oct 2007 16:34:12 +0000 (GMT) Received: (from eschrock@localhost) by zion.sfbay.sun.com (8.14.1+Sun/8.14.1/Submit) id l9JGYCJX307366; Fri, 19 Oct 2007 09:34:12 -0700 (PDT) Date: Fri, 19 Oct 2007 09:34:11 -0700 From: Eric Schrock Subject: Re: ZFS cachefile property [PSARC/2007/607 FastTrack timeout 10/24/2007] In-reply-to: <4718840D.3010601@Sun.COM> To: Darren J Moffat Cc: Matthew Ahrens , PSARC-ext@Sun.COM, zfs-team@Sun.COM Message-id: <20071019163411.GE306619@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.2.0.264296 References: <200710182257.l9IMvFkK028179@localhost.sfbay.sun.com> <4718840D.3010601@Sun.COM> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 742 On Fri, Oct 19, 2007 at 11:16:45AM +0100, Darren J Moffat wrote: > What is the behaviour of zpool import with -c if the file doesn't exist > or isn't a valid cache file (eg zpool import -c /dev/zero) It returns an error. I will clarify that in the manpages. > I think the man page should actually say what the default cachefile > location actually is, it isn't any old file in the root filesystem but a > very specific one. We have avoided doing this because we do not want the actual location (/etc/zfs/zpool.cache) to become a supported interface. That being said, it may be worth documenting it with the caveat that the location is unstable. - Eric -- Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock From sommerfeld@sun.com Fri Oct 19 09:52:26 2007 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l9JGqPe7014331 for ; Fri, 19 Oct 2007 09:52:26 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l9JGmr9m022737; Fri, 19 Oct 2007 17:49:01 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JQ60040541P5E00@nwk-avmta-2.sfbay.sun.com>; Fri, 19 Oct 2007 09:49:01 -0700 (PDT) Received: from localhost.east.sun.com ([129.148.19.3]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JQ6002GQ41NQAB0@nwk-avmta-2.sfbay.sun.com>; Fri, 19 Oct 2007 09:49:00 -0700 (PDT) Received: from localhost.east.sun.com (localhost [127.0.0.1]) by localhost.east.sun.com (8.14.1+Sun/8.14.1) with ESMTP id l9JGmkV0002424; Fri, 19 Oct 2007 12:48:46 -0400 (EDT) Received: (from sommerfeld@localhost) by localhost.east.sun.com (8.14.1+Sun/8.14.1/Submit) id l9JGmjIv002423; Fri, 19 Oct 2007 12:48:45 -0400 (EDT) Date: Fri, 19 Oct 2007 12:48:45 -0400 From: Bill Sommerfeld Subject: Re: ZFS cachefile property [PSARC/2007/607 FastTrack timeout 10/24/2007] In-reply-to: <200710182257.l9IMvFkK028179@localhost.sfbay.sun.com> To: Matthew Ahrens Cc: PSARC-ext@sun.com, zfs-team@sun.com Message-id: <1192812525.1748.27.camel@localhost> MIME-version: 1.0 X-Mailer: Evolution 2.12.0 Content-type: text/plain Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200710182257.l9IMvFkK028179@localhost.sfbay.sun.com> X-Authentication-warning: localhost.east.sun.com: sommerfeld set sender to sommerfeld@sun.com using -f Status: RO Content-Length: 1083 a couple use-case questions: 1) are the file contents as an opaque binary blob intended to be usefully mobile between systems which share access to the pool storage? in other words, would it be meaningful for clustering infrastructure to replicate a cachefile between nodes which may import a pool, or is the expected usage of this feature that each node will maintain caches of the cluster-managed pools it has imported in the past and may import again in the future? 2) in disaster recovery situations, I've been instructed to do something like: boot -m milestone=none ... (mount root writeable) ... mv /etc/zfs/zpool.cache /etc/zfs/zpool.cache.old ... as a mechanism to boot without opening the pool. I think this means that the filename of /etc/zfs/zpool.cache is turning into a de-facto administrative interface because there is no more-stable interface available to "export" a potentially-toxic pool. Does that mean I can check to see that I have all disks making up the pools I would have opened on boot using: zpool import -c /etc/zfs/zpool.cache.old ? > From eschrock@zion.sfbay.sun.com Fri Oct 19 10:00:47 2007 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l9JH0ljb015031 for ; Fri, 19 Oct 2007 10:00:47 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l9JGvORV007427; Fri, 19 Oct 2007 09:57:24 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JQ60040B4FOKR00@nwk-avmta-2.sfbay.sun.com>; Fri, 19 Oct 2007 09:57:24 -0700 (PDT) Received: from zion.sfbay.sun.com ([129.146.17.75]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JQ6002844FNQAF0@nwk-avmta-2.sfbay.sun.com>; Fri, 19 Oct 2007 09:57:23 -0700 (PDT) Received: from zion.sfbay.sun.com (localhost [127.0.0.1]) by zion.sfbay.sun.com (8.14.1+Sun/8.14.1) with ESMTP id l9JGvN54307994; Fri, 19 Oct 2007 16:57:23 +0000 (GMT) Received: (from eschrock@localhost) by zion.sfbay.sun.com (8.14.1+Sun/8.14.1/Submit) id l9JGvN3w307993; Fri, 19 Oct 2007 09:57:23 -0700 (PDT) Date: Fri, 19 Oct 2007 09:57:23 -0700 From: Eric Schrock Subject: Re: ZFS cachefile property [PSARC/2007/607 FastTrack timeout 10/24/2007] In-reply-to: <1192812525.1748.27.camel@localhost> To: Bill Sommerfeld Cc: Matthew Ahrens , PSARC-ext@sun.com, zfs-team@sun.com Message-id: <20071019165723.GI306619@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.2.0.264296 References: <200710182257.l9IMvFkK028179@localhost.sfbay.sun.com> <1192812525.1748.27.camel@localhost> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 1898 On Fri, Oct 19, 2007 at 12:48:45PM -0400, Bill Sommerfeld wrote: > a couple use-case questions: > > 1) are the file contents as an opaque binary blob intended to be > usefully mobile between systems which share access to the pool storage? Yes. > in other words, would it be meaningful for clustering infrastructure to > replicate a cachefile between nodes which may import a pool, or is the > expected usage of this feature that each node will maintain caches of > the cluster-managed pools it has imported in the past and may import > again in the future? The intent is to replicate this cachefile to shared storage in some cluster-specific way. I'm not sure how the second scenario would work, but it definitely would be possible. > 2) in disaster recovery situations, I've been instructed to do > something like: > > boot -m milestone=none > > ... (mount root writeable) ... > > mv /etc/zfs/zpool.cache /etc/zfs/zpool.cache.old > ... > as a mechanism to boot without opening the pool. > > I think this means that the filename of /etc/zfs/zpool.cache is turning > into a de-facto administrative interface because there is no more-stable > interface available to "export" a potentially-toxic pool. Yes, but this won't work in a ZFS root world. Obviously the right answer is to not panic on toxic pools, but we will need some non-trivial infrastructure to make this operation possible in the brave new world of ZFS boot. I'm fine with it being a "de-facto adminstrative interface", as long it's not a "committed interface". > Does that mean I can check to see that I have all disks making up the > pools I would have opened on boot using: > > zpool import -c /etc/zfs/zpool.cache.old Yes, you can do this. Depending on the toxicity, this may or may not trigger the original problem ;-) - Eric -- Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock From sommerfeld@sun.com Fri Oct 19 11:46:50 2007 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l9JIkoep017457 for ; Fri, 19 Oct 2007 11:46:50 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l9JIhNo4010949; Fri, 19 Oct 2007 19:43:25 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JQ600A0H9CAN300@nwk-avmta-2.sfbay.sun.com>; Fri, 19 Oct 2007 11:43:22 -0700 (PDT) Received: from dm-east-01.east.sun.com ([129.148.9.192]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JQ6004PN9CALV60@nwk-avmta-2.sfbay.sun.com>; Fri, 19 Oct 2007 11:43:22 -0700 (PDT) Received: from thunk.east.sun.com (thunk.East.Sun.COM [129.148.174.66]) by dm-east-01.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id l9JIhLJg014558; Fri, 19 Oct 2007 14:43:21 -0400 (EDT) Received: from [IPv6:::1] (localhost [IPv6:::1]) by thunk.east.sun.com (8.14.1+Sun/8.14.1) with ESMTP id l9JIhLaw019794; Fri, 19 Oct 2007 14:43:21 -0400 (EDT) Date: Fri, 19 Oct 2007 14:43:20 -0400 From: Bill Sommerfeld Subject: Re: ZFS cachefile property [PSARC/2007/607 FastTrack timeout 10/24/2007] In-reply-to: <20071019165723.GI306619@eng.sun.com> To: Eric Schrock Cc: Matthew Ahrens , PSARC-ext@sun.com, zfs-team@sun.com Message-id: <1192819400.19556.24.camel@thunk> MIME-version: 1.0 X-Mailer: Evolution 2.12.0 Content-type: text/plain Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200710182257.l9IMvFkK028179@localhost.sfbay.sun.com> <1192812525.1748.27.camel@localhost> <20071019165723.GI306619@eng.sun.com> Status: RO Content-Length: 619 On Fri, 2007-10-19 at 09:57 -0700, Eric Schrock wrote: > Yes, you can do this. Depending on the toxicity, this may or may not > trigger the original problem ;-) so, the implication I get from the proposal is that zpool import -c like zpool import merely reports on what's out there and importable but doesn't actually open the pools, and you'd have to do something like: zpool import -c foo-cache foo-pool to actually import the pool. Is that correct? if so, that answers my question; hardening the former case against toxic crud seems a lot easier than hardening the latter... - Bill From eschrock@zion.sfbay.sun.com Fri Oct 19 13:02:22 2007 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l9JK2M1w019505 for ; Fri, 19 Oct 2007 13:02:22 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l9JJww3a059300; Fri, 19 Oct 2007 13:58:59 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JQ60080JCUAGO00@brm-avmta-1.central.sun.com>; Fri, 19 Oct 2007 13:58:58 -0600 (MDT) Received: from zion.sfbay.sun.com ([129.146.17.75]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JQ6005GTCUABF10@brm-avmta-1.central.sun.com>; Fri, 19 Oct 2007 13:58:58 -0600 (MDT) Received: from zion.sfbay.sun.com (localhost [127.0.0.1]) by zion.sfbay.sun.com (8.14.1+Sun/8.14.1) with ESMTP id l9JJwtiE311803; Fri, 19 Oct 2007 19:58:55 +0000 (GMT) Received: (from eschrock@localhost) by zion.sfbay.sun.com (8.14.1+Sun/8.14.1/Submit) id l9JJwt8f311802; Fri, 19 Oct 2007 12:58:55 -0700 (PDT) Date: Fri, 19 Oct 2007 12:58:54 -0700 From: Eric Schrock Subject: Re: ZFS cachefile property [PSARC/2007/607 FastTrack timeout 10/24/2007] In-reply-to: <1192819400.19556.24.camel@thunk> To: Bill Sommerfeld Cc: Matthew Ahrens , PSARC-ext@sun.com, zfs-team@sun.com Message-id: <20071019195854.GM306619@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.2.0.264296 References: <200710182257.l9IMvFkK028179@localhost.sfbay.sun.com> <1192812525.1748.27.camel@localhost> <20071019165723.GI306619@eng.sun.com> <1192819400.19556.24.camel@thunk> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 978 On Fri, Oct 19, 2007 at 02:43:20PM -0400, Bill Sommerfeld wrote: > > so, the implication I get from the proposal is that > > zpool import -c > > like > > zpool import > > merely reports on what's out there and importable but doesn't actually > open the pools, and you'd have to do something like: > > zpool import -c foo-cache foo-pool > > to actually import the pool. > > Is that correct? if so, that answers my question; hardening the former > case against toxic crud seems a lot easier than hardening the latter... > Sort of. The '-c' option only changes the source of the configuration. The rest of the import command behaves in the same manner. We still do a ZPOOL_IOC_TRYIMPORT to read the true MOS config as well as determine vdev state, so we call spa_load() in either case. This is required to correctly determine whether the pool is importable or not. - Eric -- Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock