From ahrens@sac.sfbay.sun.com Mon May 11 15:28:24 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n4BMSNBr028256 for ; Mon, 11 May 2009 15:28:24 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n4BMSLR4027787; Mon, 11 May 2009 23:28:23 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KJI0090F3R9CR00@nwk-avmta-2.sfbay.sun.com>; Mon, 11 May 2009 15:28:21 -0700 (PDT) Received: from localhost.sfbay.sun.com ([129.146.17.46]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KJI008PW3R9J000@nwk-avmta-2.sfbay.sun.com>; Mon, 11 May 2009 15:28:21 -0700 (PDT) Received: from localhost.sfbay.sun.com (localhost [127.0.0.1] (may be forged)) by localhost.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n4BMSKOR004681; Mon, 11 May 2009 15:28:20 -0700 (PDT) Received: (from ahrens@localhost) by localhost.sfbay.sun.com (8.14.3+Sun/8.14.3/Submit) id n4BMSKhR004677; Mon, 11 May 2009 15:28:20 -0700 (PDT) Date: Mon, 11 May 2009 15:28:20 -0700 (PDT) From: Matthew Ahrens Subject: zfs snapshot holds [PSARC/2009/297 FastTrack timeout 05/20/2009] To: PSARC-ext@sun.com Cc: zfs-team@sun.com Message-id: <200905112228.n4BMSKhR004677@localhost.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 6816 Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: zfs snapshot holds 1.2. Name of Document Author/Supplier: Author: Chris Kirby 1.3 Date of This Document: 11 May, 2009 4. Technical Description ZFS snapshot reference counts A. SUMMARY: This case adds support to ZFS for user-initiated reference counts on snapshots. B. PROBLEM: Remote replication of ZFS datasets can result in different automatic snapshot policies on the two sides of a replication pair. For example, the sending side might want to keep five snapshots at one-minute intervals, while the receiving side might want to keep ten snapshots at one-minute intervals. This can result in the older snapshots being destroyed inadvertently by zfs receive since they no longer exist on the sending side. Also, when an administrator wants to destroy a snapshot for which there are clones, the admin must remember to come back and destroy the snapshot once the last clone has been destroyed. It would be handy to have an automated way of doing this deferred removal. C. PROPOSED SOLUTION C.1. Overview This new facility would permit users or applications to place holds on snapshots that would prevent them from being deleted. Further, this facility would allow a snapshot with clones to be deleted pending the removal of the last clone using the new "zfs destroy -d" option. Each snapshot will have an associated user-reference count, initialized to zero. This count increases by one whenever a hold is taken on the snapshot and decreases by one whenever a hold is released. In the current model, in order for a snapshot to be destroyed with "zfs destroy", it must have no clones. In the new model, the snapshot must also have a zero user-reference count. In the example in section B, the receiving side could place holds on the ten snapshots it wants to keep, which will prevent the zfs receive operation from destroying them. C.2. Version Compatibility Using these features does not require a pool upgrade. Snapshots created on earlier pool versions will be eligible for user reference counts and will be treated as if their initial reference count is zero. C.3. Changes to Existing Subcommands C.3.1 zfs destroy -d The "zfs destroy" subcommand now takes a -d option, applicable only to snapshots. If specified, the snapshot will be destroyed right away if and only if "zfs destroy" without the -d option would have deleted it. i.e. immediate deletion by "zfs destroy -d" requires the following: 1) The snapshot has no clones. AND 2) The user-initiated reference count is zero. Otherwise, the snapshot is marked for deferred deletion, where it will exist as a normal (usable/visible) snapshot until both of the preconditions listed above are met, at which point the snapshot will be destroyed. C.3.2. zfs destroy When a snapshot's last clone is destroyed with "zfs destroy", if the snapshot's user-initiated reference count is zero; and the snapshot was previously marked for deferred destruction with "zfs destroy -d"; the snapshot and the clone will be destroyed as part of a single transaction group. That is, either they will both be destroyed or neither of them will be. If the snapshot has an elevated user reference count or other children, the clone will be destroyed and the snapshot will remain. C.3.3. zfs receive In the current model, receiving a "zfs send -R" (replication) stream will result in a "zfs destroy" on any snapshots that don't exist on the sending side. In the new model, the receiving side will do a "zfs destroy -d" instead. C.4. New Subcommands Three new subcommands are added: "zfs hold", "zfs release", and "zfs holds": C.4.1. zfs hold [-r] ... The zfs hold subcommand adds a single reference, named with the argument, to the given snapshot(s). Each snapshot has its own tag namespace, and tags must be unique within that space. As long as a hold exists on a snapshot, attempts to "zfs destroy" that snapshot will return EBUSY. If the -r option is given, a hold with the given tag will be applied recursively to the snapshots of all descendent file systems. For example, given these snapshots: pool/fs1@snap1 pool/fs1/fs2@snap1 pool/fs2@snap1 This command: # zfs hold -r mytag pool/fs1@snap1 will place a hold named "mytag" on "pool/fs1@snap1" and "pool/fs1/fs2@snap1", but not on "pool/fs2@snap1". The actions taken by -r are atomic; either all of the snapshots will be held with the given tag, or none of them will. C.4.1.1. Delegated Administration The "zfs hold" subcommand requires the new 'hold' delegation permission. C.4.2. zfs release [-r] ... The zfs release subcommand removes a single reference named by the argument from the named snapshot(s). The tag must already exist for each snapshot. The -r option does a recursive release, similar to the recursive hold in section C.4.1. When the last user reference has been removed from a snapshot, the snapshot will be automatically destroyed if and only if: 1) The snapshot has no clones. AND 2) The snapshot has already been marked for deferred deletion with 'zfs destroy -d'. C.4.2.1. Delegated Administration The "zfs release" subcommand requires the new 'release' delegation permission. C.4.3. zfs holds [-r] ... This subcommand lists all of the existing user references for the given snapshot(s) or family of snapshots. For example: # zfs snapshot tank@snap2 # zfs hold com.sun.sometag tank@snap2 # zfs hold mytag tank@snap2 # zfs holds tank@snap2 tank@snap2: TIMESTAMP TAG Mon Apr 27 12:15:49 2009 com.sun.sometag Mon Apr 27 12:16:02 2009 mytag The -r option will list holds on the named snapshots of descendents, in addition to listing holds on the named dataset. If "zfs hold -r com.sun.sometag pool@snap" was used to recursively apply a tag, "zfs holds -r pool@snap" will display those tags. C.4.3.1. Delegated Administration The "zfs holds" subcommand requires the new 'hold' delegation permission. C.5. Tag Naming Convention We will suggest (but not enforce) that users use reverse DNS form for user-reference tagging. e.g. "com.sun.fishworks.replication" Tag names are subject to the same restrictions as snapshot names, i.e. they are limited to alphanumeric characters plus [-_.: ]. We will reserve tag names starting with "." for use by direct callers of the relevant libzfs routines. Stability This case requests patch/micro release binding. The new interfaces are committed. 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From Darren.Moffat@sun.com Tue May 12 01:35:02 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n4C8VIJE029632 for ; Tue, 12 May 2009 01:35:02 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n4C8V69W026014; Tue, 12 May 2009 01:31:17 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KJI00B03VO4RP00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 12 May 2009 01:31:16 -0700 (PDT) Received: from gmp-eb-inf-1.sun.com ([192.18.6.21]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KJI00991VO2X110@nwk-avmta-1.sfbay.Sun.COM>; Tue, 12 May 2009 01:31:14 -0700 (PDT) Received: from fe-emea-09.sun.com (gmp-eb-lb-2-fe3.eu.sun.com [192.18.6.12]) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n4C8VDCt017507; Tue, 12 May 2009 08:31:13 +0000 (GMT) Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) id <0KJI00L00V9JBH00@fe-emea-09.sun.com>; Tue, 12 May 2009 09:31:13 +0100 (BST) Received: from [192.168.1.103] (cpc2-rdng20-2-0-cust917.15-3.cable.virginmedia.com [86.28.167.150]) by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) with ESMTPSA id <0KJI00GNHVNWCO70@fe-emea-09.sun.com>; Tue, 12 May 2009 09:31:08 +0100 (BST) Date: Tue, 12 May 2009 09:31:03 +0100 From: Darren J Moffat Subject: Re: zfs snapshot holds [PSARC/2009/297 FastTrack timeout 05/20/2009] In-reply-to: <200905112228.n4BMSKhR004677@localhost.sfbay.sun.com> Sender: Darren.Moffat@sun.com To: Matthew Ahrens Cc: PSARC-ext@sun.com, zfs-team@sun.com Message-id: <4A0933C7.7080602@Sun.COM> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200905112228.n4BMSKhR004677@localhost.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.18 (X11/20090211) Status: RO Content-Length: 939 Matthew Ahrens wrote: > C.2. Version Compatibility > > Using these features does not require a pool upgrade. > Snapshots created on earlier pool versions will be eligible > for user reference counts and will be treated as if their > initial reference count is zero. > C.4.2.1. Delegated Administration > > The "zfs release" subcommand requires the new 'release' delegation permission. Don't these two statements conflict ? Or is the delegation code on older pool versions smart enough to ignore delegations it doesn't know about ? The concern I have is if for some reason a delegation is added for hold/release to a dataset but the pool is not upgraded and is used on an older system than knows about the new hold/release delegations. Other than that this seems to make perfect sense and is a very useful feature, it seems complete and appropriate for the release binding and interface taxonomy requested. -- Darren J Moffat From chris.kirby@sun.com Tue May 12 09:37:15 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n4CGXVEm003309 for ; Tue, 12 May 2009 09:37:15 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n4CGXUiO028823; Tue, 12 May 2009 09:33:30 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KJJ00705HZU4Y00@nwk-avmta-2.sfbay.sun.com>; Tue, 12 May 2009 09:33:30 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KJJ001KKHZU4RB0@nwk-avmta-2.sfbay.sun.com>; Tue, 12 May 2009 09:33:30 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n4CGXTnn023416; Tue, 12 May 2009 16:33:29 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) id <0KJJ00400GZP6T00@mail-amer.sun.com>; Tue, 12 May 2009 10:33:29 -0600 (MDT) Received: from [10.1.202.246] ([unknown] [10.1.202.246]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) with ESMTPSA id <0KJJ005GAHZ9HHD0@mail-amer.sun.com>; Tue, 12 May 2009 10:33:11 -0600 (MDT) Date: Tue, 12 May 2009 11:28:40 -0500 From: Christopher Kirby Subject: Re: zfs snapshot holds [PSARC/2009/297 FastTrack timeout 05/20/2009] In-reply-to: <4A0933C7.7080602@Sun.COM> Sender: Christopher.Kirby@sun.com To: Darren J Moffat Cc: Matthew Ahrens , PSARC-ext@sun.com, zfs-team@sun.com Message-id: MIME-version: 1.0 X-Mailer: Apple Mail (2.930.3) Content-type: text/plain; CHARSET=US-ASCII; delsp=yes; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200905112228.n4BMSKhR004677@localhost.sfbay.sun.com> <4A0933C7.7080602@Sun.COM> Status: RO Content-Length: 1285 On May 12, 2009, at 3:31 AM, Darren J Moffat wrote: > Matthew Ahrens wrote: >> C.2. Version Compatibility >> Using these features does not require a pool upgrade. >> Snapshots created on earlier pool versions will be eligible >> for user reference counts and will be treated as if their >> initial reference count is zero. > >> C.4.2.1. Delegated Administration >> The "zfs release" subcommand requires the new 'release' delegation >> permission. > > Don't these two statements conflict ? Or is the delegation code on > older pool versions smart enough to ignore delegations it doesn't > know about ? The concern I have is if for some reason a delegation > is added for hold/release to a dataset but the pool is not upgraded > and is used on an older system than knows about the new hold/release > delegations. Older code will just ignore any delegations that it doesn't know about. They're just ZAP entries and won't be used by the older code. I've done some limited testing to confirm this, and I'll ask the zfs- test folks to take a look at it too. > > Other than that this seems to make perfect sense and is a very > useful feature, it seems complete and appropriate for the release > binding and interface taxonomy requested. Great, thanks Darren! -Chris From Darren.Moffat@sun.com Tue May 12 09:38:59 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n4CGZEk2003329 for ; Tue, 12 May 2009 09:38:59 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n4CGZ7Pu029300; Tue, 12 May 2009 09:35:14 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KJJ0080FI2QLN00@brm-avmta-1.central.sun.com>; Tue, 12 May 2009 10:35:14 -0600 (MDT) Received: from gmp-eb-inf-1.sun.com ([192.18.6.21]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KJJ002PCI2OM350@brm-avmta-1.central.sun.com>; Tue, 12 May 2009 10:35:12 -0600 (MDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-2-fe2.eu.sun.com [192.18.6.11]) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n4CGZBME009949; Tue, 12 May 2009 16:35:11 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) id <0KJJ00300HT2UA00@fe-emea-10.sun.com>; Tue, 12 May 2009 17:35:11 +0100 (BST) Received: from [192.168.1.103] (cpc2-rdng20-2-0-cust917.15-3.cable.virginmedia.com [86.28.167.150]) by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) with ESMTPSA id <0KJJ001MAI2MNK60@fe-emea-10.sun.com>; Tue, 12 May 2009 17:35:11 +0100 (BST) Date: Tue, 12 May 2009 17:35:07 +0100 From: Darren J Moffat Subject: Re: zfs snapshot holds [PSARC/2009/297 FastTrack timeout 05/20/2009] In-reply-to: Sender: Darren.Moffat@sun.com To: Christopher Kirby Cc: Matthew Ahrens , PSARC-ext@sun.com, zfs-team@sun.com Message-id: <4A09A53B.7080605@Sun.COM> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200905112228.n4BMSKhR004677@localhost.sfbay.sun.com> <4A0933C7.7080602@Sun.COM> User-Agent: Thunderbird 2.0.0.18 (X11/20090211) Status: RO Content-Length: 1276 Christopher Kirby wrote: > On May 12, 2009, at 3:31 AM, Darren J Moffat wrote: > >> Matthew Ahrens wrote: >>> C.2. Version Compatibility >>> Using these features does not require a pool upgrade. >>> Snapshots created on earlier pool versions will be eligible >>> for user reference counts and will be treated as if their >>> initial reference count is zero. >> >>> C.4.2.1. Delegated Administration >>> The "zfs release" subcommand requires the new 'release' delegation >>> permission. >> >> Don't these two statements conflict ? Or is the delegation code on >> older pool versions smart enough to ignore delegations it doesn't know >> about ? The concern I have is if for some reason a delegation is >> added for hold/release to a dataset but the pool is not upgraded and >> is used on an older system than knows about the new hold/release >> delegations. > > Older code will just ignore any delegations that it doesn't know about. > They're just ZAP entries and won't be used by the older code. > > I've done some limited testing to confirm this, and I'll ask the > zfs-test folks > to take a look at it too. That sounds safe enough not to need a pool version bump to me and is what I expected. From an ARC review point I'm done and happy. -- Darren J Moffat From edward.pilatowicz@sun.com Thu May 14 15:48:23 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n4EMmNYH016970 for ; Thu, 14 May 2009 15:48:23 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n4EMmM8B011144; Thu, 14 May 2009 15:48:23 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KJN00J19OOMHZ00@brm-avmta-1.central.sun.com>; Thu, 14 May 2009 16:48:22 -0600 (MDT) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KJN00G8OOOGUGB0@brm-avmta-1.central.sun.com>; Thu, 14 May 2009 16:48:17 -0600 (MDT) Received: from eng.sun.com (cretaceous.SFBay.Sun.COM [129.146.17.63]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n4EMmG4Y061946; Thu, 14 May 2009 15:48:16 -0700 (PDT) Date: Thu, 14 May 2009 15:48:16 -0700 From: Edward Pilatowicz Subject: Re: zfs snapshot holds [PSARC/2009/297 FastTrack timeout 05/20/2009] In-reply-to: <200905112228.n4BMSKhR004677@localhost.sfbay.sun.com> To: Matthew Ahrens Cc: PSARC-ext@sun.com, zfs-team@sun.com Message-id: <20090514224815.GF308461@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200905112228.n4BMSKhR004677@localhost.sfbay.sun.com> User-Agent: Mutt/1.5.19 (2009-01-05) Status: RO Content-Length: 7450 this is great. i just have one question. is there anyway to undo a "zfs destroy -d" request? (it seems to me that if zfs destroy -d was implmented in terms of a snapshot property, say autodelete=on|off, this would be easy.) ed On Mon, May 11, 2009 at 03:28:20PM -0700, Matthew Ahrens wrote: > > Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI > This information is Copyright 2009 Sun Microsystems > 1. Introduction > 1.1. Project/Component Working Name: > zfs snapshot holds > 1.2. Name of Document Author/Supplier: > Author: Chris Kirby > 1.3 Date of This Document: > 11 May, 2009 > 4. Technical Description > > ZFS snapshot reference counts > > A. SUMMARY: > > This case adds support to ZFS for user-initiated reference counts on snapshots. > > B. PROBLEM: > > Remote replication of ZFS datasets can result in different automatic > snapshot policies on the two sides of a replication pair. For example, > the sending side might want to keep five snapshots at one-minute intervals, > while the receiving side might want to keep ten snapshots at one-minute > intervals. > > This can result in the older snapshots being destroyed inadvertently > by zfs receive since they no longer exist on the sending side. > > Also, when an administrator wants to destroy a snapshot for which there > are clones, the admin must remember to come back and destroy the snapshot > once the last clone has been destroyed. It would be handy to have > an automated way of doing this deferred removal. > > C. PROPOSED SOLUTION > > C.1. Overview > > This new facility would permit users or applications to place > holds on snapshots that would prevent them from being deleted. Further, > this facility would allow a snapshot with clones to be deleted pending the > removal of the last clone using the new "zfs destroy -d" option. > > Each snapshot will have an associated user-reference count, initialized > to zero. This count increases by one whenever a hold is taken on the snapshot > and decreases by one whenever a hold is released. > > In the current model, in order for a snapshot to be destroyed with > "zfs destroy", it must have no clones. In the new model, the snapshot > must also have a zero user-reference count. > > In the example in section B, the receiving side could place holds > on the ten snapshots it wants to keep, which will prevent the zfs receive > operation from destroying them. > > C.2. Version Compatibility > > Using these features does not require a pool upgrade. > Snapshots created on earlier pool versions will be eligible > for user reference counts and will be treated as if their > initial reference count is zero. > > C.3. Changes to Existing Subcommands > > C.3.1 zfs destroy -d > > The "zfs destroy" subcommand now takes a -d option, applicable only > to snapshots. If specified, the snapshot will be destroyed right away > if and only if "zfs destroy" without the -d option would have deleted it. > i.e. immediate deletion by "zfs destroy -d" requires the following: > > 1) The snapshot has no clones. > AND > 2) The user-initiated reference count is zero. > > Otherwise, the snapshot is marked for deferred deletion, where it will exist > as a normal (usable/visible) snapshot until both of the preconditions listed > above are met, at which point the snapshot will be destroyed. > > C.3.2. zfs destroy > > When a snapshot's last clone is destroyed with "zfs destroy", if the > snapshot's user-initiated reference count is zero; and the snapshot > was previously marked for deferred destruction with "zfs destroy -d"; > the snapshot and the clone will be destroyed as part of a single > transaction group. That is, either they will both be destroyed > or neither of them will be. > > If the snapshot has an elevated user reference count or other children, > the clone will be destroyed and the snapshot will remain. > > C.3.3. zfs receive > > In the current model, receiving a "zfs send -R" (replication) stream > will result in a "zfs destroy" on any snapshots that don't exist on > the sending side. In the new model, the receiving side will do a > "zfs destroy -d" instead. > > C.4. New Subcommands > > Three new subcommands are added: "zfs hold", "zfs release", and "zfs holds": > > C.4.1. zfs hold [-r] ... > > The zfs hold subcommand adds a single reference, named with the > argument, to the given snapshot(s). Each snapshot has its own > tag namespace, and tags must be unique within that space. > > As long as a hold exists on a snapshot, attempts to "zfs destroy" > that snapshot will return EBUSY. > > If the -r option is given, a hold with the given tag will be > applied recursively to the snapshots of all descendent file systems. > For example, given these snapshots: > > pool/fs1@snap1 > pool/fs1/fs2@snap1 > pool/fs2@snap1 > > This command: > > # zfs hold -r mytag pool/fs1@snap1 > > will place a hold named "mytag" on "pool/fs1@snap1" and "pool/fs1/fs2@snap1", > but not on "pool/fs2@snap1". > > The actions taken by -r are atomic; either all of the snapshots will > be held with the given tag, or none of them will. > > C.4.1.1. Delegated Administration > > The "zfs hold" subcommand requires the new 'hold' delegation permission. > > C.4.2. zfs release [-r] ... > > The zfs release subcommand removes a single reference named by > the argument from the named snapshot(s). The tag must already > exist for each snapshot. > > The -r option does a recursive release, similar to the recursive > hold in section C.4.1. > > When the last user reference has been removed from a snapshot, > the snapshot will be automatically destroyed if and only if: > > 1) The snapshot has no clones. > AND > 2) The snapshot has already been marked for deferred deletion > with 'zfs destroy -d'. > > C.4.2.1. Delegated Administration > > The "zfs release" subcommand requires the new 'release' delegation permission. > > C.4.3. zfs holds [-r] ... > > This subcommand lists all of the existing user references for > the given snapshot(s) or family of snapshots. For example: > > # zfs snapshot tank@snap2 > # zfs hold com.sun.sometag tank@snap2 > # zfs hold mytag tank@snap2 > # zfs holds tank@snap2 > tank@snap2: > TIMESTAMP TAG > Mon Apr 27 12:15:49 2009 com.sun.sometag > Mon Apr 27 12:16:02 2009 mytag > > The -r option will list holds on the named snapshots of descendents, in > addition to listing holds on the named dataset. > > If "zfs hold -r com.sun.sometag pool@snap" was used to recursively apply > a tag, "zfs holds -r pool@snap" will display those tags. > > C.4.3.1. Delegated Administration > > The "zfs holds" subcommand requires the new 'hold' delegation permission. > > C.5. Tag Naming Convention > > We will suggest (but not enforce) that users use reverse DNS form > for user-reference tagging. e.g. "com.sun.fishworks.replication" > > Tag names are subject to the same restrictions as snapshot names, i.e. > they are limited to alphanumeric characters plus [-_.: ]. > > We will reserve tag names starting with "." for use by direct callers > of the relevant libzfs routines. > > Stability > > This case requests patch/micro release binding. The new interfaces are > committed. > > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > ON > 6.5. ARC review type: FastTrack > 6.6. ARC Exposure: open From chris.kirby@sun.com Thu May 14 16:02:19 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n4EN2HNx022678 for ; Thu, 14 May 2009 16:02:17 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n4EN1q3Y026441; Fri, 15 May 2009 07:02:16 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KJN00901PBP0E00@nwk-avmta-1.sfbay.Sun.COM>; Thu, 14 May 2009 16:02:13 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KJN0066JPBPNZ50@nwk-avmta-1.sfbay.Sun.COM>; Thu, 14 May 2009 16:02:13 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n4EN2DmS009952; Thu, 14 May 2009 23:02:13 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) id <0KJN00C00P4JHM00@mail-amer.sun.com>; Thu, 14 May 2009 17:02:13 -0600 (MDT) Received: from [10.0.1.197] ([unknown] [129.150.34.97]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) with ESMTPSA id <0KJN00DN5PBHSIB0@mail-amer.sun.com>; Thu, 14 May 2009 17:02:06 -0600 (MDT) Date: Thu, 14 May 2009 18:02:05 -0500 From: Chris Kirby Subject: Re: zfs snapshot holds [PSARC/2009/297 FastTrack timeout 05/20/2009] In-reply-to: <20090514224815.GF308461@eng.sun.com> Sender: Christopher.Kirby@sun.com To: Edward Pilatowicz Cc: Matthew Ahrens , PSARC-ext@sun.com, zfs-team@sun.com Message-id: MIME-version: 1.0 X-Mailer: Apple Mail (2.930.3) Content-type: text/plain; CHARSET=US-ASCII; delsp=yes; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200905112228.n4BMSKhR004677@localhost.sfbay.sun.com> <20090514224815.GF308461@eng.sun.com> Status: RO Content-Length: 510 On May 14, 2009, at 5:48 PM, Edward Pilatowicz wrote: > this is great. > > i just have one question. is there anyway to undo a "zfs destroy -d" > request? (it seems to me that if zfs destroy -d was implmented in > terms > of a snapshot property, say autodelete=on|off, this would be easy.) Ed, Just like with "zfs destroy", there's no way to undo a "zfs destroy -d". The snapshot might be destroyed immediately if there's no reason to defer the operation, and we have no way to undo that. -Chris From Richard.Matthews@sun.com Tue May 19 10:49:47 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n4JHnknj020335 for ; Tue, 19 May 2009 10:49:46 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n4JHn2Bq023050; Tue, 19 May 2009 10:49:46 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KJW00C0RK6T9E00@nwk-avmta-2.sfbay.sun.com>; Tue, 19 May 2009 10:49:41 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KJW0037BK6ST4E0@nwk-avmta-2.sfbay.sun.com>; Tue, 19 May 2009 10:49:40 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n4JHneE9008776; Tue, 19 May 2009 17:49:40 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) id <0KJW00400JMK0C00@mail-amer.sun.com>; Tue, 19 May 2009 11:49:40 -0600 (MDT) Received: from [129.152.9.11] ([unknown] [129.152.9.11]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) with ESMTPSA id <0KJW0085OK66KA60@mail-amer.sun.com>; Tue, 19 May 2009 11:49:25 -0600 (MDT) Date: Tue, 19 May 2009 12:49:18 -0500 From: Rick Matthews Subject: Re: zfs snapshot holds [PSARC/2009/297 FastTrack timeout 05/20/2009] In-reply-to: <200905112228.n4BMSKhR004677@localhost.sfbay.sun.com> Sender: Richard.Matthews@sun.com To: Matthew Ahrens Cc: PSARC-ext@sun.com, zfs-team@sun.com Reply-to: Richard.Matthews@sun.com Message-id: <4A12F11E.1020803@Sun.COM> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200905112228.n4BMSKhR004677@localhost.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.19 (X11/20090110) Status: RO Content-Length: 472 +1 -- --------------------------------------------------------------------- Rick Matthews email: Rick.Matthews@sun.com Sun Microsystems, Inc. phone:+1(651) 554-1518 1270 Eagan Industrial Road phone(internal): 54418 Suite 160 fax: +1(651) 554-1540 Eagan, MN 55121-1231 USA main: +1(651) 554-1500 --------------------------------------------------------------------- From Matthew.Ahrens@sun.com Wed May 20 10:10:53 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n4KHAqTa006374 for ; Wed, 20 May 2009 10:10:52 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n4KHAm1X008132; Wed, 20 May 2009 10:10:51 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KJY0040BD22S200@nwk-avmta-2.sfbay.sun.com>; Wed, 20 May 2009 10:10:50 -0700 (PDT) Received: from zion.sfbay.sun.com ([129.146.17.75]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KJY00M1MD228L90@nwk-avmta-2.sfbay.sun.com>; Wed, 20 May 2009 10:10:50 -0700 (PDT) Received: from stationary.local (punchin-ahrens.SFBay.Sun.COM [10.7.251.178]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id n4KHAnDV003972; Wed, 20 May 2009 17:10:49 +0000 (GMT) Date: Wed, 20 May 2009 10:10:49 -0700 From: Matthew Ahrens Subject: Re: zfs snapshot holds [PSARC/2009/297 FastTrack timeout 05/20/2009] In-reply-to: <200905112228.n4BMSKhR004677@localhost.sfbay.sun.com> To: Matthew Ahrens Cc: PSARC-ext@sun.com, zfs-team@sun.com Message-id: <4A143999.4090709@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200905112228.n4BMSKhR004677@localhost.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.21 (Macintosh/20090302) Status: RO Content-Length: 7293 This case was approved at today's psarc meeting. --matt Matthew Ahrens wrote: > Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI > This information is Copyright 2009 Sun Microsystems > 1. Introduction > 1.1. Project/Component Working Name: > zfs snapshot holds > 1.2. Name of Document Author/Supplier: > Author: Chris Kirby > 1.3 Date of This Document: > 11 May, 2009 > 4. Technical Description > > ZFS snapshot reference counts > > A. SUMMARY: > > This case adds support to ZFS for user-initiated reference counts on snapshots. > > B. PROBLEM: > > Remote replication of ZFS datasets can result in different automatic > snapshot policies on the two sides of a replication pair. For example, > the sending side might want to keep five snapshots at one-minute intervals, > while the receiving side might want to keep ten snapshots at one-minute > intervals. > > This can result in the older snapshots being destroyed inadvertently > by zfs receive since they no longer exist on the sending side. > > Also, when an administrator wants to destroy a snapshot for which there > are clones, the admin must remember to come back and destroy the snapshot > once the last clone has been destroyed. It would be handy to have > an automated way of doing this deferred removal. > > C. PROPOSED SOLUTION > > C.1. Overview > > This new facility would permit users or applications to place > holds on snapshots that would prevent them from being deleted. Further, > this facility would allow a snapshot with clones to be deleted pending the > removal of the last clone using the new "zfs destroy -d" option. > > Each snapshot will have an associated user-reference count, initialized > to zero. This count increases by one whenever a hold is taken on the snapshot > and decreases by one whenever a hold is released. > > In the current model, in order for a snapshot to be destroyed with > "zfs destroy", it must have no clones. In the new model, the snapshot > must also have a zero user-reference count. > > In the example in section B, the receiving side could place holds > on the ten snapshots it wants to keep, which will prevent the zfs receive > operation from destroying them. > > C.2. Version Compatibility > > Using these features does not require a pool upgrade. > Snapshots created on earlier pool versions will be eligible > for user reference counts and will be treated as if their > initial reference count is zero. > > C.3. Changes to Existing Subcommands > > C.3.1 zfs destroy -d > > The "zfs destroy" subcommand now takes a -d option, applicable only > to snapshots. If specified, the snapshot will be destroyed right away > if and only if "zfs destroy" without the -d option would have deleted it. > i.e. immediate deletion by "zfs destroy -d" requires the following: > > 1) The snapshot has no clones. > AND > 2) The user-initiated reference count is zero. > > Otherwise, the snapshot is marked for deferred deletion, where it will exist > as a normal (usable/visible) snapshot until both of the preconditions listed > above are met, at which point the snapshot will be destroyed. > > C.3.2. zfs destroy > > When a snapshot's last clone is destroyed with "zfs destroy", if the > snapshot's user-initiated reference count is zero; and the snapshot > was previously marked for deferred destruction with "zfs destroy -d"; > the snapshot and the clone will be destroyed as part of a single > transaction group. That is, either they will both be destroyed > or neither of them will be. > > If the snapshot has an elevated user reference count or other children, > the clone will be destroyed and the snapshot will remain. > > C.3.3. zfs receive > > In the current model, receiving a "zfs send -R" (replication) stream > will result in a "zfs destroy" on any snapshots that don't exist on > the sending side. In the new model, the receiving side will do a > "zfs destroy -d" instead. > > C.4. New Subcommands > > Three new subcommands are added: "zfs hold", "zfs release", and "zfs holds": > > C.4.1. zfs hold [-r] ... > > The zfs hold subcommand adds a single reference, named with the > argument, to the given snapshot(s). Each snapshot has its own > tag namespace, and tags must be unique within that space. > > As long as a hold exists on a snapshot, attempts to "zfs destroy" > that snapshot will return EBUSY. > > If the -r option is given, a hold with the given tag will be > applied recursively to the snapshots of all descendent file systems. > For example, given these snapshots: > > pool/fs1@snap1 > pool/fs1/fs2@snap1 > pool/fs2@snap1 > > This command: > > # zfs hold -r mytag pool/fs1@snap1 > > will place a hold named "mytag" on "pool/fs1@snap1" and "pool/fs1/fs2@snap1", > but not on "pool/fs2@snap1". > > The actions taken by -r are atomic; either all of the snapshots will > be held with the given tag, or none of them will. > > C.4.1.1. Delegated Administration > > The "zfs hold" subcommand requires the new 'hold' delegation permission. > > C.4.2. zfs release [-r] ... > > The zfs release subcommand removes a single reference named by > the argument from the named snapshot(s). The tag must already > exist for each snapshot. > > The -r option does a recursive release, similar to the recursive > hold in section C.4.1. > > When the last user reference has been removed from a snapshot, > the snapshot will be automatically destroyed if and only if: > > 1) The snapshot has no clones. > AND > 2) The snapshot has already been marked for deferred deletion > with 'zfs destroy -d'. > > C.4.2.1. Delegated Administration > > The "zfs release" subcommand requires the new 'release' delegation permission. > > C.4.3. zfs holds [-r] ... > > This subcommand lists all of the existing user references for > the given snapshot(s) or family of snapshots. For example: > > # zfs snapshot tank@snap2 > # zfs hold com.sun.sometag tank@snap2 > # zfs hold mytag tank@snap2 > # zfs holds tank@snap2 > tank@snap2: > TIMESTAMP TAG > Mon Apr 27 12:15:49 2009 com.sun.sometag > Mon Apr 27 12:16:02 2009 mytag > > The -r option will list holds on the named snapshots of descendents, in > addition to listing holds on the named dataset. > > If "zfs hold -r com.sun.sometag pool@snap" was used to recursively apply > a tag, "zfs holds -r pool@snap" will display those tags. > > C.4.3.1. Delegated Administration > > The "zfs holds" subcommand requires the new 'hold' delegation permission. > > C.5. Tag Naming Convention > > We will suggest (but not enforce) that users use reverse DNS form > for user-reference tagging. e.g. "com.sun.fishworks.replication" > > Tag names are subject to the same restrictions as snapshot names, i.e. > they are limited to alphanumeric characters plus [-_.: ]. > > We will reserve tag names starting with "." for use by direct callers > of the relevant libzfs routines. > > Stability > > This case requests patch/micro release binding. The new interfaces are > committed. > > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > ON > 6.5. ARC review type: FastTrack > 6.6. ARC Exposure: open From chris.kirby@oracle.com Wed Jul 14 10:15:02 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o6EHF25M006091 for ; Wed, 14 Jul 2010 10:15:02 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o6EHF2qh023811 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 14 Jul 2010 10:15:02 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L5K00L05592O300@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 14 Jul 2010 10:15:02 -0700 (PDT) Received: from sca-ea-mail-4.sun.com ([192.18.43.22]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L5K00CLB591C9B0@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 14 Jul 2010 10:15:01 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by sca-ea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o6EHF1Go006535 for ; Wed, 14 Jul 2010 17:15:01 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o6EHEwRt006211 for ; Wed, 14 Jul 2010 17:14:59 +0000 (GMT) Received: from abhmt016.oracle.com by acsmt353.oracle.com with ESMTP id 405337901279127695; Wed, 14 Jul 2010 10:14:55 -0700 Received: from punchin-client-10-7-250-116.sfbay.sun.com (/10.7.250.116) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 14 Jul 2010 10:14:55 -0700 Date: Wed, 14 Jul 2010 12:15:28 -0500 From: Chris Kirby Subject: Re: zfs snapshot holds [PSARC/2009/297 FastTrack timeout 05/20/2009] In-reply-to: <200905112228.n4BMSKhR004677@localhost.sfbay.sun.com> To: PSARC-ext@sun.com Message-id: MIME-version: 1.0 X-Mailer: Apple Mail (2.1081) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090204.4C3DF094.00FE:SCFMA4539814,ss=1,fgs=0 References: <200905112228.n4BMSKhR004677@localhost.sfbay.sun.com> Status: RO Content-Length: 427 I'd like to correct this case for the archives: > C.2. Version Compatibility > > Using these features does not require a pool upgrade. > Snapshots created on earlier pool versions will be eligible > for user reference counts and will be treated as if their > initial reference count is zero. The deferred destroy and snapshot holds features did in fact require an on-disk version change, which is zpool version 18. -Chris