From ahrens@sac.sfbay.sun.com Tue May 25 13:30:57 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o4PKUuXP022004 for ; Tue, 25 May 2010 13:30:57 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o4PKUuCt019190; Tue, 25 May 2010 14:30:56 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L2Z00803SZKRN00@nwk-avmta-2.sfbay.sun.com>; Tue, 25 May 2010 13:30:56 -0700 (PDT) Received: from localhost.sfbay.sun.com ([129.146.17.46]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L2Z005NASZJFZ90@nwk-avmta-2.sfbay.sun.com>; Tue, 25 May 2010 13:30:55 -0700 (PDT) Received: from localhost.sfbay.sun.com (localhost [127.0.0.1] (may be forged)) by localhost.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id o4PKUtWt002786; Tue, 25 May 2010 13:30:55 -0700 (PDT) Received: (from ahrens@localhost) by localhost.sfbay.sun.com (8.14.3+Sun/8.14.3/Submit) id o4PKUt0t002782; Tue, 25 May 2010 13:30:55 -0700 (PDT) Date: Tue, 25 May 2010 13:30:55 -0700 (PDT) From: Matthew Ahrens Subject: ZFS backup options [PSARC/2010/193 FastTrack timeout 06/02/2010] To: PSARC-ext@sun.com Cc: zfs-eng@sun.com Message-id: <201005252030.o4PKUt0t002782@localhost.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 26584 Template Version: @(#)sac_nextcase 1.70 03/30/10 SMI This information is Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. 1. Introduction 1.1. Project/Component Working Name: ZFS backup options 1.2. Name of Document Author/Supplier: Author: Thomas Erickson 1.3 Date of This Document: 25 May, 2010 4. Technical Description 4. Technical Description ZFS backup options A. SUMMARY Dataset properties can interfere with the use of 'zfs send' and 'zfs receive' as a backup solution because the properties of the original dataset may not work in the context of the backup dataset, or they may not be the right settings in terms of how you want to back up your data. Local properties on the backup dataset also interfere with restores, since there is no way to tell 'zfs send' to send the properties received from the original rather than local settings that may apply only to the backup. This case adds a -o option to the 'zfs receive' command that works like the existing 'zfs create -o property=value' to specify the initial properties of a newly created dataset, making it possible to override properties received from a non-incremental 'zfs send' stream. Similar to -o, a -x option for 'zfs receive' is added to override received properties without specifying a value, simply to ensure that they have no effect on the behavior of the received dataset (as if they had been excluded from the send stream), even though the received values can still be restored. This case also adds a -b (restore from backup) option to the 'zfs send' command to restore properties received from the original dataset rather than local settings specified on the backup dataset with 'zfs set' or 'zfs receive -o'. For the convenience of related administrator actions, this case adds a -r option to 'zfs set' similar to 'zfs inherit -r' and new aliases for use with 'zfs list -t'. B. PROBLEM B.1. Backup Today we can set properties locally with 'zfs set' to override properties received from incremental 'zfs send' streams, but this is no help with non-incremental streams because there is no opportunity to run 'zfs set' on a dataset before it is newly created by 'zfs receive'. This is particularly awkward when a sent dataset has a non-default mountpoint, since the receive has no place to replicate the sent data when it fails to mount the received dataset on a directory that may not exist or that is already in use by the sent dataset. For example: % zfs create -o mountpoint=/tank/mnt tank/fs/mnt % zfs snapshot tank/fs/mnt@1 % zfs create tank/foo % zfs send -R tank/fs/mnt@1 | zfs recv -d tank/foo cannot mount 'tank/foo/fs/mnt': mountpoint or dataset is busy % Similar problems can interfere with the use of send/receive as a backup solution when there is a property in the source dataset whose validity on the target depends on inherited settings different from those of the source. For example, the following interference from the source quota prevents a successful receive when the target inherits copies differently: % zfs create tank/a % zfs create tank/z % zfs set copies=3 tank/z % zfs create tank/a/b % zfs set quota=2m tank/a/b % mkfile 1m /tank/a/b/file % zfs snapshot tank/a/b@1 % zfs send -R tank/a/b@1 | zfs recv -e tank/z cannot receive new filesystem stream: destination tank/z space quota exceeded % The lack of a receive option to set initial property values also hinders administrators from applying settings like compression and deduplication to backups, since local settings that differ from those in the send stream can only be applied after the fact and therefore only apply to incremental updates. Sometimes the administrator has a value in mind to override the received property, but other times the administrator may simply want the default. In that case we don't want to task the administrator with finding out the default in order to set it explicitly, so an option to override a received property without providing an explicit value is more helpful (and preserves inheritance). B.2. Restore Restoring properties from a backup is also troublesome because 'zfs send' assumes that the effective property values are the ones to include in the send stream (if specified on the sent dataset and not inherited). Since local settings override received properties, the administrator will not get back the original settings received on the backup dataset because 'zfs send' will choose the effective local settings instead. Local settings on the backup likely specify how the administrator wants to store backups and have nothing to do with the restored dataset being like the original. To clarify the problem of restoring properties from a backup, it helps to think of the effective value of a property is the topmost value in the following stack: local received --------- inherited default Each level of the stack overrides the one below it. If a property does not have a specified value at the topmost level, ZFS resorts to the value at the next lower level, if it exists, and so on. (The default value always exists.) The first value it finds determines the effective value of the property. This is how ZFS already works and this case does nothing to change that. Values above the dashed line (local and received) are explicit and included in a 'zfs send' stream unless overridden. Values below the dashed line (inherited and default) are implicit and never included in a 'zfs send' stream. If you say nothing about a dataset property, neither by setting it nor by receiving it, your dataset inherits the property from its parent. If the parent does not have a set or received value and neither do any of its ancestors, then your dataset inherits the default. Inheriting properties is the way to get the behavior you want without including those settings in a 'zfs send' stream. Again, this case does not change that. The reason that sending the effective value is a problem for backups is best understood by considering the outcome of a replication chain. Suppose that we send from dataset A and receive to dataset B, then send from dataset B and receive to dataset C. Clearly we want A->B to replicate the effective properties of A, but what about B->C? If B overrides a property locally, should C get the original value from A or the local value on B? Currently 'zfs send' can only replicate the effective properties of B. It cannot pass along the original properties from A without interference from local settings on B, thereby finishing the relay of A->B->C with the same properties it had at the start. The semantics of restoring from a backup are not possible unless you can get back the properties you started with. B.3. Set recursively In the case of a recursive 'zfs send' stream, the option to set the initial values of specific properties needs to apply those properties recursively. For consistency and convenience, 'zfs set' should also support this ability to apply a local setting recursively so it can override a property received in a recursive incremental update. B.4. Query It's inconvenient to spell out the full names of the dataset types "filesystem", "volume", and "snapshot" when querying received datasets with 'zfs list -t'. C. PROPOSED SOLUTION C.1. Overview The proposed solution adds a -o option to 'zfs receive' to specify the initial values of any number of properties as '-o property=value', just like we do with 'zfs create'. To avoid the unwanted effects of a property in a 'zfs send' stream without specifying a value for that property, a similar '-x property' option overrides the named property in a way that preserves the received value for a possible future restore (or inherit -S) but has no effect on the effective value of the property on the received dataset, as if the named property had been excluded from the send stream. Both of these receive options, -o and -x, work with recursive send streams by overriding the received property recursively. For convenience, they also work with incremental streams, even though an administrator can use 'zfs set' and/or 'zfs inherit' to override received properties before receiving an incremental update. For consistency with 'zfs inherit -r' as well as the -o and -x receive options, a '-r' option added to 'zfs set' makes it possible to apply a property setting recursively before receiving a recursive, incremental update. A new -b option added to 'zfs send' specifies received rather than effective property values to be included in the send stream, making it possible to restore the original, backed-up settings without interference from local settings. Finally, the proposed solution adds the type aliases "fs" for "filesystem", "snap" for "snapshot", and "vol" for "volume" to use with 'zfs list -t' when querying datasets. C.1.1. Backup with 'receive -o', restore with 'send -b' To support the use of send/receive as a backup solution, ZFS will set a property's local value to that specified by receive -o, whereas its received value (overridden by the local value) will be set to the value obtained from the send stream (if any). For example: % zfs set compression=off a/fs % zfs snapshot a/fs@1 % zfs send -R a/fs@1 | zfs recv -o compression=on -d z % zfs get -o all compression z/fs NAME PROPERTY VALUE RECEIVED SOURCE z/fs compression on off local % This way, the backup dataset created by 'zfs receive' can use whatever settings the administrator specifies, but the original settings are not lost and can still be restored from the backup. To do that requires a new -b option for 'zfs send' that tells ZFS to send received properties regardless of local settings. This is not the default 'zfs send' behavior because it might be confusing to send property values that are not currently effective on the source dataset. The following table describes what 'zfs send -b' actually sends compared to 'zfs send' without -b, depending on whether the source property has a local value, a received value, both, or neither: local received | sent with -b sent without -b ----- -------- | ----------------- --------------- yes no | nothing local value no yes | received value received value yes yes | received value local value no no | nothing nothing An internal "$hasrecvd" property tracks whether a dataset was received on or after pool version 22 (received properties). If that internal property is not found, 'zfs send' treats the dataset as the start of a replication chain by sending its effective property values, as if the dataset was sent without -b. This behavior is consistent in the sense that 'send -b' always preserves original properties. In the replication chain A->B->C, the initial replication A->B is a special case in which the original properties are the effective and not the received properties, so the column under "sent without -b" in the above table applies. The alternative was to send no properties, since there are no received properties if the dataset has never been received and this is the initial replication. While that may be the more consistent behavior, it lacks utility. The chosen behavior potentially frees backup applications from deciding when to use -b since they can use -b by default and get the desired behavior in both the A->B and B->C cases. In the case where a received dataset needs to become the start of a new replication chain, that can be indicated by a user property. C.1.2. Recursive streams and 'zfs set -r' If the sent stream is recursive (ie 'zfs send -R'); that is, it includes one or more levels of descendant datasets, then 'zfs recv -o property=value' ensures that the effective value of the property is the same in all received datasets. It does this in the most sensible way it can, depending on the property. If the property is inheritable, it sets the named property to the specified value only on the top level dataset and inherits the named property on all the datasets below that. That way, all received datasets get the setting from one place. The received values, however, reflect the settings of the source datasets. For example: % zfs create a/b % zfs create a/b/c % zfs create a/b/c/d % zfs set compression=off a/b % zfs set compression=gzip a/b/c/d % zfs get compression a/b a/b/c a/b/c/d NAME PROPERTY VALUE SOURCE a/b compression off local a/b/c compression off inherited from a/b a/b/c/d compression gzip local % zfs snapshot -r a/b@1 % zfs send -R a/b@1 | zfs recv -o compression=on -d z % zfs get -o all compression z/b z/b/c z/b/c/d NAME PROPERTY VALUE RECEIVED SOURCE z/b compression on off local z/b/c compression on - inherited from z/b z/b/c/d compression on gzip inherited from z/b % Note that the value "gzip" in the RECEIVED column for z/b/c/d reflects the local setting of the source dataset, a/b/c/d. If the property is not inheritable, 'zfs recv -o property=value' instead sets the property recursively. For example, if we override canmount instead of compression: % zfs create a/b % zfs create a/b/c % zfs create a/b/c/d % zfs set canmount=off a/b % zfs set canmount=noauto a/b/c/d % zfs get canmount a/b a/b/c a/b/c/d NAME PROPERTY VALUE SOURCE a/b canmount off local a/b/c canmount on default a/b/c/d canmount noauto local % zfs snapshot -r a/b@1 % zfs send -R a/b@1 | zfs recv -o canmount=on -d z % zfs get -o all canmount z/b z/b/c z/b/c/d NAME PROPERTY VALUE RECEIVED SOURCE z/b canmount on off local z/b/c canmount on - local z/b/c/d canmount on noauto local % The new 'zfs set -r' option uses the same algorithm to make the setting effective throughout the subtree of child datasets. If the property is inheritable, it sets the property on the top-level dataset only and inherits the setting the rest of the way down the subtree. In the above example involving compression, receiving without -o and correcting after the fact with 'zfs set -r compression=on z/b' would still result in the same property values: % zfs send -R a/b@1 | zfs recv -d z % zfs set -r compression=on z/b % zfs get -o all compression z/b z/b/c z/b/c/d NAME PROPERTY VALUE RECEIVED SOURCE z/b compression on off local z/b/c compression on - inherited from z/b z/b/c/d compression on gzip inherited from z/b % The difference is that in the case of 'zfs recv -o compression=on' the initial received data is actually compressed, whereas 'set -r' only applies to future updates. Without -o to specify compression, 'zfs receive' uses the received property values ("off" and "gzip" in this case) to determine how the initial receive should or should not compress data: % zfs send -R a/b@1 | zfs recv -d z % zfs get -o all compression z/b z/b/c z/b/c/d NAME PROPERTY VALUE RECEIVED SOURCE z/b compression off off received z/b/c compression off - inherited from z/b z/b/c/d compression gzip gzip received % That's how the data is actually received before compression is overridden locally with 'set -r'. Either way, the final outcome is exactly the same as if the administrator had typed the commands % zfs inherit -r compression z/b % zfs set compression=on z/b If the property is not inheritable, as in the example involving canmount above, then 'zfs set -r' sets the property all the way down the subtree rooted at the specified dataset. A subset of non-inheritable properties, like quota and reservation, take a size value. In that case it does not make sense to set the property recursively, since the size already applies to the entire subtree, so 'zfs receive -o' would only set the property on the top-level dataset. For example: % zfs create a/b % zfs create a/b/c % zfs create a/b/c/d % zfs set quota=1g a/b % zfs set quota=1m a/b/c/d % zfs get quota a/b a/b/c a/b/c/d NAME PROPERTY VALUE SOURCE a/b quota 1G local a/b/c quota none default a/b/c/d quota 1M local % zfs snapshot -r a/b@1 % zfs send -R a/b@1 | zfs recv -o quota=5g -d z % zfs get -o all quota z/b z/b/c z/b/c/d NAME PROPERTY VALUE RECEIVED SOURCE z/b quota 5G 1G local z/b/c quota none - default z/b/c/d quota 1M 1M received % However, in the case of overriding received quota with quota=none, it does make sense to set the property recursively, so the above example turns out somewhat differently: % zfs send -R a/b@1 | zfs recv -o quota=none -d z % zfs get -o all quota z/b z/b/c z/b/c/d NAME PROPERTY VALUE RECEIVED SOURCE z/b quota none 1G local z/b/c quota none - local z/b/c/d quota none 1M local % 'zfs set -r' treats non-inheritable size properties like quota in exactly the same way. Regardless of the property, 'zfs receive -o' of a recursive stream and 'zfs set -r' both use the same algorithm to do what makes the most sense depending on the property. The only potential difference is when a snapshot is missing from the send stream. In that case, 'zfs send' prints a warning about the missing snapshot and 'zfs receive -o' does not update the property in the target dataset corresponding to the missing snapshot. In the normal case, however, 'zfs receive -o' visits the entire subtree of child datasets and the outcome (as far as properties are concerned) is the same as that of 'zfs set -r'. C.1.3. 'zfs receive -x' The idea of 'receive -x' is to ensure that the received property does not determine the effective value of the property on the received dataset; that is, to get the behavior that you would have gotten if the property had been excluded from the send stream, and to do that recursively in the case of a recursive stream. How it accomplishes that depends on the property. If the property is inheritable, 'receive -x' explicitly inherits the property to override the received value. The effect is the same as 'zfs inherit -r' if the send stream is recursive, for example: % zfs create a/b % zfs create a/b/c % zfs create a/b/c/d % zfs set compression=off a/b % zfs set compression=gzip a/b/c/d % zfs get compression a/b a/b/c a/b/c/d NAME PROPERTY VALUE SOURCE a/b compression off local a/b/c compression off inherited from a/b a/b/c/d compression gzip local % zfs snapshot -r a/b@1 % zfs send -R a/b@1 | zfs recv -x compression -d z % zfs get -o all compression z/b z/b/c z/b/c/d NAME PROPERTY VALUE RECEIVED SOURCE z/b compression off off default z/b/c compression off - default z/b/c/d compression off gzip default % Explicit inheritance causes ZFS to ignore the received value and resort to the parent if there is no local setting. In this case, since the parent pool z does not specify a value, all the children inherit the default. If the property is not inheritable, 'receive -x' sets the default value recursively. Since the default canmount value happens to be "on", the canmount example in C.1.2 above turns out the same when we replace % zfs send -R a/b@1 | zfs recv -o canmount=on -d z with % zfs send -R a/b@1 | zfs recv -x canmount -d z In the case of a non-inheritable size property like quota, the default value "none" is set recursively (since it makes sense to do so), so '-x quota' is the same as '-o quota=none' for non-incrementals. In the case of 'receive -x volsize' where there is no default value, the command fails with an error message unless the send stream is an incremental update. In the case of an incremental update, 'zfs receive -x' does nothing if the received property is already overridden by explicit inheritance or a local setting. Checking for an existing setting and updating the property is atomic. If the property is not present in the send stream, -x does nothing. C.1.4. Uneditable, set-once, and special properties Specifying an uneditable property with 'receive -o' or 'receive -x' fails the command and prints an error message. Even set-once properties normally settable by 'zfs create -o' fail with an error message when specified with 'zfs receive -o' because they are bound to the sent data. These include normalization casesensitivity utf8only volblocksize A set-once property that is independent of the sent data might be specifiable with 'receive -o' or 'receive -x'. The following property is editable, but modifications to the property only affect subsequent writes, not subsequent receives: recordsize Specifying recordsize with 'receive -o' or 'receive -x' (default is 128K) succeeds without a warning message and has no effect on received data. C.2. Version Compatibility To get the full benefit of the options described in this case, the ZFS pool needs to be at version 22 (received properties) or later. Before that, the following options behave differently: receive -o If the specified property is present in the send stream, it is replaced by the value specified after the -o option, since the sent value cannot be overridden locally. receive -x If the specified property is present in the send stream, it is simply ignored, since it cannot be overridden locally. send -b This option is ignored. Received properties cannot be sent in favor of local settings because the two cannot be distinguished. All datasets appear to have never been received and their settings are sent as original properties. Otherwise the options described in this case work the same on all pool versions. C.3. Manpage diffs C.3.1. zfs receive < zfs receive [-vnFu] filesystem|volume|snapshot > zfs receive [-vnFu] [[-o property=value] | [-x property]] ... > filesystem|volume|snapshot < zfs receive [-vnFu] [-d | -e] filesystem > zfs receive [-vnFu] [[-o property=value] | [-x property]] ... > [-d | -e] filesystem < zfs receive [-vnFu] filesystem|volume|snapshot < zfs receive [-vnFu] [-d | -e] filesystem > zfs receive [-vnFu] [[-o property=value] | [-x property]] ... > filesystem|volume|snapshot > zfs receive [-vnFu] [[-o property=value] | [-x property]] ... > [-d | -e] filesystem ... When a snapshot replication package stream that is gen- erated by using the zfs send -R command is received, any snapshots that do not exist on the sending location are < destroyed by using the zfs destroy -d command. > destroyed by using the zfs destroy -d command. If -o > property=value or -x property is specified, it applies to the > effective value of the property throughout the entire subtree > of replicated datasets. Effective property values may be set or > inherited, depending on the property and whether the dataset is > the topmost in the replicated subtree. Received properties are > retained in spite of being overridden and may be restored with > zfs inherit -rS or zfs send -Rb. ... > -o property=value > > Sets the specified property as if the command zfs > set property=value is invoked at the same time the > received dataset is created from the non-incremental send > stream or updated from the incremental send stream. Any > editable ZFS property can also be set at receive > time. Set-once properties bound to the received data, such > as normalization and casesensitivity, cannot be set at > receive time even when the datasets are newly created by > zfs receive. Multiple -o options can be specified. An > error results if the same property is specified in > multiple -o or -x options. > > -x property > > Ensures that the effective value of the specified property > after the receive is unaffected by the value of that > property in the send stream (if any), as if the property > had been excluded from the send stream. If the specified > property is not present in the send stream, this option > does nothing. If a received property needs to be > overridden, the effective value may be set or inherited, > depending on the property. In the case of an incremental > update, -x leaves any existing local setting or explicit > inheritance unchanged (since the received property is > already overridden). All -o restrictions apply equally to > -x. C.3.2. zfs send < zfs send [-vR] [-[iI] snapshot] snapshot > zfs send [-vRb] [-[iI] snapshot] snapshot < zfs send [-vR] [-[iI] snapshot] snapshot > zfs send [-vRb] [-[iI] snapshot] snapshot ... > -b > > Sends only received property values whether or not they > are overridden by local settings, but only if the dataset > has ever been received. Use this option when you want 'zfs > receive' to restore received properties backed up on the > sent dataset and to avoid sending local settings that may > have nothing to do with the source dataset, but only with > how the data is backed up. C.3.3. zfs set < zfs set property=value filesystem|volume|snapshot ... > zfs set [-r] property=value filesystem|volume|snapshot ... < zfs set property=value filesystem|volume|snapshot ... > zfs set [-r] property=value filesystem|volume|snapshot ... ... > -r > > Recursively apply the effective value of the setting > throughout the subtree of child datasets. The effective > value may be set or inherited, depending on the property. C.3.4. Native Properties: recordsize Changing the file system's recordsize affects only files < created afterward; existing files are unaffected. > created afterward; existing files and received data are > unaffected. Stability This case requests patch/micro release binding. The new interfaces are committed. 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From thomas.erickson@Oracle.com Wed May 26 17:04:14 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o4R04E7O013286 for ; Wed, 26 May 2010 17:04:14 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o4R04D5m000563; Wed, 26 May 2010 17:04:14 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L310003LXJ1VN00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 26 May 2010 17:04:13 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L3100JX7XJ0VDD0@nwk-avmta-1.sfbay.Sun.COM>; Wed, 26 May 2010 17:04:13 -0700 (PDT) Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o4R04BnF004599; Thu, 27 May 2010 00:04:11 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4R049gA017691; Thu, 27 May 2010 00:04:09 +0000 (GMT) Received: from abhmt019.oracle.com by acsmt354.oracle.com with ESMTP id 301126241274918629; Wed, 26 May 2010 17:03:49 -0700 Received: from [129.146.229.245] (/129.146.229.245) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 26 May 2010 17:03:47 -0700 Date: Wed, 26 May 2010 17:03:48 -0700 From: Tom Erickson Subject: Re: ZFS backup options [PSARC/2010/193 FastTrack timeout 06/02/2010] In-reply-to: <201005252030.o4PKUt0t002782@localhost.sfbay.sun.com> To: PSARC-ext@sun.com Cc: Matthew Ahrens , zfs-eng@sun.com, Dave Pacheco , janice.chang@Oracle.com, John Harres Message-id: <4BFDB6E4.8010701@oracle.com> MIME-version: 1.0 Content-type: multipart/mixed; boundary="Boundary_(ID_zlLlXrC0VklajEoAI7CaDg)" X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090207.4BFDB6FB.00B6:SCFMA4539814,ss=1,fgs=0 References: <201005252030.o4PKUt0t002782@localhost.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.21 (X11/20090323) Status: RO Content-Length: 29238 This is a multi-part message in MIME format. --Boundary_(ID_zlLlXrC0VklajEoAI7CaDg) Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT I attached an updated case with the following changes: Dave Pacheco wrote: > In section A, paragraph 4: I still find the terminology of saying that > "zfs send -b" is used to "restore properties" from the original > dataset confusing, since it's really just generating a stream that > will later be restored. For customers who understand that a NDMP > backup = "zfs send" and NDMP restore = "zfs restore" (which it is), > this is an important distinction. I would just change "restore" to > "backup" in that sentence. > I Changed that to "This case also adds a -b (restore from backup) option to the 'zfs send' command to generate a stream that, when received, restores properties from the original dataset ... " > The header of section C.1.1 is similarly confusing. > I changed that from C.1.1. Backup with 'receive -o', restore with 'send -b' to C.1.1. 'zfs receive -o' and 'zfs send -b' > Section B.1: For clarification in the first sentence, you might say > "_subsequent_ incremental 'zfs send' streams". > Done. > You mention the "-t" aliases in sections A, B, and the overview in C, > but not in the detailed solutions part of C or the manpage diffs. > Nice catch. I meant to include that in the manpage diffs, but I had nothing further to say about them in the detailed solutions part of C. C.3.5. zfs list -t -t type A comma-separated list of types to display, where type is one of filesystem, snapshot , volume, or all. For example, specifying -t snapshot displays - only snapshots. + only snapshots. The following aliases can be used in + place of the type specifiers: fs (filesystem), snap + (snapshot), and vol (volume). Thanks, Tom --Boundary_(ID_zlLlXrC0VklajEoAI7CaDg) Content-type: text/plain; name=initprops.PSARC Content-transfer-encoding: 7BIT Content-disposition: inline; filename=initprops.PSARC Template Version: @(#)sac_nextcase 1.70 03/30/10 SMI This information is Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. 1. Introduction 1.1. Project/Component Working Name: ZFS backup options 1.2. Name of Document Author/Supplier: Author: Thomas Erickson 1.3 Date of This Document: 26 May, 2010 4. Technical Description ZFS backup options A. SUMMARY Dataset properties can interfere with the use of 'zfs send' and 'zfs receive' as a backup solution because the properties of the original dataset may not work in the context of the backup dataset, or they may not be the right settings in terms of how you want to back up your data. Local properties on the backup dataset also interfere with restores, since there is no way to tell 'zfs send' to send the properties received from the original rather than local settings that may apply only to the backup. This case adds a -o option to the 'zfs receive' command that works like the existing 'zfs create -o property=value' to specify the initial properties of a newly created dataset, making it possible to override properties received from a non-incremental 'zfs send' stream. Similar to -o, a -x option for 'zfs receive' is added to override received properties without specifying a value, simply to ensure that they have no effect on the behavior of the received dataset (as if they had been excluded from the send stream), even though the received values can still be restored. This case also adds a -b (restore from backup) option to the 'zfs send' command to generate a stream that, when received, restores properties from the original dataset rather than local settings specified on the backup dataset with 'zfs set' or 'zfs receive -o'. For the convenience of related administrator actions, this case adds a -r option to 'zfs set' similar to 'zfs inherit -r' and new aliases for use with 'zfs list -t'. B. PROBLEM B.1. Backup Today we can set properties locally with 'zfs set' to override properties received from subsequent incremental 'zfs send' streams, but this is no help with non-incremental streams because there is no opportunity to run 'zfs set' on a dataset before it is newly created by 'zfs receive'. This is particularly awkward when a sent dataset has a non-default mountpoint, since the receive has no place to replicate the sent data when it fails to mount the received dataset on a directory that may not exist or that is already in use by the sent dataset. For example: % zfs create -o mountpoint=/tank/mnt tank/fs/mnt % zfs snapshot tank/fs/mnt@1 % zfs create tank/foo % zfs send -R tank/fs/mnt@1 | zfs recv -d tank/foo cannot mount 'tank/foo/fs/mnt': mountpoint or dataset is busy % Similar problems can interfere with the use of send/receive as a backup solution when there is a property in the source dataset whose validity on the target depends on inherited settings different from those of the source. For example, the following interference from the source quota prevents a successful receive when the target inherits copies differently: % zfs create tank/a % zfs create tank/z % zfs set copies=3 tank/z % zfs create tank/a/b % zfs set quota=2m tank/a/b % mkfile 1m /tank/a/b/file % zfs snapshot tank/a/b@1 % zfs send -R tank/a/b@1 | zfs recv -e tank/z cannot receive new filesystem stream: destination tank/z space quota exceeded % The lack of a receive option to set initial property values also hinders administrators from applying settings like compression and deduplication to backups, since local settings that differ from those in the send stream can only be applied after the fact and therefore only apply to incremental updates. Sometimes the administrator has a value in mind to override the received property, but other times the administrator may simply want the default. In that case we don't want to task the administrator with finding out the default in order to set it explicitly, so an option to override a received property without providing an explicit value is more helpful (and preserves inheritance). B.2. Restore Restoring properties from a backup is also troublesome because 'zfs send' assumes that the effective property values are the ones to include in the send stream (if specified on the sent dataset and not inherited). Since local settings override received properties, the administrator will not get back the original settings received on the backup dataset because 'zfs send' will choose the effective local settings instead. Local settings on the backup likely specify how the administrator wants to store backups and have nothing to do with the restored dataset being like the original. To clarify the problem of restoring properties from a backup, it helps to think of the effective value of a property is the topmost value in the following stack: local received --------- inherited default Each level of the stack overrides the one below it. If a property does not have a specified value at the topmost level, ZFS resorts to the value at the next lower level, if it exists, and so on. (The default value always exists.) The first value it finds determines the effective value of the property. This is how ZFS already works and this case does nothing to change that. Values above the dashed line (local and received) are explicit and included in a 'zfs send' stream unless overridden. Values below the dashed line (inherited and default) are implicit and never included in a 'zfs send' stream. If you say nothing about a dataset property, neither by setting it nor by receiving it, your dataset inherits the property from its parent. If the parent does not have a set or received value and neither do any of its ancestors, then your dataset inherits the default. Inheriting properties is the way to get the behavior you want without including those settings in a 'zfs send' stream. Again, this case does not change that. The reason that sending the effective value is a problem for backups is best understood by considering the outcome of a replication chain. Suppose that we send from dataset A and receive to dataset B, then send from dataset B and receive to dataset C. Clearly we want A->B to replicate the effective properties of A, but what about B->C? If B overrides a property locally, should C get the original value from A or the local value on B? Currently 'zfs send' can only replicate the effective properties of B. It cannot pass along the original properties from A without interference from local settings on B, thereby finishing the relay of A->B->C with the same properties it had at the start. The semantics of restoring from a backup are not possible unless you can get back the properties you started with. B.3. Set recursively In the case of a recursive 'zfs send' stream, the option to set the initial values of specific properties needs to apply those properties recursively. For consistency and convenience, 'zfs set' should also support this ability to apply a local setting recursively so it can override a property received in a recursive incremental update. B.4. Query It's inconvenient to spell out the full names of the dataset types "filesystem", "volume", and "snapshot" when querying received datasets with 'zfs list -t'. C. PROPOSED SOLUTION C.1. Overview The proposed solution adds a -o option to 'zfs receive' to specify the initial values of any number of properties as '-o property=value', just like we do with 'zfs create'. To avoid the unwanted effects of a property in a 'zfs send' stream without specifying a value for that property, a similar '-x property' option overrides the named property in a way that preserves the received value for a possible future restore (or inherit -S) but has no effect on the effective value of the property on the received dataset, as if the named property had been excluded from the send stream. Both of these receive options, -o and -x, work with recursive send streams by overriding the received property recursively. For convenience, they also work with incremental streams, even though an administrator can use 'zfs set' and/or 'zfs inherit' to override received properties before receiving an incremental update. For consistency with 'zfs inherit -r' as well as the -o and -x receive options, a '-r' option added to 'zfs set' makes it possible to apply a property setting recursively before receiving a recursive, incremental update. A new -b option added to 'zfs send' specifies received rather than effective property values to be included in the send stream, making it possible to restore the original, backed-up settings without interference from local settings. Finally, the proposed solution adds the type aliases "fs" for "filesystem", "snap" for "snapshot", and "vol" for "volume" to use with 'zfs list -t' when querying datasets. C.1.1. 'zfs receive -o' and 'zfs send -b' To support the use of send/receive as a backup solution, ZFS will set a property's local value to that specified by receive -o, whereas its received value (overridden by the local value) will be set to the value obtained from the send stream (if any). For example: % zfs set compression=off a/fs % zfs snapshot a/fs@1 % zfs send -R a/fs@1 | zfs recv -o compression=on -d z % zfs get -o all compression z/fs NAME PROPERTY VALUE RECEIVED SOURCE z/fs compression on off local % This way, the backup dataset created by 'zfs receive' can use whatever settings the administrator specifies, but the original settings are not lost and can still be restored from the backup. To do that requires a new -b option for 'zfs send' that tells ZFS to send received properties regardless of local settings. This is not the default 'zfs send' behavior because it might be confusing to send property values that are not currently effective on the source dataset. The following table describes what 'zfs send -b' actually sends compared to 'zfs send' without -b, depending on whether the source property has a local value, a received value, both, or neither: local received | sent with -b sent without -b ----- -------- | ----------------- --------------- yes no | nothing local value no yes | received value received value yes yes | received value local value no no | nothing nothing An internal "$hasrecvd" property tracks whether a dataset was received on or after pool version 22 (received properties). If that internal property is not found, 'zfs send' treats the dataset as the start of a replication chain by sending its effective property values, as if the dataset was sent without -b. This behavior is consistent in the sense that 'send -b' always preserves original properties. In the replication chain A->B->C, the initial replication A->B is a special case in which the original properties are the effective and not the received properties, so the column under "sent without -b" in the above table applies. The alternative was to send no properties, since there are no received properties if the dataset has never been received and this is the initial replication. While that may be the more consistent behavior, it lacks utility. The chosen behavior potentially frees backup applications from deciding when to use -b since they can use -b by default and get the desired behavior in both the A->B and B->C cases. In the case where a received dataset needs to become the start of a new replication chain, that can be indicated by a user property. C.1.2. Recursive streams and 'zfs set -r' If the sent stream is recursive (ie 'zfs send -R'); that is, it includes one or more levels of descendant datasets, then 'zfs recv -o property=value' ensures that the effective value of the property is the same in all received datasets. It does this in the most sensible way it can, depending on the property. If the property is inheritable, it sets the named property to the specified value only on the top level dataset and inherits the named property on all the datasets below that. That way, all received datasets get the setting from one place. The received values, however, reflect the settings of the source datasets. For example: % zfs create a/b % zfs create a/b/c % zfs create a/b/c/d % zfs set compression=off a/b % zfs set compression=gzip a/b/c/d % zfs get compression a/b a/b/c a/b/c/d NAME PROPERTY VALUE SOURCE a/b compression off local a/b/c compression off inherited from a/b a/b/c/d compression gzip local % zfs snapshot -r a/b@1 % zfs send -R a/b@1 | zfs recv -o compression=on -d z % zfs get -o all compression z/b z/b/c z/b/c/d NAME PROPERTY VALUE RECEIVED SOURCE z/b compression on off local z/b/c compression on - inherited from z/b z/b/c/d compression on gzip inherited from z/b % Note that the value "gzip" in the RECEIVED column for z/b/c/d reflects the local setting of the source dataset, a/b/c/d. If the property is not inheritable, 'zfs recv -o property=value' instead sets the property recursively. For example, if we override canmount instead of compression: % zfs create a/b % zfs create a/b/c % zfs create a/b/c/d % zfs set canmount=off a/b % zfs set canmount=noauto a/b/c/d % zfs get canmount a/b a/b/c a/b/c/d NAME PROPERTY VALUE SOURCE a/b canmount off local a/b/c canmount on default a/b/c/d canmount noauto local % zfs snapshot -r a/b@1 % zfs send -R a/b@1 | zfs recv -o canmount=on -d z % zfs get -o all canmount z/b z/b/c z/b/c/d NAME PROPERTY VALUE RECEIVED SOURCE z/b canmount on off local z/b/c canmount on - local z/b/c/d canmount on noauto local % The new 'zfs set -r' option uses the same algorithm to make the setting effective throughout the subtree of child datasets. If the property is inheritable, it sets the property on the top-level dataset only and inherits the setting the rest of the way down the subtree. In the above example involving compression, receiving without -o and correcting after the fact with 'zfs set -r compression=on z/b' would still result in the same property values: % zfs send -R a/b@1 | zfs recv -d z % zfs set -r compression=on z/b % zfs get -o all compression z/b z/b/c z/b/c/d NAME PROPERTY VALUE RECEIVED SOURCE z/b compression on off local z/b/c compression on - inherited from z/b z/b/c/d compression on gzip inherited from z/b % The difference is that in the case of 'zfs recv -o compression=on' the initial received data is actually compressed, whereas 'set -r' only applies to future updates. Without -o to specify compression, 'zfs receive' uses the received property values ("off" and "gzip" in this case) to determine how the initial receive should or should not compress data: % zfs send -R a/b@1 | zfs recv -d z % zfs get -o all compression z/b z/b/c z/b/c/d NAME PROPERTY VALUE RECEIVED SOURCE z/b compression off off received z/b/c compression off - inherited from z/b z/b/c/d compression gzip gzip received % That's how the data is actually received before compression is overridden locally with 'set -r'. Either way, the final outcome is exactly the same as if the administrator had typed the commands % zfs inherit -r compression z/b % zfs set compression=on z/b If the property is not inheritable, as in the example involving canmount above, then 'zfs set -r' sets the property all the way down the subtree rooted at the specified dataset. A subset of non-inheritable properties, like quota and reservation, take a size value. In that case it does not make sense to set the property recursively, since the size already applies to the entire subtree, so 'zfs receive -o' would only set the property on the top-level dataset. For example: % zfs create a/b % zfs create a/b/c % zfs create a/b/c/d % zfs set quota=1g a/b % zfs set quota=1m a/b/c/d % zfs get quota a/b a/b/c a/b/c/d NAME PROPERTY VALUE SOURCE a/b quota 1G local a/b/c quota none default a/b/c/d quota 1M local % zfs snapshot -r a/b@1 % zfs send -R a/b@1 | zfs recv -o quota=5g -d z % zfs get -o all quota z/b z/b/c z/b/c/d NAME PROPERTY VALUE RECEIVED SOURCE z/b quota 5G 1G local z/b/c quota none - default z/b/c/d quota 1M 1M received % However, in the case of overriding received quota with quota=none, it does make sense to set the property recursively, so the above example turns out somewhat differently: % zfs send -R a/b@1 | zfs recv -o quota=none -d z % zfs get -o all quota z/b z/b/c z/b/c/d NAME PROPERTY VALUE RECEIVED SOURCE z/b quota none 1G local z/b/c quota none - local z/b/c/d quota none 1M local % 'zfs set -r' treats non-inheritable size properties like quota in exactly the same way. Regardless of the property, 'zfs receive -o' of a recursive stream and 'zfs set -r' both use the same algorithm to do what makes the most sense depending on the property. The only potential difference is when a snapshot is missing from the send stream. In that case, 'zfs send' prints a warning about the missing snapshot and 'zfs receive -o' does not update the property in the target dataset corresponding to the missing snapshot. In the normal case, however, 'zfs receive -o' visits the entire subtree of child datasets and the outcome (as far as properties are concerned) is the same as that of 'zfs set -r'. C.1.3. 'zfs receive -x' The idea of 'receive -x' is to ensure that the received property does not determine the effective value of the property on the received dataset; that is, to get the behavior that you would have gotten if the property had been excluded from the send stream, and to do that recursively in the case of a recursive stream. How it accomplishes that depends on the property. If the property is inheritable, 'receive -x' explicitly inherits the property to override the received value. The effect is the same as 'zfs inherit -r' if the send stream is recursive, for example: % zfs create a/b % zfs create a/b/c % zfs create a/b/c/d % zfs set compression=off a/b % zfs set compression=gzip a/b/c/d % zfs get compression a/b a/b/c a/b/c/d NAME PROPERTY VALUE SOURCE a/b compression off local a/b/c compression off inherited from a/b a/b/c/d compression gzip local % zfs snapshot -r a/b@1 % zfs send -R a/b@1 | zfs recv -x compression -d z % zfs get -o all compression z/b z/b/c z/b/c/d NAME PROPERTY VALUE RECEIVED SOURCE z/b compression off off default z/b/c compression off - default z/b/c/d compression off gzip default % Explicit inheritance causes ZFS to ignore the received value and resort to the parent if there is no local setting. In this case, since the parent pool z does not specify a value, all the children inherit the default. If the property is not inheritable, 'receive -x' sets the default value recursively. Since the default canmount value happens to be "on", the canmount example in C.1.2 above turns out the same when we replace % zfs send -R a/b@1 | zfs recv -o canmount=on -d z with % zfs send -R a/b@1 | zfs recv -x canmount -d z In the case of a non-inheritable size property like quota, the default value "none" is set recursively (since it makes sense to do so), so '-x quota' is the same as '-o quota=none' for non-incrementals. In the case of 'receive -x volsize' where there is no default value, the command fails with an error message unless the send stream is an incremental update. In the case of an incremental update, 'zfs receive -x' does nothing if the received property is already overridden by explicit inheritance or a local setting. Checking for an existing setting and updating the property is atomic. If the property is not present in the send stream, -x does nothing. C.1.4. Uneditable, set-once, and special properties Specifying an uneditable property with 'receive -o' or 'receive -x' fails the command and prints an error message. Even set-once properties normally settable by 'zfs create -o' fail with an error message when specified with 'zfs receive -o' because they are bound to the sent data. These include normalization casesensitivity utf8only volblocksize A set-once property that is independent of the sent data might be specifiable with 'receive -o' or 'receive -x'. The following property is editable, but modifications to the property only affect subsequent writes, not subsequent receives: recordsize Specifying recordsize with 'receive -o' or 'receive -x' (default is 128K) succeeds without a warning message and has no effect on received data. C.2. Version Compatibility To get the full benefit of the options described in this case, the ZFS pool needs to be at version 22 (received properties) or later. Before that, the following options behave differently: receive -o If the specified property is present in the send stream, it is replaced by the value specified after the -o option, since the sent value cannot be overridden locally. receive -x If the specified property is present in the send stream, it is simply ignored, since it cannot be overridden locally. send -b This option is ignored. Received properties cannot be sent in favor of local settings because the two cannot be distinguished. All datasets appear to have never been received and their settings are sent as original properties. Otherwise the options described in this case work the same on all pool versions. C.3. Manpage diffs C.3.1. zfs receive < zfs receive [-vnFu] filesystem|volume|snapshot > zfs receive [-vnFu] [[-o property=value] | [-x property]] ... > filesystem|volume|snapshot < zfs receive [-vnFu] [-d | -e] filesystem > zfs receive [-vnFu] [[-o property=value] | [-x property]] ... > [-d | -e] filesystem < zfs receive [-vnFu] filesystem|volume|snapshot < zfs receive [-vnFu] [-d | -e] filesystem > zfs receive [-vnFu] [[-o property=value] | [-x property]] ... > filesystem|volume|snapshot > zfs receive [-vnFu] [[-o property=value] | [-x property]] ... > [-d | -e] filesystem ... When a snapshot replication package stream that is gen- erated by using the zfs send -R command is received, any snapshots that do not exist on the sending location are < destroyed by using the zfs destroy -d command. > destroyed by using the zfs destroy -d command. If -o > property=value or -x property is specified, it applies to the > effective value of the property throughout the entire subtree > of replicated datasets. Effective property values may be set or > inherited, depending on the property and whether the dataset is > the topmost in the replicated subtree. Received properties are > retained in spite of being overridden and may be restored with > zfs inherit -rS or zfs send -Rb. ... > -o property=value > > Sets the specified property as if the command zfs > set property=value is invoked at the same time the > received dataset is created from the non-incremental send > stream or updated from the incremental send stream. Any > editable ZFS property can also be set at receive > time. Set-once properties bound to the received data, such > as normalization and casesensitivity, cannot be set at > receive time even when the datasets are newly created by > zfs receive. Multiple -o options can be specified. An > error results if the same property is specified in > multiple -o or -x options. > > -x property > > Ensures that the effective value of the specified property > after the receive is unaffected by the value of that > property in the send stream (if any), as if the property > had been excluded from the send stream. If the specified > property is not present in the send stream, this option > does nothing. If a received property needs to be > overridden, the effective value may be set or inherited, > depending on the property. In the case of an incremental > update, -x leaves any existing local setting or explicit > inheritance unchanged (since the received property is > already overridden). All -o restrictions apply equally to > -x. C.3.2. zfs send < zfs send [-vR] [-[iI] snapshot] snapshot > zfs send [-vRb] [-[iI] snapshot] snapshot < zfs send [-vR] [-[iI] snapshot] snapshot > zfs send [-vRb] [-[iI] snapshot] snapshot ... > -b > > Sends only received property values whether or not they > are overridden by local settings, but only if the dataset > has ever been received. Use this option when you want 'zfs > receive' to restore received properties backed up on the > sent dataset and to avoid sending local settings that may > have nothing to do with the source dataset, but only with > how the data is backed up. C.3.3. zfs set < zfs set property=value filesystem|volume|snapshot ... > zfs set [-r] property=value filesystem|volume|snapshot ... < zfs set property=value filesystem|volume|snapshot ... > zfs set [-r] property=value filesystem|volume|snapshot ... ... > -r > > Recursively apply the effective value of the setting > throughout the subtree of child datasets. The effective > value may be set or inherited, depending on the property. C.3.4. Native Properties: recordsize Changing the file system's recordsize affects only files < created afterward; existing files are unaffected. > created afterward; existing files and received data are > unaffected. C.3.5. zfs list -t -t type A comma-separated list of types to display, where type is one of filesystem, snapshot , volume, or all. For example, specifying -t snapshot displays < only snapshots. > only snapshots. The following aliases can be used in > place of the type specifiers: fs (filesystem), snap > (snapshot), and vol (volume). Stability This case requests patch/micro release binding. The new interfaces are committed. 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open --Boundary_(ID_zlLlXrC0VklajEoAI7CaDg)-- From matthew.ahrens@oracle.com Wed Jun 2 09:30:11 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o52GUBie005234 for ; Wed, 2 Jun 2010 09:30:11 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o52GUAwE006418; Wed, 2 Jun 2010 11:30:10 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L3E00711B6AOD00@nwk-avmta-2.sfbay.sun.com>; Wed, 02 Jun 2010 09:30:10 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L3E006PMB69BM50@nwk-avmta-2.sfbay.sun.com>; Wed, 02 Jun 2010 09:30:09 -0700 (PDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o52GU8Qh001465; Wed, 02 Jun 2010 16:30:08 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o52F4f5w007619; Wed, 02 Jun 2010 16:30:05 +0000 (GMT) Received: from abhmt004.oracle.com by acsmt354.oracle.com with ESMTP id 316641321275496152; Wed, 02 Jun 2010 09:29:12 -0700 Received: from matthew-ahrenss-macbook-pro.local (/10.7.251.178) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 02 Jun 2010 09:29:11 -0700 Date: Wed, 02 Jun 2010 09:29:09 -0700 From: Matthew Ahrens Subject: Re: ZFS backup options [PSARC/2010/193 FastTrack timeout 06/02/2010] In-reply-to: <4BFDB6E4.8010701@oracle.com> To: Tom Erickson Cc: PSARC-ext@sun.com, Matthew Ahrens , zfs-eng@sun.com, Dave Pacheco , janice.chang@oracle.com, John Harres Message-id: <4C0686D5.5020403@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090205.4C068710.0021:SCFMA4539814,ss=1,fgs=0 References: <201005252030.o4PKUt0t002782@localhost.sfbay.sun.com> <4BFDB6E4.8010701@oracle.com> User-Agent: Thunderbird 2.0.0.24 (Macintosh/20100228) Status: RO Content-Length: 1934 Can I get a +1 on this? --matt Tom Erickson wrote: > I attached an updated case with the following changes: > > Dave Pacheco wrote: > > In section A, paragraph 4: I still find the terminology of saying that > > "zfs send -b" is used to "restore properties" from the original > > dataset confusing, since it's really just generating a stream that > > will later be restored. For customers who understand that a NDMP > > backup = "zfs send" and NDMP restore = "zfs restore" (which it is), > > this is an important distinction. I would just change "restore" to > > "backup" in that sentence. > > > > I Changed that to > > "This case also adds a -b (restore from backup) option to the 'zfs > send' > command to generate a stream that, when received, restores properties > from the original dataset ... " > > > The header of section C.1.1 is similarly confusing. > > > > I changed that from > C.1.1. Backup with 'receive -o', restore with 'send -b' > to > C.1.1. 'zfs receive -o' and 'zfs send -b' > > > > Section B.1: For clarification in the first sentence, you might say > > "_subsequent_ incremental 'zfs send' streams". > > > > Done. > > > You mention the "-t" aliases in sections A, B, and the overview in C, > > but not in the detailed solutions part of C or the manpage diffs. > > > > Nice catch. I meant to include that in the manpage diffs, but I had > nothing further to say about them in the detailed solutions part of C. > > C.3.5. zfs list -t > > -t type > > A comma-separated list of types to display, where > type is one of filesystem, snapshot , volume, or > all. For example, specifying -t snapshot displays > - only snapshots. > + only snapshots. The following aliases can be used in > + place of the type specifiers: fs (filesystem), snap > + (snapshot), and vol (volume). > > Thanks, > Tom From sebastien.roy@oracle.com Wed Jun 2 09:51:07 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o52Gp7Z5006549 for ; Wed, 2 Jun 2010 09:51:07 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o52Gp3RE016619; Wed, 2 Jun 2010 11:51:07 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L3E0080ZC56N800@nwk-avmta-2.sfbay.sun.com>; Wed, 02 Jun 2010 09:51:06 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L3E006YUC55BM90@nwk-avmta-2.sfbay.sun.com>; Wed, 02 Jun 2010 09:51:06 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o52Gp4jb021404; Wed, 02 Jun 2010 16:51:05 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o52G1Ew1013849; Wed, 02 Jun 2010 16:51:03 +0000 (GMT) Received: from abhmt017.oracle.com by acsmt353.oracle.com with ESMTP id 290853861275497377; Wed, 02 Jun 2010 09:49:37 -0700 Received: from [129.148.19.4] (/129.148.19.4) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 02 Jun 2010 09:49:36 -0700 Date: Wed, 02 Jun 2010 12:49:34 -0400 From: Sebastien Roy Subject: Re: ZFS backup options [PSARC/2010/193 FastTrack timeout 06/02/2010] In-reply-to: <4C0686D5.5020403@oracle.com> To: Matthew Ahrens Cc: Tom Erickson , zfs-eng@sun.com, janice.chang@oracle.com, Matthew Ahrens , PSARC-ext@sun.com, John Harres , Dave Pacheco Message-id: <4C068B9E.9080105@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090202.4C068BF8.00E4:SCFMA4539814,ss=1,fgs=0 References: <201005252030.o4PKUt0t002782@localhost.sfbay.sun.com> <4BFDB6E4.8010701@oracle.com> <4C0686D5.5020403@oracle.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.9) Gecko/20100510 Lightning/1.0b1 Thunderbird/3.0.4 Status: RO Content-Length: 1176 I only have one comment regarding the zfs pool version handling: > C.2. Version Compatibility > > To get the full benefit of the options described in this case, the ZFS > pool needs to be at version 22 (received properties) or later. Before > that, the following options behave differently: > > receive -o If the specified property is present in the send stream, > it is replaced by the value specified after the -o option, > since the sent value cannot be overridden locally. > > receive -x If the specified property is present in the send stream, > it is simply ignored, since it cannot be overridden > locally. > > send -b This option is ignored. Received properties cannot be sent > in favor of local settings because the two cannot be > distinguished. All datasets appear to have never been > received and their settings are sent as original > properties. Is there not a concern that the handling of the latter two usages will lead to surprising results if the administrator is not immediately aware of the version of the appropriate pool? Would failing these operations be a viable option? -Seb From thomas.erickson@oracle.com Wed Jun 2 13:09:03 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o52K920d016574 for ; Wed, 2 Jun 2010 13:09:02 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o52K91tI000882; Wed, 2 Jun 2010 13:09:02 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L3E00B15LB2GG00@brm-avmta-1.central.sun.com>; Wed, 02 Jun 2010 14:09:02 -0600 (MDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L3E00155LB1SS40@brm-avmta-1.central.sun.com>; Wed, 02 Jun 2010 14:09:01 -0600 (MDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o52K90b6023956; Wed, 02 Jun 2010 20:09:00 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o52JaNve019907; Wed, 02 Jun 2010 20:08:59 +0000 (GMT) Received: from abhmt018.oracle.com by acsmt355.oracle.com with ESMTP id 291438401275509320; Wed, 02 Jun 2010 13:08:40 -0700 Received: from [10.7.250.186] (/10.7.250.186) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 02 Jun 2010 13:08:39 -0700 Date: Wed, 02 Jun 2010 13:08:33 -0700 From: Tom Erickson Subject: Re: ZFS backup options [PSARC/2010/193 FastTrack timeout 06/02/2010] In-reply-to: <4C068B9E.9080105@oracle.com> To: Sebastien Roy Cc: Matthew Ahrens , zfs-eng@sun.com, janice.chang@oracle.com, Matthew Ahrens , PSARC-ext@sun.com, John Harres , Dave Pacheco Message-id: <4C06BA41.9060103@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090206.4C06BA5C.002E:SCFMA4539814,ss=1,fgs=0 References: <201005252030.o4PKUt0t002782@localhost.sfbay.sun.com> <4BFDB6E4.8010701@oracle.com> <4C0686D5.5020403@oracle.com> <4C068B9E.9080105@oracle.com> User-Agent: Thunderbird 2.0.0.21 (X11/20090623) Status: RO Content-Length: 2143 Sebastien Roy wrote: > I only have one comment regarding the zfs pool version handling: > >> C.2. Version Compatibility >> >> To get the full benefit of the options described in this case, the ZFS >> pool needs to be at version 22 (received properties) or later. Before >> that, the following options behave differently: >> >> receive -o If the specified property is present in the send stream, >> it is replaced by the value specified after the -o option, >> since the sent value cannot be overridden locally. >> >> receive -x If the specified property is present in the send stream, >> it is simply ignored, since it cannot be overridden >> locally. >> >> send -b This option is ignored. Received properties cannot be sent >> in favor of local settings because the two cannot be >> distinguished. All datasets appear to have never been >> received and their settings are sent as original >> properties. > > Is there not a concern that the handling of the latter two usages will > lead to surprising results if the administrator is not immediately aware > of the version of the appropriate pool? Would failing these operations > be a viable option? > In the 'receive -x' case, the behavior is not surprising. On all pool versions, -x says treat the specified property as if it had been excluded from the send stream. The only difference is that before version 22, the received property value is not saved, but it would not have affected the behavior of the dataset anyway. The effective behavior resulting from receive -x is the same on all pool versions. Since -x is useful on all pool versions, I don't think it makes sense to fail the command before version 22 (that would be more surprising). In the 'send -b' case, the -b doesn't do anything useful before version 22. The option assumes that we can distinguish received properties, so I think it makes sense to fail the command before version 22. If no one objects, I'll go with Sebastian's suggestion and fail 'send -b' with an error message before version 22. Thanks, Tom From sebastien.roy@oracle.com Wed Jun 2 13:32:42 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o52KWgpJ017231 for ; Wed, 2 Jun 2010 13:32:42 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o52KWfUT011207; Wed, 2 Jun 2010 14:32:41 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L3E00701MEHRE00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 02 Jun 2010 13:32:41 -0700 (PDT) Received: from sca-ea-mail-1.sun.com ([192.18.43.24]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L3E00DLUMEHMT60@nwk-avmta-1.sfbay.Sun.COM>; Wed, 02 Jun 2010 13:32:41 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by sca-ea-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o52KWZKd017487; Wed, 02 Jun 2010 20:32:35 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o52KWTTC032294; Wed, 02 Jun 2010 20:32:30 +0000 (GMT) Received: from abhmt008.oracle.com by acsmt355.oracle.com with ESMTP id 291509211275510747; Wed, 02 Jun 2010 13:32:27 -0700 Received: from [129.148.19.4] (/129.148.19.4) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 02 Jun 2010 13:32:26 -0700 Date: Wed, 02 Jun 2010 16:32:24 -0400 From: Sebastien Roy Subject: Re: ZFS backup options [PSARC/2010/193 FastTrack timeout 06/02/2010] In-reply-to: <4C06BA41.9060103@oracle.com> To: Tom Erickson Cc: Matthew Ahrens , zfs-eng@sun.com, janice.chang@oracle.com, Matthew Ahrens , PSARC-ext@sun.com, John Harres , Dave Pacheco Message-id: <4C06BFD8.2070707@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090203.4C06BFE2.00F3:SCFMA4539814,ss=1,fgs=0 References: <201005252030.o4PKUt0t002782@localhost.sfbay.sun.com> <4BFDB6E4.8010701@oracle.com> <4C0686D5.5020403@oracle.com> <4C068B9E.9080105@oracle.com> <4C06BA41.9060103@oracle.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.9) Gecko/20100510 Lightning/1.0b1 Thunderbird/3.0.4 Status: RO Content-Length: 2205 On 06/ 2/10 04:08 PM, Tom Erickson wrote: > Sebastien Roy wrote: >> I only have one comment regarding the zfs pool version handling: >> >>> C.2. Version Compatibility >>> >>> To get the full benefit of the options described in this case, the ZFS >>> pool needs to be at version 22 (received properties) or later. Before >>> that, the following options behave differently: >>> >>> receive -o If the specified property is present in the send stream, >>> it is replaced by the value specified after the -o option, >>> since the sent value cannot be overridden locally. >>> >>> receive -x If the specified property is present in the send stream, >>> it is simply ignored, since it cannot be overridden >>> locally. >>> >>> send -b This option is ignored. Received properties cannot be sent >>> in favor of local settings because the two cannot be >>> distinguished. All datasets appear to have never been >>> received and their settings are sent as original >>> properties. >> >> Is there not a concern that the handling of the latter two usages will >> lead to surprising results if the administrator is not immediately >> aware of the version of the appropriate pool? Would failing these >> operations be a viable option? >> > > In the 'receive -x' case, the behavior is not surprising. On all pool > versions, -x says treat the specified property as if it had been > excluded from the send stream. The only difference is that before > version 22, the received property value is not saved, but it would not > have affected the behavior of the dataset anyway. The effective behavior > resulting from receive -x is the same on all pool versions. Since -x is > useful on all pool versions, I don't think it makes sense to fail the > command before version 22 (that would be more surprising). Okay. > > In the 'send -b' case, the -b doesn't do anything useful before version > 22. The option assumes that we can distinguish received properties, so I > think it makes sense to fail the command before version 22. If no one > objects, I'll go with Sebastian's suggestion and fail 'send -b' with an > error message before version 22. Yes, that makes sense to me. +1 on the case given this clarification. -Seb From matthew.ahrens@oracle.com Wed Jun 2 14:21:04 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o52LL4sJ018618 for ; Wed, 2 Jun 2010 14:21:04 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o52LL3sX036338; Wed, 2 Jun 2010 15:21:03 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L3E00K09ON3KG00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 02 Jun 2010 14:21:03 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L3E00DMOON2MTB0@nwk-avmta-1.sfbay.Sun.COM>; Wed, 02 Jun 2010 14:21:02 -0700 (PDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o52LL1sI027988; Wed, 02 Jun 2010 21:21:01 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o52LKxb6025869; Wed, 02 Jun 2010 21:20:59 +0000 (GMT) Received: from abhmt004.oracle.com by acsmt354.oracle.com with ESMTP id 291643261275513658; Wed, 02 Jun 2010 14:20:58 -0700 Received: from matthew-ahrenss-macbook-pro.local (/10.7.251.178) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 02 Jun 2010 14:20:57 -0700 Date: Wed, 02 Jun 2010 14:20:55 -0700 From: Matthew Ahrens Subject: Re: ZFS backup options [PSARC/2010/193 FastTrack timeout 06/02/2010] In-reply-to: <4C06BFD8.2070707@oracle.com> To: Sebastien Roy Cc: Tom Erickson , zfs-eng@sun.com, janice.chang@oracle.com, Matthew Ahrens , PSARC-ext@sun.com, John Harres , Dave Pacheco Message-id: <4C06CB37.3050008@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090207.4C06CB3D.00F6:SCFMA4539814,ss=1,fgs=0 References: <201005252030.o4PKUt0t002782@localhost.sfbay.sun.com> <4BFDB6E4.8010701@oracle.com> <4C0686D5.5020403@oracle.com> <4C068B9E.9080105@oracle.com> <4C06BA41.9060103@oracle.com> <4C06BFD8.2070707@oracle.com> User-Agent: Thunderbird 2.0.0.24 (Macintosh/20100228) Status: RO Content-Length: 147 Sebastien Roy wrote: > Yes, that makes sense to me. +1 on the case given this clarification. Thanks. I have marked this case approved. --matt