From sacadmin Wed Oct 3 10:33:19 2007 Received: from sfbaymail1sca.SFBay.Sun.COM (sfbaymail1sca [129.145.154.35]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l93HXJ67002997 for ; Wed, 3 Oct 2007 10:33:19 -0700 (PDT) Received: from sca-es-mail-1.sun.com (sca-es-mail-1.Sun.COM [192.18.43.132]) by sfbaymail1sca.SFBay.Sun.COM (8.13.6+Sun/8.13.6/ENSMAIL,v2.2) with ESMTP id l93HU9Kx014996 for ; Wed, 3 Oct 2007 10:30:09 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l93HU4mB010835 for ; Wed, 3 Oct 2007 10:30:04 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPC00B01IRGKZ00@fe-sfbay-09.sun.com> (original mail from John.Plocher@Sun.COM) for psarc-record@sac.sfbay.sun.com; Wed, 03 Oct 2007 10:30:04 -0700 (PDT) Received: from wp668.local ([208.74.177.212]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPC002FWJA1DO00@fe-sfbay-09.sun.com> for psarc-record@sac.sfbay.sun.com; Wed, 03 Oct 2007 10:30:01 -0700 (PDT) Date: Wed, 03 Oct 2007 10:29:54 -0700 From: John Plocher Subject: zfs send -R [PSARC/2007/574 FastTrack timeout 10/10/2007] Sender: John.Plocher@Sun.COM To: psarc-record@sac.sfbay.sun.com Message-id: <4703D192.2080101@Sun.Com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) Status: RO Content-Length: 2859 -------- Original Message -------- Subject: zfs send -R [PSARC/2007/574 FastTrack timeout 10/10/2007] Date: Wed, 03 Oct 2007 10:24:39 -0700 (PDT) From: Matthew Ahrens To: PSARC-EXT@sun.com CC: zfs-eng@sun.com Template Version: @(#)sac_nextcase 1.64 07/13/07 SMI This information is Copyright 2007 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: zfs send -R 1.2. Name of Document Author/Supplier: Author: Matthew Ahrens 1.3 Date of This Document: 03 October, 2007 4. Technical Description This case adds new flags to "zfs send". The stability of the flags are committed, and the release binding is patch/micro. A. INTRODUCTION Currently, "zfs send" can send only a single snapshot at a time. To send all the snapshots of a given filesystem, or snapshots of multiple filesystems, would require multiple "zfs send" commands. Additionally, "zfs send" can not transfer property settings, nor can it replicate namespace changes (eg, snapshot/fs renaming or deletion). This case enhances "zfs send" to implement these features. B. DESCRIPTION This case adds several new capabilities to "zfs send": 1. zfs send -I @snapA pool/fs@snapD Sends all incrementals from snapA to snapD (ie, snapA to snapB, snapB to snapC, and snapC to snapD). 2. zfs send -[iI] pool/fs@origin pool/clone@snap Sends an incremental stream from the origin snapshot to create a clone. (To be received, the origin snapshot must already exist on the receiving end.) 3. zfs send -R pool/fs@snap Sends a "replication" stream, which will replicate pool/fs, and all descendant filesystems, up to the named snap. When received, all properties, snapshots, descendent filesystems, and clones will be preserved. 4. zfs send -R -[iI] @snapA pool/fs@snapD Sends an "incremental replication" stream. Changes to properties, and snapshot and filesystem renames and destroys will be preserved. When receiving, if -F is not specified, then destroys will be ignored. (-F also retains its "rollback if necessary" meaning.) As with other (non-R) -i/-I cases, if -I is used, all snapshots between snapA and snapD will be sent; if -i is used, only snapD (for all descendents) will be sent. Note, to receive any of these new type of "zfs send" streams, the receiving system must be running a software version capable of sending them. Ie, the stream version is incremented. However, no on-disk changes are required, so these streams can be received into old-version pools. There are no changes needed to the user interface for "zfs recv" to receive these streams. C. MANPAGE CHANGES TBD, based on the above text. 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From sommerfeld@sun.com Wed Oct 3 11:00:00 2007 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l93I00m5005150 for ; Wed, 3 Oct 2007 11:00:00 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l93HuoWM011195; Wed, 3 Oct 2007 10:56:50 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPC00D05KIQQC00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 03 Oct 2007 10:56:50 -0700 (PDT) Received: from dm-east-01.east.sun.com ([129.148.9.192]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPC006D0KIPWN40@nwk-avmta-1.sfbay.Sun.COM>; Wed, 03 Oct 2007 10:56:50 -0700 (PDT) Received: from thunk.east.sun.com (thunk.East.Sun.COM [129.148.174.66]) by dm-east-01.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id l93HuirO039380; Wed, 03 Oct 2007 13:56:44 -0400 (EDT) Received: from [IPv6:::1] (localhost [IPv6:::1]) by thunk.east.sun.com (8.14.1+Sun/8.14.1) with ESMTP id l93Huif2028342; Wed, 03 Oct 2007 13:56:44 -0400 (EDT) Date: Wed, 03 Oct 2007 13:56:43 -0400 From: Bill Sommerfeld Subject: Re: zfs send -R [PSARC/2007/574 FastTrack timeout 10/10/2007] In-reply-to: <200710031724.l93HOduQ984796@jurassic-x4600.sfbay.sun.com> To: Matthew Ahrens Cc: PSARC-EXT@sun.com, zfs-eng@sun.com Message-id: <1191434203.27431.20.camel@thunk> MIME-version: 1.0 X-Mailer: Evolution 2.10.2 Content-type: text/plain Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200710031724.l93HOduQ984796@jurassic-x4600.sfbay.sun.com> Status: RO Content-Length: 1345 On Wed, 2007-10-03 at 10:24 -0700, Matthew Ahrens wrote: > Note, to receive any of these new type of "zfs send" streams, the > receiving system must be running a software version capable of sending > them. Ie, the stream version is incremented. However, no on-disk > changes are required, so these streams can be received into old-version > pools. There are no changes needed to the user interface for "zfs recv" > to receive these streams. Q1: with respect to this stream version change alone, can an old "zfs send" stream still be received by new software? Q2: with the changes proposed by this case, is a "zfs send" not using the new features sent using the older stream version or the newer version? This is motivated by an operational question: I run a couple servers which have large zfs pools and which track nevada closely. Another group, which has a server running an s10 update, wants to do backups for disaster recovery to one of my servers. zfs replication of a small number of filesystems would be the way to do it but for the current lack of commitment to the stream format. The ability for a new pool to receive old-format streams would seem to be sufficient for this sort of use -- restores would (hopefully!) be so infrequent that we could just use rsync or equivalent to push file contents across. - Bill From Matthew.Ahrens@sun.com Wed Oct 3 11:58:34 2007 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l93IwYKn007574 for ; Wed, 3 Oct 2007 11:58:34 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l93IsP85065436; Wed, 3 Oct 2007 12:54:26 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPC00J03N8ALK00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 03 Oct 2007 11:55:22 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPC0062BN87WN80@nwk-avmta-1.sfbay.Sun.COM>; Wed, 03 Oct 2007 11:55:19 -0700 (PDT) Received: from dhcp-ubrm05-51-162.central.sun.com (dhcp-ubrm05-51-162.Central.Sun.COM [129.147.51.162]) by jurassic-x4600.sfbay.sun.com (8.14.1+Sun/8.14.1) with ESMTP id l93ItGxn103175 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 03 Oct 2007 11:55:19 -0700 (PDT) Date: Wed, 03 Oct 2007 12:55:16 -0600 From: Matthew Ahrens Subject: Re: zfs send -R [PSARC/2007/574 FastTrack timeout 10/10/2007] In-reply-to: <1191434203.27431.20.camel@thunk> To: Bill Sommerfeld Cc: Matthew Ahrens , PSARC-EXT@sun.com, zfs-eng@sun.com Message-id: <4703E594.4090901@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200710031724.l93HOduQ984796@jurassic-x4600.sfbay.sun.com> <1191434203.27431.20.camel@thunk> User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) Status: RO Content-Length: 1446 Bill Sommerfeld wrote: > On Wed, 2007-10-03 at 10:24 -0700, Matthew Ahrens wrote: >> Note, to receive any of these new type of "zfs send" streams, the >> receiving system must be running a software version capable of sending >> them. Ie, the stream version is incremented. However, no on-disk >> changes are required, so these streams can be received into old-version >> pools. There are no changes needed to the user interface for "zfs recv" >> to receive these streams. > > Q1: with respect to this stream version change alone, can an old "zfs > send" stream still be received by new software? Yes. > Q2: with the changes proposed by this case, is a "zfs send" not using > the new features sent using the older stream version or the newer > version? The older version. > This is motivated by an operational question: I run a couple servers > which have large zfs pools and which track nevada closely. Another > group, which has a server running an s10 update, wants to do backups for > disaster recovery to one of my servers. zfs replication of a small > number of filesystems would be the way to do it but for the current lack > of commitment to the stream format. Yep, this should just work. Despite our lack of commitment, it's our goal for new software versions to always be able to receive old streams. However, at some point "zfs send" may only send a new stream format that can not be received with old software. --matt From sommerfeld@Sun.COM Wed Oct 3 13:24:49 2007 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l93KOntx009840 for ; Wed, 3 Oct 2007 13:24:49 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l93KKcU1023518; Wed, 3 Oct 2007 14:20:41 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPC0050PR82FV00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 03 Oct 2007 13:21:38 -0700 (PDT) Received: from dm-east-01.east.sun.com ([129.148.9.192]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPC006D4R82WHB0@nwk-avmta-1.sfbay.Sun.COM>; Wed, 03 Oct 2007 13:21:38 -0700 (PDT) Received: from thunk.east.sun.com (thunk.East.Sun.COM [129.148.174.66]) by dm-east-01.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id l93KLb1c022121; Wed, 03 Oct 2007 16:21:37 -0400 (EDT) Received: from [IPv6:::1] (localhost [IPv6:::1]) by thunk.east.sun.com (8.14.1+Sun/8.14.1) with ESMTP id l93KLbdZ028787; Wed, 03 Oct 2007 16:21:37 -0400 (EDT) Date: Wed, 03 Oct 2007 16:21:37 -0400 From: Bill Sommerfeld Subject: Re: zfs send -R [PSARC/2007/574 FastTrack timeout 10/10/2007] In-reply-to: <4703E594.4090901@sun.com> To: Matthew Ahrens Cc: PSARC-EXT@Sun.COM, zfs-eng@Sun.COM Message-id: <1191442897.27431.58.camel@thunk> MIME-version: 1.0 X-Mailer: Evolution 2.10.2 Content-type: text/plain Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200710031724.l93HOduQ984796@jurassic-x4600.sfbay.sun.com> <1191434203.27431.20.camel@thunk> <4703E594.4090901@sun.com> Status: RO Content-Length: 816 On Wed, 2007-10-03 at 12:55 -0600, Matthew Ahrens wrote: > Yep, this should just work. Despite our lack of commitment, it's our goal for > new software versions to always be able to receive old streams. However, at > some point "zfs send" may only send a new stream format that can not be > received with old software. Okay, great. My observation is that the explicit lack of commitment to the stream format in the zfs(1m) man page is actively scaring people away from using zfs send. I wouldn't insist on it but given the stated goal above, this might be a good time to formally commit (both as part of this case and in the man page) to being able to restore old streams with new software. (I can understand not wanting to commit to always being able to send in older stream formats). - Bill