From ahl@zday.sfbay.sun.com Mon Oct 19 18:13:13 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9K1DCaP019563 for ; Mon, 19 Oct 2009 18:13:12 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n9K1DBZE022615; Mon, 19 Oct 2009 19:13:11 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRS00M0DGPX9B00@brm-avmta-1.central.sun.com>; Mon, 19 Oct 2009 19:13:09 -0600 (MDT) Received: from zday.sfbay.sun.com ([10.6.46.178]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRS00AP5GPWSQE0@brm-avmta-1.central.sun.com>; Mon, 19 Oct 2009 19:13:08 -0600 (MDT) Received: from zday.sfbay.sun.com (localhost [127.0.0.1]) by zday.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n9K1DMGv766371; Mon, 19 Oct 2009 18:13:22 -0700 (PDT) Received: (from ahl@localhost) by zday.sfbay.sun.com (8.14.3+Sun/8.14.3/Submit) id n9K1DM84766368; Mon, 19 Oct 2009 18:13:22 -0700 (PDT) Date: Mon, 19 Oct 2009 18:13:22 -0700 (PDT) From: Adam Leventhal Subject: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] To: PSARC-ext@sun.com Cc: Jeff.Bonwick@sun.com, ahl@eng.sun.com, zfs-team@sun.com Message-id: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 5354 I'm sponsoring the following fasttrack on behalf of Jeff Bonwick, and the ZFS team. The binding is patch and the commitment level is Committed. Apologies for the late notice, but if it is possible to review this case at the 10/21/2009 meeting that would be much appreciated. We believe the interfaces as defined are in keeping with other ZFS interfaces. Please take particular note of the question at the end of B.1 where we're unsure of the best path and hope the ARC can provide guidance. Thanks. Adam Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: ZFS Deduplication Properties 1.2. Name of Document Author/Supplier: Author: Jeffrey Bonwick 1.3 Date of This Document: 19 October, 2009 4. Technical Description A. Background Deduplication is a feature of modern storage platforms by which varying mechanisms are employed to reduce the amount of total data stored by eliminating and sharing common components. We are adding deduplication to ZFS in order to further enable market penetration with ZFS and the Sun Storage 7000 series. The algorithm employed by ZFS deduplication uses checksum based comparison of blocks with optional verification (for example with non- cryptographically secure checksums). Deduplication is performed across the entire ZFS storage pool; administrators can select if individual datasets have deduplication enabled or not. This is useful in mixed- mode environments in which some datasets have highly duplicated data (e.g. VMware images, VDI, home directories, or email folders) and others are unique (e.g. databases). With this case we propose the user interface for enabling deduplication in ZFS. B. Interface B.1 zfs(1M) The interface for enabling and disabling deduplication is simple and straightforward, and follows the convention of other similar ZFS settings. We simply add a new per-dataset property, dedup: zfs set dedup=[,verify] zfs get dedup The acceptable values for the dedup property are as follows: off (the default) on (see below) on,verify verify sha256 sha256,verify fletcher4,verify fletcher2,verify The dedup property can be set to any of the cryptographically strong checksums supported by ZFS (today just sha256). In this mode we rely on the checksum alone to ensure no data collisions. Alternatively the dedup property can be set to ',verify' in which the given checksum is used for comparison, the blocks are compared to ensure against collisions. This is strictly relevant only for non- cryptographically secure checksums but we offer it as an option for customers who seek that reassurance. The value of 'on' uses the zpool- wide default defined by the zpool property dedupchecksum (see B.2.1). As an explicit request for input from the ARC, our fletcher2 implementation has been shown to be suboptimal and results in a large number of collisions (as a result, the default checksum has been changed to fletcher4). Should 'fletcher2,verify' be permitted as an option for consistency or should we eliminate that option since it would rarely be an attractive choice for users due to the high number of hash collisions. B.2 zpool(1M) B.2.1 Mutable properties Two new mutable pool-wide properties will be added: zpool set dedupchecksum= zpool set dedupditto= The first selects the pool-wide default to be used when a dataset's dedup value is set to 'on' or 'on,verify'. The default value for dedupchecksum is 'sha256'. The second allows the administrator to select a threshhold afterwhich 2 copies of a block are stored rather than 1. For example, if many duplicate blocks exist deduplication would reduce that count to just 1; at some threshhold, it becomes desirable to have multiple copies to guard against the multiplied effects of the loss of a single block. The default value is '100'. B.2.2 Statistics Two new read-only pool-wide properties will be added to track deduplication efficacy: deduptotal # the amount of deduplicated data on disk dedupinflated # deduplicated data had duplicates not be removed With these two properties and the pool's size property one could compute: dedup efficacy = dedupinflated / deduptotal dedup savings = dedupinflated - deduptotal dedup ratio = (size + dedupinflated) / (size + deduptotal) Note that efficacy measures only data that was a candidate for deduplication (i.e. on which the dedup dataset property was enabled) whereas the ratio measures a similar value for all data regardless of whether it was a candidate for deduplication. The 'zpool status' command will be modified to present the size and dedup ratio and efficacy for the give pool or pools: # zpool status tank pool: tank state: ONLINE size: 464G dedup: 1.90x (total) / 5.41x (dedup enabled) ... C. Man Page Changes The zfs(1M) and zpool(1M) man pages will be modified to include the descriptions above for the new properties as well as an overview of the deduplication feature. 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: OS/Net 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From gdamore@sun.com Mon Oct 19 18:52:32 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9K1qVIH019684 for ; Mon, 19 Oct 2009 18:52:31 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n9K1qU8o038221; Mon, 19 Oct 2009 19:52:30 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRS00301IJGL100@brm-avmta-1.central.sun.com>; Mon, 19 Oct 2009 19:52:28 -0600 (MDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRS00MWYIJGTOC0@brm-avmta-1.central.sun.com>; Mon, 19 Oct 2009 19:52:28 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n9K1qRdH000428; Mon, 19 Oct 2009 18:52:27 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KRS00I00IJ9KS00@fe-sfbay-09.sun.com>; Mon, 19 Oct 2009 18:52:27 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KRS00FB8IJF9L10@fe-sfbay-09.sun.com>; Mon, 19 Oct 2009 18:52:27 -0700 (PDT) Date: Mon, 19 Oct 2009 18:52:26 -0700 From: "Garrett D'Amore" Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> Sender: Garrett.Damore@sun.com To: Adam Leventhal Cc: PSARC-ext@sun.com, Jeff.Bonwick@sun.com, ahl@eng.sun.com, zfs-team@sun.com Message-id: <4ADD17DA.9070104@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.22 (X11/20090909) Status: RO Content-Length: 6967 This case looks good. I have a few thoughts though. First, wrt. fletcher2 ... I don't think it is a good idea to support the algorithm if it has these undesirable characteristics. Its not the case that you need to support it for compatibility. Actually, that brings into question fletcher4 as well. I guess my question is, why not just support sha256 for now? I'm assuming that the motivation here is reduced computation time for fletcher vs. sha256? The second question is about the default for dedupditto. It seems to me at least that a default value of "2" might be better than "100", as it means that dedup wins with the 3rd copy, and also wins by adding yet another level of data redundancy. (In other words, dedup actually improves data safety this way.) Where did the value of "100" come from? In any case, none of the above items are enough to prevent me from issuing a +1 to the case. Technically the case should falls under our 48 hour timeline for fast tracks, but I think we could probably review it, and move the timeout to Thursday morning, at which point it would be approved by successful time with the necessary +1s and no derails. This would only cause a 24 hour delay, but allow the case to still fall within existing ARC practice. Would that be acceptable? - Garrett Adam Leventhal wrote: > I'm sponsoring the following fasttrack on behalf of Jeff Bonwick, and > the ZFS team. The binding is patch and the commitment level is Committed. > > Apologies for the late notice, but if it is possible to review this case > at the 10/21/2009 meeting that would be much appreciated. We believe the > interfaces as defined are in keeping with other ZFS interfaces. > > Please take particular note of the question at the end of B.1 where we're > unsure of the best path and hope the ARC can provide guidance. > > Thanks. > > Adam > > > Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI > This information is Copyright 2009 Sun Microsystems > 1. Introduction > 1.1. Project/Component Working Name: > ZFS Deduplication Properties > 1.2. Name of Document Author/Supplier: > Author: Jeffrey Bonwick > 1.3 Date of This Document: > 19 October, 2009 > 4. Technical Description > A. Background > > Deduplication is a feature of modern storage platforms by which > varying mechanisms are employed to reduce the amount of total data > stored by eliminating and sharing common components. We are adding > deduplication to ZFS in order to further enable market penetration > with ZFS and the Sun Storage 7000 series. > > The algorithm employed by ZFS deduplication uses checksum based > comparison of blocks with optional verification (for example with non- > cryptographically secure checksums). Deduplication is performed across > the entire ZFS storage pool; administrators can select if individual > datasets have deduplication enabled or not. This is useful in mixed- > mode environments in which some datasets have highly duplicated data > (e.g. VMware images, VDI, home directories, or email folders) and > others are unique (e.g. databases). > > With this case we propose the user interface for enabling > deduplication in ZFS. > > > B. Interface > > B.1 zfs(1M) > > The interface for enabling and disabling deduplication is simple and > straightforward, and follows the convention of other similar ZFS > settings. We simply add a new per-dataset property, dedup: > > zfs set dedup=[,verify] > zfs get dedup > > The acceptable values for the dedup property are as follows: > > off (the default) > on (see below) > on,verify > verify > sha256 > sha256,verify > fletcher4,verify > fletcher2,verify > > The dedup property can be set to any of the cryptographically strong > checksums supported by ZFS (today just sha256). In this mode we rely > on the checksum alone to ensure no data collisions. Alternatively the > dedup property can be set to ',verify' in which the given > checksum is used for comparison, the blocks are compared to ensure > against collisions. This is strictly relevant only for non- > cryptographically secure checksums but we offer it as an option for > customers who seek that reassurance. The value of 'on' uses the zpool- > wide default defined by the zpool property dedupchecksum (see B.2.1). > > As an explicit request for input from the ARC, our fletcher2 implementation > has been shown to be suboptimal and results in a large number of > collisions (as a result, the default checksum has been changed to > fletcher4). Should 'fletcher2,verify' be permitted as an option for > consistency or should we eliminate that option since it would rarely > be an attractive choice for users due to the high number of hash > collisions. > > > B.2 zpool(1M) > > B.2.1 Mutable properties > > Two new mutable pool-wide properties will be added: > > zpool set dedupchecksum= > zpool set dedupditto= > > The first selects the pool-wide default to be used when a dataset's dedup > value is set to 'on' or 'on,verify'. The default value for dedupchecksum > is 'sha256'. > > The second allows the administrator to select a threshhold afterwhich > 2 copies of a block are stored rather than 1. For example, if many > duplicate blocks exist deduplication would reduce that count to just 1; > at some threshhold, it becomes desirable to have multiple copies to > guard against the multiplied effects of the loss of a single block. > The default value is '100'. > > > B.2.2 Statistics > > Two new read-only pool-wide properties will be added to track > deduplication efficacy: > > deduptotal # the amount of deduplicated data on disk > dedupinflated # deduplicated data had duplicates not be removed > > With these two properties and the pool's size property one could > compute: > > dedup efficacy = dedupinflated / deduptotal > dedup savings = dedupinflated - deduptotal > dedup ratio = (size + dedupinflated) / (size + deduptotal) > > Note that efficacy measures only data that was a candidate for > deduplication (i.e. on which the dedup dataset property was enabled) > whereas the ratio measures a similar value for all data regardless of > whether it was a candidate for deduplication. > > The 'zpool status' command will be modified to present the size and > dedup ratio and efficacy for the give pool or pools: > > # zpool status tank > pool: tank > state: ONLINE > size: 464G > dedup: 1.90x (total) / 5.41x (dedup enabled) > ... > > > C. Man Page Changes > > The zfs(1M) and zpool(1M) man pages will be modified to include the > descriptions above for the new properties as well as an overview of the > deduplication feature. > > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > OS/Net > 6.5. ARC review type: FastTrack > 6.6. ARC Exposure: open > > From Matthew.Ahrens@sun.com Mon Oct 19 19:12:07 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9K2C60f020241 for ; Mon, 19 Oct 2009 19:12:06 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n9K2C408003229; Mon, 19 Oct 2009 19:12:05 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRS00M03JG5LW00@nwk-avmta-1.sfbay.Sun.COM>; Mon, 19 Oct 2009 19:12:05 -0700 (PDT) Received: from zion.sfbay.sun.com ([129.146.17.75]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRS00I8XJG43LB0@nwk-avmta-1.sfbay.Sun.COM>; Mon, 19 Oct 2009 19:12:04 -0700 (PDT) Received: from stationary.local (punchin-ahrens.SFBay.Sun.COM [10.7.251.178]) by zion.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n9K2AsZH017155; Tue, 20 Oct 2009 02:10:55 +0000 (GMT) Date: Mon, 19 Oct 2009 19:12:03 -0700 From: Matthew Ahrens Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> To: Adam Leventhal Cc: PSARC-ext@sun.com, Jeff.Bonwick@sun.com, ahl@eng.sun.com, zfs-team@sun.com Message-id: <4ADD1C73.5090407@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) Status: RO Content-Length: 6407 A few suggestions: 1. don't allow fletcher2,verify. (we only allow checksum=fletcher2 for backwards compatability anyway) 2. expose the dedup ratio as a pool property (like compression ratio is a zfs property) 3. Are the valid values for the zpool dedupchecksum property "sha256" and "sha256,verify"? I'm wondering what the rationale is behind introducing the idea of a pool-wide default for "zfs set X=on", and why we wouldn't extend this to all ZFS properties of the form "zfs set X=on | off | specific value" (namely, compression, checksum, and share*)? It seems simpler to continue using the existing zfs property inheritance model, rather than introducing new pool-wide "what does on mean" properties. --matt Adam Leventhal wrote: > I'm sponsoring the following fasttrack on behalf of Jeff Bonwick, and > the ZFS team. The binding is patch and the commitment level is Committed. > > Apologies for the late notice, but if it is possible to review this case > at the 10/21/2009 meeting that would be much appreciated. We believe the > interfaces as defined are in keeping with other ZFS interfaces. > > Please take particular note of the question at the end of B.1 where we're > unsure of the best path and hope the ARC can provide guidance. > > Thanks. > > Adam > > > Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI > This information is Copyright 2009 Sun Microsystems > 1. Introduction > 1.1. Project/Component Working Name: > ZFS Deduplication Properties > 1.2. Name of Document Author/Supplier: > Author: Jeffrey Bonwick > 1.3 Date of This Document: > 19 October, 2009 > 4. Technical Description > A. Background > > Deduplication is a feature of modern storage platforms by which > varying mechanisms are employed to reduce the amount of total data > stored by eliminating and sharing common components. We are adding > deduplication to ZFS in order to further enable market penetration > with ZFS and the Sun Storage 7000 series. > > The algorithm employed by ZFS deduplication uses checksum based > comparison of blocks with optional verification (for example with non- > cryptographically secure checksums). Deduplication is performed across > the entire ZFS storage pool; administrators can select if individual > datasets have deduplication enabled or not. This is useful in mixed- > mode environments in which some datasets have highly duplicated data > (e.g. VMware images, VDI, home directories, or email folders) and > others are unique (e.g. databases). > > With this case we propose the user interface for enabling > deduplication in ZFS. > > > B. Interface > > B.1 zfs(1M) > > The interface for enabling and disabling deduplication is simple and > straightforward, and follows the convention of other similar ZFS > settings. We simply add a new per-dataset property, dedup: > > zfs set dedup=[,verify] > zfs get dedup > > The acceptable values for the dedup property are as follows: > > off (the default) > on (see below) > on,verify > verify > sha256 > sha256,verify > fletcher4,verify > fletcher2,verify > > The dedup property can be set to any of the cryptographically strong > checksums supported by ZFS (today just sha256). In this mode we rely > on the checksum alone to ensure no data collisions. Alternatively the > dedup property can be set to ',verify' in which the given > checksum is used for comparison, the blocks are compared to ensure > against collisions. This is strictly relevant only for non- > cryptographically secure checksums but we offer it as an option for > customers who seek that reassurance. The value of 'on' uses the zpool- > wide default defined by the zpool property dedupchecksum (see B.2.1). > > As an explicit request for input from the ARC, our fletcher2 implementation > has been shown to be suboptimal and results in a large number of > collisions (as a result, the default checksum has been changed to > fletcher4). Should 'fletcher2,verify' be permitted as an option for > consistency or should we eliminate that option since it would rarely > be an attractive choice for users due to the high number of hash > collisions. > > > B.2 zpool(1M) > > B.2.1 Mutable properties > > Two new mutable pool-wide properties will be added: > > zpool set dedupchecksum= > zpool set dedupditto= > > The first selects the pool-wide default to be used when a dataset's dedup > value is set to 'on' or 'on,verify'. The default value for dedupchecksum > is 'sha256'. > > The second allows the administrator to select a threshhold afterwhich > 2 copies of a block are stored rather than 1. For example, if many > duplicate blocks exist deduplication would reduce that count to just 1; > at some threshhold, it becomes desirable to have multiple copies to > guard against the multiplied effects of the loss of a single block. > The default value is '100'. > > > B.2.2 Statistics > > Two new read-only pool-wide properties will be added to track > deduplication efficacy: > > deduptotal # the amount of deduplicated data on disk > dedupinflated # deduplicated data had duplicates not be removed > > With these two properties and the pool's size property one could > compute: > > dedup efficacy = dedupinflated / deduptotal > dedup savings = dedupinflated - deduptotal > dedup ratio = (size + dedupinflated) / (size + deduptotal) > > Note that efficacy measures only data that was a candidate for > deduplication (i.e. on which the dedup dataset property was enabled) > whereas the ratio measures a similar value for all data regardless of > whether it was a candidate for deduplication. > > The 'zpool status' command will be modified to present the size and > dedup ratio and efficacy for the give pool or pools: > > # zpool status tank > pool: tank > state: ONLINE > size: 464G > dedup: 1.90x (total) / 5.41x (dedup enabled) > ... > > > C. Man Page Changes > > The zfs(1M) and zpool(1M) man pages will be modified to include the > descriptions above for the new properties as well as an overview of the > deduplication feature. > > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > OS/Net > 6.5. ARC review type: FastTrack > 6.6. ARC Exposure: open From George.Wilson@sun.com Mon Oct 19 20:12:24 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9K3CNop021548 for ; Mon, 19 Oct 2009 20:12:23 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n9K3CM4b002577; Mon, 19 Oct 2009 21:12:23 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRS00801M8M7A00@nwk-avmta-1.sfbay.Sun.COM>; Mon, 19 Oct 2009 20:12:22 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRS007FVM8LL610@nwk-avmta-1.sfbay.Sun.COM>; Mon, 19 Oct 2009 20:12:21 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n9K3CL1L016350; Tue, 20 Oct 2009 03:12:21 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KRS00900LZVX300@mail-amer.sun.com>; Mon, 19 Oct 2009 21:12:21 -0600 (MDT) Received: from george-wilsons-macbook-pro.local ([unknown] [69.181.229.222]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KRS009ZNM8KLH00@mail-amer.sun.com>; Mon, 19 Oct 2009 21:12:21 -0600 (MDT) Date: Mon, 19 Oct 2009 20:12:20 -0700 From: George Wilson Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <4ADD17DA.9070104@sun.com> Sender: George.Wilson@sun.com To: "Garrett D'Amore" Cc: Adam Leventhal , PSARC-ext@sun.com, Jeff.Bonwick@sun.com, ahl@eng.sun.com, zfs-team@sun.com Message-id: <4ADD2A94.5060909@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD17DA.9070104@sun.com> User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) Status: RO Content-Length: 7786 Garrett D'Amore wrote: > This case looks good. I have a few thoughts though. > > First, wrt. fletcher2 ... I don't think it is a good idea to support the > algorithm if it has these undesirable characteristics. Its not the case > that you need to support it for compatibility. > > Actually, that brings into question fletcher4 as well. I guess my > question is, why not just support sha256 for now? I'm assuming that the > motivation here is reduced computation time for fletcher vs. sha256? Exactly the reason. We wanted to provide the flexibility. > > The second question is about the default for dedupditto. It seems to me > at least that a default value of "2" might be better than "100", as it > means that dedup wins with the 3rd copy, and also wins by adding yet > another level of data redundancy. (In other words, dedup actually > improves data safety this way.) Where did the value of "100" come from? The idea is that blocks with fewer than 100 references aren't as critical and thus we don't need to make an extra copy. It boils down to a policy decision between safety vs. space savings. We decided that 100 references seemed like a valid threshold for making the extra copy. A value of '2' would result in 0% saving until a block has 3 or more references. Setting the value too low would defeat the benefits of de-dup. Thanks, George > > In any case, none of the above items are enough to prevent me from > issuing a +1 to the case. > > Technically the case should falls under our 48 hour timeline for fast > tracks, but I think we could probably review it, and move the timeout to > Thursday morning, at which point it would be approved by successful time > with the necessary +1s and no derails. This would only cause a 24 hour > delay, but allow the case to still fall within existing ARC practice. > Would that be acceptable? > > - Garrett > > Adam Leventhal wrote: >> I'm sponsoring the following fasttrack on behalf of Jeff Bonwick, and >> the ZFS team. The binding is patch and the commitment level is Committed. >> >> Apologies for the late notice, but if it is possible to review this case >> at the 10/21/2009 meeting that would be much appreciated. We believe the >> interfaces as defined are in keeping with other ZFS interfaces. >> >> Please take particular note of the question at the end of B.1 where we're >> unsure of the best path and hope the ARC can provide guidance. >> >> Thanks. >> >> Adam >> >> >> Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI >> This information is Copyright 2009 Sun Microsystems >> 1. Introduction >> 1.1. Project/Component Working Name: >> ZFS Deduplication Properties >> 1.2. Name of Document Author/Supplier: >> Author: Jeffrey Bonwick >> 1.3 Date of This Document: >> 19 October, 2009 >> 4. Technical Description >> A. Background >> >> Deduplication is a feature of modern storage platforms by which >> varying mechanisms are employed to reduce the amount of total data >> stored by eliminating and sharing common components. We are adding >> deduplication to ZFS in order to further enable market penetration >> with ZFS and the Sun Storage 7000 series. >> >> The algorithm employed by ZFS deduplication uses checksum based >> comparison of blocks with optional verification (for example with non- >> cryptographically secure checksums). Deduplication is performed >> across the entire ZFS storage pool; administrators can select if >> individual datasets have deduplication enabled or not. This is useful >> in mixed- mode environments in which some datasets have highly >> duplicated data (e.g. VMware images, VDI, home directories, or email >> folders) and others are unique (e.g. databases). >> >> With this case we propose the user interface for enabling >> deduplication in ZFS. >> >> >> B. Interface >> >> B.1 zfs(1M) >> >> The interface for enabling and disabling deduplication is simple and >> straightforward, and follows the convention of other similar ZFS >> settings. We simply add a new per-dataset property, dedup: >> >> zfs set dedup=[,verify] >> zfs get dedup >> >> The acceptable values for the dedup property are as follows: >> >> off (the default) >> on (see below) >> on,verify >> verify >> sha256 >> sha256,verify >> fletcher4,verify >> fletcher2,verify >> >> The dedup property can be set to any of the cryptographically strong >> checksums supported by ZFS (today just sha256). In this mode we rely >> on the checksum alone to ensure no data collisions. Alternatively the >> dedup property can be set to ',verify' in which the given >> checksum is used for comparison, the blocks are compared to ensure >> against collisions. This is strictly relevant only for non- >> cryptographically secure checksums but we offer it as an option for >> customers who seek that reassurance. The value of 'on' uses the zpool- >> wide default defined by the zpool property dedupchecksum (see B.2.1). >> >> As an explicit request for input from the ARC, our fletcher2 >> implementation >> has been shown to be suboptimal and results in a large number of >> collisions (as a result, the default checksum has been changed to >> fletcher4). Should 'fletcher2,verify' be permitted as an option for >> consistency or should we eliminate that option since it would rarely >> be an attractive choice for users due to the high number of hash >> collisions. >> >> >> B.2 zpool(1M) >> >> B.2.1 Mutable properties >> >> Two new mutable pool-wide properties will be added: >> >> zpool set dedupchecksum= >> zpool set dedupditto= >> >> The first selects the pool-wide default to be used when a dataset's dedup >> value is set to 'on' or 'on,verify'. The default value for dedupchecksum >> is 'sha256'. >> >> The second allows the administrator to select a threshhold afterwhich >> 2 copies of a block are stored rather than 1. For example, if many >> duplicate blocks exist deduplication would reduce that count to just 1; >> at some threshhold, it becomes desirable to have multiple copies to >> guard against the multiplied effects of the loss of a single block. >> The default value is '100'. >> >> >> B.2.2 Statistics >> >> Two new read-only pool-wide properties will be added to track >> deduplication efficacy: >> >> deduptotal # the amount of deduplicated data on disk >> dedupinflated # deduplicated data had duplicates not be removed >> >> With these two properties and the pool's size property one could >> compute: >> >> dedup efficacy = dedupinflated / deduptotal >> dedup savings = dedupinflated - deduptotal >> dedup ratio = (size + dedupinflated) / (size + deduptotal) >> >> Note that efficacy measures only data that was a candidate for >> deduplication (i.e. on which the dedup dataset property was enabled) >> whereas the ratio measures a similar value for all data regardless of >> whether it was a candidate for deduplication. >> >> The 'zpool status' command will be modified to present the size and >> dedup ratio and efficacy for the give pool or pools: >> >> # zpool status tank >> pool: tank >> state: ONLINE >> size: 464G >> dedup: 1.90x (total) / 5.41x (dedup enabled) >> ... >> >> >> C. Man Page Changes >> >> The zfs(1M) and zpool(1M) man pages will be modified to include the >> descriptions above for the new properties as well as an overview of the >> deduplication feature. >> >> 6. Resources and Schedule >> 6.4. Steering Committee requested information >> 6.4.1. Consolidation C-team Name: >> OS/Net >> 6.5. ARC review type: FastTrack >> 6.6. ARC Exposure: open >> >> > From George.Wilson@sun.com Mon Oct 19 20:22:53 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9K3Mqpw021628 for ; Mon, 19 Oct 2009 20:22:52 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n9K3Mmrn012126; Tue, 20 Oct 2009 04:22:51 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRS00I01MQ2MW00@nwk-avmta-2.sfbay.sun.com>; Mon, 19 Oct 2009 20:22:50 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRS00EF9MQ1IZ60@nwk-avmta-2.sfbay.sun.com>; Mon, 19 Oct 2009 20:22:49 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n9K3MnNO027486; Tue, 20 Oct 2009 03:22:49 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KRS00G00MGXE300@mail-amer.sun.com>; Mon, 19 Oct 2009 21:22:49 -0600 (MDT) Received: from george-wilsons-macbook-pro.local ([unknown] [69.181.229.222]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KRS009YLMQ0LH20@mail-amer.sun.com>; Mon, 19 Oct 2009 21:22:49 -0600 (MDT) Date: Mon, 19 Oct 2009 20:22:48 -0700 From: George Wilson Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <4ADD1C73.5090407@sun.com> Sender: George.Wilson@sun.com To: Matthew Ahrens Cc: Adam Leventhal , PSARC-ext@sun.com, Jeff.Bonwick@sun.com, ahl@eng.sun.com, zfs-team@sun.com Message-id: <4ADD2D08.50805@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD1C73.5090407@sun.com> User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) Status: RO Content-Length: 7219 Matthew Ahrens wrote: > A few suggestions: > > 1. don't allow fletcher2,verify. (we only allow checksum=fletcher2 for > backwards compatability anyway) > > 2. expose the dedup ratio as a pool property (like compression ratio is > a zfs property) > > 3. Are the valid values for the zpool dedupchecksum property "sha256" > and "sha256,verify"? I'm wondering what the rationale is behind > introducing the idea of a pool-wide default for "zfs set X=on", and why > we wouldn't extend this to all ZFS properties of the form "zfs set X=on > | off | specific value" (namely, compression, checksum, and share*)? It > seems simpler to continue using the existing zfs property inheritance > model, rather than introducing new pool-wide "what does on mean" > properties. The reason we added dedupchecksum was to allow us to change the default checksum should we ever find something better than sha256 (in anticipation of sha-3). Normally, we would just make the new algorithm the default and then allow future blocks to leverage that checksum function. The problem is that if the pool has already been using sha256 and we change the default then new blocks would no longer de-dup against existing blocks. By having the dedupchecksum we use this to always define what the default 'on' value maps to. Thanks, George > > --matt > > Adam Leventhal wrote: >> I'm sponsoring the following fasttrack on behalf of Jeff Bonwick, and >> the ZFS team. The binding is patch and the commitment level is Committed. >> >> Apologies for the late notice, but if it is possible to review this case >> at the 10/21/2009 meeting that would be much appreciated. We believe the >> interfaces as defined are in keeping with other ZFS interfaces. >> >> Please take particular note of the question at the end of B.1 where we're >> unsure of the best path and hope the ARC can provide guidance. >> >> Thanks. >> >> Adam >> >> >> Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI >> This information is Copyright 2009 Sun Microsystems >> 1. Introduction >> 1.1. Project/Component Working Name: >> ZFS Deduplication Properties >> 1.2. Name of Document Author/Supplier: >> Author: Jeffrey Bonwick >> 1.3 Date of This Document: >> 19 October, 2009 >> 4. Technical Description >> A. Background >> >> Deduplication is a feature of modern storage platforms by which >> varying mechanisms are employed to reduce the amount of total data >> stored by eliminating and sharing common components. We are adding >> deduplication to ZFS in order to further enable market penetration >> with ZFS and the Sun Storage 7000 series. >> >> The algorithm employed by ZFS deduplication uses checksum based >> comparison of blocks with optional verification (for example with non- >> cryptographically secure checksums). Deduplication is performed >> across the entire ZFS storage pool; administrators can select if >> individual datasets have deduplication enabled or not. This is useful >> in mixed- mode environments in which some datasets have highly >> duplicated data (e.g. VMware images, VDI, home directories, or email >> folders) and others are unique (e.g. databases). >> >> With this case we propose the user interface for enabling >> deduplication in ZFS. >> >> >> B. Interface >> >> B.1 zfs(1M) >> >> The interface for enabling and disabling deduplication is simple and >> straightforward, and follows the convention of other similar ZFS >> settings. We simply add a new per-dataset property, dedup: >> >> zfs set dedup=[,verify] >> zfs get dedup >> >> The acceptable values for the dedup property are as follows: >> >> off (the default) >> on (see below) >> on,verify >> verify >> sha256 >> sha256,verify >> fletcher4,verify >> fletcher2,verify >> >> The dedup property can be set to any of the cryptographically strong >> checksums supported by ZFS (today just sha256). In this mode we rely >> on the checksum alone to ensure no data collisions. Alternatively the >> dedup property can be set to ',verify' in which the given >> checksum is used for comparison, the blocks are compared to ensure >> against collisions. This is strictly relevant only for non- >> cryptographically secure checksums but we offer it as an option for >> customers who seek that reassurance. The value of 'on' uses the zpool- >> wide default defined by the zpool property dedupchecksum (see B.2.1). >> >> As an explicit request for input from the ARC, our fletcher2 >> implementation >> has been shown to be suboptimal and results in a large number of >> collisions (as a result, the default checksum has been changed to >> fletcher4). Should 'fletcher2,verify' be permitted as an option for >> consistency or should we eliminate that option since it would rarely >> be an attractive choice for users due to the high number of hash >> collisions. >> >> >> B.2 zpool(1M) >> >> B.2.1 Mutable properties >> >> Two new mutable pool-wide properties will be added: >> >> zpool set dedupchecksum= >> zpool set dedupditto= >> >> The first selects the pool-wide default to be used when a dataset's dedup >> value is set to 'on' or 'on,verify'. The default value for dedupchecksum >> is 'sha256'. >> >> The second allows the administrator to select a threshhold afterwhich >> 2 copies of a block are stored rather than 1. For example, if many >> duplicate blocks exist deduplication would reduce that count to just 1; >> at some threshhold, it becomes desirable to have multiple copies to >> guard against the multiplied effects of the loss of a single block. >> The default value is '100'. >> >> >> B.2.2 Statistics >> >> Two new read-only pool-wide properties will be added to track >> deduplication efficacy: >> >> deduptotal # the amount of deduplicated data on disk >> dedupinflated # deduplicated data had duplicates not be removed >> >> With these two properties and the pool's size property one could >> compute: >> >> dedup efficacy = dedupinflated / deduptotal >> dedup savings = dedupinflated - deduptotal >> dedup ratio = (size + dedupinflated) / (size + deduptotal) >> >> Note that efficacy measures only data that was a candidate for >> deduplication (i.e. on which the dedup dataset property was enabled) >> whereas the ratio measures a similar value for all data regardless of >> whether it was a candidate for deduplication. >> >> The 'zpool status' command will be modified to present the size and >> dedup ratio and efficacy for the give pool or pools: >> >> # zpool status tank >> pool: tank >> state: ONLINE >> size: 464G >> dedup: 1.90x (total) / 5.41x (dedup enabled) >> ... >> >> >> C. Man Page Changes >> >> The zfs(1M) and zpool(1M) man pages will be modified to include the >> descriptions above for the new properties as well as an overview of the >> deduplication feature. >> >> 6. Resources and Schedule >> 6.4. Steering Committee requested information >> 6.4.1. Consolidation C-team Name: >> OS/Net >> 6.5. ARC review type: FastTrack >> 6.6. ARC Exposure: open > From Jeff.Bonwick@Sun.COM Mon Oct 19 23:28:59 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9K6Swbg024377 for ; Mon, 19 Oct 2009 23:28:58 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n9K6SqbW023070; Tue, 20 Oct 2009 07:28:53 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRS00C01VC4Z200@nwk-avmta-1.sfbay.Sun.COM>; Mon, 19 Oct 2009 23:28:52 -0700 (PDT) Received: from cathy.local ([129.146.228.63]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRS0090RVC4MR50@nwk-avmta-1.sfbay.Sun.COM>; Mon, 19 Oct 2009 23:28:52 -0700 (PDT) Received: from cathy.sfbay.sun.com (localhost [127.0.0.1]) by cathy.local (8.14.3+Sun/8.14.3) with ESMTP id n9K6Sq0E028185; Mon, 19 Oct 2009 23:28:52 -0700 (PDT) Received: (from bonwick@localhost) by cathy.sfbay.sun.com (8.14.3+Sun/8.14.3/Submit) id n9K6Spsa028184; Mon, 19 Oct 2009 23:28:51 -0700 (PDT) Date: Mon, 19 Oct 2009 23:28:51 -0700 From: Jeff Bonwick Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <4ADD2A94.5060909@sun.com> To: George Wilson Cc: "Garrett D'Amore" , Adam Leventhal , PSARC-ext@Sun.COM, Jeff.Bonwick@Sun.COM, ahl@eng.sun.com, zfs-team@Sun.COM Message-id: <20091020062851.GD28054@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD17DA.9070104@sun.com> <4ADD2A94.5060909@sun.com> X-Authentication-warning: cathy.sfbay.sun.com: bonwick set sender to Jeff.Bonwick@sun.com using -f User-Agent: Mutt/1.5.14 (2007-02-12) Status: RO Content-Length: 1924 > >First, wrt. fletcher2 ... I don't think it is a good idea to support the > >algorithm if it has these undesirable characteristics. Its not the case > >that you need to support it for compatibility. I'm fine with that. fletcher4 is almost as fast and far more robust. As you surmise, it's all about computation speed vs. sha256, which is an order of magnitude slower. > The idea is that blocks with fewer than 100 references aren't as > critical and thus we don't need to make an extra copy. It boils down to > a policy decision between safety vs. space savings. There are a couple of factors that lead to this particular number: - We always ditto indirect blocks, which represent up to 128 data blocks. The rationale is that loss of an indirect block would imply loss of 128 data blocks, so it's worth the cost. The same reasoning applies to a block that's deduped and referenced 128 times. We're rounding to 100 to be less geeky. - Unlike all other user data, dedupditto blocks are not billed to any particular user (because they do not belong to any particular dataset). Therefore, we would like to limit the maximum possible space overhead to something comparable to the overall metadata overhead, which is about 1%. Hence 100:1. I am open to the possibility of dedupditto being off by default, with 100 as the minimum legal value. If it's on by default, we can claim better data integrity out of the box, which is cool. However, it also means that when your build machine goes from 99 workspaces to 100, the space consumption suddenly doubles. That said, it's double the size of *one* workspace, not double the size of 99 workspaces, so it's no worse than if you simply added another workspace that wasn't deduped. (Of course, we could "fuzz" the refcnt threshold so that there is no sharp spike at the 100th copy -- which would be cool, but impossible to explain.) Jeff From ahl@eng.sun.com Mon Oct 19 23:48:43 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9K6mgUV025067 for ; Mon, 19 Oct 2009 23:48:43 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n9K6mdrv005399; Tue, 20 Oct 2009 07:48:42 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRS00A03W94BI00@nwk-avmta-2.sfbay.sun.com>; Mon, 19 Oct 2009 23:48:40 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRS009BLW93Z000@nwk-avmta-2.sfbay.sun.com>; Mon, 19 Oct 2009 23:48:39 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n9K6mdXY004271; Tue, 20 Oct 2009 06:48:39 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KRS00800VXZ7R00@mail-amer.sun.com>; Tue, 20 Oct 2009 00:48:39 -0600 (MDT) Received: from [192.168.1.100] ([unknown] [24.130.173.33]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KRS008YUW92H700@mail-amer.sun.com>; Tue, 20 Oct 2009 00:48:39 -0600 (MDT) Date: Mon, 19 Oct 2009 23:48:38 -0700 From: Adam Leventhal Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <4ADD1C73.5090407@sun.com> Sender: Adam.Leventhal@sun.com To: Matthew Ahrens Cc: PSARC-ext@sun.com, Jeff Bonwick , zfs-team@sun.com Message-id: <1714E736-768F-4F46-9DCB-DBEBB1AF9132@eng.sun.com> MIME-version: 1.0 X-Mailer: Apple Mail (2.1076) Content-type: text/plain; CHARSET=US-ASCII; delsp=yes; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD1C73.5090407@sun.com> Status: RO Content-Length: 6934 > 1. don't allow fletcher2,verify. (we only allow checksum=fletcher2 > for backwards compatability anyway) I think you'll see this was called out explicitly in section B.1. > 2. expose the dedup ratio as a pool property (like compression ratio > is a zfs property) Deduplication efficacy is a bit more complex than compression which is why we present the raw data as pool properties. We can let our users guide us with regard to adding further computation on those properties. > 3. Are the valid values for the zpool dedupchecksum property > "sha256" and "sha256,verify"? No, today, just 'sha256'. > I'm wondering what the rationale is behind introducing the idea of a > pool-wide default for "zfs set X=on", and why we wouldn't extend > this to all ZFS properties of the form "zfs set X=on | off | > specific value" (namely, compression, checksum, and share*)? It > seems simpler to continue using the existing zfs property > inheritance model, rather than introducing new pool-wide "what does > on mean" properties. See Jeff's and George's follow-ups. Adam > > --matt > > Adam Leventhal wrote: >> I'm sponsoring the following fasttrack on behalf of Jeff Bonwick, and >> the ZFS team. The binding is patch and the commitment level is >> Committed. >> Apologies for the late notice, but if it is possible to review this >> case >> at the 10/21/2009 meeting that would be much appreciated. We >> believe the >> interfaces as defined are in keeping with other ZFS interfaces. >> Please take particular note of the question at the end of B.1 where >> we're >> unsure of the best path and hope the ARC can provide guidance. >> Thanks. >> Adam >> Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI >> This information is Copyright 2009 Sun Microsystems >> 1. Introduction >> 1.1. Project/Component Working Name: >> ZFS Deduplication Properties >> 1.2. Name of Document Author/Supplier: >> Author: Jeffrey Bonwick >> 1.3 Date of This Document: >> 19 October, 2009 >> 4. Technical Description >> A. Background >> Deduplication is a feature of modern storage platforms by which >> varying mechanisms are employed to reduce the amount of total data >> stored by eliminating and sharing common components. We are adding >> deduplication to ZFS in order to further enable market penetration >> with ZFS and the Sun Storage 7000 series. >> The algorithm employed by ZFS deduplication uses checksum based >> comparison of blocks with optional verification (for example with >> non- cryptographically secure checksums). Deduplication is >> performed across the entire ZFS storage pool; administrators can >> select if individual datasets have deduplication enabled or not. >> This is useful in mixed- mode environments in which some datasets >> have highly duplicated data (e.g. VMware images, VDI, home >> directories, or email folders) and others are unique (e.g. >> databases). >> With this case we propose the user interface for enabling >> deduplication in ZFS. >> B. Interface >> B.1 zfs(1M) >> The interface for enabling and disabling deduplication is simple >> and straightforward, and follows the convention of other similar >> ZFS settings. We simply add a new per-dataset property, dedup: >> zfs set dedup=[,verify] >> zfs get dedup >> The acceptable values for the dedup property are as follows: >> off (the default) >> on (see below) >> on,verify >> verify >> sha256 >> sha256,verify >> fletcher4,verify >> fletcher2,verify >> The dedup property can be set to any of the cryptographically >> strong checksums supported by ZFS (today just sha256). In this >> mode we rely on the checksum alone to ensure no data collisions. >> Alternatively the dedup property can be set to ',verify' >> in which the given checksum is used for comparison, the blocks are >> compared to ensure against collisions. This is strictly relevant >> only for non- cryptographically secure checksums but we offer it as >> an option for customers who seek that reassurance. The value of >> 'on' uses the zpool- wide default defined by the zpool property >> dedupchecksum (see B.2.1). >> As an explicit request for input from the ARC, our fletcher2 >> implementation >> has been shown to be suboptimal and results in a large number of >> collisions (as a result, the default checksum has been changed to >> fletcher4). Should 'fletcher2,verify' be permitted as an option for >> consistency or should we eliminate that option since it would rarely >> be an attractive choice for users due to the high number of hash >> collisions. >> B.2 zpool(1M) >> B.2.1 Mutable properties >> Two new mutable pool-wide properties will be added: >> zpool set dedupchecksum= >> zpool set dedupditto= >> The first selects the pool-wide default to be used when a dataset's >> dedup >> value is set to 'on' or 'on,verify'. The default value for >> dedupchecksum >> is 'sha256'. >> The second allows the administrator to select a threshhold afterwhich >> 2 copies of a block are stored rather than 1. For example, if many >> duplicate blocks exist deduplication would reduce that count to >> just 1; >> at some threshhold, it becomes desirable to have multiple copies to >> guard against the multiplied effects of the loss of a single block. >> The default value is '100'. >> B.2.2 Statistics >> Two new read-only pool-wide properties will be added to track >> deduplication efficacy: >> deduptotal # the amount of deduplicated data on disk >> dedupinflated # deduplicated data had duplicates not be removed >> With these two properties and the pool's size property one could >> compute: >> dedup efficacy = dedupinflated / deduptotal >> dedup savings = dedupinflated - deduptotal >> dedup ratio = (size + dedupinflated) / (size + deduptotal) >> Note that efficacy measures only data that was a candidate for >> deduplication (i.e. on which the dedup dataset property was >> enabled) whereas the ratio measures a similar value for all data >> regardless of whether it was a candidate for deduplication. >> The 'zpool status' command will be modified to present the size >> and dedup ratio and efficacy for the give pool or pools: >> # zpool status tank >> pool: tank >> state: ONLINE >> size: 464G >> dedup: 1.90x (total) / 5.41x (dedup enabled) >> ... >> C. Man Page Changes >> The zfs(1M) and zpool(1M) man pages will be modified to include the >> descriptions above for the new properties as well as an overview of >> the >> deduplication feature. >> 6. Resources and Schedule >> 6.4. Steering Committee requested information >> 6.4.1. Consolidation C-team Name: >> OS/Net >> 6.5. ARC review type: FastTrack >> 6.6. ARC Exposure: open > -- Adam Leventhal, Fishworks http://blogs.sun.com/ahl From casper@holland.sun.com Tue Oct 20 00:55:35 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9K7tX56007971 for ; Tue, 20 Oct 2009 00:55:34 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n9K7tUeK006116; Tue, 20 Oct 2009 15:55:32 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRS0010ZZCKNH00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 20 Oct 2009 00:55:32 -0700 (PDT) Received: from dm-holland-02.uk.sun.com ([129.156.101.225]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRS00JQJZCI3590@nwk-avmta-1.sfbay.Sun.COM>; Tue, 20 Oct 2009 00:55:31 -0700 (PDT) Received: from holland (room101.Holland.Sun.COM [10.16.117.40]) by dm-holland-02.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n9K7tSZB019251; Tue, 20 Oct 2009 08:55:28 +0100 (BST) Date: Tue, 20 Oct 2009 09:55:28 +0200 From: Casper.Dik@sun.com Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <1714E736-768F-4F46-9DCB-DBEBB1AF9132@eng.sun.com> Sender: casper@holland.sun.com To: Adam Leventhal Cc: Matthew Ahrens , PSARC-ext@sun.com, Jeff Bonwick , zfs-team@sun.com Message-id: <200910200755.n9K7tSZB019251@dm-holland-02.uk.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD1C73.5090407@sun.com> <1714E736-768F-4F46-9DCB-DBEBB1AF9132@eng.sun.com> Status: RO Content-Length: 253 >> 1. don't allow fletcher2,verify. (we only allow checksum=fletcher2 >> for backwards compatability anyway) > >I think you'll see this was called out explicitly in section B.1. Is fletcher2 still the default? The zfs manual claims that. Casper From Darren.Moffat@sun.com Tue Oct 20 02:51:45 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9K9pjdV010718 for ; Tue, 20 Oct 2009 02:51:45 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n9K9piVE026129; Tue, 20 Oct 2009 02:51:45 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRT00G0D4Q9XI00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 20 Oct 2009 02:51:45 -0700 (PDT) Received: from gmp-eb-inf-2.sun.com ([192.18.6.24]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRT00CIE4Q72T30@nwk-avmta-1.sfbay.Sun.COM>; Tue, 20 Oct 2009 02:51:44 -0700 (PDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-1-fe1.eu.sun.com [192.18.6.7] (may be forged)) by gmp-eb-inf-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n9K9pgIe014787; Tue, 20 Oct 2009 09:51:43 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KRT00C003WL1V00@fe-emea-10.sun.com>; Tue, 20 Oct 2009 10:51:40 +0100 (BST) Received: from [192.168.1.105] (cpc2-rdng20-2-0-cust917.15-3.cable.virginmedia.com [86.28.167.150]) by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KRT007XE4PTKE60@fe-emea-10.sun.com>; Tue, 20 Oct 2009 10:51:29 +0100 (BST) Date: Tue, 20 Oct 2009 10:51:29 +0100 From: Darren J Moffat Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> Sender: Darren.Moffat@sun.com To: Adam Leventhal Cc: PSARC-ext@sun.com, Jeff.Bonwick@sun.com, zfs-team@sun.com Message-id: <4ADD8821.9010806@Sun.COM> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090929) Status: RO Content-Length: 4233 Adam Leventhal wrote: > B.1 zfs(1M) > > The interface for enabling and disabling deduplication is simple and > straightforward, and follows the convention of other similar ZFS > settings. We simply add a new per-dataset property, dedup: > > zfs set dedup=[,verify] > zfs get dedup I'm happy with this. > The acceptable values for the dedup property are as follows: > > off (the default) > on (see below) > on,verify > verify > sha256 > sha256,verify > fletcher4,verify > fletcher2,verify Given that dedup allows specifying a checksum does this mean that there need not be a relationship between the checksum used for the block on disk (ie the one stored in blkptr_t) and the one used for dedup ? Is this valid: zfs set checksum=fletcher4 tank zfs set dedup=sha256 If so what is stored on disk in the blkptr_t ? I assume it is a fletecher4 stored there. Where is the sha256 checksum stored then ? In the DDT ? Does this mean that deduplication is not using the blkptr checksum at all even if the blkptr checksum and dedup checksum are the same ? When in the ZIO pipeline is the checksum specified with the dedup property calculated ? I'm assuming it is in zio_write_bp_init() after compression and after encryption so it is on the state of the block exactly as it will be written to disk. Can gang blocks be deduplicated ? > The dedup property can be set to any of the cryptographically strong > checksums supported by ZFS (today just sha256). In this mode we rely > on the checksum alone to ensure no data collisions. Alternatively the > dedup property can be set to ',verify' in which the given > checksum is used for comparison, the blocks are compared to ensure > against collisions. This is strictly relevant only for non- > cryptographically secure checksums but we offer it as an option for > customers who seek that reassurance. The value of 'on' uses the zpool- > wide default defined by the zpool property dedupchecksum (see B.2.1). Glad that you do offer verify as a choice. It would be very useful to provide some sort of log output for the cases where verify found a collision - ie the checksum hashes matched but the verify said they were different. Not useful to end users so it could be a DTrace SDT or only in a DEBUG kernel. If this ever shows up a "hit" when dedup=sha256,verify it will make ZFS famous for finding collisions in SHA256. > As an explicit request for input from the ARC, our fletcher2 implementation > has been shown to be suboptimal and results in a large number of > collisions (as a result, the default checksum has been changed to > fletcher4). Should 'fletcher2,verify' be permitted as an option for > consistency or should we eliminate that option since it would rarely > be an attractive choice for users due to the high number of hash > collisions. I don't think fletcher2,verify should be provided. > B.2 zpool(1M) > > B.2.1 Mutable properties > > Two new mutable pool-wide properties will be added: > > zpool set dedupchecksum= Why is this needed when we don't have a pool level property for the default checksum or compression or encryption (or any other property that is inherited and has an "on | off | ..." style of setting ? Under what circumstances would this be needed rather than setting dedup to the required value ? Is this a new precendent that all properties with an "on" value should have a pool level property to determine what "on" is ? > zpool set dedupditto= > The second allows the administrator to select a threshhold afterwhich > 2 copies of a block are stored rather than 1. For example, if many > duplicate blocks exist deduplication would reduce that count to just 1; > at some threshhold, it becomes desirable to have multiple copies to > guard against the multiplied effects of the loss of a single block. > The default value is '100'. I think I understand that this needs to be pool wide because this is a SPA level concept not a dataset level one. Is it actually necessary to expose this tunable ? Given there is already a per dataset copies property how does this interact with that ? -- Darren J Moffat From gdamore@sun.com Tue Oct 20 07:07:08 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9KE78P0015908 for ; Tue, 20 Oct 2009 07:07:08 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n9KE6tGO006198; Tue, 20 Oct 2009 07:07:07 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRT00C2VGJTHI00@brm-avmta-1.central.sun.com>; Tue, 20 Oct 2009 08:07:05 -0600 (MDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRT000QXGJSOU80@brm-avmta-1.central.sun.com>; Tue, 20 Oct 2009 08:07:05 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n9KE74CH020428; Tue, 20 Oct 2009 07:07:04 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KRT00I00GBRIV00@fe-sfbay-09.sun.com>; Tue, 20 Oct 2009 07:07:04 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KRT003KYGJNRVF0@fe-sfbay-09.sun.com>; Tue, 20 Oct 2009 07:06:59 -0700 (PDT) Date: Tue, 20 Oct 2009 07:06:59 -0700 From: "Garrett D'Amore" Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <20091020062851.GD28054@eng.sun.com> Sender: Garrett.Damore@sun.com To: Jeff Bonwick Cc: George Wilson , Adam Leventhal , PSARC-ext@sun.com, ahl@eng.sun.com, zfs-team@sun.com Message-id: <4ADDC403.3040705@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD17DA.9070104@sun.com> <4ADD2A94.5060909@sun.com> <20091020062851.GD28054@eng.sun.com> User-Agent: Thunderbird 2.0.0.22 (X11/20090909) Status: RO Content-Length: 2674 Jeff Bonwick wrote: >>> First, wrt. fletcher2 ... I don't think it is a good idea to support the >>> algorithm if it has these undesirable characteristics. Its not the case >>> that you need to support it for compatibility. >>> > > I'm fine with that. fletcher4 is almost as fast and far more robust. > As you surmise, it's all about computation speed vs. sha256, which is > an order of magnitude slower. > > >> The idea is that blocks with fewer than 100 references aren't as >> critical and thus we don't need to make an extra copy. It boils down to >> a policy decision between safety vs. space savings. >> > > There are a couple of factors that lead to this particular number: > > - We always ditto indirect blocks, which represent up to 128 > data blocks. The rationale is that loss of an indirect block > would imply loss of 128 data blocks, so it's worth the cost. > The same reasoning applies to a block that's deduped and > referenced 128 times. We're rounding to 100 to be less geeky. > > - Unlike all other user data, dedupditto blocks are not billed > to any particular user (because they do not belong to any > particular dataset). Therefore, we would like to limit the > maximum possible space overhead to something comparable to the > overall metadata overhead, which is about 1%. Hence 100:1. > > I am open to the possibility of dedupditto being off by default, with > 100 as the minimum legal value. If it's on by default, we can claim > better data integrity out of the box, which is cool. However, it also > means that when your build machine goes from 99 workspaces to 100, > the space consumption suddenly doubles. That said, it's double the > size of *one* workspace, not double the size of 99 workspaces, so it's > no worse than if you simply added another workspace that wasn't deduped. > (Of course, we could "fuzz" the refcnt threshold so that there is no sharp > spike at the 100th copy -- which would be cool, but impossible to explain.) > If you set it at "2", then you don't have a sudden "oh usage doubled" moment. It means that you immediately gain a redundancy advantage. The drawback is the space savings of the ditto doesn't reap rewards until you have the third reference, and of course for blocks that aren't heavily duplicated, you do wind up using double the space than you would without it. But the dedup is *still* a significant win over traditional storage. All that said, I'm ok with the rationale you've presented -- I think the different values here are almost subject to administrator preference. The case has my +1 in any event. - Garrett > Jeff > From Nicolas.Williams@sun.com Tue Oct 20 07:50:35 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9KEoYJf016876 for ; Tue, 20 Oct 2009 07:50:35 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n9KEoG7H025405; Tue, 20 Oct 2009 15:50:32 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRT00803IK4GY00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 20 Oct 2009 07:50:28 -0700 (PDT) Received: from binky.Central.Sun.COM ([129.153.128.104]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRT00KMJIK38890@nwk-avmta-1.sfbay.Sun.COM>; Tue, 20 Oct 2009 07:50:27 -0700 (PDT) Received: from binky.Central.Sun.COM (localhost [127.0.0.1]) by binky.Central.Sun.COM (8.14.3+Sun/8.14.3) with ESMTP id n9KEkYfk003699; Tue, 20 Oct 2009 09:46:34 -0500 (CDT) Received: (from nw141292@localhost) by binky.Central.Sun.COM (8.14.3+Sun/8.14.3/Submit) id n9KEkYwC003698; Tue, 20 Oct 2009 09:46:34 -0500 (CDT) Date: Tue, 20 Oct 2009 09:46:34 -0500 From: Nicolas Williams Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <4ADD8821.9010806@Sun.COM> To: Darren J Moffat Cc: Adam Leventhal , PSARC-ext@sun.com, Jeff.Bonwick@sun.com, zfs-team@sun.com Message-id: <20091020144634.GQ892@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD8821.9010806@Sun.COM> X-Authentication-warning: binky.Central.Sun.COM: nw141292 set sender to Nicolas.Williams@sun.com using -f User-Agent: Mutt/1.5.7i Status: RO Content-Length: 1750 On Tue, Oct 20, 2009 at 10:51:29AM +0100, Darren J Moffat wrote: > Glad that you do offer verify as a choice. It would be very useful to > provide some sort of log output for the cases where verify found a > collision - ie the checksum hashes matched but the verify said they were > different. Not useful to end users so it could be a DTrace SDT or only > in a DEBUG kernel. If this ever shows up a "hit" when > dedup=sha256,verify it will make ZFS famous for finding collisions in > SHA256. A collision log for debug purposes would be nice. But collision stats should be provided in any case because such stats can be useful to estimitating the usefulness of a hash function for this purpose (how much time is spent computing hashes vs. how much time is spent verifying blocks, and how many blocks do collide). Collision stats could also be a useful way to build confidence in a hash function ("look! 0 collisions for SHA-3 candidate X on a 1PB pool with random and real data!"). Of course, a SHA-3 candidate must build confidence by surviving known cryptanalysis techniques + any new ones that cryptographers throw at it, and no collisions in 1PB hardly constitutes proof, but N>0 collisions in 1PB would be likely be worrisome and indicative that additional analysis is needed. Yes, a flight of fancy, maybe just eye candy ("ah, SHA-256 seems to be working as advertised"), but if so, it'd be cheap eye candy. % zpool get ddhashcolls,ddcollrate rpool NAME PROPERTY VALUE SOURCE rpool ddhashcolls 5 - rpool ddcollrate .0135 - % (One prop would count total collisions ever seen, the other would be a ration of the first and the pool size.) Nico -- From Darren.Moffat@sun.com Tue Oct 20 09:37:43 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9KGbgkV019731 for ; Tue, 20 Oct 2009 09:37:42 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n9KGbdBt009786; Tue, 20 Oct 2009 17:37:41 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRT0011HNIQVK00@nwk-avmta-2.sfbay.sun.com>; Tue, 20 Oct 2009 09:37:38 -0700 (PDT) Received: from gmp-eb-inf-1.sun.com ([192.18.6.21]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRT00EY3NIPTND0@nwk-avmta-2.sfbay.sun.com>; Tue, 20 Oct 2009 09:37:38 -0700 (PDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-1-fe1.eu.sun.com [192.18.6.7] (may be forged)) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n9KGbaZN021674; Tue, 20 Oct 2009 16:37:37 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KRT00E00NCVNM00@fe-emea-10.sun.com>; Tue, 20 Oct 2009 17:37:23 +0100 (BST) Received: from [192.168.1.105] (cpc2-rdng20-2-0-cust917.15-3.cable.virginmedia.com [86.28.167.150]) by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KRT001DENHWNUD0@fe-emea-10.sun.com>; Tue, 20 Oct 2009 17:37:09 +0100 (BST) Date: Tue, 20 Oct 2009 17:37:07 +0100 From: Darren J Moffat Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <4ADD2D08.50805@sun.com> Sender: Darren.Moffat@sun.com To: George Wilson Cc: Matthew Ahrens , zfs-team@sun.com, PSARC-ext@sun.com, ahl@eng.sun.com, Jeff.Bonwick@sun.com, Adam Leventhal Message-id: <4ADDE733.5030708@Sun.COM> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD1C73.5090407@sun.com> <4ADD2D08.50805@sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090929) Status: RO Content-Length: 998 George Wilson wrote: > The reason we added dedupchecksum was to allow us to change the default > checksum should we ever find something better than sha256 (in > anticipation of sha-3). Normally, we would just make the new algorithm > the default and then allow future blocks to leverage that checksum > function. The problem is that if the pool has already been using sha256 > and we change the default then new blocks would no longer de-dup against > existing blocks. By having the dedupchecksum we use this to always > define what the default 'on' value maps to. That makes sense and isn't needed for checksum/compression. A similar thing might be useful for the encryption property since unlike checksum/compression it uses only a single bit in the May I suggest that these types of properties be named to make it clear they are just a default for the "on" value, eg one of the following: default-dedupchecksum dedupdefaultchecksum dedup-default-checksum -- Darren J Moffat From ahl@eng.sun.com Tue Oct 20 10:11:25 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9KHBO4n021087 for ; Tue, 20 Oct 2009 10:11:25 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n9KHBLun058097; Tue, 20 Oct 2009 11:11:24 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRT0032BP2YXJ00@nwk-avmta-2.sfbay.sun.com>; Tue, 20 Oct 2009 10:11:22 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRT003R6P2XFV00@nwk-avmta-2.sfbay.sun.com>; Tue, 20 Oct 2009 10:11:21 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n9KHBLVS007050; Tue, 20 Oct 2009 17:11:21 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KRT00700N0L1B00@mail-amer.sun.com>; Tue, 20 Oct 2009 11:11:21 -0600 (MDT) Received: from [192.168.221.125] ([unknown] [198.144.208.55]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KRT00BENP2SYJ10@mail-amer.sun.com>; Tue, 20 Oct 2009 11:11:17 -0600 (MDT) Date: Tue, 20 Oct 2009 10:11:17 -0700 From: Adam Leventhal Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <200910200755.n9K7tSZB019251@dm-holland-02.uk.sun.com> Sender: Adam.Leventhal@sun.com To: Casper.Dik@sun.com Cc: Matthew Ahrens , PSARC-ext@sun.com, Jeff Bonwick , zfs-team@sun.com Message-id: <41159E80-FE82-4504-8157-49DC65A6A036@eng.sun.com> MIME-version: 1.0 X-Mailer: Apple Mail (2.1076) Content-type: text/plain; CHARSET=US-ASCII; delsp=yes; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD1C73.5090407@sun.com> <1714E736-768F-4F46-9DCB-DBEBB1AF9132@eng.sun.com> <200910200755.n9K7tSZB019251@dm-holland-02.uk.sun.com> Status: RO Content-Length: 477 >>> 1. don't allow fletcher2,verify. (we only allow checksum=fletcher2 >>> for backwards compatability anyway) >> >> I think you'll see this was called out explicitly in section B.1. > > Is fletcher2 still the default? The zfs manual claims that. It is no longer the default. If you wish to discuss this further, I suggest you take it off this list as it's off-topic for the PSARC case. Adam -- Adam Leventhal, Fishworks http://blogs.sun.com/ahl From ahl@basura.sf.fishpong.com Tue Oct 20 10:24:55 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9KHOsbD021573 for ; Tue, 20 Oct 2009 10:24:55 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n9KHOnpk011985; Tue, 20 Oct 2009 18:24:52 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRT0040TPPE4P00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 20 Oct 2009 10:24:50 -0700 (PDT) Received: from zion.sfbay.sun.com ([129.146.17.75]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRT00DAOPPDS3E0@nwk-avmta-1.sfbay.Sun.COM>; Tue, 20 Oct 2009 10:24:49 -0700 (PDT) Received: from basura.sf.fishpong.com (fgw.SFBay.Sun.COM [10.7.251.167]) by zion.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n9KHNc3a027877; Tue, 20 Oct 2009 17:23:38 +0000 (GMT) Received: from basura.sf.fishpong.com (localhost [127.0.0.1]) by basura.sf.fishpong.com (8.14.2+Sun/8.14.2) with ESMTP id n9KHQMxe129240; Tue, 20 Oct 2009 17:26:22 +0000 (GMT) Received: (from ahl@localhost) by basura.sf.fishpong.com (8.14.2+Sun/8.14.2/Submit) id n9KHQMwK129239; Tue, 20 Oct 2009 10:26:22 -0700 (PDT) Date: Tue, 20 Oct 2009 10:26:22 -0700 From: Adam Leventhal Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <4ADD8821.9010806@Sun.COM> To: Darren J Moffat Cc: PSARC-ext@sun.com, Jeff.Bonwick@sun.com, zfs-team@sun.com Message-id: <20091020172622.GH118655@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD8821.9010806@Sun.COM> User-Agent: Mutt/1.5.17 (2007-11-01) Status: RO Content-Length: 3939 On Tue, Oct 20, 2009 at 10:51:29AM +0100, Darren J Moffat wrote: > Is this valid: > > zfs set checksum=fletcher4 tank > zfs set dedup=sha256 > > If so what is stored on disk in the blkptr_t ? I assume it is a fletecher4 > stored there. Where is the sha256 checksum stored then ? In the DDT ? The dedup checksum supercedes the value of checksum if dedup is set to anything other than 'off'. Thanks for the question -- that's an important clarification. > When in the ZIO pipeline is the checksum specified with the dedup property > calculated ? I'm assuming it is in zio_write_bp_init() after compression > and after encryption so it is on the state of the block exactly as it will > be written to disk. > > Can gang blocks be deduplicated ? These are obviously not questions about the interface so may be straying a bit far afield, but there is only one checksum which is computed after compression and encryption, and gang blocks can be deduplicated. > Glad that you do offer verify as a choice. It would be very useful to > provide some sort of log output for the cases where verify found a > collision - ie the checksum hashes matched but the verify said they were > different. Not useful to end users so it could be a DTrace SDT or only in > a DEBUG kernel. If this ever shows up a "hit" when dedup=sha256,verify it > will make ZFS famous for finding collisions in SHA256. Agreed. I'd like to defer that discussion for a later PSARC case as it is not part of the initial deliverable. >> As an explicit request for input from the ARC, our fletcher2 implementation >> has been shown to be suboptimal and results in a large number of >> collisions (as a result, the default checksum has been changed to >> fletcher4). Should 'fletcher2,verify' be permitted as an option for >> consistency or should we eliminate that option since it would rarely >> be an attractive choice for users due to the high number of hash >> collisions. > > I don't think fletcher2,verify should be provided. That seems to be the growing concensus. > Why is this needed when we don't have a pool level property for > the default checksum or compression or encryption (or any other property > that is inherited and has an "on | off | ..." style of setting ? > > Under what circumstances would this be needed rather than setting dedup to > the required value ? > > Is this a new precendent that all properties with an "on" value should have > a pool level property to determine what "on" is ? That's a fair point. The rationale behind this is that we think a) it's more likely to change given the advances in cryptographic checksums and b) while we wouldn't hesitate to change the meaning of other 'on' values, changing the meaning of this one would mean that a completely different lookup table were used. Therefore we want to encode the dataset default at the level of the pool (again, since the dedup table (DDT) is pool-wide). >> The second allows the administrator to select a threshhold afterwhich >> 2 copies of a block are stored rather than 1. For example, if many >> duplicate blocks exist deduplication would reduce that count to just 1; >> at some threshhold, it becomes desirable to have multiple copies to >> guard against the multiplied effects of the loss of a single block. >> The default value is '100'. > > Is it actually necessary to expose this tunable ? Given there is already a > per dataset copies property how does this interact with that ? Good questions. The per-dataset ncopies property is obeyed in that if one were to dedup many blocks with ncopies set to 2 there would result in 2 copies total after deduplication. The 'dedupditto' property guides what we're calling auto-ditto in which ZFS chooses to store an additional copy once some threshhold is reached. This is independent of the per-dataset 'ncopies' property. Adam -- Adam Leventhal, Fishworks http://blogs.sun.com/ahl From ahl@basura.sf.fishpong.com Tue Oct 20 10:58:21 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9KHwKef022538 for ; Tue, 20 Oct 2009 10:58:20 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n9KHwEgU006671; Tue, 20 Oct 2009 18:58:15 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRT0080RR92CW00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 20 Oct 2009 10:58:14 -0700 (PDT) Received: from zion.sfbay.sun.com ([129.146.17.75]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRT00403R92K020@nwk-avmta-1.sfbay.Sun.COM>; Tue, 20 Oct 2009 10:58:14 -0700 (PDT) Received: from basura.sf.fishpong.com (fgw.SFBay.Sun.COM [10.7.251.167]) by zion.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n9KHv3lp028857; Tue, 20 Oct 2009 17:57:04 +0000 (GMT) Received: from basura.sf.fishpong.com (localhost [127.0.0.1]) by basura.sf.fishpong.com (8.14.2+Sun/8.14.2) with ESMTP id n9KHxlPH129278; Tue, 20 Oct 2009 17:59:47 +0000 (GMT) Received: (from ahl@localhost) by basura.sf.fishpong.com (8.14.2+Sun/8.14.2/Submit) id n9KHxlfV129277; Tue, 20 Oct 2009 10:59:47 -0700 (PDT) Date: Tue, 20 Oct 2009 10:59:47 -0700 From: Adam Leventhal Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <4ADDC403.3040705@sun.com> To: "Garrett D'Amore" Cc: Jeff Bonwick , George Wilson , PSARC-ext@sun.com, zfs-team@sun.com Message-id: <20091020175947.GK118655@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD17DA.9070104@sun.com> <4ADD2A94.5060909@sun.com> <20091020062851.GD28054@eng.sun.com> <4ADDC403.3040705@sun.com> User-Agent: Mutt/1.5.17 (2007-11-01) Status: RO Content-Length: 907 On Tue, Oct 20, 2009 at 07:06:59AM -0700, Garrett D'Amore wrote: > If you set it at "2", then you don't have a sudden "oh usage doubled" > moment. It means that you immediately gain a redundancy advantage. The > drawback is the space savings of the ditto doesn't reap rewards until you > have the third reference, and of course for blocks that aren't heavily > duplicated, you do wind up using double the space than you would without > it. But the dedup is *still* a significant win over traditional storage. To clarify, dedupditto is a threshhold. With the value set to 2 the following would apply: 1 instance 1 copy on disk 2 instances 1 copy on disk 3 instances 2 copies on disk (since the threshhold of two is exceeded) 4 instances 2 copies on disk ... > The case has my +1 in any event. Excellent. Thanks. Adam -- Adam Leventhal, Fishworks http://blogs.sun.com/ahl From gdamore@sun.com Tue Oct 20 11:51:16 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9KIpGh1023911 for ; Tue, 20 Oct 2009 11:51:16 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n9KIpF8Q007136; Tue, 20 Oct 2009 11:51:15 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRT00H09TPF9J00@brm-avmta-1.central.sun.com>; Tue, 20 Oct 2009 12:51:15 -0600 (MDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRT00C4PTPEDCF0@brm-avmta-1.central.sun.com>; Tue, 20 Oct 2009 12:51:15 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n9KIpEH7021838; Tue, 20 Oct 2009 11:51:14 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KRT00A00T8UYC00@fe-sfbay-09.sun.com>; Tue, 20 Oct 2009 11:51:14 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KRT00C2MTPDH1B0@fe-sfbay-09.sun.com>; Tue, 20 Oct 2009 11:51:14 -0700 (PDT) Date: Tue, 20 Oct 2009 11:51:13 -0700 From: "Garrett D'Amore" Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <20091020144634.GQ892@Sun.COM> Sender: Garrett.Damore@sun.com To: Nicolas Williams Cc: Darren J Moffat , Adam Leventhal , PSARC-ext@sun.com, jeff.bonwick@sun.com, zfs-team@sun.com Message-id: <4ADE06A1.8050204@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD8821.9010806@Sun.COM> <20091020144634.GQ892@Sun.COM> User-Agent: Thunderbird 2.0.0.22 (X11/20090909) Status: RO Content-Length: 1973 Nicolas Williams wrote: > On Tue, Oct 20, 2009 at 10:51:29AM +0100, Darren J Moffat wrote: > >> Glad that you do offer verify as a choice. It would be very useful to >> provide some sort of log output for the cases where verify found a >> collision - ie the checksum hashes matched but the verify said they were >> different. Not useful to end users so it could be a DTrace SDT or only >> in a DEBUG kernel. If this ever shows up a "hit" when >> dedup=sha256,verify it will make ZFS famous for finding collisions in >> SHA256. >> > > A collision log for debug purposes would be nice. But collision stats > should be provided in any case because such stats can be useful to > estimitating the usefulness of a hash function for this purpose (how > much time is spent computing hashes vs. how much time is spent verifying > blocks, and how many blocks do collide). > > Collision stats could also be a useful way to build confidence in a hash > function ("look! 0 collisions for SHA-3 candidate X on a 1PB pool with > random and real data!"). Of course, a SHA-3 candidate must build > confidence by surviving known cryptanalysis techniques + any new ones > that cryptographers throw at it, and no collisions in 1PB hardly > constitutes proof, but N>0 collisions in 1PB would be likely be > worrisome and indicative that additional analysis is needed. Yes, a > flight of fancy, maybe just eye candy ("ah, SHA-256 seems to be working > as advertised"), but if so, it'd be cheap eye candy. > > % zpool get ddhashcolls,ddcollrate rpool > NAME PROPERTY VALUE SOURCE > rpool ddhashcolls 5 - > rpool ddcollrate .0135 - > % > > (One prop would count total collisions ever seen, the other would be a > ration of the first and the pool size.) > How about just a kstat where it can be located easily for debug, without polluting normal zfs properties? - Garrett > Nico > From Darren.Moffat@sun.com Tue Oct 20 12:07:03 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9KJ73Xc024565 for ; Tue, 20 Oct 2009 12:07:03 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n9KJ72aO013843; Tue, 20 Oct 2009 12:07:03 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRT00B0NUFR0B00@nwk-avmta-2.sfbay.sun.com>; Tue, 20 Oct 2009 12:07:03 -0700 (PDT) Received: from gmp-eb-inf-1.sun.com ([192.18.6.21]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRT003PCUFPFPB0@nwk-avmta-2.sfbay.sun.com>; Tue, 20 Oct 2009 12:07:02 -0700 (PDT) Received: from fe-emea-09.sun.com (gmp-eb-lb-1-fe1.eu.sun.com [192.18.6.7] (may be forged)) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n9KJ714p029588; Tue, 20 Oct 2009 19:07:01 +0000 (GMT) Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KRT00800UAYT200@fe-emea-09.sun.com>; Tue, 20 Oct 2009 20:06:41 +0100 (BST) Received: from [192.168.1.105] (cpc2-rdng20-2-0-cust917.15-3.cable.virginmedia.com [86.28.167.150]) by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KRT00EEEUF4LOE0@fe-emea-09.sun.com>; Tue, 20 Oct 2009 20:06:41 +0100 (BST) Date: Tue, 20 Oct 2009 20:06:40 +0100 From: Darren J Moffat Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <4ADE06A1.8050204@sun.com> Sender: Darren.Moffat@sun.com To: "Garrett D'Amore" Cc: Nicolas Williams , Adam Leventhal , PSARC-ext@sun.com, Jeff.Bonwick@sun.com, zfs-team@sun.com Message-id: <4ADE0A40.7000008@Sun.COM> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD8821.9010806@Sun.COM> <20091020144634.GQ892@Sun.COM> <4ADE06A1.8050204@sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090929) Status: RO Content-Length: 2247 Garrett D'Amore wrote: > Nicolas Williams wrote: >> On Tue, Oct 20, 2009 at 10:51:29AM +0100, Darren J Moffat wrote: >> >>> Glad that you do offer verify as a choice. It would be very useful >>> to provide some sort of log output for the cases where verify found a >>> collision - ie the checksum hashes matched but the verify said they >>> were different. Not useful to end users so it could be a DTrace SDT >>> or only in a DEBUG kernel. If this ever shows up a "hit" when >>> dedup=sha256,verify it will make ZFS famous for finding collisions in >>> SHA256. >>> >> >> A collision log for debug purposes would be nice. But collision stats >> should be provided in any case because such stats can be useful to >> estimitating the usefulness of a hash function for this purpose (how >> much time is spent computing hashes vs. how much time is spent verifying >> blocks, and how many blocks do collide). >> >> Collision stats could also be a useful way to build confidence in a hash >> function ("look! 0 collisions for SHA-3 candidate X on a 1PB pool with >> random and real data!"). Of course, a SHA-3 candidate must build >> confidence by surviving known cryptanalysis techniques + any new ones >> that cryptographers throw at it, and no collisions in 1PB hardly >> constitutes proof, but N>0 collisions in 1PB would be likely be >> worrisome and indicative that additional analysis is needed. Yes, a >> flight of fancy, maybe just eye candy ("ah, SHA-256 seems to be working >> as advertised"), but if so, it'd be cheap eye candy. >> >> % zpool get ddhashcolls,ddcollrate rpool >> NAME PROPERTY VALUE SOURCE >> rpool ddhashcolls 5 - >> rpool ddcollrate .0135 - >> % >> (One prop would count total collisions ever seen, the other would be a >> ration of the first and the pool size.) >> > > How about just a kstat where it can be located easily for debug, without > polluting normal zfs properties? kstat's don't persist over reboot or pool export/import. But I agree with Adam this is a future nice to have feature that is more about debugging than run time stats not a requirement for dedup's first integration. -- Darren J Moffat From Scott.Rotondo@sun.com Tue Oct 20 12:08:41 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9KJ8ek7024586 for ; Tue, 20 Oct 2009 12:08:40 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n9KJ8dln009835; Tue, 20 Oct 2009 12:08:40 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRT00B0ZUIE3Q00@nwk-avmta-2.sfbay.sun.com>; Tue, 20 Oct 2009 12:08:38 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRT0031BUIEFMA0@nwk-avmta-2.sfbay.sun.com>; Tue, 20 Oct 2009 12:08:38 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n9KJ8b6T028985; Tue, 20 Oct 2009 19:08:37 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KRT00200TYTU800@mail-amer.sun.com>; Tue, 20 Oct 2009 13:08:37 -0600 (MDT) Received: from [129.146.108.62] ([unknown] [129.146.108.62]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KRT00I3UUI0FS80@mail-amer.sun.com>; Tue, 20 Oct 2009 13:08:25 -0600 (MDT) Date: Tue, 20 Oct 2009 12:08:24 -0700 From: Scott Rotondo Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <20091020062851.GD28054@eng.sun.com> Sender: Scott.Rotondo@sun.com To: Jeff Bonwick Cc: George Wilson , "Garrett D'Amore" , Adam Leventhal , PSARC-ext@sun.com, ahl@eng.sun.com, zfs-team@sun.com Message-id: <4ADE0AA8.6030309@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD17DA.9070104@sun.com> <4ADD2A94.5060909@sun.com> <20091020062851.GD28054@eng.sun.com> User-Agent: Thunderbird 2.0.0.21 (X11/20090323) Status: RO Content-Length: 1715 Jeff Bonwick wrote: >> The idea is that blocks with fewer than 100 references aren't as >> critical and thus we don't need to make an extra copy. It boils down to >> a policy decision between safety vs. space savings. > > There are a couple of factors that lead to this particular number: > > - We always ditto indirect blocks, which represent up to 128 > data blocks. The rationale is that loss of an indirect block > would imply loss of 128 data blocks, so it's worth the cost. > The same reasoning applies to a block that's deduped and > referenced 128 times. We're rounding to 100 to be less geeky. > > - Unlike all other user data, dedupditto blocks are not billed > to any particular user (because they do not belong to any > particular dataset). Therefore, we would like to limit the > maximum possible space overhead to something comparable to the > overall metadata overhead, which is about 1%. Hence 100:1. > > I am open to the possibility of dedupditto being off by default, with > 100 as the minimum legal value. The first factor would seem to argue for 100 as a maximum, not minimum, legal value. You've already decided that it's always unacceptable to risk 100 (actually, 128) data blocks on a single copy of an indirect block. Why would you be willing to risk a larger number of dedup data blocks on a single copy? Similarly, a 1% overhead for dedup ditto blocks sounds like a reasonable default. But I could imagine someone willingly paying a 2% overhead by dropping the value to 50, for example. Scott -- Scott Rotondo Principal Engineer, Solaris Security Technologies President, Trusted Computing Group Phone/FAX: +1 408 850 3655 (Internal x68278) From Jeff.Bonwick@sun.com Tue Oct 20 15:16:17 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9KMGG8A000624 for ; Tue, 20 Oct 2009 15:16:17 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n9KMFjAd057993; Tue, 20 Oct 2009 16:16:15 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRU00F1L36R5L00@brm-avmta-1.central.sun.com>; Tue, 20 Oct 2009 16:16:03 -0600 (MDT) Received: from cathy.local ([129.146.228.63]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRU00HDO36QBEC0@brm-avmta-1.central.sun.com>; Tue, 20 Oct 2009 16:16:02 -0600 (MDT) Received: from cathy.sfbay.sun.com (localhost [127.0.0.1]) by cathy.local (8.14.3+Sun/8.14.3) with ESMTP id n9KMG25I022805; Tue, 20 Oct 2009 15:16:02 -0700 (PDT) Received: (from bonwick@localhost) by cathy.sfbay.sun.com (8.14.3+Sun/8.14.3/Submit) id n9KMG13C022804; Tue, 20 Oct 2009 15:16:01 -0700 (PDT) Date: Tue, 20 Oct 2009 15:16:01 -0700 From: Jeff Bonwick Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <4ADE0AA8.6030309@sun.com> To: Scott Rotondo Cc: Jeff Bonwick , George Wilson , "Garrett D'Amore" , Adam Leventhal , PSARC-ext@sun.com, ahl@eng.sun.com, zfs-team@sun.com Message-id: <20091020221601.GB22776@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD17DA.9070104@sun.com> <4ADD2A94.5060909@sun.com> <20091020062851.GD28054@eng.sun.com> <4ADE0AA8.6030309@sun.com> X-Authentication-warning: cathy.sfbay.sun.com: bonwick set sender to Jeff.Bonwick@sun.com using -f User-Agent: Mutt/1.5.14 (2007-02-12) Status: RO Content-Length: 422 > Similarly, a 1% overhead for dedup ditto blocks sounds like a reasonable > default. But I could imagine someone willingly paying a 2% overhead by > dropping the value to 50, for example. Quite possible. It's just a question of what we're willing to test/support. If we find that there's demand for a lower threshold, it will be trivial to accomodate -- it's just a #define with no on-disk format implications. Jeff From sommerfeld@sun.com Tue Oct 20 16:23:27 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9KNNQ4b001561 for ; Tue, 20 Oct 2009 16:23:27 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n9KNNHA6017564; Wed, 21 Oct 2009 00:23:25 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRU0040D6B0UO00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 20 Oct 2009 16:23:24 -0700 (PDT) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRU001I56AZ1QC0@nwk-avmta-1.sfbay.Sun.COM>; Tue, 20 Oct 2009 16:23:23 -0700 (PDT) Received: from thunk-west.local (dhcp-umpk17-109-232.SFBay.Sun.COM [129.146.109.232]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n9KNNNM2000648; Tue, 20 Oct 2009 16:23:23 -0700 (PDT) Received: from thunk-west.local (thunk-west [127.0.0.1]) by thunk-west.local (8.14.3+Sun/8.14.3) with ESMTP id n9KNNM2G000464; Tue, 20 Oct 2009 16:23:23 -0700 (PDT) Received: (from sommerfeld@localhost) by thunk-west.local (8.14.3+Sun/8.14.3/Submit) id n9KNNMqc000463; Tue, 20 Oct 2009 16:23:22 -0700 (PDT) Date: Tue, 20 Oct 2009 16:23:22 -0700 From: Bill Sommerfeld Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> To: Adam Leventhal Cc: PSARC-ext@sun.com, Jeff.Bonwick@sun.com, ahl@eng.sun.com, zfs-team@sun.com Message-id: <1256081002.23138.36.camel@thunk-west> MIME-version: 1.0 Content-type: text/plain; charset=ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> X-Authentication-warning: thunk-west.local: sommerfeld set sender to sommerfeld@sun.com using -f Status: RO Content-Length: 746 On Mon, 2009-10-19 at 18:13 -0700, Adam Leventhal wrote: > [dedupditto] allows the administrator to select a threshhold after > which 2 copies of a block are stored rather than 1. For example, if > many duplicate blocks exist deduplication would reduce that count to > just 1; at some threshhold, it becomes desirable to have multiple > copies to guard against the multiplied effects of the loss of a single > block. The default value is '100'. What's the interaction between the zfs "copies" property and "dedupditto" ? (my hope would be that if any filesystem referencing a block has copies=2 or copies=3 at the time the reference is created, that block will be dittoed the appropriate number of times if it wasn't already). - Bill From ahl@eng.sun.com Tue Oct 20 16:31:12 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9KNVB3K001661 for ; Tue, 20 Oct 2009 16:31:11 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n9KNUxkl000081; Wed, 21 Oct 2009 07:31:10 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRU0000R6NU9C00@brm-avmta-1.central.sun.com>; Tue, 20 Oct 2009 17:31:06 -0600 (MDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRU00GYR6NURC40@brm-avmta-1.central.sun.com>; Tue, 20 Oct 2009 17:31:06 -0600 (MDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n9KNV61x012990; Tue, 20 Oct 2009 23:31:06 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KRU006006EKYE00@mail-amer.sun.com>; Tue, 20 Oct 2009 17:31:06 -0600 (MDT) Received: from [192.168.221.125] ([unknown] [198.144.208.55]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KRU0016Y6NTBH30@mail-amer.sun.com>; Tue, 20 Oct 2009 17:31:06 -0600 (MDT) Date: Tue, 20 Oct 2009 16:31:05 -0700 From: Adam Leventhal Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <1256081002.23138.36.camel@thunk-west> Sender: Adam.Leventhal@sun.com To: Bill Sommerfeld Cc: Adam Leventhal , PSARC-ext@sun.com, Jeff.Bonwick@sun.com, zfs-team@sun.com Message-id: <71C8AFBC-65A7-4C1F-95FF-42C3FA1073F3@eng.sun.com> MIME-version: 1.0 X-Mailer: Apple Mail (2.1076) Content-type: text/plain; CHARSET=US-ASCII; delsp=yes; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <1256081002.23138.36.camel@thunk-west> Status: RO Content-Length: 936 >> [dedupditto] allows the administrator to select a threshhold after >> which 2 copies of a block are stored rather than 1. For example, if >> many duplicate blocks exist deduplication would reduce that count to >> just 1; at some threshhold, it becomes desirable to have multiple >> copies to guard against the multiplied effects of the loss of a >> single >> block. The default value is '100'. > > What's the interaction between the zfs "copies" property and > "dedupditto" ? > > (my hope would be that if any filesystem referencing a block has > copies=2 or copies=3 at the time the reference is created, that block > will be dittoed the appropriate number of times if it wasn't already). That's correct. A block with copies set to N will be always have at least N copies on the system regardless of the number of deduplicated references. Adam -- Adam Leventhal, Fishworks http://blogs.sun.com/ahl From Victor.Latushkin@sun.com Wed Oct 21 04:17:24 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9LBHNsi024587 for ; Wed, 21 Oct 2009 04:17:23 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n9LBHKWW024053; Wed, 21 Oct 2009 19:17:22 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRV00A013CX3C00@nwk-avmta-2.sfbay.sun.com>; Wed, 21 Oct 2009 04:17:21 -0700 (PDT) Received: from gmp-eb-inf-1.sun.com ([192.18.6.21]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRV009V33CV4G80@nwk-avmta-2.sfbay.sun.com>; Wed, 21 Oct 2009 04:17:20 -0700 (PDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-1-fe1.eu.sun.com [192.18.6.7] (may be forged)) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n9LBHJLM014039; Wed, 21 Oct 2009 11:17:19 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KRV0030031S1Z00@fe-emea-10.sun.com>; Wed, 21 Oct 2009 12:17:13 +0100 (BST) Received: from [129.156.248.32] ([unknown] [129.156.248.32]) by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KRV004OP3CJ9V40@fe-emea-10.sun.com>; Wed, 21 Oct 2009 12:17:09 +0100 (BST) Date: Wed, 21 Oct 2009 15:17:07 +0400 From: Victor Latushkin Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <20091020172622.GH118655@eng.sun.com> Sender: Victor.Latushkin@sun.com To: Adam Leventhal Cc: Darren J Moffat , PSARC-ext@sun.com, Jeff.Bonwick@sun.com, zfs-team@sun.com Reply-to: Victor.Latushkin@sun.com Message-id: <4ADEEDB3.8090101@Sun.COM> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD8821.9010806@Sun.COM> <20091020172622.GH118655@eng.sun.com> User-Agent: Thunderbird 2.0.0.21 (X11/20090311) Status: RO Content-Length: 1193 On 20.10.09 21:26, Adam Leventhal wrote: >>> The second allows the administrator to select a threshhold afterwhich >>> 2 copies of a block are stored rather than 1. For example, if many >>> duplicate blocks exist deduplication would reduce that count to just 1; >>> at some threshhold, it becomes desirable to have multiple copies to >>> guard against the multiplied effects of the loss of a single block. >>> The default value is '100'. >> Is it actually necessary to expose this tunable ? Given there is already a >> per dataset copies property how does this interact with that ? > > Good questions. > > The per-dataset ncopies property is obeyed in that if one were to dedup many > blocks with ncopies set to 2 there would result in 2 copies total after > deduplication. > > The 'dedupditto' property guides what we're calling auto-ditto in which ZFS > chooses to store an additional copy once some threshhold is reached. This is > independent of the per-dataset 'ncopies' property. What does happen to the auto-ditto copies once reference count drops below threshold? Are auto-ditto-copies freed immediately or when reference count drops below e.g. half-threshold or never? Victor From ahl@eng.sun.com Wed Oct 21 14:22:39 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9LLMb1W011726 for ; Wed, 21 Oct 2009 14:22:37 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n9LLMV6q008175; Wed, 21 Oct 2009 22:22:36 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KRV00119VDMQG00@nwk-avmta-2.sfbay.sun.com>; Wed, 21 Oct 2009 14:22:34 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KRV00ISDVDLMZA0@nwk-avmta-2.sfbay.sun.com>; Wed, 21 Oct 2009 14:22:33 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n9LLMXa3027826; Wed, 21 Oct 2009 21:22:33 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KRV00600V7HWZ00@mail-amer.sun.com>; Wed, 21 Oct 2009 15:22:33 -0600 (MDT) Received: from [192.168.221.129] ([unknown] [198.144.208.55]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KRV007GSVD13R10@mail-amer.sun.com>; Wed, 21 Oct 2009 15:22:14 -0600 (MDT) Date: Wed, 21 Oct 2009 14:22:12 -0700 From: Adam Leventhal Subject: Re: ZFS Deduplication Properties [PSARC/2009/571 FastTrack timeout 10/21/2009] In-reply-to: <4ADEEDB3.8090101@Sun.COM> Sender: Adam.Leventhal@sun.com To: Victor.Latushkin@sun.com Cc: Darren J Moffat , PSARC-ext@sun.com, Jeff.Bonwick@sun.com, zfs-team@sun.com Message-id: MIME-version: 1.0 X-Mailer: Apple Mail (2.1076) Content-type: text/plain; CHARSET=US-ASCII; delsp=yes; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910200113.n9K1DM84766368@zday.sfbay.sun.com> <4ADD8821.9010806@Sun.COM> <20091020172622.GH118655@eng.sun.com> <4ADEEDB3.8090101@Sun.COM> Status: RO Content-Length: 1035 >> The per-dataset ncopies property is obeyed in that if one were to >> dedup many >> blocks with ncopies set to 2 there would result in 2 copies total >> after >> deduplication. >> The 'dedupditto' property guides what we're calling auto-ditto in >> which ZFS >> chooses to store an additional copy once some threshhold is >> reached. This is >> independent of the per-dataset 'ncopies' property. > > What does happen to the auto-ditto copies once reference count drops > below threshold? Are auto-ditto-copies freed immediately or when > reference count drops below e.g. half-threshold or never? When the reference count drops below the threshold the second block is deleted. Yes, the possibility of creating and deleting exists if we're right on the boundary of the threshold, but that degenerate case is identical to what would happen without dedup: deletion of a block deletes a block and creation of a block creates a block. Adam -- Adam Leventhal, Fishworks http://blogs.sun.com/ahl