From Tim.Haley@sun.com Fri Jun 20 11:26:03 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5KIQ28e003203 for ; Fri, 20 Jun 2008 11:26:02 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m5KIPxD1024019; Sat, 21 Jun 2008 02:26:01 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K2R0090BXVB3T00@nwk-avmta-2.sfbay.sun.com>; Fri, 20 Jun 2008 11:25:59 -0700 (PDT) Received: from dm-central-01.central.sun.com ([129.147.62.4]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K2R007GDXVAQP20@nwk-avmta-2.sfbay.sun.com>; Fri, 20 Jun 2008 11:25:58 -0700 (PDT) Received: from spidey.Central.Sun.COM (spidey.Central.Sun.COM [172.20.25.27]) by dm-central-01.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m5KIPwV7018505; Fri, 20 Jun 2008 12:25:58 -0600 (MDT) Received: from spidey.Central.Sun.COM (localhost [127.0.0.1]) by spidey.Central.Sun.COM (8.14.3+Sun/8.14.3) with ESMTP id m5KIISgO028826; Fri, 20 Jun 2008 12:18:28 -0600 (MDT) Received: (from timh@localhost) by spidey.Central.Sun.COM (8.14.3+Sun/8.14.3/Submit) id m5KIISO0028824; Fri, 20 Jun 2008 12:18:28 -0600 (MDT) Date: Fri, 20 Jun 2008 12:18:28 -0600 (MDT) From: Tim Haley Subject: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] To: PSARC-ext@sun.com Cc: zfs-team@sun.com Message-id: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 3526 I am sponsoring the following fast-track on behalf of Eric Kustarz. Requested binding is micro/patch. Timeout 06/27/2008 (June 27). Template Version: @(#)sac_nextcase 1.66 04/17/08 SMI This information is Copyright 2008 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: zfs primarycache and secondarycache properties 1.2. Name of Document Author/Supplier: Author: Eric Kustarz 1.3 Date of This Document: 20 June, 2008 4. Technical Description A. SUMMARY This case adds two new dataset-level properties, 'primarycache' and 'secondarycache', to the existing zfs property infrastructure. These properties control what is cached in the primary cache (ARC) and the secondary cache (l2ARC). B. USE CASES One use case for not caching reads of user data is a large buffer cache (part of the SGA) for Oracle. If the admin wants to cache at the database level, then it's pointless and wasteful to attempt to cache the same data at the file system level as well. One use case for not caching writes of user data is the log writer for Oracle. This data is only needed to redo transactions in the case of a panic / unplanned shutdown. We've also had users on zfs-discuss ask for the ability to control what is cached. Two such examples are noted here: http://www.opensolaris.org/jive/thread.jspa?threadID=52159&tstart=0 http://www.opensolaris.org/jive/thread.jspa?messageID=202791𱠧 One reason we need to separate the control of the primary cache from the secondary cache is to allow customers the ability to charge for use of the l2ARC. C. PROPOSED SOLUTION The introduction of the 'primarycache' property will allow administrators the ability to control what is cached in the ARC. The possible choices are: all, none, and metadata. If 'all' is chosen, then both user data and metadata are cached. If 'none' is chosen, then neither user data nor metadata is cached. If 'metadata' is chosen, then only metadata is cached. The default is 'all'. The introduction of the 'secondarycache' property will allow administrators the ability to control what is cached in the l2ARC. The possible choices are the exact same as 'primarycache'. The default is also 'all'. The syntax for setting these zfs properties is as follows: # zfs set primarycache=metadata # zfs create -o primarycache=metadata # zfs set secondarycache=none # zfs create -o secondarycache=none D. MANPAGE DIFFS The following text will be added under the "Properties" section in zfs(1M): primarycache=all | none | metadata Controls what is cached in the primary cache (ARC). If set to "all", then both user data and metadata is cached. If set to "none", then neither user data nor metadata is cached. If set to "metadata", then only metadata is cached. The default behavior is "all". secondarycache=all | none | metadata Controls what is cached in the secondary cache (l2ARC). If set to "all", then both user data and metadata is cached. If set to "none", then neither user data nor metadata is cached. If set to "metadata", then only metadata is cached. The default behavior is "all". Also, 'primarycache' and 'secondarycache' will be listed under allowable permissions for 'zfs allow' in zfs(1M). 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From sommerfeld@sun.com Fri Jun 20 13:20:51 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5KKKogn006820 for ; Fri, 20 Jun 2008 13:20:51 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m5KKKh8e012585; Fri, 20 Jun 2008 21:20:49 +0100 (BST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K2S00E0B36ODJ00@brm-avmta-1.central.sun.com>; Fri, 20 Jun 2008 14:20:48 -0600 (MDT) Received: from dm-east-02.east.sun.com ([129.148.13.5]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K2S00MYQ36MDTA0@brm-avmta-1.central.sun.com>; Fri, 20 Jun 2008 14:20:46 -0600 (MDT) Received: from thunk.east.sun.com (thunk.East.Sun.COM [129.148.174.66]) by dm-east-02.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m5KKKkhv030949; Fri, 20 Jun 2008 16:20:46 -0400 (EDT) Received: from [IPv6:::1] (localhost [IPv6:::1]) by thunk.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m5KKKkIm027401; Fri, 20 Jun 2008 16:20:46 -0400 (EDT) Date: Fri, 20 Jun 2008 16:20:45 -0400 From: Bill Sommerfeld Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> To: Tim Haley Cc: PSARC-ext@sun.com, zfs-team@sun.com Message-id: <1213993245.10441.13.camel@thunk> MIME-version: 1.0 X-Mailer: Evolution 2.12.2 Content-type: text/plain Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> Status: RO Content-Length: 325 Does setting these properties to a more-restrictive setting drop now-excluded cache entries immediately? (in other words, would setting primarycache=none and then setting it back to all serve to empty the cache of any entries from the affected datasets, and free up the corresponding system memory or cache disk space?) From eric.kustarz@sun.com Fri Jun 20 13:40:00 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5KKe03B007630 for ; Fri, 20 Jun 2008 13:40:00 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m5KKdxxa003422; Fri, 20 Jun 2008 13:39:59 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K2S00F0142N1K00@nwk-avmta-2.sfbay.sun.com>; Fri, 20 Jun 2008 13:39:59 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K2S007K142NQE90@nwk-avmta-2.sfbay.sun.com>; Fri, 20 Jun 2008 13:39:59 -0700 (PDT) Received: from punchin-client-10-7-250-182.SFBay.Sun.COM (punchin-client-10-7-250-182.SFBay.Sun.COM [10.7.250.182]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m5KKdwrZ950481; Fri, 20 Jun 2008 13:39:58 -0700 (PDT) Date: Fri, 20 Jun 2008 13:39:57 -0700 From: eric kustarz Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <1213993245.10441.13.camel@thunk> To: Bill Sommerfeld Cc: Tim Haley , PSARC-ext@sun.com, zfs-team@sun.com Message-id: MIME-version: 1.0 X-Mailer: Apple Mail (2.924) Content-type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <1213993245.10441.13.camel@thunk> Status: RO Content-Length: 463 On Jun 20, 2008, at 1:20 PM, Bill Sommerfeld wrote: > Does setting these properties to a more-restrictive setting drop > now-excluded cache entries immediately? (in other words, would > setting > primarycache=none and then setting it back to all serve to empty the > cache of any entries from the affected datasets, and free up the > corresponding system memory or cache disk space?) Clever, but no it does not - the properties only apply to new I/O. eric From Jeff.Bonwick@sun.com Sat Jun 21 00:06:48 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5L76l6Y017266 for ; Sat, 21 Jun 2008 00:06:48 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m5L76gQV018050; Sat, 21 Jun 2008 15:06:44 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K2S00007X37GV00@nwk-avmta-2.sfbay.sun.com>; Sat, 21 Jun 2008 00:06:43 -0700 (PDT) Received: from zion.sfbay.sun.com ([129.146.17.75]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K2S00879X376W90@nwk-avmta-2.sfbay.sun.com>; Sat, 21 Jun 2008 00:06:43 -0700 (PDT) Received: from zion.sfbay.sun.com (localhost [127.0.0.1]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m5L76hUN006572; Sat, 21 Jun 2008 07:06:43 +0000 (GMT) Received: (from bonwick@localhost) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2/Submit) id m5L76gbZ006571; Sat, 21 Jun 2008 00:06:42 -0700 (PDT) Date: Sat, 21 Jun 2008 00:06:42 -0700 From: Jeff Bonwick Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: To: eric kustarz Cc: Bill Sommerfeld , Tim Haley , PSARC-ext@sun.com, zfs-team@sun.com Message-id: <20080621070641.GA6495@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <1213993245.10441.13.camel@thunk> X-Authentication-warning: zion.sfbay.sun.com: bonwick set sender to Jeff.Bonwick@sun.com using -f User-Agent: Mutt/1.5.14 (2007-02-12) Status: RO Content-Length: 865 > >Does setting these properties to a more-restrictive setting drop > >now-excluded cache entries immediately? (in other words, would > >setting > >primarycache=none and then setting it back to all serve to empty the > >cache of any entries from the affected datasets, and free up the > >corresponding system memory or cache disk space?) > > Clever, but no it does not - the properties only apply to new I/O. Nor should it. The ARC and L2ARC are both physical caches, whereas ZFS datasets are logical. There can be many-to-one mappings between datasets and cached blocks, whether because of snapshots (today) or deduplication (coming). Trying to flush the ARC or L2ARC in response to turning off caching in a particular dataset would be like trying to flush physically tagged e-cache lines in respose to a process with shared memory segments exiting. Jeff From Darren.Moffat@sun.com Mon Jun 23 02:34:22 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5N9YLUl027373 for ; Mon, 23 Jun 2008 02:34:21 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m5N9Y9L1012924; Mon, 23 Jun 2008 10:34:19 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K2W00811T953K00@nwk-avmta-1.sfbay.Sun.COM>; Mon, 23 Jun 2008 02:34:17 -0700 (PDT) Received: from gmp-eb-inf-1.sun.com ([192.18.6.21]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K2W005H5T93S290@nwk-avmta-1.sfbay.Sun.COM>; Mon, 23 Jun 2008 02:34:16 -0700 (PDT) Received: from fe-emea-09.sun.com (gmp-eb-lb-2-fe3.eu.sun.com [192.18.6.12]) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m5N9YFFM015278; Mon, 23 Jun 2008 09:34:15 +0000 (GMT) Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K2W00201T4M7200@fe-emea-09.sun.com> (original mail from Darren.Moffat@Sun.COM); Mon, 23 Jun 2008 10:34:15 +0100 (BST) Received: from [129.156.173.21] by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K2W00DQ3T8ZA2D0@fe-emea-09.sun.com>; Mon, 23 Jun 2008 10:34:12 +0100 (BST) Date: Mon, 23 Jun 2008 10:34:11 +0100 From: Darren J Moffat Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> Sender: Darren.Moffat@sun.com To: Tim Haley Cc: PSARC-ext@sun.com, zfs-team@sun.com Message-id: <485F6E13.9090005@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> User-Agent: Thunderbird 2.0.0.14 (X11/20080507) Status: RO Content-Length: 461 Is data cached or not in the following case: $ zfs create -o primarycache=all tank/foo $ zfs snapshot tank/foo@s1 $ zfs clone tank/foo@s1 tank/bar $ zfs set primarycache=none tank/bar # zpool export tank && zpool import tank Now read from tank/bar the data which is shared with tank/foo, does that get cached ? I'm assuming not because of the primarycache option for tank/bar. But if we read it via tank/foo it would be cached, right ? -- Darren J Moffat From eric.kustarz@sun.com Mon Jun 23 09:49:16 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5NGnGBq006872 for ; Mon, 23 Jun 2008 09:49:16 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m5NGnBpZ007791; Mon, 23 Jun 2008 09:49:15 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K2X00611DE2SG00@nwk-avmta-1.sfbay.Sun.COM>; Mon, 23 Jun 2008 09:49:14 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.63]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K2X004FVDE1SK10@nwk-avmta-1.sfbay.Sun.COM>; Mon, 23 Jun 2008 09:49:13 -0700 (PDT) Received: from punchin-client-10-7-250-182.SFBay.Sun.COM (punchin-client-10-7-250-182.SFBay.Sun.COM [10.7.250.182]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m5NGnDV3285930; Mon, 23 Jun 2008 09:49:13 -0700 (PDT) Date: Mon, 23 Jun 2008 09:49:12 -0700 From: eric kustarz Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <485F6E13.9090005@Sun.COM> To: Darren J Moffat Cc: Tim Haley , PSARC-ext@sun.com, zfs-team@sun.com Message-id: <0C252D0A-0837-4743-8A36-61DE7F018DCA@sun.com> MIME-version: 1.0 X-Mailer: Apple Mail (2.924) Content-type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <485F6E13.9090005@Sun.COM> Status: RO Content-Length: 544 On Jun 23, 2008, at 2:34 AM, Darren J Moffat wrote: > Is data cached or not in the following case: > > $ zfs create -o primarycache=all tank/foo > $ zfs snapshot tank/foo@s1 > $ zfs clone tank/foo@s1 tank/bar > $ zfs set primarycache=none tank/bar > # zpool export tank && zpool import tank > > Now read from tank/bar the data which is shared with tank/foo, does > that get cached ? I'm assuming not because of the primarycache > option for tank/bar. But if we read it via tank/foo it would be > cached, right ? That's correct. eric From ahl@basura.sf.fishpong.com Mon Jun 23 10:10:12 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5NHAClw007410 for ; Mon, 23 Jun 2008 10:10:12 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m5NHAA8J034457; Mon, 23 Jun 2008 11:10:11 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K2X00F0BECYU000@brm-avmta-1.central.sun.com>; Mon, 23 Jun 2008 11:10:10 -0600 (MDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K2X00ESUECX9S20@brm-avmta-1.central.sun.com>; Mon, 23 Jun 2008 11:10:10 -0600 (MDT) Received: from relay44i.sun.com ([192.5.209.118]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m5NHA9lb004660; Mon, 23 Jun 2008 17:10:09 +0000 (GMT) Received: from mms48es.mms.us.syntegra.com ([160.41.221.230] [160.41.221.230]) by relay44i.sun.com with ESMTP id BT-MMP-877196; Mon, 23 Jun 2008 17:10:09 +0000 (Z) Received: from relay41i.sun.com (relay41i.sun.com [192.5.209.70]) by mms48es.mms.us.syntegra.com with ESMTP id BT-MMP-46822869; Mon, 23 Jun 2008 17:10:09 +0000 (Z) Received: from basura.sf.fishpong.com ([198.144.208.55] [198.144.208.55]) by relay4i.sun.com with ESMTP id BT-MMP-2101428; Mon, 23 Jun 2008 17:10:08 +0000 (Z) Received: from basura.sf.fishpong.com (localhost [127.0.0.1]) by basura.sf.fishpong.com (8.14.2+Sun/8.14.2) with ESMTP id m5NHD43m357809; Mon, 23 Jun 2008 17:13:04 +0000 (GMT) Received: (from ahl@localhost) by basura.sf.fishpong.com (8.14.2+Sun/8.14.2/Submit) id m5NHD4NS357808; Mon, 23 Jun 2008 10:13:04 -0700 (PDT) Date: Mon, 23 Jun 2008 10:13:03 -0700 From: Adam Leventhal Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <20080621070641.GA6495@eng.sun.com> Sender: ahl@basura.sf.fishpong.com To: Jeff Bonwick Cc: eric kustarz , Bill Sommerfeld , Tim Haley , PSARC-ext@sun.com, zfs-team@sun.com Message-id: <20080623171303.GC179603@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=0.0/5.0, scanned in 0.096sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <1213993245.10441.13.camel@thunk> <20080621070641.GA6495@eng.sun.com> User-Agent: Mutt/1.5.17 (2007-11-01) Status: RO Content-Length: 830 On Sat, Jun 21, 2008 at 12:06:42AM -0700, Jeff Bonwick wrote: > > Clever, but no it does not - the properties only apply to new I/O. > > Nor should it. The ARC and L2ARC are both physical caches, whereas > ZFS datasets are logical. There can be many-to-one mappings between > datasets and cached blocks, whether because of snapshots (today) or > deduplication (coming). Trying to flush the ARC or L2ARC in response > to turning off caching in a particular dataset would be like trying > to flush physically tagged e-cache lines in respose to a process > with shared memory segments exiting. All true. However, we could (and should?) not check the caches if caching is not enabled for the dataset. This would have a similar effect as flushing. Adam -- Adam Leventhal, Fishworks http://blogs.sun.com/ahl From Matthew.Ahrens@sun.com Mon Jun 23 10:14:57 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5NHEvEa007452 for ; Mon, 23 Jun 2008 10:14:57 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m5NHEuxP016438; Mon, 23 Jun 2008 10:14:57 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K2X00G0FEKV4800@brm-avmta-1.central.sun.com>; Mon, 23 Jun 2008 11:14:55 -0600 (MDT) Received: from zion.sfbay.sun.com ([129.146.17.75]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K2X00E9AEKT9S30@brm-avmta-1.central.sun.com>; Mon, 23 Jun 2008 11:14:54 -0600 (MDT) Received: from stationary.local (punchin-ahrens.SFBay.Sun.COM [10.7.251.178]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m5NHErgL025949; Mon, 23 Jun 2008 17:14:53 +0000 (GMT) Date: Mon, 23 Jun 2008 10:14:41 -0700 From: Matthew Ahrens Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> To: Tim Haley Cc: PSARC-ext@sun.com, zfs-team@sun.com Message-id: <485FDA01.5080107@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) Status: RO Content-Length: 333 Tim Haley wrote: > A. SUMMARY > > This case adds two new dataset-level properties, 'primarycache' and > 'secondarycache', to the existing zfs property infrastructure. These > properties control what is cached in the primary cache (ARC) and the > secondary cache (l2ARC). I presume that these properties will be inherited? --matt From eric.kustarz@sun.com Mon Jun 23 10:21:15 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5NHLE6q007622 for ; Mon, 23 Jun 2008 10:21:15 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m5NHKqbP001144; Tue, 24 Jun 2008 01:21:12 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K2X00H2VEV9Z600@nwk-avmta-2.sfbay.sun.com>; Mon, 23 Jun 2008 10:21:09 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.63]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K2X00DW1EV8K0B0@nwk-avmta-2.sfbay.sun.com>; Mon, 23 Jun 2008 10:21:08 -0700 (PDT) Received: from punchin-client-10-7-250-182.SFBay.Sun.COM (punchin-client-10-7-250-182.SFBay.Sun.COM [10.7.250.182]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m5NHL8dA292883; Mon, 23 Jun 2008 10:21:08 -0700 (PDT) Date: Mon, 23 Jun 2008 10:21:06 -0700 From: eric kustarz Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <485FDA01.5080107@sun.com> To: Matthew Ahrens Cc: Tim Haley , PSARC-ext@sun.com, zfs-team@sun.com Message-id: MIME-version: 1.0 X-Mailer: Apple Mail (2.924) Content-type: text/plain; charset=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <485FDA01.5080107@sun.com> Status: RO Content-Length: 402 On Jun 23, 2008, at 10:14 AM, Matthew Ahrens wrote: > Tim Haley wrote: >> A. SUMMARY >> This case adds two new dataset-level properties, 'primarycache' and >> 'secondarycache', to the existing zfs property infrastructure. These >> properties control what is cached in the primary cache (ARC) and the >> secondary cache (l2ARC). > > I presume that these properties will be inherited? Correct. eric From Darren.Reed@sun.com Mon Jun 23 13:07:33 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5NK7X8J011981 for ; Mon, 23 Jun 2008 13:07:33 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m5NK7Ult009564; Mon, 23 Jun 2008 13:07:33 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K2X00501MKKVF00@brm-avmta-1.central.sun.com>; Mon, 23 Jun 2008 14:07:32 -0600 (MDT) Received: from sineb-mail-1.sun.com ([192.18.19.6]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K2X00MUBMKIXPC0@brm-avmta-1.central.sun.com>; Mon, 23 Jun 2008 14:07:31 -0600 (MDT) Received: from fe-apac-06.sun.com (fe-apac-06.sun.com [192.18.19.177] (may be forged)) by sineb-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m5NK7tap000424; Mon, 23 Jun 2008 20:07:55 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) id <0K2X00701MGOKT00@mail-apac.sun.com> (original mail from Darren.Reed@Sun.COM); Tue, 24 Jun 2008 04:05:35 +0800 (SGT) Received: from [192.168.2.120] ([213.220.197.34]) by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) with ESMTPSA id <0K2X00FWEMGUR11I@mail-apac.sun.com>; Tue, 24 Jun 2008 04:05:35 +0800 (SGT) Date: Mon, 23 Jun 2008 22:07:11 +0200 From: Darren Reed Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> Sender: Darren.Reed@sun.com To: Tim Haley Cc: psarc-ext@sun.com, zfs-team@sun.com Message-id: <4860026F.3040408@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) Status: RO Content-Length: 634 Tim Haley wrote: > .... > primarycache=all | none | metadata > > Controls what is cached in the primary cache (ARC). If set to > "all", then both user data and metadata is cached. If set to > "none", then neither user data nor metadata is cached. If set to > "metadata", then only metadata is cached. The default behavior is > "all". > The description above kind of implies that user data is somehow separate to metadata but it isn't possible to say cache only user data (with the text given.) Is this just an oversight or is this really saying you cannot cache only the user data? Darren From eric.kustarz@sun.com Mon Jun 23 13:12:27 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5NKCRAx012471 for ; Mon, 23 Jun 2008 13:12:27 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m5NKCFm3002199; Mon, 23 Jun 2008 21:12:24 +0100 (BST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K2X00601MSN7E00@brm-avmta-1.central.sun.com>; Mon, 23 Jun 2008 14:12:23 -0600 (MDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.63]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K2X00M0XMSMXPE0@brm-avmta-1.central.sun.com>; Mon, 23 Jun 2008 14:12:22 -0600 (MDT) Received: from punchin-client-10-7-250-182.SFBay.Sun.COM (punchin-client-10-7-250-182.SFBay.Sun.COM [10.7.250.182]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m5NKCMNv354380; Mon, 23 Jun 2008 13:12:22 -0700 (PDT) Date: Mon, 23 Jun 2008 13:12:21 -0700 From: eric kustarz Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <4860026F.3040408@Sun.COM> To: Darren Reed Cc: Tim Haley , psarc-ext@sun.com, zfs-team@sun.com Message-id: MIME-version: 1.0 X-Mailer: Apple Mail (2.924) Content-type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <4860026F.3040408@Sun.COM> Status: RO Content-Length: 955 On Jun 23, 2008, at 1:07 PM, Darren Reed wrote: > Tim Haley wrote: >> .... >> primarycache=all | none | metadata >> >> Controls what is cached in the primary cache (ARC). If set to >> "all", then both user data and metadata is cached. If set to >> "none", then neither user data nor metadata is cached. If set >> to >> "metadata", then only metadata is cached. The default >> behavior is >> "all". >> > > The description above kind of implies that user data is somehow > separate to metadata > but it isn't possible to say cache only user data (with the text > given.) Is this just an > oversight or is this really saying you cannot cache only the user > data? We couldn't come up with any realistic workload that would want to cache user data but not metadata, so we're not allowing it. We can always add the option later, but if someone has a realistic use case for it, i'd be happy to add it now. eric From Darren.Reed@sun.com Mon Jun 23 13:21:01 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5NKL0la012949 for ; Mon, 23 Jun 2008 13:21:00 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m5NKKtbU004456; Tue, 24 Jun 2008 04:20:58 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K2X00207N6W8H00@nwk-avmta-2.sfbay.sun.com>; Mon, 23 Jun 2008 13:20:56 -0700 (PDT) Received: from sineb-mail-2.sun.com ([192.18.19.7]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K2X00LO2N6VWJ40@nwk-avmta-2.sfbay.sun.com>; Mon, 23 Jun 2008 13:20:56 -0700 (PDT) Received: from fe-apac-06.sun.com (fe-apac-06.sun.com [192.18.19.177] (may be forged)) by sineb-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m5NKLref023495; Mon, 23 Jun 2008 20:21:53 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) id <0K2X00C01N3F3R00@mail-apac.sun.com> (original mail from Darren.Reed@Sun.COM); Tue, 24 Jun 2008 04:18:59 +0800 (SGT) Received: from [192.168.2.120] ([213.220.197.34]) by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) with ESMTPSA id <0K2X00FV0N3IR13I@mail-apac.sun.com>; Tue, 24 Jun 2008 04:18:58 +0800 (SGT) Date: Mon, 23 Jun 2008 22:20:48 +0200 From: Darren Reed Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: Sender: Darren.Reed@sun.com To: eric kustarz Cc: Tim Haley , psarc-ext@sun.com, zfs-team@sun.com Message-id: <486005A0.8080104@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <4860026F.3040408@Sun.COM> User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) Status: RO Content-Length: 1269 eric kustarz wrote: > > On Jun 23, 2008, at 1:07 PM, Darren Reed wrote: > >> Tim Haley wrote: >>> .... >>> primarycache=all | none | metadata >>> >>> Controls what is cached in the primary cache (ARC). If set to >>> "all", then both user data and metadata is cached. If set to >>> "none", then neither user data nor metadata is cached. If set to >>> "metadata", then only metadata is cached. The default behavior is >>> "all". >>> >> >> The description above kind of implies that user data is somehow >> separate to metadata >> but it isn't possible to say cache only user data (with the text >> given.) Is this just an >> oversight or is this really saying you cannot cache only the user data? > > We couldn't come up with any realistic workload that would want to > cache user data but not metadata, so we're not allowing it. > > We can always add the option later, but if someone has a realistic use > case for it, i'd be happy to add it now. It's not so much the "why", but maybe I'd like to say the primarycache gets metadata and the secondary cache gets user data (or vice versa.) If that make sense? Or would that require linkage between metadata and user data (across cache boundaries) in order to maintain sanity? Darren From Eric.Kustarz@sun.com Mon Jun 23 13:38:15 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5NKcFqK013505 for ; Mon, 23 Jun 2008 13:38:15 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m5NKcCue003968; Mon, 23 Jun 2008 13:38:13 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K2X00709NZOWC00@nwk-avmta-1.sfbay.Sun.COM>; Mon, 23 Jun 2008 13:38:12 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.63]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K2X006VUNZOXN00@nwk-avmta-1.sfbay.Sun.COM>; Mon, 23 Jun 2008 13:38:12 -0700 (PDT) Received: from punchin-client-10-7-250-182.SFBay.Sun.COM (punchin-client-10-7-250-182.SFBay.Sun.COM [10.7.250.182]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m5NKcBWo374652; Mon, 23 Jun 2008 13:38:11 -0700 (PDT) Date: Mon, 23 Jun 2008 13:38:11 -0700 From: eric kustarz Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <486005A0.8080104@Sun.COM> To: Darren Reed Cc: Tim Haley , psarc-ext@sun.com, zfs-team@sun.com Message-id: <11C78B9F-F57B-49DD-8997-AC47C2F676DD@sun.com> MIME-version: 1.0 X-Mailer: Apple Mail (2.924) Content-type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <4860026F.3040408@Sun.COM> <486005A0.8080104@Sun.COM> Status: RO Content-Length: 1615 On Jun 23, 2008, at 1:20 PM, Darren Reed wrote: > eric kustarz wrote: >> >> On Jun 23, 2008, at 1:07 PM, Darren Reed wrote: >> >>> Tim Haley wrote: >>>> .... >>>> primarycache=all | none | metadata >>>> >>>> Controls what is cached in the primary cache (ARC). If set to >>>> "all", then both user data and metadata is cached. If set to >>>> "none", then neither user data nor metadata is cached. If >>>> set to >>>> "metadata", then only metadata is cached. The default >>>> behavior is >>>> "all". >>>> >>> >>> The description above kind of implies that user data is somehow >>> separate to metadata >>> but it isn't possible to say cache only user data (with the text >>> given.) Is this just an >>> oversight or is this really saying you cannot cache only the user >>> data? >> >> We couldn't come up with any realistic workload that would want to >> cache user data but not metadata, so we're not allowing it. >> >> We can always add the option later, but if someone has a realistic >> use case for it, i'd be happy to add it now. > > It's not so much the "why", but maybe I'd like to say the primarycache > gets metadata and the secondary cache gets user data (or vice versa.) > If that make sense? Or would that require linkage between metadata > and user data (across cache boundaries) in order to maintain sanity? It is the "why". If there's no reason to do it, then we shouldn't allow it (adds more complexity, more confusion, more ways for a customer to shoot themselves in the foot). However, if there is a legitimate use case, let's discuss that. eric From Darren.Reed@sun.com Mon Jun 23 19:34:11 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5O2YB4d025296 for ; Mon, 23 Jun 2008 19:34:11 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m5O2YA6Z053652; Mon, 23 Jun 2008 20:34:11 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K2Y00J074GYT600@nwk-avmta-2.sfbay.sun.com>; Mon, 23 Jun 2008 19:34:10 -0700 (PDT) Received: from sineb-mail-1.sun.com ([192.18.19.6]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K2Y00BJP4GXIQ60@nwk-avmta-2.sfbay.sun.com>; Mon, 23 Jun 2008 19:34:09 -0700 (PDT) Received: from fe-apac-05.sun.com (fe-apac-05.sun.com [192.18.19.176] (may be forged)) by sineb-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m5O2YY5x014889; Tue, 24 Jun 2008 02:34:34 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) id <0K2Y00G014DTJB00@mail-apac.sun.com> (original mail from Darren.Reed@Sun.COM); Tue, 24 Jun 2008 10:33:43 +0800 (SGT) Received: from [200.200.0.213] ([85.207.19.34]) by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) with ESMTPSA id <0K2Y003B54G3JP6S@mail-apac.sun.com>; Tue, 24 Jun 2008 10:33:43 +0800 (SGT) Date: Tue, 24 Jun 2008 04:34:02 +0200 From: Darren Reed Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <11C78B9F-F57B-49DD-8997-AC47C2F676DD@sun.com> Sender: Darren.Reed@sun.com To: eric kustarz Cc: Tim Haley , psarc-ext@sun.com, zfs-team@sun.com Message-id: <48605D1A.5000900@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <4860026F.3040408@Sun.COM> <486005A0.8080104@Sun.COM> <11C78B9F-F57B-49DD-8997-AC47C2F676DD@sun.com> User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) Status: RO Content-Length: 1859 eric kustarz wrote: > > On Jun 23, 2008, at 1:20 PM, Darren Reed wrote: > >> eric kustarz wrote: >>> >>> On Jun 23, 2008, at 1:07 PM, Darren Reed wrote: >>> >>>> Tim Haley wrote: >>>>> .... >>>>> primarycache=all | none | metadata >>>>> >>>>> Controls what is cached in the primary cache (ARC). If set to >>>>> "all", then both user data and metadata is cached. If set to >>>>> "none", then neither user data nor metadata is cached. If set to >>>>> "metadata", then only metadata is cached. The default >>>>> behavior is >>>>> "all". >>>>> >>>> >>>> The description above kind of implies that user data is somehow >>>> separate to metadata >>>> but it isn't possible to say cache only user data (with the text >>>> given.) Is this just an >>>> oversight or is this really saying you cannot cache only the user >>>> data? >>> >>> We couldn't come up with any realistic workload that would want to >>> cache user data but not metadata, so we're not allowing it. >>> >>> We can always add the option later, but if someone has a realistic >>> use case for it, i'd be happy to add it now. >> >> It's not so much the "why", but maybe I'd like to say the primarycache >> gets metadata and the secondary cache gets user data (or vice versa.) >> If that make sense? Or would that require linkage between metadata >> and user data (across cache boundaries) in order to maintain sanity? > > It is the "why". If there's no reason to do it, then we shouldn't > allow it (adds more complexity, more confusion, more ways for a > customer to shoot themselves in the foot). > > However, if there is a legitimate use case, let's discuss that. I think we can move that discussion to one of the ZFS discussion forums on OpenSolaris.org. The important point I was curious about was if there was an architectural ommission or mistake. Darren From Timothy.Haley@sun.com Wed Jun 25 11:47:10 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5PIlAAb010904 for ; Wed, 25 Jun 2008 11:47:10 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m5PIl9Gf034905; Wed, 25 Jun 2008 12:47:09 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K310070786KO400@nwk-avmta-2.sfbay.sun.com>; Wed, 25 Jun 2008 11:47:08 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K310017V86JA1D0@nwk-avmta-2.sfbay.sun.com>; Wed, 25 Jun 2008 11:47:07 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m5PIl7k0012563; Wed, 25 Jun 2008 18:47:07 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K31001017BAR800@mail-amer.sun.com> (original mail from Timothy.Haley@Sun.COM); Wed, 25 Jun 2008 12:47:07 -0600 (MDT) Received: from [172.20.25.27] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K3100LZV85F7O90@mail-amer.sun.com>; Wed, 25 Jun 2008 12:46:31 -0600 (MDT) Date: Wed, 25 Jun 2008 12:46:29 -0600 From: Tim Haley Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> Sender: Timothy.Haley@sun.com To: PSARC-ext@sun.com Cc: zfs-team@sun.com Message-id: <48629285.7040305@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> User-Agent: Thunderbird 2.0.0.14 (X11/20080602) Status: RO Content-Length: 55 This case was approved in today's PSARC meeting. -tim From Darren.Reed@sun.com Wed Jun 25 11:49:50 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5PInoHc010991 for ; Wed, 25 Jun 2008 11:49:50 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m5PInnvD035806 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 25 Jun 2008 12:49:49 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K31009098B08D00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 25 Jun 2008 11:49:48 -0700 (PDT) Received: from sineb-mail-2.sun.com ([192.18.19.7]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K3100G1F8AZIXC0@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 25 Jun 2008 11:49:48 -0700 (PDT) Received: from fe-apac-05.sun.com (fe-apac-05.sun.com [192.18.19.176] (may be forged)) by sineb-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m5PIojr6029656 for ; Wed, 25 Jun 2008 18:50:45 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) id <0K3100F01856HS00@mail-apac.sun.com> (original mail from Darren.Reed@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 26 Jun 2008 02:49:21 +0800 (SGT) Received: from [200.200.0.213] ([85.207.19.34]) by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) with ESMTPSA id <0K3100MF98A6LR5Q@mail-apac.sun.com>; Thu, 26 Jun 2008 02:49:21 +0800 (SGT) Date: Wed, 25 Jun 2008 20:49:42 +0200 From: Darren Reed Subject: Re: [zfs-discuss] zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <4860BE21.9070103@Sun.COM> Sender: Darren.Reed@sun.com To: eric kustarz Cc: PSARC-ext Message-id: <48629346.5040307@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <4860026F.3040408@Sun.COM> <486005A0.8080104@Sun.COM> <11C78B9F-F57B-49DD-8997-AC47C2F676DD@sun.com> <48606712.3040503@Sun.COM> <4860BE21.9070103@Sun.COM> User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) Status: RO Content-Length: 1444 This would seem to be a significant use case for the model of having non-overlapping data types in each of the two caches. Since no reply was received on zfs-discuss, I'm redirecting it to psarc to indicate that this question isn't closed. Darren J Moffat wrote: > Darren Reed wrote: > >> So I spent some time thinking about different directions you could build >> on this in the future, for example: >> 1) controlling the size of the ARC/L2ARC by controlling the cache size >> 2) specifying different backing storage for primary/secondary cache >> 3) having more than two levels of cache >> ...none of which is precluded by current efforts. >> >> With (2), if the backing storage for each cache is different and it is >> slower >> to access the secondary cache than the primary, then you may not want >> metadata to be stored in the secondary cache for performance reasons. >> >> As an example, you might be using NVRAM (be it flash or otherwise) >> for the primary cache and ordinary RAM for the secondary. In this case >> you probably don't want any metadata to be stored in the secondary >> cache (power failure issues) but the same may not hold for user data. >> But I'm probably wrong about that. >> > > I doubt you would be, the primarycache is system memory not a cache > device. The secondarycache is the L2ARC devices specified with the > "cache" vdev type to zpool so your examle would be the otherway around. > > From Eric.Kustarz@sun.com Wed Jun 25 11:56:01 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5PIu0s6011384 for ; Wed, 25 Jun 2008 11:56:01 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m5PItnSS011053; Wed, 25 Jun 2008 19:55:58 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K31009038L7W400@nwk-avmta-1.sfbay.Sun.COM>; Wed, 25 Jun 2008 11:55:55 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.63]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K3100G0Y8L7J3E0@nwk-avmta-1.sfbay.Sun.COM>; Wed, 25 Jun 2008 11:55:55 -0700 (PDT) Received: from punchin-client-10-7-251-94.SFBay.Sun.COM (punchin-client-10-7-251-94.SFBay.Sun.COM [10.7.251.94]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m5PItstA458348; Wed, 25 Jun 2008 11:55:54 -0700 (PDT) Date: Wed, 25 Jun 2008 11:55:54 -0700 From: eric kustarz Subject: Re: [zfs-discuss] zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <48629346.5040307@Sun.COM> To: Darren Reed Cc: PSARC-ext Message-id: <67A74B2A-5735-46AC-BBF3-5E239E2A10C4@sun.com> MIME-version: 1.0 X-Mailer: Apple Mail (2.924) Content-type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <4860026F.3040408@Sun.COM> <486005A0.8080104@Sun.COM> <11C78B9F-F57B-49DD-8997-AC47C2F676DD@sun.com> <48606712.3040503@Sun.COM> <4860BE21.9070103@Sun.COM> <48629346.5040307@Sun.COM> Status: RO Content-Length: 1651 On Jun 25, 2008, at 11:49 AM, Darren Reed wrote: > This would seem to be a significant use case for the model of having > non-overlapping data types in each of the two caches. Since no reply > was received on zfs-discuss, I'm redirecting it to psarc to indicate > that > this question isn't closed. I see some comments, but no direct question. So what is the question? eric > > > Darren J Moffat wrote: >> Darren Reed wrote: >> >>> So I spent some time thinking about different directions you could >>> build >>> on this in the future, for example: >>> 1) controlling the size of the ARC/L2ARC by controlling the cache >>> size >>> 2) specifying different backing storage for primary/secondary cache >>> 3) having more than two levels of cache >>> ...none of which is precluded by current efforts. >>> >>> With (2), if the backing storage for each cache is different and >>> it is slower >>> to access the secondary cache than the primary, then you may not >>> want >>> metadata to be stored in the secondary cache for performance >>> reasons. >>> >>> As an example, you might be using NVRAM (be it flash or otherwise) >>> for the primary cache and ordinary RAM for the secondary. In this >>> case >>> you probably don't want any metadata to be stored in the secondary >>> cache (power failure issues) but the same may not hold for user >>> data. >>> But I'm probably wrong about that. >>> >> >> I doubt you would be, the primarycache is system memory not a cache >> device. The secondarycache is the L2ARC devices specified with the >> "cache" vdev type to zpool so your examle would be the otherway >> around. >> >> From Darren.Reed@Sun.COM Wed Jun 25 12:02:18 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5PJ2H1Q011641 for ; Wed, 25 Jun 2008 12:02:17 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m5PJ2GEN039769 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 25 Jun 2008 13:02:17 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K3100A0N8VTHQ00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 25 Jun 2008 12:02:17 -0700 (PDT) Received: from sineb-mail-2.sun.com ([192.18.19.7]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K3100GGX8VSJ2B0@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 25 Jun 2008 12:02:17 -0700 (PDT) Received: from fe-apac-06.sun.com (fe-apac-06.sun.com [192.18.19.177] (may be forged)) by sineb-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m5PJ3DPT029742 for ; Wed, 25 Jun 2008 19:03:13 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) id <0K31006018NWPM00@mail-apac.sun.com> (original mail from Darren.Reed@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 26 Jun 2008 03:00:18 +0800 (SGT) Received: from [200.200.0.213] ([85.207.19.34]) by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) with ESMTPSA id <0K3100FCQ8SFR1DU@mail-apac.sun.com>; Thu, 26 Jun 2008 03:00:18 +0800 (SGT) Date: Wed, 25 Jun 2008 21:02:11 +0200 From: Darren Reed Subject: Re: [zfs-discuss] zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <67A74B2A-5735-46AC-BBF3-5E239E2A10C4@sun.com> Sender: Darren.Reed@Sun.COM To: eric kustarz Cc: PSARC-ext Message-id: <48629633.2030504@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <4860026F.3040408@Sun.COM> <486005A0.8080104@Sun.COM> <11C78B9F-F57B-49DD-8997-AC47C2F676DD@sun.com> <48606712.3040503@Sun.COM> <4860BE21.9070103@Sun.COM> <48629346.5040307@Sun.COM> <67A74B2A-5735-46AC-BBF3-5E239E2A10C4@sun.com> User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) Status: RO Content-Length: 1969 eric kustarz wrote: > > On Jun 25, 2008, at 11:49 AM, Darren Reed wrote: > >> This would seem to be a significant use case for the model of having >> non-overlapping data types in each of the two caches. Since no reply >> was received on zfs-discuss, I'm redirecting it to psarc to indicate >> that >> this question isn't closed. > > I see some comments, but no direct question. So what is the question? If the primary and secondary cache are different media, especially in the case of one being non-volatile, shouldn't it be possible to allow the user to specify that they want to use the non-volatile cache for meta data without requiring them to forgo caching user data in a volatile cache? Darren >> Darren J Moffat wrote: >>> Darren Reed wrote: >>> >>>> So I spent some time thinking about different directions you could >>>> build >>>> on this in the future, for example: >>>> 1) controlling the size of the ARC/L2ARC by controlling the cache size >>>> 2) specifying different backing storage for primary/secondary cache >>>> 3) having more than two levels of cache >>>> ...none of which is precluded by current efforts. >>>> >>>> With (2), if the backing storage for each cache is different and it >>>> is slower >>>> to access the secondary cache than the primary, then you may not want >>>> metadata to be stored in the secondary cache for performance reasons. >>>> >>>> As an example, you might be using NVRAM (be it flash or otherwise) >>>> for the primary cache and ordinary RAM for the secondary. In this >>>> case >>>> you probably don't want any metadata to be stored in the secondary >>>> cache (power failure issues) but the same may not hold for user data. >>>> But I'm probably wrong about that. >>>> >>> >>> I doubt you would be, the primarycache is system memory not a cache >>> device. The secondarycache is the L2ARC devices specified with the >>> "cache" vdev type to zpool so your examle would be the otherway around. >>> >>> > From Eric.Kustarz@sun.com Wed Jun 25 12:28:13 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5PJSDJ6012009 for ; Wed, 25 Jun 2008 12:28:13 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m5PJS8YK008816; Wed, 25 Jun 2008 12:28:11 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K3100G05A2YRR00@brm-avmta-1.central.sun.com>; Wed, 25 Jun 2008 13:28:10 -0600 (MDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K31004LWA2X16A0@brm-avmta-1.central.sun.com>; Wed, 25 Jun 2008 13:28:09 -0600 (MDT) Received: from punchin-client-10-7-251-94.SFBay.Sun.COM (punchin-client-10-7-251-94.SFBay.Sun.COM [10.7.251.94]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m5PJS9q6464672; Wed, 25 Jun 2008 12:28:09 -0700 (PDT) Date: Wed, 25 Jun 2008 12:28:09 -0700 From: eric kustarz Subject: Re: [zfs-discuss] zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <48629633.2030504@Sun.COM> To: Darren Reed Cc: PSARC-ext Message-id: <881409D6-4F71-4951-ACC8-DE80F96B6D46@Sun.COM> MIME-version: 1.0 X-Mailer: Apple Mail (2.924) Content-type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <4860026F.3040408@Sun.COM> <486005A0.8080104@Sun.COM> <11C78B9F-F57B-49DD-8997-AC47C2F676DD@sun.com> <48606712.3040503@Sun.COM> <4860BE21.9070103@Sun.COM> <48629346.5040307@Sun.COM> <67A74B2A-5735-46AC-BBF3-5E239E2A10C4@sun.com> <48629633.2030504@Sun.COM> Status: RO Content-Length: 2324 On Jun 25, 2008, at 12:02 PM, Darren Reed wrote: > eric kustarz wrote: >> >> On Jun 25, 2008, at 11:49 AM, Darren Reed wrote: >> >>> This would seem to be a significant use case for the model of having >>> non-overlapping data types in each of the two caches. Since no >>> reply >>> was received on zfs-discuss, I'm redirecting it to psarc to >>> indicate that >>> this question isn't closed. >> >> I see some comments, but no direct question. So what is the >> question? > > If the primary and secondary cache are different media, especially > in the case > of one being non-volatile, shouldn't it be possible to allow the > user to specify > that they want to use the non-volatile cache for meta data without > requiring > them to forgo caching user data in a volatile cache? Sure: # zfs set primarycache=all tank/fs # zfs set secondarycache=metadata tank/fs ARC (server memory) is the primary cache, l2ARC (SSD) is the secondary cache. eric > > > Darren > >>> Darren J Moffat wrote: >>>> Darren Reed wrote: >>>> >>>>> So I spent some time thinking about different directions you >>>>> could build >>>>> on this in the future, for example: >>>>> 1) controlling the size of the ARC/L2ARC by controlling the >>>>> cache size >>>>> 2) specifying different backing storage for primary/secondary >>>>> cache >>>>> 3) having more than two levels of cache >>>>> ...none of which is precluded by current efforts. >>>>> >>>>> With (2), if the backing storage for each cache is different and >>>>> it is slower >>>>> to access the secondary cache than the primary, then you may not >>>>> want >>>>> metadata to be stored in the secondary cache for performance >>>>> reasons. >>>>> >>>>> As an example, you might be using NVRAM (be it flash or otherwise) >>>>> for the primary cache and ordinary RAM for the secondary. In >>>>> this case >>>>> you probably don't want any metadata to be stored in the secondary >>>>> cache (power failure issues) but the same may not hold for user >>>>> data. >>>>> But I'm probably wrong about that. >>>>> >>>> >>>> I doubt you would be, the primarycache is system memory not a >>>> cache device. The secondarycache is the L2ARC devices specified >>>> with the "cache" vdev type to zpool so your examle would be the >>>> otherway around. >>>> >>>> >> From Darren.Reed@sun.com Wed Jun 25 12:53:52 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5PJrpVk012282 for ; Wed, 25 Jun 2008 12:53:52 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m5PJrlwQ013202 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 26 Jun 2008 03:53:50 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K3100I0VB9QPL00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 25 Jun 2008 13:53:50 -0600 (MDT) Received: from sineb-mail-1.sun.com ([192.18.19.6]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K31004JLB9O0VA0@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 25 Jun 2008 13:53:49 -0600 (MDT) Received: from fe-apac-05.sun.com (fe-apac-05.sun.com [192.18.19.176] (may be forged)) by sineb-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m5PJsDP2010770 for ; Wed, 25 Jun 2008 19:54:13 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) id <0K3100601B6GUY00@mail-apac.sun.com> (original mail from Darren.Reed@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 26 Jun 2008 03:53:22 +0800 (SGT) Received: from [200.200.0.213] ([85.207.19.34]) by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) with ESMTPSA id <0K3100MD1B8VLRFQ@mail-apac.sun.com>; Thu, 26 Jun 2008 03:53:22 +0800 (SGT) Date: Wed, 25 Jun 2008 21:53:42 +0200 From: Darren Reed Subject: Re: [zfs-discuss] zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <881409D6-4F71-4951-ACC8-DE80F96B6D46@Sun.COM> Sender: Darren.Reed@sun.com To: eric kustarz Cc: PSARC-ext Message-id: <4862A246.4000301@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <4860026F.3040408@Sun.COM> <486005A0.8080104@Sun.COM> <11C78B9F-F57B-49DD-8997-AC47C2F676DD@sun.com> <48606712.3040503@Sun.COM> <4860BE21.9070103@Sun.COM> <48629346.5040307@Sun.COM> <67A74B2A-5735-46AC-BBF3-5E239E2A10C4@sun.com> <48629633.2030504@Sun.COM> <881409D6-4F71-4951-ACC8-DE80F96B6D46@Sun.COM> User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) Status: RO Content-Length: 2620 eric kustarz wrote: > > On Jun 25, 2008, at 12:02 PM, Darren Reed wrote: > >> eric kustarz wrote: >>> >>> On Jun 25, 2008, at 11:49 AM, Darren Reed wrote: >>> >>>> This would seem to be a significant use case for the model of having >>>> non-overlapping data types in each of the two caches. Since no reply >>>> was received on zfs-discuss, I'm redirecting it to psarc to >>>> indicate that >>>> this question isn't closed. >>> >>> I see some comments, but no direct question. So what is the question? >> >> If the primary and secondary cache are different media, especially in >> the case >> of one being non-volatile, shouldn't it be possible to allow the user >> to specify >> that they want to use the non-volatile cache for meta data without >> requiring >> them to forgo caching user data in a volatile cache? > > Sure: > # zfs set primarycache=all tank/fs > # zfs set secondarycache=metadata tank/fs > > ARC (server memory) is the primary cache, l2ARC (SSD) is the secondary > cache. > > eric Oh. are you saying that because metadata is directly s[ecofoed to be cached in one place, it won't also be cached in the other? The case didn't make that behaviour clear, if so. the desire would be primary=user data, secondary=meta data... Darren >>>> Darren J Moffat wrote: >>>>> Darren Reed wrote: >>>>> >>>>>> So I spent some time thinking about different directions you >>>>>> could build >>>>>> on this in the future, for example: >>>>>> 1) controlling the size of the ARC/L2ARC by controlling the cache >>>>>> size >>>>>> 2) specifying different backing storage for primary/secondary cache >>>>>> 3) having more than two levels of cache >>>>>> ...none of which is precluded by current efforts. >>>>>> >>>>>> With (2), if the backing storage for each cache is different and >>>>>> it is slower >>>>>> to access the secondary cache than the primary, then you may not >>>>>> want >>>>>> metadata to be stored in the secondary cache for performance >>>>>> reasons. >>>>>> >>>>>> As an example, you might be using NVRAM (be it flash or otherwise) >>>>>> for the primary cache and ordinary RAM for the secondary. In >>>>>> this case >>>>>> you probably don't want any metadata to be stored in the secondary >>>>>> cache (power failure issues) but the same may not hold for user >>>>>> data. >>>>>> But I'm probably wrong about that. >>>>>> >>>>> >>>>> I doubt you would be, the primarycache is system memory not a >>>>> cache device. The secondarycache is the L2ARC devices specified >>>>> with the "cache" vdev type to zpool so your examle would be the >>>>> otherway around. >>>>> >>>>> >>> > From eric.kustarz@sun.com Wed Jun 25 13:06:40 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5PK6dYJ012987 for ; Wed, 25 Jun 2008 13:06:40 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m5PK6bdQ017952; Thu, 26 Jun 2008 04:06:38 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K3100H1ZBV19600@nwk-avmta-1.sfbay.Sun.COM>; Wed, 25 Jun 2008 13:06:37 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.63]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K3100EU8BV15D30@nwk-avmta-1.sfbay.Sun.COM>; Wed, 25 Jun 2008 13:06:37 -0700 (PDT) Received: from punchin-client-10-7-251-94.SFBay.Sun.COM (punchin-client-10-7-251-94.SFBay.Sun.COM [10.7.251.94]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m5PK6ad5471701; Wed, 25 Jun 2008 13:06:37 -0700 (PDT) Date: Wed, 25 Jun 2008 13:06:37 -0700 From: eric kustarz Subject: Re: [zfs-discuss] zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <4862A246.4000301@Sun.COM> To: Darren Reed Cc: PSARC-ext Message-id: <203B789F-0F01-40A7-AC88-34193466D97E@sun.com> MIME-version: 1.0 X-Mailer: Apple Mail (2.924) Content-type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <4860026F.3040408@Sun.COM> <486005A0.8080104@Sun.COM> <11C78B9F-F57B-49DD-8997-AC47C2F676DD@sun.com> <48606712.3040503@Sun.COM> <4860BE21.9070103@Sun.COM> <48629346.5040307@Sun.COM> <67A74B2A-5735-46AC-BBF3-5E239E2A10C4@sun.com> <48629633.2030504@Sun.COM> <881409D6-4F71-4951-ACC8-DE80F96B6D46@Sun.COM> <4862A246.4000301@Sun.COM> Status: RO Content-Length: 1769 On Jun 25, 2008, at 12:53 PM, Darren Reed wrote: > eric kustarz wrote: >> >> On Jun 25, 2008, at 12:02 PM, Darren Reed wrote: >> >>> eric kustarz wrote: >>>> >>>> On Jun 25, 2008, at 11:49 AM, Darren Reed wrote: >>>> >>>>> This would seem to be a significant use case for the model of >>>>> having >>>>> non-overlapping data types in each of the two caches. Since no >>>>> reply >>>>> was received on zfs-discuss, I'm redirecting it to psarc to >>>>> indicate that >>>>> this question isn't closed. >>>> >>>> I see some comments, but no direct question. So what is the >>>> question? >>> >>> If the primary and secondary cache are different media, especially >>> in the case >>> of one being non-volatile, shouldn't it be possible to allow the >>> user to specify >>> that they want to use the non-volatile cache for meta data without >>> requiring >>> them to forgo caching user data in a volatile cache? >> >> Sure: >> # zfs set primarycache=all tank/fs >> # zfs set secondarycache=metadata tank/fs >> >> ARC (server memory) is the primary cache, l2ARC (SSD) is the >> secondary cache. >> >> eric > > Oh. are you saying that because metadata is directly s[ecofoed to be > cached in one place, it won't also be cached in the other? The case > didn't make that behaviour clear, if so. No - the ARC will cache both data and metadata. The l2ARC will only cache metadata. > > > the desire would be primary=user data, secondary=meta data... Desire for what workload? You would have to *always* go to the secondary cache (or disk) for metadata in order to get to the data cached in the primary cache. I don't see a sensible use case for this - this is why we are not allowing a data only option. But we've been over this already. eric From Darren.Reed@sun.com Wed Jun 25 13:11:23 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m5PKBMc3013430 for ; Wed, 25 Jun 2008 13:11:23 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m5PKBHNs013278 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 25 Jun 2008 21:11:22 +0100 (BST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K3100K0FC2T1T00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 25 Jun 2008 14:11:17 -0600 (MDT) Received: from sineb-mail-2.sun.com ([192.18.19.7]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K31004VGC2S15B0@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 25 Jun 2008 14:11:17 -0600 (MDT) Received: from fe-apac-06.sun.com (fe-apac-06.sun.com [192.18.19.177] (may be forged)) by sineb-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m5PKCD5q000378 for ; Wed, 25 Jun 2008 20:12:13 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) id <0K3100301BXYFR00@mail-apac.sun.com> (original mail from Darren.Reed@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 26 Jun 2008 04:09:18 +0800 (SGT) Received: from [200.200.0.213] ([85.207.19.34]) by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) with ESMTPSA id <0K3100FWIBZER1NU@mail-apac.sun.com>; Thu, 26 Jun 2008 04:09:18 +0800 (SGT) Date: Wed, 25 Jun 2008 22:11:09 +0200 From: Darren Reed Subject: Re: [zfs-discuss] zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <203B789F-0F01-40A7-AC88-34193466D97E@sun.com> Sender: Darren.Reed@sun.com To: eric kustarz Cc: PSARC-ext Message-id: <4862A65D.9010603@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <4860026F.3040408@Sun.COM> <486005A0.8080104@Sun.COM> <11C78B9F-F57B-49DD-8997-AC47C2F676DD@sun.com> <48606712.3040503@Sun.COM> <4860BE21.9070103@Sun.COM> <48629346.5040307@Sun.COM> <67A74B2A-5735-46AC-BBF3-5E239E2A10C4@sun.com> <48629633.2030504@Sun.COM> <881409D6-4F71-4951-ACC8-DE80F96B6D46@Sun.COM> <4862A246.4000301@Sun.COM> <203B789F-0F01-40A7-AC88-34193466D97E@sun.com> User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) Status: RO Content-Length: 1926 eric kustarz wrote: > > On Jun 25, 2008, at 12:53 PM, Darren Reed wrote: > >> eric kustarz wrote: >>> >>> On Jun 25, 2008, at 12:02 PM, Darren Reed wrote: >>> >>>> eric kustarz wrote: >>>>> >>>>> On Jun 25, 2008, at 11:49 AM, Darren Reed wrote: >>>>> >>>>>> This would seem to be a significant use case for the model of having >>>>>> non-overlapping data types in each of the two caches. Since no >>>>>> reply >>>>>> was received on zfs-discuss, I'm redirecting it to psarc to >>>>>> indicate that >>>>>> this question isn't closed. >>>>> >>>>> I see some comments, but no direct question. So what is the >>>>> question? >>>> >>>> If the primary and secondary cache are different media, especially >>>> in the case >>>> of one being non-volatile, shouldn't it be possible to allow the >>>> user to specify >>>> that they want to use the non-volatile cache for meta data without >>>> requiring >>>> them to forgo caching user data in a volatile cache? >>> >>> Sure: >>> # zfs set primarycache=all tank/fs >>> # zfs set secondarycache=metadata tank/fs >>> >>> ARC (server memory) is the primary cache, l2ARC (SSD) is the >>> secondary cache. >>> >>> eric >> >> Oh. are you saying that because metadata is directly s[ecofoed to be >> cached in one place, it won't also be cached in the other? The case >> didn't make that behaviour clear, if so. > > No - the ARC will cache both data and metadata. The l2ARC will only > cache metadata. > >> >> >> the desire would be primary=user data, secondary=meta data... > > Desire for what workload? You would have to *always* go to the > secondary cache (or disk) for metadata in order to get to the data > cached in the primary cache. I don't see a sensible use case for this > - this is why we are not allowing a data only option. But we've been > over this already. Ugh, brain fade... I was thinking of the caches as being in parallel rather than layered. Darren From Jeff.Bonwick@sun.com Tue Jul 1 00:47:31 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m617lVS8013582 for ; Tue, 1 Jul 2008 00:47:31 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m617lUwr020354; Tue, 1 Jul 2008 00:47:31 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K3B00E03HN5QF00@brm-avmta-1.central.sun.com>; Tue, 01 Jul 2008 01:47:29 -0600 (MDT) Received: from zion.sfbay.sun.com ([129.146.17.75]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K3B007S3HN43040@brm-avmta-1.central.sun.com>; Tue, 01 Jul 2008 01:47:28 -0600 (MDT) Received: from zion.sfbay.sun.com (localhost [127.0.0.1]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m617lRVu024966; Tue, 01 Jul 2008 07:47:27 +0000 (GMT) Received: (from bonwick@localhost) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2/Submit) id m617lReP024965; Tue, 01 Jul 2008 00:47:27 -0700 (PDT) Date: Tue, 01 Jul 2008 00:47:27 -0700 From: Jeff Bonwick Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: To: eric kustarz Cc: Matthew Ahrens , Tim Haley , PSARC-ext@sun.com, zfs-team@sun.com Message-id: <20080701074727.GC22698@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <485FDA01.5080107@sun.com> X-Authentication-warning: zion.sfbay.sun.com: bonwick set sender to Jeff.Bonwick@sun.com using -f User-Agent: Mutt/1.5.14 (2007-02-12) Status: RO Content-Length: 1264 > >>This case adds two new dataset-level properties, 'primarycache' and > >>'secondarycache', to the existing zfs property infrastructure. These > >>properties control what is cached in the primary cache (ARC) and the > >>secondary cache (l2ARC). > > > >I presume that these properties will be inherited? > > Correct. Hmmm... the ARC and L2ARC are both physical caches. With snapshots, clones, and (in the future) dedup, the same block can be shared by many datasets. What will determine whether a block is cached in the ARC? What will determine whether the L2ARC feed thread should write or ignore an ARC-cached block? In the case of the ARC, the answer seems simple: only datasets with caching enabled would allocate in the ARC; datasets with caching disabled could skip the ARC, or could consult it anyway (in case someone else has already done the I/O anyway). The L2ARC seems more complex because allocation is done by the feed thread, not in the context of any particular dataset doing I/O. The feed thread has no way of knowing on whose behalf blocks exist. So it would have to operate as it does today, caching everything that falls out of the ARC. It's not clear to me how the L2ARC policy can be anything other than a pool-wide property. Jeff From Darren.Moffat@sun.com Tue Jul 1 03:19:12 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m61AJC6g018372 for ; Tue, 1 Jul 2008 03:19:12 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m61AJC4H026463; Tue, 1 Jul 2008 03:19:12 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K3B00313ONZ5G00@brm-avmta-1.central.sun.com>; Tue, 01 Jul 2008 04:19:11 -0600 (MDT) Received: from gmp-eb-inf-2.sun.com ([192.18.6.24]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K3B007UZONJ35B0@brm-avmta-1.central.sun.com>; Tue, 01 Jul 2008 04:18:56 -0600 (MDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-2-fe1.eu.sun.com [192.18.6.10]) by gmp-eb-inf-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m61AItNS020810; Tue, 01 Jul 2008 10:18:55 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K3B00E01NYPXK00@fe-emea-10.sun.com> (original mail from Darren.Moffat@Sun.COM); Tue, 01 Jul 2008 11:18:55 +0100 (BST) Received: from [129.156.173.21] by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K3B0095MONH9370@fe-emea-10.sun.com>; Tue, 01 Jul 2008 11:18:55 +0100 (BST) Date: Tue, 01 Jul 2008 11:18:53 +0100 From: Darren J Moffat Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <20080701074727.GC22698@eng.sun.com> Sender: Darren.Moffat@sun.com To: Jeff Bonwick Cc: eric kustarz , Matthew Ahrens , PSARC-ext@sun.com, zfs-team@sun.com, Tim Haley Message-id: <486A048D.70901@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <485FDA01.5080107@sun.com> <20080701074727.GC22698@eng.sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080602) Status: RO Content-Length: 758 Jeff Bonwick wrote: > The L2ARC seems more complex because allocation is done by the feed thread, > not in the context of any particular dataset doing I/O. The feed thread > has no way of knowing on whose behalf blocks exist. So it would have to > operate as it does today, caching everything that falls out of the ARC. > It's not clear to me how the L2ARC policy can be anything other than a > pool-wide property. I had to solve a very similar problem for enabling encryption support in the L2ARC. I did this by using the internal flags field in the ARC header to record wither or not the the initial read/write in the ARC was for an encrypted dataset. I think this case is slightly tricker but I can see how it could be done. -- Darren J Moffat From Mark.Maybee@sun.com Tue Jul 1 09:27:04 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m61GR3eO027092 for ; Tue, 1 Jul 2008 09:27:03 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m61GQibc003734; Wed, 2 Jul 2008 00:27:02 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K3C0061B5P09Y00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 01 Jul 2008 09:27:00 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K3C00N245OZ0E60@nwk-avmta-1.sfbay.Sun.COM>; Tue, 01 Jul 2008 09:27:00 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m61GQx93017317; Tue, 01 Jul 2008 16:26:59 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K3C00M012ZII400@mail-amer.sun.com> (original mail from Mark.Maybee@Sun.COM); Tue, 01 Jul 2008 10:26:59 -0600 (MDT) Received: from [172.20.25.54] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K3C001RJ5OKWBB0@mail-amer.sun.com>; Tue, 01 Jul 2008 10:26:44 -0600 (MDT) Date: Tue, 01 Jul 2008 10:26:44 -0600 From: Mark Maybee Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <486A048D.70901@Sun.COM> Sender: Mark.Maybee@sun.com To: Darren J Moffat Cc: Jeff Bonwick , eric kustarz , Matthew Ahrens , psarc-ext@sun.com, zfs-team@sun.com, Tim Haley Message-id: <486A5AC4.7090304@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-Accept-Language: en-us, en X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <485FDA01.5080107@sun.com> <20080701074727.GC22698@eng.sun.com> <486A048D.70901@Sun.COM> User-Agent: Mozilla Thunderbird 1.0.2 (X11/20050323) Status: RO Content-Length: 907 Darren J Moffat wrote: > Jeff Bonwick wrote: > >> The L2ARC seems more complex because allocation is done by the feed >> thread, >> not in the context of any particular dataset doing I/O. The feed thread >> has no way of knowing on whose behalf blocks exist. So it would have to >> operate as it does today, caching everything that falls out of the ARC. >> It's not clear to me how the L2ARC policy can be anything other than a >> pool-wide property. > > > I had to solve a very similar problem for enabling encryption support in > the L2ARC. I did this by using the internal flags field in the ARC > header to record wither or not the the initial read/write in the ARC was > for an encrypted dataset. I think this case is slightly tricker but I > can see how it could be done. > Yup, this was Eric's proposed solution: an L2ARC_IS_CACHABLE flag that is added when the arc buffer is allocated. From Jeff.Bonwick@sun.com Tue Jul 1 14:29:45 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m61LTjOS009789 for ; Tue, 1 Jul 2008 14:29:45 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m61LTfhX015143; Tue, 1 Jul 2008 14:29:42 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K3C00E13JPHKY00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 01 Jul 2008 14:29:41 -0700 (PDT) Received: from zion.sfbay.sun.com ([129.146.17.75]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K3C00DIYJPHO500@nwk-avmta-1.sfbay.Sun.COM>; Tue, 01 Jul 2008 14:29:41 -0700 (PDT) Received: from zion.sfbay.sun.com (localhost [127.0.0.1]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m61LTft0012435; Tue, 01 Jul 2008 21:29:41 +0000 (GMT) Received: (from bonwick@localhost) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2/Submit) id m61LTesp012434; Tue, 01 Jul 2008 14:29:40 -0700 (PDT) Date: Tue, 01 Jul 2008 14:29:40 -0700 From: Jeff Bonwick Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <486A5AC4.7090304@Sun.COM> To: Mark Maybee Cc: Darren J Moffat , Jeff Bonwick , eric kustarz , Matthew Ahrens , psarc-ext@sun.com, zfs-team@sun.com, Tim Haley Message-id: <20080701212940.GD10570@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <485FDA01.5080107@sun.com> <20080701074727.GC22698@eng.sun.com> <486A048D.70901@Sun.COM> <486A5AC4.7090304@Sun.COM> X-Authentication-warning: zion.sfbay.sun.com: bonwick set sender to Jeff.Bonwick@sun.com using -f User-Agent: Mutt/1.5.14 (2007-02-12) Status: RO Content-Length: 1124 > Yup, this was Eric's proposed solution: an L2ARC_IS_CACHABLE flag that > is added when the arc buffer is allocated. That doesn't seem quite right. Consider this sequence of events, where C and NC are two datasets with and without L2ARC caching: NC: reads block B, puts it in the ARC with L2ARC_IS_CACHEABLE clear C: reads block B, but since it's not the one doing the allocation, doesn't set L2ARC_IS_CACHEABLE feed thread: discards block B C: reads block B, going to disk instead of L2ARC as it should have But if C had read the block before NC, everything would have been fine. It seems to me that the order of access should not determine the policy. Whenever a block is looked up in the ARC, if the caller wants that block to be L2ARC cached, then the L2ARC_IS_CACHEABLE flag should be ORed in. In short, there are three possibly policies: (1) If *anyone* wants the block cached, it is cached. (2) If *anyone* doesn't want the block cached, it is not cached. (3) It might or might not be cached depending who gets there first. To me, (1) is the only policy that makes sense, or is even defensible. Jeff From Matthew.Ahrens@sun.com Tue Jul 1 14:40:19 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m61LeJbp010038 for ; Tue, 1 Jul 2008 14:40:19 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m61LeFKq039128; Tue, 1 Jul 2008 15:40:17 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K3C0070JK74SU00@nwk-avmta-2.sfbay.sun.com>; Tue, 01 Jul 2008 14:40:16 -0700 (PDT) Received: from zion.sfbay.sun.com ([129.146.17.75]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K3C007KYK73BS00@nwk-avmta-2.sfbay.sun.com>; Tue, 01 Jul 2008 14:40:15 -0700 (PDT) Received: from stationary.local (punchin-ahrens.SFBay.Sun.COM [10.7.251.178]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m61LeFTZ012782; Tue, 01 Jul 2008 21:40:15 +0000 (GMT) Date: Tue, 01 Jul 2008 14:39:43 -0700 From: Matthew Ahrens Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <20080701212940.GD10570@eng.sun.com> To: Jeff Bonwick Cc: Mark Maybee , Darren J Moffat , eric kustarz , psarc-ext@sun.com, zfs-team@sun.com, Tim Haley Message-id: <486AA41F.1010108@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <485FDA01.5080107@sun.com> <20080701074727.GC22698@eng.sun.com> <486A048D.70901@Sun.COM> <486A5AC4.7090304@Sun.COM> <20080701212940.GD10570@eng.sun.com> User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) Status: RO Content-Length: 1326 Jeff Bonwick wrote: >> Yup, this was Eric's proposed solution: an L2ARC_IS_CACHABLE flag that >> is added when the arc buffer is allocated. > > That doesn't seem quite right. Consider this sequence of events, > where C and NC are two datasets with and without L2ARC caching: > > NC: reads block B, puts it in the ARC with L2ARC_IS_CACHEABLE clear > C: reads block B, but since it's not the one doing the allocation, > doesn't set L2ARC_IS_CACHEABLE > feed thread: discards block B > C: reads block B, going to disk instead of L2ARC as it should have > > But if C had read the block before NC, everything would have been fine. > > It seems to me that the order of access should not determine the policy. > Whenever a block is looked up in the ARC, if the caller wants that block > to be L2ARC cached, then the L2ARC_IS_CACHEABLE flag should be ORed in. > In short, there are three possibly policies: > > (1) If *anyone* wants the block cached, it is cached. > (2) If *anyone* doesn't want the block cached, it is not cached. > (3) It might or might not be cached depending who gets there first. > > To me, (1) is the only policy that makes sense, or is even defensible. Agreed. Seems like this could be accomplished in the above scenario by having C set L2ARC_IS_CACHEABLE when it hits in the cache. --matt From Mark.Maybee@sun.com Tue Jul 1 14:49:09 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m61Ln8bx010272 for ; Tue, 1 Jul 2008 14:49:09 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m61LmdqZ011196; Wed, 2 Jul 2008 05:48:56 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K3C00817KLHAR00@brm-avmta-1.central.sun.com>; Tue, 01 Jul 2008 15:48:53 -0600 (MDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K3C00071KLGEM50@brm-avmta-1.central.sun.com>; Tue, 01 Jul 2008 15:48:52 -0600 (MDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m61Lmq8e015181; Tue, 01 Jul 2008 21:48:52 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K3C00301K3UVL00@mail-amer.sun.com> (original mail from Mark.Maybee@Sun.COM); Tue, 01 Jul 2008 15:48:52 -0600 (MDT) Received: from [172.20.25.54] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K3C00AZPKL21H80@mail-amer.sun.com>; Tue, 01 Jul 2008 15:48:38 -0600 (MDT) Date: Tue, 01 Jul 2008 15:48:38 -0600 From: Mark Maybee Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <486AA41F.1010108@sun.com> Sender: Mark.Maybee@sun.com To: Matthew Ahrens Cc: Jeff Bonwick , Darren J Moffat , eric kustarz , psarc-ext@sun.com, zfs-team@sun.com, Tim Haley Message-id: <486AA636.2040503@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-Accept-Language: en-us, en X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <485FDA01.5080107@sun.com> <20080701074727.GC22698@eng.sun.com> <486A048D.70901@Sun.COM> <486A5AC4.7090304@Sun.COM> <20080701212940.GD10570@eng.sun.com> <486AA41F.1010108@sun.com> User-Agent: Mozilla Thunderbird 1.0.2 (X11/20050323) Status: RO Content-Length: 1611 Matthew Ahrens wrote: > Jeff Bonwick wrote: > >>> Yup, this was Eric's proposed solution: an L2ARC_IS_CACHABLE flag that >>> is added when the arc buffer is allocated. >> >> >> That doesn't seem quite right. Consider this sequence of events, >> where C and NC are two datasets with and without L2ARC caching: >> >> NC: reads block B, puts it in the ARC with L2ARC_IS_CACHEABLE clear >> C: reads block B, but since it's not the one doing the allocation, >> doesn't set L2ARC_IS_CACHEABLE >> feed thread: discards block B >> C: reads block B, going to disk instead of L2ARC as it should have >> >> But if C had read the block before NC, everything would have been fine. >> >> It seems to me that the order of access should not determine the policy. >> Whenever a block is looked up in the ARC, if the caller wants that block >> to be L2ARC cached, then the L2ARC_IS_CACHEABLE flag should be ORed in. >> In short, there are three possibly policies: >> >> (1) If *anyone* wants the block cached, it is cached. >> (2) If *anyone* doesn't want the block cached, it is not cached. >> (3) It might or might not be cached depending who gets there first. >> >> To me, (1) is the only policy that makes sense, or is even defensible. > > > Agreed. Seems like this could be accomplished in the above scenario by > having C set L2ARC_IS_CACHEABLE when it hits in the cache. > Indeed... and looking back at the actual webrev (instead of relying on my memory) I see that Eric actually implemented something like (2) above. That is, he uses a ARC_DONT_L2CACHE flag and sets it when NC gets a hit. From Jeff.Bonwick@sun.com Tue Jul 1 15:01:32 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m61M1Vht010832 for ; Tue, 1 Jul 2008 15:01:32 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m61M0tUl015131; Wed, 2 Jul 2008 06:01:25 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K3C00I07L6C3T00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 01 Jul 2008 15:01:24 -0700 (PDT) Received: from zion.sfbay.sun.com ([129.146.17.75]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K3C00D7AL6BNV20@nwk-avmta-1.sfbay.Sun.COM>; Tue, 01 Jul 2008 15:01:23 -0700 (PDT) Received: from zion.sfbay.sun.com (localhost [127.0.0.1]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m61M1N8l013277; Tue, 01 Jul 2008 22:01:23 +0000 (GMT) Received: (from bonwick@localhost) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2/Submit) id m61M1NAo013276; Tue, 01 Jul 2008 15:01:23 -0700 (PDT) Date: Tue, 01 Jul 2008 15:01:22 -0700 From: Jeff Bonwick Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <486AA636.2040503@Sun.COM> To: Mark Maybee Cc: Matthew Ahrens , Jeff Bonwick , Darren J Moffat , eric kustarz , psarc-ext@sun.com, zfs-team@sun.com, Tim Haley Message-id: <20080701220122.GA13213@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <485FDA01.5080107@sun.com> <20080701074727.GC22698@eng.sun.com> <486A048D.70901@Sun.COM> <486A5AC4.7090304@Sun.COM> <20080701212940.GD10570@eng.sun.com> <486AA41F.1010108@sun.com> <486AA636.2040503@Sun.COM> X-Authentication-warning: zion.sfbay.sun.com: bonwick set sender to Jeff.Bonwick@sun.com using -f User-Agent: Mutt/1.5.14 (2007-02-12) Status: RO Content-Length: 814 > >>(1) If *anyone* wants the block cached, it is cached. > >>(2) If *anyone* doesn't want the block cached, it is not cached. > >>(3) It might or might not be cached depending who gets there first. > >> > >>To me, (1) is the only policy that makes sense, or is even defensible. > > > >Agreed. Seems like this could be accomplished in the above scenario by > >having C set L2ARC_IS_CACHEABLE when it hits in the cache. > > > Indeed... and looking back at the actual webrev (instead of relying on > my memory) I see that Eric actually implemented something like (2) > above. That is, he uses a ARC_DONT_L2CACHE flag and sets it when NC > gets a hit. OK. So it seems like the solution is to invert the logic there, and OR in ARC_DO_L2CACHE on any secondarily-cacheable cache hit. Everyone cool with that? Jeff From Mark.Maybee@Sun.COM Tue Jul 1 15:44:21 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m61MiLq2011849 for ; Tue, 1 Jul 2008 15:44:21 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m61MiL8t056495; Tue, 1 Jul 2008 16:44:21 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K3C00C01N5WND00@brm-avmta-1.central.sun.com>; Tue, 01 Jul 2008 16:44:20 -0600 (MDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K3C000WRN5WEO60@brm-avmta-1.central.sun.com>; Tue, 01 Jul 2008 16:44:20 -0600 (MDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m61MiKV2025872; Tue, 01 Jul 2008 22:44:20 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K3C00801MYCT700@mail-amer.sun.com> (original mail from Mark.Maybee@Sun.COM); Tue, 01 Jul 2008 16:44:20 -0600 (MDT) Received: from [172.20.25.54] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K3C006QIN5TT290@mail-amer.sun.com>; Tue, 01 Jul 2008 16:44:17 -0600 (MDT) Date: Tue, 01 Jul 2008 16:44:17 -0600 From: Mark Maybee Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <20080701220122.GA13213@eng.sun.com> Sender: Mark.Maybee@Sun.COM To: Jeff Bonwick Cc: Matthew Ahrens , Darren J Moffat , eric kustarz , psarc-ext@Sun.COM, zfs-team@Sun.COM, Tim Haley Message-id: <486AB341.7090806@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-Accept-Language: en-us, en X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <485FDA01.5080107@sun.com> <20080701074727.GC22698@eng.sun.com> <486A048D.70901@Sun.COM> <486A5AC4.7090304@Sun.COM> <20080701212940.GD10570@eng.sun.com> <486AA41F.1010108@sun.com> <486AA636.2040503@Sun.COM> <20080701220122.GA13213@eng.sun.com> User-Agent: Mozilla Thunderbird 1.0.2 (X11/20050323) Status: RO Content-Length: 859 Jeff Bonwick wrote: >>>>(1) If *anyone* wants the block cached, it is cached. >>>>(2) If *anyone* doesn't want the block cached, it is not cached. >>>>(3) It might or might not be cached depending who gets there first. >>>> >>>>To me, (1) is the only policy that makes sense, or is even defensible. >>> >>>Agreed. Seems like this could be accomplished in the above scenario by >>>having C set L2ARC_IS_CACHEABLE when it hits in the cache. >>> >> >>Indeed... and looking back at the actual webrev (instead of relying on >>my memory) I see that Eric actually implemented something like (2) >>above. That is, he uses a ARC_DONT_L2CACHE flag and sets it when NC >>gets a hit. > > > OK. So it seems like the solution is to invert the logic there, > and OR in ARC_DO_L2CACHE on any secondarily-cacheable cache hit. > Everyone cool with that? > Works for me. From Darren.Moffat@sun.com Wed Jul 2 03:26:43 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m62AQgXX002572 for ; Wed, 2 Jul 2008 03:26:42 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m62AQI5A027597; Wed, 2 Jul 2008 18:26:29 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K3D00A01JO4PO00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 02 Jul 2008 03:26:28 -0700 (PDT) Received: from gmp-eb-inf-2.sun.com ([192.18.6.24]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K3D005JDJO3XE20@nwk-avmta-1.sfbay.Sun.COM>; Wed, 02 Jul 2008 03:26:28 -0700 (PDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-2-fe1.eu.sun.com [192.18.6.10]) by gmp-eb-inf-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m62AQRol029306; Wed, 02 Jul 2008 10:26:27 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K3D00601J88PM00@fe-emea-10.sun.com> (original mail from Darren.Moffat@Sun.COM); Wed, 02 Jul 2008 11:26:27 +0100 (BST) Received: from [129.156.173.21] by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K3D00J91JNZD0D0@fe-emea-10.sun.com>; Wed, 02 Jul 2008 11:26:25 +0100 (BST) Date: Wed, 02 Jul 2008 11:26:23 +0100 From: Darren J Moffat Subject: Re: zfs primarycache and secondarycache properties [PSARC/2008/393 FastTrack timeout 06/27/2008] In-reply-to: <20080701220122.GA13213@eng.sun.com> Sender: Darren.Moffat@sun.com To: Jeff Bonwick Cc: Mark Maybee , Matthew Ahrens , eric kustarz , psarc-ext@sun.com, zfs-team@sun.com, Tim Haley Message-id: <486B57CF.3040104@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200806201818.m5KIISO0028824@spidey.Central.Sun.COM> <485FDA01.5080107@sun.com> <20080701074727.GC22698@eng.sun.com> <486A048D.70901@Sun.COM> <486A5AC4.7090304@Sun.COM> <20080701212940.GD10570@eng.sun.com> <486AA41F.1010108@sun.com> <486AA636.2040503@Sun.COM> <20080701220122.GA13213@eng.sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080602) Status: RO Content-Length: 883 Jeff Bonwick wrote: >>>> (1) If *anyone* wants the block cached, it is cached. >>>> (2) If *anyone* doesn't want the block cached, it is not cached. >>>> (3) It might or might not be cached depending who gets there first. >>>> >>>> To me, (1) is the only policy that makes sense, or is even defensible. >>> Agreed. Seems like this could be accomplished in the above scenario by >>> having C set L2ARC_IS_CACHEABLE when it hits in the cache. >>> >> Indeed... and looking back at the actual webrev (instead of relying on >> my memory) I see that Eric actually implemented something like (2) >> above. That is, he uses a ARC_DONT_L2CACHE flag and sets it when NC >> gets a hit. > > OK. So it seems like the solution is to invert the logic there, > and OR in ARC_DO_L2CACHE on any secondarily-cacheable cache hit. > Everyone cool with that? Makes sense to me. -- Darren J Moffat