From tim.haley@Sun.COM Thu Apr 1 08:26:39 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o31FQdCg027717 for ; Thu, 1 Apr 2010 08:26:39 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o31FQZFK012483; Thu, 1 Apr 2010 09:26:38 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0700705EW98V00@brm-avmta-1.central.sun.com>; Thu, 01 Apr 2010 09:26:33 -0600 (MDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L070049EEW6GU20@brm-avmta-1.central.sun.com>; Thu, 01 Apr 2010 09:26:30 -0600 (MDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o31FQUQi018343; Thu, 01 Apr 2010 15:26:30 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0L0700400ETOUD00@mail-amer.sun.com>; Thu, 01 Apr 2010 09:26:30 -0600 (MDT) Received: from Giles.local ([unknown] [71.196.247.195]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0L0700HBDEW57T20@mail-amer.sun.com>; Thu, 01 Apr 2010 09:26:30 -0600 (MDT) Date: Thu, 01 Apr 2010 09:26:29 -0600 From: Tim Haley Subject: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] Sender: Timothy.Haley@Sun.COM To: PSARC-ext@Sun.COM Cc: zfs-team@Sun.COM, milek@task.gda.pl Message-id: <4BB4BB25.6080607@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 Status: RO Content-Length: 3786 Template Version: @(#)sac_nextcase 1.70 03/30/10 SMI This information is Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. 1. Introduction 1.1. Project/Component Working Name: zil synchronicity 1.2. Name of Document Author/Supplier: Author: Neil Perrin 1.3 Date of This Document: 01 April, 2010 4. Technical Description I am sponsoring the following fast-track for Robert Milkowski and Neil Perrin. It introduces a new dataset property for controlling synchronous behavior. The case requests micro/patch binding. 1. Summary Provide administrators of zfs with the ability to control the behavior of synchronous requests (e.g. fsync, O_DSYNC). In particular, new capabilities are proposed to 1) delay executing the synchronous request and 2) force all requests to be synchronous. The current POSIX behavior of ensuring all synchronous requests are written to stable storage would remain the default. 2. Background Currently ZFS have no official control over synchronous behaviour. However, unfortunately, a zfs module switch (zil_disable) has been fairly well publicized to disable the ZIL, the code which enforces synchronous requests. This is a global kernel variable and so affects all file systems within all pools. It should be noted that ZFS pools are always consistent. That is, the intent log is not required for pool integrity. This is due to the transactional behavior of ZFS and in particular its transaction group commit (txg) model. What the ZIL does is simply to ensure that synchronous write requests are committed to stable storage prior to returning from the system call. There are reasonable cases where the administrator understands the consequences of disabling synchronous behavior. For example, if the system crashes they start again from scratch. This is quicker than enabling synchronous behavior. There are also reasons an administrator may want to enable synchronous behavior for all writes. This might help debug an issue where synchronous writes are needed. 3. Proposal The options and semantics proposed for a new zfs dataset property: sync=default This is the default option. Synchronous file system transactions (fsync, O_DSYNC, O_SYNC, etc) are written out (to the intent log) and then secondly all devices written are flushed to ensure the data is stable (not cached by device controllers). sync=always For the ultra-cautious, every file system transaction would be written and flushed to stable storage by system call return. This obviously has a big performance penalty. sync=disabled Synchronous requests are disabled. File system transactions only commit to stable storage on the next DMU transaction group commit which can be many seconds. This option will give the highest performance. However, it is very dangerous as ZFS would be ignoring the synchronous transaction demands of applications such as databases or NFS. Administrators should only use this when these risks are understood. The property can be set when the dataset is created, or dynamically, and will take effect immediately. To change the property, an administrator can use the standard 'zfs' command. For example: # zfs create -o sync=disabled whirlpool/milek # zfs set sync=always whirlpool/perrin The current value of 'sync' can be retrieved in the usual manner with 'zfs get sync' or 'zfs list -o sync'. The sync property is not inherited from parent datasets. 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From casper@holland.sun.com Thu Apr 1 08:34:36 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o31FYaVp027772 for ; Thu, 1 Apr 2010 08:34:36 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o31FYaL2021207; Thu, 1 Apr 2010 09:34:36 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0700H09F9N4S00@nwk-avmta-2.sfbay.sun.com>; Thu, 01 Apr 2010 08:34:35 -0700 (PDT) Received: from dm-holland-01.uk.sun.com ([129.156.101.192]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0700FGYF9MVZ20@nwk-avmta-2.sfbay.sun.com>; Thu, 01 Apr 2010 08:34:35 -0700 (PDT) Received: from holland (room101.Holland.Sun.COM [10.16.117.40]) by dm-holland-01.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o31FYXDv026514; Thu, 01 Apr 2010 16:34:33 +0100 (BST) Date: Thu, 01 Apr 2010 17:34:33 +0200 From: Casper.Dik@sun.com Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BB4BB25.6080607@sun.com> Sender: casper@holland.sun.com To: Tim Haley Cc: PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <201004011534.o31FYXDv026514@dm-holland-01.uk.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <4BB4BB25.6080607@sun.com> Status: RO Content-Length: 742 >sync=disabled > Synchronous requests are disabled. File system transactions > only commit to stable storage on the next DMU transaction group > commit which can be many seconds. This option will give the > highest performance. However, it is very dangerous as ZFS > would be ignoring the synchronous transaction demands of > applications such as databases or NFS. Administrators should > only use this when these risks are understood. Is there a partialy order defined for all filesystem operations? Specifically, will ZFS guarantee that when fsync()/O_DATA happens on a file, that later transactions will not be in an earlier transaction group? (Or is this already the case?) Casper From Darren.Moffat@Oracle.COM Thu Apr 1 08:41:23 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o31FfNB2027809 for ; Thu, 1 Apr 2010 08:41:23 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o31FfM6F008552; Thu, 1 Apr 2010 08:41:22 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L070090DFKYCI00@nwk-avmta-1.sfbay.Sun.COM>; Thu, 01 Apr 2010 08:41:22 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0700KHSFKYAT60@nwk-avmta-1.sfbay.Sun.COM>; Thu, 01 Apr 2010 08:41:22 -0700 (PDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o31FfLLJ022192; Thu, 01 Apr 2010 15:41:21 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o31FfCwV029566; Thu, 01 Apr 2010 15:41:12 +0000 (GMT) Received: from abhmt005.oracle.com by acsmt354.oracle.com with ESMTP id 138525091270136409; Thu, 01 Apr 2010 08:40:09 -0700 Received: from [10.7.251.221] (/10.7.251.221) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 01 Apr 2010 08:40:09 -0700 Date: Thu, 01 Apr 2010 16:40:06 +0100 From: Darren J Moffat Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BB4BB25.6080607@sun.com> To: Tim Haley Cc: PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <4BB4BE56.6010801@Oracle.COM> Organization: Oracle Solaris Security MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090204.4BB4BE9C.018D:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.7) Gecko/20100214 Lightning/1.0b1 Thunderbird/3.0.1 Status: RO Content-Length: 555 On 01/04/2010 16:26, Tim Haley wrote: > The current value of 'sync' can be retrieved in the usual manner > with 'zfs get sync' or 'zfs list -o sync'. The sync property is not > inherited from parent datasets. Why isn't it inherited from the parent dataset ? The logbias property which effects ZIL behaviour is inherited. It can see having it inherited being useful for setting on a parent dataset that contains a large number of ZVOLs being used for COMSTAR or filesystems for NFS where all of them need the same sync behaviour. -- Darren J Moffat From andrew.gabriel@oracle.com Thu Apr 1 08:54:28 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o31FsSFR028762 for ; Thu, 1 Apr 2010 08:54:28 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o31FsSuh015224; Thu, 1 Apr 2010 08:54:28 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0700905G6RTC00@brm-avmta-1.central.sun.com>; Thu, 01 Apr 2010 09:54:27 -0600 (MDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L07004JAG6RGO40@brm-avmta-1.central.sun.com>; Thu, 01 Apr 2010 09:54:27 -0600 (MDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o31FsRFM008868; Thu, 01 Apr 2010 15:54:27 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o31EDLkO022269; Thu, 01 Apr 2010 15:54:24 +0000 (GMT) Received: from abhmt020.oracle.com by acsmt354.oracle.com with ESMTP id 138578581270137262; Thu, 01 Apr 2010 08:54:22 -0700 Received: from [192.168.2.135] (/81.187.74.206) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 01 Apr 2010 08:54:22 -0700 Date: Thu, 01 Apr 2010 16:54:42 +0100 From: Andrew Gabriel Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BB4BB25.6080607@sun.com> To: Tim Haley Cc: PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <4BB4C1C2.9050503@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090209.4BB4C1B0.01CE:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> User-Agent: Thunderbird 2.0.0.22 (X11/20090908) Status: RO Content-Length: 855 Tim Haley wrote: > I am sponsoring the following fast-track for Robert Milkowski and Neil > Perrin. It introduces a new dataset property for controlling > synchronous behavior. The case requests micro/patch binding. Thanks for this. One nit... > The options and semantics proposed for a new zfs dataset property: > > sync=default > This is the default option. Synchronous file system transactions > (fsync, O_DSYNC, O_SYNC, etc) are written out (to the intent log) > and then secondly all devices written are flushed to ensure > the data is stable (not cached by device controllers). The naming of this one seems strange. I would have had something like sync=posix, and document that sync=posix is the default. (Maybe posix isn't the right term.) sync=default doesn't actually tell you what the behavior is. -- Andrew From Nicolas.Williams@sun.com Thu Apr 1 09:17:17 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o31GHHqx029235 for ; Thu, 1 Apr 2010 09:17:17 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o31GHF8S023506; Thu, 1 Apr 2010 11:17:16 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0700I01H8SQR00@nwk-avmta-1.sfbay.Sun.COM>; Thu, 01 Apr 2010 09:17:16 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0700K2MH8RASB0@nwk-avmta-1.sfbay.Sun.COM>; Thu, 01 Apr 2010 09:17:16 -0700 (PDT) Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o31GHFLB026514; Thu, 01 Apr 2010 16:17:15 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o31C7Hsi024056; Thu, 01 Apr 2010 16:17:14 +0000 (GMT) Received: from abhmt008.oracle.com by acsmt354.oracle.com with ESMTP id 130963331270138633; Thu, 01 Apr 2010 09:17:13 -0700 Received: from Sun.COM (/129.153.128.104) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 01 Apr 2010 09:17:12 -0700 Date: Thu, 01 Apr 2010 11:17:06 -0500 From: Nicolas Williams Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BB4C1C2.9050503@oracle.com> To: Andrew Gabriel Cc: Tim Haley , PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <20100401161706.GE4225@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090204.4BB4C70B.0054,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <4BB4C1C2.9050503@oracle.com> User-Agent: Mutt/1.5.20 (2010-03-02) Status: RO Content-Length: 573 On Thu, Apr 01, 2010 at 04:54:42PM +0100, Andrew Gabriel wrote: > >The options and semantics proposed for a new zfs dataset property: > > > >sync=default > > [...] > > The naming of this one seems strange. > I would have had something like sync=posix, and document that > sync=posix is the default. (Maybe posix isn't the right term.) > sync=default doesn't actually tell you what the behavior is. I agree that POSIX isn't the right term. I don't mind "default", but if others do then I recommend "explicit" (as in, when the application requests sync explicitly). From neil.perrin@oracle.com Thu Apr 1 10:28:36 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o31HSa20001146 for ; Thu, 1 Apr 2010 10:28:36 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o31HSZs9051248; Thu, 1 Apr 2010 11:28:36 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L070000DKJNMA00@nwk-avmta-2.sfbay.sun.com>; Thu, 01 Apr 2010 10:28:35 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0700FOTKJMVUC0@nwk-avmta-2.sfbay.sun.com>; Thu, 01 Apr 2010 10:28:34 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o31HSYb8016709; Thu, 01 Apr 2010 17:28:34 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o316RxbY027736; Thu, 01 Apr 2010 17:28:28 +0000 (GMT) Received: from abhmt016.oracle.com by acsmt354.oracle.com with ESMTP id 138916001270142899; Thu, 01 Apr 2010 10:28:19 -0700 Received: from [129.147.9.21] (/129.147.9.21) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 01 Apr 2010 10:28:19 -0700 Date: Thu, 01 Apr 2010 11:28:18 -0600 From: Neil Perrin Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BB4BE56.6010801@Oracle.COM> To: Darren J Moffat Cc: PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <4BB4D7B2.8070307@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090204.4BB4D7BD.00A8:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> User-Agent: Thunderbird 2.0.0.23 (X11/20090910) Status: RO Content-Length: 920 On 04/01/10 09:40, Darren J Moffat wrote: > On 01/04/2010 16:26, Tim Haley wrote: >> The current value of 'sync' can be retrieved in the usual manner >> with 'zfs get sync' or 'zfs list -o sync'. The sync property is not >> inherited from parent datasets. > > Why isn't it inherited from the parent dataset ? > > The logbias property which effects ZIL behaviour is inherited. > > It can see having it inherited being useful for setting on a parent > dataset that contains a large number of ZVOLs being used for COMSTAR > or filesystems for NFS where all of them need the same sync behaviour. We've flip-flopped on whether it should be inherited. It's currently coded as inherited, and I know Robert believes it should stay that way. Anyway, it was generally felt by the zfs group that sync=disabled was sufficiently dangerous to require explicit setting on each dataset. I would be ok with it being inherited. Neil. From Darren.Moffat@oracle.com Thu Apr 1 10:34:36 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o31HYaun001199 for ; Thu, 1 Apr 2010 10:34:36 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o31HTNjJ005936; Thu, 1 Apr 2010 10:34:25 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0700019KT9YS00@nwk-avmta-2.sfbay.sun.com>; Thu, 01 Apr 2010 10:34:21 -0700 (PDT) Received: from sca-ea-mail-4.sun.com ([192.18.43.22]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0700F1DKT8W0D0@nwk-avmta-2.sfbay.sun.com>; Thu, 01 Apr 2010 10:34:20 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by sca-ea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o31HYK3J018310; Thu, 01 Apr 2010 17:34:20 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o31H8SA3010739; Thu, 01 Apr 2010 17:34:16 +0000 (GMT) Received: from abhmt012.oracle.com by acsmt354.oracle.com with ESMTP id 138936821270143217; Thu, 01 Apr 2010 10:33:37 -0700 Received: from [10.7.251.221] (/10.7.251.221) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 01 Apr 2010 10:33:37 -0700 Date: Thu, 01 Apr 2010 18:33:33 +0100 From: Darren J Moffat Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BB4D7B2.8070307@oracle.com> To: Neil Perrin Cc: PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <4BB4D8ED.4000402@Oracle.COM> Organization: Oracle Solaris Security MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090204.4BB4D918.0103:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> <4BB4D7B2.8070307@oracle.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.7) Gecko/20100214 Lightning/1.0b1 Thunderbird/3.0.1 Status: RO Content-Length: 630 On 01/04/2010 18:28, Neil Perrin wrote: > We've flip-flopped on whether it should be inherited. It's currently > coded as inherited, and I know Robert believes it should stay that way. > Anyway, it was generally felt by the zfs group that sync=disabled > was sufficiently dangerous to require explicit setting on each dataset. > I would be ok with it being inherited. I agree that it is dangerous but I'm not sure that the change in semantics means it shouldn't be inherited. On the other hand I can't think of any other equivalently "dangerous" (to applications view of the world) per dataset property. -- Darren J Moffat From bill.sommerfeld@oracle.com Thu Apr 1 11:34:28 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o31IYRKE002173 for ; Thu, 1 Apr 2010 11:34:28 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o31IYN5Q034712; Thu, 1 Apr 2010 12:34:27 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0700403NLEMQ00@nwk-avmta-2.sfbay.sun.com>; Thu, 01 Apr 2010 11:34:26 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0700225NLD5440@nwk-avmta-2.sfbay.sun.com>; Thu, 01 Apr 2010 11:34:25 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o31IYO1q016069; Thu, 01 Apr 2010 18:34:25 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o2VEKXRM019381; Thu, 01 Apr 2010 18:34:20 +0000 (GMT) Received: from abhmt021.oracle.com by acsmt355.oracle.com with ESMTP id 131325131270146793; Thu, 01 Apr 2010 11:33:13 -0700 Received: from [129.146.109.133] (/129.146.109.133) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 01 Apr 2010 11:33:12 -0700 Date: Thu, 01 Apr 2010 11:33:11 -0700 From: Bill Sommerfeld Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BB4D7B2.8070307@oracle.com> To: Neil Perrin Cc: Darren J Moffat , PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <4BB4E6E7.9080207@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090205.4BB4E72D.011B:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> <4BB4D7B2.8070307@oracle.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.8) Gecko/20100302 Lightning/1.0b1 Thunderbird/3.0.2 Status: RO Content-Length: 896 On 04/01/10 10:28, Neil Perrin wrote: > We've flip-flopped on whether it should be inherited. It's currently > coded as inherited, and I know Robert believes it should stay that way. > Anyway, it was generally felt by the zfs group that sync=disabled > was sufficiently dangerous to require explicit setting on each dataset. > I would be ok with it being inherited. IMHO, sync=disabled is less dangerous than checksum=off, which is inherited. Both can result in silent data corruption. In the case of sync=disabled, only recently written data can be lost, and the loss occurs in conjuction with a pretty obvious event (a crash and reboot). In the case of checksum=off, any data written while the setting is in effect can be lost. Inheriting sync=always is actually useful and non-dangerous; on balance, IMHO it's less confusing and more useful for sync to be inherited. - Bill From neil.perrin@oracle.com Thu Apr 1 11:57:39 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o31Ivc9X002734 for ; Thu, 1 Apr 2010 11:57:38 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o31Ivaac052597; Thu, 1 Apr 2010 12:57:36 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0700E1NOO00B00@nwk-avmta-1.sfbay.Sun.COM>; Thu, 01 Apr 2010 11:57:36 -0700 (PDT) Received: from sca-ea-mail-3.sun.com ([192.18.43.21]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0700CP7ONZFE70@nwk-avmta-1.sfbay.Sun.COM>; Thu, 01 Apr 2010 11:57:35 -0700 (PDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by sca-ea-mail-3.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o31IvZbI021732; Thu, 01 Apr 2010 18:57:35 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o31IvVbE022196; Thu, 01 Apr 2010 18:57:31 +0000 (GMT) Received: from abhmt001.oracle.com by acsmt354.oracle.com with ESMTP id 131386851270148184; Thu, 01 Apr 2010 11:56:24 -0700 Received: from [129.147.9.21] (/129.147.9.21) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 01 Apr 2010 11:56:24 -0700 Date: Thu, 01 Apr 2010 12:56:23 -0600 From: Neil Perrin Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <201004011534.o31FYXDv026514@dm-holland-01.uk.sun.com> To: Casper.Dik@sun.com Cc: PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <4BB4EC57.8@oracle.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_YbsJXcTAVW4mjzX06rpWcw)" X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090202.4BB4EC9D.0082:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <201004011534.o31FYXDv026514@dm-holland-01.uk.sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090910) Status: RO Content-Length: 3170 This is a multi-part message in MIME format. --Boundary_(ID_YbsJXcTAVW4mjzX06rpWcw) Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT On 04/01/10 09:34, Casper.Dik@sun.com wrote: >> sync=disabled >> Synchronous requests are disabled. File system transactions >> only commit to stable storage on the next DMU transaction group >> commit which can be many seconds. This option will give the >> highest performance. However, it is very dangerous as ZFS >> would be ignoring the synchronous transaction demands of >> applications such as databases or NFS. Administrators should >> only use this when these risks are understood. >> > > Is there a partialy order defined for all filesystem operations? > File system operations will be written in order for all settings of the sync flag. > Specifically, will ZFS guarantee that when fsync()/O_DATA happens on a file, > (I assume by O_DATA you meant O_DSYNC). > that later transactions will not be in an earlier transaction group? > (Or is this already the case?) > This is already the case. > Casper --Boundary_(ID_YbsJXcTAVW4mjzX06rpWcw) Content-type: text/html; charset=ISO-8859-1 Content-transfer-encoding: 7BIT On 04/01/10 09:34, Casper.Dik@sun.com wrote: 
sync=disabled
       Synchronous requests are disabled.  File system transactions
       only commit to stable storage on the next DMU transaction group
       commit which can be many seconds.  This option will give the
       highest performance.  However, it is very dangerous as ZFS
       would be ignoring the synchronous transaction demands of
       applications such as databases or NFS.  Administrators should
       only use this when these risks are understood.
    

Is there a partialy order defined for all filesystem operations?

File system operations  will be written in order for all settings of the sync flag.

Specifically, will ZFS guarantee that when fsync()/O_DATA happens on a file,
(I assume by O_DATA you meant O_DSYNC).

that later transactions will not be in an earlier transaction group?
(Or is this already the case?)
This is already the case. 

Casper


--Boundary_(ID_YbsJXcTAVW4mjzX06rpWcw)-- From scott.rotondo@oracle.com Thu Apr 1 13:53:46 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o31Krk03004349 for ; Thu, 1 Apr 2010 13:53:46 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o31KrjLc000691; Thu, 1 Apr 2010 14:53:45 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0700D01U1L8700@nwk-avmta-2.sfbay.sun.com>; Thu, 01 Apr 2010 13:53:45 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L070025IU1K55F0@nwk-avmta-2.sfbay.sun.com>; Thu, 01 Apr 2010 13:53:45 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o31Kri5K006089; Thu, 01 Apr 2010 20:53:44 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o31CiHqL020266; Thu, 01 Apr 2010 20:53:41 +0000 (GMT) Received: from abhmt005.oracle.com by acsmt355.oracle.com with ESMTP id 139634401270155204; Thu, 01 Apr 2010 13:53:24 -0700 Received: from [129.146.108.62] (/129.146.108.62) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 01 Apr 2010 13:53:23 -0700 Date: Thu, 01 Apr 2010 13:53:26 -0700 From: Scott Rotondo Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BB4E6E7.9080207@oracle.com> To: Bill Sommerfeld Cc: Neil Perrin , Darren J Moffat , PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <4BB507C6.5020101@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090206.4BB507D6.0069:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> <4BB4D7B2.8070307@oracle.com> <4BB4E6E7.9080207@oracle.com> User-Agent: Thunderbird 2.0.0.21 (X11/20090323) Status: RO Content-Length: 1475 Bill Sommerfeld wrote: > On 04/01/10 10:28, Neil Perrin wrote: >> We've flip-flopped on whether it should be inherited. It's currently >> coded as inherited, and I know Robert believes it should stay that way. >> Anyway, it was generally felt by the zfs group that sync=disabled >> was sufficiently dangerous to require explicit setting on each dataset. >> I would be ok with it being inherited. > > IMHO, sync=disabled is less dangerous than checksum=off, which is > inherited. > > Both can result in silent data corruption. In the case of > sync=disabled, only recently written data can be lost, and the loss > occurs in conjuction with a pretty obvious event (a crash and reboot). > > In the case of checksum=off, any data written while the setting is in > effect can be lost. > > Inheriting sync=always is actually useful and non-dangerous; on balance, > IMHO it's less confusing and more useful for sync to be inherited. > > - Bill +1. I'm generally skeptical of the argument that administration should become more cumbersome if the action you're trying to take is dangerous. For one thing, it's hard to implement this principle consistently. As Bill points out, you usually don't have to look too hard to find something else that is just as dangerous but not equally cumbersome. Scott -- Scott Rotondo Senior Principal Engineer, Solaris Engineering President, Trusted Computing Group Phone/FAX: +1 408 850 3655 (Internal x68278) From scott.rotondo@oracle.com Thu Apr 1 14:00:14 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o31L0EE3004742 for ; Thu, 1 Apr 2010 14:00:14 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o31L091S019404; Thu, 1 Apr 2010 14:00:11 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0700I2JUCADS00@brm-avmta-1.central.sun.com>; Thu, 01 Apr 2010 15:00:10 -0600 (MDT) Received: from sca-ea-mail-4.sun.com ([192.18.43.22]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L070079VUC92L50@brm-avmta-1.central.sun.com>; Thu, 01 Apr 2010 15:00:09 -0600 (MDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by sca-ea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o31L09Fv007185; Thu, 01 Apr 2010 21:00:09 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o31HtVxt011208; Thu, 01 Apr 2010 21:00:06 +0000 (GMT) Received: from abhmt010.oracle.com by acsmt353.oracle.com with ESMTP id 139653591270155537; Thu, 01 Apr 2010 13:58:57 -0700 Received: from [129.146.108.62] (/129.146.108.62) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 01 Apr 2010 13:58:56 -0700 Date: Thu, 01 Apr 2010 13:59:00 -0700 From: Scott Rotondo Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <20100401161706.GE4225@Sun.COM> To: Tim Haley Cc: Nicolas Williams , Andrew Gabriel , PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <4BB50914.3080603@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090204.4BB50957.0020:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <4BB4C1C2.9050503@oracle.com> <20100401161706.GE4225@Sun.COM> User-Agent: Thunderbird 2.0.0.21 (X11/20090323) Status: RO Content-Length: 1148 Nicolas Williams wrote: > On Thu, Apr 01, 2010 at 04:54:42PM +0100, Andrew Gabriel wrote: >>> The options and semantics proposed for a new zfs dataset property: >>> >>> sync=default >>> [...] >> The naming of this one seems strange. >> I would have had something like sync=posix, and document that >> sync=posix is the default. (Maybe posix isn't the right term.) >> sync=default doesn't actually tell you what the behavior is. > > I agree that POSIX isn't the right term. I don't mind "default", but if > others do then I recommend "explicit" (as in, when the application > requests sync explicitly). There is another problem with the term "default." Someone not well-versed in UNIX file semantics might well imagine that "sync=default" means that all writes are synchronous by default, requiring some special action to open the file in asynchronous mode. That's obviously not the meaning you intend to convey. I think "explicit" is a good choice; no doubt there are others too. Scott -- Scott Rotondo Senior Principal Engineer, Solaris Engineering President, Trusted Computing Group Phone/FAX: +1 408 850 3655 (Internal x68278) From jeff.bonwick@oracle.com Thu Apr 1 18:32:02 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o321W18Y009046 for ; Thu, 1 Apr 2010 18:32:02 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o321W1g9024766; Thu, 1 Apr 2010 20:32:01 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L080090N6XD8R00@nwk-avmta-2.sfbay.sun.com>; Thu, 01 Apr 2010 18:32:01 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L080012B6XC6M60@nwk-avmta-2.sfbay.sun.com>; Thu, 01 Apr 2010 18:32:00 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o321W07X012159; Fri, 02 Apr 2010 01:32:00 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o321VugF018762; Fri, 02 Apr 2010 01:31:56 +0000 (GMT) Received: from abhmt010.oracle.com by acsmt353.oracle.com with ESMTP id 140380881270171910; Thu, 01 Apr 2010 18:31:50 -0700 Received: from [10.0.1.8] (/24.6.60.71) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 01 Apr 2010 18:31:49 -0700 Date: Thu, 01 Apr 2010 18:31:48 -0700 From: Jeff Bonwick Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BB4D8ED.4000402@Oracle.COM> To: Darren J Moffat Cc: Neil Perrin , PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <81E2698C-FB12-47F0-9A45-91959AF3B85E@oracle.com> MIME-version: 1.0 X-Mailer: Apple Mail (2.936) Content-type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A0B0206.4BB5490E.0016:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> <4BB4D7B2.8070307@oracle.com> <4BB4D8ED.4000402@Oracle.COM> Status: RO Content-Length: 1079 Yes. It should be inherited. For the property values, I'd prefer: "standard" - we honor the standard (posix) semantics, without saying posix. "always" - make every system call synchronous. "barrier" - treat fsync() only as a write barrier, which ensures that all previous writes/updates will be on stable storage before any subsequent updates. Jeff On Apr 1, 2010, at 10:33 AM, Darren J Moffat wrote: > On 01/04/2010 18:28, Neil Perrin wrote: >> We've flip-flopped on whether it should be inherited. It's currently >> coded as inherited, and I know Robert believes it should stay that >> way. >> Anyway, it was generally felt by the zfs group that sync=disabled >> was sufficiently dangerous to require explicit setting on each >> dataset. >> I would be ok with it being inherited. > > I agree that it is dangerous but I'm not sure that the change in > semantics means it shouldn't be inherited. On the other hand I > can't think of any other equivalently "dangerous" (to applications > view of the world) per dataset property. > > -- > Darren J Moffat From neil.perrin@oracle.com Thu Apr 1 22:04:18 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o3254In4011838 for ; Thu, 1 Apr 2010 22:04:18 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o3254H3u043661; Thu, 1 Apr 2010 23:04:17 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0800303GR5RA00@brm-avmta-1.central.sun.com>; Thu, 01 Apr 2010 23:04:17 -0600 (MDT) Received: from sca-ea-mail-1.sun.com ([192.18.43.24]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0800LC0GR4XT20@brm-avmta-1.central.sun.com>; Thu, 01 Apr 2010 23:04:16 -0600 (MDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by sca-ea-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o3254Fsc013728; Fri, 02 Apr 2010 05:04:16 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o324TZS3027429; Fri, 02 Apr 2010 05:04:13 +0000 (GMT) Received: from abhmt009.oracle.com by acsmt353.oracle.com with ESMTP id 140720251270184631; Thu, 01 Apr 2010 22:03:51 -0700 Received: from [129.147.9.21] (/129.147.9.21) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 01 Apr 2010 22:03:51 -0700 Date: Thu, 01 Apr 2010 23:03:50 -0600 From: Neil Perrin Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <81E2698C-FB12-47F0-9A45-91959AF3B85E@oracle.com> To: Jeff Bonwick Cc: PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <4BB57AB6.8040300@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090202.4BB57ACE.0016:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> <4BB4D7B2.8070307@oracle.com> <4BB4D8ED.4000402@Oracle.COM> <81E2698C-FB12-47F0-9A45-91959AF3B85E@oracle.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090910) Status: RO Content-Length: 1584 On 04/01/10 19:31, Jeff Bonwick wrote: > Yes. It should be inherited. Ok, that seems to be the consensus and I'm also fine with it. > > For the property values, I'd prefer: > > "standard" - we honor the standard (posix) semantics, without saying > posix. - Seems a reasonable name. > > "always" - make every system call synchronous. > > "barrier" - treat fsync() only as a write barrier, which ensures that > all previous > writes/updates will be on stable storage before any subsequent updates. Are there cases where an admin might want an applications fsyncs to be honored but O_DSYNC/O_SYNC/O_RSYNC to be ignored? Are you suggesting this new barrier option replaces disable or is in addition to it? I think we still need to provide a disable option that is equivalent to zil_disable and ignores all sync operation including fsync. > > Jeff > > On Apr 1, 2010, at 10:33 AM, Darren J Moffat wrote: > >> On 01/04/2010 18:28, Neil Perrin wrote: >>> We've flip-flopped on whether it should be inherited. It's currently >>> coded as inherited, and I know Robert believes it should stay that way. >>> Anyway, it was generally felt by the zfs group that sync=disabled >>> was sufficiently dangerous to require explicit setting on each dataset. >>> I would be ok with it being inherited. >> >> I agree that it is dangerous but I'm not sure that the change in >> semantics means it shouldn't be inherited. On the other hand I can't >> think of any other equivalently "dangerous" (to applications view of >> the world) per dataset property. >> >> -- >> Darren J Moffat > From jeff.bonwick@oracle.com Thu Apr 1 22:57:13 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o325vDxP012362 for ; Thu, 1 Apr 2010 22:57:13 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o325vDYr003358; Thu, 1 Apr 2010 22:57:13 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0800I07J7DKY00@nwk-avmta-1.sfbay.Sun.COM>; Thu, 01 Apr 2010 22:57:13 -0700 (PDT) Received: from sca-ea-mail-4.sun.com ([192.18.43.22]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0800M71J7CGG80@nwk-avmta-1.sfbay.Sun.COM>; Thu, 01 Apr 2010 22:57:12 -0700 (PDT) Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by sca-ea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o325vC1R018397; Fri, 02 Apr 2010 05:57:12 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o325v8Ft001277; Fri, 02 Apr 2010 05:57:08 +0000 (GMT) Received: from abhmt010.oracle.com by acsmt353.oracle.com with ESMTP id 140800521270187765; Thu, 01 Apr 2010 22:56:05 -0700 Received: from [10.0.1.8] (/24.6.60.71) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 01 Apr 2010 22:56:04 -0700 Date: Thu, 01 Apr 2010 22:56:03 -0700 From: Jeff Bonwick Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BB57AB6.8040300@oracle.com> To: Neil Perrin Cc: PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: MIME-version: 1.0 X-Mailer: Apple Mail (2.936) Content-type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090203.4BB58736.00B8:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> <4BB4D7B2.8070307@oracle.com> <4BB4D8ED.4000402@Oracle.COM> <81E2698C-FB12-47F0-9A45-91959AF3B85E@oracle.com> <4BB57AB6.8040300@oracle.com> Status: RO Content-Length: 1899 Yeah, I'm just suggesting "barrier" as a name rather than "disable". It's a more accurate description of the semantic ZFS provides when zil_commit() is a no-op. Jeff On Apr 1, 2010, at 10:03 PM, Neil Perrin wrote: > On 04/01/10 19:31, Jeff Bonwick wrote: >> Yes. It should be inherited. > > Ok, that seems to be the consensus and I'm also fine with it. > > >> >> For the property values, I'd prefer: >> >> "standard" - we honor the standard (posix) semantics, without >> saying posix. > > - Seems a reasonable name. > >> >> "always" - make every system call synchronous. >> >> "barrier" - treat fsync() only as a write barrier, which ensures >> that all previous >> writes/updates will be on stable storage before any subsequent >> updates. > > Are there cases where an admin might want an applications fsyncs to > be honored but > O_DSYNC/O_SYNC/O_RSYNC to be ignored? > > Are you suggesting this new barrier option replaces disable or is in > addition to it? > I think we still need to provide a disable option that is equivalent > to zil_disable > and ignores all sync operation including fsync. > >> >> Jeff >> >> On Apr 1, 2010, at 10:33 AM, Darren J Moffat wrote: >> >>> On 01/04/2010 18:28, Neil Perrin wrote: >>>> We've flip-flopped on whether it should be inherited. It's >>>> currently >>>> coded as inherited, and I know Robert believes it should stay >>>> that way. >>>> Anyway, it was generally felt by the zfs group that sync=disabled >>>> was sufficiently dangerous to require explicit setting on each >>>> dataset. >>>> I would be ok with it being inherited. >>> >>> I agree that it is dangerous but I'm not sure that the change in >>> semantics means it shouldn't be inherited. On the other hand I >>> can't think of any other equivalently "dangerous" (to applications >>> view of the world) per dataset property. >>> >>> -- >>> Darren J Moffat >> > From Roch.Bourbonnais@sun.com Fri Apr 2 01:00:26 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o3280PWs001076 for ; Fri, 2 Apr 2010 01:00:26 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o3280K8O016292; Fri, 2 Apr 2010 03:00:25 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L080071ROWO4F00@nwk-avmta-1.sfbay.Sun.COM>; Fri, 02 Apr 2010 01:00:24 -0700 (PDT) Received: from gmp-eb-inf-1.sun.com ([192.18.6.21]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0800M7BOWMQVA0@nwk-avmta-1.sfbay.Sun.COM>; Fri, 02 Apr 2010 01:00:23 -0700 (PDT) Received: from fe-emea-13.sun.com (gmp-eb-lb-1-fe1.eu.sun.com [192.18.6.7] (may be forged)) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o3280Lsw024239; Fri, 02 Apr 2010 08:00:22 +0000 (GMT) Received: from conversion-daemon.fe-emea-13.sun.com by fe-emea-13.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0L0800F00OQVH600@fe-emea-13.sun.com>; Fri, 02 Apr 2010 09:00:17 +0100 (BST) Received: from bode ([unknown] [129.157.192.177]) by fe-emea-13.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0L0800BB6OWGK740@fe-emea-13.sun.com>; Fri, 02 Apr 2010 09:00:17 +0100 (BST) Date: Fri, 02 Apr 2010 01:00:16 -0700 From: Roch Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BB4BB25.6080607@sun.com> Sender: Roch.Bourbonnais@sun.com To: Tim Haley Cc: PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <19381.42000.633475.2548@gargle.gargle.HOWL> Organization: SUN Microsystems MIME-version: 1.0 X-Mailer: VM 7.07 under 21.4 (patch 12) "Portable Code" XEmacs Lucid Content-type: text/plain; CHARSET=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <4BB4BB25.6080607@sun.com> Phone: (+33).4.76.18.83.20 (x[70]38320) Status: RO Content-Length: 4559 Tim Haley writes: > Template Version: @(#)sac_nextcase 1.70 03/30/10 SMI > This information is Copyright (c) 2010, Oracle and/or its affiliates. > All rights > reserved. > 1. Introduction > 1.1. Project/Component Working Name: > zil synchronicity > 1.2. Name of Document Author/Supplier: > Author: Neil Perrin > 1.3 Date of This Document: > 01 April, 2010 > > 4. Technical Description > > I am sponsoring the following fast-track for Robert Milkowski and Neil > Perrin. It introduces a new dataset property for controlling > synchronous behavior. The case requests micro/patch binding. > > 1. Summary > > Provide administrators of zfs with the ability to control > the behavior of synchronous requests (e.g. fsync, O_DSYNC). > In particular, new capabilities are proposed to 1) delay executing the > synchronous request and 2) force all requests to be synchronous. > The current POSIX behavior of ensuring all synchronous requests are > written to stable storage would remain the default. > > 2. Background > > Currently ZFS have no official control over synchronous behaviour. > However, unfortunately, a zfs module switch (zil_disable) has been > fairly well publicized to disable the ZIL, the code which enforces > synchronous requests. This is a global kernel variable and so affects > all file systems within all pools. > > It should be noted that ZFS pools are always consistent. That is, the > intent log is not required for pool integrity. This is due to the > transactional behavior of ZFS and in particular its transaction > group commit (txg) model. What the ZIL does is simply to ensure that > synchronous write requests are committed to stable storage prior to > returning from the system call. > > There are reasonable cases where the administrator understands the > consequences of disabling synchronous behavior. For example, if the > system crashes they start again from scratch. This is quicker > than enabling synchronous behavior. > > There are also reasons an administrator may want to enable synchronous > behavior for all writes. This might help debug an issue where > synchronous writes are needed. > > 3. Proposal > > The options and semantics proposed for a new zfs dataset property: > > sync=default > This is the default option. Synchronous file system transactions > (fsync, O_DSYNC, O_SYNC, etc) are written out (to the intent log) > and then secondly all devices written are flushed to ensure > the data is stable (not cached by device controllers). > > sync=always > For the ultra-cautious, every file system transaction would > be written and flushed to stable storage by system call return. > This obviously has a big performance penalty. > It's possible that sync=always could be used by NFSD become helpful for linux client behavior. So I would say : This obviously has a significant performance impact. More below > sync=disabled > Synchronous requests are disabled. File system transactions > only commit to stable storage on the next DMU transaction group > commit which can be many seconds. This option will give the > highest performance. However, it is very dangerous as ZFS > would be ignoring the synchronous transaction demands of > applications such as databases or NFS. Administrators should > only use this when these risks are understood. > > The property can be set when the dataset is created, or dynamically, > and will take effect immediately. To change the property, an > administrator can use the standard 'zfs' command. For example: > > # zfs create -o sync=disabled whirlpool/milek > # zfs set sync=always whirlpool/perrin > > The current value of 'sync' can be retrieved in the usual manner > with 'zfs get sync' or 'zfs list -o sync'. The sync property is not > inherited from parent datasets. > > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > ON > 6.5. ARC review type: FastTrack > 6.6. ARC Exposure: open > A zvol can accept DKIOCSETWCE which has similar effect and counter effect. We should be specifiying what the interactions will be. I guess that the non default dataset sync flag will render DKIOCSETWCE a noop. Will the sync flag impact the return value of a DKIOCGETWCE ? -r From milek@task.gda.pl Fri Apr 2 01:58:40 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o328weqv001645 for ; Fri, 2 Apr 2010 01:58:40 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o328we7U027781 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 2 Apr 2010 01:58:40 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0800003RLSLT00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Fri, 02 Apr 2010 01:58:40 -0700 (PDT) Received: from sca-ea-mail-3.sun.com ([192.18.43.21]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0800LD8RLRAP10@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Fri, 02 Apr 2010 01:58:39 -0700 (PDT) Received: from relay44i.sun.com ([192.5.209.118]) by sca-ea-mail-3.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o328thU8023412 for ; Fri, 02 Apr 2010 08:58:39 +0000 (GMT) Received: from mms49es.mms.us.syntegra.com ([160.41.221.232] [160.41.221.232]) by relay44i.sun.com with ESMTP id BT-MMP-3477959; Fri, 02 Apr 2010 08:58:20 +0000 (Z) Received: from relay42i.sun.com (relay42i.sun.com [192.5.209.72]) by mms49es.mms.us.syntegra.com with ESMTP id BT-MMP-68477508; Fri, 02 Apr 2010 08:58:20 +0000 (Z) Received: from bojer.task.gda.pl ([153.19.250.250] [153.19.250.250]) by relay4i.sun.com with ESMTP id BT-MMP-6595955; Fri, 02 Apr 2010 08:58:19 +0000 (Z) Received: from localhost (localhost [127.0.0.1]) by bojer.task.gda.pl (Postfix) with ESMTP id 38DC4BC05; Fri, 02 Apr 2010 10:58:19 +0200 (CEST) Received: from bojer.task.gda.pl ([127.0.0.1]) by localhost (bojer.task.gda.pl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CLMISOciOmOY; Fri, 02 Apr 2010 10:58:08 +0200 (CEST) Received: from [IPv6:::1] (rudy.mif.pg.gda.pl [153.19.42.16]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by bojer.task.gda.pl (Postfix) with ESMTP id AC58ABC03; Fri, 02 Apr 2010 10:58:07 +0200 (CEST) Date: Fri, 02 Apr 2010 09:58:06 +0100 From: Robert Milkowski Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: To: Jeff Bonwick Cc: Neil Perrin , PSARC-ext@sun.com, zfs-team@sun.com Message-id: <4BB5B19E.2040005@task.gda.pl> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Virus-Scanned: amavisd-new at task.gda.pl X-Antispam: No, score=-0.2/5.0, scanned in 0.075sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> <4BB4D7B2.8070307@oracle.com> <4BB4D8ED.4000402@Oracle.COM> <81E2698C-FB12-47F0-9A45-91959AF3B85E@oracle.com> <4BB57AB6.8040300@oracle.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.5) Gecko/20100117 Lightning/1.0b1 Thunderbird/3.0 Status: RO Content-Length: 2305 but with current implementation when sync=disable/barrier fsync() is basically a no-op. The exception is when a sync of all file systems or a file system is requested (sync())- it will be honored regardless of the sync= setting. Unless you meant something else. -- Robert Milkowski http://milek.blogspot.com On 02/04/2010 06:56, Jeff Bonwick wrote: > Yeah, I'm just suggesting "barrier" as a name rather than "disable". > It's a more accurate description of the semantic ZFS provides > when zil_commit() is a no-op. > > Jeff > > On Apr 1, 2010, at 10:03 PM, Neil Perrin wrote: > >> On 04/01/10 19:31, Jeff Bonwick wrote: >>> Yes. It should be inherited. >> >> Ok, that seems to be the consensus and I'm also fine with it. >> >> >>> >>> For the property values, I'd prefer: >>> >>> "standard" - we honor the standard (posix) semantics, without >>> saying posix. >> >> - Seems a reasonable name. >> >>> >>> "always" - make every system call synchronous. >>> >>> "barrier" - treat fsync() only as a write barrier, which ensures >>> that all previous >>> writes/updates will be on stable storage before any subsequent updates. >> >> Are there cases where an admin might want an applications fsyncs to >> be honored but >> O_DSYNC/O_SYNC/O_RSYNC to be ignored? >> >> Are you suggesting this new barrier option replaces disable or is in >> addition to it? >> I think we still need to provide a disable option that is equivalent >> to zil_disable >> and ignores all sync operation including fsync. >> >>> >>> Jeff >>> >>> On Apr 1, 2010, at 10:33 AM, Darren J Moffat wrote: >>> >>>> On 01/04/2010 18:28, Neil Perrin wrote: >>>>> We've flip-flopped on whether it should be inherited. It's currently >>>>> coded as inherited, and I know Robert believes it should stay that >>>>> way. >>>>> Anyway, it was generally felt by the zfs group that sync=disabled >>>>> was sufficiently dangerous to require explicit setting on each >>>>> dataset. >>>>> I would be ok with it being inherited. >>>> >>>> I agree that it is dangerous but I'm not sure that the change in >>>> semantics means it shouldn't be inherited. On the other hand I >>>> can't think of any other equivalently "dangerous" (to applications >>>> view of the world) per dataset property. >>>> >>>> -- >>>> Darren J Moffat >>> >> > > From rmilkowski@task.gda.pl Fri Apr 2 02:12:22 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o329CMAr001832 for ; Fri, 2 Apr 2010 02:12:22 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o329CMbS005206 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 2 Apr 2010 02:12:22 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0800I0FS8MRX00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Fri, 02 Apr 2010 02:12:22 -0700 (PDT) Received: from sca-ea-mail-3.sun.com ([192.18.43.21]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L08000FHS8LALE0@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Fri, 02 Apr 2010 02:12:21 -0700 (PDT) Received: from relay42i.sun.com ([192.5.209.72]) by sca-ea-mail-3.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o329CLlU027553 for ; Fri, 02 Apr 2010 09:12:21 +0000 (GMT) Received: from mmp41es.mmp.us.syntegra.com ([160.41.221.10] [160.41.221.10]) by relay42i.sun.com with ESMTP id BT-MMP-3499173; Fri, 02 Apr 2010 09:11:57 +0000 (Z) Received: from relay44i.sun.com (relay44i.sun.com [192.5.209.118]) by mmp41es.mmp.us.syntegra.com with ESMTP id BT-MMP-70335878; Fri, 02 Apr 2010 09:11:57 +0000 (Z) Received: from bojer.task.gda.pl ([153.19.250.250] [153.19.250.250]) by relay4i.sun.com with ESMTP id BT-MMP-12329442; Fri, 02 Apr 2010 09:11:57 +0000 (Z) Received: from localhost (localhost [127.0.0.1]) by bojer.task.gda.pl (Postfix) with ESMTP id 8AC18BC05; Fri, 02 Apr 2010 11:11:56 +0200 (CEST) Received: from bojer.task.gda.pl ([127.0.0.1]) by localhost (bojer.task.gda.pl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UMgSBWc3VM+G; Fri, 02 Apr 2010 11:11:46 +0200 (CEST) Received: from [IPv6:::1] (rudy.mif.pg.gda.pl [153.19.42.16]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by bojer.task.gda.pl (Postfix) with ESMTP id BB01DBC03; Fri, 02 Apr 2010 11:11:45 +0200 (CEST) Date: Fri, 02 Apr 2010 10:11:44 +0100 From: Robert Milkowski Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: To: Jeff Bonwick Cc: Neil Perrin , PSARC-ext@sun.com, zfs-team@sun.com Message-id: <4BB5B4D0.8030704@task.gda.pl> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Virus-Scanned: amavisd-new at task.gda.pl X-Antispam: No, score=0.0/5.0, scanned in 0.249sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> <4BB4D7B2.8070307@oracle.com> <4BB4D8ED.4000402@Oracle.COM> <81E2698C-FB12-47F0-9A45-91959AF3B85E@oracle.com> <4BB57AB6.8040300@oracle.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.5) Gecko/20100117 Lightning/1.0b1 Thunderbird/3.0 Status: RO Content-Length: 266 but with current implementation when sync=disable/barrier fsync() is basically a no-op. The exception is when a sync of all file systems or a file system is requested (sync())- it will be honored regardless of the sync= setting. Unless you meant something else. From casper@holland.sun.com Fri Apr 2 02:15:29 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o329FTmW001855 for ; Fri, 2 Apr 2010 02:15:29 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o329FSmJ012938; Fri, 2 Apr 2010 02:15:29 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L080080BSDSYF00@brm-avmta-1.central.sun.com>; Fri, 02 Apr 2010 03:15:28 -0600 (MDT) Received: from dm-holland-01.uk.sun.com ([129.156.101.192]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0800766SDRM210@brm-avmta-1.central.sun.com>; Fri, 02 Apr 2010 03:15:28 -0600 (MDT) Received: from holland (room101.Holland.Sun.COM [10.16.117.40]) by dm-holland-01.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o329FQef024371; Fri, 02 Apr 2010 10:15:26 +0100 (BST) Date: Fri, 02 Apr 2010 11:15:26 +0200 From: Casper.Dik@sun.com Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BB4C1C2.9050503@oracle.com> Sender: casper@holland.sun.com To: Andrew Gabriel Cc: Tim Haley , PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <201004020915.o329FQef024371@dm-holland-01.uk.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <4BB4BB25.6080607@sun.com> <4BB4C1C2.9050503@oracle.com> Status: RO Content-Length: 980 >Tim Haley wrote: >> I am sponsoring the following fast-track for Robert Milkowski and Neil >> Perrin. It introduces a new dataset property for controlling >> synchronous behavior. The case requests micro/patch binding. > >Thanks for this. > >One nit... >> The options and semantics proposed for a new zfs dataset property: >> >> sync=default >> This is the default option. Synchronous file system transactions >> (fsync, O_DSYNC, O_SYNC, etc) are written out (to the intent log) >> and then secondly all devices written are flushed to ensure >> the data is stable (not cached by device controllers). > >The naming of this one seems strange. >I would have had something like sync=posix, and document that sync=posix >is the default. (Maybe posix isn't the right term.) sync=default doesn't >actually tell you what the behavior is. I agree; you can never change the default either :-) I propose "sync=standard"; posix sounds so dated. Casper From casper@holland.sun.com Fri Apr 2 02:32:37 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o329WbEJ001912 for ; Fri, 2 Apr 2010 02:32:37 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o329Wbe6014184; Fri, 2 Apr 2010 02:32:37 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0800A07T6DPW00@brm-avmta-1.central.sun.com>; Fri, 02 Apr 2010 03:32:37 -0600 (MDT) Received: from dm-holland-01.uk.sun.com ([129.156.101.192]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L08007N3T6CM220@brm-avmta-1.central.sun.com>; Fri, 02 Apr 2010 03:32:36 -0600 (MDT) Received: from holland (room101.Holland.Sun.COM [10.16.117.40]) by dm-holland-01.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o329WYQd027438; Fri, 02 Apr 2010 10:32:34 +0100 (BST) Date: Fri, 02 Apr 2010 11:32:34 +0200 From: Casper.Dik@sun.com Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <81E2698C-FB12-47F0-9A45-91959AF3B85E@oracle.com> Sender: casper@holland.sun.com To: Jeff Bonwick Cc: Darren J Moffat , Neil Perrin , PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <201004020932.o329WYQd027438@dm-holland-01.uk.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> <4BB4D7B2.8070307@oracle.com> <4BB4D8ED.4000402@Oracle.COM> <81E2698C-FB12-47F0-9A45-91959AF3B85E@oracle.com> Status: RO Content-Length: 524 >Yes. It should be inherited. > >For the property values, I'd prefer: > >"standard" - we honor the standard (posix) semantics, without saying >posix. > >"always" - make every system call synchronous. > >"barrier" - treat fsync() only as a write barrier, which ensures that >all previous >writes/updates will be on stable storage before any subsequent updates. +1. I like barrier as it clearly explains what happens. Is this actually what happens now when you disable the ZIL or is this a new feature? Casper From rmilkowski@task.gda.pl Fri Apr 2 02:38:56 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o329cuFs001928 for ; Fri, 2 Apr 2010 02:38:56 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o329csjX024399; Fri, 2 Apr 2010 03:38:55 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0800C0JTGUAL00@nwk-avmta-1.sfbay.Sun.COM>; Fri, 02 Apr 2010 02:38:54 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0800LFJTGUAP90@nwk-avmta-1.sfbay.Sun.COM>; Fri, 02 Apr 2010 02:38:54 -0700 (PDT) Received: from relay15i.sun.com (ip125.net129179-4.block1.us.syntegra.com [129.179.4.125]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o329XkqK024686; Fri, 02 Apr 2010 09:38:54 +0000 (GMT) Received: from mmp13es.mmp.us.syntegra.com ([160.41.208.13] [160.41.208.13]) by relay15i.sun.com with ESMTP id BT-MMP-3537745; Fri, 02 Apr 2010 09:38:54 +0000 (Z) Received: from relay15i.sun.com (relay15i.sun.com [129.179.4.125]) by mmp13es.mmp.us.syntegra.com with ESMTP id BT-MMP-104256541; Fri, 02 Apr 2010 09:38:53 +0000 (Z) Received: from bojer.task.gda.pl ([153.19.250.250] [153.19.250.250]) by relay1i.sun.com with ESMTP id BT-MMP-29067965; Fri, 02 Apr 2010 09:38:53 +0000 (Z) Received: from localhost (localhost [127.0.0.1]) by bojer.task.gda.pl (Postfix) with ESMTP id BDD7DBC08; Fri, 02 Apr 2010 11:38:52 +0200 (CEST) Received: from bojer.task.gda.pl ([127.0.0.1]) by localhost (bojer.task.gda.pl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BfnEaBdSQxgq; Fri, 02 Apr 2010 11:38:42 +0200 (CEST) Received: from [IPv6:::1] (rudy.mif.pg.gda.pl [153.19.42.16]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by bojer.task.gda.pl (Postfix) with ESMTP id 97E06BC03; Fri, 02 Apr 2010 11:38:41 +0200 (CEST) Date: Fri, 02 Apr 2010 10:38:40 +0100 From: Robert Milkowski Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <19381.42000.633475.2548@gargle.gargle.HOWL> To: Roch Cc: Tim Haley , PSARC-ext@sun.com, zfs-team@sun.com Message-id: <4BB5BB20.8060607@task.gda.pl> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Virus-Scanned: amavisd-new at task.gda.pl X-Antispam: No, score=0.0/5.0, scanned in 0.072sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <4BB4BB25.6080607@sun.com> <19381.42000.633475.2548@gargle.gargle.HOWL> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.5) Gecko/20100117 Lightning/1.0b1 Thunderbird/3.0 Status: RO Content-Length: 2143 On 02/04/2010 09:00, Roch wrote: > > sync=disabled > > Synchronous requests are disabled. File system transactions > > only commit to stable storage on the next DMU transaction group > > commit which can be many seconds. This option will give the > > highest performance. However, it is very dangerous as ZFS > > would be ignoring the synchronous transaction demands of > > applications such as databases or NFS. Administrators should > > only use this when these risks are understood. > > > > The property can be set when the dataset is created, or dynamically, > > and will take effect immediately. To change the property, an > > administrator can use the standard 'zfs' command. For example: > > > > # zfs create -o sync=disabled whirlpool/milek > > # zfs set sync=always whirlpool/perrin > > > > The current value of 'sync' can be retrieved in the usual manner > > with 'zfs get sync' or 'zfs list -o sync'. The sync property is not > > inherited from parent datasets. > > > > 6. Resources and Schedule > > 6.4. Steering Committee requested information > > 6.4.1. Consolidation C-team Name: > > ON > > 6.5. ARC review type: FastTrack > > 6.6. ARC Exposure: open > > > > > A zvol can accept DKIOCSETWCE which has similar effect and > counter effect. We should be specifiying what the > interactions will be. > > if sync=default wce is obeyed else wce is not obeyed however the moment a sync is set back to the default value for a given zvol the WCE kicks in. Is DKIOSETWCE documented somewhere? > I guess that the non default dataset sync flag will render > DKIOCSETWCE a noop. > > yes, > Will the sync flag impact the return value of a DKIOCGETWCE ? > No, it won't. GETWCE will return the current WCE setting regardless of the sync value. SETWCE will also set its value regardless of the sync value. From API point of view manipulating WCE and sync properties are orthogonal. -- Robert Milkowski http://milek.blogspot.com From garrett.damore@oracle.com Fri Apr 2 08:52:23 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o32FqN6I007030 for ; Fri, 2 Apr 2010 08:52:23 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o32FqLCZ027635; Fri, 2 Apr 2010 10:52:21 -0500 (CDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0900609AR9VS00@brm-avmta-1.central.sun.com>; Fri, 02 Apr 2010 09:52:21 -0600 (MDT) Received: from sca-ea-mail-1.sun.com ([192.18.43.24]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L090084HAR8LKB0@brm-avmta-1.central.sun.com>; Fri, 02 Apr 2010 09:52:21 -0600 (MDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by sca-ea-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o32FqKdm009564; Fri, 02 Apr 2010 15:52:20 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o32FqGVf032316; Fri, 02 Apr 2010 15:52:16 +0000 (GMT) Received: from abhmt017.oracle.com by acsmt355.oracle.com with ESMTP id 133421061270223485; Fri, 02 Apr 2010 08:51:25 -0700 Received: from [10.7.251.172] (/10.7.251.172) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 02 Apr 2010 08:51:24 -0700 Date: Fri, 02 Apr 2010 08:51:23 -0700 From: "Garrett D'Amore" Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <201004020932.o329WYQd027438@dm-holland-01.uk.sun.com> To: Casper.Dik@sun.com Cc: Jeff Bonwick , Darren J Moffat , Neil Perrin , PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <4BB6127B.5080704@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090207.4BB612B1.017B:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> <4BB4D7B2.8070307@oracle.com> <4BB4D8ED.4000402@Oracle.COM> <81E2698C-FB12-47F0-9A45-91959AF3B85E@oracle.com> <201004020932.o329WYQd027438@dm-holland-01.uk.sun.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.7) Gecko/20100131 Lightning/1.0b1 Thunderbird/3.0.1 Status: RO Content-Length: 1027 On 04/ 2/10 02:32 AM, Casper.Dik@sun.com wrote: > >> Yes. It should be inherited. >> >> For the property values, I'd prefer: >> >> "standard" - we honor the standard (posix) semantics, without saying >> posix. >> >> "always" - make every system call synchronous. >> >> "barrier" - treat fsync() only as a write barrier, which ensures that >> all previous >> writes/updates will be on stable storage before any subsequent updates. >> > > +1. I like barrier as it clearly explains what happens. Is this > actually what happens now when you disable the ZIL or is this a new > feature? > > Casper > > I like this too. IMO, anyone messing around with this tunable ought to understand what a barrier is. If they don't, then they probably shouldn't be messing here. I've not reviewed the case in its entirety, so I'm explicitly not giving this case a +1 to defer to other filesystem experts. If a +1 is required from someone else, let me know and I'll take the time to review it properly. - Garrett From neil.perrin@oracle.com Fri Apr 2 09:49:07 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o32Gn7TO008506 for ; Fri, 2 Apr 2010 09:49:07 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o32Gn6DW006364; Fri, 2 Apr 2010 09:49:07 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0900D0FDDUDU00@brm-avmta-1.central.sun.com>; Fri, 02 Apr 2010 10:49:06 -0600 (MDT) Received: from sca-ea-mail-4.sun.com ([192.18.43.22]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L09008LNDDTLYD0@brm-avmta-1.central.sun.com>; Fri, 02 Apr 2010 10:49:06 -0600 (MDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by sca-ea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o32Gn5ES022484; Fri, 02 Apr 2010 16:49:05 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o32FZ4oJ028218; Fri, 02 Apr 2010 16:48:59 +0000 (GMT) Received: from abhmt010.oracle.com by acsmt354.oracle.com with ESMTP id 142095501270226832; Fri, 02 Apr 2010 09:47:12 -0700 Received: from [129.147.9.21] (/129.147.9.21) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 02 Apr 2010 09:47:12 -0700 Date: Fri, 02 Apr 2010 10:47:11 -0600 From: Neil Perrin Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: To: Jeff Bonwick Cc: PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <4BB61F8F.6090701@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090203.4BB61FFF.00FB:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> <4BB4D7B2.8070307@oracle.com> <4BB4D8ED.4000402@Oracle.COM> <81E2698C-FB12-47F0-9A45-91959AF3B85E@oracle.com> <4BB57AB6.8040300@oracle.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090910) Status: RO Content-Length: 2192 Jeff, We also ignore fsyncs when the zil is disabled, so there is no barrier. That was why I thought you were proposing a new option. I think disabled reflects the zil_disable semantics accurately. Neil. On 04/01/10 23:56, Jeff Bonwick wrote: > Yeah, I'm just suggesting "barrier" as a name rather than "disable". > It's a more accurate description of the semantic ZFS provides > when zil_commit() is a no-op. > > Jeff > > On Apr 1, 2010, at 10:03 PM, Neil Perrin wrote: > >> On 04/01/10 19:31, Jeff Bonwick wrote: >>> Yes. It should be inherited. >> >> Ok, that seems to be the consensus and I'm also fine with it. >> >> >>> >>> For the property values, I'd prefer: >>> >>> "standard" - we honor the standard (posix) semantics, without >>> saying posix. >> >> - Seems a reasonable name. >> >>> >>> "always" - make every system call synchronous. >>> >>> "barrier" - treat fsync() only as a write barrier, which ensures >>> that all previous >>> writes/updates will be on stable storage before any subsequent updates. >> >> Are there cases where an admin might want an applications fsyncs to >> be honored but >> O_DSYNC/O_SYNC/O_RSYNC to be ignored? >> >> Are you suggesting this new barrier option replaces disable or is in >> addition to it? >> I think we still need to provide a disable option that is equivalent >> to zil_disable >> and ignores all sync operation including fsync. >> >>> >>> Jeff >>> >>> On Apr 1, 2010, at 10:33 AM, Darren J Moffat wrote: >>> >>>> On 01/04/2010 18:28, Neil Perrin wrote: >>>>> We've flip-flopped on whether it should be inherited. It's currently >>>>> coded as inherited, and I know Robert believes it should stay that >>>>> way. >>>>> Anyway, it was generally felt by the zfs group that sync=disabled >>>>> was sufficiently dangerous to require explicit setting on each >>>>> dataset. >>>>> I would be ok with it being inherited. >>>> >>>> I agree that it is dangerous but I'm not sure that the change in >>>> semantics means it shouldn't be inherited. On the other hand I >>>> can't think of any other equivalently "dangerous" (to applications >>>> view of the world) per dataset property. >>>> >>>> -- >>>> Darren J Moffat >>> >> > From jeff.bonwick@oracle.com Fri Apr 2 16:07:22 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o32N7MEB015503 for ; Fri, 2 Apr 2010 16:07:22 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o32N7MEi001257; Fri, 2 Apr 2010 16:07:22 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0900F09UW9UM00@nwk-avmta-1.sfbay.Sun.COM>; Fri, 02 Apr 2010 16:07:21 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0900FDLUW7G6D0@nwk-avmta-1.sfbay.Sun.COM>; Fri, 02 Apr 2010 16:07:19 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o32N7JPk001008; Fri, 02 Apr 2010 23:07:19 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o32FwNBn019389; Fri, 02 Apr 2010 23:07:15 +0000 (GMT) Received: from abhmt015.oracle.com by acsmt354.oracle.com with ESMTP id 134288021270249570; Fri, 02 Apr 2010 16:06:10 -0700 Received: from punchin-client-10-7-250-23.sfbay.sun.com (/10.7.250.23) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 02 Apr 2010 16:06:09 -0700 Date: Fri, 02 Apr 2010 16:06:13 -0700 From: Jeff Bonwick Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BB61F8F.6090701@oracle.com> To: Neil Perrin Cc: PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: MIME-version: 1.0 X-Mailer: Apple Mail (2.1078) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090209.4BB678A5.0047:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> <4BB4D7B2.8070307@oracle.com> <4BB4D8ED.4000402@Oracle.COM> <81E2698C-FB12-47F0-9A45-91959AF3B85E@oracle.com> <4BB57AB6.8040300@oracle.com> <4BB61F8F.6090701@oracle.com> Status: RO Content-Length: 2594 What I mean is that the barrier semantic is implicit even with no ZIL at all. In ZFS, if event A happens before event B, and you lose power, then what you'll see on disk is either nothing, A, or both A and B. Never just B. It is impossible for us not to have at least barrier semantics. Jeff On Apr 2, 2010, at Friday, April 2 9:47 AM, Neil Perrin wrote: > Jeff, > > We also ignore fsyncs when the zil is disabled, so there is no barrier. > That was why I thought you were proposing a new option. > I think disabled reflects the zil_disable semantics accurately. > > Neil. > > On 04/01/10 23:56, Jeff Bonwick wrote: >> Yeah, I'm just suggesting "barrier" as a name rather than "disable". >> It's a more accurate description of the semantic ZFS provides >> when zil_commit() is a no-op. >> >> Jeff >> >> On Apr 1, 2010, at 10:03 PM, Neil Perrin wrote: >> >>> On 04/01/10 19:31, Jeff Bonwick wrote: >>>> Yes. It should be inherited. >>> >>> Ok, that seems to be the consensus and I'm also fine with it. >>> >>> >>>> >>>> For the property values, I'd prefer: >>>> >>>> "standard" - we honor the standard (posix) semantics, without saying posix. >>> >>> - Seems a reasonable name. >>> >>>> >>>> "always" - make every system call synchronous. >>>> >>>> "barrier" - treat fsync() only as a write barrier, which ensures that all previous >>>> writes/updates will be on stable storage before any subsequent updates. >>> >>> Are there cases where an admin might want an applications fsyncs to be honored but >>> O_DSYNC/O_SYNC/O_RSYNC to be ignored? >>> >>> Are you suggesting this new barrier option replaces disable or is in addition to it? >>> I think we still need to provide a disable option that is equivalent to zil_disable >>> and ignores all sync operation including fsync. >>> >>>> >>>> Jeff >>>> >>>> On Apr 1, 2010, at 10:33 AM, Darren J Moffat wrote: >>>> >>>>> On 01/04/2010 18:28, Neil Perrin wrote: >>>>>> We've flip-flopped on whether it should be inherited. It's currently >>>>>> coded as inherited, and I know Robert believes it should stay that way. >>>>>> Anyway, it was generally felt by the zfs group that sync=disabled >>>>>> was sufficiently dangerous to require explicit setting on each dataset. >>>>>> I would be ok with it being inherited. >>>>> >>>>> I agree that it is dangerous but I'm not sure that the change in semantics means it shouldn't be inherited. On the other hand I can't think of any other equivalently "dangerous" (to applications view of the world) per dataset property. >>>>> >>>>> -- >>>>> Darren J Moffat >>>> >>> >> > From jeff.bonwick@oracle.com Fri Apr 2 16:08:24 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o32N8NPJ015515 for ; Fri, 2 Apr 2010 16:08:24 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o32N8MTU010071; Fri, 2 Apr 2010 17:08:22 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0900707UXYNB00@nwk-avmta-2.sfbay.sun.com>; Fri, 02 Apr 2010 16:08:22 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0900GQZUXXQQ70@nwk-avmta-2.sfbay.sun.com>; Fri, 02 Apr 2010 16:08:21 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o32N8Kkg019529; Fri, 02 Apr 2010 23:08:20 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o32LFvC3023599; Fri, 02 Apr 2010 23:08:17 +0000 (GMT) Received: from abhmt015.oracle.com by acsmt355.oracle.com with ESMTP id 143099811270249622; Fri, 02 Apr 2010 16:07:02 -0700 Received: from punchin-client-10-7-250-23.sfbay.sun.com (/10.7.250.23) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 02 Apr 2010 16:07:01 -0700 Date: Fri, 02 Apr 2010 16:07:06 -0700 From: Jeff Bonwick Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <201004020932.o329WYQd027438@dm-holland-01.uk.sun.com> To: Casper.Dik@sun.com Cc: Darren J Moffat , Neil Perrin , PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <2AD439FA-D9E2-440B-BF91-8FE3D7E6C056@oracle.com> MIME-version: 1.0 X-Mailer: Apple Mail (2.1078) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090206.4BB678E3.0041:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> <4BB4D7B2.8070307@oracle.com> <4BB4D8ED.4000402@Oracle.COM> <81E2698C-FB12-47F0-9A45-91959AF3B85E@oracle.com> <201004020932.o329WYQd027438@dm-holland-01.uk.sun.com> Status: RO Content-Length: 732 This is how it actually works already. It's a side effect of ZFS being transactional. Jeff On Apr 2, 2010, at Friday, April 2 2:32 AM, Casper.Dik@sun.com wrote: > >> Yes. It should be inherited. >> >> For the property values, I'd prefer: >> >> "standard" - we honor the standard (posix) semantics, without saying >> posix. >> >> "always" - make every system call synchronous. >> >> "barrier" - treat fsync() only as a write barrier, which ensures that >> all previous >> writes/updates will be on stable storage before any subsequent updates. > > > +1. I like barrier as it clearly explains what happens. Is this > actually what happens now when you disable the ZIL or is this a new > feature? > > Casper > From milek@task.gda.pl Sat Apr 3 03:41:57 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o33AfuhN009706 for ; Sat, 3 Apr 2010 03:41:57 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o33AfuFe012469 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Sat, 3 Apr 2010 03:41:56 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0A00L05R1WCJ00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sat, 03 Apr 2010 04:41:56 -0600 (MDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0A00H0YR1WCC20@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sat, 03 Apr 2010 04:41:56 -0600 (MDT) Received: from relay41i.sun.com ([192.5.209.70]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o33AftMw002302 for ; Sat, 03 Apr 2010 10:41:55 +0000 (GMT) Received: from mms48es.mms.us.syntegra.com ([160.41.221.230] [160.41.221.230]) by relay41i.sun.com with ESMTP id BT-MMP-3603395; Sat, 03 Apr 2010 10:41:55 +0000 (Z) Received: from relay42i.sun.com (relay42i.sun.com [192.5.209.72]) by mms48es.mms.us.syntegra.com with ESMTP id BT-MMP-70284738; Sat, 03 Apr 2010 10:41:54 +0000 (Z) Received: from bojer.task.gda.pl ([153.19.250.250] [153.19.250.250]) by relay4i.sun.com with ESMTP id BT-MMP-8359252; Sat, 03 Apr 2010 10:41:54 +0000 (Z) Received: from localhost (localhost [127.0.0.1]) by bojer.task.gda.pl (Postfix) with ESMTP id 07EF9BC05; Sat, 03 Apr 2010 12:41:54 +0200 (CEST) Received: from bojer.task.gda.pl ([127.0.0.1]) by localhost (bojer.task.gda.pl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7zeutpuVOr04; Sat, 03 Apr 2010 12:41:43 +0200 (CEST) Received: from [IPv6:::1] (rudy.mif.pg.gda.pl [153.19.42.16]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by bojer.task.gda.pl (Postfix) with ESMTP id 0D488BC03; Sat, 03 Apr 2010 12:41:42 +0200 (CEST) Date: Sat, 03 Apr 2010 11:41:43 +0100 From: Robert Milkowski Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: To: Jeff Bonwick Cc: Neil Perrin , PSARC-ext@sun.com, zfs-team@sun.com Message-id: <4BB71B67.3030707@task.gda.pl> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Virus-Scanned: amavisd-new at task.gda.pl X-Antispam: No, score=0.0/5.0, scanned in 0.057sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> <4BB4D7B2.8070307@oracle.com> <4BB4D8ED.4000402@Oracle.COM> <81E2698C-FB12-47F0-9A45-91959AF3B85E@oracle.com> <4BB57AB6.8040300@oracle.com> <4BB61F8F.6090701@oracle.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.5) Gecko/20100117 Lightning/1.0b1 Thunderbird/3.0 Status: RO Content-Length: 940 On 03/04/2010 00:06, Jeff Bonwick wrote: > What I mean is that the barrier semantic is implicit even with no ZIL at all. > In ZFS, if event A happens before event B, and you lose power, then > what you'll see on disk is either nothing, A, or both A and B. Never just B. > It is impossible for us not to have at least barrier semantics. > > I'm not entirely sure but I think there might be an exception to this. A happens as an asynchronous write, then B happens as a synchronous write and is written to ZIL. Now before A is committed a server looses power, after reboot B will be committed to a pool. But A would never make it there. Or am I missing something? I think that sync=barrier would be confusing as it suggest that fsync() is synchronous while it isn't with this setting. However an extra explanation as the one above would be useful to be in the zfs/zpool man page. -- Robert Milkowski http://milek.blogspot.com From rmilkowski@task.gda.pl Sat Apr 3 04:03:59 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o33B3xiD010138 for ; Sat, 3 Apr 2010 04:03:59 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o33B3xRo015391 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Sat, 3 Apr 2010 06:03:59 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0A00H01S2MA800@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sat, 03 Apr 2010 04:03:58 -0700 (PDT) Received: from sca-ea-mail-2.sun.com ([192.18.43.25]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0A00K02S2MQR80@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sat, 03 Apr 2010 04:03:58 -0700 (PDT) Received: from relay43i.sun.com ([192.5.209.74]) by sca-ea-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o33AwB8q008728 for ; Sat, 03 Apr 2010 11:03:57 +0000 (GMT) Received: from mmp42es.mmp.us.syntegra.com ([160.41.221.11] [160.41.221.11]) by relay43i.sun.com with ESMTP id BT-MMP-475327; Sat, 03 Apr 2010 11:03:57 +0000 (Z) Received: from relay45i.sun.com (relay45i.sun.com [192.5.209.94]) by mmp42es.mmp.us.syntegra.com with ESMTP id BT-MMP-70706630; Sat, 03 Apr 2010 11:03:57 +0000 (Z) Received: from bojer.task.gda.pl ([153.19.250.250] [153.19.250.250]) by relay4i.sun.com with ESMTP id BT-MMP-23637389; Sat, 03 Apr 2010 11:03:57 +0000 (Z) Received: from localhost (localhost [127.0.0.1]) by bojer.task.gda.pl (Postfix) with ESMTP id C23AFBC05; Sat, 03 Apr 2010 13:03:56 +0200 (CEST) Received: from bojer.task.gda.pl ([127.0.0.1]) by localhost (bojer.task.gda.pl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5jU5Xa9GnYXB; Sat, 03 Apr 2010 13:03:55 +0200 (CEST) Received: from [IPv6:::1] (rudy.mif.pg.gda.pl [153.19.42.16]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by bojer.task.gda.pl (Postfix) with ESMTP id E8CB7BC03; Sat, 03 Apr 2010 13:03:54 +0200 (CEST) Date: Sat, 03 Apr 2010 12:03:55 +0100 From: Robert Milkowski Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: To: Jeff Bonwick Cc: Neil Perrin , PSARC-ext@sun.com, zfs-team@sun.com Message-id: <4BB7209B.2010904@task.gda.pl> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Virus-Scanned: amavisd-new at task.gda.pl X-Antispam: No, score=0.0/5.0, scanned in 0.118sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> <4BB4D7B2.8070307@oracle.com> <4BB4D8ED.4000402@Oracle.COM> <81E2698C-FB12-47F0-9A45-91959AF3B85E@oracle.com> <4BB57AB6.8040300@oracle.com> <4BB61F8F.6090701@oracle.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.5) Gecko/20100117 Lightning/1.0b1 Thunderbird/3.0 Status: RO Content-Length: 887 On 03/04/2010 00:06, Jeff Bonwick wrote: > What I mean is that the barrier semantic is implicit even with no ZIL at all. > In ZFS, if event A happens before event B, and you lose power, then > what you'll see on disk is either nothing, A, or both A and B. Never just B. > It is impossible for us not to have at least barrier semantics. > > I'm not entirely sure but I think there might be an exception to this. A happens as an asynchronous write, then B happens as a synchronous write and is written to ZIL. Now before A is committed a server looses power, after reboot B will be committed to a pool. But A would never make it there. Or am I missing something? I think that sync=barrier would be confusing as it suggest that fsync() is synchronous while it isn't with this setting. However an extra explanation as the one above would be useful to be in the zfs/zpool man page. From jeff.bonwick@oracle.com Sat Apr 3 12:57:07 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o33Jv7As014617 for ; Sat, 3 Apr 2010 12:57:07 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o33Jv7V1024049; Sat, 3 Apr 2010 12:57:07 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0B00B0NGR7OB00@nwk-avmta-2.sfbay.sun.com>; Sat, 03 Apr 2010 12:57:07 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0B00AU5GR5ZI80@nwk-avmta-2.sfbay.sun.com>; Sat, 03 Apr 2010 12:57:06 -0700 (PDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o33Jv5Fs019655; Sat, 03 Apr 2010 19:57:05 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o33Jv34m027688; Sat, 03 Apr 2010 19:57:03 +0000 (GMT) Received: from abhmt001.oracle.com by acsmt353.oracle.com with ESMTP id 144545841270324600; Sat, 03 Apr 2010 12:56:40 -0700 Received: from [10.0.1.8] (/24.6.60.71) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sat, 03 Apr 2010 12:56:40 -0700 Date: Sat, 03 Apr 2010 12:56:39 -0700 From: Jeff Bonwick Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BB71B67.3030707@task.gda.pl> To: Robert Milkowski Cc: Neil Perrin , PSARC-ext@sun.com, zfs-team@sun.com Message-id: <761FCEA3-7E16-4E74-B9E4-28714092F33A@oracle.com> MIME-version: 1.0 X-Mailer: Apple Mail (2.936) Content-type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090208.4BB79D8F.0188:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <4BB4BE56.6010801@Oracle.COM> <4BB4D7B2.8070307@oracle.com> <4BB4D8ED.4000402@Oracle.COM> <81E2698C-FB12-47F0-9A45-91959AF3B85E@oracle.com> <4BB57AB6.8040300@oracle.com> <4BB61F8F.6090701@oracle.com> <4BB71B67.3030707@task.gda.pl> Status: RO Content-Length: 1175 That is correct -- but that can only happen when the ZIL is enabled. With ZIL disabled, we always preserve event ordering. Jeff On Apr 3, 2010, at 3:41 AM, Robert Milkowski wrote: > On 03/04/2010 00:06, Jeff Bonwick wrote: >> What I mean is that the barrier semantic is implicit even with no >> ZIL at all. >> In ZFS, if event A happens before event B, and you lose power, then >> what you'll see on disk is either nothing, A, or both A and B. >> Never just B. >> It is impossible for us not to have at least barrier semantics. >> >> > > I'm not entirely sure but I think there might be an exception to this. > > A happens as an asynchronous write, then B happens as a synchronous > write and is written to ZIL. > Now before A is committed a server looses power, after reboot B will > be committed to a pool. But A would never make it there. > > Or am I missing something? > > I think that sync=barrier would be confusing as it suggest that > fsync() is synchronous while it isn't with this setting. > > However an extra explanation as the one above would be useful to be > in the zfs/zpool man page. > > -- > Robert Milkowski > http://milek.blogspot.com > From Nicolas.Williams@sun.com Sat Apr 3 19:07:51 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o3427pU7018431 for ; Sat, 3 Apr 2010 19:07:51 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o3427pbj015874; Sat, 3 Apr 2010 19:07:51 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0B00H01XX3NB00@nwk-avmta-2.sfbay.sun.com>; Sat, 03 Apr 2010 19:07:51 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0B004RUXX2L460@nwk-avmta-2.sfbay.sun.com>; Sat, 03 Apr 2010 19:07:50 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o3427otg018466; Sun, 04 Apr 2010 02:07:50 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o33N3S8T006524; Sun, 04 Apr 2010 02:07:48 +0000 (GMT) Received: from abhmt016.oracle.com by acsmt353.oracle.com with ESMTP id 144762511270346772; Sat, 03 Apr 2010 19:06:12 -0700 Received: from Sun.COM (/129.153.128.104) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sat, 03 Apr 2010 19:06:12 -0700 Date: Sat, 03 Apr 2010 21:06:07 -0500 From: Nicolas Williams Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BB4BB25.6080607@sun.com> To: Tim Haley Cc: PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <20100404020607.GI4225@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090206.4BB7F474.004E,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> User-Agent: Mutt/1.5.20 (2010-03-02) Status: RO Content-Length: 581 It would be a bad idea to place Kerberos V5 replay caches on datasets with sync=barrier. Would it be possible to make /var/krb5/rcache a separate dataset with sync explicitly set (to prevent inherittance)? I'd be happy with an RFE for the installer to do this, or perhaps it could be done by a self-assembly SMF service on first boot (advice welcome). (Incidentally, I've a fix for 6794523 rcache could skip fsync(2)s ... that causes the rcache to use fsync() only in some cases while remaining secure. See the CR, but keep in mind that its comments are out of date.) Nico -- From Nicolas.Williams@sun.com Mon Apr 5 10:01:20 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o35H1Krl022527 for ; Mon, 5 Apr 2010 10:01:20 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o35H1I4Q024427; Mon, 5 Apr 2010 10:01:20 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0E00K11XY7B800@brm-avmta-1.central.sun.com>; Mon, 05 Apr 2010 11:01:19 -0600 (MDT) Received: from sca-ea-mail-3.sun.com ([192.18.43.21]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0E00CB9XY533D0@brm-avmta-1.central.sun.com>; Mon, 05 Apr 2010 11:01:17 -0600 (MDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by sca-ea-mail-3.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o35H1GT4018259; Mon, 05 Apr 2010 17:01:17 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o35FXuvZ029562; Mon, 05 Apr 2010 17:01:14 +0000 (GMT) Received: from abhmt003.oracle.com by acsmt355.oracle.com with ESMTP id 147474381270486805; Mon, 05 Apr 2010 10:00:05 -0700 Received: from Sun.COM (/129.153.128.104) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 05 Apr 2010 10:00:05 -0700 Date: Mon, 05 Apr 2010 11:59:56 -0500 From: Nicolas Williams Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <20100404020607.GI4225@Sun.COM> To: Tim Haley Cc: PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <20100405165956.GA7007@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090209.4BBA175B.007E,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <20100404020607.GI4225@Sun.COM> User-Agent: Mutt/1.5.20 (2010-03-02) Status: RO Content-Length: 558 On Sat, Apr 03, 2010 at 09:06:07PM -0500, Nicolas Williams wrote: > It would be a bad idea to place Kerberos V5 replay caches on datasets > with sync=barrier. Would it be possible to make /var/krb5/rcache a > separate dataset with sync explicitly set (to prevent inherittance)? > I'd be happy with an RFE for the installer to do this, or perhaps it > could be done by a self-assembly SMF service on first boot (advice > welcome). Will F. suggests that setting sync=barrier on /var should yield an unsupported system. I'd be happy with that too. Nico -- From sommerfeld@sun.com Mon Apr 5 15:40:59 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o35Mewpi028767 for ; Mon, 5 Apr 2010 15:40:58 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o35Mewbt015095; Mon, 5 Apr 2010 16:40:58 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0F00D01DO9Q400@nwk-avmta-2.sfbay.sun.com>; Mon, 05 Apr 2010 15:40:58 -0700 (PDT) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0F004IKDO9L890@nwk-avmta-2.sfbay.sun.com>; Mon, 05 Apr 2010 15:40:57 -0700 (PDT) Received: from thunk-west.local (dhcp-umpk17-109-54.SFBay.Sun.COM [129.146.109.54]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o35MeuF5018708; Mon, 05 Apr 2010 15:40:56 -0700 (PDT) Received: from [IPv6:::1] (thunk-west [IPv6:::1]) by thunk-west.local (8.14.4+Sun/8.14.4) with ESMTP id o35MeuRO009140; Mon, 05 Apr 2010 15:40:56 -0700 (PDT) Date: Mon, 05 Apr 2010 15:40:56 -0700 From: Bill Sommerfeld Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <20100405165956.GA7007@Sun.COM> To: Nicolas Williams Cc: Tim Haley , PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <4BBA66F8.6030203@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <4BB4BB25.6080607@sun.com> <20100404020607.GI4225@Sun.COM> <20100405165956.GA7007@Sun.COM> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.8) Gecko/20100315 Lightning/1.0b1 Thunderbird/3.0.3 Status: RO Content-Length: 1153 On 04/05/10 09:59, Nicolas Williams wrote: > Will F. suggests that setting sync=barrier on /var should yield an > unsupported system. I'd be happy with that too. In PSARC contexts, I generally prefer to avoid using words like "supported" and "unsupported", since they are statements about the likely future behavior of support personnel rather than direct promises about the behavior of the system. How about: "Setting sync=barrier on the currently active root or /var filesystem may result in out-of-spec behavior, application data loss and increased vulnerability to replay attacks". Another instance where sync=barrier could subtly cause trouble is for any SMTP listener (sendmail or alternative). SMTP requires that the receiver have the message in stable storage before sending the "ok" to the sender, so that in the event of a crash, messages get duplicated rather than dropped. But, on the other hand, it seems like it might be clever to temporarily set sync=barrier during package operations on a freshly-cloned alternate BE -- if the package operation doesn't complete, you have to torch the BE, anyway. - Bill From bill.sommerfeld@oracle.com Mon Apr 5 15:48:21 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o35MmLbL028921 for ; Mon, 5 Apr 2010 15:48:21 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o35MmJ20029691; Mon, 5 Apr 2010 17:48:20 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0F00E0BE0J7Q00@nwk-avmta-2.sfbay.sun.com>; Mon, 05 Apr 2010 15:48:19 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0F0040WE0IL5A0@nwk-avmta-2.sfbay.sun.com>; Mon, 05 Apr 2010 15:48:19 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o35MmIoB028484; Mon, 05 Apr 2010 22:48:18 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o35MRYL2013029; Mon, 05 Apr 2010 22:48:15 +0000 (GMT) Received: from abhmt001.oracle.com by acsmt354.oracle.com with ESMTP id 138341281270507632; Mon, 05 Apr 2010 15:47:12 -0700 Received: from [129.146.109.54] (/129.146.109.54) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 05 Apr 2010 15:47:12 -0700 Date: Mon, 05 Apr 2010 15:47:11 -0700 From: Bill Sommerfeld Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <20100405165956.GA7007@Sun.COM> To: Nicolas Williams Cc: Tim Haley , PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <4BBA686F.8060804@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090208.4BBA68B0.018C:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <20100404020607.GI4225@Sun.COM> <20100405165956.GA7007@Sun.COM> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.8) Gecko/20100315 Lightning/1.0b1 Thunderbird/3.0.3 Status: RO Content-Length: 1153 On 04/05/10 09:59, Nicolas Williams wrote: > Will F. suggests that setting sync=barrier on /var should yield an > unsupported system. I'd be happy with that too. In PSARC contexts, I generally prefer to avoid using words like "supported" and "unsupported", since they are statements about the likely future behavior of support personnel rather than direct promises about the behavior of the system. How about: "Setting sync=barrier on the currently active root or /var filesystem may result in out-of-spec behavior, application data loss and increased vulnerability to replay attacks". Another instance where sync=barrier could subtly cause trouble is for any SMTP listener (sendmail or alternative). SMTP requires that the receiver have the message in stable storage before sending the "ok" to the sender, so that in the event of a crash, messages get duplicated rather than dropped. But, on the other hand, it seems like it might be clever to temporarily set sync=barrier during package operations on a freshly-cloned alternate BE -- if the package operation doesn't complete, you have to torch the BE, anyway. - Bill From Nicolas.Williams@sun.com Mon Apr 5 15:52:00 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o35Mpxf0028987 for ; Mon, 5 Apr 2010 15:52:00 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o35MpwcT001706; Mon, 5 Apr 2010 17:51:59 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0F0040JE6NCY00@nwk-avmta-1.sfbay.Sun.COM>; Mon, 05 Apr 2010 15:51:59 -0700 (PDT) Received: from sca-ea-mail-1.sun.com ([192.18.43.24]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0F008W1E6LFB90@nwk-avmta-1.sfbay.Sun.COM>; Mon, 05 Apr 2010 15:51:58 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by sca-ea-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o35Mpv0e014760; Mon, 05 Apr 2010 22:51:57 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o35JSLoE017733; Mon, 05 Apr 2010 22:51:54 +0000 (GMT) Received: from abhmt005.oracle.com by acsmt353.oracle.com with ESMTP id 148400051270507894; Mon, 05 Apr 2010 15:51:34 -0700 Received: from Sun.COM (/129.153.128.104) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 05 Apr 2010 15:51:34 -0700 Date: Mon, 05 Apr 2010 17:51:29 -0500 From: Nicolas Williams Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BBA66F8.6030203@sun.com> To: Bill Sommerfeld Cc: Tim Haley , PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <20100405225129.GI4225@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090203.4BBA698B.016D,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <20100404020607.GI4225@Sun.COM> <20100405165956.GA7007@Sun.COM> <4BBA66F8.6030203@sun.com> User-Agent: Mutt/1.5.20 (2010-03-02) Status: RO Content-Length: 1746 On Mon, Apr 05, 2010 at 03:40:56PM -0700, Bill Sommerfeld wrote: > On 04/05/10 09:59, Nicolas Williams wrote: > >Will F. suggests that setting sync=barrier on /var should yield an > >unsupported system. I'd be happy with that too. > > In PSARC contexts, I generally prefer to avoid using words like > "supported" and "unsupported", since they are statements about the > likely future behavior of support personnel rather than direct > promises about the behavior of the system. > > How about: > > "Setting sync=barrier on the currently active root or /var > filesystem may result in out-of-spec behavior, application data loss > and increased vulnerability to replay attacks". Yes, I like that. Datasets are cheap, so I wouldn't mind an actual /var/krb5/rcache dataset. > Another instance where sync=barrier could subtly cause trouble is > for any SMTP listener (sendmail or alternative). SMTP requires that > the receiver have the message in stable storage before sending the > "ok" to the sender, so that in the event of a crash, messages get > duplicated rather than dropped. Yes. Here too we could use having separate datasets for sendmail (and freinds) from the rest of the system. It'd be very nice if POSIX had always had an fbarrier()/fsync() distinction. Alas, it does not. > But, on the other hand, it seems like it might be clever to > temporarily set sync=barrier during package operations on a > freshly-cloned alternate BE -- if the package operation doesn't > complete, you have to torch the BE, anyway. :) sync=barrier on home directories will also help, but you also get into the mail delivery problem in cases where you download e-mail onto local folders, immediately deleting the copies on the server. Nico -- From neil.perrin@oracle.com Tue Apr 6 16:00:45 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o36N0jab009353 for ; Tue, 6 Apr 2010 16:00:45 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o36N0j1R008192; Tue, 6 Apr 2010 16:00:45 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0H00E0J9986W00@nwk-avmta-2.sfbay.sun.com>; Tue, 06 Apr 2010 16:00:44 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0H002Y89983S80@nwk-avmta-2.sfbay.sun.com>; Tue, 06 Apr 2010 16:00:44 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o36N0idx014115; Tue, 06 Apr 2010 23:00:44 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o35JugYc009959; Tue, 06 Apr 2010 23:00:39 +0000 (GMT) Received: from abhmt003.oracle.com by acsmt355.oracle.com with ESMTP id 141298481270594783; Tue, 06 Apr 2010 15:59:43 -0700 Received: from [129.147.9.21] (/129.147.9.21) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 06 Apr 2010 15:59:43 -0700 Date: Tue, 06 Apr 2010 16:59:42 -0600 From: Neil Perrin Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BBA66F8.6030203@sun.com> To: Bill Sommerfeld Cc: PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <4BBBBCDE.2030208@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A0B020B.4BBBBD19.011D:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <20100404020607.GI4225@Sun.COM> <20100405165956.GA7007@Sun.COM> <4BBA66F8.6030203@sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090910) Status: RO Content-Length: 697 On 04/05/10 16:40, Bill Sommerfeld wrote: > On 04/05/10 09:59, Nicolas Williams wrote: >> Will F. suggests that setting sync=barrier on /var should yield an >> unsupported system. I'd be happy with that too. > > In PSARC contexts, I generally prefer to avoid using words like > "supported" and "unsupported", since they are statements about the > likely future behavior of support personnel rather than direct > promises about the behavior of the system. > > How about: > > "Setting sync=barrier on the currently active root or /var filesystem > may result in out-of-spec behavior, application data loss and > increased vulnerability to replay attacks". I like that wording as well. neil. From neil.perrin@oracle.com Tue Apr 6 16:49:14 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o36NnDrw009921 for ; Tue, 6 Apr 2010 16:49:14 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o36NnDtr009395; Tue, 6 Apr 2010 18:49:13 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0H00H01BI1IK00@nwk-avmta-2.sfbay.sun.com>; Tue, 06 Apr 2010 16:49:13 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0H002GWBI03CA0@nwk-avmta-2.sfbay.sun.com>; Tue, 06 Apr 2010 16:49:12 -0700 (PDT) Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o36NnCTK013586; Tue, 06 Apr 2010 23:49:12 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o36FxDN5020286; Tue, 06 Apr 2010 23:49:00 +0000 (GMT) Received: from abhmt004.oracle.com by acsmt353.oracle.com with ESMTP id 152213251270597729; Tue, 06 Apr 2010 16:48:49 -0700 Received: from [129.147.9.21] (/129.147.9.21) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 06 Apr 2010 16:48:49 -0700 Date: Tue, 06 Apr 2010 17:48:48 -0600 From: Neil Perrin Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BB4BB25.6080607@sun.com> To: PSARC-ext@sun.com Cc: zfs-team@sun.com, milek@task.gda.pl Message-id: <4BBBC860.1070800@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090202.4BBBC870.0036:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090910) Status: RO Content-Length: 4402 After reviewing the many comments on this case and after further discussion with the zfs engineers, we are making the following changes to the PSARC case: 1. The sync property will be inherited. 2. sync=default is replaced by sync=standard Note, sync=disabled will remain. 3. Additional text has been added to warn about setting sync=disabled on root or /var file systems. Here's the updated proposal: ------------------------------------ Template Version: @(#)sac_nextcase 1.70 03/30/10 SMI This information is Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. 1. Introduction 1.1. Project/Component Working Name: zil synchronicity 1.2. Name of Document Author/Supplier: Author: Neil Perrin 1.3 Date of This Document: 06 April, 2010 4. Technical Description I am sponsoring the following fast-track for Robert Milkowski and Neil Perrin. It introduces a new dataset property for controlling synchronous behavior. The case requests micro/patch binding. 1. Summary Provide administrators of zfs with the ability to control the behavior of synchronous requests (e.g. fsync, O_DSYNC). In particular, new capabilities are proposed to 1) delay executing the synchronous request and 2) force all requests to be synchronous. The current POSIX behavior of ensuring all synchronous requests are written to stable storage would remain the default. 2. Background Currently ZFS have no official control over synchronous behaviour. However, unfortunately, a zfs module switch (zil_disable) has been fairly well publicized to disable the ZIL, the code which enforces synchronous requests. This is a global kernel variable and so affects all file systems within all pools. It should be noted that ZFS pools are always consistent. That is, the intent log is not required for pool integrity. This is due to the transactional behavior of ZFS and in particular its transaction group commit (txg) model. What the ZIL does is simply to ensure that synchronous write requests are committed to stable storage prior to returning from the system call. There are reasonable cases where the administrator understands the consequences of disabling synchronous behavior. For example, if the system crashes they start again from scratch. This is quicker than enabling synchronous behavior. There are also reasons an administrator may want to enable synchronous behavior for all writes. This might help debug an issue where synchronous writes are needed. 3. Proposal The options and semantics proposed for a new zfs dataset property: sync=standard This is the default option. Synchronous file system transactions (fsync, O_DSYNC, O_SYNC, etc) are written out (to the intent log) and then secondly all devices written are flushed to ensure the data is stable (not cached by device controllers). sync=always For the ultra-cautious, every file system transaction would be written and flushed to stable storage by system call return. This obviously has a big performance penalty. sync=disabled Synchronous requests are disabled. File system transactions only commit to stable storage on the next DMU transaction group commit which can be many seconds. This option will give the highest performance. However, it is very dangerous as ZFS would be ignoring the synchronous transaction demands of applications such as databases or NFS. Setting sync=disabled on the currently active root or /var filesystem may result in out-of-spec behavior, application data loss and increased vulnerability to replay attacks Administrators should only use this when these risks are understood. The property can be set when the dataset is created, or dynamically, and will take effect immediately. To change the property, an administrator can use the standard 'zfs' command. For example: # zfs create -o sync=disabled whirlpool/milek # zfs set sync=always whirlpool/perrin The current value of 'sync' can be retrieved in the usual manner with 'zfs get sync' or 'zfs list -o sync'. The sync property is inherited from parent datasets. 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From Darren.Moffat@oracle.com Wed Apr 7 10:14:51 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o37HEpv4016210 for ; Wed, 7 Apr 2010 10:14:51 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o37HEntF001481; Wed, 7 Apr 2010 10:14:50 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0I00N0JNWQ4E00@brm-avmta-1.central.sun.com>; Wed, 07 Apr 2010 11:14:50 -0600 (MDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0I006GDNWQOLC0@brm-avmta-1.central.sun.com>; Wed, 07 Apr 2010 11:14:50 -0600 (MDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o37HEna0008580; Wed, 07 Apr 2010 17:14:49 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o36LwBnE022128; Wed, 07 Apr 2010 17:14:39 +0000 (GMT) Received: from abhmt018.oracle.com by acsmt354.oracle.com with ESMTP id 143554641270660477; Wed, 07 Apr 2010 10:14:37 -0700 Received: from [10.7.251.221] (/10.7.251.221) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 07 Apr 2010 10:14:36 -0700 Date: Wed, 07 Apr 2010 18:14:34 +0100 From: Darren J Moffat Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BBBC860.1070800@oracle.com> To: Neil Perrin Cc: PSARC-ext@sun.com, zfs-team@sun.com, milek@task.gda.pl Message-id: <4BBCBD7A.2040506@Oracle.COM> Organization: Oracle Solaris Security MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090203.4BBCBD7F.0123:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <4BBBC860.1070800@oracle.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.8) Gecko/20100315 Lightning/1.0b1 Thunderbird/3.0.3 Status: RO Content-Length: 504 On 07/04/2010 00:48, Neil Perrin wrote: > After reviewing the many comments on this case and after further discussion > with the zfs engineers, we are making the following changes to the PSARC > case: > > 1. The sync property will be inherited. > 2. sync=default is replaced by sync=standard > Note, sync=disabled will remain. > 3. Additional text has been added to warn about setting sync=disabled > on root or /var file systems. I'm happy with the updated proposal it gets my +1. -- Darren J Moffat From tim.haley@oracle.com Fri Apr 9 10:21:00 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o39HL0jM010654 for ; Fri, 9 Apr 2010 10:21:00 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o39HL08g010707; Fri, 9 Apr 2010 10:21:00 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0M00F05DJ04100@nwk-avmta-2.sfbay.sun.com>; Fri, 09 Apr 2010 10:21:00 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0M00D8DDIZW010@nwk-avmta-2.sfbay.sun.com>; Fri, 09 Apr 2010 10:21:00 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o39HKxHF010462; Fri, 09 Apr 2010 17:20:59 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o39EeNeM019754; Fri, 09 Apr 2010 17:20:56 +0000 (GMT) Received: from abhmt021.oracle.com by acsmt353.oracle.com with ESMTP id 149569431270833632; Fri, 09 Apr 2010 10:20:32 -0700 Received: from [172.20.25.27] (/10.85.25.27) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 09 Apr 2010 10:20:31 -0700 Date: Fri, 09 Apr 2010 11:20:31 -0600 From: Tim Haley Subject: Re: zil synchronicity [PSARC/2010/108 FastTrack timeout 04/08/2010] In-reply-to: <4BBCBD7A.2040506@Oracle.COM> To: PSARC-ext@sun.com Cc: zfs-team@sun.com, milek@task.gda.pl Message-id: <4BBF61DF.2030602@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090203.4BBF61F9.0056:SCFMA4539814,ss=1,fgs=0 References: <4BB4BB25.6080607@sun.com> <4BBBC860.1070800@oracle.com> <4BBCBD7A.2040506@Oracle.COM> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.8) Gecko/20100315 Thunderbird/3.0.3 Status: RO Content-Length: 101 The case has timed out and received the requisite +1. I am marking it as closed and approved. -tim