From gww@eng.sun.com Mon Oct 1 22:10:19 2007 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l925AJ7I013756 for ; Mon, 1 Oct 2007 22:10:19 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l9257AgO008121; Mon, 1 Oct 2007 22:07:12 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JP900K0PQ7ZYK00@brm-avmta-1.central.sun.com>; Mon, 01 Oct 2007 23:07:11 -0600 (MDT) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JP900HB7Q7YPZ10@brm-avmta-1.central.sun.com>; Mon, 01 Oct 2007 23:07:11 -0600 (MDT) Received: from marduk.eng.sun.com (marduk.SFBay.Sun.COM [129.146.108.224]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id l9257AuG030636; Mon, 01 Oct 2007 22:07:10 -0700 (PDT) Received: from marduk.eng.sun.com (localhost [127.0.0.1]) by marduk.eng.sun.com (8.13.6+Sun/8.12.11) with ESMTP id l9258sGE029999; Mon, 01 Oct 2007 22:08:54 -0700 (PDT) Received: (from gww@localhost) by marduk.eng.sun.com (8.13.6+Sun/8.12.11/Submit) id l9258srj029998; Mon, 01 Oct 2007 22:08:54 -0700 (PDT) Date: Mon, 01 Oct 2007 22:08:54 -0700 (PDT) From: Gary Winiger Subject: Re: zpool failmode property [PSARC/2007/567 FastTrack timeout 10/08/2007] To: psarc-ext@sun.com, maybee@moosylvania.central.sun.com Cc: zfs-eng@sun.com Message-id: <200710020508.l9258srj029998@marduk.eng.sun.com> Content-transfer-encoding: 7BIT X-Sun-Charset: US-ASCII X-PMX-Version: 5.2.0.264296 Status: RO Content-Length: 4006 This never made it to the case directory, as I still have it, I'm resending. Gary.. ======= > From psarc-member-list-request@Sun.COM Mon Oct 1 14:45:21 2007 > Date: Mon, 01 Oct 2007 15:42:48 -0600 (MDT) > From: Mark Maybee > Subject: zpool failmode property [PSARC/2007/567 FastTrack timeout 10/08/2007] > To: PSARC-ext@Sun.COM > Cc: zfs-eng@Sun.COM > Content-transfer-encoding: 7BIT > X-PMX-Version: 5.2.0.264296 > X-PMX-Version: 5.2.0.264296 > > > Template Version: @(#)sac_nextcase 1.64 07/13/07 SMI > This information is Copyright 2007 Sun Microsystems > 1. Introduction > 1.1. Project/Component Working Name: > zpool failmode property > 1.2. Name of Document Author/Supplier: > Author: George Wilson > 1.3 Date of This Document: > 01 October, 2007 > 4. Technical Description > > The stability of this new property is committed, and the release binding > is patch/micro. > > A. SUMMARY > > This case adds a new pool-level property, 'failmode', to the existing zpool > property infrastructure (PSARC 2007/342). > > B. PROBLEM > > ZFS was designed to panic a system in the event of a catastrophic write failure > to a pool. From the initial design, non-replicated writes and critical reads > that could not be satisfied resulted in a system panic. This behavior was > commonly seen during error injection testing, as administrators would often > remove all paths to their storage arrays. > > Additionally, customer that configured ZFS on large HW RAID arrays, like > Hitachi, found themselves in a more susceptible situation since these pools > were typically created from a large single LUNs rather than a series of > replicated devices. This meant that a single write failure on this type of > pool would result in a system crash. > > C. PROPOSED SOLUTION > > This solution introduces a new pool property, "failmode", which allows the > administrator to determine the recovery behavior in the event of such a > failure. The property can be set to one of three options: "wait", "continue", > or "panic". > > The default behavior will be to "wait" for manual intervention before > allowing any further I/O attempts. Any I/O that was already queued would > remain in memory until the condition is resolved. This error condition can > be cleared by using the 'zpool clear' subcommand, which will attempt to resume > any queued I/Os. > > The "continue" mode returns EIO to any new write request but attempts to > satisfy reads. Any write I/Os that were already in-flight at the time > of the failure will be queued and maybe resumed using 'zpool clear'. > > Finally, the "panic" mode provides the existing behavior that was explained > above. > > The syntax for setting the pool property utilizes the "set" subcommand defined > in PSARC 2006/577: > > # zpool set failmode=continue pool > # zpool create -o failmode=continue pool > > D. MANPAGE DIFFS > > The following text will be added under the "Properties" section: > > failmode=continue | wait | panic > > Controls the system behavior in the event of catastrophic pool > failure. This condition is typically a result of a loss of connectivity > to the underlying storage device[s] or a failure of all devices > within the pool. The behavior in the event of such an event will > be determined as follows: > > wait Block all I/O access until the device connectivity > is recovered and the errors are cleared. This is the > default behavior. > > continue Returns EIO to any new write I/O requests but allows > reads to any of the remaining healthy devices. Any > write requests that has yet be be committed to > disk would block. > > panic Prints out a messages to the console and generates > a system crash dump. > > > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > ON > 6.5. ARC review type: FastTrack > 6.6. ARC Exposure: open > > From Darren.Moffat@sun.com Tue Oct 2 04:30:01 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92BU0hj029767 for ; Tue, 2 Oct 2007 04:30:01 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l92BQigJ024857; Tue, 2 Oct 2007 19:26:52 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPA006017SQV200@nwk-avmta-1.sfbay.Sun.COM>; Tue, 02 Oct 2007 04:26:50 -0700 (PDT) Received: from gmp-eb-mail-2.sun.com ([192.18.6.24]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPA002ZH7SPOQ80@nwk-avmta-1.sfbay.Sun.COM>; Tue, 02 Oct 2007 04:26:50 -0700 (PDT) Received: from fe-emea-09.sun.com (gmp-eb-lb-2-fe1.eu.sun.com [192.18.6.10]) by gmp-eb-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l92BQnT4002518; Tue, 02 Oct 2007 11:26:49 +0000 (GMT) Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPA00G0178CAA00@fe-emea-09.sun.com> (original mail from Darren.Moffat@Sun.COM); Tue, 02 Oct 2007 12:26:49 +0100 (BST) Received: from [129.156.173.21] by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPA008E17SFWL10@fe-emea-09.sun.com>; Tue, 02 Oct 2007 12:26:40 +0100 (BST) Date: Tue, 02 Oct 2007 12:26:39 +0100 From: Darren J Moffat Subject: Re: zpool failmode property [PSARC/2007/567 FastTrack timeout 10/08/2007] In-reply-to: <200710012142.l91LgmV2013403@moosylvania.Central.Sun.COM> Sender: Darren.Moffat@sun.com To: Mark Maybee Cc: PSARC-ext@sun.com, zfs-eng@sun.com Message-id: <47022AEF.40805@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200710012142.l91LgmV2013403@moosylvania.Central.Sun.COM> User-Agent: Thunderbird 2.0.0.4 (X11/20070731) Status: RO Content-Length: 365 Mark Maybee wrote: > wait Block all I/O access until the device connectivity > is recovered and the errors are cleared. This is the > default behavior. It isn't clear from the case material but I assume that reads that can be satisfied from the cache (ARC) will succeed regardless of the setting of failmode. Is that correct ? -- Darren J Moffat From George.Wilson@sun.com Tue Oct 2 07:12:14 2007 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92ECETa004467 for ; Tue, 2 Oct 2007 07:12:14 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l92E9112021254; Tue, 2 Oct 2007 07:09:06 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPA00B4TFB6D400@nwk-avmta-2.sfbay.sun.com>; Tue, 02 Oct 2007 07:09:06 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPA00J8TFB3P1B0@nwk-avmta-2.sfbay.sun.com>; Tue, 02 Oct 2007 07:09:04 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l92E93oi026648; Tue, 02 Oct 2007 14:09:03 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPA00601F0CVI00@mail-amer.sun.com> (original mail from George.Wilson@Sun.COM); Tue, 02 Oct 2007 08:09:03 -0600 (MDT) Received: from [172.16.0.188] ([64.135.171.115]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPA002GMFAA3KC0@mail-amer.sun.com>; Tue, 02 Oct 2007 08:08:35 -0600 (MDT) Date: Tue, 02 Oct 2007 10:02:33 -0400 From: George Wilson Subject: Re: zpool failmode property [PSARC/2007/567 FastTrack timeout 10/08/2007] In-reply-to: <47022AEF.40805@Sun.COM> Sender: George.Wilson@sun.com To: Darren J Moffat Cc: Mark Maybee , PSARC-ext@sun.com, zfs-eng@sun.com Message-id: <47024F79.7020301@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200710012142.l91LgmV2013403@moosylvania.Central.Sun.COM> <47022AEF.40805@Sun.COM> User-Agent: Thunderbird 2.0.0.4 (X11/20070730) Status: RO Content-Length: 525 Darren J Moffat wrote: > Mark Maybee wrote: >> wait Block all I/O access until the device connectivity >> is recovered and the errors are cleared. This is the >> default behavior. > > It isn't clear from the case material but I assume that reads that can > be satisfied from the cache (ARC) will succeed regardless of the setting > of failmode. Is that correct ? > > Darren, That's correct, the blocking behavior occurs when we need to issue an I/O to the disk. Thanks, George