From rsb@sac.sfbay.sun.com Fri Nov 2 15:02:35 2007 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id lA2M2ZbA026101 for ; Fri, 2 Nov 2007 15:02:35 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id lA2Lwv6Q022609; Fri, 2 Nov 2007 15:58:59 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JQW0071VFQ9QN00@nwk-avmta-2.sfbay.sun.com>; Fri, 02 Nov 2007 14:58:57 -0700 (PDT) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JQW00H55FQ9FJB0@nwk-avmta-2.sfbay.sun.com>; Fri, 02 Nov 2007 14:58:57 -0700 (PDT) Received: from sac.sfbay.sun.com (new-sac.SFBay.Sun.COM [129.146.175.65]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id lA2Lwu6X044103; Fri, 02 Nov 2007 14:58:56 -0700 (PDT) Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id lA2M2UTZ026096; Fri, 02 Nov 2007 15:02:30 -0700 (PDT) Received: (from rsb@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id lA2M2U65026092; Fri, 02 Nov 2007 17:02:30 -0500 (CDT) Date: Fri, 02 Nov 2007 17:02:30 -0500 (CDT) From: Rich.Brown@sun.com Subject: Caller context flags [PSARC/2007/632 FastTrack timeout 11/09/2007] To: PSARC-ext@sun.com Cc: james.wahlig@sun.com Message-id: <200711022202.lA2M2U65026092@sac.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 Status: RO Content-Length: 3228 I'm sponsoring this fast-track for Jim Wahlig. This case seeks Minor binding. Template Version: @(#)sac_nextcase 1.64 07/13/07 SMI This information is Copyright 2007 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: Caller context flags 1.2. Name of Document Author/Supplier: Author: Jim Wahlig 1.3 Date of This Document: 02 November, 2007 4. Technical Description One of the uses of the caller_context structure is to pass information between the caller of a vnode operation (VOP) and a File Event Monitor (FEM). It is used by both NFS and CIFS servers. Monitors often need to perform operations that would block the caller. For example, an NFSv4 delegation monitor may need to perform an over-the-wire operation to recall a delegation. The problem is that the caller may not be in a position to block and has no way to communicate that state to the monitor. Proposed Solution This case proposes to add a flags field (cc_flags) to the caller_context structure as well as values to communicate the behavior needed by the caller. The new caller context structure looks like this: typedef struct caller_context { pid_t cc_pid; /* Process ID of the caller */ int cc_sysid; /* System ID, used for remote calls */ u_longlong_t cc_caller_id; /* Identifier for (set of) caller(s) */ uint64_t cc_flags; <-- NEW FLAG FIELD } caller_context_t; The first two new flags to be defined: #define CC_WOULDBLOCK 0x1 /* set upon return by monitor */ #define CC_DONTBLOCK 0x2 /* set by caller */ The caller sets CC_DONTBLOCK in cc_flags to direct the monitor not to perform an operation that might block. In the case where a monitor would perform a blocking operation and CC_DONTBLOCK is set, the monitor sets CC_WOULDBLOCK in the cc_flags and returns EAGAIN to the caller. The first consumer of this new field would be the NFS server. The flags passed would inform the monitors on delegated files whether to wait for the delegation to be returned or just kick off the recall and return an error. The NFS server will set CC_DONTBLOCK to inform the delegation monitors not to wait for a delegation to be returned when there is a conflict. Instead, the monitors will return EAGAIN and set the CC_WOULDBLOCK flag after issuing the delegation recall. Exported Interfaces | | Interface Name | Classification | Comments ================================================================= | | CC_WOULDBLOCK | consolidation | set when returning EAGAIN to an op | private | that would have been blocked. | | CC_DONTBLOCK | | set by caller to indicate that op's | | should not block. 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From carlsonj@phorcys.east.sun.com Mon Nov 5 04:25:04 2007 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id lA5CP4Hg004762 for ; Mon, 5 Nov 2007 04:25:04 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id lA5CLCWs025674; Mon, 5 Nov 2007 04:21:26 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JR100K058ZOB900@nwk-avmta-1.sfbay.Sun.COM>; Mon, 05 Nov 2007 04:21:24 -0800 (PST) Received: from phorcys.east.sun.com ([129.148.174.143]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JR1007AB8ZN5X50@nwk-avmta-1.sfbay.Sun.COM>; Mon, 05 Nov 2007 04:21:24 -0800 (PST) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.1+Sun/8.14.1) with ESMTP id lA5CKkHl020734; Mon, 05 Nov 2007 07:20:46 -0500 (EST) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.1+Sun/8.14.1/Submit) id lA5CKkq1020731; Mon, 05 Nov 2007 07:20:46 -0500 (EST) Date: Mon, 05 Nov 2007 07:20:46 -0500 From: James Carlson Subject: Re: Caller context flags [PSARC/2007/632 FastTrack timeout 11/09/2007] In-reply-to: <200711022202.lA2M2U65026092@sac.sfbay.sun.com> To: Rich.Brown@sun.com Cc: PSARC-ext@sun.com, James.Wahlig@sun.com Message-id: <18223.2718.390678.550681@gargle.gargle.HOWL> MIME-version: 1.0 X-Mailer: VM 7.01 under Emacs 21.3.1 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200711022202.lA2M2U65026092@sac.sfbay.sun.com> Status: RO Content-Length: 1026 Rich.Brown@Sun.COM writes: > The first two new flags to be defined: > #define CC_WOULDBLOCK 0x1 /* set upon return by monitor */ > #define CC_DONTBLOCK 0x2 /* set by caller */ > > The caller sets CC_DONTBLOCK in cc_flags to direct the monitor not to > perform an operation that might block. In the case where a monitor would > perform a blocking operation and CC_DONTBLOCK is set, the monitor > sets CC_WOULDBLOCK in the cc_flags and returns EAGAIN to the caller. I'm not sure I understand how this is intended to be used. What exactly is the caller supposed to do when he gets CC_WOULDBLOCK? How long should he wait before trying again? Or is it always possible to detect this case and defer the operation until at can be run without CC_DONTBLOCK? -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From James.Wahlig@sun.com Mon Nov 5 12:16:27 2007 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id lA5KGRUb016435 for ; Mon, 5 Nov 2007 12:16:27 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id lA5KCUPr002013 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Mon, 5 Nov 2007 20:12:48 GMT Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JR100403UT9CS00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 05 Nov 2007 12:12:46 -0800 (PST) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JR1000A9UT91B50@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 05 Nov 2007 12:12:45 -0800 (PST) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id lA5KCjhx026982 for ; Mon, 05 Nov 2007 20:12:45 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JR100H01SF50H00@mail-amer.sun.com> (original mail from James.Wahlig@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 05 Nov 2007 13:12:45 -0700 (MST) Received: from [129.153.131.94] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JR100K2JUSYU320@mail-amer.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 05 Nov 2007 13:12:34 -0700 (MST) Date: Mon, 05 Nov 2007 14:12:33 -0600 From: james wahlig Subject: Re: Caller context flags [PSARC/2007/632 FastTrack timeout 11/09/2007] In-reply-to: <18223.2718.390678.550681@gargle.gargle.HOWL> Sender: James.Wahlig@sun.com To: James Carlson Cc: Rich.Brown@sun.com, PSARC-ext@sun.com Reply-to: james.wahlig@central.sun.com Message-id: <472F7931.6060905@Sun.COM> Organization: Sun Microsystems MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-Accept-Language: en-us, en X-PMX-Version: 5.2.0.264296 References: <200711022202.lA2M2U65026092@sac.sfbay.sun.com> <18223.2718.390678.550681@gargle.gargle.HOWL> User-Agent: Mozilla/5.0 (X11; U; SunOS sun4u; en-US; rv:1.7) Gecko/20070109 Status: RO Content-Length: 2431 James Carlson wrote: >Rich.Brown@Sun.COM writes: > > >> The first two new flags to be defined: >> #define CC_WOULDBLOCK 0x1 /* set upon return by monitor */ >> #define CC_DONTBLOCK 0x2 /* set by caller */ >> >> The caller sets CC_DONTBLOCK in cc_flags to direct the monitor not to >> perform an operation that might block. In the case where a monitor would >> perform a blocking operation and CC_DONTBLOCK is set, the monitor >> sets CC_WOULDBLOCK in the cc_flags and returns EAGAIN to the caller. >> >> > >I'm not sure I understand how this is intended to be used. > > It is intended to be used by callers that do not want to block inside a monitor while waiting for a delegation to be returned. >What exactly is the caller supposed to do when he gets CC_WOULDBLOCK? > > I'm not certain that there is a general rule to follow here, however, I can give you examples of what NFS does. Each version of the NFS server will use the CC_DONTBLOCK flag on calls to VOP_READ, VOP_WRITE, VOP_SETATTR, and (in the case of v4) VOP_OPEN. When a monitor on a delegated file detects a conflict in one of the ops, it will issue a recall of the delegation and then check to see if the CC_DONTBLOCK flag is set. If it is set, it will then set the CC_WOULDBLOCK flag and return EAGAIN. When the NFSv2 server gets this, it will simply drop the response, causing the client to reissue the request. When the NFSv3 server gets the EAGAIN w/CC_WOULDBLOCK, it will return the error NFS3ERR_JUKEBOX to the client. The NFSv3 client knows to retry when it sees this error. The NFSv4 server will return NFS4ERR_DELAY when it gets this error from the monitor. >How long should he wait before trying again? Or is it always possible >to detect this case and defer the operation until at can be run >without CC_DONTBLOCK? > > > The way the NFSv2&v3 server currently work is to use cross calls into the NFSv4 server to check if a file is delegated. If so, the delegation is recalled and the v2 & v3 server do what is described above. The reason for this fast track is to allow us to remove those cross calls from the v2 & v3 server and have all conflict detection handled by the monitors. The point is, the only caller that will use these flags are ones that are built to handle it. That is, if they get EAGAIN, they have a mechanism for retrying. I hope this helped. Jim From carlsonj@phorcys.east.sun.com Mon Nov 5 13:56:38 2007 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id lA5LubC8019588 for ; Mon, 5 Nov 2007 13:56:38 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id lA5Lqqdp014489; Mon, 5 Nov 2007 21:52:57 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JR100H14ZG65G00@nwk-avmta-1.sfbay.Sun.COM>; Mon, 05 Nov 2007 13:52:54 -0800 (PST) Received: from phorcys.east.sun.com ([129.148.174.143]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JR1006KPZG4ZK50@nwk-avmta-1.sfbay.Sun.COM>; Mon, 05 Nov 2007 13:52:53 -0800 (PST) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.1+Sun/8.14.1) with ESMTP id lA5LqEPo024157; Mon, 05 Nov 2007 16:52:14 -0500 (EST) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.1+Sun/8.14.1/Submit) id lA5LqEqq024154; Mon, 05 Nov 2007 16:52:14 -0500 (EST) Date: Mon, 05 Nov 2007 16:52:14 -0500 From: James Carlson Subject: Re: Caller context flags [PSARC/2007/632 FastTrack timeout 11/09/2007] In-reply-to: <472F7931.6060905@Sun.COM> To: james.wahlig@central.sun.com Cc: Rich.Brown@sun.com, PSARC-ext@sun.com Message-id: <18223.37006.431243.420773@gargle.gargle.HOWL> MIME-version: 1.0 X-Mailer: VM 7.01 under Emacs 21.3.1 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200711022202.lA2M2U65026092@sac.sfbay.sun.com> <18223.2718.390678.550681@gargle.gargle.HOWL> <472F7931.6060905@Sun.COM> Status: RO Content-Length: 1197 james wahlig writes: > return EAGAIN. When the NFSv2 server gets this, it will simply drop the > response, causing the client to reissue the request. When the NFSv3 > server gets the EAGAIN w/CC_WOULDBLOCK, it will return the error > NFS3ERR_JUKEBOX to the client. The NFSv3 client knows to retry when it > sees this error. The NFSv4 server will return NFS4ERR_DELAY when it > gets this error from the monitor. That seems to answer the question; thanks. The answer is that the wait period is handled by forcing the client to retry. > The point is, the only caller that will use these flags are ones that > are built to handle it. That is, if they get EAGAIN, they have a > mechanism for retrying. Right ... the part I was missing was how they know _when_ to retry, because the mechanism doesn't specify a means to allow for that. Knowing that the retries are pushed back to the client which will just try every now and again answers the question. Thanks. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From Rich.Brown@sun.com Wed Nov 7 20:35:43 2007 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id lA84Zh7x020264 for ; Wed, 7 Nov 2007 20:35:43 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id lA84W2uO021473 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 7 Nov 2007 20:32:03 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JR60060H79FZ700@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Wed, 07 Nov 2007 20:32:03 -0800 (PST) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JR600LL979EXY70@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Wed, 07 Nov 2007 20:32:02 -0800 (PST) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id lA84W2AA023554 for ; Thu, 08 Nov 2007 04:32:02 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JR60070178KP500@mail-amer.sun.com> (original mail from Rich.Brown@Sun.COM) for PSARC-ext@Sun.Com (ORCPT PSARC-ext@Sun.Com); Wed, 07 Nov 2007 21:32:02 -0700 (MST) Received: from [129.147.9.135] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JR6001GI79EQFH0@mail-amer.sun.com> for PSARC-ext@Sun.Com (ORCPT PSARC-ext@Sun.Com); Wed, 07 Nov 2007 21:32:02 -0700 (MST) Date: Wed, 07 Nov 2007 22:32:02 -0600 From: Rich Brown Subject: Re: Caller context flags [PSARC/2007/632 FastTrack timeout 11/09/2007] In-reply-to: <200711022202.lA2M2U65026092@sac.sfbay.sun.com> Sender: Rich.Brown@sun.com To: PSARC-ext@sun.com Cc: Rich.Brown@sun.com, James.Wahlig@sun.com Message-id: <47329142.7030301@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200711022202.lA2M2U65026092@sac.sfbay.sun.com> User-Agent: Mail/News 1.5.0.5 (X11/20060813) Status: RO Content-Length: 78 This was approved at today's PSARC meeting. I've updated the IAM file. Rich