From darrenm@sac.sfbay.sun.com Tue Nov 10 16:00:10 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nAB00AGE028271 for ; Tue, 10 Nov 2009 16:00:10 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id nAB009Ut003136; Tue, 10 Nov 2009 16:00:09 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSX0001N409V400@nwk-avmta-2.sfbay.sun.com>; Tue, 10 Nov 2009 16:00:09 -0800 (PST) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSX00KSM408IH80@nwk-avmta-2.sfbay.sun.com>; Tue, 10 Nov 2009 16:00:08 -0800 (PST) Received: from sac.sfbay.sun.com (sac.SFBay.Sun.COM [129.146.226.132]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id nAB008Yv004379; Tue, 10 Nov 2009 16:00:08 -0800 (PST) Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nAB006W3028249; Tue, 10 Nov 2009 16:00:06 -0800 (PST) Received: (from darrenm@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id nAB006i9028245; Tue, 10 Nov 2009 16:00:06 -0800 (PST) Date: Tue, 10 Nov 2009 16:00:06 -0800 (PST) From: Darren J Moffat Subject: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] To: PSARC-ext@sun.com Cc: Jonathan.Adams@sun.com, blake.jones@sun.com Message-id: <200911110000.nAB006i9028245@sac.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 4797 I'm submitting this as closed approved automatic, it has already been reviewed by core ZFS and kernel scheduling teams. Given the taxonomy of the new interfaces I believe this is sufficient review and closed approved automatic is sufficient. If any member disagrees then I'll set a fast-track timer. Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: System Duty Cycle Scheduling Class and ZFS IO Observability 1.2. Name of Document Author/Supplier: Author: Jonathan Adams 1.3 Date of This Document: 10 November, 2009 4. Technical Description This case covers two user-visible changes associated with the bug: 6806882 need a less brutal I/O scheduler The changes are: 1. A new kernel-thread-only scheduling class, SDC (System Duty Cycle) is introduced. Since the list of scheduling classes is user-visible through dispadmin(1M) and priocntl(1), we feel an ARC case is appropriate. The new scheduling class is only used by kernel threads, so the only manpage changes are a new SDC(7) page covering the existence of the new loadable module, and an update to the discussion of the SYS class in priocntl(2). A new function "sysdc_thread_enter()", puts a kernel thread into SDC. 2. In order to improve the observability of ZFS IO processing, we introduce a new system process per-zpool, with a u_comm and u_psargs of zpool- (i.e. zpool-foo). The LWPs in the process are kernel threads associated with IO processing. Since these threads have LWPs, they get microstate accounting data. Thus, CPU usage associated with IO processing can be observed using prstat(1M) and ps(1). This process will always be in the global zone, even if the pool was imported into a local zone. These changes, while user-visible, are essentially implementation details, and are not committed interfaces. We've tried to make this clear in the manpage changes. Interface Table --------------- Interface Stability --------------------------------------- ----------------- SDC module and scheduling class Consolidation Private sysdc_thread_enter() Consolidation Private zpool- process Volatile We seek Micro/Patch binding for these interfaces. See Also -------- 6806882 need a less brutal I/O scheduler Man Page Diffs -------------- SDC(7) +Device and Network Interfaces SDC(7) + + +NAME + SDC - System Duty Cycle scheduling class + +DESCRIPTION + The System Duty Cycle (SDC) scheduling class is used for some + CPU-intensive kernel thread workloads. Like the SYS class, + it cannot be used for user processes. + +ATTRIBUTES + See attributes(5) for descriptions of the following attri- + butes: + ____________________________________________________________ + | ATTRIBUTE TYPE | ATTRIBUTE VALUE | + |_____________________________|_____________________________| + +SEE ALSO + attributes(5) + priocntl(2) ... that class and/or when setting class-specific scheduling parameters. Two special scheduling classes, SYS and SDC, exist for the purpose of | scheduling the execution of certain special system processes (such as the swapper process). It is not possible to change the class of any LWP to SYS or SDC. In addition, any processes in | the SYS or SDC classes that are included in a specified set of | processes are disregarded by priocntl(). For example, an idtype of P_UID and an id value of 0 would specify all processes with a user ID of 0 except processes in the SYS and | SDC classes and (if changing the parameters using PC_SETPARMS or | PC_SETXPARMS) the init(1M) process. ... zpool(1M) --- zpool.1m.dist Tue Nov 3 14:00:53 2009 +++ zpool.1m Tue Nov 3 14:49:47 2009 @@ -947,6 +947,15 @@ version number, and only up to the most recent version supported by this software. +PROCESSES + Each imported pool has an associated process, named + zpool-poolname. The threads in this process are the + pool's IO processing threads, which handle the compression, + checksumming, and other tasks for all IO associated with the + pool. This process exists to provides visibility into the CPU + utilization of the system's storage pools. The existence + of this process is an unstable interface. + EXAMPLES Example 1 Creating a RAID-Z Storage Pool 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: On 6.5. ARC review type: Automatic 6.6. ARC Exposure: open From carlsonj@workingcode.com Wed Nov 11 05:04:32 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nABD4WkA022591 for ; Wed, 11 Nov 2009 05:04:32 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id nABD4Vwv019434; Wed, 11 Nov 2009 05:04:32 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSY00D014BI8J00@brm-avmta-1.central.sun.com>; Wed, 11 Nov 2009 06:04:30 -0700 (MST) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSY0018A4BH5380@brm-avmta-1.central.sun.com>; Wed, 11 Nov 2009 06:04:29 -0700 (MST) Received: from relay15i.sun.com (ip125.net129179-4.block1.us.syntegra.com [129.179.4.125]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id nABCtiPt012482; Wed, 11 Nov 2009 13:04:29 +0000 (GMT) Received: from mmp14es.mmp.us.syntegra.com ([160.41.208.14] [160.41.208.14]) by relay15i.sun.com with ESMTP id BT-MMP-2106718; Wed, 11 Nov 2009 13:04:29 +0000 (Z) Received: from relay11i.sun.com (relay11i.sun.com [129.179.4.121]) by mmp14es.mmp.us.syntegra.com with ESMTP id BT-MMP-358499; Wed, 11 Nov 2009 13:04:29 +0000 (Z) Received: from carlson.workingcode.com ([75.150.68.97] [75.150.68.97]) by relay1i.sun.com with ESMTP id BT-MMP-6150765; Wed, 11 Nov 2009 13:04:28 +0000 (Z) Received: from [10.50.24.188] (gate.abinitio.com [65.170.40.132]) (authenticated bits=0) by carlson.workingcode.com (8.14.2+Sun/8.14.3) with ESMTP id nABD4RV9017874 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 11 Nov 2009 08:04:27 -0500 (EST) Date: Wed, 11 Nov 2009 08:04:27 -0500 From: James Carlson Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: <200911110000.nAB006i9028245@sac.sfbay.sun.com> To: Darren J Moffat Cc: PSARC-ext@sun.com, blake.jones@sun.com Message-id: <4AFAB65B.6050804@workingcode.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-DCC-EATSERVER-Metrics: carlson 1166; Body=3 Fuz1=3 Fuz2=3 References: <200911110000.nAB006i9028245@sac.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.22 (X11/20090605) Status: RO Content-Length: 1161 Darren J Moffat wrote: > 1. A new kernel-thread-only scheduling class, SDC > (System Duty Cycle) is introduced. Since the > list of scheduling classes is user-visible through > dispadmin(1M) and priocntl(1), we feel an ARC case > is appropriate. User visible or not, I'd think a scheduling class is an interesting architectural attribute. "User visible" isn't the only reason for ARC review. > 2. In order to improve the observability of ZFS IO processing, > we introduce a new system process per-zpool, with a > u_comm and u_psargs of zpool- (i.e. zpool-foo). > The LWPs in the process are kernel threads associated > with IO processing. Since these threads have LWPs, they > get microstate accounting data. Thus, CPU usage associated > with IO processing can be observed using prstat(1M) and ps(1). Is this new class usable only by ZFS, or is it potentially usable by other kernel tasks as well? It looks to me like it could be used to make work done by networking squeues and STREAMS service procedures more observable. -- James Carlson 42.703N 71.076W From Blake.Jones@sun.com Wed Nov 11 06:10:25 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nABEAOco023865 for ; Wed, 11 Nov 2009 06:10:24 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id nABEALdV043908; Wed, 11 Nov 2009 07:10:23 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSY00H0H7DAJB00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 11 Nov 2009 06:10:22 -0800 (PST) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.63]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSY00FBK7D9LH20@nwk-avmta-1.sfbay.Sun.COM>; Wed, 11 Nov 2009 06:10:21 -0800 (PST) Received: from jurassic-x4600.sfbay.sun.com (localhost [127.0.0.1]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id nABEAKcU442095; Wed, 11 Nov 2009 06:10:20 -0800 (PST) Date: Wed, 11 Nov 2009 06:10:20 -0800 From: Blake Jones Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: "Your message of Wed, 11 Nov 2009 08:04:27 EST." <4AFAB65B.6050804@workingcode.com> Sender: Blake.Jones@sun.com To: James Carlson Cc: Darren J Moffat , PSARC-ext@sun.com, Blake.Jones@sun.com, jonathan.adams@sun.com Reply-to: Blake Jones Message-id: <442094.1257948620@jurassic-x4600.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 1899 > Darren J Moffat wrote: > > 1. A new kernel-thread-only scheduling class, SDC > > (System Duty Cycle) is introduced. Since the > > list of scheduling classes is user-visible through > > dispadmin(1M) and priocntl(1), we feel an ARC case > > is appropriate. > > User visible or not, I'd think a scheduling class is an interesting > architectural attribute. "User visible" isn't the only reason for ARC > review. > > > 2. In order to improve the observability of ZFS IO processing, > > we introduce a new system process per-zpool, with a > > u_comm and u_psargs of zpool- (i.e. zpool-foo). > > The LWPs in the process are kernel threads associated > > with IO processing. Since these threads have LWPs, they > > get microstate accounting data. Thus, CPU usage associated > > with IO processing can be observed using prstat(1M) and ps(1). > > Is this new class usable only by ZFS, or is it potentially usable by > other kernel tasks as well? It looks to me like it could be used to > make work done by networking squeues and STREAMS service procedures more > observable. It is potentially usable by other kernel tasks. There is nothing ZFS-specific about the class right now, although it makes some very mild assumptions about the characteristics of ZFS' workload (e.g. that it's processing many fairly small tasks, rather than a few large ones). These assumptions basically just have performance implications, rather than correctness ones, and changing them would be fairly straightforward -- on the order of adding of a new flag to an interface -- but we didn't do so because there were no other current consumers of such generalization. The "zpool-" process is created separately from the scheduling class. It allows the CPU time from the ZFS task queues (which are now running under SDC) to be more easily observed. Blake From carlsonj@workingcode.com Wed Nov 11 06:38:22 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nABEcLet023943 for ; Wed, 11 Nov 2009 06:38:21 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id nABEcKKP060818; Wed, 11 Nov 2009 07:38:21 -0700 (MST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSY00M098NWHR00@brm-avmta-1.central.sun.com>; Wed, 11 Nov 2009 07:38:20 -0700 (MST) Received: from sca-ea-mail-3.sun.com ([192.18.43.21]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSY001RH8NV59E0@brm-avmta-1.central.sun.com>; Wed, 11 Nov 2009 07:38:19 -0700 (MST) Received: from relay41i.sun.com ([192.5.209.70]) by sca-ea-mail-3.sun.com (8.13.6+Sun/8.12.9) with ESMTP id nABEIZke006823; Wed, 11 Nov 2009 14:38:19 +0000 (GMT) Received: from mms48es.mms.us.syntegra.com ([160.41.221.230] [160.41.221.230]) by relay41i.sun.com with ESMTP id BT-MMP-184765; Wed, 11 Nov 2009 14:38:19 +0000 (Z) Received: from relay44i.sun.com (relay44i.sun.com [192.5.209.118]) by mms48es.mms.us.syntegra.com with ESMTP id BT-MMP-50788104; Wed, 11 Nov 2009 14:38:18 +0000 (Z) Received: from carlson.workingcode.com ([75.150.68.97] [75.150.68.97]) by relay4i.sun.com with ESMTP id BT-MMP-4996083; Wed, 11 Nov 2009 14:38:18 +0000 (Z) Received: from [10.50.24.188] (gate.abinitio.com [65.170.40.132]) (authenticated bits=0) by carlson.workingcode.com (8.14.2+Sun/8.14.3) with ESMTP id nABEcHSZ018138 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 11 Nov 2009 09:38:17 -0500 (EST) Date: Wed, 11 Nov 2009 09:38:17 -0500 From: James Carlson Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: <442094.1257948620@jurassic-x4600.sfbay.sun.com> To: Blake Jones Cc: Darren J Moffat , PSARC-ext@sun.com, jonathan.adams@sun.com Message-id: <4AFACC59.907@workingcode.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-DCC-dmv.com-Metrics: carlson 1181; Body=4 Fuz1=4 Fuz2=4 X-Antispam: No, score=-0.7/5.0, scanned in 0.135sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <442094.1257948620@jurassic-x4600.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.22 (X11/20090605) Status: RO Content-Length: 1285 Blake Jones wrote: >> Is this new class usable only by ZFS, or is it potentially usable by >> other kernel tasks as well? It looks to me like it could be used to >> make work done by networking squeues and STREAMS service procedures more >> observable. > > It is potentially usable by other kernel tasks. There is nothing > ZFS-specific about the class right now, although it makes some very mild > assumptions about the characteristics of ZFS' workload (e.g. that it's > processing many fairly small tasks, rather than a few large ones). > These assumptions basically just have performance implications, rather > than correctness ones, and changing them would be fairly straightforward > -- on the order of adding of a new flag to an interface -- but we didn't > do so because there were no other current consumers of such > generalization. Nifty; thanks! It sounds like it could end up being quite useful. > The "zpool-" process is created separately from the scheduling > class. It allows the CPU time from the ZFS task queues (which are now > running under SDC) to be more easily observed. Just curious: is that process created with a non-zero zoneid_t when the pool is inside a zone? -- James Carlson 42.703N 71.076W From jonathan.adams@sun.com Wed Nov 11 09:13:42 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nABHDf6n028181 for ; Wed, 11 Nov 2009 09:13:41 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id nABHDeqk021417; Wed, 11 Nov 2009 09:13:41 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSY00G05FUTWT00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 11 Nov 2009 09:13:41 -0800 (PST) Received: from norton.eng.sun.com ([129.146.228.124]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSY0066EFUSWLE0@nwk-avmta-1.sfbay.Sun.COM>; Wed, 11 Nov 2009 09:13:40 -0800 (PST) Received: from norton.sfbay.sun.com (localhost [127.0.0.1]) by norton.eng.sun.com (8.14.3+Sun/8.14.3) with ESMTP id nABHDdVh116491; Wed, 11 Nov 2009 09:13:39 -0800 (PST) Received: (from jwadams@localhost) by norton.sfbay.sun.com (8.14.3+Sun/8.14.3/Submit) id nABHDcsP116490; Wed, 11 Nov 2009 09:13:38 -0800 (PST) Date: Wed, 11 Nov 2009 09:13:37 -0800 From: Jonathan Adams Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: <4AFACC59.907@workingcode.com> To: James Carlson Cc: Blake Jones , Darren J Moffat , PSARC-ext@sun.com Message-id: <20091111171337.GA116375@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <442094.1257948620@jurassic-x4600.sfbay.sun.com> <4AFACC59.907@workingcode.com> X-Authentication-warning: norton.sfbay.sun.com: jwadams set sender to jonathan.adams@sun.com using -f User-Agent: Mutt/1.5.14 (2007-02-12) Status: RO Content-Length: 2057 On Wed, Nov 11, 2009 at 09:38:17AM -0500, James Carlson wrote: > Blake Jones wrote: > >> Is this new class usable only by ZFS, or is it potentially usable by > >> other kernel tasks as well? It looks to me like it could be used to > >> make work done by networking squeues and STREAMS service procedures more > >> observable. > > > > It is potentially usable by other kernel tasks. There is nothing > > ZFS-specific about the class right now, although it makes some very mild > > assumptions about the characteristics of ZFS' workload (e.g. that it's > > processing many fairly small tasks, rather than a few large ones). > > These assumptions basically just have performance implications, rather > > than correctness ones, and changing them would be fairly straightforward > > -- on the order of adding of a new flag to an interface -- but we didn't > > do so because there were no other current consumers of such > > generalization. > > Nifty; thanks! It sounds like it could end up being quite useful. The system process and associated LWPs are what give you observability; the regular process framework handles reporting all of the CPU use and microstate accounting data. The trade-off is the cost of microstate accounting; currently, kernel threads don't have it. Once you have an LWP, its recorded. The SDC class itself has no direct effect on observability; it is entirely about CPU-intensive kernel threads "playing well with others". SDC does require microstate accounting, though, which implies an LWP and the attendant observability. > > The "zpool-" process is created separately from the scheduling > > class. It allows the CPU time from the ZFS task queues (which are now > > running under SDC) to be more easily observed. > > Just curious: is that process created with a non-zero zoneid_t when the > pool is inside a zone? No; one future direction would be to make zone-created pools both show up as children of zsched and have their worker threads bound to the processor set the zone is bound to. Cheers, - jonathan From olga.kryzhanovska@gmail.com Wed Nov 11 09:57:33 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nABHvWBJ000314 for ; Wed, 11 Nov 2009 09:57:33 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id nABHvPvA024394; Wed, 11 Nov 2009 17:57:30 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSY00M0BHVT9300@nwk-avmta-1.sfbay.Sun.COM>; Wed, 11 Nov 2009 09:57:29 -0800 (PST) Received: from sca-ea-mail-2.sun.com ([192.18.43.25]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSY00H1ZHVTU460@nwk-avmta-1.sfbay.Sun.COM>; Wed, 11 Nov 2009 09:57:29 -0800 (PST) Received: from relay11i.sun.com (ip121.net129179-4.block1.us.syntegra.com [129.179.4.121]) by sca-ea-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id nABHmFZ8013516; Wed, 11 Nov 2009 17:57:29 +0000 (GMT) Received: from mmp14es.mmp.us.syntegra.com ([160.41.208.14] [160.41.208.14]) by relay11i.sun.com with ESMTP id BT-MMP-6418515; Wed, 11 Nov 2009 17:57:28 +0000 (Z) Received: from relay11i.sun.com (relay11i.sun.com [129.179.4.121]) by mmp14es.mmp.us.syntegra.com with ESMTP id BT-MMP-294793; Wed, 11 Nov 2009 17:57:28 +0000 (Z) Received: from mail-fx0-f213.google.com ([209.85.220.213] [209.85.220.213]) by relay1i.sun.com with ESMTP id BT-MMP-6728618; Wed, 11 Nov 2009 17:57:28 +0000 (Z) Received: by mail-fx0-f213.google.com with SMTP id 5so1431540fxm.8 for ; Wed, 11 Nov 2009 09:57:20 -0800 (PST) Received: by 10.223.16.72 with SMTP id n8mr266519faa.26.1257962240151; Wed, 11 Nov 2009 09:57:20 -0800 (PST) Date: Wed, 11 Nov 2009 18:57:20 +0100 From: =?KOI8-R?B?z8zYx8Egy9LZ1sHOz9fTy8HR?= Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: <200911110000.nAB006i9028245@sac.sfbay.sun.com> To: Darren J Moffat Cc: PSARC-ext@sun.com, blake.jones@sun.com Message-id: MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7BIT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=BhAKvsIhX4jHmuo4DuSnCxkbmnUAxqvINZHBfZshmzM=; b=f5NE372vMVkB0/nrak/B0aUK6lexHMnW+YhgJ8MFNzZmKPsJoit1lHqYP3J+9JH737 sm3XD9w5kXfM1PAc8F0YFVokShOE3XTvLkd5lqRisiU/ABVyGPDjgEq/bkMj8Gojqwfy gL7QwNmaSUwNNFz+5YSgwurNzJsnNz4IZ3pRQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=Va0Frhnf12qThz++9sVxQ4G+Vwq88GUZ26EYTVhNBG/T4qiHYykYDCQ1PLxmXnw2ag ND8aeUynN8u2iYjnoSoPf/yMuzhvX0fKkgI14CIpmNuXk9IHMyuOOk/0MrjfU4IebqVM hTgwwnb43DThC4DI+26AmWuoonyNdsoTcrSyA= X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== References: <200911110000.nAB006i9028245@sac.sfbay.sun.com> Status: RO Content-Length: 6095 Why should this class only available for kernel threads? There are daemons ported from micro kernel platforms which act like a kernel daemon but run in user land and could benefit from such a scheduler class. On Wed, Nov 11, 2009 at 1:00 AM, Darren J Moffat wrote: > I'm submitting this as closed approved automatic, it has already been > reviewed by core ZFS and kernel scheduling teams. Given the taxonomy > of the new interfaces I believe this is sufficient review and closed > approved automatic is sufficient. If any member disagrees then I'll > set a fast-track timer. > > Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI > This information is Copyright 2009 Sun Microsystems > 1. Introduction > 1.1. Project/Component Working Name: > System Duty Cycle Scheduling Class and ZFS IO Observability > 1.2. Name of Document Author/Supplier: > Author: Jonathan Adams > 1.3 Date of This Document: > 10 November, 2009 > 4. Technical Description > > This case covers two user-visible changes associated with the bug: > > 6806882 need a less brutal I/O scheduler > > The changes are: > > 1. A new kernel-thread-only scheduling class, SDC > (System Duty Cycle) is introduced. Since the > list of scheduling classes is user-visible through > dispadmin(1M) and priocntl(1), we feel an ARC case > is appropriate. > > The new scheduling class is only used by kernel threads, > so the only manpage changes are a new SDC(7) page covering > the existence of the new loadable module, and an update > to the discussion of the SYS class in priocntl(2). > > A new function "sysdc_thread_enter()", puts a kernel thread > into SDC. > > 2. In order to improve the observability of ZFS IO processing, > we introduce a new system process per-zpool, with a > u_comm and u_psargs of zpool- (i.e. zpool-foo). > The LWPs in the process are kernel threads associated > with IO processing. Since these threads have LWPs, they > get microstate accounting data. Thus, CPU usage associated > with IO processing can be observed using prstat(1M) and ps(1). > > This process will always be in the global zone, even if > the pool was imported into a local zone. > > These changes, while user-visible, are essentially implementation > details, and are not committed interfaces. We've tried to make this > clear in the manpage changes. > > > Interface Table > --------------- > > Interface Stability > --------------------------------------- ----------------- > SDC module and scheduling class Consolidation Private > sysdc_thread_enter() Consolidation Private > zpool- process Volatile > > We seek Micro/Patch binding for these interfaces. > > See Also > -------- > > 6806882 need a less brutal I/O scheduler > > > Man Page Diffs > -------------- > > SDC(7) > +Device and Network Interfaces SDC(7) > + > + > +NAME > + SDC - System Duty Cycle scheduling class > + > +DESCRIPTION > + The System Duty Cycle (SDC) scheduling class is used for some > + CPU-intensive kernel thread workloads. Like the SYS class, > + it cannot be used for user processes. > + > +ATTRIBUTES > + See attributes(5) for descriptions of the following attri- > + butes: > + ____________________________________________________________ > + | ATTRIBUTE TYPE | ATTRIBUTE VALUE | > + |_____________________________|_____________________________| > + > +SEE ALSO > + attributes(5) > + > > priocntl(2) > ... > that class and/or when setting class-specific scheduling > parameters. > > Two special scheduling classes, SYS and SDC, exist for the purpose of | > scheduling the execution of certain special system processes > (such as the swapper process). It is not possible to change > the class of any LWP to SYS or SDC. In addition, any processes in | > the SYS or SDC classes that are included in a specified set of | > processes are disregarded by priocntl(). For example, an > idtype of P_UID and an id value of 0 would specify all > processes with a user ID of 0 except processes in the SYS and | > SDC classes and (if changing the parameters using PC_SETPARMS or | > PC_SETXPARMS) the init(1M) process. > ... > > zpool(1M) > --- zpool.1m.dist Tue Nov 3 14:00:53 2009 > +++ zpool.1m Tue Nov 3 14:49:47 2009 > @@ -947,6 +947,15 @@ > version number, and only up to the most > recent version supported by this software. > > +PROCESSES > + Each imported pool has an associated process, named > + zpool-poolname. The threads in this process are the > + pool's IO processing threads, which handle the compression, > + checksumming, and other tasks for all IO associated with the > + pool. This process exists to provides visibility into the CPU > + utilization of the system's storage pools. The existence > + of this process is an unstable interface. > + > EXAMPLES > Example 1 Creating a RAID-Z Storage Pool > > > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > On > 6.5. ARC review type: Automatic > 6.6. ARC Exposure: open > > _______________________________________________ > opensolaris-arc mailing list > opensolaris-arc@opensolaris.org > -- , _ _ , { \/`o;====- Olga Kryzhanovska -====;o`\/ } .----'-/`-/ olga.kryzhanovska@gmail.com \-`\-'----. `'-..-| / Solaris/BSD//C/C++ programmer \ |-..-'` /\/\ /\/\ `--` `--` From jonathan.adams@sun.com Wed Nov 11 11:00:01 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nABJ01vE002620 for ; Wed, 11 Nov 2009 11:00:01 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id nABIxvSc001699; Wed, 11 Nov 2009 11:00:01 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSY00119KS0W900@brm-avmta-1.central.sun.com>; Wed, 11 Nov 2009 12:00:00 -0700 (MST) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSY00ELFKRZKL90@brm-avmta-1.central.sun.com>; Wed, 11 Nov 2009 11:59:59 -0700 (MST) Received: from jurassic-x4600.sfbay.sun.com (localhost [127.0.0.1]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id nABIxwFh533775 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 11 Nov 2009 10:59:58 -0800 (PST) Received: (from jwadams@localhost) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3/Submit) id nABIxwLa533761; Wed, 11 Nov 2009 10:59:58 -0800 (PST) Date: Wed, 11 Nov 2009 10:59:48 -0800 From: Jonathan Adams Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: To: ????? ???????????? Cc: Darren J Moffat , PSARC-ext@sun.com, blake.jones@sun.com Message-id: <20091111185948.GA533685@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200911110000.nAB006i9028245@sac.sfbay.sun.com> X-Authentication-warning: jurassic-x4600.sfbay.sun.com: jwadams set sender to jonathan.adams@sun.com using -f User-Agent: Mutt/1.5.14 (2007-02-12) Status: RO Content-Length: 6695 On Wed, Nov 11, 2009 at 06:57:20PM +0100, ????? ???????????? wrote: > Why should this class only available for kernel threads? There are > daemons ported from micro kernel platforms which act like a kernel > daemon but run in user land and could benefit from such a scheduler > class. Because it is modeled on the SYS class, which is also not available to user threads. Supporting user threads involves a bunch of additional work which isn't relevant to the problem at hand. Cheers, - jonathan > On Wed, Nov 11, 2009 at 1:00 AM, Darren J Moffat > wrote: > > I'm submitting this as closed approved automatic, it has already been > > reviewed by core ZFS and kernel scheduling teams. Given the taxonomy > > of the new interfaces I believe this is sufficient review and closed > > approved automatic is sufficient. If any member disagrees then I'll > > set a fast-track timer. > > > > Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI > > This information is Copyright 2009 Sun Microsystems > > 1. Introduction > > 1.1. Project/Component Working Name: > > System Duty Cycle Scheduling Class and ZFS IO Observability > > 1.2. Name of Document Author/Supplier: > > Author: Jonathan Adams > > 1.3 Date of This Document: > > 10 November, 2009 > > 4. Technical Description > > > > This case covers two user-visible changes associated with the bug: > > > > 6806882 need a less brutal I/O scheduler > > > > The changes are: > > > > 1. A new kernel-thread-only scheduling class, SDC > > (System Duty Cycle) is introduced. Since the > > list of scheduling classes is user-visible through > > dispadmin(1M) and priocntl(1), we feel an ARC case > > is appropriate. > > > > The new scheduling class is only used by kernel threads, > > so the only manpage changes are a new SDC(7) page covering > > the existence of the new loadable module, and an update > > to the discussion of the SYS class in priocntl(2). > > > > A new function "sysdc_thread_enter()", puts a kernel thread > > into SDC. > > > > 2. In order to improve the observability of ZFS IO processing, > > we introduce a new system process per-zpool, with a > > u_comm and u_psargs of zpool- (i.e. zpool-foo). > > The LWPs in the process are kernel threads associated > > with IO processing. Since these threads have LWPs, they > > get microstate accounting data. Thus, CPU usage associated > > with IO processing can be observed using prstat(1M) and ps(1). > > > > This process will always be in the global zone, even if > > the pool was imported into a local zone. > > > > These changes, while user-visible, are essentially implementation > > details, and are not committed interfaces. We've tried to make this > > clear in the manpage changes. > > > > > > Interface Table > > --------------- > > > > Interface Stability > > --------------------------------------- ----------------- > > SDC module and scheduling class Consolidation Private > > sysdc_thread_enter() Consolidation Private > > zpool- process Volatile > > > > We seek Micro/Patch binding for these interfaces. > > > > See Also > > -------- > > > > 6806882 need a less brutal I/O scheduler > > > > > > Man Page Diffs > > -------------- > > > > SDC(7) > > +Device and Network Interfaces SDC(7) > > + > > + > > +NAME > > + SDC - System Duty Cycle scheduling class > > + > > +DESCRIPTION > > + The System Duty Cycle (SDC) scheduling class is used for some > > + CPU-intensive kernel thread workloads. Like the SYS class, > > + it cannot be used for user processes. > > + > > +ATTRIBUTES > > + See attributes(5) for descriptions of the following attri- > > + butes: > > + ____________________________________________________________ > > + | ATTRIBUTE TYPE | ATTRIBUTE VALUE | > > + |_____________________________|_____________________________| > > + > > +SEE ALSO > > + attributes(5) > > + > > > > priocntl(2) > > ... > > that class and/or when setting class-specific scheduling > > parameters. > > > > Two special scheduling classes, SYS and SDC, exist for the purpose of | > > scheduling the execution of certain special system processes > > (such as the swapper process). It is not possible to change > > the class of any LWP to SYS or SDC. In addition, any processes in | > > the SYS or SDC classes that are included in a specified set of | > > processes are disregarded by priocntl(). For example, an > > idtype of P_UID and an id value of 0 would specify all > > processes with a user ID of 0 except processes in the SYS and | > > SDC classes and (if changing the parameters using PC_SETPARMS or | > > PC_SETXPARMS) the init(1M) process. > > ... > > > > zpool(1M) > > --- zpool.1m.dist Tue Nov 3 14:00:53 2009 > > +++ zpool.1m Tue Nov 3 14:49:47 2009 > > @@ -947,6 +947,15 @@ > > version number, and only up to the most > > recent version supported by this software. > > > > +PROCESSES > > + Each imported pool has an associated process, named > > + zpool-poolname. The threads in this process are the > > + pool's IO processing threads, which handle the compression, > > + checksumming, and other tasks for all IO associated with the > > + pool. This process exists to provides visibility into the CPU > > + utilization of the system's storage pools. The existence > > + of this process is an unstable interface. > > + > > EXAMPLES > > Example 1 Creating a RAID-Z Storage Pool > > > > > > 6. Resources and Schedule > > 6.4. Steering Committee requested information > > 6.4.1. Consolidation C-team Name: > > On > > 6.5. ARC review type: Automatic > > 6.6. ARC Exposure: open > > > > _______________________________________________ > > opensolaris-arc mailing list > > opensolaris-arc@opensolaris.org > > > > > > -- > , _ _ , > { \/`o;====- Olga Kryzhanovska -====;o`\/ } > .----'-/`-/ olga.kryzhanovska@gmail.com \-`\-'----. > `'-..-| / Solaris/BSD//C/C++ programmer \ |-..-'` > /\/\ /\/\ > `--` `--` From olga.kryzhanovska@gmail.com Wed Nov 11 11:17:37 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nABJHa66003003 for ; Wed, 11 Nov 2009 11:17:37 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id nABJHYmI019483; Wed, 11 Nov 2009 19:17:35 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSY00809LLAYX00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 11 Nov 2009 11:17:34 -0800 (PST) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSY006JGLL9MP60@nwk-avmta-1.sfbay.Sun.COM>; Wed, 11 Nov 2009 11:17:33 -0800 (PST) Received: from relay42i.sun.com ([192.5.209.72]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id nABJ2EWn012162; Wed, 11 Nov 2009 19:17:33 +0000 (GMT) Received: from mmp43es.mmp.us.syntegra.com ([160.41.221.12] [160.41.221.12]) by relay42i.sun.com with ESMTP id BT-MMP-208881; Wed, 11 Nov 2009 19:17:33 +0000 (Z) Received: from relay43i.sun.com (relay43i.sun.com [192.5.209.74]) by mmp43es.mmp.us.syntegra.com with ESMTP id BT-MMP-51197228; Wed, 11 Nov 2009 19:17:28 +0000 (Z) Received: from mail-fx0-f213.google.com ([209.85.220.213] [209.85.220.213]) by relay4i.sun.com with ESMTP id BT-MMP-4224379; Wed, 11 Nov 2009 19:17:26 +0000 (Z) Received: by fxm5 with SMTP id 5so1522679fxm.8 for ; Wed, 11 Nov 2009 11:16:35 -0800 (PST) Received: by 10.223.103.25 with SMTP id i25mr284197fao.64.1257966995707; Wed, 11 Nov 2009 11:16:35 -0800 (PST) Date: Wed, 11 Nov 2009 20:16:35 +0100 From: =?KOI8-R?B?z8zYx8Egy9LZ1sHOz9fTy8HR?= Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: <20091111185948.GA533685@eng.sun.com> To: Jonathan Adams Cc: Darren J Moffat , PSARC-ext@sun.com, blake.jones@sun.com Message-id: MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7BIT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=xJkS5WQBzZ6VU3UsE0NyZjRxBIOEdUIdPeN7Dvp5koU=; b=ctejBwGwOV4aKXWSlONb/GjXi4QBb/5Xudn3eSDoPX34DG8JZiLpsqBoi8kfKo74x5 DI5nqdaMipZ2Gg9yujLzIA1KAjNNdwy/cOd/mOmeGxBpmvOKRdX8GnnlAjlnfPNHGIOl OxsVJ7rFYVwclF+JTwvIq8fs3JLcs7IcGUG5A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=sGW2I9nocl7vSmWFFZLDlVk0DZNshSdMWCW6G6ps2Y3mzgDqsy9e02pCvYxMUcVMdA 18FwKTaKd/w7eHS50BvT0MNMrVapaB5r28/JK2ttjSQcFQT3nOYWO9yMpYDBhwTsv2aT WzmCJDFIdhGKZ5eaOEvtjtq2/2VCSJl9H5tzU= X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=-1.1/5.0, scanned in 0.130sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200911110000.nAB006i9028245@sac.sfbay.sun.com> <20091111185948.GA533685@eng.sun.com> Status: RO Content-Length: 7641 Why don't you rename the class to zfsiosched If the class is tailored to support zfs only or kerneliosched if the class is tailored to support kernel io only? In my eyes its is a clear design fault to restrict such a class to kernel threads only. You are locking out user land daemons which have similar duties as the zfs threads you describe in this case. On Wed, Nov 11, 2009 at 7:59 PM, Jonathan Adams wrote: > On Wed, Nov 11, 2009 at 06:57:20PM +0100, ????? ???????????? wrote: >> Why should this class only available for kernel threads? There are >> daemons ported from micro kernel platforms which act like a kernel >> daemon but run in user land and could benefit from such a scheduler >> class. > > Because it is modeled on the SYS class, which is also not available to > user threads. Supporting user threads involves a bunch of additional > work which isn't relevant to the problem at hand. > > Cheers, > - jonathan > >> On Wed, Nov 11, 2009 at 1:00 AM, Darren J Moffat >> wrote: >> > I'm submitting this as closed approved automatic, it has already been >> > reviewed by core ZFS and kernel scheduling teams. Given the taxonomy >> > of the new interfaces I believe this is sufficient review and closed >> > approved automatic is sufficient. If any member disagrees then I'll >> > set a fast-track timer. >> > >> > Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI >> > This information is Copyright 2009 Sun Microsystems >> > 1. Introduction >> > 1.1. Project/Component Working Name: >> > System Duty Cycle Scheduling Class and ZFS IO Observability >> > 1.2. Name of Document Author/Supplier: >> > Author: Jonathan Adams >> > 1.3 Date of This Document: >> > 10 November, 2009 >> > 4. Technical Description >> > >> > This case covers two user-visible changes associated with the bug: >> > >> > 6806882 need a less brutal I/O scheduler >> > >> > The changes are: >> > >> > 1. A new kernel-thread-only scheduling class, SDC >> > (System Duty Cycle) is introduced. Since the >> > list of scheduling classes is user-visible through >> > dispadmin(1M) and priocntl(1), we feel an ARC case >> > is appropriate. >> > >> > The new scheduling class is only used by kernel threads, >> > so the only manpage changes are a new SDC(7) page covering >> > the existence of the new loadable module, and an update >> > to the discussion of the SYS class in priocntl(2). >> > >> > A new function "sysdc_thread_enter()", puts a kernel thread >> > into SDC. >> > >> > 2. In order to improve the observability of ZFS IO processing, >> > we introduce a new system process per-zpool, with a >> > u_comm and u_psargs of zpool- (i.e. zpool-foo). >> > The LWPs in the process are kernel threads associated >> > with IO processing. Since these threads have LWPs, they >> > get microstate accounting data. Thus, CPU usage associated >> > with IO processing can be observed using prstat(1M) and ps(1). >> > >> > This process will always be in the global zone, even if >> > the pool was imported into a local zone. >> > >> > These changes, while user-visible, are essentially implementation >> > details, and are not committed interfaces. We've tried to make this >> > clear in the manpage changes. >> > >> > >> > Interface Table >> > --------------- >> > >> > Interface Stability >> > --------------------------------------- ----------------- >> > SDC module and scheduling class Consolidation Private >> > sysdc_thread_enter() Consolidation Private >> > zpool- process Volatile >> > >> > We seek Micro/Patch binding for these interfaces. >> > >> > See Also >> > -------- >> > >> > 6806882 need a less brutal I/O scheduler >> > >> > >> > Man Page Diffs >> > -------------- >> > >> > SDC(7) >> > +Device and Network Interfaces SDC(7) >> > + >> > + >> > +NAME >> > + SDC - System Duty Cycle scheduling class >> > + >> > +DESCRIPTION >> > + The System Duty Cycle (SDC) scheduling class is used for some >> > + CPU-intensive kernel thread workloads. Like the SYS class, >> > + it cannot be used for user processes. >> > + >> > +ATTRIBUTES >> > + See attributes(5) for descriptions of the following attri- >> > + butes: >> > + ____________________________________________________________ >> > + | ATTRIBUTE TYPE | ATTRIBUTE VALUE | >> > + |_____________________________|_____________________________| >> > + >> > +SEE ALSO >> > + attributes(5) >> > + >> > >> > priocntl(2) >> > ... >> > that class and/or when setting class-specific scheduling >> > parameters. >> > >> > Two special scheduling classes, SYS and SDC, exist for the purpose of | >> > scheduling the execution of certain special system processes >> > (such as the swapper process). It is not possible to change >> > the class of any LWP to SYS or SDC. In addition, any processes in | >> > the SYS or SDC classes that are included in a specified set of | >> > processes are disregarded by priocntl(). For example, an >> > idtype of P_UID and an id value of 0 would specify all >> > processes with a user ID of 0 except processes in the SYS and | >> > SDC classes and (if changing the parameters using PC_SETPARMS or | >> > PC_SETXPARMS) the init(1M) process. >> > ... >> > >> > zpool(1M) >> > --- zpool.1m.dist Tue Nov 3 14:00:53 2009 >> > +++ zpool.1m Tue Nov 3 14:49:47 2009 >> > @@ -947,6 +947,15 @@ >> > version number, and only up to the most >> > recent version supported by this software. >> > >> > +PROCESSES >> > + Each imported pool has an associated process, named >> > + zpool-poolname. The threads in this process are the >> > + pool's IO processing threads, which handle the compression, >> > + checksumming, and other tasks for all IO associated with the >> > + pool. This process exists to provides visibility into the CPU >> > + utilization of the system's storage pools. The existence >> > + of this process is an unstable interface. >> > + >> > EXAMPLES >> > Example 1 Creating a RAID-Z Storage Pool >> > >> > >> > 6. Resources and Schedule >> > 6.4. Steering Committee requested information >> > 6.4.1. Consolidation C-team Name: >> > On >> > 6.5. ARC review type: Automatic >> > 6.6. ARC Exposure: open >> > >> > _______________________________________________ >> > opensolaris-arc mailing list >> > opensolaris-arc@opensolaris.org >> > >> >> >> >> -- >> , _ _ , >> { \/`o;====- Olga Kryzhanovska -====;o`\/ } >> .----'-/`-/ olga.kryzhanovska@gmail.com \-`\-'----. >> `'-..-| / Solaris/BSD//C/C++ programmer \ |-..-'` >> /\/\ /\/\ >> `--` `--` > -- , _ _ , { \/`o;====- Olga Kryzhanovska -====;o`\/ } .----'-/`-/ olga.kryzhanovska@gmail.com \-`\-'----. `'-..-| / Solaris/BSD//C/C++ programmer \ |-..-'` /\/\ /\/\ `--` `--` From jonathan.adams@sun.com Wed Nov 11 11:39:02 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nABJd14S003428 for ; Wed, 11 Nov 2009 11:39:01 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id nABJcq7F011651; Thu, 12 Nov 2009 03:38:58 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSY00C07MKX2000@nwk-avmta-2.sfbay.sun.com>; Wed, 11 Nov 2009 11:38:57 -0800 (PST) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSY000WZMKWE1E0@nwk-avmta-2.sfbay.sun.com>; Wed, 11 Nov 2009 11:38:56 -0800 (PST) Received: from jurassic-x4600.sfbay.sun.com (localhost [127.0.0.1]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id nABJctFd541961 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 11 Nov 2009 11:38:55 -0800 (PST) Received: (from jwadams@localhost) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3/Submit) id nABJctVW541960; Wed, 11 Nov 2009 11:38:55 -0800 (PST) Date: Wed, 11 Nov 2009 11:38:55 -0800 From: Jonathan Adams Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: To: olga.kryzhanovska@gmail.com Cc: Darren J Moffat , PSARC-ext@sun.com, blake.jones@sun.com Message-id: <20091111193855.GH123068@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200911110000.nAB006i9028245@sac.sfbay.sun.com> <20091111185948.GA533685@eng.sun.com> X-Authentication-warning: jurassic-x4600.sfbay.sun.com: jwadams set sender to jonathan.adams@sun.com using -f User-Agent: Mutt/1.5.14 (2007-02-12) Status: RO Content-Length: 1182 On Wed, Nov 11, 2009 at 08:16:35PM +0100, ????? ???????????? wrote: > Why don't you rename the class to zfsiosched If the class is tailored > to support zfs only or kerneliosched if the class is tailored to > support kernel io only? Because it is not; it's designed for CPU-intensive kernel threads for which the default SYS class is inadequate. The fact that ZFS IO threads are the first ones to use it is because they can be fairly extreme CPU hogs. The name already reflects this; SDC (System Duty Cycle), modeled on SYS (System Class), which is also restricted to kernel threads only. > In my eyes its is a clear design fault to restrict such a class to > kernel threads only. You are locking out user land daemons which have > similar duties as the zfs threads you describe in this case. As an aside, userland daemons needing to run at system priorities can use the FX or RT classes to do so. The reason this is Consolidation-Private and kernel-only is that we are still evolving the exact details of how the class works. When there are more consumers, investigation into making it possible for user threads to join the class would be appropriate. Cheers, - jonathan From gdamore@sun.com Wed Nov 11 11:47:36 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nABJlZlc003476 for ; Wed, 11 Nov 2009 11:47:36 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id nABJlPg4008151 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 11 Nov 2009 19:47:35 GMT Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSY00C1DMZ9N400@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Nov 2009 11:47:33 -0800 (PST) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSY00CHFMZ7C900@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Nov 2009 11:47:31 -0800 (PST) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id nABJlVtR021456 for ; Wed, 11 Nov 2009 11:47:31 -0800 (PST) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KSY00L00MQWR600@fe-sfbay-10.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Nov 2009 11:47:31 -0800 (PST) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KSY00IZ7MZ6YYG0@fe-sfbay-10.sun.com>; Wed, 11 Nov 2009 11:47:31 -0800 (PST) Date: Wed, 11 Nov 2009 11:47:30 -0800 From: "Garrett D'Amore" Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: Sender: Garrett.Damore@sun.com To: =?windows-1252?Q?=3F=3F=3F=3F=3F_=3F=3F=3F=3F=3F=3F=3F=3F=3F=3F=3F=3F?= Cc: Jonathan Adams , Darren J Moffat , PSARC-ext@sun.com, Blake.Jones@sun.com Message-id: <4AFB14D2.3090906@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200911110000.nAB006i9028245@sac.sfbay.sun.com> <20091111185948.GA533685@eng.sun.com> User-Agent: Thunderbird 2.0.0.22 (X11/20090909) Status: RO Content-Length: 7760 ????? ???????????? wrote: > Why don't you rename the class to zfsiosched If the class is tailored > to support zfs only or kerneliosched if the class is tailored to > support kernel io only? > > In my eyes its is a clear design fault to restrict such a class to > kernel threads only. You are locking out user land daemons which have > similar duties as the zfs threads you describe in this case. > User threads already have a number of different scheduling classes available to them -- including even real-time scheduling, I don't see why they need this particular class to be available. As far as this case goes, I'd better like to understand why we needed a new class, instead of just using SYS. - Garrett > On Wed, Nov 11, 2009 at 7:59 PM, Jonathan Adams wrote: > >> On Wed, Nov 11, 2009 at 06:57:20PM +0100, ????? ???????????? wrote: >> >>> Why should this class only available for kernel threads? There are >>> daemons ported from micro kernel platforms which act like a kernel >>> daemon but run in user land and could benefit from such a scheduler >>> class. >>> >> Because it is modeled on the SYS class, which is also not available to >> user threads. Supporting user threads involves a bunch of additional >> work which isn't relevant to the problem at hand. >> >> Cheers, >> - jonathan >> >> >>> On Wed, Nov 11, 2009 at 1:00 AM, Darren J Moffat >>> wrote: >>> >>>> I'm submitting this as closed approved automatic, it has already been >>>> reviewed by core ZFS and kernel scheduling teams. Given the taxonomy >>>> of the new interfaces I believe this is sufficient review and closed >>>> approved automatic is sufficient. If any member disagrees then I'll >>>> set a fast-track timer. >>>> >>>> Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI >>>> This information is Copyright 2009 Sun Microsystems >>>> 1. Introduction >>>> 1.1. Project/Component Working Name: >>>> System Duty Cycle Scheduling Class and ZFS IO Observability >>>> 1.2. Name of Document Author/Supplier: >>>> Author: Jonathan Adams >>>> 1.3 Date of This Document: >>>> 10 November, 2009 >>>> 4. Technical Description >>>> >>>> This case covers two user-visible changes associated with the bug: >>>> >>>> 6806882 need a less brutal I/O scheduler >>>> >>>> The changes are: >>>> >>>> 1. A new kernel-thread-only scheduling class, SDC >>>> (System Duty Cycle) is introduced. Since the >>>> list of scheduling classes is user-visible through >>>> dispadmin(1M) and priocntl(1), we feel an ARC case >>>> is appropriate. >>>> >>>> The new scheduling class is only used by kernel threads, >>>> so the only manpage changes are a new SDC(7) page covering >>>> the existence of the new loadable module, and an update >>>> to the discussion of the SYS class in priocntl(2). >>>> >>>> A new function "sysdc_thread_enter()", puts a kernel thread >>>> into SDC. >>>> >>>> 2. In order to improve the observability of ZFS IO processing, >>>> we introduce a new system process per-zpool, with a >>>> u_comm and u_psargs of zpool- (i.e. zpool-foo). >>>> The LWPs in the process are kernel threads associated >>>> with IO processing. Since these threads have LWPs, they >>>> get microstate accounting data. Thus, CPU usage associated >>>> with IO processing can be observed using prstat(1M) and ps(1). >>>> >>>> This process will always be in the global zone, even if >>>> the pool was imported into a local zone. >>>> >>>> These changes, while user-visible, are essentially implementation >>>> details, and are not committed interfaces. We've tried to make this >>>> clear in the manpage changes. >>>> >>>> >>>> Interface Table >>>> --------------- >>>> >>>> Interface Stability >>>> --------------------------------------- ----------------- >>>> SDC module and scheduling class Consolidation Private >>>> sysdc_thread_enter() Consolidation Private >>>> zpool- process Volatile >>>> >>>> We seek Micro/Patch binding for these interfaces. >>>> >>>> See Also >>>> -------- >>>> >>>> 6806882 need a less brutal I/O scheduler >>>> >>>> >>>> Man Page Diffs >>>> -------------- >>>> >>>> SDC(7) >>>> +Device and Network Interfaces SDC(7) >>>> + >>>> + >>>> +NAME >>>> + SDC - System Duty Cycle scheduling class >>>> + >>>> +DESCRIPTION >>>> + The System Duty Cycle (SDC) scheduling class is used for some >>>> + CPU-intensive kernel thread workloads. Like the SYS class, >>>> + it cannot be used for user processes. >>>> + >>>> +ATTRIBUTES >>>> + See attributes(5) for descriptions of the following attri- >>>> + butes: >>>> + ____________________________________________________________ >>>> + | ATTRIBUTE TYPE | ATTRIBUTE VALUE | >>>> + |_____________________________|_____________________________| >>>> + >>>> +SEE ALSO >>>> + attributes(5) >>>> + >>>> >>>> priocntl(2) >>>> ... >>>> that class and/or when setting class-specific scheduling >>>> parameters. >>>> >>>> Two special scheduling classes, SYS and SDC, exist for the purpose of | >>>> scheduling the execution of certain special system processes >>>> (such as the swapper process). It is not possible to change >>>> the class of any LWP to SYS or SDC. In addition, any processes in | >>>> the SYS or SDC classes that are included in a specified set of | >>>> processes are disregarded by priocntl(). For example, an >>>> idtype of P_UID and an id value of 0 would specify all >>>> processes with a user ID of 0 except processes in the SYS and | >>>> SDC classes and (if changing the parameters using PC_SETPARMS or | >>>> PC_SETXPARMS) the init(1M) process. >>>> ... >>>> >>>> zpool(1M) >>>> --- zpool.1m.dist Tue Nov 3 14:00:53 2009 >>>> +++ zpool.1m Tue Nov 3 14:49:47 2009 >>>> @@ -947,6 +947,15 @@ >>>> version number, and only up to the most >>>> recent version supported by this software. >>>> >>>> +PROCESSES >>>> + Each imported pool has an associated process, named >>>> + zpool-poolname. The threads in this process are the >>>> + pool's IO processing threads, which handle the compression, >>>> + checksumming, and other tasks for all IO associated with the >>>> + pool. This process exists to provides visibility into the CPU >>>> + utilization of the system's storage pools. The existence >>>> + of this process is an unstable interface. >>>> + >>>> EXAMPLES >>>> Example 1 Creating a RAID-Z Storage Pool >>>> >>>> >>>> 6. Resources and Schedule >>>> 6.4. Steering Committee requested information >>>> 6.4.1. Consolidation C-team Name: >>>> On >>>> 6.5. ARC review type: Automatic >>>> 6.6. ARC Exposure: open >>>> >>>> _______________________________________________ >>>> opensolaris-arc mailing list >>>> opensolaris-arc@opensolaris.org >>>> >>>> >>> >>> -- >>> , _ _ , >>> { \/`o;====- Olga Kryzhanovska -====;o`\/ } >>> .----'-/`-/ olga.kryzhanovska@gmail.com \-`\-'----. >>> `'-..-| / Solaris/BSD//C/C++ programmer \ |-..-'` >>> /\/\ /\/\ >>> `--` `--` >>> > > > > From blakej@rift.sfbay.sun.com Wed Nov 11 13:07:55 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nABL7twh006666 for ; Wed, 11 Nov 2009 13:07:55 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id nABL7rKR042065 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 11 Nov 2009 14:07:54 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSY00N01QP62200@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Nov 2009 13:07:54 -0800 (PST) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSY00DDCQP56O80@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Nov 2009 13:07:53 -0800 (PST) Received: from rift.sfbay.sun.com (rift.SFBay.Sun.COM [129.146.228.163]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id nABL7rk1002808; Wed, 11 Nov 2009 13:07:53 -0800 (PST) Received: from rift (localhost [127.0.0.1]) by rift.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id nABL0NB3028272; Wed, 11 Nov 2009 13:00:23 -0800 (PST) Date: Wed, 11 Nov 2009 13:00:23 -0800 From: Blake Jones Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: "Your message of Wed, 11 Nov 2009 09:13:37 PST." <20091111171337.GA116375@eng.sun.com> Sender: blakej@rift.sfbay.sun.com To: James Carlson Cc: Jonathan Adams , Darren J Moffat , PSARC-ext@sun.com Reply-to: Blake Jones Message-id: <28271.1257973223@rift> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 926 Jonathan Adams wrote: >> Nifty; thanks! It sounds like it could end up being quite useful. > > The system process and associated LWPs are what give you observability; > the regular process framework handles reporting all of the CPU use and > microstate accounting data. The trade-off is the cost of microstate > accounting; currently, kernel threads don't have it. Once you have an > LWP, its recorded. > > The SDC class itself has no direct effect on observability; it is > entirely about CPU-intensive kernel threads "playing well with others". > SDC does require microstate accounting, though, which implies an LWP and > the attendant observability. Observability aside, we do expect that SDC will be useful for other kernel consumers over time. Higher-throughput I/O devices require more processing from the kernel, and the SYS class isn't designed to deal with threads hogging the CPU for extended periods. Blake From blakej@rift.sfbay.sun.com Wed Nov 11 13:43:07 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nABLh6B3007840 for ; Wed, 11 Nov 2009 13:43:06 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id nABLh486018284 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 12 Nov 2009 05:43:05 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSY00K0FSBS5000@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Nov 2009 13:43:04 -0800 (PST) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSY00CD2SBSCL80@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Nov 2009 13:43:04 -0800 (PST) Received: from rift.sfbay.sun.com (rift.SFBay.Sun.COM [129.146.228.163]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id nABLh37U026028; Wed, 11 Nov 2009 13:43:03 -0800 (PST) Received: from rift (localhost [127.0.0.1]) by rift.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id nABLD5iB028348; Wed, 11 Nov 2009 13:13:05 -0800 (PST) Date: Wed, 11 Nov 2009 13:13:05 -0800 From: Blake Jones Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: "Your message of Wed, 11 Nov 2009 11:38:55 PST." <20091111193855.GH123068@eng.sun.com> Sender: blakej@rift.sfbay.sun.com To: olga.kryzhanovska@gmail.com Cc: Jonathan Adams , Darren J Moffat , PSARC-ext@sun.com Reply-to: Blake Jones Message-id: <28347.1257973985@rift> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 616 >> In my eyes its is a clear design fault to restrict such a class to >> kernel threads only. You are locking out user land daemons which have >> similar duties as the zfs threads you describe in this case. > > As an aside, userland daemons needing to run at system priorities can > use the FX or RT classes to do so. Moreover, userland threads that want to use a certain percentage of the CPU may be able to use the FSS scheduling class. FSS makes some assumptions which caused us to decide to make a different class to solve our problem, but those assumptions may work for the workload you have in mind. Blake From Scott.Rotondo@sun.com Wed Nov 11 14:11:16 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nABMBFVn008873 for ; Wed, 11 Nov 2009 14:11:15 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id nABMBEXo027932 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 11 Nov 2009 14:11:15 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSY00L09TMRXP00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Nov 2009 14:11:15 -0800 (PST) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSY00C1XTMPCBA0@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Nov 2009 14:11:14 -0800 (PST) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id nABMBDUQ025619 for ; Wed, 11 Nov 2009 22:11:13 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KSY00M00SF0RF00@mail-amer.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Nov 2009 15:11:13 -0700 (MST) Received: from [129.146.108.62] ([unknown] [129.146.108.62]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KSY0074ZTMJBQ30@mail-amer.sun.com>; Wed, 11 Nov 2009 15:11:08 -0700 (MST) Date: Wed, 11 Nov 2009 14:11:07 -0800 From: Scott Rotondo Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: <4AFB14D2.3090906@sun.com> Sender: Scott.Rotondo@sun.com To: "Garrett D'Amore" Cc: ????? ???????????? , Jonathan Adams , Darren J Moffat , PSARC-ext@sun.com, Blake.Jones@sun.com Message-id: <4AFB367B.1040300@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200911110000.nAB006i9028245@sac.sfbay.sun.com> <20091111185948.GA533685@eng.sun.com> <4AFB14D2.3090906@sun.com> User-Agent: Thunderbird 2.0.0.21 (X11/20090323) Status: RO Content-Length: 842 Garrett D'Amore wrote: > As far as this case goes, I'd better like to understand why we needed > a new class, instead of just using SYS. That's the key question, which I haven't really seen anyone address in this PSARC email thread. However, you can find a high-level answer in CR 6806882: "The basic issue is that the SYS scheduling class was not designed for long-running CPU intensive workloads. We introduce a new SDC (System Duty Cycle) scheduling class, which adjusts its threads' priority in order to give them a specified percentage of the processor. It also gives them a scheduling quantum, which gives other threads at the same priority on the same CPU a chance to run." Scott -- Scott Rotondo Principal Engineer, Solaris Security Technologies President, Trusted Computing Group Phone/FAX: +1 408 850 3655 (Internal x68278) From jonathan.adams@sun.com Wed Nov 11 14:49:23 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nABMnM1v009704 for ; Wed, 11 Nov 2009 14:49:23 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id nABMnGFn024575; Thu, 12 Nov 2009 06:49:18 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSY0010FVE4B300@nwk-avmta-2.sfbay.sun.com>; Wed, 11 Nov 2009 14:49:16 -0800 (PST) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.63]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSY00CU6VE4CHD0@nwk-avmta-2.sfbay.sun.com>; Wed, 11 Nov 2009 14:49:16 -0800 (PST) Received: from jurassic-x4600.sfbay.sun.com (localhost [127.0.0.1]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id nABMnFu0627147 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 11 Nov 2009 14:49:15 -0800 (PST) Received: (from jwadams@localhost) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3/Submit) id nABMnFmv627145; Wed, 11 Nov 2009 14:49:15 -0800 (PST) Date: Wed, 11 Nov 2009 14:49:14 -0800 From: Jonathan Adams Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: <4AFB367B.1040300@sun.com> To: Scott Rotondo Cc: "Garrett D'Amore" , ????? ???????????? , Darren J Moffat , PSARC-ext@sun.com, Blake.Jones@sun.com Message-id: <20091111224914.GJ123068@eng.sun.com> MIME-version: 1.0 Content-type: multipart/mixed; boundary="Boundary_(ID_c6QFv283QcIO+i2GkGRW3g)" Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200911110000.nAB006i9028245@sac.sfbay.sun.com> <20091111185948.GA533685@eng.sun.com> <4AFB14D2.3090906@sun.com> <4AFB367B.1040300@sun.com> X-Authentication-warning: jurassic-x4600.sfbay.sun.com: jwadams set sender to jonathan.adams@sun.com using -f User-Agent: Mutt/1.5.14 (2007-02-12) Status: RO Content-Length: 6334 --Boundary_(ID_c6QFv283QcIO+i2GkGRW3g) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline On Wed, Nov 11, 2009 at 02:11:07PM -0800, Scott Rotondo wrote: > Garrett D'Amore wrote: > >As far as this case goes, I'd better like to understand why we needed > >a new class, instead of just using SYS. > > That's the key question, which I haven't really seen anyone address in > this PSARC email thread. However, you can find a high-level answer in CR > 6806882: > > "The basic issue is that the SYS scheduling class was not designed for > long-running CPU intensive workloads. We introduce a new SDC (System > Duty Cycle) scheduling class, which adjusts its threads' priority in > order to give them a specified percentage of the processor. > It also gives them a scheduling quantum, which gives other threads at > the same priority on the same CPU a chance to run." That being said, I'm looking into just having this be another mode the SYS class can run in, so that we don't pollute the namespace. Attached is the intro to the big theory statement from the project gate. Cheers, - jonathan --Boundary_(ID_c6QFv283QcIO+i2GkGRW3g) Content-type: text/plain; NAME=bigtheory; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: attachment; filename=bigtheory /* * The System Duty Cycle (SDC) scheduling class * -------------------------------------------- * * Background * * Kernel threads in Solaris have traditionally not been large consumers * of CPU time. They typically wake up, perform a small amount of * work, then go back to sleep waiting for either a timeout or another * signal. On the assumption that the small amount of work that they do * is important for the behavior of the whole system, these threads are * treated kindly by the dispatcher and the SYS scheduling class: they run * without preemption from anything other than real-time and interrupt * threads; when preempted, they are put at the front of the queue, so they * generally do not migrate between CPUs; and they are allowed to stay * running until they voluntarily give up the CPU. * * As Solaris has evolved, new workloads have emerged which require the * kernel to perform significant amounts of CPU-intensive work. One * example of such a workload is ZFS's transaction group sync processing. * Each sync operation generates a large batch of I/Os, and each I/O * may need to be compressed and/or checksummed before it is written to * storage. The taskq threads which perform the compression and checksums * will run nonstop as long as they have work to do; a large sync operation * on a compression-heavy dataset can keep them busy for seconds on end. * This causes human-time-scale dispatch latency bubbles for any other * threads which have the misfortune to share a CPU with the taskq threads. * * The SDC scheduling class is a solution to this problem. * * * Overview * * SDC is centered around the concept of a thread's duty cycle (DC): * * ONPROC time * Duty Cycle = ---------------------- * ONPROC + Runnable time * * This is the ratio of the time that the thread spent running on a CPU * divided by the time it spent running or trying to run. It is unaffected * by any time the thread spent sleeping, stopped, etc. * * A thread joining the SDC class specifies a "target" DC that it wants * to run at. To implement this policy, the routine sysdc_update() scans * the list of active SDC threads every few ticks and uses each thread's * microstate data to compute the actual duty cycle that that thread * has experienced recently. If the thread is under its target DC, its * priority is increased to the maximum available (sysdc_maxpri, which is * 99 by default). If the thread is over its target DC, its priority is * reduced to the minimum available (sysdc_minpri, 0 by default). This * is a fairly primitive approach, in that it doesn't use any of the * intermediate priorities, but it's not completely inappropriate. Even * though threads in the SDC class might take a while to do their job, they * are by some definition important if they're running inside the kernel, * so it is reasonable that they should get to run at priority 99. * * If a thread is running when sysdc_update() calculates its actual duty * cycle, and there are other threads of equal or greater priority on its * CPU's dispatch queue, sysdc_update() preempts that thread. The thread * acknowledges the preemption by calling sysdc_preempt(), which calls * setbackdq(), which gives other threads with the same priority a chance * to run. This creates a de facto time quantum for threads in the SDC * scheduling class. * * An SDC thread which is assigned priority 0 can continue to run if * nothing else needs to use the CPU that it's running on. Similarly, an * SDC thread at priority 99 might not get to run as much as it wants to * if there are other priority-99 or higher threads on its CPU. These * situations would cause the thread to get ahead of or behind its target * DC; the longer the situations lasted, the further ahead or behind the * thread would get. Rather than condemning a thread to a lifetime of * paying for its youthful indiscretions, SDC keeps "base" values for * ONPROC and Runnable times in each thread's sysdc data, and updates these * values periodically. The duty cycle is then computed using the elapsed * amount of ONPROC and Runnable times since those base times. * * Since sysdc_update() scans SDC threads fairly frequently, it tries to * keep the list of "active" threads small by pruning out threads which * have been asleep for a brief time. They are not pruned immediately upon * going to sleep, since some threads may bounce back and forth between * sleeping and being runnable. * * * Interfaces * * void sysdc_thread_enter(t, dc, flags) * * Moves a kernel thread from the SYS scheduling class to the * SDC class. t must have an associated LWP (created by calling * lwp_kernel_create()). The thread will have a target DC of dc. * Flags should be either 0 or SYSDC_THREAD_BATCH. If * SYSDC_THREAD_BATCH is specified, the thread will run with a * slightly lower priority (see "Batch threads", below). * --Boundary_(ID_c6QFv283QcIO+i2GkGRW3g)-- From gdamore@sun.com Wed Nov 11 14:52:18 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nABMqHhY009774 for ; Wed, 11 Nov 2009 14:52:17 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id nABMqGju012929 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 11 Nov 2009 14:52:17 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSY0010ZVJ4IV00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Nov 2009 14:52:16 -0800 (PST) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSY00C1AVJ4CJD0@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Nov 2009 14:52:16 -0800 (PST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id nABMqGGB010758 for ; Wed, 11 Nov 2009 14:52:16 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KSY00M00V5NVU00@fe-sfbay-09.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Nov 2009 14:52:16 -0800 (PST) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KSY00GG6VJ2KEC0@fe-sfbay-09.sun.com>; Wed, 11 Nov 2009 14:52:15 -0800 (PST) Date: Wed, 11 Nov 2009 14:52:14 -0800 From: "Garrett D'Amore" Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: <4AFB367B.1040300@sun.com> Sender: Garrett.Damore@sun.com To: Scott Rotondo Cc: ????? ???????????? , Jonathan Adams , Darren J Moffat , PSARC-ext@sun.com, Blake.Jones@sun.com Message-id: <4AFB401E.1040009@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200911110000.nAB006i9028245@sac.sfbay.sun.com> <20091111185948.GA533685@eng.sun.com> <4AFB14D2.3090906@sun.com> <4AFB367B.1040300@sun.com> User-Agent: Thunderbird 2.0.0.22 (X11/20090909) Status: RO Content-Length: 1025 Scott Rotondo wrote: > Garrett D'Amore wrote: >> As far as this case goes, I'd better like to understand why we needed >> a new class, instead of just using SYS. > > That's the key question, which I haven't really seen anyone address in > this PSARC email thread. However, you can find a high-level answer in > CR 6806882: > > "The basic issue is that the SYS scheduling class was not designed for > long-running CPU intensive workloads. We introduce a new SDC (System > Duty Cycle) scheduling class, which adjusts its threads' priority in > order to give them a specified percentage of the processor. > It also gives them a scheduling quantum, which gives other threads at > the same priority on the same CPU a chance to run." > > Scott > So my read of the above is that this is an attempt to give userland-like scheduling properties to kernel threads (perhaps somewhat like the FX class?). Which further underscores the fact that making this class available to user threads is probably pointless. - Garrett From gdamore@sun.com Wed Nov 11 14:56:38 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nABMubS1010207 for ; Wed, 11 Nov 2009 14:56:37 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id nABMuJi9008243 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 11 Nov 2009 22:56:36 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSY00E03VQA4T00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Nov 2009 14:56:34 -0800 (PST) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSY00CDTVQ9E410@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Nov 2009 14:56:34 -0800 (PST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id nABMuXso016607 for ; Wed, 11 Nov 2009 14:56:33 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KSY00D00VPQR900@fe-sfbay-09.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 11 Nov 2009 14:56:33 -0800 (PST) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KSY00GYWVQ8KED0@fe-sfbay-09.sun.com>; Wed, 11 Nov 2009 14:56:33 -0800 (PST) Date: Wed, 11 Nov 2009 14:56:32 -0800 From: "Garrett D'Amore" Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: <20091111224914.GJ123068@eng.sun.com> Sender: Garrett.Damore@sun.com To: Jonathan Adams Cc: Scott Rotondo , ????? ???????????? , Darren J Moffat , PSARC-ext@sun.com, Blake.Jones@sun.com Message-id: <4AFB4120.1070504@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200911110000.nAB006i9028245@sac.sfbay.sun.com> <20091111185948.GA533685@eng.sun.com> <4AFB14D2.3090906@sun.com> <4AFB367B.1040300@sun.com> <20091111224914.GJ123068@eng.sun.com> User-Agent: Thunderbird 2.0.0.22 (X11/20090909) Status: RO Content-Length: 1159 Jonathan Adams wrote: > On Wed, Nov 11, 2009 at 02:11:07PM -0800, Scott Rotondo wrote: > >> Garrett D'Amore wrote: >> >>> As far as this case goes, I'd better like to understand why we needed >>> a new class, instead of just using SYS. >>> >> That's the key question, which I haven't really seen anyone address in >> this PSARC email thread. However, you can find a high-level answer in CR >> 6806882: >> >> "The basic issue is that the SYS scheduling class was not designed for >> long-running CPU intensive workloads. We introduce a new SDC (System >> Duty Cycle) scheduling class, which adjusts its threads' priority in >> order to give them a specified percentage of the processor. >> It also gives them a scheduling quantum, which gives other threads at >> the same priority on the same CPU a chance to run." >> > > That being said, I'm looking into just having this be another mode the SYS > class can run in, so that we don't pollute the namespace. > > Attached is the intro to the big theory statement from the project gate. > > Cheers, > - jonathan > > Thanks, this makes the case much easier to understand. - Garrett From jonathan.adams@Sun.COM Wed Nov 11 14:57:46 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nABMvjGP010281 for ; Wed, 11 Nov 2009 14:57:45 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id nABMvS6j028079; Thu, 12 Nov 2009 06:57:41 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSY00E05VS39Z00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 11 Nov 2009 14:57:39 -0800 (PST) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSY00CJUVS3E710@nwk-avmta-1.sfbay.Sun.COM>; Wed, 11 Nov 2009 14:57:39 -0800 (PST) Received: from jurassic-x4600.sfbay.sun.com (localhost [127.0.0.1]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id nABMvc3l628533 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 11 Nov 2009 14:57:38 -0800 (PST) Received: (from jwadams@localhost) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3/Submit) id nABMvYs2628363; Wed, 11 Nov 2009 14:57:34 -0800 (PST) Date: Wed, 11 Nov 2009 14:55:39 -0800 From: Jonathan Adams Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: <4AFB401E.1040009@sun.com> To: "Garrett D'Amore" Cc: Scott Rotondo , ????? ???????????? , Darren J Moffat , PSARC-ext@Sun.COM, Blake.Jones@Sun.COM Message-id: <20091111225539.GK123068@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200911110000.nAB006i9028245@sac.sfbay.sun.com> <20091111185948.GA533685@eng.sun.com> <4AFB14D2.3090906@sun.com> <4AFB367B.1040300@sun.com> <4AFB401E.1040009@sun.com> X-Authentication-warning: jurassic-x4600.sfbay.sun.com: jwadams set sender to jonathan.adams@sun.com using -f User-Agent: Mutt/1.5.14 (2007-02-12) Status: RO Content-Length: 1337 On Wed, Nov 11, 2009 at 02:52:14PM -0800, Garrett D'Amore wrote: > Scott Rotondo wrote: > >Garrett D'Amore wrote: > >>As far as this case goes, I'd better like to understand why we needed > >>a new class, instead of just using SYS. > > > >That's the key question, which I haven't really seen anyone address in > >this PSARC email thread. However, you can find a high-level answer in > >CR 6806882: > > > >"The basic issue is that the SYS scheduling class was not designed for > >long-running CPU intensive workloads. We introduce a new SDC (System > >Duty Cycle) scheduling class, which adjusts its threads' priority in > >order to give them a specified percentage of the processor. > >It also gives them a scheduling quantum, which gives other threads at > >the same priority on the same CPU a chance to run." > > > > Scott > > > So my read of the above is that this is an attempt to give userland-like > scheduling properties to kernel threads (perhaps somewhat like the FX > class?). Which further underscores the fact that making this class > available to user threads is probably pointless. See the big theory statement I attached in response; basically, we lower the priority to 0 whenever we're ahead of our duty cycle, which gives user threads a chance to use the CPU, preventing latency bubbles. Cheers, - jonathan From jonathan.adams@sun.com Thu Nov 12 16:21:48 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nAD0Lll9024792 for ; Thu, 12 Nov 2009 16:21:48 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id nAD0Lfub000536; Fri, 13 Nov 2009 00:21:44 GMT Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KT00010HUC8FB00@brm-avmta-1.central.sun.com>; Thu, 12 Nov 2009 17:21:44 -0700 (MST) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KT0008FKUC7EFA0@brm-avmta-1.central.sun.com>; Thu, 12 Nov 2009 17:21:43 -0700 (MST) Received: from jurassic-x4600.sfbay.sun.com (localhost [127.0.0.1]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id nAD0LguQ920696 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 12 Nov 2009 16:21:42 -0800 (PST) Received: (from jwadams@localhost) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3/Submit) id nAD0LgOg920694; Thu, 12 Nov 2009 16:21:42 -0800 (PST) Date: Thu, 12 Nov 2009 16:21:42 -0800 From: Jonathan Adams Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: <20091111224914.GJ123068@eng.sun.com> To: Scott Rotondo Cc: "Garrett D'Amore" , ????? ???????????? , Darren J Moffat , PSARC-ext@sun.com, Blake.Jones@sun.com Message-id: <20091113002142.GP123068@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200911110000.nAB006i9028245@sac.sfbay.sun.com> <20091111185948.GA533685@eng.sun.com> <4AFB14D2.3090906@sun.com> <4AFB367B.1040300@sun.com> <20091111224914.GJ123068@eng.sun.com> X-Authentication-warning: jurassic-x4600.sfbay.sun.com: jwadams set sender to jonathan.adams@sun.com using -f User-Agent: Mutt/1.5.14 (2007-02-12) Status: RO Content-Length: 1321 On Wed, Nov 11, 2009 at 02:49:14PM -0800, Jonathan Adams wrote: > On Wed, Nov 11, 2009 at 02:11:07PM -0800, Scott Rotondo wrote: > > Garrett D'Amore wrote: > > >As far as this case goes, I'd better like to understand why we needed > > >a new class, instead of just using SYS. > > > > That's the key question, which I haven't really seen anyone address in > > this PSARC email thread. However, you can find a high-level answer in CR > > 6806882: > > > > "The basic issue is that the SYS scheduling class was not designed for > > long-running CPU intensive workloads. We introduce a new SDC (System > > Duty Cycle) scheduling class, which adjusts its threads' priority in > > order to give them a specified percentage of the processor. > > It also gives them a scheduling quantum, which gives other threads at > > the same priority on the same CPU a chance to run." > > That being said, I'm looking into just having this be another mode the SYS > class can run in, so that we don't pollute the namespace. After investigating this and talking with Eric Saxe, we decided that this would only confuse things; the reason we have different scheduling classes is to classify the scheduling behavior. Making sysdc be just a mode of SYS would run counter to this. So we're leaving the case as written. Cheers, - jonathan From jonathan.adams@sun.com Fri Nov 13 13:30:12 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nAD0Lll9024792 for ; Thu, 12 Nov 2009 16:21:48 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id nAD0Lfub000536; Fri, 13 Nov 2009 00:21:44 GMT Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KT00010HUC8FB00@brm-avmta-1.central.sun.com>; Thu, 12 Nov 2009 17:21:44 -0700 (MST) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KT0008FKUC7EFA0@brm-avmta-1.central.sun.com>; Thu, 12 Nov 2009 17:21:43 -0700 (MST) Received: from jurassic-x4600.sfbay.sun.com (localhost [127.0.0.1]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id nAD0LguQ920696 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 12 Nov 2009 16:21:42 -0800 (PST) Received: (from jwadams@localhost) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3/Submit) id nAD0LgOg920694; Thu, 12 Nov 2009 16:21:42 -0800 (PST) Date: Thu, 12 Nov 2009 16:21:42 -0800 From: Jonathan Adams Subject: Re: System Duty Cycle Scheduling Class and ZFS IO Observability [PSARC/2009/615 Self Review] In-reply-to: <20091111224914.GJ123068@eng.sun.com> To: Scott Rotondo Cc: "Garrett D'Amore" , ????? ???????????? , Darren J Moffat , PSARC-ext@sun.com, Blake.Jones@sun.com Message-id: <20091113002142.GP123068@eng.sun.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200911110000.nAB006i9028245@sac.sfbay.sun.com> <20091111185948.GA533685@eng.sun.com> <4AFB14D2.3090906@sun.com> <4AFB367B.1040300@sun.com> <20091111224914.GJ123068@eng.sun.com> X-Authentication-warning: jurassic-x4600.sfbay.sun.com: jwadams set sender to jonathan.adams@sun.com using -f User-Agent: Mutt/1.5.14 (2007-02-12) Status: RO Content-Length: 1321 On Wed, Nov 11, 2009 at 02:49:14PM -0800, Jonathan Adams wrote: > On Wed, Nov 11, 2009 at 02:11:07PM -0800, Scott Rotondo wrote: > > Garrett D'Amore wrote: > > >As far as this case goes, I'd better like to understand why we needed > > >a new class, instead of just using SYS. > > > > That's the key question, which I haven't really seen anyone address in > > this PSARC email thread. However, you can find a high-level answer in CR > > 6806882: > > > > "The basic issue is that the SYS scheduling class was not designed for > > long-running CPU intensive workloads. We introduce a new SDC (System > > Duty Cycle) scheduling class, which adjusts its threads' priority in > > order to give them a specified percentage of the processor. > > It also gives them a scheduling quantum, which gives other threads at > > the same priority on the same CPU a chance to run." > > That being said, I'm looking into just having this be another mode the SYS > class can run in, so that we don't pollute the namespace. After investigating this and talking with Eric Saxe, we decided that this would only confuse things; the reason we have different scheduling classes is to classify the scheduling behavior. Making sysdc be just a mode of SYS would run counter to this. So we're leaving the case as written. Cheers, - jonathan