From sacadmin Thu Mar 12 08:21:13 2009 Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CFLD1p007632; Thu, 12 Mar 2009 08:21:13 -0700 (PDT) Received: (from casper@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id n2CFLDVR007628; Thu, 12 Mar 2009 08:21:13 -0700 (PDT) Date: Thu, 12 Mar 2009 08:21:13 -0700 (PDT) From: Casper Dik Message-Id: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> To: PSARC@sac.sfbay.sun.com Cc: Casper.Dik@Sun.COM Subject: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] Status: RO Content-Length: 3237 Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI Copyright 2009 Sun Microsystems, Inc. 1. Introduction 1.1. Project/Component Working Name: Fasttrack for turbo-charging SVr4 package install. 1.2. Name of Document Author/Supplier: Author: Casper Dik 1.3 Date of This Document: 12 March, 2009 4. Technical Description Fasttrack for turbo-charging SVr4 package install. Release binding: patch/micro Our customers using Solaris 10 have several problems: - installing patches is slow - liveupgrade is slow - upgrade is slow - liveupgrade with zones is even slower. - installing zones is slow Much of this is caused by the contents database /var/sadm/install/contents. Most of the procedures listed above cause around 50-150GB of I/O to the contents file. Earlier attempts to get rid of the file didn't pan out. This project proposes to keep the contents file but delay writing it. The contents file is managed by a daemon. Package commands no longer manipulate the contents file themselves, they call the daemon. The pkgserv daemon stores all modifications in a logfile (/var/sadm/install/pkglog). The "virtual contents file" is the on-disk "contents" file and the on-disk "pkglog" file. The daemon will lock out older versions of the package commands by grabbing the contents lock file. (Older package commands can run again after the daemon has exited) The fasttrack adds the following commands/subcommands: /usr/sadm/install/bin/pkgserv - Consolidation private - invoked automatically by the pkgcmds. /usr/bin/pkgadm sync [-R root] [-q] Writes the contents file and rolls the contents log file. Optionally forces the contents file server to quit [-q]. - invoked automatically by the pkgcmds, lu commands (such as luumount), install, upgrade, liveupgrade, zoneadm install /var/svc/manifest/system/pkgserv.xml - defines svc:/system/pkgserv:default - runs "pkgadm sync -q" at boot and shutdown time. - write the contents file if needed, picks up the pieces if pkgserv is killed, makes sure the pkgserv exits. The new implementation uses at most as much memory (both real and reserved) as the current implementation, but typically less. Each zone will run its own pkgserv. The daemon will automatically flush the contents file and it will shut down. (By default, the contents file is written within minutes after the last package commands exits) The performance change is considerable: install 25-50% (1 1/3-2 as fast) upgrade 50-75% (2 - 4 times faster) liveupgrade 50-75% (w/ or w/o zones) (higher gain with more zones) zone install 50-75% Security: the daemon verifies the caller's identity using door_ucred(). Users which cannot modify the contents database will run their own read-only version of the daemon. It is possible to back out the new pkgcmds; the contents file stays as before, and after the daemon is stopped, the older package commands can be reinstalled. 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: admininstall 6.5. ARC review type: FastTrack 6.6. ARC Exposure: closed 6.6.1. Rationale: It's about Solaris 10 sustaining (mostly) From sacadmin Thu Mar 12 08:32:02 2009 Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CFW1Fx007952; Thu, 12 Mar 2009 08:32:02 -0700 (PDT) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n2CFVxZO016617; Thu, 12 Mar 2009 11:31:59 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n2CFVxGC016614; Thu, 12 Mar 2009 11:31:59 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18873.10991.72939.781690@gargle.gargle.HOWL> Date: Thu, 12 Mar 2009 11:31:59 -0400 From: James Carlson To: Casper Dik Cc: PSARC@sac.sfbay.sun.com, Casper.Dik@sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-Reply-To: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 972 Casper Dik writes: > The pkgserv daemon stores all modifications in a logfile > (/var/sadm/install/pkglog). The "virtual contents file" is > the on-disk "contents" file and the on-disk "pkglog" file. I assume that if we boot up and find a non-empty pkglog, then we'll take the time to sync up the contents file (playing the log) at that point, right? > 6.6. ARC Exposure: closed > 6.6.1. Rationale: It's about Solaris 10 sustaining (mostly) This all sounds great, but I'm not sure I agree with the above statements. The old packaging commands aren't going away, and there are other distributions besides Sun's that rely on them in Nevada. Is there a real reason to make this closed, other than the potential embarrassment factor? -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From sacadmin Thu Mar 12 08:43:19 2009 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CFhJDo008091; Thu, 12 Mar 2009 08:43:19 -0700 (PDT) Received: from gmp-eb-inf-2.sun.com (gmp-eb-inf-2.EU.Sun.COM [192.18.6.24]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2CFhIYi061931; Thu, 12 Mar 2009 08:43:18 -0700 (PDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-2-fe3.eu.sun.com [192.18.6.12]) by gmp-eb-inf-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n2CFhC5T023682; Thu, 12 Mar 2009 15:43:12 GMT MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; charset=ISO-8859-15 Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KGE00J00GMVLX00@fe-emea-10.sun.com>; Thu, 12 Mar 2009 15:43:12 +0000 (GMT) Received: from [129.157.207.210] ([unknown] [129.157.207.210]) by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KGE005CRGZLNN10@fe-emea-10.sun.com>; Thu, 12 Mar 2009 15:42:57 +0000 (GMT) Date: Thu, 12 Mar 2009 16:43:07 +0100 From: Renaud Manus Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-reply-to: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> Sender: Renaud.Manus@Sun.COM To: Casper Dik Cc: PSARC@sac.sfbay.sun.com, Casper.Dik@Sun.COM Message-id: <49B92D8B.1080900@sun.com> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.18 (X11/20090127) Status: RO Content-Length: 287 Casper Dik wrote: > The pkgserv daemon stores all modifications in a logfile > (/var/sadm/install/pkglog). The "virtual contents file" is > the on-disk "contents" file and the on-disk "pkglog" file. Shouldn't the logfile be specified as $PKG_ROOT/var/sadm/install/pkglog ? -- Renaud From sacadmin Thu Mar 12 08:45:17 2009 Received: from dm-holland-02.uk.sun.com (dm-holland-02.UK.Sun.COM [129.156.101.225]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CFjGNO008115 for ; Thu, 12 Mar 2009 08:45:17 -0700 (PDT) Received: from holland (room101.Holland.Sun.COM [10.16.117.40]) by dm-holland-02.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2CFjFkI060189; Thu, 12 Mar 2009 15:45:15 GMT Message-Id: <200903121545.n2CFjFkI060189@dm-holland-02.uk.sun.com> From: Casper.Dik@sun.com To: James Carlson cc: PSARC@sac.sfbay.sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-Reply-To: <18873.10991.72939.781690@gargle.gargle.HOWL> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> <18873.10991.72939.781690@gargle.gargle.HOWL> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 12 Mar 2009 16:45:15 +0100 Sender: casper@holland.sun.com Status: RO Content-Length: 890 >Casper Dik writes: >> The pkgserv daemon stores all modifications in a logfile >> (/var/sadm/install/pkglog). The "virtual contents file" is >> the on-disk "contents" file and the on-disk "pkglog" file. > >I assume that if we boot up and find a non-empty pkglog, then we'll >take the time to sync up the contents file (playing the log) at that >point, right? Correct. >> 6.6. ARC Exposure: closed >> 6.6.1. Rationale: It's about Solaris 10 sustaining (mostly) > >This all sounds great, but I'm not sure I agree with the above >statements. The old packaging commands aren't going away, and there >are other distributions besides Sun's that rely on them in Nevada. > >Is there a real reason to make this closed, other than the potential >embarrassment factor? I don't really case all that much, but I figured it was sufficiently like PSARC/2008/644 to copy it. Casper From sacadmin Thu Mar 12 08:50:26 2009 Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CFoQfD008226 for ; Thu, 12 Mar 2009 08:50:26 -0700 (PDT) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n2CFoNvb018458; Thu, 12 Mar 2009 11:50:23 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n2CFoNs7018451; Thu, 12 Mar 2009 11:50:23 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18873.12080.589318.350699@gargle.gargle.HOWL> Date: Thu, 12 Mar 2009 11:50:08 -0400 From: James Carlson To: Casper.Dik@sun.com Cc: PSARC@sac.sfbay.sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-Reply-To: <200903121545.n2CFjFkI060189@dm-holland-02.uk.sun.com> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> <18873.10991.72939.781690@gargle.gargle.HOWL> <200903121545.n2CFjFkI060189@dm-holland-02.uk.sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 639 Casper.Dik@Sun.COM writes: > >Is there a real reason to make this closed, other than the potential > >embarrassment factor? > > > I don't really case all that much, but I figured it was sufficiently like > PSARC/2008/644 to copy it. It's not, though. 2008/644 was about the patching tools, which don't (and due to IPS won't) exist in Nevada. This case is about the packaging tools, which *do* exist in Nevada. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From sacadmin Thu Mar 12 09:00:46 2009 Received: from dm-holland-02.uk.sun.com (dm-holland-02.UK.Sun.COM [129.156.101.225]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CG0jWm008915; Thu, 12 Mar 2009 09:00:45 -0700 (PDT) Received: from holland (room101.Holland.Sun.COM [10.16.117.40]) by dm-holland-02.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2CG0iO4065445; Thu, 12 Mar 2009 16:00:44 GMT Message-Id: <200903121600.n2CG0iO4065445@dm-holland-02.uk.sun.com> From: Casper.Dik@sun.com To: Renaud Manus cc: Casper Dik , PSARC@sac.sfbay.sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-Reply-To: <49B92D8B.1080900@sun.com> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> <49B92D8B.1080900@sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 12 Mar 2009 17:00:44 +0100 Sender: casper@holland.sun.com Status: RO Content-Length: 612 >Casper Dik wrote: >> The pkgserv daemon stores all modifications in a logfile >> (/var/sadm/install/pkglog). The "virtual contents file" is >> the on-disk "contents" file and the on-disk "pkglog" file. > >Shouldn't the logfile be specified as > $PKG_ROOT/var/sadm/install/pkglog ? It's in the same directory where the contents file is, yes. (But earlier on in the text I refer to the contents file as "/var/sadm/install/contents") BTW, I think you wanted to say "$PKG_INSTALL_ROOT". There's one pkgserv per copy of the contents file and there's one pkglog, if needed, next to that contents file. Casper From sacadmin Thu Mar 12 09:17:24 2009 Received: from binky.Central.Sun.COM (binky.Central.Sun.COM [129.153.128.104]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CGHOLq009287; Thu, 12 Mar 2009 09:17:24 -0700 (PDT) Received: from binky.Central.Sun.COM (localhost [127.0.0.1]) by binky.Central.Sun.COM (8.14.3+Sun/8.14.3) with ESMTP id n2CGFkif022257; Thu, 12 Mar 2009 11:15:46 -0500 (CDT) Received: (from nw141292@localhost) by binky.Central.Sun.COM (8.14.3+Sun/8.14.3/Submit) id n2CGFkcn022256; Thu, 12 Mar 2009 11:15:46 -0500 (CDT) X-Authentication-Warning: binky.Central.Sun.COM: nw141292 set sender to Nicolas.Williams@sun.com using -f Date: Thu, 12 Mar 2009 11:15:46 -0500 From: Nicolas Williams To: Casper Dik Cc: PSARC@sac.sfbay.sun.com, Casper.Dik@sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] Message-ID: <20090312161546.GC9992@Sun.COM> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> User-Agent: Mutt/1.5.7i Status: RO Content-Length: 746 On Thu, Mar 12, 2009 at 08:21:13AM -0700, Casper Dik wrote: > Security: the daemon verifies the caller's identity using door_ucred(). But what is it checking? That the caller has euid == 0? > Users which cannot modify the contents database will run their own read-only > version of the daemon. That would be for pkginfo(1)/pkgchk(1)? And only if the daemon's not already running? Does a read-only pkgserv "tail" the log to keep an in-memory copy of the contents file up to date? Where are the doors for the privileged and read-only daemons placed? For the former I expect /var/run/ (is that a namespace that the ARC controls, so that the door name needs to be documented?). For the latter, what, $TMPDIR/? Nico -- From sacadmin Thu Mar 12 09:34:27 2009 Received: from paravon.sfbay.sun.com (paravon.SFBay.Sun.COM [192.168.83.160]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CGYRd4010574; Thu, 12 Mar 2009 09:34:27 -0700 (PDT) Received: from paravon.sfbay.sun.com (localhost [127.0.0.1]) by paravon.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CGYRu8019301; Thu, 12 Mar 2009 09:34:27 -0700 (PDT) Received: (from johnf@localhost) by paravon.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id n2CGYRqO019300; Thu, 12 Mar 2009 09:34:27 -0700 (PDT) Date: Thu, 12 Mar 2009 09:34:27 -0700 (PDT) From: John Fischer Message-Id: <200903121634.n2CGYRqO019300@paravon.sfbay.sun.com> To: casper.dik@sac.sfbay.sun.com Cc: PSARC@sac.sfbay.sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] Status: RO Content-Length: 515 Casper, Since this is about the packaging tools/installer and we are talking about a Patch release of Solaris has this project been funded by the PAC? As you are well aware there are no more S10 installer changes allowed within the Updates. What are your plans with regards to the PAC? I know not strictly Architectural but ... By the way this is the issue (installer changes) that the Solaris 10 Update P-Team is upset at the committees for not taking into consideration prior to approval. Thanks, John From sacadmin Thu Mar 12 09:34:49 2009 Received: from dm-holland-02.uk.sun.com (dm-holland-02.UK.Sun.COM [129.156.101.225]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CGYmtb010644; Thu, 12 Mar 2009 09:34:49 -0700 (PDT) Received: from holland (room101.Holland.Sun.COM [10.16.117.40]) by dm-holland-02.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2CGYleI012475; Thu, 12 Mar 2009 16:34:47 GMT Message-Id: <200903121634.n2CGYleI012475@dm-holland-02.uk.sun.com> From: Casper.Dik@sun.com To: Nicolas Williams cc: Casper Dik , PSARC@sac.sfbay.sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-Reply-To: <20090312161546.GC9992@Sun.COM> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> <20090312161546.GC9992@Sun.COM> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 12 Mar 2009 17:34:47 +0100 Sender: casper@holland.sun.com Status: RO Content-Length: 1213 >On Thu, Mar 12, 2009 at 08:21:13AM -0700, Casper Dik wrote: >> Security: the daemon verifies the caller's identity using door_ucred(). > >But what is it checking? That the caller has euid == 0? That the caller's id matches the id of the pkgserv. >> Users which cannot modify the contents database will run their own read-only >> version of the daemon. > >That would be for pkginfo(1)/pkgchk(1)? And only if the daemon's not >already running? They'll start one. >Does a read-only pkgserv "tail" the log to keep an in-memory copy of the >contents file up to date? It re-creates the virtual contents file and uses that. >Where are the doors for the privileged and read-only daemons placed? >For the former I expect /var/run/ (is that a namespace that >the ARC controls, so that the door name needs to be documented?). For >the latter, what, $TMPDIR/? The .door file is created in the same directory as the contents file. This is required because many pkgserv can run inside the same system instance (e.g., when creating a zone) The "temporary" read-only pkgservs create a .door in /tmp. This is removed when it is no longer needed by the temporary pkgserv when it exits. Casper From sacadmin Thu Mar 12 09:38:10 2009 Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CGcAcN011188; Thu, 12 Mar 2009 09:38:10 -0700 (PDT) Received: (from gww@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id n2CGcAIP011187; Thu, 12 Mar 2009 09:38:10 -0700 (PDT) Date: Thu, 12 Mar 2009 09:38:10 -0700 (PDT) From: Gary Winiger Message-Id: <200903121638.n2CGcAIP011187@sac.sfbay.sun.com> To: Casper.Dik@sun.com, Renaud.Manus@sun.com Cc: PSARC@sac.sfbay.sun.com, casper@sac.sfbay.sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] Status: RO Content-Length: 312 > >Shouldn't the logfile be specified as > > $PKG_ROOT/var/sadm/install/pkglog ? > > It's in the same directory where the contents file is, yes. > (But earlier on in the text I refer to the contents file as > "/var/sadm/install/contents") > > > BTW, I think you wanted to say "$PKG_INSTALL_ROOT". +1 Gary.. From sacadmin Thu Mar 12 09:45:11 2009 Received: from binky.Central.Sun.COM (binky.Central.Sun.COM [129.153.128.104]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CGjAJK011830; Thu, 12 Mar 2009 09:45:11 -0700 (PDT) Received: from binky.Central.Sun.COM (localhost [127.0.0.1]) by binky.Central.Sun.COM (8.14.3+Sun/8.14.3) with ESMTP id n2CGhW98022281; Thu, 12 Mar 2009 11:43:32 -0500 (CDT) Received: (from nw141292@localhost) by binky.Central.Sun.COM (8.14.3+Sun/8.14.3/Submit) id n2CGhWMW022280; Thu, 12 Mar 2009 11:43:32 -0500 (CDT) X-Authentication-Warning: binky.Central.Sun.COM: nw141292 set sender to Nicolas.Williams@sun.com using -f Date: Thu, 12 Mar 2009 11:43:32 -0500 From: Nicolas Williams To: Casper.Dik@sun.com Cc: Casper Dik , PSARC@sac.sfbay.sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] Message-ID: <20090312164332.GD9992@Sun.COM> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> <20090312161546.GC9992@Sun.COM> <200903121634.n2CGYleI012475@dm-holland-02.uk.sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200903121634.n2CGYleI012475@dm-holland-02.uk.sun.com> User-Agent: Mutt/1.5.7i Status: RO Content-Length: 427 On Thu, Mar 12, 2009 at 05:34:47PM +0100, Casper.Dik@Sun.COM wrote: > The "temporary" read-only pkgservs create a .door in /tmp. This is removed > when it is no longer needed by the temporary pkgserv when it exits. Since different users could be creating read-only pkgservs presumably each one or each user has its own door file name. The read-only pkg commands should check that this door is really owned by the same user. From sacadmin Thu Mar 12 10:03:55 2009 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CH3tTf012808; Thu, 12 Mar 2009 10:03:55 -0700 (PDT) Received: from gmp-eb-inf-1.sun.com (gmp-eb-inf-1.EU.Sun.COM [192.18.6.21]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2CH3s0J062651; Thu, 12 Mar 2009 10:03:54 -0700 (PDT) Received: from fe-emea-09.sun.com (gmp-eb-lb-1-fe3.eu.sun.com [192.18.6.10]) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n2CH3mp7025135; Thu, 12 Mar 2009 17:03:48 GMT MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; charset=ISO-8859-1 Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KGE00J00JR06U00@fe-emea-09.sun.com>; Thu, 12 Mar 2009 17:03:48 +0000 (GMT) Received: from [129.157.207.210] ([unknown] [129.157.207.210]) by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KGE00K47KQBU150@fe-emea-09.sun.com>; Thu, 12 Mar 2009 17:03:48 +0000 (GMT) Date: Thu, 12 Mar 2009 18:03:57 +0100 From: Renaud Manus Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-reply-to: <200903121600.n2CG0iO4065445@dm-holland-02.uk.sun.com> Sender: Renaud.Manus@Sun.COM To: Casper.Dik@Sun.COM Cc: Casper Dik , PSARC@sac.sfbay.sun.com Message-id: <49B9407D.8050702@sun.com> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> <49B92D8B.1080900@sun.com> <200903121600.n2CG0iO4065445@dm-holland-02.uk.sun.com> User-Agent: Thunderbird 2.0.0.18 (X11/20090127) Status: RO Content-Length: 801 Casper.Dik@Sun.COM wrote: >> Casper Dik wrote: >>> The pkgserv daemon stores all modifications in a logfile >>> (/var/sadm/install/pkglog). The "virtual contents file" is >>> the on-disk "contents" file and the on-disk "pkglog" file. >> Shouldn't the logfile be specified as >> $PKG_ROOT/var/sadm/install/pkglog ? > > It's in the same directory where the contents file is, yes. > (But earlier on in the text I refer to the contents file as > "/var/sadm/install/contents") > > > BTW, I think you wanted to say "$PKG_INSTALL_ROOT". correct > > There's one pkgserv per copy of the contents file and there's one > pkglog, if needed, next to that contents file. Well, if zones aren't running, it seems that you can have one pkgserv daemon for multiple copy of the contents file, no ? -- Renaud From sacadmin Thu Mar 12 10:36:15 2009 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CHaFZi014088; Thu, 12 Mar 2009 10:36:15 -0700 (PDT) Received: from gmp-eb-inf-1.sun.com (gmp-eb-inf-1.EU.Sun.COM [192.18.6.21]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2CHaEDt032864; Thu, 12 Mar 2009 10:36:15 -0700 (PDT) Received: from fe-emea-09.sun.com (gmp-eb-lb-2-fe2.eu.sun.com [192.18.6.11]) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n2CHa9De028691; Thu, 12 Mar 2009 17:36:09 GMT MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; charset=ISO-8859-1 Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KGE00000LK2XJ00@fe-emea-09.sun.com>; Thu, 12 Mar 2009 17:36:09 +0000 (GMT) Received: from [129.156.173.239] ([unknown] [129.156.173.239]) by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KGE00EEUM82GR30@fe-emea-09.sun.com>; Thu, 12 Mar 2009 17:36:05 +0000 (GMT) Date: Thu, 12 Mar 2009 17:36:02 +0000 From: Brian Ruthven - Sun UK Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-reply-to: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> Sender: Brian.Ruthven@Sun.COM To: Casper Dik Cc: PSARC@sac.sfbay.sun.com, Casper.Dik@Sun.COM Message-id: <49B94802.7040706@sun.com> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.9 (X11/20071119) Status: RO Content-Length: 1474 Hi Casper, Is single-user mode catered for? (or doesn't matter?) Is the "-R" option to pkgadd and friends catered for? In another thread, there was mention of a check that "That the caller's id matches the id of the pkgserv". Will this work with multiple privileged but different users? i.e. if there are two admins for the box which both had the appropriate privilege to apply packages, would the second one have to wait until the deamon started by the first had exited? Under what conditions does the daemon exit? Is the daemon purely started on demand on first invocation of a pkg command? Casper Dik wrote: > It is possible to back out the new pkgcmds; the contents file stays as before, > and after the daemon is stopped, the older package commands can be reinstalled. > Presumably this is simply saying that this is an implementation change, and that the pkgadd, pkgrm, patchadd, etc... UI experience remains the same (albeit sped up somewhat)? If so, then I would agree that customers are crying out for this kind of speed bump. There are still stories of multi-hour patching sessions, especially with zones involved. Will this case be compatible with the parallelism of PSARC/2008/644 (assuming it gets/has been implemented)? Thanks, Brian -- Brian Ruthven Sun Microsystems UK Solaris Revenue Product Engineering Tel: +44 (0)1252 422 312 Sparc House, Guillemont Park, Camberley, GU17 9QG From sacadmin Thu Mar 12 11:28:34 2009 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CISYK9016518 for ; Thu, 12 Mar 2009 11:28:34 -0700 (PDT) Received: from brmea-mail-1.sun.com (brmea-mail-1.Sun.COM [192.18.98.31]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2CISY5C012928 for ; Thu, 12 Mar 2009 11:28:34 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n2CISXku013913 for ; Thu, 12 Mar 2009 18:28:33 GMT MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; charset=ISO-8859-1 Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KGE00100M9B8Y00@mail-amer.sun.com> for PSARC@sac.sfbay.sun.com; Thu, 12 Mar 2009 12:28:33 -0600 (MDT) Received: from [129.146.108.62] ([unknown] [129.146.108.62]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KGE00M36ONF58D0@mail-amer.sun.com> for PSARC@sac.sfbay.sun.com; Thu, 12 Mar 2009 12:28:27 -0600 (MDT) Date: Thu, 12 Mar 2009 11:28:27 -0700 From: Scott Rotondo Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-reply-to: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> Sender: Scott.Rotondo@Sun.COM To: Casper.Dik@Sun.COM Cc: PSARC@sac.sfbay.sun.com Message-id: <49B9544B.8090306@sun.com> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.12 (X11/20080422) Status: RO Content-Length: 2325 Casper Dik wrote: > Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI > Copyright 2009 Sun Microsystems, Inc. > 1. Introduction > 1.1. Project/Component Working Name: > Fasttrack for turbo-charging SVr4 package install. > 1.2. Name of Document Author/Supplier: > Author: Casper Dik > 1.3 Date of This Document: > 12 March, 2009 > 4. Technical Description > Fasttrack for turbo-charging SVr4 package install. > > Release binding: patch/micro > > Our customers using Solaris 10 have several problems: > > - installing patches is slow > - liveupgrade is slow > - upgrade is slow > - liveupgrade with zones is even slower. > - installing zones is slow > > Much of this is caused by the contents database /var/sadm/install/contents. > Most of the procedures listed above cause around 50-150GB of I/O > to the contents file. > > Earlier attempts to get rid of the file didn't pan out. This project > proposes to keep the contents file but delay writing it. > The contents file is managed by a daemon. Package commands no > longer manipulate the contents file themselves, they call the daemon. > > The pkgserv daemon stores all modifications in a logfile > (/var/sadm/install/pkglog). The "virtual contents file" is > the on-disk "contents" file and the on-disk "pkglog" file. > > The daemon will lock out older versions of the package commands by grabbing > the contents lock file. (Older package commands can run again after the > daemon has exited) > > The fasttrack adds the following commands/subcommands: > > /usr/sadm/install/bin/pkgserv > - Consolidation private > - invoked automatically by the pkgcmds. > > /usr/bin/pkgadm sync [-R root] [-q] > Writes the contents file and rolls the contents log file. > Optionally forces the contents file server to quit [-q]. > > - invoked automatically by the pkgcmds, > lu commands (such as luumount), install, upgrade, > liveupgrade, zoneadm install Would it make sense for pkgserv to sync automatically after some suitably long period of inactivity? This would help guard against package utilities that crash just before the sync step, for example. Scott -- Scott Rotondo Principal Engineer, Solaris Security Technologies President, Trusted Computing Group Phone/FAX: +1 408 850 3655 (Internal x68278) From sacadmin Thu Mar 12 13:01:06 2009 Received: from dm-holland-02.uk.sun.com (dm-holland-02.UK.Sun.COM [129.156.101.225]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CK15Er013946; Thu, 12 Mar 2009 13:01:05 -0700 (PDT) Received: from holland (room101.Holland.Sun.COM [10.16.117.40]) by dm-holland-02.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2CK13VY008279; Thu, 12 Mar 2009 20:01:03 GMT Message-Id: <200903122001.n2CK13VY008279@dm-holland-02.uk.sun.com> From: Casper.Dik@sun.com To: Nicolas Williams cc: Casper Dik , PSARC@sac.sfbay.sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-Reply-To: <20090312164332.GD9992@Sun.COM> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> <20090312161546.GC9992@Sun.COM> <200903121634.n2CGYleI012475@dm-holland-02.uk.sun.com> <20090312164332.GD9992@Sun.COM> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 12 Mar 2009 21:01:03 +0100 Sender: casper@holland.sun.com Status: RO Content-Length: 454 >On Thu, Mar 12, 2009 at 05:34:47PM +0100, Casper.Dik@Sun.COM wrote: >> The "temporary" read-only pkgservs create a .door in /tmp. This is removed >> when it is no longer needed by the temporary pkgserv when it exits. > >Since different users could be creating read-only pkgservs presumably >each one or each user has its own door file name. The read-only pkg >commands should check that this door is really owned by the same user. It does. Casper From sacadmin Thu Mar 12 13:05:38 2009 Received: from dm-holland-02.uk.sun.com (dm-holland-02.UK.Sun.COM [129.156.101.225]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CK5bnN014505; Thu, 12 Mar 2009 13:05:37 -0700 (PDT) Received: from holland (room101.Holland.Sun.COM [10.16.117.40]) by dm-holland-02.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2CK5aDU009284; Thu, 12 Mar 2009 20:05:36 GMT Message-Id: <200903122005.n2CK5aDU009284@dm-holland-02.uk.sun.com> From: Casper.Dik@sun.com To: Renaud Manus cc: Casper Dik , PSARC@sac.sfbay.sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-Reply-To: <49B9407D.8050702@sun.com> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> <49B92D8B.1080900@sun.com> <200903121600.n2CG0iO4065445@dm-holland-02.uk.sun.com> <49B9407D.8050702@sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 12 Mar 2009 21:05:36 +0100 Sender: casper@holland.sun.com Status: RO Content-Length: 453 >> There's one pkgserv per copy of the contents file and there's one >> pkglog, if needed, next to that contents file. > >Well, if zones aren't running, it seems that you can have one pkgserv >daemon for multiple copy of the contents file, no ? The daemon is bound to the contents file. If you multiple contents file, e.g., when installing a new zone, a new package daemon is bound to that copy. AsLo when you use "pkgadd -R /some/root". Casper From sacadmin Thu Mar 12 13:16:28 2009 Received: from dm-holland-02.uk.sun.com (dm-holland-02.UK.Sun.COM [129.156.101.225]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CKGRGO015394 for ; Thu, 12 Mar 2009 13:16:28 -0700 (PDT) Received: from holland (room101.Holland.Sun.COM [10.16.117.40]) by dm-holland-02.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2CKGRn3011454; Thu, 12 Mar 2009 20:16:27 GMT Message-Id: <200903122016.n2CKGRn3011454@dm-holland-02.uk.sun.com> From: Casper.Dik@sun.com To: Brian Ruthven - Sun UK cc: PSARC@sac.sfbay.sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-Reply-To: <49B94802.7040706@sun.com> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> <49B94802.7040706@sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 12 Mar 2009 21:16:27 +0100 Sender: casper@holland.sun.com Status: RO Content-Length: 2058 > >Hi Casper, > >Is single-user mode catered for? (or doesn't matter?) Doesn't really matter. When running "pkg*" cmds, the pkgserv is started. (It is required, pretty much like libc and libpkg is required) The worst that could happen is that "reboot" is done and this may kill the pkgserv. However, we can recover from that. ("pkgadm sync" will fix that) >Is the "-R" option to pkgadd and friends catered for? Yes. I run all the pkg install tests and they use the -R option continuously. >In another thread, there was mention of a check that "That the caller's >id matches the id of the pkgserv". Will this work with multiple >privileged but different users? i.e. if there are two admins for the box >which both had the appropriate privilege to apply packages, would the >second one have to wait until the deamon started by the first had exited? As the Solaris privilege model works, it is NOT possible to install packages will not being root. See, e.g., the pkgadd entry in the exec_attr file: Software Installation:suser:cmd:::/usr/sbin/pkgadd:uid=0;gid=bin >Under what conditions does the daemon exit? When: - "pkgadm sync -q" is run - when it is idle for some time and the last client has exited (we record the last pid from the last caller) >Is the daemon purely started on demand on first invocation of a pkg command? Yes; it is not started in any other circumstances. >Presumably this is simply saying that this is an implementation change, >and that the pkgadd, pkgrm, patchadd, etc... UI experience remains the >same (albeit sped up somewhat)? If so, then I would agree that customers >are crying out for this kind of speed bump. There are still stories of >multi-hour patching sessions, especially with zones involved. Yes. >Will this case be compatible with the parallelism of PSARC/2008/644 >(assuming it gets/has been implemented)? No problem with that; it installs patches in multiple zones in parallel. Each zone has its own pkgserv and wouldn't know or care about other zones running pkgserv. Casper From sacadmin Thu Mar 12 13:19:37 2009 Received: from dm-holland-02.uk.sun.com (dm-holland-02.UK.Sun.COM [129.156.101.225]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CKJaU7015570 for ; Thu, 12 Mar 2009 13:19:37 -0700 (PDT) Received: from holland (room101.Holland.Sun.COM [10.16.117.40]) by dm-holland-02.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2CKJaOw012004; Thu, 12 Mar 2009 20:19:36 GMT Message-Id: <200903122019.n2CKJaOw012004@dm-holland-02.uk.sun.com> From: Casper.Dik@sun.com To: Scott Rotondo cc: PSARC@sac.sfbay.sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-Reply-To: <49B9544B.8090306@sun.com> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> <49B9544B.8090306@sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 12 Mar 2009 21:19:36 +0100 Sender: casper@holland.sun.com Status: RO Content-Length: 376 >Would it make sense for pkgserv to sync automatically after some >suitably long period of inactivity? This would help guard against >package utilities that crash just before the sync step, for example. It does, actually. This is what I meant to say when I wrote: By default, the contents file is written within minutes after the last package commands exits. Casper From sacadmin Thu Mar 12 13:43:35 2009 Received: from zion.sfbay.sun.com (zion.SFBay.Sun.COM [129.146.17.75]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CKhZJh016982 for ; Thu, 12 Mar 2009 13:43:35 -0700 (PDT) Received: from [129.146.228.109] (cyber.SFBay.Sun.COM [129.146.228.109]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id n2CKhZ7s004219; Thu, 12 Mar 2009 20:43:35 GMT Message-ID: <49B973F3.7020405@Sun.COM> Date: Thu, 12 Mar 2009 13:43:31 -0700 From: Bart Smaalders Organization: Sun Microsystems User-Agent: Thunderbird 2.0.0.18 (X11/20090113) MIME-Version: 1.0 To: Casper.Dik@Sun.COM CC: Scott Rotondo , PSARC@sac.sfbay.sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> <49B9544B.8090306@sun.com> <200903122019.n2CKJaOw012004@dm-holland-02.uk.sun.com> In-Reply-To: <200903122019.n2CKJaOw012004@dm-holland-02.uk.sun.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 851 Casper.Dik@sun.com wrote: > >> Would it make sense for pkgserv to sync automatically after some >> suitably long period of inactivity? This would help guard against >> package utilities that crash just before the sync step, for example. > > It does, actually. This is what I meant to say when I wrote: > > By default, the contents file is written within minutes after > the last package commands exits. > > Casper > > It's interesting to see the discussion on this case as compared to 2006/120. What happens if someone types: patchadd .... ; reboot Does a reboot (as opposed to shutdown) cause the contents file to miss the patches/packages that have been added? - Bart -- Bart Smaalders Solaris Kernel Performance barts@cyber.eng.sun.com http://blogs.sun.com/barts "You will contribute more with mercurial than with thunderbird." From sacadmin Thu Mar 12 13:59:59 2009 Received: from dm-holland-02.uk.sun.com (dm-holland-02.UK.Sun.COM [129.156.101.225]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2CKxwsQ018377 for ; Thu, 12 Mar 2009 13:59:58 -0700 (PDT) Received: from holland (room101.Holland.Sun.COM [10.16.117.40]) by dm-holland-02.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2CKxu7R020958; Thu, 12 Mar 2009 20:59:56 GMT Message-Id: <200903122059.n2CKxu7R020958@dm-holland-02.uk.sun.com> From: Casper.Dik@sun.com To: Bart Smaalders cc: PSARC@sac.sfbay.sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-Reply-To: <49B973F3.7020405@Sun.COM> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> <49B9544B.8090306@sun.com> <200903122019.n2CKJaOw012004@dm-holland-02.uk.sun.com> <49B973F3.7020405@Sun.COM> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 12 Mar 2009 21:59:56 +0100 Sender: casper@holland.sun.com Status: RO Content-Length: 427 >It's interesting to see the discussion on this case as >compared to 2006/120. > >What happens if someone types: > >patchadd .... ; reboot > >Does a reboot (as opposed to shutdown) cause the contents file to >miss the patches/packages that have been added? No, it doesn't matter. You can even "pkill -9 pkgserv" and then reboot. (Powerfailure will not work so well because none of the archivers we use run fsync()) Casper From sacadmin Mon Mar 16 11:27:07 2009 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2GIR7sj005275; Mon, 16 Mar 2009 11:27:07 -0700 (PDT) Received: from brmea-mail-1.sun.com (brmea-mail-1.Sun.COM [192.18.98.31]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2GIR7AI038645; Mon, 16 Mar 2009 11:27:07 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n2GIR7sE013365; Mon, 16 Mar 2009 18:27:07 GMT MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; charset=ISO-8859-1 Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KGM0000021SMV00@mail-amer.sun.com>; Mon, 16 Mar 2009 12:27:07 -0600 (MDT) Received: from [192.168.0.11] ([unknown] [206.53.29.107]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KGM000WH391MU70@mail-amer.sun.com>; Mon, 16 Mar 2009 12:27:01 -0600 (MDT) Date: Mon, 16 Mar 2009 12:27:01 -0600 From: Jerry Jelinek Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-reply-to: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> Sender: Gerald.Jelinek@Sun.COM To: Casper Dik Cc: PSARC@sac.sfbay.sun.com, Casper.Dik@Sun.COM Message-id: <49BE99F5.4010603@sun.com> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.18 (X11/20090211) Status: RO Content-Length: 361 Casper, Will this only work with the public pkgadd and pkgrm commands or will it work with the private pkgremove and pkginstall commands as well? The zones team has a contract with install for the use of pkgremove and pkginstall (PSARC/2007/621) for use with zones 'update on attach' and I'm wondering how this will interact with that feature? Thanks, Jerry From sacadmin Mon Mar 16 12:12:30 2009 Received: from dm-holland-02.uk.sun.com (dm-holland-02.UK.Sun.COM [129.156.101.225]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2GJCTIo021975; Mon, 16 Mar 2009 12:12:30 -0700 (PDT) Received: from holland (room101.Holland.Sun.COM [10.16.117.40]) by dm-holland-02.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2GJCS9Y036936; Mon, 16 Mar 2009 19:12:28 GMT Message-Id: <200903161912.n2GJCS9Y036936@dm-holland-02.uk.sun.com> From: Casper.Dik@sun.com To: Jerry Jelinek cc: Casper Dik , PSARC@sac.sfbay.sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-Reply-To: <49BE99F5.4010603@sun.com> References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> <49BE99F5.4010603@sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 16 Mar 2009 20:12:28 +0100 Sender: casper@holland.sun.com Status: RO Content-Length: 523 >Will this only work with the public pkgadd and pkgrm >commands or will it work with the private pkgremove >and pkginstall commands as well? > >The zones team has a contract with install for the >use of pkgremove and pkginstall (PSARC/2007/621) >for use with zones 'update on attach' and I'm >wondering how this will interact with that feature? "pkgadd" just runs "pkginstall"; all the commands which manipulate the contents database has been rewritten. Installing a zone will work with the modified commands. Casper From casper@holland.sun.com Fri Mar 20 03:00:51 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2KA0pOJ002568 for ; Fri, 20 Mar 2009 03:00:51 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n2KA0f4Y054710 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Fri, 20 Mar 2009 04:00:51 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KGS00E3MUHBJ500@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Fri, 20 Mar 2009 03:00:47 -0700 (PDT) Received: from dm-holland-02.uk.sun.com ([129.156.101.225]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KGS00E0CUH8JQ00@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Fri, 20 Mar 2009 03:00:45 -0700 (PDT) Received: from holland (room101.Holland.Sun.COM [10.16.117.40]) by dm-holland-02.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2KA0gig013891; Fri, 20 Mar 2009 10:00:42 +0000 (GMT) Date: Fri, 20 Mar 2009 11:00:42 +0100 From: Casper.Dik@sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-reply-to: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> Sender: casper@holland.sun.com To: Casper Dik Cc: psarc-ext@sun.com Message-id: <200903201000.n2KA0gig013891@dm-holland-02.uk.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> Status: RO Content-Length: 2644 This case is now open. Fasttrack for turbo-charging SVr4 package install. Release binding: patch/micro Our customers using Solaris 10 have several problems: - installing patches is slow - liveupgrade is slow - upgrade is slow - liveupgrade with zones is even slower. - installing zones is slow Much of this is caused by the contents database /var/sadm/install/contents. Most of the procedures listed above cause around 50-150GB of I/O to the contents file. Earlier attempts to get rid of the file didn't pan out. This project proposes to keep the contents file but delay writing it. The contents file is managed by a daemon. Package commands no longer manipulate the contents file themselves, they call the daemon. The pkgserv daemon stores all modifications in a logfile (/var/sadm/install/pkglog). The "virtual contents file" is the on-disk "contents" file and the on-disk "pkglog" file. The daemon will lock out older versions of the package commands by grabbing the contents lock file. (Older package commands can run again after the daemon has exited) The fasttrack adds the following commands/subcommands: /usr/sadm/install/bin/pkgserv - Consolidation private - invoked automatically by the pkgcmds. /usr/bin/pkgadm sync [-R root] [-q] Writes the contents file and rolls the contents log file. Optionally forces the contents file server to quit [-q]. - invoked automatically by the pkgcmds, lu commands (such as luumount), install, upgrade, liveupgrade, zoneadm install /var/svc/manifest/system/pkgserv.xml - defines svc:/system/pkgserv:default - runs "pkgadm sync -q" at boot and shutdown time. - write the contents file if needed, picks up the pieces if pkgserv is killed, makes sure the pkgserv exits. The new implementation uses at most as much memory (both real and reserved) as the current implementation, but typically less. Each zone will run its own pkgserv. The daemon will automatically flush the contents file and it will shut down. (By default, the contents file is written within minutes after the last package commands exits) The performance change is considerable: install 25-50% (1.5-2 as fast) upgrade 50-75% (2 - 4 times faster) liveupgrade 50-75% (w/ or w/o zones) (higher gain with more zones) zone install 50-75% Security: the daemon verifies the caller's identity using door_ucred(). Users which cannot modify the contents database will run their own read-only version of the daemon. It is possible to back out the new pkgcmds; the contents file stays as before, and after the daemon is stopped, the older package commands can be reinstalled. From gww@eng.sun.com Fri Mar 20 08:08:39 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2KF8cMV024271 for ; Fri, 20 Mar 2009 08:08:38 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2KF8ZN2007426 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Fri, 20 Mar 2009 15:08:37 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KGT00I058QC7R00@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Fri, 20 Mar 2009 08:08:36 -0700 (PDT) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KGT00FOE8QCZ670@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Fri, 20 Mar 2009 08:08:36 -0700 (PDT) Received: from marduk.eng.sun.com (marduk.SFBay.Sun.COM [129.146.108.224]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2KF8ZST019320; Fri, 20 Mar 2009 08:08:35 -0700 (PDT) Received: from marduk.eng.sun.com (localhost [127.0.0.1]) by marduk.eng.sun.com (8.13.6+Sun/8.12.11) with ESMTP id n2KF9Bm4015568; Fri, 20 Mar 2009 07:09:11 -0800 (PST) Received: (from gww@localhost) by marduk.eng.sun.com (8.13.6+Sun/8.12.11/Submit) id n2KF9B3Y015567; Fri, 20 Mar 2009 08:09:11 -0700 (PDT) Date: Fri, 20 Mar 2009 08:09:11 -0700 (PDT) From: Gary Winiger Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] To: casper@sac.sfbay.sun.com, Casper.Dik@sun.com Cc: psarc-ext@sun.com Message-id: <200903201509.n2KF9B3Y015567@marduk.eng.sun.com> Content-transfer-encoding: 7BIT X-Sun-Charset: US-ASCII X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 68 > This case is now open. +1 in the open as well as closed. Gary.. From gdamore@sun.com Fri Mar 20 08:28:40 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2KFSdHi008991 for ; Fri, 20 Mar 2009 08:28:39 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n2KFSZRV027239 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Fri, 20 Mar 2009 23:28:38 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KGT00K0H9NPFG00@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Fri, 20 Mar 2009 08:28:37 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KGT00F999NOZ6F0@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Fri, 20 Mar 2009 08:28:36 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n2KFSacv010501 for ; Fri, 20 Mar 2009 08:28:36 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KGT003009CCHN00@fe-sfbay-09.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Fri, 20 Mar 2009 08:28:36 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KGT0084U9NBQJ70@fe-sfbay-09.sun.com>; Fri, 20 Mar 2009 08:28:24 -0700 (PDT) Date: Fri, 20 Mar 2009 08:28:23 -0700 From: "Garrett D'Amore" Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] In-reply-to: <200903201000.n2KA0gig013891@dm-holland-02.uk.sun.com> Sender: Garrett.Damore@sun.com To: Casper.Dik@sun.com Cc: Casper Dik , psarc-ext@sun.com Message-id: <49C3B617.5030300@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> <200903201000.n2KA0gig013891@dm-holland-02.uk.sun.com> User-Agent: Thunderbird 2.0.0.18 (X11/20081201) Status: RO Content-Length: 2824 +1. -- Garrett Casper.Dik@Sun.COM wrote: > This case is now open. > > Fasttrack for turbo-charging SVr4 package install. > > Release binding: patch/micro > > Our customers using Solaris 10 have several problems: > > - installing patches is slow > - liveupgrade is slow > - upgrade is slow > - liveupgrade with zones is even slower. > - installing zones is slow > > Much of this is caused by the contents database /var/sadm/install/contents. > Most of the procedures listed above cause around 50-150GB of I/O > to the contents file. > > Earlier attempts to get rid of the file didn't pan out. This project > proposes to keep the contents file but delay writing it. > The contents file is managed by a daemon. Package commands no > longer manipulate the contents file themselves, they call the daemon. > > The pkgserv daemon stores all modifications in a logfile > (/var/sadm/install/pkglog). The "virtual contents file" is > the on-disk "contents" file and the on-disk "pkglog" file. > > The daemon will lock out older versions of the package commands by grabbing > the contents lock file. (Older package commands can run again after the > daemon has exited) > > The fasttrack adds the following commands/subcommands: > > /usr/sadm/install/bin/pkgserv > - Consolidation private > - invoked automatically by the pkgcmds. > > /usr/bin/pkgadm sync [-R root] [-q] > Writes the contents file and rolls the contents log file. > Optionally forces the contents file server to quit [-q]. > > - invoked automatically by the pkgcmds, > lu commands (such as luumount), install, upgrade, > liveupgrade, zoneadm install > > /var/svc/manifest/system/pkgserv.xml > - defines svc:/system/pkgserv:default > - runs "pkgadm sync -q" at boot and shutdown time. > - write the contents file if needed, picks up the pieces > if pkgserv is killed, makes sure the pkgserv exits. > > The new implementation uses at most as much memory (both real and reserved) > as the current implementation, but typically less. > > Each zone will run its own pkgserv. > > The daemon will automatically flush the contents file and it will > shut down. (By default, the contents file is written within minutes after > the last package commands exits) > > The performance change is considerable: > > install 25-50% (1.5-2 as fast) > upgrade 50-75% (2 - 4 times faster) > liveupgrade 50-75% > (w/ or w/o zones) (higher gain with more zones) > zone install 50-75% > > Security: the daemon verifies the caller's identity using door_ucred(). > Users which cannot modify the contents database will run their own read-only > version of the daemon. > > It is possible to back out the new pkgcmds; the contents file stays as before, > and after the daemon is stopped, the older package commands can be reinstalled. > From casper@holland.sun.com Wed Mar 25 17:23:16 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2Q0NFwl017308 for ; Wed, 25 Mar 2009 17:23:16 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2Q0NE86024225 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Thu, 26 Mar 2009 00:23:15 GMT Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KH300A017QQA600@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 25 Mar 2009 17:23:14 -0700 (PDT) Received: from dm-holland-02.uk.sun.com ([129.156.101.225]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KH3000TF7QOBXE0@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 25 Mar 2009 17:23:13 -0700 (PDT) Received: from holland (room101.Holland.Sun.COM [10.16.117.40]) by dm-holland-02.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2Q0NCjA023003 for ; Thu, 26 Mar 2009 00:23:12 +0000 (GMT) Date: Thu, 26 Mar 2009 01:23:12 +0100 From: Casper.Dik@sun.com Subject: Re: Fasttrack for turbo-charging SVr4 package install. [PSARC/2009/173 FastTrack timeout 03/18/2009] Sender: casper@holland.sun.com To: psarc-ext@sun.com Message-id: <200903260023.n2Q0NCjA023003@dm-holland-02.uk.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200903121521.n2CFLDVR007628@sac.sfbay.sun.com> <49B92D8B.1080900@sun.com> Status: RO Content-Length: 48 This case was approved at PSARC today. Casper