From sacadmin Mon Sep 25 04:27:22 2006 Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.6+Sun/8.13.6) with ESMTP id k8PBRM3s002304; Mon, 25 Sep 2006 04:27:22 -0700 (PDT) Received: (from darrenm@localhost) by sac.sfbay.sun.com (8.13.6+Sun/8.13.6/Submit) id k8PBRMvE002301; Mon, 25 Sep 2006 04:27:22 -0700 (PDT) Date: Mon, 25 Sep 2006 04:27:22 -0700 (PDT) From: Darren J Moffat Message-Id: <200609251127.k8PBRMvE002301@sac.sfbay.sun.com> To: PSARC@sac.sfbay.sun.com Cc: Rhodri.Davies@Sun.COM Subject: setproject(3PROJECT) defining and enhancing [PSARC/2006/554 Timeout: 10/02/2006] Status: RO Content-Length: 18223 Subject: PSARC FastTrack [10/02/2006]: setproject(3PROJECT) defining and enhancing Template Version: @(#)sac_nextcase %I% %G% SMI 1. Introduction 1.1. Project/Component Working Name: setproject(3PROJECT) defining and enhancing 1.2. Name of Document Author/Supplier: Author: Rhodri Davies 1.3 Date of This Document: 25 September, 2006 4. Technical Description SUMMARY setproject(3PROJECT) defining, and enhancing behaviour libproject's setproject() was introduced by PSARC 1999/119 (Jobs, Sessions, Projects and Accounting). We are currently seeing many customer escalations related to setproject() boundary conditions: 6194864 simultaneous setproject()'s on the same project can fail to set rctl 6449567 setproject(3PROJECT) deletes resource controls set through prctl(1M) and a usability concern: 6450539 projmod(1M) does not provide a mechanism to refresh "in-core" enforced resource controls BACKGROUND Resource controls (rctls) allow the system administrator to limit the resources available to a specific process, task, project or zone, eg setting a limit on the amount of shared memory that may be used by a process. There are currently five categories of rctls: +----------+----------------------------------+----------------------+ | Category | Kernel resource | Example | +----------+----------------------------------+----------------------+ | project | proc->p_task->tk_proj->kpj_rctls | project.cpu-shares | | task | proc->p_task->tk_rctls | task.max-cpu-time | | process | proc->p_rctls | process.max-cpu-time | | zone | proc->p_zone->zone_rctls | zone.cpu-shares | | rcap | n/a (managed by rcapd(1M) | rcap.max-rss | +----------+----------------------------------+----------------------+ This case specifically looks at how project category rctls are applied. This case DOES NOT affect task, process or zone rctls. rctls are manipulate via prctl(1M), eg # prctl -t priv -n project.max-sem-ids -s -v 600 -i project foo prctl uses setrctl(2) to set the appropriate rctls for the project foo. prctl and getrctl(2) may be used to view the active rctls. Solaris provides a database - project(4) - to map resource controls to a specific project id. eg foo:100::rd::project.max-sem-ids=(priv,500,deny); \ task.max-lwps=(privileged,100,signal=SIGTERM) The project database does not reflect the resource controls currently in effect, rather these rctls will be applied when setproject(3PROJECT) is next called against the project. A process may join a project by calling setproject(), by the user starting the process via newtask: $ newtask -p foo /usr/bin/sleep 100 or by running newtask against a process id: $ newtask -p foo -c 1234 setproject() is called by newtask(1M), pam_unix_cred(5) and may be potentially called from 3rd party applications. The contents of /etc/project may be modified with the projadd(1M) and projmod(1M) commands, eg # projmod -sK "project.max-sem-ids=(priv,100,deny)" foo There are three distinct privilege of resource control, as identified in resource_controls(5): +-------------+---------------------------------------------------------+ | Privilege | Comment | +-------------+---------------------------------------------------------+ | basic | Can be modified by the owner of the calling process | | privileged | Can be modified only by privileged (superuser) callers | | system | Fixed for the duration of the operating system instance | +-------------+---------------------------------------------------------+ ISSUES 1. In kernel, the rctl values are referenced via a rctl_set_t. rctl_set_t has a hash (rcs_ctls) of rctl_t, proc_t +---------+ | . | task_t | p_task -+----> +----------+ | . | | . | kproject_t | p_rctls |--+ | tk_proj -+----> +-----------+ | . | | | . | | . | rctl_set_t +---------+ | | tk_rctls |--+ | kpj_rctls |----> +--------+ | | . | | | . | | | | +----------+ | +-----------+ +--------+ | | | | rctl_set_t | +--> +--------+ | | | | +--------+ | rctl_set_t +--> +--------+ | | +--------+ We are only interested in resource controls hanging off kproject_t. Assuming an entry in the project database such as project.max-sem-ids=(priv,6,signal=SIGHUP),(priv,100,deny) this would be represented in kernel as: rctl_t +------------+ | . | rctl_val_t | rc_values |----> +----------------------------+ | . | | rcv_value = 6 | | rc_id | | rcv_flagaction = signal | | . | | rcv_action_signal = SIGHUP | +------------+ | . | | rcv_next |---+ +----------------------------+ | | +------------------------------------+ | rctl_val_t +-> +----------------------------+ | rcv_value = 100 | | rcv_flagaction = deny | | rcv_action_signal = 0 | | . | | rcv_next |---> NULL +----------------------------+ rctl_t's rc_id is a unique identifier that identifies the rctl - on the current system, eg # mdb -k > ::rctl_dict ID NAME ADDR TYPE GLOBAL_FLAGS . 17 project.max-sem-ids 0000030000387908 project 0xa0100000 . setproject() will create a new task, the process will be its only member. The proc_t's p_rctls, and task_t's tk_rctl,s will only be manipulated by this process. However, the kproject_t (and kpj_rctls) will be shared between all processes in the project. Multiple processes may call setproject() against the same project, at the same time. Given that they are referencing the same kproject_t, this may lead to a race condition. The current behaviour of setproject(), is such that it will: 1) delete the basic and privileged rctls - setrctl(..., RCTL_DELETE) 2) inserts the new rctls - setrctl(..., RCTL_INSERT) Should another process insert a new rctl between steps 1 and 2 (e.g. a process calling setproject()), then we would expect setrctl(..., RCTL_INSERT) to return an EINVAL, as an attempt has been made to insert an rctl that is already set. A failing setrctl() results in the following behaviour: newtask(1M): # newtask -p foo /usr/bin/sleep 2 newtask: warning, project.max-sem-ids resource control assignment failed for project "foo" pam_unix_cred(5): pam_unix_cred: project.max-shm-memory resource control assignment failed for project "foo" The failure in pam_unix_cred is relatively serious, as it causes the pam stack to return an error, resulting in a login failure, or a cron job that fail to run. 2. As mentioned above, setproject() will delete all basic, and privileged rctls before inserting the rctls defined in project(4). If a system administrator has inserted or modified rctls directly via prctl, then these changes will be lost - in a matter of seconds, days or never, depending on when setproject() is called. 3. Resource controls are updated in a "hap-hazard" fashion, ie a change in /etc/project is applied only if some process happens to call setproject(). To counter this many system administrator following a two step process to apply a resource control: # projmod -sK "project.max-sem-ids=(priv,100,deny)" wibble # newtask -p wibble /usr/bin/true where newtask(1M) will call setproject() against the project name specified. There is no well-defined, supported mechanism of applying resource controls defined in the project database. SOLUTIONS: Proposed solutions: 1. Ensure that project rctl deletion, and insertion (previously accomplished, by setproject()'s use of setrctl(.., .., RCTL_DELETE) and setrctl(.., .., RCTL_INSERT)), are done atomically within kernel. 2. The default behaviour of setproject(..., 0) shall be that only rctl values set by previous calls to setproject() will be changed. However, if the project database entry has changed since the last setproject("foo", ...), then the new rctl values from the database will be applied. Values set by prctl(1M) remain unchanged. It will also be possible to call setproject(..., TASK_PROJ_PURGE) to purge all currently assigned rctl values, and replace them with the values found in the project database. 3. Addition of a -A flag to projmod(1M) that will apply the rctl values found in the project database to the active project. This -A flag will call setproject(..., TASK_PROJ_PURGE) --- rctl values set manually by prctl will be lost. SOLUTIONS IN DETAIL: 1. Adding a new sub-command to the rtclsys() system call. This will be accessed by the project private libc.so.1`setprojrctl() function. libproc.so.1`pr_setprojrctl() is introduced as a mechanism for calling setprojrctl() in the context of another process. int setprojrctl(const char *name, rctlblk_t *new_rblk, size_t size, int flags); int pr_setprojrctl(const char *name, rctlblk_t *new_rblk, size_t size, int flags); These functions accept an array of rctlblk_t, of _size_ number of elements. _flags_ defines the behaviour expected of setprojrctl(), whether it is to delete all rctls, or just those set through previous invocations of this function (see TASK_PROJ_PURGE below). setprojrctl() allows a number of resource blocks to be passed to the kernel at one time, and allows the kernel to perform the necessary locking while applying these rctl values. Within the kernel, rctlsys() will call a new function, rctlsys_projset(), static long rctlsys_projset(char *name, rctl_opaque_t *rblk, size_t size, int flags) which makes use of the following new functions: int rctl_local_replace_projdb(rctl_hndl_t hndl, rctl_val_t *val, rctl_val_t *cval, struct proc *p) int rctl_local_replace_all(rctl_hndl_t hndl, rctl_val_t *val, rctl_val_t *cval, struct proc *p) The former replaces rctls previously set through setprojrctl() for the resource control; the latter replaces all rctls for the resource control. val is a linked list of new rctls to be set, while cval is a linked list of pre-allocated rctl_val_t. This is needed so that we can make copies of the new rctl_val_t, in order to insert into rctl_t's rc_projdb (see below). 2. Amendment to struct rctl_val's rcv_flagaction member: this may also have the following bit set: #define RCTL_LOCAL_PROJDB 0x40000000 This signifies that the rctl_val was created by the projects framework, as opposed to prctl, or setrctl(). We also add the rc_projdb member to the kernel's struct rctl_t: typedef struct rctl { struct rctl *rc_next; /* next in set hash chain */ rctl_val_t *rc_values; /* list of enforced value */ rctl_val_t *rc_cursor; /* currently enforced value */ struct rctl_dict_entry *rc_dict_entry; /* global control properties */ rctl_hndl_t rc_id; /* control handle (hash key) */ rctl_val_t *rc_projdb; /* values set via projects */ } rctl_t; rc_projdb represents the rctl values set by setproject(), via the rctl_local_replace_projdb() or rctl_local_replace_all() kernel functions. To clarify with an example, setproject() applies the following entry from the project database: project.max-sem-ids=(priv,6,signal=SIGHUP),(priv,100,deny) setprojrctl() passes through an array of two elements; each element represents a resource value; within rctlsys_projset() these are transformed into a liked list of rctl_val_t: +----------------------------+ +----------------------------+ | rcv_value = 6 | | rcv_value = 100 | | rcv_flagaction = signal | | rcv_flagaction = deny | | rcv_action_signal = SIGHUP | | rcv_action_signal = 0 | | . | | . | | rcv_next |----->| rcv_next |-----> NULL +----------------------------+ +----------------------------+ The linked list is passed into rctl_local_replace_projdb(). We perform a comparison between these new rctl_val_t with those previously set through this mechanism. The behaviour of rctl_local_replace_projdb() is as follows: - if the rc_projdb linked list matches the new list, then we do nothing - if the rc_projdb contains an rctl_val_t that is not in the "new" linked list, then it implies that the entry has been removed from the project database. The entry is removed from rc_projdb. We will then search through the rc_values (the active rctl_val_t), and remove any matching rctl_val_t, (as long as the rctl_val_t does have the RCTL_LOCAL_PROJDB bit set in rcv_flagaction, as that would imply that it was set through prctl/setrctl(), and not through setproject()). - if the new list has a new value, then it is placed in the rc_projdb linked list, and a copy of this rctl_val_t is made and added to the active rctl values (rctl_t's rcv_values). It is necessary to be able to perform two actions through setprojrctl(): 1) remove rctl values previously inserted by setprojrctl() or 2) remove all values previously inserted, be that via setprojrctl(), or prctl(1M). (pr_)setprojrctl() is called from only setproject(3PROJECT). The default behavior of setproject() will be the first action noted above. In order to communicate the desire to perform the latter action, we must "overload" setproject()'s flags argument. As currently implemented, setproject()'s flag argument is such: int setproject(const char *project_name, const char *user_name, uint_t flags); . . setproject() will create a new task with settaskid(2) using task flags specified by flags, i.e. the flags argument to setproject() is passed directly through to settaskid(). It is proposed that some additional values for flags be allowed: task.h: #define TASK_NORMAL 0x0 /* task may create tasks via settaskid() */ #define TASK_FINAL 0x1 /* task finalized, settaskid() will fail */ #define TASK_MASK 0xfffff /* task flags mask */ #define TASK_PROJ_PURGE 0x100000 /* purge project.* rctl entities before resetting */ TASK_MASK and TASK_PROJ_PURGE are added. While TASK_PROJ_PURGE has no meaning in the context of settaskid(2), we include it in task.h for the sake of simplicity, and to ensure that subsequent changes to this include file do not result in incompatible behaviour. NB: There exists a Perl module to setproject(), exported via Project.pm, this module, along with Task.pm, are impacted by these changes. 3. Addition of a "-A" flag to projmod(1M), that will apply the rctls defined in the project database to the current running system: # projmod -A foo # projmod -AsK "project.max-sem-ids=(priv,100,deny)" foo projmod(1M) projmod [-n] [-A | -f filename | -] projmod [-n] [-A | -f filename | -] [-p projid [-o]] [-c comment] [-a|-s|-r] [-U user [,user...] ] [-G group [,group...] ] [ [-K name [=value [,value...]...]]] [-l new_projectname] project -A Apply the project's resource controls, as defined in the system's project database, to the project if it is active. INTERFACE TABLE +---------------------------------------------------------------+ | Interface Stability | +---------------------------------------------------------------+ | projmod(1M) Committed | | libc`setprojrctl Project Private | | libproc`pr_setprojrctl Project Private | | rctlsys_projset Project Private | | rctl_local_replace_projrctl Project Private | | rctl_local_replace_all Project Private | | TASK_PROJ_PURGE Project Private | | TASK_MASK Project Private | | RCTL_LOCAL_PROJDB Project Private | +---------------------------------------------------------------+ RELEASE TAXONOMY micro/patch 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack From sacadmin Mon Sep 25 16:49:59 2006 Received: from engmail3mpk.sfbay.Sun.COM (engmail3mpk.SFBay.Sun.COM [129.146.11.26]) by sac.sfbay.sun.com (8.13.6+Sun/8.13.6) with ESMTP id k8PNnxgF004199; Mon, 25 Sep 2006 16:49:59 -0700 (PDT) Received: from marduk.eng.sun.com (marduk.SFBay.Sun.COM [129.146.108.224]) by engmail3mpk.sfbay.Sun.COM (8.13.6+Sun/8.13.6/ENSMAIL,v2.2) with ESMTP id k8PNnwDt015231; Mon, 25 Sep 2006 16:49:58 -0700 (PDT) Received: from marduk.eng.sun.com (localhost [127.0.0.1]) by marduk.eng.sun.com (8.13.6+Sun/8.12.11) with ESMTP id k8PNqBXV019499; Mon, 25 Sep 2006 16:52:11 -0700 (PDT) Received: (from gww@localhost) by marduk.eng.sun.com (8.13.6+Sun/8.12.11/Submit) id k8PNqB8T019498; Mon, 25 Sep 2006 16:52:11 -0700 (PDT) Date: Mon, 25 Sep 2006 16:52:11 -0700 (PDT) From: Gary Winiger Message-Id: <200609252352.k8PNqB8T019498@marduk.eng.sun.com> To: PSARC@sac.sfbay.sun.com, darrenm@sac.sfbay.sun.com Cc: Rhodri.Davies@sun.com Subject: Re: setproject(3PROJECT) defining and enhancing [PSARC/2006/554 Timeout: 10/02/2006] Status: RO Content-Length: 878 > There are three distinct privilege of resource control, as > identified in resource_controls(5): > > +-------------+---------------------------------------------------------+ > | Privilege | Comment | > +-------------+---------------------------------------------------------+ > | basic | Can be modified by the owner of the calling process | > | privileged | Can be modified only by privileged (superuser) callers | ^^^^^^^^^ Nit: This should really relate the appropriate privilege from privileges(5). Neither resource_controls(5) nor setproject(3PROJECT) seem to discuss the appropriate privileges. Please file man page bugs to correct them. I suspect approprite privilege will include one or more of proc_taskid, sys_resource, sys_res_config, proc_priocntl. Gary.. From sacadmin Mon Oct 2 06:45:04 2006 Received: from beep.UK.Sun.COM (beep.UK.Sun.COM [129.156.173.75]) by sac.sfbay.sun.com (8.13.6+Sun/8.13.6) with ESMTP id k92Dj3WT029971; Mon, 2 Oct 2006 06:45:04 -0700 (PDT) Received: from beep (localhost [127.0.0.1]) by beep.UK.Sun.COM (8.13.7+Sun/8.13.7) with ESMTP id k92Divtq027723; Mon, 2 Oct 2006 14:44:57 +0100 (BST) To: Gary Winiger Cc: PSARC@sac.sfbay.sun.com, darrenm@sac.sfbay.sun.com Subject: Re: setproject(3PROJECT) defining and enhancing [PSARC/2006/554 Timeout: 10/02/2006] References: <200609252352.k8PNqB8T019498@marduk.eng.sun.com> From: Rhodri Davies - Sun UK Date: Mon, 02 Oct 2006 14:44:57 +0100 In-Reply-To: <200609252352.k8PNqB8T019498@marduk.eng.sun.com> (Gary Winiger's message of "Mon, 25 Sep 2006 16:52:11 -0700 (PDT)") Message-ID: User-Agent: Gnus/5.1008 (Gnus v5.10.8) Emacs/21.3 (usg-unix-v) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Status: RO Content-Length: 753 On Tuesday, Gary Winiger wrote: >> | privileged | Can be modified only by privileged (superuser) callers | > ^^^^^^^^^ > > Nit: This should really relate the appropriate privilege from > privileges(5). > Neither resource_controls(5) nor setproject(3PROJECT) seem to discuss > the appropriate privileges. Please file man page bugs to correct > them. I suspect approprite privilege will include one or more > of proc_taskid, sys_resource, sys_res_config, proc_priocntl. I've raised the following CRs for the above nits. CR 6477234 resource_controls(5) - clarification on requirements to modify a privileged resource control CR 6477244 setproject(3PROJECT) - clarification on required privileges Thanks. -- Rhodri From sacadmin Tue Oct 17 06:18:43 2006 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.6+Sun/8.13.6) with ESMTP id k9HDIgQ5026269 for ; Tue, 17 Oct 2006 06:18:42 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id k9HDIZQG020140 for <@sunmail1brm.central.sun.com:psarc@sun.com>; Tue, 17 Oct 2006 21:18:41 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0J7A007037N43200@brm-avmta-1.central.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Tue, 17 Oct 2006 07:18:40 -0600 (MDT) Received: from gmp-ea-fw-1.sun.com ([129.156.42.6]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0J7A004ZN7N31K30@brm-avmta-1.central.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Tue, 17 Oct 2006 07:18:40 -0600 (MDT) Received: from d1-emea-09.sun.com ([192.18.2.119]) by gmp-ea-fw-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id k9HDIcH4006087 for ; Tue, 17 Oct 2006 14:18:38 +0100 (BST) Received: from conversion-daemon.d1-emea-09.sun.com by d1-emea-09.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) id <0J7A007017MEQ300@d1-emea-09.sun.com> (original mail from Darren.Moffat@Sun.COM) for psarc@sun.com (ORCPT psarc@sun.com); Tue, 17 Oct 2006 14:18:38 +0100 (BST) Received: from [129.156.173.21] by d1-emea-09.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) with ESMTPSA id <0J7A006RB7N2CV10@d1-emea-09.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Tue, 17 Oct 2006 14:18:38 +0100 (BST) Date: Tue, 17 Oct 2006 14:18:38 +0100 From: Darren J Moffat Subject: setproject(3PROJECT) defining and enhancing [PSARC/2006/554 Close approved] Sender: Darren.Moffat@sun.com To: psarc@sun.com Cc: Rhodri.Davies@sun.com Message-id: <4534D82E.1090405@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 User-Agent: Thunderbird 1.5.0.5 (X11/20060912) Status: RO Content-Length: 44 This case is approved. -- Darren J Moffat