From Artem.Kachitchkin@sun.com Tue Oct 7 19:52:53 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m982qqr6019831 for ; Tue, 7 Oct 2008 19:52:52 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m982qojF021882; Wed, 8 Oct 2008 10:52:50 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8E00B03G01S300@brm-avmta-1.central.sun.com>; Tue, 07 Oct 2008 20:52:49 -0600 (MDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8E007PFG00HQD0@brm-avmta-1.central.sun.com>; Tue, 07 Oct 2008 20:52:48 -0600 (MDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m982qmXq007128; Tue, 07 Oct 2008 19:52:48 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8E00401FSYWH00@fe-sfbay-10.sun.com> (original mail from Artem.Kachitchkin@Sun.COM); Tue, 07 Oct 2008 19:52:48 -0700 (PDT) Received: from [129.150.16.68] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8E000V3FZWZW10@fe-sfbay-10.sun.com>; Tue, 07 Oct 2008 19:52:48 -0700 (PDT) Date: Tue, 07 Oct 2008 19:52:06 -0700 From: Artem Kachitchkine Subject: PSARC/2008/628 Interrupt Resource Management Sender: Artem.Kachitchkin@sun.com To: psarc-ext@sun.com Cc: ddi-irm-interest@sun.com Message-id: <48EC2056.9000505@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 User-Agent: Mozilla/5.0 Gecko/20040113 Status: RO Content-Length: 12870 I'm sponsoring this fasttrack for Scott Carter. It is set to timeout on 10/15/2008. Given that this is a relatively minor amendment to a larger, approved case PSARC/2004/253 "Advanced DDI Interrupt Functions", I think it qualifies as a fasttrack. As the discussion progresses, extending the timer or even promoting to a full case may be considered. In addition to this proposal, three man page files can be found in the case directory: ddi_cb_register.9f.txt ddi_intr_set_nreq.9f.txt ndi_irm_create.9f.txt -Artem Template Version: @(#)sac_nextcase %I% %G% SMI This information is Copyright 2008 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: Interrupt Resource Management 1.2. Name of Document Author/Supplier: Author: Scott Carter 1.3 Date of This Document: 07 October, 2008 4. Technical Description 4.1 Project Description This project delivers the resource management feature originally defined in PSARC/2004/253 "Advanced DDI Interrupt Functions" with minor changes (see 4.1.3 for details). It provides a mechanism for device drivers to get more interrupt vectors (and increased performance). By participating with the new feature, their number of available interrupts becomes dynamic and can increase or decrease. The goal is to maximize utilization of interrupt vectors in a fair manner, and rebalance the allocations whenever devices are added or removed from the system. 4.1.1 Definition The project defines new DDI interfaces for device drivers to register and unregister a generic callback handler. If a driver has registered a handler, then it can be notified when its number of interrupts has been increased or decreased. How many interrupt vectors a device driver wants is set by the initial number it attempts to allocate through ddi_intr_alloc(). And a driver can explicitly adjust this number at later times with a new DDI function that is introduced by this project. The project also defines new NDI interfaces for nexus drivers to define the supplies of interrupt vectors that are available in the system. A supply can be associated with a specific IO bus (e.g. PCIe root complex) or it could be a global supply shared by all devices. Along with the new DDI and NDI interfaces, the project also delivers an MDB debugging module with two new debug macros. One macro is used to display statistics on all of the defined interrupt supplies, and the other displays statistics about how a supply is divided up between the drivers that map to it. 4.1.2 Motivation, Goals, and Requirements Currently, interrupt vectors are given to device drivers in very small numbers. This is to avoid exhausting a system's supply before all the devices have been attached, and to keep interrupt vectors in reserve for later hotplugs. Interrupts are given so conservatively because there is no mechanism to take them back later unless a driver detaches. Fewer interrupt vectors means less IO performance. More interrupt vectors means more parallelism for handling interrupt conditions. So the motivation for this project is to increase IO performance. And the goal of this project is to maximize the allocation of interrupts given to each device, but in a way which is fair and balanced across the full set of attached devices. The requirement of this project is: - Provide a mechanism for drivers to get more interrupts. The feature is optional so drivers that don't use it still work even if the system has implemented support for the feature. And conversely, drivers that do use it also work if the system does not implement the support. A full implementation of the platform level support will be provided by this project for PCIe IO bus drivers on SPARC. 4.1.3 Changes From the Previous Case This project is a micro change to the approved case PSARC/2004/253 "Advanced DDI Interrupt Interfaces". The existing DDI interrupt interfaces are not changed. But some new DDI interrupt interfaces are added, which extend the capabilities of the existing interfaces. The interfaces described in section 6.3 of the specifications for PSARC/2004/253 "Advanced DDI Interrupt Functions" have already been approved, but those resource management interfaces have never been implemented. This project is providing the implementation of those interfaces, with minor changes. The changes include: - Generalization of the proposed callback interfaces, so they can be shared by both IRM and any future projects. - Removed semantics originally proposed to temporarily disable callbacks. - An additional function (ddi_intr_set_nreq()) for explicitly setting how many interrupt vectors a device driver requests. The interface described in PSARC/2007/453 "MSI-X interrupt limit override" provides a temporary workaround for device drivers to request more interrupts. And the workaround is currently used. This project supercedes the functionality of this workaround, and drivers using the workaround should ultimately be converted. But in the meantime, the workaround is preserved and still works in conjunction with this project. 4.1.4 Competitive Analysis This project is important for the overall IO performance on all of Sun's platforms in order to remain competitive in the marketplace. Modern IO bus technologies support large numbers of interrupts. A single PCI or PCIe device could use up to 32 MSI interrupts, or 2048 MSI-X interrupts. Without this project, Solaris only gives at most 2 interrupts per each device. We are limiting the IO throughput of our systems by not having this feature, and by not giving some devices nearly as many interrupt vectors as they can support and utilize. Current Sun platforms have limited numbers of interrupts to give to the devices. SPARC PCIe platforms have 256 interrupts per root complex, and x64 systems only have 256 interrupts per each processor. But we have future platforms in development with many thousands of interrupt vectors available. We need this feature so that we can achieve the full potential of advanced MSI and MSI-X devices on such platforms. And in the meantime we already have some devices which already could benefit from getting more than the current 2 interrupt vector limit. 4.2 Technical Description 4.2.1 Architecture The basic strategy is to organize representations of each supply of interrupt vectors in the system with the drivers who consume them. And to compute the optimal way to divide each supply amongst those devices. When a driver is attached or detached, the computations are performed to rebalance the system. And, drivers can explicitly change how many interrupts they want in response to load. The computations seek to maximize the use of the system's interrupts and derive fair allocations for each device. Drivers are notified using a callback mechanism when the computations result in giving the driver more or less interrupts. The main components of this project are: - NDI interfaces for nexus drivers to create or destroy the descriptions of individual supplies of interrupt vectors. - DDI interfaces for drivers to register or unregister callback mechanisms. The callbacks notify them of changes to their interrupt availability. - DDI interface for drivers to explicitly set their requested number of interrupts dynamically. - A mechanism to map individual devices to interrupt supplies. - Background threads which keep the allocations of interrupts from each supply to each device optimized and balanced. - Routines to initialize the new feature at boottime. - An MDB debugging module to display the status of how the interrupt subsystem has been balanced. Each supply of interrupt vectors in the system is described by a data structure (ddi_irm_pool_t), representing one pool of interrupts that can be shared by multiple devices. And for each device that maps to an interrupt pool, a data structure (ddi_irm_req_t) represents how many interrupts it wants versus how many it received. Nexus drivers create the interrupt pools, and then they map individual devices to them through the existing bus nexus driver INTROP feature. The request data structures are created internally and associated with the interrupt pools when devices are attached and mapped to a pool. Existing device drivers do not benefit, and they continue to only get the same small number of interrupt vectors that they currently receive. In order to benefit, they must be modified with optional enhancements so that they can participate. To participate, they must provide a new callback mechanism so that the system can notify them when they have been given more or less interrupt vectors. A modified driver first registers a generic callback handler so it can receive notifications of interrupt availability changes. Then it calls ddi_intr_alloc() to request an initial number of interrupt vectors. If the system has the necessary support (from nexus drivers), then it associates the requesting driver with an interrupt pool. Through this association, the system will compute if the driver gets more interrupts. A driver may initially receive all the interrupts it requested, or it may receive callbacks at a later time (post attach(9F)) notifying it when more interrupt vectors are available. More could be available if other devices were removed, or if workload changes cause other drivers to reduce their requests. But to qualify for additional interrupts, a driver must also yield and call ddi_intr_free() when necessary. This may occur if another device is inserted, or if changes in workload cause other drivers to need more. This project introduces the interfaces for nexus drivers to describe the interrupt pools, and for device drivers to engage with those interrupt pools to possibly receive more resources. Plus all of the additional implementation behind the scenes to perform the related computations when necessary. 4.2.2 Interfaces 4.2.2.1 Exported Interfaces Interface Stability Comments -----------------------------------+----------+------------------------- ndi_irm_create() Committed Create an IRM pool ndi_irm_destroy() Committed Destroy an IRM pool DDI_INTROP_GETPOOL Committed Get IRM pool INTROP ddi_cb_register() Committed Install callback handler ddi_cb_unregister() Committed Remove callback handler ddi_cb_action_t Committed Callback action type ddi_cb_flags_t Committed Callback flags type ddi_cb_func_t Committed Callback function type ddi_cb_handle_t Committed Callback handle type ddi_intr_set_nreq() Committed Set IRM request size -----------------------------------+----------+------------------------- 4.2.2.2 Imported Interfaces Interface Stability Comments -----------------------------------+----------+------------------------- ddi_intr_alloc() Committed Added hooks into IRM ddi_intr_free() Committed Added hooks into IRM -----------------------------------+----------+------------------------- 4.2.2.3 Removed Interfaces These interfaces were previously approved, but never implemented. And they are now superceded by the interfaces of this project. Interface Stability Comments -----------------------------------+----------+------------------------- ddi_intr_register_management_cb() Committed Register callback ddi_intr_unregister_management_cb() Committed Unregister callback ddi_intr_enable_management_cb() Committed Enable callback ddi_intr_disable_management_cb() Committed Disable callback -----------------------------------+----------+------------------------- 5. References This Project Implements, Extends, or Replaces these Projects: - PSARC/2004/253: Advanced DDI Interrupt Functions - PSARC/2007/453: MSI-X interrupt limit override (A future RFE will remove the workaround once all consumers have been converted to use this project.) Consumers of this Project: - PSARC/2008/181: Solaris Hotplug Framework (Uses this project's generic callback mechanism) - IRM Enhancements for Atlas/Neptune Driver (Convert from existing workaround to use this project) - x86 APIC Expansion and IRM Support (Provide interrupt pool definitions on x86. The scope of this project is to only deliver interrupt pools on SPARC.) Design and Implementation Specification of this Project: - http://pciexpress.sfbay/intr/docs/irm/irm_design.txt 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From David.Kahn@sun.com Tue Oct 7 20:13:01 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m983D019020453 for ; Tue, 7 Oct 2008 20:13:00 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m983CwR3027209; Wed, 8 Oct 2008 11:12:59 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8E00H01GXMED00@nwk-avmta-2.sfbay.sun.com>; Tue, 07 Oct 2008 20:12:58 -0700 (PDT) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8E00CX1GXMY8D0@nwk-avmta-2.sfbay.sun.com>; Tue, 07 Oct 2008 20:12:58 -0700 (PDT) Received: from dtmail.sfbay.sun.com (pkg.SFBay.Sun.COM [129.146.90.56]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m983Cv7r041143; Tue, 07 Oct 2008 20:12:57 -0700 (PDT) Received: from [192.168.0.9] (noho.SFBay.Sun.COM [10.6.92.101]) by dtmail.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m983CvBY011097; Tue, 07 Oct 2008 20:12:57 -0700 (PDT) Date: Tue, 07 Oct 2008 20:12:56 -0700 From: David Kahn Subject: Re: PSARC/2008/628 Interrupt Resource Management To: Artem Kachitchkine Cc: psarc-ext@sun.com, ddi-irm-interest@sun.com Message-id: <48EC2538.80207@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) Status: RO Content-Length: 703 I would like to see a minor clarification to the nreq argument in ddi_set_nreq(9f). It's not clear to me if nreq means I want nreq more interrupts than I have now, or if it means I want nreq total interrupts, including the n that are already allocated to me. I think it's the latter, but the man page should be more specific about it by stating that nreq is the total number of interrupt resources (including those already allocated to the device instance) that this instance of the device driver would like to use. Sorry I must have missed this in my pre-submission review of the materials. I think it can be fixed easily without extending the timer, since it's a relatively minor comment. -David From David.Kahn@Sun.COM Tue Oct 7 20:26:38 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m983QcJJ020607 for ; Tue, 7 Oct 2008 20:26:38 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m983QaIc063563; Tue, 7 Oct 2008 21:26:37 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8E0090XHKCKI00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 07 Oct 2008 20:26:36 -0700 (PDT) Received: from dm-sfbay-01.sfbay.sun.com ([129.145.155.118]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8E000I0HKB12C0@nwk-avmta-1.sfbay.Sun.COM>; Tue, 07 Oct 2008 20:26:35 -0700 (PDT) Received: from dtmail.sfbay.sun.com (pkg.SFBay.Sun.COM [129.146.90.56]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m983QYdi034134; Tue, 07 Oct 2008 20:26:34 -0700 (PDT) Received: from [192.168.0.9] (noho.SFBay.Sun.COM [10.6.92.101]) by dtmail.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m983QYL1011176; Tue, 07 Oct 2008 20:26:34 -0700 (PDT) Date: Tue, 07 Oct 2008 20:26:33 -0700 From: David Kahn Subject: Re: PSARC/2008/628 Interrupt Resource Management To: Artem Kachitchkine Cc: psarc-ext@Sun.COM, ddi-irm-interest@Sun.COM Message-id: <48EC2869.8070609@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) Status: RO Content-Length: 216 one more question. It's not clear to me how the irm pool stuff works. Who actually allocates MSI/X address/data values when these pools are used/defined? Did I miss that part of the design or interface? -David From Govinda.Tatti@sun.com Tue Oct 7 20:45:15 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m983jEe7020710 for ; Tue, 7 Oct 2008 20:45:15 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m983j90o005101; Wed, 8 Oct 2008 11:45:13 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8E00J0JIFC0S00@nwk-avmta-2.sfbay.sun.com>; Tue, 07 Oct 2008 20:45:12 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8E00IPQIFCA200@nwk-avmta-2.sfbay.sun.com>; Tue, 07 Oct 2008 20:45:12 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m983jBA8012674; Tue, 07 Oct 2008 20:45:11 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8E00801IB2RB00@fe-sfbay-10.sun.com> (original mail from Govinda.Tatti@Sun.COM); Tue, 07 Oct 2008 20:45:11 -0700 (PDT) Received: from [127.0.0.1] ([67.188.235.218]) by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8E0008OIFAZWA0@fe-sfbay-10.sun.com>; Tue, 07 Oct 2008 20:45:11 -0700 (PDT) Date: Tue, 07 Oct 2008 20:45:21 -0700 From: Govinda Tatti Subject: Re: PSARC/2008/628 Interrupt Resource Management In-reply-to: <48EC2538.80207@sun.com> Sender: Govinda.Tatti@sun.com To: David Kahn Cc: Artem Kachitchkine , psarc-ext@sun.com, ddi-irm-interest@sun.com Message-id: <48EC2CD1.6050804@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EC2538.80207@sun.com> User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) Status: RO Content-Length: 768 > I would like to see a minor clarification to the > nreq argument in ddi_set_nreq(9f). > > It's not clear to me if nreq means I want nreq > more interrupts than I have now, or if it means > I want nreq total interrupts, including the n > that are already allocated to me. I think it's > the latter, but the man page should be more > specific about it by stating that nreq is the > total number of interrupt resources (including > those already allocated to the device instance) > that this instance of the device driver would > like to use. Thanks David. yes, nreq is the total number of interrupt resources that the driver would like to use including those already allocated to it. ddi_intr_set_nreq(9f) man page will be updated with this change. Cheers GOVINDA From Govinda.Tatti@sun.com Tue Oct 7 21:01:05 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m98414bt021250 for ; Tue, 7 Oct 2008 21:01:05 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m98411TX024379; Wed, 8 Oct 2008 05:01:03 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8E00J01J5QTY00@nwk-avmta-2.sfbay.sun.com>; Tue, 07 Oct 2008 21:01:02 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8E00IPIJ5QA210@nwk-avmta-2.sfbay.sun.com>; Tue, 07 Oct 2008 21:01:02 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m984117p013232; Tue, 07 Oct 2008 21:01:01 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8E00L01IYVBA00@fe-sfbay-09.sun.com> (original mail from Govinda.Tatti@Sun.COM); Tue, 07 Oct 2008 21:01:01 -0700 (PDT) Received: from [127.0.0.1] ([67.188.235.218]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8E00MMJJ5OZB70@fe-sfbay-09.sun.com>; Tue, 07 Oct 2008 21:01:01 -0700 (PDT) Date: Tue, 07 Oct 2008 21:01:11 -0700 From: Govinda Tatti Subject: Re: PSARC/2008/628 Interrupt Resource Management In-reply-to: <48EC2869.8070609@sun.com> Sender: Govinda.Tatti@sun.com To: David Kahn Cc: Artem Kachitchkine , psarc-ext@sun.com, ddi-irm-interest@sun.com Message-id: <48EC3087.5070302@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EC2869.8070609@sun.com> User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) Status: RO Content-Length: 580 > one more question. It's not clear to me how > the irm pool stuff works. > > Who actually allocates MSI/X address/data values > when these pools are used/defined? > > Did I miss that part of the design or interface? This is more of an implementation issue. DDI interrupt framework just manages the interrupt pool (# of MSI/X vectors), but it is not responsible for assigning or programming any MSI/X address and data values. This project is not changing this part and it is still all done by the platform specific layers such PCIe nexus drivers (px, pci, npe). Cheers GOVINDA From gdamore@sun.com Tue Oct 7 21:15:49 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m984FmY7022178 for ; Tue, 7 Oct 2008 21:15:48 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m984FhcY028508; Wed, 8 Oct 2008 05:15:47 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8E00F07JUANI00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 07 Oct 2008 21:15:46 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8E000P5JUA1ED0@nwk-avmta-1.sfbay.Sun.COM>; Tue, 07 Oct 2008 21:15:46 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m984FkOr009318; Tue, 07 Oct 2008 21:15:46 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8E00801JU7VE00@fe-sfbay-09.sun.com> (original mail from gdamore@sun.com) ; Tue, 07 Oct 2008 21:15:46 -0700 (PDT) Received: from [10.7.251.172] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8E00M9UJU9ZBA0@fe-sfbay-09.sun.com>; Tue, 07 Oct 2008 21:15:45 -0700 (PDT) Date: Tue, 07 Oct 2008 21:12:31 -0700 From: "Garrett D'Amore" Subject: Re: PSARC/2008/628 Interrupt Resource Management In-reply-to: <48EC2056.9000505@sun.com> Sender: Garrett.Damore@sun.com To: Artem Kachitchkine Cc: psarc-ext@sun.com, ddi-irm-interest@sun.com Message-id: <48EC332F.2020905@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EC2056.9000505@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 15068 (Preface: I've not read the material in 2004/253.) The interfaces are marked Committed. I have some concerns with this, as I read the project details. One of the areas that really concerns me is the fact that drivers have to deal with a situation where the total number of interrupts is reduced. (*Increasing* interrupts seems free from the issues I am concerned with.) My read is that the interrupts are taken from them, without any way for drivers to decline (or even get a warning) in advance. I'm interested to know how this is used with real hardware. For example, one potential use of this facility is to support multiple receive rings. However, now the problem is that there is a window of time where an extra ring may be "orphaned". What happens to those packets that are received there? (What happens to the interrupt that the hardware issues, for that matter?) What if a device has to have a minimum number of interrupts once it configures them? Should a driver be able to specify that once allocated, certain interrupts cannot ever be returned to the system? How will this feature interact with Crossbow resource allocation? Do we have any examples of the two working together yet? I'd feel a lot better if we had some more complete description of how this is used some typical device drivers, with some real experience with them, before raising the commitment. If the project team has some sample implementations that can use this, then I might change my position. But in the absence of that, I'd feel better with an Uncommitted binding while we get some experience with the APIs at hand. -- Garrett Artem Kachitchkine wrote: > I'm sponsoring this fasttrack for Scott Carter. It is set to timeout > on 10/15/2008. Given that this is a relatively minor amendment to a > larger, approved case PSARC/2004/253 "Advanced DDI Interrupt > Functions", I think it qualifies as a fasttrack. As the discussion > progresses, extending the timer or even promoting to a full case may > be considered. > > In addition to this proposal, three man page files can be found in the > case directory: > > ddi_cb_register.9f.txt > ddi_intr_set_nreq.9f.txt > ndi_irm_create.9f.txt > > -Artem > > Template Version: @(#)sac_nextcase %I% %G% SMI > This information is Copyright 2008 Sun Microsystems > 1. Introduction > 1.1. Project/Component Working Name: > Interrupt Resource Management > 1.2. Name of Document Author/Supplier: > Author: Scott Carter > 1.3 Date of This Document: > 07 October, 2008 > 4. Technical Description > 4.1 Project Description > > This project delivers the resource management feature originally > defined in PSARC/2004/253 "Advanced DDI Interrupt Functions" > with minor changes (see 4.1.3 for details). > > It provides a mechanism for device drivers to get more interrupt > vectors (and increased performance). By participating with the new > feature, their number of available interrupts becomes dynamic and > can increase or decrease. The goal is to maximize utilization of > interrupt vectors in a fair manner, and rebalance the allocations > whenever devices are added or removed from the system. > > > 4.1.1 Definition > > The project defines new DDI interfaces for device drivers to register > and unregister a generic callback handler. If a driver has registered > a handler, then it can be notified when its number of interrupts has > been increased or decreased. > > How many interrupt vectors a device driver wants is set by the initial > number it attempts to allocate through ddi_intr_alloc(). And a driver > can explicitly adjust this number at later times with a new DDI function > that is introduced by this project. > > The project also defines new NDI interfaces for nexus drivers to define > the supplies of interrupt vectors that are available in the system. A > supply can be associated with a specific IO bus (e.g. PCIe root complex) > or it could be a global supply shared by all devices. > > Along with the new DDI and NDI interfaces, the project also delivers > an MDB debugging module with two new debug macros. One macro is used > to display statistics on all of the defined interrupt supplies, and the > other displays statistics about how a supply is divided up between the > drivers that map to it. > > > 4.1.2 Motivation, Goals, and Requirements > > Currently, interrupt vectors are given to device drivers in very small > numbers. This is to avoid exhausting a system's supply before all the > devices have been attached, and to keep interrupt vectors in reserve for > later hotplugs. Interrupts are given so conservatively because there > is no mechanism to take them back later unless a driver detaches. > > Fewer interrupt vectors means less IO performance. More interrupt > vectors means more parallelism for handling interrupt conditions. So > the motivation for this project is to increase IO performance. And > the goal of this project is to maximize the allocation of interrupts > given to each device, but in a way which is fair and balanced across > the full set of attached devices. > > The requirement of this project is: > > - Provide a mechanism for drivers to get more interrupts. > > The feature is optional so drivers that don't use it still work even > if the system has implemented support for the feature. And conversely, > drivers that do use it also work if the system does not implement the > support. > > A full implementation of the platform level support will be provided > by this project for PCIe IO bus drivers on SPARC. > > > 4.1.3 Changes From the Previous Case > > This project is a micro change to the approved case PSARC/2004/253 > "Advanced DDI Interrupt Interfaces". The existing DDI interrupt > interfaces are not changed. But some new DDI interrupt interfaces > are added, which extend the capabilities of the existing interfaces. > > The interfaces described in section 6.3 of the specifications for > PSARC/2004/253 "Advanced DDI Interrupt Functions" have already been > approved, but those resource management interfaces have never been > implemented. This project is providing the implementation of those > interfaces, with minor changes. > > The changes include: > > - Generalization of the proposed callback interfaces, so they > can be shared by both IRM and any future projects. > > - Removed semantics originally proposed to temporarily disable > callbacks. > > - An additional function (ddi_intr_set_nreq()) for explicitly > setting how many interrupt vectors a device driver requests. > > The interface described in PSARC/2007/453 "MSI-X interrupt limit > override" provides a temporary workaround for device drivers to request > more interrupts. And the workaround is currently used. This project > supercedes the functionality of this workaround, and drivers using the > workaround should ultimately be converted. But in the meantime, the > workaround is preserved and still works in conjunction with this > project. > > > 4.1.4 Competitive Analysis > > This project is important for the overall IO performance on all of > Sun's platforms in order to remain competitive in the marketplace. > > Modern IO bus technologies support large numbers of interrupts. A > single PCI or PCIe device could use up to 32 MSI interrupts, or 2048 > MSI-X interrupts. Without this project, Solaris only gives at most > 2 interrupts per each device. We are limiting the IO throughput of > our systems by not having this feature, and by not giving some devices > nearly as many interrupt vectors as they can support and utilize. > > Current Sun platforms have limited numbers of interrupts to give to > the devices. SPARC PCIe platforms have 256 interrupts per root complex, > and x64 systems only have 256 interrupts per each processor. But we > have future platforms in development with many thousands of interrupt > vectors available. We need this feature so that we can achieve the > full potential of advanced MSI and MSI-X devices on such platforms. > And in the meantime we already have some devices which already could > benefit from getting more than the current 2 interrupt vector limit. > > > 4.2 Technical Description > > 4.2.1 Architecture > > The basic strategy is to organize representations of each supply of > interrupt vectors in the system with the drivers who consume them. And > to compute the optimal way to divide each supply amongst those devices. > > When a driver is attached or detached, the computations are performed > to rebalance the system. And, drivers can explicitly change how many > interrupts they want in response to load. The computations seek to > maximize the use of the system's interrupts and derive fair allocations > for each device. Drivers are notified using a callback mechanism when > the computations result in giving the driver more or less interrupts. > > The main components of this project are: > > - NDI interfaces for nexus drivers to create or destroy the > descriptions of individual supplies of interrupt vectors. > > - DDI interfaces for drivers to register or unregister callback > mechanisms. The callbacks notify them of changes to their > interrupt availability. > > - DDI interface for drivers to explicitly set their requested > number of interrupts dynamically. > > - A mechanism to map individual devices to interrupt supplies. > > - Background threads which keep the allocations of interrupts > from each supply to each device optimized and balanced. > > - Routines to initialize the new feature at boottime. > > - An MDB debugging module to display the status of how the > interrupt subsystem has been balanced. > > Each supply of interrupt vectors in the system is described by a data > structure (ddi_irm_pool_t), representing one pool of interrupts that > can be shared by multiple devices. And for each device that maps to > an interrupt pool, a data structure (ddi_irm_req_t) represents how > many interrupts it wants versus how many it received. > > Nexus drivers create the interrupt pools, and then they map individual > devices to them through the existing bus nexus driver INTROP feature. > The request data structures are created internally and associated with > the interrupt pools when devices are attached and mapped to a pool. > > Existing device drivers do not benefit, and they continue to only get > the same small number of interrupt vectors that they currently receive. > In order to benefit, they must be modified with optional enhancements > so that they can participate. To participate, they must provide a new > callback mechanism so that the system can notify them when they have > been given more or less interrupt vectors. > > A modified driver first registers a generic callback handler so it can > receive notifications of interrupt availability changes. Then it calls > ddi_intr_alloc() to request an initial number of interrupt vectors. If > the system has the necessary support (from nexus drivers), then it > associates the requesting driver with an interrupt pool. Through this > association, the system will compute if the driver gets more interrupts. > > A driver may initially receive all the interrupts it requested, or it > may receive callbacks at a later time (post attach(9F)) notifying it > when more interrupt vectors are available. More could be available if > other devices were removed, or if workload changes cause other drivers > to reduce their requests. > > But to qualify for additional interrupts, a driver must also yield and > call ddi_intr_free() when necessary. This may occur if another device > is inserted, or if changes in workload cause other drivers to need more. > > This project introduces the interfaces for nexus drivers to describe > the interrupt pools, and for device drivers to engage with those > interrupt pools to possibly receive more resources. Plus all of the > additional implementation behind the scenes to perform the related > computations when necessary. > > > 4.2.2 Interfaces > > 4.2.2.1 Exported Interfaces > > Interface Stability Comments > -----------------------------------+----------+------------------------- > ndi_irm_create() Committed Create an IRM pool > ndi_irm_destroy() Committed Destroy an IRM pool > DDI_INTROP_GETPOOL Committed Get IRM pool INTROP > > ddi_cb_register() Committed Install callback handler > ddi_cb_unregister() Committed Remove callback handler > ddi_cb_action_t Committed Callback action type > ddi_cb_flags_t Committed Callback flags type > ddi_cb_func_t Committed Callback function type > ddi_cb_handle_t Committed Callback handle type > > ddi_intr_set_nreq() Committed Set IRM request size > -----------------------------------+----------+------------------------- > > > 4.2.2.2 Imported Interfaces > > Interface Stability Comments > -----------------------------------+----------+------------------------- > ddi_intr_alloc() Committed Added hooks into IRM > ddi_intr_free() Committed Added hooks into IRM > -----------------------------------+----------+------------------------- > > > 4.2.2.3 Removed Interfaces > > These interfaces were previously approved, but never implemented. > And they are now superceded by the interfaces of this project. > > Interface Stability Comments > -----------------------------------+----------+------------------------- > ddi_intr_register_management_cb() Committed Register callback > ddi_intr_unregister_management_cb() Committed Unregister callback > ddi_intr_enable_management_cb() Committed Enable callback > ddi_intr_disable_management_cb() Committed Disable callback > -----------------------------------+----------+------------------------- > > > 5. References > > This Project Implements, Extends, or Replaces these Projects: > > - PSARC/2004/253: Advanced DDI Interrupt Functions > - PSARC/2007/453: MSI-X interrupt limit override > (A future RFE will remove the workaround once all > consumers have been converted to use this project.) > > Consumers of this Project: > > - PSARC/2008/181: Solaris Hotplug Framework > (Uses this project's generic callback mechanism) > - IRM Enhancements for Atlas/Neptune Driver > (Convert from existing workaround to use this project) > - x86 APIC Expansion and IRM Support > (Provide interrupt pool definitions on x86. The scope of > this project is to only deliver interrupt pools on SPARC.) > > Design and Implementation Specification of this Project: > - http://pciexpress.sfbay/intr/docs/irm/irm_design.txt > > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > ON > 6.5. ARC review type: FastTrack > 6.6. ARC Exposure: open > From Scott.Carter@sun.com Tue Oct 7 23:17:22 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m986HMZ7025363 for ; Tue, 7 Oct 2008 23:17:22 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m986HLTT009993; Tue, 7 Oct 2008 23:17:22 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8E00407PGXX700@brm-avmta-1.central.sun.com>; Wed, 08 Oct 2008 00:17:21 -0600 (MDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8E00FCDPGWZB50@brm-avmta-1.central.sun.com>; Wed, 08 Oct 2008 00:17:21 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m986HK6h017267; Tue, 07 Oct 2008 23:17:20 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8E00601PDUSR00@fe-sfbay-09.sun.com> (original mail from Scott.Carter@Sun.COM); Tue, 07 Oct 2008 23:17:20 -0700 (PDT) Received: from [129.150.17.136] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8E00AEWPGS8UC0@fe-sfbay-09.sun.com>; Tue, 07 Oct 2008 23:17:20 -0700 (PDT) Date: Tue, 07 Oct 2008 23:18:40 -0700 From: Scott Carter Subject: Re: PSARC/2008/628 Interrupt Resource Management In-reply-to: <48EC2538.80207@sun.com> Sender: Scott.Carter@sun.com To: David Kahn Cc: Artem Kachitchkine , psarc-ext@sun.com, ddi-irm-interest@sun.com Message-id: <48EC50C0.4080408@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-Accept-Language: en-us, en X-PMX-Version: 5.4.1.325704 References: <48EC2538.80207@sun.com> User-Agent: Mozilla Thunderbird 0.7.2 (Windows/20040707) Status: RO Content-Length: 1754 David Kahn wrote: > I would like to see a minor clarification to the > nreq argument in ddi_set_nreq(9f). > > It's not clear to me if nreq means I want nreq > more interrupts than I have now, or if it means > I want nreq total interrupts, including the n > that are already allocated to me. I think it's > the latter, but the man page should be more > specific about it by stating that nreq is the > total number of interrupt resources (including > those already allocated to the device instance) > that this instance of the device driver would > like to use. You are correct, it is the latter. The 'nreq' value is the total number of interrupts that the device wants. If a driver initially wanted 8 interrupts, it would have called ddi_intr_alloc(9F) with a 'count' of 8 to request then. That value becomes the initial 'nreq' parameter for the driver. If at some later time the driver changes its mind and decides it only needs to use a total of 4 interrupts, it should call ddi_intr_set_nreq(9F) with an 'nreq' value of 4. There is a paragraph in the ddi_intr_set_nreq(9F) manpage that tries to define what 'nreq' is. I will improve it, perhaps like this: The nreq parameter is the total number of interrupts that the driver requests from the system. Initially it is set to be equal to the count parameter when the driver first calls the ddi_intr_alloc(9F) function. If the driver detects changes in its workload and it decides it needs more or less interrupt vectors, then it may call the function ddi_intr_set_nreq() to change its nreq parameter. The function ddi_intr_set_nreq() replaces the current nreq value with whatever new value is specified. I hope that wording is clear enough? -- Scott From Scott.Carter@sun.com Tue Oct 7 23:35:37 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m986Za5s025588 for ; Tue, 7 Oct 2008 23:35:36 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m986Z3b5009650; Wed, 8 Oct 2008 07:35:34 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8E0080ZQB9GM00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 07 Oct 2008 23:35:33 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8E004MVQB4GV40@nwk-avmta-1.sfbay.Sun.COM>; Tue, 07 Oct 2008 23:35:28 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m986ZS6U012454; Tue, 07 Oct 2008 23:35:28 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8E00101Q7TMO00@fe-sfbay-10.sun.com> (original mail from Scott.Carter@Sun.COM); Tue, 07 Oct 2008 23:35:28 -0700 (PDT) Received: from [129.150.17.136] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8E00IXXQB376A0@fe-sfbay-10.sun.com>; Tue, 07 Oct 2008 23:35:27 -0700 (PDT) Date: Tue, 07 Oct 2008 23:36:51 -0700 From: Scott Carter Subject: Re: PSARC/2008/628 Interrupt Resource Management In-reply-to: <48EC2869.8070609@sun.com> Sender: Scott.Carter@sun.com To: David Kahn Cc: Artem Kachitchkine , psarc-ext@sun.com, ddi-irm-interest@sun.com Message-id: <48EC5503.4000203@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-Accept-Language: en-us, en X-PMX-Version: 5.4.1.325704 References: <48EC2869.8070609@sun.com> User-Agent: Mozilla Thunderbird 0.7.2 (Windows/20040707) Status: RO Content-Length: 1576 David Kahn wrote: > one more question. It's not clear to me how > the irm pool stuff works. > > Who actually allocates MSI/X address/data values > when these pools are used/defined? > > Did I miss that part of the design or interface? > > -David The "Design and Implementation" spec has more about the mechanics of defining interrupt pools, mapping requests from devices into the pools, and keeping the pools balanced. Although in spite of all of its excruciating detail, I can see how the main point you are asking isn't explained there in a simple way. The interrupt pool is just an abstraction to represent some supply of interrupt vectors in the system. It describes how many they are, of what type, and who supplied them. And the request structures are an abstraction of how many interrupts a particular driver wants and gets from its respective interrupt pool. By mapping the two types of structures together, IRM can generically compute a fair number of interrupts to give to each device from its pool, in a way which is balancing together the needs of all the requests in the same pool. IRM then uses callbacks to tell the drivers when there is a change in those computations. During the callbacks, when the drivers then respond by calling ddi_intr_alloc(9F) or ddi_intr_free(9F), those functions are implemented by sending INTROPs to whichever nexus driver owns and manages the actual vectors represented by the interrupt pool. And there are private implementations in those nexus drivers to implement the actual vector assignments at that level. -- Scott From David.Kahn@sun.com Wed Oct 8 00:47:51 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m987lpnb027096 for ; Wed, 8 Oct 2008 00:47:51 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m987lnjL007211; Wed, 8 Oct 2008 00:47:51 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8E00B03TNRQ300@brm-avmta-1.central.sun.com>; Wed, 08 Oct 2008 01:47:51 -0600 (MDT) Received: from dm-sfbay-01.sfbay.sun.com ([129.145.155.118]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8E00F75TNQZFA0@brm-avmta-1.central.sun.com>; Wed, 08 Oct 2008 01:47:50 -0600 (MDT) Received: from dtmail.sfbay.sun.com (pkg.SFBay.Sun.COM [129.146.90.56]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m987loWa002922; Wed, 08 Oct 2008 00:47:50 -0700 (PDT) Received: from [192.168.0.9] (noho.SFBay.Sun.COM [10.6.92.101]) by dtmail.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m987lnM5017866; Wed, 08 Oct 2008 00:47:49 -0700 (PDT) Date: Wed, 08 Oct 2008 00:47:48 -0700 From: David Kahn Subject: Re: PSARC/2008/628 Interrupt Resource Management In-reply-to: <48EC50C0.4080408@Sun.COM> To: Scott Carter Cc: Artem Kachitchkine , psarc-ext@sun.com, ddi-irm-interest@sun.com Message-id: <48EC65A4.2040400@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EC2538.80207@sun.com> <48EC50C0.4080408@Sun.COM> User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) Status: RO Content-Length: 3520 Scott, The problem with the description of the nreq parameter in the man page is that it doesn't say what it is in that function call. I don't know what it means that it's "initially set to be equal ...". That isn't how we specify an interface. You might want to replace DESCRIPTION section with something like this: (basically replacing your 2nd paragraph with 2 new paragraphs.) The ddi_intr_set_nreq() function changes the number of interrupts requested by a device driver instance. The nreq parameter is the total number of interrupt resources that this instance of the device driver would like to have available. nreq includes any interrupt resources already allocated by the driver. For example, the driver instance is currently using two MSI-X vectors, and it wants access to four additional unique MSI-X vectors. In that case it should call this function with nreq parameter set to the value six. The nreq parameter can be any value between 1 and the maximum number of interrupts supported by the device hardware, as reported by a call to the ddi_intr_get_nintrs(9F) function. The driver will receive a callback notifying it in cases when it must release any previously allocated interrupts, or when it is allowed to allocate more interrupts as a result of its new nreq parameter. The ddi_intr_set_nreq() function is not supported unless a driver is already consuming interrupts, and if it has a registered callback handler that can process actions related to changes in interrupt availability. See ddi_cb_register(9F) for an explanation on how to enable this functionality. (and this part might go into an implementation note section, because it's not part of the interface, but describes how the interface is implemented.) The implementation tracks the number of interrupt resources that a device driver instance wants by initially using the .... As the driver processes more data streams and i/o load, it may decide that it wants more interrupt resources later. As that data and i/o load diminishes, it may decide that it has extra interrupt resources that can be returned to the system for use by other devices. ... Here's what's there now: DESCRIPTION The ddi_intr_set_nreq() function changes the number of interrupts requested by a device driver. The nreq parameter is initially set to be equal to the count parameter when the device driver first calls the ddi_intr_alloc(9F) function. If the device driver detects changes in its workload, and it decides it needs more or less interrupt vectors, then it may use the ddi_intr_set_nreq() function to adjust its nreq parameter at arbitrary times. The nreq parameter can be any value between 1 and the maximum number of interrupts supported by the device hardware, as reported by a call to the ddi_intr_get_nintrs(9F) function. The driver will receive a callback notifying it in cases when it must release any previously allocated interrupts, or when it is allowed to allocate more interrupts as a result of its new nreq parameter. The ddi_intr_set_nreq() function is not supported unless a driver is already consuming interrupts, and if it has a registered callback handler that can process actions related to changes in interrupt availability. See ddi_cb_register(9F) for an explanation on how to enable this functionality. From David.Kahn@sun.com Wed Oct 8 00:58:04 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m987w3tx027386 for ; Wed, 8 Oct 2008 00:58:03 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m987vxGn008256; Wed, 8 Oct 2008 08:58:02 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8E0080XU4N5H00@nwk-avmta-2.sfbay.sun.com>; Wed, 08 Oct 2008 00:57:59 -0700 (PDT) Received: from dm-sfbay-01.sfbay.sun.com ([129.145.155.118]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8E00I7EU4MA0F0@nwk-avmta-2.sfbay.sun.com>; Wed, 08 Oct 2008 00:57:59 -0700 (PDT) Received: from dtmail.sfbay.sun.com (pkg.SFBay.Sun.COM [129.146.90.56]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m987vw2X006025; Wed, 08 Oct 2008 00:57:58 -0700 (PDT) Received: from [192.168.0.9] (noho.SFBay.Sun.COM [10.6.92.101]) by dtmail.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m987vs9x017929; Wed, 08 Oct 2008 00:57:58 -0700 (PDT) Date: Wed, 08 Oct 2008 00:57:54 -0700 From: David Kahn Subject: Re: PSARC/2008/628 Interrupt Resource Management In-reply-to: <48EC50C0.4080408@Sun.COM> To: Scott Carter Cc: Artem Kachitchkine , psarc-ext@sun.com, ddi-irm-interest@sun.com Message-id: <48EC6802.4000001@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EC2538.80207@sun.com> <48EC50C0.4080408@Sun.COM> User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) Status: RO Content-Length: 1057 Just in case this wasn't clear from my last email. Scott Carter wrote: > The nreq parameter is the total number of interrupts > that the driver requests from the system. Initially > it is set to be equal to the count parameter when the > driver first calls the ddi_intr_alloc(9F) function. How is an argument to a function initially set to some value? It isn't. It doesn't have any value or persistence as an argument to an interface. "it" is vague and it doesn't belong in the definition. At best, it's an implementation note that defines what the implementation is supposed to do. We are mixing the internal ddi implementation with the normative definition of this interface. Also, is "it" set from the first successful call to ddi_intr_alloc()? What if the strict behaviour flag is set and the function fails, for example? What if no resources are available at all? How much detail do you need to give here? (I don't really care what the answers are to these rhetorical questions, only that the interface is well-defined.) -David From pdurrant@solarflare.com Wed Oct 8 02:35:59 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m989Zx1m000988 for ; Wed, 8 Oct 2008 02:35:59 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m989Zulr027854; Wed, 8 Oct 2008 03:35:58 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8E00C05YNX9F00@nwk-avmta-2.sfbay.sun.com>; Wed, 08 Oct 2008 02:35:57 -0700 (PDT) Received: from sca-ea-mail-1.sun.com ([192.18.43.24]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8E00BWNYNXLP00@nwk-avmta-2.sfbay.sun.com>; Wed, 08 Oct 2008 02:35:57 -0700 (PDT) Received: from relay24.sun.com (relay24.sun.com [192.12.251.74] (may be forged)) by sca-ea-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m989X46q024856; Wed, 08 Oct 2008 09:35:56 +0000 (GMT) Received: from mms25es.mms.us.syntegra.com ([150.143.232.90] [150.143.232.90]) by relay24i.sun.com with ESMTP id BT-MMP-5131021; Wed, 08 Oct 2008 09:35:56 +0000 (Z) Received: from relay22.sun.com (relay22.sun.com [192.12.251.34]) by mms25es.mms.us.syntegra.com with ESMTP id BT-MMP-126783982; Wed, 08 Oct 2008 09:35:56 +0000 (Z) Received: from exchange.solarflare.com ([216.237.3.220] [216.237.3.220]) by relay22i.sun.com with ESMTP id BT-MMP-58228544; Wed, 08 Oct 2008 09:35:56 +0000 (Z) Received: from [10.17.20.48] ([10.17.20.48]) by exchange.solarflare.com over TLS secured channel with Microsoft SMTPSVC(6.0.3790.3959); Wed, 08 Oct 2008 02:35:20 -0700 Date: Wed, 08 Oct 2008 10:35:52 +0100 From: Paul Durrant Subject: Re: PSARC/2008/628 Interrupt Resource Management In-reply-to: <48EC2056.9000505@sun.com> To: Artem Kachitchkine Cc: psarc-ext@sun.com, ddi-irm-interest@sun.com Message-id: <48EC7EF8.6090708@solarflare.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=0.0/5.0, scanned in 0.056sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <48EC2056.9000505@sun.com> User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) X-OriginalArrivalTime: 08 Oct 2008 09:35:20.0841 (UTC) FILETIME=[2E90CF90:01C92929] Status: RO Content-Length: 1508 Artem Kachitchkine wrote: > I'm sponsoring this fasttrack for Scott Carter. It is set to timeout on > 10/15/2008. Given that this is a relatively minor amendment to a larger, > approved case PSARC/2004/253 "Advanced DDI Interrupt Functions", I think > it qualifies as a fasttrack. As the discussion progresses, extending > the timer or even promoting to a full case may be considered. > I can't find any mention in the specification of an interface for a device driver to discover or control interrupt CPU binding. In general, for a driver with high data throughput, multiple interrupts on the same CPU are pointless at best and in many cases harmful to performance; in fact, multiple interrupts using the same CPU core, cache or even chip/package can be harmful since they may cause needless CPU contention. So, if a device driver is being given an interface to request more interrupts then that interface really should allow some control of which CPUs those interrupts are bound to whether this is specific or by specification of a policy (e.g. one-per-cpu, one-per-cache, one-per-chip, etc.). Also, for interrupts allocated using the existing ddi_intr_alloc() command there really should be a means to discover the CPU binding unless this call can be superceded by a call that also gives control over CPU binding for that initial allocation. Paul -- ============================== Paul Durrant Senior Staff Engineer Solarflare Communications Inc. ============================== From Artem.Kachitchkin@sun.com Wed Oct 8 13:57:43 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m98KvgtB000025 for ; Wed, 8 Oct 2008 13:57:43 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m98Kvgja001411; Wed, 8 Oct 2008 13:57:42 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8F00G05U86CV00@nwk-avmta-2.sfbay.sun.com>; Wed, 08 Oct 2008 13:57:42 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8F00D0YU86BV50@nwk-avmta-2.sfbay.sun.com>; Wed, 08 Oct 2008 13:57:42 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m98Kvg53023631; Wed, 08 Oct 2008 13:57:42 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8F00D01QV3D200@fe-sfbay-10.sun.com> (original mail from Artem.Kachitchkin@Sun.COM); Wed, 08 Oct 2008 13:57:42 -0700 (PDT) Received: from [129.146.104.83] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8F00JLSU7ZAP30@fe-sfbay-10.sun.com>; Wed, 08 Oct 2008 13:57:35 -0700 (PDT) Date: Wed, 08 Oct 2008 13:56:03 -0700 From: Artem Kachitchkine Subject: Re: PSARC/2008/628 Interrupt Resource Management In-reply-to: <48EC2056.9000505@sun.com> Sender: Artem.Kachitchkin@sun.com To: psarc-ext@sun.com Cc: ddi-irm-interest@sun.com Message-id: <48ED1E63.4060208@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EC2056.9000505@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 527 I missed this during case preparation: what is the requested release binding? > - Provide a mechanism for drivers to get more interrupts. > > The feature is optional so drivers that don't use it still work even > if the system has implemented support for the feature. And conversely, > drivers that do use it also work if the system does not implement the > support. > > A full implementation of the platform level support will be provided > by this project for PCIe IO bus drivers on SPARC. No support for x86? -Artem From Matthew.Jacob@sun.com Wed Oct 8 14:00:42 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m98L0gRp000237 for ; Wed, 8 Oct 2008 14:00:42 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m98L0afc015697; Wed, 8 Oct 2008 15:00:42 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8F00G07UD5HN00@nwk-avmta-2.sfbay.sun.com>; Wed, 08 Oct 2008 14:00:41 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8F00DEAUD4BX50@nwk-avmta-2.sfbay.sun.com>; Wed, 08 Oct 2008 14:00:40 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m98L0eF0012033; Wed, 08 Oct 2008 14:00:40 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8F00701TS5LQ00@fe-sfbay-09.sun.com> (original mail from Matthew.Jacob@Sun.COM); Wed, 08 Oct 2008 14:00:40 -0700 (PDT) Received: from [129.145.154.105] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8F003DPUD1TY10@fe-sfbay-09.sun.com>; Wed, 08 Oct 2008 14:00:38 -0700 (PDT) Date: Wed, 08 Oct 2008 14:00:37 -0700 From: Matthew Jacob Subject: Re: PSARC/2008/628 Interrupt Resource Management In-reply-to: <48ED1E63.4060208@sun.com> Sender: Matthew.Jacob@sun.com To: Artem Kachitchkine Cc: psarc-ext@sun.com, ddi-irm-interest@sun.com Reply-to: Matthew.Jacob@sun.com Message-id: <48ED1F75.1010108@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EC2056.9000505@sun.com> <48ED1E63.4060208@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 221 On 10/ 8/08 01:56 PM, Artem Kachitchkine wrote: >> >> A full implementation of the platform level support will be provided >> by this project for PCIe IO bus drivers on SPARC. > > No support for x86? Aiiee! No! Need it! From Richard.Zatorski@sun.com Wed Oct 8 14:11:07 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m98LB5LF000427 for ; Wed, 8 Oct 2008 14:11:06 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m98LB16C022427; Thu, 9 Oct 2008 05:11:04 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8F00G0RUUDV000@nwk-avmta-2.sfbay.sun.com>; Wed, 08 Oct 2008 14:11:01 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8F00DDJUUDBZ50@nwk-avmta-2.sfbay.sun.com>; Wed, 08 Oct 2008 14:11:01 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m98LB1kt025486; Wed, 08 Oct 2008 14:11:01 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8F00F01UMEXR00@fe-sfbay-09.sun.com> (original mail from Richard.Zatorski@Sun.COM); Wed, 08 Oct 2008 14:11:01 -0700 (PDT) Received: from dhcp-umpk14-97-195.sfbay.sun.com ([129.146.97.195]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8F0038JUU8TY20@fe-sfbay-09.sun.com>; Wed, 08 Oct 2008 14:10:57 -0700 (PDT) Date: Wed, 08 Oct 2008 14:10:54 -0700 From: Rich Zatorski Subject: Re: PSARC/2008/628 Interrupt Resource Management In-reply-to: <48ED1F75.1010108@Sun.COM> Sender: Richard.Zatorski@sun.com To: Matthew.Jacob@sun.com Cc: Artem Kachitchkine , PSARC-ext@sun.com, ddi-irm-interest@sun.com Message-id: <48ED21DE.4080707@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EC2056.9000505@sun.com> <48ED1E63.4060208@sun.com> <48ED1F75.1010108@Sun.COM> User-Agent: Thunderbird 2.0.0.17 (Macintosh/20080914) Status: RO Content-Length: 645 Matthew Jacob wrote: > On 10/ 8/08 01:56 PM, Artem Kachitchkine wrote: >>> >>> A full implementation of the platform level support will be provided >>> by this project for PCIe IO bus drivers on SPARC. >> >> No support for x86? > > Aiiee! No! Need it! We know that, but the low-level underpinnings of x86 interrupt support are messed up. We have another engineer working on that. Once we have the interrupt vector support in x86 (Project is called x86 APIC), we'll then marry it into the IRM framework. The IRM design has been done such that it should be a simple plug n play exercise for x86 once the APIC project is complete. (RAZ) From Scott.Carter@sun.com Wed Oct 8 15:19:57 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m98MJveN002230 for ; Wed, 8 Oct 2008 15:19:57 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m98MJuUo014810; Wed, 8 Oct 2008 15:19:57 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8F00J0JY16T800@nwk-avmta-2.sfbay.sun.com>; Wed, 08 Oct 2008 15:19:54 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8F00D4UY15BZB0@nwk-avmta-2.sfbay.sun.com>; Wed, 08 Oct 2008 15:19:53 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m98MJrs7021367; Wed, 08 Oct 2008 15:19:53 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8F00F01XPOZV00@fe-sfbay-10.sun.com> (original mail from Scott.Carter@Sun.COM); Wed, 08 Oct 2008 15:19:53 -0700 (PDT) Received: from [129.153.85.37] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8F006MAY0Y6Z50@fe-sfbay-10.sun.com>; Wed, 08 Oct 2008 15:19:48 -0700 (PDT) Date: Wed, 08 Oct 2008 15:19:46 -0700 From: Scott Carter Subject: Re: PSARC/2008/628 Interrupt Resource Management In-reply-to: <48EC65A4.2040400@sun.com> Sender: Scott.Carter@sun.com To: David Kahn Cc: Artem Kachitchkine , psarc-ext@sun.com, ddi-irm-interest@sun.com Reply-to: Scott.Carter@sun.com Message-id: <48ED3202.70405@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EC2538.80207@sun.com> <48EC50C0.4080408@Sun.COM> <48EC65A4.2040400@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 1656 David, This looks like a good suggestion to me. I will update the manpage content like you suggest, and get the case materials updated. On 10/08/08 00:47, David Kahn wrote: > You might want to replace DESCRIPTION section > with something like this: (basically replacing > your 2nd paragraph with 2 new paragraphs.) > > The ddi_intr_set_nreq() function changes the number of > interrupts requested by a device driver instance. > > The nreq parameter is the total number of interrupt > resources that this instance of the device driver > would like to have available. nreq includes any > interrupt resources already allocated by the driver. > > For example, the driver instance is currently using > two MSI-X vectors, and it wants access to four additional > unique MSI-X vectors. In that case it should call > this function with nreq parameter set to the value six. > > The nreq parameter can be any value between 1 and the maximum > number of interrupts supported by the device hardware, as reported > by a call to the ddi_intr_get_nintrs(9F) function. The driver will > receive a callback notifying it in cases when it must release any > previously allocated interrupts, or when it is allowed to allocate > more interrupts as a result of its new nreq parameter. > > The ddi_intr_set_nreq() function is not supported unless a driver > is already consuming interrupts, and if it has a registered callback > handler that can process actions related to changes in interrupt > availability. See ddi_cb_register(9F) for an explanation on how > to enable this functionality. -- Scott From Scott.Carter@sun.com Wed Oct 8 15:58:38 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m98MwbjV003418 for ; Wed, 8 Oct 2008 15:58:38 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m98Mwa57061678; Wed, 8 Oct 2008 16:58:37 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8F00L09ZTOOD00@nwk-avmta-2.sfbay.sun.com>; Wed, 08 Oct 2008 15:58:36 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8F00DZ3ZTMBUC0@nwk-avmta-2.sfbay.sun.com>; Wed, 08 Oct 2008 15:58:34 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m98MwYRZ008790; Wed, 08 Oct 2008 15:58:34 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8F00601ZNB3J00@fe-sfbay-10.sun.com> (original mail from Scott.Carter@Sun.COM); Wed, 08 Oct 2008 15:58:34 -0700 (PDT) Received: from [129.153.85.37] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8F006E0ZTL6Z80@fe-sfbay-10.sun.com>; Wed, 08 Oct 2008 15:58:34 -0700 (PDT) Date: Wed, 08 Oct 2008 15:58:32 -0700 From: Scott Carter Subject: Re: PSARC/2008/628 Interrupt Resource Management In-reply-to: <48EC332F.2020905@sun.com> Sender: Scott.Carter@sun.com To: "Garrett D'Amore" Cc: Artem Kachitchkine , psarc-ext@sun.com, ddi-irm-interest@sun.com Reply-to: Scott.Carter@sun.com Message-id: <48ED3B18.50908@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EC2056.9000505@sun.com> <48EC332F.2020905@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 4758 On 10/07/08 21:12, Garrett D'Amore wrote: > The interfaces are marked Committed. I have some concerns with this, as > I read the project details. One of the areas that really concerns me is > the fact that drivers have to deal with a situation where the total > number of interrupts is reduced. (*Increasing* interrupts seems free > from the issues I am concerned with.) You're right, we have very limited experimentation with these interfaces. And I should not be casting the callback semantics into stone. So I will change the interfaces to Uncommitted. > I'm interested to know how this is used with real hardware. > > For example, one potential use of this facility is to support multiple > receive rings. However, now the problem is that there is a window of > time where an extra ring may be "orphaned". What happens to those > packets that are received there? (What happens to the interrupt that > the hardware issues, for that matter?) I've been working with the Neptune driver developers, to modify it to use these interfaces. So let's use that real hardware example. The attach(9F) routine for Neptune sets up interrupt handling with an algorithm like this: 1) nintrs <-- ddi_intr_get_nintrs(); 2) nactual <-- ddi_intr_alloc(nintrs); 3) nrequired <-- nxge_ldgv_init(nactual); 4) for (i = [0..nrequired]) { inthandler = (lookup handler for inum i) ddi_intr_add_handler(i, inthandler); ddi_intr_enable(i); } The function nxge_ldgv_init() takes a number of interrupts available as an input, and decides: 1) how many to actually use (nrequired), and 2) what composition of handlers to use on each vector. (Some are a 1-to-1 handler, others multiplex many conditions on one vector.) I believe that function also does hardware reprogramming of some sort to match what it decided. To modify this driver to use IRM, the enhancements are: - First insert a step 0 in the attach(9F) procedure, to register an IRM callback handler prior to any other interrupt setup. - Then implement a callback handler which basically tears down all handlers, adjust allocations in response to the callback, and then restarts interrupts in the same way they were first setup in attach. With pseudo code like this: nxge_cbfunc(dip, action, cbarg, arg1, arg2) { nxgep = (nxge_t *)arg1; case DDI_CB_INTR_ADD: count = (int)cbarg; nxge_quiesce(dip); for (i = 0; i < nxgep->nactual; i++) { ddi_intr_disable(nxgep->hdls[i]); ddi_intr_remove_handler(nxgep->hdls[i]); } ddi_intr_alloc(dip, &nxgep->hdls[nxgep->nactual], count, &nactual); nxgep->nactual += nactual; /**** Repeat steps 3 & 4 from attach routine ****/ nxge_unquiesce(dip); return (SUCCESS); case DDI_CB_INTR_REMOVE: count = (int)cbarg; nxge_quiesce(dip); for (i = 0; i < nxgep->nactual; i++) { ddi_intr_disable(nxgep->hdls[i]); ddi_intr_remove_handler(nxgep->hdls[i]); } for (i = 0; i < count; i++) { ddi_intr_free(nxgep->hdls[nxgep->nactual - 1]); nxgep->nactual--; } /**** Repeat steps 3 & 4 from attach routine ****/ nxge_unquiesce(dip); return (SUCCESS); } In general, drivers should already have an ability to quiesce their hardware. They should have this for other reasons, like OS quiesce during certain DR operations. And, drivers that can utilize more interrupt vectors must already have logic to adapt themselves to arbitrary numbers of vectors. As is done by that nxge_ldgv_init() function in Neptune. The two bits of logic put together and the driver should be able to quiesce, re-calculate, and unquiesce as necessary during a callback. Things should not get orphaned. And there is no need for a driver to try and create a policy to lock in a specific number of interrupt vectors. To get and use more interrupt vectors, they must have the flexibility to dynamically adapt in either direction. Either to scale up and use more vectors, or scale down and use less. > How will this feature interact with Crossbow resource allocation? Do we > have any examples of the two working together yet? Our team has met with the Crossbow team. Their basic concern was that we must not regress the performance of certain NIC drivers. The drivers they rely upon have existing workaround mechanisms to get more interrupt vectors. My project preserves those workarounds, so they can still get more interrupts before they are converted to use my new interfaces. -- Scott From gdamore@sun.com Wed Oct 8 16:29:04 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m98NT452003963 for ; Wed, 8 Oct 2008 16:29:04 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m98NT1u9025958; Wed, 8 Oct 2008 16:29:03 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8G00A0H18EIX00@brm-avmta-1.central.sun.com>; Wed, 08 Oct 2008 17:29:02 -0600 (MDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8G009P518DA850@brm-avmta-1.central.sun.com>; Wed, 08 Oct 2008 17:29:02 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m98NT1Mc028556; Wed, 08 Oct 2008 16:29:01 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8G00C0114O8000@fe-sfbay-09.sun.com> (original mail from gdamore@sun.com) ; Wed, 08 Oct 2008 16:29:01 -0700 (PDT) Received: from [10.7.251.172] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8G0099N1801AF0@fe-sfbay-09.sun.com>; Wed, 08 Oct 2008 16:28:49 -0700 (PDT) Date: Wed, 08 Oct 2008 16:25:31 -0700 From: "Garrett D'Amore" Subject: Re: PSARC/2008/628 Interrupt Resource Management In-reply-to: <48ED3B18.50908@Sun.COM> Sender: Garrett.Damore@sun.com To: Scott.Carter@sun.com Cc: Artem Kachitchkine , psarc-ext@sun.com, ddi-irm-interest@sun.com Message-id: <48ED416B.3080908@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EC2056.9000505@sun.com> <48EC332F.2020905@sun.com> <48ED3B18.50908@Sun.COM> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 4194 Scott Carter wrote: > On 10/07/08 21:12, Garrett D'Amore wrote: > >> The interfaces are marked Committed. I have some concerns with this, >> as I read the project details. One of the areas that really concerns >> me is the fact that drivers have to deal with a situation where the >> total number of interrupts is reduced. (*Increasing* interrupts >> seems free from the issues I am concerned with.) > > You're right, we have very limited experimentation with these interfaces. > And I should not be casting the callback semantics into stone. So I will > change the interfaces to Uncommitted. Thank you. > >> I'm interested to know how this is used with real hardware. >> >> For example, one potential use of this facility is to support >> multiple receive rings. However, now the problem is that there is a >> window of time where an extra ring may be "orphaned". What happens >> to those packets that are received there? (What happens to the >> interrupt that the hardware issues, for that matter?) > > I've been working with the Neptune driver developers, to modify it to > use these interfaces. So let's use that real hardware example. > > [ neptune details snipped ] > > > > In general, drivers should already have an ability to quiesce their > hardware. They should have this for other reasons, like OS quiesce > during certain DR operations. Agreed about quiesce. But there is a new semantic here, which is that drivers no longer have any way to enforce the QoS guarantees that they may have previously done. For example, with Neptune, you can imagine that you had four interrupts, two general purpose (one send, one receive), and two that were dedicated to providing QoS guarantees for realtime data. These guarantees are passed up the stack to the mac clients via crossbow. If the neptune driver gets a callback that says it only has two interrupts available now, it might (depending on hardware limitations) have to collapse its rings and renege on the guarantees it has made to other clients. Unlike with other things, the driver probably relied upon its ability to make the guarantees in the first place by learning what interrupts and rings were available at attach time. Drivers today are designed such that those resources are immutable -- once interrupts are available, they don't disappear later. The new behavior potentially breaks these semantics in significant ways that may limit what drivers can do. > > And, drivers that can utilize more interrupt vectors must already > have logic to adapt themselves to arbitrary numbers of vectors. > As is done by that nxge_ldgv_init() function in Neptune. > > The two bits of logic put together and the driver should be able to > quiesce, re-calculate, and unquiesce as necessary during a callback. > Things should not get orphaned. And there is no need for a driver > to try and create a policy to lock in a specific number of interrupt > vectors. To get and use more interrupt vectors, they must have the > flexibility to dynamically adapt in either direction. Either to > scale up and use more vectors, or scale down and use less. > >> How will this feature interact with Crossbow resource allocation? Do >> we have any examples of the two working together yet? > > Our team has met with the Crossbow team. Their basic concern was that > we must not regress the performance of certain NIC drivers. The drivers > they rely upon have existing workaround mechanisms to get more interrupt > vectors. My project preserves those workarounds, so they can still get > more interrupts before they are converted to use my new interfaces. > My concerns go beyond just preserving the existing hack. I want to make sure that they can implement drivers to the new framework and not regress either. (I.e. it shouldn't be required to use a hack to get the performance.) See my description above of Crossbow resource guarantees to see what I'm talking about. I think that there needs to be much closer interaction between the resource allocation done by Crossbow (and there may be other similar projects outside of the networking space!) and what this project proposes. -- Garrett From Scott.Carter@sun.com Wed Oct 8 17:34:35 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m990YYkd005734 for ; Wed, 8 Oct 2008 17:34:35 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m990YSnN021615; Thu, 9 Oct 2008 08:34:33 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8G0080149I3A00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 08 Oct 2008 17:34:30 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8G006XJ49I7G50@nwk-avmta-1.sfbay.Sun.COM>; Wed, 08 Oct 2008 17:34:30 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m990YU9E003854; Wed, 08 Oct 2008 17:34:30 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8G00B0141J0F00@fe-sfbay-09.sun.com> (original mail from Scott.Carter@Sun.COM); Wed, 08 Oct 2008 17:34:30 -0700 (PDT) Received: from [129.153.85.37] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8G0097V49GZT00@fe-sfbay-09.sun.com>; Wed, 08 Oct 2008 17:34:30 -0700 (PDT) Date: Wed, 08 Oct 2008 17:34:28 -0700 From: Scott Carter Subject: Re: PSARC/2008/628 Interrupt Resource Management In-reply-to: <48EC7EF8.6090708@solarflare.com> Sender: Scott.Carter@sun.com To: Paul Durrant Cc: Artem Kachitchkine , psarc-ext@sun.com, ddi-irm-interest@sun.com Reply-to: Scott.Carter@sun.com Message-id: <48ED5194.6070004@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EC2056.9000505@sun.com> <48EC7EF8.6090708@solarflare.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 1566 Paul, Features to discover or control interrupt CPU binding are beyond the scope of this project. This project is just managing the number of available interrupts. Just as I mentioned to David Kahn about how the actual assignment of individual interrupt vectors is done at a different implementation layer, so is the binding of those interrupt vectors to specific CPUs. The interfaces or functionality you're seeking is going to be from some other project or feature. Perhaps intrd, for example. On 10/08/08 02:35, Paul Durrant wrote: > I can't find any mention in the specification of an interface for a > device driver to discover or control interrupt CPU binding. In general, > for a driver with high data throughput, multiple interrupts on the same > CPU are pointless at best and in many cases harmful to performance; in > fact, multiple interrupts using the same CPU core, cache or even > chip/package can be harmful since they may cause needless CPU contention. > So, if a device driver is being given an interface to request more > interrupts then that interface really should allow some control of which > CPUs those interrupts are bound to whether this is specific or by > specification of a policy (e.g. one-per-cpu, one-per-cache, > one-per-chip, etc.). Also, for interrupts allocated using the existing > ddi_intr_alloc() command there really should be a means to discover the > CPU binding unless this call can be superceded by a call that also gives > control over CPU binding for that initial allocation. > > Paul > -- Scott From Scott.Carter@Sun.COM Wed Oct 8 17:42:28 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m990gSJv006233 for ; Wed, 8 Oct 2008 17:42:28 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m990gR0H016942; Wed, 8 Oct 2008 17:42:27 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8G003014MRKZ00@nwk-avmta-2.sfbay.sun.com>; Wed, 08 Oct 2008 17:42:27 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8G0001O4MRRXC0@nwk-avmta-2.sfbay.sun.com>; Wed, 08 Oct 2008 17:42:27 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m990gRhB004416; Wed, 08 Oct 2008 17:42:27 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8G000014LLVZ00@fe-sfbay-09.sun.com> (original mail from Scott.Carter@Sun.COM); Wed, 08 Oct 2008 17:42:27 -0700 (PDT) Received: from [129.153.85.37] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8G009JD4MQZT00@fe-sfbay-09.sun.com>; Wed, 08 Oct 2008 17:42:27 -0700 (PDT) Date: Wed, 08 Oct 2008 17:42:25 -0700 From: Scott Carter Subject: Re: PSARC/2008/628 Interrupt Resource Management In-reply-to: <48ED1E63.4060208@sun.com> Sender: Scott.Carter@Sun.COM To: Artem Kachitchkine Cc: psarc-ext@Sun.COM, ddi-irm-interest@Sun.COM Reply-to: Scott.Carter@Sun.COM Message-id: <48ED5371.2090708@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EC2056.9000505@sun.com> <48ED1E63.4060208@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 1565 On 10/08/08 13:56, Artem Kachitchkine wrote: > I missed this during case preparation: what is the requested release > binding? This project seeks approval for a micro release binding. We are planning to integrate into Nevada and S10U8. > No support for x86? The project provides a generic DDI mechanism for drivers to create and manage requests for interrupts. And the project provides a generic NDI mechanism for the system to define interrupt pools, map requests into pools, and then balance together all the requests that were mapped to the same pool. If there is any mismatch, the framework adapts. That is, if a driver uses the new DDI interfaces but the system does not have any defined interrupt pools for the driver to use, then behavior reverts back to what currently exists. Likewise, if a driver is not using the new DDI interfaces but an interrupt pool is defined, the framework can again translate the driver's use of interrupts into the pool in such a way as to provide the previous type of behavior for that driver. We are providing a specific implementation for the PCIe nexus driver on SPARC to define interrupt pools so that drivers using the new DDI interfaces will get more interrupts through IRM when plugged into PCIe buses on SPARC based systems. But we are not providing a specific implementation to define interrupt pools on x86. There is a lot more work required to overhaul the interrupt handling on x86 first, and then the NDI interfaces can be used to define interrupt pools and plugin to IRM when that is done. -- Scott From Kais.Belgaied@sun.com Thu Oct 9 02:20:31 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m999KUAE017476 for ; Thu, 9 Oct 2008 02:20:30 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m999KM9f003148; Thu, 9 Oct 2008 17:20:25 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8G00M03SLZ2P00@nwk-avmta-1.sfbay.Sun.COM>; Thu, 09 Oct 2008 02:20:23 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8G0005TSLZ9RD0@nwk-avmta-1.sfbay.Sun.COM>; Thu, 09 Oct 2008 02:20:23 -0700 (PDT) Received: from [129.146.11.144] (sr1-jurassic-01.SFBay.Sun.COM [129.146.11.144]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m999KMpA755002; Thu, 09 Oct 2008 02:20:23 -0700 (PDT) Date: Thu, 09 Oct 2008 02:20:22 -0700 From: Kais Belgaied Subject: Derailing PSARC/2008/628 Interrupt Resource Management In-reply-to: <48ED416B.3080908@sun.com> To: Artem Kachitchkine Cc: Scott.Carter@sun.com, psarc-ext@sun.com, ddi-irm-interest@sun.com Reply-to: Kais.Belgaied@sun.com Message-id: <48EDCCD6.7020401@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EC2056.9000505@sun.com> <48EC332F.2020905@sun.com> <48ED3B18.50908@Sun.COM> <48ED416B.3080908@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 1766 I am derailing this case on grounds of non-obviousness of its architectural impact, and possible incompleteness. The discussion already uncovered that there is more than a minor amendment to PSARC/2004/253 "Advanced DDI Interrupt Functions" . To prepare for the full review, the architecture should address the impact on device drivers and on the subsystems they are part of. If the scope of the project is intended to remain generic enough, the material needs to reflect that more than one class of device drivers were considered in the architecture. To elaborate (see Garrett's previous email), the interrupt handles that a NIC driver acquired are actually exposed to the MAC layer (see PSARC/2006/357 - Crossbow), for enabling/disable the interrupts on demand. The proposal should be clear on how the behavior of such drivers is intended to be modified when ported to the IRM interfaces. Should there be an extra notification event between MAC and the drivers to invalidate the interrupt handles registered with MAC? Are drivers supposed to insulate MAC from the real interrupt handles instead, and, internally map to real handles that can be added/removed? are they supposed to start "faking" the polling mode in software on rx rings that lost their real interrupts for example? Cryptographic accelerators are another class of I/O where an external framework (the Solaris crypto framework) relies on driver notifications coming from job completion interrupts. See PSARC/2001/557. What such drivers are supposed to do for proper handling DDI_CB_INTR_REMOVE ? Should they block until the jobs drain and they get to call crypto_provider_notification(READY), should they immediately notify an error for all pending crypto requests? Kais. From roamer@sun.com Fri Oct 10 19:12:47 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9B2CkcF016801 for ; Fri, 10 Oct 2008 19:12:46 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m9B2Cg4t005264; Sat, 11 Oct 2008 10:12:45 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8J00001Y579I00@nwk-avmta-2.sfbay.sun.com>; Fri, 10 Oct 2008 19:12:43 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8J00KDPY574V70@nwk-avmta-2.sfbay.sun.com>; Fri, 10 Oct 2008 19:12:43 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m9B2Chon029084; Fri, 10 Oct 2008 19:12:43 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8J00F01XZXM700@fe-sfbay-09.sun.com> (original mail from roamer@sun.com) ; Fri, 10 Oct 2008 19:12:43 -0700 (PDT) Received: from [192.18.98.63] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8J0029CY55DOC0@fe-sfbay-09.sun.com>; Fri, 10 Oct 2008 19:12:42 -0700 (PDT) Date: Fri, 10 Oct 2008 19:14:01 -0700 From: "Yunsong (Roamer) Lu" Subject: Re: Derailing PSARC/2008/628 Interrupt Resource Management In-reply-to: <48EDCCD6.7020401@Sun.COM> Sender: Roamer.Lu@sun.com To: Artem Kachitchkine , Scott.Carter@sun.com Cc: Kais.Belgaied@sun.com, psarc-ext@sun.com, ddi-irm-interest@sun.com Message-id: <48F00BE9.5090907@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EC2056.9000505@sun.com> <48EC332F.2020905@sun.com> <48ED3B18.50908@Sun.COM> <48ED416B.3080908@sun.com> <48EDCCD6.7020401@Sun.COM> User-Agent: Thunderbird 2.0.0.14 (X11/20080714) Status: RO Content-Length: 3882 A few more concerns about the IRM proposed interfaces. 1. When the material talks about current interface limitation, 4.1.2, why it's a problem to allow a driver to get more that *2* MSI-X? Those integrated device drivers should be prepared that it can not get any MSI-X interrupt vector, and it might try the legacy INTX instead. So it should not be a problem even all MSI-X vectors have been given to those attached drivers. Late-attached drivers will just use legacy INTX interrupts. The justification for current *hard-coded* limitation doesn't make sense. 2. How the IRM framework decide to decrease the number of interrupt vectors that have been given to a driver? 4.2.1 talk about how driver participate the IRM interfaces, but it's obscure how the framework can wisely move interrupt resources around drivers. 3. How the IRM framework make *wise* decision about which driver can take more interrupt vectors than others? For example, when you have a 10GbE NIC and a 1GbE NIC in the box, both drivers ask for 16 vectors when you don't have enough vectors left. To give the same amount of interrupt vectors to two driver instances are unreasonable. As part of Crossbow project, hardware resources are allocated depending on the real link speed and bandwidth need. But as the low level I/O framework, IRM don't have knowledge about those information. How do you prove that your "management" is reasonable? 4. What's the perimeter of IRM? In a virtualized environment, interrupts might have been bound to CPUs in an exclusive zone or a guest domain, when IRM asks such interrupt vectors back from the driver, who will take care of the interrupt re-targeting? It's out of driver's control, and I can not find any relevant information from this document. Thanks, Roamer Kais Belgaied wrote: > I am derailing this case on grounds of non-obviousness of its > architectural impact, and possible incompleteness. The discussion > already uncovered that there is more than a minor amendment to > PSARC/2004/253 "Advanced DDI Interrupt Functions" . > > To prepare for the full review, the architecture should address the > impact on device drivers and on the subsystems they are part of. > If the scope of the project is intended to remain generic enough, the > material needs to reflect that more than one class of > device drivers were considered in the architecture. > > To elaborate (see Garrett's previous email), the interrupt handles that > a NIC driver acquired are actually exposed to the MAC layer (see > PSARC/2006/357 - Crossbow), for enabling/disable the interrupts on demand. > The proposal should be clear on how the behavior of such drivers is > intended to be modified when ported to the IRM interfaces. > Should there be an extra notification event between MAC and the drivers > to invalidate the interrupt handles registered with MAC? > Are drivers supposed to insulate MAC from the real interrupt handles > instead, and, internally map to real handles that can be > added/removed? are they supposed to start "faking" the polling mode in > software on rx rings that lost their real interrupts for > example? > > Cryptographic accelerators are another class of I/O where an external > framework (the Solaris crypto framework) relies > on driver notifications coming from job completion interrupts. See > PSARC/2001/557. > What such drivers are supposed to do for proper handling > DDI_CB_INTR_REMOVE ? > Should they block until the jobs drain and they get to call > crypto_provider_notification(READY), should they immediately > notify an error for all pending crypto requests? > > Kais. > -- # telnet (650)-786-6759 (x86759) Connected to Solaris.Sun.COM. login: Lu, Yunsong Last login: January 2, 2007 from beyond.sfbay Yunsong.Lu@Sun.COM v1.04 Since Mon Dec. 22, 2003 [Roamer@Solaris Networking]# cd .. From Kais.Belgaied@sun.com Wed Oct 15 09:48:11 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9FGmAQW000689 for ; Wed, 15 Oct 2008 09:48:11 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m9FGlxRV015780; Wed, 15 Oct 2008 17:48:08 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8S0061FHC6TT00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 15 Oct 2008 09:48:06 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.63]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8S006APHC5LJ00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 15 Oct 2008 09:48:05 -0700 (PDT) Received: from [129.146.11.144] (sr1-jurassic-01.SFBay.Sun.COM [129.146.11.144]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m9FGm4gk316378; Wed, 15 Oct 2008 09:48:05 -0700 (PDT) Date: Wed, 15 Oct 2008 09:48:04 -0700 From: Kais Belgaied Subject: Re: Derailing PSARC/2008/628 Interrupt Resource Management In-reply-to: <48F00BE9.5090907@Sun.COM> To: "Yunsong (Roamer) Lu" Cc: psarc-ext@sun.com Reply-to: Kais.Belgaied@sun.com Message-id: <48F61EC4.5080400@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EC2056.9000505@sun.com> <48EC332F.2020905@sun.com> <48ED3B18.50908@Sun.COM> <48ED416B.3080908@sun.com> <48EDCCD6.7020401@Sun.COM> <48F00BE9.5090907@Sun.COM> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 2051 Roamer since this is a full case now, the procedure is to add the comments to the issues file (for internal contributors). Kais. On 10/10/08 19:14, Yunsong (Roamer) Lu wrote: > A few more concerns about the IRM proposed interfaces. > > 1. When the material talks about current interface limitation, 4.1.2, > why it's a problem to allow a driver to get more that *2* MSI-X? Those > integrated device drivers should be prepared that it can not get any > MSI-X interrupt vector, and it might try the legacy INTX instead. So > it should not be a problem even all MSI-X vectors have been given to > those attached drivers. Late-attached drivers will just use legacy > INTX interrupts. The justification for current *hard-coded* limitation > doesn't make sense. > > 2. How the IRM framework decide to decrease the number of interrupt > vectors that have been given to a driver? 4.2.1 talk about how driver > participate the IRM interfaces, but it's obscure how the framework can > wisely move interrupt resources around drivers. > > 3. How the IRM framework make *wise* decision about which driver can > take more interrupt vectors than others? For example, when you have a > 10GbE NIC and a 1GbE NIC in the box, both drivers ask for 16 vectors > when you don't have enough vectors left. To give the same amount of > interrupt vectors to two driver instances are unreasonable. As part of > Crossbow project, hardware resources are allocated depending on the > real link speed and bandwidth need. But as the low level I/O > framework, IRM don't have knowledge about those information. How do > you prove that your "management" is reasonable? > > 4. What's the perimeter of IRM? In a virtualized environment, > interrupts might have been bound to CPUs in an exclusive zone or a > guest domain, when IRM asks such interrupt vectors back from the > driver, who will take care of the interrupt re-targeting? It's out of > driver's control, and I can not find any relevant information from > this document. > > Thanks, > > Roamer From sacadmin Thu Oct 16 09:52:45 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9GGqjJW000895 for ; Thu, 16 Oct 2008 09:52:45 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m9GGqgnf000705 for <@sunmail2sca.sfbay.sun.com:psarc-members@sun.com>; Thu, 16 Oct 2008 09:52:45 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8U00D19C7VK900@brm-avmta-1.central.sun.com> for psarc-members@sun.com (ORCPT psarc-members@sun.com); Thu, 16 Oct 2008 10:52:43 -0600 (MDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8U004CUC7VTCF0@brm-avmta-1.central.sun.com> for psarc-members@sun.com (ORCPT psarc-members@sun.com); Thu, 16 Oct 2008 10:52:43 -0600 (MDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m9GGqgnL029722 for ; Thu, 16 Oct 2008 16:52:42 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8U00801BRN1X00@mail-amer.sun.com> (original mail from Aarti.Pai@Sun.COM) for psarc-members@sun.com (ORCPT psarc-members@sun.com); Thu, 16 Oct 2008 10:52:42 -0600 (MDT) Received: from [129.150.37.74] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8U00E92C7J7OC0@mail-amer.sun.com>; Thu, 16 Oct 2008 10:52:36 -0600 (MDT) Date: Thu, 16 Oct 2008 09:52:30 -0700 From: Aarti Pai Subject: Re: [Fwd: Re: [Fwd: Re: PSARC/2008/628 Interrupt Resource Management]] In-reply-to: <48F76B2B.7030404@sun.com> Sender: Aarti.Pai@sun.com To: Tina Bracksher Cc: Govinda Tatti , Scott Carter , Richard Zatorski , Kais Belgaied , psarc-members@sun.com Message-id: <48F7714E.2020406@Sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EEDC2F.9000507@sun.com> <48EF9406.5020606@Sun.COM> <48EF9628.1000700@Sun.COM> <48F622AA.9040402@sun.com> <48F62749.5030004@Sun.COM> <48F6308C.50203@sun.com> <48F68C81.5050102@sun.com> <48F693CC.7010101@sun.com> <48F6F8A0.2090608@sun.com> <48F76B2B.7030404@sun.com> User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) Status: RO Content-Length: 833 Hi Tina, The 10am or 4pm timing is not the main issue here. PSARC is low on membership / resources at the moment. The PSARC member that is able to take this case (as the case owner) is available for the 10/29 agenda or thereafter. Is Govinda available 11/5 or is that too far out to schedule this case? Aarti Tina Bracksher wrote: > Hi Aarti - Is there absolutely no way to have this review in the late > afternoon? > > thx, > tb > Govinda Tatti wrote: >> >> Too bad. I don't think I can make Oct 29th but I will try to attend >> it from Dubai. >> >>> I think it was a matter of not getting ARC members to meet late in >>> the day? >>> >>> Govinda Tatti wrote: >>>> Tina Bracksher wrote: >>>>> Rich is returning from Bejing on the 29th, and Govinda will be in >>>>> India then. >>>> What happened to Oct 22nd slot?. >>>> > From sacadmin Thu Oct 16 10:37:14 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9GHbDVT008089 for ; Thu, 16 Oct 2008 10:37:13 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m9GHbAjW011663 for <@sunmail2sca.sfbay.sun.com:psarc-members@sun.com>; Fri, 17 Oct 2008 01:37:12 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8U0090HE9Y3300@nwk-avmta-2.sfbay.sun.com> for psarc-members@sun.com (ORCPT psarc-members@sun.com); Thu, 16 Oct 2008 10:37:10 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8U003B3E9X9YA0@nwk-avmta-2.sfbay.sun.com> for psarc-members@sun.com (ORCPT psarc-members@sun.com); Thu, 16 Oct 2008 10:37:10 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m9GHb9V1018626 for ; Thu, 16 Oct 2008 17:37:09 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8U00501DRHUY00@mail-amer.sun.com> (original mail from Aarti.Pai@Sun.COM) for psarc-members@sun.com (ORCPT psarc-members@sun.com); Thu, 16 Oct 2008 11:37:09 -0600 (MDT) Received: from [129.150.37.74] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8U00JG0E9W8490@mail-amer.sun.com>; Thu, 16 Oct 2008 11:37:09 -0600 (MDT) Date: Thu, 16 Oct 2008 10:37:07 -0700 From: Aarti Pai Subject: Re: [Fwd: Re: [Fwd: Re: PSARC/2008/628 Interrupt Resource Management]] In-reply-to: <48F77679.8040401@sun.com> Sender: Aarti.Pai@sun.com To: Rich Zatorski Cc: Tina Bracksher , Govinda Tatti , Scott Carter , Kais Belgaied , psarc-members@sun.com Message-id: <48F77BC3.9070806@Sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EEDC2F.9000507@sun.com> <48EF9406.5020606@Sun.COM> <48EF9628.1000700@Sun.COM> <48F622AA.9040402@sun.com> <48F62749.5030004@Sun.COM> <48F6308C.50203@sun.com> <48F68C81.5050102@sun.com> <48F693CC.7010101@sun.com> <48F6F8A0.2090608@sun.com> <48F76B2B.7030404@sun.com> <48F77679.8040401@sun.com> User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) Status: RO Content-Length: 814 I have to defer to the PSARC voting members and/or chair on this one. Aarti Rich Zatorski wrote: > Is 10/22 totally out? Even at 10am PT? Govinda and I will just stay > up late if need be. I'd prefer to wrap this up sooner as opposed to > drawing it out. > > (RAZ) > > Tina Bracksher wrote: >> Hi Aarti - Is there absolutely no way to have this review in the late >> afternoon? >> >> thx, >> tb >> Govinda Tatti wrote: >>> >>> Too bad. I don't think I can make Oct 29th but I will try to attend >>> it from Dubai. >>> >>>> I think it was a matter of not getting ARC members to meet late in >>>> the day? >>>> >>>> Govinda Tatti wrote: >>>>> Tina Bracksher wrote: >>>>>> Rich is returning from Bejing on the 29th, and Govinda will be in >>>>>> India then. >>>>> What happened to Oct 22nd slot?. >>>>> >> From sacadmin Thu Oct 16 10:46:06 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9GHk5ep009113 for ; Thu, 16 Oct 2008 10:46:05 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m9GHk18B012459 for <@sunmail2sca.sfbay.sun.com:psarc-members@sun.com>; Thu, 16 Oct 2008 18:46:03 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8U00801EOQN500@nwk-avmta-1.sfbay.Sun.COM> for psarc-members@sun.com (ORCPT psarc-members@sun.com); Thu, 16 Oct 2008 10:46:02 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8U0019JEOQKT60@nwk-avmta-1.sfbay.Sun.COM> for psarc-members@sun.com (ORCPT psarc-members@sun.com); Thu, 16 Oct 2008 10:46:02 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m9GHk2VW019983 for ; Thu, 16 Oct 2008 10:46:02 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8U00J01EL5HN00@fe-sfbay-09.sun.com> (original mail from gdamore@sun.com) for psarc-members@sun.com (ORCPT psarc-members@sun.com); Thu, 16 Oct 2008 10:46:02 -0700 (PDT) Received: from [10.7.251.172] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8U00AWOEOKID30@fe-sfbay-09.sun.com>; Thu, 16 Oct 2008 10:45:57 -0700 (PDT) Date: Thu, 16 Oct 2008 10:42:28 -0700 From: "Garrett D'Amore" Subject: Re: [Fwd: Re: [Fwd: Re: PSARC/2008/628 Interrupt Resource Management]] In-reply-to: <48F77BC3.9070806@Sun.com> Sender: Garrett.Damore@sun.com To: Aarti Pai Cc: Rich Zatorski , Tina Bracksher , Govinda Tatti , Scott Carter , Kais Belgaied , psarc-members@sun.com Message-id: <48F77D04.6090606@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EEDC2F.9000507@sun.com> <48EF9406.5020606@Sun.COM> <48EF9628.1000700@Sun.COM> <48F622AA.9040402@sun.com> <48F62749.5030004@Sun.COM> <48F6308C.50203@sun.com> <48F68C81.5050102@sun.com> <48F693CC.7010101@sun.com> <48F6F8A0.2090608@sun.com> <48F76B2B.7030404@sun.com> <48F77679.8040401@sun.com> <48F77BC3.9070806@Sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 985 Aarti Pai wrote: > I have to defer to the PSARC voting members and/or chair on this one. I can be there, but it will be difficult for me. Pushing it out would make it a lot easier for me. -- Garrett > Aarti > > Rich Zatorski wrote: >> Is 10/22 totally out? Even at 10am PT? Govinda and I will just stay >> up late if need be. I'd prefer to wrap this up sooner as opposed to >> drawing it out. >> >> (RAZ) >> >> Tina Bracksher wrote: >>> Hi Aarti - Is there absolutely no way to have this review in the >>> late afternoon? >>> >>> thx, >>> tb >>> Govinda Tatti wrote: >>>> >>>> Too bad. I don't think I can make Oct 29th but I will try to attend >>>> it from Dubai. >>>> >>>>> I think it was a matter of not getting ARC members to meet late in >>>>> the day? >>>>> >>>>> Govinda Tatti wrote: >>>>>> Tina Bracksher wrote: >>>>>>> Rich is returning from Bejing on the 29th, and Govinda will be >>>>>>> in India then. >>>>>> What happened to Oct 22nd slot?. >>>>>> >>> > From sacadmin Thu Oct 16 10:58:36 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9GHwaKT009654 for ; Thu, 16 Oct 2008 10:58:36 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m9GHwYTW060682; Thu, 16 Oct 2008 11:58:35 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8U0090BF9LV100@nwk-avmta-1.sfbay.Sun.COM>; Thu, 16 Oct 2008 10:58:33 -0700 (PDT) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8U001IZF9LKQ60@nwk-avmta-1.sfbay.Sun.COM>; Thu, 16 Oct 2008 10:58:33 -0700 (PDT) Received: from marduk.eng.sun.com (marduk.SFBay.Sun.COM [129.146.108.224]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m9GHwMEp009469; Thu, 16 Oct 2008 10:58:22 -0700 (PDT) Received: from marduk.eng.sun.com (localhost [127.0.0.1]) by marduk.eng.sun.com (8.13.6+Sun/8.12.11) with ESMTP id m9GHxWFU020242; Thu, 16 Oct 2008 10:59:32 -0700 (PDT) Received: (from gww@localhost) by marduk.eng.sun.com (8.13.6+Sun/8.12.11/Submit) id m9GHxVWi020241; Thu, 16 Oct 2008 10:59:31 -0700 (PDT) Date: Thu, 16 Oct 2008 10:59:31 -0700 (PDT) From: Gary Winiger Subject: Re: [Fwd: Re: [Fwd: Re: PSARC/2008/628 Interrupt Resource Management]] To: Aarti.Pai@sun.com, gdamore@sun.com Cc: Richard.Zatorski@sun.com, Tina.Bracksher@sun.com, Govinda.Tatti@sun.com, Scott.Carter@sun.com, Kais.Belgaied@sun.com, psarc-members@sun.com Message-id: <200810161759.m9GHxVWi020241@marduk.eng.sun.com> Content-transfer-encoding: 7BIT X-Sun-Charset: US-ASCII X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 1305 > Aarti Pai wrote: > > I have to defer to the PSARC voting members and/or chair on this one. > > I can be there, but it will be difficult for me. Pushing it out would > make it a lot easier for me. I think the point is ARC member/intern resources, not time of day. Perhaps the Director/VP of the project team's area can ensure that the ARC members/inters they are contributing to SAC have the bandwidth to handle this case. Gary.. > > -- Garrett > > Aarti > > > > Rich Zatorski wrote: > >> Is 10/22 totally out? Even at 10am PT? Govinda and I will just stay > >> up late if need be. I'd prefer to wrap this up sooner as opposed to > >> drawing it out. > >> > >> (RAZ) > >> > >> Tina Bracksher wrote: > >>> Hi Aarti - Is there absolutely no way to have this review in the > >>> late afternoon? > >>> > >>> thx, > >>> tb > >>> Govinda Tatti wrote: > >>>> > >>>> Too bad. I don't think I can make Oct 29th but I will try to attend > >>>> it from Dubai. > >>>> > >>>>> I think it was a matter of not getting ARC members to meet late in > >>>>> the day? > >>>>> > >>>>> Govinda Tatti wrote: > >>>>>> Tina Bracksher wrote: > >>>>>>> Rich is returning from Bejing on the 29th, and Govinda will be > >>>>>>> in India then. > >>>>>> What happened to Oct 22nd slot?. > >>>>>> > >>> > > > > From sacadmin Thu Oct 16 11:10:17 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9GIAHvr010356 for ; Thu, 16 Oct 2008 11:10:17 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m9GIAGOa017863 for <@sunmail2sca.sfbay.sun.com:psarc-members@sun.com>; Thu, 16 Oct 2008 11:10:17 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8U00A0JFT2LO00@nwk-avmta-2.sfbay.sun.com> for psarc-members@sun.com (ORCPT psarc-members@sun.com); Thu, 16 Oct 2008 11:10:14 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8U003QQFT29IC0@nwk-avmta-2.sfbay.sun.com> for psarc-members@sun.com (ORCPT psarc-members@sun.com); Thu, 16 Oct 2008 11:10:14 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m9GIAEew007380 for ; Thu, 16 Oct 2008 18:10:14 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8U00M01FFCK400@mail-amer.sun.com> (original mail from Richard.Matthews@Sun.COM) for psarc-members@sun.com (ORCPT psarc-members@sun.com); Thu, 16 Oct 2008 12:10:14 -0600 (MDT) Received: from [129.152.9.11] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8U00GNNFSLVN20@mail-amer.sun.com>; Thu, 16 Oct 2008 12:10:04 -0600 (MDT) Date: Thu, 16 Oct 2008 13:09:57 -0500 From: Rick Matthews Subject: Re: [Fwd: Re: [Fwd: Re: PSARC/2008/628 Interrupt Resource Management]] In-reply-to: <48F77D04.6090606@sun.com> Sender: Richard.Matthews@sun.com To: "Garrett D'Amore" Cc: Aarti Pai , Rich Zatorski , Tina Bracksher , Govinda Tatti , Scott Carter , Kais Belgaied , psarc-members@sun.com Reply-to: Richard.Matthews@sun.com Message-id: <48F78375.1080809@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EEDC2F.9000507@sun.com> <48EF9406.5020606@Sun.COM> <48EF9628.1000700@Sun.COM> <48F622AA.9040402@sun.com> <48F62749.5030004@Sun.COM> <48F6308C.50203@sun.com> <48F68C81.5050102@sun.com> <48F693CC.7010101@sun.com> <48F6F8A0.2090608@sun.com> <48F76B2B.7030404@sun.com> <48F77679.8040401@sun.com> <48F77BC3.9070806@Sun.com> <48F77D04.6090606@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 2538 Project team, You could review the audio from the PSARC meeting to correctly identify the issues here. As I understand them, there were two cases that needed owners. I expressed a preference in owning PSARC 2008/532 NWAM. My primary reason is that I am traveling on 10/21 so will be unable to prepare adequately for your case. Another member (currently on sabbatical) expressed interest in the NWAM case, if I could own your case. Again (see reason above), I didn't feel I could be adequately prepared for your case next week, regardless of the time slot. Aarti was tasked with contacting you. You could do a meeting next week, if 1) you could find a ARC member to own your case, and 2) you could get a quorum of members to review your case. PSARC is having issues with participation and particularly with management support of membership. Your case is being "bitten" by this issue. Notify you management of the issues, and that it is affecting your ability to move forward. -- Rick On 10/16/08 12:42, Garrett D'Amore wrote: > Aarti Pai wrote: >> I have to defer to the PSARC voting members and/or chair on this one. > > I can be there, but it will be difficult for me. Pushing it out would > make it a lot easier for me. > > -- Garrett >> Aarti >> >> Rich Zatorski wrote: >>> Is 10/22 totally out? Even at 10am PT? Govinda and I will just >>> stay up late if need be. I'd prefer to wrap this up sooner as >>> opposed to drawing it out. >>> >>> (RAZ) >>> >>> Tina Bracksher wrote: >>>> Hi Aarti - Is there absolutely no way to have this review in the >>>> late afternoon? >>>> >>>> thx, >>>> tb >>>> Govinda Tatti wrote: >>>>> >>>>> Too bad. I don't think I can make Oct 29th but I will try to >>>>> attend it from Dubai. >>>>> >>>>>> I think it was a matter of not getting ARC members to meet late >>>>>> in the day? >>>>>> >>>>>> Govinda Tatti wrote: >>>>>>> Tina Bracksher wrote: >>>>>>>> Rich is returning from Bejing on the 29th, and Govinda will be >>>>>>>> in India then. >>>>>>> What happened to Oct 22nd slot?. >>>>>>> >>>> >> > -- --------------------------------------------------------------------- Rick Matthews email: Rick.Matthews@sun.com Sun Microsystems, Inc. phone:+1(651) 554-1518 1270 Eagan Industrial Road phone(internal): 54418 Suite 160 fax: +1(651) 554-1540 Eagan, MN 55121-1231 USA main: +1(651) 554-1500 --------------------------------------------------------------------- From sacadmin Thu Oct 16 12:15:58 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9GJFwvA017233 for ; Thu, 16 Oct 2008 12:15:58 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m9GJFl4J010309; Thu, 16 Oct 2008 12:15:57 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8U0001PIUJM100@brm-avmta-1.central.sun.com>; Thu, 16 Oct 2008 13:15:55 -0600 (MDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8U00DGYIUILQA0@brm-avmta-1.central.sun.com>; Thu, 16 Oct 2008 13:15:54 -0600 (MDT) Received: from [129.146.11.144] (sr1-jurassic-01.SFBay.Sun.COM [129.146.11.144]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m9GJFZ2q552526; Thu, 16 Oct 2008 12:15:42 -0700 (PDT) Date: Thu, 16 Oct 2008 12:15:35 -0700 From: Kais Belgaied Subject: Re: [Fwd: Re: [Fwd: Re: PSARC/2008/628 Interrupt Resource Management]] In-reply-to: <48F78375.1080809@Sun.COM> To: Rich Zatorski , Tina Bracksher Cc: Aarti Pai , Govinda Tatti , Scott Carter , psarc-members@sun.com Reply-to: Kais.Belgaied@sun.com Message-id: <48F792D7.6070807@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EEDC2F.9000507@sun.com> <48EF9406.5020606@Sun.COM> <48EF9628.1000700@Sun.COM> <48F622AA.9040402@sun.com> <48F62749.5030004@Sun.COM> <48F6308C.50203@sun.com> <48F68C81.5050102@sun.com> <48F693CC.7010101@sun.com> <48F6F8A0.2090608@sun.com> <48F76B2B.7030404@sun.com> <48F77679.8040401@sun.com> <48F77BC3.9070806@Sun.com> <48F77D04.6090606@sun.com> <48F78375.1080809@Sun.COM> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 583 two points: - PSARC cannot accommodate a review next week, regardless the time. Risk of no quorum, and no one ready to own the case next week. Members are available for the week after. There will be an owner by then. - To the project team: Let's be clear about the goal here: We're trying our best to schedule an architectural review for the project. Please designate who exactly is/are responsible for answering questions about the architecture. He/she/they need to be present at the review. Everyone else's presence at the review is irrelevant. Kais. From sacadmin Thu Oct 16 13:00:00 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9GJxwVJ021574 for ; Thu, 16 Oct 2008 12:59:59 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m9GJxscE005243 for <@sunmail2sca.sfbay.sun.com:psarc-members@sun.com>; Fri, 17 Oct 2008 03:59:58 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8U00E01KVXQ500@nwk-avmta-2.sfbay.sun.com> for psarc-members@sun.com (ORCPT psarc-members@sun.com); Thu, 16 Oct 2008 12:59:57 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8U00C0RKVX9A50@nwk-avmta-2.sfbay.sun.com> for psarc-members@sun.com (ORCPT psarc-members@sun.com); Thu, 16 Oct 2008 12:59:57 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m9GJxvxd007744 for ; Thu, 16 Oct 2008 12:59:57 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8U00701KJU7200@fe-sfbay-09.sun.com> (original mail from Richard.Zatorski@Sun.COM) for psarc-members@sun.com (ORCPT psarc-members@sun.com); Thu, 16 Oct 2008 12:59:57 -0700 (PDT) Received: from rich-zatorskis-computer.local (c-24-5-209-117.hsd1.ca.comcast.net [24.5.209.117]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8U00GVXKVU9YD0@fe-sfbay-09.sun.com>; Thu, 16 Oct 2008 12:59:56 -0700 (PDT) Date: Thu, 16 Oct 2008 12:59:53 -0700 From: Rich Zatorski Subject: Re: [Fwd: Re: [Fwd: Re: PSARC/2008/628 Interrupt Resource Management]] In-reply-to: <48F792D7.6070807@Sun.COM> Sender: Richard.Zatorski@sun.com To: Kais.Belgaied@sun.com Cc: Tina Bracksher , Aarti Pai , Govinda Tatti , Scott Carter , psarc-members@sun.com, Wesley Shao Message-id: <48F79D39.6040905@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48EEDC2F.9000507@sun.com> <48EF9406.5020606@Sun.COM> <48EF9628.1000700@Sun.COM> <48F622AA.9040402@sun.com> <48F62749.5030004@Sun.COM> <48F6308C.50203@sun.com> <48F68C81.5050102@sun.com> <48F693CC.7010101@sun.com> <48F6F8A0.2090608@sun.com> <48F76B2B.7030404@sun.com> <48F77679.8040401@sun.com> <48F77BC3.9070806@Sun.com> <48F77D04.6090606@sun.com> <48F78375.1080809@Sun.COM> <48F792D7.6070807@Sun.COM> User-Agent: Thunderbird 2.0.0.17 (Macintosh/20080914) Status: RO Content-Length: 789 Govinda Tatti and/or Wes Shao are the architects who need to be there. I'll double-check and see what time he's returning from Beijing on the 29th. (RAZ) Kais Belgaied wrote: > two points: > > - PSARC cannot accommodate a review next week, regardless the time. > Risk of no quorum, and no one ready to own the case next week. > Members are available for the week after. There will be an owner by > then. > > - To the project team: Let's be clear about the goal here: We're > trying our best to schedule an architectural > review for the project. > Please designate who exactly is/are responsible for answering > questions about > the architecture. He/she/they need to be present at the review. > Everyone else's presence at the review is irrelevant. > > Kais. > > From Richard.Matthews@sun.com Thu Oct 23 05:57:30 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9NCvUs4000059 for ; Thu, 23 Oct 2008 05:57:30 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m9NCvRn8016634 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Thu, 23 Oct 2008 06:57:29 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K9600B0JZZRWB00@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Thu, 23 Oct 2008 05:57:27 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K9600FBLZZP44D0@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Thu, 23 Oct 2008 05:57:25 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m9NCvOsR015497 for ; Thu, 23 Oct 2008 12:57:24 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K9600301ZOQ9S00@mail-amer.sun.com> (original mail from Richard.Matthews@Sun.COM) for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Thu, 23 Oct 2008 06:57:24 -0600 (MDT) Received: from rick-matthews-powerbook-g4-15.local ([129.150.33.239]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K9600BSEZZ30260@mail-amer.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Thu, 23 Oct 2008 06:57:11 -0600 (MDT) Date: Thu, 23 Oct 2008 07:57:03 -0500 From: Rick Matthews Subject: PSARC/2008/628 - Interrupt Resource Management Sender: Richard.Matthews@sun.com To: psarc-ext@sun.com Message-id: <4900749F.4000102@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 User-Agent: Thunderbird 2.0.0.17 (Macintosh/20080914) Status: RO Content-Length: 606 Materials have been posted in the case directory under a "materials" directory. This case is scheduled for review at next weeks meeting. -- --------------------------------------------------------------------- Rick Matthews email: Rick.Matthews@sun.com Sun Microsystems, Inc. phone:+1(651) 554-1518 1270 Eagan Industrial Road phone(internal): 54418 Suite 160 fax: +1(651) 554-1540 Eagan, MN 55121-1231 USA main: +1(651) 554-1500 --------------------------------------------------------------------- From sac-owner Wed Nov 19 09:20:49 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAJHKnlY023027 for ; Wed, 19 Nov 2008 09:20:49 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAJHKmYW007073 for <@sunmail2sca.sfbay.sun.com:All-ARCs@sun.com>; Wed, 19 Nov 2008 09:20:48 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAL00M03C6OW100@brm-avmta-1.central.sun.com> for All-ARCs@sun.com (ORCPT All-ARCs@Sun.COM); Wed, 19 Nov 2008 10:20:48 -0700 (MST) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAL00E2SC6N4QB0@brm-avmta-1.central.sun.com> for All-ARCs@sun.com (ORCPT All-ARCs@Sun.COM); Wed, 19 Nov 2008 10:20:47 -0700 (MST) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mAJHKlbk012159 for ; Wed, 19 Nov 2008 17:20:47 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAL00L01B889Q00@mail-amer.sun.com> (original mail from Aarti.Pai@Sun.COM) for All-ARCs@Sun.COM (ORCPT All-ARCs@Sun.COM); Wed, 19 Nov 2008 10:20:47 -0700 (MST) Received: from [129.145.154.82] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAL00D29C6IH230@mail-amer.sun.com> for All-ARCs@Sun.COM (ORCPT All-ARCs@Sun.COM); Wed, 19 Nov 2008 10:20:44 -0700 (MST) Date: Wed, 19 Nov 2008 09:20:43 -0800 From: Aarti Pai Subject: PSARC Approved 10/29/2008 - Interrupt Resource Management (2008/628) Sender: Aarti.Pai@sun.com To: Aarti Pai Reply-to: Aarti.Pai@sun.com Message-id: <49244AEB.2050609@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 212 PSARC approved 10/29/2008 Case: Eclipse For Java Developers (2008/626) Incompatible Changes: none Precedent: none If more information is needed, please contact the case owner/intern. Aarti From sac-owner Wed Nov 19 09:34:43 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAJHYh9W001843 for ; Wed, 19 Nov 2008 09:34:43 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAJHYfXZ016116 for <@sunmail2sca.sfbay.sun.com:All-ARCs@sun.com>; Wed, 19 Nov 2008 09:34:43 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAL00717CTVNK00@nwk-avmta-1.sfbay.Sun.COM> for All-ARCs@sun.com (ORCPT All-ARCs@Sun.COM); Wed, 19 Nov 2008 09:34:43 -0800 (PST) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAL00LXUCTUCN50@nwk-avmta-1.sfbay.Sun.COM> for All-ARCs@sun.com (ORCPT All-ARCs@Sun.COM); Wed, 19 Nov 2008 09:34:42 -0800 (PST) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mAJHYgJc017753 for ; Wed, 19 Nov 2008 17:34:42 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAL00D01C02XC00@mail-amer.sun.com> (original mail from Aarti.Pai@Sun.COM) for All-ARCs@Sun.COM (ORCPT All-ARCs@Sun.COM); Wed, 19 Nov 2008 10:34:42 -0700 (MST) Received: from [129.145.154.82] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAL00D5SCTPH240@mail-amer.sun.com> for All-ARCs@Sun.COM (ORCPT All-ARCs@Sun.COM); Wed, 19 Nov 2008 10:34:38 -0700 (MST) Date: Wed, 19 Nov 2008 09:34:37 -0800 From: Aarti Pai Subject: Resend: PSARC Approved 10/29/2008 - Interrupt Resource Management (2008/628) In-reply-to: <49244AEB.2050609@Sun.COM> Sender: Aarti.Pai@sun.com To: Aarti.Pai@sun.com Reply-to: Aarti.Pai@sun.com Message-id: <49244E2D.5080502@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <49244AEB.2050609@Sun.COM> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 258 On 11/19/08 09:20 AM, Aarti Pai wrote: PSARC approved 10/29/2008 Case: Interrupt Resource Management (2008/628) <===Corrected case info. Incompatible Changes: none Precedent: none If more information is needed, please contact the case owner/intern.