From gd78059@sac.sfbay.sun.com Wed Jun 3 22:02:06 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n54525Y6022228 for ; Wed, 3 Jun 2009 22:02:06 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n545228L021727; Thu, 4 Jun 2009 13:02:05 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KKP000057BGH000@nwk-avmta-1.sfbay.Sun.COM>; Wed, 03 Jun 2009 22:02:04 -0700 (PDT) Received: from dm-sfbay-01.sfbay.sun.com ([129.145.155.118]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KKP00M4P7BG4H90@nwk-avmta-1.sfbay.Sun.COM>; Wed, 03 Jun 2009 22:02:04 -0700 (PDT) Received: from sac.sfbay.sun.com (sac.SFBay.Sun.COM [129.146.226.132]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n545226H023192; Wed, 03 Jun 2009 22:02:02 -0700 (PDT) Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n54521li022223; Wed, 03 Jun 2009 22:02:01 -0700 (PDT) Received: (from gd78059@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id n54521w6022219; Wed, 03 Jun 2009 22:02:01 -0700 (PDT) Date: Wed, 03 Jun 2009 22:02:01 -0700 (PDT) From: "Garrett D'Amore - sun microsystems" Subject: Interrupt affinity interfaces and PCITool enhancements [PSARC/2009/340 FastTrack timeout 06/17/2009] To: PSARC-ext@sun.com Cc: evan.yan@sun.com, govinda.tatti@sun.com Message-id: <200906040502.n54521w6022219@sac.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 12054 I'm sponsoring this fast track for Govinda Tatti and the PCI team. This project introduces new DDI interfaces, and changes PCITool's command line syntax in an incompatible way. However, the change is intended to correct an incompatibility with respect to CLIP, and the original code has only been integrated in the last couple of builds of Nevada, so we believe it is an opportune time to fix this. The project is seeking Minor Commitment, since the interfaces are primarily intended for consumption by Crossbow which is not available in Solaris 10. Man pages, headers, and supporting materials are also located in the case directory under "materials/" - Garrett Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: Interrupt affinity interfaces and PCITool enhancements 1.2. Name of Document Author/Supplier: Author: Govinda Tatti 1.3 Date of This Document: 03 June, 2009 4. Technical Description Template Version: @(#)sac_nextcase 1.9 06/02/09 SMI This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1 Project/Component Working Name: Interrupt Affinity Interfaces and PCITool Enhancements 1.2 Name of Document Author/Supplier: Author: Govinda Tatti 1.3 Date of This Document: 02 June, 2009 4. Technical Description 4.1 Project Summary This project provides a mechanism for device drivers, IO frameworks such as Crossbow, and for the users who want to know the current CPU binding for their interrupts and fine tune those bindings to achieve maximum IO performance. The first phase of this project delivers the simple DDI interrupt affinity interfaces to allow a device driver to retrieve the current interrupt target CPU and to express its interrupt target preference. In addition, it will deliver some PCITool enhancement to retarget MSI/X interrupts. In the next phase, these simple DDI interrupt affinity interfaces will be replaced with hint or preference based interfaces. Plus, the DDI interrupt framework and platform specific implementation will be modified to query the NUMA-IO framework for optimal interrupt target CPU before configuring the platform interrupt targeting hardware logic. 4.2 Problem and Requirements Modern IO bus technologies support large numbers of interrupts. A single PCI or PCIe device could use up to 32 MSI interrupts, or 2048 MSI-X interrupts. The IRM project (PSARC/2008/628) fixed the MSI-X allocation limit issue and solved part of an IO performance problem. Other part of this problem is how to fine tune the CPU bindings for these multiple MSI-X interrupts to achieve the expected IO performance. Currently there is a need for Solaris device drivers such as NIC (10G), HBA (Emulex) and IO frameworks such as Crossbow to retrieve and reroute the target CPU for their interrupts. For example, Crossbow provides a framework by which NIC resources such as Rx and Tx rings are exposed to the MAC layer. The MAC layer doles out these resources to VNICs when they get created while reserving a fixed amount for the primary NIC. CPUs, on which the processing of packets take place, can be specified at VNIC creation time or later. If they are specified, the interrupts associated with the Rx/Tx rings need to be re-targeted to the specified CPUs. A mechanism by which we can re-target a specific MSI-X interrupt to a different CPU is needed. This is for the virtualization part of Crossbow. For optimal performance of regular NICs (as well as VNICs), the poll thread associated with an Rx ring should be bound to the same CPU as the interrupt CPU. So given an interrupt handle and a CPU, we need a mechanism to retarget the interrupt to the specified CPU. This has become a major issue (on Maramba) for performance when multiple 10 Gig NICs are present. The poll threads belonging to one NIC can end up running on CPUs which is taking interrupts from another NIC. Presently Crossbow uses the PCITool ioctls (sys/pci_tools.h) to re-target fixed interrupts from inside the kernel. The interface provided is not ideal for doing this kind of work from inside the kernel. A better interface is needed here. Also this mechanism currently does not work for MSI-Xs on SPARC platforms. This should be addressed. To achieve the above objectives, the following interfaces are required: 1. Given an interrupt handle (ddi_intr_handle_t) that is associated with an Rx/Tx ring, provide the CPU (processorid_t) to which interrupt is going. 2. Given an interrupt handle (ddi_intr_handle_t) that is associated with an Rx/Tx ring and a CPU, bind the interrupt to the specified CPU. 4.3 Changes From the Previous Case This project is an extension to the approved cases, PSARC/2004/253 "Advanced DDI Interrupt Interfaces" and PSARC/2008/628 "Interrupt Resource Management". The existing DDI interrupt interfaces are not changed. But some new DDI interrupt interfaces are added, which extend the capabilities of the existing interfaces. The changes include: - A new function (ddi_intr_get_affinity(9f)) to return the interrupt target CPU for a given DDI interrupt handle h. - A new function (ddi_intr_set_affinity(9f)) to set the interrupt target CPU for a given DDI interrupt handle h. - Modify ddi_intr_get_cap(9f) function to return the new capability flag DDI_INTR_FLAG_RETARGETABLE indicating all the interrupts are retargetable for the current interrupt type in use. - A new PCITool option, -m to retarget MSI/X interrupts. 4.4 Competitive Analysis Linux and Microsoft OSs already provides the interrupt retarget interfaces of some fashion to their device drivers. So, it is important to provide similar features to Solaris device drivers to achieve individual device performance and also, overall IO performance on all of Sun's platforms in order to remain competitive in the marketplace. 4.5 Project Description 4.5.1 Interrupt Affinity Interfaces The basic strategy is to provide an opportunity for device drivers to provide its input in selecting the proper interrupt target CPU (such as CPU# or preference) for its interrupts. The device drivers or IO frameworks will call the proposed affinity interfaces either during its initialization or run time to optimize its IO performance based on the available resources such as DMA channels, rings, interrupts allocated and current CPU bindings. typedef processorid_t ddi_intr_target_t; int ddi_intr_get_affinity(ddi_intr_handle_t h, ddi_intr_target_t *tgt_p); int ddi_intr_set_affinity(ddi_intr_handle_t h, ddi_intr_target_t tgt); These interfaces are optional to the device drivers, so drivers that don't use it still work even if the system has implemented this feature. And conversely, drivers that do use it also work if the system does not implement the support. This case also includes the contract for Crossbow framework to use these interrupt affinity interfaces in place of existing PCITool ioctl interfaces. Constraints: a) Set affinity limitations for certain interrupt types Fixed or INTx interrupts could be either exclusive or sharable depending on hardware. Because there is no good way to detect that, the current implementation will refuse any set affinity requests for INTx interrupts. On x86 platforms, multiple MSI interrupts of a single PCI function need to be rerouted together since all MSI interrupts share the same MSI address, which in turn includes same CPU number. Hence the current x86 implementation will refuse any set affinity requests for MSI interrupts. The future phase of this project may support MSI group retarget, similar to PCITool method. b) CPU offline considerations CPUs may be online/offlined through administrative interfaces. When a CPU is offlined, all of the interrupts targeting it are re-targeted. The OS will pick any set of the surviving CPUs for re-targeting. The OS is under no obligation to maintain drivers' interrupt affinity preferences. The first phase of this project will not provide any callback on CPU online/offline events. Such callback events need to be defined in the future. If a driver or framework is interested in maintaining optimal CPU targeting, it should monitor its interrupt CPU bindings on a regular basis using ddi_intr_get_affinity(9f) or register a callback to receive various CPU specific events using register_cpu_setup_func(). Where as, the userland entities should subscribe to CPU DR specific sysevents. 4.5.2 PCITool Enhancements Current syntax: pcitool pci@ -i ino=ino [ -r [ -c ] | -w cpu=CPU [ -g ] ] [ -v ] [ -q ] Proposed syntax: pcitool pci@ -i | all [ -r [ -c ] | -w [ -g ] ] [ -v ] [ -q ] pcitool pci@ -m | all [ -r [ -c ] | -w [ -g ] ] [ -v ] [ -q ] The PCItool is a low-level tool which provides a facility for getting and setting interrupt routing information. This project is making some minor syntax changes to PCITool since the current syntax is not compliant with existing userland guidelines. In addition, this project is adding a new "-m" option to retrieve and reroute the interrupt target CPU for MSI/Xs on SPARC platforms. On SPARC platforms, the INO is mapped to an interrupt mondo, and where as one or more MSI/Xs are mapped to an INO. So, INO and MSI/Xs are individually retargetable. Use "-i " option to retrieve or reroute a given INO, and where as use "-m" option for MSI/Xs. On x86 platforms, both INOs and MSI/Xs are mapped to the same interrupt vectors. Use "-i" option to retrieve and reroute any interrupt vectors (both INO and MSI/Xs). So, "-m" option is not required on x86 platforms. Hence it is not supported. 4.6 Interfaces 4.6.1 Exported Interfaces Interface Stability Comments ----------------------------+---------------+-------------------------- ddi_intr_target_t Project Interrupt target CPU Private ddi_intr_get_affinity Project Get interrupt target CPU Private ddi_intr_set_affinity Project Set interrupt target CPU Private ----------------------------------------------------------------------- 4.6.2 Imported Interfaces Interface Stability Comments ----------------------------+---------------+-------------------------- DDI_INTR_FLAG_RETARGETABLE Project Return this new flag (RO) to Private ddi_intr_get_cap() callers if current interrupt type in use is retargetable pcitool Project Minor syntax changes. Added Private new -m option for MSI/Xs. ----------------------------------------------------------------------- 5. References [1] Solaris Interrupt Project Webpage http://pciexpress.sfbay/intr [2] Advanced DDI Interrupt Functions - PSARC/2004/253 http://sac.sfbay.sun.com/PSARC/2004/253 [3] Interrupt Resource Management - PSARC/2008/628 http://sac.sfbay.sun.com/PSARC/2008/628 [4] PCITool and its nexus ioctl support - PSARC/2005/232 http://sac.sfbay.sun.com/PSARC/2005/232 [5] PCITool Public Interrupts - PSARC/2009/215 http://sac.sfbay.sun.com/PSARC/2009/215 6. Resources and Schedule 6.4 Steering Committee requested information 6.4.1 Consolidation C-team Name: ON 6.5 ARC review type: FastTrack 6.6 ARC Exposure: open 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From Kais.Belgaied@sun.com Mon Jun 15 15:45:07 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n5FMj6VH006927 for ; Mon, 15 Jun 2009 15:45:06 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n5FMiuln008011; Tue, 16 Jun 2009 06:45:03 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KLA00C0DXV1QD00@nwk-avmta-1.sfbay.Sun.COM>; Mon, 15 Jun 2009 15:45:01 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.63]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KLA00727XV1M220@nwk-avmta-1.sfbay.Sun.COM>; Mon, 15 Jun 2009 15:45:01 -0700 (PDT) Received: from [129.146.11.146] (sr1-jurassic-03.SFBay.Sun.COM [129.146.11.146]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n5FMj07i643201; Mon, 15 Jun 2009 15:45:01 -0700 (PDT) Date: Mon, 15 Jun 2009 15:45:00 -0700 From: Kais Belgaied Subject: Re: Interrupt affinity interfaces and PCITool enhancements [PSARC/2009/340 FastTrack timeout 06/17/2009] In-reply-to: <200906040502.n54521w6022219@sac.sfbay.sun.com> To: "Garrett D'Amore" Cc: PSARC-ext@sun.com, evan.yan@sun.com, govinda.tatti@sun.com Reply-to: Kais.Belgaied@sun.com Message-id: <4A36CEEC.106@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200906040502.n54521w6022219@sac.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.19 (X11/20090110) Status: RO Content-Length: 6578 > This case also includes the contract for Crossbow framework to use these > interrupt affinity interfaces in place of existing PCITool ioctl interfaces. > If I look at the this case in isolation from its expected consumers, and with pcitool as the only consumer of the CPU affinity APIs, I have no trouble sending a +1. However, when considering the overall architecture that includes both this case's deliverables as well as the changes expected imminently from its external consumers, I am unclear on how the system will behave when we use both pcitool and those consumers' interrupt settings. I'll use the interaction with Crossbow as an example. The point is similar for the interaction with other tools (intd(1m), etc). Say the system has a physical NIC nxge0, whose interrupts are bound the cpu's 1,2,3,4 using pcitool modified by this case. Later one creates vnic1 over nxge0 which used a couple of hardware rings out of nxge0's, and uses dladm set-linkprop cpus=5,6 vnic1. With the changes imported from this case, the implementation of that call of dladm will attempt to have the MSI/X interrupts assigned to the rings (thus the vnic1) bound to CPUs 5 and 6. Will such setting of vnic1's interrupts, fail because of a conflict with a previous binding by pcitool? will it succeed silently? what will the call to pcitool querying about the interrupt binding for nxge0 return then? CPUs 1,2,3,4 only, as set ? or will it surprisingly show 1,2,3,4,5,6? how about the other way around? will dladm get-linkprop vnic1 see the settings that were previously done by pcitool? Kais. > Constraints: > a) Set affinity limitations for certain interrupt types > Fixed or INTx interrupts could be either exclusive or sharable depending > on hardware. Because there is no good way to detect that, the current > implementation will refuse any set affinity requests for INTx interrupts. > > On x86 platforms, multiple MSI interrupts of a single PCI function need > to be rerouted together since all MSI interrupts share the same MSI > address, which in turn includes same CPU number. Hence the current x86 > implementation will refuse any set affinity requests for MSI interrupts. > The future phase of this project may support MSI group retarget, similar > to PCITool method. > > b) CPU offline considerations > CPUs may be online/offlined through administrative interfaces. When > a CPU is offlined, all of the interrupts targeting it are re-targeted. > The OS will pick any set of the surviving CPUs for re-targeting. The > OS is under no obligation to maintain drivers' interrupt affinity > preferences. > > The first phase of this project will not provide any callback on CPU > online/offline events. Such callback events need to be defined in the > future. If a driver or framework is interested in maintaining optimal > CPU targeting, it should monitor its interrupt CPU bindings on a regular > basis using ddi_intr_get_affinity(9f) or register a callback to receive > various CPU specific events using register_cpu_setup_func(). Where as, > the userland entities should subscribe to CPU DR specific sysevents. > > 4.5.2 PCITool Enhancements > > Current syntax: > pcitool pci@ -i ino=ino > [ -r [ -c ] | -w cpu=CPU [ -g ] ] [ -v ] [ -q ] > > Proposed syntax: > pcitool pci@ -i | all > [ -r [ -c ] | -w [ -g ] ] [ -v ] [ -q ] > > pcitool pci@ -m | all > [ -r [ -c ] | -w [ -g ] ] [ -v ] [ -q ] > > The PCItool is a low-level tool which provides a facility for getting and > setting interrupt routing information. This project is making some minor > syntax changes to PCITool since the current syntax is not compliant with > existing userland guidelines. > > In addition, this project is adding a new "-m" option to retrieve and > reroute the interrupt target CPU for MSI/Xs on SPARC platforms. > > On SPARC platforms, the INO is mapped to an interrupt mondo, and where as > one or more MSI/Xs are mapped to an INO. So, INO and MSI/Xs are individually > retargetable. Use "-i " option to retrieve or reroute a given INO, and > where as use "-m" option for MSI/Xs. > > On x86 platforms, both INOs and MSI/Xs are mapped to the same interrupt > vectors. Use "-i" option to retrieve and reroute any interrupt vectors > (both INO and MSI/Xs). So, "-m" option is not required on x86 platforms. > Hence it is not supported. > > 4.6 Interfaces > > 4.6.1 Exported Interfaces > > Interface Stability Comments > ----------------------------+---------------+-------------------------- > ddi_intr_target_t Project Interrupt target CPU > Private > ddi_intr_get_affinity Project Get interrupt target CPU > Private > ddi_intr_set_affinity Project Set interrupt target CPU > Private > ----------------------------------------------------------------------- > > 4.6.2 Imported Interfaces > > Interface Stability Comments > ----------------------------+---------------+-------------------------- > DDI_INTR_FLAG_RETARGETABLE Project Return this new flag (RO) to > Private ddi_intr_get_cap() callers if > current interrupt type in use > is retargetable > > pcitool Project Minor syntax changes. Added > Private new -m option for MSI/Xs. > ----------------------------------------------------------------------- > > 5. References > [1] Solaris Interrupt Project Webpage > http://pciexpress.sfbay/intr > > [2] Advanced DDI Interrupt Functions - PSARC/2004/253 > http://sac.sfbay.sun.com/PSARC/2004/253 > > [3] Interrupt Resource Management - PSARC/2008/628 > http://sac.sfbay.sun.com/PSARC/2008/628 > > [4] PCITool and its nexus ioctl support - PSARC/2005/232 > http://sac.sfbay.sun.com/PSARC/2005/232 > > [5] PCITool Public Interrupts - PSARC/2009/215 > http://sac.sfbay.sun.com/PSARC/2009/215 > > 6. Resources and Schedule > 6.4 Steering Committee requested information > 6.4.1 Consolidation C-team Name: > ON > 6.5 ARC review type: FastTrack > 6.6 ARC Exposure: open > > > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > ON > 6.5. ARC review type: FastTrack > 6.6. ARC Exposure: open > > > > From Evan.Yan@Sun.COM Mon Jun 15 20:41:30 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n5G3fUBH011431 for ; Mon, 15 Jun 2009 20:41:30 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n5G3fU44025766 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Mon, 15 Jun 2009 20:41:30 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KLB00801BL67300@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 15 Jun 2009 20:41:30 -0700 (PDT) Received: from sineb-mail-1.sun.com ([192.18.19.6]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KLB006L5BL56240@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 15 Jun 2009 20:41:29 -0700 (PDT) Received: from fe-apac-06.sun.com (fe-apac-06.sun.com [192.18.19.177] (may be forged)) by sineb-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n5G3fR3v015971 for ; Tue, 16 Jun 2009 03:41:28 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) id <0KLB00G00BH54I00@mail-apac.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Tue, 16 Jun 2009 11:41:27 +0800 (SGT) Received: from [10.13.26.108] ([unknown] [10.13.26.108]) by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) with ESMTPSA id <0KLB00HQUBK0UD10@mail-apac.sun.com>; Tue, 16 Jun 2009 11:41:27 +0800 (SGT) Date: Tue, 16 Jun 2009 11:38:45 +0800 From: Evan Yan Subject: Re: Interrupt affinity interfaces and PCITool enhancements [PSARC/2009/340 FastTrack timeout 06/17/2009] In-reply-to: <4A36CEEC.106@Sun.COM> Sender: Evan.Yan@Sun.COM To: Kais.Belgaied@Sun.COM Cc: "Garrett D'Amore" , PSARC-ext@Sun.COM, Govinda.Tatti@Sun.COM Message-id: <4A3713C5.8000405@Sun.COM> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200906040502.n54521w6022219@sac.sfbay.sun.com> <4A36CEEC.106@Sun.COM> User-Agent: Thunderbird 2.0.0.18 (X11/20090127) Status: RO Content-Length: 2025 Hi Kais, Thanks for the comments. Pcitool and the interrupt affinity interfaces use the same under-layer implementation to re-target interrupts to some cpu. Whatever read operation will reflect the current binding status and whatever write operation will override the former settings. Thanks, -Evan Kais Belgaied wrote: >> This case also includes the contract for Crossbow framework to use these >> interrupt affinity interfaces in place of existing PCITool ioctl interfaces. >> >> > If I look at the this case in isolation from its expected consumers, and > with pcitool as the > only consumer of the CPU affinity APIs, I have no trouble sending a +1. > However, when considering the overall architecture that includes both > this case's deliverables as well as the changes > expected imminently from its external consumers, I am unclear on how > the system will behave when > we use both pcitool and those consumers' interrupt settings. > > I'll use the interaction with Crossbow as an example. The point is > similar for the interaction with other tools > (intd(1m), etc). Say the system has a physical NIC nxge0, whose > interrupts are bound the cpu's 1,2,3,4 > using pcitool modified by this case. Later one creates vnic1 over nxge0 > which > used a couple of hardware rings out of nxge0's, and uses dladm > set-linkprop cpus=5,6 vnic1. > With the changes imported from this case, the implementation of that > call of dladm will attempt > to have the MSI/X interrupts assigned to the rings (thus the vnic1) > bound to CPUs 5 and 6. > Will such setting of vnic1's interrupts, fail because of a conflict > with a previous binding by pcitool? > will it succeed silently? what will the call to pcitool querying about > the interrupt binding for > nxge0 return then? CPUs 1,2,3,4 only, as set ? or will it surprisingly > show 1,2,3,4,5,6? > how about the other way around? will dladm get-linkprop vnic1 see the > settings that were previously done > by pcitool? > > > Kais. > From Kais.Belgaied@sun.com Wed Jun 17 15:28:49 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n5HMSm6k002993 for ; Wed, 17 Jun 2009 15:28:48 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n5HMSTlx006696; Wed, 17 Jun 2009 23:28:44 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KLE00701MFVA900@nwk-avmta-2.sfbay.sun.com>; Wed, 17 Jun 2009 15:28:43 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.63]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KLE003D3MFUHJ30@nwk-avmta-2.sfbay.sun.com>; Wed, 17 Jun 2009 15:28:42 -0700 (PDT) Received: from [129.146.11.146] (sr1-jurassic-03.SFBay.Sun.COM [129.146.11.146]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n5HMSgIl112039; Wed, 17 Jun 2009 15:28:42 -0700 (PDT) Date: Wed, 17 Jun 2009 15:28:42 -0700 From: Kais Belgaied Subject: Re: Interrupt affinity interfaces and PCITool enhancements [PSARC/2009/340 FastTrack timeout 06/17/2009] In-reply-to: <4A3713C5.8000405@Sun.COM> To: Evan Yan Cc: "Garrett D'Amore" , PSARC-ext@sun.com, Govinda.Tatti@sun.com Reply-to: Kais.Belgaied@sun.com Message-id: <4A396E1A.9090302@Sun.COM> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_KKRRHO6M6Pvuc2DozVdSEg)" X-PMX-Version: 5.4.1.325704 References: <200906040502.n54521w6022219@sac.sfbay.sun.com> <4A36CEEC.106@Sun.COM> <4A3713C5.8000405@Sun.COM> User-Agent: Thunderbird 2.0.0.19 (X11/20090110) Status: RO Content-Length: 5765 This is a multi-part message in MIME format. --Boundary_(ID_KKRRHO6M6Pvuc2DozVdSEg) Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT On 06/15/09 20:38, Evan Yan wrote: > Hi Kais, > > Thanks for the comments. > > Pcitool and the interrupt affinity interfaces use the same under-layer > implementation to re-target interrupts to some cpu. Whatever read > operation will reflect the current binding status and whatever write > operation will override the former settings. > so, back the example, let's say you you use pcitool to bind the interrupts from a physical NIC nxge0 to cpu1, the usecrossbow's dladm to set-linprop cpus=2 vnic1 and cpus=3 vnic2 (where vnic1 and vnic2 are built over nxge0) will pcitool show that nxge0's interrupts are bound to cpus 1, 2 and 3? Kais > Thanks, > -Evan > > Kais Belgaied wrote: > >>> This case also includes the contract for Crossbow framework to use these >>> interrupt affinity interfaces in place of existing PCITool ioctl interfaces. >>> >>> >>> >> If I look at the this case in isolation from its expected consumers, and >> with pcitool as the >> only consumer of the CPU affinity APIs, I have no trouble sending a +1. >> However, when considering the overall architecture that includes both >> this case's deliverables as well as the changes >> expected imminently from its external consumers, I am unclear on how >> the system will behave when >> we use both pcitool and those consumers' interrupt settings. >> >> I'll use the interaction with Crossbow as an example. The point is >> similar for the interaction with other tools >> (intd(1m), etc). Say the system has a physical NIC nxge0, whose >> interrupts are bound the cpu's 1,2,3,4 >> using pcitool modified by this case. Later one creates vnic1 over nxge0 >> which >> used a couple of hardware rings out of nxge0's, and uses dladm >> set-linkprop cpus=5,6 vnic1. >> With the changes imported from this case, the implementation of that >> call of dladm will attempt >> to have the MSI/X interrupts assigned to the rings (thus the vnic1) >> bound to CPUs 5 and 6. >> Will such setting of vnic1's interrupts, fail because of a conflict >> with a previous binding by pcitool? >> will it succeed silently? what will the call to pcitool querying about >> the interrupt binding for >> nxge0 return then? CPUs 1,2,3,4 only, as set ? or will it surprisingly >> show 1,2,3,4,5,6? >> how about the other way around? will dladm get-linkprop vnic1 see the >> settings that were previously done >> by pcitool? >> >> >> Kais. >> >> > > --Boundary_(ID_KKRRHO6M6Pvuc2DozVdSEg) Content-type: text/html; charset=ISO-8859-1 Content-transfer-encoding: 7BIT On 06/15/09 20:38, Evan Yan wrote: 
Hi Kais,

Thanks for the comments.

Pcitool and the interrupt affinity interfaces use the same under-layer
implementation to re-target interrupts to some cpu. Whatever read
operation will reflect the current binding status and whatever write
operation will override the former settings.

so, back the example,  let's say you you use pcitool to bind the interrupts from a physical NIC nxge0 to
cpu1, the usecrossbow's dladm  to  set-linprop cpus=2 vnic1 and cpus=3 vnic2 (where vnic1 and vnic2
are built over nxge0) will pcitool show that nxge0's interrupts are bound to cpus 1, 2 and 3?


    Kais

Thanks,
-Evan

Kais Belgaied wrote:
  
    This case also includes the contract for Crossbow framework to use these
    interrupt affinity interfaces in place of existing PCITool ioctl interfaces.
  
    
      
If I look at the this case in isolation from its expected consumers, and 
with pcitool as the
only consumer of the CPU affinity APIs, I have no trouble sending a +1.
However, when considering the overall architecture  that includes both 
this case's deliverables as well as the changes
expected imminently  from its external consumers,  I am unclear on how 
the system will behave when
we  use both pcitool and those consumers' interrupt settings.

I'll use the interaction with Crossbow as an example. The point is 
similar for the interaction with other tools
(intd(1m), etc). Say the system has a physical NIC nxge0, whose 
interrupts are bound the cpu's 1,2,3,4
using pcitool modified by this case. Later one creates vnic1 over nxge0 
which
used a couple of hardware rings out of nxge0's, and uses dladm 
set-linkprop cpus=5,6 vnic1.
With the changes imported from this case, the implementation of that 
call of dladm will attempt
to have the MSI/X interrupts  assigned to  the rings (thus the vnic1)  
bound to CPUs 5 and 6.
Will such setting of vnic1's  interrupts, fail because of a conflict 
with a previous binding by pcitool?
will it succeed silently? what will the call to pcitool querying about 
the interrupt binding for
nxge0 return then? CPUs 1,2,3,4 only, as set ? or will it surprisingly 
show 1,2,3,4,5,6?
how about the other way around? will dladm  get-linkprop vnic1 see the 
settings that were previously done
by pcitool?


    Kais.

--Boundary_(ID_KKRRHO6M6Pvuc2DozVdSEg)-- From Evan.Yan@sun.com Wed Jun 17 16:25:36 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n5HNPZdU009535 for ; Wed, 17 Jun 2009 16:25:36 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n5HNPWje000244 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 18 Jun 2009 07:25:35 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KLE00B05P2LCX00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Jun 2009 16:25:33 -0700 (PDT) Received: from sineb-mail-1.sun.com ([192.18.19.6]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KLE00344P2KH870@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Jun 2009 16:25:32 -0700 (PDT) Received: from fe-apac-05.sun.com (fe-apac-05.sun.com [192.18.19.176] (may be forged)) by sineb-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n5HNPVmc025748 for ; Wed, 17 Jun 2009 23:25:31 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) id <0KLE00900P0DQS00@mail-apac.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 18 Jun 2009 07:25:31 +0800 (SGT) Received: from [127.0.0.1] ([unknown] [222.131.191.82]) by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) with ESMTPSA id <0KLE00J6XP2EZ6L0@mail-apac.sun.com>; Thu, 18 Jun 2009 07:25:31 +0800 (SGT) Date: Thu, 18 Jun 2009 07:25:28 +0800 From: Evan Yan Subject: Re: Interrupt affinity interfaces and PCITool enhancements [PSARC/2009/340 FastTrack timeout 06/17/2009] In-reply-to: <4A396E1A.9090302@Sun.COM> Sender: Evan.Yan@sun.com To: Kais.Belgaied@sun.com Cc: "Garrett D'Amore" , psarc-ext@sun.com, Govinda.Tatti@sun.com Message-id: <4A397B68.5030701@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=UTF-8 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 References: <200906040502.n54521w6022219@sac.sfbay.sun.com> <4A36CEEC.106@Sun.COM> <4A3713C5.8000405@Sun.COM> <4A396E1A.9090302@Sun.COM> User-Agent: Thunderbird 2.0.0.21 (Windows/20090302) Status: RO Content-Length: 3032 Kais Belgaied 写道: > On 06/15/09 20:38, Evan Yan wrote: >> Hi Kais, >> >> Thanks for the comments. >> >> Pcitool and the interrupt affinity interfaces use the same under-layer >> implementation to re-target interrupts to some cpu. Whatever read >> operation will reflect the current binding status and whatever write >> operation will override the former settings. >> > > so, back the example, let's say you you use pcitool to bind the > interrupts from a physical NIC nxge0 to > cpu1, the usecrossbow's dladm to set-linprop cpus=2 vnic1 and cpus=3 > vnic2 (where vnic1 and vnic2 > are built over nxge0) will pcitool show that nxge0's interrupts are > bound to cpus 1, 2 and 3? It depends on how the mapping between vnic cpu binding and it's physical nic's interrupt cpu binding is implemented, either in nxge driver or crossbow. Say nxge0 has two MSI-X interrupts, which are bound to cpu 0 via pcitool. When you use dladm set-linkprop cpus=2 vnic1 and cpus=3 vnic2, if the nxge driver call the interrupt affinity interface to bind its two interrupts to cpu 2 and cpu 3, then pcitool show nxge0's interrupts are bound to cpu 2 and 3. Thanks, -Evan > > > Kais > >> Thanks, >> -Evan >> >> Kais Belgaied wrote: >> >>>> This case also includes the contract for Crossbow framework to use these >>>> interrupt affinity interfaces in place of existing PCITool ioctl interfaces. >>>> >>>> >>>> >>> If I look at the this case in isolation from its expected consumers, and >>> with pcitool as the >>> only consumer of the CPU affinity APIs, I have no trouble sending a +1. >>> However, when considering the overall architecture that includes both >>> this case's deliverables as well as the changes >>> expected imminently from its external consumers, I am unclear on how >>> the system will behave when >>> we use both pcitool and those consumers' interrupt settings. >>> >>> I'll use the interaction with Crossbow as an example. The point is >>> similar for the interaction with other tools >>> (intd(1m), etc). Say the system has a physical NIC nxge0, whose >>> interrupts are bound the cpu's 1,2,3,4 >>> using pcitool modified by this case. Later one creates vnic1 over nxge0 >>> which >>> used a couple of hardware rings out of nxge0's, and uses dladm >>> set-linkprop cpus=5,6 vnic1. >>> With the changes imported from this case, the implementation of that >>> call of dladm will attempt >>> to have the MSI/X interrupts assigned to the rings (thus the vnic1) >>> bound to CPUs 5 and 6. >>> Will such setting of vnic1's interrupts, fail because of a conflict >>> with a previous binding by pcitool? >>> will it succeed silently? what will the call to pcitool querying about >>> the interrupt binding for >>> nxge0 return then? CPUs 1,2,3,4 only, as set ? or will it surprisingly >>> show 1,2,3,4,5,6? >>> how about the other way around? will dladm get-linkprop vnic1 see the >>> settings that were previously done >>> by pcitool? >>> >>> >>> Kais. >>> >>> >> >> > From gdamore@sun.com Wed Jun 17 17:20:19 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n5I0KJCq012353 for ; Wed, 17 Jun 2009 17:20:19 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n5I0KHQD059722 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 17 Jun 2009 18:20:18 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KLE00D03RLUR700@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Jun 2009 17:20:18 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KLE00B3PRLUY110@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Jun 2009 17:20:18 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n5I0KHNe004626 for ; Wed, 17 Jun 2009 17:20:17 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) id <0KLE00I00R1LZH00@fe-sfbay-10.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Jun 2009 17:20:17 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) with ESMTPSA id <0KLE00HZ6RLTSOF0@fe-sfbay-10.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Jun 2009 17:20:17 -0700 (PDT) Date: Wed, 17 Jun 2009 17:20:17 -0700 From: "Garrett D'Amore" Subject: PSARC 2009/340 Interrupt affinity interfaces and PCITool enhancements Sender: Garrett.Damore@sun.com To: PSARC-ext Message-id: <4A398841.4030802@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 User-Agent: Thunderbird 2.0.0.18 (X11/20081201) Status: RO Content-Length: 54 This case was approved at PSARC today. - Garrett From Govinda.Tatti@oracle.com Wed Jun 2 19:44:21 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o532iL1W026521 for ; Wed, 2 Jun 2010 19:44:21 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o532iK44001539; Wed, 2 Jun 2010 19:44:20 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L3F0010J3LWK000@brm-avmta-1.central.sun.com>; Wed, 02 Jun 2010 20:44:20 -0600 (MDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L3F00BH23LWSJ40@brm-avmta-1.central.sun.com>; Wed, 02 Jun 2010 20:44:20 -0600 (MDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o532iJ5i002696; Thu, 03 Jun 2010 02:44:20 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o52KlFxT013249; Thu, 03 Jun 2010 02:44:18 +0000 (GMT) Received: from abhmt012.oracle.com by acsmt355.oracle.com with ESMTP id 292218141275532952; Wed, 02 Jun 2010 19:42:32 -0700 Received: from [10.7.250.167] (/10.7.250.167) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 02 Jun 2010 19:42:32 -0700 Date: Wed, 02 Jun 2010 19:42:31 -0700 From: Govinda Tatti Subject: Patch binding approval request for PSARC/2009/340 [PSARC/2010/203 FastTrack timeout 06/09/2010] To: PSARC-ext@sun.com Cc: "pci-core@Sun.COM" , ddi-intr-iteam Message-id: <4C071697.3000506@Oracle.COM> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090209.4C071703.00FA:SCFMA4539814,ss=1,fgs=0 User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.7) Gecko/20100214 Lightning/1.0b1 Thunderbird/3.0.1 Status: RO Content-Length: 1417 I'm sponsoring the following fast track for Evan Yan. The timer is set to expire on 06/09/2010. - Govinda Template Version: @(#)sac_nextcase 1.70 03/30/10 SMI This information is Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. 1. Introduction 1.1. Project/Component Working Name: Patch binding approval request for PSARC/2009/340 1.2. Name of Document Author/Supplier: Author: Evan Yan 1.3 Date of This Document: 02 June, 2010 4. Technical Description 4. Technical Description: 4.1 Project Summary Requesting patch binding approval for the PCITool changes described in PSARC/2009/340. The business requirement is for RF out-of-box performance and benchmark publication. The part of affinity interfaces described in PSARC/2009/340 will also be backported to S10, but it has already been covered by PSARC/2010/172. So, only the part of PCITool changes needs patch binding approval here. 4.2 References [1] Interrupt Affinity Interfaces and PCITool Enhancements - PSARC/2009/340 http://sac.sfbay.sun.com/PSARC/2009/340 [2] Renaming interrupt affinity interfaces - PSARC/2010/172 http://sac.sfbay.sun.com/PSARC/2010/172 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From Govinda.Tatti@oracle.com Wed Jun 16 12:51:50 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o5GJpo5b029693 for ; Wed, 16 Jun 2010 12:51:50 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o5GJpmLk015838; Wed, 16 Jun 2010 14:51:49 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L440040HHUD7Q00@nwk-avmta-2.sfbay.sun.com>; Wed, 16 Jun 2010 12:51:49 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L44003FDHUCC410@nwk-avmta-2.sfbay.sun.com>; Wed, 16 Jun 2010 12:51:49 -0700 (PDT) Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o5GJpmjx001653; Wed, 16 Jun 2010 19:51:48 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5GJpjYb005593; Wed, 16 Jun 2010 19:51:45 +0000 (GMT) Received: from abhmt021.oracle.com by acsmt354.oracle.com with ESMTP id 350731151276717900; Wed, 16 Jun 2010 12:51:40 -0700 Received: from [10.6.93.67] (/10.6.93.67) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 16 Jun 2010 12:51:40 -0700 Date: Wed, 16 Jun 2010 12:51:31 -0700 From: Govinda Tatti Subject: PSARC/2010/203 Patch binding approval request for PSARC/2009/340 To: PSARC-ext@sun.com Cc: pci-core@sun.com, ddi-intr-iteam@sun.com Message-id: <4C192B43.8030104@Oracle.COM> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090208.4C192B54.00AC:SCFMA4539814,ss=1,fgs=0 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Lightning/1.0b1 Thunderbird/3.0.4 Status: RO Content-Length: 65 This case was approved in today's PSARC meeting. - Govinda