From Artem.Kachitchkin@sun.com Tue Aug 7 14:24:04 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l77LO3ui025081 for ; Tue, 7 Aug 2007 14:24:04 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l77LLd0h005657 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 8 Aug 2007 05:21:42 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JMF00909A041E00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Tue, 07 Aug 2007 14:21:40 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JMF008BUA034910@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Tue, 07 Aug 2007 14:21:40 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l77LLdLh025789 for ; Tue, 07 Aug 2007 14:21:39 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JMF006019KSZ900@fe-sfbay-09.sun.com> (original mail from Artem.Kachitchkin@Sun.COM) for PSARC-ext@Sun.Com (ORCPT PSARC-ext@Sun.Com); Tue, 07 Aug 2007 14:21:39 -0700 (PDT) Received: from [129.150.20.223] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JMF00GP89ZBXEC0@fe-sfbay-09.sun.com>; Tue, 07 Aug 2007 14:21:12 -0700 (PDT) Date: Tue, 07 Aug 2007 14:20:35 -0700 From: Artem Kachitchkine Subject: PSARC/2007/453 MSI-X interrupt limit override Sender: Artem.Kachitchkin@sun.com To: PSARC-ext@sun.com Cc: Edward.Gillett@sun.com, Govinda.Tatti@sun.com Message-id: <46B8E223.2040107@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 User-Agent: Mozilla/5.0 Gecko/20040113 Status: RO Content-Length: 4142 I am sponsoring this case for Edward Gillett. Requested binding is patch/micro, timeout 08/15/2007. -Artem Template Version: @(#)sac_nextcase 1.64 07/13/07 SMI This information is Copyright 2007 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: MSI-X interrupt limit override 1.2. Name of Document Author/Supplier: Author: Edward Gillett 1.3 Date of This Document: 07 August, 2007 4. Technical Description 4.1. Summary This proposal provides a short-term solution for device drivers to request and receive more MSI-X interrupt resources than the current default limit. The future long-term solution will be provided by the Interrupt Resource Management (IRM) project. Proposed solution will be implemented on SPARC only. 4.2. Problem Even though drivers can request arbitrary number of interrupts via ddi_intr_alloc(9F), currently the number of returned MSI-X interrupts is limited to 2. Some high-throughput drivers, such as nxge and qlc, are forced to share 2 interrupts among multiple DMA channels, leading to negative performance impact. The reason for the limit of 2 is that the initial phase of the Advanced DDI interrupt project (PSARC 2004/253) did not implement any IRM interfaces. The IRM project is expected to integrate in the early 2008, but there is immediate need to address performance issues on Sun's latest platforms. 4.3. Proposal Provide a boolean device property, "#msix-request", that indicates to the DDI framework to attempt to allocate more interrupts than the default limit, should the driver request it. A device driver can create the proposed property if and only if it requires more MSI-X interrupts than the default limit. It should create this property before registering any of its interrupts. Sample code: (void) ddi_prop_create(DDI_DEV_T_NONE, dip, DDI_PROP_CANSLEEP, "#msix-request", NULL, 0); The DDI framework will look for the proposed property as part of the driver's interrupt registration process and will try to satisfy the driver's MSI-X allocation request. The number of MSI-X interrupts that can be allocated in this case may be less than or equal to what the driver has requested. This depends on the availability of MSI-X interrupt resources at that moment and also based on other platform specific limitations. So, the driver must continue to check the returned number of MSI-X interrupts per ddi_intr_alloc(9F) and must not assume its requests will be honored. Constraints: 1) This solution is limited to MSI-X interrupts only and it will not be extended to MSI interrupts. This means device drivers will continue to receive the current default number of MSI interrupts irrespective of this special driver property. 2) Device drivers using MSI-X interrupts without this property will only receive the default MSI-X allocation. 3) At present, there is no plan to implement this feature on non-SPARC platforms. The property, if present, will be ignored on x86 platforms. Business priority is to address performance issues on SPARC systems urgently, while the x86 platforms can wait for the long-term solution. Also, x86 systems provide much more limited interrupt resources and can suffer from allocating too many interrupts. The proposed interface is Contracted Consolidation Private. Each driver owner group is expected to contract the interfaces and transition to the new IRM based interfaces when they are available. 4.4. Interfaces interface | stability | description --------------+----------------------------------+---------------------- #msix-request | Contracted Consolidation Private | boolean property to | | override MSI-X limit --------------+----------------------------------+---------------------- Binding: patch/micro 4.5. References PSARC 2004/253 Advanced DDI Interrupt Interfaces 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From jek3@sun.com Tue Aug 7 15:30:42 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l77MUfvX028343 for ; Tue, 7 Aug 2007 15:30:41 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l77MSDeI029199; Wed, 8 Aug 2007 06:28:18 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JMF00503D33Z400@nwk-avmta-1.sfbay.Sun.COM>; Tue, 07 Aug 2007 15:28:15 -0700 (PDT) Received: from jurassic.eng.sun.com ([129.146.226.130]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JMF00156D33KG20@nwk-avmta-1.sfbay.Sun.COM>; Tue, 07 Aug 2007 15:28:15 -0700 (PDT) Received: from [129.150.12.124] (vpn-129-150-12-124.SFBay.Sun.COM [129.150.12.124]) by jurassic.eng.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l77MSEwv962580; Tue, 07 Aug 2007 15:28:14 -0700 (PDT) Date: Tue, 07 Aug 2007 12:28:57 -1000 From: Joseph Kowalski Subject: Re: PSARC/2007/453 MSI-X interrupt limit override In-reply-to: <46B8E223.2040107@sun.com> To: Artem Kachitchkine Cc: PSARC-ext@sun.com, Edward.Gillett@sun.com, Govinda.Tatti@sun.com Message-id: <46B8F229.20107@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <46B8E223.2040107@sun.com> User-Agent: Thunderbird 1.5.0.5 (X11/20060911) Status: RO Content-Length: 672 Artem Kachitchkine wrote: > I am sponsoring this case for Edward Gillett. > Requested binding is patch/micro, timeout 08/15/2007. > > -Artem ... > This proposal provides a short-term solution for device drivers to > request and receive more MSI-X interrupt resources than the current > default limit. The future long-term solution will be provided by the > Interrupt Resource Management (IRM) project. Proposed solution will be > implemented on SPARC only. Just to be sure, its only this "short-term" solution defined in this fast-track is SPARC only. The "longer-term" solution doesn't have an ISA restriction. I think that's what this says. Just checking... - jek3 From Edward.Gillett@sun.com Tue Aug 7 19:26:22 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l782QLit004160 for ; Tue, 7 Aug 2007 19:26:22 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l782O0ED011057 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 8 Aug 2007 10:24:02 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JMF00L17O010900@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Tue, 07 Aug 2007 19:24:01 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JMF00HT3NZWZJ60@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Tue, 07 Aug 2007 19:23:56 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l782NuBd023393 for ; Tue, 07 Aug 2007 19:23:56 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JMF00L01NZOGB00@fe-sfbay-10.sun.com> (original mail from Edward.Gillett@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Tue, 07 Aug 2007 19:23:56 -0700 (PDT) Received: from [129.146.96.116] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JMF009RANZWQTB0@fe-sfbay-10.sun.com>; Tue, 07 Aug 2007 19:23:56 -0700 (PDT) Date: Tue, 07 Aug 2007 19:23:39 -0700 From: Edward Gillett Subject: Re: PSARC/2007/453 MSI-X interrupt limit override In-reply-to: <46B8F229.20107@sun.com> Sender: Edward.Gillett@sun.com To: Joseph Kowalski Cc: Artem Kachitchkine , PSARC-ext@sun.com, Govinda.Tatti@sun.com Message-id: <46B9292B.9080001@sun.com> Organization: Sun Microsystems, Inc. MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <46B8E223.2040107@sun.com> <46B8F229.20107@sun.com> User-Agent: Thunderbird 2.0.0.4 (X11/20070615) Status: RO Content-Length: 741 > Artem Kachitchkine wrote: >> I am sponsoring this case for Edward Gillett. >> Requested binding is patch/micro, timeout 08/15/2007. >> >> -Artem > ... >> This proposal provides a short-term solution for device drivers to >> request and receive more MSI-X interrupt resources than the current >> default limit. The future long-term solution will be provided by the >> Interrupt Resource Management (IRM) project. Proposed solution will be >> implemented on SPARC only. > Just to be sure, its only this "short-term" solution defined in this > fast-track is SPARC only. The "longer-term" solution doesn't > have an ISA restriction. I think that's what this says. Just checking... yes, only the short term solution is SPARC only. Edward From Garrett.Damore@sun.com Tue Aug 7 22:38:52 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l785cpgM006636 for ; Tue, 7 Aug 2007 22:38:52 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l785aTh9020462 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 8 Aug 2007 13:36:31 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JMF0060NWWT2C00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Tue, 07 Aug 2007 23:36:29 -0600 (MDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JMF0036WWWTCOD0@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Tue, 07 Aug 2007 23:36:29 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l785aT7w028209 for ; Tue, 07 Aug 2007 22:36:29 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JMF00201WTLQC00@fe-sfbay-09.sun.com> (original mail from Garrett.Damore@Sun.COM) for PSARC-ext@Sun.Com (ORCPT PSARC-ext@Sun.Com); Tue, 07 Aug 2007 22:36:29 -0700 (PDT) Received: from [192.168.251.21] ([76.174.83.55]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JMF00IH9WWSX630@fe-sfbay-09.sun.com> for PSARC-ext@Sun.Com (ORCPT PSARC-ext@Sun.Com); Tue, 07 Aug 2007 22:36:28 -0700 (PDT) Date: Tue, 07 Aug 2007 22:32:41 -0700 From: "Garrett D'Amore" Subject: Re: PSARC/2007/453 MSI-X interrupt limit override In-reply-to: <46B8E223.2040107@sun.com> Sender: Garrett.Damore@sun.com To: Artem Kachitchkine Cc: PSARC-ext@sun.com, Edward.Gillett@sun.com, Govinda.Tatti@sun.com Message-id: <46B95579.6020505@sun.com> Organization: Sun Microsystems MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <46B8E223.2040107@sun.com> User-Agent: Thunderbird 2.0.0.0 (X11/20070618) Status: RO Content-Length: 5443 This is urgently needed for nxge. What I don't understand, however, is why or how IRM plays here. Are we concerned that there will be devices requesting more interrupts than are present on the system? What is so different from a device exporting this property versus requesting them via ddi_intr_alloc()? (In other words, having an arbitrary limit, with a trivial override, seems somewhat pointless to me, unless -- and quite possibly -- I'm missing some larger point.) Note that nxge will suffer even on x86 if the default is less than the number of cores available. I realize nxge performance on x86 isn't as interesting, but I'm uncomfortable with the idea of ignoring this. And future IHV-based products are likely to want/need more MSI-X interrupts as well. (The default of 2 is woefully inadequate, especially for large systems like x4600.) Perhaps the default value for this should be based upon the number of cores available in the system? (Admittedly, I am not intimately familiar with how MSI-X interrupts are allocated on x86 hardware...) -- Garrett Artem Kachitchkine wrote: > I am sponsoring this case for Edward Gillett. > Requested binding is patch/micro, timeout 08/15/2007. > > -Artem > > Template Version: @(#)sac_nextcase 1.64 07/13/07 SMI > This information is Copyright 2007 Sun Microsystems > 1. Introduction > 1.1. Project/Component Working Name: > MSI-X interrupt limit override > 1.2. Name of Document Author/Supplier: > Author: Edward Gillett > 1.3 Date of This Document: > 07 August, 2007 > 4. Technical Description > > 4.1. Summary > > This proposal provides a short-term solution for device drivers to > request and receive more MSI-X interrupt resources than the current > default limit. The future long-term solution will be provided by the > Interrupt Resource Management (IRM) project. Proposed solution will be > implemented on SPARC only. > > > 4.2. Problem > > Even though drivers can request arbitrary number of interrupts via > ddi_intr_alloc(9F), currently the number of returned MSI-X interrupts > is limited to 2. Some high-throughput drivers, such as nxge and qlc, > are forced to share 2 interrupts among multiple DMA channels, leading > to negative performance impact. > > The reason for the limit of 2 is that the initial phase of the Advanced > DDI interrupt project (PSARC 2004/253) did not implement any IRM > interfaces. The IRM project is expected to integrate in the early 2008, > but there is immediate need to address performance issues on Sun's > latest platforms. > > > 4.3. Proposal > > Provide a boolean device property, "#msix-request", that indicates to > the DDI framework to attempt to allocate more interrupts than the > default limit, should the driver request it. > > A device driver can create the proposed property if and only if it > requires more MSI-X interrupts than the default limit. It should create > this property before registering any of its interrupts. Sample code: > > (void) ddi_prop_create(DDI_DEV_T_NONE, dip, > DDI_PROP_CANSLEEP, "#msix-request", NULL, 0); > > The DDI framework will look for the proposed property as part of the > driver's interrupt registration process and will try to satisfy the > driver's MSI-X allocation request. The number of MSI-X interrupts that > can be allocated in this case may be less than or equal to what the > driver has requested. This depends on the availability of MSI-X > interrupt resources at that moment and also based on other platform > specific limitations. So, the driver must continue to check the > returned number of MSI-X interrupts per ddi_intr_alloc(9F) and must > not assume its requests will be honored. > > Constraints: > > 1) This solution is limited to MSI-X interrupts only and it will not be > extended to MSI interrupts. This means device drivers will continue > to receive the current default number of MSI interrupts irrespective > of this special driver property. > > 2) Device drivers using MSI-X interrupts without this property will > only receive the default MSI-X allocation. > > 3) At present, there is no plan to implement this feature on non-SPARC > platforms. The property, if present, will be ignored on x86 > platforms. Business priority is to address performance issues on > SPARC systems urgently, while the x86 platforms can wait for the > long-term solution. Also, x86 systems provide much more limited > interrupt resources and can suffer from allocating too many > interrupts. > > The proposed interface is Contracted Consolidation Private. Each driver > owner group is expected to contract the interfaces and transition to > the new IRM based interfaces when they are available. > > 4.4. Interfaces > > interface | stability | description > --------------+----------------------------------+---------------------- > #msix-request | Contracted Consolidation Private | boolean property to > | | override MSI-X limit > --------------+----------------------------------+---------------------- > > Binding: patch/micro > > 4.5. References > > PSARC 2004/253 Advanced DDI Interrupt Interfaces > > > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > ON > 6.5. ARC review type: FastTrack > 6.6. ARC Exposure: open > From Edward.Gillett@Sun.COM Wed Aug 8 13:57:36 2007 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l78KvZOh028099 for ; Wed, 8 Aug 2007 13:57:36 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l78KtA9C019028 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 8 Aug 2007 21:55:14 +0100 (BST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JMH00D173FYRX00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Wed, 08 Aug 2007 14:55:10 -0600 (MDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JMH004153FWBH80@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Wed, 08 Aug 2007 14:55:09 -0600 (MDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l78Kt8hr025863 for ; Wed, 08 Aug 2007 13:55:08 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JMH00J0139CQE00@fe-sfbay-10.sun.com> (original mail from Edward.Gillett@Sun.COM) for PSARC-ext@Sun.Com (ORCPT PSARC-ext@Sun.Com); Wed, 08 Aug 2007 13:55:08 -0700 (PDT) Received: from [129.146.96.116] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JMH00H9I3FWQ0D0@fe-sfbay-10.sun.com> for PSARC-ext@Sun.Com (ORCPT PSARC-ext@Sun.Com); Wed, 08 Aug 2007 13:55:08 -0700 (PDT) Date: Wed, 08 Aug 2007 13:54:51 -0700 From: Edward Gillett Subject: Re: PSARC/2007/453 MSI-X interrupt limit override In-reply-to: <46B95579.6020505@sun.com> Sender: Edward.Gillett@Sun.COM To: "Garrett D'Amore" Cc: Artem Kachitchkine , PSARC-ext@Sun.COM, Govinda.Tatti@Sun.COM Message-id: <46BA2D9B.4000206@sun.com> Organization: Sun Microsystems, Inc. MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <46B8E223.2040107@sun.com> <46B95579.6020505@sun.com> User-Agent: Thunderbird 2.0.0.4 (X11/20070615) Status: RO Content-Length: 6666 On 08/07/07 22:32, Garrett D'Amore wrote: > This is urgently needed for nxge. > > What I don't understand, however, is why or how IRM plays here. Are we > concerned that there will be devices requesting more interrupts than are > present on the system? Yes, that is one of the concerns here. IRM will be managing the interrupt resources, allocating and reclaiming them as necessary. Basically a contract between drivers and the ddi interrupt framework where a driver has to be aware that if it has requested additional interrupt resources, it may need to return them at a future time as system conditions change. The x86 architecture is much more limited than what we have on SPARC. Some priority levels have a very limited number of interrupt resources and without management it's possible for 1 or 2 devices to request everything and leave nothing for anyone else at that PIL level. You can contact the ddi-intr-iteam alias for more info. > What is so different from a device exporting this property versus > requesting them via ddi_intr_alloc()? (In other words, having an > arbitrary limit, with a trivial override, seems somewhat pointless to > me, unless -- and quite possibly -- I'm missing some larger point.) The framework can change the limit of how many interrupts can be allocated through ddi_intr_alloc(), but there is no contract or anything between the drivers and framework. You'll still need to make the request through ddi_intr_alloc(), but this property will provide an interim solution/contract for the drivers. Edward > Note that nxge will suffer even on x86 if the default is less than the > number of cores available. I realize nxge performance on x86 isn't as > interesting, but I'm uncomfortable with the idea of ignoring this. And > future IHV-based products are likely to want/need more MSI-X interrupts > as well. (The default of 2 is woefully inadequate, especially for large > systems like x4600.) > > Perhaps the default value for this should be based upon the number of > cores available in the system? (Admittedly, I am not intimately > familiar with how MSI-X interrupts are allocated on x86 hardware...) > > -- Garrett > > Artem Kachitchkine wrote: >> I am sponsoring this case for Edward Gillett. >> Requested binding is patch/micro, timeout 08/15/2007. >> >> -Artem >> >> Template Version: @(#)sac_nextcase 1.64 07/13/07 SMI >> This information is Copyright 2007 Sun Microsystems >> 1. Introduction >> 1.1. Project/Component Working Name: >> MSI-X interrupt limit override >> 1.2. Name of Document Author/Supplier: >> Author: Edward Gillett >> 1.3 Date of This Document: >> 07 August, 2007 >> 4. Technical Description >> >> 4.1. Summary >> >> This proposal provides a short-term solution for device drivers to >> request and receive more MSI-X interrupt resources than the current >> default limit. The future long-term solution will be provided by the >> Interrupt Resource Management (IRM) project. Proposed solution will be >> implemented on SPARC only. >> >> >> 4.2. Problem >> >> Even though drivers can request arbitrary number of interrupts via >> ddi_intr_alloc(9F), currently the number of returned MSI-X interrupts >> is limited to 2. Some high-throughput drivers, such as nxge and qlc, >> are forced to share 2 interrupts among multiple DMA channels, leading >> to negative performance impact. >> >> The reason for the limit of 2 is that the initial phase of the Advanced >> DDI interrupt project (PSARC 2004/253) did not implement any IRM >> interfaces. The IRM project is expected to integrate in the early 2008, >> but there is immediate need to address performance issues on Sun's >> latest platforms. >> >> >> 4.3. Proposal >> >> Provide a boolean device property, "#msix-request", that indicates to >> the DDI framework to attempt to allocate more interrupts than the >> default limit, should the driver request it. >> >> A device driver can create the proposed property if and only if it >> requires more MSI-X interrupts than the default limit. It should create >> this property before registering any of its interrupts. Sample code: >> >> (void) ddi_prop_create(DDI_DEV_T_NONE, dip, >> DDI_PROP_CANSLEEP, "#msix-request", NULL, 0); >> >> The DDI framework will look for the proposed property as part of the >> driver's interrupt registration process and will try to satisfy the >> driver's MSI-X allocation request. The number of MSI-X interrupts that >> can be allocated in this case may be less than or equal to what the >> driver has requested. This depends on the availability of MSI-X >> interrupt resources at that moment and also based on other platform >> specific limitations. So, the driver must continue to check the >> returned number of MSI-X interrupts per ddi_intr_alloc(9F) and must >> not assume its requests will be honored. >> >> Constraints: >> >> 1) This solution is limited to MSI-X interrupts only and it will not be >> extended to MSI interrupts. This means device drivers will continue >> to receive the current default number of MSI interrupts irrespective >> of this special driver property. >> >> 2) Device drivers using MSI-X interrupts without this property will >> only receive the default MSI-X allocation. >> >> 3) At present, there is no plan to implement this feature on non-SPARC >> platforms. The property, if present, will be ignored on x86 >> platforms. Business priority is to address performance issues on >> SPARC systems urgently, while the x86 platforms can wait for the >> long-term solution. Also, x86 systems provide much more limited >> interrupt resources and can suffer from allocating too many >> interrupts. >> >> The proposed interface is Contracted Consolidation Private. Each driver >> owner group is expected to contract the interfaces and transition to >> the new IRM based interfaces when they are available. >> >> 4.4. Interfaces >> >> interface | stability | description >> --------------+----------------------------------+---------------------- >> #msix-request | Contracted Consolidation Private | boolean property to >> | | override MSI-X limit >> --------------+----------------------------------+---------------------- >> >> Binding: patch/micro >> >> 4.5. References >> >> PSARC 2004/253 Advanced DDI Interrupt Interfaces >> >> >> 6. Resources and Schedule >> 6.4. Steering Committee requested information >> 6.4.1. Consolidation C-team Name: >> ON >> 6.5. ARC review type: FastTrack >> 6.6. ARC Exposure: open >> > From Artem.Kachitchkin@Sun.COM Wed Aug 15 13:04:01 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l7FK40iq020004 for ; Wed, 15 Aug 2007 13:04:01 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l7FK1VQl024080 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 16 Aug 2007 04:01:33 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JMT0053RZMIZH00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Wed, 15 Aug 2007 13:01:30 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JMT00HAUZMD53D0@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Wed, 15 Aug 2007 13:01:25 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l7FK1Pto000708 for ; Wed, 15 Aug 2007 13:01:25 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JMT00801ZCLOS00@fe-sfbay-10.sun.com> (original mail from Artem.Kachitchkin@Sun.COM) for PSARC-ext@Sun.Com (ORCPT PSARC-ext@Sun.Com); Wed, 15 Aug 2007 13:01:25 -0700 (PDT) Received: from [129.150.21.54] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JMT00AIWZM6WT10@fe-sfbay-10.sun.com> for PSARC-ext@Sun.Com (ORCPT PSARC-ext@Sun.Com); Wed, 15 Aug 2007 13:01:19 -0700 (PDT) Date: Wed, 15 Aug 2007 13:00:49 -0700 From: Artem Kachitchkine Subject: Re: PSARC/2007/453 MSI-X interrupt limit override In-reply-to: <46B8E223.2040107@sun.com> Sender: Artem.Kachitchkin@Sun.COM To: PSARC-EXT@Sun.COM Cc: Edward.Gillett@Sun.COM, Govinda.Tatti@Sun.COM Message-id: <46C35B71.8040209@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <46B8E223.2040107@sun.com> User-Agent: Mozilla/5.0 Gecko/20040113 Status: RO Content-Length: 79 Timeout was reached without open issues and the case is now approved. -Artem