Subject: MSI-X interrupt limit override [PSARC/2007/453 FastTrack timeout 08/15/2007] To: PSARC-ext@Sun.Com Cc: Edward.Gillett@Sun.COM, Govinda.Tatti@Sun.COM Bcc: one-pager-list@sac.sfbay one-pager-log@sac.sfbay sac-bar@sac.sfbay Template Version: @(#)sac_nextcase 1.64 07/13/07 SMI This information is Copyright 2007 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: MSI-X interrupt limit override 1.2. Name of Document Author/Supplier: Author: Edward Gillett 1.3 Date of This Document: 07 August, 2007 4. Technical Description 4.1. Summary This proposal provides a short-term solution for device drivers to request and receive more MSI-X interrupt resources than the current default limit. The future long-term solution will be provided by the Interrupt Resource Management (IRM) project. Proposed solution will be implemented on SPARC only. 4.2. Problem Even though drivers can request arbitrary number of interrupts via ddi_intr_alloc(9F), currently the number of returned MSI-X interrupts is limited to 2. Some high-throughput drivers, such as nxge and qlc, are forced to share 2 interrupts among multiple DMA channels, leading to very negative performance impact. The reason for the limit of 2 is that the initial phase of the Advanced DDI interrupt project (PSARC 2004/253) did not implement any IRM interfaces. The IRM project is expected to integrate in the early 2008, but there is immediate need to address performance issues on Sun's latest platforms. 4.3. Proposal Provide a boolean device property, "#msix-request", that indicates to the DDI framework to attempt to allocate more interrupts than the default limit, should the driver request it. A device driver can create the proposed property if and only if it requires more MSI-X interrupts than the default limit. It should create this property before registering any of its interrupts. Sample code: (void) ddi_prop_create(DDI_DEV_T_NONE, dip, DDI_PROP_CANSLEEP, "#msix-request", NULL, 0); The DDI framework will look for the proposed property as part of the driver's interrupt registration process and will try to satisfy the driver's MSI-X allocation request. The number of MSI-X interrupts that can be allocated in this case may be less than or equal to what the driver has requested. This depends on the availability of MSI-X interrupt resources at that moment and also based on other platform specific limitations. So, the driver must continue to check the returned number of MSI-X interrupts per ddi_intr_alloc(9F) and must not assume its requests will be honored. Constraints: 1) This solution is limited to MSI-X interrupts only and it will not be extended to MSI interrupts. This means device drivers will continue to receive the current default number of MSI interrupts irrespective of this special driver property. 2) Device drivers using MSI-X interrupts without this property will only receive the default MSI-X allocation. 3) At present, there is no plan to implement this feature on non-SPARC platforms. The property, if present, will be ignored on x86 platforms. Business priority is to address peformance issues on SPARC systems urgently, while the x86 platforms can wait for the long-term solution. Also, x86 systems provide much more limited interrupt resources and can suffer from allocating too many interrupts. The proposed interface is Contracted Consolidation Private. Each driver owner group is expected to contract the interfaces and transition to the new IRM based interfaces when they are available. 4.4. Interfaces interface | stability | description --------------+----------------------------------+----------------------- #msix-request | Contracted Consolidation Private | boolean property for | | allocating MSI-X intrs --------------+----------------------------------+----------------------- Binding: patch/micro 4.5. References PSARC 2004/253 Advanced DDI Interrupt Interfaces 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open