From sacadmin Thu May 24 21:27:34 2007 Received: from sfbaymail2sca.sfbay.sun.com (sfbaymail2sca.SFBay.Sun.COM [129.145.155.42]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l4P4RYAH021951 for ; Thu, 24 May 2007 21:27:34 -0700 (PDT) Received: from nwk-ea-fw-1.sun.com (nwkes-gis-mail-1.SFBay.Sun.COM [10.4.134.5]) by sfbaymail2sca.sfbay.sun.com (8.13.6+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id l4P4QJIc026330 for ; Thu, 24 May 2007 21:26:19 -0700 (PDT) Received: from d1-sfbay-09.sun.com ([192.18.39.119]) by nwk-ea-fw-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l4P4QEvl025177 for ; Thu, 24 May 2007 21:26:14 -0700 (PDT) Received: from conversion-daemon.d1-sfbay-09.sun.com by d1-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) id <0JIK00L01XIB5H00@d1-sfbay-09.sun.com> (original mail from John.Plocher@Sun.COM) for PSARC-record@sac.sfbay.sun.com; Thu, 24 May 2007 21:26:14 -0700 (PDT) Received: from [192.168.168.4] ([66.166.204.98]) by d1-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) with ESMTPSA id <0JIK002QKXNPDIXI@d1-sfbay-09.sun.com> for PSARC-record@sac.sfbay.sun.com; Thu, 24 May 2007 21:26:13 -0700 (PDT) Date: Thu, 24 May 2007 21:26:09 -0700 From: John Plocher Subject: PSARC/2007/302 PSM_INTR_OPS extensions for handling groups of interrupt vectors] Sender: John.Plocher@Sun.COM To: PSARC-record@sac.sfbay.sun.com Message-id: <46566561.4050801@Sun.Com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT User-Agent: Thunderbird 1.5.0.10 (Macintosh/20070221) Status: RO Content-Length: 6525 resend -------- Original Message -------- Date: Thu, 24 May 2007 15:59:12 -0700 From: Artem Kachitchkine Subject: PSARC/2007/302 PSM_INTR_OPS extensions for handling groups of To: psarc-ext@Sun.COM Cc: Jack.A.Schwartz@Sun.COM I am sponsoring this case for Jack Schwartz. Requested binding is patch/micro, timeout 06/01/2007. -Artem. Template Version: @(#)sac_nextcase 1.58 05/23/07 SMI This information is Copyright 2007 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: PSM_INTR_OPS extensions for handling groups of interrupt vectors 1.2. Name of Document Author/Supplier: Author: Jack Schwartz 1.3 Date of This Document: 24 May, 2007 4. Technical Description 4.1 Summary This case extends PSM_INTR_OPS (platform specific module interrupt operations) to be able to change interrupt routing on X86 systems for groups of MSIs (Message Signaled Interrupts) belonging to a single device instance, and to be able to return interrupt controller information. This is a further extension of "PSARC/2005/584 PSM_INTR_OPS extensions". Interfaces affected are consolidation private. Patch/micro binding is requested. 4.2. Details Problem: On X86 platforms with Intel 82093 or other IO Advanced Programmable Interrupt Controllers (APICs) controlled by the pcplusmp module, vectors of multi-MSI-vector devices cannot be moved individually; they must be moved as a group. Trying to move them singly, as is currently being done, throws off the interrupt bookkeeping done by Solaris. This leads tools such as mdb, pcitool and kstats to incorrectly report these vectors as running on the wrong CPU. intrd(1M), an interrupt load balancing daemon, can misfire based on incorrect information, causing potential performance issues. Solution: Provide kernel support so logic can be applied in userland to do the right thing. A separate case will be filed to enhance the kernel/user ioctl interfaces. 1) Support atomic movement of a group of MSI vectors all belonging to the same device. 2) Provide a way of telling whether or not the system requires special handling for groups of MSI vectors belonging to the same device. (Currently only X86 systems with IO APICs supported by pcplusmp require this special handling.) The APIC module (pcplusmp) already does CPU migration of single interrupts. It has PSM_INTR_OPS called from the PCI (pci) and PCI express (npe) drivers to handle existing PCItool interrupt requests. This case extends PCItool support from the pcplusmp module with two new PSM_INTR_OPS, which the above ioctl interfaces can invoke in order to carry out their requests. Constraints: 1) Constraints on group moves (handled by PSM_INTR_OP_GRP_SET_CPU): - Vector passed in must be the lowest-numbered vector of the group. - Specifying a group move for a single vector is allowed and will be treated as a non-group move. 2) Constraint on a single move (handled by PSM_INTR_OP_SET_CPU) is the same as for the PCItool extension ioctls: - The vector to be moved must not be part of a group of (> 1) MSI vectors belonging to the same device. 3) This extension is limited to pci bus and its follow-on interconnects only. Interface Details: Proposed data structure and definition enhancements/changes: PSM_INTR_OP_GRP_SET_CPU is introduced to change the CPU<->interrupt-vector routing for a group of MSI vectors all belonging to the same device instance. It checks that the vector passed in is the lowest-numbered vector of an MSI vector group. PSM_INTR_OP_APIC_TYPE is introduced to return the name of the interrupt controller module as a string, and returns the minimum versions of the Local and IO APIC(s). PSM_INTR_OP_SET_CPU is enhanced to do extra checking that a vector passed to it is not part of a group of MSI vectors all belonging to a single device instance. Both new PSM_INTR_OPS make use of existing data structures for passing information. The psm_intr_op_t structure is augmented as follows: typedef enum psm_intr_op_e { PSM_INTR_OP_ALLOC_VECTORS = 0, /* 0. Allocate vectors */ PSM_INTR_OP_FREE_VECTORS, /* 1. Free vectors */ PSM_INTR_OP_NAVAIL_VECTORS, /* 2. Get # of available vectors */ PSM_INTR_OP_XLATE_VECTOR, /* 3. Translate vector */ PSM_INTR_OP_GET_PENDING, /* 4. Get pending information */ PSM_INTR_OP_CLEAR_MASK, /* 5. Clear interrupt mask */ PSM_INTR_OP_SET_MASK, /* 6. Set interrupt mask */ PSM_INTR_OP_GET_CAP, /* 7. Get devices's capabilities */ PSM_INTR_OP_SET_CAP, /* 8. Set devices's capabilities */ PSM_INTR_OP_SET_PRI, /* 9. Set the interrupt priority */ PSM_INTR_OP_GET_SHARED, /* 10. Get the shared intr info */ PSM_INTR_OP_CHECK_MSI, /* 11. Chk if device supports MSI */ PSM_INTR_OP_SET_CPU, /* 12. Set vector's CPU */ PSM_INTR_OP_GET_INTR, /* 13. Get vector's info */ -----> PSM_INTR_OP_GRP_SET_CPU, /* 14. Set all dev's vectors' CPU */ -----> PSM_INTR_OP_APIC_TYPE /* 15. Returns APIC type and info */ } psm_intr_op_t; PSM_INTR_OP_APIC_TYPE returns versioning info in a single platform-independent field. The two X86-specific subfields for Local and IO APIC versions are accessed via APIC_LOCAL_VER and APIC_IO_VER macros. 4.3. Interface table All affected interfaces will be/remain consolidation private (CP). ------------------------+-----+------------------------------------------- PSM_INTR_OP_GRP_SET_CPU | CP | Command to APIC to remap CPU of group of | | MSIs belonging to the same device instance. PSM_INTR_OP_APIC_TYPE | CP | Command to APIC to retrieve APIC name and | | HW version information. PSMAT_LOCAL_APIC_VER | CP | Returns min version of local APIC(s) PSMAT_IO_APIC_VER | CP | Returns min version of IO APIC(s) ------------------------+-----+------------------------------------------- 4.4. References PSARC/2005/232 pcitool and its nexus ioctl support PSARC/2005/584 PSM_INTR_OPS extensions PSARC/2005/091 PSMI Extension for ADII 6458838 Once intrd performs reassignment, MSI interrupts stop coming 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack ---- end of original message ----