Template Version: @(#)sac_nextcase 1.58 05/23/07 SMI
This information is Copyright 2007 Sun Microsystems
1. Introduction
    1.1. Project/Component Working Name:
	 PCItool extensions for handling groups of interrupt vectors
    1.2. Name of Document Author/Supplier:
	 Author:  Jack Schwartz
    1.3  Date of This Document:
	24 May, 2007
4. Technical Description

4.1. Summary

Enhance PCI and PCI-express driver interrupt ioctls introduced in
"PSARC/2005/232 pcitool and its nexus ioctl support" to support CPU
migration of groups of interrupt vectors, where the vectors of such
groups have to be moved as a unit. A separate case will be filed to
facilitate kernel implementation of these ioctls.

Interfaces affected are consolidation private.
Patch/micro binding is requested.

4.2. Details

Problem: On some platforms, such as X86, MSI (Message Signaled
Interrupt) vectors of multi-MSI-vector devices cannot be moved
individually; they must be moved as a group.  Trying to move them
singly throws off the bookkeeping done by Solaris, and tools such as
mdb, pcitool and kstats will then incorrectly report these vectors as
running on the wrong CPU.  intrd(1M), an interrupt load balancing
daemon, can misfire based on incorrect information, causing potential
performance issues.

Solution: Introduce new/replacement ioctls which:

1) Support atomic movement of a group of vectors all belonging to the
same device.  (Whether or not this is implemented on a particular
platform or for a particular type of vector is out of the scope of this
case.)

2) Provide a way of telling whether or not the system requires special
handling for groups of vectors belonging to the same device. 
(Currently only X86 systems with IO APICs supported by pcplusmp require
this special handling for groups of MSI vectors.)

Constraints:

1) Group of vectors operations require the same privileges as
respective single vector operations.

2) Specific platforms or interrupt types may have additional
constraints on group moves which are out of the scope of this case.

Interface Details:

These new ioctls will utilize the existing PCItool interrupt minor
nodes made available from PCI and PCI-express nexus drivers.

PCITOOL_DEVICE_SET_INTR ioctl (enhanced)

After the proposed changes, this ioctl remaps an vector or group of
vectors belonging to a single device to a new CPU.

Add a new flags field, and the definition of a flag which specifies
migration of a group of vectors instead of a single vector migration.

After the proposed changes, the following information will be exchanged:

Ino (unchanged)
	- to kernel
	The interrupt number to remap

CPU ID (unchanged)
	- to/from kernel
	input: The CPU to remap to.
	output: The originally mapped CPU.

Status (unchanged)
	- from kernel

Flags (new)
	- to kernel
	Accepts the following flag:
		PCITOOL_INTR_SET_FLAG_GROUP - when set, the remap
			request is for a group of vectors.  When clear,
			the request is for a single vector.


PCITOOL_SYSTEM_INTR_INFO ioctl (new)

This new ioctl returns system interrupt information.  The following
information will be exchanged:

Controller version
	- from kernel
	Platform independent interrupt controller hardware versioning info.

Controller type
	- from kernel
	Type of system interrupt controller(s).  Possible types to return:
		PCITOOL_CTLR_TYPE_RISC	  - Returned on SPARC systems
		PCITOOL_CTLR_TYPE_UPPC	  - Returned on X86 systems
						without IOAPICs 
		PCITOOL_CTLR_TYPE_PCPLUSMP- Returned on X86 systems
						with IOAPICs managed by
						a PCPLUSMP software module.
		PCITOOL_CTLR_TYPE_UNKNOWN - Returned when intr controller
						type is unknown

Number of interrupt vectors
	- from kernel
	The number of supported interrupt vectors on a PCI or PCI express bus.


PCITOOL_DEVICE_NUM_INTR ioctl (removed)

The information returned formerly by this ioctl is now returned by the
new PCITOOL_SYSTEM_INTR_INFO ioctl.


4.3. Interface table

All affected interfaces will be/remain consolidation private (CP).

---------------------------+----+-------------------------------------------
PCITOOL_DEVICE_SET_INTR    | CP | ioctl to remap CPU of intr or grp of intr
PCITOOL_SYSTEM_INTR_INFO   | CP | ioctl to retrieve system interrupt info  
PCITOOL_DEVICE_NUM_INTR    | CP | (removed) ioctl to get num intr per nexus
pcitool_intr_set_t         | CP | PCITOOL_DEVICE_SET_INTR arg data structure
pcitool_intr_info_t        | CP | PCITOOL_SYSTEM_INTR_INFO arg data structure
PCITOOL_CTLR_TYPE_RISC	   | CP | SPARC system interrupt controller type
PCITOOL_CTLR_TYPE_UPPC     | CP | Interrupt controller type for X86 system
                           |    | w/o IO APIC
PCITOOL_CTLR_TYPE_PCPLUSMP | CP | Interrupt controller type for X86 system
                           |    | w/ PCPLUSMP IO APIC
PCITOOL_CTLR_TYPE_UNKNOWN  | CP | Unknown interrupt controller
PCITOOL_INTR_SET_FLAG_GROUP| CP | Flag specifying when set to move a group
                           |    | of MSI vectors
---------------------------+----+--------------------------------------------

4.4. References

PSARC/2005/232 pcitool and its nexus ioctl support
PSARC/2005/584 PSM_INTR_OPS extensions
6458838 Once intrd performs reassignment, MSI interrupts stop coming

6. Resources and Schedule
    6.4. Steering Committee requested information
   	6.4.1. Consolidation C-team Name:
		ON
    6.5. ARC review type: FastTrack