sun microsystems Systems Architecture Committee _________________________________________________________________ Subject: PCI Hot-Plug Support Submitted by: Prasad Singamsetty File: PSARC/1998/327/opinion.ms Date: November 4th, 1998 Committee: Steve Z., Joseph Kowalski, David R., Brian W. Steering Committee: Foundation Steering Committee sfsc-opinion@scs.eng 1. Summary This project provides support within Solaris for hot- plugging PCI devices based on the PCI Hot-Plug Specification [2]. The project delivers the framework changes, the com- ponents common to all Solaris platforms, and platform- specific components for some Intel (x86) platforms. It sup- ports Hot Expansion and Hot Removal, but not the Like-for- Like Replacement feature, of the PCI Hot-Plug technology. 2. Decision & Precedence Information The project is approved as specified in reference [1]. The project introduces no incompatible changes and may be delivered in a minor release of Solaris. The project depends on the following project and may not be delivered before it: PSARC/1998/471 add-drv extensions 3. Interfaces The project exports the following interfaces. ______________________________________________________________________________ | Libdevice.so.1 Interfaces Exported (user level) | |_____________________|_________________|____________________________________| |Interface | Classification | Comments | |_____________________|_________________|____________________________________| |devctl_ap_acquire | Project Private| Acquire access to attachment point| |devctl_ap_connect | Project Private| CONNECT on attachment point | |devctl_ap_disconnect | Project Private| DISCONNECT on attachment point | |_____________________|_________________|____________________________________| PSARC/1998/327 Copyright 1998 Sun Microsystems, Inc. - 2 - ______________________________________________________________________________ | Libdevice.so.1 Interfaces Exported (user level) | |_____________________|_________________|____________________________________| |Interface | Classification | Comments | |_____________________|_________________|____________________________________| |devctl_ap_configure | Project Private| CONFIGURE on attachment point | |devctl_ap_unconfigure| Project Private| UNCONFIGURE on attachment point | |devctl_ap_insert | Project Private| INSERT on attachment point | |devctl_ap_remove | Project Private| REMOVE on attachment point | |devctl_ap_getstate | Project Private| GETSTATE of attachment point | |devctl_ap_state_t | Project Private| State describing attachment point | |_____________________|_________________|____________________________________| _________________________________________________________________________ | Bus Nexus IOCTL Interfaces Exported | |_____________________|_________________|_______________________________| |Interface | Classification | Comments | |_____________________|_________________|_______________________________| |DEVCTL_AP_ACQUIRE | Project Private| ACQUIRE attachment point | |DEVCTL_AP_CONNECT | Project Private| CONNECT attachment point. | |DEVCTL_AP_DISCONNECT | Project Private| DISCONNECT attachment point | |DEVCTL_AP_INSERT | Project Private| INSERT attachment point | |DEVCTL_AP_REMOVE | Project Private| REMOVE attachment point | |DEVCTL_AP_CONFIGURE | Project Private| CONFIGURE attachment point | |DEVCTL_AP_UNCONFIGURE| Project Private| UNCONFIGURE attachment point | |DEVCTL_AP_GETSTATE | Project Private| Get state of attachment point| |DEVCTL_AP_CONTROL | Project Private| HPC specific control commands| |_____________________|_________________|_______________________________| ________________________________________________________________________ | NDI Interfaces Exported | |___________________|_________________|________________________________| |Interface | Classification | Comments | |___________________|_________________|________________________________| |ndi_dc_return_state| Project Private| Copy the attachment state info| |___________________|_________________|________________________________| _______________________________________________________________________________ | Hot-Plug Controller (HPCI) Interfaces Exported | |_________________________|_________________|_________________________________| |Interface | Classification | Comments | |_________________________|_________________|_________________________________| |hpc_register_bus | Project Private| Register the pci bus node | |hpc_unregister_bus | Project Private| Unregister the pci bus node | |hpc_install_event_handler| Project Private| Install an event handler | |hpc_remove_event_handler | Project Private| Remove an event handler | |hpc_connect | Project Private| Connect slot to bus | |hpc_disconnect | Project Private| Disconnect slot from bus | |_________________________|_________________|_________________________________| PSARC/1998/327 Copyright 1998 Sun Microsystems, Inc. - 3 - _______________________________________________________________________________ | Hot-Plug Controller (HPCI) Interfaces Exported | |_________________________|_________________|_________________________________| |Interface | Classification | Comments | |_________________________|_________________|_________________________________| |hpc_insert | Project Private| Prepare slot for card insertion| |hpc_remove | Project Private| Prepare slot for card removal. | |hpc_control | Project Private| HW specific control operations | |hpc_register_slot | Project Private| Register the hot-plug slot | |hpc_unregister_slot | Project Private| Unregister the hot-plug slot | |hpc_event_notify | Project Private| Notify the event occurrence | |hpc_alloc_slot_ops | Project Private| Allocate slot_ops structure | |hpc_free_slot_ops | Project Private| Free slot_ops structure | |hpc_slot_ops_t | Project Private| HPC ops structure | |hpc_slot_info_t | Project Private| Slot info type | |hpc_pci_slotinfo_t | Project Private| PCI slot information structure | |_________________________|_________________|_________________________________| _____________________________________________________________________________________ | PCI Resource Allocator and Configurator (PRAC) Interfaces Exported | |_____________________|_________________|___________________________________________| |Interface | Classification | Comments | |_____________________|_________________|___________________________________________| |prac_setup | Project Private| Setup allocator for bus instance | |prac_teardown | Project Private| Teardown allocator for bus instance | |prac_configure | Project Private| Create devinfo tree and configure devices| |prac_unconfigure | Project Private| Remove the devinfo subtree | |prac_alloc_space | Project Private| Allocate space from I/O or memory | |prac_free_space | Project Private| Free space to I/O or memory | |prac_alloc_bus_number| Project Private| Allocate new bus number | |prac_free_bus_number | Project Private| Free bus back to pool | |prac_handle_t | Project Private| PRA handle for the bus | |_____________________|_________________|___________________________________________| The project imports the following interfaces. ____________________________________________________ | Interfaces Imported | |_________________|________________|_______________| |Interface | Classification| Comments | |_________________|________________|_______________| |PCI Hot-Plug Spec| Standard | Reference [2]| |_________________|________________|_______________| 4. Opinion No issues of any substance were raised by the committee. Most of the discussion was related to issues posed by the team in section 3.10 of [1] and, in the days following the review, by the DDI team. PSARC/1998/327 Copyright 1998 Sun Microsystems, Inc. - 4 - 4.1. Loading of the HPS module and the HPC driver module The team noted that, since no application directly accesses the hot-plug controller (HPC) device, some means must be provided for causing the HPC driver module to be loaded. It was agreed that the mechanism would be specified by another case. That case, PSARC/1998/471, add-drv extensions, has been filed and is listed as a dependency for this case. 4.2. Need for PROM devinfo nodes for hot-plugged devices The team pointed out that the prtconf utility does not show PROM-defined properties, so programs that parse its output may not work as expected if the PROM devinfo tree is not updated for hot-plugged devices. This observation led, after some discussion, to the advisory to the Steering Committee (below) that boot and kernel devinfo nodes should be treated uniformly. 4.3. Association of hot-plug slot with bus nexus node The association between a hot-plug slot, the device that controls the slot, and the bus nexus driver that is the parent of the device in the hot-plug slot is platform depen- dent. The team noted that this requires platform-specific means of associating these components, and that this case cannot provide a standard solution to this problem. One means, which will be used for some Intel platforms, is to put the required information in properties in the configura- tion (.conf) file for the hot-plug controller (HPC) device. On SPARC platforms, this information could be provided directly by firmware-generated properties. 4.4. Support for partial configure/unconfigure operation The team noted that some PCI cards (``multi-function'') con- tain multiple independent devices (e.g., a SCSI controller and an ethernet controller). While such devices are hot- plugged ``together'', they are otherwise logically indepen- dent. The committee agreed with the team's assertion that, although some work is needed to provide interfaces for con- figuring and unconfiguring individual device functions on an attachment point, this work need not be a part of this pro- ject. Existing projects (PSARC/1998/423, cfgadm extensions, and PSARC/1998/451, devctl(1M)) will provide the required interfaces, but this project is not directly dependent on them. 4.5. Commitment Levels When the case was approved at the inception review, the com- mitment level of the devctl_ap interfaces was Evolving, and that of the ioctls and the ndi interfaces was Sun Private. Subsequent to the review, the DDI group asked that the PSARC/1998/327 Copyright 1998 Sun Microsystems, Inc. - 5 - commitment level be lowered to Project Private, so long as the PCI Hot-plug project is the only one using the inter- faces, with the intent upgrade the commitment level later. 5. Minority Opinion(s) None 6. Advisory Information With the advent of hot-plugging, the distinction between the parts of the device tree provided by the ``PROM'' (the firmware on SPARC or the booting system on Intel platforms) and the parts discovered by the kernel becomes somewhat artificial. As a result, the already confusing difference in output provided by the prtconf utility for boot vs. ker- nel nodes (e.g., some properties are not copied from the boot nodes into the kernel tree) is exacerbated. The Steer- ing Committee is advised to sponsor a project to bring uni- formity to the treatment of boot and kernel nodes. 7. Appendices 7.1. Appendix A: Technical Changes Required None. 7.2. Appendix B: Technical Changes Advised None. 7.3. Appendix C: Reference Material Unless stated otherwise, path names are relative to the case directory, PSARC/1998/327. 1. Project Specification File: incept.materials/project.overview 2. PCI Hot-Plug Specification, Revision 1.0 File: incept.materials/Pcihp10.ps 3. Project One-Pager File: incept.materials/1pager 4. Response to PSARC 20 Questions File: incept.materials/20questions PSARC/1998/327 Copyright 1998 Sun Microsystems, Inc.