PSARC case: 1998/327 (PCI Hot Plug Support) Description: A Technical Overview of "PCI Hot Plug Support" Project Date: October 29, 1998. Contents: Chapter 1. Overview of Hot-Plug PCI Architecture Chapter 2. Goals of the project Chapter 3. Project Design Overview Chapter 4. Project Interfaces Chapter 5. References 1.0 Overview of Hot-Plug PCI Architecture Hot Plug PCI is a new feature being added as an option for PCI busses. It makes no changes to existing PCI add-on cards or to PCI bridges. The changes consists of an active device being interposed between the PCI bridge and the slot in which a PCI card would be inserted. This device is controlled and managed by a separate Hot Plug Controller device. For hardware implementation details, see the Hot Plug PCI Specification. +-----------------------+ | PCI Bridge | +-----------------------+ +------------+ | | | | | | .....|.........|.................| Hot Plug | | : | ........|.................| Controller | | : | : | ................| Hardware | / \: / \: / \: | | \ / \ / \ / +------------+ | | | | | | +---+ +---+ +---+ / \ Indicates the | S | | S | | S | \ / controller switches | l | | l | | l | and indicators. | o | | o | | o | | t | | t |.....| t | | | | | | | | 0 | | 1 | | N | +---+ +---+ +---+ Figure 1: Hot Plug Controller hardware 1.1 PCI Hot Plug Specification 1.0 This specification is published by PCI SIG. It standardizes the key aspects of the PCI Hot Plug functionality by specifying the requirements for the hot plug platform and the OS that provides programmatic interface to the hot plug technology. It also specifies the minimum electrical requirements for the PCI adapters to be hot pluggable. The specification does not specify the details of any platform specific implementation or details of how the OS should implement the hot plug features. 1.2 PCI Hot Plug Controller Hardware The Hot-Plug controller is a separate device from the PCI bus itself. It may be implemented in a number of different ways as long as it provides the required functionality defined in the Hot-Plug PCI Specification. Some examples of platform specific implementations of Hot Plug Controller include: . Hot-Plug Controller as a PCI device attached to the the same bus that has hot pluggable slots. (e.g. Compaq Proliant 6500/7000). . Hot-Plug Controller is not a PCI device attached to to some other bus in the system. (e.g. Daktari platform) The controller essentially controls the power supply to each hot-pluggable slot, controls the connection of the slot pins to the PCI bus and controls the indicators present on each slot. It must also provide status information such as whether a card is present or not and what type of card (33 vs 66 MHZ, 32 bit vs 64bit, etc.). It may optionally provide an asynchronous notification of a card insertion but does not support asynchronous removal of a card. The full details can be found in the Hot-Plug PCI Specification. A platform may have multiple instances of the Hot-Plug Controller, each instance supporting a set of slots. Note that the architecture supports the implementation where a single controller controls slots from multiple PCI busses as well as multiple controllers on a single bus. The implementation of Hot Plug Controller is platform vendor specific so no details are presented here. 1.3 Hot Plug Operations Removal and Insertion of PCI cards is done in an orderly fashion where the Operating System vendor and platform vendor define the sequence of user actions and system management facilities to insert or remove a card from the slot. Inserting or removing an adapter card without following the proper sequence may lead to unpredictable results, including data corruption and system panics. Hot-Plug PCI differs from an active hot-plug bus like PCMCIA in that it is not designed to have devices removed from slots at just any time. The user must notify the operating system first and then wait until the OS indicates that removal/insertion can be safely performed. PCMCIA devices were designed in such a way that hot insertion/removal does not cause a problem or damage the devices. For PCI, the slot is actually electrically isolated from the bus during insertion/removal. 1.3.1 Hot Expansion This is the ability to add PCI adapter cards to a system and load the respective device drivers without bringing the system down. Normally the boot firmware configures the PCI devices and assigns bus resources (memory, i/o space and interrupt assignment if any). In the case of Hot Expansion the OS is expected to support configuration of the new PCI device(s) on the expansion card. To support this hot plug feature, the device drivers must be capable of dynamically attaching to new device instances. 1.3.2 Hot Removal This is the ability to remove PCI adapter cards from a system while the system is running, without bringing down the system. This requires detaching the device drivers for all the devices involved and freeing the PCI bus resources used by the card. For this feature, the device drivers must be capable of detaching themselves dynamically from the device instances. 1.3.3 Like-for-Like Replacement This is the ability to replace a failing or failed PCI card from a running system without bringing down the system or requiring the applications/drivers to detach from the device(s). Here the assumption is that the adapter replaced is identical to the failing adapter. During the replacement operation the device(s) on the adapter are basically suspended for the duration of hot replacement operation. 1.3.4 Hot Upgrade The ability to upgrade existing PCI controllers and drivers to next generation controllers and drivers while a system is running, without bringing down the system. 1.4 OS Support To support PCI Hot Plug operations, the OS must provide support for dynamically configuring/unconfiguring devices from the system, user interface to perform hot plug operations, probe and configuration of PCI bus/device(s), and hot plug aware device drivers. Also, the hot plug controller driver must be provided which controls the hot plug slots. 1.4.1 PCI Bus Probing and PCI Bus Resource Allocation Normally the boot firmware probes and configures devices on the PCI busses and programs the PCI configuration registers of the PCI devices and PCI bridge devices. The OS needs to support this dynamically for PCI Hot Plug functionality. This requires PCI bus resource management (memory and i/o spaces, new bus numbers for PCI-to-PCI bridges, etc.) support in the kernel. 1.4.2 FCODE PROM Normally the interpretation of FCODE on the PCI card is done by the boot firmware. If the FCODE PROM has hardware (vendor) specific code (e.g initialization of hardware) then to support that hardware, OS needs to provide support for interpreting the FCODE. 1.4.3 User Interface A command level or GUI interface is necessary to perform the PCI hot plug operations. The sequence of steps necessary to perform the operation is OS and platform vendor dependent. 1.4.4 Hot Plug Aware Device Drivers The device drivers are dynamically loaded to attach new device instances. So, the device drivers must be hot plug aware to attach/detach a specific device instance without affecting other device instances. The driver may also need to perform any hardware specific initialization, e.g. downloading firmware to the card. 1.4.5 Hot Plug Controller Driver The implementation of the hardware is platform dependent. Typically the platform vendor provides the driver, written using the OS specific DDI to support the connect/disconnect operations on the PCI hot plug slots. 2.0 Goals of PCI Hot Plug Support project The main goal of this project is to design and implement an architecture that supports PCI Hot Plug functionality in Solaris for both PCI and CompactPCI (cPCI) based hardware platforms. Solaris already has hot plug framework which is enhanced by this project to support PCI Hot Plug. The project also provides implementation of platform specific components for predefined x86 and SPARC platforms. The project documents the changes needed to convert Solaris device drivers to support PCI Hot Plug functionality. It also documents writing Hot Plug Controller drivers for new platforms. It is not the goal of this project to define a complete High Availability solution using this technology. But, the support of the PCI Hot Plug technology added by this project to Solaris is expected to work in a HA solution like Alternate Pathing software. Even without a HA software the PCI Hot Plug features add value to Solaris product by supporting Hot Expansion and Hot Upgrade of PCI adapters. In the context of HA software like Alternate Pathing with redundant hardware, a failing adapter can be replaced after moving the device functions to the redundant hardware and then doing Hot Remove and Hot Addition to replace the card. Note: For Solaris HA related information see http://ha.eng web page. 2.1 Implementation Strategy Due to OEM hardware introduction requirements and time constraints, the project will be implemented in multiple phases. The ARC review for this project is only for the phase-I implementation. The definition of other phases including resources/schedules are not known at this time. o Phase-I: To support Hot Add and Hot Remove features. For the Hot Add case, the operation will fail if there are not enough bus resources to configure the new card. The Hot Upgrade feature is supported as Hot Remove followed by Hot Add operation. It is assumed that the Like-for-Like replacement functionality is achieved by Hot Remove followed by Hot Add operation in the context of High Availability solution like Alternate Pathing or Failover software. The cfgadm(1M) command interface is used to support administration of hot plug slots. This phase may also include an FCODE interpreter for SPARC platforms. o Phase-NN: Other features that may be supported in the subsequent phases include: . Support for ACPI systems. . Support for a GUI admin tool like SyMON. . I2O Hot Plug support. . Automation of hot plug operations without the need for user commands. For example, when a new card is inserted the system should automatically configure the card and inform the user about it. This will require event framework that notifies the user applications about the hot plug events. Currently there is a 1pager on Resource Event Manager framework that supports this functionality. . Additional platforms and PCI adapters. Note: No resources or schedules are defined for this work. The strategy for this project (phase-I only) is to develop and integrate the basic framework necessary to support PCI Hot Plug on all Solaris platform architectures plus platform support for production-level Intel platforms prior to the 2.8 functionality freeze. Subsequently, additional platform-specific support for SPARC and Intel systems, consisting of hot-plug controller drivers and updated PCI device drivers, can be added to Solaris with little risk. 2.2 Platforms Supported Intel: Compaq Proliant 6500/7000, Intel ESG "Polar" (Q3CY98), NCR platform (TBD), SNI platform (TBD). SPARC: Daktari platform (CY00), Serengeti platform (CY00), Starcat platform (CY00). 3.0 Project Design Overview This chapter gives an overview of the software architecture and the functionality of software components involved in supporting the PCI hot plug operations. Section 3.9 discusses the limitations and assumptions on the design; section 3.10 lists the issues that need to be addressed in the context of this project. 3.1 PCI Hot Plug Software Architecture +-----------------+ | cfgadm (1M) | +-----------------+ | | v +-----------------+ | Hot Plug PCI | hardware specific | library | plug-in library +-----------------+ | | | User ======================================|============================== | Kernel | +------------+ +----------+ +----v------+ +--------------+ | Hot Plug | | Hot Plug | | PCI Nexus | | PCI bus | | controller <---> Services <----> driver <------> configurator | | driver | | module | | | | module | +------------+ +----------+ +-----------+ +--------------+ | | /\ / \ v v ( Hot Plug capable drivers including HBAs) Figure 2: PCI Hot Plug Software Architecture The Figure 2 shows the block diagram of software components involved in supporting the PCI hot plug operations. The software components are discussed further in this section. 3.1.1 cfgadm(1M) command Cfgadm(1M) is a user level command interface as defined in the PSARC case 1996/285. It uses the generic library, libcfgadm, and hot plug hardware specific plug-in library to manage hot plug slots on that bus. The hot plug operations are performed on Attachment Point nodes (PSARC case 1996/285) in the devinfo tree which identify the logical/physical location of hot plug slots in the system. For PCI Hot Plug, each hot plug slot is represented by an Attachment Point of that bus. The node type of attachment point is defined as DDI_NT_PCI_ATTACHMENT_POINT which is a string of value, "ddi_ctl:attachment_point:hp_pci". Cfgadm uses the class string 'hp_pci' in the node type to load the plug-in library for PCI. (See section 4.4 for details of the cfgadm(1M) extensions.) 3.1.2 Hot Plug PCI (plug-in) library Hot Plug PCI library is the cfgadm plug-in library which is specific to the PCI bus. This uses devctl_* interfaces defined in libdevice.so to communicate with the nexus driver to perform hot plug operations on the attachment point. The strategy is to have this library as a generic library for PCI with no platform specific code, but the cfgadm(1M) model still allows for platform specific plug-in to work in place of this generic library. This project defines extensions to devctl_* interfaces (see section 4.1) to support operations on Attachment Points. 3.1.3 Hot Plug Services module The Hot Plug Services (HPS) module provides the bus independent transport services between a nexus driver and hot plug controller driver. The HPS module is designed to be a generic module that provides interfaces and communication mechanism between the nexus driver for an instance of a hot-pluggable bus and the device driver for an instance of a hot-plug controller responsible for the slots on that bus. The hot-plug services module works by providing an abstract set of hot-plug operations to a bus nexus driver. Examples of these operations are connect and disconnect in the cfgadm model. A driver for a hot-plug controller registers its device specific hot-plug operations with the hot-plug services module, and module vectors abstract operations requested by the bus nexus driver to the device specific operations registered by the hot-plug controller driver. The hot-plug services module also provides a way for a bus nexus driver to register interest in events on a hot-plug controller such as a slot insertion, slot removal, etc. This module is developed as a generic kernel/misc module. Both the PCI nexus and platform specific HPC driver will have dependency on this module. This module framework is similar to SCSA and the I2O message module. (See section 4.2 for details of interfaces provided by HPS module.) 3.1.4 Hot Plug Controller driver Hot Plug Controller (HPC) driver is a Solaris device driver written for the platform specific Hot Plug Controller hardware. The primary purpose of the device driver for a hot-plug controller is to provide the hot-plug services module abstraction for its underlying hardware. This abstraction means providing an implementation for the connect and disconnect operations, as well as translating controller interrupts to hot-plug service events such as insertion and removal. The HPC driver is also responsible for registering the controllers slots with the HPS module. Note that the driver for the controller must be provided with the information necessary to map its slots to a bus node and slot name. See section 3.10 for the issue in establishing this mapping. Hot-plug controller device drivers are implemented as ordinary device drivers. These drivers are not required to implement all the standard driver entry points. Once attached, the driver is invoked through the receptacle functions it registered with the hot-plug services module via the hpc_register_slot interface. The driver may also need to process interrupts for the underlying hot-plug controller and translate those interrupts into the invocation of slot event handlers in the nexus. It is expected that the OEMs or IHVs who provide hot plug controller hardware write the HPC driver for that platform hardware. (See section 4.2 for the details on the interfaces between HPC driver and the framework.) 3.1.5 PCI configurator module This is a kernel misc module to support PCI device configuration and resource allocation for hot plugged PCI adapters. The PCI nexus drivers depend on this module. This may use platform specific interfaces to implement the required functionality. The basic functionality provided by this module includes: . Enumeration of PCI devices on the hot plugged adapter. . PCI bus resource (bus number for pci-to-pci bridges, memory and i/o space) allocation for all the PCI devices on the hot plugged adapter. . Configuration of the PCI devices. i.e programming the configuration header of the PCI devices including pci-to-pci bridges. . Resource management of PCI bus resources when adapters are removed or added to the system. . Creation of devinfo nodes for the PCI devices on the hot plugged card. The devinfo nodes created are compliant with the "PCI Bus Binding to IEEE 1275-1994 Boot Firmware" specification. . Uses Built-in Self Test (BIST) feature if the device supports it to verify the device function. The following are the limitations on the initial implementation of this module: . It uses the "available" and "bus-ranges" properties of the PCI bus node to allocate resources for the devices plugged into the bus. If there are no resources available to configure new device(s) then the hot plug operation returns failure. . Enumeration of other types of bridges (e.g pci-to-cardbus) are not supported in the initial implementation. This module can be considered as an extension to PCI nexus drivers. (Section 4.3 describes the interfaces exported by this module.) 3.1.6 PCI Nexus Driver PCI nexus drivers (pci and pci_to_pci nexi) are modified to support PCI Hot Plug functionality. The changes include: . Implementation of DEVCTL_* ioctl interfaces (including the interfaces defined in section 4.1) to support configure/unconfigure operations on the occupant. It depends on the PCI Configurator module to perform these configuration operations. . Creation of Attachment Point nodes for all the hot plug slots on the bus. It depends on the HPC driver to know which slots are hot pluggable slots on the bus. Note that the association of hot plug slot and the PCI bus instance are defined by the platform so the platform specific HPC driver provides this information. This information may come from the boot firmware as properties to the HPC devinfo node. . Communication with the HPC driver using the HPS framework to control the power for connect/disconnect operations, to get the status of the receptacle, etc. . For x86, the nexus need to translate the PCI interrupt pin to system IRQ. This is platform hardware dependent. At INITCHILD time the nexus needs to program the Interrupt Line field with the IRQ value. Note: It is assumed that the Power Management features specified by the PCI bus Power Management Interface Specification is not addressed in the context of PCI Hot Plug project. It is outside the scope of this project. 3.2 Device Driver requirements The requirements from the device drivers is the same as what was proposed in PSARC case 1998/075 (see the white paper at http://www.sun.com/developers/driver/wps/hotplug/hotplug.html). Basically, the requirements are: o PCI device drivers (including nexi) need to support D_HOTPLUG flag (PSARC 1998/075). This feature allows the device driver to support Hot Add and Hot Remove features of PCI Hot Plug. o Drivers need to be robust and hardened for recovering from hardware failures. Drivers must be able to detach from a device instance even when the device is not functional. 3.3 User Interface This project uses the cfgadm(1M) user command interface to manually perform the hot plug operations. But, some platforms may be able to support automated operations with the help of hardware support. This section discusses the details of performing hot plug operations. The error messages are reported on the console and system log file. 3.3.1 Hot Expansion The following are the sequence of steps followed by the user in performing the Hot Expansion: 1. User determines the empty slot for hot plugging the adapter. 2. User inserts the adapter in the empty PCI slot. Note: For PCI bus, a slot must be powered down and isolated from the bus before an adapter can be inserted. In Solaris, an empty PCI hot plug slot is always kept powered down and isolated from the bus so no slot preparation is needed to insert the card. 3. Adapter is powered up by the user using the command: cfgadm -c connect Note: On cPCI systems this step is not necessary because the hardware turns on the power and connects the adapter to the bus when inserted. 4. The user configures the adapter by using the command: cfgadm -c configure Note: For cPCI systems that support the Full Hot Swap feature or platforms that support the auto configure feature may not require this step. The user is notified of the result. If successful all the device functions of the adapter are available for use. The following is the sequence of events that occur during Hot Expansion operation: . For 'cfgadm -c connect' command the plug-in sends the command to the nexus driver. The PCI nexus will send the request to HPC driver to power up the slot and connect the slot to the bus. The HPC driver connects the slot to the bus and turns on the power LED. . For 'cfgadm -c configure' command, the plug-in sends the command to the nexus driver. The PCI nexus calls the PCI Configurator to probe and create the devinfo tree. If implemented, the FCODE interpreter is called to create the devinfo properties. The drivers for the child devices are also attached. If the operation failed then the nexus will turn on the fault LED on the slot. Note: Platforms (e.g Serengeti) that support the automatic configure feature may perform this operation automatically when the user inserts the card. i.e both connect and configure operations are done automatically. Auto configure option can be implemented if the hardware supports notification of the 'card inserted' event to the OS. 3.3.2 Hot Removal The following are the sequence of steps followed by the user in performing the Hot Removal: 1. The user determines the card that needs to be removed and makes sure that no applications are using the device functions of the card. 2. User unconfigures the card by using the command: cfgadm -c unconfigure The system notifies the caller of the result. If successful the user can continue the sequence. Otherwise, the user needs to correct the problem and retry this step. Note: Platforms (e.g Serengeti) that support the automatic unconfigure feature may perform this operation automatically when the user attempts to open the latch on the slot. This feature can be implemented if the platform hardware (i.e the HPC) supports notification of the event, 'card removal is attempted', to the OS. 3. User disconnects the slot from the bus using the command: cfgadm -c disconnect The system powers down the card and isolates the slot from the bus. The user is notified when it is safe to remove the card. The system will turn off the power LED when the disconnect operation is complete. Note: On cPCI systems this step is not necessary because the hardware allows the removal of the card when the latch is opened. 4. User removes the card from the slot. The following is the sequence of events that occur during Hot Removal operation: . For the 'cfgadm -c unconfigure' command, the plug-in sends the command to the nexus driver using the devctl_ap_unconfigure interface. The nexus walks down the devinfo tree and offlines all the devices in a bottom up fashion. It uses the PCI configurator to perform this operation. Basically it undoes the 'configure' operation. If it fails at any point during the operation (e.g fails to offline a device node) it aborts the operation and returns failure to the user. When successful, the devinfo tree that corresponds to the adapter is destroyed. . For the 'cfgadm -c disconnect' command, the plug-in verifies that the occupant is in the 'unconfigured' state and then sends the command to the nexus driver using devctl_ap_disconnect interface. The nexus sends the command to the HPC driver to power off the adapter and isolate the slot from the bus. The HPC driver, after isolating the slot from the bus, turns off the power LED to indicate that it is safe to remove the card from the slot. The operator can then remove the card from the slot. Note: Platforms that support the auto unconfigure feature, the nexus gets the 'card removal is attempted' event and performs the unconfigure operation as described above. If successful, it then performs the 'disconnect' operation as described above. 3.3.3 Behavior of slot LEDs The PCI Hot Plug specification supports the following LEDs which are controlled by the HPC hardware. Some platforms may support additional LEDs and operation of these additional LEDs is not specified here. (Green) Power LED: This is turned ON when the slot is connected to the bus for normal operation. It is turned off when the slot is disconnected from the bus. (Amber) Fault LED: This is used to indicate a fault condition on the card. This LED is turned ON when the hardware or software detects hardware failure. The cPCI systems may have an additional (Amber) LED to indicate removal-OK for removing the card from the slot. A specific platform may support a flashing feature for LEDs. If this feature is available then its use (i.e how the flashing is used) is defined by the platform specific software. 3.3.4 Automated Hot Plug operations The user interface model and the framework defined by this project is sufficient to support auto configuration/ unconfiguration of hot plugged cards to work if the platform supports it. For platforms that support generation of events for card-insertion and removal-attempted events, the OS can automatically configure/unconfigure the card without the need for cfgadm commands from the user. For example, on the Serengeti platform the hardware generates a 'latch open' event to indicate that the user wants to unplug the card. The nexus can try to unconfigure the card and disconnect the slot from the bus. If it is successful then it will indicate to the user (the second Amber LED is turned ON) that the card can be unplugged. If it fails to perform the operation then the fault LED is turned ON. Similarly when the user inserts the card (i.e the latch is closed) then the HPC generates an 'card inserted' event. The nexus will perform the connect and configure operations. 3.4 Boot System changes This project is not proposing any extensions to the boot system at this time except those noted in this section. The main issue that is addressed here is the PCI bus resources for PCI devices. The boot firmware (OBP on SPARC platforms and BIOS on x86 platforms) configures all PCI devices present at POST and the kernel doesn't change the bus configuration. With Hot Plugged PCI device the kernel need to configure devices and that requires bus resources. 3.4.1 OBP changes The kernel depends on the boot firmware to allocate sufficient resources to the bus bridges such that the nexus can find unused resources to configure new devices on the bus. A specific platform may determine the policy on preallocation of resources for hot plug slots based on the type of adapters that may be hot plugged into the system. Basically, the plaftform specific OBP needs to implement the above and pass the information in standard 1275 properties of the PCI bus nodes. 3.4.2 Devconf changes (x86) Currently the Devconf system is not creating the devinfo properties like "available", "bus-ranges", etc., for PCI bus nodes. It needs to be fixed to support the Hot Add feature. Also, currently for x86 platforms Generic Names are not used for PCI devices. The Devconf system need changes to support this such that Hot Upgrade would work correctly. Also, platform specific changes may be necessary to gather information about the available resources and pass that information as 1275 properties. Note that the Devconf changes are implemented as a separate project. 3.4.3 BIOS considerations The issue is same as the OBP requirements for supporting the Hot Add feature. It is expected that the BIOS on platforms that have PCI Hot Plug slots support bus resource allocation for dynamically adding PCI devices. This project assumes that there is BIOS support for supporting the Hot Add feature. Please note that the BIOS changes are platform specific and there is no standard specification for BIOS interfaces that provide the resource information. This may not be the case for ACPI based systems. 3.4.4 ACPI considerations At this time it is not clear what interfaces or features ACPI supports. The main difference between ACPI and non-ACPI systems w.r.t PCI Hot Plug is the access to hot plug controller hardware. Currently there is a project for ACPI (PSARC 1998/300) under development but this phase will not be addressing any framework support within the Solaris kernel. It is expected that the next phase of the PCI Hot Plug project will support both ACPI and non-ACPI systems. 3.5 FCODE interpreter The project may add an FCODE interpreter for SPARC platforms. Currently it is not determined how this will be integrated into the framework. It is expected that the PCI configurator module (Project Private) will use the FCODE interpreter to intepret the FCODE PROM on the card. 3.6 Requirements for Hot Swap on cPCI systems The proposed interfaces and the framework provided by this project is sufficient to support Hot Swap feature on cPCI systems. 3.7 Dependencies on other Solaris projects The project depends on the functionality provided by the following projects: PSARC 1997/202 Devfsadm(1M) PSARC 1998/275 Devfsadm(1M) Update For x86, the project depends on the following RFE's for the Devconf system. Devconf RFEs: 4164759, 4164755, 4164754, 4164753. 3.8 Documentation The project will provide the following Solaris documents: . Modified man pages: cfgadm(1M), config_admin(3X). . New document "PCI Hot Plug in Solaris - User's guide". . The marketing group may provide a white paper on the subject like, "PCI Hot Plug functionality in Solaris". . The project may need to provide a supplement to WDD for "Hot Plug Controller drivers". This is required for OEMs to write HPC drivers. . The project may need to provide man pages for HPC driver interfaces. This is also required for OEMs to write HPC drivers. Note that initially the HPC interfaces (section 4.2) are "Project Private" so these interfaces will not be documented in the standard Solaris documentation. 3.9 Assumptions and Limitations 3.9.1 Assumptions . The PCI bus inherently is not a fault-tolerant bus, so it is assumed that no hardware failures occur during the hot plug operation. Therefore the integrity of the system is not guaranteed for hardware failures. For example, a broken/faulty adapter when plugged in could cause the PCI bus hang and in turn could crash the Solaris system. . The PCI Hot Plug functionality has no direct impact on the HA software like Alternate Pathing. It is assumed that the testing of PCI Hot Plug functionality in the context of HA software is a requirement for specific platforms. The independent platform groups will be responsible for testing this functionality in the context of HA software. . It is assumed that the PCI adapters hot plugged into the system are compliant with version 2.1 (or later) PCI Bus spec and meet the requirements specified in the PCI Hot Plug Specification 1.0. . If the PCI adapter card has option ROM that contains code that is normally executed at power-on time to initialize the card then it is the responsibility of the adapter driver to use some other means to achieve the same result. For SPARC platforms if an FCODE interpreter is made available, then it may be possible to run the fcode. 3.9.2 Limitations . Currently there is no hardware support for testing a PCI adapter before it is configured into the Solaris system, so this project will not have any test requirements from within Solaris. But, if the card supports BIST feature then the PCI nexus may use it to verify the hardware. . Currently Solaris doesn't have any framework/driver support to dynamically reconfigure devices when bus resources may be reassigned. For the case of Hot Add, if the PCI bus node doesn't have enough resources required by adding a new device to the bus, then the operation will fail. To guarantee allocation of bus resources to a PCI device, the system must be capable of relocating the peer devices on the bus which requires support from device drivers. This is not possible with in the current framework. Therfore, to support Hot Add feature the kernel depends on the boot firmware to preallocate enough resources for the bus bridge such that the kernel can find unused resources for configuring new devices dynamically. . Currently in Solaris there is no simple way to find out what applications are using a particular device. This will be an issue when the operator tries to unconfigure the adapter and it fails because the device is busy. This is not an issue in the case of HA software like AP where a different device path is used before unconfiguring the broken device. This project will not try to resolve this issue or propose any new mechanisms in Solaris. 3.10 Issues For Discussion 1. Dynamic loading of HPS module and HPC driver module. Issue: For PCI Hot Plug, the bus nexus doesn't know whether there are any hot plug slots on the bus. So, it simply registers itself with the HPS module so that when the HPC driver is loaded it communicates the presence of hot plug hardware and the slot information. Now, the issue is that if the platform doesn't have hot plug hardware then the HPS module doesn't have to stay in memory. Other issue is w.r.t loading of HPC driver. Since there is no application directly accessing the HPC device, the driver doesn't get loaded automatically. 2. Need for PROM devinfo nodes for hot plugged devices? Issue: The current prtconf(1M) command has the option to dump PROM version of devinfo tree. The prtconf output for kernel version of devinfo tree doesn't include PROM defined properties so programs that parse prtconf output may not work correctly if PROM devinfo tree is not updated for hot plugged devices. If the PCI Hot Plug framework need to support prtconf then we may need to define new promif_* interfaces to keep track of changes to devinfo trees. Note: The current implementation of USB project doesn't use PROM support to update PROM version of the tree for hot plug events. 3. Association of hot plug slot with the bus nexus node. Issue: The problem is that the PCI bus nexus has no knowledge of which slots are hot pluggable slots. Since the hot plug controller hardware can be separate from the bus itself the association is only known to the hot plug controller hardware which is platform specific. The issue is how this association information is created and made available to the nexus. (see section 4.5 for the initial proposal.) 4. Support for partial configure/unconfigure operation. Issue: The cfgadm(1M) model supports operations only on Attachment Points. In practice, it may be necessary to configure/unconfigure individual device functions of an attachment point. Currently, there is no command level interface to support this. The following are some of the proposals: Option-1: Extend the devctl_ap_*configure interfaces to take an additional argument which specifies the path to the device function on which the configure/ unconfigure operation is to be applied. Option-2: SCSI Hot Plug project is proposing an extension (fast-track PSARC 1998/423) to the cfgadm(1M) command interface to add sub_ap_id concept where a specific platform can define sub_ap_ids which can specify individual devices of the attachment point. This requires changes to the plug-in library to support this feature. Option-3: There is another proposal (PSARC 1998/451) to add a generic command, devctl, to Solaris that will directly talk to the nexus to peform devctl command operations on a device or bus. 5. Need a DDI interface for device drivers to notify hardware failures to the framework. Issue: The hot plug platforms support fault LED which can be used to indicate hardware failures on the device. Device drivers when they detect hardware errors can notify the system such that the system (nexus) may turn on fault LED on the slot if it is available. This is useful to get attention of an operator. 6. Documentation of "-x" option extensions to cfgadm(1M). Issue: The extensions proposed in section 4.4.3 need to be documented. It is not clear where these changes are to be documented. cfgadm(1M) man page? 4.0 Project Interfaces The following is the table of new interfaces exported by this project: Interface Level Comments +-------------------------------------------------------------------------+ | Libdevice.so.1 Interfaces (user level) | +-------------------------------------------------------------------------+ | devctl_ap_acquire | Evolving | Acquire access to attachment Point.| | devctl_ap_connect | Evolving | Perform a CONNECT operation on | | | | the attachment point. | | devctl_ap_disconnect | Evolving | Perform a DISCONNECT operation | | | | on the attachment point. | | devctl_ap_configure | Evolving | Perform a CONFIGURE operation on | | | | the attachment point. | | devctl_ap_unconfigure | Evolving | Perform an UNCONFIGURE operation | | | | on the attachment point. | | devctl_ap_insert | Evolving | Perform an INSERT operation on | | | | the attachment point. | | devctl_ap_remove | Evolving | Perform a REMOVE operation on | | | | the attachment point. | | devctl_ap_getstate | Evolving | Obtain the current state of an | | | | attachment point. | | devctl_ap_state_t | Evolving | State of attachment point. | |-------------------------------------------------------------------------| | Bus Nexus IOCTL Interfaces | |-------------------------------------------------------------------------| | DEVCTL_AP_ACQUIRE | Sun Private | ACQUIRE attachment point. | | DEVCTL_AP_CONNECT | Sun Private | CONNECT attachment point. | | DEVCTL_AP_DISCONNECT | Sun Private | DISCONNECT attachment point. | | DEVCTL_AP_INSERT | Sun Private | INSERT attachment point. | | DEVCTL_AP_REMOVE | Sun Private | REMOVE attachment point. | | DEVCTL_AP_CONFIGURE | Sun Private | CONFIGURE attachment point. | | DEVCTL_AP_UNCONFIGURE | Sun Private | UNCONFIGURE attachment point. | | DEVCTL_AP_GETSTATE | Sun Private | Get state of attachment point. | | DEVCTL_AP_CONTROL | Sun Private | HPC specific control commands. | |-------------------------------------------------------------------------| | NDI Interfaces | |-------------------------------------------------------------------------| | ndi_dc_return_state | Sun Private | Copy the attachment state info. | |-------------------------------------------------------------------------| | Hot Plug Controller Interfaces (HPCI) | |-------------------------------------------------------------------------| | hpc_register_bus | Project Private | Register the pci bus | | | | node. | | hpc_unregister_bus | Project Private | Unregister the pci | | | | bus node. | | hpc_install_event_handler| Project Private | Install an event | | | | handler of the nexus.| | hpc_remove_event_handler | Project Private | Remove an event | | | | handler of the nexus.| | hpc_connect | Project Private | Connect the slot to | | | | the bus. | | hpc_disconnect | Project Private | Disconnect the slot | | | | from the bus. | | hpc_insert | Project Private | Prepare the slot for | | | | card insertion. | | hpc_remove | Project Private | Prepare the slot for | | | | card removal. | | hpc_control | Project Private | HW specific control | | | | operations. | | hpc_register_slot | Project Private | Register the hot plug| | | | slot. | | hpc_unregister_slot | Project Private | Unregister the hot | | | | plug slot. | | hpc_event_notify | Project Private | Notify the event | | | | occurrence. | | hpc_alloc_slot_ops | Project Private | Allocate slot_ops | | | | structure. | | hpc_free_slot_ops | Project Private | Free slot_ops | | | | structure. | | hpc_slot_ops_t | Project Private | HPC ops structure | | hpc_slot_info_t | Project Private | Slot info type. | | hpc_pci_slotinfo_t | Project Private | PCI slot information | | | | structure. | |-------------------------------------------------------------------------| | PCI Resource Allocator and Configurator (PRAC) Interfaces | |-------------------------------------------------------------------------| | prac_setup | Project Private | Setup allocator for the | | | | bus instance. | | prac_teardown | Project Private | Teardown allocator for | | | | the bus instance. | | prac_configure | Project Private | Create the devinfo treed| | | | and configure the devices| | prac_unconfigure | Project Private | Remove the devinfo sub | | | | tree. | | prac_alloc_space | Project Private | Allocate space from I/O | | | | or memory. | | prac_free_space | Project Private | Free space to I/O | | | | or memory. | | prac_alloc_bus_number | Project Private | Allocate new bus number | | prac_free_bus_number | Project Private | Free bus back to pool | | prac_handle_t | Project Private | PRA handle for the bus. | +-------------------------------------------------------------------------+ 4.1 Interfaces to Attachment Points The library libdevice provides a common set of interfaces for performing hot-plug type operations such as onlining or configuring devices and busses. This section presents a proposal for extensions to the libdevice interface for supporting the concept of attachment points. The current commitment level for these interfaces is Sun Private. The proposed extensions would be Sun Private as well. The motivation for supporting attachment points through libdevice is to provide a common set of interfaces between cfgadm plug-in libraries and the underlying drivers which implement operations required by cfgadm. At the present these interfaces are implemented as ioctl commands that are specific to the underlying driver. These interface enhancements will be particularly useful to CS platforms such as Daktari and Serengeti which will provide hot-plug functionality on multiple bus types (i.e. PCI, Safari, FC-AL, etc...). The framework provided by cfgadm(1M) and config_admin(3X) outlines a basic set of operations on attachment points. These operations are: o connect - put the receptacle in a connected state. o disconnect - put the receptacle in disconnected state. o configure - configure the occupant to be usable by Solaris. o unconfigure - logically remove the occupant's resources from the system. o insert - perform operations required to allow manual insertion of an occupant. o remove - perform operations required to allow manual removal of an occupant. o get state - return the state of an attachment point. The libdevice enhancements for attachment points introduce a new interfaces for each of the operations listed above, as well as an interface for obtaining a handle to any attachment point. Aside from the new libdevice interfaces, the proposal also requires the following changes: o bus nexus IOCTL command words for attachment point operations. o ndi (nexus driver interface) - support function for attachment points. Most libdevice operations are implemented as IOCTL operations on the device or bus. New IOCTL command words are required for each of the operations listed in section 2. New ndi interfaces are required for supporting the implementation of some of these new IOCTL words. 4.1.1 Changes to libdevice Interfaces The user level changes for supporting attachment points consists of introducing a new libdevice interface for each of the attachment point operations listed in section 4.1, as well as an interface for obtaining a handle to an attachment point. 4.1.1.1 devctl_ap_acquire() The current implementation of libdevice provides separate interface for obtaining a handle to a device and bus. These interfaces are devctl_device_acquire and devctl_bus_acquire. Both interface return a handle represented by the type devctl_handle_t. The libdevice framework tags the handle to indicate whether it's a device or bus. Attachment points can be supported extending the interfaces to include a new interface for acquiring a handle to an attachment point. The parameters, return value and setting of the errno variable for this new interface are the same as the interfaces devctl_device_acquire and devctl_bus_acquire. Synopsis #include devctl_hdl_t devctl_ap_acquire(char *devfs_path, uint_t flags) Interface Level Evolving Description Acquire a handle to an attachment point. The handle can be released with devctl_release. Parameters devfs_path path name to attachment point flags acquire flags (only DC_EXCL supported) Return Value A handle to attachment point on success or NULL on failure. 4.1.1.2 devctl_ap_connect()/devctl_ap_disconnect() The following interfaces would be introduced to provide the connect and unconnect operations on an attachment point. Synopsis #include int devctl_ap_connect(devctl_hdl_t handle) int devctl_ap_disconnect(devctl_hdl_t handle) Interface Level Evolving Description Perform a connect (or disconnect) operation on the attachment point. Parameters handle handle to the attachment point to be connected (or disconnected) Return Value Returns zero on success or -1 on failure. 4.1.1.3 devctl_ap_configure()/devctl_ap_unconfigure() The following interfaces would be introduced to provide the configure and unconfigure operations on an attachment point. Synopsis #include int devctl_ap_configure(devctl_hdl_t handle) int devctl_ap_unconfigure(devctl_hdl_t handle) Interface Level Evolving Description Perform a configure (or unconfigure) operation on the attachment point. Parameters handle handle to the attachment point to be connected (or disconnected). Return Value Returns zero on success or -1 on failure. 4.1.1.4 devctl_ap_insert()/devctl_ap_remove() The following interfaces would be introduced to provide the insert and remove operations. Synopsis #include int devctl_ap_insert(devctl_hdl_t handle) int devctl_ap_remove(devctl_hdl_t handle) Interface Level Evolving Description Perform an insert (or remove) operation on the attachment point. Parameters handle handle to the attachment point to be connected (or disconnected). Return Value Returns zero on success or -1 on failure. 4.1.1.5 devctl_ap_getstate() The following interface would be introduced to provide the operation of obtaining the state of an attachment point. Synopsis #include int devctl_ap_getstate(devctl_hdl_t handle, devctl_ap_state_t *ap) Interface Level Evolving Description Obtain the current state of an attachment point. Parameters handle handle to the attachment point ap pointer to a structure to which the state is to be copied Return Value Returns zero on success or -1 on failure. 4.1.1.6 devctl_ap_state_t The type devctl_ap_state_t which will be added to devctl.h is used to represent the state of an attachment point. This structure is defined as follows: typedef struct { ap_rstate_t ap_rstate; ap_ostate_t ap_ostate; ap_condition_t ap_condition; time32_t ap_last_change; uint32_t ap_error_code; uchar_t ap_in_transition; } devctl_ap_state_t; The supporting types ap_rstate_t, ap_ostate_t and ap_condition_t are also defined in devctl.h as follows: typedef enum { AP_RSTATE_EMPTY, AP_RSTATE_DISCONNECT, AP_RSTATE_CONNECTED } ap_rstate_t; typedef enum { AP_OSTATE_UNCONFIGURED, AP_OSTATE_CONFIGURED } ap_ostate_t; typedef enum { AP_COND_OK, AP_COND, FAILING, AP_COND_FAILED, AP_COND_UNUSABLE, AP_COND_UNKNOWN } ap_condition_t; The field ap_rstate holds the receptacle state. The possible states for a receptacle are represented by the enumeration type ap_rstate_t. The field ap_ostate holds the occupant state. The possible states for an occupant are represented by the enumeration type ap_ostate_t. The field ap_condition indicates the condition of the attachment point. The possible conditions are represented by the enumeration type ap_condition_t. The value AP_COND_OK indicates that the attachment point is operational. The field ap_last_change indicates the time of the last attachment point operation (i.e. connect, disconnect, configure, unconfigure, etc...) on the attachment point. The field ap_error_code provides a mechanism for passing error codes from the driver supporting the attachment point to a user-level client such as a cfgadm plug-in library. These enhancements make no attempt to provide standard error codes. It is expected that the error codes will be a contract between the driver and user-level client. This field is only intended for providing a vehicle for those error codes to be passed. The field ap_in_transition is used to indicate that an attachment point operation is in progress on the specified attachment point. This proposal attempts to separate information about the attachment point into bus (or implementation) independent and dependent components. It is intended that the information returned through the devctl_ap_state_t type be independent. Dependent information such as the system label of the attachment point (i.e. slot name), type of slot (i.e. PCI, CompactPCI, Safari, etc...), type of occupant, etc... should be provided by device node properties of the bus supporting the attachment points or device node properties of the devices corresponding to the attachment point occupant. 4.1.2 Bus Nexus Driver ioctl Command Words The device and bus operations provided by the libdevice interfaces are implemented by ioctl calls into the driver responsible for that device or bus. A new ioctl word is required for each attachment point operation listed on section 4.1.1. These ioctl words will be defined in the include file devctl.h along with the existing ioctl command words for device and bus operations. These command words will use the devctl_iocdata struct used by existing ioctl command words for passing parameters. 4.1.2.1 DECTL_AP_CONNECT/DEVCTL_AP_DISCONNECT The ioctl command words DEVCTL_AP_CONNECT and DEVCTL_AP_DISCONNECT provide connect and disconnect operations on an attachment point device. Synopsis #include Interface Level Sun Private Description Perform a connect (or disconnect) operation on the attachment point. Parameters All information is passed via a devctl_iocdata struct. Return Value Zero on success or one of the following Errno codes on failure: EIO Failed to connect (or disconnect) the attachment point. ENODEV No such attachment point. EINVAL Attachment point is not in a valid state for this operation. ENOTSUP Operation not supported on this device. EBUSY An operation is in progress on this attachment point. 4.1.2.2 DEVCTL_AP_CONFIGURE/DEVCTL_AP_UNCONFIGURE The ioctl command words DEVCTL_AP_CONFIGURE and DEVCTL_AP_UNCONFIGURE provide configure and unconfigure operations on an attachment point device. Synopsis #include Interface Level Sun Private Description Perform a configure (or unconfigure) operation on the attachment point. Parameters All information is passed via a devctl_iocdata struct. Return Value Zero on success or one of the following Errno codes on failure: EIO Failed to configure (or unconfigure) the attachment point. ENODEV No such attachment point. EINVAL Attachment point is not in a valid state for this operation. ENOTSUP Operation not supported on this device. EBUSY An operation is in progress on this attachment point. 4.1.2.3 DEVCTL_AP_INSERT/DEVCTL_AP_REMOVE The ioctl command words DEVCTL_AP_INSERT and DEVCTL_AP_REMOVE provide configure and unconfigure operations on an attachment point device. Synopsis #include Interface Level Sun Private Description Perform an insert (or remove) operation on the attachment point. Parameters All information is passed via a devctl_iocdata struct. Return Value Zero on success or one of the following Errno codes on failure: EIO Attachment point operation failed.. ENODEV No such attachment point. EINVAL Attachment point is not in a valid state for this operation. ENOTSUP Operation not supported on this device. EBUSY An operation is in progress on this attachment point. EIO Failed to connect (or disconnect) the attachment point. ENODEV No such attachment point. EINVAL Attachment point is not in a valid state for this operation. ENOTSUP Operation not supported on this device. EBUSY An operation is in progress on this attachment point. 4.1.2.4 DEVCTL_AP_GETSTATE The ioctl command word DEVCTL_AP_GETSTATE obtains the state of an attachment point device. Synopsis #include Interface Level Sun Private Description Obtain the current state of an attachment point. The ret_state field of the devctl_iocdata struct passed is treated as a pointer to devctl_ap_state_t. The state information for that attachment point is copied to the location specified by that pointer. Parameters All information is passed via a devctl_iocdata struct. Return Value Zero on success or one of the following Errno codes on failure: ENODEV No such attachment point. ENOTSUP Operation not supported on this device. 4.1.2.5 DEVCTL_AP_CONTROL The ioctl command word DEVCTL_AP_CONTROL obtains the state of an attachment point device. Synopsis #include Interface Level Sun Private Description Send the hardware specific "-x" option command to the platform specific hot plug controller driver. Parameters All information is passed via a devctl_iocdata struct. Return Value Zero on success or one of the following Errno codes on failure: ENODEV No such attachment point. ENOTSUP Operation not supported on this device. 4.1.3 NDI Interfaces A new ndi routine is needed for supporting the attachment point get state operation. The purpose of this new routine is to provide the driver associated with the attachment point with a mechanism for copying the attachment point state (as represented by an devctl_ap_state_t) to user space. Synopsis #include int ndi_dc_return_state(devctl_ap_state_t *ap, struct devctl_iocdata *dcp) Interface Level Sun Private Description Copy the attachment point state specified the kernel virtual address ap into the user space address specified by the ret_state field of the devctl_iocdata struct pointed to by dcp. Parameters ap pointer to an devctl_ap_state_t in kernel space containing the state of the attachment point dcp pointer to a struct devctl_iocdata containing the parameters for a libdevice ioctl command. The ret_state field of this structure is assumed to contain the user space virtual address to which the attachment point state is to be copied. Return Value NDI_SUCCESS Operation was successful. NDI_FAILURE dcp is NULL NDI_FAULT Copy of state information for kernel to user space failed. 4.2 Hot Plug Controller Interfaces (HPCI) There are two sets of interfaces provided by the HPS module. The first set of interfaces are for the nexus driver and the second set of interfaces for the HPC driver. The nexus interfaces include registration of bus, hardware event handling and hot plug operations on the slot. The second set of interfaces consists of slot registration, notification of events and performing hot plug operations on the slot. Initially all these interfaces are at the 'Project Private' level. Basically, the HPS module is a generic module that allows bus-dependent communication services between the HPC driver and the bus nexus driver. The HPS module is expected to be used by other hot plug bus implementations, so the interfaces may be promoted to a higher commitment level in future. Also, the set of interfaces used by the HPC driver are expected to be promoted to the DDI level to allow OEMs and IHVs to write HPC drivers. o Nexus uses the following interfaces to the HPS framework: hpc_register_bus hpc_unregister_bus hpc_install_event_handler hpc_remove_event_handler hpc_connect hpc_disconnect hpc_insert hpc_remove hpc_control o A HPC driver uses the following interfaces to the HPS framework: hpc_register_slot hpc_unregister_slot hpc_event_notify hpc_alloc_slot_ops hpc_free_slot_ops hpc_slot_ops_t hpc_slot_info_t Note: hpc_pci_slotinfo_t type is PCI specific. 4.2.1 Registration Interfaces In order for a bus nexus to use the hot-plug services interfaces to operate on a slot of a hot-plug controller, the bus nexus driver must have registered its bus node with the hot-plug services module and the hot-plug controller driver must have registered that slot with the hot-plug services module. The registration model does not impose an order on registration of bus node or hot-plug slot, simply that both registrations must take place before the bus nexus can perform receptacle operations on the hot-plug slot. The registration model permits the system configuration of bus nodes and hot-plug to expand and contract dynamically after boot time. It also allows multiple hot-plug controllers to register slots with the same bus node, multiple bus nodes to share the same hot-plug controller. 4.2.1.1 hpc_register_bus() Synopsis #include int hpc_register_bus(dev_info_t *dip, void (* nexus_slot_state)(dev_info_t *dip, hpc_slot_t hdl, hpc_slot_info_t slot_info, int slot_state), uint_t flags) Interface Level Project Private Description Register the bus instance with the HPS module. The callback function nexus_slot_state is used to callback into the nexus whenever the framework adds hot plug slots to the bus or removes the hot plug slots from the bus. i.e when HPC driver registers/unregisters the slots, the nexus is notified by the framework. Parameters dip pointer to the device node for the bus to be registered. nexus_slot_state Callback function to be called for hot plug slots that are enabled for hot plugging. flags UNUSED. (it is provided for future expansion). The arguments to nexus_slot_state function: dip dev_info pointer of the bus node. hdl Handle to slot. This is to be used when nexus requests the framework to perform hot plug operations on the slot. slot_info Opaque pointer to bus specific slot information. slot_state HPC_SLOT_ONLINE (slot registered) HPC_SLOT_OFFLINE (slot unregistered) Return Values This interface returns zero if the registration is successful or one of the following error codes on failure: ERR_HPC_BUSDUPLICATE The bus node has already been registered. ERR_HPC_INVALID The argument are invalid. Context Called in the kernel or user context. Implementation Notes If there are any hot plug slots already registered for this bus instance (this is possible if the HPC driver is attached first) then the HPS framework will call the nexus_slot_state callback function for each slot that is already registered. Typically a bus nexus driver would register its device node with the hot-plug services module during its attach(9E) operation. 4.2.1.2 hpc_slot_info_t and hpc_pci_slotinfo_t The slot information structure is provided by the HPC driver when it registers a hot plug slot to the HPS framework. This information is passed on to the nexus driver by the framework. Synopsis #include typedef void *hpc_slot_info_t; typedef struct hpc_pci_slotinfo { uint_t device_number; char *slot_logical_name; } hpc_pci_slotinfo_t; Interface Level Project Private Description hpc_slot_info_t is defined as an opaque pointer that describes bus specific slot information. hpc_pci_slotinfo_t describes the slot specific to PCI bus slot. This is known only to the HPC driver and the bus nexus driver. The fields in hpc_pci_slotinfo_t are described as follows: device_number PCI device number for the slot. slot_logical_name Logical name for the slot which may be used as the minor node name for the attachment point. Note: The slot information structure is bus specific so it may be different for other hot plug busses (e.g Safari bus). 4.2.1.3 hpc_unregister_bus() Synopsis #include int hpc_unregister_bus(dev_info_t *dip) Interface Level Project Private Description Unregister the bus instance with the HPS module. Parameters dip pointer to the device node for the bus to be unregistered. Return Values This interface returns zero if the unregistration is successful or one of the following error codes on failure: ERR_HPC_BUSNOTREGISTERED The bus node isn't registered. ERR_HPC_INVALID The argument dip is NULL or does not point to a valid dev_info_t node. Context This is called from the kernel or user context. Implementation Notes The HPS framework will uninstall any event handlers setup by the bus nexus for slots on this bus. A bus nexus driver should unregister its device node during its detach(9E) operation. 4.2.1.4 hpc_register_slot() Synopsis #include int hpc_register_slot(char *bus, hpc_slot_info_t slot, hpc_slot_t *handle, hpc_slot_ops_t *ops, uint_t flags) Interface Level Project Private Description Register a hot-plug slot with the HPS module. Parameters bus Fully qualified path name of the bus node associated with the hot-plug slot. slot Opaque pointer to bus specific slot information structure. handle Pointer to return an opaque handle for the slot which can be used in subsequent hot-plug services operations. ops Pointer to a structure (see section 4.2.2) containing addresses of the hot-plug driver functions providing the services for the corresponding hot-plug controller. flags UNUSED. (it is provided for future expansion). Return Values This interface returns zero if successful or one of the following error codes on failure: ERR_HPC_SLOTDUPLICATE The slot has already been registered. ERR_HPC_INVALID Invlaid arguments. Context This interface is called in the user or kernel context. Implementation Notes The HPS framework notifies the nexus of the registration of new hot plug slot if the nexus has registered the bus with the framework. If the nexus hasn't already registered (this can happen if the HPC driver is attached first before the bus nexus is attached) then the HPS framework does this when the nexus registers the bus. The driver for a hot-plug controller would typically register its known slots during the attach(9E) operation. This interface also allows a driver to register as they become available. Note that this interface permits a different ops vector to be registered for each slot. Also, the driver will use hpc_alloc_slot_ops() interface to allocate the slot ops structure. 4.2.1.5 hpc_unregister_slot() Synopsis #include int hpc_unregister_slot(hpc_slot_t *handle) Interface Level Project Private Description Unregister the slot with the HPS framework. Parameters handle pointer to the slot handle. Return Values This interface returns zero if the unregistration is successful or one of the following error codes on failure: ERR_HPC_SLOTNOTREGISTERED The slot isn't registered. ERR_HPC_INVALID Invalid argument. Implementation Notes The HPS framework should notify the nexus about the unregistration of slot. This enables the nexus to free up any data associated with the slot and remove the minor node for the attachment point, etc. Typically a hot-plug controller driver would unregister all known slots during its detach(9E) operation. A hot-plug controller driver should also unregister a slot once it's no longer available. 4.2.2 hpc_slot_ops_t - Hot Plug Controller Slot Operations The type hpc_slot_ops_t holds the set of hot-plug controller driver functions that implement the hot-plug services for a set of slots of that controller. This type is a struct containing fields that are private to the framework as well as the fields which represent the hot-plug services: Synopsis #include typedef struct hpc_slot_ops { int hpc_version; void *hpc_private_data; int (*hpc_op_connect)(dev_info_t *dip, hpc_slot_t slot, void *data, uint_t flags); int (*hpc_op_disconnect)(dev_info_t *dip, hpc_slot_t slot, void *data, uint_t flags); int (*hpc_op_insert)(dev_info_t *dip, hpc_slot_t slot, void *data, uint_t flags); int (*hpc_op_remove)(dev_info_t *dip, hpc_slot_t slot, void *data, uint_t flags); int (*hpc_op_control)(dev_info_t *dip, hpc_slot_t slot, int request, caddr_t arg); } hpc_slot_ops_t; Interface Level Project Private Description The following is a description of the fields in the hpc_slot_ops structure. If the field value is NULL then the HPC driver doesn't support that specific function. hpc_version: Version number of this data structure. hpc_private_data: An opaque pointer to a data structure maintained by the framework. hpc_op_connect: The hot-plug controller driver function that implements a connect operation on the slot. hpc_op_disconnect: The hot-plug controller driver function that implements a disconnect operation on the slot. hpc_op_insert: The hot-plug controller driver function to prepare a slot for an insertion. hpc_op_remove: The hot-plug controller driver function to prepare a slot for a removal. hpc_op_control: The hpc_op_control field provides a mechanism for supporting hardware specific cfgadm(1M) functions. These hardware specific functions are specified by the "-x" option to cfgadm(1M). The argument 'request' should be used to specify the specific operation, and the argument 'arg' should be used to pass the data for that operation. Parameters dip dev_info pointer of the requesting driver. slot handle to the slot on which operation is be performed. data UNUSED. (provided for future expansion) flags UNUSED. (provided for future expansion) Return Values These service functions return zero on success or one of the ERR_HPC_ error codes on failure. Context All the hot plug operations defined above are performed from the kernel or user context. 4.2.3 hpc_alloc_slot_ops()/hpc_free_slot_ops() Synopsis #include #include hpc_slot_ops_t *hpc_alloc_slot_ops(int flag); void hpc_free_slot_ops(hpc_slot_ops *ops); Interface Level Project Private Description The hpc_alloc_slot_ops() allocates the hpc_slot_ops structure. Parameters flag KM_SLEEP or KM_NOSLEEP to indicate if the caller can wait if no resources are available. ops pointer to hpc_slot_ops structure to be freed. Context Called from the kernel or user context. 4.2.4 Slot Hot Plug Event Interfaces The hot-plug services module allows a bus nexus driver to register a function to be invoked in the case of various hot-plug controller events. The hot-plug services module provides a bus nexus driver with interfaces to install and remove handlers for hot-plug controller events. The services module also provides hot-plug controller device drivers with an interface for informing the hot-plug services module of the occurrence of a hot-plug controller event. 4.2.4.1 hpc_install_event_handler() Synopsis #include int hpc_install_event_handler( hpc_slot_t handle, uint_t event_mask, int (*event_handler)(caddr_t, int), caddr_t arg); Interface Level Project Private Description Register an event handler for the hardware events on the specified slot. The HPS framework calls the event_handler whenever the framework is notified of the hardware events that are of interest to the nexus. The first argument to the event handler is the 'arg' itself and the second argument is the event number that occurred. The event handler may return the value HPC_EVENT_CLAIMED or HPC_EVENT_UNCLAIMED. This interface may be called by the nexus multiple times. The HPS framework keeps the event mask only for the last call to this interface. This would be useful if the nexus need to block some or all events temporarily and re-enable them. Also, the framework updates the event_handler address in case the nexus specifies a different handler. Parameters handle Handle to the slot. event_mask Bit mask that specifies the events that the nexus driver is interested for this slot. If the value is -1 then all events are to be notified. event_handler Event handler to be called for hardware events specified. arg Argument that is to be passed to the event handler. flags UNUSED. (provided for future expansion) Return Values The interface hpc_install_event_handler returns zero if successful or one of the following error codes on failure: ERR_HPC_INVALID For invalid arguments. Context The hpc_install_event_handler interface is called from the kernel or user context. Implementation Notes The event handler is called at interrupt context so the nexus may need to do minimum processing without blocking and schedule a lower level software interrupt context or finish the processing from a kernel context. 4.2.4.2 hpc_remove_event_handler() Synopsis #include int hpc_remove_event_handler(hpc_slot_t handle) Interface Level Project Private Description Remove the callback for all the hardware events on the slot. Parameters handle Handle to the slot. Return Values The interface hpc_remove_event_handler returns zero if successful or one of the following error codes on failure: ERR_HPC_INVALID For invalid argument. Context The interface is called from the kernel or user context. 4.2.4.3 Hot Plug Slot Events for PCI The hot plug events are generated by the HPC driver. The following events are defined for PCI Hot Plug slots. These are defined as single bit fields in the hpcsvc.h. HPC_SLOT_INSERTION A card was inserted into the slot. HPC_SLOT_REMOVAL A card was removed from the slot. HPC_SLOT_POWER_ON Power has been applied to the slot. HPC_SLOT_POWER_OFF Power to the slot has been removed. HPC_SLOT_LATCH_SHUT The slot's latch was placed in the shut position. HPC_SLOT_LATCH_OPEN The slot's latch was placed in the open position. The platform specific Hot Plug controller may not support all of the events. Synopsis #include #define HPC_SLOT_INSERTION 0x00000001 #define HPC_SLOT_REMOVAL 0x00000002 #define HPC_SLOT_POWER_ON 0x00000004 #define HPC_SLOT_POWER_OFF 0x00000008 #define HPC_SLOT_LATCH_SHUT 0x00000010 #define HPC_SLOT_LATCH_OPEN 0x00000020 Interface Level Project Private 4.2.4.4 hpc_event_notify() Synopsis #include int hpc_event_notify(hpc_slot_t handle, uint_t event, uint_t flags) Interface Level Project Private Description Called by the HPC driver to notify the occurrence of a hardware event to the framework. The HPS framework would call the nexus event handler(s) if it was requested by the nexus. Parameters handle Handle to the slot. event Event number for the hardware event. flags UNUSED. (provided for future expansion) Return Values The interface hpc_event_notify returns one of the following: HPC_EVENT_CLAIMED Event has been claimed by the nexus. HPC_EVENT_UNCLAIMED Event is not claimed by the nexus. ERR_HPC_INVALID For invalid argument. Context The interface is called from the interrupt context only. Implementation Notes The return value may be ignored by the HPC driver, but a specific platform driver may use the return value (i.e EVENT_CLAIMED or EVENT_UNCLIAMED) to take a specific action. The HPC driver cannot make any assumptions about the state of the occupant upon return from this interface call. 4.2.5 Hot-Plug Services Interfaces The hot-plug services interfaces allow a bus nexus driver to perform a basic set of hot-plug operations on a hot-plug controller. All hot-plug services interfaces are callable from kernel or user context only. 4.2.5.1 hpc_connect()/hpc_disconnect() Synopsis #include int hpc_connect(hpc_slot_t handle, void *data, uint_t flags); int hpc_disconnect(hpc_slot_t handle, void *data, uint_t flags); Interface Level Project Private Description hpc_connect() performs the CONNECT operation on the specified hot plug slot. hpc_disconnect() performs the DISCONNECT operation on the specified hot plug slot. Parameters handle Handle to the slot. data UNUSED. (provided for future use) flags UNUSED. (provided for future use) Return Values These interfaces return zero if successful or one of the following error codes on failure: ERR_HPC_INVALID One or more of the arguments is invalid. ERR_HPC_NOTSUPPORTED The hot-plug controller does not support this operation. ERR_HPC_FAILED The hot-plug controller operation failed. Context Called from kernel or user context. Implementation Notes None. 4.2.5.2 hpc_insert()/hpc_remove() Synopsis #include int hpc_insert(hpc_slot_t handle, void *data, uint_t flags); int hpc_remove(hpc_slot_t handle, void *data, uint_t flags); Interface Level Project Private Description hpc_insert() performs the INSERT operation (i.e preparing the slot for card insertion) on the specified hot plug slot. hpc_remove() performs the REMOVE operation (i.e preparing the slot for card removal) on the specified hot plug slot. Parameters handle Handle to the slot. data UNUSED. (provided for future use) flags UNUSED. (provided for future use) Return Values These interfaces return zero if successful or one of the following error codes on failure: ERR_HPC_INVALID One or more of the arguments is invalid. ERR_HPC_NOTSUPPORTED The hot-plug controller does not support this operation. ERR_HPC_FAILED The hot-plug controller operation failed. Context Called from kernel or user context. Implementation Notes None. 4.2.5.3 hpc_control() Synopsis #include int hpc_control(hpc_slot_t handle, int request, caddr_t arg); Interface Level Project Private Description hpc_control() provides a way for the bus nexus driver to invoke hardware specific operation on the hot plug controller. Examples of hardware specific operations that could be provided by this interface might include operations for LED control on hot-plug slots. Parameters handle Handle to the slot. request hot-plug controller driver specific command to be performed on the slot. arg provides the argument for the command. Return Values These interfaces return zero if successful or one of the following error codes on failure: ERR_HPC_INVALID One or more of the arguments is invalid. ERR_HPC_NOTSUPPORTED The hot-plug controller does not support this operation. ERR_HPC_FAILED The hot-plug controller operation failed. Context Called from kernel or user context. Implementation Notes None. 4.3 PCI Resource Allocator and Configurator interfaces This section describes the PCI Resource Allocator and Configurator (PRAC) interfaces. As described in section 3.1.5 the PRAC allocates pci bus resources, programs the configuration headers of devices and creates the devinfo tree for the hot plugged adapter. The interfaces defined in this section are committed to 'Project Private' level. 4.3.1 prac_setup() Synopsis #include int prac_setup (dev_info_t *dip, prac_handle_t *handle); Interface Level Project Private Description This is called when the PCI nexus is attached. It associates the bus instance with a handle which is passed as a parameter in further PCI resource allocator operations. This routine will allocate and fill in the handle structure. Parameters dip Pointer to the dev_info node of the bus nexus. handle Pointer to the PRAC access handle. Return values: PRAC_SUCCESS - Allocation of the handle was successful. PRAC_FAILURE - Allocation of the handle failed. Context prac_setup() can be called from user or kernel context. Implementation Notes This function also sets up internal data structures for PCI bus resource management of bus resources for this node. 4.3.2 prac_teardown() Synopsis #include int prac_teardown (prac_handle_t *handle, uint_t flags); Interface Level Project Private Description De-allocates the resources associated with the handle. This is called when the configurator will no longer be used for a bus nexus instance. Parameters handle Pointer to the PRAC access handle. flags 1 - if the bus node is going away. 0 - otherwise Return values: PRAC_SUCCESS - Memory associated with the resource allocator handle is returned. PRAC_FAILURE - If the de-allocation of memory fails. Context prac_teardown() can be called from any context. Implementation Notes If the bus node itself is going away, any data structure maintained by the PRAC module for bus resource management are freed and the PCI bus resources used by the bus itself are putback back into its parent pci bus node. Typically this happens when a pci-to-pci bridge is going away and the resources used by the pci-to-pci bus instance are putback into its parent bus. The implicit assumption here is that PRAC manages bus resources for each pci bus node involved in the hot plug operation. 4.3.3 prac_configure() Synopsis #include int prac_configure (prac_handle_t handle, uint_t bus, uint_t device ); Interface Level Project Private Description This will probe and create a devinfo tree for the PCI devices associated with the slot ('device' parameter). It allocates bus resources and programs the configuration headers of the device functions. Only the pci-to-pci bridges are enumerated. When it returns successfully the devinfo tree is created and the drivers are bound to the device nodes. Any failures in attach(9E) of leaf drivers are ignored. Parameters handle Pointer to the PRAC access handle. device PCI device number for the slot. Return values: PRAC_SUCCESS All devices under the specified slot will have a device tree node created for them. All information will be filled into the device tree node. Finally, devices will have their base address registers programmed. PRAC_FAILURE There was a problem at some level of the device tree creation. Any nodes created will be destroyed and the device tree will be left like it was found. No devices will be left partially configured. The specific failure will be logged on the console. Implementation Notes This routine is called to create kernel device tree nodes that represent devices on the specified bus, in the specified slot. This includes building device nodes for single devices as well as multi-function. It will make recursive calls down the device heirarchy when PCI-to-PCI bridges are discovered. As it recurses down the device hierarchy, it determines the memory requirements for each device and records that in the "reg" property of the corresponding device node. 4.3.4 prac_unconfigure() Synopsis: #include int prac_unconfigure(prac_handle_t handle, uint_t device); Interface Level Project Private Description: This performs the opposite of the "prac_configure()" operation. It will remove the kernel device tree nodes associated with the slot (i.e 'device' arg). It assumes that all the device nodes for the slot are in OFFLINE state (i.e no driver is attached). Parameters handle Pointer to the PRAC access handle. device PCI device number for the slot. Return values: PRAC_SUCCESS - The whole tree (from the specified starting point) is removed. PRAC_FAILURE - If any failure to free the resources, or if any devices are not already in the OFFLINE state. Context: User or kernel. 4.3.5 prac_alloc_space() Synopsis #include #include caddr_t prac_alloc_space(dev_info_t *dip, uint_t space, uint_t type, uint32_t size, uint32_t align) Interface Level Project Private Description: Allocate I/O or memory space from a common pool which is associated with the "dip". Internally a pool of memory will be setup from the "available" property when this routine is first called on behalf of a particular "dip". This pool will use a first fit algorithm to minimize fragmentation. Parameters dip Devinfo node pointer to the bus node from which resource is to be allocated. space PCI_BASE_SPACE_MEM - PCI memory space PCI_BASE_SPACE_IO - PCI I/O space type For memory space the type is one of the following: PCI_BASE_TYPE_MEM - 32-bit memory address PCI_BASE_TYPE_LOW - less than 1Mb address PCI_BASE_TYPE_ALL - 64-bit memory address size Size of the space requested. align Address alignement for the space requested. Return values: Returns the address of the memory or I/O space if the request can be satisfied. Otherwise, it returns NULL. Context: User or kernel context. 4.3.6 prac_free_space() Synopsis #include #include void prac_free_space(dev_info_t *dip, caddr_t addr, uint_t space, uint32_t size) Interface Level Project Private Description: Free the allocated I/O or memory. This is the complement to the (above) allocation routine. Parameters dip Devinfo node pointer to the bus node from which resource was allocated. addr Address of the space to be freed. space PCI_BASE_SPACE_MEM - PCI memory space PCI_BASE_SPACE_IO - PCI I/O space size Size of the space to be freed. Return value: (none) Context: User or kernel. 4.3.7 prac_alloc_bus_number() Synopsis: #include prac_alloc_bus_number(dev_info_t *dip) Interface Level Project Private Description: Allocate the lowest bus number available in the PCI domain associated with "dip". Parameters dip Devinfo node pointer to the bus node from which resource is to be allocated. Return value: This interface returns -1 if no bus-number is available. Otherwise it returns a valid bus-number. Context: User or kernel. 4.3.8 prac_free_bus_number() Synopsis: #include void prac_free_bus_number(dev_info_t *dip, int bus_number) Interface Level Project Private Description: Put back the specified bus number into the pool associated with "dip". Parameters dip Devinfo node pointer of the bus from where the resource was allocated. bus_number bus number to be freed. Context: User or kernel. 4.4 cfgadm(1M) extensions 4.4.1 Attachment Point Type for PCI slots The existing cfgadm command uses the driver name (e.g sysctrl) to find the plug-in library (e.g libsysctrl.so.1). For PCI Hot Plug we expect to have a single plug-in library that is platform independent but the driver that exports the attachment points may be different (e.g pci, pci_to_pci). So, we propose to extend the mechanism in cfgadm to locate the correct plug-in library by providing additional information in the attachment point node. We propose to add a new node type that include class information. For example the node type for PCI Hot Plug APs: DDI_NT_PCI_ATTACHMENT_POINT == "ddi_ctl:attachment_point:hp_pci" And we propose that the cfgadm locates the plug-in library in the following sequence: . Uses the node name (e.g SUNW,envctrl) to find the library libSUNWenvctrl.so.1. If it fails to find this library then the search continues with driver name. . Uses the driver name (e.g sysctrl) to find the library libsysctrl.so.1. If this fails then it tries with class information if it is available in the AP node type. . If the AP node type has class information (3rd component in the node type string), e.g hp_pci, then it looks for the libhp_pci.so.1 library. . Otherwise failure is reported to the user. The above proposal would allow platform specific library to be located first before looking for class specific library. NOTE: A similar proposal is being made by the SCSI Hot Plug project team. 4.4.2 Logical AP ID Specification The logical AP ID is the logical name used to identify a specific Attachment Point. In the current cfgadm(1M) model the logical name of an attachment point is in the form: : where 'instance' is the instance number of the driver that is attached to the attachment point. 'driver' is the name of the driver. And 'minorname' is the minor node name of the attachement point. The same mechanism will be used to name the PCI Hot Plug slots. The 'minorname' of the attachment point is defined by the platform so it can be descriptive enough to uniquely identify a specific slot. So, the HPC driver will define the minor name for the attachment point based on the location of the slot in the system. 4.4.3 Extensions to "-x" hardware functions The following are the options to control platform hardware specific features. [ -x enable_slot | disable_slot ] This command changes the state of the slot and preserves the state of slot across reboot. This is useful to disable slots that have hardware problems. [ -x enable_autoconfig | disable_autoconfig ] This command enables or disables the ability to autoconfigure the occupant of the slot. This command is useful for the user to to have the control if the platform supports auto configuration. [ -x led [ fault | power | active | attn ] mode [ on | off ] ] This command allows the user to set the mode of a specific led for a slot. Note: changing the state of the led does not change the state of the receptacle or occupant. Normally, the leds are controlled by the hotplug controller, no user intervention is necessary. See section 3.3.3 for the LED behavior. 4.4.4 Behavior of "-v" option When "-v" is used with "-l" option the cfgadm outputs information about the attachment point. For PCI Hot Plug, the "Information" field will be the slot's system label. This string will be obtained from the "slot-name" property of the slot's bus node. The occupant "Type" field will describe the contents of the slot. There are 3 possible values: NULL The slot is empty multi-function The card in the slot is a multi-function card. , The card in the slot is a single-function device. is a string representing class code of the device (i.e. "storage", "network", etc...). is a string representing the subclass code of the device (i.e. "SCSI", "ethernet", "token-ring", etc...). 4.5 Association of HPC slot with PCI bus instance Problem Description: Since the hot plug hardware is an extension of standard PCI bus, the association between the slot and the bus instance is only known to the hot plug controller hardware, which is platform specific. So, the issue is how this information is passed and used within the framework. Proposal: The project proposes that the HPC driver should establish the association of a slot with the respective bus nexus node. For x86, this information is passed in the .conf file. For SPARC, this information may come from the OBP as a property of the HPC node. The format of association is simply the pair: where is the PCI device number that corresponds to the slot and is simply the device path (e.g /pci@0,0) to the nexus node of that slot. This solution is assumed in defining the interface hpc_register_slot() in section 4.2. 4.6 OBP interfaces This project is not proposing any new interfaces, but the OBP for a specific platform should support allocation of bus resources as discussed in section 3.4.1. 4.7 Devconf interfaces For x86, the interrupt mapping information is needed when the PCI device configuration registers are programmed. The Interrupt Line field of the configuration header is programmed with the IRQ (an ISA style IRQ in the range 0-15). The project will use the solution proposed by the Devconf group (RFE id 4164754). The details of the interface is not available at this time. 5.0 Reference Documents 1. PCI Hot Plug Specification (Revision 1.0, October 6,1997) published by PCI SIG. 2. White paper on converting device drivers for PCI Hot Plug. http://www.sun.com/developers/driver/wps/hotplug/hotplug.html 3. CompactPCI Hot Swap Specification, PICMG 2.1 R1.0 published by PICMG.