PCI Hot Plug Support One Pager 1. Introduction 1.1. Project/Component Working Name: PCI Hot Plug Support 1.2. Name of Document Author/Supplier: Prasad Singamsetty 1.3. Date of This Document: 7/23/98 1.4. Name of Major Document Customer(s)/Consumer(s): Platform Steering Committee, PSARC, ONSC, SFSC 2. Project Summary 2.1 Project Description This project is to implement the PCI hot plug functionality in Solaris for the platforms that have PCI hot plug hardware. PCI Hot-Plug specification revision 1.0 from PCI SIG is a standard specification that defines the capabilities of hot plug hardware that can be implemented such that the standard PCI adapters can be hot plugged into a running system. Solaris 2.7 release has hot plug framework introduced by Sunfire DR and I/O Hot Plug projects. This framework needs some extensions to fully support PCI hot plug features. The basic capabilities of PCI hot plug functionality are: o Like-for-Like or Hot Replacement; this feature supports hot replacement of failing or failed adapter with an identical adapter without bringing down the system or requiring the applications and/or drivers to detach from the device. o Hot Expansion or Addition; this feature supports hot addition of PCI adapters to the system without bringing down the system. o Hot Upgrade; this feature supports the hot upgrade of adapters and drivers to next generation while the system is running. The goal of this project is to design and implement an architecture that supports hot plug capabilities for both PCI and CompactPCI based hardware platforms. Due to OEM requirements and time constraints the project may be done in multiple phases. o Phase-I: All three capabilities of PCI Hot Plug is supported with some limitations on Hot Expansion. For Like-for-Like support, a failing card is replaced with an exact duplicate while the driver stack is still configured. It requires the suspend and resume operation on the device. For the Hot-Add case, the operation may fail if there are not enough bus resources to configure the new card. The Hot Upgrade feature is really supported as Hot Remove followed by Hot Add operation. The cfgadm(1M) command is used to support administration of hot plug slots. o Phase-II: To support Hot-Add feature where the system is capable of moving PCI bus resource mappings allocated for other peer devices on the bus in order to satisfy the resource allocations for the new device. Other functionalities for this phase may include I2O Hot Plug support, FCODE interpreter and ACPI support. Also, a GUI model (e.g SyMON) will be supported in this phase. o Phase-III: To support automation of hot plug operations without the need for user commands. For example, when a new card is inserted the system should automatically configure it into the system and inform the user about it. This phase may require extensions to the existing event framework in Solaris kernel. 2.2 Assumptions o It is assumed that the GUI tool like SyMON is part of a overall Solaris strategy for console administration. Porting SyMON to Solaris is outside the scope of this project. o It is assumed that this project does not provide a complete solution for High Availability feature and it requires special software like Alternate Pathing to provide a complete solution. The scope of porting Alternate Pathing software to Solaris is outside the scope of this project. o It is assumed that "driver hardening" is a separate project (outside the scope of this project) to make Solaris drivers more robust when faulty adapters are plugged into the system. 3. Business Summary 3.1. Problem Area The Solaris hot plug framework in 2.7 needs extensions to support PCI Hot Plug functionality. This project is to enhance the framework and to support key SPARC and X86 platforms to provide the PCI hot plug functionality in Solaris. 3.2. Market/Requester It is a requirement from our OEMs that PCI Hot Plug functionality supported in Solaris. 3.3. Business Justification PCI Bus has become the de facto I/O bus on Sun SPARC and Intel-based servers and the Hot Plug capability is an important technology to provide RAS capabilities for these systems. It is a requirement of our OEMs that the PCI Hot Plug functionality is supported in Solaris. Our competitors in the Enterprise Operating System market have announced plans for PCI Hot Plug support and a limited functionality is already supported in their current products. Microsoft Windows NT 5.0, update release to SCO Unixware version 7 and the next release of Novell IntraNetWare will have full functionality support for PCI Hot Plug. It is very likely that their implementations are ahead of our implementation in Solaris. 3.4. Competitive Analysis Microsoft, SCO and Novell will have full PCI Hot Plug support including GUI tools to manage the hot plug slots in a major or updated release of their server-class operating systems. Compaq is shipping server hardware that has PCI Hot Plug hardware and is supported by all the major server OSes. Like-for-Like replacement feature support is considered the minimum functionality by these vendors. 3.5. Opportunity Window/Exposure It is expected that almost all Intel OEMs will be introducing server platforms that support PCI Hot Plug hardware and expect all the major OSs to support this functionality. This functionality provides simple means to support I/O hot plug capability on the standard PCI bus and to enhance the Availability feature on server platforms. It is important that Solaris support this technology as soon as possible to be competitive in the server market with other major OSs. 4. Technical Description This project basically uses the hot plug framework introduced by Sunfire DR and I/O Hot Plug projects to extend the support for PCI Hot Plug functionality. The Figure-1 below illustrates the cfgadm model used for supporting PCI Hot Plug functionality in Solaris. +-----------------+ | cfgadm | +-----------------+ | | v +-----------------+ | Hot Plug PCI | hardware specific | library | plug-in library +-----------------+ | | | | | | User =================|===========|=============================== | | Kernel | | +----------v---+ +--v----------+ +--------------+ | Hot Plug | | PCI Nexus | | | | controller | | driver <------> PCI bus | | driver | | | | configurator | +--------------+ +-------------+ +--------------+ | | /\ / \ v v ( Hot Plug capable drivers including HBAs) Figure 1: Block Diagram of PCI Hot Plug software The project requires changes in the following areas: (1) DDI framework extensions to support dynamic configuration of like-for-like feature. (2) Platform independent cfgadm plug-in library for PCI Hot Plug slots. (3) Bus nexus drivers to support D_HOTPLUG. In addition, PCI nexus drivers to support devctl interfaces for dynamic configuration of devices on hot plug slots. (4) Platform dependent PCI bus configurator and resource allocator kernel module to support dynamic configuration of PCI adapters. (5) Device drivers need to support D_HOTPLUG flag. (6) Device drivers including bus nexus drivers need to support suspend/resume interfaces for dynamic reconfiguration to support like-for-like replacement feature. (7) OBP support for SPARC platforms: The changes include devinfo properties for hot plug controller hardware and/or hot plug slots, platform specific bus resource allocation for hot plug slots, etc. (8) X86 devconf system: The changes include generic device name support for PCI devices, devinfo properties that describe available bus resources for PCI bus nodes, PCI interrupt routing information, etc. (9) There may be need to add an FCODE interpreter to provide device specific devinfo properties. This project will modify only selected key drivers for the supported platforms and document the porting effort needed to convert device drivers to support hot plug operation. The PCI Hot Plug support added by this project in Solaris provides foundation to support High Availability feature with additional software support like Alternate Pathing. Initial phase of this project would use the current cfgadm(1M) user interface for administration of PCI hot plug slots. The work done will allow intergration of the PCI Hot Plug system with a graphical user interface model like SyMON. 5. Reference Documents o PCI Bus Hot Plug Specification Revision 1.0 from PCI SIG. o Hot Swap Specification PICMG 2.1 R1.0 from PICMG. 6. Resources and Schedule 6.1. Projected Availability Second quarter of CY99. 6.2. Cost of Effort Estimated cost of the project is 3 staff years. 6.3. Cost of Capital Resources The hardware is currently available in house for development and testing on Intel platform. Additional hardware to be provided by Intel OEMs. SPARC platforms will be made available by the platform groups: Daktari, Serengeti, Wildcat and Starcat. 7. Prototype Availability 7.1. Prototype Availability Available in the fourth quarter of CY98. 7.2. Prototype Cost Estimated cost is about 8 staff months.