Template Version: @(#)onepager.txt 1.35 07/11/07 SMI Copyright 2008 Sun Microsystem 1. Introduction 1.1. Project/Component Working Name: Solaris Hotplug Framework 1.2. Name of Document Author/Supplier: Govinda Tatti (govinda.tatti@sun.com) 1.3. Date of This Document: 03/10/08 1.4. Name of Major Document Customer(s)/Consumer(s): 1.4.1. The PAC or CPT you expect to review your project: Solaris PAC 1.4.2. The ARC(s) you expect to review your project: PSARC 1.4.3. The Director/VP who is "Sponsoring" this project: Quresh Dhoon (quresh.dhoon@sun.com) 1.4.4. The name of your business unit: Solaris Platform IO Software 1.5. Email Aliases: 1.5.1. Responsible Manager: Richard Zatorski (richard.zatorski@sun.com) 1.5.2. Responsible Engineer: Govinda Tatti (govinda.tatti@sun.com) Scott Carter (scott.carter@sun.com) Wesley Shao (wesley.shao@sun.com) 1.5.3. Marketing Manager: Honglin Su (honglin.su@sun.com) Jeff McMeekin (jeff.mcmeekin@sun.com) 1.5.4. Interest List: ddi-hotplug-framework@sun.com 2. Project Summary 2.1. Project Description: The main goal of this project is to provide a generic common Solaris hotplug framework, a foundation which can support hotplug functionality for any hotpluggable bus. The key features include: o A state machine based, bus independent hotplug framework which will interact with other frameworks (such as PM, FMA and Devfs). It will manage the hotplug slot states, interact with bus specific hotplug controller modules and configuators (including kernel driven auto-configuration), deliver hotplug events through the Device Contract, LDI and Sysevent frameworks, and handle all user initiated hotplug operations. o Well defined, common interfaces to the physical and virtual hotplug controller drivers. This will simplify writing a new driver with hotplug controller functionality, because drivers can implement their hotplug requirements by calling into the new framework. o DDI hotplug interfaces for leaf device drivers to support features such as surprise removal, hot replacement, dynamic resource re-balance etc., and framework notification of any asynchronous events such as errors and brute force user operations. At the same time, the new hotplug framework will be compatible with the existing DDI compliant leaf device drivers. o A new resource allocator, which will manage bus resources through device tree and provides support for dynamic resource re-balance operations. o Generic hotplug management interfaces, so that various types of management applications, including GUIs, can be written to these interfaces. The project will deliver a new GUI based hotplug tool and may also modify cfgadm or deliver a new command line (CLI) tool. o A hotplug daemon to deliver hotplug events to all hotplug aware applications and userland modules through Device Contract and Sysevent frameworks. o Support for PCI (SHPC based) and PCI Express hotplug functionality under the new Solaris Hotplug Framework. In addition, a set of new features will be supported such as surprise removal, hot replacement, power fault, hotplug FMA, and hotplug operation on an individual device or function (in addition to the current hotplug support for slot or attachment point). Please note that the old or proprietary hotplug functionality will not be migrated to new hotplug framework. o Support for virtual hotplug under virtual environments like IOV, Ldoms, Xen etc. To support hotplug for virtualized devices requires a virtual hotplug controller driver. These virtual HPC drivers may use different interfaces or methods for accessing physical/IOV compliant devices (v/s para-virtualized (PV) devices). The project will be implemented and delivered in multiple phases due to the priority of some key features & time constraints. For clarification of the big picture, all phases are presented here. However, the scope of ARC review requested for this case is limited to the first phase. The full definition of other phases including resources and schedules are not known at this time. Subsequent ARC cases will be submitted when those project phases commence. o Phase-I Support for existing standard PCI, Native and ACPI based PCI Express hotplug functionality under the new Solaris Hotplug Framework along with several new key features such as: - New hotplug DDI interfaces for leaf drivers - New resource allocator interfaces - Kernel initiated auto-configuration - Hotplug events through device contract, LDI and sysevent frameworks - Dynamic resource re-balance - Surprise removal, hot replacement, hotplug FMA - Modified cfgadm or new userland admin tool (CLI) - Virtual hotplug support for non-IOV virtual devices (physical, para-virtualized (PV) devices) under virtual environments such as Ldoms, Xen. o Phase-II - Virtual hotplug support for IOV capable devices under virtual environments such as Ldoms, Xen. - Support for a GUI admin tool o Phase-III - Support for Express Cards 2.2. Risks and Assumptions: o While this project has the goal of replacing all current hotplug frameworks, it will be delivered in phases with PCI and PCI Express Hotplug as the focus for the first phase. The virtual hotplug support will be delivered either as part of first phase or in sub-sequent phase. The remaining work will follow provided the underlying IO technology has not been obsoleted. o The new DDI Hotplug framework will expose a small number of DDI interfaces to leaf drivers that want to be hotplug aware. Driver modifications will need to be done by driver owners to take advantage of the new hotplug features. 3. Business Summary 3.1. Problem Area: Solaris needs a generic common hotplug framework that can support the hotplug functionality for all industry standard PCI, PCI Express, IOV, Express Card, Cardbus and other virtual IO devices. 3.2. Market/Requester: Any customer using systems with PCI Express hotplug, DR, and IOV capabilities. This currently includes OPL, various SPARC and x64 Blades such as Andromeda, St.Paul, Montoya, Glendale and Wolf. 3.3. Business Justification: PCI Express has become the de facto I/O bus on Sun SPARC and x64 based servers. Full support for hotplug capability is important for providing RAS capabilities for these systems. Our OEM systems require full hotplug support for PCI Express, IOV capable and other Virtual IO devices. Our competitors in the enterprise OS market have announced or have plans for PCI Express and IOV hotplug support. Indeed, they already support limited functionality in their current products. Linux and Microsoft OSs either already have or will be releasing full hotplug functionality for PCI Express, IOV, and Express Card devices. It is very likely that their implementations are already ahead of our implementation in Solaris. 3.4. Competitive Analysis: Linux and Microsoft, in a major or updated release of their server-class operating systems, already support full hotplug functionality. This includes GUI tools to manage various hotplug capable bus slots. 3.5. Opportunity Window/Exposure: It is expected that almost all x64 OEMs will be introducing server and mobile platforms that support various derivatives of PCI Express hotplug hardware. Thus it is expected that all major OSs will support hotplug functionality, at least to the extent of supporting I/O hotplug capability on the standard PCIe bus and enhancing the Availability feature on server and mobile platforms. It is important that Solaris support this technology as soon as possible to be competitive in the server market with other major OSs. 3.6. How will you know when you are done?: Complete delivery of Solaris Hotplug Framework for PCI Express, IOV and Virtual hotplug capable platforms. 4. Technical Description: 4.1. Details: In Solaris today, every hotplug capable IO bus technology (e.g.PCI, USB, PCMCIA, Storage) has its own hotplug framework. Some of these frameworks are very old and have been built based on other even-older technologies. For example, the PCI hotplug framework was developed about 10 years back and was leveraged from IO DR and PM frameworks. Since then there have been no significant changes made to this PCI hotplug framework except the recent enhancements to support some features of PCI Express hotplug. In short, the existing hotplug frameworks, including their management applications (e.g. cfgadm), are not generic and modern enough to support full current hotplug functionality for all industry standard devices. Neither are they modern enough to address future hotplug requirements such as fast-boot support, hotplug support for Express Cards, or hotplug under virtual environments like IOV, Ldoms or Xen. The time has come to cease patching the existing frameworks to get the required support. This case proposes a new generic common Solaris hotplug framework that lays the foundation to address current and future hotplug requirements. +-------+ +----------+ +--------+ +------+ +-------+ +-------+ | GUI | | SHPC CLI | | Cfgadm | | PICL | | RCM | | Apps | +-------+ +----------+ +--------+ +------+ +---^---+ +---^---+ | | | | | | ----V----------V-----------V---------V---- ----V-^----------------V---- Management APIs | | Hotplug Events APIs +---V-----------------------------V---+ | Hotplug Library | +-------------------------------------+ | +-----------------V-------------------+ | Hotplug Daemon | +-------------------------------------+ | | +------V------+ +------V------+ | libdevice | | libcontract | +-------------+ +-------------+ | | User _______________________V_______________________V________________________ | | Kernel +---------+ DDI +-------------------+ +-------------+ | Leaf | I/Fs| Hotplug Framework | | Contract | | Drivers |<--->|(State Machine, | | Framework | +---------+ |Resource Allocator,| +-------------+ |Resource Rebalance,| | +---------+ |Auto-Configuration,| +------V-------+ |Cfgrator |<--->|Event Generation) |<->|IO Subsystem | +---------+ |Ex:pcicfg| +-------^-----------+ |(including |<--->| Kernel | +---------+ | |LDI framework)| | Modules | | +--------------+ +---------+ Hotplug Controller APIs | -----^---------------^----V---------^---------------^-------------^----- | | | | | +----V-----+ +----V----+ +----V-----+ +----V----+ +----V----+ | Virtual | | PCIe | | Express | | Cardbus | | Prop | | HPC Drv | | Module | | HPC Drv | | HPC Drv | | HPC Drv | +----------+ +---------+ +----------+ +---------+ +---------+ | IOV | |PCI/PCIe | | Exp Card | | Cardbus | | Prop | | Framework| | Nexus | | Nexus | | Nexus | | Driver | +----------+ +---------+ +----------+ +---------+ +---------+ Figure 1: Block diagram of new Solaris Hotplug Framework The goal of this project is to design and implement an architecture that supports the hotplug functionality on any industry standard IO bus technology, even though the primary motivation of this project is to support the hotplug functionality on all derivatives of PCI and PCI Express bus technologies. This new hotplug project will be based on a state machine model, providing various NDI, DDI and user interfaces to Hotplug controller drivers, DDI-complaint drivers and to management applications. This framework will also be interacting with other frameworks like FMA and PM to provide better error handling and power management features for the hotplug capable devices. It will also address dynamic resource allocation and other limitations which exist in the current hotplug frameworks. Figure 1 shows the block diagram of software components involved. The new Solaris hotplug framework will be based on the SHPC (Standard Hot Plug Controller) usage model. It provides a set of generic and common hotplug interfaces, which includes: o Management Interfaces Generic or bus specific hotplug admin applications including GUIs, can be written using the newly defined management interfaces provided by the hotplug library. o Hotplug Event Interfaces Hotplug-aware user applications or userland modules can register and receive the hotplug events using the newly defined interfaces provided by the hotplug library/daemon. o Device Driver Interfaces Device drivers can work with the hotplug framework to participate in hotplug activities such as surprise removal, hot replacement, dynamic resource re-balance, error handling etc., using the newly defined DDI interfaces. o Hotplug Controller Driver Interfaces Hotplug controller drivers can be written easily for any hotpluggable bus (physical or virtual) using the newly defined bus_op and NDI interfaces. Please refer to [1] for more technical details of this project. 4.2. Bug/RFE Number(s): This project will fix the following existing RFE/Bugs: R6651339 P3 pcicfg does not ensure that the desired amount of IO/Mem space is available R6555777 P3 source code duplication for pcicfg on x86 and sparc platform 6561239 P3 Configuration Retry Mechanism support is necessary to support hot plugging of some PCIe cards 6559506 P4 Need workaround for PLX 8532 errata #62 4.3. In Scope: Refer to section 2.1 4.4. Out of Scope: o UEFI support o Support for power budgeting o Enhancements or modifications to existing DR framework o Obsoleting RCM and EFCode frameworks 4.5. Interfaces: Please refer to [1] for all details about new and modified interfaces. 4.6. Doc Impact: o Writing Device Drivers guide o Man pages for new DDI, NDI, Hotplug library and other interfaces 4.7. Admin/Config Impact: TBD 4.8. HA Impact: TBD 4.9. I18N/L10N Impact: This project is planning to deliver a set of new admin tools, such as SHPC CLI and a GUI based tool. These tools will be properly globalized. 4.10. Packaging & Delivery: TBD 4.11. Security Impact: TBD 4.12. Dependencies: None 5. Reference Documents: [1] Solaris Hotplug Project - Proposal http://pciexpress.sfbay/hotplug/docs/psarc/shp/shp-proposal.txt [2] Solaris Hotplug Project Webpage http://pciexpress.sfbay/hotplug [3] PCI Hotplug Support - PSARC/1998/327 http://sac.sfbay.sun.com/PSARC/1998/327 [4] PCI Bus Resource Allocator - PSARC/1999/122 http://sac.sfbay.sun.com/PSARC/1999/122 [5] RCM Framework - PSARC/1998/460 http://sac.sfbay.sun.com/PSARC/1998/460 [6] Solaris Contracts and restart agreements - PSARC/2003/193 http://sac.sfbay.sun.com/PSARC/2003/193 [7] PCI Hotplug Framework Extensions for PCI Express - PSARC/2005/375 http://sac.sfbay.sun.com/PSARC/2005/375 [8] PCI Hotplug Resource Preallocation - FWARC/2006/198 http://sac.sfbay.sun.com/FWARC/2006/198 [9] Retire Agent for IO devices - PSARC/2007/290 http://sac.sfbay.sun.com/PSARC/2007/290 [10] PCI Hot-Plug Specification, Revision 1.0, June 20, 2001 [11] PCI Standard Hot-Plug Controller Specification, Revision 1.0, June 20, 2001 [12] PCI Express Base Specification, Revision 2.0, Dec 20, 2006 [13] Solaris Operating System - Hardware Virtualization Product Architecture, Revision 1.0, November 2007 6. Resources and Schedule: 6.1. Projected Availability: Q2FY10 6.2. Cost of Effort: Development 3.0 Engineers 12 Months Testing 2.0 Engineers 06 Months Documentation 1.0 Engineer 03 Months 6.3. Cost of Capital Resources: Need access to platforms that we want to support new Solaris hotplug framework or its features on. If we are to support a platform or device that we don't already have, such capital will need to be acquired, and could result in additional capital expense. 6.4. Product Approval Committee requested information: 6.4.1. Consolidation or Component Name: ON 6.4.3. Type of CPT Review and Approval expected: Standard 6.4.4. Project Boundary Conditions: N/A 6.4.5. Is this a necessary project for OEM agreements: No 6.4.6. Notes: None 6.4.7. Target RTI Date/Release: TBD 6.4.8. Target Code Design Review Date: TBD 6.4.9. Update approval addition: N/A 6.5. ARC review type: Standard 6.6. ARC Exposure: 6.6.1. Rationale: Open 7. Prototype Availability: 7.1. Prototype Availability: Q2FY09 7.2. Prototype Cost: 2.0 Engineers - 6 Months