#ident "@(#)shp-userland.txt 1.19 09/08/19 SMI" Copyright 2009 Sun Microsystems Title: Userland Components of Solaris Hotplug. Date: August 9, 2009 Author: scott.carter@sun.com Abstract: This document highlights the userland components of the Solaris Hotplug Framework, and is an extension of the main proposal for that project. Contents: 1. Introduction 2. Architecture 3. Technical Details 4. Interfaces 1.0 Introduction This document only describes the userland components of the Solaris Hotplug Framework project. And specifically only for phase 1 of the project. Kernel details are documented separately. The first phase of the userland implementation supports the following new features introduced by the Solaris Hotplug Framework: virtual hotplug support, and an improved SHPC state model. 1.1 Scope and Roadmap The Solaris Hotplug Framework project is a multi-phase project. The first phase introduces the following features into userland: - New hotplug(1M) CLI to support physical and virtual hotplugging. - New cfgadm plugin to support physical hotplugging of connectors. - New libhotplug(3LIB) library shared by hotplug(1M) and cfgadm. - New hotplugd(1M) daemon to centrally manage hotplug operations. - RBAC based authentication. Userland features that are planned for a second phase include: - Management of hotplug events for third party consumers. - Management of user friendly aliases to hotplug connectors and ports. - Management of blacklisted components. And in a third phase of userland functionality it is foreseen that a GUI would be developed based upon the libhotplug(3LIB) interfaces. 1.2 References - PSARC/2008/181 Solaris Hotplug Framework: Architecture and Design - PSARC/1998/460 RCM Framework - PSARC/2000/517 Thread-Safe Audit API 2.0 Architecture Here is a block diagram of the various userland components: +-------------+ +-------------------------------+ | hotplug(1M) | | cfgadm(1M) | | (SHP CLI) | +-------------------------------+ +-------------+ | libcfgadm | | libhotplug | +------------+------------------+ +-------------+ | SHP plugin | Other Plugins... | | +------------+------+-----------+ | | libhotplug | | librcm | | Door +------------+ +-----------+ | | | | | | | | Door | | | | +-----------------------------------+ | | | | | hotplugd(1M) | | | | | | (SHP Daemon) | | +---------------------+-------------+ | | libdevinfo / modctl | librcm | | +---------------------+-------------+ V | | +------------+ | ------------------->| RCM Daemon | | +------------+ | | | Device +----------------+ | Contract | Other Consumer | | ------------->| Applications | | | Events +----------------+ | | | | Userland . . . . . .|. . . . . . . . .|. . . . . . . . . . . . . . . . . . V | Kernel The architecture includes the hotplugd(1M) daemon that centrally manages all ongoing hotplug operations, both synchronous and asynchronous. An asynchronous operation is one detected by the kernel (e.g. a surprise removal or an ATTN button event). A synchronous operation is one that a user initiated through a CLI or GUI. The daemon serializes operations to ensure there are no conflicting operations. And it fully sequences all operations by coordinating with in-kernel portions of the SHP framework through modctl APIs, and with other consumer applications through the RCM framework. Users can initiate operations, manage hotplug connections, or view the status and dependency relationships of hotplug connections through the hotplug(1M) CLI or the legacy cfgadm(1M) CLI. In either case, the application uses the libhotplug(3LIB) library to communicate through a private doors based IPC mechanism to hotplugd(1M) where the operations are actually implemented. 3.0 Technical Details 3.1 Hotplug CLI The hotplug(1M) CLI allows a user to: - View a list of defined hotplug connectors and ports, their status, dependencies, and usage. - Initiate state change operations on hotplug connectors and ports. - Perform private, bus specific functions on a hotplug connector. This section gives a high level summary of the capabilities of the hotplug(1M) CLI. Refer to the hotplug(1M) man page for full details. 3.1.1 Displaying Hotplug Connectors Consider the following about hotplug connections: - Each hotplug connector has a name and a current state. - Each hotplug connector has one or more dependent hotplug ports. - Each hotplug port also has a name and a current state. - Each hotplug port has a dependent device node. - There is a hierarchy of connectors and ports, some dependent upon others, depending upon the physical composition of the hotpluggable components in the hardware. - Connectors and ports are integrated in the device tree hierarchy, each one being associated with a specific device node. - Beyond device tree dependencies, additional layers of dependency occur from other subsystems (e.g. filesystem mounts, networks). All these details are of interest to a system administrator when evaluating the impact of a hotplug operation on a system. This necessary information is all displayed by the hotplug(1M) CLI. The natural way to represent such hierarchical relationships is as a graph or tree. The hotplug(1M) CLI displays these details as a tree, and in a manner consistent with other existing CLIs (such as prtconf(1M) and prtpicl(1M)), by indenting each subsequent layer to show the dependency relationships. In reality the structure is a graph and not a tree, because multi-pathed resources have multiple parents. In these cases, the hotplug(1M) CLI will display some resources multiple times, once per path. A user may display the entire tree of all connectors, ports, and their dependencies in the system. Or the user may indicate a path (and possibly a connector or port) to display just a subset. For each hotplug connector or port, the hotplug(1M) CLI displays: 1) its name, and 2) its current state. The hotplug(1M) CLI displays information with varying levels of verbosity. Detailed usage information can be gathered from the RCM framework and included in the output. RCM usage information includes higher level usage of devices (e.g. filesystem mounts, plumbed network interfaces, storage volumes, multipathed devices, etc.) 3.1.2 Initiating State Change Operations In general, initiating a state change operation involves specifying the following: o Target device path. o Target hotplug connector or port associated with the device path. o What hotplug state the target should be transitioned to. Initiating a state change operation on a hotplug connection will effect the full hierarchy of dependents below the target. 3.1.3 Virtual Hotplug Support There are physical hotplug connectors, and physical components that can be inserted or removed in those connectors. Traditionally the hotplug features in Solaris were centered on this physical style of hotplugging. The terminology used "attachment points" to describe physical receptacles and their occupants. But except for the physical actions of inserting or removing the components, the remainder of a hotplug operation is quite generic. It mostly entails probing or de-probing devices, attaching or detaching device drivers, and reconfiguring the higher levels of resource consumption. It is not necessary to limit hotplugging to the boundaries of physical components. Especially when components may be multi-function devices whose resources could be divided and managed separately. Virtual hotplugging improves the situation by giving an administrator finer grained control over the system's configuration, and also allows allocating resources individually to virtualized environments. Virtual hotplugging introduces new terminology. Hotplug connectors describe physical locations where hotpluggable components can be inserted or removed. Devices in the Solaris device tree represent the logical hardware functions that each have their own attached driver. A hotplug port manages the connection of a device to the system. There exists a hotplug port for each device, regardless if it can be physically hotplugged. Each bus nexus is associated with one or more hotplug ports to represent its dependent devices. And virtual hotplug operations can be performed on each port. If physical hotplugging is possible, then the bus nexus will also have an extra layer of hotplug connectors upon which the ports depend, to manage related physical hotplug operations. The hotplug(1M) CLI can perform hotplug operations on connectors and ports. The legacy cfgadm(1M) CLI is limited only to physical hotplug operations on connectors. 3.1.4 Private Bus Functions There will always be extra functionality implemented privately by bus controllers that just doesn't fit well in a generic state model for hotplugging. Therefore the Solaris Hotplug Framework provides a mechanism to initiate private, bus-specific functions on hotplug connectors. There are 'set' and 'get' subcommands in the hotplug(1M) CLI which implement this mechanism. Each subcommand accepts a comma-separated list of private options, as could be parsed by getsubopt(3C). For the 'set' subcommand, the options represent a bus-specific function and its parameters that will be implemented by a bus controller. For the 'get' subcommand, the options are bus specific properties to be queried and displayed to the user. The userland software does no interpretation or implementation of the commands aside from splitting the comma separated list of options and packing it into an nvlist_t data structure for delivery to the kernel. 3.2 Hotplug Library The hotplug library connects administrative commands to the hotplugd(1M) daemon, where all hotplug operations are then implemented. The library exports a management API to its clients, allowing them to locate, list, and initiate operations on hotplug connectors and ports. A private doors based IPC mechanism is then used to communicate with the daemon internally to actually retrieve hotplug status information and perform commands. 3.2.1 Management API The information about hotplug connectors, ports, and their devices is represented as a graph. The interfaces to get and process all this information is as follows: - An application first gets a snapshot of hotplug information. A path and connection name can be specified to select a subset. And flags are used to indicate the level of detail included. - There are functions to traverse the nodes of the snapshot. The caller can manually traverse through child and sibling nodes, or perform an automated traversal with a callback function. - For each node in the snapshot, there are accessor functions to get individual data items for each node. Including the node's name, full path, current state, what type of node it is, and any verbose usage description associated with the node. - There are control functions to perform actions on a node. To initiate a state change operation, or to set/get bus specific properties. - Once finished, a final interface exists to clean up and destroy the snapshot. 3.2.2 Doors Based IPC Mechanism The hotplug library opens the following door file to access a private doors based IPC mechanism to the hotplugd(1M) daemon: - /var/run/hotplugd_door Door calls are used by libhotplug(3LIB) to get hotplug information snapshots, bus specific parameters, or initiate state changes and bus specific functions. In each case the parameters of the call are packed into an nvlist_t data structure and transmitted through the door to the hotplugd(1M) daemon. The hotplugd(1M) daemon then transmits the results of the operation in a similarly packed nvlist data structure. The door is only opened temporarily to implement the following libhotplug(3LIB) functions: - hp_init(): To get a hotplug information snapshot. - hp_set_state(): To initiate a state change operation. - hp_set_private(): To set bus specific properties. - hp_get_private(): To get bus specific properties. The remainder of the functionality implemented by libhotplug(3LIB) operates upon hotplug information snapshots independent of any active communications with the hotplugd(1M) daemon. 3.3 Hotplug Daemon (hotplugd(1M)) The hotplug daemon is an SMF managed service, which operates as a door server to receive incoming libhotplug.so commands from local entities. It also receives system events from the kernel which indicate when any asynchronous hotplug operations occur, such as ATTN button events or surprise removals. There are certain architectural reasons why a hotplug daemon is actually required, versus just implementing shared functionality in the hotplug library. These reasons are: - If a client application crashes while performing a hotplug operation, the system may be left in an inconsistent state. Because the hotplug daemon is managed by SMF, and because all operations are implemented by the hotplug daemon, recovery scenarios can then be ensured and automated. - RCM interactions require all privileges. The hotplug daemon runs with all privileges so that it can perform RCM interactions while still allowing client applications to have basic privilege. The hotplug daemon is the point of enforcement for security. It uses RBAC to perform authorization checks on incoming calls on its doors interface, and performs all auditing actions. 3.3.1 Service Management Facility (SMF) The hotplugd(1M) daemon is managed by SMF: the Service Management Facility. Using SMF simplifies the efforts to manage the daemon as a long running service. It will ensure that the hotplug daemon is automatically restarted, thus making the hotplug feature more robust. The hotplug daemon can maintain a transaction log so it can automatically recover if it ever crashes during an operation. The SMF implementation includes the following details: - FMRI: svc:/system/hotplug:default - SMF manifest: /var/svc/manifest/system/hotplug.xml - SMF method script: /lib/svc/method/svc-hotplug - The service is local only. - There are no service configuration properties. - The method context executes the hotplugd(1M) daemon as root. - The action_authorization is: solaris.smf.manage.hotplug - The value_authorization is: solaris.smf.manage.hotplug 3.3.2 Doors Based IPC Mechanism The IPC mechanism for libhotplug.so clients to communicate with the hotplugd(1M) daemon is based on doors. This restricts service to local clients only. All data for commands and their results are transmitted through the door in packed nvlist_t data structures. 3.3.3 Serialization of Hotplug Operations Internally, the hotplugd(1M) daemon serializes incoming requests to do synchronous hotplug operations. It does the necessary locking to avoid interference between simultaneous operations that collide or overlap. It is the central arbiter for all hotplug operations that ultimately go through a libdevinfo(3LIB) and modctl based interface to the in-kernel portions of the Solaris Hotplug Framework. Higher level sequencing with other frameworks is managed either through the new device contract event interfaces directly consumed by other consumer applications, or by interactions with the RCM framework. 3.3.4 RCM Interactions The hotplugd(1M) daemon interacts with the RCM framework for two separate reasons. It collects detailed resource usage information from the RCM clients which is then integrated with other hotplug connection information when providing verbose listings. And it sequences RCM offline operations on the resources affected by any hotplug connector/port state change operations. The RCM framework must be informed of the root of an operation before the hotplugd(1M) daemon initiates a state change operation in the kernel. And RCM operations are transactional, which means that the hotplugd(1M) must further interact with the RCM framework at the end of an operation to notify RCM if the operation succeeded or failed. RCM clients take specific actions to restore the use of a resource if a hotplug attempt has failed. Or when a hotplug succeeds, RCM clients need to be informed that it is now safe to discard the information they retained in case restoration was necessary. 3.3.5 Gathering Hotplug Information Information about hotplug connectors, ports, and their states is managed in the kernel and exported to userland by libdevinfo(3LIB). Because the information is distributed throughout the device tree as additional properties of device nodes, this technique naturally supports gathering the hierarchical relationships between hotplug connectors, ports, and their dependent device nodes. Basically, the libdevinfo(3LIB) extensions include: - A new flag for di_init(3DEVINFO) to include hotplug data in a device tree snapshot. - Hotplug data will appear as a series of hotplug connection nodes (di_hp_t) associated with each individual device node (di_node_t) that supports the hotplug feature. - New functions facilitate traversing either all hotplug connection nodes in the entire libdevinfo snapshot, or just the hotplug connection nodes associated with a device. - New accessor functions are added to query individual fields of information about hotplug connection nodes. See the following manpages for additional details: - libdevinfo(3LIB) - di_init(3DEVINFO) - di_walk_hp(3DEVINFO) - di_hp_next(3DEVINFO) - di_hp_name(3DEVINFO) If a client requests additional details about how devices are used, then the hotplugd(1M) daemon also gathers information from the RCM framework and integrates the resulting RCM information with the information already gathered from libdevinfo(3LIB). Information returned from the hotplugd(1M) daemon is formatted as a different type of libhotplug(3LIB) snapshot for further processing by the hotplug(1M) command or other clients. 3.3.6 modctl API A libhotplug.so caller initiates a state change operation by referencing a node in its hotplug information snapshot. The snapshot includes a name and path associated with each hotplug connector. This path and name is transmitted by the libhotplug.so client through the doors based IPC mechanism to the hotplugd(1M) daemon to identify the target of a state change operation. The hotplugd(1M) daemon then uses a modctl API to initiate the operation in the kernel, referencing the target of the operation by the given hotplug connector name. The modctl API is described in other documentation that defines the kernel interfaces of the Solaris Hotplug Framework. 3.3.7 RBAC Authorizations The hotplugd(1M) daemon performs RBAC authorization checks on each door call. It uses door_ucred(3C) to obtain the credentials of the client process that is calling the door. And from these credentials it obtains the effective UID of the calling process. Based on the caller's effective UID, it uses RBAC to perform the authorization checks before implementing door services. The following new authorizations are added: - solaris.hotplug.:::Solaris Hotplug::help=HotplugHeader.html - solaris.hotplug.modify:::Modify Hotplug Connections:: \ help=HotplugModify.html - solaris.smf.manage.hotplug:::Manage Hotplug Service:: \ help=SmfManageHotplug.html The "solaris.hotplug.modify" authorization is required to perform a change of state operation on a hotplug connector or port. Or to set bus private properties on a hotplug connector. The "solaris.smf.manage.hotplug" authorization is required to manage the SMF service for the hotplug daemon. The following new rights profile is added to contain these authorizations: - Hotplug Management:::Manage Hotplug Connections: \ auths=solaris.smf.manage.hotplug,solaris.hotplug.*; \ help=RtHotplugMngmnt.html The rights profile is granted to the existing rights profile of "Maintenance and Repair", which in turn is granted by default to the existing rights profile of "System Administrator." 3.3.8 Auditing The hotplugd(1M) daemon is managed by SMF, so events to start and stop the daemon are automatically audited. In addition, the following actions are also audited: - RBAC authorization checks for incoming door operations. - Calls to modctl() to perform change of state operations. - Calls to RCM to perform offline, online, and remove operations. All auditing will record details about which event occurred, who initiated the operation, whether the activity succeeded or failed, and all relevant parameters. 3.4 Libcfgadm Plugin (cfgadm_shp.so) Existing libcfgadm plugins for other DR and hotplug features will remain unchanged. But to integrate support for the new hotplug connectors defined by the Solaris Hotplug Framework into the existing cfgadm framework, a new libcfgadm plugin is required. Like the hotplug(1M) CLI, it will use libhotplug.so to interact with the hotplugd(1M) daemon. It will only operate upon physical connectors, and not virtual ports. Virtual hotplugging is not supported. Its interactions with hotplugd(1M) will include gathering connector information which can then be integrated into the cfgadm(1M) output, and initiating state change operations on those connectors. In addition to the plugin, other changes are also required in the generic libcfgadm.so library. Currently, libcfgadm.so searches the device tree for "attachment points", and dispatches operations on these targets to class-specific plugins. This information is found in libdevinfo(3LIB). The libcfgadm.so library is modified to also recognize the new hotplug connectors in libdevinfo(3LIB), and to dispatch operations on these new targets to the SHP specific plugin. 3.5 Other Consumer Applications High level consumers of hotpluggable devices include features such as the Solaris network stack, filesystem mounts, clustering features, storage volumes, and various multipathing features for storage and networks. Many of these applications already integrate with the RCM framework to synchronize their reconfiguration with DR or hotplug operations. The hotplugd(1M) daemon maintains interactions when appropriate with the RCM framework to synchronize these consumers, and to gather their usage information for verbose hotplug information displays. 4.0 Interfaces 4.1 Exported Interfaces Main Deliverables: Interface Stability Comments ---------------------------------------------------------------------- /usr/sbin/hotplug Committed hotplug(1M) CLI /usr/lib/hotplugd Project Private hotplugd(1M) daemon /lib/libhotplug.so.1 Consolidation Private libhotplug(3LIB) /var/run/hotplugd_door Project Private Door file, hotplugd(1M) RBAC Authorizations: Interface Stability Comments --------------------------------------------------------------------------- Hotplug Management Uncommitted Rights Prof. solaris.hotplug. Uncommitted Hierarchy solaris.hotplug.modify Uncommitted Modify Auth. solaris.smf.manage.hotplug Uncommitted SMF Auth. /usr/lib/help/auths/locale/HotplugHeader.html Uncommitted Help Files /usr/lib/help/auths/locale/HotplugModify.html Uncommitted " " /usr/lib/help/auths/locale/SmfManageHotplug.html Uncommitted " " /usr/lib/help/auths/locale/C/HotplugHeader.html Uncommitted " " /usr/lib/help/auths/locale/C/HotplugModify.html Uncommitted " " /usr/lib/help/auths/locale/C/SmfManageHotplug.html Uncommitted " " /usr/lib/help/profiles/locale/RtHotplugMngmnt.html Uncommitted " " /usr/lib/help/profiles/locale/C/RtHotplugMngmnt.html Uncommitted " " SMF Service for hotplugd(1M): Interface Stability Comments ---------------------------------------------------------------------- svc:/system/hotplug Committed FMRI /var/svc/manifest/system/hotplug.xml Project Private Service Manifest /lib/svc/method/svc-hotplug Project Private Method Script Libdevinfo(3LIB) Extensions: Interface Stability Comments ---------------------------------------------------------------------- DINFOHP Consol. Priv. Flag for di_init() to include hotplug information in snapshot. DI_HP_NIL Consol. Priv. A NULL di_hp_t structure. DI_HP_CONNECTOR Consol. Priv. Flag for di_walk_hp() to traverse hotplug connectors. DI_HP_PORT Consol. Priv. Flag for di_walk_hp() to traverse hotplug ports. di_hp_t Consol. Priv. Structure of a hotplug connector associated with a di_node_t. di_walk_hp() Consol. Priv. Traverse hotplug connectors associated with a di_node_t. di_hp_next() Consol. Priv. Get next di_hp_t in a list. di_hp_name() Consol. Priv. Get name of a di_hp_t connector. di_hp_state() Consol. Priv. Get state of a di_hp_t connector. di_hp_last_change() Consol. Priv. Get timestamp of last state change of a di_hp_t connection. di_hp_type() Consol. Priv. Return connection type. di_hp_description() Consol. Priv. Return description of the hotplug handle (e.g. "PCI Slot", etc.) di_hp_child() Consol. Priv. Return child device node of a virtual hotplug port. di_hp_connection() Consol. Priv. Return connection number. di_hp_depends_on() Consol. Priv. Return depended upon connection number Management Interfaces (libhotplug(3LIB)): Interface Stability Comments ---------------------------------------------------------------------- HP_NODE_DEVICE Consol. Priv. Node type for a device node. HP_NODE_CONNECTOR Consol. Priv. Node type for physical connector HP_NODE_PORT Consol. Priv. Node type for virtual port. HP_NODE_USAGE Consol. Priv. Node type for usage record. hp_node_t Consol. Priv. Structure of a node in snapshot. hp_init() Consol. Priv. Initialize a hotplug snapshot. hp_fini() Consol. Priv. Cleanup/Remove hotplug snapshot. hp_traverse() Consol. Priv. Traverse nodes in a snapshot. hp_name() Consol. Priv. Get a node's name. hp_path() Consol. Priv. Get a node's full path name. hp_type() Consol. Priv. Get a node's type. hp_description() Consol. Priv. Get description of connector or port. hp_state() Consol. Priv. Get a node's current state. hp_last_change() Consol. Priv. Get timestamp when a connector or port's state was last changed. hp_usage() Consol. Priv. Get a node's usage description. hp_parent() Consol. Priv. Get a node's parent. hp_child() Consol. Priv. Get a node's first child. hp_sibling() Consol. Priv. Get a node's next sibling. hp_set_state() Consol. Priv. Command to initiate state change. hp_set_private() Consol. Priv. Bus specific 'set' function. hp_get_private() Consol. Priv. Bus specific 'get' function. 4.2 Imported Interfaces The following interfaces are imported: - libdevinfo (Committed): to access device and connection information. - librcm (Consolidation Private): to get RCM usage, and RCM offline. - libsecdb (Committed): to perform RBAC authorization checks. - libbsm (Contract Private): to perform auditing. 4.3 Packaging The SHP userland components are platform independent and have strong dependencies on kernel APIs. The SHP hotplug feature is also similar to the existing cfgadm and RCM frameworks which are currently delivered in core Solaris packages. For all these reasons the SHP userland components are delivered in core Solaris packages as follows: SUNWarc: -------- - /usr/lib/llib-lhotplug - /usr/lib/llib-lhotplug.ln SUNWcsu: -------- - /usr/sbin/hotplug - /usr/lib/hotplugd - /usr/lib/help/auths/locale/C/HotplugHeader.html - /usr/lib/help/auths/locale/C/HotplugModify.html - /usr/lib/help/auths/locale/C/SmfManageHotplug.html - /usr/lib/help/profiles/locale/C/RtHotplugMngmnt.html SUNW0on: -------- - /usr/lib/help/auths/locale/HotplugHeader.html - /usr/lib/help/auths/locale/HotplugModify.html - /usr/lib/help/auths/locale/SmfManageHotplug.html - /usr/lib/help/profiles/locale/RtHotplugMngmnt.html SUNWcsr: -------- - /var/svc/manifest/system/hotplug.xml - /lib/svc/method/svc-hotplug SUNWcsl: -------- - /lib/libhotplug.so.1 - /lib/amd64/libhotplug.so.1 - /lib/sparcv9/libhotplug.so.1