Driver Update Project DDU Programming Interface Specification (V6, 1/13/10) 1 Overview 1.1 Introduction The Driver Update project makes it possible for OpenSolaris installers to have full access to a system's devices. This includes storage devices which could be used as installation targets. There are three OpenSolaris installers planned, and each will be supported by the Driver Update project. The GUI and text-mode installers run in an interactive environment; the Device Driver Utility (DDU) in graphic and text-mode versions will support these installers as standalone programs (programs separate from the installer itself). The Automated Installer (AI) runs non-interactively; it will be supported by a library of DDU functionality. Please see the Driver Update Functional Specification at http://www.opensolaris.org/os/project/caiman/Driver_Update/ for details. This document defines the interfaces between the three DDU deliverables and the environment in which they will operate. 1.2 Definition of Terms AI: Automated Installer AI Manifest: Automated Installer Input specification containing an installation blueprint and a group of systems to install it on. compatible name: A device ID string built from the product ID, vendor ID, revision ID and/or device class ID strings returned from the device. Drivers are mapped to devices based on compatible names. OpenSolaris generates a series of compatible names for each device, ranging from general to specific. The driver which maps to the most specific compatible name is the one used for that device. DDU: Device Driver Utility. Utility used to install drivers for devices which are missing them. DU-Image: Driver Update Image. A legacy format used to deploy driver updates, used by Install Time Update. ITU: Install Time Update. The ability to install additional drivers in the running kernel used during an installation. Traditionally this functionality was started when an installation CD was booted, via a menu early in the boot process. Additional drivers can allow the installer to find additional devices and install support for them. P5I: File format used by pkg(5) to fully identify a package (including name and location) or group of packages. pkg(5): New package format of the Image Packaging System (IPS), introduced with OpenSolaris and the proposed standard going forward. SVR4: Package format used for Solaris 10 and before. Third-party drivers: drivers which run on OpenSolaris, which are delivered by means other than an official OpenSolaris repository (e.g. pkg.opensolaris.org) 2 GUI DDU GUI interactive environments, most notably the Live CD, will include the GUI DDU. The Driver Update Functional Specification describes two ways of invoking the GUI DDU on the live CD: in a silent mode at boot time, and manually via an icon or through the application menu. The desired end result of running the DDU is an environment in which all devices are available. While a focus of the Driver Update project is on installation and the installer can make use of these new devices, the installer is decoupled from the DDU; and any application, not just the installer, can benefit from the new devices made available by the DDU. 2.1 Invocation 2.1.1 Silent Mode The DDU will be invoked in "silent mode" as the last phase of system boot. When the "jack" account is started on the live CD and the Gnome desktop is brought up for it, Gnome will auto-start the DDU in silent mode as follows: /usr/bin/ddu --silent Later, when the installer creates the "jack" and/or other user accounts on the target boot image, it will remove the ability of Gnome to autostart the DDU. 2.1.2 Add-Drivers Mode Add-Drivers mode is an interactive mode of the DDU, and is invoked when the DDU is started without the "--silent" switch. The GUI DDU is started this way from the Gnome applications menu. Any error code or status returned from the DDU upon termination is ignored. Errors should be made apparent to the user while the GUI is running. 2.2 Results and Their Propagation to the Installer The DDU installs new drivers into the booted kernel. This enables the devices which are operated by those new drivers. This in turn enables the applications, including the installer, to see and use the newly-enabled devices. The Live CD installer copies the contents of the running system when it creates its target boot image. Stored copies of the DDU-installed packages are not required. An install into the booted environment is all that is required of the DDU; the installer will pick up the new drivers automatically. 3. Text-Mode DDU Non-GUI interactive installation environments will be supported by the text-mode DDU. The text-mode DDU has one mode of operation: an interactive mode that is functionally equivalent to the Add-Drivers mode of the GUI DDU. 3.1 Invocation A menu process will be invoked as one of the last SMF services in the boot procedure. The menu process will display a menu on the console as boot completes. The menu will have the following options: The DDU will be invoked from a menu which appears on the console shortly after boot. The menu will have the following options: 1. Run installer 2. Install additional drivers 3. Shell 4. Reboot The DDU will be run in a separate process forked by the menu when "Install additional drivers" is selected. 3.2 Operating Environment Being a text-mode program, the DDU will communicate to the console (user) via the standard three file descriptors: stdin, stdout and stderr. The menu will not have console control while the DDU is operational, and ^Z (SIGTSTP) will be disabled. Thus it is important that the DDU always handle a ^C (SIGINT) signal. After cleanup, as appropriate, the DDU should exit upon receipt of ^C. The menu interface will ignore any return code from the DDU. Error messages generated by the DDU are to be displayed within the DDU's screens. Upon termination, the screen will be cleared and the menu redisplayed. 3.3 Results and Their Propagation to the Installer The text-mode installer, like the Live CD installer, will copy the contents of the running system when it creates its target boot image. Thus an install into the booted environment is all that is required of the DDU. The installer will pick up the new drivers automatically when installing the target. 4. Automated Installer The AI client will invoke the DDU library to add drivers to the installation environment, after the AI manifest has been syntactically validated, but before semantic validation. This is because a needed driver may be missing, and its device, which would be checked for during semantic validation, would also be missing. AI can keep the DDU library data structures from the first phase around to reuse during target installation. The same DDU library methods can be used during both installation-environment and target installation phases. 4.1 DDU library API 4.1.1 Definitions The following methods and object classes would be exported from the ddu library in support of AI. ------------------------------------------------------------------------ Name: ddu_devscan() Summary: - Do a device scan. Returns a list of devices in the form of ddu_dev_data objects. Given a device type, this function can return a list of all devices of that device type, or only devices of that device type which are missing their driver. Arguments: - return_missing_only: Boolean; default value is True - When True, ddu_devscan returns only the devices of the given device_type which are missing drivers. - When False, all devices of the given device_type found are returned. - device_type: string: type of device to return; default value is "all" - Possible items are: audio, battery, cpu, cd_dvd, memory, network, storage, usb, video, other - item can also be "all" to return all device categories. - string can be a comma-separated list of items above. Returns: - python list of ddu_dev_data objects corresponding to relevant devices. - an empty list (possible only when Return_missing_only is True) represents a system with no missing drivers. - notes on specific data fields of ddu_dev_data objects returned: - device_type returned will never be "all" - driver_name returned will be an empty string for devices which are missing drivers. - parent_item_ID will not be set for controllers. - instance_ID returned will be < 0 if no driver is attached to the device. - attach_status returned: - will be "detached" if driver is not attached. - will be "attached" if driver is attached. Raises discrete exceptions for: - could not start the scan - specific errors caught along the way of the scan Functionality: Scans the system for devices which are missing drivers. Returns a list of tuples. Each tuple has a compatible name string and a device description (e.g. "audio device"). Each compatible name string is unique, and corresponds to a type of device which is missing a driver. ------------------------------------------------------------------------ Name: ddu_package_lookup() Summary: - Lookup a package containing a device driver for a device. Arguments: - ddu_dev_data object to define the device - ddu_repo_list built from ddu_build_repo_list() Returns: - a ddu_package_object containing the package name and location, matching compatible name, driver name and any other information retrievable from the driver.db database. It also has the device description filled in from the passed argument. Raises discrete exceptions for: - invalid ddu_dev_data argument - invalid ddu_repo_list argument - no package found - error retrieving information from stored database files Functionality: Lookup a package containing a device driver for a device. Device is identified by the ddu_dev_data object passed in. Names in the compatible name string (generated via the ddu_dev_data object) are traversed from beginning to end until a suitable package is found or the string is exhausted. Search order will be: for each repository: for each name in compatible name string: if match found: return it Note about search order: It will be least confusing to users if the search checks one repository for all compatible names, before advancing to the next repository. This way, the repositories are searched in the order given by the user. This is the order suggested for this method. An alternative way of searching, which could yield better results (in the form of better matched drivers) but which would confuse users, would be to search all repositories for a match on the most specific compatible name, before searching all repositories for the next most specific compatible name. This would yield better results if one repository had a better match for a specific device, even if that repository was near the end of the repository list. However, it would appear to the user to violate the repository search order, since the package would not be taken from the first repository which had a match. ------------------------------------------------------------------------ Name: ddu_build_repo_list() Summary: - Build a list of repository objects to be used by ddu_package_lookup() Arguments: - List of IPS package repository (name, URL) tuples. Tuple names and URLs are strings. Returns: - ddu_repo_list: a list of ddu_repo_objects, one object per repository. List is ordered the same as the repository (name, URL) tuples given for input. Raises discrete exceptions for: - Repository not found - Publisher name does not conform to the URL. - error getting database file from a repository. (Repository to be listed in message Functionality: The DDU database of each repository in the repository list is obtained, uniquely named and stored in /tmp. A new ddu_repo_object is created; it contains the name of the repo and pointer to the stored database for that repo. After all repositories are processed, an ordered list of ddu_repo_objects is returned. ddu_package_lookup() will use the list of ddu_repo_objects to find a package containing a suitable driver for a given device. Note: No delete_from_repo_list() is mentioned here as AI won't need it, but if China Team needs it internally for the GUI or text-mode DDU, they can add it. ------------------------------------------------------------------------ Name: ddu_install_package() Summary: - Install the package specified by the ddu_package_object given Arguments: - A ddu_package_object that defines what to install - The root of the installation area. - A boolean set to True when it is OK to install a third party driver - Defaults to False when not specified. Returns: None Raises discrete exceptions for: - invalid package type in ddu_package_object - location and pkgname fields in ddu_package_object inappropriately blank - invalid db flags field in ddu_package_object - error trying to install a third party driver when not allowed - installation area inaccessible - error when trying to install package (capture specific tool error) except when trying to install a package which is already installed Functionality: - Installs a package as defined by a ddu_package_object. - Wildcards in the pkgname are accepted; multiple packages can be installed as a result of wildcards. - Capable of figuring out what type of package the ddu_package_object points to, if the package type is set to UNK (unknown). - Knows how to unbundle the DU images and extract packages from them. Installs all packages (of appropriate version) from a DU image. pkgname field ignored: no way to specify single packages of DU image. - Capable of reading all accepted types of packaging. Calls the appropriate installation tool (pkg, pkgadd, etc) to do the actual package installation. - Screens out installation of third party drivers when their installation is not desired. When the third argument is not specified or set to False, and the third_party_from_search field in the ddu_package_object is True, an exception is raised. - Silently returns if the specified package is already installed. ------------------------------------------------------------------------ Name: ddu_dev_data object Purpose: - An object containing device data. A list of these is returned by ddu_devscan(). Attributes: - description: String device description. - Can be used for error messages and status reporting. - device_type: String: One of the device types passed into ddu_devscan() - not "all" - driver_name: Name of the driver operating the device. - Set to an empty string for devices which are missing drivers. - item_ID: numeric string ID used by DDU to identify the device. - parent_item_ID: numeric string ID used by DDU to identify the parent device (controller) of the device. - instance_ID: int: Instance ID assigned to the device. - Will be < 0 if no driver is attached to the device. - attach_status: Status of whether or not a driver is attached to the device. - Will be "detached" if driver is not attached. - Will be "attached" if driver is attached. - vendor_id: Device's vendor ID. - device_id: Device's device ID. - class_code: Device's class code. - pci_path: Device path. Methods: - Constructor which takes args to initialize the fields. - get_compatible_name_string(): Summary: - Return a string of device compatible names. Arguments: None: Gets info from object itself. Returns: - string of device compatible names. Functionality: - Generates a device compatible name string from the object's data. - A device compatible name string is a space-separated string of all compatible names for a device. Compatible names in the string are ordered from the most specific name to the most general name (i.e. the "pciclass" definitions are listed last). - It is OK if two devices have one or more compatible names in common, but the full compatible name string of each different kind of device (as distinguished by vendor_id, device_id and/or class_code) should be unique. - Note: this method exists and the compatible name string not automatically generated with the rest of the data returned by the ddu_dev_data object itself, since the compatible name string is not always needed, and takes time and resources to generate. ------------------------------------------------------------------------ Name: ddu_repo_object Purpose: - An object which represents a repository to the DDU library. Attributes: - The publisher name of the repository (string) - The URL of the repository (string) - A pointer to a locally-stored copy of the database file from that repository (string) Methods: - Constructor which takes args to initialize the fields. ------------------------------------------------------------------------ Name: ddu_package_object Purpose: - Representation of a package found from a DDU database, used to specify a package to DDU library methods. Attributes: - type: can be set to "PKG", "SVR4", "DU", "P5I", "UNK" Set to "UNK" if type is unknown. - name: string, may be blank if location completely specifies path - location: - URL of repo for PKG type, - pathname of package datastream file or parent directory for SVR4 type, - pathname or device name of DUimage for "DU" type, - URI of p5i file for "P5I" type, - download-link from db if third party (URI, URL (ftp or http), pathname) - driver_name: string if single driver, or list if multiple drivers may be left blank) Used to help locate the right driver in 3rd party webpages which contain links to many drivers. - info_link: string, can be set blank; informational link from database - compatible_name: compatible name for which the match was found in database (string) - db_flags: flags found in database (e.g. 64 bit capability; type: TBD) - dev_description: device description; string, used for messages - third_party_from_search: boolean set to True when this object is result of db search and is a third party driver. Set only by ddu_package_lookup(). Used for rejecting third-party driver installation requests when ddu_install_package() args specifies this Methods: - Constructor which takes args to initialize the fields. - Have some fields take default values if not given. They will be provided to the constructor only if object is constructed from a database search. - Have the following fields default to blank if not given: - info_link - compatible_name - driver_name - dev_description - db_flags - Have the following fields default to False if not given: - third_party_from_search ------------------------------------------------------------------------ 4.1.2 Programming Use Cases Here are the use cases of the functions and classes above. They describe how to add a driver to the installation environment. AI will have to perform installation of added drivers twice: in the installation environment and in the target. Once the ddu_package_object objects are set up, they can be used for both. The second arg to ddu_install_package() will say where to install. Note that all of the code below will be inside try/except clauses to catch errors. 4.1.2.1 specified in the manifest. Perform a database search for missing devices. Do not install third party drivers. # Get the list of devices which are missing drivers. device_list[] = ddu_devscan(return_missing_only, "all") # Build the repo list from the repo name and URL taken from the # manifest. # If no repo supplied in tag, then use the list of # (name, URL) pairs from where normal packages will be # installed. repos[] = ddu_build_repo_list(list of (name, URL) tuples) for device in device_list: # Look up the package for the given device. ddu_package_obj = ddu_package_lookup(device, repos[]) # The returned ddu_package_obj comes back # filled in as follows: # type: PKG # pkgname: package name # drivername: driver name # location: URL # info link: blank # compatible name: filled in with device ID # device descr: filled in from dev_descr argument # db flags: filled in per database # third_party_from_search: False if ((ddu_package_obj.get_location() != "") and (ddu_package_obj.get_name() != "")): # Now install the package. # Note that if it is OK to include third party # drivers in the search and install, set the # third arg to True. ddu_install_package(ddu_package_obj, "/") else: log("Info on driver for %s is at %s" % ( ddu_package_obj.get_dev_descr(), ddu_package_obj.get_info_link()) 4.1.2.2 specified in the manifest. Perform a database search for missing devices. Allow installation of third party drivers (addall). Code is the same as above, except the third argument to ddu_install_package() is passed explicitly, and is set to True to allow installation of third party drivers. Suppose that the search returns a third party driver, and that its entry has a download link. The link points to a file, but the type of file is unknown. This means that ddu_install_package() will have to figure out the file type, which may slow things down (as in the worst case it may need to try all packaging tools in succession until the right one is found), but is still doable. In this scenario, the ddu_package_object will have different contents: type: UNK (unknown type) pkgname: "" ("location" field has full path) location: download link from db drivername: driver name info link: info link if available from db compatible name: filled in device descr: filled in db flags: filled in per database third_party_from_search: True 4.1.2.3 Same scenario as 4.1.2.2 but the driver entry found does not have a download link. It does have an informational link. In this case, the ddu_package_object will have the following: type: UNK (unknown type) pkgname: "" location: "" drivername: driver name info link: info link from db compatible name: filled in device descr: filled in db flags: "" third_party_from_search: True Code checks for blank name and location fields before attempting to install the package. Other parts of the code can check for non-blank informational links and display them as appropriate. 4.1.2.4 An P5I file is specified in the manifest, as # AI creates a ddu_package_object based on the manifest # contents. blocn = bundle.get_location() bname = bundle.get_name() btype = bundle.get_type() ddu_package_obj = ddu_package_object( type=btype, pkgname=bname, location=blocn) # This will result in the following ddu_package_obj: # type: P5I # pkgname: "" # location: "http://repo.raidco.com/p5i/p3raid" # drivername: "" # info link: "" # compatible name: "" # device descr: "" # db flags: "" # third_party_from_search: False # Install the package ddu_install_package(ddu_package_obj, "/") 4.1.2.5 A DUimage is specified in the manifest, as This case is similar to 4.1.2.4. # AI creates a ddu_package_object based on the manifest # contents. blocn = bundle.get_location() bname = bundle.get_name() btype = bundle.get_type() ddu_package_obj = ddu_package_object( type=btype, pkgname=bname, location=blocn) # This will result in the following ddu_package_obj: # type: DU # pkgname: "" # location: "/media/USBSTICK" # drivername: "" # info link: "" # compatible name: "" # device descr: "" # db flags: "" # third_party_from_search: False # Install the package ddu_install_package(ddu_package_obj, "/") 4.1.2.6 Two SVR4 packages are specified in the manifest, as This case is also similar to 4.1.2.4. Call the code below in a loop to get all of the manifest contents. for bundle in bundles: # AI creates a ddu_package_object based on the # manifest contents. blocn = bundle.get_location() bname = bundle.get_name() btype = bundle.get_type() ddu_package_obj = ddu_package_object( type=btype, pkgname=bname, location=blocn) # For the first bundle, this will result in # the following ddu_package_obj: # type: SVR4 # pkgname: "3Praiddev" # location: "/media/USBSTICK" # drivername: "" # info link: "" # compatible name: "" # device descr: "" # db flags: "" # third_party_from_search: False # Install the package ddu_install_package(ddu_package_obj, "/") 4.1.2.7 Other Possibilities There are some subcases of case 4.1.2.3, not relevant now, but could be in the future, which are easily handled. These cases are when a third party driver is returned and the type of package is known (as opposed to unknown). ddu_install_package() can then skip the cycle of trying all of the installation tools, and just use the appropriate one because it knows the package type. 4.1.2.8 Putting it All Together The functional specification allows the clause in the manifest to have zero or more , and then an optional at the end. The code for handling this would be similar to the following. Note that all of this would be in try/except clauses to handle errors. return_missing_only = True # List of packages to install on target. target_package_list = [] for bundle in bundles: # AI creates a ddu_package_object based on the # manifest contents. blocn = bundle.get_location() bname = bundle.get_name() btype = bundle.get_type() ddu_package_obj = ddu_package_object( type=btype, pkgname=bname, location=blocn) target_package_list.append(ddu_package_obj) # Install the package ddu_install_package(ddu_package_obj, "/") # Now process the if present. if (searchall != None): # Get the list of devices which are missing drivers. # It is important to do this after installing bundles. device_list[] = ddu_devscan(return_missing_only, "all") # Build the repo list from the repo name and URL taken # from the manifest. # If no repo supplied in tag, then use the # list of (name, URL) tuples from where normal packages # will be installed. repos[] = ddu_build_repo_list(list of (name,URL) tuples) for device in device_list: # Look up the package for the given device. ddu_package_obj = ddu_package_lookup(device, repos[]) if ((ddu_package_obj.get_location() != "") and (ddu_package_obj.get_name() != "")): target_package_list.append( ddu_package_obj) # Now install the package. # Note that if it is OK to include third # party drivers in the search and # install, set the third arg to True. ddu_install_package(ddu_package_obj, "/", searchall.addall) else: log("Info on driver for %s is at %s" % ( ddu_package_obj.get_dev_descr(), ddu_package_obj.get_info_link()) Later, when installing on the target image, just cycle through the target_package_list, calling ddu_install_package() with its second argument set to the target image root.