Subject: Paravirtualized Drivers for Fully Virtualized xVM Domains [PSARC/2007/664 FastTrack timeout 11/30/2007] To: PSARC-ext@Sun.Com Cc: nils.nieuwejaar@sun.com Bcc: one-pager-list@sac.sfbay one-pager-log@sac.sfbay sac-bar@sac.sfbay Template Version: @(#)sac_nextcase 1.64 07/13/07 SMI 1. Introduction 1.1. Project/Component Working Name: Paravirtualized Drivers for Fully Virtualized xVM Domains 1.2. Name of Document Author/Supplier: Author: Nils Nieuwejaar 1.3 Date of This Document: 04 November, 2007 4. Technical Description 4.1 Summary The Xen hypervisor and the port of Solaris to that hypervisor are described in PSARC/2006/260. That case is considered required pre-reading for this. The Solaris port to the Xen hypervisor included disk and network drivers for paravirtualized (PV) domUs. These PV drivers provide standard disk and network interfaces to Solaris, but communicate with back-end drivers in the control domain (dom0) rather than interacting directly with hardware. When running in a fully virtualized (HVM) domain, Solaris continues to use standard device drivers. These drivers believe they are interacting with hardware, but are actually interacting with emulated devices that run in dom0. These emulated physical devices deliver significantly lower performance than the corresponding virtual devices in a PV domain. For example, the emulated network device may run at 1/10th the speed of a virtual device. This project proposes to provide the infrastructure necessary to allow the existing Solaris PV drivers to function in HVM domains, delivering significantly higher performance to applications running in those domains. We intend to backport these PV drivers to Solaris 10, enabling it to be a performant guest operating system. Therefore, we are requesting micro/patch binding for this project. 4.2 Proposal The project is comprised of several components. There is one new driver that provides the infrastructure necessary to allow software in an HVM domain to communicate with the hypervisor and the control domain. There are also minor modifications to the existing drivers, that will allow them to plug into the new infrastructure. 4.2.1 xpv xpv is a new leaf driver that binds to Xen's PCI ID (pci5853,1). This driver provides all the plumbing needed for PV drivers to drill through the bottom of a fully virtualized domain, and interact directly with the hypervisor. The driver uses HVM-specific hypercalls to: o map in the 'Xen shared info' page, which is used for communication between the hypervisor and OS. o map pages for grant tables, which are used to enable temporary transfers of page ownership between domains o register a single interrupt with the hypervisor, which it will trigger when there are events pending for the OS to process. After initialization, the driver's primary responsibility is to receive emulated 'physical' interrupts from the hypervisor when events are pending, and then forward those event notifications to the correct PV driver(s). 4.2.2 Modified Drivers cmdk is an existing Solaris driver that provides a common framework for block devices. This driver presents a simple high-level disk model to Solaris while abstracting away the distinctions between disks, CD drives, etc. This project modifies cmdk to be PV-aware. When loaded in an HVM domain, this driver uses the Layered Driver framework to forward all disk requests to the PV xdf driver rather than to the standard driver(s) for emulated physical devices. This forwarding mechanism allows us to add the PV drivers after a domain has already been installed, without having to modify the path to the boot device. In addition, the paravirtualized xdf, xnf, xpvd, and xenbus drivers must all be modified to register event handlers with a new 'xpv' driver, rather than registering interrupt handlers with Solaris. Other than this event delivery mechanism, all of a driver's interaction with the hypervisor and control plane is exactly the same in both PV and HVM domains. The scope of these changes is on the order of about a dozen lines per driver. 4.3 User Experience 4.3.1 Background Xen has adopted two very different models for supporting PV devices in HVM domains. These models are built into the architecture of Xen itself, and are not specific to Sun's implementation. For block devices, each device configured in a domU has two interfaces: an emulated ATA device on the PCI bus, and an explicit PV interface accessible through the Xenbus. The two interfaces provide access to the same storage on the back end. For network devices, each NIC is configured in dom0 to be either PV or HVM. If a device is configured to be HVM, Xen provides an emulated RealTek NIC on the PCI bus. If it is configured to be PV, then it provides an explicit PV interface on the Xenbus. A single NIC cannot be configured in both modes at once. 4.3.2 Initial Installation When the PV drivers are integrated into the WOS, they will function as any other block or network driver. When a domU is installed, the disks will automatically and transparently be accessed using the PV interfaces. Users will install to /dev/dsk/cXdYsZ as they do now. All the normal tools, such as format, prtvtoc, and fdisk will work as expected. When the installer probes for network devices, any PV NICs configured into the domain will be made available as xnf0 through xnf, just as (for example) Intel NICS are made available as e1000g0 through e1000g. 4.3.3 Upgrades or post-install additions When the PV drivers are added to an existing domain, our modifications to the cmdk driver will allow the PV disk drivers to fall into place transparently. When the driver is first installed, the system will continue to access disks using the existing emulated metal interfaces. When the domain is rebooted, the disks will automatically be accessed through the PV interfaces. The domain's bootpath remains unchanged, so the transition from HVM to PV should be completely transparent to the user. For network devices, however, the user experience will be less satisfying. As a NIC must be explicitly configured in dom0 to be either PV or HVM, simply installing the PV drivers will not deliver PV performance to the user. The domU must be reconfig- ured to contain PV NICs rather than HVM. When the domain is reconfigured, the 'brand' of the NIC changes from rtls (i.e. "RealTek") to xnf. Since Solaris binds the identity of a system to its NIC name, the user will have to sys-unconfig and reconfigure the domain. A natural question is: can't we transparently remap the rtls devices to xnf devices under the hood (as we do for disks), making this reconfiguration step go away. Unfortunately, no. Whether a NIC is configured to be PV or HVM is not simply a cosmetic issue. When Xen instantiates an HVM NIC for a domain, it does not create a back-end device that is capable of interacting with a front-end PV driver. Even if the domU had sufficient information about the back-end configuration to understand how this remapping should be done, there is simply nothing in the dom0 for our PV drivers to talk to. While this upgrade experience is certainly less smooth than one would hope, it is not dissimilar to the experience of upgrading physical hardware. Replacing a disk in a physical machine is a simple process, and the device naming is not affected by the brand of the hard drive. Replacing an Intel NIC with a Broadcom NIC, on other hand, requires exactly the same sys-unconfig as that required by replacing a fully virtualized NIC with a paravirtualized NIC. 4.4 Interface Table Imported Interfaces --------------------------------------------------------------------- | Interface Level Comments |-------------------------------------------------------------------- | Layered Driver Evolving | Framework | | Xen Interfaces External | Hypercall interface | Shared page format | Xenbus protocol | | PCI ID pci5853,1 External Xen's registered | PCI ID |-------------------------------------------------------------------- Exported Interfaces --------------------------------------------------------------------- | Interface Level Comments |-------------------------------------------------------------------- | ec_bind_evtchn_to_handler() Project Used by the PV | ec_unbind_evtchn() private drivers to plug into | hypervisor_mask_event() the new PV-on-HVM | hypervisor_unmask_event() framework. |-------------------------------------------------------------------- 4.4 References [1] PSARC 2006/260: Solaris on Xen [2] PSARC 2001/769: LDF: Layered Driver Framework 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open