This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: Bandwidth limit for virtual interface 1.2. Name of Document Author/Supplier: Author: Max Zhen 1.3 Date of This Document: 25 February, 2009 4. Technical Description 4.1. Summary In this fast-track, we plan to support applying bandwidth limit for virtual network interfaces attached to a guest domain running on Xen hypervisor. Minor release binding is requested. 4.2. Discussion After Crossbow(PSARC/2006/357)'s integration into Nevada, it's possible to enforce bandwidth limit for a guest domain from Solaris dom0 by setting bandwidth property of a back end NIC device attached to the domain. In this fast-track, we plan to enhance existing management tools to leverage dladm(1M) and its 'maxbw' property to set bandwidth limit for virtual network interfaces for domains running on top of Xen hypervisor. With the integration of porting Solaris to run on Xen(PSARC/2006/260), three external management tools are also ported and integrated into Solaris: virsh(1M)(PSARC 2007/157), xm(1M)(PSARC/2006/260) and virt-install(1M) (LSARC/2007/175). Virsh(1M) and xm(1M) commands are user interfaces of domain management and virt-install(1M) are used to install guest domains. These three tools currently have user interfaces for specifying configurations for virtual interfaces for a guest domain. So, they are going to be enhanced to support specifying bandwidth limit while defining a configuration of a virtual network interface. I will discuss about each of them in following sections. Note: please refer to the design doc[1] in case directory for detailed information. 4.2.1 Management tool architecture The architecture of management tools for virtual network interface can be illustrated as below: virsh ---------+ v virt-install ----> virtd | V xend --> vif-vnic/vif-dedicated ^ xm ---------+ So, from above graph, we can see that virtual interface configuration will be passed to xend(1M) directly by xm. While, configuration will be passed to virtd (see PSARC 2008/165), who will pass it to xend. But, no matter where does the configuration come from, xend will collect all configuration and pass them to either /usr/lib/xen/scripts/vif-vnic or /usr/lib/xen/scripts/vif-dedicated, which are shell scripts to set up back end NIC device based on the configuraion for the corresponding virtual interface of the guest domain. 4.2.2 Xm(1M) Xm command has already supported specifying bandwidth limit for virtual interface. But, due to the limitation of vif-vnic and vif-dedicated scripts, the bandwidth specified in xm command line or in '.py' configuration file for guest domain has no effect. So, we need to enhance vif-vnic and vif-dedicated scripts to be able to parse out bandwidth information in configuration from xend and issue appropriate dladm command to set bandwidth value as 'maxbw' property for the corresponding NIC device serve as the back end device for the virtual network interface. Thus, we can apply bandwidth limit for a virtual interface of guest domain. There is no intention to do any change to xm UI and the data format of the configuration from xend. The command line tool used by vif-vnic and vif-dedicated will be switched from /usr/lib/vna to /sbin/dladm due to lack of needed functionality of vna. Note: please refer to the xm.man.diff.bw.txt[2] in case directory for the difference. 4.2.3 Virsh(1M) There are two ways for end user to provide bandwidth limit information to virsh: + via 'virsh attach-interface' command line + via guest domain configuration file in XML format But, neither of them support specifying bandwidth limit information. So, we need to enhance both of them. We need to add one more option, "--capped-bandwidth", to 'virsh attach-interface' command line syntax to allow end user to provide bandwidth limit information while attaching a new virtual interface to a guest domain: --capped-bandwidth , where 'bandwidth_string' is an integer with one of the scale suffixes K, M, or G for Kbps, Mbps, or Gbps (the same format as used by dladm). 'Virsh create' and 'virsh define' are two commands that interact with XML format guest domain configuration file as a whole. So, we also need to extend current configuration file format to allow end user to provide bandwidth limit. In order to insert bandwidth information into XML file, we create a new element, "networkresource", inside "interface" element. Inside "networkresource", users provide bandwidth by setting "cappedbandwidth" element with three attributes, "unit", "period" and "value" to express the bandwidth limit. For example (100Mb/s): [...cut...] [...cut...] The supported unit can be 'gigabit', 'megabit' and 'kilobit'. The supported period can be 'second', 'millisecond' and 'microsecond'. And value is an integer to express the amount of data in unit allowed to be transferred in the specified period of time. We can easily add more network resource limit in it by adding more element inside "networkresource" element in this format later, if needed. We also communicate this extention back to community and there is no objection. Note: please refer to the virsh.man.diff.bw.txt[3] in case directory for the difference. 4.2.4 virt-install(1M) Virt-install currently does not support specifying bandwidth limit. We need to extend its command line syntax to allow it to be specified by end users. We may simply add one more option, "--capped-bandwidth=", to current virt-install option list to allow providing bandwidth information. But, since the option list of virt-install are already too long, keep adding options to it will just make situation worse. After consult with the community, we decide to group network related options into one option, "-w or --network", and turns original options into properties of this option separated by comma. So, before, a virt-install command line can look like: #virt-install ...--mac a:b:c:d:e:f --bridge bge0 --capped-bandwidth 100M/s \ --mac g:h:i:j:k:l --bridge bge1 --capped-bandwidth 200M/s... Now, it looks like: #virt-install ...--network mac=a:b:c:d:e:f,bridge=bge0,capped-bandwidth=100M/s \ --network mac=g:h:i:j:k:l,bridge=bge1,capped-bandwidth=200M/s... So, by using new syntax, it will be easier to add more properties for virtual interfaces in the future and will be clearer when we try to specify configurations for multiple virtual interfaces. Currently, we support three properties of "-w/--network" option: 'mac', 'bridge' and 'capped-bandwidth'. The first two are converted from current options, while the last one, 'capped-bandwidth', is added for specifying bandwidth limit (in the same format used in dladm command line). We will only support specifying bandwidth limit as an property, not as an option. Note: please refer to the virt-install.man.diff.bw.txt[4] in case directory for the difference. 4.3. Interfaces Exported interfaces: ---------------------------------------------------------------------------- | Interface | Stability | Comments | +-------------------------------+---------------+--------------------------+ | --capped-bandwidth option | | | | for virsh(1M) | Committed | | | | | | | -m/--mac, -b/--bridge | Obsolete | Old-style options | | | Uncommitted | | | | | | | new -w/--network syntax and | | | | 'capped-bandwidth' property | | | | of virt-install(1M) | Committed | | | | | | | "networkresource" and "capped | | | | bandwidth" elements in XML | | | | configuration file | Committed | | +-------------------------------+---------------+--------------------------+ Imported interfaces: ---------------------------------------------------------------------------- | Interface | Stability | Comments | +-------------------------------+---------------+--------------------------+ | dladm(1M) | Committed | | | | | | | bandwidth representation | | | | from xend(1M) (in xenstore) | External | | +-------------------------------+---------------+--------------------------+ 5. References PSARC 2006/260 Solaris on Xen PSARC 2006/357 Crossbow - Network Virtualization and Resource Management PSARC 2007/157 libvirt - a LGPL library to control guest domains PSARC 2008/165 xVM Hypervisor Remote Access (virtd) LSARC 2007/175 Virtual Machine Manager 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open FOOTNOTE: ========= [1] design doc: http://sac.eng/Archives/CaseLog/arc/PSARC/2009/137/spec/design.bw.txt [2] xm.man.diff.bw.txt: http://sac.eng/Archives/CaseLog/arc/PSARC/2009/137/spec/xm.man.diff.bw.txt [3] virsh.man.diff.bw.txt: http://sac.eng/Archives/CaseLog/arc/PSARC/2009/137/spec/virsh.man.diff.bw.txt [4] virt-install.man.diff.bw.txt http://sac.eng/Archives/CaseLog/arc/PSARC/2009/137/spec/virt-install.man.diff.bw.txt