From sac-owner Fri May 26 18:42:08 2006 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.6+Sun/8.13.6) with ESMTP id k4R1g7ig019469 for ; Fri, 26 May 2006 18:42:08 -0700 (PDT) Received: from sunmail5.uk.sun.com (localhost [127.0.0.1]) by sunmail5.uk.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id k4R1g6oe014834 for ; Sat, 27 May 2006 02:42:06 +0100 (BST) Received: (from noaccess@localhost) by sunmail5.uk.sun.com (8.13.4+Sun/8.13.3/Submit) id k4R1g6gI014832 for one-pager-not-2b-used-directly; Sat, 27 May 2006 02:42:06 +0100 (BST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.149.247.22]) by sunmail5.uk.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id k4R1g5Qw014823; Sat, 27 May 2006 02:42:05 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IZW00101I25L600@nwk-avmta-2.sfbay.sun.com>; Fri, 26 May 2006 18:42:05 -0700 (PDT) Received: from jurassic.eng.sun.com ([129.146.226.130]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IZW00K19I24AD20@nwk-avmta-2.sfbay.sun.com>; Fri, 26 May 2006 18:42:04 -0700 (PDT) Received: from [129.146.11.175] (sr1-umpk-09.SFBay.Sun.COM [129.146.11.175]) by jurassic.eng.sun.com (8.13.6+Sun/8.13.6) with ESMTP id k4R1g4GU936528; Fri, 26 May 2006 18:42:04 -0700 (PDT) Date: Fri, 26 May 2006 18:42:04 -0700 From: Kais Belgaied Subject: [/]Crossbow - Network Virtualization and Resource Management To: one-pager@Sun.Com Cc: crossbow-iteam@Sun.Com Message-id: <4477AE6C.1020300@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-Accept-Language: ar-eg, en-us, en, ar, ar-dz, ar-bh, ar-iq, ar-jo, ar-kw, ar-lb, ar-ly, ar-ma, ar-om, ar-qa, ar-sa, ar-sy, ar-tn, ar-ae, ar-ye X-PMX-Version: 5.1.2.240295 User-Agent: Mozilla/5.0 (X11; U; SunOS sun4u; en-US; rv:1.7) Gecko/20050530 Status: RO Content-Length: 11497 This information is Copyright 2006 Sun Microsystems, Inc. 1. Introduction 1.1. Project/Component Working Name: "Crossbow: Network Virtualization and Resource Management" 1.2. Name of Document Author/Supplier: Kais Belgaied 1.3. Date of This Document: 05/26/06 1.4. Name of Major Document Customer(s)/Consumer(s): 1.4.1. The PAC or CPT you expect to review your project: Solaris PAC solaris-pac@sun.com 1.4.2. The ARC(s) you expect to review your project: PSARC 1.4.3. The Director/VP who is "Sponsoring" this project: Glenn Weinberg 1.4.4. The name of your business unit: OPG 1.5. Email Aliases: 1.5.1. Responsible Manager: darrin.johnson@sun.com,markus.flierl@sun.com 1.5.2. Responsible Engineer: crossbow-core@sun.com 1.5.3. Marketing Manager: Paul.Steeves@sun.com 1.5.4. Interest List: crossbow-interest@sun.com 2. Project Summary 2.1. Project Description: Crossbow is a network virtualization technology that greatly improves resource control, performance and network utilization needed to achieve true OS virtualization, utility computing and server consolidation. Crossbow will be the foundation for future innovation in network security (DDOS, IDM, etc.), consolidated appliances, and end-to-end resource control. 2.2. Risks and Assumptions: . Increased demand for sharing servers in the data center between multiple clients, while assuring fairness and separation. . OS virtualization continues to be Sun's stategy for achieving server consolidation in the data center. . NIC vendors keep and improve virtualization features on their devices (e.g. Multiple queues, Recieve Traffic Steering) 3. Business Summary 3.1. Problem Area: Networking devices currently are not securely sharable in Solaris. Traffic from multiple zones or virtual domains is all mixed inside the TCP/IP stack, which accesses the same NIC. Although rudimentary sharing is possible at a high level, it doesn't expose separate network device nodes that can be accessed independently or assigned to zones or virtual domains on the same system. Furthermore, the Solaris offers an egalitarian treatment to all packets. It causes critical server to be overwheled by non-critical traffic types, services and virtual systems. It lacks a usable mechanism for expressing flows prioroty and bandwidth guanrantees and limits. 3.2. Market/Requester: Extensive customer and partners input about Crossbow is available in https://ctg.central.sun.com:713//wiki/index.php/Customer_Visits They can be categorized in four types ofd customers and ISVs that are asking for the virtualization and resource management features: . Financial services - Lehman Brothers, Morgan Stanley, Credit Suisse First Boston, Merill Lynch, Thompson, Deutsche Bank, DE Shaw, Goldman, Bloomberg, SwissRE . Large ISPs - Comcast, EarthLink. . ISVs and IHVs Motorola, Veraz Networks, Zeus ... . Enterprise Computing Singapore MoD, Aramco, ... 3.3. Business Justification: Consolidating multiple servers in the data center into a single easy to manage server is at the heart of Sun's value proposition for the Niagara based line of servers that is starting. Virtualization is a key feature in enabling multiple services that were formerly hosted on separate machines, to be securely co-hosted on a single system. Lack of proper virtualization solution, which includes a true answer to the network resources virtualization is an inhibitor for selling T2000s' and future Niagara II and beyond. Sun wishes to win WallStreet customers, and offering accurate control over the network resource allocation is a clear differentiator for Solaris. It directly addresses their needs for deploying multiple revenue generating and free services on the same IT infrastructure. See https://ctg.central.sun.com:713//wiki/index.php/Customer_Visits and http://netvirt.sfbay.sun.com/presos/crossbow_sunlabs-1.sxi 3.4. Competitive Analysis: Many OS virtualization technologies are emerging, some in native mode (Xen) or emulated mode (VMWare). (See "Virtualization is Everywhere" section of PSARC/2006/260 specs) Although all provide a solution to access the networking resource from the virtual domains, none offers fair and controled sharing of those resources between multiple domains. The expected competitive advantage is two-fold: Saving customers acquisition costs and reduction of the total cost of operation in the long run. 3.5. Opportunity Window/Exposure: Virtualization is at the growing stage of its hype cycle, Systems (Sun, IBM, HP, VMware) and Nic vendors (Intel, Neterion) are all announcing virtualization-ready NICs in the next two years. 3.6. How will you know when you are done?: Demonstrate the ability to consolidate multiple small servers into a single CMT multi-zoned or one with multiple Xen instances, while offering the same level of separation, and without performance regression. Show case actual control of the bandwidth allocation between flows based on services, virtual domains, or traffic types. 4. Technical Description: 4.1. Details: 4.1.1 Access to advanced NICs capabilities The Nemo interface is extended to expose advanced NIC capabilities that are related to traffic separation and that control the NIC interrupts. This includes . the ability to reserve and poll receive rings individually. . an interface for programming the NICs hardware classifier to steer incoming traffic to different rings, based on the packet's flow. . interface for using of multiple transmit fifos. . toggling interrupts per ring when MSI/X is supported. 4.1.2 NIC Virtualization A layer is added to the control path allowing the creation of multiple virtual NICs on top of the same physical MAC device. A VNIC acts like any regular MAC device, it can be assigned to a zone, or a stack instance, IP interfaces can be plumbed on it, or it may be assigned to a virtual machine with a separate address space co-existing on the host. When hardware classification is available, Hardware rings may be assigned to the VNIC, and the classifier is programmed to steer incoming packeets to that VNIC's flow. Alternatively, the classification occurs in software. 4.1.3 Flows Sets of packets sharing common criteria are called 'flows'. The criteria may be any subset of the L2, L3 and L4 headers. A flow belongs to a MAC entity (Physical NIC, VNIC or aggregation) Flows are defined for two purposes: . Proactively, to differentiate the treatment for incoming and outgoing packets that meet the flow's criteria. . Reactively, to contain surges of offending traffic of a some type. 4.1.4 Resource Control Bandwidth limits, shares, and guarantees may be assigned to a single flow, or to a link. For the incoming path, bandwidth allocation enforcement is based on switching off the interrupts from the physical NIC, then periodically polling packets off the receive rings at the right rate. Controlling the outgoing bandwith involves blocking user processes attempting to write beyond the allotted amount of bytes per period, and resume their execution when allowed. Additionally, relative a priority is defined between flows. 4.1.5 Administration The project will deliver administrative interfaces for - creating/deleting and changing attributes of VNICs and flows - extensions to zonecfg to include the creation of VNIC for the zone, its priority and bandwidth allocation. - extension to existing tools for reporting statistic and historic usage per VNIC and flow. The new admin tool will be based on an API for VNIC and flow management. The API will offer the ability to sign-up for notification on events pertaining to the flows (e.g. surges, extended dropping or blocking, etc ...). 4.2. Bug/RFE Number(s): N/A 4.3. In Scope: Integration with intrd2.0 Integration with Stack Instances. 4.4. Out of Scope: GUI for the administrative interface. 4.5. Interfaces: . SPI (Nemo based) for classification capabilities . New and modified CLI for the admin interface . API for flows manipulation. 4.6. Doc Impact: . dladm(1M), ifconfig(1M), flowadm(1M), zonecfg(1M), netstat(1) . Writing Device Drivers. . Solaris Administrator guide 4.7. Admin/Config Impact: See 4.1.5 4.8. HA Impact: No new requirements, however the ability to provide priority and bandwidth guarantee can be leveraged by the SunCluster's heartbeat traffic. 4.9. I18N/L10N Impact: N/A 4.10. Packaging & Delivery: The product's features will be delivered in the Core distribution. 4.11. Security Impact: The new privileged actions will require the appropriate privilege, and will generate audit record. 4.12. Dependencies: PSARC/2006/248 Nemo MAC-Type Plugin Architecture PSARC/2006/249 Nemo Changes for Binary Compatibility PSARC/2006/265 Multiple MAC address support 5. Reference Documents: http://www.opensolaris.org/os/project/crossbow http://crossbow.eng http://sac.sfbay.sun.com//PSARC/2006/260/inception.materials/specification 6. Resources and Schedule: 6.1. Projected Availability: Q2 CY07 6.2. Cost of Effort: 9 person.year for dev 1 person for 3 month for Doc 3 persons.year for QA 6.3. Cost of Capital Resources: N/A 6.4. Product Approval Committee requested information: 6.4.1. Consolidation or Component Name: ON 6.4.3. Type of CPT Review and Approval expected: Standard 6.4.4. Project Boundary Conditions: N/A 6.4.5. Is this a necessary project for OEM agreements: No 6.4.6. Notes: N/A 6.4.7. Target RTI Date/Release: onnv, with a possible back-port to the soonest s10-update possible 6.4.8. Target Code Design Review Date: Q1 CY07 6.4.9. Update approval addition: Not yet. 6.5. ARC review type: Standard 7. Prototype Availability: 7.1. Prototype Availability: April 2006 7.2. Prototype Cost: 4 people for 2.5 month.