This information is Copyright 2005 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: BrandX: Support for non-native zones 1.2. Name of Document Author/Supplier: Nils Nieuwejaar (nils.nieuwejaar@sun.com) 1.3. Date of This Document: 08/11/05 1.4. Name of Major Document Customer(s)/Consumer(s): 1.4.1. The PAC or CPT you expect to review your project: Solaris PAC 1.4.2. The ARC(s) you expect to review your project: PSARC 1.4.3. The Director/VP who is "Sponsoring" this project: Barry Cooks 1.4.4. The name of your business unit: Solaris Core Technologies 1.5. Email Aliases: 1.5.1. Responsible Manager: Jerri-Ann Meyer 1.5.2. Responsible Engineer: Nils Nieuwejaar 1.5.3. Marketing Manager: Matthew Baier 1.5.4. Interest List: brandx-interest 2. Project Summary 2.1. Project Description: BrandX is an extension of the zones infrastructure that allows the creation of zones that emulate non-native operating system environments, such as Linux, FreeBSD, or older versions of Solaris. Each type of environment supported is referred to as a "brand". The initial release of BrandX will include a single brand: Red Hat Advanced Server. This brand comprises the user-visible feature referred to as the Linux Application Environment (LAE). Future brands may include other Linux versions and/or distributions or older versions of Solaris. The OpenSolaris community has expressed an interest in adding support for a FreeBSD zone. The goal of the project is to enable users to run applications that are compiled for a different OS; the goal is not for a branded zone to deliver the full functionality of that operating system. So, a Linux zone should be able to run acroread, Websphere, Mathematica, and so on. It will not function as an NFS server, support ext2fs, or run Quake. 2.2. Risks and Assumptions: 2.2.1. Risk: Compatability Likelihood: High We will not be able to provide 100% compatibility with any Linux environment. Applications that work correctly on Red Hat may fail, either spectacularly or quietly, when run under LAE. Mitigation: We can run a full complement of Linux test suites, document areas in which we are known to be incompatible and provide diagnostic information when asked to perform an unsupported operation, but there are apt to be more subtle incompatibilities as well. 2.2.2. Risk: Support costs Likelihood: High Since a brand is in many ways a new operating system, fully testing and supporting any brand will require considerable time and resources. The sustaining organization has estimated that costs will be comparable to that of maintaining Sun Cluster, which is roughly $1M per year. Mitigation: Marketing will survey customers, SEs, ambassadors, etc. to ensure that we select the proper brands to support. Selecting the right brands will allow us to satisfy the most customers while minimizing the number of brands that must be supported. 2.2.3. Risk: Product overlap Likelihood: Medium The user-visible features of this project has significant overlap with the Xen project, which is expected to be available at roughly the same time. Both projects provide a way to run non-native applications on Solaris. Xen provides a fully virtualized machine which will run a complete OS instance, while BrandX simply provides an environment for foreign binaries to run within a zone. There are distinct tradeoffs to each solution which will dictate which is appropriate for a given customer environment, but the superficial similarity will cause confusion among sales people and customers. Mitigation: work with Marketing to develop a clear and comprehensive description of our virtualization strategy, including Zones, BrandX, Xen, and Resource Management solutions (e.g., memory sets and swap sets). Coordinate messaging among the teams to ensure that all these projects are consistently represented in roadmaps presented internally and externally. 2.2.4. Risk: Reduce ISV incentive to port Likelihood: Medium Availability of a reliable Linux emulation environment on Solaris may reduce ISV's incentive to port applications to Solaris. Mitigation: We view LAE primarily as a transition tool for customers migrating to Solaris. We do not anticipate ISV's officially supporting their applications in this environment. Note: LAE may also reduce a customer's incentive to port a homegrown application from Linux to Solaris. We don't view that as inherently bad, as they will be running on Solaris in either case. 2.2.5. Risk: Potential 'sabotage' from Linux vendors Likelihood: Low Red Hat could deliberately try to break our compatibility. Linux applications are written to the interface provided by glibc and other libraries, while we emulate the interface between glibc and the kernel. By modifying that interface, Red Hat could break our emulation without affecting their application compatibility. Mitigation: In addition to testing each new release from Red Hat, the sustaining organization will monitor their patch stream and correct for any incompatibilities. 3. Business Summary 3.1. Problem Area: A large number of applications are available or being developed for Linux. Many of these applications can or will not be ported to Solaris in a timely manner, hampering its market adoption. 3.2. Market/Requester: Any customer who is considering deploying Linux and those currently using Linux but wishing to migrate to Solaris. 3.3. Business Justification: Linux is generally considered to be building market share in the server segment particularly at the low end. This popularity has resulted in many ISVs publishing software for Linux either before or rather than for Solaris. 3.4. Competitive Analysis: Competition comes from the main commercial Linux distributions such as RedHat and SuSE who work with hardware vendors such as IBM and HP to provide Linux servers. Red Hat: http://www.redhat.com/ Suse: http://www.novell.com/linux/suse/ SCO includes a 'Linux Kernel Personality' with Unixware. LKP follows the basic Janus approach of introducing a new 'int80' system call handler and translating Linux to native system calls in the kernel. With LKP, executing a Linux application implicitly does a chroot(2) to the /linux directory. This allows Linux applications to access system filesystems without the pathmapping introduced by the Janus project. Unlike LAE, they do not support running Linux daemons such as httpd, cron, and sshd. Maintaining consistency between the native /etc/passwd, /etc/shadow, and /etc/groups is done by a new daemon in the background. Our zones-based approach lets us run all the standard Linux daemons. Since each zone has its own /etc files, there is no question of maintaining consistency. SCO: http://uw714doc.sco.com/en/Navpages/LKPhome.html FreeBSD also has a Linux Binary Compatibility feature. It is less complete and correct than LAE intends to be. Rather than using Linux libraries and binaries from a known Linux distribution, they provide their own home-grown collection of support libraries. It appears that they are unable to recognize many Linux executables automatically, and require that the end user run a utility that modifies the binary's ELF header to add the supporting information that they need. FreeBSD: http://www.freebsd.org/ Finally, there are projects/products that provide a complete virtual machine environment, which allows the end user to run a full Linux system including the kernel. This approach delivers the highest fidelity Linux support, but limits the flexibility of resource sharing beteen system instances, and often reduces the performance of both the Linux and native operating systems. VMWare allows a native, unmodified Linux environment to run. Xen will run a complete Linux system, but requires a kernel that has been modified to run in a 'paravirtualized' environment. VMWare: http://www.vmware.com/products/vplatform/ Xen: http://xen.sf.net/ 3.5. Opportunity Window/Exposure: Given the popularity of Linux on x86 at the low end of the server market and the need to facilitate Solaris adoption in the X86 space, it is important that this project delivers in a timely manner. As other virtualization technologies mature, increased competition from them should be expected. While the LAE functionality was intended to ship with S10 GA, a design review determined that the approach taken by the original team would have made both LAE and core Solaris significantly harder to maintain over the long term. It was decided that the quality of the project was more important than schedule considerations, so we now expect to ship with an S10 update. From experience, we know that many customers choose to wait for at least the first update release before upgrading to a new version of Solaris. So, despite not being available at GA, we believe this will still be a useful tool for increasing the S10 adoption rate. 3.6. How will you know when you are done?: It is anticipated that the project will deliver in phases. There are two key metrics that the project success is measured by: - performance of programs compiled for Linux and executed in the BrandX environment must be within 5% of the same program when executed under Linux directly and on identical hardware. - it is possible to install and run a defined set of Linux applications (definition to occur early in project life cycle) 4. Technical Description: 4.1. Details: This project consists of two parts: the BrandX framework, which allows for plug-in support of new brands, and an initial brand that supports the execution of Linux applications. 4.1.1 BrandX Framework The framework extends the zones infrastructure to include a brand 'attribute', which identifies a zone as providing a non-native execution environment. A brand may also provide scripts to be automatically executed before and after zone creation or zone boot, to allow for any specific setup operations to be executed. The brand framework introduces a number of interposition points within the kernel, allowing a brand to supplement or replace standard Solaris functionality. The most important of these interposition points is in the system call handler. When an application enters the kernel via a system call trap, we check the process's proc_t structure to see whether the process has been branded. If so, control is passed to that brand's system call handler. If not, control continues along the standard system call path. Other interposition points will be placed in the code paths for process creation/exit, thread creation/teardown, signal delivery, and so on. 4.1.2 'lx' Brand The brand we will be delivering with the framework allows for the execution of Linux applications. This brand is specifically designed to support the kernel and glibc versions that form the base of the Red Hat Advanced Server distribution. This brand's ability to support applications built for other Linux distributions will be limited by how far from RHAS those distributions diverge. If necessary (for technical and marketing reasons), we will deliver variant brands that support those other distributions. 4.2. Bug/RFE Number(s): None (yet). 4.3. In Scope: - Add Dtrace providers to allow probing of Linux applications - Add support to mdb and Solaris /proc tools to allow debugging of Linux applications 4.4. Out of Scope: - Generating Linux core files on an application crash. The resulting core file will be in the Solaris format and only debuggable with Solaris tools. - Running non-native applications in the global zone. - Running native Solaris applications in a branded zone. - Accessing Linux filesystems - Supporting Linux device drivers or kernel modules - LSB compliance. We will be using the LSB compliance test suite to identify problems, but we do not anticipate achieving full compliance. It is worth noting that the latest Red Hat server release is not LSB compliant, and it is not clear that it ever will be. Clearly we cannot be more compliant than our underlying Linux distribution. - The initial release of LAE will not support 64-bit applications. This functionality is expected to be provided in a subsequent release. 4.5. Interfaces: This project introduces two new interfaces for brand developers and extends one interface that is visible to users. 4.5.1. User Interface This project extends the existing interfaces for managing zones (PSARC 2002/174) by adding a new option to the zone creation CLI: The 'create' command will accept "-B ", which will set the brand attribute on the zone. In addition, the 'zoneadm list' output will be modified to display a zone's brand. 4.5.2. Zones interfaces This project introduces the notion of per-brand install, pre-boot and post-boot scripts. These scripts allow the brand to dictate which software is installed in the zone, and to modify the default zone configuration to establish the correct environment for the brand. The initial Linux brand makes use of all three of these script hooks. lx_install: - Extract Linux software from customer-provided Red Hat install media and installs it in the local zone's root directory. - Copy the necessary Solaris components (ld.so.1 and brand library) into the local zone - Modify the vendor-provided startup scripts to prevent low-level hardware-related tasks (e.g., downloading CPU microcode, fscking disks, etc). from executing lx_preboot: - Ensure that the supporting kernel module is loaded. lx_postboot: - Create a FIFO for communicating with init(1M) The project also moves certain zone attributes, such as which devices are exported and which filesystems are mounted, into a per-brand XML configuration file. Each brand, including native zones, will provide a configuration file in /usr/lib/brand//config.xml containing these settings. 4.5.3. Kernel Interfaces To support the emulation of foreign operating systems, we must have a way for Solaris to redirect control to the emulation software at certain points. The most obvious place is at the system call interface, but there are a number of others as well. At these specific points, we allow a brand to interpose on the standard Solaris actions, replacing them with brand-specific behavior. This interface is exported to kernel modules through an ops vector, similar to that used for VFS or VM segments: struct brand_ops { long b_version; int (*b_attach_proc)(struct proc *); void (*b_detach_proc)(struct proc *); int (*b_brandsys)(int, int64_t *, uintptr_t, uintptr_t, uintptr_t, uintptr_t, uintptr_t, uintptr_t); int (*b_clearbrand)(struct proc *); int (*b_setbrand)(struct proc *); void (*b_copy_procdata)(struct proc *, struct proc *); void (*b_proc_exit)(struct proc *, klwp_t *); void (*b_setregs)(struct uarg *); void (*b_initlwp)(klwp_t *); void (*b_forklwp)(klwp_t *); void (*b_freelwp)(klwp_t *); void (*b_lwpexit)(klwp_t *); int (*b_sendsig)(int, k_siginfo_t *, void (*)()); void *padding[18]; }; This ops vector is sufficient for the initial Linux brand. Adding support for a new Linux distribution based on the 2.4 kernel can likely also be done without modifying this interface. It is likely that adding any new brand that is substantially different from the initial Linux brand will require additional interposition points. For example, adding support for a 2.6-based Linux distribution could require modifications. Supporting a whole new operating system such as FreeBSD or Apple's OS X would almost certainly require modifying this interface. Since this interface is expected to change, we included a version number to allow the kernel to identify brand modules that make use of an unsupported ops vector. We have also added sufficient padding to allow for substantial future growth without breaking compatibility with older versions. 4.6. Doc Impact: This project will require modification to the following man pages: - zones(5) - zonecfg(1M) - zoneadm(1M) In addition, the zones section of the System Administration Guide will be extended to describe the new brand-related functionality. More specifically material will be added to "Part 2, Zones" in the book: System Administration Guide: Solaris Containers--Resource Management and Solaris Zones. The Solaris 10 Update part number is 817-1592. The Solaris 11 (Nevada) part number is 819-2450. Each brand is expected to include its own manual page, documenting its features and limitations. This project will be delivering the lx(5) manual page. The What's New release document will be modified to include information on this project. 4.7. Admin/Config Impact: This project introduces a single option to the zonefg(1M) create command and will add a column to the output of the zoneadm(1M) subcommand 'list -v'. There are no other changes to system administration or configuration. 4.8. HA Impact: None 4.9. I18N/L10N Impact: None 4.10. Packaging & Delivery: This project modifies the following packages: SUNWckr SUNWcakr.i SUNWcakr.u SUNWcsu SUNWhea SUNWmdb SUNWzoner SUNWzoneu This project adds the following packages: SUNWbrand - Brand infrastructure SUNWlx - The initial 'lx' brand This project will require modifying the zones upgrade infrastructure. Since a branded zone will contain non-standard software, it should not be patched or upgraded in the same way as standard non-global zones. The precise mechanism for upgrading a branded zone is still under investigation. 4.11. Security Impact: None 4.12. Dependencies: Upgrading a branded zone will depend on the functionality provided by the Zones Upgrade project. 5. Reference Documents: The project web page is http://brandx.eng The in-progress design document is at: http://perf.eng/twiki/bin/view/BrandX/DesignDoc This project replaces PSARC case 2003/445: "Janus: Linux binary compatibility for Solaris x86". 6. Resources and Schedule: 6.1. Projected Availability: Q4 CY 2005 OpenSolaris Q1 CY 2006 Beta Q2 CY 2006 SX/Nevada 2H CY 2006 S10 Update 6.2. Cost of Effort: 6.2.1. Initial development cost: All estimates are for the delivery of the BrandX framework and the initial 'lx' brand. - On average 5 development engineers for 18 months - On average 3 QA engineers for 18 months for ongoing QA work and test development - Majority time of 1 Program manager for 18 months - Part time of 1 Marketing product manager for 18 months - 1 documentation writer for at most one month to update the System Administration Guide. - 1 documentation writer for at most one week to update each release's What's New document. - 1 documentation writer for at most one month for manpage changes and additions. 6.2.2. Ongoing sustaining cost: Brandx infrastructure: - One sustaining engineer on an ongoing basis. - Each bug fix costs $32K plus testing. Per-brand cost: - Total cost for each brand is $1M per year. - each bug fix costs $32K plus testing. - For testing the estimate is one sustaining engineer per two Linux releases on an ongoing basis, plus one sustaining engineer per two supported applications on an ongoing basis. Note that major releases such as Oracle 9 to 10 are each counted as a new application. 6.3. Cost of Capital Resources: This project will mostly leverage existing capital resources. Additional equipment is needed for QA, estimated to be 6-8 additional x64 systems. It will probably be sufficient for these additional x64 systems to be a v20z or v40z or v65x. Additional equipment is needed for sustaining, estimated to be 6 additional x64 systems. This estimate will be refined one the specific set of applications is more clearly defined. 6.4. Product Approval Committee requested information: 6.4.1. Consolidation or Component Name: OS/Networking Consolidation 6.4.3. Type of CPT Review and Approval expected: Standard 6.4.4. Project Boundary Conditions: Not yet available 6.4.5. Is this a necessary project for OEM agreements: No 6.4.6. Notes: See dependencies section above 6.4.7. Target RTI Date/Release: See section 6.1 for release target timeframes. 6.4.8. Target Code Design Review Date: Target Design Review is August 30, 2005. Target Code Review timeframe for Nevada integration is end of Q1CY2006 or beginning of Q2CY2006. 6.4.9. Update approval addition: This project is targeting an S10 Update release. 6.5. ARC review type: Standard 7. Prototype Availability: 7.1. Prototype Availability: The work done as part of the Janus project has been released to a small number of selected customers as the "Linux Application Environment technical preview." 7.2. Prototype Cost: No additional cost