> Inception: BrandX: Support for non-native zones (2005/471) > Submitter: Nils Nieuwejaar > Owner: Ed Gould > Intern: Alan Hargreaves > > > SUMMARY > ======= > * Need spec - Full list of things that will keep this project from being LSB > compliant. This is still in progress. The list will be available prior to putback, and it will be made publicly available on the OpenSolaris web site. > * Need spec - List what features are not going to work and what are. See the 'what_works' document included with the materials for this case. All of the information in this document will be reflected in the lx(5) man page and in the 'lx' brand section of the "System Administration Guide: Solaris Containers--Resource Management and Solaris Zones" > * Admin issues - More precise boundaries of what you can do and what you > cannot do would be better for everyone to understand (list apps for > example or a one paragraph explanation) Covered in gcs-0b. > * Man page clarification for wes-3 Done. See wes-3. > * Need spec - list how system calls work Done. See gcs-2 and Section 3.4 of the design document. > * fix the init(1M) invocation Done. See eg-4. > * ARC members: go back and take a look into the Janus case again > > > ISSUES > ======= > wes-1 20q9: can you explain in a little more detail how the > brand-specific hooks in mdb are plugged in? > do you have to run mdb in a solaris-branded zone or a > linux-branded zone? if the latter, how does it coexist? > if the former, how does it figure out which brand library > to load? > > * mdb would only run under Solaris brand but others can run Solaris to > run mdb. > - This should be documented as well in what works and what doesn't Section 3.9.2 of the design has been expanded to include these details. > wes-2 brandz zones and networking: > > will there be any supported/documented configuration which > works in cases where the system (e.g., standalone developer's > laptop using typical ISP) only has one externally reachable ip > address? (alternatives here include sharing ip address & port > space with global zone; NAT; ...?) > > * This would be able to show you how to run and be supported in the global > zone. > * If this was a commitment review, Advice to the PAC - support for global zone We have had extensive discussions with the networking group about providing this functionality, and it is their contention that this cannot be achieved until they deliver 'per-zone stack instances'. This project does not have a formal plan yet, but it would certainly not be available until at least late this year. There is an unsupported mechanism for achieving this functionality today, which has been documented in blog entries by engineers on the BrandZ and networking teams. This is absolutely not a production-ready solution and we do not propose to make this part of any official documentation, but it might be a usable workaround for the development environment discussed in this issue. > wes-3 non-solaris zones, smf, and zoneadm boot -s? > "Boots only to milestone svc:/milestone/single-user:default" > if the lx zone doesn't have SMF, ??? > or are you making the lx brand bits SMF-aware? > > * Booting a linux zone usually works. This may be a man page clarification. The "-s" option is passed through to the zone's init(1M) routine. We just got lucky in that both Solaris and Linux interpret this as "boot to single-user mode". The Zone Boot Arguments case (PSARC 2006/269) deprecates the use of this flag, and introduces a mechanism allowing brand-specific boot arguments. The lx brand's boot arguments will be described on the lx(5) man page. > wes-4 NIT: zonecfg(1m) starts with a list of properties. > I would have expected to see > > (global) brand > > in there. Fixed > wes-5 NIT: zonecfg(1m): I would expect to see an example of setting "brand" Fixed > wes-6 NIT: zones(5): "The software may include Solaris software > configured our laid out differently" > s/our/or/ ? Fixed > wes-7 NIT: zonecfg(1m) doesn't document the "create -B brand" > syntax found in design_doc.pdf page 1 Fixed > wes-8 NIT? design_doc.pdf page 2: %Z or %R for zone root path? > > * Project team will check this. %R. Fixed in the doc. > wes-9 design_doc.pdf page 4: when to run "preboot" is obvious, but > how does the framework know when to run "postboot" - i.e., > what event in the zone boot sequence triggers it and what > is the postboot script guaranteed is set up? > > * This has been changed to a single boot - there is no pre-boot and post boot. The design doc has been updated to reflect this change. > wes-10 design_doc.pdf 2.1.5: interposing on kill may not be sufficient > (e.g., kernel-originated signals?) > > * Project team will go back and look at this. This should also be documented. The Linux documentation is silent on whether the Linux kernel can in fact send unhandled signals to init(1M), so we have chosen to interpret the limitation as one imposed on the sender of the signal rather than on init(1M). We prevent Linux processes from sending unhandled signals to init(1M), but place no restrictions on the kernel or on processes running in the global zone. The design doc has been modified to reflect this interpretation. > wes-11 design_doc.pdf page 5 (and perhaps elsewhere): text is clipped > on the right end of the line? I think I can guess what it's > saying but can't be sure.. > > * This has already been fixed. This was an issue with the process used to convert the web-based design doc to a .pdf file. > wes-12 design_doc.pdf page 6: "it again demonstrates that the ability > for third parties to develop radically new brands will be limited"? > unless they use the opensolaris dev process to request > integration, right? > > * People can develop and send it to the project team if needed. Clarified in the design doc. > gcs-0) transition from Janus: How does this project address the > deficiencies noted in the Janus opinion? What does it give up > that Janus provided? (And I assume that the intent is to have > this project supercede Janus -- right?) > > * The project team is based on the zone model right now. What the project > team loses through this is running Linux and Solaris running zones side > by side. The project team does own these zones. There will be some stuff > not accessible such as D-Trace inside the zone. This project does supercede Janus. This was originally noted in the "Reference Documents" section of the onepager, but I moved the notice up front to the "Project Summary" section. The key difference between BrandZ and Janus is that BrandZ is a zones-based project and Janus was not. From that one change, the following benefits fall out: - We can easily support multiple brands simultaneously - No need for 'pathmapping' to handle different device models, /bin directories, etc. - Simplified administrative model, since we don't need two versions of /etc/passwd, /etc/netmasks, and so on in a single zone - Easy to support non-Linux brands, including Solaris variants - Can deliver a more complete Linux emulation - Buggy Linux applications are isolated from Solaris environment, reducing security risk - Simplifies integration with layered products. For example, Solaris Container Manager can manage Linux zones, SunCluster can support failover between Linux zones, etc. The most obvious disadvantage of this approach is that we cannot run Solaris and Linux applications "side by side". Two other disadvantages are that users must have at least a rudimentary understanding of zones to run Linnux applications, and those zones have to have resources (specifically IP addresses) assigned to them. > gcs-0a) messiness and incompleteness of emulation: How will we know > that the emulation is good enough? When something doesn't > work, how will support be handled? (Does the design include > provisions for helping assign blame?) > > * When something doesn't work, there will be a log message if something doesn't > work or is not supported. If for some reason there is a feature that is not > supported, again there will be a message displayed. > * The project team is running a bunch of Linux applications and trying to be > LSB compliant. However, there are things that run on LSB that does not run on > Linux. > The source is opened to the community (OpenSource) and so the code is public. There are two different failure modes for intentionally unsupported Linux features. First: if an application executes a system call we don't support, we display an explicit error message and return an error. Sample error messages: unimplemented syscall #101 (ioperm): No equivalent Solaris functionality unimplemented syscall #86 (swapon): Reads/modifies Linux kernel state clone(2) only supports CLONE_PARENT for threads. lx_ioctl(): unsupported linux ioctl For unsupported ioctl()s we return EINVAL. For all other unsupported operations, we return ENOTSUP. The other failure mode occurs when a Linux application tries to use some unsupported device or /proc feature. In both cases, the application will fail to find the special file it is trying to open. To evaluate the completeness of the emulation, we are using the LTP and LSB Linux test suites. We are also running the 'connectathon' test suite inside a Linux zone to validate our NFS support, and have modified the zones test suite to exercise the Branded Zones functionality. Finally, BrandZ has been available through OpenSolaris for 6 months, and we have a fairly active user community which has subjected BrandZ to a variety of applications. > gcs-0b) administration issues: Sketch out how attempts to use Linux > administrative tools in a lx-brand zone will play out. What > will work? What won't? > > * Anything that doesn't try to touch real hardware should work. > - ping does however work in the zone (project team will go and look into > this) > - More precise boundaries of what you can do and what you cannot do would be > better for everyone to understand (list apps for example or a one paragraph > explanation) Administrative tools that manipulate files, processes, users, and packages will work. You can use redhat-config-services to start/stop and enable/disable services. useradd/userdel will add and remove users. 'yum' and 'up2date' can determine the status of packages inside the zone, and can connect to CentOS/RedHat to upgrade those packages. Administrative tools that manipulate hardware will not work. You cannot change the zone's IP address or routing information. We don't support disk or serial devices, so fdisk and local printer administration tools will not work. Some networking tools fall into a gray area. ifconfig cannot modify the network configuration, but it can report it. Similarly, route cannot modify the zone's routing tables, but it can display them. Should the stack instances project allow zones to modify their own network configuration, we could implement the Linux ioctl(2)s needed to enable this functionality inside a Linux zone as well. > gcs-1) brand-specific librtld_db.so: delivered in brand-specific > package? Installed where? (If in the branded zone itself, is > there risk of conflict with a native file of the same name?) > > * Already discussed. See wes-1. > gcs-2) emulating complex system calls (section 3.4.3): So how _are_ > they emulated? Do you introduce new (non-branded, native > Solaris) system calls and do the dirty work in the kernel? > > * This might not be embedded in the design document right now. The project team > is introducing a single system call. > - What kind of machinery, etc. should be listed...otherwise it will be hard > to find out how this is working The short answer is that we introduce a single system call, brandsys(), which each brand is free to use however it wishes. The most complex system call we have is clone(2), which we implement as follows: 1) In the brand library, we prep the necessary state for the new thread. 2) We create a new thread with thr_create(), and start it running our clone_start() routine. 3) The new thread calls brandsys(LX_SYS_clone, , , ...) 4) The common brandsys() code extracts the process's brand from the proc_t structure and calls the brand-specific brandsys() routine. 5) The brand kernel module allocates a new PID for the thread and initializes some other kernel state. 6) The brandsys() syscall returns 7) The new thread returns from our clone_start() to the thread- creation code inside of glibc() The full details of this system call can be seen in section 3.5.1 of the design document. > gcs-3) interface table is way too sketchy; will need to be fleshed out > for commitment (e.g., uucopy() isn't mentioned in it, but > should be). Done. See 20q13. > gcs-4) returning from signals (section 3.6.2): Are you saying that > non-branded Solaris processes will experience a change in > behavior due to changing how %gs is restored? If so, is there > any way to avoid the change? We have made this behavior selectable on a per-process basis. Solaris processes will get the old behavior and Linux processes will get the new behavior. The design doc has been updated accordingly. > eg-0 > There are several places in the design where the authors are vague > about actual Linux behavior. Should we expect further inception > review(s) as these details are investigated? > > * Nils: I don't think so - it might just be text that is lingering that needs > to be cleaned up perhaps. However, where ever possible we track down where > Linux is running but we cannot look at that code so this is how we document > what it may be doing. > - Possible chance to ask externally about what Linux is doing? > - Tracking down what Linux could be doing regarding this project will be > better The document has been cleaned up. Most of the vagueness was in text that was written during the early design phase, as we were still identifying issues to be resolved. > eg-1 20q6 > It strikes me as folly to integrate only the x86/x64 code. How can we > ensure that the SPARC code actually works, and continues to work, if > nothing is integrated with which to test it. There may be a business > case that says not to support this functionality, but it does not seem > architecturally sound to leave it out. > > * Nils: If we put back the SPARC code, it is potentially dead code. This > is more or less frowned upon. If the ARC would like us to put back SPARC > support, we'd be willing to do that. > * In the kernel nothing is 32 specific. Basically, a tiny fraction is 32 > specific. Assuming we get C-team approval, the SPARC code will be putback. We will include a fake "solaris 10" brand, which can be used to exercise the framework on both SPARC and x86. The zones test suite will be extended to include tests of this "solaris 10" brand. The 20 questions doc has been updated accordingly. > eg-2 20q10 > Why is a non-Linux tool being introduced into a Linux zone? Shouldn't > everything inside that zone just be Linux? > > * ptchmod is an existing tool in Solaris right now. NFS is working right now > but locking doesn't. The Solaris tool (ptchmod) originally mentioned is not going to be needed. We are going to have to run two Solaris daemons in the Linux zone to support NFS locking. The lockd and statd daemons run in userspace, but are very tightly integrated with the kernel. There was no feasible way to support NFS locking without running the daemons in the kernel, and we decided that maintaining data integrity was more important than the purity of our branded zone model. The ugly details are all in section 3.8 of the design doc. > eg-3 20q15 > The answer suggests that multiple versions of the Linux syscall > interface will *not* be supported. Does this mean that users may not > retain an old version of a Linux zone if a Solaris update contains a > newer Linux interface? Doesn't this put customers into an unnecessary > bind? (Consider NetBSD - it has compatibility with all previous > syscall versions, back to the start of [its] time.) > > * It might be a suggestion to do your system call early. > > > [observation from sommerfeld: actually, NetBSD's compatibility is via > kernel-compile-time options -- COMPAT_x for various values of > x -- which on solaris are typically handled via boot-time or > run-time configiration]. The answer to this question was poorly worded and insufficient. The document has been updated. The short version is: we expect that the 'lx' brand kernel module can be extended to support future Linux releases without breaking support for current releases. We should be able to support both RHEL-3 and RHEL-4 simultaneously with a single kernel module. If we do move forward with a RHEL-4 brand, we will absolutely support both RHEL-3 and RHEL-4 zones on the same system. > eg-4 Design 2.1.5 > Why not just fix the init(1M) invocation now? Without knowing the > details, I would expect it to be fairly easy to do, and not doing so > leaves a large hole in the architecture. > > Also, does Linux auto-reboot when it discovers that init has died? > BSD systems do. > > * The project team can make the change if needed right now. After we prototyped this in our gate, the core zones team found this functionality interesting enough that they decided to integrate this themselves. This will be part of their Boot Arguments (PSARC 2006/269) putback. > eg-5 Design 2.2.1 > The new struct brand appears to contain enough information to support > multiple versions of a given brand simultaneously, yet the text seems > to forbid this. Why? Isn't the (name, version) pair sufficient to > identify the underlying implementation? > > * Already discussed The versioning described in this section is not intended to detect changes between different releases of the same brand. The versioning described here is intended to ensure that the brand module being loaded is compatible with the two 'brand op' structures defined by the BrandZ framework. As to the question of whether we can support multiple versions of a brand simultaneously: there should be no reason to do so, just like there is no reason to support multiple versions of the UFS module. A single instance of the 'lx' brand should be able to support multiple Linux variants. If/when we modify the 'lx' brand to support RHEL4, it will be a requirement that we maintain backwards compatibility with RHEL3. > eg-6 Design 3.5.1 > What happens if a Linux process calls clone() with an unsupported set > of flags? > > * 2 of the 4 are supported and should be detailed. Fixed in the design doc. > eg-7 Design 3.7.1.1 > The text suggests that the investigation of actual Linux behavior is > incomplete. Is the proposed model that networking should work from a > branded zone, but control of network devices (including such things as > IP address assignment), using branded tools, will not? > > * Already discussed See eg-0 and gcs-0b. > eg-8 Design 3.7.3.1 > Does the driver name-to-major mapping ever need to be redone at > Solaris boot? > > * This is checked dynamically when needed. the checking can be done any time. > - Doc should be cleaned up to specifically say when it it is redone. Fixed in the design doc. > eg-9 Design 3.7.4 > Is /dev/null really write only in Linux? It has always been > read-write in Unix (with read returning EOF). > > * This will be changed. It is now r/w and the design doc has been changed. > eg-10 Design 3.7.5.3 > There are uses of /dev/fd other than in shell scripts. One > common use is on the command line, when a program does not, by > default or conventional argument (e.g., "-"), read standard > input. It can then be invoked with an explicit /dev/fd/0 > argument. > > * This should be included in the design doc. Fixed in the design doc. > eg-11 Design 3.7.5.4 > What if a branded zone wants access to multiple sound devices? Is the > proposed mechanism extensible in that way? > > * You can only have one audio device in the zone. We cannot support more > than one audio device in a zone. You would need to create another zone > for more than one. The design doc has been updated to show that it would be technically possible to support multiple sound devices, but that this functionality would complicate the implementation and the administrative model. We haven't gotten any customer feedback that this is a desired feature, so it is not included in this release. > eg-997 Design 3.9.2 (typo) > Email cruft ">From ..." Fixed. > eg-998 Design 3.9.1 (nit) > Note that multi-threaded debugging in Linux has frequently been > broken, and may be broken now. It will undoubtedly break again in the > future. This has turned out to be an unsolvable problem, given our time constraints for this release. It may turn out to be unsolvable, full stop. The design doc has been updated to reflect this. > eg-999 Design 3.7.5.2 (typo) > "user and" -> "userland" Fixed > dwc-1 20Q, Q1, "By what criteria will you judge its success?" & Q17 > I see no reference to the LSB in your materials. Shouldn't we > require that the LSB test suites pass in an lx branded zone? > > * Already discussed. See gcs-0a. Also, Q1 has been updated to list the specific tests we will be running, to indicate that we do not expect to be LSB compliant, and that we will provide a list of all the issues preventing compliance. > dwc-2 NIT: zones(5): Brands heading: 1st paragraph > "configured our laid" -> "configured or laid" Fixed. > kb-1 The name "lx" leads to believe this is a generic brand for > Linux. Yet it seems that what the project id delivering is > a brand specific to Red Hat Entreprise Linux 3. > So, will future brands for other flavors of Linux also be > named 'lx' and differentiated with the int b_version? > > * The brand name is "lx". You could probably support RedHat 3 and 4 using > the same brand but not quite sure right now. The next release of Red > Hat might have the same brand name but again, it could change. > - There has been no pushback from marketing about using the word > "brand" in naming things. The project team has already gotten sign off > from marketing and from legal on this. See eg-3 and question 15 in the 20 questions doc. The 'lx' brand is our vehicle for delivering Linux functionality, and we believe that it can be extended to support both RHEL-3 and RHEL-4 simultaneously. If there is a need to distinguish between RHEL3 and RHEL4 behavior in the same machine, that would be indicated by a flag in the per-process brand structure, rather than in a per-brand data structure. > kb-2 About section 2.2.1 "The b_version number is currently used to > determine whether the brand is eligible to be loaded into the > system. If the version number doesn't match the one compiled into > the kernel, then we simply refuse to load the band module" > Any way to know up front the (list of) brand(s) that a the kernel > has built0in support for? I'm thinking about the case where I want to > order new brand modules or newer versions of the lx, and want to > know beforehand if they'll get installed sucessfully or not. > > * The project team will this about this. Brands will be delivered into /kernel/brand. You can test whether a brand is compatible by doing a 'modload /kernel/brand/'. If it isn't compatible, the operation will fail. We don't plan to make the version number visible externally. If a third party wishes to distribute a brand, they will have to make a statement as to which versions of Solaris will support it. > kb-3 I am not clear on the upgrade model and constraints for applications > running within a BrandZ emulated environemnt. > Example, say I set up an 'lx' band (based on RHEL 4) in a zone, and > install a few applications there, and deploy that in production > and populate some state in that zone. Later, I need to upgrade > one of the application, and the new version would nornally > requires an OS upgarde to RHEL 5. > In a non-BrandZ enviroenment it is just an OS upgrade, everything > is expected to be compatble. > For a BrandZ environement, how would I achive that upgrade, given > the fact that a zone's brand cannot be changed (according to > section 2.1.1 Configuration) ? > > * We do not intend to upgrade Linux and zones. Nothing like a live upgrade will > be done. > * Maybe a best practice doc should be written up about this issue from the > Linux point of view. We have added some text to question 15 in the 20q document to address this issue. The short version is that we do not plan to support the upgrade of zones, and will provide a 'best practices' document. That said, it is worth noting that we have successfully upgraded a Linux zone from CentOS 3.1 to CentOS 3.6 using the CentOS on-line upgrade tool. We may well find that upgrading to a 4.x or 5.x release is something we can support. Since we cannot be sure of that now, our initial stance has to be that this will not be supported. As to the specific question about not changing the zone's brand: it is our expectation that future Linux releases will be implemented by adding functionality to the 'lx' brand, so there would be no need to change a zone's brand to perform an upgrade. > gw-1 Nit: please communicate with the Layered Trusted Solaris team > (PSARC/2002/762). There seems to be overlap in zoneadm and > zonecfg. Perhaps this case's architecture of config.xml > is also applicable there. Contact Glenn Faden. Trusted Solaris putback before this communication happened. We have resynced with their changes, and can comfortable co-exist. > gw-2 Would it be useful to extend ucred to return the brand? > Would a network or door server ever be interested? > > * Project team will think about this. We still haven't found an occasion where this was required. It could certainly be added if the ARC thought it was a good idea, but it hasn't been necessary yet. > On white board: > --------------- > eg-12 syscall vector > > * Tied with eg-3 We discussed this issue further by email. The optimization that was discussed and drawn on the whiteboard was based on a misunderstanding of the BrandZ interposition mechanism, which in turn was caused by a deficiency in the design document. Section 2.2.2 of the design doc has been expanded and clarified, to more thoroughly explain this mechanism.