1. What specifically is the proposal that we are reviewing? - What is the technical content of the project? The RBridges project has three main components: - IS-IS routing - TRILL encapsulation and forwarding - Ethernet bridging This project builds on "Solaris Bridging" (PSARC 2008/055), "Bridging Updates" (PSARC 2009/344), and "IS-IS For OpenSolaris" (PSARC 2008/352) to provide support for the new TRILL-based "RBridging" mechanism. See the rbridges-arch.png drawing for an overview of RBridge architecture. In this drawing, the colors represent various parts of the architecture: grey existing parts of OpenSolaris green new components being added by this project pink parts from subprojects (bridging) yellow donated upstream to the Quagga repository blue future projects. RBridges use IS-IS instead of Spanning Tree (STP) and differ from traditional bridges in many ways, including: 1. Ability to use multiple, diverse, load-spreading paths through the network. 2. "Fail-safe" operation; lack of signaling means no forwarding allowed, unlike Spanning Tree. 3. Forwarding with hop-count protection against transient loops. 4. Optimal forwarding for all frames; no nodes are disadvantaged on the "wrong" side of a cut, as with Spanning Tree. More information is available on the project web page: http://www.opensolaris.org/os/project/rbridges/ - Is this a new product, or a change to a pre-existing one? If it is a change, would you consider it a "major", "minor", or "micro" change? See the Release Taxonomy in: Project consists of Minor release compatible changes. - If your project is an evolution of a previous project, what changed from one version to another? This project builds on "Solaris Bridging" to provide new functionality. - What is the motivation for it, in general as well as specific terms? (Note that not everyone on the ARC will be an expert in the area.) TRILL is new, and is in active development in the IETF. The three main motivations for implementing it are: 1. The other vendors implementing this new protocol are all major players in the bridge market. No other common operating system yet has support for TRILL. This project will give OpenSolaris a chance to ship something before Linux does. 2. The inventor of RBridges is Sun's own Radia Perlman. Implementing things that Sun invents and advances through the standards process just seems like a good idea. 3. The tunneling mechanism used by TRILL may have applications for future projects, including server consolidation and migration used for cloud computing. - What are the expected benefits for Sun? The three subcomponents of this project are each valuable in different areas: - The IS-IS component will include porting this software to OpenSolaris and testing in normal IPv4 (and IPv6, if possible) routing mode. This means that users will have the option of selecting among IS-IS, OSPF, and RIP for interior routing. - The Ethernet bridging component will allow OpenSolaris-based distributions to do traditional bridging. - The full RBridge functionality will build on the other two components and allow future projects to build on this new technology. - By what criteria will you judge its success? 1. External testing for compliance with relevant standards 2. Reuse within other OpenSolaris projects 3. Demonstrated interoperability with other implementations now in development by other IETF participants 2. Describe how your project changes the user experience, upon installation and during normal operation. - What does the user perceive when the system is upgraded from a previous release? No change on install or upgrade. The user will be able to create bridge instances via dladm administer them, assign physical and virtual connections to them, and configure forwarding areas and rules. 3. What is its plan? - What is its current status? Has a design review been done? Are there multiple delivery phases? The project work has had a design review for the bridging component, and one for the RBridges parts will start soon. The IETF work is still a moving target, but enough bridge vendors have prototype implementations (including hardware) that substantial changes in the core protocol are quite unlikely. The original plan had three delivery phases. Due to scheduling issues with other projects, this has been cut back to two: the IS-IS routing portion was delivered to SFW some time ago, and both bridging and RBridges will be delivered together. The delivery consists of new functionality in the SFW and ON consolidations. The ON portion has no dependency on SFW and will be delivered first. This case covers the full RBridge architecture. Other cases describe the independent details of the subcomponents. 4. Are there related projects in Sun? - If so, what is the proposal's relationship to their work? Which not-yet- delivered Sun (or non-Sun) projects (libraries, hardware, etc.) does this project depend upon? What other projects, if any, depend on this one? This project has relationships to at least the following projects: - Clearview (PSARC 2005/132) Clearview has changed the way network interfaces are named and administered. The administration of bridges builds on Clearview. - Crossbow (PSARC 2006/357) Crossbow substantially changed the way the Nemo/GLD interfaces work. The design of the bridging component (described in 2008055 and 2009/344) is built on the changes from Crossbow. - Volo (PSARC 2007/587 and 2008/694) Volo provides the kernel socket plug-in module mechanism used by the 'trill' module to provide AF_TRILL. AF_TRILL sockets are used by the TRILL daemon to send and receive control messages and to issue ioctls that control nickname (tunnel) forwarding paths. - Solaris Simnet (PSARC 2009/200) The simnet project provides a simulated Ethernet interface that will be useful in testing L2 features (such as aggregations and bridges) that can't be tested using using other virtual technologies. The RBridges automated tests will depend on this feature. By combining simnet and bridging, many other technologies such as NWAM and IPMP will be easier to test without using physical hardware. - Packet interception for the MAC layer (PSARC 2008/249) MAC layer filtering will interact with the bridging component of this project. This will allow filtering of packets being forwarded within a bridge instance in order to create "stealth firewalls." - Are you updating, copying or changing functional areas maintained by other groups? How are you coordinating and communicating with them? Do they "approve" of what you propose? If not, please explain the areas of disagreement. The bridging component of this project is based in part on work originally done independently and enhanced by the xVM project team. That work was abandoned. The Crossbow internal bridging functionality is closely related to (but not quite overlapping with) regular Ethernet bridging; both touch parts of MAC and DLS. We're in touch with the Clearview, Crossbow, Filtering and other teams and will be coordinating closely. 5. How is the project delivered into the system? - Identify packages, directories, libraries, databases, etc. All packages are in SUNWCall and SUNWCXall. Delivered in this project: SUNWtrill (SFW) /usr/sbin/trilld /var/svc/manifest/network/trill.xml Delivered in IS-IS subproject (2008/352): SUNWquaggau /usr/sbin/isisd (In SUNWCall and SUNWCXall.) Delivered in bridging subproject (2008/055, 2009/344): SUNWbridgeu /usr/lib/bridged /usr/lib/librstp.so.1 /usr/lib/rcm/modules/SUNW_bridge_rcm.so SUNWbridger /var/svc/manifest/network/bridge.xml Kernel modules are delivered through SUNWckr. 6. Describe the project's hardware platform dependencies. - Explain any reasons why it would not work on both SPARC and Intel? No dependencies. 7. System administration - How will the project's deliverables be installed and (re)configured? pkgadd / pkg install - How will the project's deliverables be uninstalled? pkgrm / pkg uninstall - Does it use inetd to start itself? No. - Does it need installation within any global system tables? Yes; the trill sockmod module must be in /etc/sock2path. It's added there by i.sock2path in Nevada. OpenSolaris mechanism is unclear. - Does it use a naming service such as NIS, NIS+ or LDAP? No. - What are its on-going maintenance requirements (e.g. Keeping global tables up to date, trimming files)? Internally, the bridging module needs to monitor MAC source addresses on every attached network to build a dynamic list of known destinations. This list is "unimportant" -- if a destination isn't known, the forwarding path simply sends the packet out every available interface. In other words, it's just a cache, and just a performance improvment. As such, the bridging module needs to age these entries away and be able to limit its kernel memory consumption. The mechanisms for this are described in 2008/055 and 2009/344. The TRILL module is controlled entirely by the TRILL daemon using ioctls on AF_TRILL sockets. - How does this project's administrative mechanisms fit into Sun's system administration strategies? E.g., how does it fit under the Solaris Management Console (SMC) and Web-Based Enterprise Management (WBEM), how does it make use of roles, authorizations and rights profiles? Additionally, how does it provide for administrative audit in support of the Solaris BSM configuration? The existing RBAC information will be used for the Quagga components. Bridge configuration is administered through dladm(1M). The existing Network Management RBAC profile will enable control of bridging. Auditing is through the existing common dladm mechanisms. The bridging and TRILL daemons may be controlled by the administrator for special purposes using svccfg(1M) and svcadm(1M). These parts will not be documented, as the primary administrative interface is dladm(1M). - What tunable parameters are exported? Can they be changed without rebooting the system? Examples include, but are not limited to, entries in /etc/system and ndd(8) parameters. What ranges are appropriate for each tunable? What are the commitment levels associated with each tunable (these are interfaces)? There are an array of tunables accessible through dladm(1M) as both link properties and as bridge configuration parameters. See 2008/055 and 2009/344 for details. These tunables correspond to IEEE management interfaces. All can be changed without rebooting the system. The SMF instances for bridges and for TRILL each have tunable parameters. These are normally administered via dladm(1M), so they're not documented for ordinary use. In addition to the tunables controlled by documented interfaces, there are two uncontrolled ones for bridging (config/debug and config/table-maximum) and one for TRILL (config/nickname). There are two undocumented /etc/system tunables in the bridge module: bridge_scan_interval and bridge_fwd_age. These control forwarding entry aging behavior and are unlikely to be used. 8. Reliability, Availability, Serviceability (RAS) - Does the project make any material improvement to RAS? No. - How can users/administrators diagnose failures or determine operational state? (For example, how could a user tell the difference between a failure and very slow performance?) For the IS-IS component, the existing Quagga administrative and debug interfaces are available. Snoop and ethereal may be used to observe packets on the wire, including IS-IS and TRILL. Daemons will be run from SMF; FMRIs are: svc:/network/bridge: svc:/network/routing/trill: The bridging kernel component has kstats described in 2008/055 and 2009/344. The TRILL kernel component has the following set of named kstats (class 'sock', module 'trill', instance '0', name '-'): recv TRILL and control frames received sent TRILL and control frames sent drops Frames dropped due to resources encap Ethernet frames encapsulated in TRILL decap TRILL frames decapsulated to Ethernet forward TRILL frames forwarded to next hop Each is a 64-bit unsigned integer. These are new with this project. - What are the project's effects on boot time requirements? None. - How does the project handle dynamic reconfiguration (DR) events? Links attached to bridges (or RBridges) are automatically removed from the bridge when a DR event is started, and the state is saved for replacement using SUNW_bridge_rcm.so (see PSARC 2009/344). - What mechanisms are provided for continuous availability of service? N/A - Does the project call panic()? Explain why these panics cannot be avoided. No - How are significant administrative or error conditions transmitted? SNMP traps? Email notification? Quagga has existing hooks. Both TRILL and bridging log significant events via syslog by default, and have undocumented variables that can be used to log debugging information. - How does the project deal with failure and recovery? IS-IS (used by TRILL) automatically detects interface failure and recovery, and routes around broken paths in the network. - Does it ever require reboot? If so, explain why this situation cannot be avoided. No. - How does your project deal with network failures (including partition and re- integration)? How do you handle the failure of hardware that your project depends on? That's a key component of routing, and it's what IS-IS provides. IS-IS computes the shortest path to every destination, and the TRILL forwarding component relies on that to forward L2 frames -- avoiding failures. STP can deal with network failures by reactivating links that were inactivated by the Spanning Tree mechanism. - Can it save/restore or checkpoint and recover? No. (The "nickname" mechanism in TRILL saves information via SMF, but there's no defined save/restore/checkpoint mechanism.) - Can its files be corrupted by failures? Does it clean up any locks/files after crashes? No, and yes. 9. Observability - Does the project export status, either via observable output (e.g., netstat) or via internal data structures (kstats)? Status information for bridges is described in detail in 2008/055 and 2009/344. Added to that are the following changes: dladm show-bridge -f - The OUTPUT field, which previously showed output link names, will show TRILL nicknames (automatically generated integers from 1 to 65535 that identify RBridge nodes) for each forwarding entry that points to a TRILL tunnel. Entries that are forwarded to local interfaces have local interface names instead, as with regular bridging. dladm show-bridge -t [-p] [-o ,...] [-i ] - A new variant that shows TRILL nickname entries. These are generated by the TRILL daemon from IS-IS information. Each entry maps a TRILL nickname into an output interface and a next-hop MAC address. They are used by the TRILL forwarding function. The long form is "--trill". The relevant field names are: NICK Nickname (1-65535) FLAGS Forwarding flags ("L" for local) LINK Output interface name NEXTHOP Next hop MAC address The default set of fields when -o is not specified includes all of the fields above. Example output: # dladm show-bridge -t foo NICK FLAGS LINK NEXTHOP 53234 L -- -- 57102 -- sim0 36:50:e9:a4:64:2b (The ::dladm mdb command described in 2008/055 is also updated with this new feature.) See also the kstats described in the response to question 8. - How would a user or administrator tell that this subsystem is or is not behaving as anticipated? Via administrative interfaces. - What statistics does the subsystem export, and by what mechanism? Packet in/out and drops; as described in section 8 of this document. - What state information is logged? None, except by debug. - In principle, would it be possible for a program to tune the activity of your project? Yes. 10. What are the security implications of this project? - What security issues do you address in your project? - The Solaris BSM configuration carries a Common Criteria (CC) Controlled Access Protection Profile (CAPP) -- Orange Book C2 -- and a Role Based Access Control Protection Profile (RBAC) -- rating, does the addition of your project effect this rating? E.g., does it introduce interfaces that make access or privilege decisions that are not audited, does it introduce removable media support that is not managed by the allocate subsystem, does it provide administration mechanisms that are not audited? - Is system or subsystem security compromised in any way if your project's configuration files are corrupt or missing? - Please justify the introduction of any (all) new setuid executables. - Include a thorough description of the security assumptions, capabilities and any potential risks (possible attack points) being introduced by your project. A separate Security Questionnaire http://sac.sfbay/cgi-bin/bp.cgi?NAME=Security.bp is provided for more detailed guidance on the necessary information. Cases are encouraged to fill out and include the Security questionnaire (leveraging references to existing documentation) in the case materials. Projects must highlight information for the following important areas: - What features are newly visible on the network and how are they protected from exploitation (e.g. unauthorized access, eavesdropping) - If the project makes decisions about which users, hosts, services, ... are allowed to access resources it manages, how is the requestor's identity determined and what data is used to determine if the access granted. Also how this data is protected from tampering. - What privileges beyond what a common user (e.g. 'noaccess') can perform does this project require and why those are necessary. - What parts of the project are active upon default install and how it can be turned off. See security analysis completed for 2008/055. In general, most administrators view IS-IS as "more secure" than other routing protocols, because it's not based on IP and cannot be routed. Remote attacks on it (unlike, say, BGP or OSPF) are impossible; physical network access is required. IS-IS, unlike STP, has optional features that allow for authentication of messages between peers. This feature is unlikely to be used in a TRILL environment, however, as the goal of the IETF TRILL working group is to be as "plug and play" as is traditional bridging. No configuration should be necessary, and that includes authentication keys. However, this could be used to advantage. The daemons and utilities associated with this project all have Least Privilege applied. 11. What is its UNIX operational environment: - Which Solaris release(s) does it run on? Solaris Nevada - Environment variables? Exit status? Signals issued? Signals caught? (See signal(3HEAD).) SIGHUP is used internally by the "svcadm refresh" logic. - Device drivers directly used (e.g. /dev/audio)? .rc/defaults or other resource/configuration files or databases? None - Does it use any "hidden" (filename begins with ".") or temp files? No. - Does it use any locking files? No. - Command line or calling syntax: What options are supported? (please include man pages if available) Does it conform to getopt() parsing requirements? The dladm(1M) interfaces from 2008/055 and 2009/344 are extended in ways already outlined in those cases: "dladm create-bridge" and "dladm modify-bridge" both get a new "-P" (--protect=) option that specifies the protection mode. This is "stp" or "trill". The default "dladm show-bridge" output gains a new field named "PROTECT". This field displays "stp" or "trill" for each bridge instance. See CR 6849974 for man page changes, and dladm* in case materials. - Is there support for standard forms, e.g. "-display" for X programs? Are these propagated to sub-environments? No. - What shared libraries does it use? (Hint: if you have code use "ldd" and "dump -Lv")? /usr/lib/bridged libsocket.so.1 librstp.so.1 libdlpi.so.1 libdladm.so.1 libumem.so.1 libc.so.1 /usr/sbin/trilld libzebra.so.0 libm.so.2 libresolv.so.2 libcrypt.so.1 libumem.so.1 libnsl.so.1 libsocket.so.1 libdladm.so.1 [requires contract from ON] libc.so.1 - Identify and justify the requirement for any static libraries. None. - Does it depend on kernel features not provided in your packages and not in the default kernel (e.g. Berkeley compatibility package, /usr/ccs, /usr/ucblib, optional kernel loadable modules)? No. - Is your project 64-bit clean/ready? If not, are there any architectural reasons why it would not work in a 64-bit environment? Does it interoperate with 64-bit versions? Yes. No. Yes. - Does the project depend on particular versions of supporting software (especially Java virtual machines)? If so, do you deliver a private copy? What happens if a conflicting or incompatible version is already or subsequently installed on the system? No. - Is the project internationalized and localized? Yes. - Is the project compatible with IPV6 interfaces and addresses? Yes. If a future project implements multicast snooping, then it will need to implement both MLD (IPv6) as well as IGMP (IPv4) snooping. Multicast snooping support is not part of this project. 12. What is its window/desktop operational environment? - Is it ICCCM compliant (ICCCM is the standard protocol for interacting with window managers)? - X properties: Which ones does it depend on? Which ones does it export, and what are their types? - Describe your project's support for User Interface facilities including Help, Undo, Cut/Paste, Drag and Drop, Props, Find, Stop? - How do you respond to property change notification and ICCCM client messages (e.g. Do you respond to "save workspace")? - Which window-system toolkit/desktop does your project depend on? - Can it execute remotely? Is the user aware that the tool is executing remotely? Does it matter? - Which X extensions does it use (e.g. SHM, DGA, Multi-Buffering? (Hint: use "xdpyinfo") - How does it use colormap entries? Can you share them? - Does it handle 24-bit operation? All N/A; no windowing or desktop. 13. What interfaces does your project import and export? - Please provide a table of imported and exported interfaces, including stability levels. Pay close attention to the classification of these interfaces in the Interface Taxonomy -- e.g., "Committed," "Uncommitted," and "*Private;" see: http://sac.sfbay/cgi-bin/bp.cgi?NAME=interface_taxonomy.bp Use the following format: Interfaces Imported Interface Classification Comments Nemo / GLDv3 Consolidation Private PSARC 2004/571, 2006/357 Volo Consolidation Private PSARC 2007/587, 2008/694 Interfaces Exported Interface Classification Comments TRILL Uncommitted* draft-ietf-trill-rbridge-protocol-12.txt draft-eastlake-trill-rbridge-isis-02.txt draft-eastlake-trill-rbridge-options-01.txt libdladm Contracted Consolidation Private Used by trilld AF_TRILL Contracted Project Private Used by trilld Contracted Project Private Used by trilld /etc/sock2path Committed Entry for AF_TRILL /usr/sbin/trilld Project Private /var/svc/manifest/network/trill.xml Project Private kernel/socketmod/trill Project Private kernel TRILL module kernel/socketmod/amd64/trill kernel/socketmod/sparcv9/trill - Exported public library APIs and ABIs Protocols (public or private) Drag and Drop ToolTalk Cut/Paste No exported APIs or ABIs. IS-IS with TRILL extensions are exported. - Other interfaces N/A - What other applications should it interoperate with? How will it do so? Must interoperate with traditional bridges, IS-IS routers, and RBridges. - Is it "pipeable"? How does it use stdin, stdout, stderr? Yes; the dladm(1M) interfaces support a parseable output format. - Explain the significant file formats, names, syntax, and semantics. N/A - Is there a public namespace? (Can third parties create names in your namespace?) How is this administered? No. - Are the externally visible interfaces documented clearly enough for a non-Sun client to use them successfully? Yes. 14. What are its other significant internal interfaces inter-subsystem and inter-invocation)? - Protocols (public or private) - Private ToolTalk usage - Files - Other - Are the interfaces re-entrant? Refer to rbridges-arch.png; key internal interfaces are: - AF_TRILL I/O mechanism for TRILL IS-IS messages, used by Quagga-based trilld. - AF_TRILL control interface that sets up nickname-based forwarding entries and tunnel handling, used by Quagga trilld. - dladm and SMF-based interfaces that configure and control bridge instances. 15. Is the interface extensible? How will the interface evolve? - How is versioning handled? - What was the commitment level of the previous version? - Can this version co-exist with existing standards and with earlier and later versions or with alternative implementations (perhaps by other vendors)? - What are the clients over which a change should be managed? - How is transition to a new version to be accomplished? What are the consequences to ISV's and their customers? All interfaces described in section 14 above are Project Private. They're not expected to be needed outside of this project. The libdladm interfaces used by trilld are enumerated in contract-01. 16. How do the interfaces adapt to a changing world? - What is its relationship with (or difficulties with) multimedia? 3D desktops? Nomadic computers? Storage-less clients? A networked file system model (i.e., a network-wide file manager)? N/A 17. Interoperability - If applicable, explain your project's interoperability with the other major implementations in the industry. In particular, does it interoperate with Microsoft's implementation, if one exists? - What would be different about installing your project in a heterogeneous site instead of a homogeneous one (such as Sun)? - Does your project assume that a Solaris-based system must be in control of the primary administrative node? For IS-IS and traditional bridging, this project will need to be interoperable with Linux and Cisco, as those represent common peers. This implementation will help define interoperability for TRILL. At some point in the future, there will be peers to talk to using TRILL, and our goal is to demonstrate interoperability when those other implementations are available. 18. Performance - How will the project contribute (positively or negatively) to "system load" and "perceived performance"? N/A - What are the performance goals of the project? How were they evaluated? What is the test or reference platform? Performance goals are deliberately set modestly for the first release: at least line rate without drops at 100Mbps on a blade-class machine. Future goals will be influenced by usage scenarios -- for example, performance in a filtering environment is dependent primarily on filter complexity, not bridge performance, and performance in a Crossbow flow-controlled environment is governed by the Crossbow interfaces. The machine will not become unstable under load. It will instead drop packets when stressed past its limit. Further, only packets forwarded between interfaces (and not local input/output from Solaris datalink consumers) will be affected. Primary testing for bridging compatibility will be done by UNH. Internal testing for the TRILL extensions will be done in a simulated environment using simnet links. Unit testing with multi-interface machines is also necessary. - Does the application pause for significant amounts of time? Can the user interact with the application while it is performing long-duration tasks? No pausing. - What is your project's MT model? How does it use threads internally? How does it expect its client to use threads? If it uses callbacks, can the called entity create a thread and recursively call back? The kernel portion is MT-hot. The user space components are all single-threaded. (Quagga uses a multi-process design, with common components in the central zebrad daemon, and individual protocols in separate processes, separated by sockets.) - What is the impact on overall system performance? What is the average working set of this component? How much of this is shared/sharable by other apps? The primary impact of using bridges is that network interfaces are placed into promiscuous mode all of the time, and all packets must be processed for source address updates. Using promiscuous mode is a fundamental requirement for bridging, and imposes substantial costs in terms of disabled NIC optimizations. All other considerations are secondary (at best). - Does this application "wake up" periodically? How often and under what conditions? What is the working set associated with this behavior? Yes. IS-IS sends Hello messages every 10 seconds on each configured interface by default, and this can be tuned to trade off traffic versus silent failure detection time. STP sends BPDUs once a second by default. Virtual memory consumption for bridging is modest; about 76K text, 36K stack, and 256K heap (including space for private library) for a two-link bridge. Usage for trilld is around 196K text, 12K stack, and 468K heap for a similar configuration. prstat shows RSS of 1832K for bridged and 2760K for trilld. - Will it require large files/databases (for example, new fonts)? The LSP database within IS-IS is potentially large. It has at least as many entries as there are routers in the network, and may have more for pseudonodes. This database is held in user space RAM (not in files) and is built dynamically from peer reports. The kernel forwarding database is built from IS-IS derived information and from L2 information learned from directly attached interfaces. - Do files, databases or heap space tend to grow with time/load? What mechanisms does the user have to use to control this? What happens to performance/system load? The databases do not grow with time; they grow with the number of nodes and links handled. The databases for bridging (kept in the kernel) are capped and can be tuned by the user if necessary. The databases for IS-IS are in user space and are generally a few KB for each link in the network. 19. Please identify any issues that you would like the ARC to address. - Interface classification, deviations from standards, architectural conflicts, release constraints... - Are there issues or related projects that the ARC should advise the appropriate steering committees? None. 20. Appendices to include - One-Pager. - Prototype specification. - References to other documents. (Place copies in case directory.) Normative references (in materials directory): rfc1142.txt draft-ietf-trill-rbridge-protocol-06.txt draft-ietf-trill-rbridge-arch-04.txt 802.1D-1998.pdf Informative reference: http://www.opensolaris.org/os/project/rbridges/