GLDv3-based WiFi Kernel Architecture ==================================== #ident "%Z%%M% %I% %E% SMI" 1. Introduction --------------- 1.1 Overview ------------ This document covers the proposed architectural changes to the Solaris kernel to provide first-class support for GLDv3-based WiFi (IEEE 802.11) drivers. It first summarizes the existing GLDv2-based WiFi model and its limitations in order to provide context and justification. Then, it outlines the two primary components that comprise the solution: the GLDv3 mac_wifi plugin, and the net80211 kernel support module. Note that since mac_wifi and net80211 are not intended to be public interfaces at this time, they are discussed only in enough detail to understand their architectural implications. Subsequent projects may raise the commitment level of these components, at which time a more detailed treatment will be provided. However, the architectural issues associated with making mac_wifi and net80211 available for public use are summarized. 1.2 GLDv2-based WiFi Model -------------------------- Because WiFi driver development occurred in parallel with GLDv3 development, WiFi drivers were written to the GLDv2 framework, with the understanding that they would be converted to GLDv3 once it had stabilized. To minimize changes to the GLDv2 framework and to applications using DLPI, all GLDv2-based WiFi drivers were written to appear as Ethernet devices. That is, the GLDv2 framework exchanges Ethernet packets with GLDv2-based WiFi drivers, which in-turn rewrite the packets to and from the 802.11 frame format as necessary. Unfortunately, this sleight-of-hand creates several issues: 1. The architecture precludes observation of raw 802.11 packets using traditional packet sniffers, since by the time a packet can be observed, it is already converted to Ethernet. Given the complexity associated with the 802.11 protocol, this presents an observability limitation. 2. Each data packet must be rewritten on send and receive in order to convert the Ethernet header to an 802.11 header, or vice-versa. On the send-side, where the header size must be increased, the original header must be carved off and discarded, and a new one must be allocated, filled in, and bolted on. 3. Because GLDv2 believes all WiFi drivers to be Ethernet drivers, there is no convenient way to have the driver framework publish WiFi-specific kstats. Further, it advertises Ethernet statistics for WiFi Drivers even though these statistics are meaningless. Finally, while not due to GLDv2, code duplication is another issue with the current WiFi kernel architecture. Specifically, Solaris WiFi drivers are based on several different versions of Errno Consulting's WLAN framework. Since each driver requires a slightly different version of the framework, this requires each driver to have the framework it needs directly compiled into it. Needless to say, this introduces significant architectural and maintenance issue. 1.3 Proposed GLDv3-based WiFi Model ----------------------------------- The proposed GLDv3-based WiFi model consists of two components: 1. A "mac_wifi" plugin written to the Nemo MAC-Type Plugin Architecture defined by PSARC/2006/248. This allows GLDv3 to natively support 802.11, and allows DLPI-based applications to opt-in to native 802.11 support as well. 2. A "net80211" kernel support module, based on the framework provided by Errno Consulting. It also includes a set of additional interfaces that collectively allow common code to be centralized, rather than embedded inside each driver. Note that the WiFi driver never directly interacts with the mac_wifi plugin (nor does it have a direct dependency on it), so we have chosen to keep these two components distinct to keep the mac_wifi consistent with the other Nemo MAC-Type plugins, and to keep the net80211 support module functionality consistent with other platforms. The remainder of this document discusses these components. 2. The mac_wifi GLDv3 plugin ---------------------------- As discussed above, the mac_wifi plugin is written to the Nemo MAC-Type Plugin architecture defined by PSARC/2006/248. It will implement all plugin operations defined by that case, and install under /kernel/mac. Since we worked closely with the team to define that architecture, no architectural changes were necessary in order to accommodate the mac_wifi plugin. A header file will contain the necessary information for drivers to use the plugin, namely a MAC_PLUGIN_IDENT_WIFI macro used to identify the plugin during mac_register(), and a list of WiFi statistics defined by the plugin, given in section 2.4. Note that the name follows the naming convention established by PSARC/2006/248 (). 2.1 802.11 Protocol Considerations ---------------------------------- While 802.11 is similar to Ethernet in some respects (e.g., the addressing formats and SAP spaces are identical), the actual link-layer format is quite a bit different and more complex: 1. Unlike Ethernet, not all link-layer fields are constant for a given destination. For instance, each 802.11 packet header includes a unique sequence number. 2. Unlike Ethernet, the size and contents of the link-layer header may change depending on the security mode of the connection. For instance, WEP introduces an additional link-layer subheader in each packet. 3. Unlike Ethernet, even fields that are constant for a given destination may require coordination between the WiFi driver and the framework in order to be filled in. For instance, the associated BSSID/IBSSID is constant, but traditionally only known to the driver. These differences have some subtle architectural implications on the interaction between the GLDv3 framework, the mac_wifi plugin, and the WiFi device driver, as discussed in the following sections. 2.1.1 Link-layer Header Caching ------------------------------- GLDv3 consumers such as IP request GLDv3 to generate link-layer headers so that they can cache template link-layer headers for each destination. This optimization, known as "fastpath" (aka DL_IOC_HDR_INFO), improves performance since packets passed from IP to the driver will already have the link-layer header allocated and provided with template contents. Since it is possible for the link-layer headers to become invalid, another facility, DL_NOTE_FASTPATH_FLUSH, is provided to enable the driver (or, in this case, the GLDv3 framework) to inform consumers that the cached header has been invalidated and must be regenerated. 2.1.2 Link-layer Header Generation ---------------------------------- Since for a given destination Ethernet consists entirely of fixed headers, link-layer caching has minimal impact on its link-layer header generation. However, as discussed above, this is not the case for 802.11. Accordingly, the mac_wifi plugin only fills in fixed per-destination fields, and expects the WiFi driver (and perhaps the hardware itself) to fill in the variable fields according to their state. Thus, the generated header is suitable for caching. However, as per considerations (1) and (2) in section 2.1, even filling in the fixed per-destination fields requires coordination between the mac_wifi plugin and each WiFi driver. To accomplish this, mac_wifi makes use of the PSARC/2006/248 "pdata" mechanism to allow the WiFi drivers and mac_wifi plugin to exchange information. Currently, the following state structure is defined in : /* * WiFi data passed between the drivers and the plugin. */ typedef struct wifi_data { uint_t wd_opts; uint8_t wd_bssid[IEEE80211_ADDR_LEN]; enum ieee80211_opmode wd_opmode; enum wifi_secmode wd_secalloc; } wifi_data_t; As outlined in PSARC/2006/248, initial wifi_data values are provided by the WiFi driver when it registers with GLDv3 via mac_register(), and can be updated at any time by calling mac_pdata_update() with the new wifi_data. As per PSARC/2006/248, whenever mac_pdata_update() is called, the GLDv3 framework automatically triggers flushing of any cached headers, causing all link-layer headers to be regenerated using the latest information. Note that the wifi_data passed to mac_register() and mac_pdata_update() is copied by mac_wifi. This allows information to be shared without requiring sophisticated locking techniques or double-dispatch callbacks between mac_wifi and the WiFi driver. However, it does require that the WiFi drivers and mac_wifi plugin agree on the format of the wifi_data_t. The versioning aspects of this are handled through the wd_opts field, as discussed in section 2.3. 2.2 802.11 DLPI considerations ------------------------------ Unlike other mainstream link layers supported by Solaris, 802.11 is connection-oriented. For instance, in order to send and receive data, a connection must first be established, usually with an access point. Accordingly, 802.11 contains a large set of control messages to perform a variety of tasks, such as node advertisement, solicitation, discovery, and association/disassociation. DLPI (as defined) is not sufficiently rich to support all of these concepts as first-class citizens. Moreover, even if it was, most DLPI applications are written assuming a connectionless link-layer, such as Ethernet. Further, we are not aware of any DLPI-based applications that need to directly interact with WiFi devices at this layer. Instead, most DLPI applications (such as the Solaris DHCP client) wish to simply send and receive packets at the link-layer over an already-connected WiFi link. Accordingly, we propose to maintain the current WiFi DLPI model: * The "link status" of a WiFi device reflects whether it is connected: - Connecting brings the link up, and thus generates a DL_NOTE_LINK_UP message (see dlpi(7P). - Disconnecting brings the link down, and thus generates a DL_NOTE_LINK_DOWN message. Thus, from a DLPI application perspective, disconnecting from an access point is no different from unplugging an Ethernet cable. * On an "up" link, the normal DLPI "connectionless" primitives, such as DL_UNITDATA_REQ, work as expected. This enables DLPI applications to transparently use WiFi devices with no changes. Currently, WiFi DLPI links also advertise themselves as DL_ETHER, and accordingly send and receive packets with Ethernet headers (translating as needed) in DLIOCRAW mode. We propose to maintain this behavior, but provide an additional DLIOCNATIVE ioctl that allows "native" 802.11 packets to be sent and received, if necessary. The rationale for these decisions are covered in the following sections. 2.2.1 DL_ETHER and DL_WIFI types -------------------------------- As outlined above, we propose to maintain the illusion that connected WiFi devices are DL_ETHER links. Given that WiFi uses the Ethernet SAP space and address format, is often bridged onto Ethernet subnets via access points, and is designed to appear "like" Ethernet once connected, we believe this is reasonable[1] and has minimal impact to applications. However, for applications that want to opt-in to receiving and sending raw 802.11 frames, we propose a new DLIOCNATIVE facility that will change the DLIOCRAW link-layer headers from Ethernet to 802.11. As discussed in the next section, this will also change the mac type from DL_ETHER to DL_WIFI, so that the application is aware of the new frame format. Note that DL_WIFI only indicates a frame format change, and does not change the set of DLPI primitives that are supported. It may seem surprising to advocate using DL_ETHER by default given the objections to the GLDv2 model raised in section 1.2. However, we believe the DLIOCNATIVE facility and mac_wifi kstats facility (see section 2.4) address those concerns while minimizing impact. 2.2.2 Cooking, Uncooking, and DLIOCNATIVE ----------------------------------------- In order to provide the expected DL_ETHER DLIOCRAW behavior, mac_wifi will implement the mtops_cook() and mtops_uncook() entry points from PSARC/2006/248. As the names imply, when the GLDv3 framework receives a DLIOCRAW packet for transmission, it will call the mtops_cook() callback to have mac_wifi convert the Ethernet header into an 802.11 header prior to transmission. Likewise, prior to passing the packet to a DLIOCRAW DLPI consumer, GLDv3 will call the mtops_uncook() callback to have mac_wifi convert it to Ethernet. If the DLPI consumer is interested in sending and receiving raw 802.11 frames, it can use the proposed DLIOCNATIVE ioctl to instruct GLDv3 to not invoke the mtops_cook() and mtops_uncook() callbacks. For instance, in the future, a packet sniffer could use DLIOCNATIVE in order to capture, filter, and display 802.11 frames. (Note that updating packet sniffers such as snoop is a non-trivial task that is not necessitated by our proposal, and thus is not part of this project.) The proposed interface for DLIOCNATIVE is similar to DLIOCRAW: it is enabled through ioctl(fd, DLIOCNATIVE, 0), and remains enabled for the lifetime of the stream. Upon success, the native mac type associated with the stream (in this case, DL_WIFI) is returned to the caller. In addition, any subsequent DL_INFO_REQ requests will generate responses with dl_info_ack_t`dl_mac_type set to the native mac type. To enable GLDv3 to return the native mac type, a new mtr_nativetype member will be added to the Consolidation-Private mactype_register_t structure introduced by PSARC/2006/248 (this was an oversight in the original materials). The mac_wifi plugin will set mtr_nativetype to DL_WIFI; all other plugins will be updated to set this member to mtr_type, since they always use their native type. 2.3 Considerations for Public Use --------------------------------- As previously stated, mac_wifi is proposed as Consolidation Private, and thus will be evolving in lockstep with the GLDv3 framework and the WiFi drivers that make use of it. However, the following versioning mechanisms will be provided both to fail gracefully in cases where binaries have been copied across builds, and in consideration of future public use: * The MAC-Type plugin interface is versioned in an extensible manner via the mtr_version and mtops_ops fields described in PSARC/2006/248. Thus, if mac_wifi is not compatible with the GLDv3 framework, the module will fail to load and descriptive error messages will be output. This will also cause the WiFi driver which triggered the loading of mac_wifi to fail to load. * The wifi_data_t will be versioned via the wd_opts field. The mac_wifi module will use its mtops_pdata_verify() callback to verify that wd_opts is zero. If not, mtops_pdata_verify() will fail, causing the driver to fail to register with the GLDv3 framework, and descriptive error messages to be output. In the future, any changes to the wifi_data structure will be accompanied by new values for wd_opts. 2.4 WiFi kstats --------------- The MAC-Type plugin architecture defined by PSARC/2006/248 also includes a kstat facility that enables each plugin to define a set of kstats that will be common to all drivers of that type. We propose to have mac_wifi provide a set of kstats, which mirror the "dot11Counters" MIB statistics defined by the IEEE 802.11 standard: enum wifi_stat { WIFI_STAT_TX_FRAGS = MACTYPE_STAT_MIN, WIFI_STAT_MCAST_TX, WIFI_STAT_TX_FAILED, WIFI_STAT_TX_RETRANS, WIFI_STAT_TX_RERETRANS, WIFI_STAT_RTS_SUCCESS, WIFI_STAT_RTS_FAILURE, WIFI_STAT_ACK_FAILURE, WIFI_STAT_RX_FRAGS, WIFI_STAT_MCAST_RX, WIFI_STAT_FCS_ERRORS, WIFI_STAT_WEP_ERRORS, WIFI_STAT_RX_DUPS }; The semantics of each kstat correspond to those documented in Annex D of the 802.11 standard. Each kstat, along with its user-visible name, will be documented in the ieee802.11(5) manpage[2], which mirrors the existing ieee802.3(5) manpage that covers 802.3 and Ethernet statistics. Each kstat is a 64-bit value. Note that PSARC/2003/722 also proposed a set of GLDv2 kstats with identical semantics to those above, but those kstats were never implemented. Since all WiFi drivers will be GLDv3-based, there is no need to implement the PSARC/2003/722 proposal, and it is superseded by this proposal. 3. The net80211 kernel support module -------------------------------------- As previously discussed, Solaris WiFi drivers all have origins in Errno Consulting's WLAN framework. While that framework includes a module that allows common code to be shared across the various WiFi drivers, due to version skew across the Solaris WiFi driver ports, no common module yet exists on Solaris. Further, additional functionality -- such as parsing the WLAN ioctls defined by PSARC/2003/722 -- is needed in order for proper code sharing to be realized on Solaris. The remainder of this section discusses the architectural issues associated with this module. First, we provide a brief background of its history and current functionality. Next, we summarize the proposed changes in order to accommodate both the Solaris kernel and the Solaris WiFi architecture. Finally, we outline the technical challenges associated with making the module available for public use. 3.1 Background and Functionality -------------------------------- The WLAN framework dates back to 2001 and has origins in NetBSD. It is now a standard part of NetBSD and FreeBSD, and is also used by Linux projects such as MadWiFi. Though no architectural documents appear to be available, slide presentations ([3] and [4]) are available that provide a high-level view of the provided functionality, along with history, motivation, and lessons learned. Of the several different versions of the WLAN framework that are actively in-use today, we propose to base the Solaris net80211 module on the WLAN framework included with FreeBSD 7, since: * It is most consistent with the framework currently compiled into our WiFi drivers. * It tracks the latest WLAN framework. (That is, Errno integrates to FreeBSD 7 first; changes may later be folded into NetBSD or previous releases of FreeBSD.) Of course, development may shift over time to another FreeBSD release (or elsewhere), and assuming we continue to base net80211 off of the WLAN framework (see section 3.3), we will also shift to stay current. The WiFi support module provided in FreeBSD is summarized in the wlan(4) manpage[5]. In short, it provides: * The protocol state machines for 802.11a, 802.11b, and 802.11g. * A plugin mechanism for cryptographic modules. Currently, the WEP, TKIP, and AES-CCMP modules are included. * A collection of support routines for parsing 802.11 packets, and performing other common 802.11-related operations. * Wireless Multimedia Extensions (WME aka WMA/802.11e), which provide QoS functionality on suitably capable hardware. * The WPA, WPA2, 802.11i, and 802.1x security protocols, through collaboration with userland daemons. To perform the above operations, there are two large state structures that are shared between each driver and the support module: * The ieee80211_node structure tracks the nodes broadcasting beacon frames on the WLAN. * The ieee80211com structure tracks the current state of a given WiFi device instance, and also includes a set of callbacks that the WiFi support module can invoke to interact with the underlying device (e.g., send and receive packets). These structures are allocated and freed by the driver but have content that is often shared by both the driver and the WiFi support module (access is arbitrated through embedded locks). This design is adequate for consolidation-private use, but presents challenges as a public interface, as discussed in section 3.3. 3.2 Solaris net80211 module --------------------------- The Solaris net80211 module functionality and interfaces are identical to the WiFi support module described in section 3.1, except for the following changes: * Wireless Multimedia Extensions support has not been ported since it is not yet needed by any Solaris WiFi drivers. * WPA, WPA2, 802.11i, and 802.1x support has not been ported, since they are not currently supported by Solaris. However, work is underway to add such support to Solaris. * TKIP and AES-CCMP have not been ported, since they are not useful without WPA/WPA2/802.11i support. * The FreeBSD ioctl handling has been replaced with Solaris WiFi ioctl handling. (PSARC/2003/722 defines the Solaris WiFi ioctls) * The ieee80211com structure fields that were not intended to be shared between the driver and the support module have been moved into a separate ieee80211_impl structure. In addition, changes were made to the implementation in order to tie in with the GLDv3 framework when necessary (e.g., to trigger "link up" messages when WiFi device becomes associated), and to make use of the appropriate Solaris kernel functions for mutexes, timers, and so forth. 3.3 Considerations for Public Use --------------------------------- As previously stated, net80211 is proposed as Consolidation Private, and thus will be evolving in lockstep with the GLDv3 framework and the WiFi drivers that make use of it. We expect to periodically synchronize with the latest FreeBSD WLAN framework to reduce the porting effort associated with merging new or updated WiFi drivers into Solaris. However, in the long-term, we would also like to allow WiFi drivers outside of the ON consolidation to make use of the net80211 interfaces. Unfortunately, a number of issues make such decoupling difficult today: * The WiFi driver state structures are co-allocated with the ieee80211com structures, by placing the ieee80211com structure as the first member of each WiFi driver state structure. This technique means that the drivers must be recompiled in lockstep with changes to the ieee80211com structure in order to correctly access their state structures. * The ieee80211com and ieee80211_node fields are directly accessed by the WiFi drivers. This means that any changes to the framework that necessitate changes to those fields require that the drivers be updated in lockstep. * Different wireless chipsets vary greatly in their capabilities, and thus what parts of the 802.11 specification need to be implemented in hardware or software. As a result, the framework interfaces and state structures periodically have to be changed to accommodate new chipsets. These changes have often been made in incompatible ways. * 802.11 is a complex and rapidly-changing technology, especially compared to protocols like Ethernet. Some of these changes, such as Multi-Mode devices and Multi-BSS, have led to massive overhauls of the framework, breaking backward compatibility. Moreover, Solaris is unique in its guarantee that drivers written to documented kernel interfaces will not suffer API/ABI incompatibilities. This leaves us with four options to potentially pursue: 1. Convince Errno Consulting that the API and ABI need to be stabilized. This would obviously be ideal for us, but is unlikely to be appealing to them unless they see a significant market opportunity associated with unbundled wireless drivers. 2. Modify the WiFi support module architecture to facilitate a stable API and ABI. This is feasible, but will require resources to create and maintain. Further, merging new or updated WiFi drivers into Solaris will become more difficult. 3. Introduce a new WiFi support module architecture that facilitates a stable API and ABI. This seems prohibitively resource-intensive, especially since it will likely require that we author own own drivers from scratch. 4. Do not attempt to offer a stable API and ABI, and require that all WiFi drivers integrate into ON. This may be unacceptable to some companies who are unwilling to make the source for their driver chipsets available. Option (1) seems best, but unlikely to pan out. Options (2) and (3) seem untenable given our current resources, leaving us with option (4) as (unfortunately) our most likely option. Footnotes --------- [1] In fact, some old chipsets like Prism-II (pcwl) actually exchange Ethernet frames with the driver, and perform 802.11 conversion in hardware. [2] Proposed ieee802.11 manpage: manpages/ieee802.11.5 [in materials] [3] "FreeBSD Wireless Networking" Presentation: http://bsdcan.org/2005/papers/FreeBSDWirelessNetwokringSupport.pdf [sic] [4] "Wireless Networking in the Open" presentation: http://people.freebsd.org/~sam/SANE2006-Wireless.pdf [5] FreeBSD wlan manpage: http://freebsd.org/cgi/man.cgi?query=wlan