From sacadmin Wed May 7 13:46:14 2003 From: Bill Sommerfeld To: psarc@sac.eng.sun.com cc: Adi.Masputra@Sun.COM Subject: 2003/263: DL_CAPABILITY-ignorant intermediate module detection Date: Wed, 07 May 2003 16:44:24 -0400 Content-Length: 18218 I'm sponsoring this case for Adi Masputra. Timer is set to expire 5/14/2003. Requested release binding is "patch". (please suspend any disbelief about this until you reach the BACKWARDS COMPATIBILITY section below). Following the Carlson Doctrine[1], stability level is (as with 2003/264) Consolidation Private, with the expectation that it will become Contracted Consolidation Private once contracts appear with the appropriate driver groups are finalized. - Bill [1] best summarized as "It's not Contracted until a contract exists". DL_CAPABILITY-ignorant intermediate module detection BACKGROUND ---------- In the recent years, many non-standard (non-DLPIv2) mechanisms of exchanging messages between ip module and the DLPI device driver were introduced to Solaris, e.g. M_DATA fastpath, hardware checksum offload [PSARC 1996/173], Multidata Transmit (MDT) [PSARC 2002/276], IPsec Hardware Acceleration (HA) [PSARC 2001/070], etc. These new interfaces were needed in order to get around the limitation of the DLPIv2 interface between a Data Link Provider and Consumer, mainly for performance improvements (some of which are attributed to driver/hardware offload capabilities). It is expected that many other future networking projects (zero-copy, etc.) would rely on some other similarly negotiated interfaces. IP's usage of these interfaces depends on the outcome of capability negotiations that happen between ip module and the driver. Such negotation scheme works well in cases where the driver is the module directly underneath ip. In other cases where there is one or more intermediate module between ip and the driver, this scheme does not provide a way of detecting their capabilities. Therefore, the introduction of modules in between often results in a breakage. In addition, the current scheme of capabilities negotiation typically happens during the plumbing, or construction of the IP interface stream. While this works for statically-plumbed intermediate modules, it is not sufficient to handle the case where a module is dynamically introduced into an existing IP interface stream. PROBLEMS -------- The exchange of capability negotiations are done in several ways. The M_DATA fastpath negotiation is triggered by a M_IOCTL message. The current hardware checksum offload uses a M_CTL message as vehicle, while the newer features such as MDT and IPsec HA use the newly- defined DL_CAPABILITY_{REQ,ACK} inside the M_PROTO message type. An intermediate module which does not understand any of those messages may either free them, or forward them to the next module. The former action can easily be dealt with, as IP can handle the "lost" negotiation message case (it won't enable or utilize those features). The latter action, however, is rather dangerous as the response may be interpreted to be coming from such intermediate module, which in reality was generated by a module underneath it. The following example depicts such case: +----------+ | ip | +----------+ | +----------+ | firewall | No knowledge of DL_CAPABILITY_REQ interface +----------+ | +----------+ | ce | Supports capability (MDT) +----------+ A firewall module which doesn't understand the Multidata interface (as well as the DL_CAPABILITY framework) is plumbed between ip and the ce driver, either statically or dynamically. Because it passes through any unknown messages, the DL_CAPABILITY_ACK coming from the driver gets forwarded (passed through) onto ip. Since ip has no way of determining the origin of the acknowledgment, it assumes that the module immediately below (firewall) was the source of the acknowledgement, and hence it will end up utilizing Multidata to pass down the packets. When the firewall module receives the Multidata messages, it may either forward them down to the driver or free them. Either action is considered a breakage: the former allows packets to go out of the machine without first being inspected by the firewall, and the latter will most likely break any existing network connections whose packets are encapsulated within those Multidata messages. Any STREAMS module may be interposed between ip and the driver by way of autopush(1M), if_tcp(7P), and ifconfig(1M) modinsert/modremove. In the static plumbing case, using either autopush(1M) or if_tcp(7P) method, the module is pushed on top an existing stream opened to the device driver, and the ip module is further pushed on top of it. At this point, the IP interface is considered to be plumbed or constructed, and the ip module may initiate the capability negotiation exchange process. The ability to dynamically change the plumbing of the IP stream was introduced in PSARC/1999/348 - _I_MUXID2FD, _I_INSERT, and _I_REMOVE ioctls. In PSARC/1999/375 - Re-plumbing Options for ifconfig(1M) case, 2 new options to ifconfig(1M) were introduced that can change the plumbing of the IP stream dynamically (modinsert and modremove). These options are classified as Evolving. Because a dynamic re-plumbing typically happens after the exchange of capabilities negotiation, a breakage will occur when the newly- inserted module is not capable of interpreting any previously- negotiated and enabled features. The problem space can therefore be summarized as follows: 1. The lack of module identification token in the DL_CAPABILITY framework which would allow for verification of incoming sub- capabilities within DL_CAPABILITY_ACK messages, in order to detect pass-thru intermediate modules. 2. There is currently no formal mechanism to reset and re-negotiate the capabilities due to a change in the stream's plumbing, such as by way of ifconfig(1M) modinsert and modremove. PROPOSAL -------- This fast-track proposal introduces an extension to the DL_CAPABILITY framework to allow for detecting "pass-thru" modules, as well as to turn on or off specific capabilities on a case-by-case basis by ip in order to perform the capability resetting and re-negotiating actions during a dynamic re-plumbing event. The requested release binding is micro. This proposal creates the following interfaces: Exported Interfaces: +-----------------------+----------------------------------+-----------------+ | Interface | Classification | Comments | |-----------------------+----------------------------------+-----------------| | DL_CAPAB_ID_WRAPPER | Consolidation Private | sys/dlpi.h | | dl_mid_t | Consolidation Private | sys/dlpi.h | | dl_capab_id_t | Consolidation Private | sys/dlpi.h | | dlcapabsetqid() | Consolidation Private | sys/dlpi.h | | dlcapabgetqid() | Consolidation Private | sys/dlpi.h | |-----------------------+----------------------------------+-----------------| Imported Interfaces: +-----------------------+----------------------------------+-----------------+ | Interface | Classification | Comments | |-----------------------+----------------------------------+-----------------| | DL_CAPABILITY_REQ | Consolidation Private | PSARC/2001/070 | | DL_CAPABILITY_ACK | Consolidation Private | PSARC/2001/070 | | dl_capability_sub_t | Consolidation Private | PSARC/2001/070 | | dl_capab_ipsec_t | Consolidation Private | PSARC/2001/070 | | dl_capab_mdt_t | Consolidation Private | PSARC/2002/276 | +-----------------------+----------------------------------+-----------------+ From : #define DL_CAPAB_ID_WRAPPER 0x00 /* * Module ID to be included in new sub-capability structures. * Existing sub-capabilities lacking an identification token, * e.g. MDT and IPSEC hardware acceleration, need to be * encapsulated within the ID sub-capability. Access to this * structure must be done through dlcapab{set,get}qid(). */ typedef struct { t_uscalar_t mid[4]; /* private fields */ } dl_mid_t; /* * Module ID wrapper (follows dl_capability_sub_t) */ typedef struct { dl_mid_t id_mid; /* module ID */ dl_capability_sub_t id_subcap; /* sub-capability */ } dl_capab_id_t; extern void dlcapabsetqid(dl_mid_t *, queue_t *); extern queue_t *dlcapabgetqid(dl_mid_t *); A token (dl_mid_t) which allows for the module understanding the sub- capabilities to record its identification is needed. Currently, the identification token is defined to be the upper read queue pointer value of the module. By having a module or driver go through the DL_CAPABILITY_ACK message (on its way up) and "stamp" its upper read queue value on the sub-capabilities within, ip can detect a pass-thru module by way of comparing the upper read queue of the module immediately below with the value inside the sub-capability. At the time of writing, both MDT and IPsec HA features in the networking stack and device drivers have been delivered into Solaris, and possibly deployed on customer sites. Modifying dl_capab_mdt and dl_capab_ipsec_t in order to introduce an "identification" token will certainly break existing device drivers. Therefore, a "wrapper" sub- capability (dl_capab_id_t) used to encapsulate those sub-capabilities lacking the identification token is proposed. Existing MDT and IPsec HA sub-capabilities can be wrapped with the above "wrapper" sub-capability, and ip can be modified to understand such sub-capability in addition to the existing ones that it supports. New sub-capabilities defined in future are required to incorporate the dl_mid_t structure in order to provide for an identification token. Ways to set and obtain the queue pointer (the module identification) from within the dl_mid_t, via the dlcapab{set,get}qid() routines, are also introduced. Upon detecting dynamic re-plumbing events, ip would reset (disable) previously-negotiated capabilities -- details on how this is done is beyond the scope of this case, and is documented in 4771245. The reason for the "reset" is to establish that only vanilla DLPI is acceptable (i.e. modules should revert to sending only strict DLPIv2). This is done by having ip compose and send down a DL_CAPABILITY_REQ message which consists of the previously-negotiated capabilities indicating that they be disabled and no longer used. Such indication is done by way of clearing the "enable" bit defined per sub-capability. Afterwards, ip probes its lower-stream to find out about any new (or lack of) capabilities by sending down an empty DL_CAPABILITY_REQ message per PSARC 2001/070. An intermediate module which doesn't understand the DL_CAPABILITY framework or sub-capabilities within will be able to be detected, and no breakage will occur. In order for the above to work, the rule set of PSARC 2001/070 needs to be modified to include the following: a. All sub-capabilities must be able to be disabled. The current MDT and IPsec HA sub-capabilities contain flags fields whose enable bits can be cleared. Future sub-capabilities are required to implement similar mechanism. b. The enable/disable bit in previously-advertised (those which have been enabled by ip) sub-capabilities is read-only for modules and drivers, and can only be modified by ip. This would guarantee that an intermediate module doesn't interfere with the capability reset done by ip during dynamic re-plumbing. c. Future sub-capability definitions must include the dl_mid_t structure in order to provide for module identification. d. All intermediate modules (except drivers) are required to forward the DL_CAPABILITY_REQ message (received from the module above) downstream. e. All modules and drivers which understand the DL_CAPABILITY framework must record its upper queue pointer value, by calling dlcapabsetqid(), on each of the sub-capability within the DL_CAPABILITY_ACK which it recognizes and wishes for ip to enable. This must be done prior to passing the DL_CAPABILITY_ACK upstream. f. A module which initiates a DL_CAPABILITY_REQ is required to check each sub-capability returned in a DL_CAPABILITY_ACK, by calling dlcapabgetqid(), to see if the returned queue pointer matches that of the upper read queue pointer of the module immediately below. If it matches, the sub-capability may be enabled, otherwise, it should be discarded. DETAILS ------- Prior to passing up a DL_CAPABILITY_ACK (i.e. in the rput routine), a module must go through each sub-capability looking for those which it supports, and stamp its upper read queue pointer on them. In the case of a driver, this process is done as part of initializing the sub-capability that it is about to report to ip. The following example of pseudo structures depict the current MDT sub-capability as reported by drivers: dl_capability_ack { dl_primitive = DL_CAPABILITY_ACK; dl_sub_offset = sizeof (dl_capability_ack_t); dl_sub_length = sizeof (dl_capability_sub_t) + sizeof (dl_capab_mdt_t); dl_capability_sub_t { dl_cap = DL_CAPAB_MDT; dl_length = sizeof (dl_capab_mdt_t); dl_capab_mdt_t { ... } } } When using the wrapper sub-capability, the above example can be represented as: dl_capability_ack { dl_primitive = DL_CAPABILITY_ACK; dl_sub_offset = sizeof (dl_capability_ack_t); dl_sub_length = sizeof (dl_capability_sub_t) + sizeof (dl_capability_id_t) + sizeof (dl_capab_mdt_t); dl_capability_sub_t { dl_cap = DL_CAPAB_ID_WRAPPER; dl_length = sizeof (dl_capab_id_t) + sizeof (dl_capab_mdt_t); dl_capab_id_t { id_mid = SET: dlcapabsetqid(&id_mid, RD(q)); GET: dlcapabgetqid(&id_mid); id_subcap.dl_cap = DL_CAPAB_MDT; id_subcap.dl_length = sizeof (dl_capab_mdt_t); dl_capab_mdt_t { ... } } } } A pass-thru module which doesn't understand the DL_CAPABILITY framework or any of the contained sub-capabilities will not have its upper read queue pointer stamped, and therefore ip will be able to detect this by case and discard the received DL_CAPABILITY_ACK. In order to be backwards compatible with current drivers which don't incorporate this identification sub- capability scheme, ip is modified to handle both cases: 1. An unwrapped sub-capability (current MDT and IPsec HA) will only get accepted if and only if there is at most one STREAMS module beneath ip. In other words, current MDT and IPsec HA sub- capabilities advertised by currently-shipping Cassini and Venus drivers will only be enabled if there is no intermediate module in between ip and the driver. 2. Wrapped MDT and IPsec HA, as well as other future sub-capabilities incorporating dl_mid_t structure, will only get accepted by ip if and only if the upper read queue pointer of the module beneath ip matches that of the queue pointer returned by the dlcapabgetqid() on the corresponding sub-capability. Otherwise, those sub-capabilities are discarded as their sources can't be determined. It is also noted that both M_DATA fastpath and the current Solaris hardware checksum interface don't utilize the DL_CAPABILITY negotiation framework. It is expected that the negotiation of these features be ported over to the DL_CAPABILITY framework, and their progresses be documented in 4703353, 4796051, and PSARC 2003/264. BACKWARDS COMPATIBILITY ----------------------- The entire purpose of this change is enhanced backwards compatibility. But there are conflicting requirements. We also need be compatible early drivers supporting the DL_CAPABILITY interface (cassini/MDT, venus). Fortunately, we have an existing configuration "knob" (see PSARC 2002/322) we can use to detect that a customer has explicitly requested the use of the DL_CAPABILITY feature. If this project is integrated into a patch/update release, existing behavior will be preserved if "ip_use_dl_cap" is set to 1 in /etc/system -- i.e., intermediate module detection will be disabled -- allowing the use of drivers which don't play along with intermediate module detection. If ip_use_dl_cap is left alone, DL_CAPABILITY negotiation will happen anyway but with the intermediate module detection part enabled, rendering it safe in the presence of intermediate modules. In a subsequent minor release, this mechanism will always be enabled and the use of "ip_use_dl_cap" will be deprecated. REFERENCES ---------- 4703353 ip's fast-path mechanism breaks DLPIv2 4796051 Solaris needs a more complete HW checksumming support 4771245 DL_CAPABILITY-ignorant intermediate module detection PSARC 1996/173 TCP/IP support for SAHI3 hardware checksuming PSARC 1999/348 I_MUXID2FD, I_INSERT, and I_REMOVE PSARC 1999/375 Re-plumbing Options for ifconfig(1M) PSARC 2001/070 DL_{CAPABILITY,CONTROL}_REQ PSARC 2002/276 TCP Multi-Data Transmit PSARC 2002/322 DL_CAPABILITY knob PSARC 2003/264 Extended IP checksum offload interface From sacadmin Thu May 8 16:33:27 2003 X-Authentication-Warning: jurassic.eng.sun.com: masputra owned process doing -bs Date: Thu, 8 May 2003 16:31:36 -0700 (PDT) From: Adi Masputra To: psarc@sac.eng.sun.com cc: Adi.Masputra@Sun.COM, "" , "" Subject: Re: 2003/263: DL_CAPABILITY-ignorant intermediate module detection MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Content-Length: 19789 During a private conversation, Peter Memishian recently brought up the lack of details in the Proposal which are needed by module writers. These details have to do with what needs to be done by a DL_CAPABILITY- compliant module (not driver) prior to proclaiming one or more supported sub-capabilities in the DL_CAPABILITY_ACK on its way up. To make things clearer, I've added some details (marked appropriately) at the end of the Proposal section. The updated text is included below. Adi ----- DL_CAPABILITY-ignorant intermediate module detection BACKGROUND ---------- In the recent years, many non-standard (non-DLPIv2) mechanisms of exchanging messages between ip module and the DLPI device driver were introduced to Solaris, e.g. M_DATA fastpath, hardware checksum offload [PSARC 1996/173], Multidata Transmit (MDT) [PSARC 2002/276], IPsec Hardware Acceleration (HA) [PSARC 2001/070], etc. These new interfaces were needed in order to get around the limitation of the DLPIv2 interface between a Data Link Provider and Consumer, mainly for performance improvements (some of which are attributed to driver/hardware offload capabilities). It is expected that many other future networking projects (zero-copy, etc.) would rely on some other similarly negotiated interfaces. IP's usage of these interfaces depends on the outcome of capability negotiations that happen between ip module and the driver. Such negotation scheme works well in cases where the driver is the module directly underneath ip. In other cases where there is one or more intermediate module between ip and the driver, this scheme does not provide a way of detecting their capabilities. Therefore, the introduction of modules in between often results in a breakage. In addition, the current scheme of capabilities negotiation typically happens during the plumbing, or construction of the IP interface stream. While this works for statically-plumbed intermediate modules, it is not sufficient to handle the case where a module is dynamically introduced into an existing IP interface stream. PROBLEMS -------- The exchange of capability negotiations are done in several ways. The M_DATA fastpath negotiation is triggered by a M_IOCTL message. The current hardware checksum offload uses a M_CTL message as vehicle, while the newer features such as MDT and IPsec HA use the newly- defined DL_CAPABILITY_{REQ,ACK} inside the M_PROTO message type. An intermediate module which does not understand any of those messages may either free them, or forward them to the next module. The former action can easily be dealt with, as IP can handle the "lost" negotiation message case (it won't enable or utilize those features). The latter action, however, is rather dangerous as the response may be interpreted to be coming from such intermediate module, which in reality was generated by a module underneath it. The following example depicts such case: +----------+ | ip | +----------+ | +----------+ | firewall | No knowledge of DL_CAPABILITY_REQ interface +----------+ | +----------+ | ce | Supports capability (MDT) +----------+ A firewall module which doesn't understand the Multidata interface (as well as the DL_CAPABILITY framework) is plumbed between ip and the ce driver, either statically or dynamically. Because it passes through any unknown messages, the DL_CAPABILITY_ACK coming from the driver gets forwarded (passed through) onto ip. Since ip has no way of determining the origin of the acknowledgment, it assumes that the module immediately below (firewall) was the source of the acknowledgement, and hence it will end up utilizing Multidata to pass down the packets. When the firewall module receives the Multidata messages, it may either forward them down to the driver or free them. Either action is considered a breakage: the former allows packets to go out of the machine without first being inspected by the firewall, and the latter will most likely break any existing network connections whose packets are encapsulated within those Multidata messages. Any STREAMS module may be interposed between ip and the driver by way of autopush(1M), if_tcp(7P), and ifconfig(1M) modinsert/modremove. In the static plumbing case, using either autopush(1M) or if_tcp(7P) method, the module is pushed on top an existing stream opened to the device driver, and the ip module is further pushed on top of it. At this point, the IP interface is considered to be plumbed or constructed, and the ip module may initiate the capability negotiation exchange process. The ability to dynamically change the plumbing of the IP stream was introduced in PSARC/1999/348 - _I_MUXID2FD, _I_INSERT, and _I_REMOVE ioctls. In PSARC/1999/375 - Re-plumbing Options for ifconfig(1M) case, 2 new options to ifconfig(1M) were introduced that can change the plumbing of the IP stream dynamically (modinsert and modremove). These options are classified as Evolving. Because a dynamic re-plumbing typically happens after the exchange of capabilities negotiation, a breakage will occur when the newly- inserted module is not capable of interpreting any previously- negotiated and enabled features. The problem space can therefore be summarized as follows: 1. The lack of module identification token in the DL_CAPABILITY framework which would allow for verification of incoming sub- capabilities within DL_CAPABILITY_ACK messages, in order to detect pass-thru intermediate modules. 2. There is currently no formal mechanism to reset and re-negotiate the capabilities due to a change in the stream's plumbing, such as by way of ifconfig(1M) modinsert and modremove. PROPOSAL -------- This fast-track proposal introduces an extension to the DL_CAPABILITY framework to allow for detecting "pass-thru" modules, as well as to turn on or off specific capabilities on a case-by-case basis by ip in order to perform the capability resetting and re-negotiating actions during a dynamic re-plumbing event. The requested release binding is micro. This proposal creates the following interfaces: Exported Interfaces: +-----------------------+----------------------------------+-----------------+ | Interface | Classification | Comments | |-----------------------+----------------------------------+-----------------| | DL_CAPAB_ID_WRAPPER | Contracted Consolidation Private | sys/dlpi.h | | dl_mid_t | Contracted Consolidation Private | sys/dlpi.h | | dl_capab_id_t | Contracted Consolidation Private | sys/dlpi.h | | dlcapabsetqid() | Contracted Consolidation Private | sys/dlpi.h | | dlcapabgetqid() | Contracted Consolidation Private | sys/dlpi.h | |-----------------------+----------------------------------+-----------------| Imported Interfaces: +-----------------------+----------------------------------+-----------------+ | Interface | Classification | Comments | |-----------------------+----------------------------------+-----------------| | DL_CAPABILITY_REQ | Contracted Consolidation Private | PSARC/2001/070 | | DL_CAPABILITY_ACK | Contracted Consolidation Private | PSARC/2001/070 | | dl_capability_sub_t | Contracted Consolidation Private | PSARC/2001/070 | | dl_capab_ipsec_t | Contracted Consolidation Private | PSARC/2001/070 | | dl_capab_mdt_t | Contracted Consolidation Private | PSARC/2002/276 | +-----------------------+----------------------------------+-----------------+ From : #define DL_CAPAB_ID_WRAPPER 0x00 /* * Module ID to be included in new sub-capability structures. * Existing sub-capabilities lacking an identification token, * e.g. MDT and IPSEC hardware acceleration, need to be * encapsulated within the ID sub-capability. Access to this * structure must be done through dlcapab{set,get}qid(). */ typedef struct { t_uscalar_t mid[4]; /* private fields */ } dl_mid_t; /* * Module ID wrapper (follows dl_capability_sub_t) */ typedef struct { dl_mid_t id_mid; /* module ID */ dl_capability_sub_t id_subcap; /* sub-capability */ } dl_capab_id_t; extern void dlcapabsetqid(dl_mid_t *, queue_t *); extern queue_t *dlcapabgetqid(dl_mid_t *); A token (dl_mid_t) which allows for the module understanding the sub- capabilities to record its identification is needed. Currently, the identification token is defined to be the upper read queue pointer value of the module. By having a module or driver go through the DL_CAPABILITY_ACK message (on its way up) and "stamp" its upper read queue value on the sub-capabilities within, ip can detect a pass-thru module by way of comparing the upper read queue of the module immediately below with the value inside the sub-capability. At the time of writing, both MDT and IPsec HA features in the networking stack and device drivers have been delivered into Solaris, and possibly deployed on customer sites. Modifying dl_capab_mdt and dl_capab_ipsec_t in order to introduce an "identification" token will certainly break existing device drivers. Therefore, a "wrapper" sub- capability (dl_capab_id_t) used to encapsulate those sub-capabilities lacking the identification token is proposed. Existing MDT and IPsec HA sub-capabilities can be wrapped with the above "wrapper" sub-capability, and ip can be modified to understand such sub-capability in addition to the existing ones that it supports. New sub-capabilities defined in future are required to incorporate the dl_mid_t structure in order to provide for an identification token. Ways to set and obtain the queue pointer (the module identification) from within the dl_mid_t, via the dlcapab{set,get}qid() routines, are also introduced. Upon detecting dynamic re-plumbing events, ip would reset (disable) previously-negotiated capabilities -- details on how this is done is beyond the scope of this case, and is documented in 4771245. The reason for the "reset" is to establish that only vanilla DLPI is acceptable (i.e. modules should revert to sending only strict DLPIv2). This is done by having ip compose and send down a DL_CAPABILITY_REQ message which consists of the previously-negotiated capabilities indicating that they be disabled and no longer used. Such indication is done by way of clearing the "enable" bit defined per sub-capability. Afterwards, ip probes its lower-stream to find out about any new (or lack of) capabilities by sending down an empty DL_CAPABILITY_REQ message per PSARC 2001/070. An intermediate module which doesn't understand the DL_CAPABILITY framework or sub-capabilities within will be able to be detected, and no breakage will occur. In order for the above to work, the rule set of PSARC 2001/070 needs to be modified to include the following: a. All sub-capabilities must be able to be disabled. The current MDT and IPsec HA sub-capabilities contain flags fields whose enable bits can be cleared. Future sub-capabilities are required to implement similar mechanism. b. The enable/disable bit in previously-advertised (those which have been enabled by ip) sub-capabilities is read-only for modules and drivers, and can only be modified by ip. This would guarantee that an intermediate module doesn't interfere with the capability reset done by ip during dynamic re-plumbing. c. Future sub-capability definitions must include the dl_mid_t structure in order to provide for module identification. d. All intermediate modules (except drivers) are required to forward the DL_CAPABILITY_REQ message (received from the module above) downstream. e. All modules and drivers which understand the DL_CAPABILITY framework must record its upper queue pointer value, by calling dlcapabsetqid(), on each of the sub-capability within the DL_CAPABILITY_ACK which it recognizes and wishes for ip to enable. This must be done prior to passing the DL_CAPABILITY_ACK upstream. f. A module which initiates a DL_CAPABILITY_REQ is required to check each sub-capability returned in a DL_CAPABILITY_ACK, by calling dlcapabgetqid(), to see if the returned queue pointer matches that of the upper read queue pointer of the module immediately below. If it matches, the sub-capability may be enabled, otherwise, it should be discarded. + In addition to the above rules, module (not driver) writers are + required to perform additional checks in fulfilling (e), in order to + handle the case where the underlying module/driver is not capable of + the sub-capabilities which may be proclaimed by the module in the + returning DL_CAPABILITY_ACK, as explained in the following example: + + +----------+ + | ip | Supports A,B,C, ... + +----------+ + | + +----------+ + | mod1 | Supports A,B,C + +----------+ + | + +----------+ + | mod2 | Supports A,C + +----------+ + | + +----------+ + | driver | Supports A,B,C + +----------+ + + Modules mod1 and mod2 are plumbed in between ip and the driver. + All modules are capable of supporting capabilities A and C, + while B capability is only supported by ip, mod1 and driver. + Prior to sending a DL_CAPABILITY_ACK to ip, mod1 might record + its token in all of the sub-capabilities, since it's capable + to handle them. However, this may not be correct as mod2 + doesn't have the ability to handle capability B, and a breakage + may occur when mod1 uses capability B to communicate to mod2. + + Therefore, all modules claiming to support the DL_CAPABILITY framework + are required to check to see if each of the sub-capability it supports + can be handled by the the module/driver below, by way of comparing the + value returned by dlcapabgetqid() in the returning DL_CAPABILITY_ACK + against the value of the read queue pointer of the underlying module/ + driver. If the module below doesn't support the sub-capability, i.e. + the values don't match, then the following actions may be taken: + + 1. If it is somehow possible to convert messages formatted according + to the corresponding sub-capability into those which can be + understood by the module below, then it may report to the module + above that the sub-capability is supported, by recording its token + in the sub-capability prior to sending up the DL_CAPABILITY_ACK. + However, it must perform the necessary works to convert those + messages prior to sending it downstream. + + 2. If a conversion is not possible or desirable, then the module + may not record its token in the corresponding sub-capability, + which effectively renders the sub-capability unproclaimed by + the module upon returning the DL_CAPABILITY_ACK. The module + above can then make the necessary decisions in the similar + fashion based on these outlines. + + Drivers are not required to perform the above checks, since they + are at the bottom of the stream. DETAILS ------- Prior to passing up a DL_CAPABILITY_ACK (i.e. in the rput routine), a module must go through each sub-capability looking for those which it supports, and stamp its upper read queue pointer on them. In the case of a driver, this process is done as part of initializing the sub-capability that it is about to report to ip. The following example of pseudo structures depict the current MDT sub-capability as reported by drivers: dl_capability_ack { dl_primitive = DL_CAPABILITY_ACK; dl_sub_offset = sizeof (dl_capability_ack_t); dl_sub_length = sizeof (dl_capability_sub_t) + sizeof (dl_capab_mdt_t); dl_capability_sub_t { dl_cap = DL_CAPAB_MDT; dl_length = sizeof (dl_capab_mdt_t); dl_capab_mdt_t { ... } } } When using the wrapper sub-capability, the above example can be represented as: dl_capability_ack { dl_primitive = DL_CAPABILITY_ACK; dl_sub_offset = sizeof (dl_capability_ack_t); dl_sub_length = sizeof (dl_capability_sub_t) + sizeof (dl_capability_id_t) + sizeof (dl_capab_mdt_t); dl_capability_sub_t { dl_cap = DL_CAPAB_ID_WRAPPER; dl_length = sizeof (dl_capab_id_t) + sizeof (dl_capab_mdt_t); dl_capab_id_t { id_mid = SET: dlcapabsetqid(&id_mid, RD(q)); GET: dlcapabgetqid(&id_mid); id_subcap.dl_cap = DL_CAPAB_MDT; id_subcap.dl_length = sizeof (dl_capab_mdt_t); dl_capab_mdt_t { ... } } } } A pass-thru module which doesn't understand the DL_CAPABILITY framework or any of the contained sub-capabilities will not have its upper read queue pointer stamped, and therefore ip will be able to detect this by case and discard the received DL_CAPABILITY_ACK. In order to be backwards compatible with current drivers which don't incorporate this identification sub- capability scheme, ip is modified to handle both cases: 1. An unwrapped sub-capability (current MDT and IPsec HA) will only get accepted if and only if there is at most one STREAMS module beneath ip. In other words, current MDT and IPsec HA sub- capabilities advertised by currently-shipping Cassini and Venus drivers will only be enabled if there is no intermediate module in between ip and the driver. 2. Wrapped MDT and IPsec HA, as well as other future sub-capabilities incorporating dl_mid_t structure, will only get accepted by ip if and only if the upper read queue pointer of the module beneath ip matches that of the queue pointer returned by the dlcapabgetqid() on the corresponding sub-capability. Otherwise, those sub-capabilities are discarded as their sources can't be determined. It is also noted that both M_DATA fastpath and the current Solaris hardware checksum interface don't utilize the DL_CAPABILITY negotiation framework. It is expected that the negotiation of these features be ported over to the DL_CAPABILITY framework, and their progresses be documented in 4703353, 4796051, and PSARC 2003/264. REFERENCES ---------- 4703353 ip's fast-path mechanism breaks DLPIv2 4796051 Solaris needs a more complete HW checksumming support 4771245 DL_CAPABILITY-ignorant intermediate module detection PSARC 1996/173 TCP/IP support for SAHI3 hardware checksuming PSARC 1999/348 I_MUXID2FD, I_INSERT, and I_REMOVE PSARC 1999/375 Re-plumbing Options for ifconfig(1M) PSARC 2001/070 DL_{CAPABILITY,CONTROL}_REQ PSARC 2002/276 TCP Multi-Data Transmit PSARC 2003/264 Extended IP checksum offload interface From sacadmin Thu May 8 16:58:31 2003 X-Authentication-Warning: jurassic.eng.sun.com: masputra owned process doing -bs Date: Thu, 8 May 2003 16:56:41 -0700 (PDT) From: Adi Masputra To: psarc@sac.eng.sun.com cc: peter.memishian@Sun.COM, "" , "" Subject: Re: 2003/263: DL_CAPABILITY-ignorant intermediate module detection MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Content-Length: 21198 I apologize for the previous mail. I realized that Bill had made some modifications on the original text prior to submitting the case, and that my recent changes on the proposal were inserted on the original one. I'm therefore including the "correct" version of the proposal. Adi Adi Masputra writes: > During a private conversation, Peter Memishian recently brought up > the lack of details in the Proposal which are needed by module writers. > > These details have to do with what needs to be done by a DL_CAPABILITY- > compliant module (not driver) prior to proclaiming one or more supported > sub-capabilities in the DL_CAPABILITY_ACK on its way up. > > To make things clearer, I've added some details (marked appropriately) > at the end of the Proposal section. The updated text is included below. > > Adi > > ----- DL_CAPABILITY-ignorant intermediate module detection BACKGROUND ---------- In the recent years, many non-standard (non-DLPIv2) mechanisms of exchanging messages between ip module and the DLPI device driver were introduced to Solaris, e.g. M_DATA fastpath, hardware checksum offload [PSARC 1996/173], Multidata Transmit (MDT) [PSARC 2002/276], IPsec Hardware Acceleration (HA) [PSARC 2001/070], etc. These new interfaces were needed in order to get around the limitation of the DLPIv2 interface between a Data Link Provider and Consumer, mainly for performance improvements (some of which are attributed to driver/hardware offload capabilities). It is expected that many other future networking projects (zero-copy, etc.) would rely on some other similarly negotiated interfaces. IP's usage of these interfaces depends on the outcome of capability negotiations that happen between ip module and the driver. Such negotation scheme works well in cases where the driver is the module directly underneath ip. In other cases where there is one or more intermediate module between ip and the driver, this scheme does not provide a way of detecting their capabilities. Therefore, the introduction of modules in between often results in a breakage. In addition, the current scheme of capabilities negotiation typically happens during the plumbing, or construction of the IP interface stream. While this works for statically-plumbed intermediate modules, it is not sufficient to handle the case where a module is dynamically introduced into an existing IP interface stream. PROBLEMS -------- The exchange of capability negotiations are done in several ways. The M_DATA fastpath negotiation is triggered by a M_IOCTL message. The current hardware checksum offload uses a M_CTL message as vehicle, while the newer features such as MDT and IPsec HA use the newly- defined DL_CAPABILITY_{REQ,ACK} inside the M_PROTO message type. An intermediate module which does not understand any of those messages may either free them, or forward them to the next module. The former action can easily be dealt with, as IP can handle the "lost" negotiation message case (it won't enable or utilize those features). The latter action, however, is rather dangerous as the response may be interpreted to be coming from such intermediate module, which in reality was generated by a module underneath it. The following example depicts such case: +----------+ | ip | +----------+ | +----------+ | firewall | No knowledge of DL_CAPABILITY_REQ interface +----------+ | +----------+ | ce | Supports capability (MDT) +----------+ A firewall module which doesn't understand the Multidata interface (as well as the DL_CAPABILITY framework) is plumbed between ip and the ce driver, either statically or dynamically. Because it passes through any unknown messages, the DL_CAPABILITY_ACK coming from the driver gets forwarded (passed through) onto ip. Since ip has no way of determining the origin of the acknowledgment, it assumes that the module immediately below (firewall) was the source of the acknowledgement, and hence it will end up utilizing Multidata to pass down the packets. When the firewall module receives the Multidata messages, it may either forward them down to the driver or free them. Either action is considered a breakage: the former allows packets to go out of the machine without first being inspected by the firewall, and the latter will most likely break any existing network connections whose packets are encapsulated within those Multidata messages. Any STREAMS module may be interposed between ip and the driver by way of autopush(1M), if_tcp(7P), and ifconfig(1M) modinsert/modremove. In the static plumbing case, using either autopush(1M) or if_tcp(7P) method, the module is pushed on top an existing stream opened to the device driver, and the ip module is further pushed on top of it. At this point, the IP interface is considered to be plumbed or constructed, and the ip module may initiate the capability negotiation exchange process. The ability to dynamically change the plumbing of the IP stream was introduced in PSARC/1999/348 - _I_MUXID2FD, _I_INSERT, and _I_REMOVE ioctls. In PSARC/1999/375 - Re-plumbing Options for ifconfig(1M) case, 2 new options to ifconfig(1M) were introduced that can change the plumbing of the IP stream dynamically (modinsert and modremove). These options are classified as Evolving. Because a dynamic re-plumbing typically happens after the exchange of capabilities negotiation, a breakage will occur when the newly- inserted module is not capable of interpreting any previously- negotiated and enabled features. The problem space can therefore be summarized as follows: 1. The lack of module identification token in the DL_CAPABILITY framework which would allow for verification of incoming sub- capabilities within DL_CAPABILITY_ACK messages, in order to detect pass-thru intermediate modules. 2. There is currently no formal mechanism to reset and re-negotiate the capabilities due to a change in the stream's plumbing, such as by way of ifconfig(1M) modinsert and modremove. PROPOSAL -------- This fast-track proposal introduces an extension to the DL_CAPABILITY framework to allow for detecting "pass-thru" modules, as well as to turn on or off specific capabilities on a case-by-case basis by ip in order to perform the capability resetting and re-negotiating actions during a dynamic re-plumbing event. The requested release binding is micro. This proposal creates the following interfaces: Exported Interfaces: +-----------------------+----------------------------------+-----------------+ | Interface | Classification | Comments | |-----------------------+----------------------------------+-----------------| | DL_CAPAB_ID_WRAPPER | Consolidation Private | sys/dlpi.h | | dl_mid_t | Consolidation Private | sys/dlpi.h | | dl_capab_id_t | Consolidation Private | sys/dlpi.h | | dlcapabsetqid() | Consolidation Private | sys/dlpi.h | | dlcapabgetqid() | Consolidation Private | sys/dlpi.h | |-----------------------+----------------------------------+-----------------| Imported Interfaces: +-----------------------+----------------------------------+-----------------+ | Interface | Classification | Comments | |-----------------------+----------------------------------+-----------------| | DL_CAPABILITY_REQ | Consolidation Private | PSARC/2001/070 | | DL_CAPABILITY_ACK | Consolidation Private | PSARC/2001/070 | | dl_capability_sub_t | Consolidation Private | PSARC/2001/070 | | dl_capab_ipsec_t | Consolidation Private | PSARC/2001/070 | | dl_capab_mdt_t | Consolidation Private | PSARC/2002/276 | +-----------------------+----------------------------------+-----------------+ From : #define DL_CAPAB_ID_WRAPPER 0x00 /* * Module ID to be included in new sub-capability structures. * Existing sub-capabilities lacking an identification token, * e.g. MDT and IPSEC hardware acceleration, need to be * encapsulated within the ID sub-capability. Access to this * structure must be done through dlcapab{set,get}qid(). */ typedef struct { t_uscalar_t mid[4]; /* private fields */ } dl_mid_t; /* * Module ID wrapper (follows dl_capability_sub_t) */ typedef struct { dl_mid_t id_mid; /* module ID */ dl_capability_sub_t id_subcap; /* sub-capability */ } dl_capab_id_t; extern void dlcapabsetqid(dl_mid_t *, queue_t *); extern queue_t *dlcapabgetqid(dl_mid_t *); A token (dl_mid_t) which allows for the module understanding the sub- capabilities to record its identification is needed. Currently, the identification token is defined to be the upper read queue pointer value of the module. By having a module or driver go through the DL_CAPABILITY_ACK message (on its way up) and "stamp" its upper read queue value on the sub-capabilities within, ip can detect a pass-thru module by way of comparing the upper read queue of the module immediately below with the value inside the sub-capability. At the time of writing, both MDT and IPsec HA features in the networking stack and device drivers have been delivered into Solaris, and possibly deployed on customer sites. Modifying dl_capab_mdt and dl_capab_ipsec_t in order to introduce an "identification" token will certainly break existing device drivers. Therefore, a "wrapper" sub- capability (dl_capab_id_t) used to encapsulate those sub-capabilities lacking the identification token is proposed. Existing MDT and IPsec HA sub-capabilities can be wrapped with the above "wrapper" sub-capability, and ip can be modified to understand such sub-capability in addition to the existing ones that it supports. New sub-capabilities defined in future are required to incorporate the dl_mid_t structure in order to provide for an identification token. Ways to set and obtain the queue pointer (the module identification) from within the dl_mid_t, via the dlcapab{set,get}qid() routines, are also introduced. Upon detecting dynamic re-plumbing events, ip would reset (disable) previously-negotiated capabilities -- details on how this is done is beyond the scope of this case, and is documented in 4771245. The reason for the "reset" is to establish that only vanilla DLPI is acceptable (i.e. modules should revert to sending only strict DLPIv2). This is done by having ip compose and send down a DL_CAPABILITY_REQ message which consists of the previously-negotiated capabilities indicating that they be disabled and no longer used. Such indication is done by way of clearing the "enable" bit defined per sub-capability. Afterwards, ip probes its lower-stream to find out about any new (or lack of) capabilities by sending down an empty DL_CAPABILITY_REQ message per PSARC 2001/070. An intermediate module which doesn't understand the DL_CAPABILITY framework or sub-capabilities within will be able to be detected, and no breakage will occur. In order for the above to work, the rule set of PSARC 2001/070 needs to be modified to include the following: a. All sub-capabilities must be able to be disabled. The current MDT and IPsec HA sub-capabilities contain flags fields whose enable bits can be cleared. Future sub-capabilities are required to implement similar mechanism. b. The enable/disable bit in previously-advertised (those which have been enabled by ip) sub-capabilities is read-only for modules and drivers, and can only be modified by ip. This would guarantee that an intermediate module doesn't interfere with the capability reset done by ip during dynamic re-plumbing. c. Future sub-capability definitions must include the dl_mid_t structure in order to provide for module identification. d. All intermediate modules (except drivers) are required to forward the DL_CAPABILITY_REQ message (received from the module above) downstream. e. All modules and drivers which understand the DL_CAPABILITY framework must record its upper queue pointer value, by calling dlcapabsetqid(), on each of the sub-capability within the DL_CAPABILITY_ACK which it recognizes and wishes for ip to enable. This must be done prior to passing the DL_CAPABILITY_ACK upstream. f. A module which initiates a DL_CAPABILITY_REQ is required to check each sub-capability returned in a DL_CAPABILITY_ACK, by calling dlcapabgetqid(), to see if the returned queue pointer matches that of the upper read queue pointer of the module immediately below. If it matches, the sub-capability may be enabled, otherwise, it should be discarded. + In addition to the above rules, module (not driver) writers are + required to perform additional checks in fulfilling (e), in order to + handle the case where the underlying module/driver is not capable of + the sub-capabilities which may be proclaimed by the module in the + returning DL_CAPABILITY_ACK, as explained in the following example: + + +----------+ + | ip | Supports A,B,C, ... + +----------+ + | + +----------+ + | mod1 | Supports A,B,C + +----------+ + | + +----------+ + | mod2 | Supports A,C + +----------+ + | + +----------+ + | driver | Supports A,B,C + +----------+ + + Modules mod1 and mod2 are plumbed in between ip and the driver. + All modules are capable of supporting capabilities A and C, + while B capability is only supported by ip, mod1 and driver. + Prior to sending a DL_CAPABILITY_ACK to ip, mod1 might record + its token in all of the sub-capabilities, since it's capable + to handle them. However, this may not be correct as mod2 + doesn't have the ability to handle capability B, and a breakage + may occur when mod1 uses capability B to communicate to mod2. + + Therefore, all modules claiming to support the DL_CAPABILITY framework + are required to check to see if each of the sub-capability it supports + can be handled by the the module/driver below, by way of comparing the + value returned by dlcapabgetqid() in the returning DL_CAPABILITY_ACK + against the value of the read queue pointer of the underlying module/ + driver. If the module below doesn't support the sub-capability, i.e. + the values don't match, then the following actions may be taken: + + 1. If it is somehow possible to convert messages formatted according + to the corresponding sub-capability into those which can be + understood by the module below, then it may report to the module + above that the sub-capability is supported, by recording its token + in the sub-capability prior to sending up the DL_CAPABILITY_ACK. + However, it must perform the necessary works to convert those + messages prior to sending it downstream. + + 2. If a conversion is not possible or desirable, then the module + may not record its token in the corresponding sub-capability, + which effectively renders the sub-capability unproclaimed by + the module upon returning the DL_CAPABILITY_ACK. The module + above can then make the necessary decisions in the similar + fashion based on these outlines. + + Drivers are not required to perform the above checks, since they + are at the bottom of the stream. DETAILS ------- Prior to passing up a DL_CAPABILITY_ACK (i.e. in the rput routine), a module must go through each sub-capability looking for those which it supports, and stamp its upper read queue pointer on them. In the case of a driver, this process is done as part of initializing the sub-capability that it is about to report to ip. The following example of pseudo structures depict the current MDT sub-capability as reported by drivers: dl_capability_ack { dl_primitive = DL_CAPABILITY_ACK; dl_sub_offset = sizeof (dl_capability_ack_t); dl_sub_length = sizeof (dl_capability_sub_t) + sizeof (dl_capab_mdt_t); dl_capability_sub_t { dl_cap = DL_CAPAB_MDT; dl_length = sizeof (dl_capab_mdt_t); dl_capab_mdt_t { ... } } } When using the wrapper sub-capability, the above example can be represented as: dl_capability_ack { dl_primitive = DL_CAPABILITY_ACK; dl_sub_offset = sizeof (dl_capability_ack_t); dl_sub_length = sizeof (dl_capability_sub_t) + sizeof (dl_capability_id_t) + sizeof (dl_capab_mdt_t); dl_capability_sub_t { dl_cap = DL_CAPAB_ID_WRAPPER; dl_length = sizeof (dl_capab_id_t) + sizeof (dl_capab_mdt_t); dl_capab_id_t { id_mid = SET: dlcapabsetqid(&id_mid, RD(q)); GET: dlcapabgetqid(&id_mid); id_subcap.dl_cap = DL_CAPAB_MDT; id_subcap.dl_length = sizeof (dl_capab_mdt_t); dl_capab_mdt_t { ... } } } } A pass-thru module which doesn't understand the DL_CAPABILITY framework or any of the contained sub-capabilities will not have its upper read queue pointer stamped, and therefore ip will be able to detect this by case and discard the received DL_CAPABILITY_ACK. In order to be backwards compatible with current drivers which don't incorporate this identification sub- capability scheme, ip is modified to handle both cases: 1. An unwrapped sub-capability (current MDT and IPsec HA) will only get accepted if and only if there is at most one STREAMS module beneath ip. In other words, current MDT and IPsec HA sub- capabilities advertised by currently-shipping Cassini and Venus drivers will only be enabled if there is no intermediate module in between ip and the driver. 2. Wrapped MDT and IPsec HA, as well as other future sub-capabilities incorporating dl_mid_t structure, will only get accepted by ip if and only if the upper read queue pointer of the module beneath ip matches that of the queue pointer returned by the dlcapabgetqid() on the corresponding sub-capability. Otherwise, those sub-capabilities are discarded as their sources can't be determined. It is also noted that both M_DATA fastpath and the current Solaris hardware checksum interface don't utilize the DL_CAPABILITY negotiation framework. It is expected that the negotiation of these features be ported over to the DL_CAPABILITY framework, and their progresses be documented in 4703353, 4796051, and PSARC 2003/264. BACKWARDS COMPATIBILITY ----------------------- The entire purpose of this change is enhanced backwards compatibility. But there are conflicting requirements. We also need be compatible early drivers supporting the DL_CAPABILITY interface (cassini/MDT, venus). Fortunately, we have an existing configuration "knob" (see PSARC 2002/322) we can use to detect that a customer has explicitly requested the use of the DL_CAPABILITY feature. If this project is integrated into a patch/update release, existing behavior will be preserved if "ip_use_dl_cap" is set to 1 in /etc/system -- i.e., intermediate module detection will be disabled -- allowing the use of drivers which don't play along with intermediate module detection. If ip_use_dl_cap is left alone, DL_CAPABILITY negotiation will happen anyway but with the intermediate module detection part enabled, rendering it safe in the presence of intermediate modules. In a subsequent minor release, this mechanism will always be enabled and the use of "ip_use_dl_cap" will be deprecated. REFERENCES ---------- 4703353 ip's fast-path mechanism breaks DLPIv2 4796051 Solaris needs a more complete HW checksumming support 4771245 DL_CAPABILITY-ignorant intermediate module detection PSARC 1996/173 TCP/IP support for SAHI3 hardware checksuming PSARC 1999/348 I_MUXID2FD, I_INSERT, and I_REMOVE PSARC 1999/375 Re-plumbing Options for ifconfig(1M) PSARC 2001/070 DL_{CAPABILITY,CONTROL}_REQ PSARC 2002/276 TCP Multi-Data Transmit PSARC 2002/322 DL_CAPABILITY knob PSARC 2003/264 Extended IP checksum offload interface From sacadmin Fri May 9 06:31:25 2003 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Date: Fri, 9 May 2003 09:29:32 -0400 From: James Carlson To: Adi Masputra Cc: psarc@sac.eng.sun.com, peter.memishian@sun.com, "" Subject: Re: 2003/263: DL_CAPABILITY-ignorant intermediate module detection Content-Length: 1061 Adi Masputra writes: > extern void dlcapabsetqid(dl_mid_t *, queue_t *); > extern queue_t *dlcapabgetqid(dl_mid_t *); It would seem a little safer to me to have that second function actually be: extern boolean_t dlcapabcheckqid(const dl_mid_t *, const queue_t *); That is, hide the queue pointer comparison inside the function so that (in the future) if the check is ever based on some attribute of the queue rather than the pointer itself, this won't need to be changed. (And the former should probably use 'const queue_t *' ...) > In a subsequent minor release, this mechanism will always be enabled > and the use of "ip_use_dl_cap" will be deprecated. There's that word again. ;-} Does "deprecated" mean "just plain removed" or "still there but does nothing" or "still works but we tell people not to use it?" -- James Carlson, Solaris Networking Sun Microsystems / 1 Network Drive 71.234W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.497N Fax +1 781 442 1677 From sacadmin Fri May 9 10:29:20 2003 Date: Fri, 09 May 2003 10:26:15 -0700 From: Jordan Brown X-Accept-Language: fr, en MIME-Version: 1.0 To: James Carlson CC: Adi Masputra , psarc@sac.eng.sun.com, peter.memishian@sun.com, sommerfeld@east.sun.com Subject: Re: 2003/263: DL_CAPABILITY-ignorant intermediate module detection Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Content-Length: 1376 > Does "deprecated" mean "just plain removed" or "still there but does > nothing" or "still works but we tell people not to use it?" In my opinion, only the last is a correct interpretation of the word. I believe that anybody who uses it for the first or second is simply incorrect. I believe that definitions 2 and 3 below are the ones that apply. Main Entry: dep·re·cate Pronunciation: 'de-pri-"kAt Function: transitive verb Inflected Form(s): -cat·ed; -cat·ing Etymology: Latin deprecatus, past participle of deprecari to avert by prayer, from de- + precari to pray -- more at PRAY Date: 1628 1 a archaic : to pray against (as an evil) b : to seek to avert 2 : to express disapproval of 3 a : PLAY DOWN : make little of b : BELITTLE, DISPARAGE - dep·re·cat·ing·ly /-"kA-ti[ng]-lE/ adverb - dep·re·ca·tion /"de-pri-'kA-sh&n/ noun (My point here is that we have a good word with a useful meaning. If we ever allow people to use it in the wrong way, we lose the word. Should anybody use it - or any other word - in the wrong way, they should be corrected rather than tolerated. This is especially true of words that are used in specifications.) From sacadmin Fri May 9 10:38:22 2003 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Date: Fri, 9 May 2003 13:36:29 -0400 From: James Carlson To: Jordan Brown Cc: Adi Masputra , psarc@sac.eng.sun.com, peter.memishian@sun.com Subject: Re: 2003/263: DL_CAPABILITY-ignorant intermediate module detection Content-Length: 1090 Jordan Brown writes: > (My point here is that we have a good word with a useful meaning. If we > ever allow people to use it in the wrong way, we lose the word. Should > anybody use it - or any other word - in the wrong way, they should be > corrected rather than tolerated. This is especially true of words that > are used in specifications.) As much as I agree with you on this point, I think we should still be aware that there are argots (particularly relating to standards groups) that treat these otherwise well-defined words as having special meaning. So, rather than pounce on a possible misuse of a word that we (in ARC-land) don't actually use for any special purpose, I think it's better to get the project team to tell us in our language exactly what they're intending to do. We can then safely leave the etymology to Safire and the Morrises. ;-} -- James Carlson, Solaris Networking Sun Microsystems / 1 Network Drive 71.234W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.497N Fax +1 781 442 1677 From sacadmin Fri May 9 10:44:11 2003 Date: Fri, 09 May 2003 10:41:07 -0700 From: Jordan Brown X-Accept-Language: fr, en MIME-Version: 1.0 To: James Carlson CC: Adi Masputra , psarc@sac.eng.sun.com, peter.memishian@sun.com Subject: Re: 2003/263: DL_CAPABILITY-ignorant intermediate module detection Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Length: 209 > So, rather than pounce on a possible misuse of a word that we (in > ARC-land) don't actually use for any special purpose, See, I'd say that we *do* use it for that purpose, which is why I was defending it. From sacadmin Fri May 9 10:48:50 2003 X-Authentication-Warning: jurassic.eng.sun.com: masputra owned process doing -bs Date: Fri, 9 May 2003 10:46:58 -0700 (PDT) From: Adi Masputra To: James Carlson cc: Adi Masputra , "" , "" , "" Subject: Re: 2003/263: DL_CAPABILITY-ignorant intermediate module detection MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Content-Length: 551 James Carlson writes: > That is, hide the queue pointer comparison inside the function so that > (in the future) if the check is ever based on some attribute of the > queue rather than the pointer itself, this won't need to be changed. > > (And the former should probably use 'const queue_t *' ...) OK. > Does "deprecated" mean "just plain removed" or "still there but does > nothing" or "still works but we tell people not to use it?" In this case, I'm opting for the variable to be removed, rather having it still defined yet does nothing. Adi From sacadmin Fri May 9 11:36:59 2003 Date: Fri, 09 May 2003 11:34:57 -0700 From: Frank DiMambro Subject: Re: 2003/263: DL_CAPABILITY-ignorant intermediate module detection To: James Carlson Cc: Jordan Brown , Adi Masputra , psarc@sac.eng.sun.com, peter.memishian@sun.com MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-Accept-Language: en Content-Length: 1276 Hi Jim Rather than create and 'argot' for PSARC can we all just agree to use the standard English dictionary. Frank James Carlson wrote: > > Jordan Brown writes: > > (My point here is that we have a good word with a useful meaning. If we > > ever allow people to use it in the wrong way, we lose the word. Should > > anybody use it - or any other word - in the wrong way, they should be > > corrected rather than tolerated. This is especially true of words that > > are used in specifications.) > > As much as I agree with you on this point, I think we should still be > aware that there are argots (particularly relating to standards > groups) that treat these otherwise well-defined words as having > special meaning. > > So, rather than pounce on a possible misuse of a word that we (in > ARC-land) don't actually use for any special purpose, I think it's > better to get the project team to tell us in our language exactly what > they're intending to do. We can then safely leave the etymology to > Safire and the Morrises. ;-} > > -- > James Carlson, Solaris Networking > Sun Microsystems / 1 Network Drive 71.234W Vox +1 781 442 2084 > MS UBUR02-212 / Burlington MA 01803-2757 42.497N Fax +1 781 442 1677 From sacadmin Tue May 13 11:44:18 2003 X-Authentication-Warning: jurassic.eng.sun.com: masputra owned process doing -bs Date: Tue, 13 May 2003 11:42:23 -0700 (PDT) From: Adi Masputra To: psarc@sac.eng.sun.com cc: peter.memishian@Sun.COM, "" Subject: Re: 2003/263: DL_CAPABILITY-ignorant intermediate module detection MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Content-Length: 21020 Here is the most recent proposal. The changes cover the updated retrieve function, dlcapabcheckqid(), as well as indicating that the "ip_use_dl_cap" variable is to be "removed" instead of "deprecated" for S10, i.e. the subsequent minor release. Note also that as documented in PSARC 2002/276, the /dev/ip NDD variable "ip_multidata_outbound" will be restored to 1 (ON by default for S10) once this proposal is approved and implemented/integrated via 4771245. Current micro releases (S9 Updates) aren't affected by the "ip_use_dl_cap" and "ip_multidata_outbound" changes, as both are still OFF by default. ------ DL_CAPABILITY-ignorant intermediate module detection BACKGROUND ---------- In the recent years, many non-standard (non-DLPIv2) mechanisms of exchanging messages between ip module and the DLPI device driver were introduced to Solaris, e.g. M_DATA fastpath, hardware checksum offload [PSARC 1996/173], Multidata Transmit (MDT) [PSARC 2002/276], IPsec Hardware Acceleration (HA) [PSARC 2001/070], etc. These new interfaces were needed in order to get around the limitation of the DLPIv2 interface between a Data Link Provider and Consumer, mainly for performance improvements (some of which are attributed to driver/hardware offload capabilities). It is expected that many other future networking projects (zero-copy, etc.) would rely on some other similarly negotiated interfaces. IP's usage of these interfaces depends on the outcome of capability negotiations that happen between ip module and the driver. Such negotation scheme works well in cases where the driver is the module directly underneath ip. In other cases where there is one or more intermediate module between ip and the driver, this scheme does not provide a way of detecting their capabilities. Therefore, the introduction of modules in between often results in a breakage. In addition, the current scheme of capabilities negotiation typically happens during the plumbing, or construction of the IP interface stream. While this works for statically-plumbed intermediate modules, it is not sufficient to handle the case where a module is dynamically introduced into an existing IP interface stream. PROBLEMS -------- The exchange of capability negotiations are done in several ways. The M_DATA fastpath negotiation is triggered by a M_IOCTL message. The current hardware checksum offload uses a M_CTL message as vehicle, while the newer features such as MDT and IPsec HA use the newly- defined DL_CAPABILITY_{REQ,ACK} inside the M_PROTO message type. An intermediate module which does not understand any of those messages may either free them, or forward them to the next module. The former action can easily be dealt with, as IP can handle the "lost" negotiation message case (it won't enable or utilize those features). The latter action, however, is rather dangerous as the response may be interpreted to be coming from such intermediate module, which in reality was generated by a module underneath it. The following example depicts such case: +----------+ | ip | +----------+ | +----------+ | firewall | No knowledge of DL_CAPABILITY_REQ interface +----------+ | +----------+ | ce | Supports capability (MDT) +----------+ A firewall module which doesn't understand the Multidata interface (as well as the DL_CAPABILITY framework) is plumbed between ip and the ce driver, either statically or dynamically. Because it passes through any unknown messages, the DL_CAPABILITY_ACK coming from the driver gets forwarded (passed through) onto ip. Since ip has no way of determining the origin of the acknowledgment, it assumes that the module immediately below (firewall) was the source of the acknowledgement, and hence it will end up utilizing Multidata to pass down the packets. When the firewall module receives the Multidata messages, it may either forward them down to the driver or free them. Either action is considered a breakage: the former allows packets to go out of the machine without first being inspected by the firewall, and the latter will most likely break any existing network connections whose packets are encapsulated within those Multidata messages. Any STREAMS module may be interposed between ip and the driver by way of autopush(1M), if_tcp(7P), and ifconfig(1M) modinsert/modremove. In the static plumbing case, using either autopush(1M) or if_tcp(7P) method, the module is pushed on top an existing stream opened to the device driver, and the ip module is further pushed on top of it. At this point, the IP interface is considered to be plumbed or constructed, and the ip module may initiate the capability negotiation exchange process. The ability to dynamically change the plumbing of the IP stream was introduced in PSARC/1999/348 - _I_MUXID2FD, _I_INSERT, and _I_REMOVE ioctls. In PSARC/1999/375 - Re-plumbing Options for ifconfig(1M) case, 2 new options to ifconfig(1M) were introduced that can change the plumbing of the IP stream dynamically (modinsert and modremove). These options are classified as Evolving. Because a dynamic re-plumbing typically happens after the exchange of capabilities negotiation, a breakage will occur when the newly- inserted module is not capable of interpreting any previously- negotiated and enabled features. The problem space can therefore be summarized as follows: 1. The lack of module identification token in the DL_CAPABILITY framework which would allow for verification of incoming sub- capabilities within DL_CAPABILITY_ACK messages, in order to detect pass-thru intermediate modules. 2. There is currently no formal mechanism to reset and re-negotiate the capabilities due to a change in the stream's plumbing, such as by way of ifconfig(1M) modinsert and modremove. PROPOSAL -------- This fast-track proposal introduces an extension to the DL_CAPABILITY framework to allow for detecting "pass-thru" modules, as well as to turn on or off specific capabilities on a case-by-case basis by ip in order to perform the capability resetting and re-negotiating actions during a dynamic re-plumbing event. The requested release binding is micro. This proposal creates the following interfaces: Exported Interfaces: +-----------------------+----------------------------------+-----------------+ | Interface | Classification | Comments | |-----------------------+----------------------------------+-----------------| | DL_CAPAB_ID_WRAPPER | Consolidation Private | sys/dlpi.h | | dl_mid_t | Consolidation Private | sys/dlpi.h | | dl_capab_id_t | Consolidation Private | sys/dlpi.h | | dlcapabsetqid() | Consolidation Private | sys/dlpi.h | | dlcapabcheckqid() | Consolidation Private | sys/dlpi.h | |-----------------------+----------------------------------+-----------------| Imported Interfaces: +-----------------------+----------------------------------+-----------------+ | Interface | Classification | Comments | |-----------------------+----------------------------------+-----------------| | DL_CAPABILITY_REQ | Consolidation Private | PSARC/2001/070 | | DL_CAPABILITY_ACK | Consolidation Private | PSARC/2001/070 | | dl_capability_sub_t | Consolidation Private | PSARC/2001/070 | | dl_capab_ipsec_t | Consolidation Private | PSARC/2001/070 | | dl_capab_mdt_t | Consolidation Private | PSARC/2002/276 | +-----------------------+----------------------------------+-----------------+ From : #define DL_CAPAB_ID_WRAPPER 0x00 /* * Module ID to be included in new sub-capability structures. * Existing sub-capabilities lacking an identification token, * e.g. MDT and IPSEC hardware acceleration, need to be * encapsulated within the ID sub-capability. Access to this * structure must be done through dlcapab{set,get}qid(). */ typedef struct { t_uscalar_t mid[4]; /* private fields */ } dl_mid_t; /* * Module ID wrapper (follows dl_capability_sub_t) */ typedef struct { dl_mid_t id_mid; /* module ID */ dl_capability_sub_t id_subcap; /* sub-capability */ } dl_capab_id_t; extern void dlcapabsetqid(dl_mid_t *, const queue_t *); extern boolean_t dlcapabcheckqid(const dl_mid_t *, const queue_t *); A token (dl_mid_t) which allows for the module understanding the sub- capabilities to record its identification is needed. Currently, the identification token is defined to be the upper read queue pointer value of the module. By having a module or driver go through the DL_CAPABILITY_ACK message (on its way up) and "stamp" its upper read queue value on the sub-capabilities within, ip can detect a pass-thru module by way of comparing the upper read queue of the module immediately below with the value inside the sub-capability. At the time of writing, both MDT and IPsec HA features in the networking stack and device drivers have been delivered into Solaris, and possibly deployed on customer sites. Modifying dl_capab_mdt and dl_capab_ipsec_t in order to introduce an "identification" token will certainly break existing device drivers. Therefore, a "wrapper" sub- capability (dl_capab_id_t) used to encapsulate those sub-capabilities lacking the identification token is proposed. Existing MDT and IPsec HA sub-capabilities can be wrapped with the above "wrapper" sub-capability, and ip can be modified to understand such sub-capability in addition to the existing ones that it supports. New sub-capabilities defined in future are required to incorporate the dl_mid_t structure in order to provide for an identification token. Ways to set and obtain the queue pointer (the module identification) from within the dl_mid_t, via the dlcapab{set,get}qid() routines, are also introduced. Upon detecting dynamic re-plumbing events, ip would reset (disable) previously-negotiated capabilities -- details on how this is done is beyond the scope of this case, and is documented in 4771245. The reason for the "reset" is to establish that only vanilla DLPI is acceptable (i.e. modules should revert to sending only strict DLPIv2). This is done by having ip compose and send down a DL_CAPABILITY_REQ message which consists of the previously-negotiated capabilities indicating that they be disabled and no longer used. Such indication is done by way of clearing the "enable" bit defined per sub-capability. Afterwards, ip probes its lower-stream to find out about any new (or lack of) capabilities by sending down an empty DL_CAPABILITY_REQ message per PSARC 2001/070. An intermediate module which doesn't understand the DL_CAPABILITY framework or sub-capabilities within will be able to be detected, and no breakage will occur. In order for the above to work, the rule set of PSARC 2001/070 needs to be modified to include the following: a. All sub-capabilities must be able to be disabled. The current MDT and IPsec HA sub-capabilities contain flags fields whose enable bits can be cleared. Future sub-capabilities are required to implement similar mechanism. b. The enable/disable bit in previously-advertised (those which have been enabled by ip) sub-capabilities is read-only for modules and drivers, and can only be modified by ip. This would guarantee that an intermediate module doesn't interfere with the capability reset done by ip during dynamic re-plumbing. c. Future sub-capability definitions must include the dl_mid_t structure in order to provide for module identification. d. All intermediate modules (except drivers) are required to forward the DL_CAPABILITY_REQ message (received from the module above) downstream. e. All modules and drivers which understand the DL_CAPABILITY framework must record its upper queue pointer value, by calling dlcapabsetqid(), on each of the sub-capability within the DL_CAPABILITY_ACK which it recognizes and wishes for ip to enable. This must be done prior to passing the DL_CAPABILITY_ACK upstream. f. A module which initiates a DL_CAPABILITY_REQ is required to check each sub-capability returned in a DL_CAPABILITY_ACK, by calling dlcapabcheckqid(), to see if the returned queue pointer matches that of the upper read queue pointer of the module immediately below. If it matches, the sub-capability may be enabled, otherwise, it should be discarded. In addition to the above rules, module (not driver) writers are required to perform additional checks in fulfilling (e), in order to handle the case where the underlying module/driver is not capable of the sub-capabilities which may be proclaimed by the module in the returning DL_CAPABILITY_ACK, as explained in the following example: +----------+ | ip | Supports A,B,C, ... +----------+ | +----------+ | mod1 | Supports A,B,C +----------+ | +----------+ | mod2 | Supports A,C +----------+ | +----------+ | driver | Supports A,B,C +----------+ Modules mod1 and mod2 are plumbed in between ip and the driver. All modules are capable of supporting capabilities A and C, while B capability is only supported by ip, mod1 and driver. Prior to sending a DL_CAPABILITY_ACK to ip, mod1 might record its token in all of the sub-capabilities, since it's capable to handle them. However, this may not be correct as mod2 doesn't have the ability to handle capability B, and a breakage may occur when mod1 uses capability B to communicate to mod2. Therefore, all modules claiming to support the DL_CAPABILITY framework are required to check to see if each of the sub-capability it supports can be handled by the the module/driver below, by way of comparing the value returned by dlcapabcheckqid() in the returning DL_CAPABILITY_ACK against the value of the read queue pointer of the underlying module/ driver. If the module below doesn't support the sub-capability, i.e. the values don't match, then the following actions may be taken: 1. If it is somehow possible to convert messages formatted according to the corresponding sub-capability into those which can be understood by the module below, then it may report to the module above that the sub-capability is supported, by recording its token in the sub-capability prior to sending up the DL_CAPABILITY_ACK. However, it must perform the necessary works to convert those messages prior to sending it downstream. 2. If a conversion is not possible or desirable, then the module may not record its token in the corresponding sub-capability, which effectively renders the sub-capability unproclaimed by the module upon returning the DL_CAPABILITY_ACK. The module above can then make the necessary decisions in the similar fashion based on these outlines. Drivers are not required to perform the above checks, since they are at the bottom of the stream. DETAILS ------- Prior to passing up a DL_CAPABILITY_ACK (i.e. in the rput routine), a module must go through each sub-capability looking for those which it supports, and stamp its upper read queue pointer on them. In the case of a driver, this process is done as part of initializing the sub-capability that it is about to report to ip. The following example of pseudo structures depict the current MDT sub-capability as reported by drivers: dl_capability_ack { dl_primitive = DL_CAPABILITY_ACK; dl_sub_offset = sizeof (dl_capability_ack_t); dl_sub_length = sizeof (dl_capability_sub_t) + sizeof (dl_capab_mdt_t); dl_capability_sub_t { dl_cap = DL_CAPAB_MDT; dl_length = sizeof (dl_capab_mdt_t); dl_capab_mdt_t { ... } } } When using the wrapper sub-capability, the above example can be represented as: dl_capability_ack { dl_primitive = DL_CAPABILITY_ACK; dl_sub_offset = sizeof (dl_capability_ack_t); dl_sub_length = sizeof (dl_capability_sub_t) + sizeof (dl_capability_id_t) + sizeof (dl_capab_mdt_t); dl_capability_sub_t { dl_cap = DL_CAPAB_ID_WRAPPER; dl_length = sizeof (dl_capab_id_t) + sizeof (dl_capab_mdt_t); dl_capab_id_t { id_mid = SET: dlcapabsetqid(&id_mid, RD(q)); GET: dlcapabcheckqid(&id_mid); id_subcap.dl_cap = DL_CAPAB_MDT; id_subcap.dl_length = sizeof (dl_capab_mdt_t); dl_capab_mdt_t { ... } } } } A pass-thru module which doesn't understand the DL_CAPABILITY framework or any of the contained sub-capabilities will not have its upper read queue pointer stamped, and therefore ip will be able to detect this by case and discard the received DL_CAPABILITY_ACK. In order to be backwards compatible with current drivers which don't incorporate this identification sub- capability scheme, ip is modified to handle both cases: 1. An unwrapped sub-capability (current MDT and IPsec HA) will only get accepted if and only if there is at most one STREAMS module beneath ip. In other words, current MDT and IPsec HA sub- capabilities advertised by currently-shipping Cassini and Venus drivers will only be enabled if there is no intermediate module in between ip and the driver. 2. Wrapped MDT and IPsec HA, as well as other future sub-capabilities incorporating dl_mid_t structure, will only get accepted by ip if and only if the upper read queue pointer of the module beneath ip matches that of the queue pointer returned by the dlcapabcheckqid() on the corresponding sub-capability. Otherwise, those sub-capabilities are discarded as their sources can't be determined. It is also noted that both M_DATA fastpath and the current Solaris hardware checksum interface don't utilize the DL_CAPABILITY negotiation framework. It is expected that the negotiation of these features be ported over to the DL_CAPABILITY framework, and their progresses be documented in 4703353, 4796051, and PSARC 2003/264. BACKWARDS COMPATIBILITY ----------------------- The entire purpose of this change is enhanced backwards compatibility. But there are conflicting requirements. We also need be compatible early drivers supporting the DL_CAPABILITY interface (cassini/MDT, venus). Fortunately, we have an existing configuration "knob" (see PSARC 2002/322) we can use to detect that a customer has explicitly requested the use of the DL_CAPABILITY feature. If this project is integrated into a patch/update release, existing behavior will be preserved if "ip_use_dl_cap" is set to 1 in /etc/system -- i.e., intermediate module detection will be disabled -- allowing the use of drivers which don't play along with intermediate module detection. If ip_use_dl_cap is left alone, DL_CAPABILITY negotiation will happen anyway but with the intermediate module detection part enabled, rendering it safe in the presence of intermediate modules. In a subsequent minor release, this mechanism will always be enabled and the use of "ip_use_dl_cap" will be removed. REFERENCES ---------- 4703353 ip's fast-path mechanism breaks DLPIv2 4796051 Solaris needs a more complete HW checksumming support 4771245 DL_CAPABILITY-ignorant intermediate module detection PSARC 1996/173 TCP/IP support for SAHI3 hardware checksuming PSARC 1999/348 I_MUXID2FD, I_INSERT, and I_REMOVE PSARC 1999/375 Re-plumbing Options for ifconfig(1M) PSARC 2001/070 DL_{CAPABILITY,CONTROL}_REQ PSARC 2002/276 TCP Multi-Data Transmit PSARC 2002/322 DL_CAPABILITY knob PSARC 2003/264 Extended IP checksum offload interface From sacadmin Wed May 14 08:48:56 2003 From: Bill Sommerfeld To: psarc@sac.eng.sun.com cc: adi.masputra@sun.com, mimi.wong@sun.com, Jay.Jayachandran@sun.com, Mehdi.Bonyadi@sun.com Subject: 2003/263: DL_CAPABILITY-ignorant module detection: Contracts filed. Date: Wed, 14 May 2003 11:46:57 -0400 Content-Length: 2822 Two contracts covering the use of 2003/263 and 2003/264 by network drivers have been negotiated and signed by appropriate managers within SNT, NSPG, and CPG. I've filed them in the case directory as contract-01 and contract-02. Mail aliases for the contracts have been created. (I'll send a separate note to 2003/264 shortly) contract-01: use of 2003/263 and 2003/264 by NSPG network drivers Mimi Wong writes: > Adi, > > This email signifies my signature on these contracts. Let me know if > you need a hardcopy signature. > > Mimi > > -------- Original Message -------- > Date: Mon, 12 May 2003 19:29:42 -0700 > From: Jay Jayachandran > > Adi, > > I've added a couple of drivers in Section 3 with their Bugtraq > Category/SunCategory. > Modified (and signed) contract-01.txt is attached. > > Mehdi, > > The contract looks fine to me. You may sign contract-02.txt after your > engineers' > review. > > Thanks, --jay (x14877) > > Adi Masputra wrote: > > > > Since there are two consumers of the interfaces (different managers), > > I've separated the previous contract into 2, and they can be found > > as attached. > > > > Jay: Please sign the contract named "contract-01.txt" > > Mehdi: Please sign the contract named "contract-02.txt" > > > > I need these before the end of Tuesday, so that the ARC cases > > can be approved on Wednesday. > > > > Thanks, > > Adi contract-02: use of 2003/263 and 2003/264 by CPG network drivers Mimi Wong writes: > All, > > This email signifies my signature on these contracts. Let me know if > you need a hardcopy signature. > > Mimi > > Adi Masputra wrote: > > Mimi, > > > > Please sign the contract and return the mail to me, so that > > I can forward it to Bill. > > > > Thanks, > > Adi > > > > ---------- Forwarded message ---------- > > Date: Tue, 13 May 2003 09:04:44 -0700 (PDT) > > From: Mehdi Bonyadi > > > > Hello Adi, > > We changed the group name as follows: > >>From "NSPG/Networking and security products" to "Cryptographic Products Group" > > > > I added my name and date at the bottom signifying my signature. Do you need hard > > copy with my signature on it? > > > > Thanks, > > Mehdi > > > > > >>Date: Mon, 12 May 2003 15:46:19 -0700 (PDT) > >>From: Adi Masputra > > > >>Subject: Revised contracts for PSARC/2003/263 and PSARC/2003/264 > >> > >>Since there are two consumers of the interfaces (different managers), > >>I've separated the previous contract into 2, and they can be found > >>as attached. > >> > >>Jay: Please sign the contract named "contract-01.txt" > >>Mehdi: Please sign the contract named "contract-02.txt" > >> > >>I need these before the end of Tuesday, so that the ARC cases > >>can be approved on Wednesday. > >> > >>Thanks, > >>Adi > >> From sacadmin Wed May 14 10:19:15 2003 From: Bill Sommerfeld To: psarc@sac.eng.sun.com cc: adi.masputra@sun.com Subject: 2003/263: DL_CAPABILITY-ignorant intermediate module detection Date: Wed, 14 May 2003 13:17:19 -0400 Content-Length: 90 This case (with attached contracts) was approved at today's PSARC meeting. - Bill