From Sebastien.Roy@Sun.COM Tue Jun 2 06:44:39 2009 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n52DidTK022598 for ; Tue, 2 Jun 2009 06:44:39 -0700 (PDT) Received: from brmea-mail-1.sun.com (brmea-mail-1.Sun.COM [192.18.98.31]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n52Didh3006031 for ; Tue, 2 Jun 2009 06:44:39 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n52DidLc017042 for ; Tue, 2 Jun 2009 13:44:39 GMT MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) id <0KKM00J005Z0VL00@mail-amer.sun.com> for psarc-ext@sac.sfbay.sun.com; Tue, 02 Jun 2009 07:44:39 -0600 (MDT) Received: from [129.148.174.103] ([unknown] [129.148.174.103]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) with ESMTPSA id <0KKM003RF66DOXB0@mail-amer.sun.com>; Tue, 02 Jun 2009 07:44:37 -0600 (MDT) Date: Tue, 02 Jun 2009 09:43:39 -0400 From: Sebastien Roy Subject: IP Datapath Refactoring [PSARC/2009/331 FastTrack timeout 06/09/2009] Sender: Sebastien.Roy@Sun.COM To: psarc-ext Cc: "Erik.Nordmark" , Sowmini Varadhan Message-id: <1243950219.6636.12.camel@strat> Organization: Sun Microsystems X-Mailer: Evolution 2.24.2 Status: RO Content-Length: 9667 I'm sponsoring this fast-track for Erik Nordmark, the timer expires on June 9th. IP Datapath Refactoring ======================= The project changes a lot of the IP datapath code paths, but has rather minor changes to documented interfaces. Those changes are the subject of this case. The IP datapaths are extremely hard to follow both at the micro level (ip_output_options and ip_wput_ire, and ip_input) and at the macro level (an outbound packet needing IPsec and ARP resolution goes through a large number of steps). The nature of the changes in IP derive from the classical quote "ip_newroute delenda est"; a key root cause of the complexity is from the introduction of asynchronous behavior near the top of the IP output path. Moving the asynchrony to the bottom of the IP output path, where other implementations handle ARP resolution, makes things more sane. Also, partly due to the ip_newroute asynchrony, IP has grown a large number of internal mechanisms to remember state associated with a packet that needs to be queued. We are replacing that with a single internal mechanism in the form of ip_xmit_attr_t and ip_recv_attr_t data structures. All those mechanisms and interfaces are project private. Current prototyping indicates that about 30,000 lines of code can be removed as a result of these changes (combined with the ARP/IP merge pieces). Imported interfaces: -------------------- No changes. Exported interfaces: -------------------- We are introducing a new RTF_INDIRECT in . This flag is useful for routing daemons that do BGP plus OSPF/IS-IS since it can make handling routing changes a lot more efficient. Route(1m) has a new -indirect flag to set RTF_INDIRECT. The indirect routes are represented by a new 'I' flag in the netstat -r output. Old cases state that the classification of the output of netstat as Unstable. This case doesn't change that; it just adds the 'I' flag. We are adding an informational RTF_KERNEL flag for routes, for instance interface routes, that are added by the kernel as part of configuring an IP interface. Such routes can not be accidentally deleted by applications. The implementation introduces a new Destination Cache Entry in the kernel (patterned after the description in RFC 4861). For debugging reasons it is useful to be able to display the DCE in particularly the Path MTU which is recorded in it. We are adding the -d option to netstat(1m) for this purpose. (Note that netstat -d is currently an undocumented option for debugging netstat itself. We rename that undocumented debug option to -x.) The way netstat extracts the DCE table from the kernel is using the new EXPER_IP_DCE in . The implementation of multirt/CGTP changes, and in all but one detail the interfaces remain unchanged. The change is that evolving and undocumented tunable ip_multirt_resolution_interval has been removed. Multirt/CGTP routes use the same timer as ARP/ND does for other routes. Note that PSARC/2003/041 replaced PSARC/2000/539 and the never contract didn't mention the ndd tunable; this might have been an omission. _________________________________________________________________________ | Interfaces Added by This Case | |_______________________________|_______________________|_______________| |Interface | Classification | Comments | |_______________________________|_______________________|_______________| | RTF_INDIRECT | Committed | | | route(1m) -indirect flag | Committed | | | netstat(1m) -r output | Uncommitted (unchanged)| | | RTF_KERNEL | Committed | | | netstat(1m) -d option | Committed | | | netstat(1m) -d output | Uncommitted | | | EXPER_IP_DCE | Uncommitted | | |_______________________________|_______________________|_______________| _________________________________________________________________________ | Interfaces Removed by This Case | |_______________________________|_______________________|_______________| |Interface | Classification | Comments | |_______________________________|_______________________|_______________| | ip_multirt_resolution_interval| Evolving | PSARC/2000/539| |_______________________________|_______________________|_______________| Implementation changes: ----------------------- Due to the ARP/IP merge and uniform application of Neighbor Unreachability Detection (RFC 4862) the undocumented ndd tunables for /dev/arp are replaced by undocumented ndd tunables for /dev/ip. Instead of relying on timers (with undocumented but well-known tunables like arp_cleanup_interval and ip_ire_arp_interval) this project makes ARP function the same way as Neighbor Discovery in using the RFC 4862 NUD state machine. Thus those known, but undocumented, tunables are removed. The implementation changes to track IPv4 group membership per ill_t instead of per ipif_t, but we keep the IP address around so we can preserve the output of netstat -g (which reports the logical interface name e.g., bge0:1 when an IPv4 group is joined using the IP address assigned to bge0:1) netstat -ia continues to show input counters for each local address. However, the output counters never made any sense on a per-local address (IP packets are sent out of a IP interface and not out an IP address), and this project makes them be reported as zero. The Solaris 'Use' count in netstat -r has been this unpredictable/undocumented number since Solaris 2.0 (the implementation counts the number of times ip_newroute has used the route to try to create an IRE_CACHE entry). We restore the use count to actually count the number of packets that are send out using the route in question. No IRE_CACHE entries (UHA) entries will appear in netstat -ra, since the implementation no longer has IRE_CACHE entries. This project adds a new IRE_IF_CLONE type of routes. Those routes appear in netstat -ra (but not without the 'a' option) with the new 'C' flag. While the kernel no longer uses any IRE_CACHE entries, we are keeping the #define of IRE_CACHE in the header file so that applications which use the common, but undocumented, mibget approach for retrieving the kernel routing table will still compile. The new implementation no longer has a ire_max_frag field, hence the output of Maxfrg/PMTU in the netstat -rv output is no longer useful. We are removing that output. (Note that the details of the netstat output is not a stable interface.) Currently Solaris handles IP interface MTU in odd ways in that it can be set differently for local IP address prefix; this leaves it quite undefined in what MTU is applied to multicast packets. This project fixes that by applying the IP interface MTU per interface. As a result ifconfig bge0:N mtu 1400 will fail with EINVAL. No mapping entry will appear in arp -a/netstat -p output, since the implementation of the multicast mapping has changed. Individual multicast and broadcast addresses might appear in netstat -p/arp -a. API calls which refer to interface indicies and interface addresses (IP_MULTICAST_IF, IP_BOUND_IF, IP_ADD_MEMBERSHIP, etc) currently have odd behaviors when interfaces and/or IP addresses are unplumbed and removed. To preserve kernel sanity (no stray ill and ipif pointers) the applications setting is forgotten with telling the application. From the application the behavior looks very odd. E.g., if an IP_ADD_MEMBERSHIP is followed by a correct IP_DROP_MEMBERSHIP will see a EADDRNOTAVAIL error because the kernel might have removed all memory of the IP_ADD_MEMBERSHIP when the IP address was removed. This project will instead preserve what the application has set until the application explicitly removes it. For instance, an IP_BOUND_IF will remain in effect even if the IP interface is unplumbed. Packets will be dropped and ENXIO will be returned to the sendto() system call. And received packets will be dropped since they can't possibly match the interface index specified in the IP_BOUND_IF when the interface has been unplumbed. However, when the IP address (or interface index) which was use by the application reappears, then the application's setting will be fully functional again. The project extends the kernel's ability to handle multiple routes for the same prefix; currently the kernel only does some form of round robin for default routes and the project extends that to all off-link routes (default, prefix, and host routes). We are adding an undocumented knob should there be a reason to switch back to the old behavior in the field. The project removes the usage of multidata from TCP/IP, but the interfaces specified in PSARC/2004/594 and PSARC/2002/276 remain in the system. The project changes the *use* of DL_NOTE_REPLUMB (introduced by PSARC 2008/242 as a private interface); it is only used on the IP stream(s) and not on the ARP stream. Notes: ------ This project removes the AR_* message set used by IP and ARP. Earlier there was a contract private interface with SunATM on those interfaces (established in LSARC/1993/101/ and extended in PSARC/1999/446 and PSARC/2001/023). That contract was cancelled by PSARC/2006/272 EOL of ATM device driver The integration of this project is likely to also deliver the changes related to PSARC/2008/522 EOF of 2001/070 IPsec HW Acceleration support From carlsonj@phorcys.east.sun.com Tue Jun 2 07:13:23 2009 Received: from dm-east-02.east.sun.com (dm-east-02.East.Sun.COM [129.148.13.5]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n52EDNfA023180 for ; Tue, 2 Jun 2009 07:13:23 -0700 (PDT) Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by dm-east-02.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n52EDL0p061085; Tue, 2 Jun 2009 10:13:21 -0400 (EDT) Received: from phorcys.east.sun.com (phorcys.local [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n52ECEGg007552; Tue, 2 Jun 2009 10:12:14 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n52ECEgq007549; Tue, 2 Jun 2009 10:12:14 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18981.13118.399313.321138@gargle.gargle.HOWL> Date: Tue, 2 Jun 2009 10:12:14 -0400 From: James Carlson To: Sebastien Roy Cc: psarc-ext , "Erik.Nordmark" Subject: Re: IP Datapath Refactoring [PSARC/2009/331 FastTrack timeout 06/09/2009] In-Reply-To: <1243950219.6636.12.camel@strat> References: <1243950219.6636.12.camel@strat> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 4432 Sebastien Roy writes: > I'm sponsoring this fast-track for Erik Nordmark, the timer expires on > June 9th. Looks good, but some nits: > We are introducing a new RTF_INDIRECT in . This flag is useful > for routing daemons that do BGP plus OSPF/IS-IS since it can make handling > routing changes a lot more efficient. Great to hear, but can we have some detail on how to use this? Is it the network routes from BGP that must use this flag (to make the specified next hop address be "indirect"), or is it a flag set by the BGP (or even the IGP) to specify that a given entry is a special "host route" intended as a target for BGP routes? What exactly does the flag do? How is a route with this flag different? > We are adding an informational RTF_KERNEL flag for routes, for instance > interface routes, that are added by the kernel as part of configuring an IP > interface. Such routes can not be accidentally deleted by applications. RTF_KERNEL doesn't appear for ICMP redirect, even though that's processed by the kernel, right? > | EXPER_IP_DCE | Uncommitted | | There's probably a new MIB data structure that goes with this symbol. > No IRE_CACHE entries (UHA) entries will appear in netstat -ra, since the > implementation no longer has IRE_CACHE entries. > This project adds a new IRE_IF_CLONE type of routes. Those routes appear in > netstat -ra (but not without the 'a' option) with the new 'C' flag. Are these like the BSD clone entries (i.e., effectively ARP cache entries, with *only* on-link addresses represented), or are they something else? > The new implementation no longer has a ire_max_frag field, hence the output of > Maxfrg/PMTU in the netstat -rv output is no longer useful. We are removing that > output. (Note that the details of the netstat output is not a stable > interface.) > > Currently Solaris handles IP interface MTU in odd ways in that it can be > set differently for local IP address prefix; this leaves it quite undefined > in what MTU is applied to multicast packets. > This project fixes that by applying the IP interface MTU per interface. As a > result ifconfig bge0:N mtu 1400 will fail with EINVAL. There's certainly been some customer confusion around routes and addresses with specified MTU. It's sometimes the case that users must deal with remote networks that have restricted MTUs *but* that don't support PMTU properly. What should those users be doing? (Reporting bugs against our PMTU implementation? Asking for an RFE? Yelling at the admin for that remote network?) > ENXIO will be returned to the sendto() system call. And received packets will > be dropped since they can't possibly match the interface index specified in the > IP_BOUND_IF when the interface has been unplumbed. However, when the IP address > (or interface index) which was use by the application reappears, then the > application's setting will be fully functional again. Interface indices can't normally disappear and then reappear, can they? (If they can, then I'd call that a "bug" rather than a "feature." SNMP requires that ifIndex is unique for a given engine invocation.) > The project extends the kernel's ability to handle multiple routes for the same > prefix; currently the kernel only does some form of round robin for default > routes and the project extends that to all off-link routes (default, prefix, and > host routes). We are adding an undocumented knob should there be a reason to > switch back to the old behavior in the field. Yay! What happens on forwarding? Round-robin is a bad answer for forwarding, as it ends up reordering packets. ECMP is a better answer for that case. (I agree that round-robin is best _if_ you can preserve state ... local connections can do that, but forwarding cases generally cannot.) > The project removes the usage of multidata from TCP/IP, but the interfaces > specified in PSARC/2004/594 and PSARC/2002/276 remain in the system. This part is confusing; could you please elaborate? Does the stack still generate MDT messages? If not, then how are those two previous projects not affected? (Doesn't this project just obsolete those two ... ?) -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From glenn.skinner@sun.com Tue Jun 2 10:05:30 2009 Received: from ivrel.sfbay.sun.com (ivrel.SFBay.Sun.COM [129.146.74.76]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n52H5Ue5025602 for ; Tue, 2 Jun 2009 10:05:30 -0700 (PDT) Received: from ivrel (ivrel [129.146.74.76]) by ivrel.sfbay.sun.com (8.13.8+Sun/8.13.8) with SMTP id n52H5UZl003347; Tue, 2 Jun 2009 10:05:30 -0700 (PDT) Message-Id: <200906021705.n52H5UZl003347@ivrel.sfbay.sun.com> Date: Tue, 2 Jun 2009 10:05:30 -0700 (PDT) From: Glenn Skinner Reply-To: Glenn Skinner Subject: Re: 2009/331 [IP Datapath Refactoring] To: psarc-ext@sac.sfbay.sun.com Cc: Erik.Nordmark@sun.com, Sowmini.Varadhan@sun.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: rK02LDOQ1KjyE05I73b6HQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.6_36 SunOS 5.11 sun4u sparc Status: RO Content-Length: 2008 Date: Tue, 02 Jun 2009 09:43:39 -0400 From: Sebastien Roy Subject: IP Datapath Refactoring [PSARC/2009/331 FastTrack timeout 06/09/2009] I'm sponsoring this fast-track for Erik Nordmark, the timer expires on June 9th. IP Datapath Refactoring ======================= ... _________________________________________________________________________ | Interfaces Added by This Case | |_______________________________|_______________________|_______________| |Interface | Classification | Comments | |_______________________________|_______________________|_______________| | RTF_INDIRECT | Committed | | | route(1m) -indirect flag | Committed | | | netstat(1m) -r output | Uncommitted (unchanged)| | | RTF_KERNEL | Committed | | | netstat(1m) -d option | Committed | | | netstat(1m) -d output | Uncommitted | | | EXPER_IP_DCE | Uncommitted | | |_______________________________|_______________________|_______________| _________________________________________________________________________ | Interfaces Removed by This Case | |_______________________________|_______________________|_______________| |Interface | Classification | Comments | |_______________________________|_______________________|_______________| | ip_multirt_resolution_interval| Evolving | PSARC/2000/539| |_______________________________|_______________________|_______________| Perhaps it was very well hidden and I simply missed it. But what release binding do you desire? --- Glenn From erik.nordmark@sun.com Wed Jun 3 10:23:24 2009 Received: from jurassic-x4600.sfbay.sun.com (cretaceous.SFBay.Sun.COM [129.146.17.59]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n53HNONQ014885 for ; Wed, 3 Jun 2009 10:23:24 -0700 (PDT) Received: from [10.7.251.248] (punchin-nordmark.SFBay.Sun.COM [10.7.251.248]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n53HNNH0364047 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 3 Jun 2009 10:23:23 -0700 (PDT) Message-ID: <4A26B18B.1030206@sun.com> Date: Wed, 03 Jun 2009 10:23:23 -0700 From: Erik Nordmark User-Agent: Thunderbird 2.0.0.21 (X11/20090323) MIME-Version: 1.0 To: Glenn Skinner CC: psarc-ext@sac.sfbay.sun.com, Sowmini.Varadhan@sun.com Subject: Re: 2009/331 [IP Datapath Refactoring] References: <200906021705.n52H5UZl003347@ivrel.sfbay.sun.com> In-Reply-To: <200906021705.n52H5UZl003347@ivrel.sfbay.sun.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 184 Glenn Skinner wrote: > Perhaps it was very well hidden and I simply missed it. But what > release binding do you desire? Sorry about that. Minor release binding is fine. Erik From erik.nordmark@sun.com Wed Jun 3 10:59:06 2009 Received: from jurassic-x4600.sfbay.sun.com (cretaceous.SFBay.Sun.COM [129.146.17.63]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n53Hx6nW016110 for ; Wed, 3 Jun 2009 10:59:06 -0700 (PDT) Received: from [10.7.251.248] (punchin-nordmark.SFBay.Sun.COM [10.7.251.248]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n53Hx5nZ370921 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 3 Jun 2009 10:59:05 -0700 (PDT) Message-ID: <4A26B9E9.70004@sun.com> Date: Wed, 03 Jun 2009 10:59:05 -0700 From: Erik Nordmark User-Agent: Thunderbird 2.0.0.21 (X11/20090323) MIME-Version: 1.0 To: James Carlson CC: Sebastien Roy , psarc-ext Subject: Re: IP Datapath Refactoring [PSARC/2009/331 FastTrack timeout 06/09/2009] References: <1243950219.6636.12.camel@strat> <18981.13118.399313.321138@gargle.gargle.HOWL> In-Reply-To: <18981.13118.399313.321138@gargle.gargle.HOWL> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 8302 James Carlson wrote: >> We are introducing a new RTF_INDIRECT in . This flag is useful >> for routing daemons that do BGP plus OSPF/IS-IS since it can make handling >> routing changes a lot more efficient. > > Great to hear, but can we have some detail on how to use this? Is it > the network routes from BGP that must use this flag (to make the > specified next hop address be "indirect"), or is it a flag set by the > BGP (or even the IGP) to specify that a given entry is a special "host > route" intended as a target for BGP routes? The use of RTF_INDIRECT is optional. No changes are required to routing deamons are required. However, routing daemons can be modified to take advantage of the indirect routes. > What exactly does the flag do? How is a route with this flag > different? Two things are different for externally visible behavior: - when the route is added the kernel does not require that the gateway is directly reachable. If RTF_INDIRECT is not set the gateway must be directly reachable as is the case in Solaris today. - when we lookup a route and first find an indirect route, we iterate to lookup the gateway in that route. (Today if we first find an offlink/offsubnet route i.e., one with RTF_GATEWAY set, we iterate by looking up its gateway. The indirect route in essence adds one more step in this iterative process.) From an implementation perspective there is logic to ensure that the caching we do for performance proves the above visible behavior as routes are added and deleted; just ensuring that the caching stays consistent with the collection of forwarding table entries. Do you think we should document this somewhere? If so, where? route(1m)? >> We are adding an informational RTF_KERNEL flag for routes, for instance >> interface routes, that are added by the kernel as part of configuring an IP >> interface. Such routes can not be accidentally deleted by applications. > > RTF_KERNEL doesn't appear for ICMP redirect, even though that's > processed by the kernel, right? Correct. The "as part of configuring an IP interface" is key. I don't know if a different name for the flag would make this more clear. > >> | EXPER_IP_DCE | Uncommitted | | > > There's probably a new MIB data structure that goes with this symbol. Yes, that structure is named dest_cache_entry_t and is also uncommitted. >> No IRE_CACHE entries (UHA) entries will appear in netstat -ra, since the >> implementation no longer has IRE_CACHE entries. >> This project adds a new IRE_IF_CLONE type of routes. Those routes appear in >> netstat -ra (but not without the 'a' option) with the new 'C' flag. > > Are these like the BSD clone entries (i.e., effectively ARP cache > entries, with *only* on-link addresses represented), or are they > something else? I do not know the details of the BSD code. But the IRE_IF_CLONE entries do not contain any ARP information. They are merely a place to hang a reference to the ARP information. For example, if we have an interface route for a /23 129.146.228.0 129.146.228.81 U 1 5 bge0 and we are sending packets to 129.146.228.1 the kernel would create an IRE_IF_CLONE for 129.146.228.1/32, and as the ARP entries gets created there would be a direct pointer from that entry to the arp information. This is beneficial for performance since we avoid doing a lookup of the ARP information; we can always place a direct pointer to the ARP information in the IRE that matches. >> The new implementation no longer has a ire_max_frag field, hence the output of >> Maxfrg/PMTU in the netstat -rv output is no longer useful. We are removing that >> output. (Note that the details of the netstat output is not a stable >> interface.) >> >> Currently Solaris handles IP interface MTU in odd ways in that it can be >> set differently for local IP address prefix; this leaves it quite undefined >> in what MTU is applied to multicast packets. >> This project fixes that by applying the IP interface MTU per interface. As a >> result ifconfig bge0:N mtu 1400 will fail with EINVAL. > > There's certainly been some customer confusion around routes and > addresses with specified MTU. It's sometimes the case that users must > deal with remote networks that have restricted MTUs *but* that don't > support PMTU properly. > > What should those users be doing? (Reporting bugs against our PMTU > implementation? Asking for an RFE? Yelling at the admin for that > remote network?) I think it is due that that we implement RFC 4821 for TCP and SCTP which would make PMTU discovery more robust overall. If there are remote networks with different PMTU then presumably the customers are either 1) clamping down the MTU for their interfaces i.e., reducing MTU for everybody, or 2) using route(1m) to set a specific mtu for a particular route. Both of those are unaffected by this project. >> ENXIO will be returned to the sendto() system call. And received packets will >> be dropped since they can't possibly match the interface index specified in the >> IP_BOUND_IF when the interface has been unplumbed. However, when the IP address >> (or interface index) which was use by the application reappears, then the >> application's setting will be fully functional again. > > Interface indices can't normally disappear and then reappear, can > they? (If they can, then I'd call that a "bug" rather than a > "feature." SNMP requires that ifIndex is unique for a given engine > invocation.) This is outside of the scope of this case; this case merely makes sure the system is at least as robust as today should the existing SIOCSLIFINDEX be used to change the ifindex. Until relatively recently I thought we could actually remove the ability to set the interface index (essentially removing SIOCSLIFINDEX). But I've seen a legitimate case where a customer was using it (can't recall the details). >> The project extends the kernel's ability to handle multiple routes for the same >> prefix; currently the kernel only does some form of round robin for default >> routes and the project extends that to all off-link routes (default, prefix, and >> host routes). We are adding an undocumented knob should there be a reason to >> switch back to the old behavior in the field. > > Yay! > > What happens on forwarding? Round-robin is a bad answer for > forwarding, as it ends up reordering packets. ECMP is a better answer > for that case. (I agree that round-robin is best _if_ you can > preserve state ... local connections can do that, but forwarding cases > generally cannot.) For forwarding things are very simplistic in the current project gate. A given route will select a single outgoing nexthop/interface from the ECMP routes used. Thus the only way you can get ECMP behavior in the forwarding path is for separate routes which use the same gateway, but there are ECMP routes found when looking up that gateway. A contrived example of this is with this set of routes: A/n 10.0.0.1 UG B/m 10.0.0.1 UG 10.0.0.0/24 U bge0 10.0.0.0/24 U bge1 In the example A could might pick the first U route, and B might pick the second. (Similar examples can be constructed with indirect routes.) Thus we do not do anything special should there be multiple routes for A/n. It wouldn't be hard to add on top of what we are building, e.g., using a hash of src+dest ip to select a route from the bucket in which we find the matching A/n. >> The project removes the usage of multidata from TCP/IP, but the interfaces >> specified in PSARC/2004/594 and PSARC/2002/276 remain in the system. > > This part is confusing; could you please elaborate? > > Does the stack still generate MDT messages? If not, then how are > those two previous projects not affected? (Doesn't this project just > obsolete those two ... ?) The TCP/IP stack no longer sends down MDT messages. But all the support in PSARC/2004/594 and PSARC/2002/276 remains (the definition of M_MULTIDATA, the support in copymsg, the mmd_* functions. Thus device drivers that support M_MULTIDATA and link against those functions will continue to link/load into the kernel. Other code which sends M_MULTIDATA messages will continue to work. Erik From carlsonj@phorcys.east.sun.com Wed Jun 3 11:30:03 2009 Received: from dm-east-01.east.sun.com (dm-east-01.East.Sun.COM [129.148.9.192]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n53IU26P016722 for ; Wed, 3 Jun 2009 11:30:03 -0700 (PDT) Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by dm-east-01.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n53ITwAL052269; Wed, 3 Jun 2009 14:29:58 -0400 (EDT) Received: from phorcys.east.sun.com (phorcys.local [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n53ISoSc013790; Wed, 3 Jun 2009 14:28:50 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n53ISo0d013787; Wed, 3 Jun 2009 14:28:50 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18982.49378.410578.978181@gargle.gargle.HOWL> Date: Wed, 3 Jun 2009 14:28:50 -0400 From: James Carlson To: Erik Nordmark Cc: Sebastien Roy , psarc-ext Subject: Re: IP Datapath Refactoring [PSARC/2009/331 FastTrack timeout 06/09/2009] In-Reply-To: <4A26B9E9.70004@sun.com> References: <1243950219.6636.12.camel@strat> <18981.13118.399313.321138@gargle.gargle.HOWL> <4A26B9E9.70004@sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 8248 Erik Nordmark writes: > James Carlson wrote: > > >> We are introducing a new RTF_INDIRECT in . This flag is useful > >> for routing daemons that do BGP plus OSPF/IS-IS since it can make handling > >> routing changes a lot more efficient. > > > > Great to hear, but can we have some detail on how to use this? Is it > > the network routes from BGP that must use this flag (to make the > > specified next hop address be "indirect"), or is it a flag set by the > > BGP (or even the IGP) to specify that a given entry is a special "host > > route" intended as a target for BGP routes? > > The use of RTF_INDIRECT is optional. No changes are required to routing > deamons are required. However, routing daemons can be modified to take > advantage of the indirect routes. I realize that it's optional. I'll looking for "how to" information so that someone can take advantage of the option. The materials provided don't make the usage clear. > > What exactly does the flag do? How is a route with this flag > > different? > > Two things are different for externally visible behavior: > - when the route is added the kernel does not require that the gateway > is directly reachable. If RTF_INDIRECT is not set the gateway must be > directly reachable as is the case in Solaris today. > - when we lookup a route and first find an indirect route, we iterate > to lookup the gateway in that route. (Today if we first find an > offlink/offsubnet route i.e., one with RTF_GATEWAY set, we iterate by > looking up its gateway. The indirect route in essence adds one more step > in this iterative process.) Got it. So this is a flag that I-BGP should use when adding routes that depend on an IGP. Right? > From an implementation perspective there is logic to ensure that the > caching we do for performance proves the above visible behavior as > routes are added and deleted; just ensuring that the caching stays > consistent with the collection of forwarding table entries. > > Do you think we should document this somewhere? If so, where? route(1m)? By definition, to be a public interface, it must be documented. I'd suggest both route(1M) and route(7P). The latter is what routing protocol authors are supposed to be reading. > >> We are adding an informational RTF_KERNEL flag for routes, for instance > >> interface routes, that are added by the kernel as part of configuring an IP > >> interface. Such routes can not be accidentally deleted by applications. > > > > RTF_KERNEL doesn't appear for ICMP redirect, even though that's > > processed by the kernel, right? > > Correct. The "as part of configuring an IP interface" is key. I don't > know if a different name for the flag would make this more clear. No, that's ok. I was just checking that there wasn't some other meaning behind "kernel." (At one point, Linux seems to have had RTF_INTERFACE, and it seems to have had a similar meaning. But it doesn't have that flag now, and I don't know of an equivalent flag in use elsewhere on BSD or Linux.) > >> No IRE_CACHE entries (UHA) entries will appear in netstat -ra, since the > >> implementation no longer has IRE_CACHE entries. > >> This project adds a new IRE_IF_CLONE type of routes. Those routes appear in > >> netstat -ra (but not without the 'a' option) with the new 'C' flag. > > > > Are these like the BSD clone entries (i.e., effectively ARP cache > > entries, with *only* on-link addresses represented), or are they > > something else? > > I do not know the details of the BSD code. But the IRE_IF_CLONE entries > do not contain any ARP information. They are merely a place to hang a > reference to the ARP information. > For example, if we have an interface route for a /23 > 129.146.228.0 129.146.228.81 U 1 5 bge0 > and we are sending packets to 129.146.228.1 the kernel would create an > IRE_IF_CLONE for 129.146.228.1/32, and as the ARP entries gets created > there would be a direct pointer from that entry to the arp information. Yep; that's how BSD works. It has RTF_CLONING for the former route, to indicate that when you match it, you need to create a cloned route, and RTF_CLONED to mark the entries that were created by the cloning process. http://www.daemon-systems.org/man/route.8.html > This is beneficial for performance since we avoid doing a lookup of the > ARP information; we can always place a direct pointer to the ARP > information in the IRE that matches. Yes; that's the same reason BSD does it. > > What should those users be doing? (Reporting bugs against our PMTU > > implementation? Asking for an RFE? Yelling at the admin for that > > remote network?) > > I think it is due that that we implement RFC 4821 for TCP and SCTP which > would make PMTU discovery more robust overall. > > If there are remote networks with different PMTU then presumably the > customers are either 1) clamping down the MTU for their interfaces i.e., > reducing MTU for everybody, or 2) using route(1m) to set a specific mtu > for a particular route. > Both of those are unaffected by this project. OK; it sounded to me like the latter one (specific MTU for a route) would be removed. If it's not, then no issue here. > > Interface indices can't normally disappear and then reappear, can > > they? (If they can, then I'd call that a "bug" rather than a > > "feature." SNMP requires that ifIndex is unique for a given engine > > invocation.) > > This is outside of the scope of this case; this case merely makes sure > the system is at least as robust as today should the existing > SIOCSLIFINDEX be used to change the ifindex. > > Until relatively recently I thought we could actually remove the ability > to set the interface index (essentially removing SIOCSLIFINDEX). But > I've seen a legitimate case where a customer was using it (can't recall > the details). *sigh* OK. > In the example A could might pick the first U route, and B might pick > the second. (Similar examples can be constructed with indirect routes.) > Thus we do not do anything special should there be multiple routes for A/n. The multiple A/n case is the one I was interested in. By "do not do anything special" do you mean this? Forwarding always just picks one of these routes (in some unspecified way) and uses it; there is no load spreading. The fact that you have multiple routes gives you no benefit -- other than, perhaps, some robustness if one were deleted. Local connections, on the other hand, are able to use all of the A/n routes, and each connection will be assigned to one route in a round-robin when connecting (or otherwise when establishing or using the remote address). > It wouldn't be hard to add on top of what we are building, e.g., using a > hash of src+dest ip to select a route from the bucket in which we find > the matching A/n. OK. > >> The project removes the usage of multidata from TCP/IP, but the interfaces > >> specified in PSARC/2004/594 and PSARC/2002/276 remain in the system. > > > > This part is confusing; could you please elaborate? > > > > Does the stack still generate MDT messages? If not, then how are > > those two previous projects not affected? (Doesn't this project just > > obsolete those two ... ?) > > The TCP/IP stack no longer sends down MDT messages. > > But all the support in PSARC/2004/594 and PSARC/2002/276 remains (the > definition of M_MULTIDATA, the support in copymsg, the mmd_* functions. > > Thus device drivers that support M_MULTIDATA and link against those > functions will continue to link/load into the kernel. Other code which > sends M_MULTIDATA messages will continue to work. What other code (besides the TCP/IP stack) sends M_MULTIDATA? It was a private interface, and I don't know of any other users. If we're disabling multidata, then why not start the process of removing this old stuff? We could at least start the notification process on the MDT contracts. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From erik.nordmark@sun.com Thu Jun 4 11:29:32 2009 Received: from jurassic-x4600.sfbay.sun.com (cretaceous.SFBay.Sun.COM [129.146.17.59]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n54ITWqG003698 for ; Thu, 4 Jun 2009 11:29:32 -0700 (PDT) Received: from [10.7.251.248] (punchin-nordmark.SFBay.Sun.COM [10.7.251.248]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n54ITVI3394076 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 4 Jun 2009 11:29:31 -0700 (PDT) Message-ID: <4A2804AA.4070909@sun.com> Date: Thu, 04 Jun 2009 10:30:18 -0700 From: Erik Nordmark User-Agent: Thunderbird 2.0.0.21 (X11/20090323) MIME-Version: 1.0 To: James Carlson CC: Sebastien Roy , psarc-ext Subject: Re: IP Datapath Refactoring [PSARC/2009/331 FastTrack timeout 06/09/2009] References: <1243950219.6636.12.camel@strat> <18981.13118.399313.321138@gargle.gargle.HOWL> <4A26B9E9.70004@sun.com> <18982.49378.410578.978181@gargle.gargle.HOWL> In-Reply-To: <18982.49378.410578.978181@gargle.gargle.HOWL> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 3157 James Carlson wrote: >> Two things are different for externally visible behavior: >> - when the route is added the kernel does not require that the gateway >> is directly reachable. If RTF_INDIRECT is not set the gateway must be >> directly reachable as is the case in Solaris today. >> - when we lookup a route and first find an indirect route, we iterate >> to lookup the gateway in that route. (Today if we first find an >> offlink/offsubnet route i.e., one with RTF_GATEWAY set, we iterate by >> looking up its gateway. The indirect route in essence adds one more step >> in this iterative process.) > > Got it. So this is a flag that I-BGP should use when adding routes > that depend on an IGP. Right? Yep. >> From an implementation perspective there is logic to ensure that the >> caching we do for performance proves the above visible behavior as >> routes are added and deleted; just ensuring that the caching stays >> consistent with the collection of forwarding table entries. >> >> Do you think we should document this somewhere? If so, where? route(1m)? > > By definition, to be a public interface, it must be documented. > > I'd suggest both route(1M) and route(7P). The latter is what routing > protocol authors are supposed to be reading. OK. Do you want to see draft man pages? (It would essenntially be my "two things" text above.) > Yep; that's how BSD works. > > It has RTF_CLONING for the former route, to indicate that when you > match it, you need to create a cloned route, and RTF_CLONED to mark > the entries that were created by the cloning process. > > http://www.daemon-systems.org/man/route.8.html And I think RTF_CLONED maps to 'W' if I don't misremember. Would it make sense to use 'W' instead of 'C' for Solaris in this context? > The multiple A/n case is the one I was interested in. By "do not do > anything special" do you mean this? > > Forwarding always just picks one of these routes (in some > unspecified way) and uses it; there is no load spreading. The > fact that you have multiple routes gives you no benefit -- other > than, perhaps, some robustness if one were deleted. Local > connections, on the other hand, are able to use all of the A/n > routes, and each connection will be assigned to one route in a > round-robin when connecting (or otherwise when establishing or > using the remote address). Yep. Note that right now the refactor-gate implementation for local connections doesn't have visibility to the port numbers when it does the ECMP behavior. We'd to fix that to get better spreading. > What other code (besides the TCP/IP stack) sends M_MULTIDATA? It was > a private interface, and I don't know of any other users. > > If we're disabling multidata, then why not start the process of > removing this old stuff? We could at least start the notification > process on the MDT contracts. Not enough hours in a day, and I don't want to spend those few hours on debating whether or not Cassini is the best Ethernet driver ever. (As far as I know Cassini is the only driver that uses multidata.) Erik From carlsonj@phorcys.east.sun.com Thu Jun 4 11:44:42 2009 Received: from dm-east-01.east.sun.com (dm-east-01.East.Sun.COM [129.148.9.192]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n54Iif2G003987 for ; Thu, 4 Jun 2009 11:44:41 -0700 (PDT) Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by dm-east-01.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n54IiaKl004840; Thu, 4 Jun 2009 14:44:36 -0400 (EDT) Received: from phorcys.east.sun.com (phorcys.local [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n54IhRjO005296; Thu, 4 Jun 2009 14:43:27 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n54IhRtm005293; Thu, 4 Jun 2009 14:43:27 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18984.5583.821145.700546@gargle.gargle.HOWL> Date: Thu, 4 Jun 2009 14:43:27 -0400 From: James Carlson To: Erik Nordmark Cc: Sebastien Roy , psarc-ext Subject: Re: IP Datapath Refactoring [PSARC/2009/331 FastTrack timeout 06/09/2009] In-Reply-To: <4A2804AA.4070909@sun.com> References: <1243950219.6636.12.camel@strat> <18981.13118.399313.321138@gargle.gargle.HOWL> <4A26B9E9.70004@sun.com> <18982.49378.410578.978181@gargle.gargle.HOWL> <4A2804AA.4070909@sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 2981 Erik Nordmark writes: > James Carlson wrote: > > I'd suggest both route(1M) and route(7P). The latter is what routing > > protocol authors are supposed to be reading. > > OK. > Do you want to see draft man pages? (It would essenntially be my "two > things" text above.) No need; the discussion has cleared this up for me. > > Yep; that's how BSD works. > > > > It has RTF_CLONING for the former route, to indicate that when you > > match it, you need to create a cloned route, and RTF_CLONED to mark > > the entries that were created by the cloning process. > > > > http://www.daemon-systems.org/man/route.8.html > > And I think RTF_CLONED maps to 'W' if I don't misremember. Would it make > sense to use 'W' instead of 'C' for Solaris in this context? The table that I know about is in here: http://cvsweb.netbsd.org/cgi-bin/cvsweb.cgi/~checkout~/src/usr.bin/netstat/netstat.1?rev=1.51&content-type=text/plain ... and it maps RTF_CLONING to 'C' and RTF_CLONED to 'c'. I don't know of a flag that maps to 'W', but I guess it's possible that was done on some system. Lining up with BSD would be a nice thing, if possible, but if we have to be different, I wouldn't complain too much. > Note that right now the refactor-gate implementation for local > connections doesn't have visibility to the port numbers when it does the > ECMP behavior. We'd to fix that to get better spreading. I'm not sure I follow. I thought that the selection mechanism for local connections was described as round-robin. How would port numbers or any ECMP get involved with that? (I agree that if you're doing ECMP, the more flow-identifying stuff you can get in there, the better. But that might be tending towards design review ...) > > What other code (besides the TCP/IP stack) sends M_MULTIDATA? It was > > a private interface, and I don't know of any other users. > > > > If we're disabling multidata, then why not start the process of > > removing this old stuff? We could at least start the notification > > process on the MDT contracts. > > Not enough hours in a day, and I don't want to spend those few hours on > debating whether or not Cassini is the best Ethernet driver ever. > (As far as I know Cassini is the only driver that uses multidata.) At least to me, that's not quite the point. We're _breaking_ that previous feature by permanently disabling it. I think that may well be a good thing -- I can believe that we get acceptably good performance without the complication of MDT -- but I don't think it's useful to say that those things are still "supported" but simply never work. Would we say we "support" LSO but then never allow anyone to use it? I think doing this cleanly means obsoleting those old interfaces properly. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From erik.nordmark@sun.com Thu Jun 4 16:48:42 2009 Received: from jurassic-x4600.sfbay.sun.com (cretaceous.SFBay.Sun.COM [129.146.17.63]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n54NmgR1004798 for ; Thu, 4 Jun 2009 16:48:42 -0700 (PDT) Received: from [10.7.251.248] (punchin-nordmark.SFBay.Sun.COM [10.7.251.248]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n54NmeDD615497 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 4 Jun 2009 16:48:42 -0700 (PDT) Message-ID: <4A285D58.2090102@sun.com> Date: Thu, 04 Jun 2009 16:48:40 -0700 From: Erik Nordmark User-Agent: Thunderbird 2.0.0.21 (X11/20090323) MIME-Version: 1.0 To: James Carlson CC: Sebastien Roy , psarc-ext Subject: Re: IP Datapath Refactoring [PSARC/2009/331 FastTrack timeout 06/09/2009] References: <1243950219.6636.12.camel@strat> <18981.13118.399313.321138@gargle.gargle.HOWL> <4A26B9E9.70004@sun.com> <18982.49378.410578.978181@gargle.gargle.HOWL> <4A2804AA.4070909@sun.com> <18984.5583.821145.700546@gargle.gargle.HOWL> In-Reply-To: <18984.5583.821145.700546@gargle.gargle.HOWL> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 3563 James Carlson wrote: > The table that I know about is in here: > > http://cvsweb.netbsd.org/cgi-bin/cvsweb.cgi/~checkout~/src/usr.bin/netstat/netstat.1?rev=1.51&content-type=text/plain > > ... and it maps RTF_CLONING to 'C' and RTF_CLONED to 'c'. I don't > know of a flag that maps to 'W', but I guess it's possible that was > done on some system. Ah - I looked at freebsd at http://www.freebsd.org/cgi/man.cgi?query=netstat&apropos=0&sektion=0&manpath=FreeBSD+7.2-RELEASE&format=html which has different flags: C RTF_CLONING Generate new routes on use c RTF_PRCLONING Protocol-specified generate new routes on use W RTF_WASCLONED Route was generated as a result of cloning > Lining up with BSD would be a nice thing, if possible, but if we have > to be different, I wouldn't complain too much. Which BSD seems to be the key question ;-) I'll stick with 'C'. >> Note that right now the refactor-gate implementation for local >> connections doesn't have visibility to the port numbers when it does the >> ECMP behavior. We'd to fix that to get better spreading. > > I'm not sure I follow. I thought that the selection mechanism for > local connections was described as round-robin. How would port > numbers or any ECMP get involved with that? Where is it described as round-robin? In my email where I said "currently the kernel only does some form of round robin for default routes"? The issue is that we don't want the implementation to pick a different route just because some unrelated route change caused a need to revalidate the ire cached for the connection. We do this by having a predictable implementation. My point was that that algorithm can be improved. > (I agree that if you're doing ECMP, the more flow-identifying stuff > you can get in there, the better. But that might be tending towards > design review ...) I think we already crossed that line on this particular topic ;-) >> Not enough hours in a day, and I don't want to spend those few hours on >> debating whether or not Cassini is the best Ethernet driver ever. >> (As far as I know Cassini is the only driver that uses multidata.) > > At least to me, that's not quite the point. We're _breaking_ that > previous feature by permanently disabling it. I think that may well > be a good thing -- I can believe that we get acceptably good > performance without the complication of MDT -- but I don't think it's > useful to say that those things are still "supported" but simply never > work. In which sense is it a feature? It is just a performance trick with a contract private interface. Given that the trick now cost more in support complexity than its benefits, why are we not free to stop using that trick? > Would we say we "support" LSO but then never allow anyone to use it? Do we claim to support MDT? I don't see what that would mean given that the interfaces are contract private, hence no ISV/IHV should use them. > I think doing this cleanly means obsoleting those old interfaces > properly. I do not see any harm in keeping the MDT interfaces in place so that the Cassini driver binaries continue to load into the kernel and work properly. Once Cassini is EOLed we can remove all vestiges of the MDT from the system, but removing it earlier implies addition cost to the business (maintaining another version of the Cassini driver) which doesn't seem warranted. But if you are going to block this case on this issue, then I would be forced to go spend the time to start the EO*L process. Erik From carlsonj@phorcys.east.sun.com Fri Jun 5 06:17:48 2009 Received: from dm-east-02.east.sun.com (dm-east-02.East.Sun.COM [129.148.13.5]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n55DHmTc010880 for ; Fri, 5 Jun 2009 06:17:48 -0700 (PDT) Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by dm-east-02.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n55DHj31000560; Fri, 5 Jun 2009 09:17:45 -0400 (EDT) Received: from phorcys.east.sun.com (phorcys.local [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n55DGaY6007972; Fri, 5 Jun 2009 09:16:36 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n55DGaFm007969; Fri, 5 Jun 2009 09:16:36 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18985.6836.895444.668097@gargle.gargle.HOWL> Date: Fri, 5 Jun 2009 09:16:36 -0400 From: James Carlson To: Erik Nordmark Cc: Sebastien Roy , psarc-ext Subject: Re: IP Datapath Refactoring [PSARC/2009/331 FastTrack timeout 06/09/2009] In-Reply-To: <4A285D58.2090102@sun.com> References: <1243950219.6636.12.camel@strat> <18981.13118.399313.321138@gargle.gargle.HOWL> <4A26B9E9.70004@sun.com> <18982.49378.410578.978181@gargle.gargle.HOWL> <4A2804AA.4070909@sun.com> <18984.5583.821145.700546@gargle.gargle.HOWL> <4A285D58.2090102@sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 5328 Erik Nordmark writes: > > The table that I know about is in here: > > > > http://cvsweb.netbsd.org/cgi-bin/cvsweb.cgi/~checkout~/src/usr.bin/netstat/netstat.1?rev=1.51&content-type=text/plain > > > > ... and it maps RTF_CLONING to 'C' and RTF_CLONED to 'c'. I don't > > know of a flag that maps to 'W', but I guess it's possible that was > > done on some system. > > Ah - I looked at freebsd at > http://www.freebsd.org/cgi/man.cgi?query=netstat&apropos=0&sektion=0&manpath=FreeBSD+7.2-RELEASE&format=html > which has different flags: > C RTF_CLONING Generate new routes on use > c RTF_PRCLONING Protocol-specified generate new routes on use > W RTF_WASCLONED Route was generated as a result of cloning Interesting ... I guess it's not surprising that there's mutation going on here, though it's puzzling how that RTF_PRCLONING would be used. > > Lining up with BSD would be a nice thing, if possible, but if we have > > to be different, I wouldn't complain too much. > > Which BSD seems to be the key question ;-) > > I'll stick with 'C'. OK. > >> Note that right now the refactor-gate implementation for local > >> connections doesn't have visibility to the port numbers when it does the > >> ECMP behavior. We'd to fix that to get better spreading. > > > > I'm not sure I follow. I thought that the selection mechanism for > > local connections was described as round-robin. How would port > > numbers or any ECMP get involved with that? > > Where is it described as round-robin? > In my email where I said "currently the kernel only does some form of > round robin for default routes"? You said this: The project extends the kernel's ability to handle multiple routes for the same prefix; currently the kernel only does some form of round robin for default routes and the project extends that to all off-link routes (default, prefix, and host routes). That text appears to say directly that you're extending the existing round-robin behavior seen for default routes to all off-link routes. Is that not true? > The issue is that we don't want the implementation to pick a different > route just because some unrelated route change caused a need to > revalidate the ire cached for the connection. We do this by having a > predictable implementation. My point was that that algorithm can be > improved. OK. Hashed IDs instead of round-robin is certainly fine by me. If you go back through the thread, I was just trying to find out exactly what the feature was (and wasn't) supposed to do. Nothing more. > > At least to me, that's not quite the point. We're _breaking_ that > > previous feature by permanently disabling it. I think that may well > > be a good thing -- I can believe that we get acceptably good > > performance without the complication of MDT -- but I don't think it's > > useful to say that those things are still "supported" but simply never > > work. > > In which sense is it a feature? It is just a performance trick with a > contract private interface. Given that the trick now cost more in > support complexity than its benefits, why are we not free to stop using > that trick? The IP stack has a contract extended to 'ce' to use it. > > Would we say we "support" LSO but then never allow anyone to use it? > > Do we claim to support MDT? Sure. For example, see: http://docs.sun.com/app/docs/doc/817-0404/appendixa-46?a=view That's still not really the point here. > I don't see what that would mean given that > the interfaces are contract private, hence no ISV/IHV should use them. I doubt they do. I think you're missing the point I'm making. > > I think doing this cleanly means obsoleting those old interfaces > > properly. > > I do not see any harm in keeping the MDT interfaces in place so that the > Cassini driver binaries continue to load into the kernel and work properly. > Once Cassini is EOLed we can remove all vestiges of the MDT from the > system, but removing it earlier implies addition cost to the business > (maintaining another version of the Cassini driver) which doesn't seem > warranted. > > But if you are going to block this case on this issue, then I would be > forced to go spend the time to start the EO*L process. I'm not asking about removal of the functions. I agree that'd be quite foolish. It'd break at least the 'ce' driver and do so in violation of the agreed-on contract and for no apparent benefit. But, as I said, I'm not asking for that at all. (Nor do I understand why anyone would think I'd ask for breakage like that or that by asking questions I'd somehow "block this case." I never said any such things.) What I am asking for is *merely* documentation. Please explicitly say that this project retires those previous projects (the ARC cases themselves). You could do so by: - Explicitly stating that the interfaces specified in those previous projects are now Obsolete. - Going through the process of notifying the contract holders that the interfaces are going away. That's all. It shouldn't be this hard. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From erik.nordmark@sun.com Fri Jun 5 14:48:30 2009 Received: from jurassic-x4600.sfbay.sun.com (cretaceous.SFBay.Sun.COM [129.146.17.63]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n55LmUGE021341 for ; Fri, 5 Jun 2009 14:48:30 -0700 (PDT) Received: from [10.7.251.248] (punchin-nordmark.SFBay.Sun.COM [10.7.251.248]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n55LmRJC245957 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 5 Jun 2009 14:48:30 -0700 (PDT) Message-ID: <4A2992AB.6060106@sun.com> Date: Fri, 05 Jun 2009 14:48:27 -0700 From: Erik Nordmark User-Agent: Thunderbird 2.0.0.21 (X11/20090323) MIME-Version: 1.0 To: James Carlson CC: Sebastien Roy , psarc-ext Subject: Re: IP Datapath Refactoring [PSARC/2009/331 FastTrack timeout 06/09/2009] References: <1243950219.6636.12.camel@strat> <18981.13118.399313.321138@gargle.gargle.HOWL> <4A26B9E9.70004@sun.com> <18982.49378.410578.978181@gargle.gargle.HOWL> <4A2804AA.4070909@sun.com> <18984.5583.821145.700546@gargle.gargle.HOWL> <4A285D58.2090102@sun.com> <18985.6836.895444.668097@gargle.gargle.HOWL> In-Reply-To: <18985.6836.895444.668097@gargle.gargle.HOWL> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 4193 James Carlson wrote: > You said this: > > The project extends the kernel's ability to handle multiple routes for the same > prefix; currently the kernel only does some form of round robin for default > routes and the project extends that to all off-link routes (default, prefix, and > host routes). > > That text appears to say directly that you're extending the existing > round-robin behavior seen for default routes to all off-link routes. > > Is that not true? The key is "some form" - currently we don't actually specify that the current default route logic is round-robin, nor do we specify when we actually (re)select one of the default routes. (The implementation can actually re-select each time it runs low on memory, when the IRE_CACHE bucket gets too long, or when routes or interfaces are added or deleted.) >> The issue is that we don't want the implementation to pick a different >> route just because some unrelated route change caused a need to >> revalidate the ire cached for the connection. We do this by having a >> predictable implementation. My point was that that algorithm can be >> improved. > > OK. Hashed IDs instead of round-robin is certainly fine by me. If > you go back through the thread, I was just trying to find out exactly > what the feature was (and wasn't) supposed to do. Nothing more. Understood. But it leads to the question at what detail we should document the behavior of the new implementation. We want to be helpful to users wanting to understand things, without preventing us from evolving the implementation to do better load spreading over time. >>> At least to me, that's not quite the point. We're _breaking_ that >>> previous feature by permanently disabling it. I think that may well >>> be a good thing -- I can believe that we get acceptably good >>> performance without the complication of MDT -- but I don't think it's >>> useful to say that those things are still "supported" but simply never >>> work. >> In which sense is it a feature? It is just a performance trick with a >> contract private interface. Given that the trick now cost more in >> support complexity than its benefits, why are we not free to stop using >> that trick? > > The IP stack has a contract extended to 'ce' to use it. Can you point out where in the contract it says that TCP/IP will use MDT in any particular case? I see interfaces for the driver to specify to IP that the driver can handle MDT, and a format (M_MULTIDATA) plus supporting data structures and functions that the driver can use to handle M_MULTIDATA. But there isn't anything that requires TCP/IP to use MDT in any particular case. If we look at the implementation there are a large number of conditions that apply to MDT. - It is never used with SCTP. - TCP turns it off if there are IPv4 options, IPv6 extension headers, IPsec, to IPQoS - Not used with RTF_MULTIRT My point is that the driver can not assume that any particular packet will use MDT today. Thus I can't see how it would be an interface change to reduce the set of cases when TCP/IP sends down M_MULTIDATA to the driver to the null set. > I'm not asking about removal of the functions. I agree that'd be > quite foolish. It'd break at least the 'ce' driver and do so in > violation of the agreed-on contract and for no apparent benefit. But, > as I said, I'm not asking for that at all. (Nor do I understand why > anyone would think I'd ask for breakage like that or that by asking > questions I'd somehow "block this case." I never said any such > things.) > > What I am asking for is *merely* documentation. Please explicitly say > that this project retires those previous projects (the ARC cases > themselves). You could do so by: I sure wish you would have stated that up front! Would have saved some time for all of us. > - Explicitly stating that the interfaces specified in those previous > projects are now Obsolete. > > - Going through the process of notifying the contract holders that > the interfaces are going away. > > That's all. It shouldn't be this hard. Will do. I'll send out an updated writeup shortly. Erik From erik.nordmark@sun.com Fri Jun 5 14:48:42 2009 Received: from jurassic-x4600.sfbay.sun.com (cretaceous.SFBay.Sun.COM [129.146.17.59]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n55LmgOk021353 for ; Fri, 5 Jun 2009 14:48:42 -0700 (PDT) Received: from [10.7.251.248] (punchin-nordmark.SFBay.Sun.COM [10.7.251.248]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n55LmY4k245984 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 5 Jun 2009 14:48:42 -0700 (PDT) Message-ID: <4A2992B2.5010707@sun.com> Date: Fri, 05 Jun 2009 14:48:34 -0700 From: Erik Nordmark User-Agent: Thunderbird 2.0.0.21 (X11/20090323) MIME-Version: 1.0 To: Yatin Gajjar CC: psarc-ext Subject: Obsoleting the multi-data transmit contracted interfaces [Re: IP Datapath Refactoring [PSARC/2009/331 FastTrack timeout 06/09/2009] Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 1198 Yatin, [I assume you are the best contact to notify about this.] The IP datapath refactoring project will make TCP/IP stop sending down M_MULTIDATA messages to the device drivers. However, the different ARCed interfaces remain in the system thus the Cassini driver will *not* need any changes. To accurately reflect this I've asked the interfaces in PSARC 2004/594 and PSARC 2002/276 to be marked as Obsolete. (In some future when the Cassini driver has been EOLed we can consider removing those interfaces from the system.) The two contracts have the following contacts at your end. http://sac.eng/arc/PSARC/2004/594/contract-01.txt 3. The CONSUMER is identified by the following: Product or Bundle: Solaris Consolidation: NSPG Department or Group: NCS Connectivity Bugtraq Category/SubCategory: ethernet_cassini/cassini_sw Responsible Manager: Leo Hejza http://sac.eng/arc/PSARC/2002/276/contract-001 3. The CONSUMER is identified by the following: Product or Bundle: Solaris Consolidation: OS/Networking Department or Group: N&S/Cassini Bugtraq Category/SubCategory: ethernet_cassini/cassini_sw Responsible Manager: Jay Jayachandran Erik From erik.nordmark@sun.com Fri Jun 5 14:52:13 2009 Received: from jurassic-x4600.sfbay.sun.com (cretaceous.SFBay.Sun.COM [129.146.17.63]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n55LqD46021577 for ; Fri, 5 Jun 2009 14:52:13 -0700 (PDT) Received: from [10.7.251.248] (punchin-nordmark.SFBay.Sun.COM [10.7.251.248]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n55LqAhP246580 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 5 Jun 2009 14:52:13 -0700 (PDT) Message-ID: <4A29938A.5090506@sun.com> Date: Fri, 05 Jun 2009 14:52:10 -0700 From: Erik Nordmark User-Agent: Thunderbird 2.0.0.21 (X11/20090323) MIME-Version: 1.0 To: James Carlson CC: Sebastien Roy , psarc-ext Subject: Re: IP Datapath Refactoring [PSARC/2009/331 FastTrack timeout 06/09/2009] References: <1243950219.6636.12.camel@strat> <18981.13118.399313.321138@gargle.gargle.HOWL> <4A26B9E9.70004@sun.com> <18982.49378.410578.978181@gargle.gargle.HOWL> <4A2804AA.4070909@sun.com> <18984.5583.821145.700546@gargle.gargle.HOWL> <4A285D58.2090102@sun.com> <18985.6836.895444.668097@gargle.gargle.HOWL> <4A2992AB.6060106@sun.com> In-Reply-To: <4A2992AB.6060106@sun.com> Content-Type: multipart/mixed; boundary="------------050008070900060808000901" Status: RO Content-Length: 18384 This is a multi-part message in MIME format. --------------050008070900060808000901 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Here is the updated writeup, with draft man page diffs. Hopefully this clarifies things. Erik --------------050008070900060808000901 Content-Type: text/plain; name="ipd.psarc2" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="ipd.psarc2" IP Datapath Refactoring ======================= The project changes a lot of the IP datapath code paths, but has rather minor changes to documented interfaces. Those changes are the subject of this case. The IP datapaths are extremely hard to follow both at the micro level (ip_output_options and ip_wput_ire, and ip_input) and at the macro level (an outbound packet needing IPsec and ARP resolution goes through a large number of steps). The nature of the changes in IP derive from the classical quote "ip_newroute delenda est"; a key root cause of the complexity is from the introduction of asynchronous behavior near the top of the IP output path. Moving the asynchronony to the bottom of the IP output path, where other implementations handle ARP resolution, makes things more sane. Also, partly due to the ip_newroute asynchrony, IP has grown a large number of internal mechanisms to remember state associated with a packet that needs to be queued. We are replacing that with a single internal mechanism in the form of ip_xmit_attr_t and ip_recv_attr_t data structures. All those mechanisms and interfaces are project private. Current prototyping indicates that about 30,000 lines of code can be removed as a result of these changes (combined with the ARP/IP merge pieces). Imported interfaces: -------------------- No changes. Exported interfaces: -------------------- We are introducing a new RTF_INDIRECT in . This flag is useful for routing daemons that do BGP plus OSPF/IS-IS since it can make handling routing changes a lot more efficient. Route(1m) has a new -indirect flag to set RTF_INDIRECT. The indirect routes are represented by a new 'I' flag in the netstat -r output. Old cases state that the classification of the output of netstat as Unstable. This case doesn't change that; it just adds the 'I' flag. We are adding an informational RTF_KERNEL flag for routes, for instance interface routes, that are added by the kernel as part of configuring an IP interface. Such routes can not be accidentally deleted by applications. The implementation introduces a new Destination Cache Entry in the kernel (patterned after the description in RFC 4861). For debugging reasons it is useful to be able to display the DCE in particularly the Path MTU which is recorded in it. We are adding the -d option to netstat(1m) for this purpose. (Note that netstat -d is currently an undocumented option for debugging netstat itself. We rename that undocumented debug option to -x.) The way netstat extracts the DCE table from the kernel is using the new EXPER_IP_DCE and dest_cache_entry_t in . The implementation of multirt/CGTP changes, and in all but one detail the interfaces remain unchanged. The change is that evolving and undocumented tunable ip_multirt_resolution_interval has been removed. Multirt/CGTP routes use the same timer as ARP/ND does for other routes. Note that PSARC/2003/041 replaced PSARC/2000/539 and the never contract didn't mention the ndd tunable; this might have been an omission. The project removes the usage of multidata from TCP/IP, but the interfaces specified in PSARC/2004/594 and PSARC/2002/276 remain in the system. Since TCP/IP will no longer use those interfaces this case marks them as Obsolete, with the intent that they be removed when the contracted device drivers are EOLed. _________________________________________________________________________ | Interfaces Added by This Case | |_______________________________|_______________________|_______________| |Interface | Classification | Comments | |_______________________________|_______________________|_______________| | RTF_INDIRECT | Committed | | | route(1m) -indirect flag | Committed | | | netstat(1m) -r output | Uncommitted (unchanged)| | | RTF_KERNEL | Committed | | | netstat(1m) -d option | Committed | | | netstat(1m) -d output | Uncommitted | | | EXPER_IP_DCE | Uncommitted | | | dce_cache_entry_t | Uncommitted | | |_______________________________|_______________________|_______________| _________________________________________________________________________ | Interfaces Removed by This Case | |_______________________________|_______________________|_______________| |Interface | Classification | Comments | |_______________________________|_______________________|_______________| | ip_multirt_resolution_interval| Evolving | PSARC/2000/539| |_______________________________|_______________________|_______________| _________________________________________________________________________ | Interfaces Obsoleted by This Case | |_______________________________|_______________________|_______________| |Interface | Classification | Comments | |_______________________________|_______________________|_______________| | PSARC/2004/594 | Obsolete | multidata | | PSARC/2002/276 | Obsolete | multidata | |_______________________________|_______________________|_______________| Implementation changes: ----------------------- Due to the ARP/IP merge and uniform application of Neighbor Unreachability Detection (RFC 4862) the undocumented ndd tunables for /dev/arp are replaced by undocumented ndd tunables for /dev/ip. Instead of relying on timers (with undocumented but well-known tunables like arp_cleanup_interval and ip_ire_arp_interval) this project makes ARP function the same way as Neighbor Discovery in using the RFC 4862 NUD state machine. Thus those known, but undocumented, tunables are removed. The implementation changes to track IPv4 group membership per ill_t instead of per ipif_t, but we keep the IP address around so we can preserve the output of netstat -g (which reports the logical interface name e.g., bge0:1 when an IPv4 group is joined using the IP address assigned to bge0:1) netstat -ia continues to show input counters for each local address. However, the output counters never made any sense on a per-local address (IP packets are sent out of a IP interface and not out an IP address), and this project makes them be reported as zero. The Solaris 'Use' count in netstat -r has been this unpredictable/undocumented number since Solaris 2.0 (the implementation counts the number of times ip_newroute has used the route to try to create an IRE_CACHE entry). We restore the use count to actually count the number of packets that are send out using the route in question. No IRE_CACHE entries (UHA) entries will appear in netstat -ra, since the implementation no longer has IRE_CACHE entries. This project adds a new IRE_IF_CLONE type of routes. Those routes appear in netstat -ra (but not without the 'a' option) with the new 'C' flag. While the kernel no longer uses any IRE_CACHE entries, we are keeping the #define of IRE_CACHE in the header file so that applications which use the common, but undocumented, mibget approach for retrieving the kernel routing table will still compile. The new implementation no longer has a ire_max_frag field, hence the output of Maxfrg/PMTU in the netstat -rv output is no longer useful. We are removing that output. (Note that the details of the netstat output is not a stable interface.) Currently Solaris handles IP interface MTU in odd ways in that it can be set differently for local IP address prefix; this leaves it quite undefined in what MTU is applied to multicast packets. This project fixes that by applying the IP interface MTU per interface. As a result ifconfig bge0:N mtu 1400 will fail with EINVAL. No mapping entry will appear in arp -a/netstat -p output, since the implementation of the multicast mapping has changed. Individual multicast and broadcast addresses might appear in netstat -p/arp -a. API calls which refer to interface indicies and interface addresses (IP_MULTICAST_IF, IP_BOUND_IF, IP_ADD_MEMBERSHIP, etc) currently have odd behaviors when interfaces and/or IP addresses are unplumbed and removed. To preserve kernel sanity (no stray ill and ipif pointers) the applications setting is forgotten with telling the application. From the application the behavior looks very odd. E.g., if an IP_ADD_MEMBERSHIP is followed by a correct IP_DROP_MEMBERSHIP will see a EADDRNOTAVAIL error because the kernel might have removed all memory of the IP_ADD_MEMBERSHIP when the IP address was removed. This project will instead preserve what the application has set until the application explicitly removes it. For instance, an IP_BOUND_IF will remain in effect even if the IP interface is unplumbed. Packets will be dropped and ENXIO will be returned to the sendto() system call. And received packets will be dropped since they can't possibly match the interface index specified in the IP_BOUND_IF when the interface has been unplumbed. However, when the IP address (or interface index) which was use by the application reappears, then the application's setting will be fully functional again. The project extends the kernel's ability to handle multiple routes for the same prefix; currently the kernel only does some form of round-robin attempt at load spreading for default routes. This project extends that to all off-link routes (default, prefix, and host routes). We are adding an undocumented knob should there be a reason to switch back to the old behavior in the field. (The current implementation of ECMP in the forwarding path is very simplistic and not very useful. This and the detailed algorithm for how the host path selects among different ECMP routes is likely to evolve over time. Thus we are not documenting the details of how IP selects among ECMP routes.) The project changes the *use* of DL_NOTE_REPLUMB (introduced by PSARC 2008/242 as a private interface); it is only used on the IP stream(s) and not on the ARP stream. Notes: ------ This project removes the AR_* message set used by IP and ARP. Earlier there was a contract private interface with SunATM on those interfaces (established in LSARC/1993/101/ and extended in PSARC/1999/446 and PSARC/2001/023). That contract was cancelled by PSARC/2006/272 EOL of ATM device driver The integration of this project is likely to also deliver the changes related to PSARC/2008/522 EOF of 2001/070 IPsec HW Acceleration support ---- --------------050008070900060808000901 Content-Type: text/plain; name="ipd.psarc.netstat.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="ipd.psarc.netstat.diff" *** ipd.psarc.netstat.orig Fri May 8 09:20:15 2009 --- ipd.psarc.netstat.new Fri May 8 11:06:07 2009 *************** *** 37,42 **** --- 37,43 ---- netstat -D [-I interface] [-f address_family] + netstat -d [-f address_family] DESCRIPTION The netstat command displays the contents of certain *************** *** 83,88 **** --- 84,91 ---- o The ninth form (-D option) displays the state of DHCP on one or all interfaces. + o The tenth form (-d option) displays the table of + destination cache entries. These forms are described in greater detail below. *************** *** 102,107 **** --- 105,115 ---- status of physical interfaces is shown. + -d + + Show the destination cache entry table. See DISPLAYS, below. + + -f address_family Limit all displays to those of the specified *************** *** 690,705 **** D Route was created dynamically by a redirect. If the -a option is specified, there will be routing entries with the following flags: - A Combined routing and address resolution entries. - B Broadcast addresses. L Local addresses for the host. --- 698,717 ---- D Route was created dynamically by a redirect. + I Indirect routes (gateway not directly reachable) established with + the -indirect option. + If the -a option is specified, there will be routing entries with the following flags: B Broadcast addresses. + C Clones interface host route entries for on-link destinations. + + L Local addresses for the host. *************** *** 709,720 **** address of the outgoing interface. ! The use column displays the number of packets sent using a ! combined routing and address resolution (A) or a broadcast ! (B) route. For a local (L) route, this count is the number ! of packets received, and for all other routes it is the ! number of times the routing entry has been used to create a ! new combined route and address resolution entry. --- 721,728 ---- address of the outgoing interface. ! The use column displays the number of packets sent or ! forwarded using the route in question. *************** *** 800,805 **** --- 808,821 ---- packets sent, received, and declined while attempting to obtain a future lease. + Destination Cache Entry Table (Tenth Form) + The destination cache entry display shows the recorded path MTU, the + age (in seconds) of the entry, and flags. The 'P' flag indicates that + a path MTU is recorded. The 'S' flag indicates that the path MTU is + smaller than the minumum that IP will allow. The 'U' flag indicates that + some transport metrics (round-trip time, etc) are cached in the + destination cache entry. + FILES /etc/default/inet_type DEFAULT_IP setting --------------050008070900060808000901 Content-Type: text/plain; name="ipd.psarc.route.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="ipd.psarc.route.diff" *** ipd.psarc.route.orig Fri May 8 09:28:58 2009 --- ipd.psarc.route.new Fri Jun 5 14:37:31 2009 *************** *** 409,414 **** --- 409,421 ---- requests to delete or change a route on an underlying inter- face will fail. + When the routing table contains several equal routes, that is, routes + for the same destination and mask, then IP attempts to spread the + traffic over those routes. The spreading is such that an individual + transport connection uses the same route to avoid packet reordering + as seen by e.g., TCP. The details of the spreading algoritm is not + documented and is likely to evolve over time. + Routing Flags Routes have associated flags that influence operation of the protocols when sending to destinations matched by the *************** *** 432,439 **** --- 439,454 ---- -private RTF_PRIVATE Do not advertise this route -multirt RTF_MULTIRT Creates the specified redundant route -setsrc RTF_SETSRC Assigns the default source address + -indirect RTF_INDIRECT Allows adding routes where the gateway + is not on-link. + The optional -indirect modifier allows adding routes where the + gateway is not directly reachable. When an indirect route is + the best match for a packet to be sent or forwarded, then IP + proceeds to lookup that gateway to find a route that is directly + reachable. The -indirect modifier can be used even if the + gateway is directly reachable. The optional modifiers -rtt, -rttvar, -sendpipe, -recvpipe, -mtu, -hopcount, -expire, and -ssthresh provide initial --------------050008070900060808000901 Content-Type: text/plain; name="ipd.psarc.route7p.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="ipd.psarc.route7p.diff" *** ipd.psarc.route7p.orig Fri Jun 5 14:21:15 2009 --- ipd.psarc.route7p.new Fri Jun 5 14:35:39 2009 *************** *** 341,346 **** --- 341,350 ---- #define RTF_PRIVATE 0x2000 /* do not advertise this route */ #define RTF_PROTO2 0x4000 /* protocol specific routing flag #2 */ #define RTF_PROTO1 0x8000 /* protocol specific routing flag #1 */ + #define RTF_MULTIRT 0x10000 /* multiroute */ + #define RTF_SETSRC 0x20000 /* set default outgoing src address */ + #define RTF_INDIRECT 0x40000 /* gateway not directly reachable */ + #define RTF_KERNEL 0x80000 /* created by kernel; can't delete */ /* Specifiers for metric values in rmx_locks and rtm_inits are */ *************** *** 373,379 **** --- 377,396 ---- Some of the metrics might not be implemented and return zero. The implemented metrics are set in rtm_inits. + The RTF_INDIRECT flag allows adding routes where the + gateway is not directly reachable. When an indirect route is + the best match for a packet to be sent or forwarded, then IP + proceeds to lookup that gateway to find a route that is directly + reachable. The RTF_INDIRECT flag can be used even if the + gateway is directly reachable. + When the routing table contains several equal routes, that is, routes + for the same destination and mask, then IP attempts to spread the + traffic over those routes. The spreading is such that an individual + transport connection uses the same route to avoid packet reordering + as seen by e.g., TCP. The details of the spreading algoritm is not + documented and is likely to evolve over time. + --------------050008070900060808000901-- From carlsonj@phorcys.east.sun.com Tue Jun 16 10:23:55 2009 Received: from dm-east-02.east.sun.com (dm-east-02.East.Sun.COM [129.148.13.5]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n5GHNsTi025479 for ; Tue, 16 Jun 2009 10:23:54 -0700 (PDT) Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by dm-east-02.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n5GHNqEH060861; Tue, 16 Jun 2009 13:23:52 -0400 (EDT) Received: from phorcys.east.sun.com (phorcys.local [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n5GHMYLF005292; Tue, 16 Jun 2009 13:22:34 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n5GHMYI0005289; Tue, 16 Jun 2009 13:22:34 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18999.54489.942248.529106@gargle.gargle.HOWL> Date: Tue, 16 Jun 2009 13:22:33 -0400 From: James Carlson To: Erik Nordmark Cc: Sebastien Roy , psarc-ext Subject: Re: IP Datapath Refactoring [PSARC/2009/331 FastTrack timeout 06/09/2009] In-Reply-To: <4A29938A.5090506@sun.com> References: <1243950219.6636.12.camel@strat> <18981.13118.399313.321138@gargle.gargle.HOWL> <4A26B9E9.70004@sun.com> <18982.49378.410578.978181@gargle.gargle.HOWL> <4A2804AA.4070909@sun.com> <18984.5583.821145.700546@gargle.gargle.HOWL> <4A285D58.2090102@sun.com> <18985.6836.895444.668097@gargle.gargle.HOWL> <4A2992AB.6060106@sun.com> <4A29938A.5090506@sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 377 Erik Nordmark writes: > Here is the updated writeup, with draft man page diffs. > > Hopefully this clarifies things. The updates answer my issues; +1. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From Sebastien.Roy@Sun.COM Tue Jun 16 10:28:58 2009 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n5GHSwN6025798 for ; Tue, 16 Jun 2009 10:28:58 -0700 (PDT) Received: from brmea-mail-4.sun.com (brmea-mail-4.Sun.COM [192.18.98.36]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n5GHSwT7031657 for ; Tue, 16 Jun 2009 10:28:58 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n5GHSw0W001199 for ; Tue, 16 Jun 2009 17:28:58 GMT MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) id <0KLC00800DJCBV00@mail-amer.sun.com> for psarc-ext@sac.sfbay.sun.com; Tue, 16 Jun 2009 11:28:58 -0600 (MDT) Received: from [129.148.174.103] ([unknown] [129.148.174.103]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009)) with ESMTPSA id <0KLC008ZEDVUIS30@mail-amer.sun.com>; Tue, 16 Jun 2009 11:28:43 -0600 (MDT) Date: Tue, 16 Jun 2009 13:27:35 -0400 From: Sebastien Roy Subject: Re: IP Datapath Refactoring [PSARC/2009/331 FastTrack timeout 06/09/2009] In-reply-to: <1243950219.6636.12.camel@strat> Sender: Sebastien.Roy@Sun.COM To: psarc-ext Cc: "Erik.Nordmark" , Sowmini Varadhan Message-id: <1245173255.22737.26.camel@strat> Organization: Sun Microsystems X-Mailer: Evolution 2.24.2 References: <1243950219.6636.12.camel@strat> Status: RO Content-Length: 99 The timer having expired, and all issues having been addressed, this case is now approved. -Seb