From sacadmin Mon Apr 13 09:31:29 2009 Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3DGVT5a021698; Mon, 13 Apr 2009 09:31:29 -0700 (PDT) Received: (from carlsonj@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id n3DGVT9W021690; Mon, 13 Apr 2009 09:31:29 -0700 (PDT) Date: Mon, 13 Apr 2009 09:31:29 -0700 (PDT) From: James Carlson Message-Id: <200904131631.n3DGVT9W021690@sac.sfbay.sun.com> To: PSARC-record@sac.sfbay.sun.com Subject: dladm Possible Values List [PSARC/2009/235 FastTrack timeout 04/20/2009] Status: RO Content-Length: 565 Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: dladm Possible Values List 1.2. Name of Document Author/Supplier: Author: Girish Moodalbail 1.3 Date of This Document: 13 April, 2009 4. Technical Description See the case directory for more detail 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From carlsonj@phorcys.east.sun.com Mon Apr 13 09:44:17 2009 Received: from dm-east-02.east.sun.com (dm-east-02.East.Sun.COM [129.148.13.5]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3DGiGcG021940 for ; Mon, 13 Apr 2009 09:44:16 -0700 (PDT) Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by dm-east-02.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3DGiFWL061360; Mon, 13 Apr 2009 12:44:15 -0400 (EDT) Received: from phorcys.east.sun.com (phorcys.local [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n3DGhluK010772; Mon, 13 Apr 2009 12:43:47 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n3DGhkwx010769; Mon, 13 Apr 2009 12:43:47 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18915.27586.986715.589201@gargle.gargle.HOWL> Date: Mon, 13 Apr 2009 12:43:46 -0400 From: James Carlson To: psarc-ext@sac.sfbay.sun.com cc: "Girish M. G." Subject: 2009/235 dladm Possible Values List X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 4732 I'm sponsoring this fast-track request for Girish Moodalbail. The timer is set to 04/20/2009. The consolidation is "ON" and the stability level for the interfaces being introduced is "Consolidation Private". Introduction: ------------- dladm(1m) does not display possible values for some properties like MTU. For a data-link which supports setting MTU on it, there exists a range of values (for e.g., 108-9000) that can be set as MTU. Today in ON we do not display that range (see CR 6680929) in 'dladm show-linkprop' output. MTU happens to be one such property and there could be many. To display such range of values we need to contact driver or MAC layer (based on the property being queried). However there isn't any structure to represent (a) range of values (read: integers, unsigned integers, et al) or collection of strings (b) the type of the values returned and (c) count of the range itself. We want to provide one such structure and make it part of the linkprop framework itself, so that it will be useful for implementers to send in range of values from userspace to kernel or from kernel/driver to user space. This data structure would be defined in 'mac.h' and would be used as a value-result parameter (see below for more details). Interfaces being added: --------------------------- (a) Set of data structures to represent range of values (int32, int64, uint32, uint64, et al) or collection/enumeration of values (strings). For now we define uint32 range (mac_propval_uint32_range_t) for MTU. The framework can be extended for other types. /* * Defines range of uint32 values. (108-9000) */ typedef struct mac_propval_uint32_range_s { uint32_t mpur_min; /* minimum value in the range */ uint32_t mpur_max; /* maximum value in the range */ } mac_propval_uint32_range_t; /* * Data type of the value. */ typedef enum { MAC_PROPVAL_UINT32 = 0x1 /* Various other types could be * MAC_PROPVAL_INT32 * MAC_PROPVAL_INT64 * MAC_PROPVAL_UINT64 * MAC_PROPVAL_STRING */ } mac_propval_type_t; /* * Captures range of values for a given property. For e.g., * - MTU property for a data link can have values between 68-9000. * - CPU property for a MAC client can have multiple CPU ranges * 1-6, 10-16 assigned to it. */ typedef struct mac_propval_range_s { uint_t mpr_count; /* count of ranges */ mac_propval_type_t mpr_type; /* type of value */ union { mac_propval_uint32_range_t mpr_uint32[1]; } u; } mac_propval_range_t; #define range_uint32 u.mpr_uint32 (b) MAC_PROP_POSSIBLE: In order to avoid explosion of MAC properties (read: MAC_PROP_YYY, MAC_PROP_YYY_RANGE, et al), we chose to use a new flag 'MAC_PROP_POSSIBLE' to tell the drivers that the linkprop framework is interested in the range of property values. Device driver authors should handle the above flag by populating the aforementioned data structures and returning them. This flag is in the same league as MAC_PROP_DEFAULT, which is used to tell the drivers to return default property value. NOTE: MAC_PROP_POSSIBLE flag needs to be handled 'carefully'. MAC_PROP_YYY by itself means current value of property, however MAC_PROP_YYY with MAC_PROP_POSSIBLE flag means range of property values. So when MAC_PROP_POSSIBLE flag is used for a particular property, one may have to change "all" the drivers to handle this flag for that property (i.e., either return 'ENOTSUP' when not supported or return 'range of values'). Interface interaction details: ------------------------------- The caller is responsible to allocate sufficient memory for the drivers to fill in the details. The caller provides a pointer to an array (union member 'u' of 'mac_propval_range_t') and a size of the array ('mpr_count' of the 'mac_propval_range_t') and the callee will copy the data into that array. Upon success, mpr_count will have count of range of values and the array itself will have value range. However if sufficient memory was not allocated, the driver would return ENOBUF and would set 'mpr_count' to the required size of the array. The query/reply mechanism is achieved by calling i_dladm_range_get() for query and the respective driver function _m_getprop() would reply back with the range of values. i_dladm_range_get() is a function available in libdladm.so.1, which needs to be registered as a callback function in prop_desc_t[] table for those properties interested in range of values. Following drivers would be changed to use the above structure: bge, e1000g, nxge, nge, hxge, vnic & aggr. All the changes would be in the respective _m_getprop() function. From gdamore@sun.com Mon Apr 13 11:09:22 2009 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3DI9MUd025199 for ; Mon, 13 Apr 2009 11:09:22 -0700 (PDT) Received: from sca-es-mail-1.sun.com (sca-es-mail-1.Sun.COM [192.18.43.132]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3DI9MMC034458 for ; Mon, 13 Apr 2009 11:09:22 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n3DI9Hms023042 for ; Mon, 13 Apr 2009 11:09:17 -0700 (PDT) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; charset=ISO-8859-1 Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KI100K00VKKJ300@fe-sfbay-09.sun.com> for psarc-ext@sac.sfbay.sun.com; Mon, 13 Apr 2009 11:09:17 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KI100BNYX2X0V50@fe-sfbay-09.sun.com> for psarc-ext@sac.sfbay.sun.com; Mon, 13 Apr 2009 11:08:59 -0700 (PDT) Date: Mon, 13 Apr 2009 11:08:57 -0700 From: "Garrett D'Amore" Subject: Re: 2009/235 dladm Possible Values List In-reply-to: <18915.27586.986715.589201@gargle.gargle.HOWL> Sender: Garrett.Damore@sun.com To: James Carlson Cc: psarc-ext@sac.sfbay.sun.com, "Girish M. G." Message-id: <49E37FB9.3070202@sun.com> References: <18915.27586.986715.589201@gargle.gargle.HOWL> User-Agent: Thunderbird 2.0.0.18 (X11/20081201) Status: RO Content-Length: 5173 James Carlson wrote: > I'm sponsoring this fast-track request for Girish Moodalbail. The > timer is set to 04/20/2009. > > > The consolidation is "ON" and the stability level for the interfaces > being introduced is "Consolidation Private". > > Introduction: > ------------- > > dladm(1m) does not display possible values for some properties like MTU. > For a data-link which supports setting MTU on it, there exists a range of > values (for e.g., 108-9000) that can be set as MTU. Today in ON we do > not display that range (see CR 6680929) in 'dladm show-linkprop' output. > MTU happens to be one such property and there could be many. > > To display such range of values we need to contact driver or MAC layer > (based on the property being queried). However there isn't any structure > to represent > (a) range of values (read: integers, unsigned integers, et al) or > collection of strings > (b) the type of the values returned and > (c) count of the range itself. > > We want to provide one such structure and make it part of the linkprop > framework itself, so that it will be useful for implementers to send in > range of values from userspace to kernel or from kernel/driver to user > space. This data structure would be defined in 'mac.h' and would be used > as a value-result parameter (see below for more details). > > Interfaces being added: > --------------------------- > > (a) Set of data structures to represent range of values (int32, int64, > uint32, uint64, et al) or collection/enumeration of values (strings). > For now we define uint32 range (mac_propval_uint32_range_t) for MTU. The > framework can be extended for other types. > > /* > * Defines range of uint32 values. (108-9000) > */ > typedef struct mac_propval_uint32_range_s { > uint32_t mpur_min; /* minimum value in the range */ > uint32_t mpur_max; /* maximum value in the range */ > } mac_propval_uint32_range_t; > > /* > * Data type of the value. > */ > typedef enum { > MAC_PROPVAL_UINT32 = 0x1 > > /* Various other types could be > * MAC_PROPVAL_INT32 > * MAC_PROPVAL_INT64 > * MAC_PROPVAL_UINT64 > * MAC_PROPVAL_STRING > */ > } mac_propval_type_t; > > /* > * Captures range of values for a given property. For e.g., > * - MTU property for a data link can have values between 68-9000. > * - CPU property for a MAC client can have multiple CPU ranges > * 1-6, 10-16 assigned to it. > */ > typedef struct mac_propval_range_s { > uint_t mpr_count; /* count of ranges */ > mac_propval_type_t mpr_type; /* type of value */ > union { > mac_propval_uint32_range_t mpr_uint32[1]; > } u; > } mac_propval_range_t; > > #define range_uint32 u.mpr_uint32 > > (b) MAC_PROP_POSSIBLE: > In order to avoid explosion of MAC properties (read: MAC_PROP_YYY, > MAC_PROP_YYY_RANGE, et al), we chose to use a new flag > 'MAC_PROP_POSSIBLE' to tell the drivers that the linkprop framework is > interested in the range of property values. Device driver authors should > handle the above flag by populating the aforementioned data structures > and returning them. > This flag is in the same league as MAC_PROP_DEFAULT, which is used to > tell the drivers to return default property value. > > NOTE: MAC_PROP_POSSIBLE flag needs to be handled 'carefully'. > MAC_PROP_YYY by itself means current value of property, however > MAC_PROP_YYY with MAC_PROP_POSSIBLE flag means range of property values. > So when MAC_PROP_POSSIBLE flag is used for a particular property, one > may have to change "all" the drivers to handle this flag for that > property (i.e., either return 'ENOTSUP' when not supported or return > 'range of values'). > > Interface interaction details: > ------------------------------- > > The caller is responsible to allocate sufficient memory for the drivers > to fill in the details. The caller provides a pointer to an array (union > member 'u' of 'mac_propval_range_t') and a size of the array > ('mpr_count' of the 'mac_propval_range_t') and the callee will copy the > data into that array. Upon success, mpr_count will have count of range > of values and the array itself will have value range. However if > sufficient memory was not allocated, the driver would return ENOBUF and > would set 'mpr_count' to the required size of the array. > > The query/reply mechanism is achieved by calling i_dladm_range_get() for > query and the respective driver function _m_getprop() would reply > back with the range of values. > > i_dladm_range_get() is a function available in libdladm.so.1, which > needs to be registered as a callback function in prop_desc_t[] table for > those properties interested in range of values. > > Following drivers would be changed to use the above structure: > bge, e1000g, nxge, nge, hxge, vnic & aggr. > All the changes would be in the respective _m_getprop() function. > There are other drivers which use the Brussels interfaces (e.g. afe & mxfe). What happens to those drivers that don't support this "possible" interface? What about the bnx driver? -- Garrett From carlsonj@phorcys.east.sun.com Mon Apr 13 11:16:31 2009 Received: from dm-east-01.east.sun.com (dm-east-01.East.Sun.COM [129.148.9.192]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3DIGVpI025386 for ; Mon, 13 Apr 2009 11:16:31 -0700 (PDT) Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by dm-east-01.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3DIGUaO050461; Mon, 13 Apr 2009 14:16:30 -0400 (EDT) Received: from phorcys.east.sun.com (phorcys.local [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n3DIG1Bc011189; Mon, 13 Apr 2009 14:16:01 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n3DIG1I7011186; Mon, 13 Apr 2009 14:16:01 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18915.33121.838477.635322@gargle.gargle.HOWL> Date: Mon, 13 Apr 2009 14:16:01 -0400 From: James Carlson To: "Garrett D'Amore" Cc: psarc-ext@sac.sfbay.sun.com, "Girish M. G." Subject: Re: 2009/235 dladm Possible Values List In-Reply-To: <49E37FB9.3070202@sun.com> References: <18915.27586.986715.589201@gargle.gargle.HOWL> <49E37FB9.3070202@sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 933 Garrett D'Amore writes: > James Carlson wrote: > > Following drivers would be changed to use the above structure: > > bge, e1000g, nxge, nge, hxge, vnic & aggr. > > All the changes would be in the respective _m_getprop() function. > > > There are other drivers which use the Brussels interfaces (e.g. afe & > mxfe). What happens to those drivers that don't support this "possible" > interface? What about the bnx driver? Nothing special happens. They don't have MTU as one of the _m_getprop properties, because MTU is currently cached within the MAC layer itself. Thus, when the MAC layer calls that entry point, it'll get back an error, and will have to return just the current MTU. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From Girish.Moodalbail@Sun.COM Mon Apr 13 11:18:54 2009 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3DIIsmg025400 for ; Mon, 13 Apr 2009 11:18:54 -0700 (PDT) Received: from brmea-mail-2.sun.com (brmea-mail-2.Sun.COM [192.18.98.43]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3DIIsLi051956 for ; Mon, 13 Apr 2009 11:18:54 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n3DIIsTl011068 for ; Mon, 13 Apr 2009 18:18:54 GMT MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; charset=ISO-8859-1 Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KI100L00TLIGU00@mail-amer.sun.com> for psarc-ext@sac.sfbay.sun.com; Mon, 13 Apr 2009 12:18:54 -0600 (MDT) Received: from [192.168.0.106] (108.203.171.66.subscriber.vzavenue.net [66.171.203.108]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KI1008OZXIY3J70@mail-amer.sun.com> for psarc-ext@sac.sfbay.sun.com; Mon, 13 Apr 2009 12:18:35 -0600 (MDT) Date: Mon, 13 Apr 2009 14:18:32 -0400 From: Girish Moodalbail Subject: Re: 2009/235 dladm Possible Values List In-reply-to: <49E37FB9.3070202@sun.com> Sender: Girish.Moodalbail@Sun.COM To: "Garrett D'Amore" Cc: James Carlson , psarc-ext@sac.sfbay.sun.com Message-id: <49E381F8.7000405@sun.com> References: <18915.27586.986715.589201@gargle.gargle.HOWL> <49E37FB9.3070202@sun.com> User-Agent: Thunderbird 2.0.0.17 (X11/20081023) Status: RO Content-Length: 927 >> >> Following drivers would be changed to use the above structure: >> bge, e1000g, nxge, nge, hxge, vnic & aggr. >> All the changes would be in the respective _m_getprop() function. >> > There are other drivers which use the Brussels interfaces (e.g. afe & > mxfe). What happens to those drivers that don't support this "possible" > interface? What about the bnx driver? For such drivers, which do not support "possible" values for the MTU property, the _m_getprop() would return 'ENOTSUP' and we would display the current value of the MTU (as cached by MAC layer) in the output. bash-3.2$ dladm show-linkprop -p mtu mxfe0 LINK PROPERTY VALUE DEFAULT POSSIBLE mxfe0 mtu 1500 1500 1500 The meaning of 'possible' value, from dladm(1m) man page, is 'list of values the property can have' and in this case it will be 1500. ~Girish From carlsonj@phorcys.east.sun.com Mon Apr 13 11:29:29 2009 Received: from dm-east-01.east.sun.com (dm-east-01.East.Sun.COM [129.148.9.192]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3DITS6F025671 for ; Mon, 13 Apr 2009 11:29:28 -0700 (PDT) Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by dm-east-01.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3DITR5Y057169; Mon, 13 Apr 2009 14:29:27 -0400 (EDT) Received: from phorcys.east.sun.com (phorcys.local [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n3DISxwk011261; Mon, 13 Apr 2009 14:28:59 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n3DISxkj011258; Mon, 13 Apr 2009 14:28:59 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18915.33899.347299.614439@gargle.gargle.HOWL> Date: Mon, 13 Apr 2009 14:28:59 -0400 From: James Carlson To: Girish Moodalbail Cc: "Garrett D'Amore" , psarc-ext@sac.sfbay.sun.com Subject: Re: 2009/235 dladm Possible Values List In-Reply-To: <49E381F8.7000405@sun.com> References: <18915.27586.986715.589201@gargle.gargle.HOWL> <49E37FB9.3070202@sun.com> <49E381F8.7000405@sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 1533 Girish Moodalbail writes: > For such drivers, which do not support "possible" values for the MTU > property, the _m_getprop() would return 'ENOTSUP' and we would > display the current value of the MTU (as cached by MAC layer) in the output. > > bash-3.2$ dladm show-linkprop -p mtu mxfe0 > LINK PROPERTY VALUE DEFAULT POSSIBLE > mxfe0 mtu 1500 1500 1500 > > The meaning of 'possible' value, from dladm(1m) man page, is 'list of > values the property can have' and in this case it will be 1500. One of the subtleties of this project is this section: NOTE: MAC_PROP_POSSIBLE flag needs to be handled 'carefully'. The reason this project doesn't have to modify every GLDv3 driver is that the property in question is not one that drivers actually return. Everyone who has a settable MTU has _m_setprop set up to handle MAC_PROP_MTU, but nobody has _m_getprop for this, because MTU is special-cased in mac_get_prop(). If it were otherwise -- that is, if this were a "normal" property and not special-cased in mac_get_prop -- then the proposed design would require changing all the drivers. The next person who wants to extend this (using MAC_PROP_POSSIBLE for some other property) may well have to bite that bullet. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From gdamore@sun.com Mon Apr 13 11:55:07 2009 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3DIt6aY026267 for ; Mon, 13 Apr 2009 11:55:06 -0700 (PDT) Received: from sca-es-mail-2.sun.com (sca-es-mail-2.Sun.COM [192.18.43.133]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3DIt6D0002055 for ; Mon, 13 Apr 2009 11:55:06 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n3DIt1BE013222 for ; Mon, 13 Apr 2009 11:55:01 -0700 (PDT) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; charset=ISO-8859-1 Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KI100900Z2W2Z00@fe-sfbay-10.sun.com> for psarc-ext@sac.sfbay.sun.com; Mon, 13 Apr 2009 11:55:01 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KI1001P7Z7LQT30@fe-sfbay-10.sun.com> for psarc-ext@sac.sfbay.sun.com; Mon, 13 Apr 2009 11:54:57 -0700 (PDT) Date: Mon, 13 Apr 2009 11:54:57 -0700 From: "Garrett D'Amore" Subject: Re: 2009/235 dladm Possible Values List In-reply-to: <18915.33899.347299.614439@gargle.gargle.HOWL> Sender: Garrett.Damore@sun.com To: James Carlson Cc: Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Message-id: <49E38A81.7010007@sun.com> References: <18915.27586.986715.589201@gargle.gargle.HOWL> <49E37FB9.3070202@sun.com> <49E381F8.7000405@sun.com> <18915.33899.347299.614439@gargle.gargle.HOWL> User-Agent: Thunderbird 2.0.0.18 (X11/20081201) Status: RO Content-Length: 1687 James Carlson wrote: > Girish Moodalbail writes: > >> For such drivers, which do not support "possible" values for the MTU >> property, the _m_getprop() would return 'ENOTSUP' and we would >> display the current value of the MTU (as cached by MAC layer) in the output. >> >> bash-3.2$ dladm show-linkprop -p mtu mxfe0 >> LINK PROPERTY VALUE DEFAULT POSSIBLE >> mxfe0 mtu 1500 1500 1500 >> >> The meaning of 'possible' value, from dladm(1m) man page, is 'list of >> values the property can have' and in this case it will be 1500. >> > > One of the subtleties of this project is this section: > > NOTE: MAC_PROP_POSSIBLE flag needs to be handled 'carefully'. > > The reason this project doesn't have to modify every GLDv3 driver is > that the property in question is not one that drivers actually > return. Everyone who has a settable MTU has _m_setprop set up > to handle MAC_PROP_MTU, but nobody has _m_getprop for this, > because MTU is special-cased in mac_get_prop(). > > If it were otherwise -- that is, if this were a "normal" property and > not special-cased in mac_get_prop -- then the proposed design would > require changing all the drivers. > > The next person who wants to extend this (using MAC_PROP_POSSIBLE for > some other property) may well have to bite that bullet. > > Okay, understanding all this, +1 to this project. From an implementation/ codereview standpoint, I'd like to see a great big comment in the header file (and in any actual documentation) describing this caveat. (Probably at the point that MAC_PROP_MTU is discussed.) Thanks. -- Garrett From Girish.Moodalbail@Sun.COM Mon Apr 13 12:48:51 2009 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3DJmpJn022966 for ; Mon, 13 Apr 2009 12:48:51 -0700 (PDT) Received: from brmea-mail-4.sun.com (brmea-mail-4.Sun.COM [192.18.98.36]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3DJmoRx040101 for ; Mon, 13 Apr 2009 12:48:51 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n3DJmobt005774 for ; Mon, 13 Apr 2009 19:48:50 GMT MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; charset=ISO-8859-1 Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KI200M001LE9G00@mail-amer.sun.com> for psarc-ext@sac.sfbay.sun.com; Mon, 13 Apr 2009 13:48:50 -0600 (MDT) Received: from [192.168.0.106] (108.203.171.66.subscriber.vzavenue.net [66.171.203.108]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KI20086G1P23JD0@mail-amer.sun.com> for psarc-ext@sac.sfbay.sun.com; Mon, 13 Apr 2009 13:48:39 -0600 (MDT) Date: Mon, 13 Apr 2009 15:48:38 -0400 From: Girish Moodalbail Subject: Re: 2009/235 dladm Possible Values List In-reply-to: <49E38A81.7010007@sun.com> Sender: Girish.Moodalbail@Sun.COM To: "Garrett D'Amore" Cc: James Carlson , psarc-ext@sac.sfbay.sun.com Message-id: <49E39716.7080100@sun.com> References: <18915.27586.986715.589201@gargle.gargle.HOWL> <49E37FB9.3070202@sun.com> <49E381F8.7000405@sun.com> <18915.33899.347299.614439@gargle.gargle.HOWL> <49E38A81.7010007@sun.com> User-Agent: Thunderbird 2.0.0.17 (X11/20081023) Status: RO Content-Length: 455 >> The next person who wants to extend this (using MAC_PROP_POSSIBLE for >> some other property) may well have to bite that bullet. >> >> > Okay, understanding all this, +1 to this project. > > From an implementation/ codereview standpoint, I'd like to see a great > big comment in the header file (and in any actual documentation) > describing this caveat. (Probably at the point that MAC_PROP_MTU is > discussed.) Sure thing. thanks ~Girish From mws@zion.sfbay.sun.com Tue Apr 14 09:47:31 2009 Received: from zion.sfbay.sun.com (zion.SFBay.Sun.COM [129.146.17.75]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EGlV3v005531 for ; Tue, 14 Apr 2009 09:47:31 -0700 (PDT) Received: from zion.sfbay.sun.com (localhost [127.0.0.1]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id n3EGlVMM007435; Tue, 14 Apr 2009 16:47:31 GMT Received: (from mws@localhost) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2/Submit) id n3EGlVGG007434; Tue, 14 Apr 2009 09:47:31 -0700 (PDT) From: Michael Shapiro Message-Id: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> Subject: Re: 2009/235 dladm Possible Values List In-Reply-To: <49E39716.7080100@sun.com> from Girish Moodalbail at "Apr 13, 2009 03:48:38 pm" To: Girish.Moodalbail@sun.com (Girish Moodalbail) Date: Tue, 14 Apr 2009 09:47:30 -0700 (PDT) Cc: gdamore@sun.com, James.D.Carlson@sun.com, psarc-ext@sac.sfbay.sun.com X-Mailer: ELM [version 2.4ME+ PL31H (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Status: RO Content-Length: 880 This case currently does not address the issue of how an administrator or layered software determines the optimal large MTU, as opposed to the maximum MTU. The two are not always the same. For example, on Neptune (nxge), the maximum is 9000, but the optimal large MTU is 8150, because of the size of the DMA transfers the card does in hardware. This case needs to address this issue explicitly, by either: (a) Defining an additional interface by which the optimal value can be returned from the driver as another attribute, OR (b) Making extremely clear to driver writers that if a large MTU size less than the maximum is more optimal than the maximum, that the optimal size should be returned by this interface. My preference is for option (a), but others should weigh in. -Mike -- Mike Shapiro, Sun Microsystems Open Storage / Fishworks. blogs.sun.com/mws/ From gdamore@Sun.COM Tue Apr 14 10:09:17 2009 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EH9GBB007854 for ; Tue, 14 Apr 2009 10:09:16 -0700 (PDT) Received: from sca-es-mail-2.sun.com (sca-es-mail-2.Sun.COM [192.18.43.133]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3EH9G1N016802 for ; Tue, 14 Apr 2009 10:09:16 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n3EH9BOA006979 for ; Tue, 14 Apr 2009 10:09:11 -0700 (PDT) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; charset=ISO-8859-1 Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KI300N00NM42200@fe-sfbay-10.sun.com> for psarc-ext@sac.sfbay.sun.com; Tue, 14 Apr 2009 10:09:11 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KI300LXNOYYC370@fe-sfbay-10.sun.com>; Tue, 14 Apr 2009 10:08:58 -0700 (PDT) Date: Tue, 14 Apr 2009 10:08:58 -0700 From: "Garrett D'Amore" Subject: Re: 2009/235 dladm Possible Values List In-reply-to: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> Sender: Garrett.Damore@Sun.COM To: Michael Shapiro Cc: Girish Moodalbail , James.D.Carlson@Sun.COM, psarc-ext@sac.sfbay.sun.com Message-id: <49E4C32A.5000705@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.18 (X11/20081201) Status: RO Content-Length: 1839 Michael Shapiro wrote: > This case currently does not address the issue of how an administrator > or layered software determines the optimal large MTU, as opposed to > the maximum MTU. The two are not always the same. For example, on > Neptune (nxge), the maximum is 9000, but the optimal large MTU is 8150, > because of the size of the DMA transfers the card does in hardware. > This case needs to address this issue explicitly, by either: > > (a) Defining an additional interface by which the optimal value > can be returned from the driver as another attribute, OR > > (b) Making extremely clear to driver writers that if a large MTU > size less than the maximum is more optimal than the maximum, > that the optimal size should be returned by this interface. > > My preference is for option (a), but others should weigh in. > > -Mike > > This feels like a "hardware tuning" element. MTU configuration generally shouldn't need to worry about page sizes and such. An extra DMA transfer is usually in the "noise" as far as overheads of network processing are concerned. More specifically, large MTUs are intended, as I see it, to minimize the effect of per-packet overheads found in NIC hardware, switches, routers, and most especially *hosts*. (I.e. the TCP/IP stack overheads.) I suspect that because of these overheads, that in general the largest MTU you can configure is always "optimal", even if the hardware has to perform some extra DMA transfers to make them happen. Unless some specific real world tests (e.g. TCP throughput or UDP stress tests) show otherwise, I'm disinclined to believe that there is any reason an administrator would need to know about the underlying hardware DMA limitations, or that the "optimum" value is anything other than the largest supported value. -- Garrett From mws@zion.sfbay.sun.com Tue Apr 14 10:14:24 2009 Received: from zion.sfbay.sun.com (zion.SFBay.Sun.COM [129.146.17.75]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EHENi6008308 for ; Tue, 14 Apr 2009 10:14:23 -0700 (PDT) Received: from zion.sfbay.sun.com (localhost [127.0.0.1]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id n3EHEIEn008299; Tue, 14 Apr 2009 17:14:18 GMT Received: (from mws@localhost) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2/Submit) id n3EHEHVe008298; Tue, 14 Apr 2009 10:14:17 -0700 (PDT) Date: Tue, 14 Apr 2009 10:14:17 -0700 From: Mike Shapiro To: "Garrett D'Amore" Cc: Michael Shapiro , Girish Moodalbail , James.D.Carlson@sun.com, psarc-ext@sac.sfbay.sun.com Subject: Re: 2009/235 dladm Possible Values List Message-ID: <20090414171417.GA8124@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <49E4C32A.5000705@sun.com> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 3106 On Tue, Apr 14, 2009 at 10:08:58AM -0700, Garrett D'Amore wrote: > Michael Shapiro wrote: > >This case currently does not address the issue of how an administrator > >or layered software determines the optimal large MTU, as opposed to > >the maximum MTU. The two are not always the same. For example, on > >Neptune (nxge), the maximum is 9000, but the optimal large MTU is 8150, > >because of the size of the DMA transfers the card does in hardware. > >This case needs to address this issue explicitly, by either: > > > >(a) Defining an additional interface by which the optimal value > > can be returned from the driver as another attribute, OR > > > >(b) Making extremely clear to driver writers that if a large MTU > > size less than the maximum is more optimal than the maximum, > > that the optimal size should be returned by this interface. > > > >My preference is for option (a), but others should weigh in. > > > >-Mike > > > > > This feels like a "hardware tuning" element. That's because it is a hardware tuning element :) > MTU configuration generally shouldn't need to worry about page sizes and > such. An extra DMA transfer is usually in the "noise" as far as > overheads of network processing are concerned. Not if you're trying to maximize performance of a heavily loaded server, which is precisely when this matters most. > More specifically, large MTUs are intended, as I see it, to minimize the > effect of per-packet overheads found in NIC hardware, switches, routers, > and most especially *hosts*. (I.e. the TCP/IP stack overheads.) I > suspect that because of these overheads, that in general the largest MTU > you can configure is always "optimal", even if the hardware has to > perform some extra DMA transfers to make them happen. > > Unless some specific real world tests (e.g. TCP throughput or UDP stress > tests) show otherwise, I'm disinclined to believe that there is any > reason an administrator would need to know about the underlying hardware > DMA limitations, or that the "optimum" value is anything other than the > largest supported value. > > -- Garrett It makes a rather large difference in the absolute performance numbers we achieve on the 7000 series. This is because you have a system which is executing 16 cores at 100% cpu bound when it's fully loaded and therefore cutting down on transfers and extraneous packet processing makes a rather huge difference. Here's the actual point: the knowledge of this optimal size is a function of the hardware and the driver. Our customers spend a lot of time complaining about the fact that the out-of-the-box tuning of networking on Solaris is sub-optimal and hard to understand how to do better. Jumbo MTU isn't an area we can easily enable by default, but we can make it a lot easier to achieve the fastest setting. That knowledge belongs in the driver source, not on some out-of-date wiki page where everyone has to waste more time googling around and experimenting to figure it out. -Mike -- Mike Shapiro, Sun Microsystems Open Storage / Fishworks. blogs.sun.com/mws/ From carlsonj@phorcys.east.sun.com Tue Apr 14 10:19:17 2009 Received: from dm-east-01.east.sun.com (dm-east-01.East.Sun.COM [129.148.9.192]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EHJGoe008615 for ; Tue, 14 Apr 2009 10:19:17 -0700 (PDT) Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by dm-east-01.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3EHJGe7011268; Tue, 14 Apr 2009 13:19:16 -0400 (EDT) Received: from phorcys.east.sun.com (phorcys.local [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n3EHIlkS015840; Tue, 14 Apr 2009 13:18:47 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n3EHIlpH015837; Tue, 14 Apr 2009 13:18:47 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18916.50551.493349.69795@gargle.gargle.HOWL> Date: Tue, 14 Apr 2009 13:18:47 -0400 From: James Carlson To: "Garrett D'Amore" Cc: Michael Shapiro , Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Subject: Re: 2009/235 dladm Possible Values List In-Reply-To: <49E4C32A.5000705@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 1913 Garrett D'Amore writes: > Unless some specific real world tests (e.g. TCP throughput or UDP stress > tests) show otherwise, I'm disinclined to believe that there is any > reason an administrator would need to know about the underlying hardware > DMA limitations, or that the "optimum" value is anything other than the > largest supported value. Even then, I suspect that this may be more of a man page or whitepaper issue. On a given L2 network, all of the links must be configured with *exactly* the same MTU; neither larger nor smaller. There's no good way (at least today) to tell whether all of the systems are actually configured that way. You just have to hope so, because the penalty for getting it wrong is a set of very obscure failure modes. That's part of the reason why the IEEE hasn't endorsed jumbograms -- they're fragile. The implication of that network-wide configuration issue is that the administrator must choose an MTU value based not just on local hardware considerations, but on the capabilities and configuration of all of the other nodes on his L2 network. That's hard, and it's certainly not as simple as a single "preferred" value (unless that preferred value happens to be exactly 1500, the standards-conforming and safe default). In fact, I'd expect that there are devices that have a set of multiple "sweet spot" MTU values where buffers and other detailed components all happen to line up perfectly. Getting all of that information into a user interface seems hopeless to me, so I'm in favor of having the dladm command line display what the hardware can do, and leaving high-level network design and tuning to other documentation and/or tools. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From carlsonj@phorcys.east.sun.com Tue Apr 14 10:25:35 2009 Received: from dm-east-02.east.sun.com (dm-east-02.East.Sun.COM [129.148.13.5]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EHPYH5009293 for ; Tue, 14 Apr 2009 10:25:34 -0700 (PDT) Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by dm-east-02.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3EHPXmN013269; Tue, 14 Apr 2009 13:25:33 -0400 (EDT) Received: from phorcys.east.sun.com (phorcys.local [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n3EHP5Cb015867; Tue, 14 Apr 2009 13:25:05 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n3EHP572015864; Tue, 14 Apr 2009 13:25:05 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18916.50929.207453.829796@gargle.gargle.HOWL> Date: Tue, 14 Apr 2009 13:25:05 -0400 From: James Carlson To: Mike Shapiro Cc: "Garrett D'Amore" , Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Subject: Re: 2009/235 dladm Possible Values List In-Reply-To: <20090414171417.GA8124@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <20090414171417.GA8124@sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 1013 Mike Shapiro writes: > better. Jumbo MTU isn't an area we can easily enable by default, > but we can make it a lot easier to achieve the fastest setting. > That knowledge belongs in the driver source, not on some out-of-date > wiki page where everyone has to waste more time googling around > and experimenting to figure it out. If given an interface named "foo0", I would expect a user needing detailed information (particularly for performance tuning) would type "man foo" first, rather than roaming around the Internet looking for clues. Perhaps an RFE for displaying driver-related performance tuning information (not just MTU, but use of VLANs and number of local and multicast MAC addresses and so on) would be useful, but I'm not convinced that it's really this project's issue. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From mws@zion.sfbay.sun.com Tue Apr 14 10:30:10 2009 Received: from zion.sfbay.sun.com (zion.SFBay.Sun.COM [129.146.17.75]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EHUARI009381 for ; Tue, 14 Apr 2009 10:30:10 -0700 (PDT) Received: from zion.sfbay.sun.com (localhost [127.0.0.1]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id n3EHU5Y9008744; Tue, 14 Apr 2009 17:30:05 GMT Received: (from mws@localhost) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2/Submit) id n3EHU4E2008743; Tue, 14 Apr 2009 10:30:04 -0700 (PDT) Date: Tue, 14 Apr 2009 10:30:04 -0700 From: Mike Shapiro To: James Carlson Cc: "Garrett D'Amore" , Michael Shapiro , Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Subject: Re: 2009/235 dladm Possible Values List Message-ID: <20090414173004.GA8350@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <18916.50551.493349.69795@gargle.gargle.HOWL> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <18916.50551.493349.69795@gargle.gargle.HOWL> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 3065 On Tue, Apr 14, 2009 at 01:18:47PM -0400, James Carlson wrote: > Garrett D'Amore writes: > > Unless some specific real world tests (e.g. TCP throughput or UDP stress > > tests) show otherwise, I'm disinclined to believe that there is any > > reason an administrator would need to know about the underlying hardware > > DMA limitations, or that the "optimum" value is anything other than the > > largest supported value. > > Even then, I suspect that this may be more of a man page or whitepaper > issue. > > On a given L2 network, all of the links must be configured with > *exactly* the same MTU; neither larger nor smaller. There's no good > way (at least today) to tell whether all of the systems are actually > configured that way. You just have to hope so, because the penalty > for getting it wrong is a set of very obscure failure modes. That's > part of the reason why the IEEE hasn't endorsed jumbograms -- they're > fragile. > > The implication of that network-wide configuration issue is that the > administrator must choose an MTU value based not just on local > hardware considerations, but on the capabilities and configuration of > all of the other nodes on his L2 network. > > That's hard, and it's certainly not as simple as a single "preferred" > value (unless that preferred value happens to be exactly 1500, the > standards-conforming and safe default). In fact, I'd expect that > there are devices that have a set of multiple "sweet spot" MTU values > where buffers and other detailed components all happen to line up > perfectly. > > Getting all of that information into a user interface seems hopeless > to me, so I'm in favor of having the dladm command line display what > the hardware can do, and leaving high-level network design and tuning > to other documentation and/or tools. I agree with the above, but you're describing the rest of the jumbo issue beyond what I'm getting at. There are two completely different questions: (a) What is the jumbo size I ought to set on this L2 network given what is configured elsewhere on the network? (b) What is the jumbo size I ought to use with this card for optimal large MTU performance on the card? You're talking about (a), I'm talking about (b). Indeed an overall deployment may need to consider both, but that isn't a reason to decree this is all hopeless and therefore stick to the nothing we do now. This is a significant issue for server and storage administrators, as is generally all issues of making Solaris networking perform better, and it's not effective to continue solving every problem with documentation. As I said, the optimal large transfer size is a function of the hardware and the driver -- there is no earthly reason that can't become an r/o driver property of some sort so as the h/w, firmware, or driver evolve, our users aren't sitting around looking at some out-of-date wiki or have some out-of-date config script, which is what they all have today. -Mike -- Mike Shapiro, Sun Microsystems Open Storage / Fishworks. blogs.sun.com/mws/ From gdamore@sun.com Tue Apr 14 10:35:28 2009 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EHZS9p009486 for ; Tue, 14 Apr 2009 10:35:28 -0700 (PDT) Received: from sca-es-mail-2.sun.com (sca-es-mail-2.Sun.COM [192.18.43.133]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3EHZSDx005304 for ; Tue, 14 Apr 2009 10:35:28 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n3EHZNi3010728 for ; Tue, 14 Apr 2009 10:35:23 -0700 (PDT) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; charset=ISO-8859-1 Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KI300600P2XM400@fe-sfbay-10.sun.com> for psarc-ext@sac.sfbay.sun.com; Tue, 14 Apr 2009 10:35:23 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KI300CKCQ6GX520@fe-sfbay-10.sun.com>; Tue, 14 Apr 2009 10:35:05 -0700 (PDT) Date: Tue, 14 Apr 2009 10:35:04 -0700 From: "Garrett D'Amore" Subject: Re: 2009/235 dladm Possible Values List In-reply-to: <20090414171417.GA8124@sun.com> Sender: Garrett.Damore@sun.com To: Mike Shapiro Cc: Michael Shapiro , Girish Moodalbail , James.D.Carlson@sun.com, psarc-ext@sac.sfbay.sun.com Message-id: <49E4C948.50402@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <20090414171417.GA8124@sun.com> User-Agent: Thunderbird 2.0.0.18 (X11/20081201) Status: RO Content-Length: 4678 Mike Shapiro wrote: > On Tue, Apr 14, 2009 at 10:08:58AM -0700, Garrett D'Amore wrote: > >> Michael Shapiro wrote: >> >>> This case currently does not address the issue of how an administrator >>> or layered software determines the optimal large MTU, as opposed to >>> the maximum MTU. The two are not always the same. For example, on >>> Neptune (nxge), the maximum is 9000, but the optimal large MTU is 8150, >>> because of the size of the DMA transfers the card does in hardware. >>> This case needs to address this issue explicitly, by either: >>> >>> (a) Defining an additional interface by which the optimal value >>> can be returned from the driver as another attribute, OR >>> >>> (b) Making extremely clear to driver writers that if a large MTU >>> size less than the maximum is more optimal than the maximum, >>> that the optimal size should be returned by this interface. >>> >>> My preference is for option (a), but others should weigh in. >>> >>> -Mike >>> >>> >>> >> This feels like a "hardware tuning" element. >> > > That's because it is a hardware tuning element :) > > >> MTU configuration generally shouldn't need to worry about page sizes and >> such. An extra DMA transfer is usually in the "noise" as far as >> overheads of network processing are concerned. >> > > Not if you're trying to maximize performance of a heavily > loaded server, which is precisely when this matters most. > But if you wind up having to do two packet exchanges because you had to break a transfer up into a smaller size, that penalty (I hypothesize) will hurt you more than the cost of the extra DMA to transfer the packet all at once would. Now, as Jim Carlson already pointed out, if you know something about the hardware, and its true for *all* your L2 peers (that's critical!) on the network, then you can choose a slightly lower value that allows "most" packets to avoid an extra cost (DMA or otherwise), then that's probably worthwhile. > > >> More specifically, large MTUs are intended, as I see it, to minimize the >> effect of per-packet overheads found in NIC hardware, switches, routers, >> and most especially *hosts*. (I.e. the TCP/IP stack overheads.) I >> suspect that because of these overheads, that in general the largest MTU >> you can configure is always "optimal", even if the hardware has to >> perform some extra DMA transfers to make them happen. >> >> Unless some specific real world tests (e.g. TCP throughput or UDP stress >> tests) show otherwise, I'm disinclined to believe that there is any >> reason an administrator would need to know about the underlying hardware >> DMA limitations, or that the "optimum" value is anything other than the >> largest supported value. >> >> -- Garrett >> > > It makes a rather large difference in the absolute performance > numbers we achieve on the 7000 series. This is because you have > a system which is executing 16 cores at 100% cpu bound when it's > fully loaded and therefore cutting down on transfers and extraneous > packet processing makes a rather huge difference. > But the larger MTU should reduce the total number of packets. Of course, if you never exchange a packet that is over 8K, but have to pay extra overheads to *support* the *possibility* of a 9K packet... then that's a different problem. > Here's the actual point: the knowledge of this optimal size is a > function of the hardware and the driver. Our customers spend a lot > of time complaining about the fact that the out-of-the-box tuning > of networking on Solaris is sub-optimal and hard to understand how to do > better. Jumbo MTU isn't an area we can easily enable by default, > but we can make it a lot easier to achieve the fastest setting. > That knowledge belongs in the driver source, not on some out-of-date > wiki page where everyone has to waste more time googling around > and experimenting to figure it out. > Jumbo MTU is almost *always* better than default. The tweaking of values to hardware DMA engines requires tweaking not just one driver, but the whole network. And what's true for one driver/hardware combination might not be true for another. I believe that in general, the highest supported MTU (and 9000 is often used as the default because its big enough to transfer an 8K page) will win... unless you wind up in that situation where you pay for the overhead and don't actually *use* the extra transport capacity. Anyway, since this involves (as Jim pointed out) a L2 tunable that affects the entire network configuration, I'm not sure this kind of optimization really belongs here. -- Garrett From carlsonj@phorcys.east.sun.com Tue Apr 14 10:50:51 2009 Received: from dm-east-01.east.sun.com (dm-east-01.East.Sun.COM [129.148.9.192]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EHopoq009725 for ; Tue, 14 Apr 2009 10:50:51 -0700 (PDT) Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by dm-east-01.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3EHooMC028428; Tue, 14 Apr 2009 13:50:50 -0400 (EDT) Received: from phorcys.east.sun.com (phorcys.local [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n3EHoLoS016008; Tue, 14 Apr 2009 13:50:21 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n3EHoLUR016005; Tue, 14 Apr 2009 13:50:21 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18916.52445.919903.779513@gargle.gargle.HOWL> Date: Tue, 14 Apr 2009 13:50:21 -0400 From: James Carlson To: Mike Shapiro Cc: "Garrett D'Amore" , Michael Shapiro , Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Subject: Re: 2009/235 dladm Possible Values List In-Reply-To: <20090414173004.GA8350@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <18916.50551.493349.69795@gargle.gargle.HOWL> <20090414173004.GA8350@sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 2454 Mike Shapiro writes: > I agree with the above, but you're describing the rest of the jumbo issue > beyond what I'm getting at. There are two completely different questions: > > (a) What is the jumbo size I ought to set on this L2 network > given what is configured elsewhere on the network? > > (b) What is the jumbo size I ought to use with this card > for optimal large MTU performance on the card? They're distinct questions, but they're linked together in that you can't really consider (b) without taking (a) into account as well. At best, if we do what you're proposing, we could possibly do (b), but we can't currently do the required component (a) -- at all. > This is a significant issue for server and storage administrators, > as is generally all issues of making Solaris networking perform better, > and it's not effective to continue solving every problem with documentation. This problem is much more complex than I think you're making it out to be. For instance, drivers change over time. Code maintainers sometimes discover that poor performance in a commonly-used configuration is a bad thing (even if it was documented that way), and they then try to find ways to fix the driver so that it does better. Telling administrators to suck out a number from the driver, and then spit it back at the driver to "configure" the MTU means only that users will blindly be baking in that one value for MTU on their deployed systems -- and the value used will become just as stale over time as the web sites, hack scripts, and sketchy documentation you're lamenting. Figuring out how to get all of that sort of tuning right, and do so over time, still sounds to me like something separate from this project. One possibly useful convention might help: the interface defined here allows the driver to return a list of ranges for MTU. If, by convention, we tell driver writers to make sure that, where there are non-trivial performance issues, the top number listed in any range is the most efficient for that range, then you could (for instance) display "1500-8150,8151-9000" for nxge. Would that tweak allow this project to go forward without requiring the design of a new network performance subsystem? -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From sowmini.varadhan@sun.com Tue Apr 14 11:01:12 2009 Received: from dm-east-02.east.sun.com (dm-east-02.East.Sun.COM [129.148.13.5]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EI1B8G010538 for ; Tue, 14 Apr 2009 11:01:12 -0700 (PDT) Received: from quasimodo.East.Sun.COM (quasimodo.East.Sun.COM [129.148.174.94]) by dm-east-02.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3EI1AA9032131; Tue, 14 Apr 2009 14:01:10 -0400 (EDT) Received: from quasimodo.East.Sun.COM (localhost [127.0.0.1]) by quasimodo.East.Sun.COM (8.14.3+Sun/8.14.3) with ESMTP id n3EI0GSM011695; Tue, 14 Apr 2009 14:00:16 -0400 (EDT) Received: (from sowmini@localhost) by quasimodo.East.Sun.COM (8.14.3+Sun/8.14.3/Submit) id n3EI0GpL011694; Tue, 14 Apr 2009 14:00:16 -0400 (EDT) X-Authentication-Warning: quasimodo.East.Sun.COM: sowmini set sender to sowmini.varadhan@sun.com using -f Date: Tue, 14 Apr 2009 14:00:16 -0400 From: sowmini.varadhan@sun.com To: Mike Shapiro Cc: James Carlson , "Garrett D'Amore" , Michael Shapiro , Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Subject: Re: 2009/235 dladm Possible Values List Message-ID: <20090414180016.GA11683@quasimodo.East.Sun.COM> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <18916.50551.493349.69795@gargle.gargle.HOWL> <20090414173004.GA8350@sun.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20090414173004.GA8350@sun.com> User-Agent: Mutt/1.5.17 (2008-01-17) Status: RO Content-Length: 575 On (04/14/09 10:30), Mike Shapiro wrote: > > (b) What is the jumbo size I ought to use with this card > for optimal large MTU performance on the card? > Having a read-only property for optimal value per property per driver would not scale well as we add properties. In this particular case (jumbo frames) manual intervention is unavoidable, since the optimal mtu value on one side must also match the support mtu values on the peer. In any case, I think 2009/235 addresses a different problem (that does need to be solved) than the one Mike describes. --Sowmini From mws@zion.sfbay.sun.com Tue Apr 14 11:32:20 2009 Received: from zion.sfbay.sun.com (zion.SFBay.Sun.COM [129.146.17.75]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EIWJws011239 for ; Tue, 14 Apr 2009 11:32:19 -0700 (PDT) Received: from zion.sfbay.sun.com (localhost [127.0.0.1]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id n3EIWE8j010605; Tue, 14 Apr 2009 18:32:14 GMT Received: (from mws@localhost) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2/Submit) id n3EIWEG9010604; Tue, 14 Apr 2009 11:32:14 -0700 (PDT) Date: Tue, 14 Apr 2009 11:32:13 -0700 From: Mike Shapiro To: James Carlson Cc: "Garrett D'Amore" , Michael Shapiro , Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Subject: Re: 2009/235 dladm Possible Values List Message-ID: <20090414183213.GA10442@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <18916.50551.493349.69795@gargle.gargle.HOWL> <20090414173004.GA8350@sun.com> <18916.52445.919903.779513@gargle.gargle.HOWL> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <18916.52445.919903.779513@gargle.gargle.HOWL> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 2092 > This problem is much more complex than I think you're making it out to > be. > > For instance, drivers change over time. Code maintainers sometimes > discover that poor performance in a commonly-used configuration is a > bad thing (even if it was documented that way), and they then try to > find ways to fix the driver so that it does better. > > Telling administrators to suck out a number from the driver, and then > spit it back at the driver to "configure" the MTU means only that > users will blindly be baking in that one value for MTU on their > deployed systems -- and the value used will become just as stale over > time as the web sites, hack scripts, and sketchy documentation you're > lamenting. I totally disagree: if the information is in the driver, then if for some reason it changes due to driver or hardware, then we've eliminated the problem of people having the wrong value. > Figuring out how to get all of that sort of tuning right, and do so > over time, still sounds to me like something separate from this > project. I agree the end-to-end tuning this much more complicated, which is why I'm attempting to identify a specific sub-problem that can and should be solved that improves the situation. > One possibly useful convention might help: the interface defined here > allows the driver to return a list of ranges for MTU. If, by > convention, we tell driver writers to make sure that, where there are > non-trivial performance issues, the top number listed in any range is > the most efficient for that range, then you could (for instance) > display "1500-8150,8151-9000" for nxge. > > Would that tweak allow this project to go forward without requiring > the design of a new network performance subsystem? I don't understand how algorithmically that expresses, for example, which of 8150 or 8151 or 9000 is better to use, if one intends to deploy on a network with the max MTU set at 9000. Why is having that value as a single separate attribute not simpler? -Mike -- Mike Shapiro, Sun Microsystems Open Storage / Fishworks. blogs.sun.com/mws/ From mws@zion.sfbay.sun.com Tue Apr 14 11:33:27 2009 Received: from zion.sfbay.sun.com (zion.SFBay.Sun.COM [129.146.17.75]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EIXQto011314 for ; Tue, 14 Apr 2009 11:33:26 -0700 (PDT) Received: from zion.sfbay.sun.com (localhost [127.0.0.1]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id n3EIXGUP010652; Tue, 14 Apr 2009 18:33:16 GMT Received: (from mws@localhost) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2/Submit) id n3EIXGZ8010651; Tue, 14 Apr 2009 11:33:16 -0700 (PDT) Date: Tue, 14 Apr 2009 11:33:16 -0700 From: Mike Shapiro To: sowmini.varadhan@sun.com Cc: James Carlson , "Garrett D'Amore" , Michael Shapiro , Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Subject: Re: 2009/235 dladm Possible Values List Message-ID: <20090414183316.GB10442@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <18916.50551.493349.69795@gargle.gargle.HOWL> <20090414173004.GA8350@sun.com> <20090414180016.GA11683@quasimodo.East.Sun.COM> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20090414180016.GA11683@quasimodo.East.Sun.COM> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 585 On Tue, Apr 14, 2009 at 02:00:16PM -0400, sowmini.varadhan@sun.com wrote: > On (04/14/09 10:30), Mike Shapiro wrote: > > > > (b) What is the jumbo size I ought to use with this card > > for optimal large MTU performance on the card? > > > > Having a read-only property for optimal value per property per driver > would not scale well as we add properties. I'm not asking for an optimal value for every property: I'm asking for a single property that indicates "optimal_jumbo" or whatever. -Mike -- Mike Shapiro, Sun Microsystems Open Storage / Fishworks. blogs.sun.com/mws/ From gdamore@sun.com Tue Apr 14 11:42:49 2009 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EIgnrH011587 for ; Tue, 14 Apr 2009 11:42:49 -0700 (PDT) Received: from sca-es-mail-2.sun.com (sca-es-mail-2.Sun.COM [192.18.43.133]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3EIgn95060553 for ; Tue, 14 Apr 2009 11:42:49 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n3EIgiCn019782 for ; Tue, 14 Apr 2009 11:42:44 -0700 (PDT) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; charset=us-ascii Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KI300A00T4LCA00@fe-sfbay-10.sun.com> for psarc-ext@sac.sfbay.sun.com; Tue, 14 Apr 2009 11:42:44 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KI300878TB64QH0@fe-sfbay-10.sun.com>; Tue, 14 Apr 2009 11:42:42 -0700 (PDT) Date: Tue, 14 Apr 2009 11:42:42 -0700 From: "Garrett D'Amore" Subject: Re: 2009/235 dladm Possible Values List In-reply-to: <20090414183316.GB10442@sun.com> Sender: Garrett.Damore@sun.com To: Mike Shapiro Cc: Sowmini.Varadhan@sun.com, James Carlson , Michael Shapiro , Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Message-id: <49E4D922.8070907@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <18916.50551.493349.69795@gargle.gargle.HOWL> <20090414173004.GA8350@sun.com> <20090414180016.GA11683@quasimodo.East.Sun.COM> <20090414183316.GB10442@sun.com> User-Agent: Thunderbird 2.0.0.18 (X11/20081201) Status: RO Content-Length: 2168 Mike Shapiro wrote: > On Tue, Apr 14, 2009 at 02:00:16PM -0400, sowmini.varadhan@sun.com wrote: > >> On (04/14/09 10:30), Mike Shapiro wrote: >> >>> (b) What is the jumbo size I ought to use with this card >>> for optimal large MTU performance on the card? >>> >>> >> Having a read-only property for optimal value per property per driver >> would not scale well as we add properties. >> > > I'm not asking for an optimal value for every property: I'm asking > for a single property that indicates "optimal_jumbo" or whatever. > I still remain unconvinced here. If the traffic pattern can push 9000 byte packets, then I think that is what you should use for your MTU. If your pattern (at your site) is 8K packets, and you *know* that (and the pattern can't fill a 9K packets), then *perhaps* you should consider using a smaller MTU. (Of course, there are a lot more decisions that need to come into play. For example, what kinds of 802 bridges if any are in play -- fragmenting packets is *expensive* -- and what are the efficiencies of all the other hardware in your system?) In the Jumbo Frame arena, the most commonly supported size is 9K. Hardware and device drivers should, where possible, be designed to make this as efficient as possible. What I think you're asking for, "optimal_jumbo", is really "minimum_dma_overhead", and is inherently very specific to the implementation -- its just one aspect of choosing the "optimum" MTU. Calling it "optimum" is a misnomer that I think will lead to great confusion. And, furthermore, I expect that it will lead to a greatly increased number of service calls. Overall, this feels really like a "benchmark special" to me. If your driver has some particular behavior where 9K packets take a significant penalty that 8K do not, then this rightly should be treated in two ways: 1) as a bug -- the hardware/software should have been optimized to make 9K efficient as it is the most commonly deployed frame size 2) in documentation as a note in the man page. Both of these treatments are appropriate -- this isn't an either/or situation, I think. -- Garrett > -Mike > > From Girish.Moodalbail@Sun.COM Tue Apr 14 11:45:52 2009 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EIjqfX011696 for ; Tue, 14 Apr 2009 11:45:52 -0700 (PDT) Received: from brmea-mail-4.sun.com (brmea-mail-4.Sun.COM [192.18.98.36]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3EIjpDG062857 for ; Tue, 14 Apr 2009 11:45:52 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n3EIjpKx016849 for ; Tue, 14 Apr 2009 18:45:51 GMT MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; charset=us-ascii Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KI300F00TAOOQ00@mail-amer.sun.com> for psarc-ext@sac.sfbay.sun.com; Tue, 14 Apr 2009 12:45:51 -0600 (MDT) Received: from [192.168.0.106] (108.203.171.66.subscriber.vzavenue.net [66.171.203.108]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KI3006BUTGCFO80@mail-amer.sun.com>; Tue, 14 Apr 2009 12:45:49 -0600 (MDT) Date: Tue, 14 Apr 2009 14:45:48 -0400 From: Girish Moodalbail Subject: Re: 2009/235 dladm Possible Values List In-reply-to: <20090414183316.GB10442@sun.com> Sender: Girish.Moodalbail@Sun.COM To: Mike Shapiro Cc: Sowmini.Varadhan@Sun.COM, James Carlson , "Garrett D'Amore" , Michael Shapiro , psarc-ext@sac.sfbay.sun.com Message-id: <49E4D9DC.4070800@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <18916.50551.493349.69795@gargle.gargle.HOWL> <20090414173004.GA8350@sun.com> <20090414180016.GA11683@quasimodo.East.Sun.COM> <20090414183316.GB10442@sun.com> User-Agent: Thunderbird 2.0.0.17 (X11/20081023) Status: RO Content-Length: 822 Mike Shapiro wrote: > On Tue, Apr 14, 2009 at 02:00:16PM -0400, sowmini.varadhan@sun.com wrote: >> On (04/14/09 10:30), Mike Shapiro wrote: >>> (b) What is the jumbo size I ought to use with this card >>> for optimal large MTU performance on the card? >>> >> Having a read-only property for optimal value per property per driver >> would not scale well as we add properties. > > I'm not asking for an optimal value for every property: I'm asking > for a single property that indicates "optimal_jumbo" or whatever. Would specifiying the *preferred value* in the possible set, work? For example, the display would be: 1500-9000 (8150); The preferred value can be obtained using the proposed interfaces as part of 2nd range. mac_propval_range { count = 2, MAC_PROPVAL_UINT32, { 1500-9000 8150-8150 } } ~Girish From mws@zion.sfbay.sun.com Tue Apr 14 11:47:12 2009 Received: from zion.sfbay.sun.com (zion.SFBay.Sun.COM [129.146.17.75]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EIlBKd011712 for ; Tue, 14 Apr 2009 11:47:11 -0700 (PDT) Received: from zion.sfbay.sun.com (localhost [127.0.0.1]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id n3EIl4Po011143; Tue, 14 Apr 2009 18:47:04 GMT Received: (from mws@localhost) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2/Submit) id n3EIl4Se011142; Tue, 14 Apr 2009 11:47:04 -0700 (PDT) Date: Tue, 14 Apr 2009 11:47:04 -0700 From: Mike Shapiro To: "Garrett D'Amore" Cc: Sowmini.Varadhan@sun.com, James Carlson , Michael Shapiro , Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Subject: Re: 2009/235 dladm Possible Values List Message-ID: <20090414184703.GA10898@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <18916.50551.493349.69795@gargle.gargle.HOWL> <20090414173004.GA8350@sun.com> <20090414180016.GA11683@quasimodo.East.Sun.COM> <20090414183316.GB10442@sun.com> <49E4D922.8070907@sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <49E4D922.8070907@sun.com> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 2777 On Tue, Apr 14, 2009 at 11:42:42AM -0700, Garrett D'Amore wrote: > Mike Shapiro wrote: > >On Tue, Apr 14, 2009 at 02:00:16PM -0400, sowmini.varadhan@sun.com wrote: > > > >>On (04/14/09 10:30), Mike Shapiro wrote: > >> > >>>(b) What is the jumbo size I ought to use with this card > >>> for optimal large MTU performance on the card? > >>> > >>> > >>Having a read-only property for optimal value per property per driver > >>would not scale well as we add properties. > >> > > > >I'm not asking for an optimal value for every property: I'm asking > >for a single property that indicates "optimal_jumbo" or whatever. > > > > I still remain unconvinced here. > > If the traffic pattern can push 9000 byte packets, then I think that is > what you should use for your MTU. > > If your pattern (at your site) is 8K packets, and you *know* that (and > the pattern can't fill a 9K packets), then *perhaps* you should consider > using a smaller MTU. (Of course, there are a lot more decisions that > need to come into play. For example, what kinds of 802 bridges if any > are in play -- fragmenting packets is *expensive* -- and what are the > efficiencies of all the other hardware in your system?) > > In the Jumbo Frame arena, the most commonly supported size is 9K. > Hardware and device drivers should, where possible, be designed to make > this as efficient as possible. > > What I think you're asking for, "optimal_jumbo", is really > "minimum_dma_overhead", and is inherently very specific to the > implementation -- its just one aspect of choosing the "optimum" MTU. > Calling it "optimum" is a misnomer that I think will lead to great > confusion. And, furthermore, I expect that it will lead to a greatly > increased number of service calls. Naming aside, if you set the network at 9000 and then the driver at something smaller which is better, then you get the benefit. > Overall, this feels really like a "benchmark special" to me. If your > driver has some particular behavior where 9K packets take a significant > penalty that 8K do not, then this rightly should be treated in two ways: > > 1) as a bug -- the hardware/software should have been optimized to make > 9K efficient as it is the most commonly deployed frame size > 2) in documentation as a note in the man page. > > Both of these treatments are appropriate -- this isn't an either/or > situation, I think. As a software guy I agree. However, (1) you don't get to control: that's under the control of hardware design people and subject to other factors. And (2), which is the entire point of why I brought this up, is unlikely to be seen, and prone to error or staleness. -Mike -- Mike Shapiro, Sun Microsystems Open Storage / Fishworks. blogs.sun.com/mws/ From mws@zion.sfbay.sun.com Tue Apr 14 11:48:09 2009 Received: from zion.sfbay.sun.com (zion.SFBay.Sun.COM [129.146.17.75]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EIm9Ps011724 for ; Tue, 14 Apr 2009 11:48:09 -0700 (PDT) Received: from zion.sfbay.sun.com (localhost [127.0.0.1]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id n3EIm4hQ011181; Tue, 14 Apr 2009 18:48:04 GMT Received: (from mws@localhost) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2/Submit) id n3EIm4qU011180; Tue, 14 Apr 2009 11:48:04 -0700 (PDT) Date: Tue, 14 Apr 2009 11:48:03 -0700 From: Mike Shapiro To: Girish Moodalbail Cc: Sowmini.Varadhan@sun.com, James Carlson , "Garrett D'Amore" , Michael Shapiro , psarc-ext@sac.sfbay.sun.com Subject: Re: 2009/235 dladm Possible Values List Message-ID: <20090414184803.GB10898@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <18916.50551.493349.69795@gargle.gargle.HOWL> <20090414173004.GA8350@sun.com> <20090414180016.GA11683@quasimodo.East.Sun.COM> <20090414183316.GB10442@sun.com> <49E4D9DC.4070800@sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <49E4D9DC.4070800@sun.com> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 1121 On Tue, Apr 14, 2009 at 02:45:48PM -0400, Girish Moodalbail wrote: > Mike Shapiro wrote: > >On Tue, Apr 14, 2009 at 02:00:16PM -0400, sowmini.varadhan@sun.com wrote: > >>On (04/14/09 10:30), Mike Shapiro wrote: > >>>(b) What is the jumbo size I ought to use with this card > >>> for optimal large MTU performance on the card? > >>> > >>Having a read-only property for optimal value per property per driver > >>would not scale well as we add properties. > > > >I'm not asking for an optimal value for every property: I'm asking > >for a single property that indicates "optimal_jumbo" or whatever. > > > Would specifiying the *preferred value* in the possible set, work? For > example, the display would be: > > 1500-9000 (8150); > > The preferred value can be obtained using the proposed interfaces as > part of 2nd range. > mac_propval_range { > count = 2, > MAC_PROPVAL_UINT32, > { > 1500-9000 > 8150-8150 > } > } > > ~Girish Yes, that is a perfectly simple and effective approach that addresses what I was getting at. -Mike -- Mike Shapiro, Sun Microsystems Open Storage / Fishworks. blogs.sun.com/mws/ From carlsonj@phorcys.east.sun.com Tue Apr 14 11:49:13 2009 Received: from dm-east-02.east.sun.com (dm-east-02.East.Sun.COM [129.148.13.5]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EInCZq011738 for ; Tue, 14 Apr 2009 11:49:12 -0700 (PDT) Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by dm-east-02.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3EInBLK057746; Tue, 14 Apr 2009 14:49:11 -0400 (EDT) Received: from phorcys.east.sun.com (phorcys.local [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n3EImhv9016330; Tue, 14 Apr 2009 14:48:43 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n3EImhA4016327; Tue, 14 Apr 2009 14:48:43 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18916.55947.341354.85469@gargle.gargle.HOWL> Date: Tue, 14 Apr 2009 14:48:43 -0400 From: James Carlson To: Mike Shapiro Cc: "Garrett D'Amore" , Michael Shapiro , Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Subject: Re: 2009/235 dladm Possible Values List In-Reply-To: <20090414183213.GA10442@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <18916.50551.493349.69795@gargle.gargle.HOWL> <20090414173004.GA8350@sun.com> <18916.52445.919903.779513@gargle.gargle.HOWL> <20090414183213.GA10442@sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 2713 Mike Shapiro writes: > > Telling administrators to suck out a number from the driver, and then > > spit it back at the driver to "configure" the MTU means only that > > users will blindly be baking in that one value for MTU on their > > deployed systems -- and the value used will become just as stale over > > time as the web sites, hack scripts, and sketchy documentation you're > > lamenting. > > I totally disagree: if the information is in the driver, then > if for some reason it changes due to driver or hardware, then > we've eliminated the problem of people having the wrong value. No, we haven't. The user will list out this "optimal jumbogram size" value exactly once, when he's installing the system. He'll use it to configure the interface with this magical MTU value. Days, weeks, or months later, he'll install a patch or update. This may change the optimal value. But it will have zero effect on the configured MTU: his value is now suboptimal, and we're back to the world of web sites and documentation. > > One possibly useful convention might help: the interface defined here > > allows the driver to return a list of ranges for MTU. If, by > > convention, we tell driver writers to make sure that, where there are > > non-trivial performance issues, the top number listed in any range is > > the most efficient for that range, then you could (for instance) > > display "1500-8150,8151-9000" for nxge. > > > > Would that tweak allow this project to go forward without requiring > > the design of a new network performance subsystem? > > I don't understand how algorithmically that expresses, for example, > which of 8150 or 8151 or 9000 is better to use, if one intends to > deploy on a network with the max MTU set at 9000. It expresses that the driver can go all the way up to 9000, and hints that the sweet spots are the ends of the ranges -- 8150 and 9000. If you _must_ have a value over 8150 in order to interoperate, then it seems quite obvious to me that setting 8150 anyway won't help you. You need the next larger value that the driver will handle reasonably well. If you don't specifically need a value over 8150 to do whatever it is you're doing, then the appearance of that value at the end of a range says "this is the largest value with consistent performance in this range." > Why is having > that value as a single separate attribute not simpler? In part because I believe you're really talking about a different project. File an RFE. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From Sebastien.Roy@Sun.COM Tue Apr 14 12:01:49 2009 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EJ1n9l027214 for ; Tue, 14 Apr 2009 12:01:49 -0700 (PDT) Received: from brmea-mail-4.sun.com (brmea-mail-4.Sun.COM [192.18.98.36]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3EJ1mVx049870 for ; Tue, 14 Apr 2009 12:01:49 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n3EJ1m1M024445 for ; Tue, 14 Apr 2009 19:01:48 GMT MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; charset=UTF-8 Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KI300C00TOPJ500@mail-amer.sun.com> for psarc-ext@sac.sfbay.sun.com; Tue, 14 Apr 2009 13:01:48 -0600 (MDT) Received: from [129.148.174.103] ([unknown] [129.148.174.103]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KI3003ZPU6JF970@mail-amer.sun.com>; Tue, 14 Apr 2009 13:01:31 -0600 (MDT) Date: Tue, 14 Apr 2009 15:01:00 -0400 From: Sebastien Roy Subject: Re: 2009/235 dladm Possible Values List In-reply-to: <20090414183316.GB10442@sun.com> Sender: Sebastien.Roy@Sun.COM To: Mike Shapiro Cc: Sowmini.Varadhan@Sun.COM, James Carlson , "Garrett D'Amore" , Michael Shapiro , Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Message-id: <1239735660.662.262.camel@strat> Organization: Sun Microsystems X-Mailer: Evolution 2.24.2 References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <18916.50551.493349.69795@gargle.gargle.HOWL> <20090414173004.GA8350@sun.com> <20090414180016.GA11683@quasimodo.East.Sun.COM> <20090414183316.GB10442@sun.com> Status: RO Content-Length: 1196 On Tue, 2009-04-14 at 11:33 -0700, Mike Shapiro wrote: > On Tue, Apr 14, 2009 at 02:00:16PM -0400, sowmini.varadhan@sun.com wrote: > > On (04/14/09 10:30), Mike Shapiro wrote: > > > > > > (b) What is the jumbo size I ought to use with this card > > > for optimal large MTU performance on the card? > > > > > > > Having a read-only property for optimal value per property per driver > > would not scale well as we add properties. > > I'm not asking for an optimal value for every property: I'm asking > for a single property that indicates "optimal_jumbo" or whatever. I'd like to make a suggestion unrelated to the technical merits of this idea. There is value in this fast-track as-is, on its own. If there is value in the "optimal MTU" idea, then it can also stand on its own (as Mike said, it could potentially be a separate read-only property). Therefore, the least contentious way forward would be to approve this case as-is while discussing the optimal MTU idea outside the context of this case (on networking-discuss@opensolaris.org for example). If people can agree on a proposal for this, then it would be simple enough to file a small fast-track for it separately. -Seb From sowmini.varadhan@sun.com Tue Apr 14 12:14:24 2009 Received: from dm-east-01.east.sun.com (dm-east-01.East.Sun.COM [129.148.9.192]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EJEM0r007729 for ; Tue, 14 Apr 2009 12:14:23 -0700 (PDT) Received: from quasimodo.East.Sun.COM (quasimodo.East.Sun.COM [129.148.174.94]) by dm-east-01.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3EJELb0006848; Tue, 14 Apr 2009 15:14:21 -0400 (EDT) Received: from quasimodo.East.Sun.COM (localhost [127.0.0.1]) by quasimodo.East.Sun.COM (8.14.3+Sun/8.14.3) with ESMTP id n3EJDRWG011736; Tue, 14 Apr 2009 15:13:27 -0400 (EDT) Received: (from sowmini@localhost) by quasimodo.East.Sun.COM (8.14.3+Sun/8.14.3/Submit) id n3EJDR4b011735; Tue, 14 Apr 2009 15:13:27 -0400 (EDT) X-Authentication-Warning: quasimodo.East.Sun.COM: sowmini set sender to sowmini.varadhan@sun.com using -f Date: Tue, 14 Apr 2009 15:13:27 -0400 From: sowmini.varadhan@sun.com To: Sebastien Roy Cc: Mike Shapiro , James Carlson , "Garrett D'Amore" , Michael Shapiro , Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Subject: Re: 2009/235 dladm Possible Values List Message-ID: <20090414191327.GD11683@quasimodo.East.Sun.COM> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <18916.50551.493349.69795@gargle.gargle.HOWL> <20090414173004.GA8350@sun.com> <20090414180016.GA11683@quasimodo.East.Sun.COM> <20090414183316.GB10442@sun.com> <1239735660.662.262.camel@strat> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1239735660.662.262.camel@strat> User-Agent: Mutt/1.5.17 (2008-01-17) Status: RO Content-Length: 899 On (04/14/09 15:01), Sebastien Roy wrote: > I'd like to make a suggestion unrelated to the technical merits of this > idea. There is value in this fast-track as-is, on its own. If there is agreed. > value in the "optimal MTU" idea, then it can also stand on its own (as > Mike said, it could potentially be a separate read-only property). > Therefore, the least contentious way forward would be to approve this > case as-is while discussing the optimal MTU idea outside the context of > this case (on networking-discuss@opensolaris.org for example). If > people can agree on a proposal for this, then it would be simple enough > to file a small fast-track for it separately. personally, I'd like to keep the "optimal" value as a separate system-call (to be dealt with in a separate PSARC case) because it's not clear what value a driver should return for other value ranges. ymmv --Sowmini From gdamore@sun.com Tue Apr 14 12:19:06 2009 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EJJ6EY007777 for ; Tue, 14 Apr 2009 12:19:06 -0700 (PDT) Received: from sca-es-mail-2.sun.com (sca-es-mail-2.Sun.COM [192.18.43.133]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3EJJ6D8025117 for ; Tue, 14 Apr 2009 12:19:06 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n3EJJ0EE024559 for ; Tue, 14 Apr 2009 12:19:01 -0700 (PDT) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; charset=us-ascii Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KI300K00US5KP00@fe-sfbay-10.sun.com> for psarc-ext@sac.sfbay.sun.com; Tue, 14 Apr 2009 12:19:00 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KI30049CUZJFF00@fe-sfbay-10.sun.com>; Tue, 14 Apr 2009 12:18:56 -0700 (PDT) Date: Tue, 14 Apr 2009 12:18:55 -0700 From: "Garrett D'Amore" Subject: Re: 2009/235 dladm Possible Values List In-reply-to: <20090414184703.GA10898@sun.com> Sender: Garrett.Damore@sun.com To: Mike Shapiro Cc: Sowmini.Varadhan@sun.com, James Carlson , Michael Shapiro , Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Message-id: <49E4E19F.4020306@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <18916.50551.493349.69795@gargle.gargle.HOWL> <20090414173004.GA8350@sun.com> <20090414180016.GA11683@quasimodo.East.Sun.COM> <20090414183316.GB10442@sun.com> <49E4D922.8070907@sun.com> <20090414184703.GA10898@sun.com> User-Agent: Thunderbird 2.0.0.18 (X11/20081201) Status: RO Content-Length: 3281 Mike Shapiro wrote: > On Tue, Apr 14, 2009 at 11:42:42AM -0700, Garrett D'Amore wrote: > >> Mike Shapiro wrote: >> >> [snip!] >> >> What I think you're asking for, "optimal_jumbo", is really >> "minimum_dma_overhead", and is inherently very specific to the >> implementation -- its just one aspect of choosing the "optimum" MTU. >> Calling it "optimum" is a misnomer that I think will lead to great >> confusion. And, furthermore, I expect that it will lead to a greatly >> increased number of service calls. >> > > Naming aside, if you set the network at 9000 and then the driver > at something smaller which is better, then you get the benefit. > Huh? Either I'm not parsing this correctly, or you're missing a critical point. If the rest of the network is using 9000 bytes, and your device driver uses a smaller value, then you'll have fragmentation or other network breakage. Fragmentation leads to poor performance. (Far worse than other situations.) Path MTU discovery helps here, but you can't *rely* on it. (Or at least you're not *supposed* to.) > > >> Overall, this feels really like a "benchmark special" to me. If your >> driver has some particular behavior where 9K packets take a significant >> penalty that 8K do not, then this rightly should be treated in two ways: >> >> 1) as a bug -- the hardware/software should have been optimized to make >> 9K efficient as it is the most commonly deployed frame size >> 2) in documentation as a note in the man page. >> >> Both of these treatments are appropriate -- this isn't an either/or >> situation, I think. >> > > As a software guy I agree. However, (1) you don't get to control: > that's under the control of hardware design people and subject to > other factors. And (2), which is the entire point of why I brought > this up, is unlikely to be seen, and prone to error or staleness. > But again, the "optimum" (or "preferred") value will depend on elements that go far beyond the driver. As I said, I believe if you can drive 9000 bytes packets, then you will generally get better performance. (Ignoring other issues like backing store to disk blocks...) I don't think its generally true that -- even for this hardware -- an 8150 byte mtu will *always* give better performance than a 9000 byte mtu. For some traffic (NFS?) it might... but for other kinds of traffic (streaming a big file over a raw socket) I doubt it will. But then this value really is a benchmark special, not a general purpose value. I think exposing this as a single value to customers is a mistake. I think knowledge of how to properly configure this value requires much more advanced knowledge than we can expose in a single value -- and so rightly belongs in documentation. Perhaps a man page. Perhaps a tuning guide. I'm not sure, except that I feel pretty strongly that *this* is *not* the right place. And the fact that the tuning is required to get good performance, remains, IMO, a bug. We should try hard to get the hardware and driver engineers to recognize this problem. I mean, its not like 9K packets are suddenly a new thing or such -- that has been the de-facto standard for jumbo frames on ethernet pretty much from when they were first proposed. - Garrett From gdamore@sun.com Tue Apr 14 12:20:16 2009 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EJKGIW007836 for ; Tue, 14 Apr 2009 12:20:16 -0700 (PDT) Received: from sca-es-mail-1.sun.com (sca-es-mail-1.Sun.COM [192.18.43.132]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3EJKGS8001698 for ; Tue, 14 Apr 2009 12:20:16 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n3EJKBY5023818 for ; Tue, 14 Apr 2009 12:20:11 -0700 (PDT) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; format=flowed; charset=UTF-8 Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KI300K00US5KP00@fe-sfbay-10.sun.com> for psarc-ext@sac.sfbay.sun.com; Tue, 14 Apr 2009 12:20:11 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KI3004U3V1LFF00@fe-sfbay-10.sun.com>; Tue, 14 Apr 2009 12:20:10 -0700 (PDT) Date: Tue, 14 Apr 2009 12:20:09 -0700 From: "Garrett D'Amore" Subject: Re: 2009/235 dladm Possible Values List In-reply-to: <1239735660.662.262.camel@strat> Sender: Garrett.Damore@sun.com To: Sebastien Roy Cc: Mike Shapiro , Sowmini.Varadhan@sun.com, James Carlson , Michael Shapiro , Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Message-id: <49E4E1E9.7090508@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <18916.50551.493349.69795@gargle.gargle.HOWL> <20090414173004.GA8350@sun.com> <20090414180016.GA11683@quasimodo.East.Sun.COM> <20090414183316.GB10442@sun.com> <1239735660.662.262.camel@strat> User-Agent: Thunderbird 2.0.0.18 (X11/20081201) Status: RO Content-Length: 1366 Sebastien Roy wrote: > On Tue, 2009-04-14 at 11:33 -0700, Mike Shapiro wrote: > >> On Tue, Apr 14, 2009 at 02:00:16PM -0400, sowmini.varadhan@sun.com wrote: >> >>> On (04/14/09 10:30), Mike Shapiro wrote: >>> >>>> (b) What is the jumbo size I ought to use with this card >>>> for optimal large MTU performance on the card? >>>> >>>> >>> Having a read-only property for optimal value per property per driver >>> would not scale well as we add properties. >>> >> I'm not asking for an optimal value for every property: I'm asking >> for a single property that indicates "optimal_jumbo" or whatever. >> > > I'd like to make a suggestion unrelated to the technical merits of this > idea. There is value in this fast-track as-is, on its own. If there is > value in the "optimal MTU" idea, then it can also stand on its own (as > Mike said, it could potentially be a separate read-only property). > Therefore, the least contentious way forward would be to approve this > case as-is while discussing the optimal MTU idea outside the context of > this case (on networking-discuss@opensolaris.org for example). If > people can agree on a proposal for this, then it would be simple enough > to file a small fast-track for it separately. > Agreed. I'll give a +1 to the case *as specified*, without Mike's RFE. -- Garrett From carlsonj@phorcys.east.sun.com Tue Apr 14 12:27:46 2009 Received: from dm-east-01.east.sun.com (dm-east-01.East.Sun.COM [129.148.9.192]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3EJRjYc007895 for ; Tue, 14 Apr 2009 12:27:46 -0700 (PDT) Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by dm-east-01.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3EJRiXg014261; Tue, 14 Apr 2009 15:27:44 -0400 (EDT) Received: from phorcys.east.sun.com (phorcys.local [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n3EJRGLJ016550; Tue, 14 Apr 2009 15:27:16 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n3EJRGkr016547; Tue, 14 Apr 2009 15:27:16 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18916.58260.595129.323000@gargle.gargle.HOWL> Date: Tue, 14 Apr 2009 15:27:16 -0400 From: James Carlson To: "Garrett D'Amore" Cc: Mike Shapiro , Sowmini.Varadhan@sun.com, Michael Shapiro , Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Subject: Re: 2009/235 dladm Possible Values List In-Reply-To: <49E4E19F.4020306@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <18916.50551.493349.69795@gargle.gargle.HOWL> <20090414173004.GA8350@sun.com> <20090414180016.GA11683@quasimodo.East.Sun.COM> <20090414183316.GB10442@sun.com> <49E4D922.8070907@sun.com> <20090414184703.GA10898@sun.com> <49E4E19F.4020306@sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 1582 Garrett D'Amore writes: > If the rest of the network is using 9000 bytes, and your device driver > uses a smaller value, then you'll have fragmentation or other network > breakage. You usually end up with black holes, not mere fragmentation. The nodes using the larger MTU will typically be unable to send large packets to you, and little in the network will tell you that you've got a problem. (This is the well-known jumbogram issue, and is what Mike explicitly said he doesn't need to solve. He's interested only in solving the "how does the user find out 'the' 'optimal' value for this driver?" problem, and not how to make any interoperable solution.) > I think exposing this as a single value to customers is a mistake. I > think knowledge of how to properly configure this value requires much > more advanced knowledge than we can expose in a single value -- and so > rightly belongs in documentation. Perhaps a man page. Perhaps a tuning > guide. I'm not sure, except that I feel pretty strongly that *this* is > *not* the right place. As seb suggested, networking-discuss is probably a better place to discuss the issues. Maybe there is value in having a programmatic interface (though I'm still wary of it), but I don't think this project really can or should support it. The proposals to graft it on the side look like hacks to me. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From mws@zion.sfbay.sun.com Tue Apr 14 23:01:33 2009 Received: from zion.sfbay.sun.com (zion.SFBay.Sun.COM [129.146.17.75]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3F61Xak016522 for ; Tue, 14 Apr 2009 23:01:33 -0700 (PDT) Received: from zion.sfbay.sun.com (localhost [127.0.0.1]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id n3F61NWZ022261; Wed, 15 Apr 2009 06:01:23 GMT Received: (from mws@localhost) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2/Submit) id n3F61NBu022260; Tue, 14 Apr 2009 23:01:23 -0700 (PDT) Date: Tue, 14 Apr 2009 23:01:23 -0700 From: Mike Shapiro To: Sebastien Roy Cc: Sowmini.Varadhan@sun.com, James Carlson , "Garrett D'Amore" , Michael Shapiro , Girish Moodalbail , psarc-ext@sac.sfbay.sun.com Subject: Re: 2009/235 dladm Possible Values List Message-ID: <20090415060123.GA22175@sun.com> References: <200904141647.n3EGlVGG007434@zion.sfbay.sun.com> <49E4C32A.5000705@sun.com> <18916.50551.493349.69795@gargle.gargle.HOWL> <20090414173004.GA8350@sun.com> <20090414180016.GA11683@quasimodo.East.Sun.COM> <20090414183316.GB10442@sun.com> <1239735660.662.262.camel@strat> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1239735660.662.262.camel@strat> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 2516 On Tue, Apr 14, 2009 at 03:01:00PM -0400, Sebastien Roy wrote: > > On Tue, 2009-04-14 at 11:33 -0700, Mike Shapiro wrote: > > On Tue, Apr 14, 2009 at 02:00:16PM -0400, sowmini.varadhan@sun.com wrote: > > > On (04/14/09 10:30), Mike Shapiro wrote: > > > > > > > > (b) What is the jumbo size I ought to use with this card > > > > for optimal large MTU performance on the card? > > > > > > > > > > Having a read-only property for optimal value per property per driver > > > would not scale well as we add properties. > > > > I'm not asking for an optimal value for every property: I'm asking > > for a single property that indicates "optimal_jumbo" or whatever. > > I'd like to make a suggestion unrelated to the technical merits of this > idea. There is value in this fast-track as-is, on its own. If there is > value in the "optimal MTU" idea, then it can also stand on its own (as > Mike said, it could potentially be a separate read-only property). > Therefore, the least contentious way forward would be to approve this > case as-is while discussing the optimal MTU idea outside the context of > this case (on networking-discuss@opensolaris.org for example). If > people can agree on a proposal for this, then it would be simple enough > to file a small fast-track for it separately. > > -Seb Given the back-and-forth today, I think that is best. The fundamental point I would make is that the configuration of jumbos is (a) extremely common for customers, (b) not easy to achieve optimal results, and (c) has impact on the design of the property interface. In my opinion this issue is barely addressed in Solaris today, this case as it was originally proposed solves one of the sub-problems nicely, but doesn't solve some of the other problems, including the one I mentioned. And the one I mentioned has direct impact on the interface being modified in this case, which is why it ought to be considered now. Also because we ought to be actually solving the entire problem as best it can be (given the protocol-level limitations), and not pretending that this is purely a doc issue (with no action plan), or is simply not a problem (which it is, given the many customers who ask about it). As an example, the single biggest call generator we've had around missing networking features for the 7000 series was the absence of jumbo support for LACP (now fixed) -- which indicates how frequently this is used. -Mike -- Mike Shapiro, Sun Microsystems Open Storage / Fishworks. blogs.sun.com/mws/ From carlsonj@phorcys.east.sun.com Wed Apr 15 12:49:26 2009 Received: from dm-east-02.east.sun.com (dm-east-02.East.Sun.COM [129.148.13.5]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n3FJnQdA025588 for ; Wed, 15 Apr 2009 12:49:26 -0700 (PDT) Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by dm-east-02.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n3FJnPY3025091; Wed, 15 Apr 2009 15:49:25 -0400 (EDT) Received: from phorcys.east.sun.com (phorcys.local [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n3FJmtxu019636; Wed, 15 Apr 2009 15:48:55 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n3FJmtMN019633; Wed, 15 Apr 2009 15:48:55 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18918.14887.8330.840061@gargle.gargle.HOWL> Date: Wed, 15 Apr 2009 15:48:55 -0400 From: James Carlson To: psarc-ext@sac.sfbay.sun.com cc: "Girish M. G." Subject: Re: 2009/235 dladm Possible Values List In-Reply-To: <18915.27586.986715.589201@gargle.gargle.HOWL> References: <18915.27586.986715.589201@gargle.gargle.HOWL> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 627 > I'm sponsoring this fast-track request for Girish Moodalbail. The > timer is set to 04/20/2009. This fast-track request was approved during ARC business today as originally specified. The ARC members agree that the discussion of how to handle "optimal" hardware values is important, especially for eliminating needless tuning tasks, and should be continued in the OpenSolaris Networking Community. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677