From tedk@sac.sfbay.sun.com Fri Oct 30 16:05:55 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9UN5sqW024188 for ; Fri, 30 Oct 2009 16:05:54 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n9UN5Vud025918; Sat, 31 Oct 2009 07:05:53 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSC00207O5S1R00@brm-avmta-1.central.sun.com>; Fri, 30 Oct 2009 17:05:52 -0600 (MDT) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSC008C4O5R2HB0@brm-avmta-1.central.sun.com>; Fri, 30 Oct 2009 17:05:51 -0600 (MDT) Received: from sac.sfbay.sun.com (sac.SFBay.Sun.COM [129.146.226.132]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n9UN5p5T018707; Fri, 30 Oct 2009 16:05:51 -0700 (PDT) Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9UN5nbJ024183; Fri, 30 Oct 2009 16:05:49 -0700 (PDT) Received: (from tedk@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id n9UN5nOx024179; Fri, 30 Oct 2009 16:05:49 -0700 (PDT) Date: Fri, 30 Oct 2009 16:05:49 -0700 (PDT) From: Ted Kim Subject: IPoIB Connected Mode [PSARC/2009/593 FastTrack timeout 11/06/2009] To: PSARC-ext@sun.com Message-id: <200910302305.n9UN5nOx024179@sac.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 6052 Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: IPoIB Connected Mode 1.2. Name of Document Author/Supplier: Author: Kevin Ge 1.3 Date of This Document: 30 October, 2009 4. Technical Description A. Overview ----------- This case proposes changes to the Solaris kernel to provide support for "Connected Mode" in the IPoIB driver ibd(7D) (described in [1] and [2]). The Infiniband Architecture [3] defines multiple "transport service types", including Unreliable Datagram (UD), Reliable Connected (RC) and Unreliable Connected (UC). Current ibd (based on [4]) runs in "Datagram Mode" over the UD transport service type. Connected Mode (described in [5]) can use either UC and/or RC. This IPoIB-CM project uses RC, because of the desire to inter-operate with Linux which also uses RC. The main advantage of Connected Mode is better performance (higher throughput and lower CPU utilization) based on using very large MTUs (see below for more discussion). Connected Mode, though, can have the disadvantage of consuming more resources, especially when scaling up to a large cluster (due to using an InfiniBand connection to each destination). Note that this case only covers all necessary changes to support IPoIB driver running in Connected Mode over RC. Other enhancements are outside the scope of this case. A micro/patch binding is asserted for this proposal. B. Connected Mode IPoIB driver ------------------------------ The revised ibd(7D) driver will support both Connected and Datagram mode. The features from the current Datagram mode ibd driver will be inherited. The remainder of this section discusses interface additions for the Connected mode capable driver. B.1 Switching between datagram and connected mode The existing ibd driver in OpenSolaris and Solaris 10 does not ship with a driver .conf file. However, the Connected Mode support described in this case introduces a new parameter 'enable_rc' that may be set via the ibd driver .conf file. This parameter specifies whether each ibd instance defaults to using Connected Mode over RC or not. # 1: unicast packets will be sent over Reliable Connected Mode # 0: unicast packets will be sent over Unreliable Datagram Mode # # Each element in the list below maps to the corresponding ibd # instance; the first element is for ibd instance 0, the second # element is for instance 1 and so on. # enable_rc=1,1,0,0; Please note that Connected Mode support in IPoIB is optional as per [5]. Therefore, if Connected Mode is not available for a remote node, the Datagram mode will automatically be used for that destination by the ibd driver. Therefore, the only meaning of 'enable_rc' is to decide whether to try Connected Mode first or not, and whether to advertise this as a capability supported by this instance or not. The default value for 'enable_rc' for each instance is 0. Hence without a ibd.conf file, Datagram mode will be used. We intend to ship a driver .conf file for ibd in ONNV (and hence OpenSolaris) with enable_rc set to all ones (enabling Connected Mode by default on all instances) for the best performance. However, for Solaris 10, we have received business guidance to have an "opt-in" approach due to a desire for greater stability in established enterprise environments. We will do this by not shipping the .conf file. Therefore, by default Solaris 10 will be Datagram mode. It will take an explicit administrator action (setting enable_rc) to cause Solaris 10 to use Connected Mode. OFED (Linux IB) originally made Connected Mode opt-in too. However, later OFED made it the default. We don't intend to change it later to be the default in Solaris 10. However, Solaris Next, being descended from ONNV, will have it as default. An edited ibd(7D) manpage documenting this change is in the materials directory. B.2 Change of default MTU size Connected Mode by virtue of using the RC transport service type offers link MTUs of up to 2^31-4 octets in length. Thus, the use of Connected Mode can offer benefits by supporting very large MTUs. Datagram Mode using UD is limited to 4092 (4K-4) octets, though commonly only 2044 (2K-4) is offered. Due to the limits of the TCP/IP protocol, it makes sense to only offer up to 65535 (64K-1) bytes. OFED (i.e. Linux IB) uses 65520 (64K-16) byte MTU for alignment reasons. To inter-operate with OFED at the best performance, we also adopt 65520 as the default MTU of the Connected Mode. C. Interfaces ------------- +-------------------------------------------------------------------+ | Interfaces Exported | +---------------------------+------------------+--------------------+ | Interface Name | Classification | Comment | +---------------------------+------------------+--------------------+ |/kernel/drv/ibd.conf* | Uncommitted | Configuration file | +---------------------------+------------------+--------------------+ * = only for OpenSolaris D. References ------------- [1] IP over InfiniBand, PSARC/2001/289 [2] IPoIB Conversion to GLDv3, PSARC/2007/636 [3] InfiniBand Architecture Specification Volume 1, Release 1.2.1, InfiniBand Trade Association, 2007. http://www.infinibandta.org/content/pages.php?pg=technology_download [4] Transmission of IP over InfiniBand (IPoIB), RFC 4391, IETF, http://www.ietf.org/rfc/rfc4391.txt [5] IP over InfiniBand: Connected Mode, RFC 4755, IETF, http://www.ietf.org/rfc/rfc4755.txt 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From gdamore@sun.com Fri Oct 30 16:22:16 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9UNMETq024877 for ; Fri, 30 Oct 2009 16:22:15 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n9UNM81s029784 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 30 Oct 2009 23:22:14 GMT Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSC00303OX1VJ00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Fri, 30 Oct 2009 17:22:13 -0600 (MDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSC008KSOX126A0@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Fri, 30 Oct 2009 17:22:13 -0600 (MDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n9UNMDid025058 for ; Fri, 30 Oct 2009 16:22:13 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KSC00700OW2P200@fe-sfbay-10.sun.com> for PSARC-ext@Sun.Com (ORCPT PSARC-ext@Sun.Com); Fri, 30 Oct 2009 16:22:13 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KSC00BLWOX0J5D0@fe-sfbay-10.sun.com>; Fri, 30 Oct 2009 16:22:12 -0700 (PDT) Date: Fri, 30 Oct 2009 16:22:12 -0700 From: "Garrett D'Amore" Subject: Re: IPoIB Connected Mode [PSARC/2009/593 FastTrack timeout 11/06/2009] In-reply-to: <200910302305.n9UN5nOx024179@sac.sfbay.sun.com> Sender: Garrett.Damore@sun.com To: Ted Kim Cc: PSARC-ext@sun.com Message-id: <4AEB7524.1020206@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910302305.n9UN5nOx024179@sac.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.22 (X11/20090909) Status: RO Content-Length: 7249 I feel very strongly that I'd prefer to avoid the use of a driver.conf for this, and instead handle it as a Brussels property, at least on Solaris Nevada. (This will support administration via dladm, and ultimately also ndd, though we don't like to say that. ;-) If you need to use a driver.conf for Solaris 10, that's OK I suppose (although an ndd tunable would be better there too, since it doesn't require the driver to be unloaded and reloaded to change the setting -- which can be very challenging for administrators to figure out.) I feel TCR-strong on this -- if it were a full case I'd insist that this be part of the spec before I'd vote to approve. Is the project team amenable to making this change, or do they have some other reason why driver.conf values need to be used instead. Also, I'd like the mtu to be set via Brussels as well, if it isn't already handled that way. - Garrett Ted Kim wrote: > Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI > This information is Copyright 2009 Sun Microsystems > 1. Introduction > 1.1. Project/Component Working Name: > IPoIB Connected Mode > 1.2. Name of Document Author/Supplier: > Author: Kevin Ge > 1.3 Date of This Document: > 30 October, 2009 > 4. Technical Description > > A. Overview > ----------- > > This case proposes changes to the Solaris kernel to provide support > for "Connected Mode" in the IPoIB driver ibd(7D) (described in [1] > and [2]). > > The Infiniband Architecture [3] defines multiple "transport service > types", including Unreliable Datagram (UD), Reliable Connected (RC) > and Unreliable Connected (UC). Current ibd (based on [4]) runs in > "Datagram Mode" over the UD transport service type. Connected Mode > (described in [5]) can use either UC and/or RC. > > This IPoIB-CM project uses RC, because of the desire to > inter-operate with Linux which also uses RC. The main advantage of > Connected Mode is better performance (higher throughput and lower > CPU utilization) based on using very large MTUs (see below for more > discussion). Connected Mode, though, can have the disadvantage of > consuming more resources, especially when scaling up to a large > cluster (due to using an InfiniBand connection to each destination). > > Note that this case only covers all necessary changes to support > IPoIB driver running in Connected Mode over RC. Other enhancements > are outside the scope of this case. > > A micro/patch binding is asserted for this proposal. > > B. Connected Mode IPoIB driver > ------------------------------ > > The revised ibd(7D) driver will support both Connected and Datagram > mode. The features from the current Datagram mode ibd driver will > be inherited. The remainder of this section discusses interface > additions for the Connected mode capable driver. > > > B.1 Switching between datagram and connected mode > > The existing ibd driver in OpenSolaris and Solaris 10 does not > ship with a driver .conf file. However, the Connected Mode support > described in this case introduces a new parameter 'enable_rc' that > may be set via the ibd driver .conf file. > > This parameter specifies whether each ibd instance defaults to > using Connected Mode over RC or not. > > # 1: unicast packets will be sent over Reliable Connected Mode > # 0: unicast packets will be sent over Unreliable Datagram Mode > # > # Each element in the list below maps to the corresponding ibd > # instance; the first element is for ibd instance 0, the second > # element is for instance 1 and so on. > # > enable_rc=1,1,0,0; > > Please note that Connected Mode support in IPoIB is optional as per > [5]. Therefore, if Connected Mode is not available for a remote > node, the Datagram mode will automatically be used for that > destination by the ibd driver. Therefore, the only meaning of > 'enable_rc' is to decide whether to try Connected Mode first or > not, and whether to advertise this as a capability supported by > this instance or not. > > The default value for 'enable_rc' for each instance is 0. Hence > without a ibd.conf file, Datagram mode will be used. We intend to > ship a driver .conf file for ibd in ONNV (and hence OpenSolaris) > with enable_rc set to all ones (enabling Connected Mode by > default on all instances) for the best performance. > > However, for Solaris 10, we have received business guidance to have > an "opt-in" approach due to a desire for greater stability in > established enterprise environments. We will do this by not > shipping the .conf file. Therefore, by default Solaris 10 will be > Datagram mode. It will take an explicit administrator action > (setting enable_rc) to cause Solaris 10 to use Connected Mode. > > OFED (Linux IB) originally made Connected Mode opt-in too. However, > later OFED made it the default. We don't intend to change it later > to be the default in Solaris 10. However, Solaris Next, being > descended from ONNV, will have it as default. > > An edited ibd(7D) manpage documenting this change is in the > materials directory. > > B.2 Change of default MTU size > > Connected Mode by virtue of using the RC transport service type > offers link MTUs of up to 2^31-4 octets in length. Thus, the use of > Connected Mode can offer benefits by supporting very large MTUs. > Datagram Mode using UD is limited to 4092 (4K-4) octets, though > commonly only 2044 (2K-4) is offered. > > Due to the limits of the TCP/IP protocol, it makes sense to only > offer up to 65535 (64K-1) bytes. OFED (i.e. Linux IB) uses 65520 > (64K-16) byte MTU for alignment reasons. To inter-operate with > OFED at the best performance, we also adopt 65520 as the default > MTU of the Connected Mode. > > > C. Interfaces > ------------- > +-------------------------------------------------------------------+ > | Interfaces Exported | > +---------------------------+------------------+--------------------+ > | Interface Name | Classification | Comment | > +---------------------------+------------------+--------------------+ > |/kernel/drv/ibd.conf* | Uncommitted | Configuration file | > +---------------------------+------------------+--------------------+ > * = only for OpenSolaris > > > D. References > ------------- > > [1] IP over InfiniBand, PSARC/2001/289 > > [2] IPoIB Conversion to GLDv3, PSARC/2007/636 > > [3] InfiniBand Architecture Specification Volume 1, Release 1.2.1, > InfiniBand Trade Association, 2007. > http://www.infinibandta.org/content/pages.php?pg=technology_download > > [4] Transmission of IP over InfiniBand (IPoIB), RFC 4391, IETF, > http://www.ietf.org/rfc/rfc4391.txt > > [5] IP over InfiniBand: Connected Mode, RFC 4755, IETF, > http://www.ietf.org/rfc/rfc4755.txt > > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > ON > 6.5. ARC review type: FastTrack > 6.6. ARC Exposure: open > > From Ted.Kim@Sun.COM Fri Oct 30 16:30:30 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9UNUUnE025229 for ; Fri, 30 Oct 2009 16:30:30 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n9UNUT6I001288 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 30 Oct 2009 16:30:29 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSC00401PATSX00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Fri, 30 Oct 2009 17:30:29 -0600 (MDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSC00884PAS26B0@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Fri, 30 Oct 2009 17:30:29 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n9UNUSIm005062 for ; Fri, 30 Oct 2009 16:30:28 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KSC00000P3AXE00@fe-sfbay-09.sun.com> for PSARC-ext@Sun.Com (ORCPT PSARC-ext@Sun.Com); Fri, 30 Oct 2009 16:30:25 -0700 (PDT) Received: from [129.153.3.74] ([unknown] [129.153.3.74]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KSC007MVPANE4D0@fe-sfbay-09.sun.com>; Fri, 30 Oct 2009 16:30:25 -0700 (PDT) Date: Fri, 30 Oct 2009 16:27:55 -0700 From: "Ted H. Kim" Subject: Re: IPoIB Connected Mode [PSARC/2009/593 FastTrack timeout 11/06/2009] In-reply-to: <4AEB7524.1020206@sun.com> Sender: Ted.Kim@Sun.COM To: "Garrett D'Amore" Cc: Ted Kim , PSARC-ext@Sun.COM, Kevin Ge , Sudhakar Dindukurti Message-id: <4AEB767B.4070309@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910302305.n9UN5nOx024179@sac.sfbay.sun.com> <4AEB7524.1020206@sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090910) Status: RO Content-Length: 7990 There is a extensive revision of the dladm support for IPoIB coming where Brussels support and a change in the administrative model will be dealt with. But that may be a ways off (est. 2010.Q2?) and in the meantime, people are screaming for the performance that Connected Mode gives, so we don't want to wait for that. -ted Garrett D'Amore wrote: > I feel very strongly that I'd prefer to avoid the use of a driver.conf > for this, and instead handle it as a Brussels property, at least on > Solaris Nevada. (This will support administration via dladm, and > ultimately also ndd, though we don't like to say that. ;-) > > If you need to use a driver.conf for Solaris 10, that's OK I suppose > (although an ndd tunable would be better there too, since it doesn't > require the driver to be unloaded and reloaded to change the setting -- > which can be very challenging for administrators to figure out.) > > I feel TCR-strong on this -- if it were a full case I'd insist that this > be part of the spec before I'd vote to approve. > > Is the project team amenable to making this change, or do they have some > other reason why driver.conf values need to be used instead. > > Also, I'd like the mtu to be set via Brussels as well, if it isn't > already handled that way. > > - Garrett > > Ted Kim wrote: >> Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI >> This information is Copyright 2009 Sun Microsystems >> 1. Introduction >> 1.1. Project/Component Working Name: >> IPoIB Connected Mode >> 1.2. Name of Document Author/Supplier: >> Author: Kevin Ge >> 1.3 Date of This Document: >> 30 October, 2009 >> 4. Technical Description >> >> A. Overview >> ----------- >> >> This case proposes changes to the Solaris kernel to provide support >> for "Connected Mode" in the IPoIB driver ibd(7D) (described in [1] >> and [2]). >> >> The Infiniband Architecture [3] defines multiple "transport service >> types", including Unreliable Datagram (UD), Reliable Connected (RC) >> and Unreliable Connected (UC). Current ibd (based on [4]) runs in >> "Datagram Mode" over the UD transport service type. Connected Mode >> (described in [5]) can use either UC and/or RC. >> >> This IPoIB-CM project uses RC, because of the desire to >> inter-operate with Linux which also uses RC. The main advantage of >> Connected Mode is better performance (higher throughput and lower >> CPU utilization) based on using very large MTUs (see below for more >> discussion). Connected Mode, though, can have the disadvantage of >> consuming more resources, especially when scaling up to a large >> cluster (due to using an InfiniBand connection to each destination). >> >> Note that this case only covers all necessary changes to support >> IPoIB driver running in Connected Mode over RC. Other enhancements >> are outside the scope of this case. >> >> A micro/patch binding is asserted for this proposal. >> >> B. Connected Mode IPoIB driver >> ------------------------------ >> >> The revised ibd(7D) driver will support both Connected and Datagram >> mode. The features from the current Datagram mode ibd driver will >> be inherited. The remainder of this section discusses interface >> additions for the Connected mode capable driver. >> >> >> B.1 Switching between datagram and connected mode >> >> The existing ibd driver in OpenSolaris and Solaris 10 does not >> ship with a driver .conf file. However, the Connected Mode support >> described in this case introduces a new parameter 'enable_rc' that >> may be set via the ibd driver .conf file. >> >> This parameter specifies whether each ibd instance defaults to >> using Connected Mode over RC or not. >> >> # 1: unicast packets will be sent over Reliable Connected Mode >> # 0: unicast packets will be sent over Unreliable Datagram Mode >> # >> # Each element in the list below maps to the corresponding ibd >> # instance; the first element is for ibd instance 0, the second >> # element is for instance 1 and so on. >> # >> enable_rc=1,1,0,0; >> >> Please note that Connected Mode support in IPoIB is optional as per >> [5]. Therefore, if Connected Mode is not available for a remote >> node, the Datagram mode will automatically be used for that >> destination by the ibd driver. Therefore, the only meaning of >> 'enable_rc' is to decide whether to try Connected Mode first or >> not, and whether to advertise this as a capability supported by >> this instance or not. >> >> The default value for 'enable_rc' for each instance is 0. Hence >> without a ibd.conf file, Datagram mode will be used. We intend to >> ship a driver .conf file for ibd in ONNV (and hence OpenSolaris) >> with enable_rc set to all ones (enabling Connected Mode by >> default on all instances) for the best performance. >> >> However, for Solaris 10, we have received business guidance to have >> an "opt-in" approach due to a desire for greater stability in >> established enterprise environments. We will do this by not >> shipping the .conf file. Therefore, by default Solaris 10 will be >> Datagram mode. It will take an explicit administrator action >> (setting enable_rc) to cause Solaris 10 to use Connected Mode. >> OFED (Linux IB) originally made Connected Mode opt-in too. However, >> later OFED made it the default. We don't intend to change it later >> to be the default in Solaris 10. However, Solaris Next, being >> descended from ONNV, will have it as default. >> >> An edited ibd(7D) manpage documenting this change is in the >> materials directory. >> >> B.2 Change of default MTU size >> >> Connected Mode by virtue of using the RC transport service type >> offers link MTUs of up to 2^31-4 octets in length. Thus, the use of >> Connected Mode can offer benefits by supporting very large MTUs. >> Datagram Mode using UD is limited to 4092 (4K-4) octets, though >> commonly only 2044 (2K-4) is offered. >> >> Due to the limits of the TCP/IP protocol, it makes sense to only >> offer up to 65535 (64K-1) bytes. OFED (i.e. Linux IB) uses 65520 >> (64K-16) byte MTU for alignment reasons. To inter-operate with >> OFED at the best performance, we also adopt 65520 as the default >> MTU of the Connected Mode. >> >> >> C. Interfaces >> ------------- >> +-------------------------------------------------------------------+ >> | Interfaces Exported | >> +---------------------------+------------------+--------------------+ >> | Interface Name | Classification | Comment | >> +---------------------------+------------------+--------------------+ >> |/kernel/drv/ibd.conf* | Uncommitted | Configuration file | >> +---------------------------+------------------+--------------------+ >> * = only for OpenSolaris >> >> >> D. References >> ------------- >> >> [1] IP over InfiniBand, PSARC/2001/289 >> >> [2] IPoIB Conversion to GLDv3, PSARC/2007/636 >> >> [3] InfiniBand Architecture Specification Volume 1, Release 1.2.1, >> InfiniBand Trade Association, 2007. >> http://www.infinibandta.org/content/pages.php?pg=technology_download >> >> [4] Transmission of IP over InfiniBand (IPoIB), RFC 4391, IETF, >> http://www.ietf.org/rfc/rfc4391.txt >> >> [5] IP over InfiniBand: Connected Mode, RFC 4755, IETF, >> http://www.ietf.org/rfc/rfc4755.txt >> >> 6. Resources and Schedule >> 6.4. Steering Committee requested information >> 6.4.1. Consolidation C-team Name: >> ON >> 6.5. ARC review type: FastTrack >> 6.6. ARC Exposure: open >> >> > -- Ted H. Kim Sun Microsystems, Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX From gdamore@sun.com Fri Oct 30 16:41:09 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9UNf8Mq025660 for ; Fri, 30 Oct 2009 16:41:08 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n9UNf52a014090 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Sat, 31 Oct 2009 07:41:07 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSC00007PSH3Y00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Fri, 30 Oct 2009 16:41:05 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSC00KIGPSHYW20@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Fri, 30 Oct 2009 16:41:05 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n9UNexWR026258 for ; Fri, 30 Oct 2009 16:41:05 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KSC00900POHXK00@fe-sfbay-09.sun.com> for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Fri, 30 Oct 2009 16:41:04 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KSC007S7PRME4F0@fe-sfbay-09.sun.com>; Fri, 30 Oct 2009 16:40:35 -0700 (PDT) Date: Fri, 30 Oct 2009 16:40:34 -0700 From: "Garrett D'Amore" Subject: Re: IPoIB Connected Mode [PSARC/2009/593 FastTrack timeout 11/06/2009] In-reply-to: <4AEB767B.4070309@sun.com> Sender: Garrett.Damore@sun.com To: "Ted H. Kim" Cc: Ted Kim , PSARC-ext@sun.com, Kevin Ge , Sudhakar Dindukurti Message-id: <4AEB7972.2030202@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910302305.n9UN5nOx024179@sac.sfbay.sun.com> <4AEB7524.1020206@sun.com> <4AEB767B.4070309@sun.com> User-Agent: Thunderbird 2.0.0.22 (X11/20090909) Status: RO Content-Length: 8397 Ted H. Kim wrote: > There is a extensive revision of the dladm > support for IPoIB coming where Brussels > support and a change in the administrative > model will be dealt with. But that may be > a ways off (est. 2010.Q2?) and in the > meantime, people are screaming for > the performance that Connected Mode gives, > so we don't want to wait for that. So, lets make the .conf setting Volatile, since we expect to change it in less than a year to a Brussels setting. This will allow people to use it, but with an admonition not to get too fond of the driver.conf setting. Personally, I think Brussels support is so easy to implement that I'm not sure I understand why this can't be done almost immediately. - Garrett > > -ted > > Garrett D'Amore wrote: >> I feel very strongly that I'd prefer to avoid the use of a >> driver.conf for this, and instead handle it as a Brussels property, >> at least on Solaris Nevada. (This will support administration via >> dladm, and ultimately also ndd, though we don't like to say that. ;-) >> >> If you need to use a driver.conf for Solaris 10, that's OK I suppose >> (although an ndd tunable would be better there too, since it doesn't >> require the driver to be unloaded and reloaded to change the setting >> -- which can be very challenging for administrators to figure out.) >> >> I feel TCR-strong on this -- if it were a full case I'd insist that >> this be part of the spec before I'd vote to approve. >> >> Is the project team amenable to making this change, or do they have >> some other reason why driver.conf values need to be used instead. >> >> Also, I'd like the mtu to be set via Brussels as well, if it isn't >> already handled that way. >> >> - Garrett >> >> Ted Kim wrote: >>> Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI >>> This information is Copyright 2009 Sun Microsystems >>> 1. Introduction >>> 1.1. Project/Component Working Name: >>> IPoIB Connected Mode >>> 1.2. Name of Document Author/Supplier: >>> Author: Kevin Ge >>> 1.3 Date of This Document: >>> 30 October, 2009 >>> 4. Technical Description >>> >>> A. Overview >>> ----------- >>> >>> This case proposes changes to the Solaris kernel to provide support >>> for "Connected Mode" in the IPoIB driver ibd(7D) (described in [1] >>> and [2]). >>> >>> The Infiniband Architecture [3] defines multiple "transport service >>> types", including Unreliable Datagram (UD), Reliable Connected (RC) >>> and Unreliable Connected (UC). Current ibd (based on [4]) runs in >>> "Datagram Mode" over the UD transport service type. Connected Mode >>> (described in [5]) can use either UC and/or RC. >>> >>> This IPoIB-CM project uses RC, because of the desire to >>> inter-operate with Linux which also uses RC. The main advantage of >>> Connected Mode is better performance (higher throughput and lower >>> CPU utilization) based on using very large MTUs (see below for more >>> discussion). Connected Mode, though, can have the disadvantage of >>> consuming more resources, especially when scaling up to a large >>> cluster (due to using an InfiniBand connection to each destination). >>> >>> Note that this case only covers all necessary changes to support >>> IPoIB driver running in Connected Mode over RC. Other enhancements >>> are outside the scope of this case. >>> >>> A micro/patch binding is asserted for this proposal. >>> >>> B. Connected Mode IPoIB driver >>> ------------------------------ >>> >>> The revised ibd(7D) driver will support both Connected and Datagram >>> mode. The features from the current Datagram mode ibd driver will >>> be inherited. The remainder of this section discusses interface >>> additions for the Connected mode capable driver. >>> >>> >>> B.1 Switching between datagram and connected mode >>> >>> The existing ibd driver in OpenSolaris and Solaris 10 does not >>> ship with a driver .conf file. However, the Connected Mode support >>> described in this case introduces a new parameter 'enable_rc' that >>> may be set via the ibd driver .conf file. >>> >>> This parameter specifies whether each ibd instance defaults to >>> using Connected Mode over RC or not. >>> >>> # 1: unicast packets will be sent over Reliable Connected Mode >>> # 0: unicast packets will be sent over Unreliable Datagram Mode >>> # >>> # Each element in the list below maps to the corresponding ibd >>> # instance; the first element is for ibd instance 0, the second >>> # element is for instance 1 and so on. >>> # >>> enable_rc=1,1,0,0; >>> >>> Please note that Connected Mode support in IPoIB is optional as per >>> [5]. Therefore, if Connected Mode is not available for a remote >>> node, the Datagram mode will automatically be used for that >>> destination by the ibd driver. Therefore, the only meaning of >>> 'enable_rc' is to decide whether to try Connected Mode first or >>> not, and whether to advertise this as a capability supported by >>> this instance or not. >>> >>> The default value for 'enable_rc' for each instance is 0. Hence >>> without a ibd.conf file, Datagram mode will be used. We intend to >>> ship a driver .conf file for ibd in ONNV (and hence OpenSolaris) >>> with enable_rc set to all ones (enabling Connected Mode by >>> default on all instances) for the best performance. >>> >>> However, for Solaris 10, we have received business guidance to have >>> an "opt-in" approach due to a desire for greater stability in >>> established enterprise environments. We will do this by not >>> shipping the .conf file. Therefore, by default Solaris 10 will be >>> Datagram mode. It will take an explicit administrator action >>> (setting enable_rc) to cause Solaris 10 to use Connected Mode. >>> OFED (Linux IB) originally made Connected Mode opt-in too. >>> However, >>> later OFED made it the default. We don't intend to change it later >>> to be the default in Solaris 10. However, Solaris Next, being >>> descended from ONNV, will have it as default. >>> >>> An edited ibd(7D) manpage documenting this change is in the >>> materials directory. >>> >>> B.2 Change of default MTU size >>> >>> Connected Mode by virtue of using the RC transport service type >>> offers link MTUs of up to 2^31-4 octets in length. Thus, the use of >>> Connected Mode can offer benefits by supporting very large MTUs. >>> Datagram Mode using UD is limited to 4092 (4K-4) octets, though >>> commonly only 2044 (2K-4) is offered. >>> >>> Due to the limits of the TCP/IP protocol, it makes sense to only >>> offer up to 65535 (64K-1) bytes. OFED (i.e. Linux IB) uses 65520 >>> (64K-16) byte MTU for alignment reasons. To inter-operate with >>> OFED at the best performance, we also adopt 65520 as the default >>> MTU of the Connected Mode. >>> >>> >>> C. Interfaces >>> ------------- >>> +-------------------------------------------------------------------+ >>> | Interfaces Exported | >>> +---------------------------+------------------+--------------------+ >>> | Interface Name | Classification | Comment | >>> +---------------------------+------------------+--------------------+ >>> |/kernel/drv/ibd.conf* | Uncommitted | Configuration file | >>> +---------------------------+------------------+--------------------+ >>> * = only for OpenSolaris >>> >>> >>> D. References >>> ------------- >>> >>> [1] IP over InfiniBand, PSARC/2001/289 >>> >>> [2] IPoIB Conversion to GLDv3, PSARC/2007/636 >>> >>> [3] InfiniBand Architecture Specification Volume 1, Release 1.2.1, >>> InfiniBand Trade Association, 2007. >>> >>> http://www.infinibandta.org/content/pages.php?pg=technology_download >>> >>> [4] Transmission of IP over InfiniBand (IPoIB), RFC 4391, IETF, >>> http://www.ietf.org/rfc/rfc4391.txt >>> >>> [5] IP over InfiniBand: Connected Mode, RFC 4755, IETF, >>> http://www.ietf.org/rfc/rfc4755.txt >>> >>> 6. Resources and Schedule >>> 6.4. Steering Committee requested information >>> 6.4.1. Consolidation C-team Name: >>> ON >>> 6.5. ARC review type: FastTrack >>> 6.6. ARC Exposure: open >>> >>> >> > From ted.kim@sun.com Fri Oct 30 16:43:41 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9UNhd7R025777 for ; Fri, 30 Oct 2009 16:43:40 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n9UNhcUA009981 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 30 Oct 2009 23:43:39 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSC00307PWP4200@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Fri, 30 Oct 2009 16:43:37 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSC009NWPWPJV80@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Fri, 30 Oct 2009 16:43:37 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n9UNhb0Y026433 for ; Fri, 30 Oct 2009 16:43:37 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KSC00K00PQUCV00@fe-sfbay-10.sun.com> for PSARC-ext@Sun.Com (ORCPT PSARC-ext@Sun.Com); Fri, 30 Oct 2009 16:43:37 -0700 (PDT) Received: from [129.153.3.74] ([unknown] [129.153.3.74]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KSC00HV0PWO2F10@fe-sfbay-10.sun.com>; Fri, 30 Oct 2009 16:43:37 -0700 (PDT) Date: Fri, 30 Oct 2009 16:41:07 -0700 From: "Ted H. Kim" Subject: Re: IPoIB Connected Mode [PSARC/2009/593 FastTrack timeout 11/06/2009] In-reply-to: <4AEB7972.2030202@sun.com> Sender: ted.kim@sun.com To: "Garrett D'Amore" Cc: Ted Kim , PSARC-ext@sun.com, Kevin Ge , Sudhakar Dindukurti Message-id: <4AEB7993.1020107@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910302305.n9UN5nOx024179@sac.sfbay.sun.com> <4AEB7524.1020206@sun.com> <4AEB767B.4070309@sun.com> <4AEB7972.2030202@sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090910) Status: RO Content-Length: 8845 okay, volatile it is. -ted Garrett D'Amore wrote: > Ted H. Kim wrote: >> There is a extensive revision of the dladm >> support for IPoIB coming where Brussels >> support and a change in the administrative >> model will be dealt with. But that may be >> a ways off (est. 2010.Q2?) and in the >> meantime, people are screaming for >> the performance that Connected Mode gives, >> so we don't want to wait for that. > > So, lets make the .conf setting Volatile, since we expect to change it > in less than a year to a Brussels setting. This will allow people to > use it, but with an admonition not to get too fond of the driver.conf > setting. > > Personally, I think Brussels support is so easy to implement that I'm > not sure I understand why this can't be done almost immediately. > > - Garrett > >> >> -ted >> >> Garrett D'Amore wrote: >>> I feel very strongly that I'd prefer to avoid the use of a >>> driver.conf for this, and instead handle it as a Brussels property, >>> at least on Solaris Nevada. (This will support administration via >>> dladm, and ultimately also ndd, though we don't like to say that. ;-) >>> >>> If you need to use a driver.conf for Solaris 10, that's OK I suppose >>> (although an ndd tunable would be better there too, since it doesn't >>> require the driver to be unloaded and reloaded to change the setting >>> -- which can be very challenging for administrators to figure out.) >>> >>> I feel TCR-strong on this -- if it were a full case I'd insist that >>> this be part of the spec before I'd vote to approve. >>> >>> Is the project team amenable to making this change, or do they have >>> some other reason why driver.conf values need to be used instead. >>> >>> Also, I'd like the mtu to be set via Brussels as well, if it isn't >>> already handled that way. >>> >>> - Garrett >>> >>> Ted Kim wrote: >>>> Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI >>>> This information is Copyright 2009 Sun Microsystems >>>> 1. Introduction >>>> 1.1. Project/Component Working Name: >>>> IPoIB Connected Mode >>>> 1.2. Name of Document Author/Supplier: >>>> Author: Kevin Ge >>>> 1.3 Date of This Document: >>>> 30 October, 2009 >>>> 4. Technical Description >>>> >>>> A. Overview >>>> ----------- >>>> >>>> This case proposes changes to the Solaris kernel to provide support >>>> for "Connected Mode" in the IPoIB driver ibd(7D) (described in [1] >>>> and [2]). >>>> >>>> The Infiniband Architecture [3] defines multiple "transport service >>>> types", including Unreliable Datagram (UD), Reliable Connected (RC) >>>> and Unreliable Connected (UC). Current ibd (based on [4]) runs in >>>> "Datagram Mode" over the UD transport service type. Connected Mode >>>> (described in [5]) can use either UC and/or RC. >>>> >>>> This IPoIB-CM project uses RC, because of the desire to >>>> inter-operate with Linux which also uses RC. The main advantage of >>>> Connected Mode is better performance (higher throughput and lower >>>> CPU utilization) based on using very large MTUs (see below for more >>>> discussion). Connected Mode, though, can have the disadvantage of >>>> consuming more resources, especially when scaling up to a large >>>> cluster (due to using an InfiniBand connection to each destination). >>>> >>>> Note that this case only covers all necessary changes to support >>>> IPoIB driver running in Connected Mode over RC. Other enhancements >>>> are outside the scope of this case. >>>> >>>> A micro/patch binding is asserted for this proposal. >>>> >>>> B. Connected Mode IPoIB driver >>>> ------------------------------ >>>> >>>> The revised ibd(7D) driver will support both Connected and Datagram >>>> mode. The features from the current Datagram mode ibd driver will >>>> be inherited. The remainder of this section discusses interface >>>> additions for the Connected mode capable driver. >>>> >>>> >>>> B.1 Switching between datagram and connected mode >>>> >>>> The existing ibd driver in OpenSolaris and Solaris 10 does not >>>> ship with a driver .conf file. However, the Connected Mode support >>>> described in this case introduces a new parameter 'enable_rc' that >>>> may be set via the ibd driver .conf file. >>>> >>>> This parameter specifies whether each ibd instance defaults to >>>> using Connected Mode over RC or not. >>>> >>>> # 1: unicast packets will be sent over Reliable Connected Mode >>>> # 0: unicast packets will be sent over Unreliable Datagram Mode >>>> # >>>> # Each element in the list below maps to the corresponding ibd >>>> # instance; the first element is for ibd instance 0, the second >>>> # element is for instance 1 and so on. >>>> # >>>> enable_rc=1,1,0,0; >>>> >>>> Please note that Connected Mode support in IPoIB is optional as per >>>> [5]. Therefore, if Connected Mode is not available for a remote >>>> node, the Datagram mode will automatically be used for that >>>> destination by the ibd driver. Therefore, the only meaning of >>>> 'enable_rc' is to decide whether to try Connected Mode first or >>>> not, and whether to advertise this as a capability supported by >>>> this instance or not. >>>> >>>> The default value for 'enable_rc' for each instance is 0. Hence >>>> without a ibd.conf file, Datagram mode will be used. We intend to >>>> ship a driver .conf file for ibd in ONNV (and hence OpenSolaris) >>>> with enable_rc set to all ones (enabling Connected Mode by >>>> default on all instances) for the best performance. >>>> >>>> However, for Solaris 10, we have received business guidance to have >>>> an "opt-in" approach due to a desire for greater stability in >>>> established enterprise environments. We will do this by not >>>> shipping the .conf file. Therefore, by default Solaris 10 will be >>>> Datagram mode. It will take an explicit administrator action >>>> (setting enable_rc) to cause Solaris 10 to use Connected Mode. >>>> OFED (Linux IB) originally made Connected Mode opt-in too. >>>> However, >>>> later OFED made it the default. We don't intend to change it later >>>> to be the default in Solaris 10. However, Solaris Next, being >>>> descended from ONNV, will have it as default. >>>> >>>> An edited ibd(7D) manpage documenting this change is in the >>>> materials directory. >>>> >>>> B.2 Change of default MTU size >>>> >>>> Connected Mode by virtue of using the RC transport service type >>>> offers link MTUs of up to 2^31-4 octets in length. Thus, the use of >>>> Connected Mode can offer benefits by supporting very large MTUs. >>>> Datagram Mode using UD is limited to 4092 (4K-4) octets, though >>>> commonly only 2044 (2K-4) is offered. >>>> >>>> Due to the limits of the TCP/IP protocol, it makes sense to only >>>> offer up to 65535 (64K-1) bytes. OFED (i.e. Linux IB) uses 65520 >>>> (64K-16) byte MTU for alignment reasons. To inter-operate with >>>> OFED at the best performance, we also adopt 65520 as the default >>>> MTU of the Connected Mode. >>>> >>>> >>>> C. Interfaces >>>> ------------- >>>> +-------------------------------------------------------------------+ >>>> | Interfaces Exported | >>>> +---------------------------+------------------+--------------------+ >>>> | Interface Name | Classification | Comment | >>>> +---------------------------+------------------+--------------------+ >>>> |/kernel/drv/ibd.conf* | Uncommitted | Configuration file | >>>> +---------------------------+------------------+--------------------+ >>>> * = only for OpenSolaris >>>> >>>> >>>> D. References >>>> ------------- >>>> >>>> [1] IP over InfiniBand, PSARC/2001/289 >>>> >>>> [2] IPoIB Conversion to GLDv3, PSARC/2007/636 >>>> >>>> [3] InfiniBand Architecture Specification Volume 1, Release 1.2.1, >>>> InfiniBand Trade Association, 2007. >>>> >>>> http://www.infinibandta.org/content/pages.php?pg=technology_download >>>> >>>> [4] Transmission of IP over InfiniBand (IPoIB), RFC 4391, IETF, >>>> http://www.ietf.org/rfc/rfc4391.txt >>>> >>>> [5] IP over InfiniBand: Connected Mode, RFC 4755, IETF, >>>> http://www.ietf.org/rfc/rfc4755.txt >>>> >>>> 6. Resources and Schedule >>>> 6.4. Steering Committee requested information >>>> 6.4.1. Consolidation C-team Name: >>>> ON >>>> 6.5. ARC review type: FastTrack >>>> 6.6. ARC Exposure: open >>>> >>>> >>> >> > -- Ted H. Kim Sun Microsystems, Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX From Ted.Kim@sun.com Wed Nov 4 10:26:49 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nA4IQmRi009557 for ; Wed, 4 Nov 2009 10:26:49 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id nA4IQl7l011189 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 4 Nov 2009 11:26:48 -0700 (MST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSL00L1FKKOCR00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 04 Nov 2009 11:26:48 -0700 (MST) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSL00EL6KKNNVE0@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 04 Nov 2009 11:26:47 -0700 (MST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id nA4IQkkq010946 for ; Wed, 04 Nov 2009 10:26:46 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KSL00700K77O000@fe-sfbay-09.sun.com> for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Wed, 04 Nov 2009 10:26:46 -0800 (PST) Received: from [129.153.3.74] ([unknown] [129.153.3.74]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KSL009VIKKKZ220@fe-sfbay-09.sun.com>; Wed, 04 Nov 2009 10:26:46 -0800 (PST) Date: Wed, 04 Nov 2009 10:24:07 -0800 From: "Ted H. Kim" Subject: Re: IPoIB Connected Mode [PSARC/2009/593 FastTrack timeout 11/06/2009] In-reply-to: <4AEB7993.1020107@sun.com> Sender: Ted.Kim@sun.com To: PSARC-ext@sun.com Message-id: <4AF1C6C7.1060702@sun.com> MIME-version: 1.0 Content-type: multipart/mixed; boundary="Boundary_(ID_qgAlT1bnlpuL8FLLsDtpsg)" X-PMX-Version: 5.4.1.325704 References: <200910302305.n9UN5nOx024179@sac.sfbay.sun.com> <4AEB7524.1020206@sun.com> <4AEB767B.4070309@sun.com> <4AEB7972.2030202@sun.com> <4AEB7993.1020107@sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090910) Status: RO Content-Length: 6326 This is a multi-part message in MIME format. --Boundary_(ID_qgAlT1bnlpuL8FLLsDtpsg) Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT This case, IPoIB Connected Mode (PSARC/2009/593), was approved at today's PSARC meeting, so I am marking the case closed/approved. The updated spec (with the change to Volatile) is below. Thanks, -ted -- Ted H. Kim Sun Microsystems, Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX --Boundary_(ID_qgAlT1bnlpuL8FLLsDtpsg) Content-type: text/plain; name=ipoib-cm-case.txt Content-transfer-encoding: 7BIT Content-disposition: inline; filename=ipoib-cm-case.txt A. Overview ----------- This case proposes changes to the Solaris kernel to provide support for "Connected Mode" in the IPoIB driver ibd(7D) (described in [1] and [2]). The Infiniband Architecture [3] defines multiple "transport service types", including Unreliable Datagram (UD), Reliable Connected (RC) and Unreliable Connected (UC). Current ibd (based on [4]) runs in "Datagram Mode" over the UD transport service type. Connected Mode (described in [5]) can use either UC and/or RC. This IPoIB-CM project uses RC, because of the desire to inter-operate with Linux which also uses RC. The main advantage of Connected Mode is better performance (higher throughput and lower CPU utilization) based on using very large MTUs (see below for more discussion). Connected Mode, though, can have the disadvantage of consuming more resources, especially when scaling up to a large cluster (due to using an InfiniBand connection to each destination). Note that this case only covers all necessary changes to support IPoIB driver running in Connected Mode over RC. Other enhancements are outside the scope of this case. A micro/patch binding is asserted for this proposal. B. Connected Mode IPoIB driver ------------------------------ The revised ibd(7D) driver will support both Connected and Datagram mode. The features from the current Datagram mode ibd driver will be inherited. The remainder of this section discusses interface additions for the Connected mode capable driver. B.1 Switching between datagram and connected mode The existing ibd driver in OpenSolaris and Solaris 10 does not ship with a driver .conf file. However, the Connected Mode support described in this case introduces a new parameter 'enable_rc' that may be set via the ibd driver .conf file. This parameter specifies whether each ibd instance defaults to using Connected Mode over RC or not. # 1: unicast packets will be sent over Reliable Connected Mode # 0: unicast packets will be sent over Unreliable Datagram Mode # # Each element in the list below maps to the corresponding ibd # instance; the first element is for ibd instance 0, the second # element is for instance 1 and so on. # enable_rc=1,1,0,0; Please note that Connected Mode support in IPoIB is optional as per [5]. Therefore, if Connected Mode is not available for a remote node, the Datagram mode will automatically be used for that destination by the ibd driver. Therefore, the only meaning of 'enable_rc' is to decide whether to try Connected Mode first or not, and whether to advertise this as a capability supported by this instance or not. The default value for 'enable_rc' for each instance is 0. Hence without a ibd.conf file, Datagram mode will be used. We intend to ship a driver .conf file for ibd in ONNV (and hence OpenSolaris) with enable_rc set to all ones (enabling Connected Mode by default on all instances) for the best performance. However, for Solaris 10, we have received business guidance to have an "opt-in" approach due to a desire for greater stability in established enterprise environments. We will do this by not shipping the .conf file. Therefore, by default Solaris 10 will be Datagram mode. It will take an explicit administrator action (setting enable_rc) to cause Solaris 10 to use Connected Mode. OFED (Linux IB) originally made Connected Mode opt-in too. However, later OFED made it the default. We don't intend to change it later to be the default in Solaris 10. However, Solaris Next, being descended from ONNV, will have it as default. An edited ibd(7D) manpage documenting this change is in the materials directory. B.2 Change of default MTU size Connected Mode by virtue of using the RC transport service type offers link MTUs of up to 2^31-4 octets in length. Thus, the use of Connected Mode can offer benefits by supporting very large MTUs. Datagram Mode using UD is limited to 4092 (4K-4) octets, though commonly only 2044 (2K-4) is offered. Due to the limits of the TCP/IP protocol, it makes sense to only offer up to 65535 (64K-1) bytes. OFED (i.e. Linux IB) uses 65520 (64K-16) byte MTU for alignment reasons. To inter-operate with OFED at the best performance, we also adopt 65520 as the default MTU of the Connected Mode. C. Interfaces ------------- +-------------------------------------------------------------------+ | Interfaces Exported | +---------------------------+------------------+--------------------+ | Interface Name | Classification | Comment | +---------------------------+------------------+--------------------+ |/kernel/drv/ibd.conf* | Volatile | Configuration file | +---------------------------+------------------+--------------------+ * = only for OpenSolaris D. References ------------- [1] IP over InfiniBand, PSARC/2001/289 [2] IPoIB Conversion to GLDv3, PSARC/2007/636 [3] InfiniBand Architecture Specification Volume 1, Release 1.2.1, InfiniBand Trade Association, 2007. http://www.infinibandta.org/content/pages.php?pg=technology_download [4] Transmission of IP over InfiniBand (IPoIB), RFC 4391, IETF, http://www.ietf.org/rfc/rfc4391.txt [5] IP over InfiniBand: Connected Mode, RFC 4755, IETF, http://www.ietf.org/rfc/rfc4755.txt --Boundary_(ID_qgAlT1bnlpuL8FLLsDtpsg)-- From ted.h.kim@oracle.com Fri May 21 10:17:43 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o4LHHhSI009453 for ; Fri, 21 May 2010 10:17:43 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o4LHHgtC002665 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Fri, 21 May 2010 12:17:42 -0500 (CDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L2S00A055DIK500@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Fri, 21 May 2010 11:17:42 -0600 (MDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L2S00KVD5DI5J90@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Fri, 21 May 2010 11:17:42 -0600 (MDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o4LHHfLB016486 for ; Fri, 21 May 2010 17:17:42 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4LAspCF005085 for ; Fri, 21 May 2010 17:17:41 +0000 (GMT) Received: from abhmt017.oracle.com by acsmt354.oracle.com with ESMTP id 288625121274462163; Fri, 21 May 2010 10:16:03 -0700 Received: from [129.153.3.74] (/129.153.3.74) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 21 May 2010 10:16:02 -0700 Date: Fri, 21 May 2010 10:15:45 -0700 From: "Ted H. Kim" Subject: case note for PSARC/2009/593 IPoIB Connected Mode To: psarc-ext@sun.com Message-id: <4BF6BFC1.5060808@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090208.4BF6C035.0129:SCFMA4539814,ss=1,fgs=0 User-Agent: Thunderbird 2.0.0.24 (X11/20100302) Status: RO Content-Length: 1043 Folks, Here is a minor amendment to an already approved case. I am filing this as a "case note", but in case anyone thinks this needs a real fasttrack, please pipe up, and I will re-file it as such. In PSARC/2009/593 IPoIB Connected Mode, the case said S10 would default to Datagram Mode by not delivering the .conf file, which implicitly gives a value of zero to each instance for the "enable_rc" property. Well as it turns out, the project team would rather deliver the .conf file, but with the property explicitly set to Datagram Mode (zero value for each instance). So the default setting is exactly the same, just the .conf file is delivered. This would make it slightly easier to switch to Connected Mode, if so desired, as the sysadmin is editing a file where the property is already mentioned, so there is less chance of a typo. -ted -- Ted H. Kim, PhD Sun/Oracle America Inc. ted.h.kim@oracle.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1115 FAX From james.gates@oracle.com Fri May 21 11:00:29 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o4LI0T1Y010543 for ; Fri, 21 May 2010 11:00:29 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o4LI0RYG053073 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Fri, 21 May 2010 12:00:29 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L2S0042L7CSTI00@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Fri, 21 May 2010 11:00:28 -0700 (PDT) Received: from sca-ea-mail-2.sun.com ([192.18.43.25]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L2S005747CRVBB0@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Fri, 21 May 2010 11:00:27 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by sca-ea-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o4LI0RoC002901 for ; Fri, 21 May 2010 18:00:27 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4LHPwEj030465 for ; Fri, 21 May 2010 18:00:26 +0000 (GMT) Received: from abhmt002.oracle.com by acsmt354.oracle.com with ESMTP id 288749911274464821; Fri, 21 May 2010 11:00:21 -0700 Received: from [129.150.81.30] (/129.150.81.30) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 21 May 2010 11:00:21 -0700 Date: Fri, 21 May 2010 14:00:16 -0400 From: James Gates Subject: Re: case note for PSARC/2009/593 IPoIB Connected Mode In-reply-to: <4BF6BFC1.5060808@oracle.com> To: "Ted H. Kim" Cc: psarc-ext@sun.com Message-id: <4BF6CA30.8010708@oracle.com> Organization: Oracle Corp. MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090209.4BF6CA3B.0028:SCFMA4539814,ss=1,fgs=0 References: <4BF6BFC1.5060808@oracle.com> User-Agent: Thunderbird 2.0.0.21 (X11/20090323) Status: RO Content-Length: 1252 In the original case, ibd.conf was classified as 'volatile' for Nevada (because it's expected to be replaced by switching to Brussels). That classification isn't appropriate for S10 if you intend to keep ibd.conf as the only configuration interface (which I assume is the case). So it's classification on S10 should be 'committed'? Ted H. Kim wrote: > Folks, > > Here is a minor amendment to an already approved case. > I am filing this as a "case note", but in case > anyone thinks this needs a real fasttrack, please > pipe up, and I will re-file it as such. > > In PSARC/2009/593 IPoIB Connected Mode, the case said > S10 would default to Datagram Mode by not delivering > the .conf file, which implicitly gives a value of > zero to each instance for the "enable_rc" property. > > Well as it turns out, the project team would rather > deliver the .conf file, but with the property explicitly > set to Datagram Mode (zero value for each instance). > So the default setting is exactly the same, just > the .conf file is delivered. > > This would make it slightly easier to switch to > Connected Mode, if so desired, as the sysadmin is > editing a file where the property is already mentioned, > so there is less chance of a typo. > > -ted > From ted.h.kim@oracle.com Fri May 21 11:06:50 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o4LI6nmZ010630 for ; Fri, 21 May 2010 11:06:49 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o4LI6n3k056849 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Fri, 21 May 2010 12:06:49 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L2S00E077NCUE00@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Fri, 21 May 2010 12:06:48 -0600 (MDT) Received: from sca-ea-mail-2.sun.com ([192.18.43.25]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L2S00K7E7NB5JD0@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Fri, 21 May 2010 12:06:47 -0600 (MDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by sca-ea-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o4LI6kFV005556 for ; Fri, 21 May 2010 18:06:47 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4L73JKw010059 for ; Fri, 21 May 2010 18:06:46 +0000 (GMT) Received: from abhmt001.oracle.com by acsmt354.oracle.com with ESMTP id 288771491274465191; Fri, 21 May 2010 11:06:31 -0700 Received: from [129.153.3.74] (/129.153.3.74) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 21 May 2010 11:06:31 -0700 Date: Fri, 21 May 2010 11:06:13 -0700 From: "Ted H. Kim" Subject: Re: case note for PSARC/2009/593 IPoIB Connected Mode In-reply-to: <4BF6CA30.8010708@oracle.com> To: James Gates Cc: psarc-ext@sun.com Message-id: <4BF6CB95.6010204@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090206.4BF6CBB6.011F:SCFMA4539814,ss=1,fgs=0 References: <4BF6BFC1.5060808@oracle.com> <4BF6CA30.8010708@oracle.com> User-Agent: Thunderbird 2.0.0.24 (X11/20100302) Status: RO Content-Length: 1697 In Nevada (snv_139), it was already replaced by Brussels. But you are right, the team doesn't intend to backport the Brussels stuff. So the .conf file becomes "committed" for S10. -ted James Gates wrote: > In the original case, ibd.conf was classified as 'volatile' for Nevada > (because it's expected to be replaced by switching to Brussels). > > That classification isn't appropriate for S10 if you intend to keep > ibd.conf as the only configuration interface (which I assume is the case). > > So it's classification on S10 should be 'committed'? > > > Ted H. Kim wrote: >> Folks, >> >> Here is a minor amendment to an already approved case. >> I am filing this as a "case note", but in case >> anyone thinks this needs a real fasttrack, please >> pipe up, and I will re-file it as such. >> >> In PSARC/2009/593 IPoIB Connected Mode, the case said >> S10 would default to Datagram Mode by not delivering >> the .conf file, which implicitly gives a value of >> zero to each instance for the "enable_rc" property. >> >> Well as it turns out, the project team would rather >> deliver the .conf file, but with the property explicitly >> set to Datagram Mode (zero value for each instance). >> So the default setting is exactly the same, just >> the .conf file is delivered. >> >> This would make it slightly easier to switch to >> Connected Mode, if so desired, as the sysadmin is >> editing a file where the property is already mentioned, >> so there is less chance of a typo. >> >> -ted >> -- Ted H. Kim, PhD Sun/Oracle America Inc. ted.h.kim@oracle.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1115 FAX