From kais@sac.sfbay.sun.com Fri Sep 18 12:29:40 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n8IJTdhp028886 for ; Fri, 18 Sep 2009 12:29:40 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n8IJTame017741; Fri, 18 Sep 2009 20:29:39 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KQ600I01M5DBC00@nwk-avmta-1.sfbay.Sun.COM>; Fri, 18 Sep 2009 12:29:37 -0700 (PDT) Received: from dm-sfbay-01.sfbay.sun.com ([129.145.155.118]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KQ6004KXM5DVK70@nwk-avmta-1.sfbay.Sun.COM>; Fri, 18 Sep 2009 12:29:37 -0700 (PDT) Received: from sac.sfbay.sun.com (sac.SFBay.Sun.COM [129.146.226.132]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n8IJTbto028303; Fri, 18 Sep 2009 12:29:37 -0700 (PDT) Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n8IJTZiU028394; Fri, 18 Sep 2009 12:29:35 -0700 (PDT) Received: (from kais@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id n8IJTZ1i028359; Fri, 18 Sep 2009 12:29:35 -0700 (PDT) Date: Fri, 18 Sep 2009 12:29:35 -0700 (PDT) From: Kais Belgaied Subject: Dynamic Ring Grouping on NICs [PSARC/2009/501 FastTrack timeout 09/25/2009] To: PSARC-ext@sun.com Cc: crossbow-core@sun.com, venu.iyer@sun.com Message-id: <200909181929.n8IJTZ1i028359@sac.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 10449 Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: Dynamic Ring Grouping on NICs 1.2. Name of Document Author/Supplier: Author: Venu Iyer 1.3 Date of This Document: 18 September, 2009 4. Technical Description I'm filing this fasttrack for Venu Iyer. The release binding is patch. The interface taxonomy is Uncommitted Background ========== Project Crossbow (PSARC/2006/357) enables creating hardware-based MAC clients (some of these MAC clients are data links such as VNICs) both on the RX and TX side. We define hardware-based MAC clients as having dedicated hardware resources; a RX hardware-based MAC client will have one or more RX ring for exclusive use while a TX hardware-based MAC client will have one or more TX ring for exclusive use. MAC clients that are not hardware-based (RX or TX) share hardware resources with other MAC clients, such MAC clients will not have any TX or RX rings exclusively reserved for them. MAC clients may be hardware-based on RX, but not on TX (and vice-versa). Currently, when a NIC registers with MAC it informs MAC if it supports dedicated hardware RX or TX rings. MAC assigns hardware rings to MAC clients as groups, where a group may contain 1 or more hardware rings. dladm show-phys is currently used to show how RX rings are used by MAC clients. # dladm show-phys -H nxge4 LINK GROUP GROUPTYPE RINGS CLIENTS nxge4 0 RX 3 nxge4 nxge4 1 RX 1 vnic1 which says we have 1 RX hardware-based MAC client - vnic1 with 1 ring. nxge4, the primary MAC client, is using 3 rings, but will share these with any other MAC client that is subsequently created on the data link nxge4 (i.e. if vnic2 is created on nxge4, MAC clients vnic2 and nxge4 will share group 1, and hence the 3 rings), eg: # dladm show-phys -H nxge4 LINK GROUP GROUPTYPE RINGS CLIENTS nxge4 0 RX 3 nxge4,vnic2 nxge4 1 RX 1 vnic1 Information about TX rings is not shown by the show-phys subcommand. Today, an administrator can specify that a VNIC must be hardware-based on the RX side (using the -H option to dladm create-vnic). However, there is no way for an administrator to specify o that a MAC client (VNIC or primary MAC client) should be software based, i.e. should not have any dedicated hardware resource, o that a MAC client should be hardware or software based on TX. o the number of RX or TX rings needed for a MAC client. Proposal ======== This proposal gives administrative control over whether a MAC client should be hardware-based or not (RX and TX) and also allows them to specify the number of RX or TX rings that a MAC client needs, if it is hardware-based. We introduce two properties for a link: rxringcnt: The number of RX rings needed. txringcnt: The number of TX rings needed. The values for these properties could be: 0 : This link must not assigned any hardware rings of the specified type. x > 0 : This link needs x rings. If the property is not specified for a link, the system will attempt to maxmize the hardware resource utilization by making this MAC client hardware-based depending on rings availability. E.g: # dladm create-vnic -p rxringcnt=0 -l nxge0 vnic1 Will create vnic1 which will not be RX hardware-based. # dladm create-vnic -p txringcnt=2 -l nxge0 vnic2 Will create vnic2 that will be TX hardware-based with 2 TX rings. # dladm create-vnic -p rxringcnt=2,txringcnt=2 vnic3 Will create vnic3 which will be both RX and TX hardware-based with 2 RX and TX rings resp. Modifying the RX or TX rings assigned to an existing link, say nxge0, can be done using set-linkprop, e.g. if nxge0 needs to be given 2 RX rings: # dladm set-linkprop -p rxringcnt=2 nxge0 or for a VNIC, say vnic1, as: # dladm set-linkprop -p txringcnt=2 vnic1 The rings assigned to a link can be viewed using show-linkprop as: # dladm show-linkprop nxge0 LINK PROPERTY PERM VALUE DEFAULT POSSIBLE ... nxge0 rxringscnt rw 2 -- 0-4 nxge0 txringscnt rw 5 -- 0-6 ... These new properties obsolete the -H option of dladm create-vnic (i.e. the -H option will be removed). Given that we allow specifying RX and TX rings for links, we need a way to display how many rings are available for use. Additionally, we need to provide the number of hardware-based MAC clients that can be created on the RX and TX side. We introduce 4 additional read-only properties to display this information: rxringavailcnt: The total number of RX rings available for use, i.e. not exclusively given to any MAC client. txringavailcnt: The total number of TX rings available for use. rxhwavailclnt: The total number of additional RX hardware-based MAC clients that can be created. Each of these hardware-based MAC client could have one or more RX rings assigned to them. txhwavailclnt: The total number of additional TX hardware-based MAC clients that can be created. Each of these hardware-based MAC clients could have one or more TX rings assigned to them. The counts reflects the current utilization/availability of the resources listed. The counts change as hardware-based MAC clients are created or destroyed or when their hardware properties (i.e. rxringscnt and txringscnt) are changed. e.g: # dladm show-linkprop nxge0 LINK PROPERTY PERM VALUE DEFAULT POSSIBLE ... nxge0 rxringavailcnt r- 3 -- 0-4 nxge0 txringavailcnt r- 5 -- 0-6 nxge0 rxhwavailclnt r- 1 -- 0-1 nxge0 txhwavailclnt r- 5 -- 0-5 ... This indicates that there are 3 RX ring available (out of a maximum of 4) and 5 TX ring available out of 6. Additionally only one more RX hardware based MAC client can be created while 5 more TX hardware-based MAC clients can be created. We provide two separate counters (rxhwavailclnt and txhwavailclnt) because the hardware virtualizes on RX based on groups of rings (i.e. MAC address is added to a group) and to keep it consistent on the TX we do the same. dladm show-phys will be modified to display the ring information on TX and RX. e.g: # dladm show-phys -H nxge0 LINK RINGTYPE RINGS CLIENTS nxge0 RX 0 nxge0 TX 0,5 vnic1 nxge0 RX 1-3 vnic1 which means vnic1 has exclusive use of 3 RX rings and 5 TX rings. --------------------------------- n. ! dladm show-phys [-P] [[-p] -o field[,...]] [phys-link] Show the physical device and attributes of all physical links, or of the named physical link. Without -P, only --- 414,420 ---- tion. ! dladm show-phys [-P] [[-p] -o field[,...]] [-H] [phys-link] Show the physical device and attributes of all physical links, or of the named physical link. Without -P, only *************** *** 421,426 **** --- 421,445 ---- physical links that are available on the running system are displayed. + -H + Show hardware resource usage, as returned by the NIC + driver. Output from -H displays the following elements: + + LINK + A physical device corresponding to a NIC driver. + + RINGTYPE + The type of the ring. This is either RX or TX. + + RINGS + The ring index. A ring is an hardware resource, which + typically maps to a DMA channel, that can be programmed + for specific use. E.g. an RX ring can be programmed to + receive only packets belonging to a specific MAC address. + + CLIENTS + MAC clients using the rings. + -o field, --output=field A case-insensitive, comma-separated list of output *************** *** 499,505 **** purge the link's configuration from the system. - dladm create-aggr [-t] [-R root-dir] [-P policy] [-L mode] [-T time] [-u address] -l ether-link1 [-l ether-link2...] aggr-link --- 518,523 ---- *************** *** 2432,2440 **** --- 2450,2494 ---- default is high. + rxringavailcnt + A read-only property that specifies the number of rings available + on the receive side. + rxringcnt + Specifies the number of receive rings side for the MAC client. + A value of 0 means this MAC client should not be assigned any RX + ring. A non-0 value means reserve that many rings for this MAC + client, if available, and fail if not. If this property is not + specified the MAC client may get one RX ring, if available, or + will be software based. + + rxhwavailclnt + + A read-only property that specifies the number of additional + RX hardware-based MAC clients that can be created. + + txringavailcnt + + A read-only property that specifies the number of rings available + on the transmit side. + + txringcnt + + Specifies the number of transmit rings for the MAC client. + A value of 0 means this MAC client should not be assigned need any + TX ring. A non-0 value means reserve that many rings for this MAC + client, if available, and fail if not. If this property is not + specified the MAC client may get one TX ring, if available, or + will be software based. + + txhwavailclnt + + A read-only property that specifics the number of additional + TX hardware-based MAC clients that can be created. + + SunOS 5.11 Last change: 16 Mar 2009 37 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From Darren.Reed@sun.com Mon Sep 21 19:26:33 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n8M2QWLT003720 for ; Mon, 21 Sep 2009 19:26:33 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n8M2QUD3063235; Mon, 21 Sep 2009 20:26:32 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KQC0050FPG7MQ00@nwk-avmta-2.sfbay.sun.com>; Mon, 21 Sep 2009 19:26:31 -0700 (PDT) Received: from gmp-eb-inf-2.sun.com ([192.18.6.24]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KQC0032KPG58F10@nwk-avmta-2.sfbay.sun.com>; Mon, 21 Sep 2009 19:26:30 -0700 (PDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-1-fe1.eu.sun.com [192.18.6.7] (may be forged)) by gmp-eb-inf-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n8M2QTYB005372; Tue, 22 Sep 2009 02:26:29 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KQC00800PFDD400@fe-emea-10.sun.com>; Tue, 22 Sep 2009 03:26:09 +0100 (BST) Received: from [129.146.106.55] ([unknown] [129.146.106.55]) by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KQC00KQNPFJI270@fe-emea-10.sun.com>; Tue, 22 Sep 2009 03:26:09 +0100 (BST) Date: Mon, 21 Sep 2009 19:26:07 -0700 From: Darren Reed Subject: Re: Dynamic Ring Grouping on NICs [PSARC/2009/501 FastTrack timeout 09/25/2009] In-reply-to: <200909181929.n8IJTZ1i028359@sac.sfbay.sun.com> Sender: Darren.Reed@sun.com To: Kais Belgaied Cc: PSARC-ext@sun.com, crossbow-core@sun.com, Venu.Iyer@sun.com Message-id: <4AB835BF.8020802@Sun.COM> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200909181929.n8IJTZ1i028359@sac.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.21 (X11/20090608) Status: RO Content-Length: 2104 On 18/09/09 12:29 PM, Kais Belgaied wrote: > ... > > + rxringavailcnt > > + A read-only property that specifies the number of rings available > + on the receive side. > > + rxringcnt > > + Specifies the number of receive rings side for the MAC client. > + A value of 0 means this MAC client should not be assigned any RX > + ring. A non-0 value means reserve that many rings for this MAC > + client, if available, and fail if not. If this property is not > + specified the MAC client may get one RX ring, if available, or > + will be software based. > + > + rxhwavailclnt > + > + A read-only property that specifies the number of additional > + RX hardware-based MAC clients that can be created. > + > + txringavailcnt > + > + A read-only property that specifies the number of rings available > + on the transmit side. > + > + txringcnt > + > + Specifies the number of transmit rings for the MAC client. > + A value of 0 means this MAC client should not be assigned need any > + TX ring. A non-0 value means reserve that many rings for this MAC > + client, if available, and fail if not. If this property is not > + specified the MAC client may get one TX ring, if available, or > + will be software based. > + > + txhwavailclnt > + > + A read-only property that specifics the number of additional > + TX hardware-based MAC clients that can be created. > + > + > The only comment I have is the naming, specifically the "cnt" and "clnt" on the end, seems ... I don't know... awkward/cumbersome? For example, "rxringavailclnt" and "rxringcnt" both are associated with mac client but only one mentions "clnt" in its name. Additionally, they are all a "count" of something, so "cnt" should be present on all, right? I'd like to suggest thinking of simpler names that do not include redundant information such as "clnt" and "cnt." For example, if "rxringcnt" became "rxrings", is any meaning really lost? Darren From venu@sun.com Tue Sep 22 07:39:53 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n8MEdq5u025553 for ; Tue, 22 Sep 2009 07:39:53 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n8MEdc5v024564; Tue, 22 Sep 2009 22:39:51 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KQD00F0FNEC5L00@brm-avmta-1.central.sun.com>; Tue, 22 Sep 2009 08:39:48 -0600 (MDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KQD00CYUNEBHZ20@brm-avmta-1.central.sun.com>; Tue, 22 Sep 2009 08:39:47 -0600 (MDT) Received: from amchi-mumbai (amchi-mumbai.SFBay.Sun.COM [129.146.106.6]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n8MEdkl4338005 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 22 Sep 2009 07:39:46 -0700 (PDT) Date: Tue, 22 Sep 2009 07:52:03 -0700 (PDT) From: venugopal iyer Subject: Re: Dynamic Ring Grouping on NICs [PSARC/2009/501 FastTrack timeout 09/25/2009] In-reply-to: <4AB835BF.8020802@Sun.COM> X-X-Sender: vi117747@amchi-mumbai To: Darren Reed Cc: Kais Belgaied , PSARC-ext@sun.com, crossbow-core@sun.com Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; charset=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200909181929.n8IJTZ1i028359@sac.sfbay.sun.com> <4AB835BF.8020802@Sun.COM> Status: RO Content-Length: 2741 Thanks, Darren. On Mon, 21 Sep 2009, Darren Reed wrote: > On 18/09/09 12:29 PM, Kais Belgaied wrote: >> ... >> >> + rxringavailcnt >> >> + A read-only property that specifies the number of rings >> available >> + on the receive side. >> >> + rxringcnt >> >> + Specifies the number of receive rings side for the MAC client. >> + A value of 0 means this MAC client should not be assigned any RX >> + ring. A non-0 value means reserve that many rings for this MAC >> + client, if available, and fail if not. If this property is not >> + specified the MAC client may get one RX ring, if available, or >> + will be software based. >> + >> + rxhwavailclnt >> + >> + A read-only property that specifies the number of additional >> + RX hardware-based MAC clients that can be created. >> + >> + txringavailcnt >> + >> + A read-only property that specifies the number of rings >> available >> + on the transmit side. >> + >> + txringcnt >> + >> + Specifies the number of transmit rings for the MAC client. >> + A value of 0 means this MAC client should not be assigned need any >> + TX ring. A non-0 value means reserve that many rings for this MAC >> + client, if available, and fail if not. If this property is not >> + specified the MAC client may get one TX ring, if available, or >> + will be software based. >> + >> + txhwavailclnt >> + >> + A read-only property that specifics the number of additional >> + TX hardware-based MAC clients that can be created. >> + >> + >> > > The only comment I have is the naming, specifically the "cnt" and "clnt" on > the end, seems ... I don't know... awkward/cumbersome? > > For example, "rxringavailclnt" and "rxringcnt" both are associated with mac > client but only one mentions "clnt" in its name. I think you mean rxhwavailclnt and rxringcnt, right? If so, then rxhwavailclnt is the number of clients that can be created and we wanted to have the client in the name, I can add cnt to rxhwavailclnt if that makes it any better. > > Additionally, they are all a "count" of something, so "cnt" should be present > on all, right? yes, they are counts. As mentioned above if rxhwavailclntcnt and txhwavailclntcnt makes it better, I am fine with it. > > I'd like to suggest thinking of simpler names that do not include redundant > information such as "clnt" and "cnt." > > For example, if "rxringcnt" became "rxrings", is any meaning really lost? We started off with rxrings, but wanted to make it explicit that it is the number or rx rings and not, say, as ring index. thanks, -venu > > Darren > > From Darren.Reed@sun.com Tue Sep 22 11:36:28 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n8MIaRjp018494 for ; Tue, 22 Sep 2009 11:36:27 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n8MIaMYW019494; Tue, 22 Sep 2009 12:36:27 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KQD0000RYCQHF00@nwk-avmta-2.sfbay.sun.com>; Tue, 22 Sep 2009 11:36:26 -0700 (PDT) Received: from gmp-eb-inf-2.sun.com ([192.18.6.24]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KQD00IKSYCPGL70@nwk-avmta-2.sfbay.sun.com>; Tue, 22 Sep 2009 11:36:26 -0700 (PDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-1-fe1.eu.sun.com [192.18.6.7] (may be forged)) by gmp-eb-inf-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n8MIaOcW018129; Tue, 22 Sep 2009 18:36:24 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KQD00500Y2ZV100@fe-emea-10.sun.com>; Tue, 22 Sep 2009 19:36:09 +0100 (BST) Received: from [76.191.212.119] ([unknown] [76.191.212.119]) by fe-emea-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KQD004T6YC4CA10@fe-emea-10.sun.com>; Tue, 22 Sep 2009 19:36:05 +0100 (BST) Date: Tue, 22 Sep 2009 11:36:01 -0700 From: Darren Reed Subject: Re: Dynamic Ring Grouping on NICs [PSARC/2009/501 FastTrack timeout 09/25/2009] In-reply-to: Sender: Darren.Reed@sun.com To: venugopal iyer Cc: Kais Belgaied , PSARC-ext@sun.com, crossbow-core@sun.com Message-id: <4AB91911.4060503@Sun.COM> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200909181929.n8IJTZ1i028359@sac.sfbay.sun.com> <4AB835BF.8020802@Sun.COM> User-Agent: Thunderbird 2.0.0.23 (Windows/20090812) Status: RO Content-Length: 3409 venugopal iyer wrote: > > Thanks, Darren. > > On Mon, 21 Sep 2009, Darren Reed wrote: > >> On 18/09/09 12:29 PM, Kais Belgaied wrote: >>> ... >>> >>> + rxringavailcnt >>> >>> + A read-only property that specifies the number of rings >>> available >>> + on the receive side. >>> >>> + rxringcnt >>> >>> + Specifies the number of receive rings side for the MAC >>> client. >>> + A value of 0 means this MAC client should not be assigned >>> any RX >>> + ring. A non-0 value means reserve that many rings for this >>> MAC >>> + client, if available, and fail if not. If this property is not >>> + specified the MAC client may get one RX ring, if available, or >>> + will be software based. >>> + >>> + rxhwavailclnt >>> + >>> + A read-only property that specifies the number of additional >>> + RX hardware-based MAC clients that can be created. >>> + >>> + txringavailcnt >>> + >>> + A read-only property that specifies the number of rings >>> available >>> + on the transmit side. >>> + >>> + txringcnt >>> + >>> + Specifies the number of transmit rings for the MAC client. >>> + A value of 0 means this MAC client should not be assigned >>> need any >>> + TX ring. A non-0 value means reserve that many rings for >>> this MAC >>> + client, if available, and fail if not. If this property is not >>> + specified the MAC client may get one TX ring, if available, or >>> + will be software based. >>> + >>> + txhwavailclnt >>> + >>> + A read-only property that specifics the number of additional >>> + TX hardware-based MAC clients that can be created. >>> + >>> + >>> >> >> The only comment I have is the naming, specifically the "cnt" and >> "clnt" on the end, seems ... I don't know... awkward/cumbersome? >> >> For example, "rxringavailclnt" and "rxringcnt" both are associated >> with mac client but only one mentions "clnt" in its name. > > I think you mean rxhwavailclnt and rxringcnt, right? > If so, then rxhwavailclnt is the number of clients that can be created > and we > wanted to have the client in the name, I can add cnt to rxhwavailclnt > if that > makes it any better. > >> >> Additionally, they are all a "count" of something, so "cnt" should be >> present on all, right? > > yes, they are counts. As mentioned above if rxhwavailclntcnt and > txhwavailclntcnt makes it better, I am fine with it. And that is the point, these names look awful :-( For example, if you look through any of statistics in kstat, you generally don't see "cnt" or "count". >> I'd like to suggest thinking of simpler names that do not include >> redundant information such as "clnt" and "cnt." >> >> For example, if "rxringcnt" became "rxrings", is any meaning really >> lost? > > We started off with rxrings, but wanted to make it explicit that it is > the number or rx rings and not, say, as ring index. If it was just "rxring", then perhaps I would agree that maybe it might be something else... But, for example, we have "Inbound Packets", not "Inbound Packet Count" in "netstat -i" output. To me, the label "rxrings" does not imply it could be about anything else but rx rings. If it were to be about an index, then it would be "rxringindex" or "rxringindexes" - the name becomes more specific. Darren From venu@sun.com Tue Sep 22 12:34:09 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n8MJY94L019887 for ; Tue, 22 Sep 2009 12:34:09 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n8MJY3ct028827; Tue, 22 Sep 2009 12:34:07 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KQE00K1910ULZ00@brm-avmta-1.central.sun.com>; Tue, 22 Sep 2009 13:34:07 -0600 (MDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KQE004D110UGPE0@brm-avmta-1.central.sun.com>; Tue, 22 Sep 2009 13:34:06 -0600 (MDT) Received: from amchi-mumbai (amchi-mumbai.SFBay.Sun.COM [129.146.106.6]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n8MJY52v384032 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 22 Sep 2009 12:34:05 -0700 (PDT) Date: Tue, 22 Sep 2009 12:46:24 -0700 (PDT) From: venugopal iyer Subject: Re: Dynamic Ring Grouping on NICs [PSARC/2009/501 FastTrack timeout 09/25/2009] In-reply-to: <4AB91911.4060503@Sun.COM> X-X-Sender: vi117747@amchi-mumbai To: Darren Reed Cc: PSARC-ext@sun.com, crossbow-core@sun.com Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; charset=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200909181929.n8IJTZ1i028359@sac.sfbay.sun.com> <4AB835BF.8020802@Sun.COM> <4AB91911.4060503@Sun.COM> Status: RO Content-Length: 3679 On Tue, 22 Sep 2009, Darren Reed wrote: > venugopal iyer wrote: >> >> Thanks, Darren. >> >> On Mon, 21 Sep 2009, Darren Reed wrote: >> >>> On 18/09/09 12:29 PM, Kais Belgaied wrote: >>>> ... >>>> >>>> + rxringavailcnt >>>> >>>> + A read-only property that specifies the number of rings >>>> available >>>> + on the receive side. >>>> >>>> + rxringcnt >>>> >>>> + Specifies the number of receive rings side for the MAC client. >>>> + A value of 0 means this MAC client should not be assigned any RX >>>> + ring. A non-0 value means reserve that many rings for this MAC >>>> + client, if available, and fail if not. If this property is not >>>> + specified the MAC client may get one RX ring, if available, or >>>> + will be software based. >>>> + >>>> + rxhwavailclnt >>>> + >>>> + A read-only property that specifies the number of additional >>>> + RX hardware-based MAC clients that can be created. >>>> + >>>> + txringavailcnt >>>> + >>>> + A read-only property that specifies the number of rings >>>> available >>>> + on the transmit side. >>>> + >>>> + txringcnt >>>> + >>>> + Specifies the number of transmit rings for the MAC client. >>>> + A value of 0 means this MAC client should not be assigned need >>>> any >>>> + TX ring. A non-0 value means reserve that many rings for this >>>> MAC >>>> + client, if available, and fail if not. If this property is not >>>> + specified the MAC client may get one TX ring, if available, or >>>> + will be software based. >>>> + >>>> + txhwavailclnt >>>> + >>>> + A read-only property that specifics the number of additional >>>> + TX hardware-based MAC clients that can be created. >>>> + >>>> + >>>> >>> >>> The only comment I have is the naming, specifically the "cnt" and "clnt" >>> on the end, seems ... I don't know... awkward/cumbersome? >>> >>> For example, "rxringavailclnt" and "rxringcnt" both are associated with >>> mac client but only one mentions "clnt" in its name. >> >> I think you mean rxhwavailclnt and rxringcnt, right? >> If so, then rxhwavailclnt is the number of clients that can be created and >> we >> wanted to have the client in the name, I can add cnt to rxhwavailclnt if >> that >> makes it any better. >> >>> >>> Additionally, they are all a "count" of something, so "cnt" should be >>> present on all, right? >> >> yes, they are counts. As mentioned above if rxhwavailclntcnt and >> txhwavailclntcnt makes it better, I am fine with it. > > And that is the point, these names look awful :-( > > For example, if you look through any of statistics in kstat, you generally > don't see "cnt" or "count". > > >>> I'd like to suggest thinking of simpler names that do not include >>> redundant information such as "clnt" and "cnt." >>> >>> For example, if "rxringcnt" became "rxrings", is any meaning really lost? >> >> We started off with rxrings, but wanted to make it explicit that it is >> the number or rx rings and not, say, as ring index. > > If it was just "rxring", then perhaps I would agree that maybe it might be > something else... > > But, for example, we have "Inbound Packets", not "Inbound Packet Count" in > "netstat -i" output. > > To me, the label "rxrings" does not imply it could be about anything else but > rx rings. If it were to be about an index, then it would be "rxringindex" or > "rxringindexes" - the name becomes more specific. So, do you prefer: rxrings/txrings rxringsavail/txringsavail rxhwavailclnt/txhwavailclnt thanks, -venu > > Darren > > From Darren.Reed@sun.com Tue Sep 22 12:42:09 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n8MJg8GX020050 for ; Tue, 22 Sep 2009 12:42:08 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n8MJg61M001052; Tue, 22 Sep 2009 12:42:08 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KQE00L0L1E7F100@brm-avmta-1.central.sun.com>; Tue, 22 Sep 2009 13:42:07 -0600 (MDT) Received: from gmp-eb-inf-1.sun.com ([192.18.6.21]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KQE004TU1E5GHD0@brm-avmta-1.central.sun.com>; Tue, 22 Sep 2009 13:42:06 -0600 (MDT) Received: from fe-emea-09.sun.com (gmp-eb-lb-1-fe1.eu.sun.com [192.18.6.7] (may be forged)) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n8MJg55o027103; Tue, 22 Sep 2009 19:42:05 +0000 (GMT) Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KQE0010015YVC00@fe-emea-09.sun.com>; Tue, 22 Sep 2009 20:41:52 +0100 (BST) Received: from [129.146.106.55] ([unknown] [129.146.106.55]) by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KQE00BX41DQBN20@fe-emea-09.sun.com>; Tue, 22 Sep 2009 20:41:52 +0100 (BST) Date: Tue, 22 Sep 2009 12:41:49 -0700 From: Darren Reed Subject: Re: Dynamic Ring Grouping on NICs [PSARC/2009/501 FastTrack timeout 09/25/2009] In-reply-to: Sender: Darren.Reed@sun.com To: venugopal iyer Cc: PSARC-ext@sun.com, crossbow-core@sun.com Message-id: <4AB9287D.6000907@Sun.COM> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200909181929.n8IJTZ1i028359@sac.sfbay.sun.com> <4AB835BF.8020802@Sun.COM> <4AB91911.4060503@Sun.COM> User-Agent: Thunderbird 2.0.0.21 (X11/20090608) Status: RO Content-Length: 3928 On 22/09/09 12:46 PM, venugopal iyer wrote: > > > > > On Tue, 22 Sep 2009, Darren Reed wrote: > >> venugopal iyer wrote: >>> >>> Thanks, Darren. >>> >>> On Mon, 21 Sep 2009, Darren Reed wrote: >>> >>>> On 18/09/09 12:29 PM, Kais Belgaied wrote: >>>>> ... >>>>> >>>>> + rxringavailcnt >>>>> >>>>> + A read-only property that specifies the number of rings >>>>> available >>>>> + on the receive side. >>>>> >>>>> + rxringcnt >>>>> >>>>> + Specifies the number of receive rings side for the MAC >>>>> client. >>>>> + A value of 0 means this MAC client should not be assigned >>>>> any RX >>>>> + ring. A non-0 value means reserve that many rings for >>>>> this MAC >>>>> + client, if available, and fail if not. If this property >>>>> is not >>>>> + specified the MAC client may get one RX ring, if >>>>> available, or >>>>> + will be software based. >>>>> + >>>>> + rxhwavailclnt >>>>> + >>>>> + A read-only property that specifies the number of >>>>> additional >>>>> + RX hardware-based MAC clients that can be created. >>>>> + >>>>> + txringavailcnt >>>>> + >>>>> + A read-only property that specifies the number of rings >>>>> available >>>>> + on the transmit side. >>>>> + >>>>> + txringcnt >>>>> + >>>>> + Specifies the number of transmit rings for the MAC client. >>>>> + A value of 0 means this MAC client should not be assigned >>>>> need any >>>>> + TX ring. A non-0 value means reserve that many rings for >>>>> this MAC >>>>> + client, if available, and fail if not. If this property >>>>> is not >>>>> + specified the MAC client may get one TX ring, if >>>>> available, or >>>>> + will be software based. >>>>> + >>>>> + txhwavailclnt >>>>> + >>>>> + A read-only property that specifics the number of >>>>> additional >>>>> + TX hardware-based MAC clients that can be created. >>>>> + >>>>> + >>>>> >>>> >>>> The only comment I have is the naming, specifically the "cnt" and >>>> "clnt" on the end, seems ... I don't know... awkward/cumbersome? >>>> >>>> For example, "rxringavailclnt" and "rxringcnt" both are associated >>>> with mac client but only one mentions "clnt" in its name. >>> >>> I think you mean rxhwavailclnt and rxringcnt, right? >>> If so, then rxhwavailclnt is the number of clients that can be >>> created and we >>> wanted to have the client in the name, I can add cnt to >>> rxhwavailclnt if that >>> makes it any better. >>> >>>> >>>> Additionally, they are all a "count" of something, so "cnt" should >>>> be present on all, right? >>> >>> yes, they are counts. As mentioned above if rxhwavailclntcnt and >>> txhwavailclntcnt makes it better, I am fine with it. >> >> And that is the point, these names look awful :-( >> >> For example, if you look through any of statistics in kstat, you >> generally don't see "cnt" or "count". >> >> >>>> I'd like to suggest thinking of simpler names that do not include >>>> redundant information such as "clnt" and "cnt." >>>> >>>> For example, if "rxringcnt" became "rxrings", is any meaning really >>>> lost? >>> >>> We started off with rxrings, but wanted to make it explicit that it is >>> the number or rx rings and not, say, as ring index. >> >> If it was just "rxring", then perhaps I would agree that maybe it >> might be something else... >> >> But, for example, we have "Inbound Packets", not "Inbound Packet >> Count" in "netstat -i" output. >> >> To me, the label "rxrings" does not imply it could be about anything >> else but rx rings. If it were to be about an index, then it would be >> "rxringindex" or "rxringindexes" - the name becomes more specific. > > So, do you prefer: > > rxrings/txrings much better. > rxringsavail/txringsavail much better > rxhwavailclnt/txhwavailclnt Is "clnt" required here? Darren From venu@sun.com Tue Sep 22 13:35:15 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n8MKZEJC021627 for ; Tue, 22 Sep 2009 13:35:15 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n8MKYWai010401; Tue, 22 Sep 2009 21:35:12 +0100 (BST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KQE0030B3UMJS00@brm-avmta-1.central.sun.com>; Tue, 22 Sep 2009 14:35:10 -0600 (MDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.63]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KQE000WS3ULBS30@brm-avmta-1.central.sun.com>; Tue, 22 Sep 2009 14:35:09 -0600 (MDT) Received: from amchi-mumbai (amchi-mumbai.SFBay.Sun.COM [129.146.106.6]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n8MKZ9F8394083 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 22 Sep 2009 13:35:09 -0700 (PDT) Date: Tue, 22 Sep 2009 13:47:28 -0700 (PDT) From: venugopal iyer Subject: Re: Dynamic Ring Grouping on NICs [PSARC/2009/501 FastTrack timeout 09/25/2009] In-reply-to: <4AB9287D.6000907@Sun.COM> X-X-Sender: vi117747@amchi-mumbai To: Darren Reed Cc: PSARC-ext@sun.com, crossbow-core@sun.com Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; charset=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200909181929.n8IJTZ1i028359@sac.sfbay.sun.com> <4AB835BF.8020802@Sun.COM> <4AB91911.4060503@Sun.COM> <4AB9287D.6000907@Sun.COM> Status: RO Content-Length: 4212 On Tue, 22 Sep 2009, Darren Reed wrote: > On 22/09/09 12:46 PM, venugopal iyer wrote: >> >> >> >> >> On Tue, 22 Sep 2009, Darren Reed wrote: >> >>> venugopal iyer wrote: >>>> >>>> Thanks, Darren. >>>> >>>> On Mon, 21 Sep 2009, Darren Reed wrote: >>>> >>>>> On 18/09/09 12:29 PM, Kais Belgaied wrote: >>>>>> ... >>>>>> >>>>>> + rxringavailcnt >>>>>> >>>>>> + A read-only property that specifies the number of rings >>>>>> available >>>>>> + on the receive side. >>>>>> >>>>>> + rxringcnt >>>>>> >>>>>> + Specifies the number of receive rings side for the MAC >>>>>> client. >>>>>> + A value of 0 means this MAC client should not be assigned any >>>>>> RX >>>>>> + ring. A non-0 value means reserve that many rings for this >>>>>> MAC >>>>>> + client, if available, and fail if not. If this property is not >>>>>> + specified the MAC client may get one RX ring, if available, or >>>>>> + will be software based. >>>>>> + >>>>>> + rxhwavailclnt >>>>>> + >>>>>> + A read-only property that specifies the number of additional >>>>>> + RX hardware-based MAC clients that can be created. >>>>>> + >>>>>> + txringavailcnt >>>>>> + >>>>>> + A read-only property that specifies the number of rings >>>>>> available >>>>>> + on the transmit side. >>>>>> + >>>>>> + txringcnt >>>>>> + >>>>>> + Specifies the number of transmit rings for the MAC client. >>>>>> + A value of 0 means this MAC client should not be assigned need >>>>>> any >>>>>> + TX ring. A non-0 value means reserve that many rings for this >>>>>> MAC >>>>>> + client, if available, and fail if not. If this property is not >>>>>> + specified the MAC client may get one TX ring, if available, or >>>>>> + will be software based. >>>>>> + >>>>>> + txhwavailclnt >>>>>> + >>>>>> + A read-only property that specifics the number of additional >>>>>> + TX hardware-based MAC clients that can be created. >>>>>> + >>>>>> + >>>>>> >>>>> >>>>> The only comment I have is the naming, specifically the "cnt" and "clnt" >>>>> on the end, seems ... I don't know... awkward/cumbersome? >>>>> >>>>> For example, "rxringavailclnt" and "rxringcnt" both are associated with >>>>> mac client but only one mentions "clnt" in its name. >>>> >>>> I think you mean rxhwavailclnt and rxringcnt, right? >>>> If so, then rxhwavailclnt is the number of clients that can be created >>>> and we >>>> wanted to have the client in the name, I can add cnt to rxhwavailclnt if >>>> that >>>> makes it any better. >>>> >>>>> >>>>> Additionally, they are all a "count" of something, so "cnt" should be >>>>> present on all, right? >>>> >>>> yes, they are counts. As mentioned above if rxhwavailclntcnt and >>>> txhwavailclntcnt makes it better, I am fine with it. >>> >>> And that is the point, these names look awful :-( >>> >>> For example, if you look through any of statistics in kstat, you generally >>> don't see "cnt" or "count". >>> >>> >>>>> I'd like to suggest thinking of simpler names that do not include >>>>> redundant information such as "clnt" and "cnt." >>>>> >>>>> For example, if "rxringcnt" became "rxrings", is any meaning really >>>>> lost? >>>> >>>> We started off with rxrings, but wanted to make it explicit that it is >>>> the number or rx rings and not, say, as ring index. >>> >>> If it was just "rxring", then perhaps I would agree that maybe it might be >>> something else... >>> >>> But, for example, we have "Inbound Packets", not "Inbound Packet Count" in >>> "netstat -i" output. >>> >>> To me, the label "rxrings" does not imply it could be about anything else >>> but rx rings. If it were to be about an index, then it would be >>> "rxringindex" or "rxringindexes" - the name becomes more specific. >> >> So, do you prefer: >> >> rxrings/txrings > > much better. > >> rxringsavail/txringsavail > > much better > >> rxhwavailclnt/txhwavailclnt > > Is "clnt" required here? It is the # of hw based clients, so I think it is helpful, if it makes any difference how about rxhwclntavail? -venu > > Darren > > From Sebastien.Roy@Sun.COM Mon Sep 28 12:10:39 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n8SJAcQq026970 for ; Mon, 28 Sep 2009 12:10:38 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n8SJAcgx011008; Mon, 28 Sep 2009 12:10:38 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KQP00C1D3XQEA00@nwk-avmta-2.sfbay.sun.com>; Mon, 28 Sep 2009 12:10:38 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KQP007ZN3XPCS60@nwk-avmta-2.sfbay.sun.com>; Mon, 28 Sep 2009 12:10:37 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n8SJAbBq020760; Mon, 28 Sep 2009 19:10:37 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KQP00B003M56W00@mail-amer.sun.com>; Mon, 28 Sep 2009 13:10:37 -0600 (MDT) Received: from [129.148.174.103] ([unknown] [129.148.174.103]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KQP001Q73XN2Z60@mail-amer.sun.com>; Mon, 28 Sep 2009 13:10:35 -0600 (MDT) Date: Mon, 28 Sep 2009 15:08:19 -0400 From: Sebastien Roy Subject: Re: Dynamic Ring Grouping on NICs [PSARC/2009/501 FastTrack timeout 09/25/2009] In-reply-to: <200909181929.n8IJTZ1i028359@sac.sfbay.sun.com> Sender: Sebastien.Roy@Sun.COM To: Kais Belgaied Cc: PSARC-ext@Sun.COM, crossbow-core@Sun.COM, Venu.Iyer@Sun.COM Message-id: <1254164899.12892.144.camel@strat> Organization: Sun Microsystems MIME-version: 1.0 X-Mailer: Evolution 2.26.3 Content-type: text/plain; CHARSET=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200909181929.n8IJTZ1i028359@sac.sfbay.sun.com> Status: RO Content-Length: 2825 On Fri, 2009-09-18 at 12:29 -0700, Kais Belgaied wrote: > Background > ========== > > Project Crossbow (PSARC/2006/357) enables creating hardware-based MAC > clients (some of these MAC clients are data links such as VNICs) both on > the RX and TX side. We define hardware-based MAC clients as having dedicated > hardware resources; a RX hardware-based MAC client will have one or more RX > ring for exclusive use while a TX hardware-based MAC client will have one or > more TX ring for exclusive use. MAC clients that are not hardware-based > (RX or TX) share hardware resources with other MAC clients, such MAC > clients will not have any TX or RX rings exclusively reserved for them. > MAC clients may be hardware-based on RX, but not on TX (and vice-versa). To me, that last sentence is essentially saying, "MAC clients may not be hardware-based on both RX and TX". Is that what you meant? If so, can you simply say that instead? Also, why? > Proposal > ======== > > This proposal gives administrative control over whether a MAC client > should be hardware-based or not (RX and TX) and also allows them to > specify the number of RX or TX rings that a MAC client needs, if it is > hardware-based. > > We introduce two properties for a link: > > rxringcnt: The number of RX rings needed. > txringcnt: The number of TX rings needed. > > The values for these properties could be: > > 0 : This link must not assigned any hardware rings of the > specified type. > > x > 0 : This link needs x rings. > > If the property is not specified for a link, the system will attempt > to maxmize the hardware resource utilization by making this MAC client > hardware-based depending on rings availability. This is slightly ambiguous to me. Does that mean that if I neglect to override the default, that the first MAC client gets all of the hardware resources while subsequent clients get none? I would think that unless otherwise specified, a sensible default behavior would result in evenly distributing resources among clients. > dladm show-phys will be modified to display the ring information on TX and > RX. > > e.g: > # dladm show-phys -H nxge0 > LINK RINGTYPE RINGS CLIENTS > nxge0 RX 0 > nxge0 TX 0,5 vnic1 > nxge0 RX 1-3 vnic1 > > which means vnic1 has exclusive use of 3 RX rings and 5 TX rings. I would interpret the above as "vnic1 has transmit rings 0 and 5" (because of the comma), and "vnic1 has receive rings 1, 2, and 3" (because of the hyphen). Am I reading that right? How do I view the total number of hardware resources available on nxge0? The above three lines only seem to show me what the clients are using. -Seb From venu@Sun.COM Mon Sep 28 12:25:06 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n8SJP6ga027475 for ; Mon, 28 Sep 2009 12:25:06 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n8SJP4EK009126; Mon, 28 Sep 2009 12:25:05 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KQP00H094LS5100@nwk-avmta-1.sfbay.Sun.COM>; Mon, 28 Sep 2009 12:25:04 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.63]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KQP00DHR4LSMKE0@nwk-avmta-1.sfbay.Sun.COM>; Mon, 28 Sep 2009 12:25:04 -0700 (PDT) Received: from amchi-mumbai (amchi-mumbai.SFBay.Sun.COM [129.146.106.6]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n8SJP4nO632949 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 28 Sep 2009 12:25:04 -0700 (PDT) Date: Mon, 28 Sep 2009 12:38:04 -0700 (PDT) From: venugopal iyer Subject: Re: Dynamic Ring Grouping on NICs [PSARC/2009/501 FastTrack timeout 09/25/2009] In-reply-to: <1254164899.12892.144.camel@strat> X-X-Sender: vi117747@amchi-mumbai To: Sebastien Roy Cc: Kais Belgaied , PSARC-ext@Sun.COM, crossbow-core@Sun.COM Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; charset=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200909181929.n8IJTZ1i028359@sac.sfbay.sun.com> <1254164899.12892.144.camel@strat> Status: RO Content-Length: 3724 Thanks, Seb. On Mon, 28 Sep 2009, Sebastien Roy wrote: > On Fri, 2009-09-18 at 12:29 -0700, Kais Belgaied wrote: >> Background >> ========== >> >> Project Crossbow (PSARC/2006/357) enables creating hardware-based MAC >> clients (some of these MAC clients are data links such as VNICs) both on >> the RX and TX side. We define hardware-based MAC clients as having dedicated >> hardware resources; a RX hardware-based MAC client will have one or more RX >> ring for exclusive use while a TX hardware-based MAC client will have one or >> more TX ring for exclusive use. MAC clients that are not hardware-based >> (RX or TX) share hardware resources with other MAC clients, such MAC >> clients will not have any TX or RX rings exclusively reserved for them. >> MAC clients may be hardware-based on RX, but not on TX (and vice-versa). > > To me, that last sentence is essentially saying, "MAC clients may not be > hardware-based on both RX and TX". Is that what you meant? No. It was just to note that it *need* not be h/w on both. How about "MAC clients could be hardware based on either TX, RX, both or neither" or some such. > If so, can you simply say that instead? Also, why? > >> Proposal >> ======== >> >> This proposal gives administrative control over whether a MAC client >> should be hardware-based or not (RX and TX) and also allows them to >> specify the number of RX or TX rings that a MAC client needs, if it is >> hardware-based. >> >> We introduce two properties for a link: >> >> rxringcnt: The number of RX rings needed. >> txringcnt: The number of TX rings needed. >> >> The values for these properties could be: >> >> 0 : This link must not assigned any hardware rings of the >> specified type. >> >> x > 0 : This link needs x rings. >> >> If the property is not specified for a link, the system will attempt >> to maxmize the hardware resource utilization by making this MAC client >> hardware-based depending on rings availability. > > This is slightly ambiguous to me. Does that mean that if I neglect to > override the default, that the first MAC client gets all of the hardware > resources while subsequent clients get none? That's not the case. Today, when we create a VNIC and don't specify the -H option we try to give it one ring (assuming we have free rings). That won't change, if we don't specify rxringcnt then we will try to give the new MAC client 1 ring and make it hardware based. We do this till there are free rings available. When there aren't any free rings, MAC clients that don't specify this property will be software based. > I would think that unless i otherwise specified, a sensible default behavior would result in evenly > distributing resources among clients. > >> dladm show-phys will be modified to display the ring information on TX and >> RX. >> >> e.g: >> # dladm show-phys -H nxge0 >> LINK RINGTYPE RINGS CLIENTS >> nxge0 RX 0 >> nxge0 TX 0,5 vnic1 >> nxge0 RX 1-3 vnic1 >> >> which means vnic1 has exclusive use of 3 RX rings and 5 TX rings. > > I would interpret the above as "vnic1 has transmit rings 0 and > 5" (because of the comma), and "vnic1 has receive rings 1, 2, and > 3" (because of the hyphen). Am I reading that right? yes. > > How do I view the total number of hardware resources available on nxge0? the read-only rings property (rxringavailcnt, txringavailcnt) we are introducing is for that purpose. let me know if I missed answering any of your questions. thanks! -venu > The above three lines only seem to show me what the clients are using. > > -Seb > > > From Sebastien.Roy@sun.com Mon Sep 28 12:35:23 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n8SJZNfX027977 for ; Mon, 28 Sep 2009 12:35:23 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n8SJZNVJ017749; Mon, 28 Sep 2009 12:35:23 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KQP00K0752ZBN00@brm-avmta-1.central.sun.com>; Mon, 28 Sep 2009 13:35:23 -0600 (MDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KQP00MC852Y9DE0@brm-avmta-1.central.sun.com>; Mon, 28 Sep 2009 13:35:22 -0600 (MDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n8SJZM4h022500; Mon, 28 Sep 2009 19:35:22 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KQP00F0043M0P00@mail-amer.sun.com>; Mon, 28 Sep 2009 13:35:22 -0600 (MDT) Received: from [129.148.174.103] ([unknown] [129.148.174.103]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KQP003K652X4460@mail-amer.sun.com>; Mon, 28 Sep 2009 13:35:22 -0600 (MDT) Date: Mon, 28 Sep 2009 15:33:06 -0400 From: Sebastien Roy Subject: Re: Dynamic Ring Grouping on NICs [PSARC/2009/501 FastTrack timeout 09/25/2009] In-reply-to: Sender: Sebastien.Roy@sun.com To: venugopal iyer Cc: Kais Belgaied , PSARC-ext@sun.com, crossbow-core@sun.com Message-id: <1254166386.12892.155.camel@strat> Organization: Sun Microsystems MIME-version: 1.0 X-Mailer: Evolution 2.26.3 Content-type: text/plain; CHARSET=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200909181929.n8IJTZ1i028359@sac.sfbay.sun.com> <1254164899.12892.144.camel@strat> Status: RO Content-Length: 3407 On Mon, 2009-09-28 at 12:38 -0700, venugopal iyer wrote: > Thanks, Seb. > > On Mon, 28 Sep 2009, Sebastien Roy wrote: > > > On Fri, 2009-09-18 at 12:29 -0700, Kais Belgaied wrote: > >> Background > >> ========== > >> > >> Project Crossbow (PSARC/2006/357) enables creating hardware-based MAC > >> clients (some of these MAC clients are data links such as VNICs) both on > >> the RX and TX side. We define hardware-based MAC clients as having dedicated > >> hardware resources; a RX hardware-based MAC client will have one or more RX > >> ring for exclusive use while a TX hardware-based MAC client will have one or > >> more TX ring for exclusive use. MAC clients that are not hardware-based > >> (RX or TX) share hardware resources with other MAC clients, such MAC > >> clients will not have any TX or RX rings exclusively reserved for them. > >> MAC clients may be hardware-based on RX, but not on TX (and vice-versa). > > > > To me, that last sentence is essentially saying, "MAC clients may not be > > hardware-based on both RX and TX". Is that what you meant? > > No. It was just to note that it *need* not be h/w on both. How about > "MAC clients could be hardware based on either TX, RX, both or neither" > or some such. Yes, that would be more clear. > >> If the property is not specified for a link, the system will attempt > >> to maxmize the hardware resource utilization by making this MAC client > >> hardware-based depending on rings availability. > > > > This is slightly ambiguous to me. Does that mean that if I neglect to > > override the default, that the first MAC client gets all of the hardware > > resources while subsequent clients get none? > > > That's not the case. Today, when we create a VNIC and don't specify the > -H option we try to give it one ring (assuming we have free rings). > That won't change, if we don't specify rxringcnt then we will try to > give the new MAC client 1 ring and make it hardware based. We do > this till there are free rings available. When there aren't any > free rings, MAC clients that don't specify this property will be > software based. Okay, that's fine. Please clarify that in the spec because your use of the word "maximize" could imply differently. > > I would think that unless > i otherwise specified, a sensible default behavior would result in evenly > > distributing resources among clients. > > > >> dladm show-phys will be modified to display the ring information on TX and > >> RX. > >> > >> e.g: > >> # dladm show-phys -H nxge0 > >> LINK RINGTYPE RINGS CLIENTS > >> nxge0 RX 0 > >> nxge0 TX 0,5 vnic1 > >> nxge0 RX 1-3 vnic1 > >> > >> which means vnic1 has exclusive use of 3 RX rings and 5 TX rings. > > > > I would interpret the above as "vnic1 has transmit rings 0 and > > 5" (because of the comma), and "vnic1 has receive rings 1, 2, and > > 3" (because of the hyphen). Am I reading that right? > > yes. So it does not have access to 5 TX rings as you state, but rather 2. > > > > How do I view the total number of hardware resources available on nxge0? > > the read-only rings property (rxringavailcnt, txringavailcnt) we are > introducing is for that purpose. > > let me know if I missed answering any of your questions. Got it, thanks. +1 from me. -Seb From venu@sun.com Mon Sep 28 12:40:04 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n8SJe441028012 for ; Mon, 28 Sep 2009 12:40:04 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n8SJe3mG013862; Mon, 28 Sep 2009 12:40:03 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KQP00E0B5AQEM00@nwk-avmta-2.sfbay.sun.com>; Mon, 28 Sep 2009 12:40:02 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KQP0074S5AOCW90@nwk-avmta-2.sfbay.sun.com>; Mon, 28 Sep 2009 12:40:00 -0700 (PDT) Received: from amchi-mumbai (amchi-mumbai.SFBay.Sun.COM [129.146.106.6]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n8SJdxJi635219 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 28 Sep 2009 12:40:00 -0700 (PDT) Date: Mon, 28 Sep 2009 12:52:59 -0700 (PDT) From: venugopal iyer Subject: Re: Dynamic Ring Grouping on NICs [PSARC/2009/501 FastTrack timeout 09/25/2009] In-reply-to: <1254166386.12892.155.camel@strat> X-X-Sender: vi117747@amchi-mumbai To: Sebastien Roy Cc: Kais Belgaied , PSARC-ext@sun.com, crossbow-core@sun.com Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; charset=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200909181929.n8IJTZ1i028359@sac.sfbay.sun.com> <1254164899.12892.144.camel@strat> <1254166386.12892.155.camel@strat> Status: RO Content-Length: 3629 On Mon, 28 Sep 2009, Sebastien Roy wrote: > > On Mon, 2009-09-28 at 12:38 -0700, venugopal iyer wrote: >> Thanks, Seb. >> >> On Mon, 28 Sep 2009, Sebastien Roy wrote: >> >>> On Fri, 2009-09-18 at 12:29 -0700, Kais Belgaied wrote: >>>> Background >>>> ========== >>>> >>>> Project Crossbow (PSARC/2006/357) enables creating hardware-based MAC >>>> clients (some of these MAC clients are data links such as VNICs) both on >>>> the RX and TX side. We define hardware-based MAC clients as having dedicated >>>> hardware resources; a RX hardware-based MAC client will have one or more RX >>>> ring for exclusive use while a TX hardware-based MAC client will have one or >>>> more TX ring for exclusive use. MAC clients that are not hardware-based >>>> (RX or TX) share hardware resources with other MAC clients, such MAC >>>> clients will not have any TX or RX rings exclusively reserved for them. >>>> MAC clients may be hardware-based on RX, but not on TX (and vice-versa). >>> >>> To me, that last sentence is essentially saying, "MAC clients may not be >>> hardware-based on both RX and TX". Is that what you meant? >> >> No. It was just to note that it *need* not be h/w on both. How about >> "MAC clients could be hardware based on either TX, RX, both or neither" >> or some such. > > Yes, that would be more clear. > >>>> If the property is not specified for a link, the system will attempt >>>> to maxmize the hardware resource utilization by making this MAC client >>>> hardware-based depending on rings availability. >>> >>> This is slightly ambiguous to me. Does that mean that if I neglect to >>> override the default, that the first MAC client gets all of the hardware >>> resources while subsequent clients get none? >> >> >> That's not the case. Today, when we create a VNIC and don't specify the >> -H option we try to give it one ring (assuming we have free rings). >> That won't change, if we don't specify rxringcnt then we will try to >> give the new MAC client 1 ring and make it hardware based. We do >> this till there are free rings available. When there aren't any >> free rings, MAC clients that don't specify this property will be >> software based. > > Okay, that's fine. Please clarify that in the spec because your use of > the word "maximize" could imply differently. > >>> I would think that unless >> i otherwise specified, a sensible default behavior would result in evenly >>> distributing resources among clients. >>> >>>> dladm show-phys will be modified to display the ring information on TX and >>>> RX. >>>> >>>> e.g: >>>> # dladm show-phys -H nxge0 >>>> LINK RINGTYPE RINGS CLIENTS >>>> nxge0 RX 0 >>>> nxge0 TX 0,5 vnic1 >>>> nxge0 RX 1-3 vnic1 >>>> >>>> which means vnic1 has exclusive use of 3 RX rings and 5 TX rings. >>> >>> I would interpret the above as "vnic1 has transmit rings 0 and >>> 5" (because of the comma), and "vnic1 has receive rings 1, 2, and >>> 3" (because of the hyphen). Am I reading that right? >> >> yes. > > So it does not have access to 5 TX rings as you state, but rather 2. Yes, sorry, you are right I should have said 2 TX rings, not 5. > >>> >>> How do I view the total number of hardware resources available on nxge0? >> >> the read-only rings property (rxringavailcnt, txringavailcnt) we are >> introducing is for that purpose. >> >> let me know if I missed answering any of your questions. > > Got it, thanks. > > +1 from me. thanks, I'll send a revised text addressing the above. -venu > > -Seb > > > From venu@sun.com Tue Sep 29 15:40:21 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n8TMeLXm018447 for ; Tue, 29 Sep 2009 15:40:21 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n8TMeHS0021740; Tue, 29 Sep 2009 15:40:18 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KQR004078B4YX00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 29 Sep 2009 15:40:16 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KQR000M38B45AB0@nwk-avmta-1.sfbay.Sun.COM>; Tue, 29 Sep 2009 15:40:16 -0700 (PDT) Received: from amchi-mumbai (amchi-mumbai.SFBay.Sun.COM [129.146.106.6]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n8TMeGUJ896774 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 29 Sep 2009 15:40:16 -0700 (PDT) Date: Tue, 29 Sep 2009 15:53:25 -0700 (PDT) From: venugopal iyer Subject: Re: Dynamic Ring Grouping on NICs [PSARC/2009/501 FastTrack timeout 09/25/2009] In-reply-to: <1254166386.12892.155.camel@strat> X-X-Sender: vi117747@amchi-mumbai To: Sebastien Roy , Darren.Reed@sun.com Cc: PSARC-ext@sun.com, crossbow-core@sun.com Message-id: MIME-version: 1.0 Content-type: multipart/mixed; boundary="Boundary_(ID_Soi5EzrRcpl8HUXwt2H1qw)" X-PMX-Version: 5.4.1.325704 References: <200909181929.n8IJTZ1i028359@sac.sfbay.sun.com> <1254164899.12892.144.camel@strat> <1254166386.12892.155.camel@strat> Status: RO Content-Length: 13861 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --Boundary_(ID_Soi5EzrRcpl8HUXwt2H1qw) Content-type: TEXT/PLAIN; charset=US-ASCII; format=flowed Content-transfer-encoding: 7BIT Please find attached the revised text with changes addressing your comments. Let me know if I have missed anything. thanks, -venu On Mon, 28 Sep 2009, Sebastien Roy wrote: > > On Mon, 2009-09-28 at 12:38 -0700, venugopal iyer wrote: >> Thanks, Seb. >> >> On Mon, 28 Sep 2009, Sebastien Roy wrote: >> >>> On Fri, 2009-09-18 at 12:29 -0700, Kais Belgaied wrote: >>>> Background >>>> ========== >>>> >>>> Project Crossbow (PSARC/2006/357) enables creating hardware-based MAC >>>> clients (some of these MAC clients are data links such as VNICs) both on >>>> the RX and TX side. We define hardware-based MAC clients as having dedicated >>>> hardware resources; a RX hardware-based MAC client will have one or more RX >>>> ring for exclusive use while a TX hardware-based MAC client will have one or >>>> more TX ring for exclusive use. MAC clients that are not hardware-based >>>> (RX or TX) share hardware resources with other MAC clients, such MAC >>>> clients will not have any TX or RX rings exclusively reserved for them. >>>> MAC clients may be hardware-based on RX, but not on TX (and vice-versa). >>> >>> To me, that last sentence is essentially saying, "MAC clients may not be >>> hardware-based on both RX and TX". Is that what you meant? >> >> No. It was just to note that it *need* not be h/w on both. How about >> "MAC clients could be hardware based on either TX, RX, both or neither" >> or some such. > > Yes, that would be more clear. > >>>> If the property is not specified for a link, the system will attempt >>>> to maxmize the hardware resource utilization by making this MAC client >>>> hardware-based depending on rings availability. >>> >>> This is slightly ambiguous to me. Does that mean that if I neglect to >>> override the default, that the first MAC client gets all of the hardware >>> resources while subsequent clients get none? >> >> >> That's not the case. Today, when we create a VNIC and don't specify the >> -H option we try to give it one ring (assuming we have free rings). >> That won't change, if we don't specify rxringcnt then we will try to >> give the new MAC client 1 ring and make it hardware based. We do >> this till there are free rings available. When there aren't any >> free rings, MAC clients that don't specify this property will be >> software based. > > Okay, that's fine. Please clarify that in the spec because your use of > the word "maximize" could imply differently. > >>> I would think that unless >> i otherwise specified, a sensible default behavior would result in evenly >>> distributing resources among clients. >>> >>>> dladm show-phys will be modified to display the ring information on TX and >>>> RX. >>>> >>>> e.g: >>>> # dladm show-phys -H nxge0 >>>> LINK RINGTYPE RINGS CLIENTS >>>> nxge0 RX 0 >>>> nxge0 TX 0,5 vnic1 >>>> nxge0 RX 1-3 vnic1 >>>> >>>> which means vnic1 has exclusive use of 3 RX rings and 5 TX rings. >>> >>> I would interpret the above as "vnic1 has transmit rings 0 and >>> 5" (because of the comma), and "vnic1 has receive rings 1, 2, and >>> 3" (because of the hyphen). Am I reading that right? >> >> yes. > > So it does not have access to 5 TX rings as you state, but rather 2. > >>> >>> How do I view the total number of hardware resources available on nxge0? >> >> the read-only rings property (rxringavailcnt, txringavailcnt) we are >> introducing is for that purpose. >> >> let me know if I missed answering any of your questions. > > Got it, thanks. > > +1 from me. > > -Seb > > > --Boundary_(ID_Soi5EzrRcpl8HUXwt2H1qw) Content-id: Content-type: TEXT/PLAIN; charset=US-ASCII; name=drg_psarc Content-transfer-encoding: 7BIT Content-disposition: attachment; filename=drg_psarc Content-description: drg_2009_501 Background ========== Project Crossbow (PSARC/2006/357) enables creating hardware-based MAC clients (some of these MAC clients are data links such as VNICs) both on the RX and TX side. We define hardware-based MAC clients as having dedicated hardware resources; a RX hardware-based MAC client will have one or more RX ring for exclusive use while a TX hardware-based MAC client will have one or more TX ring for exclusive use. MAC clients that are not hardware-based (RX or TX) share hardware resources with other MAC clients, such MAC clients will not have any TX or RX rings exclusively reserved for them. MAC clients could be hardware based on either TX, RX, both or neither. Currently, when a NIC registers with MAC it informs MAC if it supports dedicated hardware RX or TX rings. MAC assigns hardware rings to MAC clients as groups, where a group may contain 1 or more hardware rings. dladm show-phys is currently used to show how RX rings are used by MAC clients. # dladm show-phys -H nxge4 LINK GROUP GROUPTYPE RINGS CLIENTS nxge4 0 RX 3 nxge4 nxge4 1 RX 1 vnic1 which says we have 1 RX hardware-based MAC client - vnic1 with 1 ring. nxge4, the primary MAC client, is using 3 rings, but will share these with any other MAC client that is subsequently created on the data link nxge4 (i.e. if vnic2 is created on nxge4, MAC clients vnic2 and nxge4 will share group 1, and hence the 3 rings), eg: # dladm show-phys -H nxge4 LINK GROUP GROUPTYPE RINGS CLIENTS nxge4 0 RX 3 nxge4,vnic2 nxge4 1 RX 1 vnic1 Information about TX rings is not shown by the show-phys subcommand. Today, an administrator can specify that a VNIC must be hardware-based on the RX side (using the -H option to dladm create-vnic). However, there is no way for an administrator to specify o that a MAC client (VNIC or primary MAC client) should be software based, i.e. should not have any dedicated hardware resource, o that a MAC client should be hardware or software based on TX. o the number of RX or TX rings needed for a MAC client. Proposal ======== This proposal gives administrative control over whether a MAC client should be hardware-based or not (RX and TX) and also allows them to specify the number of RX or TX rings that a MAC client needs, if it is hardware-based. We introduce two properties for a link: rxrings: The number of RX rings needed. txrings: The number of TX rings needed. The values for these properties could be: 0 : This link must not assigned any hardware rings of the specified type. x > 0 : This link needs x rings. If the property is not specified for a link, the system will attempt to make the MAC client hardware-based depending on availability of rings. If rings are not available the MAC client will be software based. E.g: # dladm create-vnic -p rxrings=0 -l nxge0 vnic1 Will create vnic1 which will not be RX hardware-based. # dladm create-vnic -p txrings=2 -l nxge0 vnic2 Will create vnic2 that will be TX hardware-based with 2 TX rings. # dladm create-vnic -p rxrings=2,txrings=2 vnic3 Will create vnic3 which will be both RX and TX hardware-based with 2 RX and TX rings resp. Modifying the RX or TX rings assigned to an existing link, say nxge0, can be done using set-linkprop, e.g. if nxge0 needs to be given 2 RX rings: # dladm set-linkprop -p rxrings=2 nxge0 or for a VNIC, say vnic1, as: # dladm set-linkprop -p txrings=2 vnic1 The rings assigned to a link can be viewed using show-linkprop as: # dladm show-linkprop nxge0 LINK PROPERTY PERM VALUE DEFAULT POSSIBLE ... nxge0 rxrings rw 2 -- 0-4 nxge0 txrings rw 5 -- 0-6 ... These new properties obsolete the -H option of dladm create-vnic (i.e. the -H option will be removed). Given that we allow specifying RX and TX rings for links, we need a way to display how many rings are available for use. Additionally, we need to provide the number of hardware-based MAC clients that can be created on the RX and TX side. We introduce 4 additional read-only properties to display this information: rxringsavail : The total number of RX rings available for use, i.e. not exclusively given to any MAC client. txringsavail : The total number of TX rings available for use. rxhwclntavail: The total number of additional RX hardware-based MAC clients that can be created. Each of these hardware-based MAC client could have one or more RX rings assigned to them. txhwclntavail: The total number of additional TX hardware-based MAC clients that can be created. Each of these hardware-based MAC clients could have one or more TX rings assigned to them. The counts reflects the current utilization/availability of the resources listed. The counts change as hardware-based MAC clients are created or destroyed or when their hardware properties (i.e. rxrings and txrings) are changed. e.g: # dladm show-linkprop nxge0 LINK PROPERTY PERM VALUE DEFAULT POSSIBLE ... nxge0 rxringsavail r- 3 -- 0-4 nxge0 txringsavail r- 5 -- 0-6 nxge0 rxhwclntavail r- 1 -- 0-1 nxge0 txhwclntavail r- 5 -- 0-5 ... This indicates that there are 3 RX ring available (out of a maximum of 4) and 5 TX ring available out of 6. Additionally only one more RX hardware based MAC client can be created while 5 more TX hardware-based MAC clients can be created. We provide two separate counters (rxhwclntavail and txhwclntavail) because the hardware virtualizes on RX based on groups of rings (i.e. MAC address is added to a group) and to keep it consistent on the TX we do the same. dladm show-phys will be modified to display the ring information on TX and RX. e.g: # dladm show-phys -H nxge0 LINK RINGTYPE RINGS CLIENTS nxge0 RX 0 nxge0 TX 0,5 vnic1 nxge0 RX 1-3 vnic1 which means vnic1 has exclusive use of 3 RX rings and 2 TX rings. --------------------------------- dladm(1m) ! dladm show-phys [-P] [[-p] -o field[,...]] [phys-link] Show the physical device and attributes of all physical links, or of the named physical link. Without -P, only --- 414,420 ---- tion. ! dladm show-phys [-P] [[-p] -o field[,...]] [-H] [phys-link] Show the physical device and attributes of all physical links, or of the named physical link. Without -P, only *************** *** 421,426 **** --- 421,445 ---- physical links that are available on the running system are displayed. + -H + Show hardware resource usage, as returned by the NIC + driver. Output from -H displays the following elements: + + LINK + A physical device corresponding to a NIC driver. + + RINGTYPE + The type of the ring. This is either RX or TX. + + RINGS + The ring index. A ring is an hardware resource, which + typically maps to a DMA channel, that can be programmed + for specific use. E.g. an RX ring can be programmed to + receive only packets belonging to a specific MAC address. + + CLIENTS + MAC clients using the rings. + -o field, --output=field A case-insensitive, comma-separated list of output *************** *** 499,505 **** purge the link's configuration from the system. - dladm create-aggr [-t] [-R root-dir] [-P policy] [-L mode] [-T time] [-u address] -l ether-link1 [-l ether-link2...] aggr-link --- 518,523 ---- *************** *** 2432,2440 **** --- 2450,2494 ---- default is high. + rxringsavail + A read-only property that specifies the number of rings available + on the receive side. + rxrings + Specifies the number of receive rings side for the MAC client. + A value of 0 means this MAC client should not be assigned any RX + ring. A non-0 value means reserve that many rings for this MAC + client, if available, and fail if not. If this property is not + specified the MAC client may get one RX ring, if available, or + will be software based. + + rxhwclntavail + + A read-only property that specifies the number of additional + RX hardware-based MAC clients that can be created. + + txringsavail + + A read-only property that specifies the number of rings available + on the transmit side. + + txrings + + Specifies the number of transmit rings for the MAC client. + A value of 0 means this MAC client should not be assigned need any + TX ring. A non-0 value means reserve that many rings for this MAC + client, if available, and fail if not. If this property is not + specified the MAC client may get one TX ring, if available, or + will be software based. + + txhwclntavail + + A read-only property that specifics the number of additional + TX hardware-based MAC clients that can be created. + + SunOS 5.11 Last change: 16 Mar 2009 37 --Boundary_(ID_Soi5EzrRcpl8HUXwt2H1qw)-- From kais.belgaied@Sun.COM Wed Sep 30 10:06:56 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n8UH6u25023061 for ; Wed, 30 Sep 2009 10:06:56 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n8UH6m8T045003; Wed, 30 Sep 2009 11:06:55 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KQS00E0XNJI2H00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 30 Sep 2009 10:06:54 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KQS0051VNJH7TA0@nwk-avmta-1.sfbay.Sun.COM>; Wed, 30 Sep 2009 10:06:53 -0700 (PDT) Received: from [129.146.11.144] (sr1-jurassic-01.SFBay.Sun.COM [129.146.11.144]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n8UH6rME141864 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 30 Sep 2009 10:06:53 -0700 (PDT) Date: Wed, 30 Sep 2009 10:06:53 -0700 From: Kais Belgaied Subject: Re: Dynamic Ring Grouping on NICs [PSARC/2009/501 FastTrack timeout 09/25/2009] In-reply-to: To: venugopal iyer Cc: PSARC-ext@Sun.COM, crossbow-core@Sun.COM Message-id: <4AC3902D.1020801@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200909181929.n8IJTZ1i028359@sac.sfbay.sun.com> <1254164899.12892.144.camel@strat> <1254166386.12892.155.camel@strat> User-Agent: Thunderbird 2.0.0.21 (X11/20090311) Status: RO Content-Length: 60 This case was approved in today's PSARC meeting. Kais.