From sacadmin Tue Apr 24 13:45:19 2007 Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l3OKjJw4020412; Tue, 24 Apr 2007 13:45:19 -0700 (PDT) Received: (from barts@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id l3OKjJ6X020408; Tue, 24 Apr 2007 13:45:19 -0700 (PDT) Date: Tue, 24 Apr 2007 13:45:19 -0700 (PDT) From: Bart Smaalders Message-Id: <200704242045.l3OKjJ6X020408@sac.sfbay.sun.com> To: PSARC@sac.sfbay.sun.com Cc: alan.duboff@sun.com, bart.smaalders@sun.com Subject: add afe driver to Solaris [PSARC/2007/229 Timeout: 05/02/2007] Status: RO Content-Length: 6546 Subject: PSARC FastTrack [05/02/2007]: add afe driver to Solaris Template Version: @(#)sac_nextcase %I% %G% SMI 1. Introduction 1.1. Project/Component Working Name: add afe driver to Solaris 1.2. Name of Document Author/Supplier: Author: Alan Duboff 1.3 Date of This Document: 24 April, 2007 4. Technical Description 1. Introduction 1.1. Project/Component Working Name: Integrate afe driver into Solaris 1.2. Name of Document Author/Supplier: Author: Alan DuBoff/Bart Smaalders 1.3 Date of This Document: April 9, 2007 2. Project Summary This project integrates a driver for Centaur and Comet fast ethernet controllers from ADMtek into Solaris. The driver was developed by Garrett D'Amore and placed under a BSD license; it is being included in Solaris by Alan DuBoff. This is a GLDV2 driver. It is anticipated that it will be converted to a GLDV3 driver sometime during S11 development. This project seeks micro/patch binding. 4.0 Technical Description This project introduces the following files: usr/share/man/man7d/afe.7d kernel/drv/afe kernel/drv/amd64/afe kernel/drv/sparcv9/afe Interfaces imported __________________________________________________________________ GLD V2 , DDI Interfaces exported __________________________________________________________________ afe Driver Name Stable SUNWafe Package name Stable Drivers afe(7D) NAME afe - ADMtek Fast Ethernet device driver SYNOPSIS /dev/afe DESCRIPTION The afe driver is a multi-threaded, loadable, clonable, GLD-based STREAMS driver supporting the Data Link Provider Interface, dlpi(7P), on ADMtek (now Infineon) Centaur and Comet Fast Ethernet controllers. APPLICATION PROGRAMMING INTERFACE The afe driver can be used as either a "style 1" or a "style 2" Data Link Service Provider. Physical points of attachment (PPAs) are interpreted as the instance number of the afe controller as assigned by the operating system. Here are the relevant fields returned as part of a DL_INFO_ACK response: o The maximum SDU is 1500. o The minimum SDU is 0. o The DLSAP address length is 8. o The MAC type is DL_ETHER. o The SAP length is -2; the 6 byte physical address is followed immediately by a 2 byte SAP. o The service mode is DL_CLDLS. o The broadcast address is the 6-byte Ethernet broadcast address (ff:ff:ff:ff:ff:ff). If the SAP provided is zero, then IEEE 802.3 mode is assumed and outbound frames will have the frame payload length writ- ten into the type field. Likewise, inbound frames with a SAP between zero and 1500 are interpreted as IEEE 802.3 frames, and delivered to any streams that have bound to SAP zero (the 802.3 SAP). PROPERTIES The following properties may be configured using either ndd(1M), or the afe.conf configuration file as described by driver.conf(4). adv_autoneg_cap Enables (default) or disables IEEE 802.3u autonegotiation of link speed and duplex settings. If enabled, the device will negotiate among the supported (and configured, see below) link options with the link partner. If disabled, then at least one of the link options below must be specified; the driver will select the first enabled link option according to the IEEE 802.3u speci- fied preferences. adv_100T4_cap Enables the 100 BaseT4 link option. (Note that most hardware does not support this unusual link style. Also, this uses two pairs of wires for data, rather than just one.) adv_100fdx_cap Enables the 100 Base TX full-duplex link option. (This will generally be the fastest mode, if both link partners support it. Most modern equipment will support this mode.) adv_100hdx_cap Enables the 100 Base TX half-duplex link option. (This is typically used when the link partner is a 100 Mbps hub.) adv_10fdx_cap Enables the 10 Base-T full-duplex link option. (This less-frequently used mode is typically used when the link partner is a 10 Mbps switch.) adv_10hdx_cap Enables the 10 Base-T half-duplex link option. (This is the fall-back when no other option is available. It is typically used when the link partner is a 10 Mbps hub, or is an older network card.) FILES /dev/afe afe special character device /kernel/drv/afe.conf afe configuration file /kernel/drv/afe 32-bit x86 afe driver binary /kernel/drv/amd64/afe 64-bit x86 afe driver binary /kernel/drv/sparcv9/afe 64-bit SPARC afe driver binary ATTRIBUTES See attributes(5) for descriptions of the following attri- butes: ____________________________________________________________ | ATTRIBUTE TYPE ATTRIBUTE VALUE | | Architecture SPARC, x86 | |___________________________________________________________| SEE ALSO ndd(1M), driver.conf(4), attributes(5), streamio(7I), dlpi(7P) 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack From sacadmin Tue Apr 24 14:40:56 2007 Received: from sfbaymail1sca.SFBay.Sun.COM (sfbaymail1sca.SFBay.Sun.COM [129.145.154.35]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l3OLeueF022689; Tue, 24 Apr 2007 14:40:56 -0700 (PDT) Received: from [129.146.108.211] (almas.SFBay.Sun.COM [129.146.108.211]) by sfbaymail1sca.SFBay.Sun.COM (8.13.6+Sun/8.13.6/ENSMAIL,v2.2) with ESMTP id l3OLe5lc003833; Tue, 24 Apr 2007 14:40:05 -0700 (PDT) Message-ID: <462E7935.3050800@sun.com> Date: Tue, 24 Apr 2007 14:40:05 -0700 From: Alan Coopersmith User-Agent: Thunderbird 2.0.0.0 (X11/20070423) MIME-Version: 1.0 To: Bart Smaalders CC: PSARC@sac.sfbay.sun.com, Alan.Duboff@sun.com, bart.smaalders@sun.com Subject: Re: add afe driver to Solaris [PSARC/2007/229 Timeout: 05/02/2007] References: <200704242045.l3OKjJ6X020408@sac.sfbay.sun.com> In-Reply-To: <200704242045.l3OKjJ6X020408@sac.sfbay.sun.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 545 I need to write a bot to send this out (and have it poke plocher to fix sac_nextcase every time I have to send it)... Bart Smaalders wrote: > Subject: PSARC FastTrack [05/02/2007]: add afe driver to Solaris > > > Template Version: @(#)sac_nextcase %I% %G% SMI Why is incorporating an open source driver into Solaris not an open case available to the OpenSolaris community? -- -Alan Coopersmith- alan.coopersmith@sun.com Sun Microsystems, Inc. - X Window System Engineering From sacadmin Tue Apr 24 16:25:58 2007 Received: from zion.eng.sun.com (zion [129.146.17.75]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l3ONPwjR026420; Tue, 24 Apr 2007 16:25:58 -0700 (PDT) Received: from [129.146.228.109] (cyber [129.146.228.109]) by zion.eng.sun.com (8.13.7+Sun/8.13.7) with ESMTP id l3ONP8JP008910; Tue, 24 Apr 2007 16:25:08 -0700 (PDT) Message-ID: <462E9188.8050403@Sun.COM> Date: Tue, 24 Apr 2007 16:23:52 -0700 From: Bart Smaalders Organization: Sun Microsystems User-Agent: Thunderbird 2.0b2 (X11/20070227) MIME-Version: 1.0 To: Alan Coopersmith CC: Bart Smaalders , PSARC@sac.sfbay.sun.com, Alan.Duboff@sun.com Subject: Re: add afe driver to Solaris [PSARC/2007/229 Timeout: 05/02/2007] References: <200704242045.l3OKjJ6X020408@sac.sfbay.sun.com> <462E7935.3050800@sun.com> In-Reply-To: <462E7935.3050800@sun.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 674 Alan Coopersmith wrote: > I need to write a bot to send this out (and have it poke plocher > to fix sac_nextcase every time I have to send it)... > > Bart Smaalders wrote: >> Subject: PSARC FastTrack [05/02/2007]: add afe driver to Solaris >> >> >> Template Version: @(#)sac_nextcase %I% %G% SMI > > Why is incorporating an open source driver into Solaris not an > open case available to the OpenSolaris community? > Blame sac_nextcase. I edited the IAM file to say it was open while running sac_nextcase. - Bart -- Bart Smaalders Solaris Kernel Performance barts@cyber.eng.sun.com http://blogs.sun.com/barts From sacadmin Wed May 2 09:34:32 2007 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l42GYVBZ015277 for ; Wed, 2 May 2007 09:34:31 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l42GX520006246; Wed, 2 May 2007 17:33:32 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JHF00F0B9ZV5L00@nwk-avmta-2.sfbay.sun.com>; Wed, 02 May 2007 09:33:31 -0700 (PDT) Received: from nwk-ea-fw-1.sun.com ([10.4.134.6]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JHF007KY9ZTRAD0@nwk-avmta-2.sfbay.sun.com>; Wed, 02 May 2007 09:33:29 -0700 (PDT) Received: from d1-sfbay-10.sun.com ([192.18.39.120]) by nwk-ea-fw-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l42GXTA3020006; Wed, 02 May 2007 09:33:29 -0700 (PDT) Received: from conversion-daemon.d1-sfbay-10.sun.com by d1-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) id <0JHF00L019W4JZ00@d1-sfbay-10.sun.com> (original mail from Sherri.Shieh@Sun.COM); Wed, 02 May 2007 09:33:29 -0700 (PDT) Received: from [129.146.11.146] by d1-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) with ESMTPSA id <0JHF00ENA9ZTVO00@d1-sfbay-10.sun.com>; Wed, 02 May 2007 09:33:29 -0700 (PDT) Date: Wed, 02 May 2007 09:33:29 -0700 From: Sherri Shieh Subject: PSARC Fast Track: add afe driver to Solaris (2007/229) Sender: Sherri.Shieh@sun.com To: psarc@sun.com, Alan Duboff , bart Smaalders Message-id: <4638BD59.9000409@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=us-ascii Content-transfer-encoding: 7BIT X-Accept-Language: en-us, en X-PMX-Version: 5.2.0.264296 User-Agent: Mozilla/5.0 (X11; U; SunOS sun4v; en-US; rv:1.7) Gecko/20060629 Status: RO Content-Length: 338 This fast track was marked closed approved from last week's meeting. - Sherri -- ========================================================= Sherri Shieh Sun Microsystems, Inc. Program Manager Email: sherri.shieh@sun.com Systems Architecture Phone: 650-786-5245/x85245 =========================================================== From Garrett.Damore@Sun.COM Mon Jul 23 23:57:48 2007 Received: from sfbaymail1sca.SFBay.Sun.COM (sfbaymail1sca.SFBay.Sun.COM [129.145.154.35]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l6O6vlhn017346 for ; Mon, 23 Jul 2007 23:57:47 -0700 (PDT) Received: from sca-es-mail-1.sun.com (sca-es-mail-1.Sun.COM [192.18.43.132]) by sfbaymail1sca.SFBay.Sun.COM (8.13.6+Sun/8.13.6/ENSMAIL,v2.2) with ESMTP id l6O6tc9Y009053 for ; Mon, 23 Jul 2007 23:55:38 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l6O6tXwM009872 for ; Mon, 23 Jul 2007 23:55:33 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JLO00J018K8KC00@fe-sfbay-09.sun.com> (original mail from Garrett.Damore@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Mon, 23 Jul 2007 23:55:33 -0700 (PDT) Received: from [192.168.251.21] ([76.174.83.55]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JLO00EI58KKS3A0@fe-sfbay-09.sun.com>; Mon, 23 Jul 2007 23:55:33 -0700 (PDT) Date: Mon, 23 Jul 2007 23:52:13 -0700 From: "Garrett D'Amore" Subject: PSARC 2007/229 add afe driver to Solaris (update) Sender: Garrett.Damore@Sun.COM To: psarc-ext@sac.sfbay.sun.com, Alan DuBoff Message-id: <46A5A19D.3060707@sun.com> Organization: Sun Microsystems MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT User-Agent: Thunderbird 2.0.0.0 (X11/20070618) Status: RO Content-Length: 2303 I would like to update this case, as we have decided to deliver a GLDv3 version of the afe driver instead of GLDv2. This has many benefits, which the ARC is probably well familiar with so I will not expand upon them here. This means that the driver will not import the GLDv2 interfaces, but will import the Consolidation Private GLDv3 interfaces. I am also importing the MII register definitions (macros) that are in sys/miiregs.h. I do not believe that this header file has ever been properly ARC'd, but since the values are all macro definitions for constant values, it should not cause any binary compatibility issues. The file was noted in PSARC 2002/711, but no stability classification was identified. (Note that the rge, bge, and dmfe drivers also depend on this header, as do a few closed source drivers.) Also, as part of this work, I've updated the driver to perform loan-up of DMA buffers, and to attempt to use zero-copy DMA for transmit buffers. To facilitate tweaking of this, there are three new undocumented tunables (accessible via either ndd or driver.conf.) I do not plan on documenting these at this point. Here are the descriptions of them: tx_max_bcopy - transmit mblks smaller than this will not be DMA'd, but will be bcopy()'d into a premapped DMA buffer (default = 256) rx_max_bcopy - receive packets smaller than this will not be loaned up, but will be bcopy()'d into a freshly allocb()'d buffer (default = 256) rx_max_loanup - maximum number of outstanding packets that will be loaned up to the upper stack without bcopy'ing (default = 128) Tuning these would be either for performance tweaking (which we don't expect to be necessary), or (in the case of rx_max_loanup) to prevent the upper layers from holding mblks. (Setting rx_max_loanup == 0 may facilitate DDI detach, since the driver will refuse to detach while the upper layers are holding message blocks that have been loaned up... as the device's instance is required to properly tear down the DMA bindings for those buffers. Note that the device is normally DR safe, although IPv6 in.ndpd seems to like to hold on to received mblks perhaps longer than it should.) In the future we hope these tunables will be handled via the Brussels framework. -- Garrett From Darren.Reed@Sun.COM Tue Jul 24 09:02:18 2007 Received: from sfbaymail1sca.SFBay.Sun.COM (sfbaymail1sca.SFBay.Sun.COM [129.145.154.35]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l6OG2ItX028768 for ; Tue, 24 Jul 2007 09:02:18 -0700 (PDT) Received: from sineb-mail-1.sun.com ([192.18.19.6]) by sfbaymail1sca.SFBay.Sun.COM (8.13.6+Sun/8.13.6/ENSMAIL,v2.2) with ESMTP id l6OG09P3022739 for ; Tue, 24 Jul 2007 09:00:09 -0700 (PDT) Received: from fe-apac-06.sun.com (fe-apac-06.sun.com [192.18.19.177] (may be forged)) by sineb-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l6OG02LQ024647 for ; Tue, 24 Jul 2007 16:00:03 GMT Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) id <0JLO00L01XJ89V00@mail-apac.sun.com> (original mail from Darren.Reed@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Wed, 25 Jul 2007 00:00:01 +0800 (SGT) Received: from [129.157.18.96] by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) with ESMTPSA id <0JLO0077DXRZ6X2C@mail-apac.sun.com>; Wed, 25 Jul 2007 00:00:01 +0800 (SGT) Date: Tue, 24 Jul 2007 17:59:24 +0200 From: Darren Reed Subject: Re: PSARC 2007/229 add afe driver to Solaris (update) In-reply-to: <46A5A19D.3060707@sun.com> Sender: Darren.Reed@Sun.COM To: "Garrett D'Amore" Cc: psarc-ext@sac.sfbay.sun.com, Alan DuBoff Message-id: <46A621DC.5080001@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <46A5A19D.3060707@sun.com> User-Agent: Thunderbird 1.5.0.12 (Windows/20070509) Status: RO Content-Length: 1176 Garrett D'Amore wrote: > ... > rx_max_loanup - maximum number of outstanding packets that will be > loaned up to the upper stack without bcopy'ing (default = 128) > > Tuning these would be either for performance tweaking (which we don't > expect to be necessary), or (in the case of rx_max_loanup) to prevent > the upper layers from holding mblks. (Setting rx_max_loanup == 0 may > facilitate DDI detach, since the driver will refuse to detach while > the upper layers are holding message blocks that have been loaned > up... as the device's instance is required to properly tear down the > DMA bindings for those buffers. Note that the device is normally DR > safe, although IPv6 in.ndpd seems to like to hold on to received mblks > perhaps longer than it should.) Reading what you're saying about rx_max_loanup here, are there implications for modunload'ing if rx_max_loanup is non-0? Is there any way to interrogate the driver to find out how many buffers have been loaned out and are outstanding? Maybe a read-only rx_loaned? (Trying to think how you trouble shoot a problem when the driver doesn't unload and you need to dettermine "why not".) Darren From Garrett.Damore@Sun.COM Tue Jul 24 09:21:08 2007 Received: from sfbaymail2sca.sfbay.sun.com (sfbaymail2sca.SFBay.Sun.COM [129.145.155.42]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l6OGL82o029678 for ; Tue, 24 Jul 2007 09:21:08 -0700 (PDT) Received: from sca-es-mail-2.sun.com (sca-es-mail-2.Sun.COM [192.18.43.133]) by sfbaymail2sca.sfbay.sun.com (8.13.6+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id l6OGJ0FI009010 for ; Tue, 24 Jul 2007 09:19:00 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l6OGItbn029012 for ; Tue, 24 Jul 2007 09:18:55 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JLO00J01YK2XM00@fe-sfbay-09.sun.com> (original mail from Garrett.Damore@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Tue, 24 Jul 2007 09:18:55 -0700 (PDT) Received: from [192.168.251.21] ([76.174.83.55]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JLO00BHPYNIMG30@fe-sfbay-09.sun.com>; Tue, 24 Jul 2007 09:18:55 -0700 (PDT) Date: Tue, 24 Jul 2007 09:15:35 -0700 From: "Garrett D'Amore" Subject: Re: PSARC 2007/229 add afe driver to Solaris (update) In-reply-to: <46A621DC.5080001@Sun.COM> Sender: Garrett.Damore@Sun.COM To: Darren Reed Cc: psarc-ext@sac.sfbay.sun.com, Alan DuBoff Message-id: <46A625A7.6060601@sun.com> Organization: Sun Microsystems MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <46A5A19D.3060707@sun.com> <46A621DC.5080001@Sun.COM> User-Agent: Thunderbird 2.0.0.0 (X11/20070618) Status: RO Content-Length: 2459 Darren Reed wrote: > Garrett D'Amore wrote: >> ... >> rx_max_loanup - maximum number of outstanding packets that will be >> loaned up to the upper stack without bcopy'ing (default = 128) >> >> Tuning these would be either for performance tweaking (which we don't >> expect to be necessary), or (in the case of rx_max_loanup) to prevent >> the upper layers from holding mblks. (Setting rx_max_loanup == 0 may >> facilitate DDI detach, since the driver will refuse to detach while >> the upper layers are holding message blocks that have been loaned >> up... as the device's instance is required to properly tear down the >> DMA bindings for those buffers. Note that the device is normally DR >> safe, although IPv6 in.ndpd seems to like to hold on to received >> mblks perhaps longer than it should.) > > > Reading what you're saying about rx_max_loanup here, are there > implications > for modunload'ing if rx_max_loanup is non-0? If the upper layer stacks "hold" a loaned up mblk, then yes, you will be unable to modunload, or even DR detach, the device. Generally I've not found this to impact IPv4 usage at all. For IPv6 I've noticed that sometimes I have to pkill -1 in.ndpd after unplumbing in order to permit the detach to complete. (I suspect that if I wait long enough for in.ndpd to realize that the interface is no longer active, it might release the associated mblk.) > > Is there any way to interrogate the driver to find out how many > buffers have > been loaned out and are outstanding? Maybe a read-only rx_loaned? > (Trying to think how you trouble shoot a problem when the driver doesn't > unload and you need to dettermine "why not".) Possibly I could add that. Right now it just says "driver busy" or somesuch. When you find no open streams to the device, its a pretty good sign that the NIC is not in use anymore. There are some longer term solutions to this problem in general that I want to work on, but they require larger changes, and I'm not confident enough to include them in the initial release. (It should be able to unbind the buffer from DMA and release the DMA handle without doing the ddi_dma_mem_free()... and the acch associated with the DMA region should be able to outlive the devinfo originally used to allocate it. Of course, that solves *detach*, but it doesn't do much for modunload. (You absolutely cannot modunload if you desballoc'd buffers outstanding!) -- Garrett From Darren.Reed@Sun.COM Tue Jul 24 09:57:34 2007 Received: from sfbaymail2sca.sfbay.sun.com (sfbaymail2sca.SFBay.Sun.COM [129.145.155.42]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l6OGvYbs002075 for ; Tue, 24 Jul 2007 09:57:34 -0700 (PDT) Received: from sineb-mail-2.sun.com ([192.18.19.7]) by sfbaymail2sca.sfbay.sun.com (8.13.6+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id l6OGtPOu028079 for ; Tue, 24 Jul 2007 09:55:25 -0700 (PDT) Received: from fe-apac-06.sun.com (fe-apac-06.sun.com [192.18.19.177] (may be forged)) by sineb-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l6OGtJbE018435 for ; Tue, 24 Jul 2007 16:55:19 GMT Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) id <0JLP00H0106SCL00@mail-apac.sun.com> (original mail from Darren.Reed@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Wed, 25 Jul 2007 00:55:19 +0800 (SGT) Received: from [129.157.18.96] by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) with ESMTPSA id <0JLP0077L0C56S2M@mail-apac.sun.com>; Wed, 25 Jul 2007 00:55:19 +0800 (SGT) Date: Tue, 24 Jul 2007 18:54:41 +0200 From: Darren Reed Subject: Re: PSARC 2007/229 add afe driver to Solaris (update) In-reply-to: <46A625A7.6060601@sun.com> Sender: Darren.Reed@Sun.COM To: "Garrett D'Amore" Cc: psarc-ext@sac.sfbay.sun.com, Alan DuBoff Message-id: <46A62ED1.4090704@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <46A5A19D.3060707@sun.com> <46A621DC.5080001@Sun.COM> <46A625A7.6060601@sun.com> User-Agent: Thunderbird 1.5.0.12 (Windows/20070509) Status: RO Content-Length: 1999 Garrett D'Amore wrote: > Darren Reed wrote: >> Garrett D'Amore wrote: >>> ... >>> rx_max_loanup - maximum number of outstanding packets that will be >>> loaned up to the upper stack without bcopy'ing (default = 128) >>> >>> Tuning these would be either for performance tweaking (which we >>> don't expect to be necessary), or (in the case of rx_max_loanup) to >>> prevent the upper layers from holding mblks. (Setting rx_max_loanup >>> == 0 may facilitate DDI detach, since the driver will refuse to >>> detach while the upper layers are holding message blocks that have >>> been loaned up... as the device's instance is required to properly >>> tear down the DMA bindings for those buffers. Note that the device >>> is normally DR safe, although IPv6 in.ndpd seems to like to hold on >>> to received mblks perhaps longer than it should.) >> >> >> ... >> >> Is there any way to interrogate the driver to find out how many >> buffers have >> been loaned out and are outstanding? Maybe a read-only rx_loaned? >> (Trying to think how you trouble shoot a problem when the driver doesn't >> unload and you need to dettermine "why not".) > > Possibly I could add that. Right now it just says "driver busy" or > somesuch. When you find no open streams to the device, its a pretty > good sign that the NIC is not in use anymore. Well, the suggestion was just a "nice to have", not a requirement. My bigger concern is that IPFilter (and potentially other users of pfhooks) can park mblks (on an internal buffer ring) and the mblks can thereafter stay there independent of what's happening elsewhere with interfaces, etc. But then this problem will exist today with any existing drivers that do loanup too. Thinking about this and the state of affairs here makes it obvious that the architecture around using mblks needs to be improved such that analysing situations like this is easier without requiring special "addons" in drivers. A discussion for networking-discuss. Darren From randy.fishel@sun.com Tue Jul 24 10:05:05 2007 Received: from engmail3mpk.sfbay.Sun.COM (engmail3mpk.SFBay.Sun.COM [129.146.11.26]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l6OH55aQ002828 for ; Tue, 24 Jul 2007 10:05:05 -0700 (PDT) Received: from grimmy.eng.sun.com (grimmy.SFBay.Sun.COM [129.146.108.114]) by engmail3mpk.sfbay.Sun.COM (8.13.6+Sun/8.13.6/ENSMAIL,v2.2) with ESMTP id l6OH2uk2027287; Tue, 24 Jul 2007 10:02:56 -0700 (PDT) Received: from grimmy.eng.sun.com (localhost [127.0.0.1]) by grimmy.eng.sun.com (8.12.10+Sun/8.12.10) with ESMTP id l6OH2oQt016073; Tue, 24 Jul 2007 10:02:50 -0700 (PDT) Received: from localhost (randyf@localhost) by grimmy.eng.sun.com (8.12.10+Sun/8.12.10/Submit) with ESMTP id l6OH2oGn016069; Tue, 24 Jul 2007 10:02:50 -0700 (PDT) X-Authentication-Warning: grimmy.eng.sun.com: randyf owned process doing -bs Date: Tue, 24 Jul 2007 10:02:50 -0700 (PDT) From: Randy Fishel X-X-Sender: randyf@grimmy To: "Garrett D'Amore" cc: Darren Reed , psarc-ext@sac.sfbay.sun.com, Alan DuBoff Subject: Re: PSARC 2007/229 add afe driver to Solaris (update) In-Reply-To: <46A625A7.6060601@sun.com> Message-ID: References: <46A5A19D.3060707@sun.com> <46A621DC.5080001@Sun.COM> <46A625A7.6060601@sun.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: RO Content-Length: 1285 On Tue, 24 Jul 2007, Garrett D'Amore wrote: > Darren Reed wrote: > > Garrett D'Amore wrote: > > > ... > > > rx_max_loanup - maximum number of outstanding packets that will be > > > loaned up to the upper stack without bcopy'ing (default = 128) > > > > > > Tuning these would be either for performance tweaking (which we don't > > > expect to be necessary), or (in the case of rx_max_loanup) to prevent the > > > upper layers from holding mblks. (Setting rx_max_loanup == 0 may > > > facilitate DDI detach, since the driver will refuse to detach while the > > > upper layers are holding message blocks that have been loaned up... as the > > > device's instance is required to properly tear down the DMA bindings for > > > those buffers. Note that the device is normally DR safe, although IPv6 > > > in.ndpd seems to like to hold on to received mblks perhaps longer than it > > > should.) > > > > > > Reading what you're saying about rx_max_loanup here, are there implications > > for modunload'ing if rx_max_loanup is non-0? > > If the upper layer stacks "hold" a loaned up mblk, then yes, you will be > unable to modunload, or even DR detach, the device. As DR and CPR have similar entry points, will this "hold" affect the ability of a device to suspend? ---- Randy From Garrett.Damore@Sun.COM Tue Jul 24 10:17:58 2007 Received: from sfbaymail2sca.sfbay.sun.com (sfbaymail2sca.SFBay.Sun.COM [129.145.155.42]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l6OHHwcp004181 for ; Tue, 24 Jul 2007 10:17:58 -0700 (PDT) Received: from sca-es-mail-1.sun.com (sca-es-mail-1.Sun.COM [192.18.43.132]) by sfbaymail2sca.sfbay.sun.com (8.13.6+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id l6OHFoHN009207 for ; Tue, 24 Jul 2007 10:15:50 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l6OHFjQX011923 for ; Tue, 24 Jul 2007 10:15:45 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JLP0030115Y6U00@fe-sfbay-10.sun.com> (original mail from Garrett.Damore@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Tue, 24 Jul 2007 10:15:45 -0700 (PDT) Received: from [192.168.251.21] ([76.174.83.55]) by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JLP00C7K1A3FE60@fe-sfbay-10.sun.com>; Tue, 24 Jul 2007 10:15:39 -0700 (PDT) Date: Tue, 24 Jul 2007 10:12:18 -0700 From: "Garrett D'Amore" Subject: Re: PSARC 2007/229 add afe driver to Solaris (update) In-reply-to: Sender: Garrett.Damore@Sun.COM To: Randy Fishel Cc: Darren Reed , psarc-ext@sac.sfbay.sun.com, Alan DuBoff Message-id: <46A632F2.2040102@sun.com> Organization: Sun Microsystems MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <46A5A19D.3060707@sun.com> <46A621DC.5080001@Sun.COM> <46A625A7.6060601@sun.com> User-Agent: Thunderbird 2.0.0.0 (X11/20070618) Status: RO Content-Length: 1541 Randy Fishel wrote: > On Tue, 24 Jul 2007, Garrett D'Amore wrote: > > >> Darren Reed wrote: >> >>> Garrett D'Amore wrote: >>> >>>> ... >>>> rx_max_loanup - maximum number of outstanding packets that will be >>>> loaned up to the upper stack without bcopy'ing (default = 128) >>>> >>>> Tuning these would be either for performance tweaking (which we don't >>>> expect to be necessary), or (in the case of rx_max_loanup) to prevent the >>>> upper layers from holding mblks. (Setting rx_max_loanup == 0 may >>>> facilitate DDI detach, since the driver will refuse to detach while the >>>> upper layers are holding message blocks that have been loaned up... as the >>>> device's instance is required to properly tear down the DMA bindings for >>>> those buffers. Note that the device is normally DR safe, although IPv6 >>>> in.ndpd seems to like to hold on to received mblks perhaps longer than it >>>> should.) >>>> >>> Reading what you're saying about rx_max_loanup here, are there implications >>> for modunload'ing if rx_max_loanup is non-0? >>> >> If the upper layer stacks "hold" a loaned up mblk, then yes, you will be >> unable to modunload, or even DR detach, the device. >> > > As DR and CPR have similar entry points, will this "hold" affect the > ability of a device to suspend? > No. These devices should suspend just fine. In fact, this may be a differentiating point between these devices and pretty much most of the other x86 100Mbps nics. -- Garrett > > ---- Randy >