From tedk@sac.sfbay.sun.com Fri Oct 10 11:04:14 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9AI4D77026160 for ; Fri, 10 Oct 2008 11:04:13 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m9AI4ASq022795; Fri, 10 Oct 2008 19:04:12 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8J00605BIZV300@nwk-avmta-1.sfbay.Sun.COM>; Fri, 10 Oct 2008 11:04:11 -0700 (PDT) Received: from dm-sfbay-01.sfbay.sun.com ([129.145.155.118]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8J0068ABIZKY00@nwk-avmta-1.sfbay.Sun.COM>; Fri, 10 Oct 2008 11:04:11 -0700 (PDT) Received: from sac.sfbay.sun.com (new-sac.SFBay.Sun.COM [129.146.175.65]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m9AI4BRU032379; Fri, 10 Oct 2008 11:04:11 -0700 (PDT) Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9AI49SO026146; Fri, 10 Oct 2008 11:04:09 -0700 (PDT) Received: (from tedk@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id m9AI49Pg026142; Fri, 10 Oct 2008 11:04:09 -0700 (PDT) Date: Fri, 10 Oct 2008 11:04:09 -0700 (PDT) From: Ted Kim Subject: IBTF IO Memory [PSARC/2008/630 FastTrack timeout 10/17/2008] To: PSARC-ext@sun.com Message-id: <200810101804.m9AI49Pg026142@sac.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 3173 Template Version: @(#)sac_nextcase %I% %G% SMI This information is Copyright 2008 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: IBTF IO Memory 1.2. Name of Document Author/Supplier: Author: Lida Horn 1.3 Date of This Document: 10 October, 2008 4. Technical Description A. Background The DDI distinguishes between different "types" of memory. Memory from ddi_dma_mem_alloc(9F) is usable for DMA and takes into account various factors such alignment and other device attributes. Memory from kmem_(z)alloc is not guaranteed to be usable for DMA, though most of the time it does work for that purpose, because of the capability of modern platforms. Nevertheless, there is value in maintaining these DDI distinctions, especially when considering certain platform issues such as memory DR. In the context of InfiniBand, registered memory is the target of DMA operations. This case introduces new InfiniBand related interfaces analogous to the ddi_dma_mem_alloc family of functions to IBTF (InfiniBand Transport Framework, PSARC/2002/132 and follow-on cases). This addition will help the InfiniBand stack maintain the proper DDI memory distinctions important for certain types of platforms. B. Proposal The proposal is to make additions to the IBTF Channel and Transport interfaces. The functionality added to the Transport Interface (TI) is used by the ULPs to allocate memory suitable for DMA and IB memory registration. In turn, the framework uses new entry points in the Channel Interface (CI) to request memory allocation from the underlying HCA driver. These interfaces are basically a wrapper for DDI functions which on the one hand abstract away HCA device specific details at the ULP level, but at the same time allow for the HCA driver to adjust the memory attributes (alignment, etc.) as necessary for efficiency. These additions include an IBTF ABI change, so this case also marks an internal flag day, incrementing our interface version numbers for both the TI and CI as noted below. All interface additions and changes in this proposal have a micro/patch binding. Transport Interface (ON Consolidation Private): ibt_alloc_io_mem() - Allocates DMA memory (at the transport level) ibt_free_io_mem() - Deallocates DMA memory IBTI_V3 - TI version change Channel Interface (ON Consolidation Private): ibc_alloc_io_mem() - Allocates DMA memory (at the HCA driver level) ibc_free_io_mem() - Deallocates DMA memory IBCI_V3 - CI version change C. Summary of Changes by man page See materials directory for copies of man pages. Modified man pages have change bars in the left margin. ibci.9 - modfied (new CI entry points added) ibc_alloc_io_mem.9e - new (alloc & free CI entry points) ibt_alloc_io_mem.9f - new (alloc & free TI functions) ibt_clnt_modinfo_t.9s - modified (IBTI_V3 version) ibc_hca_info_t.9s - modified (IBCI_V3 version) ibc_operations_t.9s - modified (new CI entry points added) 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From Richard.Matthews@Sun.COM Wed Oct 15 10:59:16 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9FHxGX7004509 for ; Wed, 15 Oct 2008 10:59:16 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m9FHxGe7009603 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Wed, 15 Oct 2008 10:59:16 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8S00F0BKMR8900@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 15 Oct 2008 10:59:15 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8S00DHMKMQ2850@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 15 Oct 2008 10:59:14 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m9FHxEHf023710 for ; Wed, 15 Oct 2008 17:59:14 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8S00501IGFIH00@mail-amer.sun.com> (original mail from Richard.Matthews@Sun.COM) for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 15 Oct 2008 11:59:13 -0600 (MDT) Received: from [129.152.9.14] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8S00EWRKMOR750@mail-amer.sun.com>; Wed, 15 Oct 2008 11:59:12 -0600 (MDT) Date: Wed, 15 Oct 2008 12:59:12 -0500 From: Rick Matthews Subject: Re: IBTF IO Memory [PSARC/2008/630 FastTrack timeout 10/17/2008] In-reply-to: <200810101804.m9AI49Pg026142@sac.sfbay.sun.com> Sender: Richard.Matthews@Sun.COM To: Ted Kim , psarc-ext@Sun.COM Reply-to: Richard.Matthews@Sun.COM Message-id: <48F62F70.8030003@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200810101804.m9AI49Pg026142@sac.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 592 Minor nit: materials/ibc_operations_t.9s Add ibc_alloc_io_mem(9E) and ibt_alloc_io_mem(9F) to the "See also:" +1 -- --------------------------------------------------------------------- Rick Matthews email: Rick.Matthews@sun.com Sun Microsystems, Inc. phone:+1(651) 554-1518 1270 Eagan Industrial Road phone(internal): 54418 Suite 160 fax: +1(651) 554-1540 Eagan, MN 55121-1231 USA main: +1(651) 554-1500 --------------------------------------------------------------------- From Kais.Belgaied@sun.com Wed Oct 15 11:35:02 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9FIZ2mF006184 for ; Wed, 15 Oct 2008 11:35:02 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m9FIZ1VW003919 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@Sun.COM>; Wed, 15 Oct 2008 11:35:02 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8S0030RMACSG00@brm-avmta-1.central.sun.com> for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Wed, 15 Oct 2008 12:35:00 -0600 (MDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.63]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8S00FSIMABRCE0@brm-avmta-1.central.sun.com> for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Wed, 15 Oct 2008 12:34:59 -0600 (MDT) Received: from [129.146.11.144] (sr1-jurassic-01.SFBay.Sun.COM [129.146.11.144]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m9FIYwdj346338; Wed, 15 Oct 2008 11:34:58 -0700 (PDT) Date: Wed, 15 Oct 2008 11:34:58 -0700 From: Kais Belgaied Subject: Re: IBTF IO Memory [PSARC/2008/630 FastTrack timeout 10/17/2008] In-reply-to: <200810101804.m9AI49Pg026142@sac.sfbay.sun.com> To: Ted Kim Cc: PSARC-ext@sun.com Reply-to: Kais.Belgaied@sun.com Message-id: <48F637D2.5050206@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200810101804.m9AI49Pg026142@sac.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 5418 - Case boundary question: since this marks a flag day for both TI and CI, can you list the components that are affected by this flag day? - I am not clear on the consumer side of this new interface: What prompts a ULP to start using this interface? Is it expected to attempt ibt_alloc_io_mem() until it exhausts all resources? It would be easier to assess the completeness and the usefulness of the TI if you either extended the case's scope to include at least the changes on one transport consumer or gave a real example thereof. - mi_ibt_version seems to be an enumeration of apparently mutually exclusive values IBTI_V{1,2,3} yet the definition suggests a combination of independent (discrete) capabilities (FMR support, DMA wrapper support, etc.) . Is there any consumer of this interface that uses DMA wapper but not FMR? . For future evolution, is the mi_ibt_version always intended to express a monotonically increasing set of capabilities (capabs of V(n+1) includes all capabs of N(n)) ? Basically I'm trying to see if information of different nature if being encoded in the same field. Without slipping in a design discussion you should consider if two fileds are more appropriate: 1 version (number or enum) and one capabs (bitmask). - Under what condition can the caller of ibt_alloc_io_mem()/ibt_free_io_mem() expect the following error to be returned? 59 IBT_MR_ACCESS_REQ_INVALID Invalid Access Control Specified. 60 Remote Write or Remote Atomic access is 61 requested without specifying Local Write. - in ibc_alloc_io_mem.9e these two sections are in conflict: 11 ibt_status_t prefix_ibc_alloc_io_mem(ibc_hca_hdl_t hca_hdl, 12 size_t size, ibt_mr_flags_t mr_flag, caddr_t *kaddrp, 13 ibc_mem_alloc_hdl_t *mem_alloc_hdl); and 23 hca_hdl IBTF channel Interface (TI) HCA Handle previously obtained 24 by calling ibt_open_hca(9F). 25 Kais. On 10/10/08 11:04, Ted Kim wrote: > Template Version: @(#)sac_nextcase %I% %G% SMI > This information is Copyright 2008 Sun Microsystems > 1. Introduction > 1.1. Project/Component Working Name: > IBTF IO Memory > 1.2. Name of Document Author/Supplier: > Author: Lida HornI > 1.3 Date of This Document: > 10 October, 2008 > 4. Technical Description > > A. Background > > The DDI distinguishes between different "types" of memory. Memory from > ddi_dma_mem_alloc(9F) is usable for DMA and takes into account various > factors such alignment and other device attributes. Memory from > kmem_(z)alloc is not guaranteed to be usable for DMA, though most of > the time it does work for that purpose, because of the capability of > modern platforms. Nevertheless, there is value in maintaining these > DDI distinctions, especially when considering certain platform issues > such as memory DR. > > In the context of InfiniBand, registered memory is the target of DMA > operations. This case introduces new InfiniBand related interfaces > analogous to the ddi_dma_mem_alloc family of functions to IBTF > (InfiniBand Transport Framework, PSARC/2002/132 and follow-on > cases). This addition will help the InfiniBand stack maintain the > proper DDI memory distinctions important for certain types of > platforms. > > > B. Proposal > > The proposal is to make additions to the IBTF Channel and Transport > interfaces. The functionality added to the Transport Interface (TI) is > used by the ULPs to allocate memory suitable for DMA and IB memory > registration. In turn, the framework uses new entry points in the > Channel Interface (CI) to request memory allocation from the > underlying HCA driver. These interfaces are basically a wrapper for > DDI functions which on the one hand abstract away HCA device specific > details at the ULP level, but at the same time allow for the HCA > driver to adjust the memory attributes (alignment, etc.) as necessary > for efficiency. > > These additions include an IBTF ABI change, so this case also marks an > internal flag day, incrementing our interface version numbers for both > the TI and CI as noted below. > > > All interface additions and changes in this proposal have a > micro/patch binding. > > Transport Interface (ON Consolidation Private): > > ibt_alloc_io_mem() - Allocates DMA memory (at the transport level) > ibt_free_io_mem() - Deallocates DMA memory > IBTI_V3 - TI version change > > Channel Interface (ON Consolidation Private): > > ibc_alloc_io_mem() - Allocates DMA memory (at the HCA driver level) > ibc_free_io_mem() - Deallocates DMA memory > IBCI_V3 - CI version change > > > C. Summary of Changes by man page > > See materials directory for copies of man pages. Modified man pages > have change bars in the left margin. > > ibci.9 - modfied (new CI entry points added) > > ibc_alloc_io_mem.9e - new (alloc & free CI entry points) > > ibt_alloc_io_mem.9f - new (alloc & free TI functions) > > ibt_clnt_modinfo_t.9s - modified (IBTI_V3 version) > ibc_hca_info_t.9s - modified (IBCI_V3 version) > ibc_operations_t.9s - modified (new CI entry points added) > > > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > ON > 6.5. ARC review type: FastTrack > 6.6. ARC Exposure: open > > > From Ted.Kim@sun.com Wed Oct 15 11:37:09 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9FIb9vw006212 for ; Wed, 15 Oct 2008 11:37:09 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m9FIb7LR023836 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Wed, 15 Oct 2008 11:37:09 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8S00H09MDVP600@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 15 Oct 2008 11:37:07 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8S006PNMDULL70@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 15 Oct 2008 11:37:06 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m9FIb64D025411 for ; Wed, 15 Oct 2008 11:37:06 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8S00F01MD0V400@fe-sfbay-10.sun.com> (original mail from Ted.Kim@Sun.COM) for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 15 Oct 2008 11:37:06 -0700 (PDT) Received: from [129.153.3.74] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8S004BAMDMTGF0@fe-sfbay-10.sun.com>; Wed, 15 Oct 2008 11:36:59 -0700 (PDT) Date: Wed, 15 Oct 2008 11:34:25 -0700 From: "Ted H. Kim" Subject: Re: IBTF IO Memory [PSARC/2008/630 FastTrack timeout 10/17/2008] In-reply-to: <48F62F70.8030003@Sun.COM> Sender: Ted.Kim@sun.com To: Richard.Matthews@sun.com Cc: Ted Kim , PSARC-ext@sun.com, Lida Horn Message-id: <48F637B1.4030703@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200810101804.m9AI49Pg026142@sac.sfbay.sun.com> <48F62F70.8030003@Sun.COM> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 396 Okay, that's a trivial edit, we will do it no problem. Rick Matthews wrote: > Minor nit: > materials/ibc_operations_t.9s > > Add ibc_alloc_io_mem(9E) and ibt_alloc_io_mem(9F) to the "See also:" > > +1 -- Ted H. Kim Sun Microsystems, Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX From Ted.Kim@sun.com Wed Oct 15 12:10:23 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9FJAMTZ007532 for ; Wed, 15 Oct 2008 12:10:22 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m9FJAHd3021341 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 15 Oct 2008 20:10:21 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8S00I0PNX84T00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 15 Oct 2008 12:10:20 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8S00DWLNX828A0@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 15 Oct 2008 12:10:20 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m9FJAK98020238 for ; Wed, 15 Oct 2008 12:10:20 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8S00701NSH5S00@fe-sfbay-09.sun.com> (original mail from Ted.Kim@Sun.COM) for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Wed, 15 Oct 2008 12:10:20 -0700 (PDT) Received: from [129.153.3.74] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8S00ECWNX6KU50@fe-sfbay-09.sun.com>; Wed, 15 Oct 2008 12:10:19 -0700 (PDT) Date: Wed, 15 Oct 2008 12:07:45 -0700 From: "Ted H. Kim" Subject: Re: IBTF IO Memory [PSARC/2008/630 FastTrack timeout 10/17/2008] In-reply-to: <48F637D2.5050206@Sun.COM> Sender: Ted.Kim@sun.com To: Kais.Belgaied@sun.com Cc: Ted Kim , PSARC-ext@sun.com, Lida Horn Message-id: <48F63F81.6070206@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200810101804.m9AI49Pg026142@sac.sfbay.sun.com> <48F637D2.5050206@Sun.COM> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 4009 Kais, Kais Belgaied wrote: > - Case boundary question: since this marks a flag day for both TI and > CI, can you list the components > that are affected by this flag day? Most of the IB modules in ON - framework: IBTL IB ULPs (TI): IPonIB, SDP, NFS/RDMA, uDAPL HCA Drivers (CI): Tavor, Hermon > - I am not clear on the consumer side of this new interface: What > prompts a ULP to start using this interface? > Is it expected to attempt ibt_alloc_io_mem() until it exhausts all > resources? > It would be easier to assess the completeness and the usefulness of the > TI if you either extended the case's scope > to include at least the changes on one transport consumer or gave a > real example thereof. We are in the process of fixing bugs in certain IB ULPs to be good citizens in big SPARC platforms with memory DR where we have to be careful about what is in/out of the cage. The current plan for ULP usage (i.e. TI usage) is related to this motivation. > - mi_ibt_version seems to be an enumeration of apparently mutually > exclusive values IBTI_V{1,2,3} > yet the definition suggests a combination of independent (discrete) > capabilities > (FMR support, DMA wrapper support, etc.) The features are examples of what was included at each version change. More discussion of the relationship between ABI and features below ... > . Is there any consumer of this interface that uses DMA wapper but > not FMR? Well to be honest FMR in the current form turned out to be a failure. So no one uses FMR in ON right now. But I think more generally what is going to happen is that the features will be used independently of each other, since they are generally not related to each other. > . For future evolution, is the mi_ibt_version always intended to > express a monotonically increasing set > of capabilities (capabs of V(n+1) includes all capabs of N(n)) ? Yes, that is the intent, but it is not guaranteed. However, as you might imagine, it would involve a great deal of discussion/agreement to remove anything and the ARC would be in the loop. > Basically I'm trying to see if information of different nature if > being encoded in the same field. Without slipping > in a design discussion you should consider if two fileds are more > appropriate: 1 version (number or enum) and one capabs (bitmask). There are in fact capability bitmasks elsewhere. In IB there are a number of optional features. So the bitmasks generally are for saying you have these optional features. But the version number is more an ABI thing, and it is mistake to conflate the too, though the reason we have to change ABI is that new features demand more fields in the structs, etc. > - Under what condition can the caller of > ibt_alloc_io_mem()/ibt_free_io_mem() expect the following error > to be returned? > 59 IBT_MR_ACCESS_REQ_INVALID Invalid Access Control Specified. > 60 Remote Write or Remote Atomic access is > 61 requested without specifying Local > Write. It's a mistake and that text should be removed. That's only something that could happen during an IB memory registration. We will correct it. > - in ibc_alloc_io_mem.9e > these two sections are in conflict: > 11 ibt_status_t prefix_ibc_alloc_io_mem(ibc_hca_hdl_t hca_hdl, > 12 size_t size, ibt_mr_flags_t mr_flag, caddr_t *kaddrp, > 13 ibc_mem_alloc_hdl_t *mem_alloc_hdl); > and > 23 hca_hdl IBTF channel Interface (TI) HCA Handle previously > obtained > 24 by calling ibt_open_hca(9F). > 25 Yeah, it looks like a cut-n-paste mistake, since it mentioned the TI stuff instead of CI. It should say the handle comes from the ibc_hca_info_t returned via ibc_attach(). We will correct it. -ted -- Ted H. Kim Sun Microsystems, Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX From Ted.Kim@sun.com Wed Oct 15 14:42:27 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9FLgQ2C016261 for ; Wed, 15 Oct 2008 14:42:27 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m9FLgN1u000684 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Thu, 16 Oct 2008 05:42:25 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8S00I01UYO3I00@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 15 Oct 2008 15:42:24 -0600 (MDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8S005NWUYNXOB0@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 15 Oct 2008 15:42:23 -0600 (MDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m9FLgN9N017116 for ; Wed, 15 Oct 2008 14:42:23 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8S00801URE2A00@fe-sfbay-10.sun.com> (original mail from Ted.Kim@Sun.COM) for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 15 Oct 2008 14:42:23 -0700 (PDT) Received: from [129.153.3.74] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8S006X9UYEJO80@fe-sfbay-10.sun.com>; Wed, 15 Oct 2008 14:42:15 -0700 (PDT) Date: Wed, 15 Oct 2008 14:39:41 -0700 From: "Ted H. Kim" Subject: Re: IBTF IO Memory [PSARC/2008/630 FastTrack timeout 10/17/2008] Sender: Ted.Kim@sun.com To: Kais Belgaied , Richard.Matthews@sun.com Cc: "Ted H. Kim" , Lida Horn , psarc-ext@sun.com Message-id: <48F6631D.60907@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 665 Folks, We fixed up the man pages in the materials directory, so there are new versions of 1. ibt_alloc_io_mem.9f - remove the error about incompatible permissions 2. ibc_alloc_io_mem.9e - fix HCA handle description, plus the same fix as #1 here too 3. ibci.9 - add the see also ref (but only the ibc ref as this is describing CI things) I think this takes care of edits brought up so far. -ted -- Ted H. Kim Sun Microsystems, Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX From Ted.Kim@sun.com Wed Oct 15 14:46:32 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9FLkVWP016792 for ; Wed, 15 Oct 2008 14:46:31 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m9FLkQYo002167 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Thu, 16 Oct 2008 05:46:30 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K8S00203V5G1U00@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 15 Oct 2008 14:46:28 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K8S00LFLV5FAWC0@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 15 Oct 2008 14:46:27 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m9FLkRIi017597 for ; Wed, 15 Oct 2008 14:46:27 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K8S00K01V413E00@fe-sfbay-10.sun.com> (original mail from Ted.Kim@Sun.COM) for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 15 Oct 2008 14:46:27 -0700 (PDT) Received: from [129.153.3.74] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K8S006OXV5EJOA0@fe-sfbay-10.sun.com>; Wed, 15 Oct 2008 14:46:27 -0700 (PDT) Date: Wed, 15 Oct 2008 14:43:53 -0700 From: "Ted H. Kim" Subject: Re: IBTF IO Memory [PSARC/2008/630 FastTrack timeout 10/17/2008] In-reply-to: <48F6631D.60907@sun.com> Sender: Ted.Kim@sun.com To: Richard.Matthews@sun.com Cc: "Ted H. Kim" , Kais Belgaied , Lida Horn , psarc-ext@sun.com Message-id: <48F66419.7040800@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48F6631D.60907@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 776 Ted H. Kim wrote: > Folks, > > We fixed up the man pages in the materials directory, > so there are new versions of > > 1. ibt_alloc_io_mem.9f - remove the error about incompatible > permissions > > 2. ibc_alloc_io_mem.9e - fix HCA handle description, plus > the same fix as #1 here too > > 3. ibci.9 - add the see also ref (but only the ibc ref > as this is describing CI things) Sorry I mean ibc_operation_t.9s for this last item. > > I think this takes care of edits brought up so far. > > -ted > > -- Ted H. Kim Sun Microsystems, Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX From Kais.Belgaied@sun.com Mon Oct 20 14:43:23 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9KLhNgq020423 for ; Mon, 20 Oct 2008 14:43:23 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m9KLhKWw045900; Mon, 20 Oct 2008 15:43:22 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K920040F4C9E200@nwk-avmta-2.sfbay.sun.com>; Mon, 20 Oct 2008 14:43:21 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K9200LTO4C8L0C0@nwk-avmta-2.sfbay.sun.com>; Mon, 20 Oct 2008 14:43:20 -0700 (PDT) Received: from [129.146.11.145] (sr1-jurassic-02.SFBay.Sun.COM [129.146.11.145]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m9KLhJnw226871; Mon, 20 Oct 2008 14:43:20 -0700 (PDT) Date: Mon, 20 Oct 2008 14:43:19 -0700 From: Kais Belgaied Subject: Re: IBTF IO Memory [PSARC/2008/630 FastTrack timeout 10/17/2008] In-reply-to: <48F63F81.6070206@sun.com> To: "Ted H. Kim" Cc: Ted Kim , PSARC-ext@sun.com, Lida Horn Reply-to: Kais.Belgaied@sun.com Message-id: <48FCFB77.1070100@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200810101804.m9AI49Pg026142@sac.sfbay.sun.com> <48F637D2.5050206@Sun.COM> <48F63F81.6070206@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 3110 On 10/15/08 12:07, Ted H. Kim wrote: > Kais, > > > Kais Belgaied wrote: >> - Case boundary question: since this marks a flag day for both TI and >> CI, can you list the components >> that are affected by this flag day? > > Most of the IB modules in ON - > framework: IBTL > IB ULPs (TI): IPonIB, SDP, NFS/RDMA, uDAPL > HCA Drivers (CI): Tavor, Hermon at the risk of re-stating the obvious, all changes in the above 3 sets of components are in-scope of this case, right? > > >> - I am not clear on the consumer side of this new interface: What >> prompts a ULP to start using this interface? >> Is it expected to attempt ibt_alloc_io_mem() until it exhausts all >> resources? >> It would be easier to assess the completeness and the usefulness of >> the TI if you either extended the case's scope >> to include at least the changes on one transport consumer or gave a >> real example thereof. > > We are in the process of fixing bugs in certain IB ULPs to > be good citizens in big SPARC platforms with memory DR where > we have to be careful about what is in/out of the cage. > The current plan for ULP usage (i.e. TI usage) is related > to this motivation. > > >> - mi_ibt_version seems to be an enumeration of apparently mutually >> exclusive values IBTI_V{1,2,3} >> yet the definition suggests a combination of independent (discrete) >> capabilities >> (FMR support, DMA wrapper support, etc.) > > The features are examples of what was included at each version change. > More discussion of the relationship between ABI and features below ... > > >> . Is there any consumer of this interface that uses DMA wapper >> but not FMR? > > Well to be honest FMR in the current form turned out to be a failure. > So no one uses FMR in ON right now. > > But I think more generally what is going to happen is that > the features will be used independently of each other, > since they are generally not related to each other. > > >> . For future evolution, is the mi_ibt_version always intended to >> express a monotonically increasing set >> of capabilities (capabs of V(n+1) includes all capabs of N(n)) ? > > Yes, that is the intent, but it is not guaranteed. However, > as you might imagine, it would involve a great deal of > discussion/agreement to remove anything and the ARC would be > in the loop. > > >> Basically I'm trying to see if information of different nature if >> being encoded in the same field. Without slipping >> in a design discussion you should consider if two fileds are more >> appropriate: 1 version (number or enum) and one capabs (bitmask). > > There are in fact capability bitmasks elsewhere. > In IB there are a number of optional features. So the bitmasks > generally are for saying you have these optional features. > > But the version number is more an ABI thing, and it is mistake > to conflate the too, though the reason we have to change ABI > is that new features demand more fields in the structs, etc. so are you fixing the mistake of conflating the capabs + version ? I still see the updated material unchanged on that. > > Kais. From Ted.Kim@sun.com Mon Oct 20 15:04:01 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9KM40kr021463 for ; Mon, 20 Oct 2008 15:04:01 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m9KM3joV014089 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Tue, 21 Oct 2008 06:03:59 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K9200F0F5AN5Q00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Mon, 20 Oct 2008 16:03:59 -0600 (MDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K9200KMJ5ALYKE0@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Mon, 20 Oct 2008 16:03:57 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m9KM3v8Y021763 for ; Mon, 20 Oct 2008 15:03:57 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K9200C0158VGC00@fe-sfbay-09.sun.com> (original mail from Ted.Kim@Sun.COM) for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Mon, 20 Oct 2008 15:03:57 -0700 (PDT) Received: from [129.153.3.74] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K92001HP5AHIU30@fe-sfbay-09.sun.com>; Mon, 20 Oct 2008 15:03:54 -0700 (PDT) Date: Mon, 20 Oct 2008 15:01:13 -0700 From: "Ted H. Kim" Subject: Re: IBTF IO Memory [PSARC/2008/630 FastTrack timeout 10/17/2008] In-reply-to: <48FCFB77.1070100@Sun.COM> Sender: Ted.Kim@sun.com To: Kais.Belgaied@sun.com Cc: Ted Kim , PSARC-ext@sun.com, Lida Horn Message-id: <48FCFFA9.4040401@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200810101804.m9AI49Pg026142@sac.sfbay.sun.com> <48F637D2.5050206@Sun.COM> <48F63F81.6070206@sun.com> <48FCFB77.1070100@Sun.COM> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 2849 Kais Belgaied wrote: >>> - Case boundary question: since this marks a flag day for both TI and >>> CI, can you list the components >>> that are affected by this flag day? >> >> Most of the IB modules in ON - >> framework: IBTL >> IB ULPs (TI): IPonIB, SDP, NFS/RDMA, uDAPL >> HCA Drivers (CI): Tavor, Hermon > > at the risk of re-stating the obvious, all changes in the above 3 sets > of components are in-scope of this case, > right? Yes >>> - mi_ibt_version seems to be an enumeration of apparently mutually >>> exclusive values IBTI_V{1,2,3} >>> yet the definition suggests a combination of independent (discrete) >>> capabilities >>> (FMR support, DMA wrapper support, etc.) >> >> The features are examples of what was included at each version change. >> More discussion of the relationship between ABI and features below ... >> >>> . Is there any consumer of this interface that uses DMA wapper >>> but not FMR? >> >> Well to be honest FMR in the current form turned out to be a failure. >> So no one uses FMR in ON right now. >> >> But I think more generally what is going to happen is that >> the features will be used independently of each other, >> since they are generally not related to each other. >> >> >>> . For future evolution, is the mi_ibt_version always intended to >>> express a monotonically increasing set >>> of capabilities (capabs of V(n+1) includes all capabs of N(n)) ? >> >> Yes, that is the intent, but it is not guaranteed. However, >> as you might imagine, it would involve a great deal of >> discussion/agreement to remove anything and the ARC would be >> in the loop. >> >> >>> Basically I'm trying to see if information of different nature if >>> being encoded in the same field. Without slipping >>> in a design discussion you should consider if two fileds are more >>> appropriate: 1 version (number or enum) and one capabs (bitmask). >> >> There are in fact capability bitmasks elsewhere. >> In IB there are a number of optional features. So the bitmasks >> generally are for saying you have these optional features. >> >> But the version number is more an ABI thing, and it is mistake >> to conflate the too, though the reason we have to change ABI >> is that new features demand more fields in the structs, etc. > > so are you fixing the mistake of conflating the capabs + version ? I > still see the updated material unchanged > on that. I think it's a mistake in *understanding* to not distinguish API from ABI. I am trying to clarify that. I don't think it is a mistake in the design. I am also not sure where this is leading. Are you suggesting some specific change to the case? -ted -- Ted H. Kim Sun Microsystems, Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX From Ted.Kim@sun.com Mon Oct 20 15:22:24 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9KMMNOx022074 for ; Mon, 20 Oct 2008 15:22:23 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m9KMME2I020921 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Tue, 21 Oct 2008 06:22:22 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K92006096593J00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Mon, 20 Oct 2008 15:22:21 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K92005F0659VM00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Mon, 20 Oct 2008 15:22:21 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m9KMMLoW023762 for ; Mon, 20 Oct 2008 15:22:21 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K920030163ZV800@fe-sfbay-10.sun.com> (original mail from Ted.Kim@Sun.COM) for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Mon, 20 Oct 2008 15:22:21 -0700 (PDT) Received: from [129.153.3.74] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K9200KZF64TYQ10@fe-sfbay-10.sun.com>; Mon, 20 Oct 2008 15:22:06 -0700 (PDT) Date: Mon, 20 Oct 2008 15:19:25 -0700 From: "Ted H. Kim" Subject: Re: IBTF IO Memory [PSARC/2008/630 FastTrack timeout 10/17/2008] In-reply-to: <48FCFFA9.4040401@sun.com> Sender: Ted.Kim@sun.com To: "Ted H. Kim" Cc: Kais.Belgaied@sun.com, Ted Kim , PSARC-ext@sun.com, Lida Horn Message-id: <48FD03ED.2000109@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200810101804.m9AI49Pg026142@sac.sfbay.sun.com> <48F637D2.5050206@Sun.COM> <48F63F81.6070206@sun.com> <48FCFB77.1070100@Sun.COM> <48FCFFA9.4040401@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 1266 Ted H. Kim wrote: > > > Kais Belgaied wrote: >>>> - Case boundary question: since this marks a flag day for both TI >>>> and CI, can you list the components >>>> that are affected by this flag day? >>> >>> Most of the IB modules in ON - >>> framework: IBTL >>> IB ULPs (TI): IPonIB, SDP, NFS/RDMA, uDAPL >>> HCA Drivers (CI): Tavor, Hermon >> >> at the risk of re-stating the obvious, all changes in the above 3 >> sets of components are in-scope of this case, >> right? > > Yes Actually, I think I made a mistake to say this. It's true to the extent that the three sets of components will use the interfaces here. But I think saying an unqualified yes means we have to deliver all modified components at once. And so I am going to say the right answer is YES to only the framework and HCA drivers. This allows us to deliver the changes to the framework and HCA drivers first. And then phase the delivery of the ULP mods. That way if there are customer priorities, we can deliver the ones they want first without having to wait to have all of the ULPs modified. -ted -- Ted H. Kim Sun Microsystems, Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX From Kais.Belgaied@sun.com Mon Oct 20 15:51:59 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9KMpwjR023039 for ; Mon, 20 Oct 2008 15:51:59 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m9KMpmQ3009051; Mon, 20 Oct 2008 23:51:56 +0100 (BST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K9200I077IIPX00@brm-avmta-1.central.sun.com>; Mon, 20 Oct 2008 16:51:54 -0600 (MDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.63]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K9200H487II2920@brm-avmta-1.central.sun.com>; Mon, 20 Oct 2008 16:51:54 -0600 (MDT) Received: from [129.146.11.145] (sr1-jurassic-02.SFBay.Sun.COM [129.146.11.145]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m9KMprYM238270; Mon, 20 Oct 2008 15:51:54 -0700 (PDT) Date: Mon, 20 Oct 2008 15:51:53 -0700 From: Kais Belgaied Subject: Re: IBTF IO Memory [PSARC/2008/630 FastTrack timeout 10/17/2008] In-reply-to: <48FCFFA9.4040401@sun.com> To: "Ted H. Kim" Cc: PSARC-ext@sun.com, Lida Horn Reply-to: Kais.Belgaied@sun.com Message-id: <48FD0B89.4040509@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200810101804.m9AI49Pg026142@sac.sfbay.sun.com> <48F637D2.5050206@Sun.COM> <48F63F81.6070206@sun.com> <48FCFB77.1070100@Sun.COM> <48FCFFA9.4040401@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 900 On 10/20/08 15:01, Ted H. Kim wrote: > > I am also not sure where this is leading. > Are you suggesting some specific change to the case? I'm not clear on the future compatibility expectations around the interface introduced by this case: I asked whether versions are incremental all the time and you answered yes ((capabs of V(n+1) includes all capabs of V(n)) I asked if some capabs can be used independently, you also said yes, which suggests (capabs of V(n+1)) don't necessarily have to include all capabs of V(n). Capabs are independent. The former means that an IBT client module written to V(n) is guaranteed to work unmodified on a framework+HCAs that evolved to V(n+1) or later. The latter means modules may break or may continue to work. No backward compatibility is guaranteed. Choose one semantic for the interface and clearly document it in the case. Kais. > > -ted > From Ted.Kim@sun.com Mon Oct 20 16:03:47 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9KN3kTo023650 for ; Mon, 20 Oct 2008 16:03:46 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m9KN3Rui007235 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Tue, 21 Oct 2008 07:03:45 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K9200709827XL00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 20 Oct 2008 16:03:43 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K92005WL826VM30@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 20 Oct 2008 16:03:42 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m9KN3dFg028400 for ; Mon, 20 Oct 2008 16:03:42 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K92001017Z5OE00@fe-sfbay-09.sun.com> (original mail from Ted.Kim@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 20 Oct 2008 16:03:39 -0700 (PDT) Received: from [129.153.3.74] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K9200C8B81HUDB0@fe-sfbay-09.sun.com>; Mon, 20 Oct 2008 16:03:18 -0700 (PDT) Date: Mon, 20 Oct 2008 16:00:35 -0700 From: "Ted H. Kim" Subject: Re: IBTF IO Memory [PSARC/2008/630 FastTrack timeout 10/17/2008] In-reply-to: <48FD0B89.4040509@Sun.COM> Sender: Ted.Kim@sun.com To: Kais.Belgaied@sun.com Cc: PSARC-ext@sun.com, Lida Horn Message-id: <48FD0D93.7070605@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200810101804.m9AI49Pg026142@sac.sfbay.sun.com> <48F637D2.5050206@Sun.COM> <48F63F81.6070206@sun.com> <48FCFB77.1070100@Sun.COM> <48FCFFA9.4040401@sun.com> <48FD0B89.4040509@Sun.COM> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 1457 We will modify the case to say: IBTI_V3 is incompatible with other IBTI versions. IBCI_V3 is incompatible with other IBCI versions. When IBTI changes, you have to recompile all the in-kernel ULPs and the IBTF framework. When IBCI changes, you have to recompile the IBTF framework and all the HCA drivers. -ted Kais Belgaied wrote: > On 10/20/08 15:01, Ted H. Kim wrote: >> >> I am also not sure where this is leading. >> Are you suggesting some specific change to the case? > > I'm not clear on the future compatibility expectations around the > interface introduced by this case: > > I asked whether versions are incremental all the time and you answered yes > ((capabs of V(n+1) includes all capabs of V(n)) > I asked if some capabs can be used independently, you also said yes, > which suggests > (capabs of V(n+1)) don't necessarily have to include all capabs of V(n). > Capabs are independent. > > The former means that an IBT client module written to V(n) is guaranteed > to work unmodified on a framework+HCAs > that evolved to V(n+1) or later. > > The latter means modules may break or may continue to work. No backward > compatibility is guaranteed. > > Choose one semantic for the interface and clearly document it in the case. > > Kais. -- Ted H. Kim Sun Microsystems, Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX From Kais.Belgaied@sun.com Tue Oct 21 12:03:25 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9LJ3PaW016121 for ; Tue, 21 Oct 2008 12:03:25 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m9LJ3FTt007845; Tue, 21 Oct 2008 12:03:25 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K9300F31RLOZD00@nwk-avmta-2.sfbay.sun.com>; Tue, 21 Oct 2008 12:03:24 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K9300EV3RLI1760@nwk-avmta-2.sfbay.sun.com>; Tue, 21 Oct 2008 12:03:18 -0700 (PDT) Received: from [129.146.11.145] (sr1-jurassic-02.SFBay.Sun.COM [129.146.11.145]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m9LJ303v402078; Tue, 21 Oct 2008 12:03:06 -0700 (PDT) Date: Tue, 21 Oct 2008 12:02:57 -0700 From: Kais Belgaied Subject: Re: IBTF IO Memory [PSARC/2008/630 FastTrack timeout 10/17/2008] In-reply-to: <48FD0D93.7070605@sun.com> To: "Ted H. Kim" Cc: PSARC-ext@sun.com, Lida Horn Reply-to: Kais.Belgaied@sun.com Message-id: <48FE2761.3000006@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200810101804.m9AI49Pg026142@sac.sfbay.sun.com> <48F637D2.5050206@Sun.COM> <48F63F81.6070206@sun.com> <48FCFB77.1070100@Sun.COM> <48FCFFA9.4040401@sun.com> <48FD0B89.4040509@Sun.COM> <48FD0D93.7070605@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 1479 No further comments from me. Ted, since the timer expired on this case, feel free to mark it closed. Kais. On 10/20/08 16:00, Ted H. Kim wrote: > We will modify the case to say: > > IBTI_V3 is incompatible with other IBTI versions. > IBCI_V3 is incompatible with other IBCI versions. > > When IBTI changes, you have to recompile > all the in-kernel ULPs and the IBTF framework. > > When IBCI changes, you have to recompile > the IBTF framework and all the HCA drivers. > > -ted > > > > Kais Belgaied wrote: >> On 10/20/08 15:01, Ted H. Kim wrote: >>> >>> I am also not sure where this is leading. >>> Are you suggesting some specific change to the case? >> >> I'm not clear on the future compatibility expectations around the >> interface introduced by this case: >> >> I asked whether versions are incremental all the time and you >> answered yes >> ((capabs of V(n+1) includes all capabs of V(n)) >> I asked if some capabs can be used independently, you also said yes, >> which suggests >> (capabs of V(n+1)) don't necessarily have to include all capabs of >> V(n). Capabs are independent. >> >> The former means that an IBT client module written to V(n) is >> guaranteed to work unmodified on a framework+HCAs >> that evolved to V(n+1) or later. >> >> The latter means modules may break or may continue to work. No >> backward compatibility is guaranteed. >> >> Choose one semantic for the interface and clearly document it in the >> case. >> >> Kais. > > From Ted.Kim@sun.com Tue Oct 21 15:04:46 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9LM4jFU023033 for ; Tue, 21 Oct 2008 15:04:46 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m9LM4RNa011071 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Tue, 21 Oct 2008 23:04:44 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K9300009ZZWI300@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 21 Oct 2008 15:04:44 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K9300KCIZZV5JF0@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 21 Oct 2008 15:04:43 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m9LM4h3t012327 for ; Tue, 21 Oct 2008 15:04:43 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K9300E01ZLLYE00@fe-sfbay-09.sun.com> (original mail from Ted.Kim@Sun.COM) for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 21 Oct 2008 15:04:43 -0700 (PDT) Received: from [129.153.3.74] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K9300H2WZZT1C90@fe-sfbay-09.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 21 Oct 2008 15:04:42 -0700 (PDT) Date: Tue, 21 Oct 2008 15:01:58 -0700 From: "Ted H. Kim" Subject: Re: IBTF IO Memory [PSARC/2008/630 FastTrack timeout 10/17/2008] Sender: Ted.Kim@sun.com To: psarc-ext@sun.com Cc: "Ted H. Kim" , Lida Horn Message-id: <48FE5156.7070506@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 4620 Final version of the case -- including a summary of the last parts of the discussion, about motivation, versioning and scope. -ted Diffs: 11c11,14 < such as memory DR. --- > such as memory DR. In fact, one of the primary motivations of the work > is to enable fixing various IB bugs on large systems, where we need to > be careful about being in/out of the cage and the possible impact to > memory DR. 37c40,44 < the TI and CI as noted below. --- > the TI and CI as noted below. Each ABI version is incompatible with > each other and so this change will require a recompile of kernel ULPs, > framework and HCA drivers. All IB kernel components are delivered by > Sun engineering and are delivered together in our releases, so this > type of ABI change is not noticed by our customers. 38a46,49 > The scope of the project includes changes at the framework and HCA > driver level, which will be delivered together. The ULPs using these > new features, however, will be phased according to customer based > priorities. ==== A. Background The DDI distinguishes between different "types" of memory. Memory from ddi_dma_mem_alloc(9F) is usable for DMA and takes into account various factors such alignment and other device attributes. Memory from kmem_(z)alloc is not guaranteed to be usable for DMA, though most of the time it does work for that purpose, because of the capability of modern platforms. Nevertheless, there is value in maintaining these DDI distinctions, especially when considering certain platform issues such as memory DR. In fact, one of the primary motivations of the work is to enable fixing various IB bugs on large systems, where we need to be careful about being in/out of the cage and the possible impact to memory DR. In the context of InfiniBand, registered memory is the target of DMA operations. This case introduces new InfiniBand related interfaces analogous to the ddi_dma_mem_alloc family of functions to IBTF (InfiniBand Transport Framework, PSARC/2002/132 and follow-on cases). This addition will help the InfiniBand stack maintain the proper DDI memory distinctions important for certain types of platforms. B. Proposal The proposal is to make additions to the IBTF Channel and Transport interfaces. The functionality added to the Transport Interface (TI) is used by the ULPs to allocate memory suitable for DMA and IB memory registration. In turn, the framework uses new entry points in the Channel Interface (CI) to request memory allocation from the underlying HCA driver. These interfaces are basically a wrapper for DDI functions which on the one hand abstract away HCA device specific details at the ULP level, but at the same time allow for the HCA driver to adjust the memory attributes (alignment, etc.) as necessary for efficiency. These additions include an IBTF ABI change, so this case also marks an internal flag day, incrementing our interface version numbers for both the TI and CI as noted below. Each ABI version is incompatible with each other and so this change will require a recompile of kernel ULPs, framework and HCA drivers. All IB kernel components are delivered by Sun engineering and are delivered together in our releases, so this type of ABI change is not noticed by our customers. The scope of the project includes changes at the framework and HCA driver level, which will be delivered together. The ULPs using these new features, however, will be phased according to customer based priorities. All interface additions and changes in this proposal have a micro/patch binding. Transport Interface (ON Consolidation Private): ibt_alloc_io_mem() - Allocates DMA memory (at the transport level) ibt_free_io_mem() - Deallocates DMA memory IBTI_V3 - TI version change Channel Interface (ON Consolidation Private): ibc_alloc_io_mem() - Allocates DMA memory (at the HCA driver level) ibc_free_io_mem() - Deallocates DMA memory IBCI_V3 - CI version change C. Summary of Changes by man page See materials directory for copies of man pages. Modified man pages have change bars in the left margin. ibci.9 - modfied (new CI entry points added) ibc_alloc_io_mem.9e - new (alloc & free CI entry points) ibt_alloc_io_mem.9f - new (alloc & free TI functions) ibt_clnt_modinfo_t.9s - modified (IBTI_V3 version) ibc_hca_info_t.9s - modified (IBCI_V3 version) ibc_operations_t.9s - modified (new CI entry points added) -- Ted H. Kim Sun Microsystems, Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX From Ted.Kim@sun.com Wed Oct 22 07:10:40 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9MEAd4g014733 for ; Wed, 22 Oct 2008 07:10:40 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m9MEAX8B018711 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Wed, 22 Oct 2008 15:10:38 +0100 (BST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K9500E3R8PN9Y00@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 22 Oct 2008 08:10:35 -0600 (MDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K95003F38PNOG80@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 22 Oct 2008 08:10:35 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m9MEAZR0022093 for ; Wed, 22 Oct 2008 07:10:35 -0700 (PDT) Received: from sun.com ([129.145.47.28]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTP id <0K950036L8PNK2C0@fe-sfbay-09.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 22 Oct 2008 07:10:35 -0700 (PDT) Received: from [192.18.34.118] (Forwarded-For: [129.150.16.62]) by sfbay4-mail1.sfbay.sun.com (mshttpd); Wed, 22 Oct 2008 07:10:35 -0700 Date: Wed, 22 Oct 2008 07:10:35 -0700 From: Ted Kim Subject: Re: IBTF IO Memory [PSARC/2008/630 FastTrack timeout 10/17/2008] In-reply-to: <48FE5156.7070506@sun.com> To: psarc-ext@sun.com Cc: Lida Horn , "Ted H. Kim" Message-id: MIME-version: 1.0 X-Mailer: Sun Java(tm) System Messenger Express 6.2-8.04 (built Feb 28 2007) Content-type: text/plain; charset=us-ascii Content-language: en Content-transfer-encoding: 7BIT Content-disposition: inline X-Accept-Language: en Priority: normal X-PMX-Version: 5.4.1.325704 References: <48FE5156.7070506@sun.com> Status: RO Content-Length: 333 Okay we are past the timeout, plus the participants have now finished the discussion. I am going to close the case as approved. -ted Ted H. Kim Sun Microsystems, Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX