From tedk@sac.sfbay.sun.com Fri Nov 21 09:24:49 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mALHOmxh009463 for ; Fri, 21 Nov 2008 09:24:48 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mALHOlIt022408; Fri, 21 Nov 2008 10:24:48 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAP00F0Z1PBTH00@nwk-avmta-2.sfbay.sun.com>; Fri, 21 Nov 2008 09:24:47 -0800 (PST) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAP005LS1PBX8E0@nwk-avmta-2.sfbay.sun.com>; Fri, 21 Nov 2008 09:24:47 -0800 (PST) Received: from sac.sfbay.sun.com (new-sac.SFBay.Sun.COM [129.146.175.65]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mALHOlHo027322; Fri, 21 Nov 2008 09:24:47 -0800 (PST) Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mALHOjVq009458; Fri, 21 Nov 2008 09:24:45 -0800 (PST) Received: (from tedk@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id mALHOj3v009454; Fri, 21 Nov 2008 09:24:45 -0800 (PST) Date: Fri, 21 Nov 2008 09:24:45 -0800 (PST) From: Ted Kim Subject: New HCA Capabilities [PSARC/2008/726 FastTrack timeout 12/02/2008] To: PSARC-ext@sun.com Cc: bill.taylor@sun.com, ted.kim@sun.com Message-id: <200811211724.mALHOj3v009454@sac.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 15559 Template Version: @(#)sac_nextcase %I% %G% SMI This information is Copyright 2008 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: New HCA Capabilities 1.2. Name of Document Author/Supplier: Author: Bill Taylor 1.3 Date of This Document: 21 November, 2008 4. Technical Description New HCA Attributes & Capabilities --------------------------------- 4.1. Background Over time, InfiniBand (IB) Host Channel Adapters (HCAs) have accumulated a number of new features. Further, IB software has placed new demands on HCAs. In order to accommodate these developments, this proposal defines a series of new interfaces for the InfiniBand Transport Framework (IBTF, PSARC/2002/132 and follow-on cases). The items in this proposal are: firmware version information, SQD flag, inline data, Reserved L_Key, LSO, CQ Moderation, IP classification flags and detailed WR sizes. 4.2. Proposal The proposal is to make additions to the IBTF Channel Interface (CI) and Transport Interface (TI). The functionality added to the TI is for use by IB clients. CI changes affect the framework interface to HCA drivers. All interface additions and changes in this proposal have a micro/patch binding. Transport Interface (ON Consolidation Private) ibt_map_mem_iov(): new map IOV function ibt_unmap_mem_iov(): new unmap IOV function ibt_modify_cq(): new CQ moderation call ibt_query_cq(): changed function signature ibt_attr_flags_t: Reserved L_Key permission flag, LSO flag ibt_chan_sizes_t: inline size ibt_hca_attr_t: firmware version fields, SQD flag, inline data size, Reserved L_Key flag & value, LSO max payload & hdr size, CQ moderation max params, IP classification support flag, "detailed" WQE size info ibt_send_wr_t, ibt_wr_lso_t: LSO work request format ibt_status_t: WQE too small error ibt_wc_t: IP classification flags ibt_wr_flags_t: inline flag ibt_wrc_opcode_t: LSO opcode Channel Interface (ON Consolidation Private) ibc_operations_t: ibc_cq_modify entry point, ibc_map_mem_iov entry point, changed ibc_query_cq signature, ibc_unmap_mem_iov entry point ibt_attr_flags_t: Reserved L_Key permission flag, LSO flag ibt_chan_sizes_t: inline size ibt_hca_attr_t: firmware version fields, SQD flag, inline data size, Reserved L_Key flag & value, LSO max payload & hdr size, CQ moderation max params, IP classification support flag, "detailed" WQE size info ibt_send_wr_t, ibt_wr_lso_t: LSO work request format ibt_status_t: WQE too small error ibt_wc_t: IP classification flags ibt_wr_flags_t: inline flag ibt_wrc_opcode_t: LSO opcode All these changes are part of the v3 IBTF ABI first introduced in PSARC/2008/630. Copies of all modified/added man pages are in the materials directory (see section 4.3 below). A. Firmware Version Occassionally, some IB software wants to know the firmware version of the HCA. Up to now, this information was only known to the HCA driver and firmware flash tool. The firmware version information takes the form of three new fields added to the ibt_hca_attr_t struct: uint32_t hca_fw_major_version; uint16_t hca_fw_minor_version; uint16_t hca_fw_micro_version; Man page changes: ibt_hca_attr_t.9s B. SQD flag The IBTF baseline for HCA functionality is version 1.1 of the IB specification. Unfortunately, the ConnectX device (serviced by the Hermon HCA driver, PSARC/2008/497) currently does not implement some parts of the 1.1 standard: most notably the Send Queue Drain (SQD) Queue Pair (QP) state (see section 10.3.1.5 in the IB spec [1]). So IBTF will now explicitly show when the SQD state is available. To show when SQD is implemented, this flag is added to ib_hca_flags_t: IBT_HCA_SQD_STATE = 1 << 30, /* SQD QP state */ If SQD is not supported, an attempt to transition to that state will result in a IBT_CHAN_STATE_INVALID (TI) or IBT_QP_STATE_INVALID (CI) error (already defined in the interface). Man page changes: ibt_hca_attr_t.9s C. Inline Data IB work requests (WRs) include a scatter-gather list (SGL) of data segments which reference the message payload. After posting, the WR is stored in a device specific Work Queue Entry (WQE) format until processing. Some HCAs support an "inline data" WQE format where the SGL portion of the WQE is replaced by the actual payload data. Using inline data can improve performance, since fetching the WQE also retrieves the payload, eliminating subsequent DMAs to reference through the WQE SGL. Further, data stored inline does not have to be in registered memory, possibly removing a memory registration step. There are limitations on this technique. Typically a WQE cannot be very large, so this only works for small messages. Also the data must be available at posting time, so there cannot be a data dependency between WRs on a QP. Finally, this technique is limited to SEND and RDMA-WRITE operations. New field for HCA maximum size of inline data in ibt_hca_attr_t: uint_t hca_max_inline_size; /* in bytes */ A value of zero indicates there is no data inline support. New field for QP-specific size of inline data in ibt_chan_sizes_t: uint_t cs_inline; This field is used during both channel allocation/modification (to size the WQE) and during channel query (to show the current supported inline size). New WR flag in ibt_wr_flags_t: #define IBT_WR_SEND_INLINE (1 << 6) Ignore L_Keys in SGL and copy data inline when this WR is posted. Valid only for SEND and RDMA-WRITE WRs, ignored for other WRs. New ibt_status_t value: IBT_CHAN_WQE_SZ_INSUFF = 417, /* TI */ #define IBT_QP_WQE_SZ_INSUFF IBT_CHAN_WQE_SZ_INSUFF /* CI */ Indicates that the specified inline data payload was too large. Man page changes: ibc_post_send.9e., ibt_post_send.9f, ibt_chan_sizes_t.9s, ibt_hca_attr_t.9s and ibt_send_wr_t.9s D. Reserved L_Key The Reserved L_Key concept was introduced in the 1.2 IB spec (see 10.6.4.3.2 in [1]). Basically, the idea is to provide a pre-defined memory L_Key which references the whole I/O bus physical address space of the HCA. This allows physical bus addresses to be used in WRs, typically after doing a virtual to physical address mapping using ibt_map_mem_iov(). This method is faster than memory registration, because the HCA memory table state does not need to be updated. For security reasons, it should be noted this is an L_Key (not an R_Key), and therefore it cannot be used as the target of a remote RDMA. Also, this is a privileged feature, not typically enabled in a user process QP. New Reserved L_Key support flag in ibt_hca_flags2_t: IBT_HCA2_RES_LKEY = 1 << 3, /* Reserved L_Key */ New Reserved L_Key value field in ibt_hca_attr_t: ibt_lkey_t hca_reserved_lkey; /* Reserved L_Key value */ New QP permission flag in ibt_attr_flags_t: IBT_FAST_REG_RES_LKEY = (1 << 1) This flag is used on channel/QP allocation to enable Reserved L_Key. Query channel/QP operations read the value. Per the IB spec (see 11.2.4.1 in [1]), this permission also controls the Fast Memory Registration feature (which is not implemented yet). New map memory IOV operation to fill out SGL list for Reserved L_Key: ibt_status_t ibt_map_mem_iov(ibt_hca_hdl_t hca_hdl, /* TI */ ibt_iov_attr_t *iov_attr, ibt_all_wr_t *wr, ibt_mi_hdl_t *mi_hdl); ibc_status_t prefix_ibc_map_mem_iov(ibc_hca_hdl_t hca_hdl, /* CI */ ibt_iov_attr_t *iov_attr, ibt_all_wr_t *wr, ibc_mi_hdl_t *mi_hdl); New unmap memory IOV operations: ibt_status_t ibt_unmap_mem_iov(ibt_hca_hdl_t hca_hdl, /* TI */ ibt_mi_hdl_t mi_hdl); ibc_status_t prefix_ibc_unmap_mem_iov(ibt_hca_hdl_t hca_hdl, /* CI */ ibc_mi_hdl_t mi_hdl); Man page changes: ibci.9, ibti.9, ibc_alloc_qp.9e, ibc_map_mem_iov.9e, ibt_alloc_rc_chan.9f, ibt_map_mem_iov.9f ibt_alloc_ud_chan.9f and ibt_hca_attr_t.9s E. Large Send Offload (LSO) HCAs have implemented stateless offloads similar to those in Ethernet NICs. In LSO, a large IP payload is sent down to the HCA along with a "template" for the header. The header template is based on the IPonIB encapsulation format for TCP/IP (IPv4 or v6). The HCA chops up the big payload into pieces that fit with the header in MTU size packets (IB MTU is typically 2 or 4 KB). The header in each packet is generated by the HCA based on the template provided. New field for HCA maximum LSO payload & header size in ibt_hca_attr_t: uint_t hca_max_lso_size; /* in bytes */ uint_t hca_max_lso_hdr_size; /* in bytes */ Zero values mean the HCA does not support LSO. New flag for LSO usage at QP/channel allocation time in ibt_attr_flags_t: IBT_USES_LSO = (1 << 2) New LSO work request/completion opcode value in ibt_wrc_opcode_t: #define IBT_WRC_SEND_LSO 11 New WR format for LSO in ibt_send_wr_t (new "wr" field union variant): typedef struct ibt_wr_lso_s { ibt_ud_dest_hdl_t lso_ud_dest; /* address handle */ uint8_t *lso_hdr; /* header template point */ ib_msglen_t lso_hdr_sz; /* size of header */ ib_msglen_t lso_mss; /* segment payload size */ } ibt_wr_lso_t; Variant to the "wr" part of the send WR struct. The payload is segmented into lso_mss size pieces with the header generated from the template pointed to by lso_hdr (of size lso_hdr_size bytes). New ibt_status_t error: IBT_CHAN_WQE_SZ_INSUFF = 417, /* TI */ #define IBT_QP_WQE_SZ_INSUFF IBT_CHAN_WQE_SZ_INSUFF /* CI */ Indicates that the specified LSO header was too large. Man page changes: ibc_alloc_qp.9e, ibc_post_send.9e, ibt_alloc_ud_channel.9f, ibt_post_send.9f, ibt_hca_attr_t.9s, ibt_wc_t.9s, ibt_send_wr_t.9s, ibt_wr_lso_t.9s and ibt_wr_ud_t.9s F. CQ Moderation CQ Moderation is an IB version of interrupt moderation. Instead of having completion notification (i.e. an interrupt) occur when the first completion is added to a completion queue (CQ), it's now adjustable based on the number of completions or a timeout. New fields for HCA maximum CQ moderation values in ibt_hca_attr_t: uint_t hca_max_cq_mod_count; uint_t hca_max_cq_mod_usec; The maximum values for CQ moderation in terms completions (hca_max_cq_mod_count) or timeout in micro-seconds (hca_max_cq_mod_usec). Zero values for both indicate the HCA does not support this feature. Note, a value of 1 for count means the same thing as zero, but it's much easier to test for zero in code. New Modify CQ HCA operation: ibt_status_t ibt_modify_cq(ibt_cq_hdl_t cq, uint_t count, uint_t usec, uint_t reserved); /* TI */ ibt_status_t (*ibc_modify_cq)(ibt_hca_hdl_t hca, ibc_cq_hdl_t cq, uint_t count, uint_t usec, uint_t reserved); /* CI */ Set the CQ moderation parameters on a CQ. An unsupported value causes the IBT_INVALID_PARAM error. A value of zero for either count or usec disables that aspect of CQ moderation. Modified Query CQ HCA operation: ibt_status_t ibt_query_cq(ibt_cq_hdl_t ibt_cq, uint_t *entries, uint_t *count_p, uint_t *usec_p, uint_t *res_p); /* TI */ /* CI */ ibt_status_t (*ibc_query_cq)(ibc_hca_hdl_t hca, ibc_cq_hdl_t cq, uint_t *entries, uint_t *count_p, uint_t *usec_p, uint_t *res_p); The last three arguments of each are new and show the current settings on a CQ. A value of zero for either count or usec shows that aspect of CQ moderation is disabled. Man pages changed: ibci.9, ibti.9, ibc_modify_cq.9e, ibc_query_cq.9e. ibt_modify_cq.9f, ibt_query_cq.9f, ibc_operations_t.9s and ibt_hca_attr_t.9s G. IP Classification Flags If this HCA feature is supported, then when an IP packet is detected on the UD transport type (the type used by IPonIB), a number of flags are set in the work completion (WC) record to show what was found by the HCA hardware. New IP classification support flag in ibt_hca_attr_t: IBT_HCA2_IP_CLASS = 1 << 5 /* has IP classification flags */ New flags field in the ibt_wc_t: uint32_t wc_detail; /* UD: IPoIB flags */ New flag definitions for the wc_detail field: /* IPoIB flags for wc_detail field */ #define IBT_WC_DETAIL_ALL_FLAGS_MASK (0x0FC00000) #define IBT_WC_DETAIL_IPV4 (1 << 22) /* IPv4 header */ #define IBT_WC_DETAIL_IPV4_FRAG (1 << 23) /* IPv4 fragment */ #define IBT_WC_DETAIL_IPV6 (1 << 24) /* IPv6 header */ #define IBT_WC_DETAIL_IPV4_OPT (1 << 25) /* IPv4 option hdr */ #define IBT_WC_DETAIL_TCP (1 << 26) /* TCP header */ #define IBT_WC_DETAIL_UDP (1 << 27) /* UDP header */ Man page changes: ibt_hca_attr_t.9s, ibt_wc_t.9s H. Detailed WQE Sizes The reality of HCA design often means that giving a single number for SGL size per WR is not quite right. In some cases, like inline data, and using ibt_map_mem_iov (with Reserved L_Key), ULPs will want to know more detailed information about how much space there really is in the WQE. New flag for support of "detailed" WQE sizes added to ib_hca_flags_t: IBT_HCA_WQE_SIZE_INFO = 1 << 29 New fields giving exact sizes for certain transport/work-request combinations: /* Inline data sizes in bytes, valid only if inline data is supported */ uint_t hca_ud_send_inline_sz; /* UD Send */ uint_t hca_conn_send_inline_sz; /* RC Send */ uint_t hca_conn_rdmaw_inline_overhead; /* RDMA-W overhead */ /* SGL lengths */ uint_t hca_recv_sgl_sz; /* Receive */ uint_t hca_ud_send_sgl_sz; /* UD Send */ uint_t hca_conn_send_sgl_sz; /* RC Send */ uint_t hca_conn_rdma_sgl_overhead; /* RDMA-W/R overhead */ Man page changes: ibt_hca_attr_t.9s 4.3. Summary of changes by man page Man Page Disposition Reasons for change ----------------------------------------------------------- ibci.9 changed D, F ibti.9 changed D, F ibc_alloc_qp.9e changed D, E ibc_map_mem_iov.9e new D ibc_modify_cq.9e new F ibc_query_cq.9e changed F ibc_post_send.9e changed C, E ibt_alloc_rc_chan.9f changed D ibt_alloc_ud_chan.9f changed D, E ibt_map_mem_iov.9f new D ibt_modify_cq.9f new F ibt_query_cq.9f changed F ibt_post_send.9f changed C, E ibc_operations_t.9s changed F ibt_chan_sizes_t.9s changed C ibt_hca_attr_t.9s changed A, B, C, D, E, F, G, H ibt_send_wr_t.9s changed C, E ibt_wc_t.9s changed E, G ibt_wr_lso_t.9s new E ibt_wr_ud_t.9s changed E 4.4. References [1] InfiniBand Architecture Specification Volume 1, Release 1.2.1. InfiniBand Trade Association, 2007. http://www.infinibandta.org/members/spec/V1r1_2_1.Release_12062007.zip (requires IBTA member login) 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From Sebastien.Roy@sun.com Wed Dec 3 12:44:54 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mB3KisHf005268 for ; Wed, 3 Dec 2008 12:44:54 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mB3Kir9C022410 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 3 Dec 2008 12:44:54 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KBB00A11IYSC000@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Wed, 03 Dec 2008 12:44:52 -0800 (PST) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KBB003AVIYR1HB0@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Wed, 03 Dec 2008 12:44:52 -0800 (PST) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mB3Kip9x027143 for ; Wed, 03 Dec 2008 20:44:51 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KBB00901GIRJP00@mail-amer.sun.com> (original mail from Sebastien.Roy@Sun.COM) for PSARC-ext@Sun.Com (ORCPT PSARC-ext@Sun.Com); Wed, 03 Dec 2008 13:44:51 -0700 (MST) Received: from [129.148.174.103] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KBB0073WIYB6500@mail-amer.sun.com>; Wed, 03 Dec 2008 13:44:35 -0700 (MST) Date: Wed, 03 Dec 2008 15:44:29 -0500 From: Sebastien Roy Subject: Re: New HCA Capabilities [PSARC/2008/726 FastTrack timeout 12/02/2008] In-reply-to: <200811211724.mALHOj3v009454@sac.sfbay.sun.com> Sender: Sebastien.Roy@sun.com To: Ted Kim Cc: PSARC-ext@sun.com, Ted.Kim@sun.com, Bill.Taylor@sun.com Message-id: <1228337069.16428.131.camel@strat> Organization: Sun Microsystems MIME-version: 1.0 X-Mailer: Evolution 2.24.0 Content-type: text/plain Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200811211724.mALHOj3v009454@sac.sfbay.sun.com> Status: RO Content-Length: 406 I've reviewed this case and have no issues with the definition of the new interfaces. I only have two tangential questions: 1. What are the consumers of these new interfaces? 2. How is the LSO functionality defined here related to the LSO functionality provided by GLDv3? Specifically, could the ibd GLDv3 IPoIB driver provide LSO support to GLDv3 consumers by using this interface underneath? -Seb From Ted.Kim@sun.com Wed Dec 3 13:33:44 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mB3LXhWf006950 for ; Wed, 3 Dec 2008 13:33:43 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id mB3LXb9a008212 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Dec 2008 05:33:40 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KBB00K07L82OE00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Wed, 03 Dec 2008 14:33:38 -0700 (MST) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KBB0025OL817TE0@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.Com); Wed, 03 Dec 2008 14:33:38 -0700 (MST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mB3LXbDf029574 for ; Wed, 03 Dec 2008 13:33:37 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KBB00501L1KNA00@fe-sfbay-09.sun.com> (original mail from Ted.Kim@Sun.COM) for PSARC-ext@Sun.Com (ORCPT PSARC-ext@Sun.Com); Wed, 03 Dec 2008 13:33:37 -0800 (PST) Received: from [129.153.3.74] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KBB00LNOL7YLMG0@fe-sfbay-09.sun.com>; Wed, 03 Dec 2008 13:33:36 -0800 (PST) Date: Wed, 03 Dec 2008 13:32:24 -0800 From: "Ted H. Kim" Subject: Re: New HCA Capabilities [PSARC/2008/726 FastTrack timeout 12/02/2008] In-reply-to: <1228337069.16428.131.camel@strat> Sender: Ted.Kim@sun.com To: Sebastien Roy Cc: Ted Kim , psarc-ext@sun.com, Bill.Taylor@sun.com Message-id: <4936FAE8.1060005@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200811211724.mALHOj3v009454@sac.sfbay.sun.com> <1228337069.16428.131.camel@strat> User-Agent: Thunderbird 2.0.0.18 (X11/20081125) Status: RO Content-Length: 1666 Hi Sebastien, Sebastien Roy wrote: > I've reviewed this case and have no issues with the definition of the > new interfaces. I only have two tangential questions: > > 1. What are the consumers of these new interfaces? FW version: upcoming Open Fabrics User Verbs project SQD flag: ibmf (IB management msg framework) and possible future scenarios concerning proposed re-routing in the face of path failure Inline data: applies broadly to anything using short messages, IPonIB could use it, but I would also expect command/status (vs data bulk data transfer) messages for NFS-RDMA, SRP, iSER, RDS, SDP to be able to make use of it Reserved L_Key: another one which could be used broadly for medium data messages, IPonIB has a prototype using it already, and RDS is quite likely to use it; SDP, NFS-RDMA/SRP/iSER could use it too. LSO, CQ moderation, IP classification flags - IPonIB, and if we end up doing similar drivers for Eth-on-IB Detailed WQE sizes: basically needed for any Inline, Reserved L_Key users > 2. How is the LSO functionality defined here related to the LSO > functionality provided by GLDv3? Specifically, could the ibd GLDv3 > IPoIB driver provide LSO support to GLDv3 consumers by using this > interface underneath? The IPonIB team now has a prototype using IB LSO w/ GLDv3 LSO. Please tell me if you have any other follow-up or new comment to make about the case. Otherwise, I will close it up. -ted -- Ted H. Kim Sun Microsystems, Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX From Sebastien.Roy@sun.com Wed Dec 3 13:39:00 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mB3Ld0EK007147 for ; Wed, 3 Dec 2008 13:39:00 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mB3LcvHw046871 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Wed, 3 Dec 2008 14:38:59 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KBB00D01LGY9P00@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 03 Dec 2008 13:38:58 -0800 (PST) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KBB001DDLGX0RF0@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 03 Dec 2008 13:38:58 -0800 (PST) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mB3Lcvgv013531 for ; Wed, 03 Dec 2008 21:38:57 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KBB00K01LD5RI00@mail-amer.sun.com> (original mail from Sebastien.Roy@Sun.COM) for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 03 Dec 2008 14:38:34 -0700 (MST) Received: from [129.148.174.103] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KBB00927LFWE780@mail-amer.sun.com>; Wed, 03 Dec 2008 14:38:21 -0700 (MST) Date: Wed, 03 Dec 2008 16:38:15 -0500 From: Sebastien Roy Subject: Re: New HCA Capabilities [PSARC/2008/726 FastTrack timeout 12/02/2008] In-reply-to: <4936FAE8.1060005@sun.com> Sender: Sebastien.Roy@sun.com To: "Ted H. Kim" Cc: Ted Kim , psarc-ext@sun.com, Bill.Taylor@sun.com Message-id: <1228340295.16428.138.camel@strat> Organization: Sun Microsystems MIME-version: 1.0 X-Mailer: Evolution 2.24.0 Content-type: text/plain Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200811211724.mALHOj3v009454@sac.sfbay.sun.com> <1228337069.16428.131.camel@strat> <4936FAE8.1060005@sun.com> Status: RO Content-Length: 224 On Wed, 2008-12-03 at 13:32 -0800, Ted H. Kim wrote: > Please tell me if you have any other follow-up or new comment to make > about the case. Otherwise, I will close it up. No, I don't. Thanks for the information. -Seb From Ted.Kim@sun.com Wed Dec 3 13:43:58 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mB3Lhwd4007293 for ; Wed, 3 Dec 2008 13:43:58 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mB3Lhuam050357 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Wed, 3 Dec 2008 14:43:57 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KBB00D09LP8W200@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 03 Dec 2008 13:43:56 -0800 (PST) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KBB001XZLP80PE0@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 03 Dec 2008 13:43:56 -0800 (PST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mB3LhuBx016365 for ; Wed, 03 Dec 2008 13:43:56 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KBB00E01LAT8500@fe-sfbay-09.sun.com> (original mail from Ted.Kim@Sun.COM) for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 03 Dec 2008 13:43:56 -0800 (PST) Received: from [129.153.3.74] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KBB00BZZLP6RZ40@fe-sfbay-09.sun.com>; Wed, 03 Dec 2008 13:43:56 -0800 (PST) Date: Wed, 03 Dec 2008 13:42:44 -0800 From: "Ted H. Kim" Subject: Re: New HCA Capabilities [PSARC/2008/726 FastTrack timeout 12/02/2008] In-reply-to: <1228340295.16428.138.camel@strat> Sender: Ted.Kim@sun.com To: Sebastien Roy Cc: Ted Kim , psarc-ext@sun.com, Bill.Taylor@sun.com Message-id: <4936FD54.5000703@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200811211724.mALHOj3v009454@sac.sfbay.sun.com> <1228337069.16428.131.camel@strat> <4936FAE8.1060005@sun.com> <1228340295.16428.138.camel@strat> User-Agent: Thunderbird 2.0.0.18 (X11/20081125) Status: RO Content-Length: 787 ** Sebastian Roy, Thanks for your review. ** Everyone: As far as I know Sebastien was the only one at the PSARC meeting today committing to review things. Further, the case is already over the timeout. So at this point, I think there is nothing left to discuss. Therefore, I am going to mark the case as closed approved. Thanks, -ted Sebastien Roy wrote: > On Wed, 2008-12-03 at 13:32 -0800, Ted H. Kim wrote: >> Please tell me if you have any other follow-up or new comment to make >> about the case. Otherwise, I will close it up. > > No, I don't. Thanks for the information. > -Seb > > -- Ted H. Kim Sun Microsystems, Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX