From tedk@sac.sfbay.sun.com Wed Jun 30 12:38:32 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o5UJcWPA007980 for ; Wed, 30 Jun 2010 12:38:32 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o5UJcW1l022899; Wed, 30 Jun 2010 12:38:32 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L4U00407EK8KP00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 30 Jun 2010 12:38:32 -0700 (PDT) Received: from dm-sfbay-01.sfbay.sun.com ([129.145.155.118]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L4U008OYEK7Z1E0@nwk-avmta-1.sfbay.Sun.COM>; Wed, 30 Jun 2010 12:38:31 -0700 (PDT) Received: from sac.sfbay.sun.com (sac.SFBay.Sun.COM [129.146.226.132]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o5UJcVnY015561; Wed, 30 Jun 2010 12:38:31 -0700 (PDT) Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o5UJcUSB007975; Wed, 30 Jun 2010 12:38:30 -0700 (PDT) Received: (from tedk@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id o5UJcU8O007971; Wed, 30 Jun 2010 12:38:30 -0700 (PDT) Date: Wed, 30 Jun 2010 12:38:30 -0700 (PDT) From: Ted Kim Subject: Fibre Channel over IB Support [PSARC/2010/251 FastTrack timeout 07/08/2010] To: PSARC-ext@sun.com Message-id: <201006301938.o5UJcU8O007971@sac.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 11041 Template Version: @(#)sac_nextcase 1.70 03/30/10 SMI This information is Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. 1. Introduction 1.1. Project/Component Working Name: Fibre Channel over IB Support 1.2. Name of Document Author/Supplier: Author: Bill Taylor 1.3 Date of This Document: 30 June, 2010 4. Technical Description Fibre Channel over IB Support ----------------------------- 4.1 Background The Mellanox BridgeX product provides a gateway between InfiniBand (IB) and 10 Gb Ethernet and/or 2/4/8 Gb Fibre Channel (FC). The bridging protocol between IB and Fibre Channel is called "Fibre Channel over IB" (FCoIB). Mellanox has added IB host adapter extensions to the ConnectX IB adapter to support the BridgeX FCoIB protocol. This case introduces interfaces into our Solaris InfiniBand Transport Framework (IBTF, PSARC/2002/132 and follow-on cases) for the Mellanox host adapter extensions used to support FCoIB initiator. The expected consumer for FCoIB support is our "hermon" ConnectX IB driver (PSARC/2008/497) and the FCoIB initiator driver. 4.2 Proposal The proposal is to make additions to the IBTF Transport Interface (TI) for IB Upper Level Protocol (ULP) clients and the Channel Interface (CI) for HCA drivers. All interface changes and additions in the proposal have a micro/patch binding. Transport Interface (ON Consolidation Private) ibt_hca_flags2_t: add IBT_HCA2_FC flags ibt_hca_attr_t: add hca_rfci_max_log2_qp, hca_fexch_max_log2_qp, hca_fexch_log2_mem fields ibt_chan_alloc_flags_t: add IBT_ACHAN_USES_RFCI, IBT_ACHAN_USES_FCMD, IBT_ACHAN_USES_FEXCH flags ibt_fc_attr_t: new struct for FC QP attributes ibt_ud_chan_alloc_args_t: add ud_fc (ibt_fc_attr_t) field ibt_fexch_query_attr_t: new struct for FEXCH query attributes ibt_ud_chan_query_attr_t: add ud_fc (ibt_fc_attr_t), ud_query_fc (ibt_fexch_query_attr_t) fields ibt_cep_modify_flags_t: add IBT_CEP_SET_FEXCH_RANGE flag ibt_ud_chan_modify_attr_t: add uc_fc (ibt_fc_attr_t) field ibt_tran_srv_t: add IBT_RFCI_SRV, IBT_FCMD_SRV, IBT_FEXCH_SRV ibt_wrc_opcode_t: add IBT_WRC_INIT_SEND_FCMD ibt_wr_flags_t: add IBT_WR_SEND_FC_CRC flag ibt_wr_rfci_send_t: new struct for RFCI version of Send WR ibt_wr_init_send_t: new struct for FCMD WR to start I/O operation ibt_wr_fc_t: new union of possible FC WRs ibt_send_wr_t wr field: add fc (ibt_wr_fc_t) to union ibt_wc_flags_t: add IBT_WC_FEXCH_FMT, IBT_WC_DIF_ERROR flags ibt_wc_t wc_detail field: add IBT_WC_DETAIL_FEXCH_INIT_XFER, IBT_WC_FEXCH_LAST, IBT_WC_DETAIL_RFCI_CRC_OK, IBT_WC_DETAIL_FC_MATCH_MASK ibt_wc_t: add wc_fexch_seq_cnt, wc_fexch_tx_bytes, wc_rx_bytes_xfer, wc_fexch_seq_id fields ibt_async_code_t: add IBT_FEXCH_ERROR ibt_fc_syndrome_t: new enum for FEXCH async errors ibt_async_event_t: add ev_fc (ibt_fc_syndrome_t) Channel Interface (ON Consolidation Private) ibt_hca_flags2_t: add IBT_HCA2_FC flags ibt_hca_attr_t: add hca_rfci_max_log2_qp, hca_fexch_max_log2_qp, hca_fexch_log2_mem fields ibt_qp_alloc_flags_t: add IBT_QP_USES_RFCI, IBT_QP_USES_FCMD, IBT_QP_USES_FEXCH flags ibt_fc_attr_t: new struct for FC QP attributes ibt_qp_alloc_attr_t: add qp_fc (ibt_fc_attr_t) ibt_fexch_query_attr_t: new struct for FEXCH query attributes ibt_qp_query_attr_t: add qp_query_fexch (ibt_fexch_query_attr_t) ibt_cep_modify_flags_t: add IBT_CEP_SET_FEXCH_RANGE flag ibt_qp_ud_attr_t: add ud_fc (ibt_fc_attr_t) ibt_tran_srv_t: add IBT_RFCI_SRV, IBT_FCMD_SRV, IBT_FEXCH_SRV ibt_wrc_opcode_t: add IBT_WRC_INIT_SEND_FCMD ibt_wr_flags_t: add IBT_WR_SEND_FC_CRC ibt_wr_rfci_send_t: new struct for RFCI version of Send WR ibt_wr_init_send_t: new struct FCMD WR to start I/O operation ibt_wr_fc_t: new union of possible FC WRs ibt_send_wr_t wr field: add fc (ibt_wr_fc_t) to union ibt_wc_flags_t: add IBT_WC_FEXCH_FMT, IBT_WC_DIF_ERROR flags ibt_wc_t wc_detail field: add IBT_WC_DETAIL_FEXCH_INIT_XFER, IBT_WC_FEXCH_LAST, IBT_WC_DETAIL_RFCI_CRC_OK, IBT_WC_DETAIL_FC_MATCH_MASK ibt_wc_t: add wc_fexch_seq_cnt, wc_fexch_tx_bytes, wc_rx_bytes_xfer, wc_fexch_seq_id fields ibt_async_code_t: add IBT_FEXCH_ERROR ibt_fc_syndrome_t: new enum for FEXCH async errors ibc_async_event_t: add ev_fc (ibt_fc_syndrome_t) The changes in this case are part of IBTF ABI version 4. Copies of all new and modified man pages are in the materials directory (see section 4.3 below). A. Detailed Description of FCoIB Support While the FCoIB support could also support targets, the extensions in this proposal are only sufficient to support initiators. Mellanox support for FCoIB involves three variants of the IB Unreliable Datagram (UD) transport service type. The first is a Raw Fibre Channel Interface (RFCI) Queue Pair (QP), which is used to establish the association between initiator and targets. When it comes to I/O operations, typically, these QPs are used to send confirmation (FCP_CONF) messages back to the target. In IB terms, these QPs are like regular UD QPs, except that they have an option for FC CRC support and End of Frame (EOF) values. The second special QP type is the Fibre Channel Command (FCMD) QP. This QP is used to initiate I/O operations through the associated FEXCH QPs (see below) and perform memory registration through Work Requests (WRs). Posting a special type of WR (Init & Send) causes a data transfer operation to be assigned to an FEXCH QP and also causes a command (FCP_CMND) message to be sent to the target. Registration is accomplished on FCMDs by using the Registration WR previously defined in PSARC/2009/060 only for Reliable Connected QPs. There is no receive half to FCMD QPs. The third special QP type is the Fibre Channel Exchange (FEXCH) QP. As mentioned before, FCMD WRs assign data transfer operations to associated FEXCH QPs. Allocation of FEXCH QPs is done via an ibt_alloc_ud_channel_range() call to assign a group/range of QPs all at once to an FCMD. The send half of these FEXCH QPs are not directly used. The receive half of these QPs are used to signal the completion of I/O operations. The work completions generated from the receive queue have additional fields specific to Fibre Channel operations. These FEXCH QPs also have associated memory regions (MRs) which define the source/sink for a data transfer. Query QP operations show which MRs are assigned to any given FEXCH QP. The MRs are registered by using Registration WRs posted on the associated FCMD QP. So pulling it all together, the typical I/O operation involves: (1) registering memory belonging to an FEXCH QP through a Registration WR on an FCMD QP, (2) initiating a data transfer with an Init & Send WR on the FCMD, (3) waiting for the FEXCH completion and (4) a possible confirmation message sent via an associated RFCI QP. Add capability flag to ibt_hca_flags2_t to indicate support: IBT_HCA2_FC = 1 << 12 /* FCoIB support */ Add other FC related HCA attributes to ibt_hca_attr_t: uint8_t hca_rfci_max_log2_qp; /* log2 num RFCI QPs */ uint8_t hca_fexch_max_log2_qp; /* log2 num FEXCH QPs */ uint8_t hca_fexch_log2_mem; /* log2 size of FEXCH MRs (in bytes) */ Add flags and attributes (ibt_fc_attr_t) to for allocate QP ops: ibt_chan_alloc_flags_t: add IBT_ACHAN_USES_RFCI, /* TI */ IBT_ACHAN_USES_FCMD, IBT_ACHAN_USES_FEXCH flags add ibt_fc_attr_t struct to ibt_ud_chan_alloc_args_t ibt_qp_alloc_flags_t: add IBT_QP_USES_RFCI, /* CI */ IBT_QP_USES_FCMD, IBT_QP_USES_FEXCH flags add ibt_fc_attr_t to ibt_qp_alloc_attr_t Add attributes (ibt_fc_attr_t & ibt_fexch_query_attr_t) for query QP ops: add ibt_fc_attr_t & ibt_fexch_query_attr_t to /* TI */ ibt_ud_chan_query_attr_t add ibt_fexch_query_attr_t to ibt_qp_query_attr_t /* CI */ add ibt_fc_attr_t to ibt_qp_ud_attr_t Add flags and attributes (ibt_fc_attr_t) to modify QP ops: add IBT_CEP_SET_FEXCH_RANGE flag to ibt_cep_modify_flags_t add ibt_fc_attr_t to ibt_ud_chan_modify_attr_t /* TI */ add ibt_fc_attr_t to ibt_qp_ud_attr_t /* CI */ Add the new work requests structs: define new transport service (QP) types for ibt_tran_srv_t: IBT_RFCI_SRV, IBT_FCMD_SRV, IBT_FEXCH_SRV for RFCI QPs, define a variant of UD Send (ibt_wr_rfci_send_t), this operation can also use a new flag (IBT_WR_SEND_FC_CRC in ibt_wr_flags_t) for FCMD QPs, define a new Init & Send WR to assign initiate I/O operations on FEXCH QPs (ibt_wr_init_send_t) and a new opcode (IBT_WRC_INIT_SEND_FCMD in ibt_wrc_opcode_t) define union of possible FC WRs (ibt_wr_fc_t) & add it to ibt_send_wr_t Add new work completion fields: add IBT_WC_DIF_ERROR flag to show Data Integrity Field errors add IBT_WC_DETAIL_RFCI_CRC_OK flag for RFCI Send completions add IBT_WC_FEXCH_FMT flag to show when using the special FEXCH format; when set, the following are defined: add wc_fexch_seq_cnt (sequence count), wc_fexch_tx_bytes (transmit bytes), wc_rx_bytes_xfer (receive byes), wc_fexch_seq_id (sequence id) fields add IBT_WC_DETAIL_FEXCH_INIT_XFER (initiative transfer) and IBT_WC_FEXCH_LAST (last sequence) flags Add a new type of async error for FEXCH syndromes: add IBT_FEXCH_ERROR to ibt_async_code_t add ibt_fc_syndrome_t field to ibt_async_event_t 4.3 Man page summary Man Page Disposition (sorted by section) ------------------------------------------- ibc_alloc_qp.9e changed ibc_alloc_qp_range.9e changed ibt_async_handler_t.9e changed ibc_async_handler.9f changed ibt_alloc_ud_channel.9f changed ibt_alloc_ud_channel_range.9f changed ibt_modify_ud_channel.9f changed ibt_query_ud_channel.9f changed ibc_qp_info_t.9s changed ibc_qp_query_attr_9.9s changed ibt_fc_attr_t.9s new ibt_fexch_query_attr_t.9s new ibt_hca_attr_t.9s changed ibt_send_wr_t.9s changed ibt_wc_t.9s changed ibt_wr_fc_t.9s new ibt_wr_rc_t.9s changed ibt_wr_reg_pmr_t.9s new Note: Since FCMD QPs can also do "Registration Work Request" operations, we split the description of that WR out of ibt_wr_rc_t.9s and put it into ibt_wr_reg_pmr_t.9s (where it can be referenced by both ibt_wr_rc_t.9s and ibt_wr_fc_t.9s). Aside from reorganizing where the information is, there is no change to the description of Registration Work Request. 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From garrett@damore.org Wed Jun 30 13:27:12 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o5UKRCsi009906 for ; Wed, 30 Jun 2010 13:27:12 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o5UKR8vj042600 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 30 Jun 2010 14:27:12 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L4U00905GTCF800@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 30 Jun 2010 13:27:12 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L4U00JWEGTBLNE0@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 30 Jun 2010 13:27:11 -0700 (PDT) Received: from relay15i.sun.com (ip125.net129179-4.block1.us.syntegra.com [129.179.4.125]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o5UK4ocm029273 for ; Wed, 30 Jun 2010 20:27:10 +0000 (GMT) Received: from mmp11es.mmp.us.syntegra.com ([160.41.208.11] [160.41.208.11]) by relay15i.sun.com with ESMTP id BT-MMP-163663 for PSARC-ext@sun.com; Wed, 30 Jun 2010 20:27:10 +0000 (Z) Received: from relay13i.sun.com (relay13i.sun.com [129.179.4.123]) by mmp11es.mmp.us.syntegra.com with ESMTP id BT-MMP-731228 for PSARC-ext@sun.com; Wed, 30 Jun 2010 20:27:10 +0000 (Z) Received: from oproxy2-pub.bluehost.com ([67.222.39.60] [67.222.39.60]) by relay1i.sun.com id BT-MMP-43911625 for PSARC-ext@sun.com; Wed, 30 Jun 2010 20:27:09 +0000 (Z) Received: (qmail 16225 invoked by uid 0); Wed, 30 Jun 2010 20:27:08 +0000 Received: from unknown (HELO box374.bluehost.com) (69.89.31.174) by oproxy2.bluehost.com with SMTP; Wed, 30 Jun 2010 20:27:08 +0000 Received: from cpe-76-93-15-33.socal.res.rr.com ([76.93.15.33] helo=[192.168.251.110]) by box374.bluehost.com with esmtpsa (SSLv3:AES256-SHA:256) (Exim 4.69) (envelope-from ) id 1OU3sC-0002Uc-CH; Wed, 30 Jun 2010 14:27:08 -0600 Date: Wed, 30 Jun 2010 13:29:36 -0700 From: "Garrett D'Amore" Subject: Re: Fibre Channel over IB Support [PSARC/2010/251 FastTrack timeout 07/08/2010] In-reply-to: <201006301938.o5UJcU8O007971@sac.sfbay.sun.com> To: Ted Kim Cc: PSARC-ext@sun.com Message-id: <1277929776.5596.4654.camel@velocity> MIME-version: 1.0 X-Mailer: Evolution 2.28.3 Content-type: text/plain; charset=UTF-8 Content-transfer-encoding: 7BIT DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=damore.org; h=Received:Subject:From:To:Cc:In-Reply-To:References:Content-Type:Date:Message-ID:Mime-Version:X-Mailer:Content-Transfer-Encoding:X-Identified-User; b=x63grbM8byF+RVin4qehOXJTAMSl5tTEjxG1Jtx44CxYm51nf1eGhf8EHn34nz2U1LT4nR4cLbRohZko907dzGU4axrRIxfbYR4hvTfIysCcjoiH+/KYKH1QIOPZrOuF; X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Identified-User: {2225:box374.bluehost.com:damoreor:damore.org} {sentby:smtp auth 76.93.15.33 authed with garrett+damore.org} X-Antispam: No, score=-0.7/5.0, scanned in 0.332sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <201006301938.o5UJcU8O007971@sac.sfbay.sun.com> Status: RO Content-Length: 12431 So, this FCoIB driver, will it be a separate driver, that lives above IBTF? Can it in theory support any IB HCA, or is hermon the only way to provide this functionality (is it custom firmware on the HCA?) Also, I'd like to understand the interaction between all of the drivers in the FCoE stack... where will the FC initiator sit? Above IBTF? Above hermon? Somewhere else? What parts of the existing FC code (SCSA or otherwise) will it use? A block diagram would be helpful here. Will storage be bootable over this? What about WWNs, multipathing, etc? I'm assuming that the the other side is basic FC, but will it be possible for folks to use the MPXIO framework with this? - Garrett On Wed, 2010-06-30 at 12:38 -0700, Ted Kim wrote: > Template Version: @(#)sac_nextcase 1.70 03/30/10 SMI > This information is Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. > 1. Introduction > 1.1. Project/Component Working Name: > Fibre Channel over IB Support > 1.2. Name of Document Author/Supplier: > Author: Bill Taylor > 1.3 Date of This Document: > 30 June, 2010 > 4. Technical Description > > Fibre Channel over IB Support > ----------------------------- > > > 4.1 Background > > The Mellanox BridgeX product provides a gateway between InfiniBand > (IB) and 10 Gb Ethernet and/or 2/4/8 Gb Fibre Channel (FC). The > bridging protocol between IB and Fibre Channel is called "Fibre > Channel over IB" (FCoIB). Mellanox has added IB host adapter > extensions to the ConnectX IB adapter to support the BridgeX FCoIB > protocol. This case introduces interfaces into our Solaris InfiniBand > Transport Framework (IBTF, PSARC/2002/132 and follow-on cases) for the > Mellanox host adapter extensions used to support FCoIB initiator. > > The expected consumer for FCoIB support is our "hermon" ConnectX IB > driver (PSARC/2008/497) and the FCoIB initiator driver. > > > 4.2 Proposal > > The proposal is to make additions to the IBTF Transport Interface (TI) > for IB Upper Level Protocol (ULP) clients and the Channel Interface > (CI) for HCA drivers. > > All interface changes and additions in the proposal have a micro/patch > binding. > > > Transport Interface (ON Consolidation Private) > > ibt_hca_flags2_t: add IBT_HCA2_FC flags > ibt_hca_attr_t: add hca_rfci_max_log2_qp, hca_fexch_max_log2_qp, > hca_fexch_log2_mem fields > > ibt_chan_alloc_flags_t: add IBT_ACHAN_USES_RFCI, IBT_ACHAN_USES_FCMD, > IBT_ACHAN_USES_FEXCH flags > ibt_fc_attr_t: new struct for FC QP attributes > ibt_ud_chan_alloc_args_t: add ud_fc (ibt_fc_attr_t) field > ibt_fexch_query_attr_t: new struct for FEXCH query attributes > ibt_ud_chan_query_attr_t: add ud_fc (ibt_fc_attr_t), ud_query_fc > (ibt_fexch_query_attr_t) fields > ibt_cep_modify_flags_t: add IBT_CEP_SET_FEXCH_RANGE flag > ibt_ud_chan_modify_attr_t: add uc_fc (ibt_fc_attr_t) field > > ibt_tran_srv_t: add IBT_RFCI_SRV, IBT_FCMD_SRV, IBT_FEXCH_SRV > ibt_wrc_opcode_t: add IBT_WRC_INIT_SEND_FCMD > ibt_wr_flags_t: add IBT_WR_SEND_FC_CRC flag > ibt_wr_rfci_send_t: new struct for RFCI version of Send WR > ibt_wr_init_send_t: new struct for FCMD WR to start I/O operation > ibt_wr_fc_t: new union of possible FC WRs > ibt_send_wr_t wr field: add fc (ibt_wr_fc_t) to union > > ibt_wc_flags_t: add IBT_WC_FEXCH_FMT, IBT_WC_DIF_ERROR flags > ibt_wc_t wc_detail field: add IBT_WC_DETAIL_FEXCH_INIT_XFER, > IBT_WC_FEXCH_LAST, IBT_WC_DETAIL_RFCI_CRC_OK, > IBT_WC_DETAIL_FC_MATCH_MASK > ibt_wc_t: add wc_fexch_seq_cnt, wc_fexch_tx_bytes, wc_rx_bytes_xfer, > wc_fexch_seq_id fields > > ibt_async_code_t: add IBT_FEXCH_ERROR > ibt_fc_syndrome_t: new enum for FEXCH async errors > ibt_async_event_t: add ev_fc (ibt_fc_syndrome_t) > > > Channel Interface (ON Consolidation Private) > > ibt_hca_flags2_t: add IBT_HCA2_FC flags > ibt_hca_attr_t: add hca_rfci_max_log2_qp, hca_fexch_max_log2_qp, > hca_fexch_log2_mem fields > > ibt_qp_alloc_flags_t: add IBT_QP_USES_RFCI, IBT_QP_USES_FCMD, > IBT_QP_USES_FEXCH flags > ibt_fc_attr_t: new struct for FC QP attributes > ibt_qp_alloc_attr_t: add qp_fc (ibt_fc_attr_t) > ibt_fexch_query_attr_t: new struct for FEXCH query attributes > ibt_qp_query_attr_t: add qp_query_fexch (ibt_fexch_query_attr_t) > ibt_cep_modify_flags_t: add IBT_CEP_SET_FEXCH_RANGE flag > ibt_qp_ud_attr_t: add ud_fc (ibt_fc_attr_t) > > ibt_tran_srv_t: add IBT_RFCI_SRV, IBT_FCMD_SRV, IBT_FEXCH_SRV > ibt_wrc_opcode_t: add IBT_WRC_INIT_SEND_FCMD > ibt_wr_flags_t: add IBT_WR_SEND_FC_CRC > ibt_wr_rfci_send_t: new struct for RFCI version of Send WR > ibt_wr_init_send_t: new struct FCMD WR to start I/O operation > ibt_wr_fc_t: new union of possible FC WRs > ibt_send_wr_t wr field: add fc (ibt_wr_fc_t) to union > > ibt_wc_flags_t: add IBT_WC_FEXCH_FMT, IBT_WC_DIF_ERROR flags > ibt_wc_t wc_detail field: add IBT_WC_DETAIL_FEXCH_INIT_XFER, > IBT_WC_FEXCH_LAST, IBT_WC_DETAIL_RFCI_CRC_OK, > IBT_WC_DETAIL_FC_MATCH_MASK > ibt_wc_t: add wc_fexch_seq_cnt, wc_fexch_tx_bytes, wc_rx_bytes_xfer, > wc_fexch_seq_id fields > > ibt_async_code_t: add IBT_FEXCH_ERROR > ibt_fc_syndrome_t: new enum for FEXCH async errors > ibc_async_event_t: add ev_fc (ibt_fc_syndrome_t) > > > The changes in this case are part of IBTF ABI version 4. > > Copies of all new and modified man pages are in the materials > directory (see section 4.3 below). > > > A. Detailed Description of FCoIB Support > > While the FCoIB support could also support targets, the extensions in > this proposal are only sufficient to support initiators. Mellanox > support for FCoIB involves three variants of the IB Unreliable > Datagram (UD) transport service type. > > The first is a Raw Fibre Channel Interface (RFCI) Queue Pair (QP), > which is used to establish the association between initiator and > targets. When it comes to I/O operations, typically, these QPs are > used to send confirmation (FCP_CONF) messages back to the target. In > IB terms, these QPs are like regular UD QPs, except that they have an > option for FC CRC support and End of Frame (EOF) values. > > The second special QP type is the Fibre Channel Command (FCMD) > QP. This QP is used to initiate I/O operations through the associated > FEXCH QPs (see below) and perform memory registration through Work > Requests (WRs). Posting a special type of WR (Init & Send) causes a > data transfer operation to be assigned to an FEXCH QP and also causes > a command (FCP_CMND) message to be sent to the target. Registration is > accomplished on FCMDs by using the Registration WR previously defined > in PSARC/2009/060 only for Reliable Connected QPs. There is no receive > half to FCMD QPs. > > The third special QP type is the Fibre Channel Exchange (FEXCH) QP. As > mentioned before, FCMD WRs assign data transfer operations to > associated FEXCH QPs. Allocation of FEXCH QPs is done via an > ibt_alloc_ud_channel_range() call to assign a group/range of QPs all > at once to an FCMD. The send half of these FEXCH QPs are not directly > used. The receive half of these QPs are used to signal the completion > of I/O operations. The work completions generated from the receive > queue have additional fields specific to Fibre Channel > operations. These FEXCH QPs also have associated memory regions (MRs) > which define the source/sink for a data transfer. Query QP operations > show which MRs are assigned to any given FEXCH QP. The MRs are > registered by using Registration WRs posted on the associated FCMD QP. > > So pulling it all together, the typical I/O operation involves: (1) > registering memory belonging to an FEXCH QP through a Registration WR > on an FCMD QP, (2) initiating a data transfer with an Init & Send WR > on the FCMD, (3) waiting for the FEXCH completion and (4) a possible > confirmation message sent via an associated RFCI QP. > > Add capability flag to ibt_hca_flags2_t to indicate support: > IBT_HCA2_FC = 1 << 12 /* FCoIB support */ > > Add other FC related HCA attributes to ibt_hca_attr_t: > uint8_t hca_rfci_max_log2_qp; /* log2 num RFCI QPs */ > uint8_t hca_fexch_max_log2_qp; /* log2 num FEXCH QPs */ > uint8_t hca_fexch_log2_mem; /* log2 size of FEXCH MRs (in bytes) */ > > Add flags and attributes (ibt_fc_attr_t) to for allocate QP ops: > ibt_chan_alloc_flags_t: add IBT_ACHAN_USES_RFCI, /* TI */ > IBT_ACHAN_USES_FCMD, IBT_ACHAN_USES_FEXCH flags > add ibt_fc_attr_t struct to ibt_ud_chan_alloc_args_t > > ibt_qp_alloc_flags_t: add IBT_QP_USES_RFCI, /* CI */ > IBT_QP_USES_FCMD, IBT_QP_USES_FEXCH flags > add ibt_fc_attr_t to ibt_qp_alloc_attr_t > > Add attributes (ibt_fc_attr_t & ibt_fexch_query_attr_t) for query QP ops: > add ibt_fc_attr_t & ibt_fexch_query_attr_t to /* TI */ > ibt_ud_chan_query_attr_t > > add ibt_fexch_query_attr_t to ibt_qp_query_attr_t /* CI */ > add ibt_fc_attr_t to ibt_qp_ud_attr_t > > Add flags and attributes (ibt_fc_attr_t) to modify QP ops: > add IBT_CEP_SET_FEXCH_RANGE flag to ibt_cep_modify_flags_t > > add ibt_fc_attr_t to ibt_ud_chan_modify_attr_t /* TI */ > add ibt_fc_attr_t to ibt_qp_ud_attr_t /* CI */ > > Add the new work requests structs: > define new transport service (QP) types for ibt_tran_srv_t: > IBT_RFCI_SRV, IBT_FCMD_SRV, IBT_FEXCH_SRV > > for RFCI QPs, define a variant of UD Send (ibt_wr_rfci_send_t), this > operation can also use a new flag (IBT_WR_SEND_FC_CRC in > ibt_wr_flags_t) > > for FCMD QPs, define a new Init & Send WR to assign initiate I/O > operations on FEXCH QPs (ibt_wr_init_send_t) and a new opcode > (IBT_WRC_INIT_SEND_FCMD in ibt_wrc_opcode_t) > > define union of possible FC WRs (ibt_wr_fc_t) & add it to ibt_send_wr_t > > Add new work completion fields: > add IBT_WC_DIF_ERROR flag to show Data Integrity Field errors > > add IBT_WC_DETAIL_RFCI_CRC_OK flag for RFCI Send completions > > add IBT_WC_FEXCH_FMT flag to show when using the special FEXCH > format; when set, the following are defined: > > add wc_fexch_seq_cnt (sequence count), wc_fexch_tx_bytes (transmit > bytes), wc_rx_bytes_xfer (receive byes), wc_fexch_seq_id > (sequence id) fields > add IBT_WC_DETAIL_FEXCH_INIT_XFER (initiative transfer) and > IBT_WC_FEXCH_LAST (last sequence) flags > > Add a new type of async error for FEXCH syndromes: > add IBT_FEXCH_ERROR to ibt_async_code_t > add ibt_fc_syndrome_t field to ibt_async_event_t > > > 4.3 Man page summary > > Man Page Disposition > (sorted by section) > ------------------------------------------- > ibc_alloc_qp.9e changed > ibc_alloc_qp_range.9e changed > ibt_async_handler_t.9e changed > > ibc_async_handler.9f changed > ibt_alloc_ud_channel.9f changed > ibt_alloc_ud_channel_range.9f changed > ibt_modify_ud_channel.9f changed > ibt_query_ud_channel.9f changed > > ibc_qp_info_t.9s changed > ibc_qp_query_attr_9.9s changed > ibt_fc_attr_t.9s new > ibt_fexch_query_attr_t.9s new > ibt_hca_attr_t.9s changed > ibt_send_wr_t.9s changed > ibt_wc_t.9s changed > ibt_wr_fc_t.9s new > ibt_wr_rc_t.9s changed > ibt_wr_reg_pmr_t.9s new > > Note: Since FCMD QPs can also do "Registration Work Request" > operations, we split the description of that WR out of ibt_wr_rc_t.9s > and put it into ibt_wr_reg_pmr_t.9s (where it can be referenced by > both ibt_wr_rc_t.9s and ibt_wr_fc_t.9s). Aside from reorganizing where > the information is, there is no change to the description of > Registration Work Request. > > > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > ON > 6.5. ARC review type: FastTrack > 6.6. ARC Exposure: open > > _______________________________________________ > opensolaris-arc mailing list > opensolaris-arc@opensolaris.org From ted.h.kim@oracle.com Wed Jun 30 13:49:56 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o5UKnt6M010085 for ; Wed, 30 Jun 2010 13:49:55 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o5UKntlZ008050 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 30 Jun 2010 13:49:55 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L4U00A09HV7O700@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 30 Jun 2010 14:49:55 -0600 (MDT) Received: from sca-ea-mail-4.sun.com ([192.18.43.22]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L4U000SFHV63250@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 30 Jun 2010 14:49:55 -0600 (MDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by sca-ea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o5UKnsKO015245; Wed, 30 Jun 2010 20:49:54 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5U2GdWp005953; Wed, 30 Jun 2010 20:49:52 +0000 (GMT) Received: from abhmt013.oracle.com by acsmt353.oracle.com with ESMTP id 386787751277930981; Wed, 30 Jun 2010 13:49:41 -0700 Received: from [129.153.3.74] (/129.153.3.74) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 30 Jun 2010 13:49:41 -0700 Date: Wed, 30 Jun 2010 13:48:10 -0700 From: "Ted H. Kim" Subject: Re: Fibre Channel over IB Support [PSARC/2010/251 FastTrack timeout 07/08/2010] In-reply-to: <1277929776.5596.4654.camel@velocity> To: "Garrett D'Amore" Cc: Ted Kim , PSARC-ext@sun.com Message-id: <4C2BAD8A.50905@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=UTF-8; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090209.4C2BADF0.0158:SCFMA4539814,ss=1,fgs=0 References: <201006301938.o5UJcU8O007971@sac.sfbay.sun.com> <1277929776.5596.4654.camel@velocity> User-Agent: Thunderbird 2.0.0.24 (X11/20100302) Status: RO Content-Length: 1436 Garrett D'Amore wrote: > So, this FCoIB driver, will it be a separate driver, that lives above > IBTF? yes, there will be a separate FCoIB ULP driver client over IBTF. What's described here is the hermon support and IBTF interface for it. > Can it in theory support any IB HCA, or is hermon the only way to > provide this functionality (is it custom firmware on the HCA?) The support described here is pretty much a Mellanox ConnectX/BridgeX thing. I am not aware of any other vendor or HCA that does this particular style. > Also, I'd like to understand the interaction between all of the drivers > in the FCoE stack... where will the FC initiator sit? Above IBTF? > Above hermon? Somewhere else? What parts of the existing FC code (SCSA > or otherwise) will it use? > > A block diagram would be helpful here. > > Will storage be bootable over this? > > What about WWNs, multipathing, etc? I'm assuming that the the other > side is basic FC, but will it be possible for folks to use the MPXIO > framework with this? There will be a separate FCoIB ULP driver over IBTF. I think in theory everything you mention is possible. But for what will actually be implemented, I think, is for the FCoIB ULP driver case to detail. -ted -- Ted H. Kim, PhD Sun/Oracle America Inc. ted.h.kim@oracle.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1115 FAX From garrett@damore.org Wed Jun 30 13:53:39 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o5UKrdk0010184 for ; Wed, 30 Jun 2010 13:53:39 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o5UKrZri009525 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 30 Jun 2010 13:53:38 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L4U00B0BI1E1500@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 30 Jun 2010 14:53:38 -0600 (MDT) Received: from sca-ea-mail-3.sun.com ([192.18.43.21]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L4U0005HI1D3350@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 30 Jun 2010 14:53:37 -0600 (MDT) Received: from relay41i.sun.com ([192.5.209.70]) by sca-ea-mail-3.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o5UKowNB015291 for ; Wed, 30 Jun 2010 20:53:36 +0000 (GMT) Received: from mms49es.mms.us.syntegra.com ([160.41.221.232] [160.41.221.232]) by relay41i.sun.com with ESMTP id BT-MMP-132568 for PSARC-ext@Sun.COM; Wed, 30 Jun 2010 20:53:36 +0000 (Z) Received: from relay43i.sun.com (relay43i.sun.com [192.5.209.74]) by mms49es.mms.us.syntegra.com with ESMTP id BT-MMP-543282 for PSARC-ext@Sun.COM; Wed, 30 Jun 2010 20:53:36 +0000 (Z) Received: from oproxy3-pub.bluehost.com ([69.89.21.8] [69.89.21.8]) by relay4i.sun.com id BT-MMP-800514 for PSARC-ext@Sun.COM; Wed, 30 Jun 2010 20:53:36 +0000 (Z) Received: (qmail 29110 invoked by uid 0); Wed, 30 Jun 2010 20:53:35 +0000 Received: from unknown (HELO box374.bluehost.com) (69.89.31.174) by oproxy3.bluehost.com with SMTP; Wed, 30 Jun 2010 20:53:35 +0000 Received: from cpe-76-93-15-33.socal.res.rr.com ([76.93.15.33] helo=[192.168.251.110]) by box374.bluehost.com with esmtpsa (SSLv3:AES256-SHA:256) (Exim 4.69) (envelope-from ) id 1OU4Hn-0008Iy-EO; Wed, 30 Jun 2010 14:53:35 -0600 Date: Wed, 30 Jun 2010 13:56:06 -0700 From: "Garrett D'Amore" Subject: Re: Fibre Channel over IB Support [PSARC/2010/251 FastTrack timeout 07/08/2010] In-reply-to: <4C2BAD8A.50905@oracle.com> To: "Ted H. Kim" Cc: Ted Kim , PSARC-ext@sun.com Message-id: <1277931366.5596.4677.camel@velocity> MIME-version: 1.0 X-Mailer: Evolution 2.28.3 Content-type: text/plain; charset=UTF-8 Content-transfer-encoding: 7BIT DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=damore.org; h=Received:Subject:From:To:Cc:In-Reply-To:References:Content-Type:Date:Message-ID:Mime-Version:X-Mailer:Content-Transfer-Encoding:X-Identified-User; b=xrYWnxN1Jx9k6eS70GWA1zOuhDziinXUjEt+CL8aYunoNa9a4mNVBDVTuva0iDxjRFsQkkXVSR9J/YTcogh8phZXVI5b6i/Z+5x/qzOy1oamVSdCh65TK3dmvcm3Cy6O; X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Identified-User: {2225:box374.bluehost.com:damoreor:damore.org} {sentby:smtp auth 76.93.15.33 authed with garrett+damore.org} X-Antispam: No, score=-0.2/5.0, scanned in 0.269sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <201006301938.o5UJcU8O007971@sac.sfbay.sun.com> <1277929776.5596.4654.camel@velocity> <4C2BAD8A.50905@oracle.com> Status: RO Content-Length: 1695 So this case is just plumbing details? I'm a bit concerned that the case is incomplete without the FCoIB ULP driver as part of it. Specifying an API without a consumer makes this feel like an incomplete spec/case, at least to me. Is there some reason to run this case now instead of together with the ULP driver? - Garrett On Wed, 2010-06-30 at 13:48 -0700, Ted H. Kim wrote: > Garrett D'Amore wrote: > > So, this FCoIB driver, will it be a separate driver, that lives above > > IBTF? > > yes, there will be a separate FCoIB ULP driver client over IBTF. > What's described here is the hermon support and > IBTF interface for it. > > > Can it in theory support any IB HCA, or is hermon the only way to > > provide this functionality (is it custom firmware on the HCA?) > > The support described here is pretty much a Mellanox ConnectX/BridgeX > thing. I am not aware of any other vendor or HCA that does > this particular style. > > > Also, I'd like to understand the interaction between all of the drivers > > in the FCoE stack... where will the FC initiator sit? Above IBTF? > > Above hermon? Somewhere else? What parts of the existing FC code (SCSA > > or otherwise) will it use? > > > > A block diagram would be helpful here. > > > > Will storage be bootable over this? > > > > What about WWNs, multipathing, etc? I'm assuming that the the other > > side is basic FC, but will it be possible for folks to use the MPXIO > > framework with this? > > There will be a separate FCoIB ULP driver over IBTF. > I think in theory everything you mention is possible. > But for what will actually be implemented, I think, > is for the FCoIB ULP driver case to detail. > > -ted > From ted.h.kim@oracle.com Wed Jun 30 14:08:38 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o5UL8chV011631 for ; Wed, 30 Jun 2010 14:08:38 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o5UL8bn3015466 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 30 Jun 2010 14:08:38 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L4U00C11IQEBT00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 30 Jun 2010 15:08:38 -0600 (MDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L4U000ZTIQD3750@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 30 Jun 2010 15:08:37 -0600 (MDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o5UL8bIS008910; Wed, 30 Jun 2010 21:08:37 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5UAeuLu014174; Wed, 30 Jun 2010 21:08:33 +0000 (GMT) Received: from abhmt020.oracle.com by acsmt353.oracle.com with ESMTP id 386827711277932017; Wed, 30 Jun 2010 14:06:57 -0700 Received: from [129.153.3.74] (/129.153.3.74) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 30 Jun 2010 14:06:55 -0700 Date: Wed, 30 Jun 2010 14:05:28 -0700 From: "Ted H. Kim" Subject: Re: Fibre Channel over IB Support [PSARC/2010/251 FastTrack timeout 07/08/2010] In-reply-to: <1277931366.5596.4677.camel@velocity> To: "Garrett D'Amore" Cc: Ted Kim , PSARC-ext@sun.com Message-id: <4C2BB198.7000901@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=UTF-8; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090205.4C2BB252.00CD:SCFMA4539814,ss=1,fgs=0 References: <201006301938.o5UJcU8O007971@sac.sfbay.sun.com> <1277929776.5596.4654.camel@velocity> <4C2BAD8A.50905@oracle.com> <1277931366.5596.4677.camel@velocity> User-Agent: Thunderbird 2.0.0.24 (X11/20100302) Status: RO Content-Length: 819 Garrett D'Amore wrote: > So this case is just plumbing details? If that is what you want to call IBTF and hermon. > I'm a bit concerned that the case is incomplete without the FCoIB ULP > driver as part of it. Specifying an API without a consumer makes this > feel like an incomplete spec/case, at least to me. > > Is there some reason to run this case now instead of together with the > ULP driver? I think the ULP driver owner has been waiting for us to submit, so he can reference our case. You should be aware in this forum, we are not likely to address all the details that Nexenta would be interested in. -ted -- Ted H. Kim, PhD Sun/Oracle America Inc. ted.h.kim@oracle.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1115 FAX From garrett@damore.org Wed Jun 30 14:43:05 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o5ULh5fF012306 for ; Wed, 30 Jun 2010 14:43:05 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o5ULh4Ud011974 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 30 Jun 2010 14:43:05 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L4U0080XKBSDW00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 30 Jun 2010 14:43:04 -0700 (PDT) Received: from sca-ea-mail-3.sun.com ([192.18.43.21]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L4U005ETKBRHX90@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 30 Jun 2010 14:43:04 -0700 (PDT) Received: from relay13i.sun.com (ip123.net129179-4.block1.us.syntegra.com [129.179.4.123]) by sca-ea-mail-3.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o5ULh3uJ003549 for ; Wed, 30 Jun 2010 21:43:03 +0000 (GMT) Received: from mmp12es.mmp.us.syntegra.com ([160.41.208.12] [160.41.208.12]) by relay13i.sun.com with ESMTP id BT-MMP-4018117 for PSARC-ext@Sun.COM; Wed, 30 Jun 2010 21:43:03 +0000 (Z) Received: from relay15i.sun.com (relay15i.sun.com [129.179.4.125]) by mmp12es.mmp.us.syntegra.com with ESMTP id BT-MMP-805365 for PSARC-ext@Sun.COM; Wed, 30 Jun 2010 21:43:02 +0000 (Z) Received: from oproxy2-pub.bluehost.com ([67.222.39.60] [67.222.39.60]) by relay1i.sun.com id BT-MMP-7271168 for PSARC-ext@Sun.COM; Wed, 30 Jun 2010 21:43:02 +0000 (Z) Received: (qmail 14645 invoked by uid 0); Wed, 30 Jun 2010 21:43:02 +0000 Received: from unknown (HELO box374.bluehost.com) (69.89.31.174) by oproxy2.bluehost.com with SMTP; Wed, 30 Jun 2010 21:43:02 +0000 Received: from cpe-76-93-15-33.socal.res.rr.com ([76.93.15.33] helo=[192.168.251.110]) by box374.bluehost.com with esmtpsa (SSLv3:AES256-SHA:256) (Exim 4.69) (envelope-from ) id 1OU53e-0002vh-4r; Wed, 30 Jun 2010 15:43:02 -0600 Date: Wed, 30 Jun 2010 14:45:32 -0700 From: "Garrett D'Amore" Subject: Re: Fibre Channel over IB Support [PSARC/2010/251 FastTrack timeout 07/08/2010] In-reply-to: <4C2BB198.7000901@oracle.com> To: "Ted H. Kim" Cc: Ted Kim , PSARC-ext@sun.com Message-id: <1277934332.5596.4680.camel@velocity> MIME-version: 1.0 X-Mailer: Evolution 2.28.3 Content-type: text/plain; charset=UTF-8 Content-transfer-encoding: 7BIT DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=damore.org; h=Received:Subject:From:To:Cc:In-Reply-To:References:Content-Type:Date:Message-ID:Mime-Version:X-Mailer:Content-Transfer-Encoding:X-Identified-User; b=YScb1QjonjHTwqHaJTLP+rEOCWKrN+URg0/9PvHd6cp0WUBk9AqooOM3YTnlx0BvbkZVCG0g33zwsOlBBZWFdR9vuvjAUUJCuOBx8SiCQFUflR6aGRIhCa+kuvhM1ToH; X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Identified-User: {2225:box374.bluehost.com:damoreor:damore.org} {sentby:smtp auth 76.93.15.33 authed with garrett+damore.org} X-Antispam: No, score=0.0/5.0, scanned in 0.215sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <201006301938.o5UJcU8O007971@sac.sfbay.sun.com> <1277929776.5596.4654.camel@velocity> <4C2BAD8A.50905@oracle.com> <1277931366.5596.4677.camel@velocity> <4C2BB198.7000901@oracle.com> Status: RO Content-Length: 1144 On Wed, 2010-06-30 at 14:05 -0700, Ted H. Kim wrote: > Garrett D'Amore wrote: > > So this case is just plumbing details? > > If that is what you want to call IBTF and hermon. > > > I'm a bit concerned that the case is incomplete without the FCoIB ULP > > driver as part of it. Specifying an API without a consumer makes this > > feel like an incomplete spec/case, at least to me. > > > > Is there some reason to run this case now instead of together with the > > ULP driver? > > I think the ULP driver owner has been waiting for us to submit, > so he can reference our case. I think you guys should collaborate and submit just single case covering the entire project. Breaking this into two separate fast tracks feels just plain wrong to me. We can't really consider one case fully without knowing more about the other, I think. > > You should be aware in this forum, we are not likely > to address all the details that Nexenta would be > interested in. I'm not concerned about Nexenta (well I am, but that wasn't my motivation here. :-) I'm more interested in seeing an architecturally complete case. - Garrett > > -ted > From ted.h.kim@oracle.com Wed Jul 7 12:11:00 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o67JB07L013093 for ; Wed, 7 Jul 2010 12:11:00 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o67JB0XQ006407 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 7 Jul 2010 14:11:00 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L5700203BYCOK00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 07 Jul 2010 12:11:00 -0700 (PDT) Received: from sca-ea-mail-4.sun.com ([192.18.43.22]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L5700L4KBYBXJ90@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 07 Jul 2010 12:10:59 -0700 (PDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by sca-ea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o67JAxmv012924 for ; Wed, 07 Jul 2010 19:10:59 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o67IeaHf004115; Wed, 07 Jul 2010 19:10:57 +0000 (GMT) Received: from abhmt018.oracle.com by acsmt355.oracle.com with ESMTP id 406994831278529779; Wed, 07 Jul 2010 12:09:39 -0700 Received: from [129.153.3.74] (/129.153.3.74) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 07 Jul 2010 12:09:38 -0700 Date: Wed, 07 Jul 2010 12:07:56 -0700 From: "Ted H. Kim" Subject: Re: Fibre Channel over IB Support [PSARC/2010/251 FastTrack timeout 07/08/2010] In-reply-to: <4C2BB198.7000901@oracle.com> To: "Garrett D'Amore" Cc: PSARC-ext@sun.com Message-id: <4C34D08C.2030500@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=UTF-8; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090209.4C34D141.014E:SCFMA4539814,ss=1,fgs=0 References: <201006301938.o5UJcU8O007971@sac.sfbay.sun.com> <1277929776.5596.4654.camel@velocity> <4C2BAD8A.50905@oracle.com> <1277931366.5596.4677.camel@velocity> <4C2BB198.7000901@oracle.com> User-Agent: Thunderbird 2.0.0.24 (X11/20100302) Status: RO Content-Length: 1309 Just to continue the discussion started at the 7/7 PSARC meeting ... Ted H. Kim wrote: > Garrett D'Amore wrote: >> So this case is just plumbing details? If I understand what you mean by that, the answer is yes. >> I'm a bit concerned that the case is incomplete without the FCoIB ULP >> driver as part of it. Specifying an API without a consumer makes this >> feel like an incomplete spec/case, at least to me. >> >> Is there some reason to run this case now instead of together with the >> ULP driver? > > I think the ULP driver owner has been waiting for us to submit, > so he can reference our case. Let's look at how a "regular" FC driver case would have been presented. The driver case would have talked about all the stuff you are asking about (multipathing, boot, etc.). They would not have described the register set of the HBA card per se. This case more or less talks about the "register set" for the FCoIB ULP driver. Yes the operations are mapped over IB, instead of some PCI registers. But the only reason is coming to the ARC's attention is that this is an interface between projects. -ted -- Ted H. Kim, PhD Sun/Oracle America Inc. ted.h.kim@oracle.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1115 FAX From garrett@damore.org Wed Jul 7 13:43:00 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o67Kh0MH016503 for ; Wed, 7 Jul 2010 13:43:00 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o67Kgxid024784 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 7 Jul 2010 14:42:59 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L570071LG7NK500@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 07 Jul 2010 13:42:59 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L57005P1G7M8Z10@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 07 Jul 2010 13:42:59 -0700 (PDT) Received: from relay41i.sun.com ([192.5.209.70]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o67KcUx4027034 for ; Wed, 07 Jul 2010 20:42:58 +0000 (GMT) Received: from mmp43es.mmp.us.syntegra.com ([160.41.221.12] [160.41.221.12]) by relay41i.sun.com with ESMTP id BT-MMP-745 for PSARC-ext@Sun.COM; Wed, 07 Jul 2010 20:42:58 +0000 (Z) Received: from relay44i.sun.com (relay44i.sun.com [192.5.209.118]) by mmp43es.mmp.us.syntegra.com with ESMTP id BT-MMP-12851212 for PSARC-ext@Sun.COM; Wed, 07 Jul 2010 20:42:57 +0000 (Z) Received: from oproxy1-pub.bluehost.com ([66.147.249.253] [66.147.249.253]) by relay4i.sun.com id BT-MMP-912715 for PSARC-ext@Sun.COM; Wed, 07 Jul 2010 20:42:53 +0000 (Z) Received: (qmail 29764 invoked by uid 0); Wed, 07 Jul 2010 20:42:51 +0000 Received: from unknown (HELO box374.bluehost.com) (69.89.31.174) by oproxy1.bluehost.com.bluehost.com with SMTP; Wed, 07 Jul 2010 20:42:51 +0000 Received: from cpe-76-93-15-33.socal.res.rr.com ([76.93.15.33] helo=[192.168.251.110]) by box374.bluehost.com with esmtpsa (SSLv3:AES256-SHA:256) (Exim 4.69) (envelope-from ) id 1OWbSF-00004u-BY; Wed, 07 Jul 2010 14:42:51 -0600 Date: Wed, 07 Jul 2010 13:43:32 -0700 From: "Garrett D'Amore" Subject: Re: Fibre Channel over IB Support [PSARC/2010/251 FastTrack timeout 07/08/2010] In-reply-to: <4C34D08C.2030500@oracle.com> To: "Ted H. Kim" Cc: PSARC-ext@sun.com Message-id: <1278535412.2510.141.camel@velocity> MIME-version: 1.0 X-Mailer: Evolution 2.28.3 Content-type: text/plain; charset=UTF-8 Content-transfer-encoding: 7BIT DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=damore.org; h=Received:Subject:From:To:Cc:In-Reply-To:References:Content-Type:Date:Message-ID:Mime-Version:X-Mailer:Content-Transfer-Encoding:X-Identified-User; b=oB1sMMkq8nrFv5+qT6kjK3zfrb72c5zEO34mtEKdtWT9NbLlqH75KPxVcgxRxlsT4CaMLxEH7qu7xqy9Uv2s1wzV+M1FLbqrvA79yNmwk/FJIuGxKM2exWTB8lBGPGAJ; X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Identified-User: {2225:box374.bluehost.com:damoreor:damore.org} {sentby:smtp auth 76.93.15.33 authed with garrett+damore.org} X-Antispam: No, score=-2.6/5.0, scanned in 0.267sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <201006301938.o5UJcU8O007971@sac.sfbay.sun.com> <1277929776.5596.4654.camel@velocity> <4C2BAD8A.50905@oracle.com> <1277931366.5596.4677.camel@velocity> <4C2BB198.7000901@oracle.com> <4C34D08C.2030500@oracle.com> Status: RO Content-Length: 2789 On Wed, 2010-07-07 at 12:07 -0700, Ted H. Kim wrote: > Just to continue the discussion started at the 7/7 PSARC meeting ... > > Ted H. Kim wrote: > > Garrett D'Amore wrote: > >> So this case is just plumbing details? > > If I understand what you mean by that, the answer is yes. > > >> I'm a bit concerned that the case is incomplete without the FCoIB ULP > >> driver as part of it. Specifying an API without a consumer makes this > >> feel like an incomplete spec/case, at least to me. > >> > >> Is there some reason to run this case now instead of together with the > >> ULP driver? > > > > I think the ULP driver owner has been waiting for us to submit, > > so he can reference our case. > > Let's look at how a "regular" FC driver case would have been > presented. The driver case would have talked about all the > stuff you are asking about (multipathing, boot, etc.). > They would not have described the register set of the HBA card > per se. > > This case more or less talks about the "register set" for > the FCoIB ULP driver. Yes the operations are mapped over IB, > instead of some PCI registers. But the only reason is coming > to the ARC's attention is that this is an interface between > projects. I'm sorry, I don't quite buy this. We don't normally expose "register sets" directly to consuming software -- drivers perform that purpose, and generally they have a DDI to which they conform. This case is essentially (if I understand it correctly), creating a new DDI (albeit one that is closely tied to specific hardware), without presenting an actual first consumer. Some of the questions I asked here are really relevant to the broader picture of things. I'd like to be able to review the cases together, either as a single case (which is what I think is most appropriate here, since as I understand it neither case presents a "generic" interface but instead is directly tied to the other), or possibly as two separate cases presented at the same time. Until we know what the overall picture is, how can we know if these interfaces are architecturally complete? Alternatively, if these interfaces are really private between the IBTF, hermon, and the FCoIB driver, then perhaps listing these changes *briefly* in the FCoIB case (along with Consolidation or Project Private) would have been better. A lot of the detail could probably then simply be elided instead of spelled out in excruciating detail here. It may be that my concerns here are entirely pedantic, and don't warrant further consideration. *Alternatively*, if you were to mark all these changes as Project Private, then you could get this case closed easily without further review. You would then need to setup a contract between the FCoIB project and this project. - Garrett > > -ted > From ted.h.kim@oracle.com Wed Jul 7 14:12:59 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o67LCwCf017110 for ; Wed, 7 Jul 2010 14:12:58 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o67LCwR4026210 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 7 Jul 2010 16:12:58 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L5700I0XHLMF100@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 07 Jul 2010 14:12:58 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L570044AHLLZQ60@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 07 Jul 2010 14:12:57 -0700 (PDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o67LCuMM013307 for ; Wed, 07 Jul 2010 21:12:56 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o6777aZY015160; Wed, 07 Jul 2010 21:12:54 +0000 (GMT) Received: from abhmt010.oracle.com by acsmt355.oracle.com with ESMTP id 407299311278537102; Wed, 07 Jul 2010 14:11:42 -0700 Received: from [129.153.3.74] (/129.153.3.74) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 07 Jul 2010 14:11:42 -0700 Date: Wed, 07 Jul 2010 14:09:59 -0700 From: "Ted H. Kim" Subject: Re: Fibre Channel over IB Support [PSARC/2010/251 FastTrack timeout 07/08/2010] In-reply-to: <1278535412.2510.141.camel@velocity> To: "Garrett D'Amore" Cc: PSARC-ext@sun.com Message-id: <4C34ED27.2060100@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=UTF-8; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090209.4C34EDD6.020C:SCFMA4539814,ss=1,fgs=0 References: <201006301938.o5UJcU8O007971@sac.sfbay.sun.com> <1277929776.5596.4654.camel@velocity> <4C2BAD8A.50905@oracle.com> <1277931366.5596.4677.camel@velocity> <4C2BB198.7000901@oracle.com> <4C34D08C.2030500@oracle.com> <1278535412.2510.141.camel@velocity> User-Agent: Thunderbird 2.0.0.24 (X11/20100302) Status: RO Content-Length: 1239 okay picking up on this part of the thread ... > Alternatively, if these interfaces are really private between the IBTF, > hermon, and the FCoIB driver, then perhaps listing these changes > *briefly* in the FCoIB case (along with Consolidation or Project > Private) would have been better. A lot of the detail could probably > then simply be elided instead of spelled out in excruciating detail > here. > > It may be that my concerns here are entirely pedantic, and don't warrant > further consideration. > > *Alternatively*, if you were to mark all these changes as Project > Private, then you could get this case closed easily without further > review. You would then need to setup a contract between the FCoIB > project and this project. Actually, what you say here makes a lot of sense, because the FCoIB ULP driver is the only consumer for this API. These are not generic operations even for other IB consumers. Anyone else going to have a problem if I take this project private and just do a contract with the FCoIB driver? -ted -- Ted H. Kim, PhD Sun/Oracle America Inc. ted.h.kim@oracle.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1115 FAX From ted.h.kim@oracle.com Fri Jul 9 11:34:47 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o69IYlwm011549 for ; Fri, 9 Jul 2010 11:34:47 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o69IYlOb007241 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 9 Jul 2010 11:34:47 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L5A00C01ZLY9A00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Fri, 09 Jul 2010 12:34:46 -0600 (MDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L5A00L0QZLYU470@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Fri, 09 Jul 2010 12:34:46 -0600 (MDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o69IYjrs029992 for ; Fri, 09 Jul 2010 18:34:46 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o699Z7H3015337 for ; Fri, 09 Jul 2010 18:34:10 +0000 (GMT) Received: from abhmt002.oracle.com by acsmt353.oracle.com with ESMTP id 393359481278700440; Fri, 09 Jul 2010 11:34:00 -0700 Received: from [129.153.3.74] (/129.153.3.74) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 09 Jul 2010 11:34:00 -0700 Date: Fri, 09 Jul 2010 11:32:17 -0700 From: "Ted H. Kim" Subject: Re: Fibre Channel over IB Support [PSARC/2010/251 FastTrack timeout 07/08/2010] In-reply-to: <4C34ED27.2060100@oracle.com> To: PSARC-ext@sun.com Message-id: <4C376B31.1070409@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=UTF-8; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090206.4C376BC0.0032:SCFMA4539814,ss=1,fgs=0 References: <201006301938.o5UJcU8O007971@sac.sfbay.sun.com> <1277929776.5596.4654.camel@velocity> <4C2BAD8A.50905@oracle.com> <1277931366.5596.4677.camel@velocity> <4C2BB198.7000901@oracle.com> <4C34D08C.2030500@oracle.com> <1278535412.2510.141.camel@velocity> <4C34ED27.2060100@oracle.com> User-Agent: Thunderbird 2.0.0.24 (X11/20100302) Status: RO Content-Length: 1579 Folks, Okay, I haven't heard objections. So I am going to make this all Project Private and change the status to closed approved automatic. The case, of course, will be remain as a reference to be used when the FCoIB ULP case comes along and a contract is put in place to use this stuff. -ted Ted H. Kim wrote: > okay picking up on this part of the thread ... > >> Alternatively, if these interfaces are really private between the IBTF, >> hermon, and the FCoIB driver, then perhaps listing these changes >> *briefly* in the FCoIB case (along with Consolidation or Project >> Private) would have been better. A lot of the detail could probably >> then simply be elided instead of spelled out in excruciating detail >> here. >> >> It may be that my concerns here are entirely pedantic, and don't warrant >> further consideration. >> >> *Alternatively*, if you were to mark all these changes as Project >> Private, then you could get this case closed easily without further >> review. You would then need to setup a contract between the FCoIB >> project and this project. > > Actually, what you say here makes a lot of sense, because the > FCoIB ULP driver is the only consumer for this API. These are not > generic operations even for other IB consumers. > > Anyone else going to have a problem if I take this > project private and just do a contract with the FCoIB driver? -- Ted H. Kim, PhD Sun/Oracle America Inc. ted.h.kim@oracle.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1115 FAX From sacadmin Fri Jul 9 11:46:44 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o69IkiUc012065 for ; Fri, 9 Jul 2010 11:46:44 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o69Ikixu003114 for <@sunmail2sca.sfbay.sun.com:psarc-record@sun.com>; Fri, 9 Jul 2010 11:46:44 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L5B00K0305WWJ00@nwk-avmta-1.sfbay.Sun.COM> for psarc-record@sun.com (ORCPT psarc-record@sun.com); Fri, 09 Jul 2010 11:46:44 -0700 (PDT) Received: from sca-ea-mail-3.sun.com ([192.18.43.21]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L5B00KVJ05VDJC0@nwk-avmta-1.sfbay.Sun.COM> for psarc-record@sun.com (ORCPT psarc-record@sun.com); Fri, 09 Jul 2010 11:46:43 -0700 (PDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by sca-ea-mail-3.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o69Ikhva028378 for ; Fri, 09 Jul 2010 18:46:43 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o69BICPg016488 for ; Fri, 09 Jul 2010 18:46:42 +0000 (GMT) Received: from abhmt016.oracle.com by acsmt353.oracle.com with ESMTP id 413774491278701178; Fri, 09 Jul 2010 11:46:18 -0700 Received: from [129.153.3.74] (/129.153.3.74) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 09 Jul 2010 11:46:18 -0700 Date: Fri, 09 Jul 2010 11:44:35 -0700 From: "Ted H. Kim" Subject: Fibre Channel over IB [ PSARC/2010/251 ] To: psarc-record@sun.com Message-id: <4C376E13.106@oracle.com> MIME-version: 1.0 Content-type: multipart/mixed; boundary="Boundary_(ID_jz+AX5dE2gIwtqtCd9Bmkg)" X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090208.4C376E92.0326:SCFMA4539814,ss=1,fgs=0 User-Agent: Thunderbird 2.0.0.24 (X11/20100302) Status: RO Content-Length: 11493 This is a multi-part message in MIME format. --Boundary_(ID_jz+AX5dE2gIwtqtCd9Bmkg) Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT Here is the final version reposted for the record. The only change is to state that these interfaces are Project Private and that it is intended that the FCoIB driver will take a contract to use these. -ted -- Ted H. Kim, PhD Sun/Oracle America Inc. ted.h.kim@oracle.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1115 FAX --Boundary_(ID_jz+AX5dE2gIwtqtCd9Bmkg) Content-type: text/plain; name=revised-FCoIB-ARC.txt Content-transfer-encoding: 7BIT Content-disposition: inline; filename=revised-FCoIB-ARC.txt 4. Technical Description Fibre Channel over IB Support ----------------------------- 4.1 Background The Mellanox BridgeX product provides a gateway between InfiniBand (IB) and 10 Gb Ethernet and/or 2/4/8 Gb Fibre Channel (FC). The bridging protocol between IB and Fibre Channel is called "Fibre Channel over IB" (FCoIB). Mellanox has added IB host adapter extensions to the ConnectX IB adapter to support the BridgeX FCoIB protocol. This case introduces interfaces into our Solaris InfiniBand Transport Framework (IBTF, PSARC/2002/132 and follow-on cases) for the Mellanox host adapter extensions used to support FCoIB initiator. The expected consumer for FCoIB support is our "hermon" ConnectX IB driver (PSARC/2008/497) and the FCoIB initiator driver. 4.2 Proposal The proposal is to make additions to the IBTF Transport Interface (TI) for IB Upper Level Protocol (ULP) clients and the Channel Interface (CI) for HCA drivers. However, unlike the majority of the TI and CI (which are ON Consolidation Private) these items are Project Private. The FCoIB ULP driver case will be taking a contract to use this interface. All interface changes and additions in the proposal have a micro/patch binding. Transport Interface (Project Private) ibt_hca_flags2_t: add IBT_HCA2_FC flags ibt_hca_attr_t: add hca_rfci_max_log2_qp, hca_fexch_max_log2_qp, hca_fexch_log2_mem fields ibt_chan_alloc_flags_t: add IBT_ACHAN_USES_RFCI, IBT_ACHAN_USES_FCMD, IBT_ACHAN_USES_FEXCH flags ibt_fc_attr_t: new struct for FC QP attributes ibt_ud_chan_alloc_args_t: add ud_fc (ibt_fc_attr_t) field ibt_fexch_query_attr_t: new struct for FEXCH query attributes ibt_ud_chan_query_attr_t: add ud_fc (ibt_fc_attr_t), ud_query_fc (ibt_fexch_query_attr_t) fields ibt_cep_modify_flags_t: add IBT_CEP_SET_FEXCH_RANGE flag ibt_ud_chan_modify_attr_t: add uc_fc (ibt_fc_attr_t) field ibt_tran_srv_t: add IBT_RFCI_SRV, IBT_FCMD_SRV, IBT_FEXCH_SRV ibt_wrc_opcode_t: add IBT_WRC_INIT_SEND_FCMD ibt_wr_flags_t: add IBT_WR_SEND_FC_CRC flag ibt_wr_rfci_send_t: new struct for RFCI version of Send WR ibt_wr_init_send_t: new struct for FCMD WR to start I/O operation ibt_wr_fc_t: new union of possible FC WRs ibt_send_wr_t wr field: add fc (ibt_wr_fc_t) to union ibt_wc_flags_t: add IBT_WC_FEXCH_FMT, IBT_WC_DIF_ERROR flags ibt_wc_t wc_detail field: add IBT_WC_DETAIL_FEXCH_INIT_XFER, IBT_WC_FEXCH_LAST, IBT_WC_DETAIL_RFCI_CRC_OK, IBT_WC_DETAIL_FC_MATCH_MASK ibt_wc_t: add wc_fexch_seq_cnt, wc_fexch_tx_bytes, wc_rx_bytes_xfer, wc_fexch_seq_id fields ibt_async_code_t: add IBT_FEXCH_ERROR ibt_fc_syndrome_t: new enum for FEXCH async errors ibt_async_event_t: add ev_fc (ibt_fc_syndrome_t) Channel Interface (Project Private) ibt_hca_flags2_t: add IBT_HCA2_FC flags ibt_hca_attr_t: add hca_rfci_max_log2_qp, hca_fexch_max_log2_qp, hca_fexch_log2_mem fields ibt_qp_alloc_flags_t: add IBT_QP_USES_RFCI, IBT_QP_USES_FCMD, IBT_QP_USES_FEXCH flags ibt_fc_attr_t: new struct for FC QP attributes ibt_qp_alloc_attr_t: add qp_fc (ibt_fc_attr_t) ibt_fexch_query_attr_t: new struct for FEXCH query attributes ibt_qp_query_attr_t: add qp_query_fexch (ibt_fexch_query_attr_t) ibt_cep_modify_flags_t: add IBT_CEP_SET_FEXCH_RANGE flag ibt_qp_ud_attr_t: add ud_fc (ibt_fc_attr_t) ibt_tran_srv_t: add IBT_RFCI_SRV, IBT_FCMD_SRV, IBT_FEXCH_SRV ibt_wrc_opcode_t: add IBT_WRC_INIT_SEND_FCMD ibt_wr_flags_t: add IBT_WR_SEND_FC_CRC ibt_wr_rfci_send_t: new struct for RFCI version of Send WR ibt_wr_init_send_t: new struct FCMD WR to start I/O operation ibt_wr_fc_t: new union of possible FC WRs ibt_send_wr_t wr field: add fc (ibt_wr_fc_t) to union ibt_wc_flags_t: add IBT_WC_FEXCH_FMT, IBT_WC_DIF_ERROR flags ibt_wc_t wc_detail field: add IBT_WC_DETAIL_FEXCH_INIT_XFER, IBT_WC_FEXCH_LAST, IBT_WC_DETAIL_RFCI_CRC_OK, IBT_WC_DETAIL_FC_MATCH_MASK ibt_wc_t: add wc_fexch_seq_cnt, wc_fexch_tx_bytes, wc_rx_bytes_xfer, wc_fexch_seq_id fields ibt_async_code_t: add IBT_FEXCH_ERROR ibt_fc_syndrome_t: new enum for FEXCH async errors ibc_async_event_t: add ev_fc (ibt_fc_syndrome_t) The changes in this case are part of IBTF ABI version 4. Copies of all new and modified man pages are in the materials directory (see section 4.3 below). A. Detailed Description of FCoIB Support While the FCoIB support could also support targets, the extensions in this proposal are only sufficient to support initiators. Mellanox support for FCoIB involves three variants of the IB Unreliable Datagram (UD) transport service type. The first is a Raw Fibre Channel Interface (RFCI) Queue Pair (QP), which is used to establish the association between initiator and targets. When it comes to I/O operations, typically, these QPs are used to send confirmation (FCP_CONF) messages back to the target. In IB terms, these QPs are like regular UD QPs, except that they have an option for FC CRC support and End of Frame (EOF) values. The second special QP type is the Fibre Channel Command (FCMD) QP. This QP is used to initiate I/O operations through the associated FEXCH QPs (see below) and perform memory registration through Work Requests (WRs). Posting a special type of WR (Init & Send) causes a data transfer operation to be assigned to an FEXCH QP and also causes a command (FCP_CMND) message to be sent to the target. Registration is accomplished on FCMDs by using the Registration WR previously defined in PSARC/2009/060 only for Reliable Connected QPs. There is no receive half to FCMD QPs. The third special QP type is the Fibre Channel Exchange (FEXCH) QP. As mentioned before, FCMD WRs assign data transfer operations to associated FEXCH QPs. Allocation of FEXCH QPs is done via an ibt_alloc_ud_channel_range() call to assign a group/range of QPs all at once to an FCMD. The send half of these FEXCH QPs are not directly used. The receive half of these QPs are used to signal the completion of I/O operations. The work completions generated from the receive queue have additional fields specific to Fibre Channel operations. These FEXCH QPs also have associated memory regions (MRs) which define the source/sink for a data transfer. Query QP operations show which MRs are assigned to any given FEXCH QP. The MRs are registered by using Registration WRs posted on the associated FCMD QP. So pulling it all together, the typical I/O operation involves: (1) registering memory belonging to an FEXCH QP through a Registration WR on an FCMD QP, (2) initiating a data transfer with an Init & Send WR on the FCMD, (3) waiting for the FEXCH completion and (4) a possible confirmation message sent via an associated RFCI QP. Add capability flag to ibt_hca_flags2_t to indicate support: IBT_HCA2_FC = 1 << 12 /* FCoIB support */ Add other FC related HCA attributes to ibt_hca_attr_t: uint8_t hca_rfci_max_log2_qp; /* log2 num RFCI QPs */ uint8_t hca_fexch_max_log2_qp; /* log2 num FEXCH QPs */ uint8_t hca_fexch_log2_mem; /* log2 size of FEXCH MRs (in bytes) */ Add flags and attributes (ibt_fc_attr_t) to for allocate QP ops: ibt_chan_alloc_flags_t: add IBT_ACHAN_USES_RFCI, /* TI */ IBT_ACHAN_USES_FCMD, IBT_ACHAN_USES_FEXCH flags add ibt_fc_attr_t struct to ibt_ud_chan_alloc_args_t ibt_qp_alloc_flags_t: add IBT_QP_USES_RFCI, /* CI */ IBT_QP_USES_FCMD, IBT_QP_USES_FEXCH flags add ibt_fc_attr_t to ibt_qp_alloc_attr_t Add attributes (ibt_fc_attr_t & ibt_fexch_query_attr_t) for query QP ops: add ibt_fc_attr_t & ibt_fexch_query_attr_t to /* TI */ ibt_ud_chan_query_attr_t add ibt_fexch_query_attr_t to ibt_qp_query_attr_t /* CI */ add ibt_fc_attr_t to ibt_qp_ud_attr_t Add flags and attributes (ibt_fc_attr_t) to modify QP ops: add IBT_CEP_SET_FEXCH_RANGE flag to ibt_cep_modify_flags_t add ibt_fc_attr_t to ibt_ud_chan_modify_attr_t /* TI */ add ibt_fc_attr_t to ibt_qp_ud_attr_t /* CI */ Add the new work requests structs: define new transport service (QP) types for ibt_tran_srv_t: IBT_RFCI_SRV, IBT_FCMD_SRV, IBT_FEXCH_SRV for RFCI QPs, define a variant of UD Send (ibt_wr_rfci_send_t), this operation can also use a new flag (IBT_WR_SEND_FC_CRC in ibt_wr_flags_t) for FCMD QPs, define a new Init & Send WR to assign initiate I/O operations on FEXCH QPs (ibt_wr_init_send_t) and a new opcode (IBT_WRC_INIT_SEND_FCMD in ibt_wrc_opcode_t) define union of possible FC WRs (ibt_wr_fc_t) & add it to ibt_send_wr_t Add new work completion fields: add IBT_WC_DIF_ERROR flag to show Data Integrity Field errors add IBT_WC_DETAIL_RFCI_CRC_OK flag for RFCI Send completions add IBT_WC_FEXCH_FMT flag to show when using the special FEXCH format; when set, the following are defined: add wc_fexch_seq_cnt (sequence count), wc_fexch_tx_bytes (transmit bytes), wc_rx_bytes_xfer (receive byes), wc_fexch_seq_id (sequence id) fields add IBT_WC_DETAIL_FEXCH_INIT_XFER (initiative transfer) and IBT_WC_FEXCH_LAST (last sequence) flags Add a new type of async error for FEXCH syndromes: add IBT_FEXCH_ERROR to ibt_async_code_t add ibt_fc_syndrome_t field to ibt_async_event_t 4.3 Man page summary Man Page Disposition (sorted by section) ------------------------------------------- ibc_alloc_qp.9e changed ibc_alloc_qp_range.9e changed ibt_async_handler_t.9e changed ibc_async_handler.9f changed ibt_alloc_ud_channel.9f changed ibt_alloc_ud_channel_range.9f changed ibt_modify_ud_channel.9f changed ibt_query_ud_channel.9f changed ibc_qp_info_t.9s changed ibc_qp_query_attr_9.9s changed ibt_fc_attr_t.9s new ibt_fexch_query_attr_t.9s new ibt_hca_attr_t.9s changed ibt_send_wr_t.9s changed ibt_wc_t.9s changed ibt_wr_fc_t.9s new ibt_wr_rc_t.9s changed ibt_wr_reg_pmr_t.9s new Note: Since FCMD QPs can also do "Registration Work Request" operations, we split the description of that WR out of ibt_wr_rc_t.9s and put it into ibt_wr_reg_pmr_t.9s (where it can be referenced by both ibt_wr_rc_t.9s and ibt_wr_fc_t.9s). Aside from reorganizing where the information is, there is no change to the description of Registration Work Request. --Boundary_(ID_jz+AX5dE2gIwtqtCd9Bmkg)--