From tedk@sac.sfbay.sun.com Wed Oct 3 14:23:59 2007 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l93LNxlV011549 for ; Wed, 3 Oct 2007 14:23:59 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l93LKnOC009642; Wed, 3 Oct 2007 14:20:49 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPC0041HTYP5O00@brm-avmta-1.central.sun.com>; Wed, 03 Oct 2007 15:20:49 -0600 (MDT) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPC003RKTYOHN00@brm-avmta-1.central.sun.com>; Wed, 03 Oct 2007 15:20:48 -0600 (MDT) Received: from sac.sfbay.sun.com (new-sac.SFBay.Sun.COM [129.146.175.65]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id l93LKlGV025559; Wed, 03 Oct 2007 14:20:47 -0700 (PDT) Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l93LNsFN011544; Wed, 03 Oct 2007 14:23:54 -0700 (PDT) Received: (from tedk@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id l93LNsga011540; Wed, 03 Oct 2007 14:23:54 -0700 (PDT) Date: Wed, 03 Oct 2007 14:23:54 -0700 (PDT) From: Ted Kim Subject: Generic HCA Support for uDAPL [PSARC/2007/575 FastTrack timeout 10/10/2007] To: PSARC-ext@sun.com Cc: bill.taylor@sun.com, brendan.doyle@sun.com, ted.kim@sun.com Message-id: <200710032123.l93LNsga011540@sac.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 Status: RO Content-Length: 4737 Template Version: @(#)sac_nextcase 1.64 07/13/07 SMI This information is Copyright 2007 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: Generic HCA Support for uDAPL 1.2. Name of Document Author/Supplier: Author: Bill Taylor 1.3 Date of This Document: 03 October, 2007 4. Technical Description Generic HCA Support for uDAPL ============================= Background ---------- One important usage mode of InfiniBand is the "OS Bypass" model. Typically what is done is to map InfiniBand adapter (HCA) objects, such as queue pairs and completion queues, into the address space of userland application processes. The application can then directly talk to the InfiniBand hardware through the mapped objects. Of course, the set up of these mappings must be done in the kernel. The current framework in Solaris used to accomplish OS Bypass is "uDAPL" (PSARC/2003/145) which is used by the Sun Clustertool MPI product. (Another OS bypass framework called "Open Fabrics User Verbs" is now under development as an OpenSolaris project.) uDAPL is structured to have a part that is generic and a part which is specific to each type of HCA (called a "plugin" library). Currently, Mellanox HCAs which run the Tavor driver (PSARC/2002/539) are supported. However, relatively soon support will be needed for Mellanox Arbel (PSARC/2006/400) and Hermon (1-pager not filed yet) HCAs. Initially, it was expected that the driver for Mellanox Arbel would use a similar structure to Tavor. However, since then it has been discovered that the Arbel hardware uses a different design. Hermon has even further changes. Also, it's possible that HCAs from other vendors may eventually be supported. Rather than try to have uDAPL understand data formats for so many different adapters, we are changing uDAPL to keep the details of each adapter private to the HCA driver and device-specific libraries. The uDAPL kernel module now has a much smaller *GENERIC* interface which is the same for all HCAs. Other device specific information is still transferred, but that information is simply treated as an opaque block of bytes to copy with only the HCA specific code understanding it (i.e. the in-kernel HCA driver and the userland plug-in library). As with other kernel usage of InfiniBand, the uDAPL kernel module calls through InfiniBand Transport Framework (IBTF) (PSARC/2002/132) "Transport interface" to the HCA driver entry points in the "Channel interface". The particular functions used by the uDAPL kernel module were introduced in "Userland HCA Object Mapping calls" of PSARC/2003/358. The specific calls are the IBTF ibt_ci_data_in() and ibt_ci_data_out() functions which in turn call ibc_ci_data_in and ibc_ci_data_out entry points on the HCA drivers. This case now codifies the semantics that uDAPL now expects for these IBTF calls within IBTF itself for all OS-bypass users (instead of treating it as a private agreement between uDAPL and the HCA drivers). Proposal -------- The interface binding for this entire proposal is micro/patch. There are three parts to this proposal: A. Add new semantics to IBTF calls to allow supporting multiple HCA types for uDAPL and other InfiniBand OS-bypass users. All calls continue to be consolidation private as before. The modified functions are: Transport interface - ibt_ci_data_in(9F) and ibt_ci_data_out(9F): both described in the ibt_ci_data_in.9f man page. Channel interface - ibc_ci_data_in(9E) and ibc_ci_data_out(9E): both described in the ibc_ci_data_in.9e man page. See materials directory for change-bar versions of these man pages. B. replace the existing Tavor/uDAPL interface from PSARC/2003/557 (main interface) and PSARC/2004/737 (adding SRQ) Not only is the content of these cases replaced by the new interface, but contracts from those older cases are obsoleted by *this* case. The contracts being terminated are: (a). "contract-01" in the case directory for PSARC/2003/557 (b). "draft-contract-01" in the case directory for PSARC/2004/737 C. replace the existing Arbel/uDAPL interface from PSARC/2006/400 The information in "arbel-udapl-interface.txt" in the PSARC/2006/400 case materials directory is replaced by the new interface, and "contract-02.txt" from PSARC/2006/400 is obsoleted by *this* case. Further, the plug-in library path will be the same as previouusly used by Tavor for compatibility reasons: /usr/lib/{64/}udapl_tavor.so.1. Note that the rest of the PSARC/2006/400 case is not altered by this proposal. 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From sacadmin Wed Oct 10 10:24:49 2007 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l9AHOmsM016478 for ; Wed, 10 Oct 2007 10:24:48 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l9AHLPoC012167 for <@sunmail2sca.sfbay.sun.com:psarc-record@sun.com>; Wed, 10 Oct 2007 10:21:33 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPP00J0RHJWSC00@brm-avmta-1.central.sun.com> for psarc-record@sun.com (ORCPT psarc-record@sun.com); Wed, 10 Oct 2007 11:21:32 -0600 (MDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPP00H7OHJVWH20@brm-avmta-1.central.sun.com> for psarc-record@sun.com (ORCPT psarc-record@sun.com); Wed, 10 Oct 2007 11:21:31 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l9AHLV5w020513 for ; Wed, 10 Oct 2007 10:21:31 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPP00001HG9J700@fe-sfbay-09.sun.com> (original mail from Ted.Kim@Sun.COM) for psarc-record@sun.com (ORCPT psarc-record@sun.com); Wed, 10 Oct 2007 10:21:31 -0700 (PDT) Received: from [129.153.3.74] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPP00E1QHJUE2D0@fe-sfbay-09.sun.com> for psarc-record@sun.com (ORCPT psarc-record@sun.com); Wed, 10 Oct 2007 10:21:30 -0700 (PDT) Date: Wed, 10 Oct 2007 10:19:46 -0700 From: "Ted H. Kim" Subject: Generic HCA Support for uDAPL [PSARC/2007/575 FastTrack timeout 10/10/2007] Sender: Ted.Kim@sun.com To: psarc-record@sun.com Cc: Bill Taylor , "Ted H. Kim" , Brendan Doyle Message-id: <470D09B2.1010208@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 User-Agent: Thunderbird 2.0.0.5 (X11/20070720) Status: RO Content-Length: 288 This case was approved at today's PSARC meeting (10/10/07). I am marking it "closed approved". -ted -- Ted H. Kim Sun Microsystems, Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX