From tedk@sac.sfbay.sun.com Wed Mar 10 15:52:37 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2ANqb4V012248 for ; Wed, 10 Mar 2010 15:52:37 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2ANqZnH028455; Wed, 10 Mar 2010 17:52:35 -0600 (CST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZ300J01BNMM500@nwk-avmta-2.sfbay.sun.com>; Wed, 10 Mar 2010 15:52:34 -0800 (PST) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZ3001X8BNMOJE0@nwk-avmta-2.sfbay.sun.com>; Wed, 10 Mar 2010 15:52:34 -0800 (PST) Received: from sac.sfbay.sun.com (sac.SFBay.Sun.COM [129.146.226.132]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2ANqXom009158; Wed, 10 Mar 2010 15:52:33 -0800 (PST) Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2ANqN4R012241; Wed, 10 Mar 2010 15:52:23 -0800 (PST) Received: (from tedk@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id o2ANqNm0012237; Wed, 10 Mar 2010 15:52:23 -0800 (PST) Date: Wed, 10 Mar 2010 15:52:23 -0800 (PST) From: Ted Kim Subject: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] To: PSARC-ext@sun.com Cc: cynthia.mcguire@sun.com, peter.memishian@sun.com Message-id: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 17064 Template Version: @(#)sac_nextcase 1.69 02/15/10 SMI This information is Copyright 2010 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: IPoIB Administration Enhancement 1.2. Name of Document Author/Supplier: Author: Sudhakar Dindukurti 1.3 Date of This Document: 10 March, 2010 4. Technical Description 4.1 Acronyms HCA : Host Channel Adaptor P_Key : Partition Key IPoIB : IP over InfiniBand GUID : Global Unique Identifier IBTF : InfiniBand Transport Framework IBCM : InfiniBand Communication Manager SDP : Sockets Direct Protocol IBD : current IPoIB Solaris driver name ULP : Upper level Protocol 4.2 Requirements/Motivation --------------------------- 4.2.1 Consistent IBD device node naming across nodes in a clusters (Amber Road Requirement (CR 6864899)) AmberRoad clustering software requires that datalink name for a specific partition on multiple nodes (identical h/w configuration) to be same. For example, if datalink name on node 1 for a P_key 0x8001 on HCA 1 and port 1 is ibnet0, then the clustering software expects that datalink name for P_Key 0x8001 on HCA 1 and port 1 is also ibnet0 on the node 2 in the cluster. This is very difficult to achieve today. The problem is that IPoIB device name is constructed automatically by IPoIB driver by appending the instance number to the driver name (ex ibd0, ibd1, etc). So, the clustering software does not have any control on the IPoIB device name space. Also, IBTF framework does not guarantee that the same names across multiple nodes. 4.2.2 Problem diagnosis issue IPoIB (PSARC 2001/289, PSARC 2009/593 and PSARC 2007/636) is implemented as the ibd(7D) driver in Solaris. When customers face problems in bringing up the IBD devices on Solaris, they are advised to take number of steps to gather information about the IPoIB link that results in usability issues. Today, users need to run cfgadm(1M) to obtain the port GUID information, do 'ls-l' on the /dev/ibd* nodes to determine the IB partition that an IBD device belongs to etc. There are also instances when it is necessary to obtain the P_key corresponding to the IBD device and HCA port that it is bound to. So, we want to improve the process of gathering IPoIB related configuration information. 4.2.3 Update IBD driver to use Brussels framework (CR 6883212) IPoIB tunables are managed today through /etc/system file or ibd.conf. For example, to modify the 'linkmode' of the IPoIB link, one needs to edit ibd.conf and reboot the system. We want to replace this interface so that user can manage these tunables using dladm(1M). 4.3 Proposal A micro/patch binding is asserted for this proposal. 4.3.0 IPoIB administration with dladm(1M) ---------------------------------------- This case proposes a new IPoIB administration mechanism for InfiniBand network datalinks using dladm(1M) command. Also, it proposes to add a consolidation private InfiniBand specific library API to libdladm(3LIB). The mechanism is very similar to the existing VLAN, VNIC, etc. administration dladm(1M) sub-commands. In the new model, two classes of IPoIB datalinks will exist: 1. Datalinks representing the physical IB ports, which will use the existing "phys" class that is used for Ethernet, WiFi, and other physical media. As with all "phys" class objects, the system will create these automatically. 2. Datalinks representing the administratively created partitions over "phys" IPoIB objects, which will use a new "part" class. Each IB partition datalink will be associated with a P_Key in a manner analogous to the way each Ethernet VLAN datalink is associated with a VLAN ID. Note that unlike other "phys" class objects, IB "phys" objects cannot send data (since IB requires a P_Key to send data) and thus cannot be plumbed. Example configuration used to explain the new IPoIB administration model -------------------------------------- | IB switch | -------------------------------------- | | | | | | Port1 | | Port2 Port1 | | Port2 -------------------------------------- | HCA1 HCA2 | | Node 1 | -------------------------------------- Port 2 of the HCA1 and Port 1 of the HCA2 are connected to IB switch and SM is running the switch. Each port is configured with two P_Key's (0xffff & 0x8001). 4.3.1 Physical datalinks ------------------------- One physical datalink will be created by default per port per HCA. These physical links serve as administrative & observability data points. These IB physical datalinks allow creating IB partitions over them similar to creating VNICs on Ethernet physical links or link aggregations. IB physical datalinks are not used for data transfers. So, the plumb and assigning a IB address are not supported on these links. In future, these physical datalinks can be used for 1) implementing the port level statistics 2) implementing port level snoop etc. Example 1. # dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE ibp0 InfiniBand up 8000 unknown ibp0 ibp1 InfiniBand down 8000 unknown ibp1 ibp2 InfiniBand down 2000 unknown ibp2 ibp3 InfiniBand up 2000 unknown ibp3 The state of the physical link is directly corresponds to state of the IB HCA port. As you might expect other generic sub-commands such as rename- link, show-link, delete-phys, etc. also work on IB datalinks. 4.3.2 IB partition Objects -------------------------- IB partition objects represent a new "part" class of datalink and these objects are managed using the new dladm(1M) sub-commands. All the new sub-command interfaces are similar to VLAN/VNIC dladm(1M) sub-commands. IB partition datalinks can be created on the top of IB physical links one per each P_Key on the port. These links are used for data transfers. The updated dladm(1M) man page describes the different commands available for managing the IB partition links. 4.3.2.1 create-part Creates a new IB partition Object with the specified datalink name. Example 2: Create a IB partition link for the P_Key 0x8001 on the top of ibp0 physical datalink # dladm create-part -l ibp0 -P 0x8001 p8001.ibp0 The above commands succeeds if the port is "up", P_Key is present on the port and IPoIB successfully completes its initialization. On subsequent reboots after successful initial creation, the partition will be available even if port is "down" or P_Key is absent, though the datalink state will be marked as "down". Example 3: Create an IB partition link for the P_Key 0x9000 on the top of ibp2 # dladm create-part -f -l ibp2 -P 0x9000 p9000.ibp2 The force option "-f" allows to create the IB partition even when the P_Key is not present or Port is down. The link state will be marked as down. The link state will be updated to "up" when P_Key is added to the port and port is activated. Example 4: Plumb and assign a IP address to IB partition p9000.ibp2 # ifconfig p9000.ibp2 plumb up # ifconfig -a p9000.ibp2: flags=1000843 mtu 2044 index 3 inet 1.1.1.1 netmask ff000000 broadcast 1.255.255.255 Example 5: Display the partition links using show-link command # dladm show-link LINK CLASS MTU STATE BRIDGE OVER p8001.ibp0 part 65520 unknown -- ibp0 p9000.ibp2 part 65520 down -- ibp2 4.3.2.2 delete-part Deletes a specified IB partition object Example 6: Delete the partition p8001.ibp0 # dladm delete-part p8001.ibp0 The above command deletes the partition. Example 7: Show the partition link information after deleting p8001.ibp0 # dldam show-part LINK PKEY OVER STATE FLAGS p9000.ibp2 9000 ibp2 down f--- 4.3.2.3 show-part Displays IB partition object information Example 8: Display the IB partition links information (below output is for the "part" operations in Example 2, 3 & 4) # dldam show-part LINK PKEY OVER STATE FLAGS p8001.ibp0 8001 ibp0 unknown ---- p9000.ibp2 9000 ibp2 down f--- The state of the IB partition link will be "unknown" after IB partition is created and before IB partition is plumbed. Once partition is plumbed the link state will be set to "up" when the link is ready to use. The state of the link will be set to "down" if 1) HCA port is down 2) P_Key is absent or 3) broadcast group is absent. 4.3.2.4 show-ib Displays IB specific information such as port#, port guid, etc. Example 9: Show IB specific information # dladm show-ib LINK HCAGUID PORTGUID PORT STATE PKEYS ibp0 3BA000100CD7C 3BA000100CD7D 1 down FFFF ibp1 3BA000100CD7C 3BA000100CD7E 2 down FFFF ibp3 5AD0000033634 5AD0000033636 2 up FFFF,8001 ibp2 5AD0000033634 5AD0000033635 1 up FFFF,8001 show-ib commands display only the physical links, port GUID, port# HCA GUID, and P_Key present on the port at the time of running the command. 4.3.3 IB partition object Administration Library The libdladm(3LIB) library is currently used to implement datalink administration for all the GLDv3 datalinks (VNIC, link aggregation, wireless, IP Tunnel, etc.). The library will be further enhanced to provide administrative interfaces of InfiniBand partition objects. All the new library extensions are similar to VLAN/VNIC libdladm extensions. IB specific functionality will be implemented by libdlib.c and its interface is provided via libdlib.h. dladm(1M) will use this library for managing IB partitions. The IB partition administration library provides a persistent repository which allows IB partition configuration to be stored across reboots. It uses the existing dladm(1M) /etc/dladm/datalink.conf repository to store the IB partition configuration. The list of InfiniBand specific extensions to libdladm library is given below. For more details of each API, see man pages in the PSARC materials directory. The man pages are only for PSARC review (not intended for public use). 4.3.3.1 dladm_part_create() Create a IB partition object 4.3.3.2 dladm_part_delete() Deletes a IB partition object 4.3.3.3 dladm_part_info() Returns IB partition Object attributes 4.3.3.4 dladm_ib_info() Returns IB specific attributes such as port number, port guid, and HCA GUID. 4.3.3.5 dladm_part_up() Brings up one or all the IB partition objects during every boot. 4.3.4 IBTF Extensions --------------------- Some of the IB ULP's such as SDP, IBCM, etc. walk the device tree and read "port-pkey", "hca-guid", and "port-guid" properties from the ibd(7D) device instance. With the new model, IB partitions are just objects (no longer device nodes). So, IB ULP's no longer can retrieve partition attributes from the device node. The following new IBTF extensions provide a mechanism to retrieve the partition attributes in the kernel. See ibt_get_part_attrs() man page in the PSARC case directory for more details about the following API. The man pages are only for PSARC review (not intended for public use). 4.3.4.1 ibt_get_part_attrs() Returns the attributes of a requested IB partition 4.3.4.2 ibt_get_all_part_attrs() Returns the attributes of all the active IB partitions 4.3.4.3 ibt_free_part_attrs() Frees the memory for partition attribute structure allocated by ibt_get_all_part_attrs() 4.3.5 Man page changes ---------------------- Updated man pages - published dladm(1M) datadm(1M) dat.conf(4) ibp(7D) (ibd(7d) name changed) New Man pages - internal only dladm_part_create(3dladm) dladm_part_delete(3dladm) dladm_part_info(3dladm) dladm_ib_info(3dladm) dladm_part_up(3dladm) ibt_get_part_attrs(9f) 4.3.6 Interface table --------------------- ----------------------------------------------------------------------- | Interface name | Commitment Level | ----------------------------------------------------------------------- | dladm(1M) extensions | ----------------------------------------------------------------------- | create-part | | | delete-part | committed | | show-ib | | | show-part | | ----------------------------------------------------------------------- | libdladm extensions (libdlib.h) | ----------------------------------------------------------------------- | dladm_part_create | | | dladm_part_delete | | | dladm_part_show | | | dladm_show_ib | ON Consolidation Private | | dladm_part_up | | | dladm_part_attr_t | | | dladm_ib_attr_t | | | DLADM_IBPART_FORCE_CREATE | | ----------------------------------------------------------------------- | InfiniBand Specific Link Properties | ----------------------------------------------------------------------- | linkmode | ON Consolidation Private | ----------------------------------------------------------------------- | libdlmgmt.h | ----------------------------------------------------------------------- | DATALINK_CLASS_IBPART | ON Consolidation Private | ----------------------------------------------------------------------- | libdladm.h | ----------------------------------------------------------------------- | DLADM_STATUS_INVALID_PORT_INSTANCE | | | DLADM_STATUS_PORT_IS_DOWN | | | DLADM_STATUS_PKEY_NOT_PRESENT | | | DLADM_STATUS_PARTITION_EXISTS | ON Consolidation Private | | DLADM_STATUS_INVALID_PKEY | | | DLADM_STATUS_NO_HW_RESOURCE | | | DLADM_STATUS_INVALID_PKEY_TBL_SIZE | | ----------------------------------------------------------------------- | IBTF extensions | ----------------------------------------------------------------------- | ibt_get_part_attrs() | | | ibt_get_all_part_attrs() | ON Consolidation private | | ibt_free_part_attrs() | | | ibt_part_attr_t | | ----------------------------------------------------------------------- | New status codes (ibt_status_t updates) | ----------------------------------------------------------------------- | IBT_NO_SUCH_OBJECT | ON Consolidation private | ----------------------------------------------------------------------- 4.3.7 References --------------------- PSARC 2001/289 IP over InfiniBand PSARC 2007/636 IPoIB Conversion to GLDv3 PSARC 2009/593 IPoIB Connected Mode 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From gdamore@sun.com Wed Mar 10 16:02:36 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2B02ZHm012984 for ; Wed, 10 Mar 2010 16:02:35 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2B02Z4J011473 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 10 Mar 2010 16:02:35 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZ300A0BC4BNH00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Mar 2010 17:02:35 -0700 (MST) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZ30085AC488LC0@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Mar 2010 17:02:33 -0700 (MST) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o2B02R8w017025 for ; Wed, 10 Mar 2010 16:02:32 -0800 (PST) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KZ300100C12MP00@fe-sfbay-10.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Mar 2010 16:02:30 -0800 (PST) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KZ3003B9C3GAIC0@fe-sfbay-10.sun.com>; Wed, 10 Mar 2010 16:02:04 -0800 (PST) Date: Wed, 10 Mar 2010 16:02:04 -0800 From: "Garrett D'Amore" Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> Sender: Garrett.Damore@sun.com To: Ted Kim Cc: PSARC-ext@sun.com, Cynthia.McGuire@sun.com, Peter.Memishian@sun.com Message-id: <4B9832FC.7040201@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.7) Gecko/20100131 Lightning/1.0b1 Thunderbird/3.0.1 Status: RO Content-Length: 1971 On 03/10/10 03:52 PM, Ted Kim wrote: > Template Version: @(#)sac_nextcase 1.69 02/15/10 SMI > This information is Copyright 2010 Sun Microsystems > 1. Introduction > 1.1. Project/Component Working Name: > IPoIB Administration Enhancement > 1.2. Name of Document Author/Supplier: > Author: Sudhakar Dindukurti > 1.3 Date of This Document: > 10 March, 2010 > 4. Technical Description > > 4.1 Acronyms > > HCA : Host Channel Adaptor > P_Key : Partition Key > IPoIB : IP over InfiniBand > GUID : Global Unique Identifier > IBTF : InfiniBand Transport Framework > IBCM : InfiniBand Communication Manager > SDP : Sockets Direct Protocol > IBD : current IPoIB Solaris driver name > ULP : Upper level Protocol > > 4.2 Requirements/Motivation > --------------------------- > > 4.2.1 Consistent IBD device node naming across nodes in a clusters > > (Amber Road Requirement (CR 6864899)) > > AmberRoad clustering software requires that datalink name for a specific > partition on multiple nodes (identical h/w configuration) to be same. For > example, if datalink name on node 1 for a P_key 0x8001 on HCA 1 and port 1 > is ibnet0, then the clustering software expects that datalink name for > P_Key 0x8001 on HCA 1 and port 1 is also ibnet0 on the node 2 in the > cluster. This is very difficult to achieve today. The problem is that > IPoIB device name is constructed automatically by IPoIB driver by > appending the instance number to the driver name (ex ibd0, ibd1, etc). So, > the clustering software does not have any control on the IPoIB device > name space. Also, IBTF framework does not guarantee that the same names > across multiple nodes. > It seems like this is a bug in Amber Road, best dealt with there. Also, how does this respond in the presence of Clearview Vanity Naming? - Garrett From peter.memishian@sun.com Wed Mar 10 16:12:19 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2B0CJqb013034 for ; Wed, 10 Mar 2010 16:12:19 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2B0CHPm016325; Wed, 10 Mar 2010 16:12:18 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZ300G0PCKHQ000@nwk-avmta-1.sfbay.Sun.COM>; Wed, 10 Mar 2010 16:12:18 -0800 (PST) Received: from triplex.local ([129.146.108.208]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZ300J98CKGOR50@nwk-avmta-1.sfbay.Sun.COM>; Wed, 10 Mar 2010 16:12:16 -0800 (PST) Received: from triplex.local (triplex [127.0.0.1]) by triplex.local (8.14.3+Sun/8.14.3) with ESMTP id o2B0CGE9469949; Wed, 10 Mar 2010 16:12:16 -0800 (PST) Received: (from meem@localhost) by triplex.local (8.14.3+Sun/8.14.3/Submit) id o2B0CGdp469938; Wed, 10 Mar 2010 16:12:16 -0800 (PST) Date: Wed, 10 Mar 2010 16:12:16 -0800 From: Peter Memishian Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <4B9832FC.7040201@sun.com> To: "Garrett D'Amore" Cc: Ted Kim , PSARC-ext@sun.com, Cynthia.McGuire@sun.com, peter.memishian@sun.com Reply-to: peter.memishian@sun.com Message-id: <19352.13664.52131.799827@gargle.gargle.HOWL> MIME-version: 1.0 X-Mailer: VM 7.19 under 21.4 (patch 21) "Educational Television" XEmacs Lucid Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> <4B9832FC.7040201@sun.com> X-Authentication-warning: triplex.local: meem set sender to peter.memishian@sun.com using -f Status: RO Content-Length: 188 > It seems like this is a bug in Amber Road, best dealt with there. False. > Also, how does this respond in the presence of Clearview Vanity Naming? Completely orthogonal. -- meem From gdamore@sun.com Wed Mar 10 16:34:01 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2B0Y1Tk013620 for ; Wed, 10 Mar 2010 16:34:01 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2B0Y185026109 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 10 Mar 2010 16:34:01 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZ300E01DKP1100@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Mar 2010 17:34:01 -0700 (MST) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZ3008ISDKO8RD0@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Mar 2010 17:34:00 -0700 (MST) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o2B0Xxc0015212 for ; Wed, 10 Mar 2010 16:33:59 -0800 (PST) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KZ300200DD6PQ00@fe-sfbay-10.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Mar 2010 16:33:59 -0800 (PST) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KZ3001ASDKMQF10@fe-sfbay-10.sun.com>; Wed, 10 Mar 2010 16:33:59 -0800 (PST) Date: Wed, 10 Mar 2010 16:33:58 -0800 From: "Garrett D'Amore" Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <19352.13664.52131.799827@gargle.gargle.HOWL> Sender: Garrett.Damore@sun.com To: Peter.Memishian@sun.com Cc: Ted Kim , PSARC-ext@sun.com, Cynthia.McGuire@sun.com Message-id: <4B983A76.4000606@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> <4B9832FC.7040201@sun.com> <19352.13664.52131.799827@gargle.gargle.HOWL> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.7) Gecko/20100131 Lightning/1.0b1 Thunderbird/3.0.1 Status: RO Content-Length: 668 On 03/10/10 04:12 PM, Peter Memishian wrote: > > It seems like this is a bug in Amber Road, best dealt with there. > > False. > Can you please elaborate. Why is it OK for Amber road to make assumptions about interface naming? > > Also, how does this respond in the presence of Clearview Vanity Naming? > > Completely orthogonal. > > I'm not sure I agree, but perhaps I'm just not understanding. To be clear, it seems like a particular product is dependent on the names of interfaces. Yet, customers ought to be able to rename those interfaces using vanity names. I'd like to understand how this is supposed to work together. - Garrett From peter.memishian@sun.com Wed Mar 10 16:51:48 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2B0pm8J014030 for ; Wed, 10 Mar 2010 16:51:48 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2B0pjo4003986; Wed, 10 Mar 2010 16:51:47 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZ300F11EEAZ900@brm-avmta-1.central.sun.com>; Wed, 10 Mar 2010 17:51:46 -0700 (MST) Received: from triplex.local ([129.146.108.208]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZ3008BWEEA8SF0@brm-avmta-1.central.sun.com>; Wed, 10 Mar 2010 17:51:46 -0700 (MST) Received: from triplex.local (triplex [127.0.0.1]) by triplex.local (8.14.3+Sun/8.14.3) with ESMTP id o2B0pjvd879154; Wed, 10 Mar 2010 16:51:45 -0800 (PST) Received: (from meem@localhost) by triplex.local (8.14.3+Sun/8.14.3/Submit) id o2B0pjfI879142; Wed, 10 Mar 2010 16:51:45 -0800 (PST) Date: Wed, 10 Mar 2010 16:51:45 -0800 From: Peter Memishian Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <4B983A76.4000606@sun.com> To: "Garrett D'Amore" Cc: peter.memishian@sun.com, Ted Kim , PSARC-ext@sun.com, Cynthia.McGuire@sun.com Reply-to: peter.memishian@sun.com Message-id: <19352.16033.752294.731929@gargle.gargle.HOWL> MIME-version: 1.0 X-Mailer: VM 7.19 under 21.4 (patch 21) "Educational Television" XEmacs Lucid Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> <4B9832FC.7040201@sun.com> <19352.13664.52131.799827@gargle.gargle.HOWL> <4B983A76.4000606@sun.com> X-Authentication-warning: triplex.local: meem set sender to peter.memishian@sun.com using -f Status: RO Content-Length: 1514 > > > It seems like this is a bug in Amber Road, best dealt with there. > > > > False. > > Can you please elaborate. Why is it OK for Amber road to make > assumptions about interface naming? Given that this is an open list, I cannot go into the details of the Fishworks clustering architecture. However, broadly speaking, the architecture requires that all clustered resources are identical across the heads. This is impossible in the current IB architecture because the partition datalinks are created by the driver itself based on events that are external to the box, and thus there is no way to guarantee that ibd0 on each head will refer to the same pkey. However, while Fishworks is the catalyst, as covered in the case materials, the work stands on its own and brings IB into line with the established administrative model for VLANs/VNICs with Ethernet. That is, IB partitions are modeled identically to Ethernet VLANs. This is the right answer for a large number of reasons (observability, administrative consistency, ability to create partitions when the IB subnet manager is down ...) that have nothing to do with Fishworks. > To be clear, it seems like a particular product is dependent on the > names of interfaces. Yet, customers ought to be able to rename those > interfaces using vanity names. When they create the partition, they choose the name. This works identically to VLANs, VNICs, aggregations, and so forth. They can then rename the link if they want. -- meem From gdamore@sun.com Wed Mar 10 17:23:36 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2B1Nat6014796 for ; Wed, 10 Mar 2010 17:23:36 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o2B1NYLp030749 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 10 Mar 2010 18:23:35 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZ300209FVBMO00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Mar 2010 17:23:35 -0800 (PST) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZ300N6HFVB2930@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Mar 2010 17:23:35 -0800 (PST) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o2B1NZmi025288 for ; Wed, 10 Mar 2010 17:23:35 -0800 (PST) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KZ300G00FOOQR00@fe-sfbay-10.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Mar 2010 17:23:35 -0800 (PST) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KZ3001AYFVAQFB0@fe-sfbay-10.sun.com>; Wed, 10 Mar 2010 17:23:34 -0800 (PST) Date: Wed, 10 Mar 2010 17:23:34 -0800 From: "Garrett D'Amore" Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <19352.16033.752294.731929@gargle.gargle.HOWL> Sender: Garrett.Damore@sun.com To: Peter.Memishian@sun.com Cc: Ted Kim , PSARC-ext@sun.com, Cynthia.McGuire@sun.com Message-id: <4B984616.6050005@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> <4B9832FC.7040201@sun.com> <19352.13664.52131.799827@gargle.gargle.HOWL> <4B983A76.4000606@sun.com> <19352.16033.752294.731929@gargle.gargle.HOWL> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.7) Gecko/20100131 Lightning/1.0b1 Thunderbird/3.0.1 Status: RO Content-Length: 1722 On 03/10/10 04:51 PM, Peter Memishian wrote: > > > > It seems like this is a bug in Amber Road, best dealt with there. > > > > > > False. > > > > Can you please elaborate. Why is it OK for Amber road to make > > assumptions about interface naming? > > Given that this is an open list, I cannot go into the details of the > Fishworks clustering architecture. However, broadly speaking, the > architecture requires that all clustered resources are identical across > the heads. This is impossible in the current IB architecture because the > partition datalinks are created by the driver itself based on events that > are external to the box, and thus there is no way to guarantee that ibd0 > on each head will refer to the same pkey. > > However, while Fishworks is the catalyst, as covered in the case > materials, the work stands on its own and brings IB into line with the > established administrative model for VLANs/VNICs with Ethernet. That is, > IB partitions are modeled identically to Ethernet VLANs. This is the > right answer for a large number of reasons (observability, administrative > consistency, ability to create partitions when the IB subnet manager is > down ...) that have nothing to do with Fishworks. > > > To be clear, it seems like a particular product is dependent on the > > names of interfaces. Yet, customers ought to be able to rename those > > interfaces using vanity names. > > When they create the partition, they choose the name. This works > identically to VLANs, VNICs, aggregations, and so forth. They can > then rename the link if they want. > > Okay, that last bit was the important part. Thanks for the clarifications. -- Garrett From gdamore@sun.com Fri Mar 12 08:40:58 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2CGewsO000034 for ; Fri, 12 Mar 2010 08:40:58 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2CGewrP023842 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 12 Mar 2010 08:40:58 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZ600G09H0AEQ00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Fri, 12 Mar 2010 08:40:58 -0800 (PST) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZ6004RRH0AO330@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Fri, 12 Mar 2010 08:40:58 -0800 (PST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o2CGevSu029148 for ; Fri, 12 Mar 2010 08:40:57 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KZ600C00GSCKM00@fe-sfbay-09.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Fri, 12 Mar 2010 08:40:57 -0800 (PST) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KZ6009DRH09QHD0@fe-sfbay-09.sun.com>; Fri, 12 Mar 2010 08:40:57 -0800 (PST) Date: Fri, 12 Mar 2010 08:40:57 -0800 From: "Garrett D'Amore" Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> Sender: Garrett.Damore@sun.com To: Ted Kim Cc: PSARC-ext@sun.com, Cynthia.McGuire@sun.com, Peter.Memishian@sun.com Message-id: <4B9A6E99.7010002@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.7) Gecko/20100131 Lightning/1.0b1 Thunderbird/3.0.1 Status: RO Content-Length: 19118 Okay, here are some questions; some of these stem from my lack of IPoIB knowledge, so I hope you'll pardon me if I ask things that seem obvious or stupid. 1) Will the new "phys" objects being created here be "snoop"-able? (I understand that they can't transmit data, but can they operate in some sort of promiscuous receive mode?) (Oh wait, it seems that this might be answered in part by 4.3.1. It looks like the answer is "not yet, but in the future". Correct?) 2) From my experience with "hermon", hermon seems mostly to treat both ports as a single registered entity to the IB framework. I'm presuming that this won't preclude IPoIB from being able to identify which port is which? Are there any special considerations here for automatic path migration? 3) Are there any constraints on the format of the name used for "part-link" in dladm? The examples seem to show a specific format, but can customers choose any name they want? 4) The way this is designed seems to depend on a notion of a physical port. Does IPoIB have anything that is morally equivalent to ethernet aggregations (trunking)? I'm thinking also about link redundancy as well as bandwidth multiplication. - Garrett On 03/10/10 03:52 PM, Ted Kim wrote: > Template Version: @(#)sac_nextcase 1.69 02/15/10 SMI > This information is Copyright 2010 Sun Microsystems > 1. Introduction > 1.1. Project/Component Working Name: > IPoIB Administration Enhancement > 1.2. Name of Document Author/Supplier: > Author: Sudhakar Dindukurti > 1.3 Date of This Document: > 10 March, 2010 > 4. Technical Description > > 4.1 Acronyms > > HCA : Host Channel Adaptor > P_Key : Partition Key > IPoIB : IP over InfiniBand > GUID : Global Unique Identifier > IBTF : InfiniBand Transport Framework > IBCM : InfiniBand Communication Manager > SDP : Sockets Direct Protocol > IBD : current IPoIB Solaris driver name > ULP : Upper level Protocol > > 4.2 Requirements/Motivation > --------------------------- > > 4.2.1 Consistent IBD device node naming across nodes in a clusters > > (Amber Road Requirement (CR 6864899)) > > AmberRoad clustering software requires that datalink name for a specific > partition on multiple nodes (identical h/w configuration) to be same. For > example, if datalink name on node 1 for a P_key 0x8001 on HCA 1 and port 1 > is ibnet0, then the clustering software expects that datalink name for > P_Key 0x8001 on HCA 1 and port 1 is also ibnet0 on the node 2 in the > cluster. This is very difficult to achieve today. The problem is that > IPoIB device name is constructed automatically by IPoIB driver by > appending the instance number to the driver name (ex ibd0, ibd1, etc). So, > the clustering software does not have any control on the IPoIB device > name space. Also, IBTF framework does not guarantee that the same names > across multiple nodes. > > 4.2.2 Problem diagnosis issue > > IPoIB (PSARC 2001/289, PSARC 2009/593 and PSARC 2007/636) is implemented > as the ibd(7D) driver in Solaris. When customers face problems in > bringing up the IBD devices on Solaris, they are advised to take number > of steps to gather information about the IPoIB link that results in > usability issues. Today, users need to run cfgadm(1M) to obtain the port > GUID information, do 'ls-l' on the /dev/ibd* nodes to determine the IB > partition that an IBD device belongs to etc. There are also instances > when it is necessary to obtain the P_key corresponding to the IBD device > and HCA port that it is bound to. So, we want to improve the process of > gathering IPoIB related configuration information. > > 4.2.3 Update IBD driver to use Brussels framework (CR 6883212) > > IPoIB tunables are managed today through /etc/system file or ibd.conf. > For example, to modify the 'linkmode' of the IPoIB link, one needs to edit > ibd.conf and reboot the system. We want to replace this interface so that > user can manage these tunables using dladm(1M). > > 4.3 Proposal > > A micro/patch binding is asserted for this proposal. > > 4.3.0 IPoIB administration with dladm(1M) > ---------------------------------------- > > This case proposes a new IPoIB administration mechanism for InfiniBand > network datalinks using dladm(1M) command. Also, it proposes to add > a consolidation private InfiniBand specific library API to libdladm(3LIB). > The mechanism is very similar to the existing VLAN, VNIC, etc. > administration dladm(1M) sub-commands. > > In the new model, two classes of IPoIB datalinks will exist: > > 1. Datalinks representing the physical IB ports, which will use the > existing "phys" class that is used for Ethernet, WiFi, and other > physical media. As with all "phys" class objects, the system > will create these automatically. > > 2. Datalinks representing the administratively created partitions > over "phys" IPoIB objects, which will use a new "part" class. > Each IB partition datalink will be associated with a P_Key in a > manner analogous to the way each Ethernet VLAN datalink is > associated with a VLAN ID. > > Note that unlike other "phys" class objects, IB "phys" objects cannot > send data (since IB requires a P_Key to send data) and thus cannot be > plumbed. > > Example configuration used to explain the new IPoIB administration model > > -------------------------------------- > | IB switch | > -------------------------------------- > | | > | | > | | > Port1 | | Port2 Port1 | | Port2 > -------------------------------------- > | HCA1 HCA2 | > | Node 1 | > -------------------------------------- > > Port 2 of the HCA1 and Port 1 of the HCA2 are connected to IB switch and > SM is running the switch. Each port is configured with two P_Key's (0xffff > & 0x8001). > > > 4.3.1 Physical datalinks > ------------------------- > > One physical datalink will be created by default per port per HCA. These > physical links serve as administrative& observability data points. These > IB physical datalinks allow creating IB partitions over them similar > to creating VNICs on Ethernet physical links or link aggregations. IB > physical datalinks are not used for data transfers. So, the plumb and > assigning a IB address are not supported on these links. In future, these > physical datalinks can be used for 1) implementing the port level > statistics 2) implementing port level snoop etc. > > Example 1. # dladm show-phys > LINK MEDIA STATE SPEED DUPLEX DEVICE > ibp0 InfiniBand up 8000 unknown ibp0 > ibp1 InfiniBand down 8000 unknown ibp1 > ibp2 InfiniBand down 2000 unknown ibp2 > ibp3 InfiniBand up 2000 unknown ibp3 > > The state of the physical link is directly corresponds to state of the IB > HCA port. As you might expect other generic sub-commands such as rename- > link, show-link, delete-phys, etc. also work on IB datalinks. > > 4.3.2 IB partition Objects > -------------------------- > > IB partition objects represent a new "part" class of datalink and these > objects are managed using the new dladm(1M) sub-commands. All the new > sub-command interfaces are similar to VLAN/VNIC dladm(1M) sub-commands. IB > partition datalinks can be created on the top of IB physical links one > per each P_Key on the port. These links are used for data transfers. > The updated dladm(1M) man page describes the different commands available > for managing the IB partition links. > > 4.3.2.1 create-part > > Creates a new IB partition Object with the specified datalink name. > > Example 2: Create a IB partition link for the P_Key 0x8001 on the > top of ibp0 physical datalink > > # dladm create-part -l ibp0 -P 0x8001 p8001.ibp0 > > The above commands succeeds if the port is "up", P_Key is present on > the port and IPoIB successfully completes its initialization. On > subsequent reboots after successful initial creation, the partition > will be available even if port is "down" or P_Key is absent, though > the datalink state will be marked as "down". > > Example 3: Create an IB partition link for the P_Key 0x9000 on the > top of ibp2 > > # dladm create-part -f -l ibp2 -P 0x9000 p9000.ibp2 > > The force option "-f" allows to create the IB partition even when > the P_Key is not present or Port is down. The link state will be > marked as down. The link state will be updated to "up" when P_Key > is added to the port and port is activated. > > Example 4: Plumb and assign a IP address to IB partition p9000.ibp2 > > # ifconfig p9000.ibp2 plumb up > > # ifconfig -a > > p9000.ibp2: flags=1000843 MULTICAST,IPv4> mtu 2044 index 3 > inet 1.1.1.1 netmask ff000000 broadcast 1.255.255.255 > > Example 5: Display the partition links using show-link command > > # dladm show-link > LINK CLASS MTU STATE BRIDGE OVER > p8001.ibp0 part 65520 unknown -- ibp0 > p9000.ibp2 part 65520 down -- ibp2 > > 4.3.2.2 delete-part > > Deletes a specified IB partition object > > Example 6: Delete the partition p8001.ibp0 > > # dladm delete-part p8001.ibp0 > > The above command deletes the partition. > > Example 7: Show the partition link information after > deleting p8001.ibp0 > > # dldam show-part > > LINK PKEY OVER STATE FLAGS > p9000.ibp2 9000 ibp2 down f--- > > 4.3.2.3 show-part > > Displays IB partition object information > > Example 8: Display the IB partition links information (below output > is for the "part" operations in Example 2, 3& 4) > > # dldam show-part > > LINK PKEY OVER STATE FLAGS > p8001.ibp0 8001 ibp0 unknown ---- > p9000.ibp2 9000 ibp2 down f--- > > The state of the IB partition link will be "unknown" after IB partition > is created and before IB partition is plumbed. Once partition is plumbed > the link state will be set to "up" when the link is ready to use. The > state of the link will be set to "down" if 1) HCA port is down 2) P_Key > is absent or 3) broadcast group is absent. > > 4.3.2.4 show-ib > > Displays IB specific information such as port#, port guid, etc. > > Example 9: Show IB specific information > > # dladm show-ib > LINK HCAGUID PORTGUID PORT STATE PKEYS > ibp0 3BA000100CD7C 3BA000100CD7D 1 down FFFF > ibp1 3BA000100CD7C 3BA000100CD7E 2 down FFFF > ibp3 5AD0000033634 5AD0000033636 2 up FFFF,8001 > ibp2 5AD0000033634 5AD0000033635 1 up FFFF,8001 > > show-ib commands display only the physical links, port GUID, port# > HCA GUID, and P_Key present on the port at the time of running the > command. > > 4.3.3 IB partition object Administration Library > > The libdladm(3LIB) library is currently used to implement datalink > administration for all the GLDv3 datalinks (VNIC, link aggregation, > wireless, IP Tunnel, etc.). The library will be further enhanced to provide > administrative interfaces of InfiniBand partition objects. All the new > library extensions are similar to VLAN/VNIC libdladm extensions. IB > specific functionality will be implemented by libdlib.c and its interface > is provided via libdlib.h. dladm(1M) will use this library for managing > IB partitions. The IB partition administration library provides a > persistent repository which allows IB partition configuration to be stored > across reboots. It uses the existing dladm(1M) /etc/dladm/datalink.conf > repository to store the IB partition configuration. > > The list of InfiniBand specific extensions to libdladm library is given below. > For more details of each API, see man pages in the PSARC materials > directory. The man pages are only for PSARC review (not intended for public > use). > > 4.3.3.1 dladm_part_create() > Create a IB partition object > > 4.3.3.2 dladm_part_delete() > Deletes a IB partition object > > 4.3.3.3 dladm_part_info() > Returns IB partition Object attributes > > 4.3.3.4 dladm_ib_info() > Returns IB specific attributes such as port number, port guid, > and HCA GUID. > > 4.3.3.5 dladm_part_up() > Brings up one or all the IB partition objects during every boot. > > 4.3.4 IBTF Extensions > --------------------- > > Some of the IB ULP's such as SDP, IBCM, etc. walk the device tree and read > "port-pkey", "hca-guid", and "port-guid" properties from the ibd(7D) device > instance. With the new model, IB partitions are just objects (no longer > device nodes). So, IB ULP's no longer can retrieve partition attributes > from the device node. The following new IBTF extensions provide a > mechanism to retrieve the partition attributes in the kernel. See > ibt_get_part_attrs() man page in the PSARC case directory for more details > about the following API. The man pages are only for PSARC review (not > intended for public use). > > 4.3.4.1 ibt_get_part_attrs() > Returns the attributes of a requested IB partition > > 4.3.4.2 ibt_get_all_part_attrs() > Returns the attributes of all the active IB partitions > > 4.3.4.3 ibt_free_part_attrs() > Frees the memory for partition attribute structure allocated by > ibt_get_all_part_attrs() > > 4.3.5 Man page changes > ---------------------- > > Updated man pages - published > dladm(1M) > datadm(1M) > dat.conf(4) > ibp(7D) (ibd(7d) name changed) > > New Man pages - internal only > dladm_part_create(3dladm) > dladm_part_delete(3dladm) > dladm_part_info(3dladm) > dladm_ib_info(3dladm) > dladm_part_up(3dladm) > > ibt_get_part_attrs(9f) > > 4.3.6 Interface table > --------------------- > > ----------------------------------------------------------------------- > | Interface name | Commitment Level | > ----------------------------------------------------------------------- > | dladm(1M) extensions | > ----------------------------------------------------------------------- > | create-part | | > | delete-part | committed | > | show-ib | | > | show-part | | > ----------------------------------------------------------------------- > | libdladm extensions (libdlib.h) | > ----------------------------------------------------------------------- > | dladm_part_create | | > | dladm_part_delete | | > | dladm_part_show | | > | dladm_show_ib | ON Consolidation Private | > | dladm_part_up | | > | dladm_part_attr_t | | > | dladm_ib_attr_t | | > | DLADM_IBPART_FORCE_CREATE | | > ----------------------------------------------------------------------- > | InfiniBand Specific Link Properties | > ----------------------------------------------------------------------- > | linkmode | ON Consolidation Private | > ----------------------------------------------------------------------- > | libdlmgmt.h | > ----------------------------------------------------------------------- > | DATALINK_CLASS_IBPART | ON Consolidation Private | > ----------------------------------------------------------------------- > | libdladm.h | > ----------------------------------------------------------------------- > | DLADM_STATUS_INVALID_PORT_INSTANCE | | > | DLADM_STATUS_PORT_IS_DOWN | | > | DLADM_STATUS_PKEY_NOT_PRESENT | | > | DLADM_STATUS_PARTITION_EXISTS | ON Consolidation Private | > | DLADM_STATUS_INVALID_PKEY | | > | DLADM_STATUS_NO_HW_RESOURCE | | > | DLADM_STATUS_INVALID_PKEY_TBL_SIZE | | > ----------------------------------------------------------------------- > | IBTF extensions | > ----------------------------------------------------------------------- > | ibt_get_part_attrs() | | > | ibt_get_all_part_attrs() | ON Consolidation private | > | ibt_free_part_attrs() | | > | ibt_part_attr_t | | > ----------------------------------------------------------------------- > | New status codes (ibt_status_t updates) | > ----------------------------------------------------------------------- > | IBT_NO_SUCH_OBJECT | ON Consolidation private | > ----------------------------------------------------------------------- > > 4.3.7 References > --------------------- > > PSARC 2001/289 IP over InfiniBand > PSARC 2007/636 IPoIB Conversion to GLDv3 > PSARC 2009/593 IPoIB Connected Mode > > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > ON > 6.5. ARC review type: FastTrack > 6.6. ARC Exposure: open > > From Sudhakar.Dindukurti@Sun.COM Fri Mar 12 09:45:53 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2CHjrqQ001874 for ; Fri, 12 Mar 2010 09:45:53 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2CHjp69005708; Fri, 12 Mar 2010 09:45:51 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZ600M07K0FJR00@nwk-avmta-2.sfbay.sun.com>; Fri, 12 Mar 2010 09:45:51 -0800 (PST) Received: from jurassic.Eng.Sun.COM ([129.146.17.63]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZ600HBCK0F55A0@nwk-avmta-2.sfbay.sun.com>; Fri, 12 Mar 2010 09:45:51 -0800 (PST) Received: from [129.146.226.78] (partha.SFBay.Sun.COM [129.146.226.78]) by jurassic.Eng.Sun.COM (8.14.4+Sun/8.14.4) with ESMTP id o2CHjoW5345034; Fri, 12 Mar 2010 09:45:50 -0800 (PST) Date: Fri, 12 Mar 2010 09:45:35 -0800 From: Sudhakar Dindukurti Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <4B9A6E99.7010002@sun.com> To: "Garrett D'Amore" Cc: Ted Kim , PSARC-ext@Sun.COM, Cynthia.McGuire@Sun.COM, Peter.Memishian@Sun.COM, Sudhakar Dindukurti Message-id: <4B9A7DBF.9020708@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> <4B9A6E99.7010002@sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090929) Status: RO Content-Length: 21148 Garrett, On 03/12/10 08:40, Garrett D'Amore wrote: > Okay, here are some questions; some of these stem from my lack of > IPoIB knowledge, so I hope you'll pardon me if I ask things that seem > obvious or stupid. > > 1) Will the new "phys" objects being created here be "snoop"-able? (I > understand that they can't transmit data, but can they operate in some > sort of promiscuous receive mode?) (Oh wait, it seems that this might > be answered in part by 4.3.1. It looks like the answer is "not yet, > but in the future". Correct?) > snoop support on the "phys" link is not supported yet but can be done in the future. > 2) From my experience with "hermon", hermon seems mostly to treat both > ports as a single registered entity to the IB framework. I'm > presuming that this won't preclude IPoIB from being able to identify > which port is which? Yes. dladm show-ib displays the port information associated with the IB phys class datalinks. > Are there any special considerations here for automatic path migration? IPoIB instances can only be created one per P_Key per port per IB HCA. Automatic path migration (APM) feature is supported from one port to another port on the same HCA. So, APM is not applicable for IPoIB. This case does not change any of this behavior from what exists today. > > 3) Are there any constraints on the format of the name used for > "part-link" in dladm? The examples seem to show a specific format, > but can customers choose any name they want? Yes. The customer can choose any name they want as long as the name format adheres to the existing "datalink" name. > > 4) The way this is designed seems to depend on a notion of a physical > port. Does IPoIB have anything that is morally equivalent to ethernet > aggregations (trunking)? I'm thinking also about link redundancy as > well as bandwidth multiplication. > No. This case does not add/change the existing IPoIB behavior on these features. regards, Sudhakar > - Garrett > > On 03/10/10 03:52 PM, Ted Kim wrote: >> Template Version: @(#)sac_nextcase 1.69 02/15/10 SMI >> This information is Copyright 2010 Sun Microsystems >> 1. Introduction >> 1.1. Project/Component Working Name: >> IPoIB Administration Enhancement >> 1.2. Name of Document Author/Supplier: >> Author: Sudhakar Dindukurti >> 1.3 Date of This Document: >> 10 March, 2010 >> 4. Technical Description >> >> 4.1 Acronyms >> >> HCA : Host Channel Adaptor >> P_Key : Partition Key >> IPoIB : IP over InfiniBand >> GUID : Global Unique Identifier >> IBTF : InfiniBand Transport Framework >> IBCM : InfiniBand Communication Manager >> SDP : Sockets Direct Protocol >> IBD : current IPoIB Solaris driver name >> ULP : Upper level Protocol >> >> 4.2 Requirements/Motivation >> --------------------------- >> >> 4.2.1 Consistent IBD device node naming across nodes in a clusters >> >> (Amber Road Requirement (CR 6864899)) >> >> AmberRoad clustering software requires that datalink name for a >> specific >> partition on multiple nodes (identical h/w configuration) to be same. >> For >> example, if datalink name on node 1 for a P_key 0x8001 on HCA 1 and >> port 1 >> is ibnet0, then the clustering software expects that datalink name >> for >> P_Key 0x8001 on HCA 1 and port 1 is also ibnet0 on the node 2 in >> the >> cluster. This is very difficult to achieve today. The problem is >> that >> IPoIB device name is constructed automatically by IPoIB >> driver by >> appending the instance number to the driver name (ex ibd0, ibd1, >> etc). So, >> the clustering software does not have any control on the IPoIB >> device >> name space. Also, IBTF framework does not guarantee that the same >> names >> across multiple nodes. >> >> 4.2.2 Problem diagnosis issue >> >> IPoIB (PSARC 2001/289, PSARC 2009/593 and PSARC 2007/636) is >> implemented >> as the ibd(7D) driver in Solaris. When customers face >> problems in >> bringing up the IBD devices on Solaris, they are advised to take >> number >> of steps to gather information about the IPoIB link that >> results in >> usability issues. Today, users need to run cfgadm(1M) to obtain the >> port >> GUID information, do 'ls-l' on the /dev/ibd* nodes to determine >> the IB >> partition that an IBD device belongs to etc. There are also >> instances >> when it is necessary to obtain the P_key corresponding to the IBD >> device >> and HCA port that it is bound to. So, we want to improve the >> process of >> gathering IPoIB related configuration information. >> >> 4.2.3 Update IBD driver to use Brussels framework (CR 6883212) >> >> IPoIB tunables are managed today through /etc/system file or >> ibd.conf. >> For example, to modify the 'linkmode' of the IPoIB link, one needs to >> edit >> ibd.conf and reboot the system. We want to replace this interface so >> that >> user can manage these tunables using dladm(1M). >> >> 4.3 Proposal >> >> A micro/patch binding is asserted for this proposal. >> >> 4.3.0 IPoIB administration with dladm(1M) >> ---------------------------------------- >> >> This case proposes a new IPoIB administration mechanism for >> InfiniBand >> network datalinks using dladm(1M) command. Also, it proposes to >> add >> a consolidation private InfiniBand specific library API to >> libdladm(3LIB). >> The mechanism is very similar to the existing VLAN, VNIC, >> etc. >> administration dladm(1M) sub-commands. >> >> In the new model, two classes of IPoIB datalinks will exist: >> >> 1. Datalinks representing the physical IB ports, which will >> use the >> existing "phys" class that is used for Ethernet, WiFi, and >> other >> physical media. As with all "phys" class objects, the >> system >> will create these automatically. >> >> 2. Datalinks representing the administratively created >> partitions >> over "phys" IPoIB objects, which will use a new "part" >> class. >> Each IB partition datalink will be associated with a P_Key >> in a >> manner analogous to the way each Ethernet VLAN >> datalink is >> associated with a VLAN ID. >> >> Note that unlike other "phys" class objects, IB "phys" objects >> cannot >> send data (since IB requires a P_Key to send data) and thus >> cannot be >> plumbed. >> >> Example configuration used to explain the new IPoIB administration >> model >> >> -------------------------------------- >> | IB switch | >> -------------------------------------- >> | | >> | | >> | | >> Port1 | | Port2 Port1 | | Port2 >> -------------------------------------- >> | HCA1 HCA2 | >> | Node 1 | >> -------------------------------------- >> >> Port 2 of the HCA1 and Port 1 of the HCA2 are connected to IB switch >> and >> SM is running the switch. Each port is configured with two P_Key's >> (0xffff >> & 0x8001). >> >> >> 4.3.1 Physical datalinks >> ------------------------- >> >> One physical datalink will be created by default per port per HCA. >> These >> physical links serve as administrative& observability data points. >> These >> IB physical datalinks allow creating IB partitions over them >> similar >> to creating VNICs on Ethernet physical links or link >> aggregations. IB >> physical datalinks are not used for data transfers. So, the plumb >> and >> assigning a IB address are not supported on these links. In future, >> these >> physical datalinks can be used for 1) implementing the port >> level >> statistics 2) implementing port level snoop etc. >> >> Example 1. # dladm show-phys >> LINK MEDIA STATE SPEED DUPLEX DEVICE >> ibp0 InfiniBand up 8000 unknown ibp0 >> ibp1 InfiniBand down 8000 unknown ibp1 >> ibp2 InfiniBand down 2000 unknown ibp2 >> ibp3 InfiniBand up 2000 unknown ibp3 >> >> The state of the physical link is directly corresponds to state of >> the IB >> HCA port. As you might expect other generic sub-commands such as >> rename- >> link, show-link, delete-phys, etc. also work on IB datalinks. >> >> 4.3.2 IB partition Objects >> -------------------------- >> >> IB partition objects represent a new "part" class of datalink and >> these >> objects are managed using the new dladm(1M) sub-commands. All the >> new >> sub-command interfaces are similar to VLAN/VNIC dladm(1M) >> sub-commands. IB >> partition datalinks can be created on the top of IB physical links >> one >> per each P_Key on the port. These links are used for data >> transfers. >> The updated dladm(1M) man page describes the different commands >> available >> for managing the IB partition links. >> >> 4.3.2.1 create-part >> >> Creates a new IB partition Object with the specified datalink >> name. >> >> Example 2: Create a IB partition link for the P_Key 0x8001 >> on the >> top of ibp0 physical datalink >> >> # dladm create-part -l ibp0 -P 0x8001 p8001.ibp0 >> >> The above commands succeeds if the port is "up", P_Key is >> present on >> the port and IPoIB successfully completes its >> initialization. On >> subsequent reboots after successful initial creation, the >> partition >> will be available even if port is "down" or P_Key is absent, >> though >> the datalink state will be marked as "down". >> >> Example 3: Create an IB partition link for the P_Key 0x9000 >> on the >> top of ibp2 >> >> # dladm create-part -f -l ibp2 -P 0x9000 p9000.ibp2 >> >> The force option "-f" allows to create the IB partition even >> when >> the P_Key is not present or Port is down. The link state >> will be >> marked as down. The link state will be updated to "up" when >> P_Key >> is added to the port and port is activated. >> >> Example 4: Plumb and assign a IP address to IB partition >> p9000.ibp2 >> >> # ifconfig p9000.ibp2 plumb up >> >> # ifconfig -a >> >> p9000.ibp2: flags=1000843> MULTICAST,IPv4> mtu 2044 index 3 >> inet 1.1.1.1 netmask ff000000 broadcast 1.255.255.255 >> >> Example 5: Display the partition links using show-link command >> >> # dladm show-link >> LINK CLASS MTU STATE BRIDGE OVER >> p8001.ibp0 part 65520 unknown -- ibp0 >> p9000.ibp2 part 65520 down -- ibp2 >> >> 4.3.2.2 delete-part >> >> Deletes a specified IB partition object >> >> Example 6: Delete the partition p8001.ibp0 >> >> # dladm delete-part p8001.ibp0 >> >> The above command deletes the partition. >> >> Example 7: Show the partition link information after >> deleting p8001.ibp0 >> >> # dldam show-part >> >> LINK PKEY OVER STATE FLAGS >> p9000.ibp2 9000 ibp2 down f--- >> >> 4.3.2.3 show-part >> >> Displays IB partition object information >> >> Example 8: Display the IB partition links information (below >> output >> is for the "part" operations in Example 2, 3& 4) >> >> # dldam show-part >> >> LINK PKEY OVER STATE FLAGS >> p8001.ibp0 8001 ibp0 unknown ---- >> p9000.ibp2 9000 ibp2 down f--- >> >> The state of the IB partition link will be "unknown" after IB >> partition >> is created and before IB partition is plumbed. Once partition is >> plumbed >> the link state will be set to "up" when the link is ready to >> use. The >> state of the link will be set to "down" if 1) HCA port is down >> 2) P_Key >> is absent or 3) broadcast group is absent. >> >> 4.3.2.4 show-ib >> >> Displays IB specific information such as port#, port guid, etc. >> >> Example 9: Show IB specific information >> >> # dladm show-ib >> LINK HCAGUID PORTGUID PORT STATE PKEYS >> ibp0 3BA000100CD7C 3BA000100CD7D 1 down FFFF >> ibp1 3BA000100CD7C 3BA000100CD7E 2 down FFFF >> ibp3 5AD0000033634 5AD0000033636 2 up FFFF,8001 >> ibp2 5AD0000033634 5AD0000033635 1 up FFFF,8001 >> >> show-ib commands display only the physical links, port GUID, >> port# >> HCA GUID, and P_Key present on the port at the time of >> running the >> command. >> >> 4.3.3 IB partition object Administration Library >> >> The libdladm(3LIB) library is currently used to implement >> datalink >> administration for all the GLDv3 datalinks (VNIC, link >> aggregation, >> wireless, IP Tunnel, etc.). The library will be further enhanced to >> provide >> administrative interfaces of InfiniBand partition objects. All >> the new >> library extensions are similar to VLAN/VNIC libdladm >> extensions. IB >> specific functionality will be implemented by libdlib.c and its >> interface >> is provided via libdlib.h. dladm(1M) will use this library for >> managing >> IB partitions. The IB partition administration library >> provides a >> persistent repository which allows IB partition configuration to be >> stored >> across reboots. It uses the existing dladm(1M) >> /etc/dladm/datalink.conf >> repository to store the IB partition configuration. >> >> The list of InfiniBand specific extensions to libdladm library is >> given below. >> For more details of each API, see man pages in the PSARC >> materials >> directory. The man pages are only for PSARC review (not intended for >> public >> use). >> >> 4.3.3.1 dladm_part_create() >> Create a IB partition object >> >> 4.3.3.2 dladm_part_delete() >> Deletes a IB partition object >> >> 4.3.3.3 dladm_part_info() >> Returns IB partition Object attributes >> >> 4.3.3.4 dladm_ib_info() >> Returns IB specific attributes such as port number, port guid, >> and HCA GUID. >> >> 4.3.3.5 dladm_part_up() >> Brings up one or all the IB partition objects during every boot. >> >> 4.3.4 IBTF Extensions >> --------------------- >> >> Some of the IB ULP's such as SDP, IBCM, etc. walk the device tree and >> read >> "port-pkey", "hca-guid", and "port-guid" properties from the ibd(7D) >> device >> instance. With the new model, IB partitions are just objects (no >> longer >> device nodes). So, IB ULP's no longer can retrieve partition >> attributes >> from the device node. The following new IBTF extensions >> provide a >> mechanism to retrieve the partition attributes in the >> kernel. See >> ibt_get_part_attrs() man page in the PSARC case directory for more >> details >> about the following API. The man pages are only for PSARC review >> (not >> intended for public use). >> >> 4.3.4.1 ibt_get_part_attrs() >> Returns the attributes of a requested IB partition >> >> 4.3.4.2 ibt_get_all_part_attrs() >> Returns the attributes of all the active IB partitions >> >> 4.3.4.3 ibt_free_part_attrs() >> Frees the memory for partition attribute structure allocated by >> ibt_get_all_part_attrs() >> >> 4.3.5 Man page changes >> ---------------------- >> >> Updated man pages - published >> dladm(1M) >> datadm(1M) >> dat.conf(4) >> ibp(7D) (ibd(7d) name changed) >> >> New Man pages - internal only >> dladm_part_create(3dladm) >> dladm_part_delete(3dladm) >> dladm_part_info(3dladm) >> dladm_ib_info(3dladm) >> dladm_part_up(3dladm) >> >> ibt_get_part_attrs(9f) >> >> 4.3.6 Interface table >> --------------------- >> >> >> ----------------------------------------------------------------------- >> | Interface name | Commitment >> Level | >> >> ----------------------------------------------------------------------- >> | dladm(1M) >> extensions | >> >> ----------------------------------------------------------------------- >> | create-part >> | | >> | delete-part | >> committed | >> | show-ib >> | | >> | show-part >> | | >> >> ----------------------------------------------------------------------- >> | libdladm extensions >> (libdlib.h) | >> >> ----------------------------------------------------------------------- >> | dladm_part_create >> | | >> | dladm_part_delete >> | | >> | dladm_part_show >> | | >> | dladm_show_ib | ON Consolidation >> Private | >> | dladm_part_up >> | | >> | dladm_part_attr_t >> | | >> | dladm_ib_attr_t >> | | >> | DLADM_IBPART_FORCE_CREATE >> | | >> >> ----------------------------------------------------------------------- >> | InfiniBand Specific Link >> Properties | >> >> ----------------------------------------------------------------------- >> | linkmode | ON Consolidation >> Private | >> >> ----------------------------------------------------------------------- >> | >> libdlmgmt.h | >> >> ----------------------------------------------------------------------- >> | DATALINK_CLASS_IBPART | ON Consolidation >> Private | >> >> ----------------------------------------------------------------------- >> | >> libdladm.h | >> >> ----------------------------------------------------------------------- >> | DLADM_STATUS_INVALID_PORT_INSTANCE >> | | >> | DLADM_STATUS_PORT_IS_DOWN >> | | >> | DLADM_STATUS_PKEY_NOT_PRESENT >> | | >> | DLADM_STATUS_PARTITION_EXISTS | ON Consolidation >> Private | >> | DLADM_STATUS_INVALID_PKEY >> | | >> | DLADM_STATUS_NO_HW_RESOURCE >> | | >> | DLADM_STATUS_INVALID_PKEY_TBL_SIZE >> | | >> >> ----------------------------------------------------------------------- >> | IBTF >> extensions | >> >> ----------------------------------------------------------------------- >> | ibt_get_part_attrs() >> | | >> | ibt_get_all_part_attrs() | ON Consolidation >> private | >> | ibt_free_part_attrs() >> | | >> | ibt_part_attr_t >> | | >> >> ----------------------------------------------------------------------- >> | New status codes (ibt_status_t >> updates) | >> >> ----------------------------------------------------------------------- >> | IBT_NO_SUCH_OBJECT | ON Consolidation >> private | >> >> ----------------------------------------------------------------------- >> >> 4.3.7 References >> --------------------- >> >> PSARC 2001/289 IP over InfiniBand >> PSARC 2007/636 IPoIB Conversion to GLDv3 >> PSARC 2009/593 IPoIB Connected Mode >> >> 6. Resources and Schedule >> 6.4. Steering Committee requested information >> 6.4.1. Consolidation C-team Name: >> ON >> 6.5. ARC review type: FastTrack >> 6.6. ARC Exposure: open >> >> > > -- ---------------------------------------------- Arise Awake Stop Not till the goal is reached ---------------------------------------------- Sudhakar Dindukurti Oracle America, Inc. 17, Network Circle, Rm#3384 (650) 786 8563 (Work) Menlo Park, CA 94025 From sebastien.roy@oracle.com Fri Mar 12 09:53:16 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2CHrFW2001920 for ; Fri, 12 Mar 2010 09:53:15 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2CHrEgD002886; Fri, 12 Mar 2010 11:53:14 -0600 (CST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZ600403KCQ5A00@brm-avmta-1.central.sun.com>; Fri, 12 Mar 2010 10:53:14 -0700 (MST) Received: from sca-ea-mail-3.sun.com ([192.18.43.21]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZ6008SQKCP5YE0@brm-avmta-1.central.sun.com>; Fri, 12 Mar 2010 10:53:14 -0700 (MST) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by sca-ea-mail-3.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o2CHrDks010824; Fri, 12 Mar 2010 17:53:13 +0000 (GMT) Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o2BF947i032682; Fri, 12 Mar 2010 17:53:11 +0000 (GMT) Received: from abhmt003.oracle.com by acsmt354.oracle.com with ESMTP id 81406861268416323; Fri, 12 Mar 2010 09:52:03 -0800 Received: from [129.148.19.4] (/129.148.19.4) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 12 Mar 2010 09:52:03 -0800 Date: Fri, 12 Mar 2010 12:52:01 -0500 From: Sebastien Roy Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> To: Ted Kim Cc: PSARC-ext@sun.com, cynthia.mcguire@sun.com, peter.memishian@sun.com Message-id: <4B9A7F41.5010000@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090209.4B9A7F88.018E:SCFMA4539814,ss=1,fgs=0 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.7) Gecko/20100214 Lightning/1.0b1 Thunderbird/3.0.1 Status: RO Content-Length: 2142 Ted, One point of minor confusion for me is that you have another ongoing PSARC case that overlaps this one (PSARC 2009/365). I can guess that you intended this case to supersede that one, but can you please make that explicit here? If that's indeed the case, please close 2009/365 appropriately. On 03/10/10 06:52 PM, Ted Kim wrote: > 4.2.3 Update IBD driver to use Brussels framework (CR 6883212) > > IPoIB tunables are managed today through /etc/system file or ibd.conf. > For example, to modify the 'linkmode' of the IPoIB link, one needs to edit > ibd.conf and reboot the system. We want to replace this interface so that > user can manage these tunables using dladm(1M). > > 4.3 Proposal > > A micro/patch binding is asserted for this proposal. It isn't entirely obvious to me if a Solaris 10 consumer of IPoIB would be required to do anything after this project is hypothetically applied as a patch. Assuming of course that there is a patch for this project (which seems somewhat unlikely given the implicit dependencies on dozens of other dladm-related projects with Minor binding), one requirement of patch binding is that there are no incompatible changes introduced by the patch. Is that the case? On a related note, the materials don't explicitly state the case dependencies, but I believe this case has a dependency on PSARC 2007/429. Pulling on that case's string, the sweater that would need to get backported into a patch gets quickly unraveled. ;-) > ----------------------------------------------------------------------- > | InfiniBand Specific Link Properties | > ----------------------------------------------------------------------- > | linkmode | ON Consolidation Private | > ----------------------------------------------------------------------- Given that it's Consolidation Private, I take it that this is a private property (as defined by 2007/429). I was under the impression that private properties had to be prefixed by a '_' character as required by the Brussels framework. Is that not the case? -Seb From gdamore@sun.com Fri Mar 12 09:57:05 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2CHv54Z002284 for ; Fri, 12 Mar 2010 09:57:05 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2CHv5ga004993 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 12 Mar 2010 11:57:05 -0600 (CST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZ600009KJ45600@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Fri, 12 Mar 2010 09:57:04 -0800 (PST) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZ600HZ6KJ255A0@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Fri, 12 Mar 2010 09:57:02 -0800 (PST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o2CHv2u7028465 for ; Fri, 12 Mar 2010 09:57:02 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KZ600H00KDXOY00@fe-sfbay-09.sun.com> for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Fri, 12 Mar 2010 09:57:02 -0800 (PST) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KZ600DYCKJ15B10@fe-sfbay-09.sun.com>; Fri, 12 Mar 2010 09:57:02 -0800 (PST) Date: Fri, 12 Mar 2010 09:57:00 -0800 From: "Garrett D'Amore" Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <4B9A7DBF.9020708@Sun.COM> Sender: Garrett.Damore@sun.com To: Sudhakar Dindukurti Cc: Ted Kim , PSARC-ext@sun.com, cynthia.mcguire@sun.com, peter.memishian@sun.com Message-id: <4B9A806C.3030100@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> <4B9A6E99.7010002@sun.com> <4B9A7DBF.9020708@Sun.COM> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.7) Gecko/20100131 Lightning/1.0b1 Thunderbird/3.0.1 Status: RO Content-Length: 21951 Thanks for the answers. +1 on the case, with the added comment (for the record), that it might be a good idea to research the possibility of providing trunking/aggregation similar features over IB links. I'd also like to see snoop and statistic support on the phys link in the near future; I hope that is actively planned for a follow-up, and not just a hypothetical "we can do it if we want to" sort of thing. - Garrett On 03/12/10 09:45 AM, Sudhakar Dindukurti wrote: > Garrett, > > On 03/12/10 08:40, Garrett D'Amore wrote: >> Okay, here are some questions; some of these stem from my lack of >> IPoIB knowledge, so I hope you'll pardon me if I ask things that seem >> obvious or stupid. >> >> 1) Will the new "phys" objects being created here be "snoop"-able? >> (I understand that they can't transmit data, but can they operate in >> some sort of promiscuous receive mode?) (Oh wait, it seems that this >> might be answered in part by 4.3.1. It looks like the answer is >> "not yet, but in the future". Correct?) >> > snoop support on the "phys" link is not supported yet but can be done > in the future. > >> 2) From my experience with "hermon", hermon seems mostly to treat >> both ports as a single registered entity to the IB framework. I'm >> presuming that this won't preclude IPoIB from being able to identify >> which port is which? > Yes. dladm show-ib displays the port information associated with the > IB phys class datalinks. >> Are there any special considerations here for automatic path migration? > IPoIB instances can only be created one per P_Key per port per IB HCA. > Automatic path migration (APM) feature is supported from one port to > another port on the same HCA. So, APM is not applicable for IPoIB. > This case does not change any of this behavior from what exists today. >> >> 3) Are there any constraints on the format of the name used for >> "part-link" in dladm? The examples seem to show a specific format, >> but can customers choose any name they want? > Yes. The customer can choose any name they want as long as the name > format adheres to the existing "datalink" name. >> >> 4) The way this is designed seems to depend on a notion of a physical >> port. Does IPoIB have anything that is morally equivalent to >> ethernet aggregations (trunking)? I'm thinking also about link >> redundancy as well as bandwidth multiplication. >> > No. This case does not add/change the existing IPoIB behavior on > these features. > > regards, > Sudhakar > >> - Garrett >> >> On 03/10/10 03:52 PM, Ted Kim wrote: >>> Template Version: @(#)sac_nextcase 1.69 02/15/10 SMI >>> This information is Copyright 2010 Sun Microsystems >>> 1. Introduction >>> 1.1. Project/Component Working Name: >>> IPoIB Administration Enhancement >>> 1.2. Name of Document Author/Supplier: >>> Author: Sudhakar Dindukurti >>> 1.3 Date of This Document: >>> 10 March, 2010 >>> 4. Technical Description >>> >>> 4.1 Acronyms >>> >>> HCA : Host Channel Adaptor >>> P_Key : Partition Key >>> IPoIB : IP over InfiniBand >>> GUID : Global Unique Identifier >>> IBTF : InfiniBand Transport Framework >>> IBCM : InfiniBand Communication Manager >>> SDP : Sockets Direct Protocol >>> IBD : current IPoIB Solaris driver name >>> ULP : Upper level Protocol >>> >>> 4.2 Requirements/Motivation >>> --------------------------- >>> >>> 4.2.1 Consistent IBD device node naming across nodes in a clusters >>> >>> (Amber Road Requirement (CR 6864899)) >>> >>> AmberRoad clustering software requires that datalink name for a >>> specific >>> partition on multiple nodes (identical h/w configuration) to be >>> same. For >>> example, if datalink name on node 1 for a P_key 0x8001 on HCA 1 and >>> port 1 >>> is ibnet0, then the clustering software expects that datalink >>> name for >>> P_Key 0x8001 on HCA 1 and port 1 is also ibnet0 on the node 2 >>> in the >>> cluster. This is very difficult to achieve today. The problem is >>> that >>> IPoIB device name is constructed automatically by IPoIB >>> driver by >>> appending the instance number to the driver name (ex ibd0, ibd1, >>> etc). So, >>> the clustering software does not have any control on the IPoIB >>> device >>> name space. Also, IBTF framework does not guarantee that the same >>> names >>> across multiple nodes. >>> >>> 4.2.2 Problem diagnosis issue >>> >>> IPoIB (PSARC 2001/289, PSARC 2009/593 and PSARC 2007/636) is >>> implemented >>> as the ibd(7D) driver in Solaris. When customers face >>> problems in >>> bringing up the IBD devices on Solaris, they are advised to take >>> number >>> of steps to gather information about the IPoIB link that >>> results in >>> usability issues. Today, users need to run cfgadm(1M) to obtain the >>> port >>> GUID information, do 'ls-l' on the /dev/ibd* nodes to determine >>> the IB >>> partition that an IBD device belongs to etc. There are also >>> instances >>> when it is necessary to obtain the P_key corresponding to the IBD >>> device >>> and HCA port that it is bound to. So, we want to improve the >>> process of >>> gathering IPoIB related configuration information. >>> >>> 4.2.3 Update IBD driver to use Brussels framework (CR 6883212) >>> >>> IPoIB tunables are managed today through /etc/system file or >>> ibd.conf. >>> For example, to modify the 'linkmode' of the IPoIB link, one needs >>> to edit >>> ibd.conf and reboot the system. We want to replace this interface so >>> that >>> user can manage these tunables using dladm(1M). >>> >>> 4.3 Proposal >>> >>> A micro/patch binding is asserted for this proposal. >>> >>> 4.3.0 IPoIB administration with dladm(1M) >>> ---------------------------------------- >>> >>> This case proposes a new IPoIB administration mechanism for >>> InfiniBand >>> network datalinks using dladm(1M) command. Also, it proposes >>> to add >>> a consolidation private InfiniBand specific library API to >>> libdladm(3LIB). >>> The mechanism is very similar to the existing VLAN, VNIC, >>> etc. >>> administration dladm(1M) sub-commands. >>> >>> In the new model, two classes of IPoIB datalinks will exist: >>> >>> 1. Datalinks representing the physical IB ports, which will >>> use the >>> existing "phys" class that is used for Ethernet, WiFi, and >>> other >>> physical media. As with all "phys" class objects, the >>> system >>> will create these automatically. >>> >>> 2. Datalinks representing the administratively created >>> partitions >>> over "phys" IPoIB objects, which will use a new "part" >>> class. >>> Each IB partition datalink will be associated with a >>> P_Key in a >>> manner analogous to the way each Ethernet VLAN >>> datalink is >>> associated with a VLAN ID. >>> >>> Note that unlike other "phys" class objects, IB "phys" objects >>> cannot >>> send data (since IB requires a P_Key to send data) and thus >>> cannot be >>> plumbed. >>> >>> Example configuration used to explain the new IPoIB administration >>> model >>> >>> -------------------------------------- >>> | IB switch | >>> -------------------------------------- >>> | | >>> | | >>> | | >>> Port1 | | Port2 Port1 | | Port2 >>> -------------------------------------- >>> | HCA1 HCA2 | >>> | Node 1 | >>> -------------------------------------- >>> >>> Port 2 of the HCA1 and Port 1 of the HCA2 are connected to IB >>> switch and >>> SM is running the switch. Each port is configured with two P_Key's >>> (0xffff >>> & 0x8001). >>> >>> >>> 4.3.1 Physical datalinks >>> ------------------------- >>> >>> One physical datalink will be created by default per port per HCA. >>> These >>> physical links serve as administrative& observability data points. >>> These >>> IB physical datalinks allow creating IB partitions over them >>> similar >>> to creating VNICs on Ethernet physical links or link >>> aggregations. IB >>> physical datalinks are not used for data transfers. So, the >>> plumb and >>> assigning a IB address are not supported on these links. In future, >>> these >>> physical datalinks can be used for 1) implementing the port >>> level >>> statistics 2) implementing port level snoop etc. >>> >>> Example 1. # dladm show-phys >>> LINK MEDIA STATE SPEED DUPLEX DEVICE >>> ibp0 InfiniBand up 8000 unknown ibp0 >>> ibp1 InfiniBand down 8000 unknown ibp1 >>> ibp2 InfiniBand down 2000 unknown ibp2 >>> ibp3 InfiniBand up 2000 unknown ibp3 >>> >>> The state of the physical link is directly corresponds to state of >>> the IB >>> HCA port. As you might expect other generic sub-commands such as >>> rename- >>> link, show-link, delete-phys, etc. also work on IB datalinks. >>> >>> 4.3.2 IB partition Objects >>> -------------------------- >>> >>> IB partition objects represent a new "part" class of datalink and >>> these >>> objects are managed using the new dladm(1M) sub-commands. All >>> the new >>> sub-command interfaces are similar to VLAN/VNIC dladm(1M) >>> sub-commands. IB >>> partition datalinks can be created on the top of IB physical >>> links one >>> per each P_Key on the port. These links are used for data >>> transfers. >>> The updated dladm(1M) man page describes the different commands >>> available >>> for managing the IB partition links. >>> >>> 4.3.2.1 create-part >>> >>> Creates a new IB partition Object with the specified datalink >>> name. >>> >>> Example 2: Create a IB partition link for the P_Key 0x8001 >>> on the >>> top of ibp0 physical datalink >>> >>> # dladm create-part -l ibp0 -P 0x8001 p8001.ibp0 >>> >>> The above commands succeeds if the port is "up", P_Key is >>> present on >>> the port and IPoIB successfully completes its >>> initialization. On >>> subsequent reboots after successful initial creation, the >>> partition >>> will be available even if port is "down" or P_Key is absent, >>> though >>> the datalink state will be marked as "down". >>> >>> Example 3: Create an IB partition link for the P_Key 0x9000 >>> on the >>> top of ibp2 >>> >>> # dladm create-part -f -l ibp2 -P 0x9000 p9000.ibp2 >>> >>> The force option "-f" allows to create the IB partition >>> even when >>> the P_Key is not present or Port is down. The link state >>> will be >>> marked as down. The link state will be updated to "up" when >>> P_Key >>> is added to the port and port is activated. >>> >>> Example 4: Plumb and assign a IP address to IB partition >>> p9000.ibp2 >>> >>> # ifconfig p9000.ibp2 plumb up >>> >>> # ifconfig -a >>> >>> p9000.ibp2: flags=1000843>> MULTICAST,IPv4> mtu 2044 index 3 >>> inet 1.1.1.1 netmask ff000000 broadcast 1.255.255.255 >>> >>> Example 5: Display the partition links using show-link command >>> >>> # dladm show-link >>> LINK CLASS MTU STATE BRIDGE OVER >>> p8001.ibp0 part 65520 unknown -- ibp0 >>> p9000.ibp2 part 65520 down -- ibp2 >>> >>> 4.3.2.2 delete-part >>> >>> Deletes a specified IB partition object >>> >>> Example 6: Delete the partition p8001.ibp0 >>> >>> # dladm delete-part p8001.ibp0 >>> >>> The above command deletes the partition. >>> >>> Example 7: Show the partition link information after >>> deleting p8001.ibp0 >>> >>> # dldam show-part >>> >>> LINK PKEY OVER STATE FLAGS >>> p9000.ibp2 9000 ibp2 down f--- >>> >>> 4.3.2.3 show-part >>> >>> Displays IB partition object information >>> >>> Example 8: Display the IB partition links information (below >>> output >>> is for the "part" operations in Example 2, 3& 4) >>> >>> # dldam show-part >>> >>> LINK PKEY OVER STATE FLAGS >>> p8001.ibp0 8001 ibp0 unknown ---- >>> p9000.ibp2 9000 ibp2 down f--- >>> >>> The state of the IB partition link will be "unknown" after IB >>> partition >>> is created and before IB partition is plumbed. Once partition is >>> plumbed >>> the link state will be set to "up" when the link is ready to >>> use. The >>> state of the link will be set to "down" if 1) HCA port is down >>> 2) P_Key >>> is absent or 3) broadcast group is absent. >>> >>> 4.3.2.4 show-ib >>> >>> Displays IB specific information such as port#, port guid, etc. >>> >>> Example 9: Show IB specific information >>> >>> # dladm show-ib >>> LINK HCAGUID PORTGUID PORT STATE PKEYS >>> ibp0 3BA000100CD7C 3BA000100CD7D 1 down FFFF >>> ibp1 3BA000100CD7C 3BA000100CD7E 2 down FFFF >>> ibp3 5AD0000033634 5AD0000033636 2 up FFFF,8001 >>> ibp2 5AD0000033634 5AD0000033635 1 up FFFF,8001 >>> >>> show-ib commands display only the physical links, port >>> GUID, port# >>> HCA GUID, and P_Key present on the port at the time of >>> running the >>> command. >>> >>> 4.3.3 IB partition object Administration Library >>> >>> The libdladm(3LIB) library is currently used to implement >>> datalink >>> administration for all the GLDv3 datalinks (VNIC, link >>> aggregation, >>> wireless, IP Tunnel, etc.). The library will be further enhanced to >>> provide >>> administrative interfaces of InfiniBand partition objects. All >>> the new >>> library extensions are similar to VLAN/VNIC libdladm >>> extensions. IB >>> specific functionality will be implemented by libdlib.c and its >>> interface >>> is provided via libdlib.h. dladm(1M) will use this library for >>> managing >>> IB partitions. The IB partition administration library >>> provides a >>> persistent repository which allows IB partition configuration to be >>> stored >>> across reboots. It uses the existing dladm(1M) >>> /etc/dladm/datalink.conf >>> repository to store the IB partition configuration. >>> >>> The list of InfiniBand specific extensions to libdladm library is >>> given below. >>> For more details of each API, see man pages in the PSARC >>> materials >>> directory. The man pages are only for PSARC review (not intended for >>> public >>> use). >>> >>> 4.3.3.1 dladm_part_create() >>> Create a IB partition object >>> >>> 4.3.3.2 dladm_part_delete() >>> Deletes a IB partition object >>> >>> 4.3.3.3 dladm_part_info() >>> Returns IB partition Object attributes >>> >>> 4.3.3.4 dladm_ib_info() >>> Returns IB specific attributes such as port number, port guid, >>> and HCA GUID. >>> >>> 4.3.3.5 dladm_part_up() >>> Brings up one or all the IB partition objects during every boot. >>> >>> 4.3.4 IBTF Extensions >>> --------------------- >>> >>> Some of the IB ULP's such as SDP, IBCM, etc. walk the device tree >>> and read >>> "port-pkey", "hca-guid", and "port-guid" properties from the ibd(7D) >>> device >>> instance. With the new model, IB partitions are just objects (no >>> longer >>> device nodes). So, IB ULP's no longer can retrieve partition >>> attributes >>> from the device node. The following new IBTF extensions >>> provide a >>> mechanism to retrieve the partition attributes in the >>> kernel. See >>> ibt_get_part_attrs() man page in the PSARC case directory for more >>> details >>> about the following API. The man pages are only for PSARC >>> review (not >>> intended for public use). >>> >>> 4.3.4.1 ibt_get_part_attrs() >>> Returns the attributes of a requested IB partition >>> >>> 4.3.4.2 ibt_get_all_part_attrs() >>> Returns the attributes of all the active IB partitions >>> >>> 4.3.4.3 ibt_free_part_attrs() >>> Frees the memory for partition attribute structure allocated by >>> ibt_get_all_part_attrs() >>> >>> 4.3.5 Man page changes >>> ---------------------- >>> >>> Updated man pages - published >>> dladm(1M) >>> datadm(1M) >>> dat.conf(4) >>> ibp(7D) (ibd(7d) name changed) >>> >>> New Man pages - internal only >>> dladm_part_create(3dladm) >>> dladm_part_delete(3dladm) >>> dladm_part_info(3dladm) >>> dladm_ib_info(3dladm) >>> dladm_part_up(3dladm) >>> >>> ibt_get_part_attrs(9f) >>> >>> 4.3.6 Interface table >>> --------------------- >>> >>> >>> ----------------------------------------------------------------------- >>> | Interface name | Commitment >>> Level | >>> >>> ----------------------------------------------------------------------- >>> | dladm(1M) >>> extensions | >>> >>> ----------------------------------------------------------------------- >>> | create-part >>> | | >>> | delete-part | >>> committed | >>> | show-ib >>> | | >>> | show-part >>> | | >>> >>> ----------------------------------------------------------------------- >>> | libdladm extensions >>> (libdlib.h) | >>> >>> ----------------------------------------------------------------------- >>> | dladm_part_create >>> | | >>> | dladm_part_delete >>> | | >>> | dladm_part_show >>> | | >>> | dladm_show_ib | ON Consolidation >>> Private | >>> | dladm_part_up >>> | | >>> | dladm_part_attr_t >>> | | >>> | dladm_ib_attr_t >>> | | >>> | DLADM_IBPART_FORCE_CREATE >>> | | >>> >>> ----------------------------------------------------------------------- >>> | InfiniBand Specific Link >>> Properties | >>> >>> ----------------------------------------------------------------------- >>> | linkmode | ON Consolidation >>> Private | >>> >>> ----------------------------------------------------------------------- >>> | >>> libdlmgmt.h | >>> >>> ----------------------------------------------------------------------- >>> | DATALINK_CLASS_IBPART | ON Consolidation >>> Private | >>> >>> ----------------------------------------------------------------------- >>> | >>> libdladm.h | >>> >>> ----------------------------------------------------------------------- >>> | DLADM_STATUS_INVALID_PORT_INSTANCE >>> | | >>> | DLADM_STATUS_PORT_IS_DOWN >>> | | >>> | DLADM_STATUS_PKEY_NOT_PRESENT >>> | | >>> | DLADM_STATUS_PARTITION_EXISTS | ON Consolidation >>> Private | >>> | DLADM_STATUS_INVALID_PKEY >>> | | >>> | DLADM_STATUS_NO_HW_RESOURCE >>> | | >>> | DLADM_STATUS_INVALID_PKEY_TBL_SIZE >>> | | >>> >>> ----------------------------------------------------------------------- >>> | IBTF >>> extensions | >>> >>> ----------------------------------------------------------------------- >>> | ibt_get_part_attrs() >>> | | >>> | ibt_get_all_part_attrs() | ON Consolidation >>> private | >>> | ibt_free_part_attrs() >>> | | >>> | ibt_part_attr_t >>> | | >>> >>> ----------------------------------------------------------------------- >>> | New status codes (ibt_status_t >>> updates) | >>> >>> ----------------------------------------------------------------------- >>> | IBT_NO_SUCH_OBJECT | ON Consolidation >>> private | >>> >>> ----------------------------------------------------------------------- >>> >>> 4.3.7 References >>> --------------------- >>> >>> PSARC 2001/289 IP over InfiniBand >>> PSARC 2007/636 IPoIB Conversion to GLDv3 >>> PSARC 2009/593 IPoIB Connected Mode >>> >>> 6. Resources and Schedule >>> 6.4. Steering Committee requested information >>> 6.4.1. Consolidation C-team Name: >>> ON >>> 6.5. ARC review type: FastTrack >>> 6.6. ARC Exposure: open >>> >> >> > > From peter.memishian@sun.com Fri Mar 12 10:19:00 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2CIJ090002664 for ; Fri, 12 Mar 2010 10:19:00 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2CIIxRJ023752; Fri, 12 Mar 2010 10:18:59 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZ600111LJNEV00@nwk-avmta-2.sfbay.sun.com>; Fri, 12 Mar 2010 10:18:59 -0800 (PST) Received: from triplex.local ([129.146.108.208]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZ600HTFLJM55D0@nwk-avmta-2.sfbay.sun.com>; Fri, 12 Mar 2010 10:18:58 -0800 (PST) Received: from triplex.local (triplex [127.0.0.1]) by triplex.local (8.14.3+Sun/8.14.3) with ESMTP id o2CIIwMP477899; Fri, 12 Mar 2010 10:18:58 -0800 (PST) Received: (from meem@localhost) by triplex.local (8.14.3+Sun/8.14.3/Submit) id o2CIIwOC477887; Fri, 12 Mar 2010 10:18:58 -0800 (PST) Date: Fri, 12 Mar 2010 10:18:58 -0800 From: Peter Memishian Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <4B9A806C.3030100@sun.com> To: "Garrett D'Amore" Cc: Sudhakar Dindukurti , Ted Kim , PSARC-ext@sun.com, Cynthia.McGuire@sun.com, peter.memishian@sun.com Reply-to: peter.memishian@sun.com Message-id: <19354.34194.241070.658085@gargle.gargle.HOWL> MIME-version: 1.0 X-Mailer: VM 7.19 under 21.4 (patch 21) "Educational Television" XEmacs Lucid Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> <4B9A6E99.7010002@sun.com> <4B9A7DBF.9020708@Sun.COM> <4B9A806C.3030100@sun.com> X-Authentication-warning: triplex.local: meem set sender to peter.memishian@sun.com using -f Status: RO Content-Length: 723 > Thanks for the answers. +1 on the case, with the added comment (for the > record), that it might be a good idea to research the possibility of > providing trunking/aggregation similar features over IB links. I'd also > like to see snoop and statistic support on the phys link in the near > future; I hope that is actively planned for a follow-up, and not just a > hypothetical "we can do it if we want to" sort of thing. I agree the observability/statistic support would be valuable. The high-order bit here is that the proposed architecture allows such as thing to be built, whereas the existing Solaris architecture does not since there is no datalink object associated with the physical port. -- meem From peter.memishian@sun.com Fri Mar 12 11:16:59 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2CJGxse003959 for ; Fri, 12 Mar 2010 11:16:59 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2CJGqjp028813; Fri, 12 Mar 2010 13:16:52 -0600 (CST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZ600C01O84KM00@brm-avmta-1.central.sun.com>; Fri, 12 Mar 2010 12:16:52 -0700 (MST) Received: from triplex.local ([129.146.108.208]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZ6007O0O845620@brm-avmta-1.central.sun.com>; Fri, 12 Mar 2010 12:16:52 -0700 (MST) Received: from triplex.local (triplex [127.0.0.1]) by triplex.local (8.14.3+Sun/8.14.3) with ESMTP id o2CJGpIn177592; Fri, 12 Mar 2010 11:16:51 -0800 (PST) Received: (from meem@localhost) by triplex.local (8.14.3+Sun/8.14.3/Submit) id o2CJGpCI177572; Fri, 12 Mar 2010 11:16:51 -0800 (PST) Date: Fri, 12 Mar 2010 11:16:51 -0800 From: Peter Memishian Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] To: Sebastien.Roy@sun.com Cc: "Garrett D'Amore" , Ted Kim , PSARC-ext@sun.com, Cynthia.McGuire@sun.com, Sudhakar Dindukurti , peter.memishian@sun.com Reply-to: peter.memishian@sun.com Message-id: <19354.37667.770160.332098@gargle.gargle.HOWL> MIME-version: 1.0 X-Mailer: VM 7.19 under 21.4 (patch 21) "Educational Television" XEmacs Lucid Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Authentication-warning: triplex.local: meem set sender to peter.memishian@sun.com using -f Status: RO Content-Length: 1445 > On a related note, the materials don't explicitly state the case > dependencies, but I believe this case has a dependency on PSARC > 2007/429. Pulling on that case's string, the sweater that would need to > get backported into a patch gets quickly unraveled. ;-) An excellent point, and one that I'm embarrased to admit I overlooked when considering the S10 aspects of this case. Further, this case is dependent on a massive number of other Nevada-only projects, going all the way back to Clearview Vanity Naming and the Devname Project. That is, I think any practical possibility of backporting this particular solution to S10 is dead in the water. > > ----------------------------------------------------------------------- > > | InfiniBand Specific Link Properties | > > ----------------------------------------------------------------------- > > | linkmode | ON Consolidation Private | > > ----------------------------------------------------------------------- > > Given that it's Consolidation Private, I take it that this is a private > property (as defined by 2007/429). I was under the impression that > private properties had to be prefixed by a '_' character as required by > the Brussels framework. Is that not the case? I think this is just an error in the interface table; `linkmode' is intended to be a Committed property for IB links, hence its inclusion into the dladm manpage Sudhakar provided. -- meem From Sudhakar.Dindukurti@Sun.COM Fri Mar 12 12:34:31 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2CKYVTd005643 for ; Fri, 12 Mar 2010 12:34:31 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2CKYUOE029404; Fri, 12 Mar 2010 12:34:30 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZ600903RTHWT00@nwk-avmta-2.sfbay.sun.com>; Fri, 12 Mar 2010 12:34:29 -0800 (PST) Received: from jurassic.Eng.Sun.COM ([129.146.17.59]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZ6004QGRTHDJ70@nwk-avmta-2.sfbay.sun.com>; Fri, 12 Mar 2010 12:34:29 -0800 (PST) Received: from [129.146.226.78] (partha.SFBay.Sun.COM [129.146.226.78]) by jurassic.Eng.Sun.COM (8.14.4+Sun/8.14.4) with ESMTP id o2CKYTXi375425; Fri, 12 Mar 2010 12:34:29 -0800 (PST) Date: Fri, 12 Mar 2010 12:34:14 -0800 From: Sudhakar Dindukurti Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <19354.37667.770160.332098@gargle.gargle.HOWL> To: peter.memishian@Sun.COM, Sebastien.Roy@Sun.COM Cc: "Garrett D'Amore" , Ted Kim , PSARC-ext@Sun.COM, Cynthia.McGuire@Sun.COM Message-id: <4B9AA546.8090908@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <19354.37667.770160.332098@gargle.gargle.HOWL> User-Agent: Thunderbird 2.0.0.23 (X11/20090929) Status: RO Content-Length: 1935 On 03/12/10 11:16, Peter Memishian wrote: > > On a related note, the materials don't explicitly state the case > > dependencies, but I believe this case has a dependency on PSARC > > 2007/429. Pulling on that case's string, the sweater that would need to > > get backported into a patch gets quickly unraveled. ;-) > > An excellent point, and one that I'm embarrased to admit I overlooked when > considering the S10 aspects of this case. Further, this case is dependent > on a massive number of other Nevada-only projects, going all the way back > to Clearview Vanity Naming and the Devname Project. That is, I think any > practical possibility of backporting this particular solution to S10 is > dead in the water. > > > > ----------------------------------------------------------------------- > > > | InfiniBand Specific Link Properties | > > > ----------------------------------------------------------------------- > > > | linkmode | ON Consolidation Private | > > > ----------------------------------------------------------------------- > > > > Given that it's Consolidation Private, I take it that this is a private > > property (as defined by 2007/429). I was under the impression that > > private properties had to be prefixed by a '_' character as required by > > the Brussels framework. Is that not the case? > > I think this is just an error in the interface table; `linkmode' is > intended to be a Committed property for IB links, hence its inclusion into > the dladm manpage Sudhakar provided. > > Yes. The "linkmode" property is intended to be a committed property for IB datalinks. regards, Sudhakar > -- > meem > > -- ---------------------------------------------- Arise Awake Stop Not till the goal is reached ---------------------------------------------- Sudhakar Dindukurti Oracle America, Inc. 17, Network Circle, Rm#3384 (650) 786 8563 (Work) Menlo Park, CA 94025 From sebastien.roy@oracle.com Fri Mar 12 12:44:17 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2CKiHjf005853 for ; Fri, 12 Mar 2010 12:44:17 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2CKiEYk029392; Fri, 12 Mar 2010 12:44:16 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZ600L0HS9RRI00@brm-avmta-1.central.sun.com>; Fri, 12 Mar 2010 13:44:15 -0700 (MST) Received: from sca-ea-mail-1.sun.com ([192.18.43.24]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZ6007CHS9R5480@brm-avmta-1.central.sun.com>; Fri, 12 Mar 2010 13:44:15 -0700 (MST) Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by sca-ea-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o2CKi9nF028532; Fri, 12 Mar 2010 20:44:09 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o2CG1E5M011567; Fri, 12 Mar 2010 20:44:06 +0000 (GMT) Received: from abhmt013.oracle.com by acsmt354.oracle.com with ESMTP id 77267541268426547; Fri, 12 Mar 2010 12:42:27 -0800 Received: from [129.148.19.4] (/129.148.19.4) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 12 Mar 2010 12:42:26 -0800 Date: Fri, 12 Mar 2010 15:42:25 -0500 From: Sebastien Roy Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <4B9AA546.8090908@Sun.COM> To: Sudhakar Dindukurti Cc: peter.memishian@sun.com, Sebastien.Roy@sun.com, PSARC-ext@sun.com, Cynthia.McGuire@sun.com, "Garrett D'Amore" , Ted Kim Message-id: <4B9AA731.80300@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090202.4B9AA799.004D:SCFMA4539814,ss=1,fgs=0 References: <19354.37667.770160.332098@gargle.gargle.HOWL> <4B9AA546.8090908@Sun.COM> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.7) Gecko/20100214 Lightning/1.0b1 Thunderbird/3.0.1 Status: RO Content-Length: 985 On 03/12/10 03:34 PM, Sudhakar Dindukurti wrote: >> > > >> ----------------------------------------------------------------------- >> > > | InfiniBand Specific Link Properties | >> > > >> ----------------------------------------------------------------------- >> > > | linkmode | ON Consolidation Private | >> > > >> ----------------------------------------------------------------------- >> > > Given that it's Consolidation Private, I take it that this is a >> private >> > property (as defined by 2007/429). I was under the impression that >> > private properties had to be prefixed by a '_' character as required by >> > the Brussels framework. Is that not the case? >> >> I think this is just an error in the interface table; `linkmode' is >> intended to be a Committed property for IB links, hence its inclusion >> into >> the dladm manpage Sudhakar provided. >> > Yes. The "linkmode" property is intended to be a committed property for > IB datalinks. Okay, sounds good. -Seb From Ted.Kim@sun.com Fri Mar 12 15:47:56 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2CNluAf009323 for ; Fri, 12 Mar 2010 15:47:56 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o2CNltc4030103 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 12 Mar 2010 16:47:55 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZ700N030RV1500@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Fri, 12 Mar 2010 15:47:55 -0800 (PST) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZ700GN40RVRC90@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Fri, 12 Mar 2010 15:47:55 -0800 (PST) Received: from fe-sfbay-08.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o2CNltkI012113 for ; Fri, 12 Mar 2010 15:47:55 -0800 (PST) Received: from sun.com ([unknown] [192.18.34.118]) by fe-sfbay-08.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTP id <0KZ70070R0RUD400@fe-sfbay-08.sun.com> for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Fri, 12 Mar 2010 15:47:54 -0800 (PST) Received: from [129.153.2.5] by fe-sfbay-08.sun.com (mshttpd); Fri, 12 Mar 2010 15:47:54 -0800 Date: Fri, 12 Mar 2010 15:47:54 -0800 From: Ted Kim Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <4B9A7F41.5010000@oracle.com> To: Sebastien Roy Cc: Ted Kim , PSARC-ext@sun.com, Cynthia.McGuire@sun.com, peter.memishian@sun.com, Sudhakar Dindukurti Message-id: <6e812fc53d87.4b9a622a@sun.com> MIME-version: 1.0 X-Mailer: Sun Java(tm) System Messenger Express 7u2-7.04 64bit (built Jul 2 2009) Content-type: text/plain; CHARSET=US-ASCII Content-language: en Content-transfer-encoding: 7BIT Content-disposition: inline X-Accept-Language: en Priority: normal X-PMX-Version: 5.4.1.325704 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> <4B9A7F41.5010000@oracle.com> Status: RO Content-Length: 858 Seb, > One point of minor confusion for me is that you have another ongoing > PSARC case that overlaps this one (PSARC 2009/365). I can guess that > you intended this case to supersede that one, but can you please make > that explicit here? If that's indeed the case, please close 2009/365 > appropriately. Good point. I will say for the record that you are correct about the intention. This case was meant to supercede the older one. I will clean up the old one and close/withdraw it. To be honest, I have never had to do that before, so if there is a specific way to resolve this which you would want to see, please tell me. Thanks, -ted Ted H. Kim, PhD Sun/Oracle America Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1115 FAX From sebastien.roy@oracle.com Fri Mar 12 18:28:37 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2D2Sb5M011708 for ; Fri, 12 Mar 2010 18:28:37 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2D2SZqn026989; Fri, 12 Mar 2010 20:28:35 -0600 (CST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZ700B0987NST00@nwk-avmta-2.sfbay.sun.com>; Fri, 12 Mar 2010 18:28:35 -0800 (PST) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZ7005D387NLP40@nwk-avmta-2.sfbay.sun.com>; Fri, 12 Mar 2010 18:28:35 -0800 (PST) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o2D2SYiS019735; Sat, 13 Mar 2010 02:28:34 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o2D2GIUN012621; Sat, 13 Mar 2010 02:28:33 +0000 (GMT) Received: from abhmt015.oracle.com by acsmt355.oracle.com with ESMTP id 77755211268447256; Fri, 12 Mar 2010 18:27:36 -0800 Received: from [129.148.19.4] (/129.148.19.4) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 12 Mar 2010 18:27:34 -0800 Date: Fri, 12 Mar 2010 21:27:29 -0500 From: Sebastien Roy Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <6e812fc53d87.4b9a622a@sun.com> To: Ted Kim Cc: Ted Kim , PSARC-ext@sun.com, Cynthia.McGuire@sun.com, Peter.Memishian@sun.com, Sudhakar Dindukurti Message-id: <4B9AF811.3060503@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090205.4B9AF852.005D:SCFMA4539814,ss=1,fgs=0 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> <4B9A7F41.5010000@oracle.com> <6e812fc53d87.4b9a622a@sun.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.7) Gecko/20100214 Lightning/1.0b1 Thunderbird/3.0.1 Status: RO Content-Length: 958 On 03/12/10 06:47 PM, Ted Kim wrote: >> One point of minor confusion for me is that you have another ongoing >> PSARC case that overlaps this one (PSARC 2009/365). I can guess that >> you intended this case to supersede that one, but can you please make >> that explicit here? If that's indeed the case, please close 2009/365 >> appropriately. > > Good point. I will say for the record > that you are correct about the intention. > This case was meant to supercede the older one. > I will clean up the old one and close/withdraw it. > > To be honest, I have never had to do that before, > so if there is a specific way to resolve this which > you would want to see, please tell me. The most appropriate thing to do would be to close that case by changing its status in the IAM file as "closed superseded 2010/085". Thanks, -Seb P.S.: Of my original email message, there is still one outstanding question regarding the validity of the release binding. From Ted.Kim@sun.com Sun Mar 14 21:35:18 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2F4ZIrs017441 for ; Sun, 14 Mar 2010 21:35:18 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o2F4ZIRW030823 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Sun, 14 Mar 2010 22:35:18 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZB007033EUF000@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Sun, 14 Mar 2010 22:35:18 -0600 (MDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZB00AKE3ETMT70@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Sun, 14 Mar 2010 22:35:17 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o2F4ZHxB010254 for ; Sun, 14 Mar 2010 21:35:17 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KZB003003DK8G00@fe-sfbay-09.sun.com> for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Sun, 14 Mar 2010 21:35:17 -0700 (PDT) Received: from [129.153.3.74] ([unknown] [129.153.3.74]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KZB00M7C3ESTK20@fe-sfbay-09.sun.com>; Sun, 14 Mar 2010 21:35:17 -0700 (PDT) Date: Sun, 14 Mar 2010 21:34:38 -0700 From: "Ted H. Kim" Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <4B9AF811.3060503@oracle.com> Sender: Ted.Kim@sun.com To: Sebastien Roy Cc: Ted Kim , PSARC-ext@sun.com, Cynthia.McGuire@sun.com, Peter.Memishian@sun.com, Sudhakar Dindukurti Message-id: <4B9DB8DE.4050206@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> <4B9A7F41.5010000@oracle.com> <6e812fc53d87.4b9a622a@sun.com> <4B9AF811.3060503@oracle.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090910) Status: RO Content-Length: 1347 Okay I closed PSARC/2009/365 as superseded by this one. Didn't forget about your other question. I or Sudhakar will get answer for you. -ted Sebastien Roy wrote: > On 03/12/10 06:47 PM, Ted Kim wrote: >>> One point of minor confusion for me is that you have another ongoing >>> PSARC case that overlaps this one (PSARC 2009/365). I can guess that >>> you intended this case to supersede that one, but can you please make >>> that explicit here? If that's indeed the case, please close 2009/365 >>> appropriately. >> >> Good point. I will say for the record >> that you are correct about the intention. >> This case was meant to supercede the older one. >> I will clean up the old one and close/withdraw it. >> >> To be honest, I have never had to do that before, >> so if there is a specific way to resolve this which >> you would want to see, please tell me. > > The most appropriate thing to do would be to close that case by changing > its status in the IAM file as "closed superseded 2010/085". > > Thanks, > -Seb > > P.S.: Of my original email message, there is still one outstanding > question regarding the validity of the release binding. -- Ted H. Kim, PhD Sun/Oracle America Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1115 FAX From sudhakar.dindukurti@sun.com Tue Mar 16 22:51:26 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2H5pQxP013800 for ; Tue, 16 Mar 2010 22:51:26 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2H5pP0E005192; Tue, 16 Mar 2010 22:51:25 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZE00303W9PIL00@nwk-avmta-2.sfbay.sun.com>; Tue, 16 Mar 2010 22:51:25 -0700 (PDT) Received: from jurassic.Eng.Sun.COM ([129.146.17.59]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZE000KKW9OLA20@nwk-avmta-2.sfbay.sun.com>; Tue, 16 Mar 2010 22:51:24 -0700 (PDT) Received: from sudhakar.local (vpn-129-150-32-53.Central.Sun.COM [129.150.32.53]) by jurassic.Eng.Sun.COM (8.14.4+Sun/8.14.4) with ESMTP id o2H5pK44948799; Tue, 16 Mar 2010 22:51:22 -0700 (PDT) Date: Tue, 16 Mar 2010 22:51:22 -0700 From: Sudhakar Dindukurti Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <4B9A7F41.5010000@oracle.com> To: Sebastien Roy Cc: Ted Kim , PSARC-ext@sun.com, cynthia.mcguire@sun.com, peter.memishian@sun.com Message-id: <4BA06DDA.8040802@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> <4B9A7F41.5010000@oracle.com> User-Agent: Thunderbird 2.0.0.24 (Macintosh/20100228) Status: RO Content-Length: 2969 Sebastien Roy wrote: > Ted, > > One point of minor confusion for me is that you have another ongoing > PSARC case that overlaps this one (PSARC 2009/365). I can guess that > you intended this case to supersede that one, but can you please make > that explicit here? If that's indeed the case, please close 2009/365 > appropriately. > > On 03/10/10 06:52 PM, Ted Kim wrote: >> 4.2.3 Update IBD driver to use Brussels framework (CR 6883212) >> >> IPoIB tunables are managed today through /etc/system file or >> ibd.conf. >> For example, to modify the 'linkmode' of the IPoIB link, one needs to >> edit >> ibd.conf and reboot the system. We want to replace this interface so >> that >> user can manage these tunables using dladm(1M). >> >> 4.3 Proposal >> >> A micro/patch binding is asserted for this proposal. > > It isn't entirely obvious to me if a Solaris 10 consumer of IPoIB > would be required to do anything after this project is hypothetically > applied as a patch. Assuming of course that there is a patch for this > project (which seems somewhat unlikely given the implicit dependencies > on dozens of other dladm-related projects with Minor binding), one > requirement of patch binding is that there are no incompatible changes > introduced by the patch. Is that the case? > Posting the short summary of the offline discussion with Seb. This case depends on the number of other cases which have minor binding but this case does not introduce any new incompatible issues. As a part of this effort, we will integrate a upgrade script which will convert the old datalinks to the new model. We will update the case material to explicitly say about the dependencies and upgrade plan. regards, Sudhakar > On a related note, the materials don't explicitly state the case > dependencies, but I believe this case has a dependency on PSARC > 2007/429. Pulling on that case's string, the sweater that would need > to get backported into a patch gets quickly unraveled. ;-) > >> >> ----------------------------------------------------------------------- >> | InfiniBand Specific Link >> Properties | >> >> ----------------------------------------------------------------------- >> | linkmode | ON Consolidation >> Private | >> >> ----------------------------------------------------------------------- > > Given that it's Consolidation Private, I take it that this is a > private property (as defined by 2007/429). I was under the impression > that private properties had to be prefixed by a '_' character as > required by the Brussels framework. Is that not the case? > > -Seb -- ---------------------------------------------- Arise Awake Stop Not till the goal is reached ---------------------------------------------- Sudhakar Dindukurti Oracle America, Inc 17, Network Circle, Rm#3384 (650) 786 8563 (Work) Menlo Park, CA 94025 From Ted.Kim@sun.com Wed Mar 17 09:28:15 2010 Received: from sunmail6brm.central.sun.com (sunmail6brm.Central.Sun.COM [129.147.4.169]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2HGSEgG012264 for ; Wed, 17 Mar 2010 09:28:15 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail6brm.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2HGSD3j005093 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 17 Mar 2010 11:28:14 -0500 (CDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZF00K0HPR23D00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Mar 2010 09:28:14 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZF00D49PR1I490@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Mar 2010 09:28:13 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o2HGSDQe024440 for ; Wed, 17 Mar 2010 09:28:13 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KZF00E00PBVN900@fe-sfbay-10.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Mar 2010 09:28:13 -0700 (PDT) Received: from [129.153.3.74] ([unknown] [129.153.3.74]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KZF009S4PQGCQB0@fe-sfbay-10.sun.com>; Wed, 17 Mar 2010 09:27:53 -0700 (PDT) Date: Wed, 17 Mar 2010 09:27:11 -0700 From: "Ted H. Kim" Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <4BA06DDA.8040802@sun.com> Sender: Ted.Kim@sun.com To: Sebastien Roy , "Garrett D'Amore" Cc: Sudhakar Dindukurti , Ted Kim , PSARC-ext@sun.com, Cynthia.McGuire@sun.com, Peter.Memishian@sun.com Message-id: <4BA102DF.8000005@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> <4B9A7F41.5010000@oracle.com> <4BA06DDA.8040802@sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090910) Status: RO Content-Length: 1279 Seb & Garrett, Here is the summary of changes made in the case as a result of the discussion - 1. clarifying the micro/patch binding 2. mentioning the closure of the older case 3. changing linkmode to committed ------- case.txt ------- 56c56,61 < A micro/patch binding is asserted for this proposal. --- > A micro/patch binding is asserted for this proposal. This case > depends on number of other networking projects which have "minor" > binding (PSARC 2007/429, PSARC 2006/499, PSARC 2003/246, etc.). > But this case does not introduce any new incompatible changes. > The project provides a upgrade script which will automatically > convert the old datalinks to the new model during upgrade. 57a63,67 > There is an older open case which started discussing IPoIB dladm > (PSARC/2009/365). This case supersedes that one, and so the older > case has been closed. > > 335c345 < | linkmode | ON Consolidation Private | --- > | linkmode | committed | Once I am sure that no other changes are needed, I will repost the updated case. So please tell me if you feel anything else needs to be addressed. Garrett, please be sure you still feel comfortable with what has happened since your "+1". Thanks, -ted From gdamore@sun.com Wed Mar 17 09:29:58 2010 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2HGTwIK012294 for ; Wed, 17 Mar 2010 09:29:58 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2HGTvEQ022032 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 17 Mar 2010 09:29:57 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZF00J03PTXTU00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Mar 2010 10:29:57 -0600 (MDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZF00CDYPTW7470@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Mar 2010 10:29:57 -0600 (MDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o2HGTurl010020 for ; Wed, 17 Mar 2010 09:29:56 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KZF00300PPG2W00@fe-sfbay-10.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Mar 2010 09:29:56 -0700 (PDT) Received: from [192.168.251.11] ([unknown] [76.93.15.33]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KZF009GGPTSCQC0@fe-sfbay-10.sun.com>; Wed, 17 Mar 2010 09:29:52 -0700 (PDT) Date: Wed, 17 Mar 2010 09:29:48 -0700 From: "Garrett D'Amore" Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <4BA102DF.8000005@sun.com> Sender: Garrett.Damore@sun.com To: "Ted H. Kim" Cc: Sebastien Roy , Sudhakar Dindukurti , Ted Kim , PSARC-ext@sun.com, Cynthia.McGuire@sun.com, Peter.Memishian@sun.com Message-id: <4BA1037C.9080507@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> <4B9A7F41.5010000@oracle.com> <4BA06DDA.8040802@sun.com> <4BA102DF.8000005@sun.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.7) Gecko/20100131 Lightning/1.0b1 Thunderbird/3.0.1 Status: RO Content-Length: 1430 I am satisfied. My +1 remains. - Garrett On 03/17/10 09:27 AM, Ted H. Kim wrote: > Seb & Garrett, > > Here is the summary of changes made in the case as a result > of the discussion - > > 1. clarifying the micro/patch binding > 2. mentioning the closure of the older case > 3. changing linkmode to committed > > ------- case.txt ------- > 56c56,61 > < A micro/patch binding is asserted for this proposal. > --- > > A micro/patch binding is asserted for this proposal. This case > > depends on number of other networking projects which have "minor" > > binding (PSARC 2007/429, PSARC 2006/499, PSARC 2003/246, etc.). > > But this case does not introduce any new incompatible changes. > > The project provides a upgrade script which will automatically > > convert the old datalinks to the new model during upgrade. > 57a63,67 > > There is an older open case which started discussing IPoIB dladm > > (PSARC/2009/365). This case supersedes that one, and so the older > > case has been closed. > > > > > 335c345 > < | linkmode | ON Consolidation Private | > --- > > | linkmode | committed | > > Once I am sure that no other changes are needed, I will repost > the updated case. > > So please tell me if you feel anything else needs to be addressed. > Garrett, please be sure you still feel comfortable with what > has happened since your "+1". > > Thanks, > -ted > From Sebastien.Roy@sun.com Wed Mar 17 09:44:56 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2HGiunG012561 for ; Wed, 17 Mar 2010 09:44:56 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o2HGitvA008788 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 17 Mar 2010 10:44:55 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZF00L0FQIVJI00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Mar 2010 10:44:55 -0600 (MDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZF00CXHQIU6Z80@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Mar 2010 10:44:54 -0600 (MDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o2HGirHC015856 for ; Wed, 17 Mar 2010 16:44:53 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KZF00000PZSPU00@mail-amer.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Mar 2010 10:44:53 -0600 (MDT) Received: from [129.148.174.103] ([unknown] [129.148.174.103]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KZF006K2QIPZT30@mail-amer.sun.com>; Wed, 17 Mar 2010 10:44:50 -0600 (MDT) Date: Wed, 17 Mar 2010 12:42:01 -0400 From: Sebastien Roy Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <4BA102DF.8000005@sun.com> Sender: Sebastien.Roy@sun.com To: "Ted H. Kim" Cc: Sebastien Roy , "Garrett D'Amore" , Sudhakar Dindukurti , Ted Kim , PSARC-ext@sun.com, Cynthia.McGuire@sun.com, Peter.Memishian@sun.com Message-id: <4BA10659.2010504@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> <4B9A7F41.5010000@oracle.com> <4BA06DDA.8040802@sun.com> <4BA102DF.8000005@sun.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.8) Gecko/20100302 Lightning/1.0b1 Thunderbird/3.0.2 Status: RO Content-Length: 275 On 03/17/10 12:27 PM, Ted H. Kim wrote: > Here is the summary of changes made in the case as a result > of the discussion - > > 1. clarifying the micro/patch binding > 2. mentioning the closure of the older case > 3. changing linkmode to committed This looks good, +1. -Seb From Ted.Kim@sun.com Thu Mar 18 13:27:07 2010 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o2IKR7uF027109 for ; Thu, 18 Mar 2010 13:27:07 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o2IKR3nu006996 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 18 Mar 2010 13:27:07 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KZH00B66VH64L00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 18 Mar 2010 14:27:06 -0600 (MDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KZH008MKVH4I610@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 18 Mar 2010 14:27:04 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o2IKR4DP021337 for ; Thu, 18 Mar 2010 13:27:04 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KZH00300VDXJ300@fe-sfbay-09.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 18 Mar 2010 13:27:04 -0700 (PDT) Received: from [129.153.3.74] ([unknown] [129.153.3.74]) by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KZH004N5VGUJKC0@fe-sfbay-09.sun.com>; Thu, 18 Mar 2010 13:27:02 -0700 (PDT) Date: Thu, 18 Mar 2010 13:26:10 -0700 From: "Ted H. Kim" Subject: Re: IPoIB Administration Enhancement [PSARC/2010/085 FastTrack timeout 03/17/2010] In-reply-to: <4BA10659.2010504@sun.com> Sender: Ted.Kim@sun.com To: PSARC-ext@sun.com Cc: Sebastien Roy , "Garrett D'Amore" , Sudhakar Dindukurti , Cynthia.McGuire@sun.com, Peter.Memishian@sun.com Message-id: <4BA28C62.6000301@sun.com> MIME-version: 1.0 Content-type: multipart/mixed; boundary="Boundary_(ID_eV+PhyOE2Pj7yWZSA2fEww)" X-PMX-Version: 5.4.1.325704 References: <201003102352.o2ANqNm0012237@sac.sfbay.sun.com> <4B9A7F41.5010000@oracle.com> <4BA06DDA.8040802@sun.com> <4BA102DF.8000005@sun.com> <4BA10659.2010504@sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20090910) Status: RO Content-Length: 17894 This is a multi-part message in MIME format. --Boundary_(ID_eV+PhyOE2Pj7yWZSA2fEww) Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT Folks, It looks like the discussion has finished with all the issues addressed. Plus, the timer has expired, and there are two "+1" votes. So I am going to close the case as approved. I have attached the final version of the case below with the changes mentioned yesterday. -ted -- Ted H. Kim, PhD Sun/Oracle America Inc. ted.kim@sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1115 FAX --Boundary_(ID_eV+PhyOE2Pj7yWZSA2fEww) Content-type: text/plain; name=case.txt Content-transfer-encoding: 7BIT Content-disposition: inline; filename=case.txt 4. Technical Description 4.1 Acronyms HCA : Host Channel Adaptor P_Key : Partition Key IPoIB : IP over InfiniBand GUID : Global Unique Identifier IBTF : InfiniBand Transport Framework IBCM : InfiniBand Communication Manager SDP : Sockets Direct Protocol IBD : current IPoIB Solaris driver name ULP : Upper level Protocol 4.2 Requirements/Motivation --------------------------- 4.2.1 Consistent IBD device node naming across nodes in a clusters (Amber Road Requirement (CR 6864899)) AmberRoad clustering software requires that datalink name for a specific partition on multiple nodes (identical h/w configuration) to be same. For example, if datalink name on node 1 for a P_key 0x8001 on HCA 1 and port 1 is ibnet0, then the clustering software expects that datalink name for P_Key 0x8001 on HCA 1 and port 1 is also ibnet0 on the node 2 in the cluster. This is very difficult to achieve today. The problem is that IPoIB device name is constructed automatically by IPoIB driver by appending the instance number to the driver name (ex ibd0, ibd1, etc). So, the clustering software does not have any control on the IPoIB device name space. Also, IBTF framework does not guarantee that the same names across multiple nodes. 4.2.2 Problem diagnosis issue IPoIB (PSARC 2001/289, PSARC 2009/593 and PSARC 2007/636) is implemented as the ibd(7D) driver in Solaris. When customers face problems in bringing up the IBD devices on Solaris, they are advised to take number of steps to gather information about the IPoIB link that results in usability issues. Today, users need to run cfgadm(1M) to obtain the port GUID information, do 'ls-l' on the /dev/ibd* nodes to determine the IB partition that an IBD device belongs to etc. There are also instances when it is necessary to obtain the P_key corresponding to the IBD device and HCA port that it is bound to. So, we want to improve the process of gathering IPoIB related configuration information. 4.2.3 Update IBD driver to use Brussels framework (CR 6883212) IPoIB tunables are managed today through /etc/system file or ibd.conf. For example, to modify the 'linkmode' of the IPoIB link, one needs to edit ibd.conf and reboot the system. We want to replace this interface so that user can manage these tunables using dladm(1M). 4.3 Proposal A micro/patch binding is asserted for this proposal. This case depends on number of other networking projects which have "minor" binding. (PSARC 2007/429, PSARC 2006/499, PSARC 2003/246, etc.). But this case does not introduce any new incompatible changes. The project provides a upgrade script which will automatically convert the old datalinks to the new model during upgrade. There is an older open case which started discussing IPoIB dladm (PSARC/2009/365). This case supersedes that one, and so the older case has been closed. 4.3.0 IPoIB administration with dladm(1M) ---------------------------------------- This case proposes a new IPoIB administration mechanism for InfiniBand network datalinks using dladm(1M) command. Also, it proposes to add a consolidation private InfiniBand specific library API to libdladm(3LIB). The mechanism is very similar to the existing VLAN, VNIC, etc. administration dladm(1M) sub-commands. In the new model, two classes of IPoIB datalinks will exist: 1. Datalinks representing the physical IB ports, which will use the existing "phys" class that is used for Ethernet, WiFi, and other physical media. As with all "phys" class objects, the system will create these automatically. 2. Datalinks representing the administratively created partitions over "phys" IPoIB objects, which will use a new "part" class. Each IB partition datalink will be associated with a P_Key in a manner analogous to the way each Ethernet VLAN datalink is associated with a VLAN ID. Note that unlike other "phys" class objects, IB "phys" objects cannot send data (since IB requires a P_Key to send data) and thus cannot be plumbed. Example configuration used to explain the new IPoIB administration model -------------------------------------- | IB switch | -------------------------------------- | | | | | | Port1 | | Port2 Port1 | | Port2 -------------------------------------- | HCA1 HCA2 | | Node 1 | -------------------------------------- Port 2 of the HCA1 and Port 1 of the HCA2 are connected to IB switch and SM is running the switch. Each port is configured with two P_Key's (0xffff & 0x8001). 4.3.1 Physical datalinks ------------------------- One physical datalink will be created by default per port per HCA. These physical links serve as administrative & observability data points. These IB physical datalinks allow creating IB partitions over them similar to creating VNICs on Ethernet physical links or link aggregations. IB physical datalinks are not used for data transfers. So, the plumb and assigning a IB address are not supported on these links. In future, these physical datalinks can be used for 1) implementing the port level statistics 2) implementing port level snoop etc. Example 1. # dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE ibp0 InfiniBand up 8000 unknown ibp0 ibp1 InfiniBand down 8000 unknown ibp1 ibp2 InfiniBand down 2000 unknown ibp2 ibp3 InfiniBand up 2000 unknown ibp3 The state of the physical link is directly corresponds to state of the IB HCA port. As you might expect other generic sub-commands such as rename- link, show-link, delete-phys, etc. also work on IB datalinks. 4.3.2 IB partition Objects -------------------------- IB partition objects represent a new "part" class of datalink and these objects are managed using the new dladm(1M) sub-commands. All the new sub-command interfaces are similar to VLAN/VNIC dladm(1M) sub-commands. IB partition datalinks can be created on the top of IB physical links one per each P_Key on the port. These links are used for data transfers. The updated dladm(1M) man page describes the different commands available for managing the IB partition links. 4.3.2.1 create-part Creates a new IB partition Object with the specified datalink name. Example 2: Create a IB partition link for the P_Key 0x8001 on the top of ibp0 physical datalink # dladm create-part -l ibp0 -P 0x8001 p8001.ibp0 The above commands succeeds if the port is "up", P_Key is present on the port and IPoIB successfully completes its initialization. On subsequent reboots after successful initial creation, the partition will be available even if port is "down" or P_Key is absent, though the datalink state will be marked as "down". Example 3: Create an IB partition link for the P_Key 0x9000 on the top of ibp2 # dladm create-part -f -l ibp2 -P 0x9000 p9000.ibp2 The force option "-f" allows to create the IB partition even when the P_Key is not present or Port is down. The link state will be marked as down. The link state will be updated to "up" when P_Key is added to the port and port is activated. Example 4: Plumb and assign a IP address to IB partition p9000.ibp2 # ifconfig p9000.ibp2 plumb up # ifconfig -a p9000.ibp2: flags=1000843 mtu 2044 index 3 inet 1.1.1.1 netmask ff000000 broadcast 1.255.255.255 Example 5: Display the partition links using show-link command # dladm show-link LINK CLASS MTU STATE BRIDGE OVER p8001.ibp0 part 65520 unknown -- ibp0 p9000.ibp2 part 65520 down -- ibp2 4.3.2.2 delete-part Deletes a specified IB partition object Example 6: Delete the partition p8001.ibp0 # dladm delete-part p8001.ibp0 The above command deletes the partition. Example 7: Show the partition link information after deleting p8001.ibp0 # dldam show-part LINK PKEY OVER STATE FLAGS p9000.ibp2 9000 ibp2 down f--- 4.3.2.3 show-part Displays IB partition object information Example 8: Display the IB partition links information (below output is for the "part" operations in Example 2, 3 & 4) # dldam show-part LINK PKEY OVER STATE FLAGS p8001.ibp0 8001 ibp0 unknown ---- p9000.ibp2 9000 ibp2 down f--- The state of the IB partition link will be "unknown" after IB partition is created and before IB partition is plumbed. Once partition is plumbed the link state will be set to "up" when the link is ready to use. The state of the link will be set to "down" if 1) HCA port is down 2) P_Key is absent or 3) broadcast group is absent. 4.3.2.4 show-ib Displays IB specific information such as port#, port guid, etc. Example 9: Show IB specific information # dladm show-ib LINK HCAGUID PORTGUID PORT STATE PKEYS ibp0 3BA000100CD7C 3BA000100CD7D 1 down FFFF ibp1 3BA000100CD7C 3BA000100CD7E 2 down FFFF ibp3 5AD0000033634 5AD0000033636 2 up FFFF,8001 ibp2 5AD0000033634 5AD0000033635 1 up FFFF,8001 show-ib commands display only the physical links, port GUID, port# HCA GUID, and P_Key present on the port at the time of running the command. 4.3.3 IB partition object Administration Library The libdladm(3LIB) library is currently used to implement datalink administration for all the GLDv3 datalinks (VNIC, link aggregation, wireless, IP Tunnel, etc.). The library will be further enhanced to provide administrative interfaces of InfiniBand partition objects. All the new library extensions are similar to VLAN/VNIC libdladm extensions. IB specific functionality will be implemented by libdlib.c and its interface is provided via libdlib.h. dladm(1M) will use this library for managing IB partitions. The IB partition administration library provides a persistent repository which allows IB partition configuration to be stored across reboots. It uses the existing dladm(1M) /etc/dladm/datalink.conf repository to store the IB partition configuration. The list of InfiniBand specific extensions to libdladm library is given below. For more details of each API, see man pages in the PSARC materials directory. The man pages are only for PSARC review (not intended for public use). 4.3.3.1 dladm_part_create() Create a IB partition object 4.3.3.2 dladm_part_delete() Deletes a IB partition object 4.3.3.3 dladm_part_info() Returns IB partition Object attributes 4.3.3.4 dladm_ib_info() Returns IB specific attributes such as port number, port guid, and HCA GUID. 4.3.3.5 dladm_part_up() Brings up one or all the IB partition objects during every boot. 4.3.4 IBTF Extensions --------------------- Some of the IB ULP's such as SDP, IBCM, etc. walk the device tree and read "port-pkey", "hca-guid", and "port-guid" properties from the ibd(7D) device instance. With the new model, IB partitions are just objects (no longer device nodes). So, IB ULP's no longer can retrieve partition attributes from the device node. The following new IBTF extensions provide a mechanism to retrieve the partition attributes in the kernel. See ibt_get_part_attrs() man page in the PSARC case directory for more details about the following API. The man pages are only for PSARC review (not intended for public use). 4.3.4.1 ibt_get_part_attrs() Returns the attributes of a requested IB partition 4.3.4.2 ibt_get_all_part_attrs() Returns the attributes of all the active IB partitions 4.3.4.3 ibt_free_part_attrs() Frees the memory for partition attribute structure allocated by ibt_get_all_part_attrs() 4.3.5 Man page changes ---------------------- Updated man pages - published dladm(1M) datadm(1M) dat.conf(4) ibp(7D) (ibd(7d) name changed) New Man pages - internal only dladm_part_create(3dladm) dladm_part_delete(3dladm) dladm_part_info(3dladm) dladm_ib_info(3dladm) dladm_part_up(3dladm) ibt_get_part_attrs(9f) 4.3.6 Interface table --------------------- ----------------------------------------------------------------------- | Interface name | Commitment Level | ----------------------------------------------------------------------- | dladm(1M) extensions | ----------------------------------------------------------------------- | create-part | | | delete-part | committed | | show-ib | | | show-part | | ----------------------------------------------------------------------- | libdladm extensions (libdlib.h) | ----------------------------------------------------------------------- | dladm_part_create | | | dladm_part_delete | | | dladm_part_show | | | dladm_show_ib | ON Consolidation Private | | dladm_part_up | | | dladm_part_attr_t | | | dladm_ib_attr_t | | | DLADM_IBPART_FORCE_CREATE | | ----------------------------------------------------------------------- | InfiniBand Specific Link Properties | ----------------------------------------------------------------------- | linkmode | committed | ----------------------------------------------------------------------- | libdlmgmt.h | ----------------------------------------------------------------------- | DATALINK_CLASS_IBPART | ON Consolidation Private | ----------------------------------------------------------------------- | libdladm.h | ----------------------------------------------------------------------- | DLADM_STATUS_INVALID_PORT_INSTANCE | | | DLADM_STATUS_PORT_IS_DOWN | | | DLADM_STATUS_PKEY_NOT_PRESENT | | | DLADM_STATUS_PARTITION_EXISTS | ON Consolidation Private | | DLADM_STATUS_INVALID_PKEY | | | DLADM_STATUS_NO_HW_RESOURCE | | | DLADM_STATUS_INVALID_PKEY_TBL_SIZE | | ----------------------------------------------------------------------- | IBTF extensions | ----------------------------------------------------------------------- | ibt_get_part_attrs() | | | ibt_get_all_part_attrs() | ON Consolidation private | | ibt_free_part_attrs() | | | ibt_part_attr_t | | ----------------------------------------------------------------------- | New status codes (ibt_status_t updates) | ----------------------------------------------------------------------- | IBT_NO_SUCH_OBJECT | ON Consolidation private | ----------------------------------------------------------------------- 4.3.7 References --------------------- PSARC 2001/289 IP over InfiniBand PSARC 2007/636 IPoIB Conversion to GLDv3 PSARC 2009/593 IPoIB Connected Mode --Boundary_(ID_eV+PhyOE2Pj7yWZSA2fEww)-- From ted.h.kim@oracle.com Tue Apr 6 14:25:10 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o36LPARm006946 for ; Tue, 6 Apr 2010 14:25:10 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o36LP7uE040903 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Tue, 6 Apr 2010 15:25:09 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L0H00E3Z4TWE600@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 06 Apr 2010 15:25:08 -0600 (MDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L0H00EAZ4TU3500@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 06 Apr 2010 15:25:07 -0600 (MDT) Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o36LP6xD028283 for ; Tue, 06 Apr 2010 21:25:06 +0000 (GMT) Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o36EsLCb027884 for ; Tue, 06 Apr 2010 21:25:03 +0000 (GMT) Received: from abhmt021.oracle.com by acsmt353.oracle.com with ESMTP id 151802811270589035; Tue, 06 Apr 2010 14:23:55 -0700 Received: from [129.153.3.74] (/129.153.3.74) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 06 Apr 2010 14:23:54 -0700 Date: Tue, 06 Apr 2010 14:22:39 -0700 From: "Ted H. Kim" Subject: Re: IPoIB Administration Enhancement [ PSARC/2010/085 ] - minor case amendment To: psarc-ext@sun.com Cc: Cynthia McGuire , "MEMISHIAN,PETER" , "DINDUKURTI,SUDHAKAR" Message-id: <4BBBA61F.8030309@oracle.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A0B020B.4BBBA6B2.0067:SCFMA4539814,ss=1,fgs=0 User-Agent: Thunderbird 2.0.0.24 (X11/20100302) Status: RO Content-Length: 1582 Folks, The project team would like to make the following minor changes to the PSARC 2010/085 which arose from our project code review. We are filing this as a note to the original PSARC/2010/085 case file, since we think the changes are pretty much just cosmetic stuff. And along those lines, I consulted the original +1 votes on the case and got their agreement that this was okay by them. However, should others disagree, please pipe up. -ted ++++ 1) New consolidation API : dladm_free_ib_info() This is a wrapper function to free() routine which deallocates memory allocated by dladm_ib_info() routine. The dladm_ib_info() man page is updated with new API. Previously, the man page said to call "free()" to free the info struct. 2) Name changes These are thought to be more consistent with existing dladm or more clear in their naming. 2.1) Added "dia_" prefix to dladm_ib_attr_t structure fields and also changed dlinkid to dia_physlinkid and plinkid to dia_partlinkid. 2.2) DATALINK_CLASS_IBPART to DATALINK_CLASS_PART 2.3) DLADM_STATUS_NO_HW_RESOURCES to DLADM_STATUS_NO_IB_HW_RESOURCES Revised man pages with change bars (in materials directory): dladm_ib_info.3dladm (add dladm_free_ib_info(), name changes) dladm_part_info.3dladm (name changes) Original man pages saved under old.materials: dladm_ib_info.3dladm.ORIG dladm_part_info.3dladm.ORIG -- Ted H. Kim, PhD Sun/Oracle America Inc. ted.h.kim@oracle.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1115 FAX