Subject: More IB Failover Support [PSARC/2009/183 FastTrack timeout 03/27/2009] To: PSARC-ext@Sun.Com Cc: ted.kim@sun.com, rajkumar.sivaprakasam@sun.com Bcc: one-pager-list@sac.sfbay one-pager-log@sac.sfbay sac-bar@sac.sfbay Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: More IB Failover Support 1.2. Name of Document Author/Supplier: Author: Rajkumar Sivaprakasam 1.3 Date of This Document: 20 March, 2009 4. Technical Description More IB Failover Support ------------------------ 4.1 Background This case introduces more InfiniBand (IB) options to the interfaces of the InfiniBand Transport Framework (IBTF, PSARC/2002/132 and follow-on cases) to support failover scenarios in Upper Level Protocols (ULPs). The items in this case are the Port Change Event, Client Reregistration Event, and the IBMF Timeout Error. Further, two minor corrections are tacked on to this case as well: adding a NO_SLEEP flag to ibt_alloc_lkey from PSARC/2009/060, and correcting the name of the Device Manager Agent in PSARC/2009/111. The expected consumers of the Port Change Event are IBTF itself and IPonIB (PSARC/2001/289 and PSARC/2007/636). The Client Registration Event would be consumed internally by IBTF. The IBMF Timeout Error would be used by IPonIB. 4.2 Proposal The propsoal is to make additions to the IBTF Channel Interface (CI) for HCA drivers and the Transport Interface (TI) for IB ULPs. All interface additions and changes in the proposal have a micro/patch binding. Transport Interface (ON Consolidation Private) ibt_alloc_lkey: add NOSLEEP flag ibt_async_code_t: add Port Change event code ibt_async_event_t: add Port Change event flags field ibt_clnt_class_t: correct DM agent name ibt_hca_flags2_t: add Port Change support flag ibt_port_change_t: new Port Change flags type definition ibt_status_t: add IBMF timeout error code Channel Interface (ON Consolidation Private) ibc_alloc_lkey: add NOSLEEP flag ibt_async_code_t: add Port Change and Client Reregistration event code ibt_async_event_t: add Port Change event flags field ibt_clnt_class_t: correct DM agent name ibt_hca_flags2_t: add Port Change support flag ibt_port_caps_t: add Client Reregistration flag ibt_port_change_t: new Port Change flags type definition All of these changes are part of the v3 IBTF ABI first introduced in PSARC/2008/630. Copies of all modified man pages are on the materials direction (see section 4.3 below). A. Port Change Event In the IB architecture, the Subnet Manager (SM) is responsible for initializing and sometimes modifying the settings on each Host Channel Adapter (HCA) port. In the 1.2.1 IB spec, a new "Port Change" event was added to signal when certain attributes of an HCA port change (see section 11.6.3.3 in [1]). This event is distinct from the earlier events which signalled the port going up or down. Of particular relevance is the signalling of changes to various parameters (SM LID and SM SL) used to contact the Subnet Administration (SA), which is the way to query IB subnet configuration. These parameters would change during SM failover. In Solaris, these change events would be consumed by IBMF (PSARC/2002/607). P_Key table changes would be consumed by IBTF (which caches the P_Key tables) and IPonIB. Subnet Timeout changes would potentially also be used by IBMF and the IB Device Manager. Finally, the IsClientReregistrationSupported indicator would be used by IBTF in support of the feature described in section B below. Add a flag indicating port change event support to ibt_hca_flags2_t: IBT_HCA2_PORT_CHANGE = 1 << 4 Add a new IBT_PORT_CHANGE_EVENT type to ibt_async_code_t: IBT_PORT_CHANGE_EVENT = 0x100000 Add a new ibt_port_change_t flags definition: typedef enum ibt_port_change_e { IBT_PORT_CHANGE_SGID = 0x000001, /* SGID table */ IBT_PORT_CHANGE_PKEY = 0x000002, /* P_Key table */ IBT_PORT_CHANGE_SM_LID = 0x000004, /* Master SM LID */ IBT_PORT_CHANGE_SM_SL = 0x000008, /* Master SM SL */ IBT_PORT_CHANGE_SUB_TIMEOUT = 0x000010, /* Subnet Timeout */ IBT_PORT_CHANGE_SM_FLAG = 0x000020, /* IsSMDisabled bit */ IBT_PORT_CHANGE_REREG = 0x000040 /* IsClientReregSupport */ } ibt_port_change_t; Add a flags field to the ibt_async_event_t structure: ibt_port_change_t ev_port_flags; /* NEW: Port Change flags */ Man page changes: ibt_async_handler.9e, ibc_async_handler.9f, ibt_hca_attr_t.9s B. Client Reregistration Event In IBTF, certain events are distributed to clients via the ibt_sm_notice_handler_t(9E). These events come from "subscriptions" to the SM/SA. When an SM restarts or fails over, subscriptions may not be preserved (i.e. the new SM doesn't know about the old subscriptions). When this happens, nodes must again sign up for subscriptions when prompted by the "Client Reregistration" event (see 11.6.3.3 in [1]). Since IBTF subscribes to events once for the whole node, this event is consumed by IBTF itself. ULPs do not need to reregister individually. Add a new flag added to ibt_port_caps_t showing client reregistration support: IBT_PORT_CAP_CLNT_REREG = 1 << 5 /* Client Rereg */ Add a new client reregistration event type to ibt_async_code_t: IBT_CLNT_REREG_EVENT = 0x200000 Man page changes: ibc_async_handler.9f, ibc_hca_portinfo_t.9s C. IBMF Timeout Error Certain IBTF functions can fail when querying the SA is unsuccessful due to timeout (i.e. no response). However, up to this point, this failure could not be explicitly distinguished. While for some ULPs, this distinction was not interesting; in others, masking this particular failure impeded triggering recovery or failover actions. Various functions using the SA now possibly return an error code indicating SA timeout. Add IBMF timeout error code to ibt_status_t: IBT_IBMF_TIMEOUT = 10 Add this error code to man pages for functions that use the SA, including ibt_get_paths() and variants. Complete list follows - Man page changes: ibt_get_alt_path.9f, ibt_get_companion_port_gids.9f, ibt_get_ip_alt_path.9f, ibt_get_ip_paths.9f, ibt_get_paths.9f, ibt_gid_to_node_info.9f, ibt_join_mcg.9f, ibt_leave_mcg.9f, ibt_query_mcg.9f, ibt_register_service.9f, ibt_set_alt_path.9f D. Alloc L_Key NOSLEEP flag fix PSARC/2009/060 incorrectly described one aspect of the Allocate L_Key operation (used by Work Request Registration). The man pages left out the SLEEP/NO_SLEEP flags. Man page changes: ibc_alloc_lkey.9e, ibt_alloc_lkey.9f E. Device Management Agent name PSARC/2009/111 incorrectly named the IB Device Management Agent as "IBT_DMA". It is actually "IBT_DM_AGENT" (enum value of ibt_clnt_class_t). Man page changes: ibt_attach.9f, ibt_clnt_modinfo_t.9s 4.3 Summary of changes by man page Man Page Disposition Reasons for change ------------------------------------------------------------------ ibc_alloc_lkey.9e changed D ibt_async_handler.9e changed A ibc_async_handler.9f changed A, B ibt_alloc_lkey.9f changed D ibt_attach.9f changed E ibt_get_alt_path.9f changed C ibt_get_companion_port_gids.9f changed C ibt_get_ip_alt_path.9f changed C ibt_get_ip_paths.9f changed C ibt_get_paths.9f changed C ibt_gid_to_node_info.9f changed C ibt_join_mcg.9f changed C ibt_leave_mcg.9f changed C ibt_query_mcg.9f changed C ibt_register_service.9f changed C ibt_set_alt_path.9f changed C ibc_hca_portinfo_t.9s changed B ibt_clnt_modinfo_t.9s changed E ibt_hca_attr_t.9s changed A 4.4 References [1] InfiniBand Architecture Specification Volume 1, Release 1.2.1. InfiniBand Trade Association, 2007. http://www.infinibandta.org/members/spec/V1r1_2_1.Release_12062007.zip (requires IBTA member login) 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open