From gww@eng.sun.com Tue Jan 15 17:02:29 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0G12S4Y006106 for ; Tue, 15 Jan 2008 17:02:29 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0G12JPd001431 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Wed, 16 Jan 2008 01:02:27 GMT Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JUP00803PK18G00@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 15 Jan 2008 18:02:25 -0700 (MST) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JUP00M26PK05SE0@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 15 Jan 2008 18:02:24 -0700 (MST) Received: from marduk.eng.sun.com (marduk.SFBay.Sun.COM [129.146.108.224]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0G12O1B055077; Tue, 15 Jan 2008 17:02:24 -0800 (PST) Received: from marduk.eng.sun.com (localhost [127.0.0.1]) by marduk.eng.sun.com (8.13.6+Sun/8.12.11) with ESMTP id m0G11JZV028646; Tue, 15 Jan 2008 17:01:19 -0800 (PST) Received: (from gww@localhost) by marduk.eng.sun.com (8.13.6+Sun/8.12.11/Submit) id m0G11JCi028645; Tue, 15 Jan 2008 17:01:19 -0800 (PST) Date: Tue, 15 Jan 2008 17:01:19 -0800 (PST) From: Gary Winiger Subject: PSARC/2008/035 - Sparks foreign backend support To: psarc-ext@sun.com Cc: Doug.Leavitt@sun.com Message-id: <200801160101.m0G11JCi028645@marduk.eng.sun.com> Content-transfer-encoding: 7BIT X-Sun-Charset: US-ASCII X-PMX-Version: 5.2.0.264296 Status: RO Content-Length: 9971 I'm sponsoring this case for Doug Leavitt and the Sparks and Duckwater project teams. Sparks: Name Service Switch 2 [PSARC/2005/133] was delivered into Nevada Build 50, see also the umbrella NSS2 (Sparks,Reno,Winchester,Duckwater) [PSARC/2006/247]. Duckwater: Simplified name services management (PSARC/2007/694) has not yet been heard by the ARC. This case wishes to bring forward a limited part of Duckwater to meet an immediate need to support clustering. The intent is to do so in a contracted way. A prototype contract is in the case directory. The proposed exported interfaces are Project Private and a Patch release binding is requested. The timer is set for 23 Jan, 2008. Gary.. =============================================================================== Background ========== The name service switch interfaces have been private [and mostly undocumented] since their development. While the private status of the interfaces has been restated multiple times, and explicitly documented as private as far back as 1994, this has not prevented uncontracted API usage outside the ON consolidation and unsupported usage by third parties. This is a well known problem that has been partly responsible for the lack of evolution of these APIs, until the initial delivery of the Sparks project [PSARC/2005/133], in snv_50. As a side note, the project team continues to learn about nsswitch backends that have been developed using the private APIs, both internal and external to Sun. It is still unclear how many "foreign" nsswitch backends actually exist. Realistically the number of "foreign" nsswitch backends is probably more than 10. The initial sparks project delivery has started down the path of documenting, updating and extending these interfaces with plans to eventually publish a set of public nsswitch interfaces. The sparks effort currently preserves backwards compatibility by keeping the old, backwards compatible, nsswitch interface code in it's original libc location. The new redesigned nsswitch is located in nscd. As discussed in PSARC/2005/133, the public APIs send requests to the new nsswitch which in turn can return a request back to the libc nsswitch via the TRYLOCAL protocol, if the new switch determines that the request must be executed by the old nsswitch to preserve backwards compatibility. Currently the name service switch has two classes of name service switch backends. The first class of backends are "foreign" or pre PSARC/2005/133 backends. Any backend that has not been updated to follow the behaviors documented in PSARC/2005/133 is considered foreign. In other words, any backend that is making unsupported or uncontracted use of the private nsswitch APIs. Foreign backends will almost always core dump the new name service switch in nscd and likely cause a disruption of service. The primary reason for nscd's inability to use a foreign backend is because prior to PSARC/2005/133 the data marshaling interfaces were undocumented, inconsistent and prone to usage error. The interface usage was very inconsistent even amongst the ON supported backends. This was a primary root cause for many naming service lookup and consistency bugs prior to Sparks. PSARC/2005/133 documented the expected behavior and spent a considerable part of the development effort towards updating all ON backends so they followed the newly documented behavior. All non-ON backends are considered 'foreign' to the new name service switch engine. The new name service switch engine returns requests that need to use a foreign backend to the original (obsolete) libc name service switch using the TRYLOCAL protocol for local processing on the client. This preserves full backwards compatibility with existing code that is using the private interfaces. However, this means databases, such as hosts or passwd, that specify a foreign backend in nsswitch.conf are not cached. The second class of backends are backends that are known have been updated to support the new interfaces. Nscd has a built-in "knowledge" of a core set of backends that are delivered in unison with nscd in the ON consolidation. These backends are required to always adhere to all the current latest [PSARC/2005/133 or later] nsswitch specifications. These are the backends that are used to test and qualify the name service switch in ON at each build/delivery via PIT and other methods. The current well known/built-in backends are those located in the ON consolidation tree: usr/src/lib/nsswitch. They currently include compat, dns, files, ldap, mdns, nis, nisplus user Problem ======= There is at least one Sun non-ON consolidation name service switch backend, nss_cluster, that should have appropriate private interface contracts with the naming service team, but doesn't, and that needs to be loadable by nscd, so nscd can cache appropriate hostname lookups. With the introduction of Sparks into s10u4, nscd has stopped using the nss_cluster backend, and has stopped caching hostname lookups when nss_cluster is one of the sources for hosts lookups. This causes a significant performance degradation on all clustering systems because the cluster nsswitch backend is used to map cluster private host names [for each node of a cluster] to the cluster public name, and host name lookups are used heavily in the clustering subsystem code base. The following CRs document this issue: CR 6632298 network/nscd nscd doesn't cache hosts for cluster after sparks project (120011-14) CR 6644077 network/nsswitch nscd rejects - foreign nsswitch backends Nscd needs a way to detect a non-ON, nsswitch backend that adheres to PSARC/2005/133 or later so that nscd can load, use and cache the results from that backend. Solution ======== This fast track specifies a new versioning API that will be delivered and used by nscd so that nscd can detect and load updated non-ON name service switch backends. Additionally, in order to close CR 6632298, the following tasks will need to be completed: 1) An interface contract needs to be written and agreed to between the cluster team and the netrep team to contract all the nsswitch private interfaces, including PSARC/2005/133 and this fasttrack. 2) The cluster team needs to revise and deliver and updated nss_cluster backend, to comply with the definitions set out by the contracted interfaces. A prototype contract is part of this case, the deliver of the cluster changes are outside the scope of this case. Versioning Details ================== The Duckwater project [PSARC/2007/694] will extend the nsswitch interfaces defined by PSARC/2005/133 with new set of versioning interfaces and extensions to the existing finder APIs so that the name service switch and name service switch configuration tools can detect extended characteristics of a nsswitch backend, including new configuration APIs. The intent of the versioning and finder APIs are to provide efficient, upward compatible and extensible APIs that allow for the addition of new XbyYs [get/put/ent etc.], new configuration interfaces (duckwater) and other extendable future nscd APIs. The specifics of the new finder interfaces will be detailed in PSARC/2007/694. These new interfaces will detect a nsswitch backend that has been versioned by looking for an externally defined version global symbol [dlsym retrievable] of the form: _nss_[backend-name]_version where backend-name corresponds to the backend name used in nsswitch.conf as in: _nss_nis_version, _nss_files_version etc. The contents of the variable are treated as a castable (void *) pointer in the native pointer size for the compiled library. This fast track defines a strict subset of the PSARC/2007/694 interfaces in the following manner: Existence of the symbol in a nsswitch backend with a value of NULL implies compliance with the specifications of PSARC/2005/133. If this NULL valued symbol exists, nscd will treat the nsswitch backend as eligible for being dlopen'd and used per PSARC/2005/133 specifications. The results from lookups will be cached along with other lookup results as applicable. Duckwater [PSARC/2007/694] will specify the additional behaviors when the value is non NULL. For this case specifically the solution proposed is: 1) to recognize _nss_[backend-name]_version loader symbols 2) to interpret a NULL value of such symbols to mean compliant with Sparks [PSARC/2005/133] 3) to contract with the cluster group. For the purposes of ARC classification, and because all the other nsswitch interfaces are currently classified as either Project Private or Committed Private, this interface is defined as project private, with a request for patch binding approval. Contracts against this interface will be signed as needed. [see above] This fast track requests that the discussion to reclassify any/all the nsswitch interfaces be held during either the PSARC/2007/694 full ARC review or until the future "sparks phase 2" ARC review. The Sparks team expects to re-classify this nsswitch interface into a stable public interface at some point in the future. Other Notes =========== It should be noted, that both Duckwater and Sparks are Open OpenSolaris development efforts, so this API and fast track are being discussed part of the respective open development efforts with the details published on the appropriate project sites and e-mail aliases. It is known that other unsupported third party nsswitch backends have been using the nsswitch private interfaces and their backends currently not being cached by nscd. Given that the Sparks and Duckwater projects are open development efforts, and the Sparks documentation is available on OpenSolaris, it is expected that some third parties might choose to use these private interfaces in an unsupported manner as well. Until these APIs are designated as public, third parties that choose to use this interface continue do so at their own risk and expense. From sacadmin Tue Jan 15 16:56:50 2008 Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0G0unqc005987; Tue, 15 Jan 2008 16:56:49 -0800 (PST) Received: (from gww@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id m0G0unW9005983; Tue, 15 Jan 2008 16:56:49 -0800 (PST) Date: Tue, 15 Jan 2008 16:56:49 -0800 (PST) From: Gary Winiger Message-Id: <200801160056.m0G0unW9005983@sac.sfbay.sun.com> To: PSARC-record@sac.sfbay.sun.com Cc: Doug.Leavitt@sun.com Subject: Sparks foreign backend support [PSARC/2008/035 FastTrack timeout 01/23/2008] Status: RO Content-Length: 566 Template Version: @(#)sac_nextcase 1.64 07/13/07 SMI This information is Copyright 2008 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: Sparks foreign backend support 1.2. Name of Document Author/Supplier: Author: Coug Leavitt 1.3 Date of This Document: 15 January, 2008 4. Technical Description See the case directory for more detail 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From gww@eng.sun.com Wed Jan 23 10:56:00 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0NItxB4027093 for ; Wed, 23 Jan 2008 10:56:00 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0NItsVS026994 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Wed, 23 Jan 2008 18:55:59 GMT Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JV400B011XAS900@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 23 Jan 2008 11:55:58 -0700 (MST) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JV4003HN1X9EI60@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 23 Jan 2008 11:55:58 -0700 (MST) Received: from marduk.eng.sun.com (marduk.SFBay.Sun.COM [129.146.108.224]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0NItvAv056805; Wed, 23 Jan 2008 10:55:57 -0800 (PST) Received: from marduk.eng.sun.com (localhost [127.0.0.1]) by marduk.eng.sun.com (8.13.6+Sun/8.12.11) with ESMTP id m0NIsihU019573; Wed, 23 Jan 2008 10:54:44 -0800 (PST) Received: (from gww@localhost) by marduk.eng.sun.com (8.13.6+Sun/8.12.11/Submit) id m0NIsiMi019572; Wed, 23 Jan 2008 10:54:44 -0800 (PST) Date: Wed, 23 Jan 2008 10:54:44 -0800 (PST) From: Gary Winiger Subject: Re: PSARC/2008/035 - Sparks foreign backend support To: gww@eng.sun.com, psarc-ext@sun.com Cc: Doug.Leavitt@sun.com Message-id: <200801231854.m0NIsiMi019572@marduk.eng.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 Status: RO Content-Length: 58 This case was approved at today's PSARC meeting. Gary..