From dhain@sac.sfbay.sun.com Thu Oct 29 13:00:15 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9TK0Fnf005741 for ; Thu, 29 Oct 2009 13:00:15 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n9TK09lP039946; Thu, 29 Oct 2009 14:00:15 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSA00B0BKWEOC00@nwk-avmta-1.sfbay.Sun.COM>; Thu, 29 Oct 2009 13:00:14 -0700 (PDT) Received: from dm-sfbay-01.sfbay.sun.com ([129.145.155.118]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSA00HOBKWESZE0@nwk-avmta-1.sfbay.Sun.COM>; Thu, 29 Oct 2009 13:00:14 -0700 (PDT) Received: from sac.sfbay.sun.com (sac.SFBay.Sun.COM [129.146.226.132]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id n9TK0C2f018270; Thu, 29 Oct 2009 13:00:12 -0700 (PDT) Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n9TK0AOV005736; Thu, 29 Oct 2009 13:00:10 -0700 (PDT) Received: (from dhain@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id n9TK09T7005732; Thu, 29 Oct 2009 13:00:10 -0700 (PDT) Date: Thu, 29 Oct 2009 13:00:10 -0700 (PDT) From: Daniel Hain Subject: LDOM-SunCluster suspend callbacks [PSARC/2009/589 FastTrack timeout 11/05/2009] To: PSARC-ext@sun.com Cc: Alexandre.Chartre@sun.com, Haik.Aftandilian@sun.com, ldom-mig-dev@sun.com Message-id: <200910292000.n9TK09T7005732@sac.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 6602 I'm sponsoring this fasttrack for Haik Aftandilian. This is an Open case seeking Patch binding (for backport to S10). Timeout on 11/05/2009. A copy of the proposed contract is in the case directory. Dan Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: LDOM-SunCluster suspend callbacks 1.2. Name of Document Author/Supplier: Author: Haik Aftandilian 1.3 Date of This Document: 29 October, 2009 4. Technical Description Introduction ------------ Solaris Cluster (SC) runs in LDoms guest domains and provides infrastructure to make applications deployed in the guest domains HA. SC cluster nodes monitor each other via (so called) heartbeats, as well associated hardware. SC manages Storage devices in a way which is specific to the server it is running on, such as performing SCSI reservations, which are meaningful only from a specific physical HBA. LDoms guest domains can be migrated from one server to another with the LDoms "Warm Migration" feature. During the migration, the domain being migrated is suspended. While a domain is suspended, which can be several minutes, the domain is totally inactive and not responsive to any requests. Thus, if a domain running SC is migrated from one server to another, other cluster nodes need to be made aware of this fact so that they can suspend monitoring of the cluster node under migration. Additionally, cluster nodes need to act co-operatively to make sure the SCSI reservations on storage devices are also migrated correctly. The proposed callbacks would allow SC to perform these tasks, thereby enabling a seamless migration of the LDoms guest domain from an end user perspective. References ---------- 1. Suspend Domain Service http://sac.eng/Archives/CaseLog/arc/FWARC/2009/559/ 2. Current list of Sun Cluster/ON contract cases in use in Solaris 10. /ws/osc-gate/usr/src/uts/sparc/cl/imported_symbols.private.Sol_10 3. Example of an existing Sun Cluster/ON contract case. http://sac.eng/Archives/CaseLog/arc/PSARC/2005/602/ Overview -------- In the Solaris kernel, hooks/callback functions will be run before and after the domain is suspended. A single callback will be made to SC before the suspension and a single callback will be made after the resumption. Note that the use of "suspend" in this contract only applies to suspend operations initiated by LDoms infrastructure using the suspend domain service on sun4v guest domains. And that today, suspend operations are only performed to facilitate LDoms domain migration. These suspend operations are entirely distinct from CPR suspend operations. Commitment level for all the interfaces: Contracted Project Private Interface Details ----------------- When SC is loaded and wishes to receive suspend notifications, it will set the callback function pointers to point to SC functions that handle the notifications. When setting these callbacks, the cl_suspend_error_decode callback should be set first, then the cl_suspend_post_callback, and then the cl_suspend_pre callback. The cl_suspend_pre_callback and cl_suspend_post_callback will never be invoked concurrently and solaris will wait indefinitely for the callbacks to return. Pre-suspend callback: int (*cl_suspend_pre_callback)(void); Called before the domain is suspended. This serves to notify SC that this domain is in the process of being suspended. SC should return 0 if it successfully suspended monitoring of this domain. If a failure occurred which should prevent the guest domain from being suspended and possibly migrated, or if SC can not support a migration at this time, SC should return a non-zero error code. If the cl_suspend_pre_callback returns an error code, the suspension will aborted. The intent is for this error to be sent back to the domain manager and then used to build an error message informing the user why the migration could not be completed. Post-suspend callback: int (*cl_suspend_post_callback)(void); Called after the guest domain has been resumed following a successful suspension. It is also called after a failed suspension attempt as well as a canceled suspension attempt. i.e., it is possible that this function will be called when the guest domain was suspended and then resumed without being migrated (as a result of a failure or cancellation). It can also be called even when the guest domain was never suspended (due to a failure before the suspension) and therefore never migrated. If the callbacks are set after a suspend operation is already in progress, since the pre callback is set after the post callback, it is also possible that this function will be called after a suspension even when the cl_suspend_pre_callback was not called. Therefore, SC should not consider it an error if cl_suspend_post_callback is called before cl_suspend_pre_callback without a corresponding call to cl_suspend_pre_callback. SC should return 0 if it successfully resumed monitoring of this domain. If a failure occurred which prevents the guest domain from resuming normal activity in the cluster, a non-zero error value should be returned. The error will be sent back to the domain manager which will display an error message informing the user that an error occurred after the migration and that manual inspection and recovery may be required for the domain to resume normal operation. The domain will have been resumed and Solaris and applications will be running. Error code decode callback: const char *(*cl_suspend_error_decode)(int); Called at any time to convert an error code returned from the cl_suspend_pre_callback or cl_suspend_post_callback into a descriptive error string suitable for use in an error message presented to the user. Returns a NULL- terminated statically allocated string of length less than or equal to 256 including the NULL terminator. The caller will consider this string immutable and will not modify it or deallocate it. This function may return NULL. When the cl_suspend_pre_callback or cl_suspend_post_callback return an error, cl_suspend_error_decode will be used to obtain an error message string that corresponds to the error. i.e., cl_suspend_error_decode will be called and its argument will be an error code returned from cl_suspend_pre_callback or cl_suspend_post_callback. 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From Sebastien.Roy@sun.com Wed Nov 4 08:16:20 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nA4GGK46002574 for ; Wed, 4 Nov 2009 08:16:20 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id nA4GGIil003482; Wed, 4 Nov 2009 08:16:20 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSL0081TEJ7F700@brm-avmta-1.central.sun.com>; Wed, 04 Nov 2009 09:16:19 -0700 (MST) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSL00NPUEJ72090@brm-avmta-1.central.sun.com>; Wed, 04 Nov 2009 09:16:19 -0700 (MST) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id nA4GGI6X022376; Wed, 04 Nov 2009 16:16:18 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KSL00K00D4SR900@mail-amer.sun.com>; Wed, 04 Nov 2009 09:16:18 -0700 (MST) Received: from [129.148.174.103] ([unknown] [129.148.174.103]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KSL005BXEIVZVF0@mail-amer.sun.com>; Wed, 04 Nov 2009 09:16:08 -0700 (MST) Date: Wed, 04 Nov 2009 11:13:33 -0500 From: Sebastien Roy Subject: Re: LDOM-SunCluster suspend callbacks [PSARC/2009/589 FastTrack timeout 11/05/2009] In-reply-to: <200910292000.n9TK09T7005732@sac.sfbay.sun.com> Sender: Sebastien.Roy@sun.com To: Daniel Hain Cc: PSARC-ext@sun.com, ldom-mig-dev@sun.com, Haik.Aftandilian@sun.com Message-id: <1257351213.26975.27.camel@strat> Organization: Sun Microsystems MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910292000.n9TK09T7005732@sac.sfbay.sun.com> Status: RO Content-Length: 302 This architecture is not extensible. Is it inconceivable that other subsystems may be interested in these events in the future (or even today)? There is no provision here for multiple callbacks, and the callbacks have names that are clustering specific. Was any consideration given to this? -Seb From Haik.Aftandilian@sun.com Wed Nov 4 09:57:55 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nA4HvsDI007244 for ; Wed, 4 Nov 2009 09:57:55 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id nA4Hvl3w024501; Thu, 5 Nov 2009 01:57:53 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSL00I03J8FO800@brm-avmta-1.central.sun.com>; Wed, 04 Nov 2009 10:57:51 -0700 (MST) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSL00EOQJ8FNX60@brm-avmta-1.central.sun.com>; Wed, 04 Nov 2009 10:57:51 -0700 (MST) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id nA4HvpFi015659; Wed, 04 Nov 2009 17:57:51 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KSL00C00IWXA500@mail-amer.sun.com>; Wed, 04 Nov 2009 10:57:51 -0700 (MST) Received: from [129.153.89.31] ([unknown] [129.153.89.31]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KSL008YHJ8EIC90@mail-amer.sun.com>; Wed, 04 Nov 2009 10:57:51 -0700 (MST) Date: Wed, 04 Nov 2009 09:58:40 -0800 From: Haik Aftandilian Subject: Re: LDOM-SunCluster suspend callbacks [PSARC/2009/589 FastTrack timeout 11/05/2009] In-reply-to: <1257351213.26975.27.camel@strat> Sender: Haik.Aftandilian@sun.com To: Sebastien Roy Cc: Daniel Hain , PSARC-ext@sun.com, ldom-mig-dev@sun.com Message-id: <4AF1C0D0.9030706@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910292000.n9TK09T7005732@sac.sfbay.sun.com> <1257351213.26975.27.camel@strat> User-Agent: Thunderbird 2.0.0.23 (X11/20091013) Status: RO Content-Length: 2010 Sebastien Roy wrote: > This architecture is not extensible. Is it inconceivable that other > subsystems may be interested in these events in the future (or even > today)? There is no provision here for multiple callbacks, and the > callbacks have names that are clustering specific. Was any > consideration given to this? Yes, we considered adding a callback registration API so that interested parties could register their own suspend callbacks, but decided against that because, at present, Sun Cluster is the only interested party that needs in-kernel notifications. The callbacks were specifically requested by Sun Cluster and are only to be used by Sun Cluster. This is the model used by Sun Cluster and ON in other kernel subsystems. We also need to control which callbacks occur. At present we don't want other kernel subsystems to register callbacks that could cause a suspend operation to fail. Users expect migrations to succeed when all the conditions outlined in the LDom documentation are met. Today, a suspend (initiated by the HV as part of a domain migration) has no Solaris hooks and Solaris is not aware of a suspend/resume. This process is being changed so that Solaris will be aware of suspend/resume, but we want to continue that model as much as possible, only making an exception for Sun Cluster here. Lastly, in LDoms, a suspend operation only occurs to permit a domain migration and this is initiated by the management software on a separate control domain. The LDom management software is not ready to account for an arbitrary number of pre/post callbacks which could take any length of time. Certain operations are blocked when a migration is in progress and so this affects usability. In the event that the notification scheme needs to expand in the future, we will address the need. This could be with an extensible callback mechanism in the kernel or an API that includes notifications issued by the domain manager on the control domain. Thanks, Haik From randy.fishel@sun.com Wed Nov 4 10:12:49 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nA4ICljB008703 for ; Wed, 4 Nov 2009 10:12:48 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id nA4ICirk064020; Wed, 4 Nov 2009 11:12:46 -0700 (MST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSL00JKZJX8KZ00@brm-avmta-1.central.sun.com>; Wed, 04 Nov 2009 11:12:44 -0700 (MST) Received: from grimmy.eng.sun.com ([129.146.108.114]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSL00E35JVKNTC0@brm-avmta-1.central.sun.com>; Wed, 04 Nov 2009 11:11:44 -0700 (MST) Received: from grimmy.eng.sun.com (localhost [127.0.0.1]) by grimmy.eng.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nA4IAPvj001531; Wed, 04 Nov 2009 10:10:25 -0800 (PST) Received: from localhost (randyf@localhost) by grimmy.eng.sun.com (8.13.8+Sun/8.13.8/Submit) with ESMTP id nA4IAPMG001528; Wed, 04 Nov 2009 10:10:25 -0800 (PST) Date: Wed, 04 Nov 2009 10:10:25 -0800 (PST) From: Randy Fishel Subject: Re: LDOM-SunCluster suspend callbacks [PSARC/2009/589 FastTrack timeout 11/05/2009] In-reply-to: <200910292000.n9TK09T7005732@sac.sfbay.sun.com> X-X-Sender: randyf@grimmy To: Daniel Hain Cc: PSARC-ext@sun.com, Alexandre.Chartre@sun.com, Haik.Aftandilian@sun.com, ldom-mig-dev@sun.com Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910292000.n9TK09T7005732@sac.sfbay.sun.com> X-Authentication-warning: grimmy.eng.sun.com: randyf owned process doing -bs User-Agent: Alpine 1.10 (GSO 962 2008-03-14) Status: RO Content-Length: 258 > These suspend operations are entirely distinct from > CPR suspend operations. > Why? It would seem to me that the problems are the same for both, and that it would be worthwhile to leverage this work, than to roll a private solution. ---- Randy From Sebastien.Roy@sun.com Wed Nov 4 10:26:41 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nA4IQfa7009540 for ; Wed, 4 Nov 2009 10:26:41 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id nA4IQdUL009878; Wed, 4 Nov 2009 10:26:40 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSL00L05KKG2400@nwk-avmta-2.sfbay.sun.com>; Wed, 04 Nov 2009 10:26:40 -0800 (PST) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSL00HA3KKFVMB0@nwk-avmta-2.sfbay.sun.com>; Wed, 04 Nov 2009 10:26:39 -0800 (PST) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id nA4IQduN009623; Wed, 04 Nov 2009 18:26:39 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KSL00200JW56900@mail-amer.sun.com>; Wed, 04 Nov 2009 11:26:39 -0700 (MST) Received: from [129.148.174.103] ([unknown] [129.148.174.103]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KSL00MWEKK2KNE0@mail-amer.sun.com>; Wed, 04 Nov 2009 11:26:27 -0700 (MST) Date: Wed, 04 Nov 2009 13:23:52 -0500 From: Sebastien Roy Subject: Re: LDOM-SunCluster suspend callbacks [PSARC/2009/589 FastTrack timeout 11/05/2009] In-reply-to: <4AF1C0D0.9030706@sun.com> Sender: Sebastien.Roy@sun.com To: Haik Aftandilian Cc: Daniel Hain , PSARC-ext@sun.com, ldom-mig-dev@sun.com Message-id: <1257359032.26975.57.camel@strat> Organization: Sun Microsystems MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910292000.n9TK09T7005732@sac.sfbay.sun.com> <1257351213.26975.27.camel@strat> <4AF1C0D0.9030706@sun.com> Status: RO Content-Length: 2185 On Wed, 2009-11-04 at 09:58 -0800, Haik Aftandilian wrote: > Sebastien Roy wrote: > > This architecture is not extensible. Is it inconceivable that other > > subsystems may be interested in these events in the future (or even > > today)? There is no provision here for multiple callbacks, and the > > callbacks have names that are clustering specific. Was any > > consideration given to this? > > Yes, we considered adding a callback registration API so that interested > parties could register their own suspend callbacks, but decided against > that because, at present, Sun Cluster is the only interested party that > needs in-kernel notifications. The callbacks were specifically requested > by Sun Cluster and are only to be used by Sun Cluster. This is the model > used by Sun Cluster and ON in other kernel subsystems. > > We also need to control which callbacks occur. At present we don't want > other kernel subsystems to register callbacks that could cause a suspend > operation to fail. Users expect migrations to succeed when all the > conditions outlined in the LDom documentation are met. Today, a suspend > (initiated by the HV as part of a domain migration) has no Solaris hooks > and Solaris is not aware of a suspend/resume. This process is being > changed so that Solaris will be aware of suspend/resume, but we want to > continue that model as much as possible, only making an exception for > Sun Cluster here. > > Lastly, in LDoms, a suspend operation only occurs to permit a domain > migration and this is initiated by the management software on a separate > control domain. The LDom management software is not ready to account for > an arbitrary number of pre/post callbacks which could take any length of > time. Certain operations are blocked when a migration is in progress and > so this affects usability. > > In the event that the notification scheme needs to expand in the future, > we will address the need. This could be with an extensible callback > mechanism in the kernel or an API that includes notifications issued by > the domain manager on the control domain. Okay, thanks for the information. +1 on the case. -Seb From randy.fishel@sun.com Wed Nov 4 11:08:09 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nA4J88XT010794 for ; Wed, 4 Nov 2009 11:08:09 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id nA4J82a7010191; Thu, 5 Nov 2009 03:08:04 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSL0001BMHF9X00@nwk-avmta-2.sfbay.sun.com>; Wed, 04 Nov 2009 11:08:03 -0800 (PST) Received: from grimmy.eng.sun.com ([129.146.108.114]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSL00NTUMHF0R00@nwk-avmta-2.sfbay.sun.com>; Wed, 04 Nov 2009 11:08:03 -0800 (PST) Received: from grimmy.eng.sun.com (localhost [127.0.0.1]) by grimmy.eng.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nA4J6iV9001643; Wed, 04 Nov 2009 11:06:44 -0800 (PST) Received: from localhost (randyf@localhost) by grimmy.eng.sun.com (8.13.8+Sun/8.13.8/Submit) with ESMTP id nA4J6i1P001640; Wed, 04 Nov 2009 11:06:44 -0800 (PST) Date: Wed, 04 Nov 2009 11:06:43 -0800 (PST) From: Randy Fishel Subject: Re: LDOM-SunCluster suspend callbacks [PSARC/2009/589 FastTrack timeout 11/05/2009] In-reply-to: <4AF1C0D0.9030706@sun.com> X-X-Sender: randyf@grimmy To: Haik Aftandilian Cc: Sebastien Roy , Daniel Hain , PSARC-ext@sun.com, ldom-mig-dev@sun.com Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910292000.n9TK09T7005732@sac.sfbay.sun.com> <1257351213.26975.27.camel@strat> <4AF1C0D0.9030706@sun.com> X-Authentication-warning: grimmy.eng.sun.com: randyf owned process doing -bs User-Agent: Alpine 1.10 (GSO 962 2008-03-14) Status: RO Content-Length: 2642 On Wed, 4 Nov 2009, Haik Aftandilian wrote: > Sebastien Roy wrote: > > This architecture is not extensible. Is it inconceivable that other > > subsystems may be interested in these events in the future (or even > > today)? There is no provision here for multiple callbacks, and the > > callbacks have names that are clustering specific. Was any > > consideration given to this? > > Yes, we considered adding a callback registration API so that interested > parties could register their own suspend callbacks, but decided against that > because, at present, Sun Cluster is the only interested party that needs > in-kernel notifications. The callbacks were specifically requested by Sun > Cluster and are only to be used by Sun Cluster. This is the model used by Sun > Cluster and ON in other kernel subsystems. > > We also need to control which callbacks occur. At present we don't want other > kernel subsystems to register callbacks that could cause a suspend operation > to fail. Users expect migrations to succeed when all the conditions outlined > in the LDom documentation are met. Today, a suspend (initiated by the HV as > part of a domain migration) has no Solaris hooks and Solaris is not aware of a > suspend/resume. This process is being changed so that Solaris will be aware of > suspend/resume, but we want to continue that model as much as possible, only > making an exception for Sun Cluster here. Solaris *is* aware of suspend/resume, it is just not obvious to me that existing work/teams are being considered the project team. As I previously mentioned, it does appear as if the problems and needs by guest migration are identical to the problems and needs by suspending bare metal machine (including "this cannot fail"). So why doesn't this proposal/project wish to align itself with that work (possibly using callbacks that already exist)? How will this project align with the Solaris core work? ---- Randy > > Lastly, in LDoms, a suspend operation only occurs to permit a domain migration > and this is initiated by the management software on a separate control domain. > The LDom management software is not ready to account for an arbitrary number > of pre/post callbacks which could take any length of time. Certain operations > are blocked when a migration is in progress and so this affects usability. > > In the event that the notification scheme needs to expand in the future, we > will address the need. This could be with an extensible callback mechanism in > the kernel or an API that includes notifications issued by the domain manager > on the control domain. > > Thanks, > Haik > From Haik.Aftandilian@sun.com Wed Nov 4 11:12:52 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nA4JCoBi010987 for ; Wed, 4 Nov 2009 11:12:50 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id nA4JCkC4027666; Wed, 4 Nov 2009 19:12:48 GMT Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSL0000HMPBIO00@nwk-avmta-2.sfbay.sun.com>; Wed, 04 Nov 2009 11:12:47 -0800 (PST) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSL00N99MP90N10@nwk-avmta-2.sfbay.sun.com>; Wed, 04 Nov 2009 11:12:45 -0800 (PST) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id nA4JCjJ5000407; Wed, 04 Nov 2009 19:12:45 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KSL00600MAK3F00@mail-amer.sun.com>; Wed, 04 Nov 2009 12:12:45 -0700 (MST) Received: from [129.153.89.31] ([unknown] [129.153.89.31]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KSL00KS4MP62XC0@mail-amer.sun.com>; Wed, 04 Nov 2009 12:12:42 -0700 (MST) Date: Wed, 04 Nov 2009 11:13:30 -0800 From: Haik Aftandilian Subject: Re: LDOM-SunCluster suspend callbacks [PSARC/2009/589 FastTrack timeout 11/05/2009] In-reply-to: Sender: Haik.Aftandilian@sun.com To: Randy Fishel Cc: Daniel Hain , PSARC-ext@sun.com, Alexandre.Chartre@sun.com, ldom-mig-dev@sun.com Message-id: <4AF1D25A.702@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910292000.n9TK09T7005732@sac.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20091013) Status: RO Content-Length: 1793 Randy Fishel wrote: > >> These suspend operations are entirely distinct from >> CPR suspend operations. >> > > Why? It would seem to me that the problems are the same for both, > and that it would be worthwhile to leverage this work, than to roll a > private solution. > > ---- Randy Re-using the CPR code was considered, but was not done because the requirements of CPR and sun4v suspend/resume are different. The current sun4v suspend occurs without Solaris' knowledge. The code in development for suspend which will utilize this case is extremely minimal compared to CPR and that is because CPR 1) has to deal with physical devices and 2) has to write the OS memory to disk. The sun4v suspend/resume and therefore domain migration is only available on domains comprised entirely of virtual I/O devices (i.e., virtual disks and virtual network devices). In sun4v, devices drivers do not need to be suspended and devices do not need to be quiesced or powered down, user threads do not need to be stopped, kernel threads do not need to be stopped, and we don't need to flush anything to disk. Leveraging the CPR code could have been done by providing several empty sun4v-specific cpr_ routines as well as making modifications to common CPR code to move more portions into platform specific areas. Some of the callbacks in CPR would have to be skipped on sun4v or we'd have to change other subsystems to not register callbacks on sun4v. And we would want to do this in a way that wouldn't make it difficult to eventually support CPR suspend on sun4v platforms. This work is targeting a Solaris update release and the amount of new code written and risk to other platforms was smallest with this approach. All that considered we chose this approach. Thanks, Haik From Haik.Aftandilian@sun.com Wed Nov 4 13:41:53 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nA4Lfqb7014016 for ; Wed, 4 Nov 2009 13:41:52 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id nA4Lfm4L004726; Wed, 4 Nov 2009 21:41:50 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KSL00303TLP1B00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 04 Nov 2009 13:41:49 -0800 (PST) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KSL00KOWTLPV530@nwk-avmta-1.sfbay.Sun.COM>; Wed, 04 Nov 2009 13:41:49 -0800 (PST) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id nA4Lfn88010597; Wed, 04 Nov 2009 21:41:49 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KSL00K00SBQ2P00@mail-amer.sun.com>; Wed, 04 Nov 2009 14:41:49 -0700 (MST) Received: from [129.153.89.31] ([unknown] [129.153.89.31]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KSL00DX0TLOH660@mail-amer.sun.com>; Wed, 04 Nov 2009 14:41:48 -0700 (MST) Date: Wed, 04 Nov 2009 13:42:36 -0800 From: Haik Aftandilian Subject: Re: LDOM-SunCluster suspend callbacks [PSARC/2009/589 FastTrack timeout 11/05/2009] In-reply-to: Sender: Haik.Aftandilian@sun.com To: Randy Fishel Cc: Sebastien Roy , Daniel Hain , PSARC-ext@sun.com, ldom-mig-dev@sun.com Message-id: <4AF1F54C.30700@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910292000.n9TK09T7005732@sac.sfbay.sun.com> <1257351213.26975.27.camel@strat> <4AF1C0D0.9030706@sun.com> User-Agent: Thunderbird 2.0.0.23 (X11/20091013) Status: RO Content-Length: 3491 Randy Fishel wrote: > On Wed, 4 Nov 2009, Haik Aftandilian wrote: > >> Sebastien Roy wrote: >>> This architecture is not extensible. Is it inconceivable that other >>> subsystems may be interested in these events in the future (or even >>> today)? There is no provision here for multiple callbacks, and the >>> callbacks have names that are clustering specific. Was any >>> consideration given to this? >> Yes, we considered adding a callback registration API so that interested >> parties could register their own suspend callbacks, but decided against that >> because, at present, Sun Cluster is the only interested party that needs >> in-kernel notifications. The callbacks were specifically requested by Sun >> Cluster and are only to be used by Sun Cluster. This is the model used by Sun >> Cluster and ON in other kernel subsystems. >> >> We also need to control which callbacks occur. At present we don't want other >> kernel subsystems to register callbacks that could cause a suspend operation >> to fail. Users expect migrations to succeed when all the conditions outlined >> in the LDom documentation are met. Today, a suspend (initiated by the HV as >> part of a domain migration) has no Solaris hooks and Solaris is not aware of a >> suspend/resume. This process is being changed so that Solaris will be aware of >> suspend/resume, but we want to continue that model as much as possible, only >> making an exception for Sun Cluster here. > > Solaris *is* aware of suspend/resume, it is just not obvious to me > that existing work/teams are being considered the project team. As I > previously mentioned, it does appear as if the problems and needs by > guest migration are identical to the problems and needs by suspending > bare metal machine (including "this cannot fail"). So why doesn't > this proposal/project wish to align itself with that work (possibly > using callbacks that already exist)? How will this project align with > the Solaris core work? I was trying to convey how Solaris is suspended today, in supported software, as part of a LDom domain migration. The hypervisor stops the CPU. Only single-CPU domains can be suspended. No Solaris changes were made for this. CPR is not used. I believe this is similar to xVM migration on x86. I bring this up because it is a stark contrast with CPR. CPR extends to userland, kernel threads, devices drivers, etc. sun4v LDom suspend touches very little. To answer your question, we are not aligning the hypervisor-based suspend for LDoms with existing Solaris "Checkpoint/Resume" CPR. Practically all of the *common* CPR code does not apply to an LDom suspend. We think the differences are significant enough for it to not be worth combining the two. Thanks, Haik > > > ---- Randy > >> Lastly, in LDoms, a suspend operation only occurs to permit a domain migration >> and this is initiated by the management software on a separate control domain. >> The LDom management software is not ready to account for an arbitrary number >> of pre/post callbacks which could take any length of time. Certain operations >> are blocked when a migration is in progress and so this affects usability. >> >> In the event that the notification scheme needs to expand in the future, we >> will address the need. This could be with an extensible callback mechanism in >> the kernel or an API that includes notifications issued by the domain manager >> on the control domain. >> >> Thanks, >> Haik >> From Daniel.Hain@sun.com Sun Nov 8 14:05:32 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id nA8M5VhS011971 for ; Sun, 8 Nov 2009 14:05:31 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id nA8M5TQx028377; Sun, 8 Nov 2009 22:05:30 GMT Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KST004019D59B00@nwk-avmta-2.sfbay.sun.com>; Sun, 08 Nov 2009 14:05:29 -0800 (PST) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KST00FD59D5EAC0@nwk-avmta-2.sfbay.sun.com>; Sun, 08 Nov 2009 14:05:29 -0800 (PST) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id nA8M5Ttx024086; Sun, 08 Nov 2009 14:05:29 -0800 (PST) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) id <0KST003008XYTB00@fe-sfbay-10.sun.com>; Sun, 08 Nov 2009 14:05:28 -0800 (PST) Received: from [192.168.1.5] ([unknown] [72.197.209.207]) by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul 2 2009)) with ESMTPSA id <0KST003YK9CZHYA0@fe-sfbay-10.sun.com>; Sun, 08 Nov 2009 14:05:28 -0800 (PST) Date: Mon, 09 Nov 2009 03:35:23 +0530 From: Daniel Hain Subject: Re: LDOM-SunCluster suspend callbacks [PSARC/2009/589 FastTrack timeout 11/05/2009] In-reply-to: <200910292000.n9TK09T7005732@sac.sfbay.sun.com> Sender: Daniel.Hain@sun.com To: PSARC-ext@sun.com Cc: Alexandre.Chartre@sun.com, Haik.Aftandilian@sun.com, ldom-mig-dev@sun.com Message-id: <4AF740A3.6030105@sun.com> MIME-version: 1.0 Content-type: text/plain; CHARSET=US-ASCII; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200910292000.n9TK09T7005732@sac.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.21 (X11/20090323) Status: RO Content-Length: 7034 This case was approved at the 11/4 meeting. Dan Daniel Hain wrote: > I'm sponsoring this fasttrack for Haik Aftandilian. This is an Open case > seeking Patch binding (for backport to S10). Timeout on 11/05/2009. > > A copy of the proposed contract is in the case directory. > > Dan > > Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI > This information is Copyright 2009 Sun Microsystems > 1. Introduction > 1.1. Project/Component Working Name: > LDOM-SunCluster suspend callbacks > 1.2. Name of Document Author/Supplier: > Author: Haik Aftandilian > 1.3 Date of This Document: > 29 October, 2009 > 4. Technical Description > > Introduction > ------------ > > Solaris Cluster (SC) runs in LDoms guest domains and > provides infrastructure to make applications deployed in the > guest domains HA. SC cluster nodes monitor each other via (so > called) heartbeats, as well associated hardware. SC manages > Storage devices in a way which is specific to the server it > is running on, such as performing SCSI reservations, which > are meaningful only from a specific physical HBA. > > LDoms guest domains can be migrated from one server to > another with the LDoms "Warm Migration" feature. During the > migration, the domain being migrated is suspended. While a > domain is suspended, which can be several minutes, the domain > is totally inactive and not responsive to any requests. Thus, > if a domain running SC is migrated from one server to > another, other cluster nodes need to be made aware of this > fact so that they can suspend monitoring of the cluster node > under migration. Additionally, cluster nodes need to act > co-operatively to make sure the SCSI reservations on storage > devices are also migrated correctly. > > The proposed callbacks would allow SC to > perform these tasks, thereby enabling a seamless migration > of the LDoms guest domain from an end user perspective. > > References > ---------- > > 1. Suspend Domain Service > http://sac.eng/Archives/CaseLog/arc/FWARC/2009/559/ > > 2. Current list of Sun Cluster/ON contract cases in use in Solaris 10. > /ws/osc-gate/usr/src/uts/sparc/cl/imported_symbols.private.Sol_10 > > 3. Example of an existing Sun Cluster/ON contract case. > http://sac.eng/Archives/CaseLog/arc/PSARC/2005/602/ > > Overview > -------- > > In the Solaris kernel, hooks/callback functions will be > run before and after the domain is suspended. A single callback > will be made to SC before the suspension and a single > callback will be made after the resumption. Note that the use of > "suspend" in this contract only applies to suspend operations > initiated by LDoms infrastructure using the suspend domain > service on sun4v guest domains. And that today, suspend > operations are only performed to facilitate LDoms domain > migration. These suspend operations are entirely distinct from > CPR suspend operations. > > Commitment level for all the interfaces: > Contracted Project Private > > Interface Details > ----------------- > > When SC is loaded and wishes to receive suspend > notifications, it will set the callback function pointers to > point to SC functions that handle the notifications. When > setting these callbacks, the cl_suspend_error_decode callback > should be set first, then the cl_suspend_post_callback, and > then the cl_suspend_pre callback. > > The cl_suspend_pre_callback and cl_suspend_post_callback > will never be invoked concurrently and solaris will wait > indefinitely for the callbacks to return. > > Pre-suspend callback: > > int (*cl_suspend_pre_callback)(void); > > Called before the domain is suspended. This serves to > notify SC that this domain is in the process of being > suspended. SC should return 0 if it successfully suspended > monitoring of this domain. If a failure occurred which > should prevent the guest domain from being suspended and > possibly migrated, or if SC can not support a migration > at this time, SC should return a non-zero error code. > If the cl_suspend_pre_callback returns an error code, > the suspension will aborted. The intent is for this error > to be sent back to the domain manager and then used to > build an error message informing the user why the > migration could not be completed. > > Post-suspend callback: > > int (*cl_suspend_post_callback)(void); > > Called after the guest domain has been resumed following a > successful suspension. It is also called after a failed > suspension attempt as well as a canceled suspension > attempt. i.e., it is possible that this function > will be called when the guest domain was suspended and > then resumed without being migrated (as a result of a > failure or cancellation). It can also be called > even when the guest domain was never suspended (due > to a failure before the suspension) and therefore never > migrated. If the callbacks are set after a suspend operation > is already in progress, since the pre callback is set after > the post callback, it is also possible that this function > will be called after a suspension even when the > cl_suspend_pre_callback was not called. Therefore, SC should > not consider it an error if cl_suspend_post_callback is > called before cl_suspend_pre_callback without a > corresponding call to cl_suspend_pre_callback. SC should > return 0 if it successfully resumed monitoring of > this domain. If a failure occurred which prevents the guest > domain from resuming normal activity in the cluster, a > non-zero error value should be returned. The error will > be sent back to the domain manager which will display an > error message informing the user that an error occurred > after the migration and that manual inspection and > recovery may be required for the domain to resume normal > operation. The domain will have been resumed and Solaris > and applications will be running. > > Error code decode callback: > > const char *(*cl_suspend_error_decode)(int); > > Called at any time to convert an error code returned from > the cl_suspend_pre_callback or cl_suspend_post_callback > into a descriptive error string suitable for use in > an error message presented to the user. Returns a NULL- > terminated statically allocated string of length less than > or equal to 256 including the NULL terminator. The caller > will consider this string immutable and will not modify it > or deallocate it. This function may return NULL. When the > cl_suspend_pre_callback or cl_suspend_post_callback return > an error, cl_suspend_error_decode will be used to obtain > an error message string that corresponds to the error. > i.e., cl_suspend_error_decode will be called and its > argument will be an error code returned from > cl_suspend_pre_callback or cl_suspend_post_callback. > > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > ON > 6.5. ARC review type: FastTrack > 6.6. ARC Exposure: open > > -- Dan Hain Solaris Revenue Product Engineering (RPE)