------- hv_sun4v_errorphilosophy-V2.1.txt ------- --- /tmp/sccs.iBaM8q Tue Feb 10 10:36:33 2009 +++ hv_sun4v_errorphilosophy-V2.1.txt Tue Feb 10 10:34:13 2009 @@ -1,7 +1,7 @@ -"@(#)hv_sun4v_errorphilosophy-V2.1.txt 1.1 09/02/05 " +"@(#)hv_sun4v_errorphilosophy-V2.1.txt 1.6 09/02/10 " -Sun4v Hypervisor Error Handling Interfaces [DRAFT FOR REVIEW] +Sun4v Hypervisor Error Handling Interfaces ------------------------------------------- NOTE: @@ -11,6 +11,10 @@ errors are still being developed and are not described in this version of the document. + This interface is being extended to include a mechanism for notifying the + sun4v guest of Service Processor (SP) state changes, ie, when the Service + Processor in a system becomes available/unavailable. + 1.0 Introduction Hardware errors which do not reset the system generate a trap to the @@ -21,6 +25,12 @@ the error is sent to the guest. Section 5 of this document describes the structure of the error report sent to the guest. +Service Processor state changes (SP becoming available again after being +offline or becoming unavailable) on systems which have the necessary +hardware support will generate an interrupt to the hypervisor. An +error report indicating this SP state change will be sent to the +guest. + Errors from devices that are directly accessed by the sun4v guest are not virtualized by the hypervisor. They are handled by the device drivers of the sun4v guest. @@ -39,6 +49,8 @@ for errors in virtual or direct I/O devices are queued on to the dev_mondo queue. +SP state change error reports are queued on to the resumable error queue. + The simplest implementation of a sun4v guest could, for example, simply perform a 'retry' on resumable error notifications and 'panic' on non-resumable error notifications. But the intent is to have enough @@ -96,8 +108,8 @@ 4.0 Brief Overview of Error Notifications -Hardware errors which do not reset the system trap to the -hypervisor. For each error handled by the hypervisor: +Hardware errors which do not reset the system trap to the hypervisor. +For each error handled by the hypervisor: (1) If the error does not reset the sun4v guest, then a sun4v error report that virtualizes the underlying hardware error and describes the impact of the error is sent to the sun4v @@ -106,6 +118,9 @@ captured by the hardware and additional diagnostic data is sent to the diagnostic service provider. +For an SP state change, a sun4v error describing the change is sent to the +sun4v guest. There is no associated service error report. + As shown in Figure 1 below, the sun4v error report is sent to the affected sun4v guest via the queues defined by the sun4v architecture. The service error report is sent via the Service Provider @@ -152,6 +167,43 @@ Fig 1. Hypervisor Error Reports to sun4v Guest and FMA Service Provider + + + _________ + ( ) + ( FMA Agent ) + (_________) + ^ FMA Error + | Report + +--------------+ + |SP state | + |error handler | + +--------------+ + ^ + +-------| + | + +-------------+ + | resumable | + |error queue | + +-------------+ + ^ + | + +------+ + | + | + | + | + +---------+ + | + | + +-------------+ + | Hypervisor | + +-------------+ + ^ + | SP state change interrupt + + Fig 1.1. Hypervisor SP Change Reports to sun4v Guest and FMA Service Provider + Some notes on Figure 1 above: (1) Virtual I/O refers to devices that cannot be directly accessed by the guest. They are either complete abstractions of @@ -186,6 +238,9 @@ the error report at the tail and updates the tail pointer to the next entry. +The SP state change error reports are sent via the sun4v resumable_error +queues. + For resumable_error and dev_mondo queues, the hardware generates a disrupting trap whenever the head and tail pointers are not equal. The disrupting trap is taken on the CPU if the interrupts are enabled @@ -286,6 +341,8 @@ Diagnosis Service Provider via the service provider interface for fault analysis. +All SP state changes are classified as resumable errors. + Some examples of hardware errors that are not reported to the guest because the error was corrected are: correctable ECC error in cache data, TLB data parity error, cache data parity error in a clean cache, and @@ -412,12 +469,13 @@ 3 NR_DF Deferred non-resumable error report 4 SHT_R Shutdown request (resumable) 5 DCORE Dump Core (non-resumable) + 6 SP SP state change (resumable) ------------------------------------------------------------ Table 5.2.3-I. Error Report Descriptors All other values are reserved. -The values R_UE and SHT_R are valid only for error reports that are queued +The values R_UE, SHT_R and SP are valid only for error reports that are queued on the resumable_error queue. The values NR_PREC, NR_DEF and DCORE are valid only for the error reports that are queued on the nonresumable_error queue. @@ -472,6 +530,10 @@ 5.2.3.v DCORE, (Dump Core). This is used to instruct the guest to initiate a dump core sequence. This report will be queued on the non-resumable error queue. +5.2.3.vi SP, (Service Processor state change). This is used to notify the guest +that the SP state has changed. The SP is now in the state denoted by the ATTR.SP_STATE value. The guest may decide to notify the user of the SP state change using some form of FMA messaging and/or perform any other actions it deems appropriate. This report will be queued on the resumable error +queue. + 5.2.4 Error Attributes (ATTR). The meaning of this field depends on the error descriptor (see 5.2.3) of the error report. It also includes the resumable queue full indicator (see 5.2.11). @@ -501,8 +563,9 @@ for future use. MODE 25:24 Execution Mode (see 5.2.5.viiii) - RSVD0 23:9 Undefined. Reserved + RSVD0 23:10 Undefined. Reserved for future use. + SP_STATE 9:9 New SP state PREG 8 Sun4v Privileged CPUID, REG Register ASI 7 Sun4v ASI register ASI, ADDR, SZ @@ -523,27 +586,28 @@ The tables 5.2.4-II below shows the applicable attibute fields for the different types of error reports. 'Y' indicates applicable. '-' indicates not applicable. -+----------------------------------------------------------------------------------------------+ -|Error| Error Attributes | | | | -|DESC |CPU |MEM |PIO |IRF |FRF |SHUT|ASR|ASI|PREG|MODE |RQFULL | Notes | -+-----|----|----|----|----|----|----|---|---|----|-----|---------------------------------------+ -|R_UE | Y | Y | - | - | - | - | - | Y | - | Y | Y | PIO, IRF, FRF, ASR, PREG | -| | | | | | | | | | | | | and REG not applicable in | -| | | | | | | | | | | | | uncorrected resumable error | -| | | | | | | | | | | | | reports. | -|NR_PR| - | Y | Y | Y | Y | - | Y | Y | Y | - | - | CPU not applicable in | -| | | | | | | | | | | | | precise non-resumable error | -| | | | | | | | | | | | | reports. PIO and MEM cannot | -| | | | | | | | | | | | | be set in the same report. | -|NR_DF| - | Y | Y | - | - | - | - | - | - | Y | - | CPU, IRF, FRF, ASR, ASI and | -| | | | | | | | | | | | | PREG not applicable | -| | | | | | | | | | | | | in deferred non-resumable | -| | | | | | | | | | | | | error reports. | -| | | | | | | | | | | | | PIO and MEM cannot be set in | -| | | | | | | | | | | | | the same report. | -|SHT_R| - | - | - | - | - | Y | - | - | - | - | - | | -|DCORE| - | - | - | - | - | - | - | - | - | - | - | No attributes for DCORE | -+----------------------------------------------------------------------------------------------+ ++----------------------------------------------------------------------------------------------------+ +|Error| Error Attributes | |SP | | | +|DESC |CPU |MEM |PIO |IRF |FRF |SHUT|ASR|ASI|PREG|STATE|MODE |RQFULL | Notes | ++-----|----|----|----|----|----|----|---|---|----|-----|-----|---------------------------------------+ +|R_UE | Y | Y | - | - | - | - | - | - | - | - | Y | Y | PIO, IRF, FRF, ASR, ASI, PREG | +| | | | | | | | | | | | | | and REG not applicable in | +| | | | | | | | | | | | | | uncorrected resumable error | +| | | | | | | | | | | | | | reports. | +|NR_PR| - | Y | Y | Y | Y | - | Y | Y | Y | - | - | - | CPU not applicable in | +| | | | | | | | | | | | | | precise non-resumable error | +| | | | | | | | | | | | | | reports. PIO and MEM cannot | +| | | | | | | | | | | | | | be set in the same report. | +|NR_DF| - | Y | Y | - | - | - | - | - | - | - | Y | - | CPU, IRF, FRF, ASR, ASI and | +| | | | | | | | | | | | | | PREG not applicable | +| | | | | | | | | | | | | | in deferred non-resumable | +| | | | | | | | | | | | | | error reports. | +| | | | | | | | | | | | | | PIO and MEM cannot be set in | +| | | | | | | | | | | | | | the same report. | +|SHT_R| - | - | - | - | - | Y | - | - | - | - | - | - | | +|DCORE| - | - | - | - | - | - | - | - | - | - | - | - | No attributes for DCORE | +|SP | - | - | - | - | - | - | - | - | - | Y | - | - | | ++----------------------------------------------------------------------------------------------------+ Table 5.2.4-II. Applicable Error Attributes Map @@ -624,18 +688,31 @@ is sent to a different CPU in the same partition indicating the ID of the CPU in error. -5.2.4.viiii Execution Mode (MODE). This field specifies the execution -mode of the operation that induced the error. +5.2.4.viiii Service Processor State (SP_STATE). This field specifies the +current state of the SP. + The table 5.2.4-III below lists the currently defined values. --------------------------------- Value Description --------------------------------- + 0b0 SP is unavailable + 0b1 SP is available + --------------------------------- + Table 5.2.4-III. Service Processor State + +5.2.4.x Execution Mode (MODE). This field specifies the execution +mode of the operation that induced the error. + +The table 5.2.4-IV below lists the currently defined values. + --------------------------------- + Value Description + --------------------------------- 0b00 Unknown 0b01 User mode 0b10 Privilege mode 0b11 Reserved --------------------------------- - Table 5.2.4-III. Execution Mode + Table 5.2.4-IV. Execution Mode The 'Unknown' execution mode will be used in error reports when the hypervisor cannot determine the CPU's state at the time of the error. @@ -955,40 +1032,55 @@ A.1 Resumable error handler - if (DESC == 1) /* Uncorrected resumable error */ - if (ATTR.CPU) + if (DESC == 1) { /* Uncorrected resumable error */ + if (ATTR.CPU) { if (ATTR.MODE == User) kill user process else panic - if (ATTR.MEM) + } + if (ATTR.MEM) { get ADDR, SZ call hypervisor to scrub memory - retry; - if (ATTR.ASI) - get ASI, ADDR, SZ - if ASI register(s) valid for this CPU - if ASI register(s) is reloadable/recoverable - reload/recover - retry - panic - - if (DESC == 4) /* Shutdown request */ - if (ATTR.SHUT) + retry; + } + } + if (DESC == 4) { /* Shutdown request */ + if (ATTR.SHUT) { get SECS delay SECS seconds shutdown + } + } + if (DESC == 6) { /* SP State change */ + if (ATTR.SP_STATE == SP_AVAILABLE) { + /* + * SP is available now after a period of being + * offline .... + */ + } else { + /* + * SP is unavailable now, disable any services which + * require SP interaction ... + */ + } + } A.2 Non-resumable error handler - if (DESC == 5) /* dump core */ + if (DESC == 5) { /* dump core */ panic - if (DESC == 3) /* deferred trap */ + } + if (DESC == 3) { /* deferred trap */ if (ATTR.MODE == User) kill user process else panic - if (ATTR.MEM) + } + + ASSERT(DESC == 2); /* Precise non-resumable error */ + + if (ATTR.MEM) { get ADDR, SZ make hypervisor call to scrub memory if (data not recoverable) @@ -995,44 +1087,49 @@ panic else retry - if (ATTR.PIO) + } + if (ATTR.PIO) { get IOADDR panic - if (ATTR.IRF or ATTR.FRF) + } + if (ATTR.IRF or ATTR.FRF) { if (user mode) kill user process else panic - - if (ATTR.ASR) + } + if (ATTR.ASR) { get ASR register from REG - if ASR valid for this CPU + if ASR valid for this CPU { if ASR is reloadable/recoverable reload/recover retry + } if (user mode) kill user process else panic - - if (ATTR.ASI) + } + if (ATTR.ASI) { get ASI, ADDR, SZ - if ASI register(s) valid for this CPU + if ASI register(s) valid for this CPU { if ASI register(s) is reloadable/recoverable reload/recover retry + } if (user mode) kill user process else panic - - if (ATTR.PREG) + } + if (ATTR.PREG) { get REG (privileged register) - if privileged register is reloadable/recoverable + if privileged register is reloadable/recoverable { reload/recover retry + } if (user mode) kill user process else panic - + }