Microcode Update for Intel Processors 1. Background The P4, Xeon and P6 family processors have the capability to correct errata by loading an Intel-supplied data block called microcode into the processor. The BIOS contains a microcode loader and is typically responsible for loading the microcode update on processors during system initialization. It is not practical to need to update the BIOS every time a new microcode update is released for a processor. Hence the need for an OS based microcode load facility as described in this document. 2. Technical Details Microcode update can be performed at early boot time by the operating system as each processing is being brought online as part of MP boot of a system. Additionally we also need a way to dynamically update the microcode when the system is already booted and running. This provides a mechanism to load critical microcode files to ensure system integrity without requiring a reboot or BIOS upgrade to update microcode for a processor. 2.1 Microcode files A microcode binary file consists of a multi-byte header, followed by encrypted data and then by an optional extended signature table. 2.1.1 Download Microcode files in text format are available for download from http://www.urbanmyth.org/microcode/ 2.1.2 Packaging Once a new microcode release is available for OS vendors, a new utility ucodeadm(1M) can be run against the file. Based on options supplied, ucodeadm(1M) will 1. Show current microcode revision. 2. Generate a set of microcode binary files, one for each processor with a unique signature and platform id, and a pkgdef file for integrating the microcode files into ON. 3. Install the microcode files on the target system 4. Apply the microcode update on a live system. The microcode binary files will be installed on a target system at /platform/i86pc/ucode/ The microcode files to be installed will follow the following naming convention. VendorStr/SignatureByte-PlatformId The Signature byte and Processor Flag byte (also called the "platform id" byte) are available in microcode file header. For example, # ucodeadm -i intel-ucode.txt where intel-ucode.txt is for processors wth signature 6f4, 6f5, and 6f6 with 0xf5 platform id mask. will generate file /platform/i86pc/ucode/GenuineIntel/.f/000006f4-f5 and hard links to /platform/i86pc/ucode/GenuineIntel/000006f4-f5 from /platform/i86pc/ucode/GenuineIntel/000006f4-01 /platform/i86pc/ucode/GenuineIntel/000006f4-04 /platform/i86pc/ucode/GenuineIntel/000006f4-10 /platform/i86pc/ucode/GenuineIntel/000006f4-20 /platform/i86pc/ucode/GenuineIntel/000006f4-40 /platform/i86pc/ucode/GenuineIntel/000006f4-80 /platform/i86pc/ucode/GenuineIntel/000006f5-01 /platform/i86pc/ucode/GenuineIntel/000006f5-04 /platform/i86pc/ucode/GenuineIntel/000006f5-10 /platform/i86pc/ucode/GenuineIntel/000006f5-20 /platform/i86pc/ucode/GenuineIntel/000006f5-40 /platform/i86pc/ucode/GenuineIntel/000006f5-80 /platform/i86pc/ucode/GenuineIntel/000006f6-01 /platform/i86pc/ucode/GenuineIntel/000006f6-04 /platform/i86pc/ucode/GenuineIntel/000006f6-10 /platform/i86pc/ucode/GenuineIntel/000006f6-20 /platform/i86pc/ucode/GenuineIntel/000006f6-40 /platform/i86pc/ucode/GenuineIntel/000006f6-80 Such design is to - keep only the latest version - keep the same file only once - not break patching An alternate path for installing these files can be specified with "-R path" option. To automate the steps above during installation and reboot, "bootadm" has been modified to invoke "ucodeadm -i" if there is a microcode text file available at /platform/i86pc/ucode/ that's newer than the timestamp file. The timestamp file is set to have the same creation time as the microcode text file on each microcode installation. 2.1.3 Microcode file format Details of the microcode file format is documented in Intel 64 and IA-32 Architectures Software Developer's Manual Section 9-11 "Microcode Update Facilities". A microcode file has a 48-byte header, followed by the binary code of size indicated in the header (uh_body_size), and optionally a (20 + n * 12)-byte extended signature block if the microcode file can be used for more than one type of processor. struct ucode_info { struct ucode_header ui_header; uint8_t *ui_body; struct ucode_ext_table *ui_ext_table; }; The 48-byte header contains the following fields: struct ucode_header { uint32_t uh_header_ver; uint32_t uh_rev; uint32_t uh_date; uint32_t uh_sigature; uint32_t uh_checksum; uint32_t uh_loader_ver; uint32_t uh_proc_flags; uint32_t uh_body_size; uint32_t uh_total_size; uint32_t uh_reserved[3]; }; The extended signature tables contains the following fields: struct ucode_ext_table { uint32_t uet_count; uint32_t uet_checksum; uint32_t uet_reserved[3]; struct ucode_proc_sigature uet_proc_sig[1]; //uet_count of them }; struct ucode_proc_sigature { uint32_t ups_signature; uint32_t ups_proc_flags; uint32_t ups_checksum; }; For example, here is the header of a microcode file for family 0x6 model 0xf and stepping 0x4 for platforms 0x01, 0x04, 0x10, 0x20, 0x40 and 0x80. 0x00000001, 0x80520000, 0x10262006, 0x000006f4, 0xf2ec076e, 0x00000001, 0x000000f5, 0x00000fd0, 0x00001000, 0x00000000, 0x00000000, 0x00000000, 2.2 Update Entry Points There are two basic points of microcode update invocation: boot time and run time. Following sections will cover details of each implementation. 2.2.1 Boot time In order to perform microcode update during boot, the microcode binary files must be available in the boot archive as a result of installing a release containing the microcode files, or post installation via "ucodeadm" and reboot. The entry point for boot path is ucode_check(). It will be called from mlsetup() in mlsetup.c for the boot CPU and mp_startup() in mp_startup.c. ucode_check() takes a single argument: pointer to the target CPU structure. void ucode_check(cpu_t *cp); Error is logged via cmn_err if update fails. 2.2.2 Run time ucodeadm(1M) can be invoked to update the microcode. A driver ucode_drv provides the following capability via ioctl: UCODE_GET_VERSION: get running microcode version UCODE_UPDATE: apply new microcode The kernel code entry point is ucode_update(). ucode_update() takes 3 arguments, and returns an error code. ucode_errno_t ucode_update(uint8_t *ucodep, int size, processorid_t *failed_cpu) 2.3 Boot Time Implementation 2.3.1 Obtain Processor Identify In order to find a match and check whether microcode update is needed, we need the following information on the target processor core: 1. vendor 2. family 3. model 4. stepping If it is indeed an Intel processor, 5. platform id (also called processor flag) 6. running microcode version The first 4 are available in cpuid_info, which is part of the CPU structure. 5 and 6 will be obtained via Model Specific Registers (MSR) MSR_INTC_PLATFORM_ID (0x17) MSR_INTC_UCODE_REV (0x8B) A natural place for storing the platform id and microcode revision is machcpu. The cpu_ucode_info structure is introduced and will be used to store the target processor's microcode related information. struct cpu_ucode_info { uint32_t cpu_ui_pf; uint32_t cpu_ui_rev; }; 2.3.2 Validate match A microcode file structure is used to cache the last used microcode. If the cached microcode file exists, check to see 1. if the signature and platform id match (in header or in extended signature table), and 2. if the revision is greater than the running version. If both conditions are met, use the binary ui_body from the microcode structure. Such design is based on the expectation that most systems will have the same type of processors, so the microcode file only needs to be loaded once. If not all conditions are met, reset the structure and locate the right file as described in 2.3.3. 2.3.3 Locate the file Construct file name based on vendor string, the signature byte, and the platform id, then look for the file using kobj_open(); (void) snprintf(name, UCODE_MAX_NAME_LEN, "/%s/%s/%08X-%02X", UCODE_INSTALL_PATH, cpuid_getvendorstr(cp), cpi_sig, uinfop->cui_platid); Return if not found. File-not-found is not a failure condition. 2.3.4 Read and validate header Once we have successfully located the file, first read 48 bytes into header structure. Perform the following validation on the header: ucode_validate_header(ucode_header_t *uhp) { 1. Verify header version is 0x1. 2. Verify total_size is 0 or corresponds to header_size, body_size and extended header size. 3. Verify body_size is either 0 or something valid. } A failure return from ucode_validate_header() will not result in microcode update. 2.3.5 Read and validate micode body Read body_size bytes into microcode data buffer. ucode_checksum() { Validate checksum. } ucode_match() { 1. Compare revision. Return if file revision is smaller. 2. Verify that signature and processor flag match. if (does-not-match && has-extended-sigature-table) do more processing } 2.3.6 Read and validate extended signature table If total_size is greater than (body_size + 48), we will also need to read and validate the signature table. This step is necessary even if a the signature and processor flag match the ones in the header as we are constructing the data structure for future comparison with other processors. First read (total_size - (body_size + 48)) from file. ucode_validate_ext_sig() { Validate checksum. } 2.3.7 Validate Match Similar to 2.3.2, but simply return if no match is found rather than looking for microcode file again. If we have come this far and not found a match, it is an indication that the packaging algorithm is faulty. 2.3.8 Apply Microcode and validate write success ucode_update_intel() { kpreempt_disable(); wrmsr(MSR_INTC_UCODE_WRITE, (uint64_t)(uintptr_t)ucode_body); ucode_read_rev(uinfop); kpreempt_enable(); } 2.4 Run Time Implementation The basic idea is the same as the boot time implementation except that we now have two more software components: ucodeadm and ucode_drv. 2.4.1 ucodeadm As mentioned earlier, ucodeadm will be the utility that generates the binary microcode files and corresponding pkgdef file for ON integration, and to update the processor microcode on a live system. # ucodeadm -h usage: ucodeadm -v Shows running microcode version. ucodeadm -u microcode-text-file Updates microcode to the latest matching version found in microcode-text-file. ucodeadm -i [-R path] [-o prototype-filename] microcode-text-file Installs microcode. The -v option can be performed by a non-privileged user. The -i option requires privilege to write to the destination. The -u option requires PRIV_ALL. More options could be added, but it is preferable to keep it simple. 2.4.2 ucode_drv The -v and -u option will call into the ucode_drv driver via ioctl with UCODE_GET_VERSION and UCODE_UPDATE commands respectively. With the -v option, ucode_drv will call the ucode_get_rev() interface. With the -u option, The ucode_drv driver will copy in the text or binary format microcode file from userspace, perform necessary validation, then call the ucode_update() interface to perform microcode update. ucode_update_lock is used make sure there is only one update in progress at a time. The reason a global lock instead of a per CPU lock is used is that, on processors that support hyper-threading, all threads on the same physical processor share the same underlying microcode hardware. To keep the locking logic simple, we will use a global lock. mutex_enter(&ucode_update_lock) ucode_update(); mutex_exit(&ucode_update_lock) 2.4.3 Apply Microcode and validate write success ucode_update() takes a different set of arguments from ucode_check() because it is invoked in a different environment. ucode_errno_t ucode_update(uint8_t *ucodep, int size, processorid_t *failed_cpu) { For each xcall capable CPU { Find the corresponding microcode file for the CPU. Add CPU to cpu_set. kpreempt_disable(); xc_sync((xc_arg_t)&ucodep_array, 0, 0, X_CALL_HIPRI, cpu_set, ucode_write); kpreempt_enable(); Verify that the processors have been updated as expected. } } All errors will be reported to user. Update message is also logged to /var/adm/messages. 3. Future work AMD might support the ability to perform microcode update the OS is able to do it early in boot. The current framework should work well to add support for AMD processors.