#pragma ident "@(#)pf-device-binding.txt 1.13 10/05/18 Oracle" Title: sun4v Single Root I/O Virtualization PF Device Binding 1. Introduction. This specification defines the firmware behavior for SR-IOV Physical Functions (PF) as defined in [1]. 2. Definitions Root Domain - A domain that owns the PCIe fabric, including configuration, management and global error handling of that physical root complex and all devices in that fabric. Function - A PCIe Function, as defined by the PCIe Base Specification. Physical Function (PF) - A PCIe base function that includes the SR IOV extended capability as defined in [1]. Virtual Function (VF) - as defined in [1]. sr-iov-cap - shorthand notation for the SR IOV Extended Capability defined in [1] Section 3.3. 3. Changes to OBP Probing Algorithm Insert the following text before the current text "If the function has an FCode program" in [2] section 2.5. If the function is a PF (if it includes the SR-IOV extended capability) then perform the procedure defined in section 3.1 of this document. In [2] Section 2.5, add the following to the end of the section that begins with the text "If the function does not have an FCode program". If the function is a PF (if it includes the SR-IOV extended capability) then create the "vf-reg" property as defined in section 5 of this document. The following text augments the paragraph in [2] section 2.5 that begins with "After all slots have been so probed". Apply address allocation to "vf-reg" properties for any PF with a #vfs property value greater than zero. For each PF with allocated vf BAR space, publish the property "vf-assigned-addresses", with entries for each VF BARx register (or register pair) for which an address was assigned, as defined in section 5 of this document. Note: When probing VF BARx, the returned value reflect the size of a single VFs space. The total space consumed by a VF BARx register or register pair will be the number of bytes decoded by each VF BARx multiplied by the value set in NumVFs in the sr-iov-cap. Note: Since it's possible to run out of 32-bit or 64-bit memory space using the algorithms defined in this document, the implementation should include a recovery policy to ensure against ending up with an unusable or unbootable system in the event it runs of PCIe memory space. VF BARx memory space may be sacrificed in the event that occurs. Implementation Note: Allocating extra bus numbers for VF expansion. Some devices that include one or more PFs may offer VF expansion using additional bus numbers, as described in the SR IOV Spec [1]. The firmware implementation must check for this after all PFs have been initialized as defined in this document, and checking First VF Offset, VF Stride and the NumVFs field in each PF and setting the bus range values in the port immediately above this device appropriately. 3.1. PF Initialization. If ARI is enabled in the immediate parent of this Function, Set ARI Capable Hierarchy in SR-IOV Control in sr-iov-cap. Set System Page Size in sr-iov-cap to the value 2 (bit 1 Set). (Sets the device's page size to 8k). let n = the value from the TotalVFs field in the sr-iov-cap. If the MD contains a matching node for this PF { If the MD node contains the unsigned integer property NumVFs { set n = min(n, NumVFs MD propval); Implementation Note: NumVFs may be defined in a platform specific location. For sun4v based systems, it is expected that this data, if defined, will be sourced from the MD. If undefined, the algorithm allocates space for the the max number of VFs. } } Set NumVFs in sr-iov-cap to n. Publish the integer property "#vfs" with the value n. Publish the pf-specific properties as defined in Section 5. initial-vfs total-vfs first-vf-offset vf-stride 4. Configuration Access Words The definition of the following config access word in [2], section 3.2.3, is modified as follows: assign-package-addresses ( phandle -- ) Assigns address (ie, creates "assigned-addresses" and possibly "vf-assigned-addresses") for the child node denoted by *phandle*. 5. Bus Specific Properties for Child Nodes Add the following property definitions for PFs: "vf-reg" prop-name, denotes VFs addressable regions in their associated PF. prop-encoded-array, same as "reg" in [2]. This property is mandatory for PFs. Each entry in "vf-reg" is defined as with "reg" in [2], however each entry describes the VF BARx registers in the SR-IOV Extended Capability in the PF. Changes from "reg" as defined in [2] and [3]. Only contains relocatable memory space entries. (no config space and no io space entries.) The interpretation of the rrrr.rrrr bits in phys.hi in "vf-reg" "vf-assigned-addresses" is as follows: rrrr.rrrr BAR# ========== ==== 0 VF BAR0 1 VF BAR1 2 VF BAR2 3 VF BAR3 4 VF BAR4 5 VF BAR5 "vf-assigned-addresses" prop-name, denotes assigned vf base addresses associated with this PF. prop-encoded-array: One to six {phys-addr, size} pairs. As with "assigned-addresses" as defined in [2] with the following changes: Each entry (ie, each phys-addr, size pair) in this property value corresponds to either one or two (in the case of 64-bit memory address space) of the PF's VF BARx in the SR-IOV Extended Capability in the PFs configuration space. The entry indicates the base physical PCI memory space address (VF BARx registers may only map PCIe memory space) of the first VF associated with this PF. Each entry indicates a multiple of *size* bytes if the value of the #vfs property is greater than 1. The size shall be a multiple of the System Page Size set in the System Page Size field in the SR-IOV extended capability. The 'n' bit of each phys-addr shall be set to 1, indicating that the address is absolute (within the PCI domain's address space), not relative to the start of a relocatable region. The type code shall be either '10' or '11' denoting 32-bit or 64-bit PCI memory space respectively. The 'bbbbbbbb,ddddd,fff,rrrrrrrr' field indicates the VF BARx register to which the entry applies, as follows: For VF BAR0, rrrrrrrrr shall contain the value 0. For VF BAR1, rrrrrrrrr shall contain the value 1. For VF BAR2, rrrrrrrrr shall contain the value 2. For VF BAR3, rrrrrrrrr shall contain the value 3. For VF BAR4, rrrrrrrrr shall contain the value 4. For VF BAR5, rrrrrrrrr shall contain the value 5. the remaining bits are as defined in [2] and [3], however, only relocatable 32 and 64 bit memory space entries are valid in this property. (The n bit in phys.hi must be zero.) Note: Each entry in "vf-assigned-addresses" contains the base PCIe memory space address and size of the first VF associated with this PF once VFs are enabled. The total allocated space can be determined by multiplying the size field by the value of the #vfs property in the this node. "#vfs" prop-name, denotes number of VFs in PF. prop-encoded-array, one prop-encoded integer. The property contains a single integer, the value of the number of Virtual Functions configured in a Physical Function by the firmware. (NumVFs). "initial-vfs" prop-name, denotes value of InitialVFs in the PFs sr-iov-cap. prop-encoded-array, one prop-encoded integer. The property contains a single integer, the value of InitialVFs field in this PFs sr-iov-cap. "total-vfs" prop-name, denotes value of TotalVFs in the PFs sr-iov-cap. prop-encoded-array, one prop-encoded integer. The property contains a single integer, the value of TotalVFs field in this PFs sr-iov-cap. "first-vf-offset" prop-name, denotes value of the FirstVFOffset field in the PFs sr-iov-cap. prop-encoded-array, one prop-encoded integer. The property contains a single integer, the value of the FirstVFOffset field in this PFs sr-iov-cap. "vf-stride" prop-name, denotes value of the VFStride field in the PFs sr-iov-cap. prop-encoded-array, one prop-encoded integer. The property contains a single integer, the value of the VFStride field in this PFs sr-iov-cap. 6. References [1] Single Root I/O Virtualization and Sharing Specification Revision 1.1 Available from pcisig.com [2] PCI Bus Binding to: IEEE 1275-1994 Standard for Boot (Initialization, Configuration) Firmware, Revision 2.1. available from http://playground.sun.com/1275/ [3] PCI Express Binding Update: FWARC/2010/063 http://sac.eng/FWARC/2010/063 http://sac.eng/FWARC/Specifications/pci-e.binding.txt