From sacadmin Wed Sep 3 16:34:02 2008 Received: from tethys.sfbay.sun.com (tethys.SFBay.Sun.COM [129.146.226.92]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m83NY2Z1016240; Wed, 3 Sep 2008 16:34:02 -0700 (PDT) Received: from tethys.sfbay.sun.com (localhost [127.0.0.1]) by tethys.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m83NTpqS001632; Wed, 3 Sep 2008 16:29:51 -0700 (PDT) Received: (from jg@localhost) by tethys.sfbay.sun.com (8.14.3+Sun/8.14.3/Submit) id m83NTpZu001629; Wed, 3 Sep 2008 16:29:51 -0700 (PDT) Date: Wed, 3 Sep 2008 16:29:51 -0700 (PDT) From: Jerry Gilliam Message-Id: <200809032329.m83NTpZu001629@tethys.sfbay.sun.com> To: PSARC-record@sac.sfbay.sun.com Subject: AMD IOMMU [PSARC/2008/561 Self Review] Status: RO Content-Length: 541 Template Version: @(#)sac_nextcase %I% %G% SMI This information is Copyright 2008 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: AMD IOMMU 1.2. Name of Document Author/Supplier: Author: Vikram Hegde 1.3 Date of This Document: 03 September, 2008 4. Technical Description See the case directory for more detail 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: Automatic 6.6. ARC Exposure: open From jg@jurassic.sfbay.Sun.COM Wed Sep 3 16:38:35 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m83NcZHC016447 for ; Wed, 3 Sep 2008 16:38:35 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m83NcY1Y003488; Wed, 3 Sep 2008 16:38:34 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6N00B018CACH00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 03 Sep 2008 16:38:34 -0700 (PDT) Received: from jurassic.eng.sun.com ([129.146.104.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6N00H1T8CAYF80@nwk-avmta-1.sfbay.Sun.COM>; Wed, 03 Sep 2008 16:38:34 -0700 (PDT) Received: from tethys (tethys.SFBay.Sun.COM [129.146.226.92]) by jurassic.eng.sun.com (8.13.8+Sun/8.13.8) with SMTP id m83NcXi3148518; Wed, 03 Sep 2008 16:38:33 -0700 (PDT) Date: Wed, 03 Sep 2008 16:34:23 -0700 (PDT) From: Jerry Gilliam Subject: AMD IOMMU [PSARC/2008/561] To: PSARC-ext@sun.com, vikram.hegde@sun.com Reply-to: Jerry Gilliam Message-id: <200809032338.m83NcXi3148518@jurassic.eng.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_94 SunOS 5.11 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-MD5: 9hZCFje0jKVkRdRAGaX7Tg== X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 9444 I am sponsoring the following fast-track for Vikram Hegde. This case presents an overview and architectural details on an IOMMU for AMD cpus. This information is being presented for information only, for discussion and future reference. As there's nothing to approve, I believe Closed Approved Automatic is appropriate. -jg AMD IOMMU ========= Introduction ============ A Memory Management Unit (CPU or I/O) is hardware that translates virtual memory addresses into "real" or physical memory addresses. A physical memory address typically reflects the actual memory installed on the system. Virtual memory on the other hand is a complete fabrication valid only for the process or other entity (such as a kernel) on the system. Using virtual memory has several benefits including providing isolation, providing the illusion of contiguous flat memory to a process, providing a large address space (that may or may not be backed by actual physical memory), providing a process the complete freedom to load code and data anywhere in the flat virtual address space. MMU for CPUs are available in almost all modern general purpose CPUs and almost all non-embedded operating systems support these MMUs. However IOMMUs i.e. MMUs for I/O devices is not yet common on most OSs. The one notable exception is SPARC CPUs and SPARC Solaris for which IOMMU has been available for quite some time now. IOMMUs are only now making their appearance on x86 (intel and AMD) CPUs and this PSARC case discusses providing Solaris x86 support for the AMD IOMMU. Background ========== An I/O MMU or IOMMU as it is commonly called provides a translator from device virtual addresses to system physical addresses. Until now on Solaris x86 a device DMA engine was programmed with physical memory addresses so that when it performed DMA reads and writes it was directly accessing physical memory. With an IOMMU, a device DMA engine is provided with device or domain specific virtual addresses and DMA accesses by the device are done with these virtual addresses. An IOMMU intercepts theses accesses and directs them to the correct physical addresses. Using an IOMMU provides several benefits including 1. The ability to isolate a device's memory access to certain limited areas of system physical memory preventing device hardware or driver software from corrupting memory belonging to the kernel or other I/O devices 2. The ability to provide the illusion of a flat contiguous virtual address space for device DMA when in fact the backing physical memory is "scattered" all over the system physical memory. This is useful for devices that don't have scatter-gather capability and cannot deal with discontiguous memory. 3. For certain legacy devices which have restrictions on memory they can access (such as only low memory), the virtual memory address space can be used to provide that illusion while mapping it to high addresses in physical memory. This allows for better use of 64 bit address spaces without needing expensive copying using "bounce buffers" 4. For virtualization software it provides the ability to isolate devices belonging to different virtual machines so that a malicious OS cannot bring down the entire system. Technical Details ================= On AMD CPUs that have IOMMU support, the IOMMU is integrated into the I/O hub. The AMD IOMMU is a standard PCI function. There may be more than 1 IOMMU per PCI function. The IOMMUs have the following capabilities 1. The ability to remap DMA accesses (Read and Write) from virtual to system physical addresses 2. The ability to remap interrupts routing them as desired (such as to VMs that controls those devices) 3. The ability to record and report faults encountered during the above remapping steps. 4. The ability to parcel out devices to various VMs. 5. The ability to virtualize the IOMMU for use by VMs and their OSes. Of these features the initial implementation in Solaris will only enable 1 and 3 i.e. DMA remapping and the ability to report faults encountered during DMA remapping. The following hardware and software elements are used for DMA remapping 1. Capability registers - There are a bunch of capability registers implemented in the IOMMU's (PCI function) configuration space. These registers point to the virtual address of the memory mapped control registers of the IOMMU. 2. Control Registers - These are a bunch of memory mapped registers which include a. Device Table base address register - The Device Table is the primary software data structure used for DMA and Interrupt remapping. The Device Table base address register contains the location and size of the Device Table b. Command Buffer Base Address Register - This register contains the base address and size of the Command Buffer - a circular buffer in system memory used to send commands to the IOMMU. c. Event Log Base Address Register - This register contains the base address and size of the Event Log - a circular buffer in system memory used by the IOMMU to report and record faults. d. Control Register - A control register used to send control commands to the IOMMU e. Status Register - A status register used by the IOMMU to report status information 3. Device Table - A software table setup in main memory by the OS for the IOMMU. The Device Table is indexed by the DeviceID a 16 bit device identifier. Each entry in the Device Table includes the following information: a. The Page Table Root Pointer - A pointer to a page table for that device b. The Interrupt table Root pointer - A pointer to the interrupt mapping table c. A mode field - This reports the number of levels in the page table d. A DomainId field - This contains the domainID i.e. the domain which contains the device. Two devices in the same domain have the same page tables. e. Read/Write Permission bits for this Translation f. Fields which indicate if the Interrupt, Translation and Device table entry valid. 4. Page Tables - These are tables in memory that are used to map specific bits in the virtual address to the next level Page Table if a Page Directory Entry (PDE) or the physical page frame if a Page Table Entry (PTE). Each level in the page tables takes as input the base address of a page table and 9 bits from the virtual address (from HI to LO). The 9 bits are used as an index into the page table to get the physical address of the next lower level page table or the final physical page frame. With 64 bits in the virtual address this yields a 6-level page table with the lowest 12 bits used as an offset into the final 4KB physical page frame. Each PDE has a next level field. If set to 0 it indicates that it is a PTE and translation has ended. Else it indicates that the level of the next table. Using 0 early allows the use of large pages (similar to the Super Pages field in Intel IOMMU). 5. Interrupt Remapping Table - This is a table in physical memory that is indexed into by using bits from the MSI interrupt data. Since this feature will not be enabled in the first phase of this project we wont discuss this further. 6. Command Buffer - The command buffer is a circular buffer in memory that is written to by the OS/driver and is read by the IOMMU. The IOMMU uses a head pointer register to get the next location to read while system software uses a tail pointer register to determine the next location to write to. The IOMMU provides a completion wait command that allows the system to wait on an interrupt until all commands prior to the command completion wait command have been completed. The command buffer is architecturally similar to the Queued Invalidation Interface used by Intel IOMMUs. 7. Event Log - The event log is a circular buffer in system memory that is written to by the IOMMU to report faults encounterred during remapping of DMA and Interrupts. There is a tail pointer register which points to the next location to write to the IOMMU and a head pointer register to be used by system software to locate the next event to read. The IOMMU can be programmed to generate an interrupt when an event occurs and the Event Log is updated. The AMD Event Log is architecturally similar to the Advanced Fault Logging capability provided by Intel IOMMUs. Operation of the IOMMU ====================== An IOMMU driver initially sets up the various data structures in memory including page tables, device table, event log and command buffer. The IOMMU is then started. When a device sets up DMA it makes a call to ddi_dma_addr_bind_handle() or ddi_dma_buf_bind_handle(). The address passed into these routines is typically the kernel virtual address. The DDI framework translates these virtual addresses to physical addresses and passes them onto the IOMMU driver. The IOMMU driver maps these into device virtual addresses, updates the device's I/O page tables and then sends a command via the command buffer to the IOMMU to invalidate any internal TLBs it may have. The OS then passes the device virtual addresses in the form of DMA cookies back to the DMA requester (the device driver). The device driver programs the device's DMA engine and starts up the DMA. The device sends data to the device virtual address programmed into it and this is then translated by the IOMMU after walking the device I/O page tables. Any errors encountered during this process are recorded in the Event Log and an interrupt is generated to notify the IOMMU driver. From Matthew.Jacob@sun.com Wed Sep 3 16:48:57 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m83NmuuP016638 for ; Wed, 3 Sep 2008 16:48:57 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m83NmoYf057923 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 3 Sep 2008 17:48:56 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6N00C1T8TJFQ00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 03 Sep 2008 16:48:55 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6N00HFU8TJY870@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 03 Sep 2008 16:48:55 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m83Nmtvw023336 for ; Wed, 03 Sep 2008 16:48:55 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6N004018RJIN00@fe-sfbay-10.sun.com> (original mail from Matthew.Jacob@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 03 Sep 2008 16:48:55 -0700 (PDT) Received: from [129.145.154.73] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6N00JH08SD9N60@fe-sfbay-10.sun.com>; Wed, 03 Sep 2008 16:48:14 -0700 (PDT) Date: Wed, 03 Sep 2008 16:48:13 -0700 From: Matthew Jacob Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <200809032338.m83NcXi3148518@jurassic.eng.sun.com> Sender: Matthew.Jacob@sun.com To: Jerry Gilliam Cc: PSARC-ext@sun.com, Vikram.Hegde@sun.com Reply-to: Matthew.Jacob@sun.com Message-id: <48BF223D.1000601@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200809032338.m83NcXi3148518@jurassic.eng.sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 765 On 09/03/08 16:34, Jerry Gilliam wrote: > I am sponsoring the following fast-track for Vikram Hegde. This > case presents an overview and architectural details on an IOMMU > for AMD cpus. This information is being presented for information > only, for discussion and future reference. As there's nothing to > approve, I believe Closed Approved Automatic is appropriate. > > > > Abilities to disable the IOMMU as a choice? Performance impact on various systems and interactions with other projects that have gone to some length to work within S/G and its performance characteristics. How will this affect them? Cache effects? Interactions with some VM work currently being undertaken? I can't imagine such a major change being closed/approved/automatic. From Vikram.Hegde@sun.com Wed Sep 3 17:07:14 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m8407ErU017982 for ; Wed, 3 Sep 2008 17:07:14 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m8407DMI063892 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 3 Sep 2008 18:07:13 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6N000219NYRT00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 03 Sep 2008 18:07:10 -0600 (MDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6N00H189NW4P60@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 03 Sep 2008 18:07:08 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m84078vA025134 for ; Wed, 03 Sep 2008 17:07:08 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6N005019NHEF00@fe-sfbay-09.sun.com> (original mail from Vikram.Hegde@Sun.COM) for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Wed, 03 Sep 2008 17:07:08 -0700 (PDT) Received: from [192.168.1.101] ([71.141.142.68]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6N007QS9NRRN60@fe-sfbay-09.sun.com>; Wed, 03 Sep 2008 17:07:07 -0700 (PDT) Date: Wed, 03 Sep 2008 17:07:07 -0700 From: Vikram Hegde Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48BF223D.1000601@Sun.COM> Sender: Vikram.Hegde@sun.com To: Matthew.Jacob@sun.com Cc: Jerry Gilliam , PSARC-ext@sun.com, Vikram.Hegde@sun.com Message-id: <48BF26AB.5080109@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200809032338.m83NcXi3148518@jurassic.eng.sun.com> <48BF223D.1000601@Sun.COM> User-Agent: Thunderbird 2.0.0.16 (Windows/20080708) Status: RO Content-Length: 1587 Hi, > Abilities to disable the IOMMU as a choice? Performance impact on > various systems and interactions with other projects that have gone to > some length to work within S/G and its performance characteristics. > How will this affect them? Cache effects? Interactions with some VM > work currently being undertaken? > > I can't imagine such a major change being closed/approved/automatic. Performance is driver dependant. There is no way a single project team can test the gamut of drivers out there and certify that the IOMMU will give acceptable performance for all needs (which is itself a subjective opinion). We have always planned to provide the ability to disable the IOMMU on a per driver basis. Does that address your performance concerns ? Vikram Matthew Jacob wrote: > On 09/03/08 16:34, Jerry Gilliam wrote: >> I am sponsoring the following fast-track for Vikram Hegde. This >> case presents an overview and architectural details on an IOMMU >> for AMD cpus. This information is being presented for information >> only, for discussion and future reference. As there's nothing to >> approve, I believe Closed Approved Automatic is appropriate. >> >> >> >> > > Abilities to disable the IOMMU as a choice? Performance impact on > various systems and interactions with other projects that have gone to > some length to work within S/G and its performance characteristics. > How will this affect them? Cache effects? Interactions with some VM > work currently being undertaken? > > I can't imagine such a major change being closed/approved/automatic. From Matthew.Jacob@sun.com Wed Sep 3 17:16:13 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m840GCkG018679 for ; Wed, 3 Sep 2008 17:16:12 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m840G666017265 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 08:16:11 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6N00309A2W0D00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 03 Sep 2008 17:16:08 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6N0012BA2W3J30@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 03 Sep 2008 17:16:08 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m840G8Zk025887 for ; Wed, 03 Sep 2008 17:16:08 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6N00A019WJUE00@fe-sfbay-10.sun.com> (original mail from Matthew.Jacob@Sun.COM) for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Wed, 03 Sep 2008 17:16:08 -0700 (PDT) Received: from [129.145.154.73] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6N00JVAA2U9N70@fe-sfbay-10.sun.com>; Wed, 03 Sep 2008 17:16:07 -0700 (PDT) Date: Wed, 03 Sep 2008 17:16:06 -0700 From: Matthew Jacob Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48BF26AB.5080109@sun.com> Sender: Matthew.Jacob@sun.com To: Vikram Hegde Cc: Jerry Gilliam , PSARC-ext@sun.com Reply-to: Matthew.Jacob@sun.com Message-id: <48BF28C6.1020807@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200809032338.m83NcXi3148518@jurassic.eng.sun.com> <48BF223D.1000601@Sun.COM> <48BF26AB.5080109@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 989 On 09/03/08 17:07, Vikram Hegde wrote: > Hi, > >> Abilities to disable the IOMMU as a choice? Performance impact on >> various systems and interactions with other projects that have gone >> to some length to work within S/G and its performance >> characteristics. How will this affect them? Cache effects? >> Interactions with some VM work currently being undertaken? >> >> I can't imagine such a major change being closed/approved/automatic. > Performance is driver dependant. There is no way a single project team > can test the gamut of drivers out there and certify that the IOMMU > will give acceptable performance for all needs (which is itself a > subjective opinion). No, it's a measurable value- not subjective. > We have always planned to provide the ability to disable the IOMMU on > a per driver basis. Does that address your performance concerns ? > Yes, but as this is a spec in advance of actual implementation, could you make sure that 'plan' is on record? From Vikram.Hegde@sun.com Wed Sep 3 17:24:24 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m840OND2018747 for ; Wed, 3 Sep 2008 17:24:24 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m840OKsc018353 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 01:24:22 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6N00G09AGK9Z00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 03 Sep 2008 17:24:20 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6N00H0RAGKY890@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 03 Sep 2008 17:24:20 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m840OKYK003601 for ; Wed, 03 Sep 2008 17:24:20 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6N00201AFP7H00@fe-sfbay-09.sun.com> (original mail from Vikram.Hegde@Sun.COM) for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Wed, 03 Sep 2008 17:24:20 -0700 (PDT) Received: from [192.168.1.101] ([71.141.142.68]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6N00714AGJRND0@fe-sfbay-09.sun.com>; Wed, 03 Sep 2008 17:24:20 -0700 (PDT) Date: Wed, 03 Sep 2008 17:24:23 -0700 From: Vikram Hegde Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48BF28C6.1020807@Sun.COM> Sender: Vikram.Hegde@sun.com To: Matthew.Jacob@sun.com Cc: Jerry Gilliam , PSARC-ext@sun.com, Vikram.Hegde@sun.com Message-id: <48BF2AB7.90906@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200809032338.m83NcXi3148518@jurassic.eng.sun.com> <48BF223D.1000601@Sun.COM> <48BF26AB.5080109@sun.com> <48BF28C6.1020807@Sun.COM> User-Agent: Thunderbird 2.0.0.16 (Windows/20080708) Status: RO Content-Length: 1869 Hi, > > No, it's a measurable value- not subjective. I wasn't talking about the performance number but rather about "acceptable". Some may argue that for certain drivers the slight reduction in performance is more than compensated by the several benefits that an IOMMU offers (note that SPARC side has been functioning with IOMMU without complaints about performance). For certain drivers it may turn out that the owners find the performance reduction undesirable. It all depends on the driver owners. >> We have always planned to provide the ability to disable the IOMMU on >> a per driver basis. Does that address your performance concerns ? >> > Yes, but as this is a spec in advance of actual implementation, could > you make sure that 'plan' is on record? ok, I will send an addendum to the spec shortly. Vikram Matthew Jacob wrote: > On 09/03/08 17:07, Vikram Hegde wrote: >> Hi, >> >>> Abilities to disable the IOMMU as a choice? Performance impact on >>> various systems and interactions with other projects that have gone >>> to some length to work within S/G and its performance >>> characteristics. How will this affect them? Cache effects? >>> Interactions with some VM work currently being undertaken? >>> >>> I can't imagine such a major change being closed/approved/automatic. >> Performance is driver dependant. There is no way a single project >> team can test the gamut of drivers out there and certify that the >> IOMMU will give acceptable performance for all needs (which is >> itself a subjective opinion). > > No, it's a measurable value- not subjective. > >> We have always planned to provide the ability to disable the IOMMU on >> a per driver basis. Does that address your performance concerns ? >> > > Yes, but as this is a spec in advance of actual implementation, could > you make sure that 'plan' is on record? > From Matthew.Jacob@sun.com Wed Sep 3 17:34:20 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m840YJuZ018992 for ; Wed, 3 Sep 2008 17:34:19 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m840YGmP021921 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 01:34:18 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6N0030NAX5RZ00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 03 Sep 2008 17:34:17 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6N0011OAX43M40@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 03 Sep 2008 17:34:16 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m840YGYb027438 for ; Wed, 03 Sep 2008 17:34:16 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6N00901ATTNB00@fe-sfbay-10.sun.com> (original mail from Matthew.Jacob@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 03 Sep 2008 17:34:16 -0700 (PDT) Received: from [129.145.154.73] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6N00JNLAWV9N80@fe-sfbay-10.sun.com>; Wed, 03 Sep 2008 17:34:08 -0700 (PDT) Date: Wed, 03 Sep 2008 17:34:07 -0700 From: Matthew Jacob Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48BF2AB7.90906@sun.com> Sender: Matthew.Jacob@sun.com To: Vikram Hegde Cc: Jerry Gilliam , PSARC-ext@sun.com Reply-to: Matthew.Jacob@sun.com Message-id: <48BF2CFF.1050009@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200809032338.m83NcXi3148518@jurassic.eng.sun.com> <48BF223D.1000601@Sun.COM> <48BF26AB.5080109@sun.com> <48BF28C6.1020807@Sun.COM> <48BF2AB7.90906@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 934 On 09/03/08 17:24, Vikram Hegde wrote: > Hi, >> >> No, it's a measurable value- not subjective. > I wasn't talking about the performance number but rather about > "acceptable". Some may argue that for certain drivers the slight > reduction in performance is more than compensated by the several > benefits that an IOMMU offers (note that SPARC side has been > functioning with IOMMU without complaints about performance). For > certain drivers it may turn out that the owners find the performance > reduction undesirable. It all depends on the driver owners. And the P-team for the platform. > >>> We have always planned to provide the ability to disable the IOMMU >>> on a per driver basis. Does that address your performance concerns ? >>> >> Yes, but as this is a spec in advance of actual implementation, could >> you make sure that 'plan' is on record? > ok, I will send an addendum to the spec shortly. > Thank you. From gdamore@sun.com Wed Sep 3 20:47:18 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m843lHg4022543 for ; Wed, 3 Sep 2008 20:47:17 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m843lBYD022594 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 04:47:16 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6N00G01JURI700@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 03 Sep 2008 20:47:15 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6N00EV2JUR0Z20@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 03 Sep 2008 20:47:15 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m843lFEF011278 for ; Wed, 03 Sep 2008 20:47:15 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6N00D01JTJ0O00@fe-sfbay-09.sun.com> (original mail from gdamore@sun.com) for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Wed, 03 Sep 2008 20:47:14 -0700 (PDT) Received: from [10.7.251.172] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6N0037XJUJ04B0@fe-sfbay-09.sun.com>; Wed, 03 Sep 2008 20:47:07 -0700 (PDT) Date: Wed, 03 Sep 2008 20:46:27 -0700 From: "Garrett D'Amore" Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48BF26AB.5080109@sun.com> Sender: Garrett.Damore@sun.com To: Vikram Hegde Cc: Matthew.Jacob@sun.com, Jerry Gilliam , PSARC-ext@sun.com Message-id: <48BF5A13.9000500@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200809032338.m83NcXi3148518@jurassic.eng.sun.com> <48BF223D.1000601@Sun.COM> <48BF26AB.5080109@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 2054 Vikram Hegde wrote: > Hi, > >> Abilities to disable the IOMMU as a choice? Performance impact on >> various systems and interactions with other projects that have gone >> to some length to work within S/G and its performance >> characteristics. How will this affect them? Cache effects? >> Interactions with some VM work currently being undertaken? >> >> I can't imagine such a major change being closed/approved/automatic. > Performance is driver dependant. There is no way a single project team > can test the gamut of drivers out there and certify that the IOMMU > will give acceptable performance for all needs (which is itself a > subjective opinion). We have always planned to provide the ability to > disable the IOMMU on a per driver basis. Does that address your > performance concerns ? Actually, for many cases, IOMMU may *improve* performance. This is because it will make it easier to satisfy device constraints for DMA resources without requiring the use of a bounce buffer or dma windows. That said, drivers which assume that setup and tear down of DMA is "cheap" may take a perf. hit when using an IOMMU. This is already the case for SPARC systems, btw. -- Garrett > > Vikram > > > Matthew Jacob wrote: >> On 09/03/08 16:34, Jerry Gilliam wrote: >>> I am sponsoring the following fast-track for Vikram Hegde. This >>> case presents an overview and architectural details on an IOMMU >>> for AMD cpus. This information is being presented for information >>> only, for discussion and future reference. As there's nothing to >>> approve, I believe Closed Approved Automatic is appropriate. >>> >>> >>> >>> >> >> Abilities to disable the IOMMU as a choice? Performance impact on >> various systems and interactions with other projects that have gone >> to some length to work within S/G and its performance >> characteristics. How will this affect them? Cache effects? >> Interactions with some VM work currently being undertaken? >> >> I can't imagine such a major change being closed/approved/automatic. > From Matthew.Jacob@Sun.COM Wed Sep 3 23:18:13 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m846IDg1026388 for ; Wed, 3 Sep 2008 23:18:13 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m846IDDk001493 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 3 Sep 2008 23:18:13 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6N00K0JQU9IS00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 03 Sep 2008 23:18:09 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6N00E17QU9R870@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@Sun.COM); Wed, 03 Sep 2008 23:18:09 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m846I9W2010973 for ; Wed, 03 Sep 2008 23:18:09 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6N00D01QP6JJ00@fe-sfbay-09.sun.com> (original mail from Matthew.Jacob@Sun.COM) for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Wed, 03 Sep 2008 23:18:09 -0700 (PDT) Received: from [192.168.1.3] ([72.164.148.65]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6N007IIQU8SY90@fe-sfbay-09.sun.com>; Wed, 03 Sep 2008 23:18:08 -0700 (PDT) Date: Wed, 03 Sep 2008 23:16:42 -0700 From: Matthew Jacob Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48BF5A13.9000500@sun.com> Sender: Matthew.Jacob@Sun.COM To: "Garrett D'Amore" Cc: Vikram Hegde , Jerry Gilliam , PSARC-ext@Sun.COM Message-id: <48BF7D4A.5020407@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200809032338.m83NcXi3148518@jurassic.eng.sun.com> <48BF223D.1000601@Sun.COM> <48BF26AB.5080109@sun.com> <48BF5A13.9000500@sun.com> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.16) Gecko/20080702 SeaMonkey/1.1.11 Status: RO Content-Length: 1873 Garrett D'Amore wrote: > Vikram Hegde wrote: >> Hi, >> >>> Abilities to disable the IOMMU as a choice? Performance impact on >>> various systems and interactions with other projects that have gone >>> to some length to work within S/G and its performance >>> characteristics. How will this affect them? Cache effects? >>> Interactions with some VM work currently being undertaken? >>> >>> I can't imagine such a major change being closed/approved/automatic. >> Performance is driver dependant. There is no way a single project >> team can test the gamut of drivers out there and certify that the >> IOMMU will give acceptable performance for all needs (which is >> itself a subjective opinion). We have always planned to provide the >> ability to disable the IOMMU on a per driver basis. Does that address >> your performance concerns ? > > Actually, for many cases, IOMMU may *improve* performance. This is > because it will make it easier to satisfy device constraints for DMA > resources without requiring the use of a bounce buffer or dma windows. That is almost always the exception now. The other issue IIRC is the interaction between the IOMMU and memory caches. In many data movement appliance contexts you use scatter-gather (scatter-gather in name only- contiguous memory is good) to physical pages which are never mapped in. The main point here is to avoid a situation that we're currently in now with respect MSI-X and Interrupt/CPU binding and make sure that there are sufficient control surfaces to allow for a broad range of possible implementation choices. > > That said, drivers which assume that setup and tear down of DMA is > "cheap" may take a perf. hit when using an IOMMU. This is already the > case for SPARC systems, btw. Except for those (ancient) ones that used 36-bit MBus physical addresses and bypassed the IOMMU. From Kais.Belgaied@sun.com Thu Sep 4 07:24:38 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m84EObQs010426 for ; Thu, 4 Sep 2008 07:24:38 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m84EOPH1016648; Thu, 4 Sep 2008 15:24:35 +0100 (BST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6O00H0HDCWEH00@brm-avmta-1.central.sun.com>; Thu, 04 Sep 2008 08:24:32 -0600 (MDT) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6O005GHDCWWOA0@brm-avmta-1.central.sun.com>; Thu, 04 Sep 2008 08:24:32 -0600 (MDT) Received: from [129.146.11.147] (sr1-jurassic-04.SFBay.Sun.COM [129.146.11.147]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id m84ENhDd002371; Thu, 04 Sep 2008 07:23:49 -0700 (PDT) Date: Thu, 04 Sep 2008 07:23:40 -0700 From: Kais Belgaied Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48BF7D4A.5020407@sun.com> To: Matthew Jacob Cc: "Garrett D'Amore" , Vikram Hegde , Jerry Gilliam , PSARC-ext@sun.com Reply-to: Kais.Belgaied@sun.com Message-id: <48BFEF6C.1030303@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200809032338.m83NcXi3148518@jurassic.eng.sun.com> <48BF223D.1000601@Sun.COM> <48BF26AB.5080109@sun.com> <48BF5A13.9000500@sun.com> <48BF7D4A.5020407@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 2294 Vikram, Jerry, So, is the ability to disable the IOMMU on a per driver basis in this case's scope? what interface is being proposed for doing this? Jerry, assuming the answer the to first question above is 'yes', the case cannot be in a "closed approved automatic" state while waiting for some of its spec. Kais. On 09/03/08 23:16, Matthew Jacob wrote: > Garrett D'Amore wrote: >> Vikram Hegde wrote: >>> Hi, >>> >>>> Abilities to disable the IOMMU as a choice? Performance impact on >>>> various systems and interactions with other projects that have gone >>>> to some length to work within S/G and its performance >>>> characteristics. How will this affect them? Cache effects? >>>> Interactions with some VM work currently being undertaken? >>>> >>>> I can't imagine such a major change being closed/approved/automatic. >>> Performance is driver dependant. There is no way a single project >>> team can test the gamut of drivers out there and certify that the >>> IOMMU will give acceptable performance for all needs (which is >>> itself a subjective opinion). We have always planned to provide the >>> ability to disable the IOMMU on a per driver basis. Does that >>> address your performance concerns ? >> >> Actually, for many cases, IOMMU may *improve* performance. This is >> because it will make it easier to satisfy device constraints for DMA >> resources without requiring the use of a bounce buffer or dma windows. > > That is almost always the exception now. The other issue IIRC is the > interaction between the IOMMU and memory caches. In many data movement > appliance contexts you use scatter-gather (scatter-gather in name > only- contiguous memory is good) to physical pages which are never > mapped in. > > The main point here is to avoid a situation that we're currently in > now with respect MSI-X and Interrupt/CPU binding and make sure that > there are sufficient control surfaces to allow for a broad range of > possible implementation choices. > >> >> That said, drivers which assume that setup and tear down of DMA is >> "cheap" may take a perf. hit when using an IOMMU. This is already >> the case for SPARC systems, btw. > > Except for those (ancient) ones that used 36-bit MBus physical > addresses and bypassed the IOMMU. > > From Vikram.Hegde@sun.com Thu Sep 4 08:18:43 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m84FIhIE011987 for ; Thu, 4 Sep 2008 08:18:43 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m84FId6X029536 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 08:18:43 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6O00K2PFV4TH00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 09:18:40 -0600 (MDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6O005ZZFV2X9E0@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 09:18:38 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m84FIc8d001904 for ; Thu, 04 Sep 2008 08:18:38 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6O00901FQXNT00@fe-sfbay-09.sun.com> (original mail from Vikram.Hegde@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 08:18:38 -0700 (PDT) Received: from [192.168.1.101] ([71.141.142.68]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6O00B34FV1OL80@fe-sfbay-09.sun.com>; Thu, 04 Sep 2008 08:18:37 -0700 (PDT) Date: Thu, 04 Sep 2008 08:18:44 -0700 From: Vikram Hegde Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48BFEF6C.1030303@Sun.COM> Sender: Vikram.Hegde@sun.com To: Kais.Belgaied@sun.com Cc: Matthew Jacob , "Garrett D'Amore" , Jerry Gilliam , PSARC-ext@sun.com, Vikram.Hegde@sun.com Message-id: <48BFFC54.7080407@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200809032338.m83NcXi3148518@jurassic.eng.sun.com> <48BF223D.1000601@Sun.COM> <48BF26AB.5080109@sun.com> <48BF5A13.9000500@sun.com> <48BF7D4A.5020407@sun.com> <48BFEF6C.1030303@Sun.COM> User-Agent: Thunderbird 2.0.0.16 (Windows/20080708) Status: RO Content-Length: 3324 Hi Kais, Below is the text I am planning to add to the PSARC case. And yes, the disable properties will be a part of this PSARC case. Vikram ===================================================================== Enabling/Disabling the IOMMU ============================= It is recognized that driver or platform developers may not in certain cases desire the IOMMU to be enabled. To accomodate such needs we propose the following global properties in the iommu driver.conf file "global-disable" - can be set to 1 to disable IOMMU platform wide. "exclude-list" - list of driver names that want the IOMMU disabled. ====================================================================== Vikram Kais Belgaied wrote: > Vikram, Jerry, > > So, is the ability to disable the IOMMU on a per driver basis in this > case's scope? > what interface is being proposed for doing this? > > Jerry, assuming the answer the to first question above is 'yes', the > case cannot be in > a "closed approved automatic" state while waiting for some of its spec. > > Kais. > > On 09/03/08 23:16, Matthew Jacob wrote: >> Garrett D'Amore wrote: >>> Vikram Hegde wrote: >>>> Hi, >>>> >>>>> Abilities to disable the IOMMU as a choice? Performance impact on >>>>> various systems and interactions with other projects that have >>>>> gone to some length to work within S/G and its performance >>>>> characteristics. How will this affect them? Cache effects? >>>>> Interactions with some VM work currently being undertaken? >>>>> >>>>> I can't imagine such a major change being closed/approved/automatic. >>>> Performance is driver dependant. There is no way a single project >>>> team can test the gamut of drivers out there and certify that the >>>> IOMMU will give acceptable performance for all needs (which is >>>> itself a subjective opinion). We have always planned to provide the >>>> ability to disable the IOMMU on a per driver basis. Does that >>>> address your performance concerns ? >>> >>> Actually, for many cases, IOMMU may *improve* performance. This is >>> because it will make it easier to satisfy device constraints for DMA >>> resources without requiring the use of a bounce buffer or dma windows. >> >> That is almost always the exception now. The other issue IIRC is the >> interaction between the IOMMU and memory caches. In many data >> movement appliance contexts you use scatter-gather (scatter-gather in >> name only- contiguous memory is good) to physical pages which are >> never mapped in. >> >> The main point here is to avoid a situation that we're currently in >> now with respect MSI-X and Interrupt/CPU binding and make sure that >> there are sufficient control surfaces to allow for a broad range of >> possible implementation choices. >> >>> >>> That said, drivers which assume that setup and tear down of DMA is >>> "cheap" may take a perf. hit when using an IOMMU. This is already >>> the case for SPARC systems, btw. >> >> Except for those (ancient) ones that used 36-bit MBus physical >> addresses and bypassed the IOMMU. >> >> > -- =============================================================================== thread(n):projecting helical rib by which parts can be screwed together =============================================================================== From Matthew.Jacob@Sun.COM Thu Sep 4 08:22:27 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m84FMRo1012066 for ; Thu, 4 Sep 2008 08:22:27 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m84FMMWt006019 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 09:22:26 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6O00J1DG1BX900@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 08:22:23 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6O00F88G1A7QE0@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 08:22:22 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m84FMMx4002918 for ; Thu, 04 Sep 2008 08:22:22 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6O00K01FJLDY00@fe-sfbay-10.sun.com> (original mail from Matthew.Jacob@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 08:22:22 -0700 (PDT) Received: from [192.168.1.3] ([72.164.148.65]) by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6O00700G118U20@fe-sfbay-10.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 08:22:14 -0700 (PDT) Date: Thu, 04 Sep 2008 08:20:48 -0700 From: Matthew Jacob Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48BFFC54.7080407@sun.com> Sender: Matthew.Jacob@Sun.COM To: Vikram Hegde Cc: PSARC-ext@Sun.COM Message-id: <48BFFCD0.9040908@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200809032338.m83NcXi3148518@jurassic.eng.sun.com> <48BF223D.1000601@Sun.COM> <48BF26AB.5080109@sun.com> <48BF5A13.9000500@sun.com> <48BF7D4A.5020407@sun.com> <48BFEF6C.1030303@Sun.COM> <48BFFC54.7080407@sun.com> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.16) Gecko/20080702 SeaMonkey/1.1.11 Status: RO Content-Length: 777 Vikram Hegde wrote: > Hi Kais, > > Below is the text I am planning to add to the PSARC case. And yes, the > disable properties will be a part of this PSARC case. > > Vikram > > ===================================================================== > Enabling/Disabling the IOMMU > ============================= > It is recognized that driver or platform developers may not in certain > cases desire the > IOMMU to be enabled. To accomodate such needs we propose the following > global properties > in the iommu driver.conf file > > "global-disable" - can be set to 1 to disable IOMMU platform wide. > "exclude-list" - list of driver names that want the IOMMU disabled. > ====================================================================== > Seems reasonable, thank you. From gdamore@Sun.COM Thu Sep 4 08:48:07 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m84Fm70l012319 for ; Thu, 4 Sep 2008 08:48:07 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m84FlmVD027907 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 08:48:07 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6O00M0PH83MM00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 09:48:03 -0600 (MDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6O00LC0H82DM20@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 09:48:03 -0600 (MDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m84Fm2IL006086 for ; Thu, 04 Sep 2008 08:48:02 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6O00801H3YHU00@fe-sfbay-10.sun.com> (original mail from gdamore@sun.com) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 08:48:02 -0700 (PDT) Received: from [10.7.251.172] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6O007Z2H7Z8UE0@fe-sfbay-10.sun.com>; Thu, 04 Sep 2008 08:48:00 -0700 (PDT) Date: Thu, 04 Sep 2008 08:47:18 -0700 From: "Garrett D'Amore" Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48BFFC54.7080407@sun.com> Sender: Garrett.Damore@Sun.COM To: Vikram Hegde Cc: Kais.Belgaied@Sun.COM, Matthew Jacob , Jerry Gilliam , PSARC-ext@Sun.COM Message-id: <48C00306.4010809@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200809032338.m83NcXi3148518@jurassic.eng.sun.com> <48BF223D.1000601@Sun.COM> <48BF26AB.5080109@sun.com> <48BF5A13.9000500@sun.com> <48BF7D4A.5020407@sun.com> <48BFEF6C.1030303@Sun.COM> <48BFFC54.7080407@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 3395 It would be cooler if this could be dealt with using ordinary DMA attributes. Won't DDI_DMA_FORCE_PHYSICAL (see the dma_attr_flags for ddi_dma_attr(9s)) work for this? - Garrett Vikram Hegde wrote: > Hi Kais, > > Below is the text I am planning to add to the PSARC case. And yes, the > disable properties will be a part of this PSARC case. > > Vikram > > ===================================================================== > Enabling/Disabling the IOMMU > ============================= > It is recognized that driver or platform developers may not in certain > cases desire the > IOMMU to be enabled. To accomodate such needs we propose the following > global properties > in the iommu driver.conf file > > "global-disable" - can be set to 1 to disable IOMMU platform wide. > "exclude-list" - list of driver names that want the IOMMU disabled. > ====================================================================== > > Vikram > > Kais Belgaied wrote: >> Vikram, Jerry, >> >> So, is the ability to disable the IOMMU on a per driver basis in this >> case's scope? >> what interface is being proposed for doing this? >> >> Jerry, assuming the answer the to first question above is 'yes', the >> case cannot be in >> a "closed approved automatic" state while waiting for some of its spec. >> >> Kais. >> >> On 09/03/08 23:16, Matthew Jacob wrote: >>> Garrett D'Amore wrote: >>>> Vikram Hegde wrote: >>>>> Hi, >>>>> >>>>>> Abilities to disable the IOMMU as a choice? Performance impact on >>>>>> various systems and interactions with other projects that have >>>>>> gone to some length to work within S/G and its performance >>>>>> characteristics. How will this affect them? Cache effects? >>>>>> Interactions with some VM work currently being undertaken? >>>>>> >>>>>> I can't imagine such a major change being closed/approved/automatic. >>>>> Performance is driver dependant. There is no way a single project >>>>> team can test the gamut of drivers out there and certify that the >>>>> IOMMU will give acceptable performance for all needs (which is >>>>> itself a subjective opinion). We have always planned to provide >>>>> the ability to disable the IOMMU on a per driver basis. Does that >>>>> address your performance concerns ? >>>> >>>> Actually, for many cases, IOMMU may *improve* performance. This is >>>> because it will make it easier to satisfy device constraints for >>>> DMA resources without requiring the use of a bounce buffer or dma >>>> windows. >>> >>> That is almost always the exception now. The other issue IIRC is the >>> interaction between the IOMMU and memory caches. In many data >>> movement appliance contexts you use scatter-gather (scatter-gather >>> in name only- contiguous memory is good) to physical pages which are >>> never mapped in. >>> >>> The main point here is to avoid a situation that we're currently in >>> now with respect MSI-X and Interrupt/CPU binding and make sure that >>> there are sufficient control surfaces to allow for a broad range of >>> possible implementation choices. >>> >>>> >>>> That said, drivers which assume that setup and tear down of DMA is >>>> "cheap" may take a perf. hit when using an IOMMU. This is already >>>> the case for SPARC systems, btw. >>> >>> Except for those (ancient) ones that used 36-bit MBus physical >>> addresses and bypassed the IOMMU. >>> >>> >> > > From Vikram.Hegde@sun.com Thu Sep 4 08:53:09 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m84Fr8u1012516 for ; Thu, 4 Sep 2008 08:53:08 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m84Fqvb7020139 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 09:53:08 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6O0010RHGINR00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 08:53:06 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6O00GJKHGIFJE0@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 08:53:06 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m84Fr6Zu006750 for ; Thu, 04 Sep 2008 08:53:06 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6O00801H3YHU00@fe-sfbay-10.sun.com> (original mail from Vikram.Hegde@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 08:53:06 -0700 (PDT) Received: from [192.168.1.101] ([71.141.142.68]) by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6O00J4WHGCZ900@fe-sfbay-10.sun.com>; Thu, 04 Sep 2008 08:53:01 -0700 (PDT) Date: Thu, 04 Sep 2008 08:53:06 -0700 From: Vikram Hegde Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C00306.4010809@sun.com> Sender: Vikram.Hegde@sun.com To: "Garrett D'Amore" Cc: Kais.Belgaied@sun.com, Matthew Jacob , Jerry Gilliam , PSARC-ext@sun.com, Vikram.Hegde@sun.com Message-id: <48C00462.40909@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200809032338.m83NcXi3148518@jurassic.eng.sun.com> <48BF223D.1000601@Sun.COM> <48BF26AB.5080109@sun.com> <48BF5A13.9000500@sun.com> <48BF7D4A.5020407@sun.com> <48BFEF6C.1030303@Sun.COM> <48BFFC54.7080407@sun.com> <48C00306.4010809@sun.com> User-Agent: Thunderbird 2.0.0.16 (Windows/20080708) Status: RO Content-Length: 4033 Garrett D'Amore wrote: > It would be cooler if this could be dealt with using ordinary DMA > attributes. Won't DDI_DMA_FORCE_PHYSICAL (see the dma_attr_flags for > ddi_dma_attr(9s)) work for this? > Yes, that would work as well and is the preferred and established solution. This flag will be respected by the IOMMU. However, it seems that an alternative mechanism that involves no touching of the driver code is desirable as well. Do you agree ? Thanks, Vikram > - Garrett > > > Vikram Hegde wrote: >> Hi Kais, >> >> Below is the text I am planning to add to the PSARC case. And yes, >> the disable properties will be a part of this PSARC case. >> >> Vikram >> >> ===================================================================== >> Enabling/Disabling the IOMMU >> ============================= >> It is recognized that driver or platform developers may not in >> certain cases desire the >> IOMMU to be enabled. To accomodate such needs we propose the >> following global properties >> in the iommu driver.conf file >> >> "global-disable" - can be set to 1 to disable IOMMU platform wide. >> "exclude-list" - list of driver names that want the IOMMU disabled. >> ====================================================================== >> >> Vikram >> >> Kais Belgaied wrote: >>> Vikram, Jerry, >>> >>> So, is the ability to disable the IOMMU on a per driver basis in >>> this case's scope? >>> what interface is being proposed for doing this? >>> >>> Jerry, assuming the answer the to first question above is 'yes', >>> the case cannot be in >>> a "closed approved automatic" state while waiting for some of its spec. >>> >>> Kais. >>> >>> On 09/03/08 23:16, Matthew Jacob wrote: >>>> Garrett D'Amore wrote: >>>>> Vikram Hegde wrote: >>>>>> Hi, >>>>>> >>>>>>> Abilities to disable the IOMMU as a choice? Performance impact >>>>>>> on various systems and interactions with other projects that >>>>>>> have gone to some length to work within S/G and its performance >>>>>>> characteristics. How will this affect them? Cache effects? >>>>>>> Interactions with some VM work currently being undertaken? >>>>>>> >>>>>>> I can't imagine such a major change being >>>>>>> closed/approved/automatic. >>>>>> Performance is driver dependant. There is no way a single project >>>>>> team can test the gamut of drivers out there and certify that the >>>>>> IOMMU will give acceptable performance for all needs (which is >>>>>> itself a subjective opinion). We have always planned to provide >>>>>> the ability to disable the IOMMU on a per driver basis. Does that >>>>>> address your performance concerns ? >>>>> >>>>> Actually, for many cases, IOMMU may *improve* performance. This >>>>> is because it will make it easier to satisfy device constraints >>>>> for DMA resources without requiring the use of a bounce buffer or >>>>> dma windows. >>>> >>>> That is almost always the exception now. The other issue IIRC is >>>> the interaction between the IOMMU and memory caches. In many data >>>> movement appliance contexts you use scatter-gather (scatter-gather >>>> in name only- contiguous memory is good) to physical pages which >>>> are never mapped in. >>>> >>>> The main point here is to avoid a situation that we're currently in >>>> now with respect MSI-X and Interrupt/CPU binding and make sure that >>>> there are sufficient control surfaces to allow for a broad range of >>>> possible implementation choices. >>>> >>>>> >>>>> That said, drivers which assume that setup and tear down of DMA is >>>>> "cheap" may take a perf. hit when using an IOMMU. This is already >>>>> the case for SPARC systems, btw. >>>> >>>> Except for those (ancient) ones that used 36-bit MBus physical >>>> addresses and bypassed the IOMMU. >>>> >>>> >>> >> >> > -- =============================================================================== thread(n):projecting helical rib by which parts can be screwed together =============================================================================== From gdamore@sun.com Thu Sep 4 09:01:32 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m84G1Vfk013111 for ; Thu, 4 Sep 2008 09:01:32 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m84G1S8e029804 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 17:01:30 +0100 (BST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6O00003HUIDZ00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 10:01:30 -0600 (MDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6O00L3ZHUHDR40@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 10:01:29 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m84G1Tnt007833 for ; Thu, 04 Sep 2008 09:01:29 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6O00901FQXNT00@fe-sfbay-09.sun.com> (original mail from gdamore@sun.com) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 09:01:29 -0700 (PDT) Received: from [10.7.251.172] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6O000WKHU405C0@fe-sfbay-09.sun.com>; Thu, 04 Sep 2008 09:01:17 -0700 (PDT) Date: Thu, 04 Sep 2008 09:00:35 -0700 From: "Garrett D'Amore" Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C00462.40909@sun.com> Sender: Garrett.Damore@sun.com To: Vikram Hegde Cc: Kais.Belgaied@sun.com, Matthew Jacob , Jerry Gilliam , PSARC-ext@sun.com Message-id: <48C00623.4000600@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200809032338.m83NcXi3148518@jurassic.eng.sun.com> <48BF223D.1000601@Sun.COM> <48BF26AB.5080109@sun.com> <48BF5A13.9000500@sun.com> <48BF7D4A.5020407@sun.com> <48BFEF6C.1030303@Sun.COM> <48BFFC54.7080407@sun.com> <48C00306.4010809@sun.com> <48C00462.40909@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 4109 Vikram Hegde wrote: > Garrett D'Amore wrote: >> It would be cooler if this could be dealt with using ordinary DMA >> attributes. Won't DDI_DMA_FORCE_PHYSICAL (see the dma_attr_flags for >> ddi_dma_attr(9s)) work for this? >> > Yes, that would work as well and is the preferred and established > solution. This flag will be respected by the IOMMU. However, it seems > that an alternative mechanism that involves no touching of the driver > code is desirable as well. Do you agree ? Yes. I'd prefer not to "document" the alternative mechanism, since I view that as a hack/workaround rather than the correct behavior of using the DMA attributes. -- Garrett > > Thanks, > Vikram >> - Garrett >> >> >> Vikram Hegde wrote: >>> Hi Kais, >>> >>> Below is the text I am planning to add to the PSARC case. And yes, >>> the disable properties will be a part of this PSARC case. >>> >>> Vikram >>> >>> ===================================================================== >>> Enabling/Disabling the IOMMU >>> ============================= >>> It is recognized that driver or platform developers may not in >>> certain cases desire the >>> IOMMU to be enabled. To accomodate such needs we propose the >>> following global properties >>> in the iommu driver.conf file >>> >>> "global-disable" - can be set to 1 to disable IOMMU platform wide. >>> "exclude-list" - list of driver names that want the IOMMU disabled. >>> ====================================================================== >>> >>> Vikram >>> >>> Kais Belgaied wrote: >>>> Vikram, Jerry, >>>> >>>> So, is the ability to disable the IOMMU on a per driver basis in >>>> this case's scope? >>>> what interface is being proposed for doing this? >>>> >>>> Jerry, assuming the answer the to first question above is 'yes', >>>> the case cannot be in >>>> a "closed approved automatic" state while waiting for some of its >>>> spec. >>>> >>>> Kais. >>>> >>>> On 09/03/08 23:16, Matthew Jacob wrote: >>>>> Garrett D'Amore wrote: >>>>>> Vikram Hegde wrote: >>>>>>> Hi, >>>>>>> >>>>>>>> Abilities to disable the IOMMU as a choice? Performance impact >>>>>>>> on various systems and interactions with other projects that >>>>>>>> have gone to some length to work within S/G and its performance >>>>>>>> characteristics. How will this affect them? Cache effects? >>>>>>>> Interactions with some VM work currently being undertaken? >>>>>>>> >>>>>>>> I can't imagine such a major change being >>>>>>>> closed/approved/automatic. >>>>>>> Performance is driver dependant. There is no way a single >>>>>>> project team can test the gamut of drivers out there and certify >>>>>>> that the IOMMU will give acceptable performance for all needs >>>>>>> (which is itself a subjective opinion). We have always planned >>>>>>> to provide the ability to disable the IOMMU on a per driver >>>>>>> basis. Does that address your performance concerns ? >>>>>> >>>>>> Actually, for many cases, IOMMU may *improve* performance. This >>>>>> is because it will make it easier to satisfy device constraints >>>>>> for DMA resources without requiring the use of a bounce buffer or >>>>>> dma windows. >>>>> >>>>> That is almost always the exception now. The other issue IIRC is >>>>> the interaction between the IOMMU and memory caches. In many data >>>>> movement appliance contexts you use scatter-gather (scatter-gather >>>>> in name only- contiguous memory is good) to physical pages which >>>>> are never mapped in. >>>>> >>>>> The main point here is to avoid a situation that we're currently >>>>> in now with respect MSI-X and Interrupt/CPU binding and make sure >>>>> that there are sufficient control surfaces to allow for a broad >>>>> range of possible implementation choices. >>>>> >>>>>> >>>>>> That said, drivers which assume that setup and tear down of DMA >>>>>> is "cheap" may take a perf. hit when using an IOMMU. This is >>>>>> already the case for SPARC systems, btw. >>>>> >>>>> Except for those (ancient) ones that used 36-bit MBus physical >>>>> addresses and bypassed the IOMMU. >>>>> >>>>> >>>> >>> >>> >> > > From Vikram.Hegde@Sun.COM Thu Sep 4 09:11:24 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m84GBNeU014010 for ; Thu, 4 Sep 2008 09:11:24 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m84GBMlB029833 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 5 Sep 2008 00:11:22 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6O00L09IAWDE00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 09:11:20 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6O00KHPIAWEG30@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 09:11:20 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m84GBKaU009159 for ; Thu, 04 Sep 2008 09:11:20 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6O00801F6YO600@fe-sfbay-10.sun.com> (original mail from Vikram.Hegde@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 09:11:20 -0700 (PDT) Received: from [192.168.1.101] ([71.141.142.68]) by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6O00JL5IASZ990@fe-sfbay-10.sun.com>; Thu, 04 Sep 2008 09:11:16 -0700 (PDT) Date: Thu, 04 Sep 2008 09:11:22 -0700 From: Vikram Hegde Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C00623.4000600@sun.com> Sender: Vikram.Hegde@Sun.COM To: "Garrett D'Amore" Cc: Kais.Belgaied@Sun.COM, Matthew Jacob , Jerry Gilliam , PSARC-ext@Sun.COM, Vikram.Hegde@Sun.COM Message-id: <48C008AA.5000904@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200809032338.m83NcXi3148518@jurassic.eng.sun.com> <48BF223D.1000601@Sun.COM> <48BF26AB.5080109@sun.com> <48BF5A13.9000500@sun.com> <48BF7D4A.5020407@sun.com> <48BFEF6C.1030303@Sun.COM> <48BFFC54.7080407@sun.com> <48C00306.4010809@sun.com> <48C00462.40909@sun.com> <48C00623.4000600@sun.com> User-Agent: Thunderbird 2.0.0.16 (Windows/20080708) Status: RO Content-Length: 4896 Hi, > Yes. I'd prefer not to "document" the alternative mechanism, since I > view that as a hack/workaround rather than the correct behavior of > using the DMA attributes. How about 3rd party drivers that driver vendors don't want to rev and re-release solely for this. Shouldn't there be a switch that they can suggest to customers to use with their existing driver binaries ? Vikram Garrett D'Amore wrote: > Vikram Hegde wrote: >> Garrett D'Amore wrote: >>> It would be cooler if this could be dealt with using ordinary DMA >>> attributes. Won't DDI_DMA_FORCE_PHYSICAL (see the dma_attr_flags >>> for ddi_dma_attr(9s)) work for this? >>> >> Yes, that would work as well and is the preferred and established >> solution. This flag will be respected by the IOMMU. However, it seems >> that an alternative mechanism that involves no touching of the driver >> code is desirable as well. Do you agree ? > > Yes. I'd prefer not to "document" the alternative mechanism, since I > view that as a hack/workaround rather than the correct behavior of > using the DMA attributes. > > -- Garrett >> >> Thanks, >> Vikram >>> - Garrett >>> >>> >>> Vikram Hegde wrote: >>>> Hi Kais, >>>> >>>> Below is the text I am planning to add to the PSARC case. And yes, >>>> the disable properties will be a part of this PSARC case. >>>> >>>> Vikram >>>> >>>> ===================================================================== >>>> Enabling/Disabling the IOMMU >>>> ============================= >>>> It is recognized that driver or platform developers may not in >>>> certain cases desire the >>>> IOMMU to be enabled. To accomodate such needs we propose the >>>> following global properties >>>> in the iommu driver.conf file >>>> >>>> "global-disable" - can be set to 1 to disable IOMMU platform wide. >>>> "exclude-list" - list of driver names that want the IOMMU disabled. >>>> ====================================================================== >>>> >>>> Vikram >>>> >>>> Kais Belgaied wrote: >>>>> Vikram, Jerry, >>>>> >>>>> So, is the ability to disable the IOMMU on a per driver basis in >>>>> this case's scope? >>>>> what interface is being proposed for doing this? >>>>> >>>>> Jerry, assuming the answer the to first question above is 'yes', >>>>> the case cannot be in >>>>> a "closed approved automatic" state while waiting for some of its >>>>> spec. >>>>> >>>>> Kais. >>>>> >>>>> On 09/03/08 23:16, Matthew Jacob wrote: >>>>>> Garrett D'Amore wrote: >>>>>>> Vikram Hegde wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>>> Abilities to disable the IOMMU as a choice? Performance impact >>>>>>>>> on various systems and interactions with other projects that >>>>>>>>> have gone to some length to work within S/G and its >>>>>>>>> performance characteristics. How will this affect them? Cache >>>>>>>>> effects? Interactions with some VM work currently being >>>>>>>>> undertaken? >>>>>>>>> >>>>>>>>> I can't imagine such a major change being >>>>>>>>> closed/approved/automatic. >>>>>>>> Performance is driver dependant. There is no way a single >>>>>>>> project team can test the gamut of drivers out there and >>>>>>>> certify that the IOMMU will give acceptable performance for >>>>>>>> all needs (which is itself a subjective opinion). We have >>>>>>>> always planned to provide the ability to disable the IOMMU on a >>>>>>>> per driver basis. Does that address your performance concerns ? >>>>>>> >>>>>>> Actually, for many cases, IOMMU may *improve* performance. This >>>>>>> is because it will make it easier to satisfy device constraints >>>>>>> for DMA resources without requiring the use of a bounce buffer >>>>>>> or dma windows. >>>>>> >>>>>> That is almost always the exception now. The other issue IIRC is >>>>>> the interaction between the IOMMU and memory caches. In many data >>>>>> movement appliance contexts you use scatter-gather >>>>>> (scatter-gather in name only- contiguous memory is good) to >>>>>> physical pages which are never mapped in. >>>>>> >>>>>> The main point here is to avoid a situation that we're currently >>>>>> in now with respect MSI-X and Interrupt/CPU binding and make sure >>>>>> that there are sufficient control surfaces to allow for a broad >>>>>> range of possible implementation choices. >>>>>> >>>>>>> >>>>>>> That said, drivers which assume that setup and tear down of DMA >>>>>>> is "cheap" may take a perf. hit when using an IOMMU. This is >>>>>>> already the case for SPARC systems, btw. >>>>>> >>>>>> Except for those (ancient) ones that used 36-bit MBus physical >>>>>> addresses and bypassed the IOMMU. >>>>>> >>>>>> >>>>> >>>> >>>> >>> >> >> > -- =============================================================================== thread(n):projecting helical rib by which parts can be screwed together =============================================================================== From Wesley.Shao@sun.com Thu Sep 4 12:50:50 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m84JonxO011250 for ; Thu, 4 Sep 2008 12:50:50 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m84JokXh007404 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 20:50:48 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6O0060DSGN7Y00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 12:50:47 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6O002A7SGMNL70@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 12:50:46 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m84Jofkv010822 for ; Thu, 04 Sep 2008 12:50:46 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6O00G01SG7WH00@fe-sfbay-10.sun.com> (original mail from Wesley.Shao@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 12:50:42 -0700 (PDT) Received: from [129.146.96.108] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6O003RDSGFMJ30@fe-sfbay-10.sun.com>; Thu, 04 Sep 2008 12:50:40 -0700 (PDT) Date: Thu, 04 Sep 2008 12:50:12 -0700 From: Wesley Shao Subject: Re: AMD IOMMU [PSARC/2008/561] Sender: Wesley.Shao@sun.com To: gdamore@sun.com, Kais.Belgaied@sun.com, Matthew Jacob , Vikram.Hegde@sun.com, Jerry Gilliam , PSARC-ext@sun.com Message-id: <48C03BF4.6080705@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 User-Agent: Thunderbird 2.0.0.14 (X11/20080714) Status: RO Content-Length: 4887 FORCE_PHYSICAL allows finer granularity and the proposed properties make the drivers non-portable to SPARC platforms. They should really be used for internal debug. Besides, they make the drivers hard to run on virtualized platforms. Wes Vikram Hedgde wrote: >Hi, > >> Yes. I'd prefer not to "document" the alternative mechanism, since I >> view that as a hack/workaround rather than the correct behavior of >> using the DMA attributes. >How about 3rd party drivers that driver vendors don't want to rev and >re-release solely for this. Shouldn't there be a switch that they can >suggest to customers to use with their existing driver binaries ? > >Vikram > > >Garrett D'Amore wrote: > Vikram Hegde wrote: >> Garrett D'Amore wrote: >>> It would be cooler if this could be dealt with using ordinary DMA >>> attributes. Won't DDI_DMA_FORCE_PHYSICAL (see the dma_attr_flags >>> for ddi_dma_attr(9s)) work for this? >>> >> Yes, that would work as well and is the preferred and established >> solution. This flag will be respected by the IOMMU. However, it seems >> that an alternative mechanism that involves no touching of the driver >> code is desirable as well. Do you agree ? > > Yes. I'd prefer not to "document" the alternative mechanism, since I > view that as a hack/workaround rather than the correct behavior of > using the DMA attributes. > > -- Garrett >> >> Thanks, >> Vikram >>> - Garrett >>> >>> >>> Vikram Hegde wrote: >>>> Hi Kais, >>>> >>>> Below is the text I am planning to add to the PSARC case. And yes, >>>> the disable properties will be a part of this PSARC case. >>>> >>>> Vikram >>>> >>>> ===================================================================== >>>> Enabling/Disabling the IOMMU >>>> ============================= >>>> It is recognized that driver or platform developers may not in >>>> certain cases desire the >>>> IOMMU to be enabled. To accomodate such needs we propose the >>>> following global properties >>>> in the iommu driver.conf file >>>> >>>> "global-disable" - can be set to 1 to disable IOMMU platform wide. >>>> "exclude-list" - list of driver names that want the IOMMU disabled. >>>> ====================================================================== >>>> >>>> Vikram >>>> >>>> Kais Belgaied wrote: >>>>> Vikram, Jerry, >>>>> >>>>> So, is the ability to disable the IOMMU on a per driver basis in >>>>> this case's scope? >>>>> what interface is being proposed for doing this? >>>>> >>>>> Jerry, assuming the answer the to first question above is 'yes', >>>>> the case cannot be in >>>>> a "closed approved automatic" state while waiting for some of its >>>>> spec. >>>>> >>>>> Kais. >>>>> >>>>> On 09/03/08 23:16, Matthew Jacob wrote: >>>>>> Garrett D'Amore wrote: >>>>>>> Vikram Hegde wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>>> Abilities to disable the IOMMU as a choice? Performance impact >>>>>>>>> on various systems and interactions with other projects that >>>>>>>>> have gone to some length to work within S/G and its >>>>>>>>> performance characteristics. How will this affect them? Cache >>>>>>>>> effects? Interactions with some VM work currently being >>>>>>>>> undertaken? >>>>>>>>> >>>>>>>>> I can't imagine such a major change being >>>>>>>>> closed/approved/automatic. >>>>>>>> Performance is driver dependant. There is no way a single >>>>>>>> project team can test the gamut of drivers out there and >>>>>>>> certify that the IOMMU will give acceptable performance for >>>>>>>> all needs (which is itself a subjective opinion). We have >>>>>>>> always planned to provide the ability to disable the IOMMU on a >>>>>>>> per driver basis. Does that address your performance concerns ? >>>>>>> >>>>>>> Actually, for many cases, IOMMU may *improve* performance. This >>>>>>> is because it will make it easier to satisfy device constraints >>>>>>> for DMA resources without requiring the use of a bounce buffer >>>>>>> or dma windows. >>>>>> >>>>>> That is almost always the exception now. The other issue IIRC is >>>>>> the interaction between the IOMMU and memory caches. In many data >>>>>> movement appliance contexts you use scatter-gather >>>>>> (scatter-gather in name only- contiguous memory is good) to >>>>>> physical pages which are never mapped in. >>>>>> >>>>>> The main point here is to avoid a situation that we're currently >>>>>> in now with respect MSI-X and Interrupt/CPU binding and make sure >>>>>> that there are sufficient control surfaces to allow for a broad >>>>>> range of possible implementation choices. >>>>>> >>>>>>> >>>>>>> That said, drivers which assume that setup and tear down of DMA >>>>>>> is "cheap" may take a perf. hit when using an IOMMU. This is >>>>>>> already the case for SPARC systems, btw. >>>>>> >>>>>> Except for those (ancient) ones that used 36-bit MBus physical >>>>>> addresses and bypassed the IOMMU. From gdamore@sun.com Thu Sep 4 13:20:46 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m84KKjPt012852 for ; Thu, 4 Sep 2008 13:20:46 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m84KKg1M065212 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 14:20:45 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6O0050XTUJD300@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 13:20:43 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6O003J5TUJEK40@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 13:20:43 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m84KKhUc016412 for ; Thu, 04 Sep 2008 13:20:43 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6O00301TD77N00@fe-sfbay-10.sun.com> (original mail from gdamore@sun.com) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 13:20:43 -0700 (PDT) Received: from [10.7.251.172] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6O00HNMTUDVU10@fe-sfbay-10.sun.com>; Thu, 04 Sep 2008 13:20:37 -0700 (PDT) Date: Thu, 04 Sep 2008 13:19:54 -0700 From: "Garrett D'Amore" Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C03BF4.6080705@sun.com> Sender: Garrett.Damore@sun.com To: Wesley Shao Cc: Kais.Belgaied@sun.com, Matthew Jacob , Vikram.Hegde@sun.com, Jerry Gilliam , PSARC-ext@sun.com Message-id: <48C042EA.20402@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48C03BF4.6080705@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 5940 Furthermore, drivers that don't want to re-spin for this should continue to function normally. The only reason to avoid use of the IOMMU is either performance optimization or to workaround incorrect driver implementation. As indicated, I don't mind have have some properties "undocumented" usage, but we really should be encouraging folks that care to avoid the IOMMU to use the documented API. (And, IMO, there really should be no reason to avoid the IOMMU if you've written your code to perform well under regular DDI compliant DMA. The IOMMU itself affords some significant benefits, making it much easier to get single contiguous DMA segments, reducing the need to deal with multiple cookies -- scatter gather -- or DMA windows. Never mind the huge benefits it affords for virtualization and the fault tolerance benefits that come for free as well.) -- Garrett Wesley Shao wrote: > FORCE_PHYSICAL allows finer granularity and the proposed properties > make the drivers non-portable to SPARC platforms. They should really be > used for internal debug. > > Besides, they make the drivers hard to run on virtualized platforms. > > Wes > > Vikram Hedgde wrote: >> Hi, >> >>> Yes. I'd prefer not to "document" the alternative mechanism, since >>> I view that as a hack/workaround rather than the correct behavior of >>> using the DMA attributes. >> How about 3rd party drivers that driver vendors don't want to rev and >> re-release solely for this. Shouldn't there be a switch that they can >> suggest to customers to use with their existing driver binaries ? >> >> Vikram >> >> >> Garrett D'Amore wrote: >> Vikram Hegde wrote: >>> Garrett D'Amore wrote: >>>> It would be cooler if this could be dealt with using ordinary DMA >>>> attributes. Won't DDI_DMA_FORCE_PHYSICAL (see the dma_attr_flags >>>> for ddi_dma_attr(9s)) work for this? >>>> >>> Yes, that would work as well and is the preferred and established >>> solution. This flag will be respected by the IOMMU. However, it >>> seems that an alternative mechanism that involves no touching of the >>> driver code is desirable as well. Do you agree ? >> >> Yes. I'd prefer not to "document" the alternative mechanism, since I >> view that as a hack/workaround rather than the correct behavior of >> using the DMA attributes. >> >> -- Garrett >>> >>> Thanks, >>> Vikram >>>> - Garrett >>>> >>>> >>>> Vikram Hegde wrote: >>>>> Hi Kais, >>>>> >>>>> Below is the text I am planning to add to the PSARC case. And yes, >>>>> the disable properties will be a part of this PSARC case. >>>>> >>>>> Vikram >>>>> >>>>> ===================================================================== >>>>> Enabling/Disabling the IOMMU >>>>> ============================= >>>>> It is recognized that driver or platform developers may not in >>>>> certain cases desire the >>>>> IOMMU to be enabled. To accomodate such needs we propose the >>>>> following global properties >>>>> in the iommu driver.conf file >>>>> >>>>> "global-disable" - can be set to 1 to disable IOMMU platform wide. >>>>> "exclude-list" - list of driver names that want the IOMMU disabled. >>>>> ====================================================================== >>>>> >>>>> >>>>> Vikram >>>>> >>>>> Kais Belgaied wrote: >>>>>> Vikram, Jerry, >>>>>> >>>>>> So, is the ability to disable the IOMMU on a per driver basis in >>>>>> this case's scope? >>>>>> what interface is being proposed for doing this? >>>>>> >>>>>> Jerry, assuming the answer the to first question above is 'yes', >>>>>> the case cannot be in >>>>>> a "closed approved automatic" state while waiting for some of its >>>>>> spec. >>>>>> >>>>>> Kais. >>>>>> >>>>>> On 09/03/08 23:16, Matthew Jacob wrote: >>>>>>> Garrett D'Amore wrote: >>>>>>>> Vikram Hegde wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>>> Abilities to disable the IOMMU as a choice? Performance >>>>>>>>>> impact on various systems and interactions with other >>>>>>>>>> projects that have gone to some length to work within S/G and >>>>>>>>>> its performance characteristics. How will this affect them? >>>>>>>>>> Cache effects? Interactions with some VM work currently being >>>>>>>>>> undertaken? >>>>>>>>>> >>>>>>>>>> I can't imagine such a major change being >>>>>>>>>> closed/approved/automatic. >>>>>>>>> Performance is driver dependant. There is no way a single >>>>>>>>> project team can test the gamut of drivers out there and >>>>>>>>> certify that the IOMMU will give acceptable performance for >>>>>>>>> all needs (which is itself a subjective opinion). We have >>>>>>>>> always planned to provide the ability to disable the IOMMU on >>>>>>>>> a per driver basis. Does that address your performance concerns ? >>>>>>>> >>>>>>>> Actually, for many cases, IOMMU may *improve* performance. >>>>>>>> This is because it will make it easier to satisfy device >>>>>>>> constraints for DMA resources without requiring the use of a >>>>>>>> bounce buffer or dma windows. >>>>>>> >>>>>>> That is almost always the exception now. The other issue IIRC is >>>>>>> the interaction between the IOMMU and memory caches. In many >>>>>>> data movement appliance contexts you use scatter-gather >>>>>>> (scatter-gather in name only- contiguous memory is good) to >>>>>>> physical pages which are never mapped in. >>>>>>> >>>>>>> The main point here is to avoid a situation that we're currently >>>>>>> in now with respect MSI-X and Interrupt/CPU binding and make >>>>>>> sure that there are sufficient control surfaces to allow for a >>>>>>> broad range of possible implementation choices. >>>>>>> >>>>>>>> >>>>>>>> That said, drivers which assume that setup and tear down of DMA >>>>>>>> is "cheap" may take a perf. hit when using an IOMMU. This is >>>>>>>> already the case for SPARC systems, btw. >>>>>>> >>>>>>> Except for those (ancient) ones that used 36-bit MBus physical >>>>>>> addresses and bypassed the IOMMU. > > From vikram.hegde@sun.com Thu Sep 4 18:00:53 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m8510qOb023825 for ; Thu, 4 Sep 2008 18:00:52 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m8510oOE018989 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 19:00:52 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6P00D016TF0H00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 18:00:51 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6P00DE86TEH9D0@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 18:00:50 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m8510oxV018536 for ; Thu, 04 Sep 2008 18:00:50 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6P00H016M9WS00@fe-sfbay-09.sun.com> (original mail from Vikram.Hegde@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 18:00:50 -0700 (PDT) Received: from [129.150.17.165] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6P00JB96TD1P80@fe-sfbay-09.sun.com>; Thu, 04 Sep 2008 18:00:50 -0700 (PDT) Date: Thu, 04 Sep 2008 18:00:58 -0700 From: Vikram Hegde Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C042EA.20402@sun.com> Sender: vikram.hegde@sun.com To: "Garrett D'Amore" Cc: Wesley Shao , Kais.Belgaied@sun.com, Matthew Jacob , Jerry Gilliam , PSARC-ext@sun.com, vikram.hegde@sun.com Message-id: <48C084CA.20401@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48C03BF4.6080705@sun.com> <48C042EA.20402@sun.com> User-Agent: Thunderbird 2.0.0.16 (Windows/20080708) Status: RO Content-Length: 6619 Hi all, ok, what I am hearing is that we keep these properties Sun internal and maybe use for debugging and testing. So I will keep this in the code but will not document it. Is this acceptable to all ? Vikram Garrett D'Amore wrote: > Furthermore, drivers that don't want to re-spin for this should > continue to function normally. The only reason to avoid use of the > IOMMU is either performance optimization or to workaround incorrect > driver implementation. > > As indicated, I don't mind have have some properties "undocumented" > usage, but we really should be encouraging folks that care to avoid > the IOMMU to use the documented API. > > (And, IMO, there really should be no reason to avoid the IOMMU if > you've written your code to perform well under regular DDI compliant > DMA. The IOMMU itself affords some significant benefits, making it > much easier to get single contiguous DMA segments, reducing the need > to deal with multiple cookies -- scatter gather -- or DMA windows. > Never mind the huge benefits it affords for virtualization and the > fault tolerance benefits that come for free as well.) > > -- Garrett > > Wesley Shao wrote: >> FORCE_PHYSICAL allows finer granularity and the proposed properties >> make the drivers non-portable to SPARC platforms. They should really be >> used for internal debug. >> >> Besides, they make the drivers hard to run on virtualized platforms. >> >> Wes >> >> Vikram Hedgde wrote: >>> Hi, >>> >>>> Yes. I'd prefer not to "document" the alternative mechanism, since >>>> I view that as a hack/workaround rather than the correct behavior >>>> of using the DMA attributes. >>> How about 3rd party drivers that driver vendors don't want to rev >>> and re-release solely for this. Shouldn't there be a switch that >>> they can suggest to customers to use with their existing driver >>> binaries ? >>> >>> Vikram >>> >>> >>> Garrett D'Amore wrote: >>> Vikram Hegde wrote: >>>> Garrett D'Amore wrote: >>>>> It would be cooler if this could be dealt with using ordinary DMA >>>>> attributes. Won't DDI_DMA_FORCE_PHYSICAL (see the dma_attr_flags >>>>> for ddi_dma_attr(9s)) work for this? >>>>> >>>> Yes, that would work as well and is the preferred and established >>>> solution. This flag will be respected by the IOMMU. However, it >>>> seems that an alternative mechanism that involves no touching of >>>> the driver code is desirable as well. Do you agree ? >>> >>> Yes. I'd prefer not to "document" the alternative mechanism, since >>> I view that as a hack/workaround rather than the correct behavior of >>> using the DMA attributes. >>> >>> -- Garrett >>>> >>>> Thanks, >>>> Vikram >>>>> - Garrett >>>>> >>>>> >>>>> Vikram Hegde wrote: >>>>>> Hi Kais, >>>>>> >>>>>> Below is the text I am planning to add to the PSARC case. And >>>>>> yes, the disable properties will be a part of this PSARC case. >>>>>> >>>>>> Vikram >>>>>> >>>>>> ===================================================================== >>>>>> >>>>>> Enabling/Disabling the IOMMU >>>>>> ============================= >>>>>> It is recognized that driver or platform developers may not in >>>>>> certain cases desire the >>>>>> IOMMU to be enabled. To accomodate such needs we propose the >>>>>> following global properties >>>>>> in the iommu driver.conf file >>>>>> >>>>>> "global-disable" - can be set to 1 to disable IOMMU platform wide. >>>>>> "exclude-list" - list of driver names that want the IOMMU >>>>>> disabled. >>>>>> ====================================================================== >>>>>> >>>>>> >>>>>> Vikram >>>>>> >>>>>> Kais Belgaied wrote: >>>>>>> Vikram, Jerry, >>>>>>> >>>>>>> So, is the ability to disable the IOMMU on a per driver basis in >>>>>>> this case's scope? >>>>>>> what interface is being proposed for doing this? >>>>>>> >>>>>>> Jerry, assuming the answer the to first question above is >>>>>>> 'yes', the case cannot be in >>>>>>> a "closed approved automatic" state while waiting for some of >>>>>>> its spec. >>>>>>> >>>>>>> Kais. >>>>>>> >>>>>>> On 09/03/08 23:16, Matthew Jacob wrote: >>>>>>>> Garrett D'Amore wrote: >>>>>>>>> Vikram Hegde wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>>> Abilities to disable the IOMMU as a choice? Performance >>>>>>>>>>> impact on various systems and interactions with other >>>>>>>>>>> projects that have gone to some length to work within S/G >>>>>>>>>>> and its performance characteristics. How will this affect >>>>>>>>>>> them? Cache effects? Interactions with some VM work >>>>>>>>>>> currently being undertaken? >>>>>>>>>>> >>>>>>>>>>> I can't imagine such a major change being >>>>>>>>>>> closed/approved/automatic. >>>>>>>>>> Performance is driver dependant. There is no way a single >>>>>>>>>> project team can test the gamut of drivers out there and >>>>>>>>>> certify that the IOMMU will give acceptable performance for >>>>>>>>>> all needs (which is itself a subjective opinion). We have >>>>>>>>>> always planned to provide the ability to disable the IOMMU on >>>>>>>>>> a per driver basis. Does that address your performance >>>>>>>>>> concerns ? >>>>>>>>> >>>>>>>>> Actually, for many cases, IOMMU may *improve* performance. >>>>>>>>> This is because it will make it easier to satisfy device >>>>>>>>> constraints for DMA resources without requiring the use of a >>>>>>>>> bounce buffer or dma windows. >>>>>>>> >>>>>>>> That is almost always the exception now. The other issue IIRC >>>>>>>> is the interaction between the IOMMU and memory caches. In many >>>>>>>> data movement appliance contexts you use scatter-gather >>>>>>>> (scatter-gather in name only- contiguous memory is good) to >>>>>>>> physical pages which are never mapped in. >>>>>>>> >>>>>>>> The main point here is to avoid a situation that we're >>>>>>>> currently in now with respect MSI-X and Interrupt/CPU binding >>>>>>>> and make sure that there are sufficient control surfaces to >>>>>>>> allow for a broad range of possible implementation choices. >>>>>>>> >>>>>>>>> >>>>>>>>> That said, drivers which assume that setup and tear down of >>>>>>>>> DMA is "cheap" may take a perf. hit when using an IOMMU. This >>>>>>>>> is already the case for SPARC systems, btw. >>>>>>>> >>>>>>>> Except for those (ancient) ones that used 36-bit MBus physical >>>>>>>> addresses and bypassed the IOMMU. >> >> > -- =============================================================================== thread(n):projecting helical rib by which parts can be screwed together =============================================================================== From Matthew.Jacob@sun.com Thu Sep 4 18:11:36 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m851BaWG023888 for ; Thu, 4 Sep 2008 18:11:36 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m851BVhD002012 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 5 Sep 2008 09:11:35 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6P00F017B8R000@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 19:11:32 -0600 (MDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6P00M367B7C8E0@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 19:11:32 -0600 (MDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m851BVNx014670 for ; Thu, 04 Sep 2008 18:11:31 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6P0040179JEE00@fe-sfbay-10.sun.com> (original mail from Matthew.Jacob@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 18:11:31 -0700 (PDT) Received: from [192.168.1.7] ([72.164.148.65]) by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6P003YD7B6A4E0@fe-sfbay-10.sun.com>; Thu, 04 Sep 2008 18:11:31 -0700 (PDT) Date: Thu, 04 Sep 2008 18:10:03 -0700 From: Matthew Jacob Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C084CA.20401@sun.com> Sender: Matthew.Jacob@sun.com To: Vikram Hegde Cc: "Garrett D'Amore" , Wesley Shao , Kais.Belgaied@sun.com, Jerry Gilliam , PSARC-ext@sun.com Message-id: <48C086EB.2020003@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48C03BF4.6080705@sun.com> <48C042EA.20402@sun.com> <48C084CA.20401@sun.com> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.16) Gecko/20080702 SeaMonkey/1.1.11 Status: RO Content-Length: 249 Vikram Hegde wrote: > Hi all, > > ok, what I am hearing is that we keep these properties Sun internal > and maybe use for debugging and testing. So I will keep this in the > code but will not document it. Is this acceptable to all ? > No, sorry. From Wesley.Shao@sun.com Thu Sep 4 18:17:46 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m851HkJZ023979 for ; Thu, 4 Sep 2008 18:17:46 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m851HhLt021655 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 18:17:44 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6P00E077LHV600@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 18:17:41 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6P00DT47LGHDD0@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 18:17:40 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m851He36019511 for ; Thu, 04 Sep 2008 18:17:40 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6P00A017GZ8N00@fe-sfbay-09.sun.com> (original mail from Wesley.Shao@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 18:17:40 -0700 (PDT) Received: from [129.146.96.108] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6P00JB17LF1PC0@fe-sfbay-09.sun.com>; Thu, 04 Sep 2008 18:17:39 -0700 (PDT) Date: Thu, 04 Sep 2008 18:17:12 -0700 From: Wesley Shao Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C086EB.2020003@sun.com> Sender: Wesley.Shao@sun.com To: Matthew Jacob Cc: Vikram Hegde , "Garrett D'Amore" , Kais.Belgaied@sun.com, Jerry Gilliam , PSARC-ext@sun.com Message-id: <48C08898.7080109@sun.com> MIME-version: 1.0 Content-type: text/html; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48C03BF4.6080705@sun.com> <48C042EA.20402@sun.com> <48C084CA.20401@sun.com> <48C086EB.2020003@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080714) Status: RO Content-Length: 821 Matthew Jacob wrote:
Vikram Hegde wrote:
Hi all,

ok, what I am hearing is that we keep these properties Sun internal and maybe use for debugging and testing. So I will keep this in the code but will not document it. Is this acceptable to all ?


No, sorry.
Would you care to explain why not? This should give you the
flexibility of measuring performance without iommu in the way.

Wes
 From Vikram.Hegde@sun.com Thu Sep 4 18:19:15 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m851JFil023993 for ; Thu, 4 Sep 2008 18:19:15 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m851JDLo021960 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 18:19:15 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6P00G057O2AF00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 19:19:14 -0600 (MDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6P00G247O18000@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 19:19:13 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m851JDk4019590 for ; Thu, 04 Sep 2008 18:19:13 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6P00A017GZ8N00@fe-sfbay-09.sun.com> (original mail from Vikram.Hegde@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 18:19:13 -0700 (PDT) Received: from [129.150.17.165] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6P00JR37O01PC0@fe-sfbay-09.sun.com>; Thu, 04 Sep 2008 18:19:13 -0700 (PDT) Date: Thu, 04 Sep 2008 18:19:21 -0700 From: Vikram Hegde Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C086EB.2020003@sun.com> Sender: Vikram.Hegde@sun.com To: Matthew Jacob Cc: "Garrett D'Amore" , Wesley Shao , Kais.Belgaied@sun.com, Jerry Gilliam , PSARC-ext@sun.com, Vikram.Hegde@sun.com Message-id: <48C08919.8040006@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48C03BF4.6080705@sun.com> <48C042EA.20402@sun.com> <48C084CA.20401@sun.com> <48C086EB.2020003@sun.com> User-Agent: Thunderbird 2.0.0.16 (Windows/20080708) Status: RO Content-Length: 777 Hi Matthew, > No, sorry. Any reason why FORCE_PHYSICAL is not acceptable ? It has been around for the SPARC side for some time now and seems to meet the needs of everyone on that side. It provides exactly the same functionality as you want. Vikram Matthew Jacob wrote: > Vikram Hegde wrote: >> Hi all, >> >> ok, what I am hearing is that we keep these properties Sun internal >> and maybe use for debugging and testing. So I will keep this in the >> code but will not document it. Is this acceptable to all ? >> > > No, sorry. -- =============================================================================== thread(n):projecting helical rib by which parts can be screwed together =============================================================================== From Matthew.Jacob@sun.com Thu Sep 4 18:41:38 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m851fcNW024252 for ; Thu, 4 Sep 2008 18:41:38 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m851fa7P028114 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 19:41:37 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6P00H0B8PCQJ00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 18:41:36 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6P00DZL8PBHDF0@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 18:41:35 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m851fZ58015948 for ; Thu, 04 Sep 2008 18:41:35 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6P001018L7PU00@fe-sfbay-10.sun.com> (original mail from Matthew.Jacob@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 18:41:35 -0700 (PDT) Received: from [192.168.1.7] ([72.164.148.65]) by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6P00AKO8PA7M50@fe-sfbay-10.sun.com>; Thu, 04 Sep 2008 18:41:35 -0700 (PDT) Date: Thu, 04 Sep 2008 18:40:07 -0700 From: Matthew Jacob Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C08919.8040006@sun.com> Sender: Matthew.Jacob@sun.com To: Vikram Hegde Cc: "Garrett D'Amore" , Wesley Shao , Kais.Belgaied@sun.com, Jerry Gilliam , PSARC-ext@sun.com Message-id: <48C08DF7.3020402@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48C03BF4.6080705@sun.com> <48C042EA.20402@sun.com> <48C084CA.20401@sun.com> <48C086EB.2020003@sun.com> <48C08919.8040006@sun.com> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.16) Gecko/20080702 SeaMonkey/1.1.11 Status: RO Content-Length: 3439 Vikram Hegde wrote: > Hi Matthew, >> No, sorry. > Any reason why FORCE_PHYSICAL is not acceptable ? It has been around > for the SPARC side for some time now and seems to meet the needs of > everyone on that side. It provides exactly the same functionality as > you want. > Because it changes the effective semantics of what it was previously. Previously it was a performance hack to get 36 bit MBus addresses and get around the IOMMU limitations (at least IIRC) and as such it encapsulated a semantic that was specific to SPARC. So much so that the man page even mentions it. S/G DMA on the other hand, has been specific more to non-SPARC and has been the default there. Turning around and now making it /not/ the default on the /hope/ that those who don't want that change will get around to recompiling and changing their drivers ignores the intent of what the DDI is about. In short, similarity in names of objects does not mean similarity in effective and assumed semantics. In general I think it's bad policy for Sun to create interfaces that consumers of those interfaces cannot disable if there are workable alternates. It's even more problematic when the default underlying implementation just changes. Furthermore, even if you classified such interfaces as evolving, things that have been long standing have at least some expectation of remaining the same from (minor) release to release. The /whole /point of the DDI is to create a framework that is friendly and usable and reliable for third parties to use. It does /not/ exist to have private side interfaces for Sun internal use only, although nearly all of the DDI has ended up that way (which was /not/ the intent of some of us when we ent through the effort of starting this). You have in some senses two goals here now, though. The first is the DDI idea. The second is to improve the underlying framework in a way that is believable and measurable and helps support broadest framework within Sun itself. As such, the IOMMU addition is probably a good idea- certainly the people proposing it believe so or they wouldn't be doing it. However, if you're not making this a major release type of change (in which case, IMHO, such as it is, lots of things should be thrown out and redone from scratch), then the addition of the IOMMU as the underlying transport instead of S/G DMA should be done in a less invasive way so that previous binaries don't actually do anything different (as a default). If it turns out that they are /provably/ correct in environments where the IOMMU is used instead, then the P-teams and the third parties could then either recompile or otherwise turn on that usage. That is prudent. So if you're going to make such a big underlying change, I would suggest instead that until it's a major release, you could add a FORCE_IOMMU flag. This would have the effect of taking driver binaries to work exactly as they have done before and allow drivers written by both Sun and third parties to /try out/ the IOMMU environment as needed or desired. The above isn't probably reasoned out as well as it should be, but it does seem to me that you should either dump the DDI and it's overcautious "let's not change things (for decades)" approach (and take an approach more like (the good parts) of Linux), or if you're going to treat the DDI in its multiple roles, err very much on the side of caution. -matt From Vikram.Hegde@sun.com Thu Sep 4 19:04:56 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m8524tI6024656 for ; Thu, 4 Sep 2008 19:04:55 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m8524oq3017523 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 5 Sep 2008 10:04:54 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6P00M0D9S4YZ00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 19:04:52 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6P00DRP9S4KN70@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 19:04:52 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m8524p03016608 for ; Thu, 04 Sep 2008 19:04:51 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6P00G019LH7N00@fe-sfbay-10.sun.com> (original mail from Vikram.Hegde@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 19:04:51 -0700 (PDT) Received: from [129.150.17.165] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6P00AZJ9S37M90@fe-sfbay-10.sun.com>; Thu, 04 Sep 2008 19:04:51 -0700 (PDT) Date: Thu, 04 Sep 2008 19:04:59 -0700 From: Vikram Hegde Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C08DF7.3020402@sun.com> Sender: Vikram.Hegde@sun.com To: Matthew Jacob Cc: "Garrett D'Amore" , Wesley Shao , Kais.Belgaied@sun.com, Jerry Gilliam , PSARC-ext@sun.com, Vikram.Hegde@sun.com Message-id: <48C093CB.2020802@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48C03BF4.6080705@sun.com> <48C042EA.20402@sun.com> <48C084CA.20401@sun.com> <48C086EB.2020003@sun.com> <48C08919.8040006@sun.com> <48C08DF7.3020402@sun.com> User-Agent: Thunderbird 2.0.0.16 (Windows/20080708) Status: RO Content-Length: 4444 Hi, I think specifiying a FORCE_IOMMU flag will turn this into an experimental feature that will never become mainstream in the forseeable feature. Driver developers will simply not take interest until it becomes mainstream and it will never become mainstream without developer interest. We will also be at a competitive disadvantage compared to Windows and Linux both of which will have IOMMU support. I agree with Garrett and Wes that using FORCE_PHYSICAL is probably the right balance of caution (there is an out for driver developers) and evolving the DDI - we can't afford a glacial pace. Vikram Matthew Jacob wrote: > Vikram Hegde wrote: >> Hi Matthew, >>> No, sorry. >> Any reason why FORCE_PHYSICAL is not acceptable ? It has been around >> for the SPARC side for some time now and seems to meet the needs of >> everyone on that side. It provides exactly the same functionality as >> you want. >> > Because it changes the effective semantics of what it was previously. > Previously it was a performance hack to get 36 bit MBus addresses and > get around the IOMMU limitations (at least IIRC) and as such it > encapsulated a semantic that was specific to SPARC. So much so that > the man page even mentions it. > > S/G DMA on the other hand, has been specific more to non-SPARC and has > been the default there. Turning around and now making it /not/ the > default on the /hope/ that those who don't want that change will get > around to recompiling and changing their drivers ignores the intent of > what the DDI is about. > > In short, similarity in names of objects does not mean similarity in > effective and assumed semantics. > > In general I think it's bad policy for Sun to create interfaces that > consumers of those interfaces cannot disable if there are workable > alternates. It's even more problematic when the default underlying > implementation just changes. Furthermore, even if you classified such > interfaces as evolving, things that have been long standing have at > least some expectation of remaining the same from (minor) release to > release. > > The /whole /point of the DDI is to create a framework that is friendly > and usable and reliable for third parties to use. It does /not/ exist > to have private side interfaces for Sun internal use only, although > nearly all of the DDI has ended up that way (which was /not/ the > intent of some of us when we ent through the effort of starting this). > > You have in some senses two goals here now, though. The first is the > DDI idea. The second is to improve the underlying framework in a way > that is believable and measurable and helps support broadest framework > within Sun itself. As such, the IOMMU addition is probably a good > idea- certainly the people proposing it believe so or they wouldn't be > doing it. > > However, if you're not making this a major release type of change (in > which case, IMHO, such as it is, lots of things should be thrown out > and redone from scratch), then the addition of the IOMMU as the > underlying transport instead of S/G DMA should be done in a less > invasive way so that previous binaries don't actually do anything > different (as a default). If it turns out that they are /provably/ > correct in environments where the IOMMU is used instead, then the > P-teams and the third parties could then either recompile or otherwise > turn on that usage. That is prudent. > > So if you're going to make such a big underlying change, I would > suggest instead that until it's a major release, you could add a > FORCE_IOMMU flag. This would have the effect of taking driver binaries > to work exactly as they have done before and allow drivers written by > both Sun and third parties to /try out/ the IOMMU environment as > needed or desired. > > The above isn't probably reasoned out as well as it should be, but it > does seem to me that you should either dump the DDI and it's > overcautious "let's not change things (for decades)" approach (and > take an approach more like (the good parts) of Linux), or if you're > going to treat the DDI in its multiple roles, err very much on the > side of caution. > > -matt > > -- =============================================================================== thread(n):projecting helical rib by which parts can be screwed together =============================================================================== From Matthew.Jacob@Sun.COM Thu Sep 4 19:08:09 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m85289bq024674 for ; Thu, 4 Sep 2008 19:08:09 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m85288oa033429 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 20:08:08 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6P00N059XK4Q00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 19:08:08 -0700 (PDT) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6P00D2F9XJKO80@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 19:08:07 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m852873Y016723 for ; Thu, 04 Sep 2008 19:08:07 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6P00J019T39P00@fe-sfbay-10.sun.com> (original mail from Matthew.Jacob@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 19:08:07 -0700 (PDT) Received: from [192.168.1.7] ([72.164.148.65]) by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6P00AMS9XI7MA0@fe-sfbay-10.sun.com>; Thu, 04 Sep 2008 19:08:07 -0700 (PDT) Date: Thu, 04 Sep 2008 19:06:40 -0700 From: Matthew Jacob Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C093CB.2020802@sun.com> Sender: Matthew.Jacob@Sun.COM To: Vikram Hegde Cc: "Garrett D'Amore" , Wesley Shao , Kais.Belgaied@Sun.COM, Jerry Gilliam , PSARC-ext@Sun.COM Message-id: <48C09430.3070109@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48C03BF4.6080705@sun.com> <48C042EA.20402@sun.com> <48C084CA.20401@sun.com> <48C086EB.2020003@sun.com> <48C08919.8040006@sun.com> <48C08DF7.3020402@sun.com> <48C093CB.2020802@sun.com> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.16) Gecko/20080702 SeaMonkey/1.1.11 Status: RO Content-Length: 698 Vikram Hegde wrote: > Hi, > > I think specifiying a FORCE_IOMMU flag will turn this into an > experimental feature that will never become mainstream in the > forseeable feature. Driver developers will simply not take interest > until it becomes mainstream and it will never become mainstream > without developer interest. We will also be at a competitive > disadvantage compared to Windows and Linux both of which will have > IOMMU support. I agree with Garrett and Wes that using FORCE_PHYSICAL > is probably the right balance of caution (there is an out for driver > developers) and evolving the DDI - we can't afford a glacial pace. *shrug* Thanks for giving this some consideration. From gdamore@sun.com Thu Sep 4 21:49:58 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m854nvXO028964 for ; Thu, 4 Sep 2008 21:49:58 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m854ntW1003836 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 5 Sep 2008 12:49:56 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6P00703HF7RI00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 21:49:55 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6P00DXBHF7KAB0@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 21:49:55 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m854nth0027101 for ; Thu, 04 Sep 2008 21:49:55 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6P00301H9V3Q00@fe-sfbay-09.sun.com> (original mail from gdamore@sun.com) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 21:49:55 -0700 (PDT) Received: from [10.7.251.172] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6P00AYMHF62F50@fe-sfbay-09.sun.com>; Thu, 04 Sep 2008 21:49:54 -0700 (PDT) Date: Thu, 04 Sep 2008 21:49:09 -0700 From: "Garrett D'Amore" Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C084CA.20401@sun.com> Sender: Garrett.Damore@sun.com To: Vikram Hegde Cc: Wesley Shao , Kais.Belgaied@sun.com, Matthew Jacob , Jerry Gilliam , PSARC-ext@sun.com Message-id: <48C0BA45.6070505@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48C03BF4.6080705@sun.com> <48C042EA.20402@sun.com> <48C084CA.20401@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 6600 Vikram Hegde wrote: > Hi all, > > ok, what I am hearing is that we keep these properties Sun internal > and maybe use for debugging and testing. So I will keep this in the > code but will not document it. Is this acceptable to all ? It is with me. - -Garrett > > Vikram > > Garrett D'Amore wrote: >> Furthermore, drivers that don't want to re-spin for this should >> continue to function normally. The only reason to avoid use of the >> IOMMU is either performance optimization or to workaround incorrect >> driver implementation. >> >> As indicated, I don't mind have have some properties "undocumented" >> usage, but we really should be encouraging folks that care to avoid >> the IOMMU to use the documented API. >> >> (And, IMO, there really should be no reason to avoid the IOMMU if >> you've written your code to perform well under regular DDI compliant >> DMA. The IOMMU itself affords some significant benefits, making it >> much easier to get single contiguous DMA segments, reducing the need >> to deal with multiple cookies -- scatter gather -- or DMA windows. >> Never mind the huge benefits it affords for virtualization and the >> fault tolerance benefits that come for free as well.) >> >> -- Garrett >> >> Wesley Shao wrote: >>> FORCE_PHYSICAL allows finer granularity and the proposed properties >>> make the drivers non-portable to SPARC platforms. They should really be >>> used for internal debug. >>> >>> Besides, they make the drivers hard to run on virtualized platforms. >>> >>> Wes >>> >>> Vikram Hedgde wrote: >>>> Hi, >>>> >>>>> Yes. I'd prefer not to "document" the alternative mechanism, >>>>> since I view that as a hack/workaround rather than the correct >>>>> behavior of using the DMA attributes. >>>> How about 3rd party drivers that driver vendors don't want to rev >>>> and re-release solely for this. Shouldn't there be a switch that >>>> they can suggest to customers to use with their existing driver >>>> binaries ? >>>> >>>> Vikram >>>> >>>> >>>> Garrett D'Amore wrote: >>>> Vikram Hegde wrote: >>>>> Garrett D'Amore wrote: >>>>>> It would be cooler if this could be dealt with using ordinary DMA >>>>>> attributes. Won't DDI_DMA_FORCE_PHYSICAL (see the dma_attr_flags >>>>>> for ddi_dma_attr(9s)) work for this? >>>>>> >>>>> Yes, that would work as well and is the preferred and established >>>>> solution. This flag will be respected by the IOMMU. However, it >>>>> seems that an alternative mechanism that involves no touching of >>>>> the driver code is desirable as well. Do you agree ? >>>> >>>> Yes. I'd prefer not to "document" the alternative mechanism, since >>>> I view that as a hack/workaround rather than the correct behavior >>>> of using the DMA attributes. >>>> >>>> -- Garrett >>>>> >>>>> Thanks, >>>>> Vikram >>>>>> - Garrett >>>>>> >>>>>> >>>>>> Vikram Hegde wrote: >>>>>>> Hi Kais, >>>>>>> >>>>>>> Below is the text I am planning to add to the PSARC case. And >>>>>>> yes, the disable properties will be a part of this PSARC case. >>>>>>> >>>>>>> Vikram >>>>>>> >>>>>>> ===================================================================== >>>>>>> >>>>>>> Enabling/Disabling the IOMMU >>>>>>> ============================= >>>>>>> It is recognized that driver or platform developers may not in >>>>>>> certain cases desire the >>>>>>> IOMMU to be enabled. To accomodate such needs we propose the >>>>>>> following global properties >>>>>>> in the iommu driver.conf file >>>>>>> >>>>>>> "global-disable" - can be set to 1 to disable IOMMU platform wide. >>>>>>> "exclude-list" - list of driver names that want the IOMMU >>>>>>> disabled. >>>>>>> ====================================================================== >>>>>>> >>>>>>> >>>>>>> Vikram >>>>>>> >>>>>>> Kais Belgaied wrote: >>>>>>>> Vikram, Jerry, >>>>>>>> >>>>>>>> So, is the ability to disable the IOMMU on a per driver basis >>>>>>>> in this case's scope? >>>>>>>> what interface is being proposed for doing this? >>>>>>>> >>>>>>>> Jerry, assuming the answer the to first question above is >>>>>>>> 'yes', the case cannot be in >>>>>>>> a "closed approved automatic" state while waiting for some of >>>>>>>> its spec. >>>>>>>> >>>>>>>> Kais. >>>>>>>> >>>>>>>> On 09/03/08 23:16, Matthew Jacob wrote: >>>>>>>>> Garrett D'Amore wrote: >>>>>>>>>> Vikram Hegde wrote: >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>>> Abilities to disable the IOMMU as a choice? Performance >>>>>>>>>>>> impact on various systems and interactions with other >>>>>>>>>>>> projects that have gone to some length to work within S/G >>>>>>>>>>>> and its performance characteristics. How will this affect >>>>>>>>>>>> them? Cache effects? Interactions with some VM work >>>>>>>>>>>> currently being undertaken? >>>>>>>>>>>> >>>>>>>>>>>> I can't imagine such a major change being >>>>>>>>>>>> closed/approved/automatic. >>>>>>>>>>> Performance is driver dependant. There is no way a single >>>>>>>>>>> project team can test the gamut of drivers out there and >>>>>>>>>>> certify that the IOMMU will give acceptable performance for >>>>>>>>>>> all needs (which is itself a subjective opinion). We have >>>>>>>>>>> always planned to provide the ability to disable the IOMMU >>>>>>>>>>> on a per driver basis. Does that address your performance >>>>>>>>>>> concerns ? >>>>>>>>>> >>>>>>>>>> Actually, for many cases, IOMMU may *improve* performance. >>>>>>>>>> This is because it will make it easier to satisfy device >>>>>>>>>> constraints for DMA resources without requiring the use of a >>>>>>>>>> bounce buffer or dma windows. >>>>>>>>> >>>>>>>>> That is almost always the exception now. The other issue IIRC >>>>>>>>> is the interaction between the IOMMU and memory caches. In >>>>>>>>> many data movement appliance contexts you use scatter-gather >>>>>>>>> (scatter-gather in name only- contiguous memory is good) to >>>>>>>>> physical pages which are never mapped in. >>>>>>>>> >>>>>>>>> The main point here is to avoid a situation that we're >>>>>>>>> currently in now with respect MSI-X and Interrupt/CPU binding >>>>>>>>> and make sure that there are sufficient control surfaces to >>>>>>>>> allow for a broad range of possible implementation choices. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> That said, drivers which assume that setup and tear down of >>>>>>>>>> DMA is "cheap" may take a perf. hit when using an IOMMU. >>>>>>>>>> This is already the case for SPARC systems, btw. >>>>>>>>> >>>>>>>>> Except for those (ancient) ones that used 36-bit MBus physical >>>>>>>>> addresses and bypassed the IOMMU. >>> >>> >> > > From gdamore@sun.com Thu Sep 4 21:59:54 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m854xsBD029255 for ; Thu, 4 Sep 2008 21:59:54 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m854xsvm013197 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Sep 2008 21:59:54 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6P00803HVT9C00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 21:59:53 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6P00DN5HVSKND0@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 21:59:52 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m854xqb4027493 for ; Thu, 04 Sep 2008 21:59:52 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6P00901HQJ3V00@fe-sfbay-09.sun.com> (original mail from gdamore@sun.com) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu, 04 Sep 2008 21:59:52 -0700 (PDT) Received: from [10.7.251.172] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6P00AWWHVR2F70@fe-sfbay-09.sun.com>; Thu, 04 Sep 2008 21:59:52 -0700 (PDT) Date: Thu, 04 Sep 2008 21:59:06 -0700 From: "Garrett D'Amore" Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C093CB.2020802@sun.com> Sender: Garrett.Damore@sun.com To: Vikram Hegde Cc: Matthew Jacob , Wesley Shao , Kais.Belgaied@sun.com, Jerry Gilliam , PSARC-ext@sun.com Message-id: <48C0BC9A.5070309@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48C03BF4.6080705@sun.com> <48C042EA.20402@sun.com> <48C084CA.20401@sun.com> <48C086EB.2020003@sun.com> <48C08919.8040006@sun.com> <48C08DF7.3020402@sun.com> <48C093CB.2020802@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 6075 As additional data points: 1) Driver developers should *never* have been writing device drivers that relied on S/G DMA. If they did, then they are in violation of the DDI, and their drivers are *buggy*. The fact that an IOMMU is involved, or not, is largely an implementation detail. 2) FORCE_PHYSICAL *is* documented in the man page! (see ddi_dma_attr(9s)) -- it says nothing about MBus addresses etc. Anyone relying on that exclusive semantic is probably using an ancient sparcv8 driver, and probably not even running on modern Solaris. 3) The Mbus semantic is *not* mutually exclusive. 4) Most of our drivers work on both SPARC and x86 -- and don't need to know about IOMMU or not. There are no #ifdef's for most of this code. There are many many examples I could point to -- in fact, I've yet to write any code that cared, and I've yet to see any code for x86 Solaris that would care (unless the code was not using the DDI at all, and that's an entirely different story.) Much more common is breakage resulting from SPARC code assuming that they can get a single contiguous DMA region -- but that code usually gets fixed to deal with either multiple cookies or DMA windows when it gets ported to x86. (So in my experience the portability problems are generally the *reverse* of what I think your concern is.) 5) The "undocumented" property as an escape hatch will still exist -- if we need to expose it in the future to workaround some buggy driver, then so be it. But we should *not* be advising developers to use this. We *should* be advising them to a) ideally not care about IOMMU or not (i.e. make your code portable!), or b) use FORCE_PHYSICAL if you have some specific requirement. -- Garrett Vikram Hegde wrote: > Hi, > > I think specifiying a FORCE_IOMMU flag will turn this into an > experimental feature that will never become mainstream in the > forseeable feature. Driver developers will simply not take interest > until it becomes mainstream and it will never become mainstream > without developer interest. We will also be at a competitive > disadvantage compared to Windows and Linux both of which will have > IOMMU support. I agree with Garrett and Wes that using FORCE_PHYSICAL > is probably the right balance of caution (there is an out for driver > developers) and evolving the DDI - we can't afford a glacial pace. > > Vikram > > Matthew Jacob wrote: >> Vikram Hegde wrote: >>> Hi Matthew, >>>> No, sorry. >>> Any reason why FORCE_PHYSICAL is not acceptable ? It has been around >>> for the SPARC side for some time now and seems to meet the needs of >>> everyone on that side. It provides exactly the same functionality as >>> you want. >>> >> Because it changes the effective semantics of what it was previously. >> Previously it was a performance hack to get 36 bit MBus addresses and >> get around the IOMMU limitations (at least IIRC) and as such it >> encapsulated a semantic that was specific to SPARC. So much so that >> the man page even mentions it. >> >> S/G DMA on the other hand, has been specific more to non-SPARC and >> has been the default there. Turning around and now making it /not/ >> the default on the /hope/ that those who don't want that change will >> get around to recompiling and changing their drivers ignores the >> intent of what the DDI is about. >> >> In short, similarity in names of objects does not mean similarity in >> effective and assumed semantics. >> >> In general I think it's bad policy for Sun to create interfaces that >> consumers of those interfaces cannot disable if there are workable >> alternates. It's even more problematic when the default underlying >> implementation just changes. Furthermore, even if you classified such >> interfaces as evolving, things that have been long standing have at >> least some expectation of remaining the same from (minor) release to >> release. >> >> The /whole /point of the DDI is to create a framework that is >> friendly and usable and reliable for third parties to use. It does >> /not/ exist to have private side interfaces for Sun internal use >> only, although nearly all of the DDI has ended up that way (which was >> /not/ the intent of some of us when we ent through the effort of >> starting this). >> >> You have in some senses two goals here now, though. The first is the >> DDI idea. The second is to improve the underlying framework in a way >> that is believable and measurable and helps support broadest >> framework within Sun itself. As such, the IOMMU addition is probably >> a good idea- certainly the people proposing it believe so or they >> wouldn't be doing it. >> >> However, if you're not making this a major release type of change (in >> which case, IMHO, such as it is, lots of things should be thrown out >> and redone from scratch), then the addition of the IOMMU as the >> underlying transport instead of S/G DMA should be done in a less >> invasive way so that previous binaries don't actually do anything >> different (as a default). If it turns out that they are /provably/ >> correct in environments where the IOMMU is used instead, then the >> P-teams and the third parties could then either recompile or >> otherwise turn on that usage. That is prudent. >> >> So if you're going to make such a big underlying change, I would >> suggest instead that until it's a major release, you could add a >> FORCE_IOMMU flag. This would have the effect of taking driver >> binaries to work exactly as they have done before and allow drivers >> written by both Sun and third parties to /try out/ the IOMMU >> environment as needed or desired. >> >> The above isn't probably reasoned out as well as it should be, but it >> does seem to me that you should either dump the DDI and it's >> overcautious "let's not change things (for decades)" approach (and >> take an approach more like (the good parts) of Linux), or if you're >> going to treat the DDI in its multiple roles, err very much on the >> side of caution. >> >> -matt >> >> > > From Matthew.Jacob@sun.com Sun Sep 7 10:30:28 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m87HURuF018516 for ; Sun, 7 Sep 2008 10:30:28 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m87HUNU3010426 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Sun, 7 Sep 2008 18:30:26 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6U00L015YP1X00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 07 Sep 2008 10:30:25 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6U00AZ35YOSQA0@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 07 Sep 2008 10:30:24 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m87HUOoY008971 for ; Sun, 07 Sep 2008 10:30:24 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6U00F015NQJ300@fe-sfbay-10.sun.com> (original mail from Matthew.Jacob@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 07 Sep 2008 10:30:24 -0700 (PDT) Received: from ns1.feral.com ([192.67.166.1]) by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6U00ASM5YO76A0@fe-sfbay-10.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 07 Sep 2008 10:30:24 -0700 (PDT) Date: Sun, 07 Sep 2008 10:30:24 -0700 From: Matthew Jacob Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C042EA.20402@sun.com> Sender: Matthew.Jacob@sun.com To: "Garrett D'Amore" Cc: PSARC-ext@sun.com Message-id: <48C40FB0.5080605@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48C03BF4.6080705@sun.com> <48C042EA.20402@sun.com> User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.1.16) Gecko/20080831 SeaMonkey/1.1.11 Status: RO Content-Length: 1421 Follow up to this as it actually turns out to be related to what I'm working on. It turns out to be a not-serious problem, but it required a small bit of thinking and planning ahead. I'm working on a driver for a board that needs to have DMA S/G lists in coherent memory (as do many boards). These S/G lists are filled in from the dma cookies you get back for mapping a command. If the underlying mapping mechanism is an IOMMU, then the number of S/G list elements to allocate equals (roughly) the maximum command load for each board. If the underlying mapping mechanism is DMA S/G, then the number of S/G list elements to allocate equals (roughly) the maximum command load for each board times the (roughly) the expected average page length for each command. Yes, a DMA S/G mechanism /may/ have coalesced pages, but more often than not doesn't. It turns out to be prudent to build an allocation of card S/G list memory a page at a time anyway as load increases- this way you avoid problems trying to allocate a bunch of contiguous consistent memory at load time. (only to have it fail as this memory, at least on x86, tends to fragment over time). I'm fortunate in that I can chain these chunks together for this card. Other cards aren't so lucky. It might helpful to know the properties of the underlying DMA mapper so different strategies could be implemented at load time. Just a thought.... From gdamore@Sun.COM Sun Sep 7 19:49:41 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m882nf6k027656 for ; Sun, 7 Sep 2008 19:49:41 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m882nf7x005292 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Sun, 7 Sep 2008 19:49:41 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6U00H03VUTFQ00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 07 Sep 2008 19:49:41 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6U00GE4VUS5E00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 07 Sep 2008 19:49:41 -0700 (PDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m882neak019063 for ; Sun, 07 Sep 2008 19:49:40 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6U00501VKXJJ00@fe-sfbay-09.sun.com> (original mail from gdamore@sun.com) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 07 Sep 2008 19:49:40 -0700 (PDT) Received: from [10.7.251.172] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6U00H4CVUS5Y40@fe-sfbay-09.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 07 Sep 2008 19:49:40 -0700 (PDT) Date: Sun, 07 Sep 2008 19:48:42 -0700 From: "Garrett D'Amore" Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C40FB0.5080605@sun.com> Sender: Garrett.Damore@Sun.COM To: Matthew Jacob Cc: PSARC-ext@Sun.COM Message-id: <48C4928A.8050404@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48C03BF4.6080705@sun.com> <48C042EA.20402@sun.com> <48C40FB0.5080605@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 2334 Matthew Jacob wrote: > Follow up to this as it actually turns out to be related to what I'm > working on. It turns out to be a not-serious problem, but it required > a small bit of thinking and planning ahead. > > I'm working on a driver for a board that needs to have DMA S/G lists > in coherent memory (as do many boards). These S/G lists are filled in > from the dma cookies you get back for mapping a command. > > If the underlying mapping mechanism is an IOMMU, then the number of > S/G list elements to allocate equals (roughly) the maximum command > load for each board. > > If the underlying mapping mechanism is DMA S/G, then the number of S/G > list elements to allocate equals (roughly) the maximum command load > for each board times the (roughly) the expected average page length > for each command. Yes, a DMA S/G mechanism /may/ have coalesced pages, > but more often than not doesn't. > > It turns out to be prudent to build an allocation of card S/G list > memory a page at a time anyway as load increases- this way you avoid > problems trying to allocate a bunch of contiguous consistent memory at > load time. (only to have it fail as this memory, at least on x86, > tends to fragment over time). > > I'm fortunate in that I can chain these chunks together for this card. > Other cards aren't so lucky. > > It might helpful to know the properties of the underlying DMA mapper > so different strategies could be implemented at load time. Knowing that the IOMMU is in play might allow you to assume that only one cookie will be needed. But it would be poor form to rely on that behavior. (This is one area where SPARC drivers sometimes have troubles being ported to x86.) The driver can "force" this using restrictive DMA attributes, but experience is that such attributes simply fail to succeed to allocate resources on x86. But in any case, the failure mode here is that going from IOMMU to physical only mapping often gives problems, but that introduction of an IOMMU layer should itself not cause any problems. (Except to a driver that poorly assumed that the system always used a single cookie per page. Which would very often likely *not* be the case, so I think even those kinds of drivers would fail occasionally without an IOMMU.) -- Garrett > > Just a thought.... > From Matthew.Jacob@sun.com Sun Sep 7 20:12:02 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m883C1v1028130 for ; Sun, 7 Sep 2008 20:12:02 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m883BvHU005715 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Mon, 8 Sep 2008 04:12:00 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6U00505WW0Y900@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 07 Sep 2008 20:12:00 -0700 (PDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6U00I78WW0CQ60@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 07 Sep 2008 20:12:00 -0700 (PDT) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m883C0XH019618 for ; Sun, 07 Sep 2008 20:12:00 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6U00801WLL1T00@fe-sfbay-10.sun.com> (original mail from Matthew.Jacob@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 07 Sep 2008 20:12:00 -0700 (PDT) Received: from [129.145.154.73] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6U005P0WVZT420@fe-sfbay-10.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 07 Sep 2008 20:12:00 -0700 (PDT) Date: Sun, 07 Sep 2008 20:11:59 -0700 From: Matthew Jacob Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C4928A.8050404@sun.com> Sender: Matthew.Jacob@sun.com To: "Garrett D'Amore" Cc: PSARC-ext@sun.com Reply-to: Matthew.Jacob@sun.com Message-id: <48C497FF.5050709@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48C03BF4.6080705@sun.com> <48C042EA.20402@sun.com> <48C40FB0.5080605@sun.com> <48C4928A.8050404@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080505) Status: RO Content-Length: 496 On 09/07/08 19:48, Garrett D'Amore wrote: > > Knowing that the IOMMU is in play might allow you to assume that only > one cookie will be needed. But it would be poor form to rely on that > behavior. You've missed my point. The point is that knowing the attributes of the underlying mapping algorithm (via properties, e.g., at load time, per instance), the driver can make allocation policy choices that make sense for that current load time, instance, etc. There is no 'relying' here. From gdamore@sun.com Sun Sep 7 22:45:02 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m885j1qe001652 for ; Sun, 7 Sep 2008 22:45:02 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m885isbN027575 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Mon, 8 Sep 2008 13:45:00 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K6V0010J3YWRM00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 07 Sep 2008 23:44:56 -0600 (MDT) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K6V00JO53YVYX20@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 07 Sep 2008 23:44:55 -0600 (MDT) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m885itZR023155 for ; Sun, 07 Sep 2008 22:44:55 -0700 (PDT) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0K6V00E013S14500@fe-sfbay-09.sun.com> (original mail from gdamore@sun.com) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 07 Sep 2008 22:44:55 -0700 (PDT) Received: from [10.7.251.172] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0K6V00K623YUVKB0@fe-sfbay-09.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 07 Sep 2008 22:44:55 -0700 (PDT) Date: Sun, 07 Sep 2008 22:43:54 -0700 From: "Garrett D'Amore" Subject: Re: AMD IOMMU [PSARC/2008/561] In-reply-to: <48C497FF.5050709@Sun.COM> Sender: Garrett.Damore@sun.com To: Matthew.Jacob@sun.com Cc: PSARC-ext@sun.com Message-id: <48C4BB9A.4010406@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <48C03BF4.6080705@sun.com> <48C042EA.20402@sun.com> <48C40FB0.5080605@sun.com> <48C4928A.8050404@sun.com> <48C497FF.5050709@Sun.COM> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 1266 Matthew Jacob wrote: > On 09/07/08 19:48, Garrett D'Amore wrote: >> >> Knowing that the IOMMU is in play might allow you to assume that only >> one cookie will be needed. But it would be poor form to rely on that >> behavior. > You've missed my point. > > The point is that knowing the attributes of the underlying mapping > algorithm (via properties, e.g., at load time, per instance), the > driver can make allocation policy choices that make sense for that > current load time, instance, etc. > > There is no 'relying' here. > But, those choices (the decision to allocate just a single descriptor entry for example) or not really need to be done carefully. If the decision is is made at attach(9e) time, then the resources allocated should match the number of cookies that are claimed to be supported by the ddi_dma_attr's sgllen field. Now, if you know that an IOMMU is in play, you could restrict that entry to 1, and only allocate a single descriptor. But you'd have to be 100% certain. When an IOMMU is *not* in play, you should avoid making too many assumptions. (You could probably assume that the # cookies or descriptors to be allocated will not exceed the (total alloc size / page size + 1) calculation, I suppose.) -- Garrett From jg@jurassic.sfbay.Sun.COM Thu Oct 30 15:11:04 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m9UMB3Hp023544 for ; Thu, 30 Oct 2008 15:11:04 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m9UMAsOs002175; Thu, 30 Oct 2008 22:11:01 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K9K00I1JOACP900@nwk-avmta-1.sfbay.Sun.COM>; Thu, 30 Oct 2008 15:11:00 -0700 (PDT) Received: from jurassic.eng.sun.com ([129.146.56.36]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K9K008CQOABVKF0@nwk-avmta-1.sfbay.Sun.COM>; Thu, 30 Oct 2008 15:10:59 -0700 (PDT) Received: from tethys (tethys.SFBay.Sun.COM [129.146.226.92]) by jurassic.eng.sun.com (8.13.8+Sun/8.13.8) with SMTP id m9UMAxiY451375; Thu, 30 Oct 2008 15:10:59 -0700 (PDT) Date: Thu, 30 Oct 2008 15:06:22 -0700 (PDT) From: Jerry Gilliam Subject: Re: AMD IOMMU [PSARC/2008/561] To: PSARC-ext@sun.com, vikram.hegde@sun.com Reply-to: Jerry Gilliam Message-id: <200810302210.m9UMAxiY451375@jurassic.eng.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_94 SunOS 5.11 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-MD5: EsLsSfplPNPXXBgYB8Z+4g== X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 292 Although the release binding for this case was not specified when submitted, the project team desires micro/patch binding so this feature can be included and enabled in an update release. As this case was self-approved, I'd like to amend it in the same manner on behalf of the team. -jg