From sacadmin Tue Mar 21 17:00:50 2006 Received: from sunmail2.sfbay.sun.com (sunmail2.SFBay.Sun.COM [129.149.246.180]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2M10oIQ022842 for ; Tue, 21 Mar 2006 17:00:50 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2.sfbay.sun.com (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k2M10ov01214 for <@sunmail1brm.central.sun.com:psarc@sun.com>; Tue, 21 Mar 2006 17:00:50 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IWI00F0N85CEH00@brm-avmta-1.central.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Tue, 21 Mar 2006 18:00:48 -0700 (MST) Received: from sfbaymail2sca.sfbay.sun.com ([129.145.155.42]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IWI00CIC858JT90@brm-avmta-1.central.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Tue, 21 Mar 2006 18:00:45 -0700 (MST) Received: from billybob.sfbay.sun.com (billybob.SFBay.Sun.COM [129.146.224.123]) by sfbaymail2sca.sfbay.sun.com (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k2M10hLn028409; Tue, 21 Mar 2006 17:00:43 -0800 (PST) Received: from billybob (billybob [129.146.224.123]) by billybob.sfbay.sun.com (8.13.5+Sun/8.13.5) with SMTP id k2M12iAH107448; Tue, 21 Mar 2006 17:02:44 -0800 (PST) Date: Tue, 21 Mar 2006 17:02:44 -0800 (PST) From: Shudong Zhou Subject: PSARC 2006/188 Memory Cache Disable Support on x86 To: psarc@sun.com Cc: eiji.ota@sun.com, mark.johnson@sun.com Reply-to: Shudong Zhou Message-id: <200603220102.k2M12iAH107448@billybob.sfbay.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_13 SunOS 5.11 i86pc i386 Content-type: TEXT/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-MD5: 3aj5mhpzaB2rK4cu/WCsTA== X-PMX-Version: 5.1.2.240295 Status: RO Content-Length: 16138 I'm sponsoring the following fasttrack for Eiji Ota. The timer is set to expire on 3/28/2006. The requested release binding is patch. Shudong Template Version: @(#)onepager.txt 1.29 04/11/15 SMI This information is Copyright 2006 Sun Microsystems, Inc. 1. Introduction 1.1. Project/Component Working Name: Allocating memory from the DDI with Cache Disable Support on x86 1.2. Name of Document Author/Supplier: Eiji Ota (eiji.ota@sun.com) Mark Johnson (mark.johnson@sun.com) 1.3. Date of This Document: 03/03/06 1.4. Name of Major Document Customer(s)/Consumer(s): 1.4.1. The PAC or CPT you expect to review your project: Solaris PAC 1.4.2. The ARC(s) you expect to review your project: PSARC 1.4.3. The Director/VP who is "Sponsoring" this project: Barry Cooks (barry.cooks@sun.com) 1.4.4. The name of your business unit: OPG, Solaris Core Technologies 1.5. Email Aliases: 1.5.1. Responsible Manager: peter.luk@sun.com 1.5.2. Responsible Engineer: eiji.ota@sun.com, mark.johnson@sun.com 1.5.3. Marketing Manager: chris.ratcliffe@sun.com 1.5.4. Interest List: iom-interest@Sun.Com 2. Project Summary 2.1. Project Description: The project provides a method to access uncacheable memory, which is required for some graphics drivers running on X86/AMD64 architecture. 2.2. Risks and Assumptions: None that we know of. 3. Business Summary 3.1. Problem Area: Some graphics drivers on X86/AMD64 require a method to have an access to memory with the uncacheable attribute to keep coherency between a graphic chip and the rest on the system, but Solaris doesn't provide any interfaces those drivers can use. As a result, they have to implement how to access the uncacheable memory in their own ways. This could be seen inconvenient by Solaris driver developers and inconsistent by Solaris kernel developers. 3.2. Market/Requester: Graphic device drivers running on X86 architecture. 3.3. Business Justification: Uncacheable memory access is required for some graphics chips on X86/AMD64 architecture these days. However, Solaris doesn't provide the DDI interface for that purpose yet. This project provides the unified and consistent method to support those graphics drivers running on X86/AMD64. 3.4. Competitive Analysis: Major operating systems running on X86/AMD64 (i.e. Linux/Windows) support similar functions already. 4. Technical Description: 4.1. Details: A new flag, DDI_CACHEDISABLE_ACC, is introduced to provide the uncacheable memory access. Though we have DDI_STRICTORDER_ACC to describe the strict ordering, if this were used for the uncacheable memory access, it could cause performance degradation since DDI_STRICTORDER_ACC is used by most drivers today. The flag should be set to devacc_attr_dataorder in struct ddi_device_acc_attr when a driver wants to access memory with the uncacheable attribute. This flag is referred to by devmap_umem_setup(), devmap_umem_remap() and ddi_dma_mem_alloc(). devmap_umem_setup() and devmap_umem_remap() are used to export kernel memory to the userland. The new flag is used to allow an application to have an access to the uncacheable memory by mmap(). ddi_dma_mem_alloc() is used to allocate non-pageable kernel memory for a driver. The new flag is used to provide a driver with an access to the uncacheable memory in the same manner. The gfx_private module is also updated to implement the uncacheable memory access by this new flag. 4.2. Bug/RFE Number(s): 6212057 ddi_dma_mem_alloc should support DDI_STRICTORDER_ACC in x86 4.3. Interfaces: +-------------------------+---------------------+------------------+ | Interface Name | Classification | Comments | +-------------------------+---------------------+------------------+ | DDI_CACHEDISABLE_ACC | Evolving | -------- | +-------------------------+---------------------+------------------+ The following example shows how to use this flag for the uncacheable memory access; a driver doesn't need any other things than just setting DDI_CACHEDISABLE_ACC to devacc_attr_dataorder as follows. struct ddi_device_acc_attr XXX_acc_attr = { DDI_DEVICE_ATTR_V0, DDI_NEVERSWAP_ACC, DDI_CACHEDISABLE_ACC /* use the uncacheable memory */ }; 4.6. Doc Impact: Man page (ddi_device_acc_attr(9S)). The difference is attached below. (the change has bars on the left) 4.7. Admin/Config Impact: No impact on Admin/Config. 4.8. HA Impact: No impact on HA. 4.9. I18N/L10N Impact: No impact on I18N/L10L 4.10. Packaging & Delivery: No impact on packages, clusters and metaclusters. No impact on install/upgrade. 4.11. Security Impact: No impact on Security. 4.12. Dependencies: None. 5. Reference Documents: PSARC 2004/151 AGPgart project PSARC 2005/475 Graphics private misc module for x86 PSARC 2005/623 gfx_private update for devmap and DMA 6. Resources and Schedule: 6.1. Projected Availability: April/2006 6.4. Product Approval Committee requested information: 6.4.1. Consolidation or Component Name: ON 6.4.3. Type of CPT Review and Approval expected: FastTrack 6.4.7. Target RTI Date/Release: onnv_b38 6.4.8. Target Code Design Review Date: 3/13 6.5. ARC review type: FastTrack (ddi_device_acc_attr(9S)) Data Structures for Drivers ddi_device_acc_attr(9S) NAME ddi_device_acc_attr - data access attributes structure SYNOPSIS #include #include INTERFACE LEVEL Solaris DDI specific (Solaris DDI). DESCRIPTION The ddi_device_acc_attr structure describes the data access characteristics and requirements of the device. STRUCTURE MEMBERS ushort_t devacc_attr_version; uchar_t devacc_attr_endian_flags; uchar_t devacc_attr_dataorder; The devacc_attr_version member identifies the version number of this structure. The current version number is DDI_DEVICE_ATTR_V0. The devacc_attr_endian_flags member describes the endian characteristics of the device. Specify one of the following values: DDI_NEVERSWAP_ACC Data access with no byte swapping DDI_STRUCTURE_BE_ACC Structural data access in big-endian format DDI_STRUCTURE_LE_ACC Structural data access in little endian format DDI_STRUCTURE_BE_ACC and DDI_STRUCTURE_LE_ACC describes the endian characteristics of the device as big-endian or little-endian, respectively. Though most of the devices will have the same endian characteristics as their buses, exam- ples of devices that have opposite endian characteristics of the buses do exist. When DDI_STRUCTURE_BE_ACC or DDI_STRUCTURE_LE_ACC is set, byte swapping is automatically performed by the system if the host machine and the device data formats have opposite endian characteristics. The implementation can take advantage of hardware platform byte swapping capabilities. When you specify DDI_NEVERSWAP_ACC, byte swapping is not invoked in the data access functions. The devacc_attr_dataorder member describes order in which the CPU will reference data. Specify one of the following values. DDI_STRICTORDER_ACC The data references must be issued by a CPU in program order. Strict ordering is the default behavior. DDI_UNORDERED_OK_ACC The CPU can re-order the data references. This includes all kinds of re- ordering. For example, a load followed by a store may be replaced by a store fol- lowed by a load. DDI_MERGING_OK_ACC The CPU can merge indivi- dual stores to consecutive locations. For example, the CPU can turn two consecutive byte stores into one half- word store. It can also batch individual loads. For example, the CPU might turn two consecutive byte loads into one halfword load. DDI_MERGING_OK_ACC also implies re-ordering. DDI_LOADCACHING_OK_ACC The CPU can cache the data it fetches and reuse it until another store occurs. The default behavior is to fetch new data on every load. DDI_LOADCACHING_OK_ACC also implies merging and re-ordering. DDI_STORECACHING_OK_ACC The CPU can keep the data in the cache and push it to the device (perhaps with other data) at a later time. The default behavior is to push the data right away. DDI_STORECACHING_OK_ACC also implies load caching, merg- ing, and re-ordering. | DDI_CACHEDISABLE_ACC The CPU can keep the data | from the cache and write the | data to memory without it. | The default behavior is to | push the data right away. | DDI_CACHEDISABLE_ACC also | implies strict ordering. These values are advisory, not mandatory. For example, data can be ordered without being merged or cached, even though a driver requests unordered, merged, and cached together. EXAMPLES The following examples illustrate the use of device register address mapping setup functions and different data access functions. Example 1: Using ddi_device_acc_attr() in ddi_regs_map_setup(9F) This example demonstrates the use of the ddi_device_acc_attr() structure in ddi_regs_map_setup(9F). It also shows the use of ddi_getw(9F) and ddi_putw(9F) functions in accessing the register contents. dev_info_t *dip; uint_t rnumber; ushort_t *dev_addr; offset_t offset; offset_t len; ushort_t dev_command; ddi_device_acc_attr_t dev_attr; ddi_acc_handle_t handle; ... /* * setup the device attribute structure for little endian, * strict ordering and 16-bit word access. */ dev_attr.devacc_attr_version = DDI_DEVICE_ATTR_V0; dev_attr.devacc_attr_endian_flags = DDI_STRUCTURE_LE_ACC; dev_attr.devacc_attr_dataorder = DDI_STRICTORDER_ACC; /* * set up the device registers address mapping */ ddi_regs_map_setup(dip, rnumber, (caddr_t *)&dev_addr, offset, len, &dev_attr, &handle); /* read a 16-bit word command register from the device */ dev_command = ddi_getw(handle, dev_addr); dev_command |= DEV_INTR_ENABLE; /* store a new value back to the device command register */ ddi_putw(handle, dev_addr, dev_command); Example 2: Accessing a Device with Different Apertures The following example illustrates the steps used to access a device with different apertures. Several apertures are assumed to be grouped under one single "reg" entry. For example, the sample device has four different apertures, each 32 Kbyte in size. The apertures represent YUV little- endian, YUV big-endian, RGB little-endian, and RGB big- endian. This sample device uses entry 1 of the "reg" pro- perty list for this purpose. The size of the address space is 128 Kbyte with each 32 Kbyte range as a separate aper- ture. In the register mapping setup function, the sample driver uses the offset and len parameters to specify one of the apertures. ulong_t *dev_addr; ddi_device_acc_attr_t dev_attr; ddi_acc_handle_t handle; uchar_t buf[256]; ... /* * setup the device attribute structure for never swap, * unordered and 32-bit word access. */ dev_attr.devacc_attr_version = DDI_DEVICE_ATTR_V0; dev_attr.devacc_attr_endian_flags = DDI_NEVERSWAP_ACC; dev_attr.devacc_attr_dataorder = DDI_UNORDERED_OK_ACC; /* * map in the RGB big-endian aperture * while running in a big endian machine * - offset 96K and len 32K */ ddi_regs_map_setup(dip, 1, (caddr_t *)&dev_addr, 96*1024, 32*1024, &dev_attr, &handle); /* * Write to the screen buffer * first 1K bytes words, each size 4 bytes */ ddi_rep_putl(handle, buf, dev_addr, 256, DDI_DEV_AUTOINCR); Example 3: Functions That Call Out the Data Word Size The following example illustrates the use of the functions that explicitly call out the data word size to override the data size in the device attribute structure. struct device_blk { ushort_t d_command; /* command register */ ushort_t d_status; /* status register */ ulong d_data; /* data register */ } *dev_blkp; dev_info_t *dip; caddr_t dev_addr; ddi_device_acc_attr_t dev_attr; ddi_acc_handle_t handle; uchar_t buf[256]; ... /* * setup the device attribute structure for never swap, * strict ordering and 32-bit word access. */ dev_attr.devacc_attr_version = DDI_DEVICE_ATTR_V0; dev_attr.devacc_attr_endian_flags = DDI_NEVERSWAP_ACC; dev_attr.devacc_attr_dataorder= DDI_STRICTORDER_ACC; ddi_regs_map_setup(dip, 1, (caddr_t *)&dev_blkp, 0, 0, &dev_attr, &handle); /* write command to the 16-bit command register */ ddi_putw(handle, &dev_blkp->d_command, START_XFER); /* Read the 16-bit status register */ status = ddi_getw(handle, &dev_blkp->d_status); if (status & DATA_READY) /* Read 1K bytes off the 32-bit data register */ ddi_rep_getl(handle, buf, &dev_blkp->d_data, 256, DDI_DEV_NO_AUTOINCR); SEE ALSO ddi_getw(9F), ddi_putw(9F), ddi_regs_map_setup(9F) Writing Device Drivers From sacadmin Tue Mar 21 17:17:47 2006 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2M1HjIQ019189 for ; Tue, 21 Mar 2006 17:17:46 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id k2M1HcdU026480 for <@sunmail1brm.central.sun.com:psarc@sun.com>; Wed, 22 Mar 2006 09:17:44 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IWI00G098XH1700@brm-avmta-1.central.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Tue, 21 Mar 2006 18:17:41 -0700 (MST) Received: from sfbaymail1sca.SFBay.Sun.COM ([129.145.154.35]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IWI00CVO8XHJQB0@brm-avmta-1.central.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Tue, 21 Mar 2006 18:17:41 -0700 (MST) Received: from ivrel.sfbay.sun.com (ivrel.SFBay.Sun.COM [129.146.74.76]) by sfbaymail1sca.SFBay.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k2M1HeEZ004878; Tue, 21 Mar 2006 17:17:40 -0800 (PST) Received: from ivrel (ivrel [129.146.74.76]) by ivrel.sfbay.sun.com (8.13.5+Sun/8.13.5) with SMTP id k2M1Hel3011848; Tue, 21 Mar 2006 17:17:40 -0800 (PST) Date: Tue, 21 Mar 2006 17:17:40 -0800 (PST) From: Glenn Skinner Subject: Re: 2006/188 [Memory Cache Disable Support on x86] To: psarc@Sun.COM Cc: eiji.ota@Sun.COM, mark.johnson@Sun.COM Reply-to: Glenn Skinner Message-id: <200603220117.k2M1Hel3011848@ivrel.sfbay.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.6_36 SunOS 5.11 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-MD5: p8DBnOKascAxunRxhhhY8A== X-PMX-Version: 5.1.2.240295 Status: RO Content-Length: 1241 Date: Tue, 21 Mar 2006 17:02:44 -0800 (PST) From: Shudong Zhou Subject: PSARC 2006/188 Memory Cache Disable Support on x86 Skipping down to the man page part of the writeup, we have: The devacc_attr_dataorder member describes order in which the CPU will reference data. Specify one of the following values. ... | DDI_CACHEDISABLE_ACC The CPU can keep the data | from the cache and write the | data to memory without it. | The default behavior is to | push the data right away. | DDI_CACHEDISABLE_ACC also | implies strict ordering. These values are advisory, not mandatory. For example, data can be ordered without being merged or cached, even though a driver requests unordered, merged, and cached together. ... Does the final paragraph quoted above hold for DDI_CACHEDISABLE_ACC? If so, is there any way for the driver to tell when it's not getting what it asked for? -- Glenn From sacadmin Tue Mar 21 17:33:38 2006 Received: from sunmail3.sfbay.sun.com (sunmail3.SFBay.Sun.COM [129.149.247.180]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2M1XcIQ019534 for ; Tue, 21 Mar 2006 17:33:38 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3.sfbay.sun.com (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k2M1Xb707561; Tue, 21 Mar 2006 17:33:37 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IWI00G079O0M400@brm-avmta-1.central.sun.com>; Tue, 21 Mar 2006 18:33:36 -0700 (MST) Received: from eastmail1bur.East.Sun.COM ([129.148.9.49]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IWI00CIV9O0JTC0@brm-avmta-1.central.sun.com>; Tue, 21 Mar 2006 18:33:36 -0700 (MST) Received: from thunk.east.sun.com (thunk.East.Sun.COM [129.148.174.66]) by eastmail1bur.East.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k2M1XXbB008011; Tue, 21 Mar 2006 20:33:33 -0500 (EST) Received: from 127.0.0.1 (localhost [127.0.0.1]) by thunk.east.sun.com (8.13.5+Sun/8.13.5) with ESMTP id k2M1XX3B018395; Tue, 21 Mar 2006 20:33:33 -0500 (EST) Date: Tue, 21 Mar 2006 20:33:33 -0500 From: Bill Sommerfeld Subject: Re: PSARC 2006/188 Memory Cache Disable Support on x86 In-reply-to: <200603220102.k2M12iAH107448@billybob.sfbay.sun.com> To: Shudong Zhou Cc: psarc@sun.com, Eiji.Ota@sun.com, Mark.Johnson@sun.com Message-id: <1142991212.17801.50.camel@thunk> MIME-version: 1.0 X-Mailer: Ximian Evolution 1.4.6.333 Content-type: text/plain; charset=iso-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.1.2.240295 References: <200603220102.k2M12iAH107448@billybob.sfbay.sun.com> Status: RO Content-Length: 834 On Tue, 2006-03-21 at 20:02, Shudong Zhou wrote: > A new flag, DDI_CACHEDISABLE_ACC, is introduced to provide > the uncacheable memory access. Though we have DDI_STRICTORDER_ACC to > describe the strict ordering, if this were used for the uncacheable > memory access, it could cause performance degradation since > DDI_STRICTORDER_ACC is used by most drivers today. I'm confused. The man page also lists a "DDI_LOADCACHING_OK_ACC"; the description of this option strongly implies that DDI_STRICTORDER_ACC (as well as UNORDERED and MERGING_OK) doesn't allow for caching. What caching is currently done by STRICTORDER? How can we fix the man page to match the actual system behavior? This looks like a case where separate feature tests would be better than an enumeration. - Bill From sacadmin Tue Mar 21 17:38:09 2006 Received: from sunmail1brm.Central.Sun.COM (sunmail1brm.Central.Sun.COM [129.147.62.17]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2M1c8IQ019706 for ; Tue, 21 Mar 2006 17:38:08 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.149.246.28]) by sunmail1brm.Central.Sun.COM (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k2M1c6C22982; Tue, 21 Mar 2006 18:38:06 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2 (built Dec 2 2004)) id <0IWI008039VHOF00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 21 Mar 2006 17:38:05 -0800 (PST) Received: from jurassic.eng.sun.com ([129.146.224.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2 (built Dec 2 2004)) with ESMTP id <0IWI004559VGK8E0@nwk-avmta-1.sfbay.Sun.COM>; Tue, 21 Mar 2006 17:38:05 -0800 (PST) Received: from localhost (ota.SFBay.Sun.COM [129.146.226.183]) by jurassic.eng.sun.com (8.13.6.Gamma0+Sun/8.13.5) with ESMTP id k2M1c4LX106106; Tue, 21 Mar 2006 17:38:04 -0800 (PST) Date: Tue, 21 Mar 2006 17:38:03 -0800 (PST) From: Eiji.Ota@sun.com Subject: Re: 2006/188 [Memory Cache Disable Support on x86] In-reply-to: <200603220117.k2M1Hel3011848@ivrel.sfbay.sun.com> Sender: Eiji Ota To: glenn@ivrel.sfbay.sun.com Cc: psarc@sun.com, Mark.Johnson@sun.com Message-id: <20060321173803N.eota@sun.com> MIME-version: 1.0 X-Mailer: Mew version 1.94.2 on XEmacs 21.1 (Cuyahoga Valley) Content-type: Text/Plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.1.2.240295 Lines: 38 References: <200603220117.k2M1Hel3011848@ivrel.sfbay.sun.com> X-Dispatcher: imput version 20000228(IM140) Status: RO Content-Length: 1645 > Skipping down to the man page part of the writeup, we have: > > The devacc_attr_dataorder member describes order in which > the CPU will reference data. Specify one of the following > values. > > ... > > | DDI_CACHEDISABLE_ACC The CPU can keep the data > | from the cache and write the > | data to memory without it. > | The default behavior is to > | push the data right away. > | DDI_CACHEDISABLE_ACC also > | implies strict ordering. > > These values are advisory, not mandatory. For example, data > can be ordered without being merged or cached, even though a > driver requests unordered, merged, and cached together. > ... > > Does the final paragraph quoted above hold for DDI_CACHEDISABLE_ACC? > If so, is there any way for the driver to tell when it's not getting > what it asked for? The value should be advisory, not mandatory because whether or not it can be achieved depends on the HW implementation somehow -- for example, disabling cache on SPARC doesn't make sense since the coherency is always maintained on the platform, then DDI_CACHEDISABLE_ACC must be visually voided on it. Unfortunately there is no explicit way (I mean, via ddi-interface) to show how your request is treated in the end except you check the TTE/PTE. I'm afraid providing such a method is a bit out of the scope this time. Cheers, -Eiji From sacadmin Tue Mar 21 17:55:12 2006 Received: from sunmail2.sfbay.sun.com (sunmail2.SFBay.Sun.COM [129.149.246.180]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2M1tCIQ020486 for ; Tue, 21 Mar 2006 17:55:12 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.149.246.28]) by sunmail2.sfbay.sun.com (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k2M1tBv16343; Tue, 21 Mar 2006 17:55:11 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2 (built Dec 2 2004)) id <0IWI00A07ANXIN00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 21 Mar 2006 17:55:09 -0800 (PST) Received: from jurassic.eng.sun.com ([129.146.226.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2 (built Dec 2 2004)) with ESMTP id <0IWI009EXANW6630@nwk-avmta-1.sfbay.Sun.COM>; Tue, 21 Mar 2006 17:55:08 -0800 (PST) Received: from localhost (ota.SFBay.Sun.COM [129.146.226.183]) by jurassic.eng.sun.com (8.13.6.Gamma0+Sun/8.13.5) with ESMTP id k2M1t8bI121506; Tue, 21 Mar 2006 17:55:08 -0800 (PST) Date: Tue, 21 Mar 2006 17:55:07 -0800 (PST) From: eiji.ota@sun.com Subject: Re: PSARC 2006/188 Memory Cache Disable Support on x86 In-reply-to: <1142991212.17801.50.camel@thunk> Sender: Eiji Ota To: sommerfeld@sun.com Cc: szhou@billybob.sfbay.sun.com, psarc@sun.com, Mark.Johnson@sun.com Message-id: <20060321175507A.eota@sun.com> MIME-version: 1.0 X-Mailer: Mew version 1.94.2 on XEmacs 21.1 (Cuyahoga Valley) Content-type: Text/Plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.1.2.240295 Lines: 28 References: <200603220102.k2M12iAH107448@billybob.sfbay.sun.com> <1142991212.17801.50.camel@thunk> X-Dispatcher: imput version 20000228(IM140) Status: RO Content-Length: 1352 > > A new flag, DDI_CACHEDISABLE_ACC, is introduced to provide > > the uncacheable memory access. Though we have DDI_STRICTORDER_ACC to > > describe the strict ordering, if this were used for the uncacheable > > memory access, it could cause performance degradation since > > DDI_STRICTORDER_ACC is used by most drivers today. > > I'm confused. > > The man page also lists a "DDI_LOADCACHING_OK_ACC"; the description of > this option strongly implies that DDI_STRICTORDER_ACC (as well as > UNORDERED and MERGING_OK) doesn't allow for caching. > > What caching is currently done by STRICTORDER? How can we fix the man > page to match the actual system behavior? The value is actually advisory, and how it should be treated is somehow dependent on HW; for memory, X86 usually assumes HAT_STORECACHING_OK should be used to gain performance (coming from caches). In this case, STRICTORDER changes to STORECACHING. However, for the IO space, it should change to non-cache. Then the problem is coming up these days that some devices (e.g. graphics) want to use non-cache memory for their framebuffer, but there is no way for that purpose. The new value, DDI_CACHEDISABLE_ACC, is prepared to meet that request. > This looks like a case where separate feature tests would be better than > an enumeration. -Eiji From sacadmin Tue Mar 21 22:13:39 2006 Received: from sunmail3.sfbay.sun.com (sunmail3.SFBay.Sun.COM [129.149.247.180]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2M6DdIQ025860 for ; Tue, 21 Mar 2006 22:13:39 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.149.247.22]) by sunmail3.sfbay.sun.com (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k2M6DY727906; Tue, 21 Mar 2006 22:13:34 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IWI00D0BMMHIW00@nwk-avmta-2.sfbay.sun.com>; Tue, 21 Mar 2006 22:13:29 -0800 (PST) Received: from sineb-mail-2.sun.com ([192.18.19.7]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IWI00CXUMMF1H70@nwk-avmta-2.sfbay.sun.com>; Tue, 21 Mar 2006 22:13:28 -0800 (PST) Received: from fe-apac-06.sun.com (fe-apac-06.sun.com [192.18.19.177] (may be forged)) by sineb-mail-2.sun.com (8.12.10+Sun/8.12.9) with ESMTP id k2M6DQIS017323; Wed, 22 Mar 2006 14:13:26 +0800 (SGT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java System Messaging Server 6.2-4.02 (built Sep 9 2005)) id <0IWI00G01MKNH700@mail-apac.sun.com> (original mail from Edward.Shu@Sun.COM); Wed, 22 Mar 2006 14:13:26 +0800 (SGT) Received: from [129.158.218.35] by mail-apac.sun.com (Sun Java System Messaging Server 6.2-4.02 (built Sep 9 2005)) with ESMTPSA id <0IWI008L1MMD6K09@mail-apac.sun.com>; Wed, 22 Mar 2006 14:13:26 +0800 (SGT) Date: Wed, 22 Mar 2006 14:09:07 +0800 From: Edward Shu Subject: Re: 2006/188 [Memory Cache Disable Support on x86] In-reply-to: <200603220117.k2M1Hel3011848@ivrel.sfbay.sun.com> Sender: Edward.Shu@Sun.COM To: Glenn Skinner Cc: psarc@Sun.COM, eiji.ota@Sun.COM, mark.johnson@Sun.COM Message-id: <4420EA03.2000506@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=us-ascii Content-transfer-encoding: 7BIT X-Accept-Language: zh-cn, en-us, en X-PMX-Version: 5.1.2.240295 References: <200603220117.k2M1Hel3011848@ivrel.sfbay.sun.com> User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; zh-CN; rv:1.7) Gecko/20050614 Status: RO Content-Length: 2074 Some comment below.. Glenn Skinner: > Date: Tue, 21 Mar 2006 17:02:44 -0800 (PST) > From: Shudong Zhou > Subject: PSARC 2006/188 Memory Cache Disable Support on x86 > > Skipping down to the man page part of the writeup, we have: > > The devacc_attr_dataorder member describes order in which > the CPU will reference data. Specify one of the following > values. > > ... > > | DDI_CACHEDISABLE_ACC The CPU can keep the data > | from the cache and write the > | data to memory without it. > | The default behavior is to > | push the data right away. > | DDI_CACHEDISABLE_ACC also > | implies strict ordering. > > These values are advisory, not mandatory. For example, data > can be ordered without being merged or cached, even though a > driver requests unordered, merged, and cached together. > ... > > Does the final paragraph quoted above hold for DDI_CACHEDISABLE_ACC? > If so, is there any way for the driver to tell when it's not getting > what it asked for? There is a rough and implicit rule here. the order of Cache control precedence is, from AMD64 manual: 1. uncachable 2. write-combining 3. write-protected 4. write-through 5. Writeback The order of this list is from the highest to the lowest. If data requests lower order, data can be higher order. But the reverse is not true. And cache disable is the strictest oreder memory in X86. So if data requests cache disable, the DDI call should be fail if the data can not be cache disable. > > -- Glenn > -- ------------------------------------------ -Edward Shu -Solaris x86 Engineering, Sun Microsystems -tele: +86-10-82618200-82100 __________________________________________ From sacadmin Tue Mar 21 23:46:40 2006 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2M7kdIQ027534 for ; Tue, 21 Mar 2006 23:46:39 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.149.246.28]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id k2M7kDX2011385; Wed, 22 Mar 2006 15:46:35 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2 (built Dec 2 2004)) id <0IWI00101QXAOZ00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 21 Mar 2006 23:46:22 -0800 (PST) Received: from jurassic.eng.sun.com ([129.146.17.57]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2 (built Dec 2 2004)) with ESMTP id <0IWI00J1IQXAVX90@nwk-avmta-1.sfbay.Sun.COM>; Tue, 21 Mar 2006 23:46:22 -0800 (PST) Received: from localhost (ota.SFBay.Sun.COM [129.146.226.183]) by jurassic.eng.sun.com (8.13.6.Gamma0+Sun/8.13.5) with ESMTP id k2M7kM4J447655; Tue, 21 Mar 2006 23:46:22 -0800 (PST) Date: Tue, 21 Mar 2006 23:46:17 -0800 (PST) From: eiji.ota@sun.com Subject: Re: 2006/188 [Memory Cache Disable Support on x86] In-reply-to: <4420EA03.2000506@sun.com> Sender: Eiji Ota To: Edward.Shu@sun.com Cc: glenn@ivrel.sfbay.sun.com, psarc@sun.com, mark.johnson@sun.com Message-id: <20060321234617C.eota@sun.com> MIME-version: 1.0 X-Mailer: Mew version 1.94.2 on XEmacs 21.1 (Cuyahoga Valley) Content-type: Text/Plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.1.2.240295 Lines: 41 References: <200603220117.k2M1Hel3011848@ivrel.sfbay.sun.com> <4420EA03.2000506@sun.com> X-Dispatcher: imput version 20000228(IM140) Status: RO Content-Length: 1712 > > | DDI_CACHEDISABLE_ACC The CPU can keep the data > > | from the cache and write the > > | data to memory without it. > > | The default behavior is to > > | push the data right away. > > | DDI_CACHEDISABLE_ACC also > > | implies strict ordering. > > > > These values are advisory, not mandatory. For example, data > > can be ordered without being merged or cached, even though a > > driver requests unordered, merged, and cached together. > > ... > > > > Does the final paragraph quoted above hold for DDI_CACHEDISABLE_ACC? > > If so, is there any way for the driver to tell when it's not getting > > what it asked for? > > There is a rough and implicit rule here. the order of > Cache control precedence is, from AMD64 manual: > 1. uncachable > 2. write-combining > 3. write-protected > 4. write-through > 5. Writeback > > The order of this list is from the highest to the lowest. > If data requests lower order, data can be higher order. > But the reverse is not true. > And cache disable is the strictest oreder memory in X86. > So if data requests cache disable, the DDI call should be > fail if the data can not be cache disable. Hi Ed, On X86, the request for uncache should be successful unless (uncacheable) memory can be allocated for some reason. Otherwise the interface returns an error as you expect. (e.g. ddi_dma_mem_alloc()) Cheers, -Eiji From sacadmin Wed Mar 22 08:11:06 2006 Received: from sunmail1brm.Central.Sun.COM (sunmail1brm.Central.Sun.COM [129.147.62.17]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2MGB2IQ018547 for ; Wed, 22 Mar 2006 08:11:02 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.149.247.22]) by sunmail1brm.Central.Sun.COM (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k2MGAvC13294; Wed, 22 Mar 2006 09:11:01 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IWJ00F19EA51B00@nwk-avmta-2.sfbay.sun.com>; Wed, 22 Mar 2006 08:10:53 -0800 (PST) Received: from eastmail1bur.East.Sun.COM ([129.148.9.49]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IWJ00DI3EA43E40@nwk-avmta-2.sfbay.sun.com>; Wed, 22 Mar 2006 08:10:53 -0800 (PST) Received: from phys-bur1-1 (phys-bur1-1.East.Sun.COM [129.148.13.15]) by eastmail1bur.East.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k2MGAqbB009981; Wed, 22 Mar 2006 11:10:52 -0500 (EST) Received: from conversion-daemon.bur-mail2.east.sun.com by bur-mail2.east.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IWJ00801E9TC6@bur-mail2.east.sun.com> (original mail from mark.johnson@sun.com); Wed, 22 Mar 2006 11:10:52 -0500 (EST) Received: from [192.168.0.7] (vpn-129-150-66-219.East.Sun.COM [129.150.66.219]) by bur-mail2.east.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with ESMTPA id <0IWJ00BTVEA3JG@bur-mail2.east.sun.com>; Wed, 22 Mar 2006 11:10:52 -0500 (EST) Date: Wed, 22 Mar 2006 11:10:48 -0500 From: Mark Johnson Subject: Re: PSARC 2006/188 Memory Cache Disable Support on x86 In-reply-to: <20060321175507A.eota@sun.com> To: Eiji.Ota@sun.com Cc: sommerfeld@sun.com, szhou@billybob.sfbay.sun.com, psarc@sun.com Message-id: <44217708.2090402@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.1.2.240295 References: <200603220102.k2M12iAH107448@billybob.sfbay.sun.com> <1142991212.17801.50.camel@thunk> <20060321175507A.eota@sun.com> User-Agent: Mail/News 1.5.0.2 (Macintosh/20060310) Status: RO Content-Length: 2939 Eiji.Ota@Sun.COM wrote: >>> A new flag, DDI_CACHEDISABLE_ACC, is introduced to provide >>> the uncacheable memory access. Though we have DDI_STRICTORDER_ACC to >>> describe the strict ordering, if this were used for the uncacheable >>> memory access, it could cause performance degradation since >>> DDI_STRICTORDER_ACC is used by most drivers today. >> I'm confused. >> >> The man page also lists a "DDI_LOADCACHING_OK_ACC"; the description of >> this option strongly implies that DDI_STRICTORDER_ACC (as well as >> UNORDERED and MERGING_OK) doesn't allow for caching. >> >> What caching is currently done by STRICTORDER? How can we fix the man >> page to match the actual system behavior? > > The value is actually advisory, and how it should be treated is somehow > dependent on HW; for memory, X86 usually assumes HAT_STORECACHING_OK > should be used to gain performance (coming from caches). In this case, > STRICTORDER changes to STORECACHING. However, for the IO space, it should > change to non-cache. Then the problem is coming up these days that some > devices (e.g. graphics) want to use non-cache memory for their framebuffer, > but there is no way for that purpose. The new value, DDI_CACHEDISABLE_ACC, > is prepared to meet that request. > >> This looks like a case where separate feature tests would be better than >> an enumeration. Let me add a little more history to this to clarify things... In Solaris x86 from the very beginning up through today, we do not have anyway for a driver to allocate memory on x86 with different cache setting. This is a bug and one of the reasons we needed a special graphics misc module to workaround that (among other deficiencies). On x86 today, when you allocate device registers with DDI_STRICTORDER_ACC, you get cache disable, but when you allocate memory with DDI_STRICTORDER_ACC, you get cacheable memory. So the behavior is inconsistent. What further complicates matters, is that virtually all drivers allocate memory with DDI_STRICTORDER_ACC (but should be using cacheable memory). So if we were to fix this, and have DDI_STRICTORDER_ACC allocate memory with the cache disable bit set, we would impact the performance of all drivers on x86. To make things more interesting, when you ask the hat to allocate memory with HAT_STRICTORDER, it will ignore you and assume you really wanted cacheable memory. They added a additional flag (HAT_PLAT_NOCACHE) late in s10 development to basically mean, "I really meant it when I said I wanted the cache disabled". DDI_CACHEDISABLE_ACC would take advantage of this flag on x86. Since changing todays behavior of DDI_STRICTORDER_ACC isn't acceptable, adding an additional attribute, which is only needed by a small number of drivers that we never payed attention to in the past, is thought to be the best solution. Not pretty, but IMO, an appropriate solution to a messy problem. MRJ From sacadmin Wed Mar 22 09:10:22 2006 Received: from sunmail3.sfbay.sun.com (sunmail3.SFBay.Sun.COM [129.149.247.180]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2MHALIQ020826 for ; Wed, 22 Mar 2006 09:10:21 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.149.246.28]) by sunmail3.sfbay.sun.com (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k2MHAI500221 for <@sunmail2.sfbay.sun.com:psarc@Sun.COM>; Wed, 22 Mar 2006 09:10:18 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2 (built Dec 2 2004)) id <0IWJ00C03H0VYL00@nwk-avmta-1.sfbay.Sun.COM> for psarc@Sun.COM (ORCPT psarc@sun.com); Wed, 22 Mar 2006 09:10:07 -0800 (PST) Received: from centralmail1brm.Central.Sun.COM ([129.147.62.1]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2 (built Dec 2 2004)) with ESMTP id <0IWJ00BN5H0V5E90@nwk-avmta-1.sfbay.Sun.COM> for psarc@Sun.COM (ORCPT psarc@sun.com); Wed, 22 Mar 2006 09:10:07 -0800 (PST) Received: from phys-aus08-1 (phys-aus08-1.Central.Sun.COM [129.153.131.88]) by centralmail1brm.Central.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k2MHA6ca013295 for ; Wed, 22 Mar 2006 10:10:07 -0700 (MST) Received: from conversion-daemon.aus08-mail1.central.sun.com by aus08-mail1.central.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IWJ00J01GWSJE@aus08-mail1.central.sun.com> (original mail from Eric.Lowe@Sun.COM) for psarc@sun.com; Wed, 22 Mar 2006 11:10:06 -0600 (CST) Received: from [192.9.61.227] (punchin-elowe.SFBay.Sun.COM [192.9.61.227]) by aus08-mail1.central.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with ESMTP id <0IWJ00J6RH0SGD@aus08-mail1.central.sun.com>; Wed, 22 Mar 2006 11:10:06 -0600 (CST) Date: Wed, 22 Mar 2006 11:11:33 -0600 From: Eric Lowe Subject: Re: PSARC 2006/188 Memory Cache Disable Support on x86 In-reply-to: <200603220102.k2M12iAH107448@billybob.sfbay.sun.com> To: Shudong Zhou Cc: psarc@Sun.COM, Eiji.Ota@Sun.COM, Mark.Johnson@Sun.COM Reply-to: Eric.Lowe@Sun.COM Message-id: <44218545.1030805@Sun.COM> Organization: Sun Microsystems MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.1.2.240295 References: <200603220102.k2M12iAH107448@billybob.sfbay.sun.com> User-Agent: Thunderbird 1.5 (X11/20060113) Status: RO Content-Length: 21738 One concern I have is that this proposal by itself doesn't fill all of the gaps in the possible values, leaving room for interpretation (still) by driver authors who have a clear set of intended semantics in mind. It leaves the door open for us to end up back here again wanting to add yet another flag down the road. This seems like a good "fix" opportunity. Another concern is around whether or not non-cached implies combining (merging) or not. I'm intentionally ignoring constraints of the implementation, and am assuming the implementation will implement the least common denominator which provides the requested semantics. Sigh -- it's unfortunate the interface doesn't use flags but instead forces us to choose one value. This makes things hard. :( More on that thought later. As I see it there are four things I might want to specify as a driver author: required ordering, whether load caching is OK, whether store caching is OK, and whether combining (merging) is allowable for my device data stream. I should be able to tell the interface what I want, unambiguously, and have it happen for me; furthermore, I want to accomplish this without having to decrypt convoluted flags and glean details of the implementation (as is certainly done now -- or else folks just give up and go with the least common denominator / most restrictive option, costing performance). Ordering: the DDI appears to support relaxed or strict ordering. The default is documented as strict ordering but relaxed is implied in all caching modes. Store caching: can be on or off. The default is documented to be off. Load caching: can be on or off. The default is documented to be off. Combining: can be on or off. The default is documented to be on. Enumerating the combinations: SO|NOSTORECACHE|NOLOADCACHE|??? -- if I understand your proposal correctly this is your proposed flag. However you don't specify whether combining is allowable. I believe you should specify. E.g. do you intend to enable write combine in the PTE without the cache enable bit? Leaving this as an implementation detail is, IMO, a big mistake, and follows a pattern of past mistakes made in this interface. :) RMO|NOSTORECACHE|NOLOADCACHE|NOCOMBINE -- this is DDI_UNORDERED_OK_ACC RMO|NOSTORECACHE|NOLOADCACHE|COMBINE -- this is DDI_MERGING_OK_ACC RMO|NOSTORECACHE|LOADCACHE|COMBINE -- this is DDI_LOADCACHING_OK_ACC RMO|STORECACHE|LOADCACHE|COMBINE -- this is DDI_STORECACHING_OK_ACC Of course there are still many other combinations which aren't implemented. I'm groaning because DDI_STRICTORDER_ACC doesn't say anything about caching. We've left a lot of room for the implementation there, and that is IMO not a good thing. I don't even know where it fits in the above in the current implementation but in terms of architecture it must be defacto SO|LOADCACHE|STORECACHE|COMBINE which is least restrictive? Some more enumerations may make sense. Others may not. For instance I can certainly envision a use for strict-ordered versions of the various combinations of caching on platforms which provide it. On the other hand, I would expect caching of any sort to imply combining since the cache line widths are almost certainly not going to match the device access size. Maybe I'm trying to impose a developer tax where I shouldn't, and adding just one more flag is the right thing, but at a minimum I'd like to see another case brought forward (and soon) which attempts to clarify once and for all what the expected behavior is with all of the various and sundry .*_ACC flags, and preferably introduce a better architecture. As it is today, the semantics are not clear, and worse the architecture gives way too much leeway to the implementation -- which in this case is a bad thing since it can lead to assumptions in behavior and resultant portability problems. IMO the following changes to the architecture would solve your problems and mine with one stone: - Rev the device attr structure to DDI_DEVICE_ATTR_V1 - Keep data order so you can specify strict or relaxed ordering; combining also logically goes there. - Add a new caching attribute which allows specification of no caching, load caching only, store caching only, or load and store caching being OK. The driver which spurred this case would use the DDI_DEVICE_ATTR_V1 to get its non-cached memory, while everybody else could safely stay at _V0 for the time being. This also eliminates the risks associated with changing semantics of existing flags whose meaning is not clear (e.g. DDI_STRICTORDER_ACC). - Eric Shudong Zhou wrote: > I'm sponsoring the following fasttrack for Eiji Ota. The timer is > set to expire on 3/28/2006. The requested release binding is patch. > > Shudong > > Template Version: @(#)onepager.txt 1.29 04/11/15 SMI > > This information is > Copyright 2006 Sun Microsystems, Inc. > > 1. Introduction > 1.1. Project/Component Working Name: > Allocating memory from the DDI with Cache Disable Support on x86 > > 1.2. Name of Document Author/Supplier: > Eiji Ota (eiji.ota@sun.com) > Mark Johnson (mark.johnson@sun.com) > > 1.3. Date of This Document: > 03/03/06 > > 1.4. Name of Major Document Customer(s)/Consumer(s): > 1.4.1. The PAC or CPT you expect to review your project: > Solaris PAC > > 1.4.2. The ARC(s) you expect to review your project: > PSARC > > 1.4.3. The Director/VP who is "Sponsoring" this project: > Barry Cooks (barry.cooks@sun.com) > > 1.4.4. The name of your business unit: > OPG, Solaris Core Technologies > > 1.5. Email Aliases: > 1.5.1. Responsible Manager: peter.luk@sun.com > 1.5.2. Responsible Engineer: eiji.ota@sun.com, mark.johnson@sun.com > 1.5.3. Marketing Manager: chris.ratcliffe@sun.com > 1.5.4. Interest List: iom-interest@Sun.Com > > 2. Project Summary > > 2.1. Project Description: > The project provides a method to access uncacheable memory, which is > required for some graphics drivers running on X86/AMD64 architecture. > > 2.2. Risks and Assumptions: > None that we know of. > > 3. Business Summary > > 3.1. Problem Area: > Some graphics drivers on X86/AMD64 require a method to have an > access to memory with the uncacheable attribute to keep coherency > between a graphic chip and the rest on the system, but Solaris > doesn't provide any interfaces those drivers can use. As a result, > they have to implement how to access the uncacheable memory in their > own ways. This could be seen inconvenient by Solaris driver developers > and inconsistent by Solaris kernel developers. > > 3.2. Market/Requester: > Graphic device drivers running on X86 architecture. > > 3.3. Business Justification: > Uncacheable memory access is required for some graphics chips > on X86/AMD64 architecture these days. However, Solaris doesn't > provide the DDI interface for that purpose yet. This project provides > the unified and consistent method to support those graphics drivers > running on X86/AMD64. > > 3.4. Competitive Analysis: > Major operating systems running on X86/AMD64 (i.e. Linux/Windows) > support similar functions already. > > 4. Technical Description: > > 4.1. Details: > A new flag, DDI_CACHEDISABLE_ACC, is introduced to provide > the uncacheable memory access. Though we have DDI_STRICTORDER_ACC to > describe the strict ordering, if this were used for the uncacheable > memory access, it could cause performance degradation since > DDI_STRICTORDER_ACC is used by most drivers today. > > The flag should be set to devacc_attr_dataorder in struct > ddi_device_acc_attr when a driver wants to access memory with the > uncacheable attribute. This flag is referred to by devmap_umem_setup(), > devmap_umem_remap() and ddi_dma_mem_alloc(). > > devmap_umem_setup() and devmap_umem_remap() are used to export kernel > memory to the userland. The new flag is used to allow an application > to have an access to the uncacheable memory by mmap(). > > ddi_dma_mem_alloc() is used to allocate non-pageable kernel memory for > a driver. The new flag is used to provide a driver with an access to > the uncacheable memory in the same manner. > > The gfx_private module is also updated to implement the uncacheable > memory access by this new flag. > > 4.2. Bug/RFE Number(s): > 6212057 ddi_dma_mem_alloc should support DDI_STRICTORDER_ACC in x86 > > 4.3. Interfaces: > +-------------------------+---------------------+------------------+ > | Interface Name | Classification | Comments | > +-------------------------+---------------------+------------------+ > | DDI_CACHEDISABLE_ACC | Evolving | -------- | > +-------------------------+---------------------+------------------+ > > The following example shows how to use this flag for the uncacheable > memory access; a driver doesn't need any other things than just setting > DDI_CACHEDISABLE_ACC to devacc_attr_dataorder as follows. > > struct ddi_device_acc_attr XXX_acc_attr = { > DDI_DEVICE_ATTR_V0, > DDI_NEVERSWAP_ACC, > DDI_CACHEDISABLE_ACC /* use the uncacheable memory */ > }; > > 4.6. Doc Impact: > Man page (ddi_device_acc_attr(9S)). > The difference is attached below. (the change has bars on the left) > > 4.7. Admin/Config Impact: > No impact on Admin/Config. > > 4.8. HA Impact: > No impact on HA. > > 4.9. I18N/L10N Impact: > No impact on I18N/L10L > > 4.10. Packaging & Delivery: > No impact on packages, clusters and metaclusters. > No impact on install/upgrade. > > 4.11. Security Impact: > No impact on Security. > > 4.12. Dependencies: > None. > > 5. Reference Documents: > PSARC 2004/151 AGPgart project > PSARC 2005/475 Graphics private misc module for x86 > PSARC 2005/623 gfx_private update for devmap and DMA > > 6. Resources and Schedule: > > 6.1. Projected Availability: > April/2006 > > 6.4. Product Approval Committee requested information: > 6.4.1. Consolidation or Component Name: > ON > > 6.4.3. Type of CPT Review and Approval expected: > FastTrack > > 6.4.7. Target RTI Date/Release: > onnv_b38 > > 6.4.8. Target Code Design Review Date: > 3/13 > > 6.5. ARC review type: > FastTrack > > > > (ddi_device_acc_attr(9S)) > > Data Structures for Drivers ddi_device_acc_attr(9S) > > NAME > ddi_device_acc_attr - data access attributes structure > > SYNOPSIS > #include > #include > > INTERFACE LEVEL > Solaris DDI specific (Solaris DDI). > > DESCRIPTION > The ddi_device_acc_attr structure describes the data access > characteristics and requirements of the device. > > STRUCTURE MEMBERS > ushort_t devacc_attr_version; > uchar_t devacc_attr_endian_flags; > uchar_t devacc_attr_dataorder; > > The devacc_attr_version member identifies the version number > of this structure. The current version number is > DDI_DEVICE_ATTR_V0. > > The devacc_attr_endian_flags member describes the endian > characteristics of the device. Specify one of the following > values: > > DDI_NEVERSWAP_ACC Data access with no byte > swapping > > DDI_STRUCTURE_BE_ACC Structural data access in > big-endian format > > DDI_STRUCTURE_LE_ACC Structural data access in > little endian format > > DDI_STRUCTURE_BE_ACC and DDI_STRUCTURE_LE_ACC describes the > endian characteristics of the device as big-endian or > little-endian, respectively. Though most of the devices will > have the same endian characteristics as their buses, exam- > ples of devices that have opposite endian characteristics of > the buses do exist. When DDI_STRUCTURE_BE_ACC or > DDI_STRUCTURE_LE_ACC is set, byte swapping is automatically > performed by the system if the host machine and the device > > data formats have opposite endian characteristics. The > implementation can take advantage of hardware platform byte > swapping capabilities. > > When you specify DDI_NEVERSWAP_ACC, byte swapping is not > invoked in the data access functions. > > The devacc_attr_dataorder member describes order in which > the CPU will reference data. Specify one of the following > values. > > DDI_STRICTORDER_ACC The data references must be > issued by a CPU in program > order. Strict ordering is > the default behavior. > > DDI_UNORDERED_OK_ACC The CPU can re-order the > data references. This > includes all kinds of re- > ordering. For example, a > load followed by a store may > be replaced by a store fol- > lowed by a load. > > DDI_MERGING_OK_ACC The CPU can merge indivi- > dual stores to consecutive > locations. For example, the > CPU can turn two consecutive > byte stores into one half- > word store. It can also > batch individual loads. For > example, the CPU might turn > two consecutive byte loads > into one halfword load. > DDI_MERGING_OK_ACC also > implies re-ordering. > > DDI_LOADCACHING_OK_ACC The CPU can cache the data > it fetches and reuse it > until another store occurs. > The default behavior is to > fetch new data on every > load. DDI_LOADCACHING_OK_ACC > also implies merging and > re-ordering. > > DDI_STORECACHING_OK_ACC The CPU can keep the data > in the cache and push it to > the device (perhaps with > other data) at a later time. > The default behavior is to > push the data right away. > DDI_STORECACHING_OK_ACC also > implies load caching, merg- > ing, and re-ordering. > > | DDI_CACHEDISABLE_ACC The CPU can keep the data > | from the cache and write the > | data to memory without it. > | The default behavior is to > | push the data right away. > | DDI_CACHEDISABLE_ACC also > | implies strict ordering. > > These values are advisory, not mandatory. For example, data > can be ordered without being merged or cached, even though a > driver requests unordered, merged, and cached together. > > EXAMPLES > The following examples illustrate the use of device register > address mapping setup functions and different data access > functions. > > Example 1: Using ddi_device_acc_attr() in > ddi_regs_map_setup(9F) > > This example demonstrates the use of the > ddi_device_acc_attr() structure in ddi_regs_map_setup(9F). > It also shows the use of ddi_getw(9F) and ddi_putw(9F) > functions in accessing the register contents. > > dev_info_t *dip; > uint_t rnumber; > ushort_t *dev_addr; > offset_t offset; > offset_t len; > ushort_t dev_command; > ddi_device_acc_attr_t dev_attr; > ddi_acc_handle_t handle; > > ... > > /* > * setup the device attribute structure for little endian, > * strict ordering and 16-bit word access. > */ > dev_attr.devacc_attr_version = DDI_DEVICE_ATTR_V0; > dev_attr.devacc_attr_endian_flags = DDI_STRUCTURE_LE_ACC; > dev_attr.devacc_attr_dataorder = DDI_STRICTORDER_ACC; > > /* > * set up the device registers address mapping > */ > > ddi_regs_map_setup(dip, rnumber, (caddr_t *)&dev_addr, offset, len, > &dev_attr, &handle); > > /* read a 16-bit word command register from the device */ > dev_command = ddi_getw(handle, dev_addr); > > dev_command |= DEV_INTR_ENABLE; > /* store a new value back to the device command register */ > ddi_putw(handle, dev_addr, dev_command); > > Example 2: Accessing a Device with Different Apertures > > The following example illustrates the steps used to access a > device with different apertures. Several apertures are > assumed to be grouped under one single "reg" entry. For > example, the sample device has four different apertures, > each 32 Kbyte in size. The apertures represent YUV little- > endian, YUV big-endian, RGB little-endian, and RGB big- > endian. This sample device uses entry 1 of the "reg" pro- > perty list for this purpose. The size of the address space > is 128 Kbyte with each 32 Kbyte range as a separate aper- > ture. In the register mapping setup function, the sample > driver uses the offset and len parameters to specify one > of the apertures. > > ulong_t *dev_addr; > ddi_device_acc_attr_t dev_attr; > ddi_acc_handle_t handle; > uchar_t buf[256]; > > ... > > /* > * setup the device attribute structure for never swap, > * unordered and 32-bit word access. > */ > dev_attr.devacc_attr_version = DDI_DEVICE_ATTR_V0; > dev_attr.devacc_attr_endian_flags = DDI_NEVERSWAP_ACC; > dev_attr.devacc_attr_dataorder = DDI_UNORDERED_OK_ACC; > > /* > * map in the RGB big-endian aperture > * while running in a big endian machine > * - offset 96K and len 32K > */ > ddi_regs_map_setup(dip, 1, (caddr_t *)&dev_addr, 96*1024, 32*1024, > &dev_attr, &handle); > > /* > * Write to the screen buffer > * first 1K bytes words, each size 4 bytes > */ > ddi_rep_putl(handle, buf, dev_addr, 256, DDI_DEV_AUTOINCR); > > Example 3: Functions That Call Out the Data Word Size > > The following example illustrates the use of the functions > that explicitly call out the data word size to override the > data size in the device attribute structure. > > struct device_blk { > ushort_t d_command; /* command register */ > ushort_t d_status; /* status register */ > ulong d_data; /* data register */ > } *dev_blkp; > dev_info_t *dip; > caddr_t dev_addr; > ddi_device_acc_attr_t dev_attr; > ddi_acc_handle_t handle; > uchar_t buf[256]; > > ... > > /* > * setup the device attribute structure for never swap, > * strict ordering and 32-bit word access. > */ > dev_attr.devacc_attr_version = DDI_DEVICE_ATTR_V0; > dev_attr.devacc_attr_endian_flags = DDI_NEVERSWAP_ACC; > dev_attr.devacc_attr_dataorder= DDI_STRICTORDER_ACC; > > ddi_regs_map_setup(dip, 1, (caddr_t *)&dev_blkp, 0, 0, > &dev_attr, &handle); > > /* write command to the 16-bit command register */ > ddi_putw(handle, &dev_blkp->d_command, START_XFER); > > /* Read the 16-bit status register */ > status = ddi_getw(handle, &dev_blkp->d_status); > > if (status & DATA_READY) > /* Read 1K bytes off the 32-bit data register */ > ddi_rep_getl(handle, buf, &dev_blkp->d_data, > 256, DDI_DEV_NO_AUTOINCR); > > SEE ALSO > ddi_getw(9F), ddi_putw(9F), ddi_regs_map_setup(9F) > > Writing Device Drivers > > -- Eric Lowe Solaris Kernel Development Austin, Texas Sun Microsystems. We make the net work. x40577/+1(512)366-9080 From sacadmin Wed Mar 22 09:14:14 2006 Received: from sunmail1brm.Central.Sun.COM (sunmail1brm.Central.Sun.COM [129.147.62.17]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2MHE8IQ021057 for ; Wed, 22 Mar 2006 09:14:09 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail1brm.Central.Sun.COM (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k2MHE8C19488 for <@sunmail1brm.central.sun.com:psarc@Sun.COM>; Wed, 22 Mar 2006 10:14:08 -0700 (MST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IWJ00A05H7JCB00@brm-avmta-1.central.sun.com> for psarc@Sun.COM (ORCPT psarc@sun.com); Wed, 22 Mar 2006 10:14:07 -0700 (MST) Received: from centralmail1brm.Central.Sun.COM ([129.147.62.1]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IWJ00AQ7H7I0V10@brm-avmta-1.central.sun.com> for psarc@Sun.COM (ORCPT psarc@sun.com); Wed, 22 Mar 2006 10:14:06 -0700 (MST) Received: from phys-aus08-1 (phys-aus08-1.Central.Sun.COM [129.153.131.88]) by centralmail1brm.Central.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k2MHE3ca014963 for ; Wed, 22 Mar 2006 10:14:03 -0700 (MST) Received: from conversion-daemon.aus08-mail1.central.sun.com by aus08-mail1.central.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IWJ00J01GWSJE@aus08-mail1.central.sun.com> (original mail from Eric.Lowe@Sun.COM) for psarc@sun.com; Wed, 22 Mar 2006 11:14:02 -0600 (CST) Received: from [192.9.61.227] (punchin-elowe.SFBay.Sun.COM [192.9.61.227]) by aus08-mail1.central.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with ESMTP id <0IWJ00JG6H7DGD@aus08-mail1.central.sun.com>; Wed, 22 Mar 2006 11:14:02 -0600 (CST) Date: Wed, 22 Mar 2006 11:15:29 -0600 From: Eric Lowe Subject: Re: 2006/188 [Memory Cache Disable Support on x86] In-reply-to: <20060321173803N.eota@sun.com> To: Eiji.Ota@Sun.COM Cc: glenn@ivrel.sfbay.sun.com, psarc@Sun.COM, Mark.Johnson@Sun.COM Reply-to: Eric.Lowe@Sun.COM Message-id: <44218631.5000005@Sun.COM> Organization: Sun Microsystems MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.1.2.240295 References: <200603220117.k2M1Hel3011848@ivrel.sfbay.sun.com> <20060321173803N.eota@sun.com> User-Agent: Thunderbird 1.5 (X11/20060113) Status: RO Content-Length: 783 > The value should be advisory, not mandatory because whether or not > it can be achieved depends on the HW implementation somehow -- for example, > disabling cache on SPARC doesn't make sense since the coherency is always > maintained on the platform, then DDI_CACHEDISABLE_ACC must be visually > voided on it. > > Unfortunately there is no explicit way (I mean, via ddi-interface) to > show how your request is treated in the end except you check the TTE/PTE. > I'm afraid providing such a method is a bit out of the scope this time. And it should remain out of scope. Since we have differing possible implementations and valid combinations of semantics, this interface is necessarily "tell me what you want", rather than "tell me what you'll get". That's by design. - Eric From sacadmin Wed Mar 22 09:34:47 2006 Received: from sunmail3.sfbay.sun.com (sunmail3.SFBay.Sun.COM [129.149.247.180]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2MHYlIQ021960 for ; Wed, 22 Mar 2006 09:34:47 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3.sfbay.sun.com (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k2MHYd514839; Wed, 22 Mar 2006 09:34:39 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IWJ00A0DI5KW900@brm-avmta-1.central.sun.com>; Wed, 22 Mar 2006 10:34:32 -0700 (MST) Received: from eastmail1bur.East.Sun.COM ([129.148.9.49]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IWJ00AVRI5J0V40@brm-avmta-1.central.sun.com>; Wed, 22 Mar 2006 10:34:32 -0700 (MST) Received: from thunk.east.sun.com (thunk.East.Sun.COM [129.148.174.66]) by eastmail1bur.East.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k2MHYTbB024618; Wed, 22 Mar 2006 12:34:29 -0500 (EST) Received: from 127.0.0.1 (localhost [127.0.0.1]) by thunk.east.sun.com (8.13.5+Sun/8.13.5) with ESMTP id k2MHYTRn021462; Wed, 22 Mar 2006 12:34:29 -0500 (EST) Date: Wed, 22 Mar 2006 12:34:29 -0500 From: Bill Sommerfeld Subject: Re: PSARC 2006/188 Memory Cache Disable Support on x86 In-reply-to: <44217708.2090402@sun.com> To: Mark Johnson Cc: Eiji.Ota@sun.com, szhou@billybob.sfbay.sun.com, psarc@sun.com Message-id: <1143048868.21056.65.camel@thunk> MIME-version: 1.0 X-Mailer: Ximian Evolution 1.4.6.333 Content-type: text/plain; charset=iso-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.1.2.240295 References: <200603220102.k2M12iAH107448@billybob.sfbay.sun.com> <1142991212.17801.50.camel@thunk> <20060321175507A.eota@sun.com> <44217708.2090402@sun.com> Status: RO Content-Length: 881 On Wed, 2006-03-22 at 11:10, Mark Johnson wrote: > Let me add a little more history to this to clarify things... > > In Solaris x86 from the very beginning up through today, we do not have anyway > for a driver to allocate memory on x86 with different cache setting. This > is a bug and one of the reasons we needed a special graphics misc module > to workaround that (among other deficiencies). > > On x86 today, when you allocate device registers with DDI_STRICTORDER_ACC, > you get cache disable, but when you allocate memory with DDI_STRICTORDER_ACC, > you get cacheable memory. So the behavior is inconsistent. So, I like Eric's suggestion of revving the interface to add an orthogonal caching attribute but there's still the quesition about how to fix the documentation to close the gap between it and the current/traditional behavior of this interface.. - Bill From sacadmin Wed Mar 22 09:53:39 2006 Received: from sunmail2.sfbay.sun.com (sunmail2.SFBay.Sun.COM [129.149.246.180]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2MHrdIQ023847 for ; Wed, 22 Mar 2006 09:53:39 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2.sfbay.sun.com (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k2MHrcv05422 for <@sunmail1brm.central.sun.com:psarc@sun.com>; Wed, 22 Mar 2006 09:53:38 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IWJ00B0DJ1EGB00@brm-avmta-1.central.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Wed, 22 Mar 2006 10:53:38 -0700 (MST) Received: from sfbaymail1sca.SFBay.Sun.COM ([129.145.154.35]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IWJ00AUNJ1D0X70@brm-avmta-1.central.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Wed, 22 Mar 2006 10:53:37 -0700 (MST) Received: from ivrel.sfbay.sun.com (ivrel.SFBay.Sun.COM [129.146.74.76]) by sfbaymail1sca.SFBay.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k2MHraEZ014001; Wed, 22 Mar 2006 09:53:36 -0800 (PST) Received: from ivrel (ivrel [129.146.74.76]) by ivrel.sfbay.sun.com (8.13.5+Sun/8.13.5) with SMTP id k2MHraee012671; Wed, 22 Mar 2006 09:53:36 -0800 (PST) Date: Wed, 22 Mar 2006 09:53:36 -0800 (PST) From: Glenn Skinner Subject: Re: 2006/188 Memory Cache Disable Support on x86 To: mark.johnson@sun.com Cc: Eiji.Ota@sun.com, psarc@sun.com Reply-to: Glenn Skinner Message-id: <200603221753.k2MHraee012671@ivrel.sfbay.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.6_36 SunOS 5.11 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-MD5: dcWsEEzzp0ldK+s7hXqG/A== X-PMX-Version: 5.1.2.240295 Status: RO Content-Length: 794 Date: Wed, 22 Mar 2006 12:34:29 -0500 From: Bill Sommerfeld Subject: Re: PSARC 2006/188 Memory Cache Disable Support on x86 So, I like Eric's suggestion of revving the interface to add an orthogonal caching attribute but there's still the quesition about how to fix the documentation to close the gap between it and the current/traditional behavior of this interface.. I like the suggestion as well. As I see them, the high level requirements are that a driver writer be able to state the desired memory properties up front, and then be able to determine what the resulting properties actually turned out to be. Given that, it's possible to write a robust driver; without it, it isn't (without relying on implicit assumptions). -- Glenn From sacadmin Wed Mar 22 10:10:10 2006 Received: from sunmail3.sfbay.sun.com (sunmail3.SFBay.Sun.COM [129.149.247.180]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2MIAAIQ025533 for ; Wed, 22 Mar 2006 10:10:10 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3.sfbay.sun.com (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k2MIA9503126 for <@sunmail1brm.central.sun.com:psarc@sun.com>; Wed, 22 Mar 2006 10:10:09 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IWJ00C0VJSV2800@brm-avmta-1.central.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Wed, 22 Mar 2006 11:10:07 -0700 (MST) Received: from eastmail1bur.East.Sun.COM ([129.148.9.49]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IWJ00ABKJST0XA0@brm-avmta-1.central.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Wed, 22 Mar 2006 11:10:06 -0700 (MST) Received: from phys-bur1-1 (phys-bur1-1.East.Sun.COM [129.148.13.15]) by eastmail1bur.East.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k2MIA5bB016188 for ; Wed, 22 Mar 2006 13:10:05 -0500 (EST) Received: from conversion-daemon.bur-mail2.east.sun.com by bur-mail2.east.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IWJ00401JNR9M@bur-mail2.east.sun.com> (original mail from mark.johnson@sun.com) for psarc@sun.com; Wed, 22 Mar 2006 13:10:05 -0500 (EST) Received: from [192.9.61.47] (punchin-mrj.SFBay.Sun.COM [192.9.61.47]) by bur-mail2.east.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with ESMTPA id <0IWJ00ESMJSR05@bur-mail2.east.sun.com>; Wed, 22 Mar 2006 13:10:05 -0500 (EST) Date: Wed, 22 Mar 2006 13:09:44 -0500 From: Mark Johnson Subject: Re: PSARC 2006/188 Memory Cache Disable Support on x86 In-reply-to: <44218545.1030805@Sun.COM> To: Eric.Lowe@sun.com Cc: Shudong Zhou , psarc@sun.com, Eiji.Ota@sun.com Message-id: <442192E8.3000101@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.1.2.240295 References: <200603220102.k2M12iAH107448@billybob.sfbay.sun.com> <44218545.1030805@Sun.COM> User-Agent: Thunderbird 1.5 (X11/20060113) Status: RO Content-Length: 24004 Eric Lowe wrote: > One concern I have is that this proposal by itself doesn't fill all of > the gaps in the possible values, leaving room for interpretation (still) > by driver authors who have a clear set of intended semantics in mind. It > leaves the door open for us to end up back here again wanting to add yet > another flag down the road. This seems like a good "fix" opportunity. > > Another concern is around whether or not non-cached implies combining > (merging) or not. > > I'm intentionally ignoring constraints of the implementation, and am > assuming the implementation will implement the least common denominator > which provides the requested semantics. > > Sigh -- it's unfortunate the interface doesn't use flags but instead > forces us to choose one value. This makes things hard. :( More on that > thought later. I believe that the interpretation can be fixed by providing documentation of how's these attributes map to actual H/W bits on the different types of hardware. In the end, that's what non Sun folks understand... I was thinking something off the the Sun website and in a blog more than in a man page. > As I see it there are four things I might want to specify as a driver > author: required ordering, whether load caching is OK, whether store > caching is OK, and whether combining (merging) is allowable for my > device data stream. > > I should be able to tell the interface what I want, unambiguously, and > have it happen for me; furthermore, I want to accomplish this without > having to decrypt convoluted flags and glean details of the > implementation (as is certainly done now -- or else folks just give up > and go with the least common denominator / most restrictive option, > costing performance). > > Ordering: the DDI appears to support relaxed or strict ordering. The > default is documented as strict ordering but relaxed is implied in all > caching modes. > Store caching: can be on or off. The default is documented to be off. > Load caching: can be on or off. The default is documented to be off. > Combining: can be on or off. The default is documented to be on. > > Enumerating the combinations: > > SO|NOSTORECACHE|NOLOADCACHE|??? -- if I understand your proposal > correctly this is your proposed flag. However you don't specify whether > combining is allowable. I believe you should specify. E.g. do you intend > to enable write combine in the PTE without the cache enable bit? > Leaving this as an implementation detail is, IMO, a big mistake, and > follows a pattern of past mistakes made in this interface. :) > RMO|NOSTORECACHE|NOLOADCACHE|NOCOMBINE -- this is DDI_UNORDERED_OK_ACC > RMO|NOSTORECACHE|NOLOADCACHE|COMBINE -- this is DDI_MERGING_OK_ACC > RMO|NOSTORECACHE|LOADCACHE|COMBINE -- this is DDI_LOADCACHING_OK_ACC > RMO|STORECACHE|LOADCACHE|COMBINE -- this is DDI_STORECACHING_OK_ACC > > Of course there are still many other combinations which aren't implemented. I agree this is cleaner. But I don't think that is the correct solution *at this point in time* for the following reasons. As you know, HAT doesn't work that way. It also uses these as enumerated types, not flags.. So the implementation won't be able to do anything with these flags.. When it comes down to real H/W bits, these concepts don't map 1-to-1 into real functionality. When it comes down to it most of this is NOP code. On x86, there are three possibilities from what I can tell.. don't set anything, set PT_NOCACHE, or set (PT_NOCACHE and PT_WRITETHRU). where: DDI_MERGING_OK_ACC => PT_NOCACHE and PT_WRITETHRU DDI_CACHEDISABLE_ACC => PT_NOCACHE all others map to do nothing. Currently on x86 all map to do nothing. For SPARC, the only bit is SFMMU_SIDEFFECT which is only used for mapping PIO space (i.e. write combining). So that's a lot of change for no extra functionality.. IMO, the correct long term solution is to replace ddi_dma_mem_alloc() with a different IO allocation routine. First, ddi_device_acc_attr_t should not be used to pass in the memory cache attributes :-). Secondly, the ddi_dma_attr_t structure used to specify both memory and bind constraints is confusing. Most of it doesn't apply to memory allocation. And lastly, a DIP should not be required to allocate memory for DMA. This has caused problems for stacks which have multiple HBA's, etc.. MRJ > I'm groaning because DDI_STRICTORDER_ACC doesn't say anything about > caching. We've left a lot of room for the implementation there, and that > is IMO not a good thing. I don't even know where it fits in the above in > the current implementation but in terms of architecture it must be > defacto SO|LOADCACHE|STORECACHE|COMBINE which is least restrictive? > > Some more enumerations may make sense. Others may not. For instance I > can certainly envision a use for strict-ordered versions of the various > combinations of caching on platforms which provide it. On the other > hand, I would expect caching of any sort to imply combining since the > cache line widths are almost certainly not going to match the device > access size. > > Maybe I'm trying to impose a developer tax where I shouldn't, and adding > just one more flag is the right thing, but at a minimum I'd like to see > another case brought forward (and soon) which attempts to clarify once > and for all what the expected behavior is with all of the various and > sundry .*_ACC flags, and preferably introduce a better architecture. As > it is today, the semantics are not clear, and worse the architecture > gives way too much leeway to the implementation -- which in this case is > a bad thing since it can lead to assumptions in behavior and resultant > portability problems. > > IMO the following changes to the architecture would solve your problems > and mine with one stone: > > - Rev the device attr structure to DDI_DEVICE_ATTR_V1 > - Keep data order so you can specify strict or relaxed ordering; > combining also logically goes there. > - Add a new caching attribute which allows specification of no caching, > load caching only, store caching only, or load and store caching being OK. > > The driver which spurred this case would use the DDI_DEVICE_ATTR_V1 to > get its non-cached memory, while everybody else could safely stay at _V0 > for the time being. This also eliminates the risks associated with > changing semantics of existing flags whose meaning is not clear (e.g. > DDI_STRICTORDER_ACC). > > - Eric > > Shudong Zhou wrote: >> I'm sponsoring the following fasttrack for Eiji Ota. The timer is >> set to expire on 3/28/2006. The requested release binding is patch. >> >> Shudong >> >> Template Version: @(#)onepager.txt 1.29 04/11/15 SMI >> >> Copyright 2006 Sun Microsystems, Inc. >> >> 1. Introduction >> 1.1. Project/Component Working Name: >> Allocating memory from the DDI with Cache Disable Support on x86 >> >> 1.2. Name of Document Author/Supplier: >> Eiji Ota (eiji.ota@sun.com) >> Mark Johnson (mark.johnson@sun.com) >> >> 1.3. Date of This Document: >> 03/03/06 >> >> 1.4. Name of Major Document Customer(s)/Consumer(s): >> 1.4.1. The PAC or CPT you expect to review your project: >> Solaris PAC >> >> 1.4.2. The ARC(s) you expect to review your project: >> PSARC >> 1.4.3. The Director/VP who is "Sponsoring" this project: >> Barry Cooks (barry.cooks@sun.com) >> >> 1.4.4. The name of your business unit: >> OPG, Solaris Core Technologies >> >> 1.5. Email Aliases: >> 1.5.1. Responsible Manager: peter.luk@sun.com >> 1.5.2. Responsible Engineer: eiji.ota@sun.com, >> mark.johnson@sun.com >> 1.5.3. Marketing Manager: chris.ratcliffe@sun.com >> 1.5.4. Interest List: iom-interest@Sun.Com >> >> 2. Project Summary >> >> 2.1. Project Description: >> The project provides a method to access uncacheable memory, >> which is >> required for some graphics drivers running on X86/AMD64 >> architecture. >> >> 2.2. Risks and Assumptions: >> None that we know of. >> >> 3. Business Summary >> >> 3.1. Problem Area: >> Some graphics drivers on X86/AMD64 require a method to have an >> access to memory with the uncacheable attribute to keep coherency >> between a graphic chip and the rest on the system, but Solaris >> doesn't provide any interfaces those drivers can use. As a >> result, >> they have to implement how to access the uncacheable memory in >> their >> own ways. This could be seen inconvenient by Solaris driver >> developers >> and inconsistent by Solaris kernel developers. >> >> 3.2. Market/Requester: >> Graphic device drivers running on X86 architecture. >> >> 3.3. Business Justification: >> Uncacheable memory access is required for some graphics chips >> on X86/AMD64 architecture these days. However, Solaris doesn't >> provide the DDI interface for that purpose yet. This project >> provides >> the unified and consistent method to support those graphics >> drivers >> running on X86/AMD64. >> >> 3.4. Competitive Analysis: >> Major operating systems running on X86/AMD64 (i.e. Linux/Windows) >> support similar functions already. >> >> 4. Technical Description: >> >> 4.1. Details: >> A new flag, DDI_CACHEDISABLE_ACC, is introduced to provide >> the uncacheable memory access. Though we have >> DDI_STRICTORDER_ACC to >> describe the strict ordering, if this were used for the >> uncacheable >> memory access, it could cause performance degradation since >> DDI_STRICTORDER_ACC is used by most drivers today. >> >> The flag should be set to devacc_attr_dataorder in struct >> ddi_device_acc_attr when a driver wants to access memory with the >> uncacheable attribute. This flag is referred to by >> devmap_umem_setup(), >> devmap_umem_remap() and ddi_dma_mem_alloc(). >> >> devmap_umem_setup() and devmap_umem_remap() are used to export >> kernel >> memory to the userland. The new flag is used to allow an >> application >> to have an access to the uncacheable memory by mmap(). >> >> ddi_dma_mem_alloc() is used to allocate non-pageable kernel >> memory for >> a driver. The new flag is used to provide a driver with an >> access to the uncacheable memory in the same manner. >> The gfx_private module is also updated to implement the >> uncacheable >> memory access by this new flag. >> >> 4.2. Bug/RFE Number(s): >> 6212057 ddi_dma_mem_alloc should support DDI_STRICTORDER_ACC >> in x86 >> >> 4.3. Interfaces: >> >> +-------------------------+---------------------+------------------+ >> | Interface Name | Classification | >> Comments | >> >> +-------------------------+---------------------+------------------+ >> | DDI_CACHEDISABLE_ACC | Evolving | >> -------- | >> >> +-------------------------+---------------------+------------------+ >> >> The following example shows how to use this flag for the >> uncacheable >> memory access; a driver doesn't need any other things than >> just setting >> DDI_CACHEDISABLE_ACC to devacc_attr_dataorder as follows. >> >> struct ddi_device_acc_attr XXX_acc_attr = { >> DDI_DEVICE_ATTR_V0, >> DDI_NEVERSWAP_ACC, >> DDI_CACHEDISABLE_ACC /* use the uncacheable memory */ >> }; >> >> 4.6. Doc Impact: >> Man page (ddi_device_acc_attr(9S)). >> The difference is attached below. (the change has bars on the >> left) >> >> 4.7. Admin/Config Impact: >> No impact on Admin/Config. >> 4.8. HA Impact: >> No impact on HA. >> 4.9. I18N/L10N Impact: >> No impact on I18N/L10L >> 4.10. Packaging & Delivery: >> No impact on packages, clusters and metaclusters. >> No impact on install/upgrade. >> 4.11. Security Impact: >> No impact on Security. >> 4.12. Dependencies: >> None. >> >> 5. Reference Documents: >> PSARC 2004/151 AGPgart project >> PSARC 2005/475 Graphics private misc module for x86 >> PSARC 2005/623 gfx_private update for devmap and DMA >> >> 6. Resources and Schedule: >> >> 6.1. Projected Availability: >> April/2006 >> >> 6.4. Product Approval Committee requested information: >> 6.4.1. Consolidation or Component Name: >> ON >> >> 6.4.3. Type of CPT Review and Approval expected: >> FastTrack >> >> 6.4.7. Target RTI Date/Release: >> onnv_b38 >> >> 6.4.8. Target Code Design Review Date: >> 3/13 >> >> 6.5. ARC review type: >> FastTrack >> >> >> >> (ddi_device_acc_attr(9S)) >> >> Data Structures for Drivers ddi_device_acc_attr(9S) >> >> NAME >> ddi_device_acc_attr - data access attributes structure >> >> SYNOPSIS >> #include >> #include >> >> INTERFACE LEVEL >> Solaris DDI specific (Solaris DDI). >> >> DESCRIPTION >> The ddi_device_acc_attr structure describes the data access >> characteristics and requirements of the device. >> >> STRUCTURE MEMBERS >> ushort_t devacc_attr_version; >> uchar_t devacc_attr_endian_flags; >> uchar_t devacc_attr_dataorder; >> >> The devacc_attr_version member identifies the version number >> of this structure. The current version number is >> DDI_DEVICE_ATTR_V0. >> >> The devacc_attr_endian_flags member describes the endian >> characteristics of the device. Specify one of the following >> values: >> >> DDI_NEVERSWAP_ACC Data access with no byte >> swapping >> >> DDI_STRUCTURE_BE_ACC Structural data access in >> big-endian format >> >> DDI_STRUCTURE_LE_ACC Structural data access in >> little endian format >> >> DDI_STRUCTURE_BE_ACC and DDI_STRUCTURE_LE_ACC describes the >> endian characteristics of the device as big-endian or >> little-endian, respectively. Though most of the devices will >> have the same endian characteristics as their buses, exam- >> ples of devices that have opposite endian characteristics of >> the buses do exist. When DDI_STRUCTURE_BE_ACC or >> DDI_STRUCTURE_LE_ACC is set, byte swapping is automatically >> performed by the system if the host machine and the device >> >> data formats have opposite endian characteristics. The >> implementation can take advantage of hardware platform byte >> swapping capabilities. >> >> When you specify DDI_NEVERSWAP_ACC, byte swapping is not >> invoked in the data access functions. >> >> The devacc_attr_dataorder member describes order in which >> the CPU will reference data. Specify one of the following >> values. >> >> DDI_STRICTORDER_ACC The data references must be >> issued by a CPU in program >> order. Strict ordering is >> the default behavior. >> >> DDI_UNORDERED_OK_ACC The CPU can re-order the >> data references. This >> includes all kinds of re- >> ordering. For example, a >> load followed by a store may >> be replaced by a store fol- >> lowed by a load. >> >> DDI_MERGING_OK_ACC The CPU can merge indivi- >> dual stores to consecutive >> locations. For example, the >> CPU can turn two consecutive >> byte stores into one half- >> word store. It can also >> batch individual loads. For >> example, the CPU might turn >> two consecutive byte loads >> into one halfword load. >> DDI_MERGING_OK_ACC also >> implies re-ordering. >> >> DDI_LOADCACHING_OK_ACC The CPU can cache the data >> it fetches and reuse it >> until another store occurs. >> The default behavior is to >> fetch new data on every >> load. DDI_LOADCACHING_OK_ACC >> also implies merging and >> re-ordering. >> >> DDI_STORECACHING_OK_ACC The CPU can keep the data >> in the cache and push it to >> the device (perhaps with >> other data) at a later time. >> The default behavior is to >> push the data right away. >> DDI_STORECACHING_OK_ACC also >> implies load caching, merg- >> ing, and re-ordering. >> >> | DDI_CACHEDISABLE_ACC The CPU can keep the data >> | from the cache and write the >> | data to memory without it. >> | The default behavior is to >> | push the data right away. >> | DDI_CACHEDISABLE_ACC also >> | implies strict ordering. >> >> These values are advisory, not mandatory. For example, data >> can be ordered without being merged or cached, even though a >> driver requests unordered, merged, and cached together. >> >> EXAMPLES >> The following examples illustrate the use of device register >> address mapping setup functions and different data access >> functions. >> >> Example 1: Using ddi_device_acc_attr() in >> ddi_regs_map_setup(9F) >> >> This example demonstrates the use of the >> ddi_device_acc_attr() structure in ddi_regs_map_setup(9F). >> It also shows the use of ddi_getw(9F) and ddi_putw(9F) >> functions in accessing the register contents. >> >> dev_info_t *dip; >> uint_t rnumber; >> ushort_t *dev_addr; >> offset_t offset; >> offset_t len; >> ushort_t dev_command; >> ddi_device_acc_attr_t dev_attr; >> ddi_acc_handle_t handle; >> >> ... >> >> /* >> * setup the device attribute structure for little endian, >> * strict ordering and 16-bit word access. >> */ >> dev_attr.devacc_attr_version = DDI_DEVICE_ATTR_V0; >> dev_attr.devacc_attr_endian_flags = DDI_STRUCTURE_LE_ACC; >> dev_attr.devacc_attr_dataorder = DDI_STRICTORDER_ACC; >> >> /* >> * set up the device registers address mapping >> */ >> >> ddi_regs_map_setup(dip, rnumber, (caddr_t *)&dev_addr, offset, len, >> &dev_attr, &handle); >> >> /* read a 16-bit word command register from the device */ >> dev_command = ddi_getw(handle, dev_addr); >> >> dev_command |= DEV_INTR_ENABLE; >> /* store a new value back to the device command register */ >> ddi_putw(handle, dev_addr, dev_command); >> >> Example 2: Accessing a Device with Different Apertures >> >> The following example illustrates the steps used to access a >> device with different apertures. Several apertures are >> assumed to be grouped under one single "reg" entry. For >> example, the sample device has four different apertures, >> each 32 Kbyte in size. The apertures represent YUV little- >> endian, YUV big-endian, RGB little-endian, and RGB big- >> endian. This sample device uses entry 1 of the "reg" pro- >> perty list for this purpose. The size of the address space >> is 128 Kbyte with each 32 Kbyte range as a separate aper- >> ture. In the register mapping setup function, the sample >> driver uses the offset and len parameters to specify one >> of the apertures. >> >> ulong_t *dev_addr; >> ddi_device_acc_attr_t dev_attr; >> ddi_acc_handle_t handle; >> uchar_t buf[256]; >> >> ... >> >> /* >> * setup the device attribute structure for never swap, >> * unordered and 32-bit word access. >> */ >> dev_attr.devacc_attr_version = DDI_DEVICE_ATTR_V0; >> dev_attr.devacc_attr_endian_flags = DDI_NEVERSWAP_ACC; >> dev_attr.devacc_attr_dataorder = DDI_UNORDERED_OK_ACC; >> >> /* >> * map in the RGB big-endian aperture >> * while running in a big endian machine >> * - offset 96K and len 32K >> */ >> ddi_regs_map_setup(dip, 1, (caddr_t *)&dev_addr, 96*1024, 32*1024, >> &dev_attr, &handle); >> >> /* >> * Write to the screen buffer >> * first 1K bytes words, each size 4 bytes >> */ >> ddi_rep_putl(handle, buf, dev_addr, 256, DDI_DEV_AUTOINCR); >> >> Example 3: Functions That Call Out the Data Word Size >> >> The following example illustrates the use of the functions >> that explicitly call out the data word size to override the >> data size in the device attribute structure. >> >> struct device_blk { >> ushort_t d_command; /* command register */ >> ushort_t d_status; /* status register */ >> ulong d_data; /* data register */ >> } *dev_blkp; >> dev_info_t *dip; >> caddr_t dev_addr; >> ddi_device_acc_attr_t dev_attr; >> ddi_acc_handle_t handle; >> uchar_t buf[256]; >> >> ... >> >> /* >> * setup the device attribute structure for never swap, >> * strict ordering and 32-bit word access. >> */ >> dev_attr.devacc_attr_version = DDI_DEVICE_ATTR_V0; >> dev_attr.devacc_attr_endian_flags = DDI_NEVERSWAP_ACC; >> dev_attr.devacc_attr_dataorder= DDI_STRICTORDER_ACC; >> >> ddi_regs_map_setup(dip, 1, (caddr_t *)&dev_blkp, 0, 0, >> &dev_attr, &handle); >> >> /* write command to the 16-bit command register */ >> ddi_putw(handle, &dev_blkp->d_command, START_XFER); >> >> /* Read the 16-bit status register */ >> status = ddi_getw(handle, &dev_blkp->d_status); >> >> if (status & DATA_READY) >> /* Read 1K bytes off the 32-bit data register */ >> ddi_rep_getl(handle, buf, &dev_blkp->d_data, >> 256, DDI_DEV_NO_AUTOINCR); >> >> SEE ALSO >> ddi_getw(9F), ddi_putw(9F), ddi_regs_map_setup(9F) >> >> Writing Device Drivers >> >> > -- Mark Johnson Sun Microsystems, Inc. (781) 442-0869 From sacadmin Wed Mar 22 11:19:28 2006 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2MJJRIQ028778 for ; Wed, 22 Mar 2006 11:19:28 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id k2MJIwPo004071; Thu, 23 Mar 2006 03:19:18 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IWJ00E07N02H400@brm-avmta-1.central.sun.com>; Wed, 22 Mar 2006 12:19:14 -0700 (MST) Received: from jurassic.eng.sun.com ([129.146.17.57]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IWJ00DTYN02G410@brm-avmta-1.central.sun.com>; Wed, 22 Mar 2006 12:19:14 -0700 (MST) Received: from hawaiian-sun (vpn-129-150-12-95.SFBay.Sun.COM [129.150.12.95]) by jurassic.eng.sun.com (8.13.6+Sun/8.13.6) with SMTP id k2MJJ8O2925407; Wed, 22 Mar 2006 11:19:13 -0800 (PST) Date: Wed, 22 Mar 2006 09:17:36 -1000 (HST) From: Joseph Kowalski Subject: Re: 2006/188 Memory Cache Disable Support on x86 To: mark.johnson@sun.com, glenn@ivrel.sfbay.sun.com Cc: Eiji.Ota@sun.com, psarc@sun.com Reply-to: Joseph Kowalski Message-id: <200603221919.k2MJJ8O2925407@jurassic.eng.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_20 SunOS 5.11 i86pc i386 Content-type: TEXT/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-MD5: MHmzNC2h72JMbWCNU3xprg== X-PMX-Version: 5.1.2.240295 Status: RO Content-Length: 1985 > From: Glenn Skinner ... > Date: Wed, 22 Mar 2006 12:34:29 -0500 > From: Bill Sommerfeld > Subject: Re: PSARC 2006/188 Memory Cache Disable Support on x86 > > So, I like Eric's suggestion of revving the interface to add an > orthogonal caching attribute but there's still the quesition about > how to fix the documentation to close the gap between it and the > current/traditional behavior of this interface.. > > I like the suggestion as well. As I see them, the high level > requirements are that a driver writer be able to state the desired > memory properties up front, and then be able to determine what the > resulting properties actually turned out to be. Given that, it's > possible to write a robust driver; without it, it isn't (without > relying on implicit assumptions). > > -- Glenn I want to abstract this discussion. This also enables: I want model "foo". Humm, I didn't get it. Well, model "bar" isn't quite as optimal, I'll ask for it. Got it! Drive on. It may even be the case that other bits of the driver code will need to respond to what model they actually got. Hence, I wonder about the appropriateness of weak semantics for the requests. I can see both sides of this. I don't wonder about weak semantics for "what you got". These need to be very exact and precise. Hence, there could be two forms of the above: I want precise model "foo". I want something like "foo" Didn't get it (fail). Got precise model "oof", can't use that! Try precise model "bar". I want something like "bar". Got it! Drive on. Got precise model "bar"! Drive on. I think its easer to code to the left column. This makes these not suggestions to the memory allocator, but explicit commands. (Because I don't want to write the code to determine if I can use what I actually got. Its easier for me to enumerate and order the precise models I can use.) - jek3 From sacadmin Wed Mar 22 11:29:13 2006 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2MJTCIQ028958 for ; Wed, 22 Mar 2006 11:29:12 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id k2MJTA7m009806 for <@sunmail1brm.central.sun.com:psarc@Sun.COM>; Thu, 23 Mar 2006 03:29:11 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IWJ00E03NGJU200@brm-avmta-1.central.sun.com> for psarc@Sun.COM (ORCPT psarc@sun.com); Wed, 22 Mar 2006 12:29:07 -0700 (MST) Received: from centralmail1brm.Central.Sun.COM ([129.147.62.1]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IWJ00DGJNGIFN20@brm-avmta-1.central.sun.com> for psarc@Sun.COM (ORCPT psarc@sun.com); Wed, 22 Mar 2006 12:29:07 -0700 (MST) Received: from phys-aus08-1 (phys-aus08-1.Central.Sun.COM [129.153.131.88]) by centralmail1brm.Central.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k2MJT6ca018774 for ; Wed, 22 Mar 2006 12:29:06 -0700 (MST) Received: from conversion-daemon.aus08-mail1.central.sun.com by aus08-mail1.central.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IWJ00901MZLEF@aus08-mail1.central.sun.com> (original mail from Eric.Lowe@Sun.COM) for psarc@sun.com; Wed, 22 Mar 2006 13:29:06 -0600 (CST) Received: from [192.9.61.227] (punchin-elowe.SFBay.Sun.COM [192.9.61.227]) by aus08-mail1.central.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with ESMTP id <0IWJ00JE7NGGGD@aus08-mail1.central.sun.com>; Wed, 22 Mar 2006 13:29:06 -0600 (CST) Date: Wed, 22 Mar 2006 13:30:32 -0600 From: Eric Lowe Subject: Re: PSARC 2006/188 Memory Cache Disable Support on x86 In-reply-to: <442192E8.3000101@sun.com> To: Mark Johnson Cc: Shudong Zhou , psarc@Sun.COM, Eiji.Ota@Sun.COM Reply-to: Eric.Lowe@Sun.COM Message-id: <4421A5D8.2090202@Sun.COM> Organization: Sun Microsystems MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.1.2.240295 References: <200603220102.k2M12iAH107448@billybob.sfbay.sun.com> <44218545.1030805@Sun.COM> <442192E8.3000101@sun.com> User-Agent: Thunderbird 1.5 (X11/20060113) Status: RO Content-Length: 2370 Mark Johnson wrote: > I believe that the interpretation can be fixed by providing > documentation of how's these attributes map to actual H/W bits > on the different types of hardware. In the end, that's what > non Sun folks understand... > > I was thinking something off the the Sun website and in a > blog more than in a man page. Yes. Documenting how each combination of attributes maps to hardware is probably a good idea (what you'll get), though better descriptions in the man page of the implied semantics of each flag (what you want) won't hurt either. [...] >> Of course there are still many other combinations which aren't >> implemented. > > I agree this is cleaner. But I don't think that is the correct > solution *at this point in time* for the following reasons. > > As you know, HAT doesn't work that way. It also uses these as > enumerated types, not flags.. So the implementation won't be > able to do anything with these flags.. That's an implementation detail. You can implement the architectural change and keep the implementation more-or-less the same as what you proposed by adding a big switch statement to convert the attributes or flags from the interface into the enumerations supported by the HAT if it's important to the task at hand. I just hate to see a broken interface get more broken. > When it comes down to real H/W bits, these concepts don't map > 1-to-1 into real functionality. When it comes down to it > most of this is NOP code. Understood. > So that's a lot of change for no extra functionality.. But it does completely remove the guesswork out of using the interface, and provides plenty of room for future expansion without having to change the architecture (instead you just plug in the new implementation). > IMO, the correct long term solution is to replace > ddi_dma_mem_alloc() with a different IO allocation routine. > First, ddi_device_acc_attr_t should not be used to > pass in the memory cache attributes :-). Secondly, the > ddi_dma_attr_t structure used to specify both memory and > bind constraints is confusing. Most of it doesn't apply to > memory allocation. And lastly, a DIP should not be required > to allocate memory for DMA. This has caused problems for > stacks which have multiple HBA's, etc.. Sure. In the mean time why not try to incrementally repair some of the damage? - Eric From sacadmin Wed Mar 22 11:32:05 2006 Received: from sunmail2.sfbay.sun.com (sunmail2.SFBay.Sun.COM [129.149.246.180]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2MJW5IQ028996 for ; Wed, 22 Mar 2006 11:32:05 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.149.247.22]) by sunmail2.sfbay.sun.com (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k2MJVxv20394; Wed, 22 Mar 2006 11:31:59 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IWJ00L1TNL9K800@nwk-avmta-2.sfbay.sun.com>; Wed, 22 Mar 2006 11:31:57 -0800 (PST) Received: from jurassic.eng.sun.com ([129.146.104.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IWJ00JZONL97530@nwk-avmta-2.sfbay.sun.com>; Wed, 22 Mar 2006 11:31:57 -0800 (PST) Received: from hawaiian-sun (vpn-129-150-12-95.SFBay.Sun.COM [129.150.12.95]) by jurassic.eng.sun.com (8.13.6+Sun/8.13.6) with SMTP id k2MJVmbO930463; Wed, 22 Mar 2006 11:31:55 -0800 (PST) Date: Wed, 22 Mar 2006 09:30:19 -1000 (HST) From: Joseph Kowalski Subject: Re: 2006/188 Memory Cache Disable Support on x86 To: mark.johnson@sun.com, glenn@ivrel.sfbay.sun.com, Joseph.Kowalski@eng.sun.com Cc: eiji.ota@sun.com, psarc@sun.com Reply-to: Joseph Kowalski Message-id: <200603221931.k2MJVmbO930463@jurassic.eng.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_20 SunOS 5.11 i86pc i386 Content-type: TEXT/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-MD5: R9oG03AqGhVsIud8czexUQ== X-PMX-Version: 5.1.2.240295 Status: RO Content-Length: 737 Replying to my own mail again. Tisk, tisk, tisk, ... > From: Joseph Kowalski ... > Hence, I wonder about the appropriateness of weak semantics for the > requests. I can see both sides of this. I don't wonder about weak > semantics for "what you got". These need to be very exact and precise. I want to be clear that this is independent of the discussion as to if these specifications should be an enumeration of models or a list of attributes (well, mostly). Its a comment on the required precision of specification in either case. I said "well mostly", because I tend to think precision would be easier to obtain with a list of attributes, but that's not necessarily a overriding factor. - jek3 From sacadmin Wed Mar 22 11:35:08 2006 Received: from sunmail2.sfbay.sun.com (sunmail2.SFBay.Sun.COM [129.149.246.180]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2MJZ7IQ029037 for ; Wed, 22 Mar 2006 11:35:07 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.149.246.28]) by sunmail2.sfbay.sun.com (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k2MJZ4v22147; Wed, 22 Mar 2006 11:35:04 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2 (built Dec 2 2004)) id <0IWJ00301NQCNB00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 22 Mar 2006 11:35:00 -0800 (PST) Received: from jurassic.eng.sun.com ([129.146.58.166]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2 (built Dec 2 2004)) with ESMTP id <0IWJ002H4NQC0V30@nwk-avmta-1.sfbay.Sun.COM>; Wed, 22 Mar 2006 11:35:00 -0800 (PST) Received: from hawaiian-sun (vpn-129-150-12-95.SFBay.Sun.COM [129.150.12.95]) by jurassic.eng.sun.com (8.13.6+Sun/8.13.6) with SMTP id k2MJYtAL931855; Wed, 22 Mar 2006 11:34:59 -0800 (PST) Date: Wed, 22 Mar 2006 09:33:22 -1000 (HST) From: Joseph Kowalski Subject: Re: PSARC 2006/188 Memory Cache Disable Support on x86 To: mark.johnson@sun.com, Eric.Lowe@sun.com Cc: szhou@billybob.sfbay.sun.com, psarc@sun.com, eiji.ota@sun.com Reply-to: Joseph Kowalski Message-id: <200603221934.k2MJYtAL931855@jurassic.eng.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_20 SunOS 5.11 i86pc i386 Content-type: TEXT/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-MD5: 9ramJQ0hUNNjq6HMiJSMyg== X-PMX-Version: 5.1.2.240295 Status: RO Content-Length: 902 > From: Eric Lowe ... > Mark Johnson wrote: > > I believe that the interpretation can be fixed by providing > > documentation of how's these attributes map to actual H/W bits > > on the different types of hardware. In the end, that's what > > non Sun folks understand... > > > > I was thinking something off the the Sun website and in a > > blog more than in a man page. > > Yes. Documenting how each combination of attributes maps to hardware is > probably a good idea (what you'll get), though better descriptions in the > man page of the implied semantics of each flag (what you want) won't hurt > either. Uh, I don't think we can let anyone set themselves up as "Ask Jeeves" to address this. Sure, blogs are helpful to supplement and clarify explicit documention, but they can't replace it. (And the existance of such a blog is beyond the concern of the ARC.) - jek3 From sacadmin Wed Mar 22 11:39:27 2006 Received: from sunmail3.sfbay.sun.com (sunmail3.SFBay.Sun.COM [129.149.247.180]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2MJdRIQ029097 for ; Wed, 22 Mar 2006 11:39:27 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3.sfbay.sun.com (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k2MJdP517001 for <@sunmail1brm.central.sun.com:psarc@sun.com>; Wed, 22 Mar 2006 11:39:25 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IWJ00F0LNXJ6H00@brm-avmta-1.central.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Wed, 22 Mar 2006 12:39:19 -0700 (MST) Received: from sfbaymail1sca.SFBay.Sun.COM ([129.145.154.35]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IWJ00D0YNXIFT30@brm-avmta-1.central.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Wed, 22 Mar 2006 12:39:18 -0700 (MST) Received: from billybob.sfbay.sun.com (billybob.SFBay.Sun.COM [129.146.224.123]) by sfbaymail1sca.SFBay.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k2MJdHEZ024400; Wed, 22 Mar 2006 11:39:17 -0800 (PST) Received: from billybob (billybob [129.146.224.123]) by billybob.sfbay.sun.com (8.13.5+Sun/8.13.5) with SMTP id k2MJfJlS110783; Wed, 22 Mar 2006 11:41:19 -0800 (PST) Date: Wed, 22 Mar 2006 11:41:19 -0800 (PST) From: Shudong Zhou Subject: Re: 2006/188 Memory Cache Disable Support on x86 To: mark.johnson@sun.com, glenn@ivrel.sfbay.sun.com, Joseph.Kowalski@eng.sun.com Cc: eiji.ota@sun.com, psarc@sun.com Reply-to: Shudong Zhou Message-id: <200603221941.k2MJfJlS110783@billybob.sfbay.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_13 SunOS 5.11 i86pc i386 Content-type: TEXT/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-MD5: lXy7nKuA0h8kGk04HWlMCQ== X-PMX-Version: 5.1.2.240295 Status: RO Content-Length: 102 There is enough confusion here. I'll work with the project team to come up with a new spec. Shudong From sacadmin Wed Mar 22 11:40:31 2006 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2MJeUIQ029110 for ; Wed, 22 Mar 2006 11:40:30 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id k2MJeGGE016497 for <@sunmail1brm.central.sun.com:psarc@Sun.COM>; Thu, 23 Mar 2006 03:40:29 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IWJ00F03NZH9200@brm-avmta-1.central.sun.com> for psarc@Sun.COM (ORCPT psarc@sun.com); Wed, 22 Mar 2006 12:40:29 -0700 (MST) Received: from centralmail1brm.Central.Sun.COM ([129.147.62.1]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IWJ00DSWNZGFZ30@brm-avmta-1.central.sun.com> for psarc@Sun.COM (ORCPT psarc@sun.com); Wed, 22 Mar 2006 12:40:28 -0700 (MST) Received: from phys-aus08-1 (phys-aus08-1.Central.Sun.COM [129.153.131.88]) by centralmail1brm.Central.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k2MJeSca023902 for ; Wed, 22 Mar 2006 12:40:28 -0700 (MST) Received: from conversion-daemon.aus08-mail1.central.sun.com by aus08-mail1.central.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IWJ00A01NNCOU@aus08-mail1.central.sun.com> (original mail from Eric.Lowe@Sun.COM) for psarc@sun.com; Wed, 22 Mar 2006 13:40:28 -0600 (CST) Received: from [192.9.61.227] (punchin-elowe.SFBay.Sun.COM [192.9.61.227]) by aus08-mail1.central.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with ESMTP id <0IWJ00J24NZEGD@aus08-mail1.central.sun.com>; Wed, 22 Mar 2006 13:40:28 -0600 (CST) Date: Wed, 22 Mar 2006 13:41:54 -0600 From: Eric Lowe Subject: Re: PSARC 2006/188 Memory Cache Disable Support on x86 In-reply-to: <200603221934.k2MJYtAL931855@jurassic.eng.sun.com> To: Joseph Kowalski Cc: Mark.Johnson@sun.com, szhou@billybob.sfbay.sun.com, psarc@sun.com, Eiji.Ota@sun.com Reply-to: Eric.Lowe@sun.com Message-id: <4421A882.7030704@Sun.COM> Organization: Sun Microsystems MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.1.2.240295 References: <200603221934.k2MJYtAL931855@jurassic.eng.sun.com> User-Agent: Thunderbird 1.5 (X11/20060113) Status: RO Content-Length: 401 > Uh, I don't think we can let anyone set themselves up as "Ask Jeeves" to > address this. Sure, blogs are helpful to supplement and clarify explicit > documention, but they can't replace it. (And the existance of such a > blog is beyond the concern of the ARC.) I overlooked that detail. :) A table in the WDD guide which gives each flag and the resulting attributes would be excellent. - Eric From sacadmin Wed Mar 29 11:04:09 2006 Received: from sunmail2.sfbay.sun.com (sunmail2.SFBay.Sun.COM [129.149.246.180]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2TJ49IQ013183 for ; Wed, 29 Mar 2006 11:04:09 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2.sfbay.sun.com (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k2TJ44013117 for <@sunmail1brm.central.sun.com:psarc@sun.com>; Wed, 29 Mar 2006 11:04:04 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IWW00D0BKYOXK00@brm-avmta-1.central.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Wed, 29 Mar 2006 12:04:00 -0700 (MST) Received: from sfbaymail1sca.SFBay.Sun.COM ([129.145.154.35]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IWW00ARAKYJ3L50@brm-avmta-1.central.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Wed, 29 Mar 2006 12:03:56 -0700 (MST) Received: from billybob.sfbay.sun.com (billybob.SFBay.Sun.COM [129.146.224.123]) by sfbaymail1sca.SFBay.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k2TJ3sEZ027301; Wed, 29 Mar 2006 11:03:54 -0800 (PST) Received: from billybob (billybob [129.146.224.123]) by billybob.sfbay.sun.com (8.13.5+Sun/8.13.5) with SMTP id k2TJ6AqO110802; Wed, 29 Mar 2006 11:06:10 -0800 (PST) Date: Wed, 29 Mar 2006 11:06:10 -0800 (PST) From: Shudong Zhou Subject: Re: PSARC 2006/188 Memory Cache Disable Support on x86 To: psarc@sun.com Cc: eiji.ota@sun.com, mark.johnson@sun.com Reply-to: Shudong Zhou Message-id: <200603291906.k2TJ6AqO110802@billybob.sfbay.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_13 SunOS 5.11 i86pc i386 Content-type: TEXT/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-MD5: 7CrxZDJzvyp9YspWnfhlew== X-PMX-Version: 5.1.2.240295 Status: RO Content-Length: 25935 Here is the updated spec from the project team. It's also available as a file spec.txt in the case directory. The timer is extended to 4/4/2006. The main change is to avoid using ddi_device_acc_attr and add new bit fields to an existing flag for HAT attributes related to system memory. Shudong Template Version: @(#)onepager.txt 1.29 04/11/15 SMI This information is Copyright 2006 Sun Microsystems, Inc. 1. Introduction 1.1. Project/Component Working Name: Allocating memory from the DDI with Cache Disable Support on X86 1.2. Name of Document Author/Supplier: Eiji Ota (eiji.ota@sun.com) Mark Johnson (mark.johnson@sun.com) 1.3. Date of This Document: 03/22/06 1.4. Name of Major Document Customer(s)/Consumer(s): 1.4.1. The PAC or CPT you expect to review your project: Solaris PAC 1.4.2. The ARC(s) you expect to review your project: PSARC 1.4.3. The Director/VP who is "Sponsoring" this project: Barry Cooks (barry.cooks@sun.com) 1.4.4. The name of your business unit: OPG, Solaris Core Technologies 1.5. Email Aliases: 1.5.1. Responsible Manager: peter.luk@sun.com 1.5.2. Responsible Engineer: eiji.ota@sun.com, mark.johnson@sun.com 1.5.3. Marketing Manager: chris.ratcliffe@sun.com 1.5.4. Interest List: iom-interest@Sun.Com 2. Project Summary 2.1. Project Description: The project provides a method to access uncacheable memory, which is required for some graphics drivers running on X86/AMD64 architecture. 2.2. Risks and Assumptions: None that we know of. 3. Business Summary 3.1. Problem Area: Some graphics drivers on X86/AMD64 require a method to have access to memory with the uncacheable attribute to keep coherency between a graphic chip and the rest on the system, but Solaris doesn't provide any interfaces those drivers can use. As a result, they have to implement how to access the uncacheable memory in their own ways. This could be seen inconvenient by Solaris driver developers and inconsistent by Solaris kernel developers. 3.2. Market/Requester: Graphic device drivers running on X86 architecture. 3.3. Business Justification: Uncacheable memory access is required for some graphics chips on X86/AMD64 architecture these days. However, Solaris doesn't provide the DDI interface for that purpose yet. This project provides the unified and consistent method to support those graphics drivers running on X86/AMD64. 3.4. Competitive Analysis: Major operating systems running on X86/AMD64 (i.e. Linux/Windows) support similar functions already. 4. Technical Description: 4.1. Details: New cache attribute flags (See 4.3) are introduced to provide the uncacheable memory access via the DDI interface. Those flags should be passed to ddi_dma_mem_alloc(), devmap_umem_setup() or devmap_pmem_setup() when a driver wants to access memory with the uncacheable attribute. If no flags is specified, IOMEM_DATA_CACHED is used to keep compatibilty. ddi_dma_mem_alloc() is used to allocate non-pageable kernel memory for a driver. The new flags are used to provide a driver with an access to the uncacheable memory in the same manner. devmap_umem_setup() and devmap_umem_remap() are used to export kernel memory to the userland. The new flag are used to allow an application to have access to the uncacheable memory by mmap(). devmap_pmem_setup() and devmap_pmem_remap() are used to export kernel physical memory to graphics drivers. The new flags are used to provide a driver with access to the uncacheable memory in the same manner. The gfx_private module is also updated to implement the uncacheable memory access by the new flags. 4.2. Bug/RFE Number(s): 6212057 ddi_dma_mem_alloc should support DDI_STRICTORDER_ACC in x86 4.3. Interfaces: +-------------------------+----------------+------------------------+ | Interface Name | Classification | Comments | +-------------------------+----------------+------------------------+ | IOMEM_DATA_CACHED | Evolving | default value. | | | | | | IOMEM_DATA_UC_WR_COMBINE| Evolving | data is not cached, but| | | | writes may occur out of| | | | order or be combined. | | | | | | IOMEM_DATA_UNCACHED | Evolving | data is not cached. | | | | strict ordering. | +-------------------------+----------------+------------------------+ The following examples show how to use this flag for the uncacheable memory access; a driver doesn't need any other things than just setting IOMEM_DATA_UNCACHED to have access to ucacheable memory as follows. o ddi_dma_mem_alloc() if (ddi_dma_mem_alloc(dma_handle, size, &dev_attr, DDI_DMA_CONSISTENT | IOMEM_DATA_UNCACHED, DDI_DMA_SLEEP, NULL, (caddr_t *)&addr, &alloc_len, &accessp) != DDI_SUCCESS) { : } o devmap_umem_setup() if (devmap_umem_setup(dhp, dip, callbackops, cookie, offset, length, PROT_ALL, IOMEM_DATA_UNCACHED, &endian_attr) < 0) { : } o devmap_pmem_setup() if (devmap_pmem_setup(cookie, dip, &cb, cookie, offset, len, PROT_ALL, IOMEM_DATA_UNCACHED, &attr)) < 0) { : } 4.6. Doc Impact: Man pages (ddi_dma_mem_alloc(9F) and devmap_devmem_setup(9F)) The differences are attached below.(the changes have bars on the left) 4.7. Admin/Config Impact: No impact on Admin/Config. 4.8. HA Impact: No impact on HA. 4.9. I18N/L10N Impact: No impact on I18N/L10L 4.10. Packaging & Delivery: No impact on packages, clusters and metaclusters. No impact on install/upgrade. 4.11. Security Impact: No impact on Security. 4.12. Dependencies: None. 5. Reference Documents: PSARC 2004/151 AGPgart project PSARC 2005/475 Graphics private misc module for x86 PSARC 2005/623 gfx_private update for devmap and DMA 6. Resources and Schedule: 6.1. Projected Availability: April/2006 6.4. Product Approval Committee requested information: 6.4.1. Consolidation or Component Name: ON 6.4.3. Type of CPT Review and Approval expected: FastTrack 6.4.7. Target RTI Date/Release: onnv_b39 6.4.8. Target Code Design Review Date: 4/4 6.5. ARC review type: FastTrack (ddi_dma_mem_alloc()) Kernel Functions for Drivers ddi_dma_mem_alloc(9F) NAME ddi_dma_mem_alloc - allocate memory for DMA transfer SYNOPSIS #include #include int ddi_dma_mem_alloc(ddi_dma_handle_t handle, size_t length, ddi_device_acc_attr_t *accattrp, uint_t flags, int (*waitfp) (caddr_t), caddr_t arg, caddr_t *kaddrp, size_t *real_length, ddi_acc_handle_t *handlep); INTERFACE LEVEL Solaris DDI specific (Solaris DDI). PARAMETERS handle The DMA handle previously allocated by a call to ddi_dma_alloc_handle(9F). length The length in bytes of the desired alloca- tion. | accattrp Pointer to a ddi_device_acc_attr(9S) struc- | ture of this device (see ddi_device_acc_attr(9S)). | The value in devacc_attr_dataorder is ignored in | the current release. The value in | devacc_attr_endian_flags is meaningful on the sparc | architecture only. | flags Used to determine the data transfer mode and/or | the cache attribute. | | Possible values of the data tranfer mode are: | | DDI_DMA_STREAMING Sequential, uni- | directional, block- | sized, and block- | aligned transfers. | | DDI_DMA_CONSISTENT Nonsequential | transfers of small | objects. | | Possible values of the cache attribute are: | | IOMEM_DATA_CACHED The CPU can cache the data | it fetches and push it to memory | at a later time. This is the | default attribute and used if | no cache attributes is specified. | | IOMEM_DATA_UC_WR_COMBINE The CPU never caches the data | but writes may occur out of order | or be combined. It implies | re-ordering. | | IOMEM_DATA_UNCACHED The CPU never caches the data but | has uncacheable access to memory. | It also implies strict ordering. | | The cache attributes are mutually exclusive, and any | combination of the values leads to a failure. | On the sparc architecture, only IOMEM_DATA_CACHED is | meaningful, but others lead to a failure. waitfp The address of a function to call back later if resources are not available now. The callback function indicates how a caller wants to handle the possibility of resources not being available. If callback is set to DDI_DMA_DONTWAIT, the caller does not care if the allocation fails, and can handle an allocation failure appropriately. If call- back is set to DDI_DMA_SLEEP, the caller wishes to have the allocation routines wait for resources to become available. If any other value is set and a DMA resource allo- cation fails, this value is assumed to be the address of a function to be called when resources become available. When the speci- fied function is called, arg is passed to it as an argument. The specified callback func- tion must return either DDI_DMA_CALLBACK_RUNOUT or DDI_DMA_CALLBACK_DONE. DDI_DMA_CALLBACK_RUNOUT indicates that the callback function attempted to allocate DMA resources but failed. In this case, the callback function is put back on a list to be called again later. DDI_DMA_CALLBACK_DONE indicates that either the allocation of DMA resources was successful or the driver no longer wishes to retry. The callback func- tion is called in interrupt context. There- fore, only system functions accessible from interrupt context are available. The callback function must take whatever steps are necessary to protect its critical resources, data structures, queues, and so on. arg Argument to be passed to the callback func- tion, if such a function is specified. kaddrp On successful return, kaddrp points to the allocated memory. real_length The amount of memory, in bytes, allocated. Alignment and padding requirements may require ddi_dma_mem_alloc() to allocate more memory than requested in length. handlep Pointer to a data access handle. DESCRIPTION ddi_dma_mem_alloc() allocates memory for DMA transfers to or from a device. The allocation will obey the alignment, pad- ding constraints and device granularity as specified by the DMA attributes (see ddi_dma_attr(9S)) passed to ddi_dma_alloc_handle(9F) and the more restrictive attributes imposed by the system. flags should be set to DDI_DMA_STREAMING if the device is doing sequential, unidirectional, block-sized, and block- aligned transfers to or from memory. The alignment and pad- ding constraints specified by the minxfer and burstsizes fields in the DMA attribute structure, ddi_dma_attr(9S) (see ddi_dma_alloc_handle(9F)) will be used to allocate the most effective hardware support for large transfers. For example, if an I/O transfer can be sped up by using an I/O cache, which has a minimum transfer of one cache line, ddi_dma_mem_alloc() will align the memory at a cache line boundary and it will round up real_length to a multiple of the cache line size. flags should be set to DDI_DMA_CONSISTENT if the device accesses memory randomly, or if synchronization steps using ddi_dma_sync(9F) need to be as efficient as possible. I/O parameter blocks used for communication between a device and a driver should be allocated using DDI_DMA_CONSISTENT. The device access attributes are specified in the location pointed by the accattrp argument (see ddi_device_acc_attr(9S)). The data access handle is returned in handlep. handlep is opaque - drivers may not attempt to interpret its value. To access the data content, the driver must invoke ddi_get8(9F) or ddi_put8(9F) (depending on the data transfer direction) with the data access handle. DMA resources must be established before performing a DMA transfer by passing kaddrp and real_length as returned from ddi_dma_mem_alloc() and the flag DDI_DMA_STREAMING or DDI_DMA_CONSISTENT to ddi_dma_addr_bind_handle(9F). In addi- tion, to ensure the consistency of a memory object shared between the CPU and the device after a DMA transfer, expli- cit synchronization steps using ddi_dma_sync(9F) or ddi_dma_unbind_handle(9F) are required. RETURN VALUES ddi_dma_mem_alloc() returns: DDI_SUCCESS Memory successfully allocated. DDI_FAILURE Memory allocation failed. CONTEXT ddi_dma_mem_alloc() can be called from user or interrupt context, except when waitfp is set to DDI_DMA_SLEEP, in which case it can be called from user context only. SEE ALSO ddi_dma_addr_bind_handle(9F), ddi_dma_alloc_handle(9F), ddi_dma_mem_free(9F), ddi_dma_sync(9F), ddi_dma_unbind_handle(9F), ddi_get8(9F), ddi_put8(9F), ddi_device_acc_attr(9S), ddi_dma_attr(9S) Writing Device Drivers WARNINGS If DDI_NEVERSWAP_ACC is specified, memory can be used for any purpose; but if either endian mode is specified, you must use ddi_get/put* and never anything else. (devmap_umem_setup()) Kernel Functions for Drivers devmap_devmem_setup(9F) NAME devmap_devmem_setup, devmap_umem_setup - set driver memory mapping parameters SYNOPSIS #include #include int devmap_devmem_setup(devmap_cookie_t dhp, dev_info_t *dip, struct devmap_callback_ctl *callbackops, uint_t rnumber, offset_t roff, size_t len, uint_t maxprot, uint_t flags, ddi_device_acc_attr_t *accattrp); int devmap_umem_setup(devmap_cookie_t dhp, dev_info_t *dip, struct devmap_callback_ctl *callbackops, ddi_umem_cookie_t cookie, offset_t koff, size_t len, uint_t maxprot, uint_t flags, ddi_device_acc_attr_t *accattrp); INTERFACE LEVEL Solaris DDI specific (Solaris DDI). PARAMETERS devmap_devmem_setup() parameters: dhp An opaque mapping handle that the system uses to describe the mapping. dip Pointer to the device's dev_info structure. callbackops Pointer to a devmap_callback_ctl(9S) struc- ture. The structure contains pointers to device driver-supplied functions that manage events on the device mapping. The framework will copy the structure to the system private memory. rnumber Index number to the register address space set. roff Offset into the register address space. len Length (in bytes) of the mapping to be mapped. maxprot Maximum protection flag possible for attempted mapping. Some combinations of pos- sible settings are: PROT_READ Read access is allowed. PROT_WRITE Write access is allowed. PROT_EXEC Execute access is allowed. PROT_USER User-level access is allowed (the mapping is being done as a result of a mmap(2) system call). PROT_ALL All access is allowed. flags Must be set to 0. accattrp Pointer to a ddi_device_acc_attr(9S) struc- ture. The structure contains the device access attributes to be applied to this range of memory. devmap_umem_setup() parameters: dhp An opaque data structure that the system uses to describe the mapping. dip Pointer to the device's dev_info structure. callbackops Pointer to a devmap_callback_ctl(9S) struc- ture. The structure contains pointers to device driver-supplied functions that manage events on the device mapping. cookie A kernel memory cookie (see ddi_umem_alloc(9F)). koff Offset into the kernel memory defined by cookie. len Length (in bytes) of the mapping to be mapped. maxprot Maximum protection flag possible for attempted mapping. Some combinations of pos- sible settings are: PROT_READ Read access is allowed. PROT_WRITE Write access is allowed. PROT_EXEC Execute access is allowed. PROT_USER User-level access is allowed (the mapping is being done as a result of a mmap(2) system call). PROT_ALL All access is allowed. | flags Used to determine the cache attribute. | | Possible values of the cache attribute are: | | IOMEM_DATA_CACHED The CPU can cache the data | it fetches and push it to memory | at a later time. This is the | default attribute and used if | no cache attributes is specified. | | IOMEM_DATA_UC_WR_COMBINE The CPU never caches the data | but writes may occur out of order | or be combined. It implies | re-ordering. | | IOMEM_DATA_UNCACHED The CPU never caches the data but | has uncacheable access to memory. | It also implies strict ordering. | | The cache attributes are mutually exclusive, and any | combination of the values leads to a failure. | On the sparc architecture, only IOMEM_DATA_CACHED is | meaningful, but others lead to a failure. | accattrp Pointer to a ddi_device_acc_attr(9S) struc- | ture of this device (see ddi_device_acc_attr(9S)). | The value in devacc_attr_dataorder is ignored in | the current release. The value in | devacc_attr_endian_flags is meaningful on the sparc | architecture only. DESCRIPTION devmap_devmem_setup() and devmap_umem_setup() are used in the devmap(9E) entry point to pass mapping parameters from the driver to the system. dhp is a device mapping handle that the system uses to store all mapping parameters of a physical contiguous memory. The system copies the data pointed to by callbackops to a system private memory. This allows the driver to free the data after returning from either devmap_devmem_setup() or devmap_umem_setup(). The driver is notified of user events on the mappings via the entry points defined by devmap_callback_ctl(9S). The driver is notified of the fol- lowing user events: Mapping Setup User has called mmap(2) to create a mapping to the device memory. Access User has accessed an address in the mapping that has no translations. Duplication User has duplicated the mapping. Mappings are duplicated when the process calls fork(2). Unmapping User has called munmap(2) on the mapping or is exiting, exit(2). See devmap_map(9E), devmap_access(9E), devmap_dup(9E), and devmap_unmap(9E) for details on these entry points. By specifying a valid callbackops to the system, device drivers can manage events on a device mapping. For example, the devmap_access(9E) entry point allows the drivers to perform context switching by unloading the mappings of other processes and to load the mapping of the calling process. Device drivers may specify NULL to callbackops which means the drivers do not want to be notified by the system. The maximum protection allowed for the mapping is specified in maxprot. accattrp defines the device access attributes. See ddi_device_acc_attr(9S) for more details. devmap_devmem_setup() is used for device memory to map in the register set given by rnumber and the offset into the register address space given by roff. The system uses rnumber and roff to go up the device tree to get the physi- cal address that corresponds to roff. The range to be affected is defined by len and roff. The range from roff to roff + len must be a physical contiguous memory and page aligned. Drivers use devmap_umem_setup() for kernel memory to map in the kernel memory described by cookie and the offset into the kernel memory space given by koff. cookie is a kernel memory pointer obtained from ddi_umem_alloc(9F). If cookie is NULL, devmap_umem_setup() returns -1. The range to be affected is defined by len and koff. The range from koff to koff + len must be within the limits of the kernel memory described by koff + len and must be page aligned. Drivers use devmap_umem_setup() to export the kernel memory allocated by ddi_umem_alloc(9F) to user space. The system selects a user virtual address that is aligned with the ker- nel virtual address being mapped to avoid cache incoherence if the mapping is not MAP_FIXED. RETURN VALUES 0 Successful completion. -1 An error occurred. CONTEXT devmap_devmem_setup() and devmap_umem_setup() can be called from user, kernel, and interrupt context. SEE ALSO exit(2), fork(2), mmap(2), munmap(2), devmap(9E), ddi_umem_alloc(9F), ddi_device_acc_attr(9S), devmap_callback_ctl(9S) Writing Device Drivers From sacadmin Wed Mar 29 12:36:46 2006 Received: from sunmail2.sfbay.sun.com (sunmail2.SFBay.Sun.COM [129.149.246.180]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k2TKakIQ015396 for ; Wed, 29 Mar 2006 12:36:46 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2.sfbay.sun.com (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k2TKag024816 for <@sunmail1brm.central.sun.com:psarc@Sun.COM>; Wed, 29 Mar 2006 12:36:42 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IWW00H07P95E100@brm-avmta-1.central.sun.com> for psarc@Sun.COM (ORCPT psarc@sun.com); Wed, 29 Mar 2006 13:36:41 -0700 (MST) Received: from centralmail2brm.Central.Sun.COM ([129.147.62.14]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IWW00ARSP943Y90@brm-avmta-1.central.sun.com> for psarc@Sun.COM (ORCPT psarc@sun.com); Wed, 29 Mar 2006 13:36:40 -0700 (MST) Received: from phys-aus08-1 (phys-aus08-1.Central.Sun.COM [129.153.131.88]) by centralmail2brm.Central.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k2TKaeda024865 for ; Wed, 29 Mar 2006 13:36:40 -0700 (MST) Received: from conversion-daemon.aus08-mail1.central.sun.com by aus08-mail1.central.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IWW00C01P054X@aus08-mail1.central.sun.com> (original mail from Eric.Lowe@Sun.COM) for psarc@sun.com; Wed, 29 Mar 2006 14:36:39 -0600 (CST) Received: from [192.9.61.227] (punchin-elowe.SFBay.Sun.COM [192.9.61.227]) by aus08-mail1.central.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with ESMTP id <0IWW00235P91JI@aus08-mail1.central.sun.com>; Wed, 29 Mar 2006 14:36:38 -0600 (CST) Date: Wed, 29 Mar 2006 14:38:20 -0600 From: Eric Lowe Subject: Re: PSARC 2006/188 Memory Cache Disable Support on x86 In-reply-to: <200603291906.k2TJ6AqO110802@billybob.sfbay.sun.com> To: Shudong Zhou Cc: psarc@Sun.COM, Eiji.Ota@Sun.COM, Mark.Johnson@Sun.COM Reply-to: Eric.Lowe@Sun.COM Message-id: <442AF03C.3030902@Sun.COM> Organization: Sun Microsystems MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.1.2.240295 References: <200603291906.k2TJ6AqO110802@billybob.sfbay.sun.com> User-Agent: Thunderbird 1.5 (X11/20060113) Status: RO Content-Length: 41 I'm happy with the updated spec. - Eric From sacadmin Wed Apr 5 13:44:49 2006 Received: from sunmail3.sfbay.sun.com (sunmail3.SFBay.Sun.COM [129.149.247.180]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k35KinIQ001818 for ; Wed, 5 Apr 2006 13:44:49 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3.sfbay.sun.com (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id k35KimO14077 for <@sunmail1brm.central.sun.com:psarc@sun.com>; Wed, 5 Apr 2006 13:44:48 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IX900K0JOAOM800@brm-avmta-1.central.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Wed, 05 Apr 2006 14:44:48 -0600 (MDT) Received: from sfbaymail1sca.SFBay.Sun.COM ([129.145.154.35]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IX900INXOANBX30@brm-avmta-1.central.sun.com> for psarc@sun.com (ORCPT psarc@sun.com); Wed, 05 Apr 2006 14:44:47 -0600 (MDT) Received: from billybob.sfbay.sun.com (billybob.SFBay.Sun.COM [129.146.224.123]) by sfbaymail1sca.SFBay.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k35Kijoc027313; Wed, 05 Apr 2006 13:44:45 -0700 (PDT) Received: from billybob (billybob [129.146.224.123]) by billybob.sfbay.sun.com (8.13.5+Sun/8.13.5) with SMTP id k35KlGkC143096; Wed, 05 Apr 2006 13:47:16 -0700 (PDT) Date: Wed, 05 Apr 2006 13:47:16 -0700 (PDT) From: Shudong Zhou Subject: Re: PSARC 2006/188 Memory Cache Disable Support on x86 To: psarc@sun.com Cc: eiji.ota@sun.com, mark.johnson@sun.com Reply-to: Shudong Zhou Message-id: <200604052047.k35KlGkC143096@billybob.sfbay.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_13 SunOS 5.11 i86pc i386 Content-type: TEXT/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-MD5: wQ3ElQz5+NKcUHLTnk49uQ== X-PMX-Version: 5.1.2.240295 Status: RO Content-Length: 59 This case was approved in today's PSARC meeting. Shudong