PSARC Questions Version 4.6 1. What specifically is the proposal that we are reviewing? Advanced DDI Interrupt Framework - PSARC/2004/253 - What is the technical content of the project? http://dtsw.sfbay.sun.com/ddi/ddi-interrupts.txt - Is this a new product, or a change to a pre-existing one? If it is a change, would you consider it a "major", "minor", or "micro" change? This project constitutes a "minor" change to Solaris. - If your project is an evolution of a previous project, what changed from one version to another? N/A - What is the motivation for it, in general as well as specific terms? (Note that not everyone on the ARC will be an expert in the area.) Add support for new interrupt type Message Signaled Interrupts (MSI). A PCI or PCI-Express device can support upto 32 MSI interrupts and upto 2048 MSI-X (Extended MSI) interrupts. This spec. collectively refers to MSI and MSI-X interrupts as MSI/X henceforth. Add get/set capability, resource management and priority management interfaces to the new framework, making new bus features available to drivers that need them. Make the new framework generic enough to support other new (and unknown) interrupt types, where possible. - What are the expected benefits for Sun? The following new platforms support MSI and MSI-X hardware features. Ontario, Erie Niagara/Fire (SUN4V) based blade servers Chicago Serrano/Fire based SPARC work station Galaxy Opteron 1U/2U/4U servers Andromeda Opteron based blade server - By what criteria will you judge its success? New platforms (as mentioned in previous question) should work with MSI/X interrupts. No boot time performance regression. 2. Describe how your project changes the user experience, upon installation and during normal operation. There is no perceived change during normal operation. - What does the user perceive when the system is upgraded from a previous release? It will be transparent to users. 3. What is its plan? See below. - What is its current status? o Development: - Prototyped new DDI interrupt interfaces as per the proposal except the resource management interfaces. Regular hardware interrupts and soft interrupts have been verified to work with the prototype. Currently, we are looking into prototyping the resource management interfaces. Also, re-implemented current DDI interrupt interfaces as wrappers that use new DDI interfaces to maintain compatibility for existing DDI compliant drivers. This is also true for soft interrupts. - Updated few existing SPARC and x86 PCI device drivers to verify the new interfaces, the MSI interrupt delivery and soft interrupts. o Testing: - SPARC: Booted fireball system successfully (Niagara+Fire and Jalapeno+Fire, using Solaris PCI Express driver (Fire)) with a Solaris disk connected to PCI-X LSI SCSI-320 HBA and also verified PCI-X Broadcom BCM5704 NIC using MSIs. - X86: Booted Grizzly system successfully with a solaris disk connected to PCI-X Adaptec HBA using MSIs. Also verified PCI-X LSI SCSI-320 HBA and PCI-X Broadcom BCM5704 NIC using MSIs. Booted PCI-Express system successfully (Intel P4+ E7525 chipset) with a Solaris disk connected to PCI-X LSI SCSI-320 HBA using MSI interrupts. - Also, verified both existing and new interrupt interfaces for the fixed interrupts. This is also true for soft interrupts. Has a design review been done? The design has been reviewed by an extended team of experts within Sun (ddi-intr@sun.com). Are there multiple delivery phases? yes, The delivery will be in two phases Phase1 - All new DDI interrupt interfaces changes except Resource Management interfaces. Phase2 - Resource Management interfaces changes. Both phases are targeted for Solaris 11 and Solaris 10 update. 4. Are there related projects in Sun? No. Other current projects in the interrupt area are as follows PSARC/2004/199 intrd: Dynamic Interrupt Redistribution PSARC/2004/630 Switching dynamically between Polling & Interrupts These projects are not related to this project, but still we exchanged design specification and other materials with these project teams. There are no formal dependencies between these projects. - If so, what is the proposal's relationship to their work? Which not-yet- delivered Sun (or non-Sun) projects (libraries, hardware, etc.) does this project depend upon? What other projects, if any, depend on this one? N/A - Are you updating, copying or changing functional areas maintained by other groups? How are you coordinating and communicating with them? Do they "approve" of what you propose? If not, please explain the areas of disagreement. N/A 5. How is the project delivered into the system? With the Solaris WOS - Identify packages, directories, libraries, databases, etc. Core kernel packages: SUNWhea SUNWckr SUNWcakr.i SUNWcakr.u SUNWos86r No new packages will be introduced. 6. Describe the project's hardware platform dependencies. None. - Explain any reasons why it would not work on both SPARC and Intel? Framework is entirely common code with no hardware platform dependencies. 7. System administration - How will the project's deliverables be installed and (re)configured? With the Solaris WOS - How will the project's deliverables be uninstalled? With the Solaris WOS - Does it use inetd to start itself? N/A - Does it need installation within any global system tables? N/A - Does it use a naming service such as NIS, NIS+ or LDAP? N/A - What are its on-going maintenance requirements (e.g. Keeping global tables up to date, trimming files)? N/A - How does this project's administrative mechanisms fit into Sun's system administration strategies? E.g., how does it fit under the Solaris Management Console (SMC) and Web-Based Enterprise Management (WBEM), how does it make use of roles, authorizations and rights profiles? Additionally, how does it provide for administrative audit in support of the Solaris BSM configuration? N/A - What tunable parameters are exported? Can they be changed without rebooting the system? Examples include, but are not limited to, entries in /etc/system and ndd(8) parameters. What ranges are appropriate for each tunable? What are the commitment levels associated with each tunable (these are interfaces)? No new tunables will be introduced. 8. Reliability, Availability, Serviceability (RAS) - Does the project make any material improvement to RAS? No. - How can users/administrators diagnose failures or determine operational state? (For example, how could a user tell the difference between a failure and very slow performance?) No new kstat for interrupt will be introduced. - What are the project's effects on boot time requirements? None. - How does the project handle dynamic reconfiguration (DR) events? N/A - What mechanisms are provided for continuous availability of service? N/A - Does the project call panic()? Explain why these panics cannot be avoided. N/A - How are significant administrative or error conditions transmitted? SNMP traps? Email notification? N/A - How does the project deal with failure and recovery? N/A - Does it ever require reboot? If so, explain why this situation cannot be avoided. No. - How does your project deal with network failures (including partition and re- integration)? How do you handle the failure of hardware that your project depends on? N/A - Can it save/restore or checkpoint and recover? N/A - Can its files be corrupted by failures? Does it clean up any locks/files after crashes? N/A 9. Observability - Does the project export status, either via observable output (e.g., netstat) or via internal data structures (kstats)? yes, via kstats, mdb A new mdb dcmd will be added to report the interrupt usage of a live system. - How would a user or administrator tell that this subsystem is or is not behaving as anticipated? By monitoring the kstats. - What statistics does the subsystem export, and by what mechanism? N/A - What state information is logged? N/A - In principle, would it be possible for a program to tune the activity of your project? N/A 10. What are the security implications of this project? - What security issues do you address in your project? N/A - The Solaris BSM configuration carries a Common Criteria (CC) Controlled Access Protection Profile (CAPP) -- Orange Book C2 -- and a Role Based Access Control Protection Profile (RBAC) -- rating, does the addition of your project effect this rating? E.g., does it introduce interfaces that make access or privilege decisions that are not audited, does it introduce removable media support that is not managed by the allocate subsystem, does it provide administration mechanisms that are not audited? N/A - Is system or subsystem security compromised in any way if your project's configuration files are corrupt or missing? N/A - Please justify the introduction of any (all) new setuid executables. N/A - Include a thorough description of the security assumptions, capabilities and any potential risks (possible attack points) being introduced by your project. A separate Security Questionnaire http://sac.sfbay/cgi-bin/bp.cgi?NAME=Security.bp is provided for more detailed guidance on the necessary information. Cases are encouraged to fill out and include the Security questionnaire (leveraging references to existing documentation) in the case materials. Projects must highlight information for the following important areas: - What features are newly visible on the network and how are they protected from exploitation (e.g. unauthorized access, eavesdropping) N/A - If the project makes decisions about which users, hosts, services, ... are allowed to access resources it manages, how is the requestor's identity determined and what data is used to determine if the access granted. Also how this data is protected from tampering. N/A - What privileges beyond what a common user (e.g. 'noaccess') can perform does this project require and why those are necessary. N/A - What parts of the project are active upon default install and how it can be turned off. N/A 11. What is its UNIX operational environment: - Which Solaris release(s) does it run on? Solaris 11 (Nevada) upwards Solaris 10 Update - Environment variables? Exit status? Signals issued? Signals caught? (See signal(3HEAD).) N/A - Device drivers directly used (e.g. /dev/audio)? .rc/defaults or other resource/configuration files or databases? N/A - Does it use any "hidden" (filename begins with ".") or temp files? No. - Does it use any locking files? No. - Command line or calling syntax: What options are supported? (please include man pages if available) Does it conform to getopt() parsing requirements? N/A - Is there support for standard forms, e.g. "-display" for X programs? Are these propagated to sub-environments? N/A - What shared libraries does it use? (Hint: if you have code use "ldd" and "dump -Lv")? N/A - Identify and justify the requirement for any static libraries. N/A - Does it depend on kernel features not provided in your packages and not in the default kernel (e.g. Berkeley compatibility package, /usr/ccs, /usr/ucblib, optional kernel loadable modules)? No. - Is your project 64-bit clean/ready? If not, are there any architectural reasons why it would not work in a 64-bit environment? Does it interoperate with 64-bit versions? Everything is 32/64 bit clean. - Does the project depend on particular versions of supporting software (especially Java virtual machines)? If so, do you deliver a private copy? What happens if a conflicting or incompatible version is already or subsequently installed on the system? N/A - Is the project internationalized and localized? N/A - Is the project compatible with IPV6 interfaces and addresses? N/A 12. What is its window/desktop operational environment? - Is it ICCCM compliant (ICCCM is the standard protocol for interacting with window managers)? N/A - X properties: Which ones does it depend on? Which ones does it export, and what are their types? N/A - Describe your project's support for User Interface facilities including Help, Undo, Cut/Paste, Drag and Drop, Props, Find, Stop? N/A - How do you respond to property change notification and ICCCM client messages (e.g. Do you respond to "save workspace")? N/A - Which window-system toolkit/desktop does your project depend on? N/A - Can it execute remotely? Is the user aware that the tool is executing remotely? Does it matter? N/A - Which X extensions does it use (e.g. SHM, DGA, Multi-Buffering? (Hint: use "xdpyinfo") N/A - How does it use colormap entries? Can you share them? N/A - Does it handle 24-bit operation? N/A 13. What interfaces does your project import and export? http://dtsw.sfbay.sun.com/ddi/ddi-interrupts.txt 14. What are its other significant internal interfaces inter-subsystem and inter-invocation)? Introducing: bus_intr_op Obsoleting: bus_get_intrspec, bus_add_intrspec, bus_remove_intrspec, bus_intr_ctl 15. Is the interface extensible? How will the interface evolve? Yes, it will be handled through the versioning of interrupt handles. - How is versioning handled? For DDI interface, our current plan is to accept the version information from leaf drivers as part of ddi_intr_alloc(), save it in the interrupt handle and process all subsequent requests based on this version number. On the bus op interface, our approach is to fail earlier version nexus drivers. - What was the commitment level of the previous version? N/A - Can this version co-exist with existing standards and with earlier and later versions or with alternative implementations (perhaps by other vendors)? Yes. - What are the clients over which a change should be managed? N/A - How is transition to a new version to be accomplished? What are the consequences to ISV's and their customers? Existing interfaces will be supported but all related documentation including man pages will be removed. It means, maintain binary compatibility for existing DDI compliant drivers. Man pages and documentation will be provided for new interfaces. 16. How do the interfaces adapt to a changing world? Interfaces can be extended to support future interrupt types. - What is its relationship with (or difficulties with) multimedia? 3D desktops? Nomadic computers? Storage-less clients? A networked file system model (i.e., a network-wide file manager)? N/A 17. Interoperability - If applicable, explain your project's interoperability with the other major implementations in the industry. In particular, does it interoperate with Microsoft's implementation, if one exists? N/A - What would be different about installing your project in a heterogeneous site instead of a homogeneous one (such as Sun)? N/A - Does your project assume that a Solaris-based system must be in control of the primary administrative node? N/A 18. Performance - How will the project contribute (positively or negatively) to "system load" and "perceived performance"? There is significant amount of performance gain with devices/drivers that support multiple MSI/X especially InfiniBand, NICs. NOTE: This would require the existing InfiniBand HCA driver and MSI/X capable NIC drivers to be modified to make use of this feature. Actual driver modification is beyond the scope of this project but the respective driver teams are in contact with us. - What are the performance goals of the project? How were they evaluated? What is the test or reference platform? No regression. - Does the application pause for significant amounts of time? Can the user interact with the application while it is performing long-duration tasks? N/A - What is your project's MT model? How does it use threads internally? How does it expect its client to use threads? If it uses callbacks, can the called entity create a thread and recursively call back? N/A - What is the impact on overall system performance? What is the average working set of this component? How much of this is shared/sharable by other apps? No impact on overall system performance. - Does this application "wake up" periodically? How often and under what conditions? What is the working set associated with this behavior? N/A - Will it require large files/databases (for example, new fonts)? No. - Do files, databases or heap space tend to grow with time/load? What mechanisms does the user have to use to control this? What happens to performance/system load? N/A 19. Please identify any issues that you would like the ARC to address. None. - Interface classification, deviations from standards, architectural conflicts, release constraints... None. - Are there issues or related projects that the ARC should advise the appropriate steering committees? None. 20. Appendices to include - Design specification. http://dtsw.sfbay.sun.com/ddi/ddi-interrupts.txt - One-Pager. http://dtsw.sfbay.sun.com/ddi/ddi-intr-1pgr.txt - References to other documents. (Place copies in case directory.) None.