1. What specifically is the proposal that we are reviewing? - What is the technical content of the project? Please provide diagrams that illustrate the project's architecture and interfaces. Be specific about deliverable components and their interfaces. Pay particular attention to describing the scope of the project, especially near the project boundaries. iSCSI Extensions for RDMA (iSER) accelerates the iSCSI protocol by mapping the data transfer phases to Remote DMA (RDMA) operations. As a result an iSER initiator can read and write data from an iSER target at high data rates with relatively low CPU utilization compared to iSCSI using TCP/IP. This project will implement both an iSER initiator and an iSER target. Since iSER is based on iSCSI this target will also implement an iSCSI port provider for the Common SCSI Target Framework (COMSTAR, PSARC PSARC 2007/523) adding iSCSI transport to the existing Fibre Channel support. Section 1.2 of the iSCSI Data Mover design document in the case materials describes the high level architecture of the project including a block diagram. - Is this a new product, or a change to a pre-existing one? If it is a change, would you consider it a "major", "minor", or "micro" change? See the Release Taxonomy in: http://sac.sfbay/BestPractices/release_taxonomy.html Micro change - If your project is an evolution of a previous project, what changed from one version to another? This is a new project and stands on its own. Solaris includes an existing iSCSI target implementation (PSARC 2005/441) which is unrelated to this project. The ISCSI target implementation described by this project is intended to become the preferred Solaris iSCSI target so that Solaris iSCSI support can leverage ongoing COMSTAR development. - What is the motivation for it, in general as well as specific terms? (Note that not everyone on the ARC will be an expert in the area.) iSER provides a high-bandwidth low-latency storage protocol for Infiniband that will facilitate the adoption of Solaris in the high performance computing (HPC) industry. In addition, the new iSCSI capability for COMSTAR will add an enterprise class iSCSI target capability to Solaris. - What are the expected benefits for Sun? Support for a new high performance Infiniband storage protocol and a high performance iSCSI target. - By what criteria will you judge its success? The success of this project is largely driven by performance. The iSER transport should provide similar performance numbers to other RDMA-based protocols like NFS over RDMA. The iSCSI target component in software mode should provide performance close to the raw TCP/IP network performance as measured by a network benchmark tool. 2. Describe how your project changes the user experience, upon installation and during normal operation. On the initiator (client) side the user should not see any difference. An iSCSI connection on a network capable of iSER transport (initially only Infiniband) will automatically use iSER instead of TCP/IP. Since the project adds a new iSCSI target component the user may need to perform some configuration of the iSCSI target. - What does the user perceive when the system is upgraded from a previous release? No user impact 3. What is its plan? - What is its current status? Has a design review been done? Are there multiple delivery phases? We have working iSER initiator and target implementations (pre-alpha). Design reviews have been done. This project has only one phase. 4. Are there related projects in Sun? - If so, what is the proposal's relationship to their work? Which not-yet- delivered Sun (or non-Sun) projects (libraries, hardware, etc.) does this project depend upon? What other projects, if any, depend on this one? SAM-QFS 5.0 (PSARC 2007/588) depends on this project - Are you updating, copying or changing functional areas maintained by other groups? How are you coordinating and communicating with them? Do they "approve" of what you propose? If not, please explain the areas of disagreement. iSER requires modification to the existing iSCSI initiator to enable the new transport. The iSCSI initiator team is aware of the changes being made and approves. 5. How is the project delivered into the system? - Identify packages, directories, libraries, databases, etc. The project adds: SUNWidmr kernel/misc//idm Kernel module containing core iSCSI code used by both initiator and target SUNWiscsitr kernel/drv//iscsit Driver providing the iSCSI transport to COMSTAR SUNWiscsitu /usr/lib/libiscsit iSCSI targetmanagement library (used by the itadm CLI) /usr/bin/itadm COMSTAR iSCSI target CLI SUNWiscsir kernel/drv//iscsi Modification to existing iSCSI initiator 6. Describe the project's hardware platform dependencies. iSER requires an Infiniband network. The new iSCSI target capability has no hardware dependencies and will work on any TCP/IP-capable network. - Explain any reasons why it would not work on both SPARC and Intel? N/A 7. System administration - How will the project's deliverables be installed and (re)configured? Deliverables will be installed using pkgadd. iSER initiator will be configured identically to conventional iSCSI using the iscsiadm CLI. iSER target and TCP/IP iSCSI target will be configured using the new itadm(1m) CLI (see man page in the case materials). - How will the project's deliverables be uninstalled? pkgrm - Does it use inetd to start itself? No - Does it need installation within any global system tables? No - Does it use a naming service such as NIS, NIS+ or LDAP? No - What are its on-going maintenance requirements (e.g. Keeping global tables up to date, trimming files)? N/A - How does this project's administrative mechanisms fit into Sun's system administration strategies? E.g., how does it fit under the Solaris Management Console (SMC) and Web-Based Enterprise Management (WBEM), how does it make use of roles, authorizations and rights profiles? Additionally, how does it provide for administrative audit in support of the Solaris BSM configuration? No new interactions. iSCSI initiator management is unchanged. The project introduces a new CLI for iSCSI target management. Users assigned the "STMF Administration" profile will be able to configure STMF iSCSI targets. - What tunable parameters are exported? Can they be changed without rebooting the system? Examples include, but are not limited to, entries in /etc/system and ndd(8) parameters. What ranges are appropriate for each tunable? What are the commitment levels associated with each tunable (these are interfaces)? TBD. In general terms the project does not expect to implement any tunables for iSCSI over TCP/IP but will likely provide some global /etc/system tunables for iSER. 8. Reliability, Availability, Serviceability (RAS) - Does the project make any material improvement to RAS? No - How can users/administrators diagnose failures or determine operational state? (For example, how could a user tell the difference between a failure and very slow performance?) Each connection has an associated connection state which users can see using the iscsiadm (for initiator) and itadm (for target) CLIs. Standard networking tools like "snoop" and "ping" are also useful for diagnosing failures. - What are the project's effects on boot time requirements? None - How does the project handle dynamic reconfiguration (DR) events? N/A - What mechanisms are provided for continuous availability of service? N/A - Does the project call panic()? Explain why these panics cannot be avoided. No - How are significant administrative or error conditions transmitted? SNMP traps? Email notification? Error conditions are logged - How does the project deal with failure and recovery? See the section below on "network failures" - Does it ever require reboot? If so, explain why this situation cannot be avoided. No reboot should be required - How does your project deal with network failures (including partition and re- integration)? How do you handle the failure of hardware that your project depends on? Network failures are handle by generating appropriate events to the connection state machine. The initiator generally attempts to re-establish a connection after a network failure (this is existing functionality unmodified by the iSER project). The target will cleanup and wait for a new connection. iSER expects the underlying networking software to handle hardware failures appropriately and relies on eventually getting some kind of failure indication that it can use to drive an appropriate connection state change. - Can it save/restore or checkpoint and recover? N/A - Can its files be corrupted by failures? Does it clean up any locks/files after crashes? N/A 9. Observability - Does the project export status, either via observable output (e.g., netstat) or via internal data structures (kstats)? iscsiadm and itadm will provide status information. Also the target component will provide the probes described by the iSCSI target provider (PSARC 2007/153). - How would a user or administrator tell that this subsystem is or is not behaving as anticipated? iscsiadm/itadm - What statistics does the subsystem export, and by what mechanism? For each active iSCSI connection: Connection State iSCSI PDUs Received iSCSI PDUs Sent SCSI commands started SCSI commands completed Data transfers started (target only) Data transfers completed (target only) - What state information is logged? No state information is logged -- dtrace can be used to monitor state transitions - In principle, would it be possible for a program to tune the activity of your project? Yes 10. What are the security implications of this project? - What security issues do you address in your project? All iSCSI components provide Challenge Handshake Authentication Protocol (CHAP) authentication. - The Solaris BSM configuration carries a Common Criteria (CC) Controlled Access Protection Profile (CAPP) -- Orange Book C2 -- and a Role Based Access Control Protection Profile (RBAC) -- rating, does the addition of your project effect this rating? E.g., does it introduce interfaces that make access or privilege decisions that are not audited, does it introduce removable media support that is not managed by the allocate subsystem, does it provide administration mechanisms that are not audited? This project does not directly implement any auditing support. - Is system or subsystem security compromised in any way if your project's configuration files are corrupt or missing? No - Please justify the introduction of any (all) new setuid executables. N/A - Include a thorough description of the security assumptions, capabilities and any potential risks (possible attack points) being introduced by your project. A separate Security Questionnaire http://sac.sfbay/cgi-bin/bp.cgi?NAME=Security.bp is provided for more detailed guidance on the necessary information. Cases are encouraged to fill out and include the Security questionnaire (leveraging references to existing documentation) in the case materials. See included security questionnaire Projects must highlight information for the following important areas: - What features are newly visible on the network and how are they protected from exploitation (e.g. unauthorized access, eavesdropping) - If the project makes decisions about which users, hosts, services, ... are allowed to access resources it manages, how is the requestor's identity determined and what data is used to determine if the access granted. Also how this data is protected from tampering. - What privileges beyond what a common user (e.g. 'noaccess') can perform does this project require and why those are necessary. - What parts of the project are active upon default install and how it can be turned off. CHAP authentication is provided to authenticate remote initiators before allowing access to an iSCSI target. iSCSI using TCP/IP transport can also take advantage of IPsec for additional security. 11. What is its UNIX operational environment: - Which Solaris release(s) does it run on? Nevada/OpenSolaris - Environment variables? Exit status? Signals issued? Signals caught? (See signal(3HEAD).) N/A - Device drivers directly used (e.g. /dev/audio)? .rc/defaults or other resource/configuration files or databases? N/A - Does it use any "hidden" (filename begins with ".") or temp files? No - Does it use any locking files? No - Command line or calling syntax: What options are supported? (please include man pages if available) Does it conform to getopt() parsing requirements? See included itadm(1m) man page - Is there support for standard forms, e.g. "-display" for X programs? Are these propagated to sub-environments? N/A - What shared libraries does it use? (Hint: if you have code use "ldd" and "dump -Lv")? libstmf.so.1 libnvpair.so.1 libuuid.so.1 - Identify and justify the requirement for any static libraries. N/A - Does it depend on kernel features not provided in your packages and not in the default kernel (e.g. Berkeley compatibility package, /usr/ccs, /usr/ucblib, optional kernel loadable modules)? No - Is your project 64-bit clean/ready? If not, are there any architectural reasons why it would not work in a 64-bit environment? Does it interoperate with 64-bit versions? Yes, the project is 64-bit clean - Does the project depend on particular versions of supporting software (especially Java virtual machines)? If so, do you deliver a private copy? What happens if a conflicting or incompatible version is already or subsequently installed on the system? N/A - Is the project internationalized and localized? Yes - Is the project compatible with IPV6 interfaces and addresses? Yes 12. What is its window/desktop operational environment? No desktop environment 13. What interfaces does your project import and export? - Please provide a table of imported and exported interfaces, including stability levels. Pay close attention to the classification of these interfaces in the Interface Taxonomy -- e.g., "Committed," "Uncommitted," and "*Private;" see: http://sac.sfbay/cgi-bin/bp.cgi?NAME=interface_taxonomy.bp Use the following format: Interfaces Imported Interface Classification Comments ------------------------------------------------------------------------ libstmf.so Committed libnvpair.so Committed libuuid.so Committed libstmf Committed VOP_IOCTL Consolidation Private Used with kernel sockfs VOP_CLOSE Consolidation Private Used with kernel sockfs VN_RELE Consolidation Private Used with kernel sockfs sockfs kernel API Consolidation Private Needs contract STMF Port Provider Committed COMSTAR Port Provider API Interfaces Exported Interface Classification Comments ------------------------------------------------------------------------ itadm Committed See itadm(1m) man page itadm CLI output Not an Interface libiscsit Committed - What other applications should it interoperate with? How will it do so? N/A - Is it "pipeable"? How does it use stdin, stdout, stderr? itadm is pipeable. Stdin/stdout/stderr will follow Solaris conventions - Explain the significant file formats, names, syntax, and semantics. itadm stores persistent configuration data using libstmf which in turn persists data in SMF. - Is there a public namespace? (Can third parties create names in your namespace?) How is this administered? No public namespace - Are the externally visible interfaces documented clearly enough for a non-Sun client to use them successfully? itadm/iscsiadm are well documented 14. What are its other significant internal interfaces inter-subsystem and inter-invocation)? itadm provides configuration updates to the iscsit kernel driver and receives status information from the iscsit kernel driver using ioctls. These ioctls are described in the iscsit design document. 15. Is the interface extensible? How will the interface evolve? No public interface outside of CLI utilities 16. How do the interfaces adapt to a changing world? - What is its relationship with (or difficulties with) multimedia? 3D desktops? Nomadic computers? Storage-less clients? A networked file system model (i.e., a network-wide file manager)? N/A 17. Interoperability - If applicable, explain your project's interoperability with the other major implementations in the industry. In particular, does it interoperate with Microsoft's implementation, if one exists? iSER and iSCSI should interoperate with other standards-compliant implemenations including Microsoft. - What would be different about installing your project in a heterogeneous site instead of a homogeneous one (such as Sun)? No difference - Does your project assume that a Solaris-based system must be in control of the primary administrative node? No primary administrative node 18. Performance - How will the project contribute (positively or negatively) to "system load" and "perceived performance"? iSER should reduce total system load compared to TCP/IP iSCSI since data transfers are performed using RDMA. - What are the performance goals of the project? How were they evaluated? What is the test or reference platform? iSER should attain at least 90% of the network bandwidth attained using a raw network benchmark. - Does the application pause for significant amounts of time? No Can the user interact with the application while it is performing long-duration tasks? No long-duration tasks - What is your project's MT model? How does it use threads internally? How does it expect its client to use threads? If it uses callbacks, can the called entity create a thread and recursively call back? Each iSCSI connection has an associated taskq to handle the connection state machine Socket-based iSCSI connections include two additional kernel threads: One for receive and one for transmit - What is the impact on overall system performance? What is the average working set of this component? How much of this is shared/sharable by other apps? System performance impact will depend on the amount of iSCSI traffic being requested - Does this application "wake up" periodically? How often and under what conditions? What is the working set associated with this behavior? No - Will it require large files/databases (for example, new fonts)? No - Do files, databases or heap space tend to grow with time/load? What mechanisms does the user have to use to control this? What happens to performance/system load? No 19. Please identify any issues that you would like the ARC to address. - Interface classification, deviations from standards, architectural conflicts, release constraints... - Are there issues or related projects that the ARC should advise the appropriate steering committees? 20. Appendices to include - One-Pager. - Prototype specification. - References to other documents. (Place copies in case directory.)