1. What specifically is the proposal that we are reviewing? - What is the technical content of the project? This project provides FCoE target functionality to Solaris COMSTAR Fibre Channel port provider, so that a Solaris host can be used as FCoE target device. This project consists of two kernel drivers and extension to the existing fcadm utility. The two new drivers are: - fcoet: A pseudo driver, it's a virtual FC HBA driver which interfaces to COMSTAR FCT module on the top. It's also an implementation of FCoE VN_Port entity as defined in FCoE Specification. - fcoe: A pseudo driver, it provides FCoE transport to fcoet. It sends/ receives FCoE frames through GLDv3 MAC Client Interface on the bottom. It's an implementation of FCoE LEP entity as defined in FCoE Specification. fcoe and fcoet work together in a service provider/client manner. There is only one instance of fcoe driver across the system which provides FCoE transport services (MAC interface management, FCoE frame encapsulation and de-encapsulation, FCoE frame TX/RX). Multiple fcoet driver instances could co-exist within the same system and consume FCoE transport services provided by fcoe through FCoE Client Inteface. The management of FCoE target is through fcadm/fcinfo utility. New sub- commands will be added to fcadm to create and delete FCoE target ports. fcadm will issue ioctls to fcoe driver to accomplish this. - A high level design diagram is at: http://opensolaris.org/os/project/fcoe/fcoe_target_arch.jpg - Project one pager is at: http://sac.sfbay/Archives/Projects/2008/20080511_zhong.wang - The opensolaris project page is at: http://opensolaris.org/os/project/fcoe/ - Is this a new product, or a change to a pre-existing one? If it is a change, would you consider it a "major", "minor", or "micro" change? Its a new product. - If your project is an evolution of a previous project, what changed from one version to another? N/A - What is the motivation for it, in general as well as specific terms? What are the expected benefits for Sun? By what criteria will you judge its success? FCoE is a newly proposed T11 standard and is advocated by all leading IT vendors. With FCoE, customers can achieve I/O consolidation by leveraging their existing FC investment and knowledge with the 10 Gigabit Ethernet infrastructure. I/O consolidation results in less adapters, less power consumption, less cooling requirement and less cabling. Adding support of FCoE will enrich Solaris open storage stacks, make Solaris competitive to other Operating Systems in terms of storage connection capability. 2. Describe how your project changes the user experience, upon installation and during normal operation. - What does the user perceive when the system is upgraded from a previous release? No Change. 3. What is its plan? - What is its current status? Has a design review been done? Are there multiple delivery phases? Design review and a prototype have been done. This project has only one phase. 4. Are there related projects in Sun? - If so, what is the proposal's relationship to their work? Which not-yet- delivered Sun (or non-Sun) projects (libraries, hardware, etc.) does this project depend upon? What other projects, if any, depend on this one? This projects depends on the following projects: - PSARC/2007/523 COMSTAR: Common Multiprotocol SCSI Target - PSARC/2004/291 Fibre Channel HBA Port Utility - PSARC/2004/571 Nemo - a.k.a. GLD v3 - PSARC/2006/357 Crossbow - Network Virtualization and Resource Management PSARC/2008/311 FCoE Initiator depends on this project. - Are you updating, copying or changing functional areas maintained by other groups? How are you coordinating and communicating with them? Do they "approve" of what you propose? If not, please explain the areas of disagreement. No updating, copying or changing functional areas maintained by other groups. 5. How is the project delivered into the system? - Identify packages, directories, libraries, databases, etc. New package SUNWfcoe, installs fcoe driver at /kernel/drv//fcoe. New package SUNWfcoet, installs fcoet driver at /kernel/drv//fcoet. 6. Describe the project's hardware platform dependencies. FCoE requires an lossless Ethernet network. This project will work with any Ethernet controller that has a GLDv3 driver. 7. System administration - How will the project's deliverables be installed and (re)configured? Deliverables will be installed using pkgadd. FCoE target will be configured using fcadm(1M). - How will the project's deliverables be uninstalled? pkgrm - Does it use inetd to start itself? No - Does it need installation within any global system tables? No - Does it use a naming service such as NIS, NIS+ or LDAP? No - What are its on-going maintenance requirements (e.g. Keeping global tables up to date, trimming files)? N/A - How does this project's administrative mechanisms fit into Sun's system administration strategies? E.g., how does it fit under the Solaris Management Console (SMC) and Web-Based Enterprise Management (WBEM), how does it make use of roles, authorizations and rights profiles? Additionally, how does it provide for administrative audit in support of the Solaris BSM configuration? Users assigned the "STMF Administration" profile will be able to configure FCoE targets. - What tunable parameters are exported? Can they be changed without rebooting the system? Examples include, but are not limited to, entries in /etc/system and ndd(8) parameters. What ranges are appropriate for each tunable? What are the commitment levels associated with each tunable (these are interfaces)? TBD. 8. Reliability, Availability, Serviceability (RAS) - Does the project make any material improvement to RAS? No - How can users/administrators diagnose failures or determine operational state? (For example, how could a user tell the difference between a failure and very slow performance?) Port state and link statistics can be retrieved via fcinfo. - What are the project's effects on boot time requirements? None - How does the project handle dynamic reconfiguration (DR) events? N/A - What mechanisms are provided for continuous availability of service? N/A - Does the project call panic()? Explain why these panics cannot be avoided. No - How are significant administrative or error conditions transmitted? SNMP traps? Email notification? Error conditions are logged. - How does the project deal with failure and recovery? Network failure and recovery will be handled, see section below. STMF will handle port online and offline. - Does it ever require reboot? If so, explain why this situation cannot be avoided. Before configuring FCoE ports, the MTU size of the corresponding Ethernet controller has to be changed to larger than 2.5KB. This is a requirement by FCoE, and some GLDv3 drivers require a reboot to make the new MTU size effective. - How does your project deal with network failures (including partition and re- integration)? How do you handle the failure of hardware that your project depends on? FCoE network failures are managed in the same way as FC SAN. Link failures between NIC port and switch are reported by MAC layer, link failures between switch and remote devices are reported through RSCN ELS to the local FCoE port. All failures will be handled accordingly by determining the nature of the change. - Can it save/restore or checkpoint and recover? N/A - Can its files be corrupted by failures? Does it clean up any locks/files after crashes? N/A 9. Observability - Does the project export status, either via observable output (e.g., netstat) or via internal data structures (kstats)? Yes, status is visisble via fcinfo(1M). - How would a user or administrator tell that this subsystem is or is not behaving as anticipated? Observing status from fcadm/fcinfo and error log. - What statistics does the subsystem export, and by what mechanism? None. - What state information is logged? FCoE port creation/removal/online/offline. - In principle, would it be possible for a program to tune the activity of your project? N/A 10. What are the security implications of this project? - What security issues do you address in your project? None - The Solaris BSM configuration carries a Common Criteria (CC) Controlled Access Protection Profile (CAPP) -- Orange Book C2 -- and a Role Based Access Control Protection Profile (RBAC) -- rating, does the addition of your project effect this rating? E.g., does it introduce interfaces that make access or privilege decisions that are not audited, does it introduce removable media support that is not managed by the allocate subsystem, does it provide administration mechanisms that are not audited? No - Is system or subsystem security compromised in any way if your project's configuration files are corrupt or missing? No - Please justify the introduction of any (all) new setuid executables. N/A - Include a thorough description of the security assumptions, capabilities and any potential risks (possible attack points) being introduced by your project. A separate Security Questionnaire http://sac.sfbay/cgi-bin/bp.cgi?NAME=Security.bp is provided for more detailed guidance on the necessary information. Cases are encouraged to fill out and include the Security questionnaire (leveraging references to existing documentation) in the case materials. A physically secured and controlled network is mandatory for FCoE to run on. However FC-SP (Fibre Channel Security Protocol standard by T11) can be used with FCoE to enforce more security. FC-SP is intended as a secure protocol which includes authentication and encryption, but it is not widely implemented in the industry and is not implemented in Solaris yet. 11. What is its UNIX operational environment: - Which Solaris release(s) does it run on? Solaris Nevada/OpenSolaris. - Environment variables? Exit status? Signals issued? Signals caught? (See signal(3HEAD).) None - Device drivers directly used (e.g. /dev/audio)? .rc/defaults or other resource/configuration files or databases? None - Does it use any "hidden" (filename begins with ".") or temp files? No - Does it use any locking files? No - Command line or calling syntax: What options are supported? (please include man pages if available) Does it conform to getopt() parsing requirements? Please refer to fcadm(1M) included with the case materials. - Is there support for standard forms, e.g. "-display" for X programs? Are these propagated to sub-environments? N/A - What shared libraries does it use? (Hint: if you have code use "ldd" and "dump -Lv")? fcadm uses: libHBAAPI libscf libc libcfgadm libdevinfo libdl libnvpair libuutil libgen libsec libnsl libavl libmp libmd libm libstmf - Identify and justify the requirement for any static libraries. N/A - Does it depend on kernel features not provided in your packages and not in the default kernel (e.g. Berkeley compatibility package, /usr/ccs, /usr/ucblib, optional kernel loadable modules)? No - Is your project 64-bit clean/ready? If not, are there any architectural reasons why it would not work in a 64-bit environment? Does it interoperate with 64-bit versions? Yes - Does the project depend on particular versions of supporting software (especially Java virtual machines)? If so, do you deliver a private copy? What happens if a conflicting or incompatible version is already or subsequently installed on the system? No - Is the project internationalized and localized? No - Is the project compatible with IPV6 interfaces and addresses? N/A 12. What is its window/desktop operational environment? N/A 13. What interfaces does your project import and export? Interfaces Imported -----------------+------------------------+---------------------- Interface | Classification | Comments -----------------+------------------------+---------------------- | | COMSTAR FCT FCA | Project Private | Interface | | | | GLDv3 MAC Client | Consolidation Private | Need contract Interface | | | | libstmf | Committed | | | -----------------+------------------------+---------------------- Interfaces Exported -----------------+------------------------+---------------------- Interface | Classification | Comments -----------------+------------------------+---------------------- | | FCoE Client | Project Private | Interface | | | | FCoE IOCTLs for | Project Private | libHBAAPI | | | | fcadm | Committed | See fcadm(1M) | | fcadm output | Not an Interface | | | -----------------+------------------------+---------------------- - What other applications should it interoperate with? How will it do so? N/A - Is it "pipeable"? How does it use stdin, stdout, stderr? fcadm is pipeable. - Explain the significant file formats, names, syntax, and semantics. fcadm stores persistent configuration data using libstmf which in turn persists data in SMF. - Is there a public namespace? (Can third parties create names in your namespace?) How is this administered? N/A - Are the externally visible interfaces documented clearly enough for a non-Sun client to use them successfully? Yes. fcadm is well documented. 14. What are its other significant internal interfaces inter-subsystem and inter-invocation)? - Protocols (public or private) HBAAPI FCoE - Private ToolTalk usage N/A - Files N/A - Other - Are the interfaces re-entrant? Existing restrictions apply. HBAAPI will be re-entrant. 15. Is the interface extensible? How will the interface evolve? - How is versioning handled? Version number is kept in FCoE client interfaec and is checked when fcoet registers to fcoe. - What was the commitment level of the previous version? N/A - Can this version co-exist with existing standards and with earlier and later versions or with alternative implementations (perhaps by other vendors)? This will be the first release of the product. All exported interfaces will maintain backward compatibility in the future. - What are the clients over which a change should be managed? - How is transition to a new version to be accomplished? What are the consequences to ISV's and their customers? N/A 16. How do the interfaces adapt to a changing world? Future change of FCoE specification won't change its FC nature, thus no external interfaces will be impacted. - What is its relationship with (or difficulties with) multimedia? 3D desktops? Nomadic computers? Storage-less clients? A networked file system model (i.e., a network-wide file manager)? N/A 17. Interoperability - If applicable, explain your project's interoperability with the other major implementations in the industry. In particular, does it interoperate with Microsoft's implementation, if one exists? FCoE Target will interoperate with any FC initiators with the existence of an FCoE switch in the SAN. FCoE Target will interoperate with any FCoE initiators, either in back-to-back mode, or with connections to an FCoE switch. - What would be different about installing your project in a heterogeneous site instead of a homogeneous one (such as Sun)? None - Does your project assume that a Solaris-based system must be in control of the primary administrative node? No 18. Performance - How will the project contribute (positively or negatively) to "system load" and "perceived performance"? FCoE target can create multiple worker thread which will consume CPU cycles based on the amount of incoming traffic from the storage network. - What are the performance goals of the project? How were they evaluated? What is the test or reference platform? When running on systems that have enough CPU power, the FCoE Target is supposed to saturate a 10G Ethernet link provided that the Ethernet controller is able to fan out to multiple tx/rx rings and load balance FCoE frames so that it can reach full line speed throughput. SCSI I/O throughput will be lower than line rate due to overheads. The performance will be measured by dd, vdbench and other SCSI benchmarking tools. - Does the application pause for significant amounts of time? No Can the user interact with the application while it is performing long-duration tasks? No long duration tasks. - What is your project's MT model? How does it use threads internally? How does it expect its client to use threads? If it uses callbacks, can the called entity create a thread and recursively call back? The FCoE target complies to COMSTAR's MT model, that is, each incoming scsi task is assigned a worker thread of its own. Multiple worker threads are created additionally to do the calculation of FC-CRC. The number of worker threads can be configured through /etc/system tunable. - What is the impact on overall system performance? What is the average working set of this component? How much of this is shared/sharable by other apps? The impact on system performance depends upon the incoming traffic from the storage network. - Does this application "wake up" periodically? How often and under what conditions? What is the working set associated with this behavior? No - Will it require large files/databases (for example, new fonts)? No - Do files, databases or heap space tend to grow with time/load? What mechanisms does the user have to use to control this? What happens to performance/system load? No 19. Please identify any issues that you would like the ARC to address. Stability of GLDv3 MAC Client interface. This interface is still under change with Crossbow project. 20. Appendices to include one-pager fcoe_target_func_spec_v102.pdf - Project functional specification fcadm.man - Manpage for fcadm(1M) writing_fca.txt - COMSTAR FCT FCA Programming Guide