SAM-QFS 5.0 Project Outline v.2008.02.01 0.0 Log of Updates to this Document 2008/01/24 - Added version ID (date) to title. - Added section 0.0 Log of Updates to this Document. - Updated section 1.0 Description of Changes to include a summary of updates to SAM-QFS since the PSARC/2001/599 case (see review comment ID gcs-0). - Added sub-section 1.6 to state the current SAM-QFS responses to the opinion from PSARC/2001/599 (see review comment ID gcs-2). 2008/01/29 - Updated section "2.0 Motivation for Changes" to clarify the short-term scalability limit of 1024 client nodes (see reviewer comment ID ram-4). - Removed internal link to documentation plan web page from section "9.2 Completion of Documentation Plan" (see reviewer comment regarding "open" inception review requirements). - Removed internal link to test matrix and Q&A web pages from section "9.3 Completion of Test and Verification Strategy" (see reviewer comment regarding "open" inception review requirements). - Removed internal link to test matrix and release criteria web pages from section "9.5 Completion of Release Criteria" (see reviewer comment regarding "open" inception review requirements). - Replaced internal link to feature matrix and functional specifications with external link to non-proprietary functional specifications in sections "1.0 Description of Changes" and "9.1 Completion of Functional Specifications" (see reviewer comment regarding "open" inception review requirements). - Removed internal links to SAM-QFS 5.0 program schedule and Sun PLC process web pages from section "5.0 Program Schedule" (see reviewer comment regarding "open" inception review requirements). - Removed internal links to SAM-QFS 5.0 PCD web page from section "4.0 Benefits of Changes" (see reviewer comment regarding "open" inception review requirements). 2008/02/01 - Added section "0.1 Disclaimer". - Updated section "1.0 Description of Changes" to include references to object-based shared SAM-QFS documentation. - Updated section "1.6 Current SAM-QFS Responses to the Opinion from PSARC/2001/599" to state the current plan of record with respect to SAM-QFS consolidation (see reviewer comment ID ram-16). 0.1 Disclaimer Any and all references in this document, and all supporting documents, to SAM-QFS 5.0 capabilities reflect current plans only. Those plans are not a commitment to deliver any of those capabilities within any particular release, nor within any time frame, nor ever. The contents and existence of this document is subject to change without notice. 1.0 Description of Changes SAM-QFS 5.0 is the first major release since SAM-QFS 4.0 (PSARC/2001/599). It is comprised of all additional and modified features, functionality, and architecture since the SAM-QFS 4.0 release. Since then, there have been 6 minor releases of SAM-QFS (identified as SAM-QFS 4.1 through 4.6). SAM-QFS 4.6 was released on 04/06/2007 and is currently supported by the PTS and RPE groups. Functional specification documents are unavailable for the features introduced in the interim minor releases since SAM-QFS 4.0. For the SAM-QFS 5.0 (post-4.6) feature functional specification documents, see: http://www.opensolaris.org/os/project/samqfs/specs/50_Specs/ The SAM-QFS GUI was introduced in SAM-QFS 4.1. Updates for the 5.0 release (post-4.6) focus on usability improvements and simplifying the management of large shared file systems. More specific information is available in the SAM-QFS 5.0 GUI Functional Specification located at: http://www.opensolaris.org/os/project/samqfs/specs/50_Specs/ Object-based shared SAM-QFS file system capabilities are planned to be introduced in SAM-QFS 5.0. For details concerning the object-based effort, see the supplied document (SAM-QFS_OSD.odt). Also, functional specifications for object-based features are located at: http://www.opensolaris.org/os/project/samqfs/specs/50_Specs/ A summary of SAM-QFS changes since SAM-QFS 4.0 (PSARC/2001/599) is provided below. Additional detail regarding each item may be found in the supplied man pages and customer documentation. This section also includes current SAM-QFS responses to the opinion from PSARC/2001/599. 1.1 Archiving - Continuous Archiving (New) - Multiple Disk Volumes Archiving (New) - Archive Data Integrity Checking (New) - High Availability Archiving (HA-SAM) w/ Sun Cluster (New) - releaser.cmd(4) (Enhanced) - archiver.cmd(4) (Enhanced) - stager.cmd(4) (Enhanced) - Supported Devices and Libraries (Enhanced) 1.2 System Administration - mount(1M) (Enhanced) - umount(1M) (Enhanced) - umount(1M) (Enhanced) - samu(1M) (Enhanced) - samfsdump(1M) (Enhanced) - qfsdump(1M) (Enhanced) - samfsrestore(1M) (Enhanced) - tapealert(1M) (New) - samtrace(1M) (Enhanced) - sammkfs(1M) (Enhanced) - samfsck(1M) (Enhanced) - StorADE Support for Diagnostics (New) - SNMP Trap Support for Diagnostics (New) - SAM-QFS GUI (New) - Online Filesystem Shrink/Grow (New) - SAM-QFS Upgrade Procedures (Enhanced) 1.3 User Commands - sfind(1) (Enhanced) - archive(1) (Enhanced) 1.4 Infrastructure - EFI Labels for Large Device Support (New) - Single Port Multiplexing (SPM) Support for SAM-QFS Daemons(New) - Sun Cluster Interoperability (New) - samioc Driver Module (New) - SANergy Heterogeneous File Sharing (New) - Linux Client for Shared QFS (New) - Solaris 10 on SPARC and AMD64 (New) - samaio Asynchronous Driver for Shared QFS (New) - WORM (New) - RBAC (New) - Solaris 8 Support Dropped (New) - Object-Based Data Management (New) - Least Privilege (New) - Directory Lookup Performance (Enhanced) - Metadata Journaling (New) - IPv6 (New) - Oracle RAC (New) 1.5 Other - Open Source SAM-QFS Software 1.6 Current SAM-QFS Responses to the Opinion from PSARC/2001/599 Most issues from the PSARC/2001/599 review have been addressed over the years, yet some remain to be addressed (see below). The current plan of record is to consolidate SAM-QFS into [TBD] sometime after the current major release. Consolidation has been delayed until then due to business-related release pressures. 1.6.1 "4.1 Dependency on Solaris Kernel Interfaces" COMMENT: "SAM-QFS depends on several unsupported and undocumented ker- nel-level interfaces into the Solaris operating system. The current imported interface bind SAM-QFS to a particular release and patch level of Solaris. Since there is no ARCa- ble interfaces, the committee and project team have taken a strategy to write contracts for the groups of imported interfaces, as documented in Table 3. Due to the overwhealming quantity of contracted interfaces, the committee asks that the project team investigate mecha- nisms to simplify detection of violation of the contracts. This mechanism should be inserted in the development process for both the Solaris and QFS consolidations, flagging when breakage occurs." RESPONSE: Contracts have been written. See the commit.materials for PSARC/2001/599. Prior to the advent of Nevada and Indiana, we were not as diligent as we should have been about keeping up with kernel changes, and we usually found out about problems long after the kernel patches were released. We are now relying heavily on features in Nevada and Indiana for our next release and we are making an effort to keep our code working with Nevada builds as they happen. 1.6.2 "4.2 Cluster Interoperability" COMMENT: "At the time of writing, SAM-QFS has not been qualified with Sun Cluster. The committee asks that the project team pro- vide documentation stating the current level of support with Sun Cluster, and investigate interoperability testing." RESPONSE: We began supporting Sun Cluster in SAM-QFS release 4.1. 1.6.3 "4.3 File System Administration Features" COMMENT: "The committee asks that SAM/QFS team enhance the QFS file system to support the important file system administration features available in Solaris 8, in accordance with UFS and other file systems. The most important of these features are re-mountability and forceable unmount." RESPONSE: Forceable unmount was added in SAM-QFS release 4.5. 1.6.4 "4.4 Attributes" COMMENT: "The committee asks that the QFS team adopt the Solaris model for attributes. Futhermore, the feasibility of moving away from the proprietary attributes in QFS's inodes towards a generic attribute model for archival meta-data should be investigated." RESPONSE: No changes have been made in this area. 1.6.5 "7.1 Appendix A: Technical Changes Required" COMMENT: "1. SAM-QFS should provide notification of reasons for events that cause DR failures." RESPONSE: This work has not been scoped. COMMENT: "2. Contracts to be written for interface requirements between SAM-QFS and the Solaris consolidation. Mechanisms should be put in place to allow easy verification of compatibility to minimize viola- tions of the contract during normal development." RESPONSE: Contracts have been written. See the commit.materials for PSARC/2001/599. COMMENT: "3. The .sh files should be removed." RESPONSE: All instances have been addressed. 1.6.6 "7.2 Appendix B: Technical Changes Advised" COMMENT: "1. Ensure that fsd daemon is only started when SAM- QFS is installed AND configured." RESPONSE: The sam-fsd daemon is now started by sammkfs, mount, samfsck, or samd config. COMMENT: "2. Consolidate the number of deamons that are required to run on the system." RESPONSE: We have changed sam_sharefsd so that it is started only when shared QFS filesystems are mounted. COMMENT: "3. Enhance the generic Solaris tar to provide required features and eliminate the need to ship a separate tar package." RESPONSE: The SAM-QFS 5.0 release includes plans to revisit our options for tar. We would like to support POSIX tar headers, for example. We have recently acquired a contractor to take this task. COMMENT: "4. Make a compatible workaround for the socket bug and have bug resolved in Solaris 9." RESPONSE: We are not aware of any outstanding socket bugs in SAM-QFS. COMMENT: "5. Provide "mount -o remount" support" RESPONSE: This work has not been scoped. COMMENT: "6. Provide forceable unmount support" RESPONSE: Forceable unmount was added in SAM-QFS release 4.5. COMMENT: "7. Simplify the multi-machine administration model" RESPONSE: The SAM-QFS GUI was added in SAM-QFS 4.1 to help address this issue. COMMENT: "8. Develop more robust semantics for the shared file system when issues like non-synchronized clocks occur" RESPONSE: We require customers to run NTP on all nodes in the cluster. COMMENT: "9. Provide extended attribute support." RESPONSE: We scoped this work for the QFS filesystem but found that the scope of the work for SAM was enormous. This feature was dropped from the SAM-QFS 5.0 plan. 2.0 Motivations for Changes The main motivations for the SAM-QFS 5.0 release are to: - Horizonally scale Shared QFS to support 1024 client nodes by moving filesystem space allocation from the MDS to storage nodes. Support iSCSI target object based storage devices on storage nodes with filesystem space allocation on the storage nodes done via the object interface of QFS. - Open Source the SAM-QFS software. - Continue to enhance performance for both SAM and QFS. - Continue to qualify new hardware. 3.0 Impact to User Experience No change to the user experience for installation. The upgrade process has been modernized. No change to the user experience for QFS block file systems. The QFS object file system is new, so there will be some differences from the existing QFS block file systems in the areas of installation, configuration, and administration. The primary change during normal operation for SAM is in the area of monitoring the archiver activity. Its worklist changed from a list of directories requiring scanning (in order to find files requiring examination), to the actual list of file inodes requiring examination. This change impacts how command output is formatted. For example, the showqueue(1M) and samcmd(1M) output. The SAM-QFS GUI simplifies the configuration and management of SAM-QFS, for new users. It is likely that the name of the SAM-QFS GUI will change for the SAM-QFS 5.0 release. The current name File System Manager does not reflect the primary purpose of the application which is to manage SAM-QFS. This issue has been raised with marketing. 4.0 Benefits of Changes The main benefits of SAM-QFS 5.0 changes are to: - Gain more popular adoption as the primary software for unstructured data management in a tiered storage environment. - Provide additional shared QFS scalability to address the HPC market. 5.0 Program Schedule SAM-QFS has adopted Sun's Product Life Cycle (PLC) process. SAM-QFS 5.0 is scheduled to go through the PLC Phase 1 (Concept Phase) and Phase 2 (Planning Phase) exit reviews. Specific schedule dates are not publicly available. A 'Go' decision is required for these exit reviews in order for us to continue through the remaining three PLC phases (Develop/Test/Integrate, System Test, and Customer Acceptance). 6.0 Incoming Dependencies - OpenSolaris Project: COMSTAR: Common Multiprotocol SCSI Target http://www.opensolaris.org/os/project/comstar/ - OpenSolaris Project: iSCSI Extensions for Remote DMA (iSER) URL not available yet (project proposal submitted for approval). - A release vehicle for the COMSTAR and iSER projects. - All Sun disk and tape drive/library products. 7.0 Outgoing Dependencies Sun HPC Growth Target. 8.0 Updates to Other Functional Areas 8.1 Description of Changes This project is not responsible for changing functional areas outside of SAM-QFS. 8.2 Cross-Functional-Area Communication Plans and Progress Not Applicable. 9.0 Criteria for Successful Release 9.1 Completion of Functional Specifications See SAM-QFS 5.0 feature functional specification documents at: http://www.opensolaris.org/os/project/samqfs/specs/50_Specs/ 9.2 Completion of Documentation Plan A preliminary documentation plan will be submitted for Pteam review. Once finalized, the documentation plan will be posted on the SAM-QFS 5.0 planning/release web page. 9.3 Completion of Test and Verification Strategy Testing of SAM-QFS will be performed against all the features documented in the test matrix which is based on the requirements requested in the PRD and documented in the PCD and functional specifications. 9.4 Completion of Quality Plan Details on the test plan, schedule, and test approach can be found at the URL noted above. To highlight the major test events in the schedule, the SAM-QFS 5.0 release includes two FVT test cycles, two IST (system test) cycles, a mid-point regression test cycle as well as a final QA test cycle. All testers produce weekly reports to summarize their activities. Weekly test metrics are also generated throughout the FVT and final QA test cycles to summarize the overall release test status. 9.5 Completion of Release Criteria SAM-QFS Release Criteria must be met, including build requirements, high-priority bug closures, documentation, quality assurance testing, and out-of-box experience. 9.6 Completion of PLC Process Requirements See PLC process documentation at: http://wikihome.sfbay.sun.com/apg-sam-qfs/Wiki.jsp?page=5.0Legal 10.0 Current Status Engineering is currently in the early stages of Phase 3 (Develop/Test/Integrate). The SAM-QFS GUI is currently 30% complete. A draft design has been completed and undergone a usability study. The outcomes of this study are currently being incorporated into the SAM-QFS 5.0 design and development work is underway.