From sac-owner Tue Nov 29 13:57:38 2005 Received: from sunmail2.sfbay.sun.com (sunmail2.SFBay.Sun.COM [129.149.246.180]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id jATLvcIQ006173 for ; Tue, 29 Nov 2005 13:57:38 -0800 (PST) Received: (from noaccess@localhost) by sunmail2.sfbay.sun.com (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) id jATLvcr09745 for one-pager-not-2b-used-directly; Tue, 29 Nov 2005 13:57:38 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.149.246.28]) by sunmail2.sfbay.sun.com (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id jATLvU509667; Tue, 29 Nov 2005 13:57:30 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2 (built Dec 2 2004)) id <0IQQ00607KZUSO00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 29 Nov 2005 13:57:30 -0800 (PST) Received: from brmea-mail-3.sun.com ([192.18.98.34]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2 (built Dec 2 2004)) with ESMTP id <0IQQ001GOKZUOC80@nwk-avmta-1.sfbay.Sun.COM>; Tue, 29 Nov 2005 13:57:30 -0800 (PST) Received: from fe-amer-10.sun.com ([192.18.108.184]) by brmea-mail-3.sun.com (8.12.10/8.12.9) with ESMTP id jATLvU3F020583; Tue, 29 Nov 2005 14:57:30 -0700 (MST) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-4.02 (built Sep 9 2005)) id <0IQQ00001KXMMD00@mail-amer.sun.com> (original mail from Alok.Aggarwal@Sun.COM); Tue, 29 Nov 2005 14:57:29 -0700 (MST) Received: from fomalhaut ([172.20.25.107]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-4.02 (built Sep 9 2005)) with ESMTPSA id <0IQQ00EK9KZTO7UG@mail-amer.sun.com>; Tue, 29 Nov 2005 14:57:29 -0700 (MST) Date: Tue, 29 Nov 2005 14:57:29 -0700 (MST) From: Alok Aggarwal Subject: [/]Over-the-wire Checksums for NFSv4 Sender: Alok.Aggarwal@Sun.COM X-X-Sender: aalok@fomalhaut To: one-pager@Sun.COM Cc: insight-seuss@Sun.COM Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.1.0.207369 Status: RO Content-Length: 9329 Template Version: @(#)onepager.txt 1.29 04/11/15 SMI This information is Copyright 2005 Sun Microsystems, Inc. 1. Introduction 1.1. Project/Component Working Name: Over-the-wire checksums for NFSv4 1.2. Name of Document Author/Supplier: Alok Aggarwal 1.3. Date of This Document: November 28, 2005 1.4. Name of Major Document Customer(s)/Consumer(s): 1.4.1. The PAC or CPT you expect to review your project: Solaris PAC 1.4.2. The ARC(s) you expect to review your project: PSARC 1.4.3. The Director/VP who is "Sponsoring" this project: Fred.Zlotnick@Sun.COM 1.4.4. The name of your business unit: KISS 1.5. Email Aliases: 1.5.1. Responsible Manager: don.traub@sun.com 1.5.2. Responsible Engineer: alok.aggarwal@sun.com 1.5.3. Marketing Manager: tim.graves@sun.com 1.5.4. Interest List: insight-seuss@sun.com 2. Project Summary 2.1. Project Description: The existing TCP checksums are weak and don't always provide strong enough error detection capabilities thereby allowing silent date corruption over-the-wire. Although NFS has strong integrity checking at the RPC layer, it doesn't expose it to the higher layers. This project aims to provide integrity protection at the NFS protocol layer by introducing checksums for over-the-wire operations. 2.2. Risks and Assumptions: Performance overhead of enabling checksums may very well turn out to be significant in which case it will be a hinderance to adoption. This risk may be mitigated by pushing for fast checksum algorithm implementation by the Solaris security group. Our initial testing shows the cost of checksumming is not prohibitive. Also, there's a risk that if an incompatible proposal for introducing checksums at the IETF NFSv4 WG might show up, we may have to go back and refactor that into our design. That said, till now, we haven't seen or heard of such a proposal. 3. Business Summary 3.1. Problem Area: The NFSv4 protocol runs over TCP and relies on TCP to provide data integrity. However, TCP uses a very weak checksum that doesn't provide strong enough error detection. As a result, if the data gets silently corrupted on the wire, TCP may not be able to detect it in some cases. A recent example of this is - in June 2004, the NFS group started getting frequent reports of users in the Sun Menlo Park campus encountering corrupt files while doing ON builds or bringovers over NFS. In almost all of the cases, a single bit was being flipped. The issue turned out to be a bad router that was periodically flipping bits. The TCP checksum wasn't able to detect all errors and as a result data corruption was going undetected. As NFSv4 is increasingly being used for integrating data within the data centers, it's incumbent upon the protocol to provide strong data integrity for all data it delivers to the upper layers (typically applications). This project provides checksum support at the NFSv4 layer thus making an attempt at delivering good data to the consumers. 3.2. Market/Requester: Internal to Sun users of NFSv4 as well as external customers who use NFSv4. 3.3. Business Justification: NFSv4 is increasingly being used for integrating data within the data centers and it's incumbent upon the protocol to provide strong data integrity for data transfers taking place through it. This is a first step towards providing integrity protected data to the consumers. Not doing this project means hours or days of lost productivity and downtime due to silent data corruption. 3.4. Competitive Analysis: Currently no implementation of NFSv4 provides this functionality. Depending on how the standardisation of checksums at the IETF working group goes, other implementors of NFSv4 might choose to implement checksums. 3.5. Opportunity Window/Exposure: Ideally this should be available prior to any large scale deployments of NFSv4, potentially an early update of Solaris 10. 3.6. How will you know when you are done?: a) Demonstrate interoperability with Solaris client/server with and without checksums enabled; with varying checksum algorithms. b) Demonstrate no regressions with any of the other implementors of NFSv4. c) Performance degradation with checksums enabled is acceptable. 4. Technical Description: 4.1. Details: The approach that this project will take is to explicitly add checksum related operations, OP_CKINFO and OP_CKSUM, to the compound. The mount compound will additionally carry the OP_CKINFO operation to query the server for it's preferences on checksum algorithms for the filesystem being mounted. The client will then pick an algorithm that meets the server's preferences as well as it's own. As part of reading data from the server, the client will encode the checksum algorithm information in OP_CKSUM and compound it with the OP_READ operation. In addition to returning the data, the server will compute a checksum on the data (using the algorithm specified by the client). The client will subsequently verify the checksum prior to flagging the read successful. While writing data to the server, the client will encode the actual checksum value on the data being written alongwith the algorithm information in OP_CKSUM and compound it with the OP_WRITE operation. There exists no standard protocol mechanism to implement data integrity in NFSv4 which limits this project to be Solaris only. As a follow-on project work, efforts will be made to integrate this solution into the other vendor's NFSv4 implementations as well as making it a standard part of the IETF NFSv4 protocol. 4.2. Bug/RFE Number(s): An RFE still needs to be filed to cover this project. 4.3. In Scope: Whenever an NFSv4 mount has integrity protection enabled via krb5i, optimisations will be done to eliminate double checksumming once at the NFS layer and then at the RPC layer. The design of this project will account for any future enhancements (for example, in the form of integrating ZFS/NFSv4 checksums) that might need to be made. We'll make sure the design doesn't limit future work. 4.4. Out of Scope: This project will not look at adding checksums for the entire NFSv4 payload. That will be accomplished by a followon project. Also, at this time, we will not be integrating the NFSv4 checksums with ZFS checksums. That will be accomplished by a followon project as well. 4.5. Interfaces: The nfs admin commands will be extended to support the 'cksum' options. A new file will be introduced in /etc/nfs that will contain a name to number mapping for checksum algorithms. 4.6. Doc Impact: Man pages that will need to be modified are those of mount_nfs(1M) share (1M). 4.7. Admin/Config Impact: This project won't change the way NFS is administered today. It will only extend it to choose integrity protection modes. 4.8. HA Impact: Checksums will need to be renegotiated in case a failover occurs - the implementation will ensure that happens. HA will continue to work seamlessly otherwise just as it does today. 4.9. I18N/L10N Impact: All affected commands will change their messages. 4.10. Packaging & Delivery: The is no impact on Install/Upgrade - the design will account for seamless upgrades on the NFS server. Packages affected are - SUNWnfscr (etc/default/nfs) SUNWnfscu (nfs commands) SUNWnfsckr (nfs, nfsd kernel modules) 4.11. Security Impact: This project does not expose any new security issues. 4.12. Dependencies: None 5. Reference Documents: Solaris NFSv4 Phase I (PSARC 1999/289) NFSv4 RFC: RFC 3530 Performance of Checksums and CRC's over real data, J. Stone M. Greenwald, C. Partridge and J. Hughes 6. Resources and Schedule: 6.1. Projected Availability: February 2006 6.2. Cost of Effort: One full-time development engineer (4 months) One half-time test engineer (2 months) One part-time docs person 6.3. Cost of Capital Resources: Shared lab resources for test and development will be used. 6.4. Product Approval Committee requested information: 6.4.1. Consolidation or Component Name: ON 6.4.3. Type of CPT Review and Approval expected: Standard 6.4.4. Project Boundary Conditions: I think this is no more with the ONSC gone. 6.4.5. Is this a necessary project for OEM agreements: No 6.4.6. Notes: None 6.4.7. Target RTI Date/Release: February 20, 2006 (snv_35) 6.4.8. Target Code Design Review Date: End of December, 2005 6.4.9. Update approval addition: This project does not have Solaris PAC approval for an update release but I would like for this project to make it into an S10 update if at all possible. 6.5. ARC review type: Standard 7. Prototype Availability: 7.1. Prototype Availability: A prototype that implements a subset of the functionality is available now. 7.2. Prototype Cost: The rest of the prototype will take almost 1 more month with one full-time engineer.