From dwc@sac.sfbay.sun.com Fri Nov 21 15:39:49 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mALNdmn0014437 for ; Fri, 21 Nov 2008 15:39:49 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mALNdhLR006612; Fri, 21 Nov 2008 23:39:47 GMT Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAP00I0FJ29OB00@nwk-avmta-2.sfbay.sun.com>; Fri, 21 Nov 2008 15:39:45 -0800 (PST) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAP004DVJ28C4F0@nwk-avmta-2.sfbay.sun.com>; Fri, 21 Nov 2008 15:39:44 -0800 (PST) Received: from sac.sfbay.sun.com (new-sac.SFBay.Sun.COM [129.146.175.65]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mALNdhdw031269; Fri, 21 Nov 2008 15:39:43 -0800 (PST) Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mALNdgB8014432; Fri, 21 Nov 2008 15:39:42 -0800 (PST) Received: (from dwc@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id mALNdgOO014428; Fri, 21 Nov 2008 15:39:42 -0800 (PST) Date: Fri, 21 Nov 2008 15:39:42 -0800 (PST) From: Don Cragun Subject: Add sparse file support to cpio [PSARC/2008/727 Self Review] To: PSARC-ext@sun.com Cc: Cynthia.Eastham@sun.com Message-id: <200811212339.mALNdgOO014428@sac.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 5123 I'm sponsoring this case for Cynthia Eastham. Since this case follows the same general practices used when sparse file support was added to the pax archiving utility, I'm marking this case as close approved automatic. If any members believe this needs to be promoted to a fast track let me know. This case seeks a patch binding. - Don Template Version: @(#)sac_nextcase %I% %G% SMI This information is Copyright 2008 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: Add sparse file support to cpio 1.2. Name of Document Author/Supplier: Author: Cynthia Eastham 1.3 Date of This Document: 21 November, 2008 4. Technical Description 4.1 Details PSARC case 2006/331 (Add holey file support to pax) created extended headers to maintain information about sparse files in some archive formats written by pax. Holes are not expunged from the file in the archive, but are recreated when the file is extracted from the archive. This case adds similar sparse file support to the cpio utility. In pass mode, (cpio -p), sparse files will be recreated at the destination with the same holes that were present in the source file, as long as the source file system supports reporting holes (as described by PSARC case 2004/770) and the destination file is seekable. Otherwise, holes in sparse files will be filled with '\0' btyes in corresponding destination files as they are now. In copy out mode (-o) the following new option arguments to the cpio -H option will be added to provide sparse file support: ascii_sparse - assumes -c is specified. Only available in copy out (-o) mode. odc_sparse - assumes -H odc is specified Only available in copy out (-o) mode. When sparse files are archived using one of the above archive formats, and the underlying file system supports the detection of holes as reported by pathconf(2), it will be compressed (holes will not be stored in the archive), and hole and data offsets as returned by lseek(2) will be saved at the start of the file data. Otherwise, sparse files will be archived as they are now. When a compressed sparse file is extracted from an archive (cpio -i), if the destination file is seekable, holes will be restored using the saved hole and data offsets. Otherwise holes in the source file will be filled with '\0' bytes in the destination file. The following will apply when either '-H ascii_sparse' or '-H odc_sparse' is specified with -o: - The c_mode field will in the archive header will indicate that the file is a sparse file. In the old stat structure, the mode field is an unsigned short (16 bit) field. To avoid conflicts with other file types, a high order bit (17) in the c_mode field of the header will be set. - the file size field of the header will be the size of the compressed sparse file (i.e., the size of the header below plus the size of the file contents after removing the holes). - A string of the following format will be prepended to the compressed file data: "%lu %llu%s", prepended_info_size, expanded_file_size, data/hole_offsets where data/hole_offsets contains 2 or more entries of the following format: " %llu %llu", data_offset, hole_offset The following is an example of the string that would be prepended to the compressed file: "64 16777223 0 8192 65536 73728 1048576 1056768 16777216 16777223" for a file that is (after expansion) 16,777,223 bytes long with file data starting at offset 0 followed by a hole starting at offset 8,192, more data starting at offset 65,536 followed by another hole starting at offset 1,048,576, ..., followed by data starting at offset 26,777,216 and ending at offset 16,777,223 (the file size). The compressed file is all of the data in the file, except that the holes in the file have been removed from the archive. When the c_mode field is set, cpio will detect the sparse file upon file extraction, and use the prepended sparse file information to restore the holes in the file if the destiination file is seekable. If the destination file is not seekable, the sparse file information will be used to fill the holes with '\0' bytes. Archivers that do not recognize the sparse file mode bit will restore the compressed file and its prepended data as a regular file. 4.3 Exported Interfaces ______________________________________ | Interface |Classification| |_____________________|______________| |/usr/bin/cpio | Committed | |_____________________|______________| 5. Reference Documents: 5.1 CR 4480319 cpio does not properly copy files that are sparse (have holes in them) 5.2 PSARC/2004/770: Holey file knowledge 5.3 PSARC/2006/361: Add holey file support to pax 5.4 PSARC/2008/727/materials/cpio.1: Updated cpio.1 man page 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: Automatic 6.6. ARC Exposure: open From Garrett.Damore@sun.com Fri Nov 21 16:07:55 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAM07sUi017385 for ; Fri, 21 Nov 2008 16:07:54 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id mAM07i70003051; Sat, 22 Nov 2008 08:07:51 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAP00G1XKD2I700@nwk-avmta-1.sfbay.Sun.COM>; Fri, 21 Nov 2008 16:07:50 -0800 (PST) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAP00DIXKD0NG90@nwk-avmta-1.sfbay.Sun.COM>; Fri, 21 Nov 2008 16:07:48 -0800 (PST) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAM07muH018600; Fri, 21 Nov 2008 16:07:48 -0800 (PST) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAP00C01K4HTY00@fe-sfbay-10.sun.com> (original mail from Garrett.Damore@Sun.COM); Fri, 21 Nov 2008 16:07:48 -0800 (PST) Received: from [129.153.2.8] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAP002U0KCZ1JB0@fe-sfbay-10.sun.com>; Fri, 21 Nov 2008 16:07:48 -0800 (PST) Date: Fri, 21 Nov 2008 16:07:47 -0800 From: "Garrett D'Amore" Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <200811212339.mALNdgOO014428@sac.sfbay.sun.com> Sender: Garrett.Damore@sun.com To: Don Cragun Cc: PSARC-ext@sun.com, Cynthia.Eastham@sun.com Message-id: <49274D53.2000504@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200811212339.mALNdgOO014428@sac.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.16 (X11/20080807) Status: RO Content-Length: 6584 A few questions, which probably mean that this case should be promoted to a fast track. 1) Given that this requires new flags to be added (as well as creates a new file format), is there a compelling reason to add these flags rather than just telling folks who want holey file support to use pax? (I guess this is a level 0 question... it seems like pax does everything cpio does, only more?) 2) (This is probably stylistic.) Why two different flags (ascii_sparse and odc_sparse)? Couldn't you just add one new flag for sparse support, and then figure out whether ascii or odc by the presence of the -H odc flag? 3) My interpretation of the last sentence in section 4 ("Archivers that do not recognize the sparse file mode bit will restore the compressed file and its prepended data as a regular file.") is that the resulting destination file will be quite different from the source -- it will not have holes, but neither will it be zero filled, and it will have the prepended information string stored instead. Correct? 4) Is there any other convention or standard for holey cpio archives in common use? (E.g. something supported by Linux or star?) -- Garrett On 11/21/08 15:39, Don Cragun wrote: > I'm sponsoring this case for Cynthia Eastham. > > Since this case follows the same general practices used when sparse > file support was added to the pax archiving utility, I'm marking this > case as close approved automatic. If any members believe this needs to > be promoted to a fast track let me know. > > This case seeks a patch binding. > > - Don > > Template Version: @(#)sac_nextcase %I% %G% SMI > This information is Copyright 2008 Sun Microsystems > 1. Introduction > 1.1. Project/Component Working Name: > Add sparse file support to cpio > 1.2. Name of Document Author/Supplier: > Author: Cynthia Eastham > 1.3 Date of This Document: > 21 November, 2008 > 4. Technical Description > 4.1 Details > > PSARC case 2006/331 (Add holey file support to pax) created > extended headers to maintain information about sparse files in > some archive formats written by pax. Holes are not expunged > from the file in the archive, but are recreated when the file > is extracted from the archive. > > This case adds similar sparse file support to the cpio utility. > > In pass mode, (cpio -p), sparse files will be recreated at the > destination with the same holes that were present in the source > file, as long as the source file system supports reporting > holes (as described by PSARC case 2004/770) and the destination > file is seekable. Otherwise, holes in sparse files will be > filled with '\0' btyes in corresponding destination files as > they are now. > > In copy out mode (-o) the following new option arguments to the > cpio -H option will be added to provide sparse file support: > ascii_sparse - assumes -c is specified. Only available > in copy out (-o) mode. > odc_sparse - assumes -H odc is specified Only available > in copy out (-o) mode. > > When sparse files are archived using one of the above archive > formats, and the underlying file system supports the detection > of holes as reported by pathconf(2), it will be compressed > (holes will not be stored in the archive), and hole and data > offsets as returned by lseek(2) will be saved at the start of > the file data. Otherwise, sparse files will be archived as > they are now. > > When a compressed sparse file is extracted from an archive > (cpio -i), if the destination file is seekable, holes will be > restored using the saved hole and data offsets. Otherwise > holes in the source file will be filled with '\0' bytes in the > destination file. > > The following will apply when either '-H ascii_sparse' or > '-H odc_sparse' is specified with -o: > - The c_mode field will in the archive header will > indicate that the file is a sparse file. In the old > stat structure, the mode field is an unsigned short > (16 bit) field. To avoid conflicts with other file > types, a high order bit (17) in the c_mode field of > the header will be set. > - the file size field of the header will be the size of > the compressed sparse file (i.e., the size of the > header below plus the size of the file contents after > removing the holes). > - A string of the following format will be prepended to > the compressed file data: > "%lu %llu%s", prepended_info_size, > expanded_file_size, data/hole_offsets > where data/hole_offsets contains 2 or more entries of the > following format: > " %llu %llu", data_offset, hole_offset > > The following is an example of the string that would be > prepended to the compressed file: > "64 16777223 0 8192 65536 73728 1048576 1056768 16777216 16777223" > for a file that is (after expansion) 16,777,223 bytes > long with file data starting at offset 0 followed by a > hole starting at offset 8,192, more data starting at > offset 65,536 followed by another hole starting at > offset 1,048,576, ..., followed by data starting at > offset 26,777,216 and ending at offset 16,777,223 (the > file size). The compressed file is all of the data in > the file, except that the holes in the file have been > removed from the archive. > > When the c_mode field is set, cpio will detect the sparse file > upon file extraction, and use the prepended sparse file > information to restore the holes in the file if the > destiination file is seekable. If the destination file is not > seekable, the sparse file information will be used to fill the > holes with '\0' bytes. Archivers that do not recognize the > sparse file mode bit will restore the compressed file and its > prepended data as a regular file. > > 4.3 Exported Interfaces > ______________________________________ > | Interface |Classification| > |_____________________|______________| > |/usr/bin/cpio | Committed | > |_____________________|______________| > > 5. Reference Documents: > 5.1 CR 4480319 cpio does not properly copy files that are > sparse (have holes in them) > 5.2 PSARC/2004/770: Holey file knowledge > 5.3 PSARC/2006/361: Add holey file support to pax > 5.4 PSARC/2008/727/materials/cpio.1: Updated cpio.1 man page > > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > ON > 6.5. ARC review type: Automatic > 6.6. ARC Exposure: open > > From dwc@spartan.eng.sun.com Fri Nov 21 21:37:22 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAM5bMI8019864 for ; Fri, 21 Nov 2008 21:37:22 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAM5bLMr025954 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@Sun.COM>; Fri, 21 Nov 2008 21:37:22 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAP00J0BZM90800@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Fri, 21 Nov 2008 21:37:21 -0800 (PST) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAP004AYZM9X350@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Fri, 21 Nov 2008 21:37:21 -0800 (PST) Received: from spartan.eng.sun.com (spartan.SFBay.Sun.COM [129.146.226.64]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAM5bEva020627; Fri, 21 Nov 2008 21:37:14 -0800 (PST) Received: from spartan.eng.sun.com (localhost [127.0.0.1]) by spartan.eng.sun.com (8.13.7+Sun/8.13.7) with ESMTP id mAM5TjCP001532; Fri, 21 Nov 2008 21:29:45 -0800 (PST) Received: (from dwc@localhost) by spartan.eng.sun.com (8.13.7+Sun/8.13.7/Submit) id mAM5TilE001531; Fri, 21 Nov 2008 21:29:44 -0800 (PST) Date: Fri, 21 Nov 2008 21:29:44 -0800 (PST) From: Don Cragun Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] To: Garrett.Damore@sun.com, don.cragun@sun.com Cc: Cynthia.Eastham@sun.com, PSARC-ext@sun.com Message-id: <200811220529.mAM5TilE001531@spartan.eng.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 3524 Garrett, Please find comments in-line below... -Don >Date: Fri, 21 Nov 2008 16:07:47 -0800 >From: "Garrett D'Amore" > >A few questions, which probably mean that this case should be promoted >to a fast track. I have seen several automatic cases that have had a few questions asked and answered without changing status. If you aren't satisfied with the responses you get to your questions, I'll change the status. > >1) Given that this requires new flags to be added (as well as creates a >new file format), is there a compelling reason to add these flags rather >than just telling folks who want holey file support to use pax? (I >guess this is a level 0 question... it seems like pax does everything >cpio does, only more?) There are no new options (aka flags); there are just new values for the -H option arguement. These changes to cpio are to fix an escalated bug report from the Live Upgrade project. The submitter insists on using cpio; not pax. The project team would be happy to deprecate tar and cpio, and to use pax as a replacement, but tar and cpio provide a few features that aren't yet duplicated by pax and the project team's priorities are to work on escalations and porting packages to Nevada; not to improve infrastructure of existing utilities. > >2) (This is probably stylistic.) Why two different flags (ascii_sparse >and odc_sparse)? Couldn't you just add one new flag for sparse support, >and then figure out whether ascii or odc by the presence of the -H odc flag? The number of available option letters (or flags) that aren't used by Solaris cpio, GNU cpio, and star is extremely limited. And, we try to avoid adding options that use the same option letter with different meanings between the various archivers. (We try, but we don't always succeed. Here it was easy to use -H option arguments to identify different archive formats rather than add a conflicting option.) > >3) My interpretation of the last sentence in section 4 ("Archivers that >do not recognize the sparse file mode bit will restore the compressed >file and its prepended data as a regular file.") is that the resulting >destination file will be quite different from the source -- it will not >have holes, but neither will it be zero filled, and it will have the >prepended information string stored instead. Correct? Yes. The additional mode bit makes it an unrecognized file type for archivers that don't know what is going on. The standards require that when an unrecognized file type is found, the data be stored as a regular file. With the data in the file, the original file can be reconstituted. (I should say that the standards did require this, cpio and tar were in XPG3 and SVID3; but were replaced by pax in POSIX.2 back in 1992. Our tar and cpio still meet the requirements of those older standards.) > >4) Is there any other convention or standard for holey cpio archives in >common use? (E.g. something supported by Linux or star?) This was discussed while we were working on PSARC/2006/361. Some other archivers have a --sparse long option that converts sequences of zero bytes into holes. The ARC gave explicit direction to the project team that archivers should "preserve" existing holes, not try to reduce disk usage by "creating" holes. None of the other archivers (except for our pax) have the ability to preserve existing holes in sparse files. I see no reason why that advice should be true for pax, but not for cpio. > > -- Garrett From Garrett.Damore@sun.com Sat Nov 22 09:35:12 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAMHYvMl017583 for ; Sat, 22 Nov 2008 09:35:12 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAMHYhQd024216; Sat, 22 Nov 2008 10:34:46 -0700 (MST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAQ00C01WTP6S00@brm-avmta-1.central.sun.com>; Sat, 22 Nov 2008 10:34:37 -0700 (MST) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAQ00FS6WTOSC90@brm-avmta-1.central.sun.com>; Sat, 22 Nov 2008 10:34:37 -0700 (MST) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAMHYadJ011700; Sat, 22 Nov 2008 09:34:36 -0800 (PST) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAQ00101WQXMP00@fe-sfbay-10.sun.com> (original mail from Garrett.Damore@Sun.COM); Sat, 22 Nov 2008 09:34:36 -0800 (PST) Received: from [192.168.251.11] ([76.93.15.33]) by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAQ006DQWTN6ED0@fe-sfbay-10.sun.com>; Sat, 22 Nov 2008 09:34:36 -0800 (PST) Date: Sat, 22 Nov 2008 09:28:30 -0800 From: "Garrett D'Amore" Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <200811220529.mAM5TilE001531@spartan.eng.sun.com> Sender: Garrett.Damore@sun.com To: Don Cragun Cc: Cynthia.Eastham@sun.com, PSARC-ext@sun.com Reply-to: Garrett.Damore@sun.com Message-id: <4928413E.40900@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200811220529.mAM5TilE001531@spartan.eng.sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 4023 I'm satisfied with your responses (but slightly saddened that we have to spend effort into doing this when pax -- the POSIX standard archiver -- could probably be used instead.) You can leave the case as automatic approval unless someone else has other questions. -- Garrett Don Cragun wrote: > Garrett, > Please find comments in-line below... > > -Don > > >> Date: Fri, 21 Nov 2008 16:07:47 -0800 >> From: "Garrett D'Amore" >> >> A few questions, which probably mean that this case should be promoted >> to a fast track. >> > > I have seen several automatic cases that have had a few questions asked > and answered without changing status. If you aren't satisfied with the > responses you get to your questions, I'll change the status. > > >> 1) Given that this requires new flags to be added (as well as creates a >> new file format), is there a compelling reason to add these flags rather >> than just telling folks who want holey file support to use pax? (I >> guess this is a level 0 question... it seems like pax does everything >> cpio does, only more?) >> > > There are no new options (aka flags); there are just new values for the > -H option arguement. > > These changes to cpio are to fix an escalated bug report from the Live > Upgrade project. The submitter insists on using cpio; not pax. The > project team would be happy to deprecate tar and cpio, and to use pax > as a replacement, but tar and cpio provide a few features that aren't > yet duplicated by pax and the project team's priorities are to work on > escalations and porting packages to Nevada; not to improve > infrastructure of existing utilities. > > >> 2) (This is probably stylistic.) Why two different flags (ascii_sparse >> and odc_sparse)? Couldn't you just add one new flag for sparse support, >> and then figure out whether ascii or odc by the presence of the -H odc flag? >> > > The number of available option letters (or flags) that aren't used by > Solaris cpio, GNU cpio, and star is extremely limited. And, we try to > avoid adding options that use the same option letter with different > meanings between the various archivers. (We try, but we don't always > succeed. Here it was easy to use -H option arguments to identify > different archive formats rather than add a conflicting option.) > > >> 3) My interpretation of the last sentence in section 4 ("Archivers that >> do not recognize the sparse file mode bit will restore the compressed >> file and its prepended data as a regular file.") is that the resulting >> destination file will be quite different from the source -- it will not >> have holes, but neither will it be zero filled, and it will have the >> prepended information string stored instead. Correct? >> > > Yes. The additional mode bit makes it an unrecognized file type for > archivers that don't know what is going on. The standards require that > when an unrecognized file type is found, the data be stored as a regular > file. With the data in the file, the original file can be > reconstituted. (I should say that the standards did require this, cpio > and tar were in XPG3 and SVID3; but were replaced by pax in POSIX.2 back > in 1992. Our tar and cpio still meet the requirements of those older > standards.) > > >> 4) Is there any other convention or standard for holey cpio archives in >> common use? (E.g. something supported by Linux or star?) >> > > This was discussed while we were working on PSARC/2006/361. Some other > archivers have a --sparse long option that converts sequences of zero > bytes into holes. The ARC gave explicit direction to the project team > that archivers should "preserve" existing holes, not try to reduce disk > usage by "creating" holes. None of the other archivers (except for our > pax) have the ability to preserve existing holes in sparse files. I > see no reason why that advice should be true for pax, but not for cpio. > > >> -- Garrett >> From Joerg.Schilling@fokus.fraunhofer.de Sun Nov 23 06:01:54 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mANE1qe4024938 for ; Sun, 23 Nov 2008 06:01:53 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id mANE1jC6022142; Sun, 23 Nov 2008 22:01:47 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAS00403HMYPO00@brm-avmta-1.central.sun.com>; Sun, 23 Nov 2008 07:01:46 -0700 (MST) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAS00JXRHMYFGC0@brm-avmta-1.central.sun.com>; Sun, 23 Nov 2008 07:01:46 -0700 (MST) Received: from relay24.sun.com (relay24.sun.com [192.12.251.74] (may be forged)) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mANE1i37010826; Sun, 23 Nov 2008 14:01:46 +0000 (GMT) Received: from mms22es.mms.us.syntegra.com ([150.143.232.30] [150.143.232.30]) by relay24i.sun.com with ESMTP id BT-MMP-133113; Sun, 23 Nov 2008 14:01:44 +0000 (Z) Received: from relay21.sun.com (relay21.sun.com [192.12.251.24]) by mms22es.mms.us.syntegra.com with ESMTP id BT-MMP-1689390; Sun, 23 Nov 2008 14:01:44 +0000 (Z) Received: from iron01.fraunhofer.de ([153.96.1.54] [153.96.1.54]) by relay21i.sun.com with ESMTP id BT-MMP-2168013; Sun, 23 Nov 2008 14:01:43 +0000 (Z) Received: from pluto.fokus.fraunhofer.de ([195.37.77.164]) by iron01.fraunhofer.de with ESMTP/TLS/DHE-RSA-AES256-SHA; Sun, 23 Nov 2008 15:01:42 +0100 Received: from EXCHSRV.fokus.fraunhofer.de (bohr [10.147.9.231]) by pluto.fokus.fraunhofer.de (8.13.7/8.13.7) with SMTP id mANE1flP007341; Sun, 23 Nov 2008 15:01:42 +0100 (MET) Received: from rigel ([10.147.65.195]) by EXCHSRV.fokus.fraunhofer.de with Microsoft SMTPSVC(6.0.3790.3959); Sun, 23 Nov 2008 15:01:42 +0100 Date: Sun, 23 Nov 2008 15:01:41 +0100 From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <200811212339.mALNdgOO014428@sac.sfbay.sun.com> To: PSARC-ext@sun.com, don.cragun@sun.com Cc: Cynthia.Eastham@sun.com Message-id: <49296245.CrsPZ9GbtigzjSJv%Joerg.Schilling@fokus.fraunhofer.de> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=-2.6/5.0, scanned in 0.289sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200811212339.mALNdgOO014428@sac.sfbay.sun.com> User-Agent: nail 11.22 3/20/05 X-OriginalArrivalTime: 23 Nov 2008 14:01:42.0209 (UTC) FILETIME=[03348B10:01C94D74] Status: RO Content-Length: 4837 Don Cragun wrote: > I'm sponsoring this case for Cynthia Eastham. > > Since this case follows the same general practices used when sparse > file support was added to the pax archiving utility, I'm marking this ??? AFAIK, the "pax" implementation that comes with Solaris does not support sparse files. > case as close approved automatic. If any members believe this needs to > be promoted to a fast track let me know. I see many small problems that need to be adressed before an implementation starts. > Template Version: @(#)sac_nextcase %I% %G% SMI > This information is Copyright 2008 Sun Microsystems > 1. Introduction > 1.1. Project/Component Working Name: > Add sparse file support to cpio > 1.2. Name of Document Author/Supplier: > Author: Cynthia Eastham > 1.3 Date of This Document: > 21 November, 2008 > 4. Technical Description > 4.1 Details > > PSARC case 2006/331 (Add holey file support to pax) created I did never see such a case and the Sun pax man page does neither include "hole" nor "sparse". > This case adds similar sparse file support to the cpio utility. Similar to what? > In pass mode, (cpio -p), sparse files will be recreated at the > destination with the same holes that were present in the source > file, as long as the source file system supports reporting > holes (as described by PSARC case 2004/770) and the destination > file is seekable. Otherwise, holes in sparse files will be > filled with '\0' btyes in corresponding destination files as > they are now. How do you intend to switch between the sparse support mode and the non-sparse mode in "copy mode"? > In copy out mode (-o) the following new option arguments to the > cpio -H option will be added to provide sparse file support: > ascii_sparse - assumes -c is specified. Only available > in copy out (-o) mode. > odc_sparse - assumes -H odc is specified Only available > in copy out (-o) mode. Adding sparse file support does not introduce a new archive format unless you create a new archive format that may be detected by reading the first archive header from a random archive. If you like to avoid to to introduce a new option, you would need to document this as a dirty hack. BTW: Where is the new man page? ... > The following will apply when either '-H ascii_sparse' or > '-H odc_sparse' is specified with -o: > - The c_mode field will in the archive header will > indicate that the file is a sparse file. In the old > stat structure, the mode field is an unsigned short > (16 bit) field. To avoid conflicts with other file > types, a high order bit (17) in the c_mode field of > the header will be set. This is beyond the cpio specs. How do you plan to mark the archives as "Sun cpio" specific to allow to avoid incorrect behavior for non-Sun archives? > - the file size field of the header will be the size of > the compressed sparse file (i.e., the size of the > header below plus the size of the file contents after > removing the holes). OK > - A string of the following format will be prepended to > the compressed file data: > "%lu %llu%s", prepended_info_size, > expanded_file_size, data/hole_offsets Is this data _inside_ the file data area or is it in conflict with the cpio extensions from David Korn and Glenn Fowler? > where data/hole_offsets contains 2 or more entries of the > following format: > " %llu %llu", data_offset, hole_offset If you ever like to debug this, I would recommend to use: " %llu,%llu", data_offset, hole_offset to make the data parsable by human eyes.. But why don't you follow existing other implementations that use offset/numbytes pairs for data chunks? This results in a lower archive size. > When the c_mode field is set, cpio will detect the sparse file > upon file extraction, and use the prepended sparse file > information to restore the holes in the file if the > destiination file is seekable. If the destination file is not > seekable, the sparse file information will be used to fill the > holes with '\0' bytes. Archivers that do not recognize the > sparse file mode bit will restore the compressed file and its > prepended data as a regular file. As it is unlikely that the first file in an archive is a sparse file, how do you intend to detect an archive that contain this Sun specific cpio extension? How do you intend to switch between the sparse support mode and the non-sparse mode in "extract mode"? Jörg -- EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin js@cs.tu-berlin.de (uni) schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily From Joerg.Schilling@fokus.fraunhofer.de Sun Nov 23 07:14:51 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mANFEpg4027461 for ; Sun, 23 Nov 2008 07:14:51 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mANFEmCb009543; Sun, 23 Nov 2008 07:14:50 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAS00C0TL0PMH00@brm-avmta-1.central.sun.com>; Sun, 23 Nov 2008 08:14:49 -0700 (MST) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAS00C2ML0OCF00@brm-avmta-1.central.sun.com>; Sun, 23 Nov 2008 08:14:49 -0700 (MST) Received: from relay11i.sun.com (ip121.net129179-4.block1.us.syntegra.com [129.179.4.121]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mANFEmbZ001971; Sun, 23 Nov 2008 15:14:48 +0000 (GMT) Received: from mmp11es.mmp.us.syntegra.com ([160.41.208.11] [160.41.208.11]) by relay11i.sun.com with ESMTP id BT-MMP-185419; Sun, 23 Nov 2008 15:14:48 +0000 (Z) Received: from relay18i.sun.com (relay18i.sun.com [129.179.4.128]) by mmp11es.mmp.us.syntegra.com with ESMTP id BT-MMP-3352314; Sun, 23 Nov 2008 15:14:47 +0000 (Z) Received: from iron01.fraunhofer.de ([153.96.1.54] [153.96.1.54]) by relay1i.sun.com with ESMTP id BT-MMP-9051875; Sun, 23 Nov 2008 15:14:47 +0000 (Z) Received: from pluto.fokus.fraunhofer.de ([195.37.77.164]) by iron01.fraunhofer.de with ESMTP/TLS/DHE-RSA-AES256-SHA; Sun, 23 Nov 2008 16:14:38 +0100 Received: from EXCHSRV.fokus.fraunhofer.de (bohr [10.147.9.231]) by pluto.fokus.fraunhofer.de (8.13.7/8.13.7) with SMTP id mANFEbE3008224; Sun, 23 Nov 2008 16:14:37 +0100 (MET) Received: from rigel ([10.147.65.195]) by EXCHSRV.fokus.fraunhofer.de with Microsoft SMTPSVC(6.0.3790.3959); Sun, 23 Nov 2008 16:14:38 +0100 Date: Sun, 23 Nov 2008 16:14:37 +0100 From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <49274D53.2000504@sun.com> To: Garrett.Damore@sun.com, don.cragun@sun.com Cc: PSARC-ext@sun.com, Cynthia.Eastham@sun.com Message-id: <4929735d.7heAfmijbOyE/9ji%Joerg.Schilling@fokus.fraunhofer.de> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=0.0/5.0, scanned in 0.203sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200811212339.mALNdgOO014428@sac.sfbay.sun.com> <49274D53.2000504@sun.com> User-Agent: nail 11.22 3/20/05 X-OriginalArrivalTime: 23 Nov 2008 15:14:38.0024 (UTC) FILETIME=[3364EC80:01C94D7E] Status: RO Content-Length: 1872 "Garrett D'Amore" wrote: > 1) Given that this requires new flags to be added (as well as creates a > new file format), is there a compelling reason to add these flags rather > than just telling folks who want holey file support to use pax? (I > guess this is a level 0 question... it seems like pax does everything > cpio does, only more?) According to the man page, Sun pax does not support sparse files. > 4) Is there any other convention or standard for holey cpio archives in > common use? (E.g. something supported by Linux or star?) The cpio archive formats (there are at least 3 completely incompatible cpio archive formats) in general are hard to extend. This was the main reason for POSIX to give up cpio based formats for future archive formats. The older cpio formats are limited to a max. file size of 8 GB, the SVr4 based formats are limited to 4 GB per file. Star supports sparse files in tar based archives and tar based archives are also used for the POSIX.1-2001 extended archive format. If you like to implement things like sparse files, you in general need to take specific care about implementation details of the archivers. While it is easy to implement many things in simple archivers like Sun tar, Sun pax, Sun cpio or GNU tar, it may be hard or slow to implement the same in a fully buffered archiver implementation like star. For this reason, I designed all star extensions in a way that no data spans over more than one 512 byte data block. The current archive format proposal does not follow this rule. Jörg -- EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin js@cs.tu-berlin.de (uni) schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily From dwc@spartan.eng.sun.com Sun Nov 23 21:57:29 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAO5vTor011601 for ; Sun, 23 Nov 2008 21:57:29 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAO5vSbW042248 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Sun, 23 Nov 2008 22:57:28 -0700 (MST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAT00L0FPVSHJ00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 23 Nov 2008 22:57:28 -0700 (MST) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAT00IUHPVSL020@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 23 Nov 2008 22:57:28 -0700 (MST) Received: from spartan.eng.sun.com (spartan.SFBay.Sun.COM [129.146.226.64]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAO5vLVu052938; Sun, 23 Nov 2008 21:57:21 -0800 (PST) Received: from spartan.eng.sun.com (localhost [127.0.0.1]) by spartan.eng.sun.com (8.13.7+Sun/8.13.7) with ESMTP id mAO5gMxL009147; Sun, 23 Nov 2008 21:42:22 -0800 (PST) Received: (from dwc@localhost) by spartan.eng.sun.com (8.13.7+Sun/8.13.7/Submit) id mAO5gM5t009146; Sun, 23 Nov 2008 21:42:22 -0800 (PST) Date: Sun, 23 Nov 2008 21:42:22 -0800 (PST) From: Don Cragun Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] To: Joerg.Schilling@fokus.fraunhofer.de Cc: Cynthia.Eastham@sun.com, PSARC-ext@sun.com Message-id: <200811240542.mAO5gM5t009146@spartan.eng.sun.com> MIME-version: 1.0 Content-type: TEXT/PLAIN Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 9639 >Date: Sun, 23 Nov 2008 15:01:41 +0100 >From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) > >Don Cragun wrote: > >> I'm sponsoring this case for Cynthia Eastham. >> >> Since this case follows the same general practices used when sparse >> file support was added to the pax archiving utility, I'm marking this > >??? > >AFAIK, the "pax" implementation that comes with Solaris does not support >sparse files. It does. > >> case as close approved automatic. If any members believe this needs to >> be promoted to a fast track let me know. > >I see many small problems that need to be adressed before an implementation >starts. > >> Template Version: @(#)sac_nextcase %I% %G% SMI >> This information is Copyright 2008 Sun Microsystems >> 1. Introduction >> 1.1. Project/Component Working Name: >> Add sparse file support to cpio >> 1.2. Name of Document Author/Supplier: >> Author: Cynthia Eastham >> 1.3 Date of This Document: >> 21 November, 2008 >> 4. Technical Description >> 4.1 Details >> >> PSARC case 2006/331 (Add holey file support to pax) created > >I did never see such a case and the Sun pax man page does neither >include "hole" nor "sparse". The case was approved before PSARC cases were handled in cases open to people who are not Sun employees. But, all of the important information is included on Sun's current pax(1) man page. The pax utility added an extended header to ustar and pax archive format archives as described in the USAGE section, where it says: "When using the -x xustar and -x -pax archive formats, if the underlying file system reports that the file being archived contains holes, the Solaris pax utility records the presence of holes in an extended header record when the file is archived. If this extended header record is associated with a file in the archive, those holes are recreated whenever that file is extracted from the archive. See the SEEK_DATA and SEEK_HOLE whence values in lseek(2). In all other cases, any NUL (\0) characters found in the archive is written to the file when it is extracted." and in the EXTENDED DESCRIPTION, where it says: "SUN.holesdata A Solaris extension to pax extended header keywords. Specifies the data and hole pairs for a sparse file. "In write or copy modes and when the xustar or pax format (see -x format) is specified, pax includes a SUN.holesdate extended header record if the underlying file system supports the detection of files with holes (see fpathconf(2)) and reports that there is at least one hole in the file being archived. value consists of two or more consecutive entries of the following form: SPACEdata_offsetSPACEhole_offset "where the data and hole offsets are the long values returned by passing SEEK_DATA and SEEK_HOLE to lseek(2), respectively. For example, the following entry is an example of the SUN.holesdata entry in the extended header for a file with data offsets at bytes 0, 24576, and 49152, and hole offsets at bytes 8192, 32768, and 49159: 49 SUN.holesdata= 0 8192 24576 32768 49152 49159: 49 SUN.holesdata= 0 8192 24576 32768 49152 49159 "When extracting a file from an archive in read or copy modes, if a SUN.holesdata = pair is found in the extended header for the file, then the file is restored with the holes identified using this data. For example, for the SUN.holesdata provided in the example above, bytes from 0 to 8192 are restored as data, a hole is created up to the next data position (24576), bytes 24576 to 32768 is restored as data, and so forth." > >> This case adds similar sparse file support to the cpio utility. > >Similar to what? Similar to pax. > > >> In pass mode, (cpio -p), sparse files will be recreated at the >> destination with the same holes that were present in the source >> file, as long as the source file system supports reporting >> holes (as described by PSARC case 2004/770) and the destination >> file is seekable. Otherwise, holes in sparse files will be >> filled with '\0' btyes in corresponding destination files as >> they are now. > >How do you intend to switch between the sparse support mode and the non-sparse >mode in "copy mode"? There is no switch in copy mode. If the source filesystem reports holes in a file, the holes will be duplicated in the destination file as long as the destination file is seekable. > >> In copy out mode (-o) the following new option arguments to the >> cpio -H option will be added to provide sparse file support: >> ascii_sparse - assumes -c is specified. Only available >> in copy out (-o) mode. >> odc_sparse - assumes -H odc is specified Only available >> in copy out (-o) mode. > >Adding sparse file support does not introduce a new archive format unless you >create a new archive format that may be detected by reading the first archive >header from a random archive. Correct. When using -H ascii_sparse and -H odc_sparse, cpio uses ascii and odc format archives, respectively; but it uses a different file type when adding a sparse file to the archive. If an archiver understands cpio ascii and odc format archives, it will understand the archives. If an archiver doesn't recognize the extended file types, the standards require that it extract the file data as a regular file (which wlll contain the data needed to recreate the file contents with holes in the proper positions). > >If you like to avoid to to introduce a new option, you would need to document >this as a dirty hack. BTW: Where is the new man page? Quoting from the references section of this case: 5.4 PSARC/2008/727/materials/cpio.1: Updated cpio.1 man page > > >... > >> The following will apply when either '-H ascii_sparse' or >> '-H odc_sparse' is specified with -o: >> - The c_mode field will in the archive header will >> indicate that the file is a sparse file. In the old >> stat structure, the mode field is an unsigned short >> (16 bit) field. To avoid conflicts with other file >> types, a high order bit (17) in the c_mode field of >> the header will be set. > >This is beyond the cpio specs. How do you plan to mark the archives >as "Sun cpio" specific to allow to avoid incorrect behavior for non-Sun >archives? It is indicated by the file type. > >> - the file size field of the header will be the size of >> the compressed sparse file (i.e., the size of the >> header below plus the size of the file contents after >> removing the holes). > >OK > >> - A string of the following format will be prepended to >> the compressed file data: >> "%lu %llu%s", prepended_info_size, >> expanded_file_size, data/hole_offsets > >Is this data _inside_ the file data area or is it in conflict with the >cpio extensions from David Korn and Glenn Fowler? It is inside the file data area as indicated above. (The file size field is the size of this header plus the size of the file contents after removing the holes.) > > >> where data/hole_offsets contains 2 or more entries of the >> following format: >> " %llu %llu", data_offset, hole_offset > >If you ever like to debug this, I would recommend to use: > > " %llu,%llu", data_offset, hole_offset > >to make the data parsable by human eyes.. Maybe to European human eyes. In the U.S., some possible data offset, hole offset pairs could look like a single number with a the "," being a thousands separator instead of as a pair separator. Besides that it matches the string given as the data in a ustar/pax SUN.holesdata extended header. > >But why don't you follow existing other implementations that use >offset/numbytes pairs for data chunks? This results in a lower archive size. I'm not going to argue decisions that were agreed upon for PSARC 2006/361. But, it follows naturally from the data provided by the lseek(2) SEEK_HOLE and SEEK_DATA operations. > > >> When the c_mode field is set, cpio will detect the sparse file >> upon file extraction, and use the prepended sparse file >> information to restore the holes in the file if the >> destiination file is seekable. If the destination file is not >> seekable, the sparse file information will be used to fill the >> holes with '\0' bytes. Archivers that do not recognize the >> sparse file mode bit will restore the compressed file and its >> prepended data as a regular file. > >As it is unlikely that the first file in an archive is a sparse file, how >do you intend to detect an archive that contain this Sun specific cpio >extension? By the file type. > >How do you intend to switch between the sparse support mode and the non-sparse >mode in "extract mode"? There is no switching. If a file is archived as a sparse file, it will be extracted as a sparse file. - Don > > >Jörg From carlsonj@phorcys.east.sun.com Mon Nov 24 03:31:27 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAOBVROY006478 for ; Mon, 24 Nov 2008 03:31:27 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAOBVMmd045083; Mon, 24 Nov 2008 04:31:26 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAU0090B5CDNS00@nwk-avmta-1.sfbay.Sun.COM>; Mon, 24 Nov 2008 03:31:25 -0800 (PST) Received: from phorcys.east.sun.com ([129.148.174.143]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAU00FZY5CC4LD0@nwk-avmta-1.sfbay.Sun.COM>; Mon, 24 Nov 2008 03:31:25 -0800 (PST) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id mAOBVOMl014554; Mon, 24 Nov 2008 06:31:24 -0500 (EST) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id mAOBVOKC014551; Mon, 24 Nov 2008 06:31:24 -0500 (EST) Date: Mon, 24 Nov 2008 06:31:24 -0500 From: James Carlson Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <200811220529.mAM5TilE001531@spartan.eng.sun.com> To: Don Cragun Cc: Garrett.Damore@sun.com, Cynthia.Eastham@sun.com, PSARC-ext@sun.com Message-id: <18730.37004.301749.965729@gargle.gargle.HOWL> MIME-version: 1.0 X-Mailer: VM 7.01 under Emacs 21.3.1 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200811220529.mAM5TilE001531@spartan.eng.sun.com> Status: RO Content-Length: 1509 Don Cragun writes: > >4) Is there any other convention or standard for holey cpio archives in > >common use? (E.g. something supported by Linux or star?) > > This was discussed while we were working on PSARC/2006/361. Some other > archivers have a --sparse long option that converts sequences of zero > bytes into holes. The ARC gave explicit direction to the project team > that archivers should "preserve" existing holes, not try to reduce disk > usage by "creating" holes. None of the other archivers (except for our > pax) have the ability to preserve existing holes in sparse files. I > see no reason why that advice should be true for pax, but not for cpio. I looked over that other case, and it seems that we discussed '--sparse' for pax as well as the GNUish compression scheme (in the absence of direct detection of holes), but not the cpio format itself. The unanswered question here is whether there's another way to encode holes in cpio stream format, and one that is used by other platforms. Apart from the detection question at the source side -- I agree that trying to "compress" files is a bit odd and certainly not the focus of this case -- are there other ways to encode them in the output format? If there are, then should we be deliberately incompatible? -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From Joerg.Schilling@fokus.fraunhofer.de Mon Nov 24 05:59:29 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAODxSBU023166 for ; Mon, 24 Nov 2008 05:59:29 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id mAODxBiU009417; Mon, 24 Nov 2008 21:59:24 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAU00601C6YLV00@brm-avmta-1.central.sun.com>; Mon, 24 Nov 2008 06:59:22 -0700 (MST) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAU000IUC6YTV30@brm-avmta-1.central.sun.com>; Mon, 24 Nov 2008 06:59:22 -0700 (MST) Received: from relay21.sun.com (relay21.sun.com [192.12.251.24] (may be forged)) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mAODwj4v002771; Mon, 24 Nov 2008 13:59:22 +0000 (GMT) Received: from mms25es.mms.us.syntegra.com ([150.143.232.90] [150.143.232.90]) by relay21i.sun.com with ESMTP id BT-MMP-174291; Mon, 24 Nov 2008 13:59:21 +0000 (Z) Received: from relay24.sun.com (relay24.sun.com [192.12.251.74]) by mms25es.mms.us.syntegra.com with ESMTP id BT-MMP-2361485; Mon, 24 Nov 2008 13:59:21 +0000 (Z) Received: from iron01.fraunhofer.de ([153.96.1.54] [153.96.1.54]) by relay24i.sun.com with ESMTP id BT-MMP-3046942; Mon, 24 Nov 2008 13:59:21 +0000 (Z) Received: from pluto.fokus.fraunhofer.de ([195.37.77.164]) by iron01.fraunhofer.de with ESMTP/TLS/DHE-RSA-AES256-SHA; Mon, 24 Nov 2008 14:59:19 +0100 Received: from EXCHSRV.fokus.fraunhofer.de (bohr [10.147.9.231]) by pluto.fokus.fraunhofer.de (8.13.7/8.13.7) with SMTP id mAODxJi5017131; Mon, 24 Nov 2008 14:59:19 +0100 (MET) Received: from rigel ([10.147.65.195]) by EXCHSRV.fokus.fraunhofer.de with Microsoft SMTPSVC(6.0.3790.3959); Mon, 24 Nov 2008 14:59:19 +0100 Date: Mon, 24 Nov 2008 14:59:19 +0100 From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <200811220529.mAM5TilE001531@spartan.eng.sun.com> To: Garrett.Damore@sun.com, don.cragun@sun.com Cc: PSARC-ext@sun.com, Cynthia.Eastham@sun.com Message-id: <492ab337.fWdWF5P9NzFeRBMv%Joerg.Schilling@fokus.fraunhofer.de> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=-2.6/5.0, scanned in 0.334sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200811220529.mAM5TilE001531@spartan.eng.sun.com> User-Agent: nail 11.22 3/20/05 X-OriginalArrivalTime: 24 Nov 2008 13:59:19.0429 (UTC) FILETIME=[D883CB50:01C94E3C] Status: RO Content-Length: 3900 Don Cragun wrote: > There are no new options (aka flags); there are just new values for the > -H option arguement. > > These changes to cpio are to fix an escalated bug report from the Live > Upgrade project. The submitter insists on using cpio; not pax. The > project team would be happy to deprecate tar and cpio, and to use pax > as a replacement, but tar and cpio provide a few features that aren't > yet duplicated by pax and the project team's priorities are to work on > escalations and porting packages to Nevada; not to improve > infrastructure of existing utilities. A strong reason for going to star. Star implements all available features in it's "star" personality. > >2) (This is probably stylistic.) Why two different flags (ascii_sparse > >and odc_sparse)? Couldn't you just add one new flag for sparse support, > >and then figure out whether ascii or odc by the presence of the -H odc flag? > > The number of available option letters (or flags) that aren't used by > Solaris cpio, GNU cpio, and star is extremely limited. And, we try to > avoid adding options that use the same option letter with different > meanings between the various archivers. (We try, but we don't always > succeed. Here it was easy to use -H option arguments to identify > different archive formats rather than add a conflicting option.) Introducing "archive formats" that cannot be properly detected from reding the first archive header from a random archive has been done in the 1980s. This is now 20 years later and we schould only define archive formats that allow to be properly detected. > >3) My interpretation of the last sentence in section 4 ("Archivers that > >do not recognize the sparse file mode bit will restore the compressed > >file and its prepended data as a regular file.") is that the resulting > >destination file will be quite different from the source -- it will not > >have holes, but neither will it be zero filled, and it will have the > >prepended information string stored instead. Correct? > > Yes. The additional mode bit makes it an unrecognized file type for > archivers that don't know what is going on. The standards require that > when an unrecognized file type is found, the data be stored as a regular This is defined for tar based archive formats but not for cpio. For cpio archives (see SUSv2), the standard just mentiones that archives that contain "additional file types" should not be transported to other systems. Will the man page include a hint on the portability issues? > >4) Is there any other convention or standard for holey cpio archives in > >common use? (E.g. something supported by Linux or star?) > > This was discussed while we were working on PSARC/2006/361. Some other > archivers have a --sparse long option that converts sequences of zero > bytes into holes. The ARC gave explicit direction to the project team > that archivers should "preserve" existing holes, not try to reduce disk > usage by "creating" holes. None of the other archivers (except for our > pax) have the ability to preserve existing holes in sparse files. I > see no reason why that advice should be true for pax, but not for cpio. Star implements -sparse (in create mode) and -force-hole (in extract mode). Star is the first archiver that allowed to correctly preserve existing holes in sparse files. Support for exact replication of sparse files has been implemented in June 1998. Sparse support in general is present in star since 1993/1994. BTW: Sun pax does not have the ability to support sparse files (see man page). Jörg -- EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin js@cs.tu-berlin.de (uni) schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily From Joerg.Schilling@fokus.fraunhofer.de Mon Nov 24 07:17:15 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAOFHF4D026134 for ; Mon, 24 Nov 2008 07:17:15 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAOFH8BN002875; Mon, 24 Nov 2008 07:17:13 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAU00505FSNQF00@nwk-avmta-2.sfbay.sun.com>; Mon, 24 Nov 2008 07:17:11 -0800 (PST) Received: from sca-ea-mail-4.sun.com ([192.18.43.22]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAU00092FSMT460@nwk-avmta-2.sfbay.sun.com>; Mon, 24 Nov 2008 07:17:10 -0800 (PST) Received: from relay13i.sun.com (ip123.net129179-4.block1.us.syntegra.com [129.179.4.123]) by sca-ea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mAOFD24x014999; Mon, 24 Nov 2008 15:17:10 +0000 (GMT) Received: from mmp12es.mmp.us.syntegra.com ([160.41.208.12] [160.41.208.12]) by relay13i.sun.com with ESMTP id BT-MMP-914548; Mon, 24 Nov 2008 15:17:10 +0000 (Z) Received: from relay16i.sun.com (relay16i.sun.com [129.179.4.126]) by mmp12es.mmp.us.syntegra.com with ESMTP id BT-MMP-3811750; Mon, 24 Nov 2008 15:17:08 +0000 (Z) Received: from iron02.fraunhofer.de ([153.96.1.56] [153.96.1.56]) by relay1ib.sun.com with ESMTP id BT-MMP-9721619; Mon, 24 Nov 2008 15:17:08 +0000 (Z) Received: from pluto.fokus.fraunhofer.de ([195.37.77.164]) by iron02.fraunhofer.de with ESMTP/TLS/DHE-RSA-AES256-SHA; Mon, 24 Nov 2008 16:17:06 +0100 Received: from EXCHSRV.fokus.fraunhofer.de (bohr [10.147.9.231]) by pluto.fokus.fraunhofer.de (8.13.7/8.13.7) with SMTP id mAOFH6rP019279; Mon, 24 Nov 2008 16:17:06 +0100 (MET) Received: from rigel ([10.147.65.195]) by EXCHSRV.fokus.fraunhofer.de with Microsoft SMTPSVC(6.0.3790.3959); Mon, 24 Nov 2008 16:17:06 +0100 Date: Mon, 24 Nov 2008 16:17:06 +0100 From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <200811240542.mAO5gM5t009146@spartan.eng.sun.com> To: don.cragun@sun.com Cc: PSARC-ext@sun.com, Cynthia.Eastham@sun.com Message-id: <492ac572.c31WBw8Hfz8QTgGM%Joerg.Schilling@fokus.fraunhofer.de> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=-2.6/5.0, scanned in 1.285sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200811240542.mAO5gM5t009146@spartan.eng.sun.com> User-Agent: nail 11.22 3/20/05 X-OriginalArrivalTime: 24 Nov 2008 15:17:06.0439 (UTC) FILETIME=[B6450170:01C94E47] Status: RO Content-Length: 7406 Don Cragun wrote: > >> PSARC case 2006/331 (Add holey file support to pax) created > > > >I did never see such a case and the Sun pax man page does neither > >include "hole" nor "sparse". > > The case was approved before PSARC cases were handled in cases open to > people who are not Sun employees. But, all of the important > information is included on Sun's current pax(1) man page. The pax > utility added an extended header to ustar and pax archive format > archives as described in the USAGE section, where it says: I canot speak for _very_ recent Solaris versions but for a case from 2006, I would expect to see results before the latest (Build89) I checked. Build 89 comes with a pax man page from 2004 > and in the EXTENDED DESCRIPTION, where it says: > > "SUN.holesdata A Solaris extension to pax extended header > keywords. Specifies the data and hole pairs > for a sparse file. > > "In write or copy modes and when the xustar > or pax format (see -x format) is specified, > pax includes a SUN.holesdate extended > header record if the underlying file system > supports the detection of files with holes > (see fpathconf(2)) and reports that there > is at least one hole in the file being > archived. value consists of two or more > consecutive entries of the following form: > > SPACEdata_offsetSPACEhole_offset > > > "where the data and hole offsets are the > long values returned by passing SEEK_DATA > and SEEK_HOLE to lseek(2), respectively. > For example, the following entry is an > example of the SUN.holesdata entry in the > extended header for a file with data > offsets at bytes 0, 24576, and 49152, and > hole offsets at bytes 8192, 32768, and > 49159: 49 SUN.holesdata= 0 8192 24576 32768 > 49152 49159: > > 49 SUN.holesdata= 0 8192 24576 32768 49152 49159 Looks like it indroduces the same problem as the cpio case. > >How do you intend to switch between the sparse support mode and the non-sparse > >mode in "copy mode"? > > There is no switch in copy mode. If the source filesystem reports > holes in a file, the holes will be duplicated in the destination file > as long as the destination file is seekable. This should be marked as deficit in the man page. > >> In copy out mode (-o) the following new option arguments to the > >> cpio -H option will be added to provide sparse file support: > >> ascii_sparse - assumes -c is specified. Only available > >> in copy out (-o) mode. > >> odc_sparse - assumes -H odc is specified Only available > >> in copy out (-o) mode. > > > >Adding sparse file support does not introduce a new archive format unless you > >create a new archive format that may be detected by reading the first archive > >header from a random archive. > > Correct. When using -H ascii_sparse and -H odc_sparse, cpio uses ascii > and odc format archives, respectively; but it uses a different file > type when adding a sparse file to the archive. If an archiver > understands cpio ascii and odc format archives, it will understand the > archives. If an archiver doesn't recognize the extended file types, > the standards require that it extract the file data as a regular file > (which wlll contain the data needed to recreate the file contents with > holes in the proper positions). Does the code sets bit 17 and clears the file type bits or does it set bit 17 in addition? > > > >If you like to avoid to to introduce a new option, you would need to document > >this as a dirty hack. BTW: Where is the new man page? > > Quoting from the references section of this case: > 5.4 PSARC/2008/727/materials/cpio.1: Updated cpio.1 man page OK, but I see no description of the archive format in this man page. > >> The following will apply when either '-H ascii_sparse' or > >> '-H odc_sparse' is specified with -o: > >> - The c_mode field will in the archive header will > >> indicate that the file is a sparse file. In the old > >> stat structure, the mode field is an unsigned short > >> (16 bit) field. To avoid conflicts with other file > >> types, a high order bit (17) in the c_mode field of > >> the header will be set. > > > >This is beyond the cpio specs. How do you plan to mark the archives > >as "Sun cpio" specific to allow to avoid incorrect behavior for non-Sun > >archives? > > It is indicated by the file type. As a result of not marking the archive, archivers that carefully implement add on features depending on the archive format will not unpack the sparse files. star and AT&T pax will ignore bit 17, other archivers may include this bit in the file type with unkown results. Vendor unique extensions that do not use explicit vendor specific tags are something we had in the 1980s. > >> - A string of the following format will be prepended to > >> the compressed file data: > >> "%lu %llu%s", prepended_info_size, > >> expanded_file_size, data/hole_offsets > > > >Is this data _inside_ the file data area or is it in conflict with the > >cpio extensions from David Korn and Glenn Fowler? > > It is inside the file data area as indicated above. (The file size > field is the size of this header plus the size of the file contents > after removing the holes.) OK; how about marking the archive in the header area past the filename? > >> where data/hole_offsets contains 2 or more entries of the > >> following format: > >> " %llu %llu", data_offset, hole_offset > > > >If you ever like to debug this, I would recommend to use: > > > > " %llu,%llu", data_offset, hole_offset > > > >to make the data parsable by human eyes.. > > Maybe to European human eyes. In the U.S., some possible data offset, > hole offset pairs could look like a single number with a the "," being > a thousands separator instead of as a pair separator. Besides that it > matches the string given as the data in a ustar/pax SUN.holesdata > extended header. It seems that you are too US centric and thus do not see the problem of being unable to see number pairs in a possiblily extremely long data stream. > >But why don't you follow existing other implementations that use > >offset/numbytes pairs for data chunks? This results in a lower archive size. > > I'm not going to argue decisions that were agreed upon for PSARC > 2006/361. But, it follows naturally from the data provided by the > lseek(2) SEEK_HOLE and SEEK_DATA operations. I offered my help in special for tar/cpio specific archive format questions even before this case was aproved. So why didn't you ask me then? Jörg -- EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin js@cs.tu-berlin.de (uni) schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily From Joerg.Schilling@fokus.fraunhofer.de Mon Nov 24 12:08:53 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAOK8psA003529 for ; Mon, 24 Nov 2008 12:08:53 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAOK8kL5019289; Mon, 24 Nov 2008 20:08:48 GMT Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAU00K3JTANP300@brm-avmta-1.central.sun.com>; Mon, 24 Nov 2008 13:08:47 -0700 (MST) Received: from sca-ea-mail-2.sun.com ([192.18.43.25]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAU00HF2TAG18A0@brm-avmta-1.central.sun.com>; Mon, 24 Nov 2008 13:08:40 -0700 (MST) Received: from relay18i.sun.com (ip128.net129179-4.block1.us.syntegra.com [129.179.4.128]) by sca-ea-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAOJeHZp025265; Mon, 24 Nov 2008 20:08:40 +0000 (GMT) Received: from mmp11es.mmp.us.syntegra.com ([160.41.208.11] [160.41.208.11]) by relay18i.sun.com with ESMTP id BT-MMP-934078; Mon, 24 Nov 2008 20:08:39 +0000 (Z) Received: from relay11i.sun.com (relay11i.sun.com [129.179.4.121]) by mmp11es.mmp.us.syntegra.com with ESMTP id BT-MMP-4029919; Mon, 24 Nov 2008 20:08:38 +0000 (Z) Received: from iron01.fraunhofer.de ([153.96.1.54] [153.96.1.54]) by relay1ib.sun.com with ESMTP id BT-MMP-1981397; Mon, 24 Nov 2008 20:08:38 +0000 (Z) Received: from pluto.fokus.fraunhofer.de ([195.37.77.164]) by iron01.fraunhofer.de with ESMTP/TLS/DHE-RSA-AES256-SHA; Mon, 24 Nov 2008 21:08:37 +0100 Received: from EXCHSRV.fokus.fraunhofer.de (bohr [10.147.9.231]) by pluto.fokus.fraunhofer.de (8.13.7/8.13.7) with SMTP id mAOK8aHg026729; Mon, 24 Nov 2008 21:08:36 +0100 (MET) Received: from rigel ([10.147.65.195]) by EXCHSRV.fokus.fraunhofer.de with Microsoft SMTPSVC(6.0.3790.3959); Mon, 24 Nov 2008 21:08:36 +0100 Date: Mon, 24 Nov 2008 21:08:32 +0100 From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <200811240542.mAO5gM5t009146@spartan.eng.sun.com> To: don.cragun@sun.com Cc: PSARC-ext@sun.com, Cynthia.Eastham@sun.com Message-id: <492b09c0.srTOnFqeoKyKzl07%Joerg.Schilling@fokus.fraunhofer.de> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=0.0/5.0, scanned in 1.233sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200811240542.mAO5gM5t009146@spartan.eng.sun.com> User-Agent: nail 11.22 3/20/05 X-OriginalArrivalTime: 24 Nov 2008 20:08:36.0267 (UTC) FILETIME=[6F04F7B0:01C94E70] Status: RO Content-Length: 1555 Don Cragun wrote: > and in the EXTENDED DESCRIPTION, where it says: > > "SUN.holesdata A Solaris extension to pax extended header > keywords. Specifies the data and hole pairs > for a sparse file. > ... > 49 SUN.holesdata= 0 8192 24576 32768 49152 49159 I just verified this on a recent build and it seems that Sun pax now contains a similar conceptional problem as the GNU tar guys did introduce a few years ago: The size of the POSIX.1-2001 file metadata is limited to 8 GB. Putting the hole information into the file meta data area thus limits the number of holes. If you e.g. create a file with the maximum holyness (Block size == 8k), you are able to archive a file of up to ~ 5 TB and aprox. 300 million holes. I know that an archiver that is going to unpack this would need to allocate 5 GB of virtual memory to remember the hole list. Files that fall into this category are already possible today, so this does not seem to be ready for the future. I recommend to fix this by moving the hole information into the file data area, to switch to a offset/datasize notation (to reduce the size) and to write the information as comma separated pairs. Jörg -- EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin js@cs.tu-berlin.de (uni) schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily From gdamore@sun.com Mon Nov 24 12:16:58 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAOKGwhi005085 for ; Mon, 24 Nov 2008 12:16:58 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAOKGvUq024539; Mon, 24 Nov 2008 12:16:58 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAU00L07TNWQT00@brm-avmta-1.central.sun.com>; Mon, 24 Nov 2008 13:16:44 -0700 (MST) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAU00HLDTNN18C0@brm-avmta-1.central.sun.com>; Mon, 24 Nov 2008 13:16:39 -0700 (MST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAOKGZMO015410; Mon, 24 Nov 2008 12:16:35 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAU00I01T1QRB00@fe-sfbay-09.sun.com> (original mail from gdamore@sun.com) ; Mon, 24 Nov 2008 12:16:35 -0800 (PST) Received: from [10.7.251.172] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAU00JT3TNLBP60@fe-sfbay-09.sun.com>; Mon, 24 Nov 2008 12:16:34 -0800 (PST) Date: Mon, 24 Nov 2008 12:10:21 -0800 From: "Garrett D'Amore" Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <492b09c0.srTOnFqeoKyKzl07%Joerg.Schilling@fokus.fraunhofer.de> Sender: Garrett.Damore@sun.com To: Joerg Schilling Cc: don.cragun@sun.com, PSARC-ext@sun.com, Cynthia.Eastham@sun.com Message-id: <492B0A2D.5090008@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 References: <200811240542.mAO5gM5t009146@spartan.eng.sun.com> <492b09c0.srTOnFqeoKyKzl07%Joerg.Schilling@fokus.fraunhofer.de> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 1725 It seems to me that there has been enough real discussion on this topic, that I think it is worth promoting to at least a fast track status. Please consider it so promoted, and set the time out for a week from Wednesday. (A longer timeout than 1 week is appropriate, IMO, given that this week represents a short week in the US.) -- Garrett Joerg Schilling wrote: > Don Cragun wrote: > > >> and in the EXTENDED DESCRIPTION, where it says: >> >> "SUN.holesdata A Solaris extension to pax extended header >> keywords. Specifies the data and hole pairs >> for a sparse file. >> >> > ... > > >> 49 SUN.holesdata= 0 8192 24576 32768 49152 49159 >> > > I just verified this on a recent build and it seems that Sun pax now contains > a similar conceptional problem as the GNU tar guys did introduce a few years > ago: > > The size of the POSIX.1-2001 file metadata is limited to 8 GB. Putting the > hole information into the file meta data area thus limits the number of holes. > > If you e.g. create a file with the maximum holyness (Block size == 8k), > you are able to archive a file of up to ~ 5 TB and aprox. 300 million holes. > I know that an archiver that is going to unpack this would need to allocate > 5 GB of virtual memory to remember the hole list. Files that fall into this > category are already possible today, so this does not seem to be ready for the > future. > > I recommend to fix this by moving the hole information into the file data area, > to switch to a offset/datasize notation (to reduce the size) and to write > the information as comma separated pairs. > > > Jörg > > From dwc@spartan.eng.sun.com Mon Nov 24 14:33:31 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAOMXUBv009604 for ; Mon, 24 Nov 2008 14:33:30 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAOMXN2k063479; Mon, 24 Nov 2008 15:33:27 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAU00805ZZR6N00@nwk-avmta-2.sfbay.sun.com>; Mon, 24 Nov 2008 14:33:27 -0800 (PST) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAU000VZZZQA380@nwk-avmta-2.sfbay.sun.com>; Mon, 24 Nov 2008 14:33:26 -0800 (PST) Received: from spartan.eng.sun.com (spartan.SFBay.Sun.COM [129.146.226.64]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAOMXK9n021020; Mon, 24 Nov 2008 14:33:20 -0800 (PST) Received: from spartan.eng.sun.com (localhost [127.0.0.1]) by spartan.eng.sun.com (8.13.7+Sun/8.13.7) with ESMTP id mAOMAnnq012067; Mon, 24 Nov 2008 14:10:49 -0800 (PST) Received: (from dwc@localhost) by spartan.eng.sun.com (8.13.7+Sun/8.13.7/Submit) id mAOMAne5012066; Mon, 24 Nov 2008 14:10:49 -0800 (PST) Date: Mon, 24 Nov 2008 14:10:49 -0800 (PST) From: Don Cragun Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Fast Track 12/03/2008] To: Joerg.Schilling@fokus.fraunhofer.de, gdamore@sun.com Cc: Cynthia.Eastham@sun.com, PSARC-ext@sun.com Message-id: <200811242210.mAOMAne5012066@spartan.eng.sun.com> MIME-version: 1.0 Content-type: TEXT/PLAIN Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 2497 >Date: Mon, 24 Nov 2008 12:10:21 -0800 >From: "Garrett D'Amore" > >It seems to me that there has been enough real discussion on this topic, >that I think it is worth promoting to at least a fast track status. >Please consider it so promoted, and set the time out for a week from >Wednesday. (A longer timeout than 1 week is appropriate, IMO, given >that this week represents a short week in the US.) The case has been recast as a fast track with timeout set to 12/03/2008. Note that this is delaying a fix for an escalation for a large external customer (whose name I cannot give on an alias with non-Sun participants). Note that a good portion of this discussion is about PSARC/2006/361; not about this case. > > -- Garrett > >Joerg Schilling wrote: >> Don Cragun wrote: >> >> >>> and in the EXTENDED DESCRIPTION, where it says: >>> >>> "SUN.holesdata A Solaris extension to pax extended header >>> keywords. Specifies the data and hole pairs >>> for a sparse file. >>> >>> >> ... >> >> >>> 49 SUN.holesdata= 0 8192 24576 32768 49152 49159 >>> >> >> I just verified this on a recent build and it seems that Sun pax now contains >> a similar conceptional problem as the GNU tar guys did introduce a few years >> ago: >> >> The size of the POSIX.1-2001 file metadata is limited to 8 GB. Putting the >> hole information into the file meta data area thus limits the number of holes. >> >> If you e.g. create a file with the maximum holyness (Block size == 8k), >> you are able to archive a file of up to ~ 5 TB and aprox. 300 million holes. >> I know that an archiver that is going to unpack this would need to allocate >> 5 GB of virtual memory to remember the hole list. Files that fall into this >> category are already possible today, so this does not seem to be ready for the >> future. >> >> I recommend to fix this by moving the hole information into the file data area, >> to switch to a offset/datasize notation (to reduce the size) and to write >> the information as comma separated pairs. >> That is not this case; you are asking to reopen 2006/361. The holesdata for cpio presented by this case is stored in the file data area as you are requesting; not in the header. However, the file size (including the holesdata) is still limited by the 11 octal digit c_filesize field in the cpio file header to a maximum size of 8Gb. >> >> Jörg From dwc@spartan.eng.sun.com Mon Nov 24 14:45:28 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAOMjRK3010602 for ; Mon, 24 Nov 2008 14:45:27 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id mAOMjBFs020780 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Tue, 25 Nov 2008 06:45:26 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAV00D2A0JPEK00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 24 Nov 2008 15:45:25 -0700 (MST) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAV00MNZ0JPJUA0@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 24 Nov 2008 15:45:25 -0700 (MST) Received: from spartan.eng.sun.com (spartan.SFBay.Sun.COM [129.146.226.64]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAOMjIN4028728; Mon, 24 Nov 2008 14:45:18 -0800 (PST) Received: from spartan.eng.sun.com (localhost [127.0.0.1]) by spartan.eng.sun.com (8.13.7+Sun/8.13.7) with ESMTP id mAOMbnq3012157; Mon, 24 Nov 2008 14:37:49 -0800 (PST) Received: (from dwc@localhost) by spartan.eng.sun.com (8.13.7+Sun/8.13.7/Submit) id mAOMbnLM012156; Mon, 24 Nov 2008 14:37:49 -0800 (PST) Date: Mon, 24 Nov 2008 14:37:49 -0800 (PST) From: Don Cragun Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] To: james.d.carlson@sun.com Cc: Cynthia.Eastham@sun.com, PSARC-ext@sun.com Message-id: <200811242237.mAOMbnLM012156@spartan.eng.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 3050 >From psarc-intern-list-request@sun.com Mon Nov 24 03:33:26 2008 >Date: Mon, 24 Nov 2008 06:31:24 -0500 >From: James Carlson >Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] >To: Don Cragun >Cc: Garrett.Damore@sun.com, Cynthia.Eastham@sun.com, PSARC-ext@sun.com >MIME-version: 1.0 >Content-transfer-encoding: 7BIT > >Don Cragun writes: >> >4) Is there any other convention or standard for holey cpio archives in >> >common use? (E.g. something supported by Linux or star?) >> >> This was discussed while we were working on PSARC/2006/361. Some other >> archivers have a --sparse long option that converts sequences of zero >> bytes into holes. The ARC gave explicit direction to the project team >> that archivers should "preserve" existing holes, not try to reduce disk >> usage by "creating" holes. None of the other archivers (except for our >> pax) have the ability to preserve existing holes in sparse files. I >> see no reason why that advice should be true for pax, but not for cpio. > >I looked over that other case, and it seems that we discussed >'--sparse' for pax as well as the GNUish compression scheme (in the >absence of direct detection of holes), but not the cpio format itself. Yes. > >The unanswered question here is whether there's another way to encode >holes in cpio stream format, and one that is used by other platforms. >Apart from the detection question at the source side -- I agree that >trying to "compress" files is a bit odd and certainly not the focus of >this case -- are there other ways to encode them in the output format? There are no extensible headers in cpio format. The only place to store data on where the holes go in cpio format is in the file data area. The project team had the option of storing hole information followed by the complete file contents or storing the hold information followed by the file with the holes removed. Since the cpio format only has 33 bits to store the size of the file data area, the project team chose to remove the holes to increase the size of a sparse file that can be archived in cpio format. While it is true that it would be possible to encode holes data in a slightly more compact form, it is nice to have a common format for the holes data in the ustar/pax extended header records for sparse files and in the data area in the cpio file data area. > >If there are, then should we be deliberately incompatible? The star and the recent AT&T pax archivers encode sparse files using ustar/pax format archives. The project team has not seen any other attempt to encode sparse files using cpio format. So, there is no other known cpio format that handles this case. We are not being deliberately incompatible; nothing else handles this case. - Don > >-- >James Carlson, Solaris Networking >Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 >MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From dwc@spartan.eng.sun.com Mon Nov 24 15:16:12 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAONGCLh013136 for ; Mon, 24 Nov 2008 15:16:12 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAONG9Sb009269 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Mon, 24 Nov 2008 15:16:12 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAV009031YKJC00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 24 Nov 2008 15:15:56 -0800 (PST) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAV001KJ1YJDY90@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 24 Nov 2008 15:15:55 -0800 (PST) Received: from spartan.eng.sun.com (spartan.SFBay.Sun.COM [129.146.226.64]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAONFngU049803; Mon, 24 Nov 2008 15:15:49 -0800 (PST) Received: from spartan.eng.sun.com (localhost [127.0.0.1]) by spartan.eng.sun.com (8.13.7+Sun/8.13.7) with ESMTP id mAON0oQn012244; Mon, 24 Nov 2008 15:00:50 -0800 (PST) Received: (from dwc@localhost) by spartan.eng.sun.com (8.13.7+Sun/8.13.7/Submit) id mAON0o7g012243; Mon, 24 Nov 2008 15:00:50 -0800 (PST) Date: Mon, 24 Nov 2008 15:00:50 -0800 (PST) From: Don Cragun Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] To: Joerg.Schilling@fokus.fraunhofer.de Cc: Cynthia.Eastham@sun.com, PSARC-ext@sun.com Message-id: <200811242300.mAON0o7g012243@spartan.eng.sun.com> MIME-version: 1.0 Content-type: TEXT/PLAIN Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 4821 >Date: Mon, 24 Nov 2008 14:59:19 +0100 >From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) > >Don Cragun wrote: > >> There are no new options (aka flags); there are just new values for the >> -H option arguement. >> >> These changes to cpio are to fix an escalated bug report from the Live >> Upgrade project. The submitter insists on using cpio; not pax. The >> project team would be happy to deprecate tar and cpio, and to use pax >> as a replacement, but tar and cpio provide a few features that aren't >> yet duplicated by pax and the project team's priorities are to work on >> escalations and porting packages to Nevada; not to improve >> infrastructure of existing utilities. > >A strong reason for going to star. Star implements all available features >in it's "star" personality. Your star archiver uses extended headers to encode the holes data. There are no extended headers in cpio format archives. The customer insists on cpio format archives. > >> >2) (This is probably stylistic.) Why two different flags (ascii_sparse >> >and odc_sparse)? Couldn't you just add one new flag for sparse support, >> >and then figure out whether ascii or odc by the presence of the -H odc flag? >> >> The number of available option letters (or flags) that aren't used by >> Solaris cpio, GNU cpio, and star is extremely limited. And, we try to >> avoid adding options that use the same option letter with different >> meanings between the various archivers. (We try, but we don't always >> succeed. Here it was easy to use -H option arguments to identify >> different archive formats rather than add a conflicting option.) > >Introducing "archive formats" that cannot be properly detected from reding the >first archive header from a random archive has been done in the 1980s. >This is now 20 years later and we schould only define archive formats that >allow to be properly detected. > > >> >3) My interpretation of the last sentence in section 4 ("Archivers that >> >do not recognize the sparse file mode bit will restore the compressed >> >file and its prepended data as a regular file.") is that the resulting >> >destination file will be quite different from the source -- it will not >> >have holes, but neither will it be zero filled, and it will have the >> >prepended information string stored instead. Correct? >> >> Yes. The additional mode bit makes it an unrecognized file type for >> archivers that don't know what is going on. The standards require that >> when an unrecognized file type is found, the data be stored as a regular > >This is defined for tar based archive formats but not for cpio. > >For cpio archives (see SUSv2), the standard just mentiones that archives that >contain "additional file types" should not be transported to other systems. >Will the man page include a hint on the portability issues? You're correct. I didn't go back to the 1990 POSIX.1 standard after reading the rationale in the current standard. Until I went back and looked at it this afternoon, I thought the requirement was to extract an unknown file type in any archive format as a regular file and to print a diagnostic message saying that the conversions to regular file had been done. Our current cpio and pax convert files of unknown type to a regular file, but don't print a diagnostic. When this case is approved, a bug report will be filed so cpio and pax will print a diagnostic in this case no matter what archive format is being read. > > >> >4) Is there any other convention or standard for holey cpio archives in >> >common use? (E.g. something supported by Linux or star?) >> >> This was discussed while we were working on PSARC/2006/361. Some other >> archivers have a --sparse long option that converts sequences of zero >> bytes into holes. The ARC gave explicit direction to the project team >> that archivers should "preserve" existing holes, not try to reduce disk >> usage by "creating" holes. None of the other archivers (except for our >> pax) have the ability to preserve existing holes in sparse files. I >> see no reason why that advice should be true for pax, but not for cpio. > >Star implements -sparse (in create mode) and -force-hole (in extract mode). >Star is the first archiver that allowed to correctly preserve existing holes >in sparse files. Support for exact replication of sparse files has been >implemented in June 1998. Sparse support in general is present in star since >1993/1994. But this uses ustar/pax format archives. That won't satisfy the customer for that filed the escalation to get this fix. They insist on a fix using cpio format archives. > >BTW: Sun pax does not have the ability to support sparse files (see man page). As seen in later email, Sun pax does support sparse files. - Don > >Jörg From john.plocher@gmail.com Mon Nov 24 15:55:40 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAONtewN015274 for ; Mon, 24 Nov 2008 15:55:40 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAONtcTQ025149 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Mon, 24 Nov 2008 15:55:39 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAV00K033SRFW00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 24 Nov 2008 16:55:39 -0700 (MST) Received: from sca-ea-mail-4.sun.com ([192.18.43.22]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAV00JD03SQWV00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 24 Nov 2008 16:55:39 -0700 (MST) Received: from relay42i.sun.com ([192.5.209.72]) by sca-ea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mAONrr6L001718 for ; Mon, 24 Nov 2008 23:55:38 +0000 (GMT) Received: from mms48es.mms.us.syntegra.com ([160.41.221.230] [160.41.221.230]) by relay42i.sun.com with ESMTP id BT-MMP-116409 for PSARC-ext@sun.com; Mon, 24 Nov 2008 23:55:38 +0000 (Z) Received: from relay41i.sun.com (relay41i.sun.com [192.5.209.70]) by mms48es.mms.us.syntegra.com with ESMTP id BT-MMP-2254254 for PSARC-ext@sun.com; Mon, 24 Nov 2008 23:55:37 +0000 (Z) Received: from mail-gx0-f13.google.com ([209.85.217.13] [209.85.217.13]) by relay4i.sun.com with ESMTP id BT-MMP-1167905 for PSARC-ext@sun.com; Mon, 24 Nov 2008 23:55:38 +0000 (Z) Received: by gxk6 with SMTP id 6so1843390gxk.14 for ; Mon, 24 Nov 2008 15:54:47 -0800 (PST) Received: by 10.90.99.6 with SMTP id w6mr2471396agb.81.1227570887689; Mon, 24 Nov 2008 15:54:47 -0800 (PST) Received: by 10.90.66.16 with HTTP; Mon, 24 Nov 2008 15:54:47 -0800 (PST) Date: Mon, 24 Nov 2008 15:54:47 -0800 From: John Plocher Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <200811242300.mAON0o7g012243@spartan.eng.sun.com> To: Don Cragun , PSARC-ext@sun.com Message-id: MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7BIT Content-disposition: inline DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=C8rQmLVgFZV2yGl8SpWGWREmVtL6hglS/IKxVQGKrhE=; b=V3XSLncvZmUVSo0WYreKYXCzQ5E2qH6Cgwd81hTt5I4mruJxfWK+FQ/NDUW2+B953Y 8bnxChuy48hxb5hnkmxgF/xdGVqQBoXU6T4iD0B3/fYvUdTBfsc9DUD4z6KmwnqAzkJP qI3WvoNqu4oFIMiRzez86Yf5fPXv+9JooXJS4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=scmSfYigqqcg7rE5rXzSIGA+FHyJsFgj+41dxUpbBzZcHkU+KBwtITC9wNKDuT8v7W 5vAdFXR9HJq5bwuHHVdtqxX2tVclQLKfeFoMhmddOmhgKMOiiDndhKvHMnJltjKagCbh xAKSYRBN5lWAasdOL8LKo/Tz766NTKc1UrWjQ= X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=0.0/5.0, scanned in 0.048sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200811242300.mAON0o7g012243@spartan.eng.sun.com> Status: RO Content-Length: 595 On Mon, Nov 24, 2008 at 3:00 PM, Don Cragun wrote: > But this uses ustar/pax format archives. That won't satisfy the > customer for that filed the escalation to get this fix. They insist on > a fix using cpio format archives. With this fix, these archives will no longer be "cpio format archives", but instead they will be "XXX Proprietary CPIO format archives". That is, the customer will not be able to extract "holey" files as holey files on non-Sun systems, rendering this a proprietary solution. Is there a plan to add this support to non-Sun archivers? -John From dwc@spartan.eng.sun.com Mon Nov 24 16:36:03 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAP0a2bs000549 for ; Mon, 24 Nov 2008 16:36:02 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAP0a0qv004220 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Tue, 25 Nov 2008 00:36:01 GMT Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAV001075NZOC00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 24 Nov 2008 17:35:59 -0700 (MST) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAV00JB75NXWQ30@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 24 Nov 2008 17:35:57 -0700 (MST) Received: from spartan.eng.sun.com (spartan.SFBay.Sun.COM [129.146.226.64]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAP0Zpht037287; Mon, 24 Nov 2008 16:35:51 -0800 (PST) Received: from spartan.eng.sun.com (localhost [127.0.0.1]) by spartan.eng.sun.com (8.13.7+Sun/8.13.7) with ESMTP id mAP0KpmO012534; Mon, 24 Nov 2008 16:20:51 -0800 (PST) Received: (from dwc@localhost) by spartan.eng.sun.com (8.13.7+Sun/8.13.7/Submit) id mAP0KpCs012533; Mon, 24 Nov 2008 16:20:51 -0800 (PST) Date: Mon, 24 Nov 2008 16:20:51 -0800 (PST) From: Don Cragun Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] To: Joerg.Schilling@fokus.fraunhofer.de Cc: Cynthia.Eastham@sun.com, PSARC-ext@sun.com Message-id: <200811250020.mAP0KpCs012533@spartan.eng.sun.com> MIME-version: 1.0 Content-type: TEXT/PLAIN Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 8837 >Date: Mon, 24 Nov 2008 16:17:06 +0100 >From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) > >Don Cragun wrote: > >> >> PSARC case 2006/331 (Add holey file support to pax) created >> > >> >I did never see such a case and the Sun pax man page does neither >> >include "hole" nor "sparse". >> >> The case was approved before PSARC cases were handled in cases open to >> people who are not Sun employees. But, all of the important >> information is included on Sun's current pax(1) man page. The pax >> utility added an extended header to ustar and pax archive format >> archives as described in the USAGE section, where it says: > >I canot speak for _very_ recent Solaris versions but for a case from 2006, >I would expect to see results before the latest (Build89) I checked. > >Build 89 comes with a pax man page from 2004 I don't know when the man page got out to OpenSolaris, but the pax fixes went into Nevada build 54. > >> and in the EXTENDED DESCRIPTION, where it says: >> >> "SUN.holesdata A Solaris extension to pax extended header >> keywords. Specifies the data and hole pairs >> for a sparse file. >> >> "In write or copy modes and when the xustar >> or pax format (see -x format) is specified, >> pax includes a SUN.holesdate extended >> header record if the underlying file system >> supports the detection of files with holes >> (see fpathconf(2)) and reports that there >> is at least one hole in the file being >> archived. value consists of two or more >> consecutive entries of the following form: >> >> SPACEdata_offsetSPACEhole_offset >> >> >> "where the data and hole offsets are the >> long values returned by passing SEEK_DATA >> and SEEK_HOLE to lseek(2), respectively. >> For example, the following entry is an >> example of the SUN.holesdata entry in the >> extended header for a file with data >> offsets at bytes 0, 24576, and 49152, and >> hole offsets at bytes 8192, 32768, and >> 49159: 49 SUN.holesdata= 0 8192 24576 32768 >> 49152 49159: >> >> 49 SUN.holesdata= 0 8192 24576 32768 49152 49159 > >Looks like it indroduces the same problem as the cpio case. As Cindy said in the case materials, the changes to cpio format archives are similar to the changes approved by PSARC/2006/361. > > >> >How do you intend to switch between the sparse support mode and the non-sparse >> >mode in "copy mode"? >> >> There is no switch in copy mode. If the source filesystem reports >> holes in a file, the holes will be duplicated in the destination file >> as long as the destination file is seekable. > >This should be marked as deficit in the man page. That is not this case. The original proposal for that case had options to add holesdata extended headers or not when creating an archive and to extract the file with or without holes reinstated. The ARC directed the project team to always store holes data when creating a ustar or pax format archive and to always restore sparse files as sparse files when they are extracted from an archive. > > >> >> In copy out mode (-o) the following new option arguments to the >> >> cpio -H option will be added to provide sparse file support: >> >> ascii_sparse - assumes -c is specified. Only available >> >> in copy out (-o) mode. >> >> odc_sparse - assumes -H odc is specified Only available >> >> in copy out (-o) mode. >> > >> >Adding sparse file support does not introduce a new archive format unless you >> >create a new archive format that may be detected by reading the first archive >> >header from a random archive. >> >> Correct. When using -H ascii_sparse and -H odc_sparse, cpio uses ascii >> and odc format archives, respectively; but it uses a different file >> type when adding a sparse file to the archive. If an archiver >> understands cpio ascii and odc format archives, it will understand the >> archives. If an archiver doesn't recognize the extended file types, >> the standards require that it extract the file data as a regular file >> (which wlll contain the data needed to recreate the file contents with >> holes in the proper positions). > >Does the code sets bit 17 and clears the file type bits or does it set bit 17 in >addition? It sets bit 17 in addition. This allows cpio to handle sparse files of other file types if the case ever comes up. > >> > >> >If you like to avoid to to introduce a new option, you would need to document >> >this as a dirty hack. BTW: Where is the new man page? >> >> Quoting from the references section of this case: >> 5.4 PSARC/2008/727/materials/cpio.1: Updated cpio.1 man page > >OK, but I see no description of the archive format in this man page. The archive format will either be documented in archive(4) or in pax(1) where details of other formats are described in detail. When that happens, a cross reference will be added to the cpio(1) man page. > > >> >> The following will apply when either '-H ascii_sparse' or >> >> '-H odc_sparse' is specified with -o: >> >> - The c_mode field will in the archive header will >> >> indicate that the file is a sparse file. In the old >> >> stat structure, the mode field is an unsigned short >> >> (16 bit) field. To avoid conflicts with other file >> >> types, a high order bit (17) in the c_mode field of >> >> the header will be set. >> > >> >This is beyond the cpio specs. How do you plan to mark the archives >> >as "Sun cpio" specific to allow to avoid incorrect behavior for non-Sun >> >archives? >> >> It is indicated by the file type. > >As a result of not marking the archive, archivers that carefully implement add >on features depending on the archive format will not unpack the sparse files. > >star and AT&T pax will ignore bit 17, other archivers may include this >bit in the file type with unkown results. That is a bug in star and AT&T pax (as well as in Sun's current cpio and pax). A bug report will be filed to correct Sun's cpio and pax when this case is resolved. > >Vendor unique extensions that do not use explicit vendor specific tags >are something we had in the 1980s. > > >> >> - A string of the following format will be prepended to >> >> the compressed file data: >> >> "%lu %llu%s", prepended_info_size, >> >> expanded_file_size, data/hole_offsets >> > >> >Is this data _inside_ the file data area or is it in conflict with the >> >cpio extensions from David Korn and Glenn Fowler? >> >> It is inside the file data area as indicated above. (The file size >> field is the size of this header plus the size of the file contents >> after removing the holes.) > >OK; how about marking the archive in the header area past the filename? As I'm sure you know, the data describing the contents of the file immediately follow the header in a cpio archive. The standard does not allow any new fields to be added to the cpio archive format. > > > >> >> where data/hole_offsets contains 2 or more entries of the >> >> following format: >> >> " %llu %llu", data_offset, hole_offset >> > >> >If you ever like to debug this, I would recommend to use: >> > >> > " %llu,%llu", data_offset, hole_offset >> > >> >to make the data parsable by human eyes.. >> >> Maybe to European human eyes. In the U.S., some possible data offset, >> hole offset pairs could look like a single number with a the "," being >> a thousands separator instead of as a pair separator. Besides that it >> matches the string given as the data in a ustar/pax SUN.holesdata >> extended header. > >It seems that you are too US centric and thus do not see the problem of >being unable to see number pairs in a possiblily extremely long data >stream. As you know, the cpio header is a string of 76 octal digits followed immediately by the pathname of the file (c_namesize bytes including a trailing NUL byte) followed immediately by the contents of the file (c_filesize bytes). This header is not easy for humans to read. This header is not intended to be read by humans; it is intended to be read by cpio and other archivers that want to extract data from cpio archives. The same is true of the holes data record both in the ustar/pax archive format extended headers and in the initial portion of the file data in the cpio archive format. - Don > ... ... ... > >Jörg From dwc@spartan.eng.sun.com Mon Nov 24 17:42:41 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAP1gfjC001781 for ; Mon, 24 Nov 2008 17:42:41 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAP1gb4j037754 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Mon, 24 Nov 2008 18:42:40 -0700 (MST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAV0080J8R2GP00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 24 Nov 2008 18:42:38 -0700 (MST) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAV00JVV8R1WM70@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 24 Nov 2008 18:42:38 -0700 (MST) Received: from spartan.eng.sun.com (spartan.SFBay.Sun.COM [129.146.226.64]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAP1gbYh007893 for ; Mon, 24 Nov 2008 17:42:37 -0800 (PST) Received: from spartan.eng.sun.com (localhost [127.0.0.1]) by spartan.eng.sun.com (8.13.7+Sun/8.13.7) with ESMTP id mAP1Rcbr012755; Mon, 24 Nov 2008 17:27:38 -0800 (PST) Received: (from dwc@localhost) by spartan.eng.sun.com (8.13.7+Sun/8.13.7/Submit) id mAP1Rccd012754; Mon, 24 Nov 2008 17:27:38 -0800 (PST) Date: Mon, 24 Nov 2008 17:27:38 -0800 (PST) From: Don Cragun Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] To: john.plocher@gmail.com Cc: PSARC-ext@sun.com Message-id: <200811250127.mAP1Rccd012754@spartan.eng.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 1736 >Date: Mon, 24 Nov 2008 15:54:47 -0800 >From: John Plocher > >On Mon, Nov 24, 2008 at 3:00 PM, Don Cragun wrote: >> But this uses ustar/pax format archives. That won't satisfy the >> customer for that filed the escalation to get this fix. They insist on >> a fix using cpio format archives. > > >With this fix, these archives will no longer be "cpio format >archives", but instead >they will be "XXX Proprietary CPIO format archives". Yes and no. They are still standards conforming cpio format archives. If cpio is asked to archive sparse files while using one of these new -H option-argumments, the archive will contain entries for file types that many archivers will not recognize. You may have noticed that the 1990 standard reserved three values (C_ISCTG, C_ISLNK, and C_ISSOCK) that many implementations of cpio could not handle at that time. C_ISCTG is still reserved without definition in the current standards. > >That is, the customer will not be able to extract "holey" files as holey files >on non-Sun systems, rendering this a proprietary solution. > >Is there a plan to add this support to non-Sun archivers? No. Since cpio format archives are by definition limited to files smaller than 8Gb, we aren't spending a lot of time extending and supporting the format (other than for customer escalations like this one). There is enough detail in this case for others to recreate sparse files in this format and to create archives containing sparse files in this format if there is demand. The project team will add a note to the spio man page stating that the -H ascii_sparse and odc_sparse option-arguments should not be used for portable archives. - Don > > -John From gdamore@sun.com Mon Nov 24 17:55:05 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAP1t44s002098 for ; Mon, 24 Nov 2008 17:55:04 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id mAP1sq6R005264; Tue, 25 Nov 2008 09:55:00 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAV00L019BLTO00@nwk-avmta-2.sfbay.sun.com>; Mon, 24 Nov 2008 17:54:57 -0800 (PST) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAV00HJI9BLEX60@nwk-avmta-2.sfbay.sun.com>; Mon, 24 Nov 2008 17:54:57 -0800 (PST) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAP1svKm019966; Mon, 24 Nov 2008 17:54:57 -0800 (PST) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAV0000197LHH00@fe-sfbay-10.sun.com> (original mail from gdamore@sun.com) ; Mon, 24 Nov 2008 17:54:57 -0800 (PST) Received: from [10.7.251.172] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAV00G0P9AQWKA0@fe-sfbay-10.sun.com>; Mon, 24 Nov 2008 17:54:26 -0800 (PST) Date: Mon, 24 Nov 2008 17:48:12 -0800 From: "Garrett D'Amore" Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <200811250127.mAP1Rccd012754@spartan.eng.sun.com> Sender: Garrett.Damore@sun.com To: Don Cragun Cc: john.plocher@gmail.com, PSARC-ext@sun.com Message-id: <492B595C.5080906@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200811250127.mAP1Rccd012754@spartan.eng.sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 2212 Don Cragun wrote: >> Date: Mon, 24 Nov 2008 15:54:47 -0800 >> From: John Plocher >> >> On Mon, Nov 24, 2008 at 3:00 PM, Don Cragun wrote: >> >>> But this uses ustar/pax format archives. That won't satisfy the >>> customer for that filed the escalation to get this fix. They insist on >>> a fix using cpio format archives. >>> >> With this fix, these archives will no longer be "cpio format >> archives", but instead >> they will be "XXX Proprietary CPIO format archives". >> > > Yes and no. They are still standards conforming cpio format archives. > If cpio is asked to archive sparse files while using one of these new > -H option-argumments, the archive will contain entries for file types > that many archivers will not recognize. > > You may have noticed that the 1990 standard reserved three values > (C_ISCTG, C_ISLNK, and C_ISSOCK) that many implementations of cpio > could not handle at that time. C_ISCTG is still reserved without > definition in the current standards. > > >> That is, the customer will not be able to extract "holey" files as holey files >> on non-Sun systems, rendering this a proprietary solution. >> >> Is there a plan to add this support to non-Sun archivers? >> > > No. Since cpio format archives are by definition limited to files > smaller than 8Gb, we aren't spending a lot of time extending and > supporting the format (other than for customer escalations like this > one). There is enough detail in this case for others to recreate > sparse files in this format and to create archives containing sparse > files in this format if there is demand. > > The project team will add a note to the spio man page stating that the > -H ascii_sparse and odc_sparse option-arguments should not be used for > portable archives. > Given these limitations, I wonder if it makes sense to mark these options and the file format Uncomitted. (Maybe you did in the original, I don't recall.) We have little other way that I can think of to politely steer customers away from this and towards something that is free from the limitations. (Well, there is Committed Obsolete, I suppose....) -- Garrett From dwc@spartan.eng.sun.com Mon Nov 24 18:37:30 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAP2bUIC003264 for ; Mon, 24 Nov 2008 18:37:30 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAP2bS6B010211; Mon, 24 Nov 2008 19:37:29 -0700 (MST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAV00E01BAG2O00@brm-avmta-1.central.sun.com>; Mon, 24 Nov 2008 19:37:28 -0700 (MST) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAV00JSQBAEWWD0@brm-avmta-1.central.sun.com>; Mon, 24 Nov 2008 19:37:26 -0700 (MST) Received: from spartan.eng.sun.com (spartan.SFBay.Sun.COM [129.146.226.64]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAP2bPR5037407; Mon, 24 Nov 2008 18:37:25 -0800 (PST) Received: from spartan.eng.sun.com (localhost [127.0.0.1]) by spartan.eng.sun.com (8.13.7+Sun/8.13.7) with ESMTP id mAP2MQ10012945; Mon, 24 Nov 2008 18:22:26 -0800 (PST) Received: (from dwc@localhost) by spartan.eng.sun.com (8.13.7+Sun/8.13.7/Submit) id mAP2MQI0012944; Mon, 24 Nov 2008 18:22:26 -0800 (PST) Date: Mon, 24 Nov 2008 18:22:26 -0800 (PST) From: Don Cragun Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] To: gdamore@sun.com Cc: PSARC-ext@sun.com, cynthia.eastham@sun.com, john.plocher@gmail.com Message-id: <200811250222.mAP2MQI0012944@spartan.eng.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 2770 >Date: Mon, 24 Nov 2008 17:48:12 -0800 >From: "Garrett D'Amore" > >Don Cragun wrote: >>> Date: Mon, 24 Nov 2008 15:54:47 -0800 >>> From: John Plocher >>> >>> On Mon, Nov 24, 2008 at 3:00 PM, Don Cragun wrote: >>> >>>> But this uses ustar/pax format archives. That won't satisfy the >>>> customer for that filed the escalation to get this fix. They insist on >>>> a fix using cpio format archives. >>>> >>> With this fix, these archives will no longer be "cpio format >>> archives", but instead >>> they will be "XXX Proprietary CPIO format archives". >>> >> >> Yes and no. They are still standards conforming cpio format archives. >> If cpio is asked to archive sparse files while using one of these new >> -H option-argumments, the archive will contain entries for file types >> that many archivers will not recognize. >> >> You may have noticed that the 1990 standard reserved three values >> (C_ISCTG, C_ISLNK, and C_ISSOCK) that many implementations of cpio >> could not handle at that time. C_ISCTG is still reserved without >> definition in the current standards. >> >> >>> That is, the customer will not be able to extract "holey" files as holey files >>> on non-Sun systems, rendering this a proprietary solution. >>> >>> Is there a plan to add this support to non-Sun archivers? >>> >> >> No. Since cpio format archives are by definition limited to files >> smaller than 8Gb, we aren't spending a lot of time extending and >> supporting the format (other than for customer escalations like this >> one). There is enough detail in this case for others to recreate >> sparse files in this format and to create archives containing sparse >> files in this format if there is demand. >> >> The project team will add a note to the spio man page stating that the >> -H ascii_sparse and odc_sparse option-arguments should not be used for >> portable archives. >> > >Given these limitations, I wonder if it makes sense to mark these >options and the file format Uncomitted. (Maybe you did in the original, >I don't recall.) We have little other way that I can think of to >politely steer customers away from this and towards something that is >free from the limitations. (Well, there is Committed Obsolete, I >suppose....) I repeat: There are no new options! There is no new archive format! There is only a new file type that may appear in archives that will be used only if requested by new option-arguments to the current cpio -H option. Other than the 8Gb file size limit and other inherent limitations of the cpio archive format, there is no problem supporting these new file type entries in spio archives for the long term. - Don > > -- Garrett > > From Joerg.Schilling@fokus.fraunhofer.de Tue Nov 25 01:31:57 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAP9VvF9013218 for ; Tue, 25 Nov 2008 01:31:57 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAP9VjHZ026753; Tue, 25 Nov 2008 09:31:53 GMT Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAV00E0NUH4WP00@brm-avmta-1.central.sun.com>; Tue, 25 Nov 2008 02:31:52 -0700 (MST) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAV00LUPUH3D8C0@brm-avmta-1.central.sun.com>; Tue, 25 Nov 2008 02:31:51 -0700 (MST) Received: from relay21.sun.com (relay21.sun.com [192.12.251.24] (may be forged)) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mAP9RFHM029737; Tue, 25 Nov 2008 09:31:51 +0000 (GMT) Received: from mms24es.mms.us.syntegra.com ([150.143.232.70] [150.143.232.70]) by relay21i.sun.com with ESMTP id BT-MMP-239347; Tue, 25 Nov 2008 09:31:51 +0000 (Z) Received: from relay23.sun.com (relay23.sun.com [192.12.251.54]) by mms24es.mms.us.syntegra.com with ESMTP id BT-MMP-3276895; Tue, 25 Nov 2008 09:31:50 +0000 (Z) Received: from iron01.fraunhofer.de ([153.96.1.54] [153.96.1.54]) by relay23i.sun.com with ESMTP id BT-MMP-4151333; Tue, 25 Nov 2008 09:31:50 +0000 (Z) Received: from pluto.fokus.fraunhofer.de ([195.37.77.164]) by iron01.fraunhofer.de with ESMTP/TLS/DHE-RSA-AES256-SHA; Tue, 25 Nov 2008 10:31:49 +0100 Received: from EXCHSRV.fokus.fraunhofer.de (bohr [10.147.9.231]) by pluto.fokus.fraunhofer.de (8.13.7/8.13.7) with SMTP id mAP9VnlC028589; Tue, 25 Nov 2008 10:31:49 +0100 (MET) Received: from rigel ([10.147.65.195]) by EXCHSRV.fokus.fraunhofer.de with Microsoft SMTPSVC(6.0.3790.3959); Tue, 25 Nov 2008 10:31:49 +0100 Date: Tue, 25 Nov 2008 10:31:49 +0100 From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Fast Track 12/03/2008] In-reply-to: <200811242210.mAOMAne5012066@spartan.eng.sun.com> To: gdamore@sun.com, don.cragun@sun.com Cc: PSARC-ext@sun.com, Cynthia.Eastham@sun.com Message-id: <492bc605.ZgYL+sVdKwvYK/Fg%Joerg.Schilling@fokus.fraunhofer.de> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=0.0/5.0, scanned in 0.234sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200811242210.mAOMAne5012066@spartan.eng.sun.com> User-Agent: nail 11.22 3/20/05 X-OriginalArrivalTime: 25 Nov 2008 09:31:49.0314 (UTC) FILETIME=[A4505A20:01C94EE0] Status: RO Content-Length: 2038 Don Cragun wrote: > The case has been recast as a fast track with timeout set to > 12/03/2008. Note that this is delaying a fix for an escalation for a > large external customer (whose name I cannot give on an alias with > non-Sun participants). I thought that for Sun quality is important... Extending archive formats is something that should not be done in a hurry. I did never extend the tar format before I was sure that I did not forget important issues. Even the GNU tar people changed their sparse format after I pointed them to the problem that results from storing the hole list in the file metadata area. I thought PSARC is to have a discussion to find the best solution. > Note that a good portion of this discussion is about PSARC/2006/361; > not about this case. Well, I mentioned before and even before PSARC/2006/361 that I am willing to offer my help. You could have aske me. > >> I recommend to fix this by moving the hole information into the file data area, > >> to switch to a offset/datasize notation (to reduce the size) and to write > >> the information as comma separated pairs. > >> > > That is not this case; you are asking to reopen 2006/361. > > The holesdata for cpio presented by this case is stored in the file > data area as you are requesting; not in the header. However, the file > size (including the holesdata) is still limited by the 11 octal digit > c_filesize field in the cpio file header to a maximum size of 8Gb. Even PSARC/2006/361 was integrated less than half a year ago (see previous pax man pages that do notinclude the feature...) Because the size of the cpio file data area is limited, I recommend to use offset/datasize pairs to reduce the list size overhead. Jörg -- EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin js@cs.tu-berlin.de (uni) schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily From Joerg.Schilling@fokus.fraunhofer.de Tue Nov 25 02:24:28 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAPAORSH014746 for ; Tue, 25 Nov 2008 02:24:27 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAPAOJfr004009; Tue, 25 Nov 2008 10:24:22 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAV00C01WWLGN00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 25 Nov 2008 02:24:21 -0800 (PST) Received: from brmea-mail-3.sun.com ([192.18.98.34]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAV008KVWWJHZC0@nwk-avmta-1.sfbay.Sun.COM>; Tue, 25 Nov 2008 02:24:19 -0800 (PST) Received: from relay16i.sun.com (ip126.net129179-4.block1.us.syntegra.com [129.179.4.126]) by brmea-mail-3.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mAPAKrRr028811; Tue, 25 Nov 2008 10:24:19 +0000 (GMT) Received: from mmp14es.mmp.us.syntegra.com ([160.41.208.14] [160.41.208.14]) by relay16i.sun.com with ESMTP id BT-MMP-965571; Tue, 25 Nov 2008 10:24:19 +0000 (Z) Received: from relay17i.sun.com (relay17i.sun.com [129.179.4.127]) by mmp14es.mmp.us.syntegra.com with ESMTP id BT-MMP-41631; Tue, 25 Nov 2008 10:24:18 +0000 (Z) Received: from iron01.fraunhofer.de ([153.96.1.54] [153.96.1.54]) by relay1ib.sun.com with ESMTP id BT-MMP-10167231; Tue, 25 Nov 2008 10:24:18 +0000 (Z) Received: from pluto.fokus.fraunhofer.de ([195.37.77.164]) by iron01.fraunhofer.de with ESMTP/TLS/DHE-RSA-AES256-SHA; Tue, 25 Nov 2008 11:24:17 +0100 Received: from EXCHSRV.fokus.fraunhofer.de (bohr [10.147.9.231]) by pluto.fokus.fraunhofer.de (8.13.7/8.13.7) with SMTP id mAPAOG6d000444; Tue, 25 Nov 2008 11:24:16 +0100 (MET) Received: from rigel ([10.147.65.195]) by EXCHSRV.fokus.fraunhofer.de with Microsoft SMTPSVC(6.0.3790.3959); Tue, 25 Nov 2008 11:24:16 +0100 Date: Tue, 25 Nov 2008 11:24:13 +0100 From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <200811242237.mAOMbnLM012156@spartan.eng.sun.com> To: james.d.carlson@sun.com, don.cragun@sun.com Cc: PSARC-ext@sun.com, Cynthia.Eastham@sun.com Message-id: <492bd24d.1NC3ClFDsNAYP83R%Joerg.Schilling@fokus.fraunhofer.de> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=-1.1/5.0, scanned in 0.341sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200811242237.mAOMbnLM012156@spartan.eng.sun.com> User-Agent: nail 11.22 3/20/05 X-OriginalArrivalTime: 25 Nov 2008 10:24:16.0801 (UTC) FILETIME=[F85CC110:01C94EE7] Status: RO Content-Length: 3339 Don Cragun wrote: > There are no extensible headers in cpio format. The only place to In theory, there is a way to extend the cpio header. Glen Fowler and David Korn use this method to add a small amount of extra data to the cpio header. The method they use is to extend the space "occupied" by the file name by setting c_namesize to a value > strlen(pathname) + 1 and to write extensions into this space. On POSIX cpio archives, you may add 262143 bytes - strlen(pathname) - 1. On SVr4 cpio archives, you may add 4294967295 bytes - strlen(pathname) - 1. I would guess that many cpio implementations will dump core in case that this method is used to set c_namesize > 1024, but this is a location that could be used to add a vendor fingerprint that allows to detect the modified archive format in a reliable way. Glen Fowler uses: d For a "long" st_rdev in POSIX cpio archives g For a "long" group ID. s For a "long" file size. u For a "long" user ID. G For a group name. U For a user name. All fields are '\0' terminated and the end of the list is a double '\0'. "long" numbers are written out as intmax_t but it seems that they are read back as native C "long" only. I recommend to use: V In our case "VSUN" as a marker that this is a cpio archive with Sun extensions. and to report this to Glen Fowler and David Korn. The current pax implementation from AT&T silently skips unknown fields. > store data on where the holes go in cpio format is in the file data > area. The project team had the option of storing hole information > followed by the complete file contents or storing the hold information > followed by the file with the holes removed. Since the cpio format > only has 33 bits to store the size of the file data area, the project > team chose to remove the holes to increase the size of a sparse file > that can be archived in cpio format. While it is true that it would be > possible to encode holes data in a slightly more compact form, it is > nice to have a common format for the holes data in the ustar/pax > extended header records for sparse files and in the data area in the > cpio file data area. The decision to put the hole information into the file data area is OK for me but the simplest way to "better" compress this information is to use data_offset/data_size instead of your current proposal. > >If there are, then should we be deliberately incompatible? > > The star and the recent AT&T pax archivers encode sparse files using > ustar/pax format archives. The project team has not seen any other > attempt to encode sparse files using cpio format. So, there is no > other known cpio format that handles this case. We are not being > deliberately incompatible; nothing else handles this case. If you mark the cpio archives using the proposal from above and if you use comma separated data_offset/data_size pairs to encode the hole list, I would be willing to implement this in star too. Jörg -- EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin js@cs.tu-berlin.de (uni) schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily From carlsonj@phorcys.east.sun.com Tue Nov 25 05:07:47 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAPD7kFD003915 for ; Tue, 25 Nov 2008 05:07:46 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id mAPD7b2C001173; Tue, 25 Nov 2008 21:07:42 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAW001254GR3X00@nwk-avmta-2.sfbay.sun.com>; Tue, 25 Nov 2008 05:07:39 -0800 (PST) Received: from phorcys.east.sun.com ([129.148.174.143]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAW008AE4GRA4F0@nwk-avmta-2.sfbay.sun.com>; Tue, 25 Nov 2008 05:07:39 -0800 (PST) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id mAPCqdrd019745; Tue, 25 Nov 2008 07:52:39 -0500 (EST) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id mAPCqdKt019742; Tue, 25 Nov 2008 07:52:39 -0500 (EST) Date: Tue, 25 Nov 2008 07:52:39 -0500 From: James Carlson Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <200811242237.mAOMbnLM012156@spartan.eng.sun.com> To: Don Cragun , "Garrett D'Amore" Cc: Cynthia.Eastham@sun.com, PSARC-ext@sun.com, john.plocher@gmail.com Message-id: <18731.62743.7176.564581@gargle.gargle.HOWL> MIME-version: 1.0 X-Mailer: VM 7.01 under Emacs 21.3.1 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200811250222.mAP2MQI0012944@spartan.eng.sun.com> <200811250127.mAP1Rccd012754@spartan.eng.sun.com> <492B595C.5080906@sun.com> <200811242237.mAOMbnLM012156@spartan.eng.sun.com> Status: RO Content-Length: 1752 Don Cragun writes: > >If there are, then should we be deliberately incompatible? > > The star and the recent AT&T pax archivers encode sparse files using > ustar/pax format archives. The project team has not seen any other > attempt to encode sparse files using cpio format. That's the part I wanted to hear: that the project team had looked for other implementations and found none. I'm satisfied with that. Garrett D'Amore writes: > Given these limitations, I wonder if it makes sense to mark these > options and the file format Uncomitted. That doesn't make sense to me. The whole point of "Uncommitted" is that you believe you'll make changes in the future and thus need to warn people away. Given the limitations, what changes would we make here in the future? > (Maybe you did in the original, > I don't recall.) We have little other way that I can think of to > politely steer customers away from this and towards something that is > free from the limitations. (Well, there is Committed Obsolete, I > suppose....) "Committed Obsolete" seems more accurate, but probably not really necessary. Don Cragun writes: > I repeat: There are no new options! Please; that's unnecessary pedantry. Garrett is clearly referring to the new option argument for -H. To an ordinary user, that's an "option." The right answer, I believe, is that there's really no need to steer people hard away from cpio format. The existing man page for cpio recommends pax instead for large file support. That should be enough. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From Joerg.Schilling@fokus.fraunhofer.de Tue Nov 25 06:06:10 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAPE69cT005062 for ; Tue, 25 Nov 2008 06:06:09 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAPE67rt011743; Tue, 25 Nov 2008 06:06:08 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAW00J0L767JN00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 25 Nov 2008 06:06:07 -0800 (PST) Received: from sca-ea-mail-1.sun.com ([192.18.43.24]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAW00JRX7606P00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 25 Nov 2008 06:06:00 -0800 (PST) Received: from relay15i.sun.com (ip125.net129179-4.block1.us.syntegra.com [129.179.4.125]) by sca-ea-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAPE3ch0021285; Tue, 25 Nov 2008 14:06:00 +0000 (GMT) Received: from mmp13es.mmp.us.syntegra.com ([160.41.208.13] [160.41.208.13]) by relay15i.sun.com with ESMTP id BT-MMP-975119; Tue, 25 Nov 2008 14:05:59 +0000 (Z) Received: from relay14i.sun.com (relay14i.sun.com [129.179.4.124]) by mmp13es.mmp.us.syntegra.com with ESMTP id BT-MMP-161626; Tue, 25 Nov 2008 14:05:59 +0000 (Z) Received: from iron01.fraunhofer.de ([153.96.1.54] [153.96.1.54]) by relay1i.sun.com with ESMTP id BT-MMP-9681144; Tue, 25 Nov 2008 14:05:59 +0000 (Z) Received: from pluto.fokus.fraunhofer.de ([195.37.77.164]) by iron01.fraunhofer.de with ESMTP/TLS/DHE-RSA-AES256-SHA; Tue, 25 Nov 2008 15:05:57 +0100 Received: from EXCHSRV.fokus.fraunhofer.de (bohr [10.147.9.231]) by pluto.fokus.fraunhofer.de (8.13.7/8.13.7) with SMTP id mAPE5uur008029; Tue, 25 Nov 2008 15:05:57 +0100 (MET) Received: from rigel ([10.147.65.195]) by EXCHSRV.fokus.fraunhofer.de with Microsoft SMTPSVC(6.0.3790.3959); Tue, 25 Nov 2008 15:05:56 +0100 Date: Tue, 25 Nov 2008 15:05:56 +0100 From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <18731.62743.7176.564581@gargle.gargle.HOWL> To: james.d.carlson@sun.com, gdamore@sun.com, don.cragun@sun.com Cc: PSARC-ext@sun.com, john.plocher@gmail.com, Cynthia.Eastham@sun.com Message-id: <492c0644.6DB93ShNMaE3COUg%Joerg.Schilling@fokus.fraunhofer.de> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=0.0/5.0, scanned in 0.310sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200811250222.mAP2MQI0012944@spartan.eng.sun.com> <200811250127.mAP1Rccd012754@spartan.eng.sun.com> <492B595C.5080906@sun.com> <200811242237.mAOMbnLM012156@spartan.eng.sun.com> <18731.62743.7176.564581@gargle.gargle.HOWL> User-Agent: nail 11.22 3/20/05 X-OriginalArrivalTime: 25 Nov 2008 14:05:56.0916 (UTC) FILETIME=[EFD93F40:01C94F06] Status: RO Content-Length: 924 James Carlson wrote: > Don Cragun writes: > > >If there are, then should we be deliberately incompatible? > > > > The star and the recent AT&T pax archivers encode sparse files using > > ustar/pax format archives. The project team has not seen any other > > attempt to encode sparse files using cpio format. > > That's the part I wanted to hear: that the project team had looked for > other implementations and found none. I'm satisfied with that. Other implementation use data_offset/data_size pairs for tar archive extension since a long time. It makes sense to try to be as similar as possible. Jörg -- EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin js@cs.tu-berlin.de (uni) schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily From Joerg.Schilling@fokus.fraunhofer.de Tue Nov 25 08:02:57 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAPG2uKd001936 for ; Tue, 25 Nov 2008 08:02:57 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAPG2lot020910; Tue, 25 Nov 2008 16:02:53 GMT Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAW00703CKSRM00@brm-avmta-1.central.sun.com>; Tue, 25 Nov 2008 09:02:52 -0700 (MST) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAW007IECKR3500@brm-avmta-1.central.sun.com>; Tue, 25 Nov 2008 09:02:51 -0700 (MST) Received: from relay24.sun.com (relay24.sun.com [192.12.251.74] (may be forged)) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mAPFrPrq029273; Tue, 25 Nov 2008 16:02:51 +0000 (GMT) Received: from mms24es.mms.us.syntegra.com ([150.143.232.70] [150.143.232.70]) by relay24i.sun.com with ESMTP id BT-MMP-260655; Tue, 25 Nov 2008 16:02:50 +0000 (Z) Received: from relay23.sun.com (relay23.sun.com [192.12.251.54]) by mms24es.mms.us.syntegra.com with ESMTP id BT-MMP-3659141; Tue, 25 Nov 2008 16:02:50 +0000 (Z) Received: from iron02.fraunhofer.de ([153.96.1.56] [153.96.1.56]) by relay23i.sun.com with ESMTP id BT-MMP-4609383; Tue, 25 Nov 2008 16:02:50 +0000 (Z) Received: from pluto.fokus.fraunhofer.de ([195.37.77.164]) by iron02.fraunhofer.de with ESMTP/TLS/DHE-RSA-AES256-SHA; Tue, 25 Nov 2008 17:02:49 +0100 Received: from EXCHSRV.fokus.fraunhofer.de (bohr [10.147.9.231]) by pluto.fokus.fraunhofer.de (8.13.7/8.13.7) with SMTP id mAPG2mmV011893; Tue, 25 Nov 2008 17:02:48 +0100 (MET) Received: from rigel ([10.147.65.195]) by EXCHSRV.fokus.fraunhofer.de with Microsoft SMTPSVC(6.0.3790.3959); Tue, 25 Nov 2008 17:02:48 +0100 Date: Tue, 25 Nov 2008 17:02:48 +0100 From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <200811242300.mAON0o7g012243@spartan.eng.sun.com> To: don.cragun@sun.com Cc: PSARC-ext@sun.com, Cynthia.Eastham@sun.com Message-id: <492c21a8.Lzt48aGfXkWnekeB%Joerg.Schilling@fokus.fraunhofer.de> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=0.0/5.0, scanned in 0.314sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200811242300.mAON0o7g012243@spartan.eng.sun.com> User-Agent: nail 11.22 3/20/05 X-OriginalArrivalTime: 25 Nov 2008 16:02:48.0759 (UTC) FILETIME=[433B9470:01C94F17] Status: RO Content-Length: 1840 Don Cragun wrote: > >A strong reason for going to star. Star implements all available features > >in it's "star" personality. > > Your star archiver uses extended headers to encode the holes data. > There are no extended headers in cpio format archives. The customer > insists on cpio format archives. Are you 100% that a customer likes to get enhancements in an outdated archive format? It sounds more reasonable to ask for the cpio(1) cmdline interface. Do you know the reason for asking for a cpio based archive format? In any case, the advantage of star is to have a single implementation that support different the CLI for tar, cpio, pax, ... > >For cpio archives (see SUSv2), the standard just mentiones that archives that > >contain "additional file types" should not be transported to other systems. > >Will the man page include a hint on the portability issues? > > You're correct. I didn't go back to the 1990 POSIX.1 standard after > reading the rationale in the current standard. Until I went back and > looked at it this afternoon, I thought the requirement was to extract > an unknown file type in any archive format as a regular file and to > print a diagnostic message saying that the conversions to regular file > had been done. Our current cpio and pax convert files of unknown type > to a regular file, but don't print a diagnostic. When this case is > approved, a bug report will be filed so cpio and pax will print a > diagnostic in this case no matter what archive format is being read. OK Jörg -- EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin js@cs.tu-berlin.de (uni) schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily From Joerg.Schilling@fokus.fraunhofer.de Tue Nov 25 08:23:53 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAPGNqmv023288 for ; Tue, 25 Nov 2008 08:23:52 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id mAPGNkMQ001795; Wed, 26 Nov 2008 00:23:47 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAW00CWNDJMDK00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 25 Nov 2008 08:23:46 -0800 (PST) Received: from brmea-mail-3.sun.com ([192.18.98.34]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAW007QWDFFLD40@nwk-avmta-1.sfbay.Sun.COM>; Tue, 25 Nov 2008 08:21:15 -0800 (PST) Received: from relay22.sun.com (relay22.sun.com [192.12.251.34] (may be forged)) by brmea-mail-3.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mAPGF60v027701; Tue, 25 Nov 2008 16:21:14 +0000 (GMT) Received: from mms25es.mms.us.syntegra.com ([150.143.232.90] [150.143.232.90]) by relay22i.sun.com with ESMTP id BT-MMP-263488; Tue, 25 Nov 2008 16:21:14 +0000 (Z) Received: from relay24.sun.com (relay24.sun.com [192.12.251.74]) by mms25es.mms.us.syntegra.com with ESMTP id BT-MMP-3677509; Tue, 25 Nov 2008 16:21:13 +0000 (Z) Received: from iron01.fraunhofer.de ([153.96.1.54] [153.96.1.54]) by relay24i.sun.com with ESMTP id BT-MMP-4629777; Tue, 25 Nov 2008 16:21:13 +0000 (Z) Received: from pluto.fokus.fraunhofer.de ([195.37.77.164]) by iron01.fraunhofer.de with ESMTP/TLS/DHE-RSA-AES256-SHA; Tue, 25 Nov 2008 17:21:12 +0100 Received: from EXCHSRV.fokus.fraunhofer.de (bohr [10.147.9.231]) by pluto.fokus.fraunhofer.de (8.13.7/8.13.7) with SMTP id mAPGLCHc012605; Tue, 25 Nov 2008 17:21:12 +0100 (MET) Received: from rigel ([10.147.65.195]) by EXCHSRV.fokus.fraunhofer.de with Microsoft SMTPSVC(6.0.3790.3959); Tue, 25 Nov 2008 17:21:12 +0100 Date: Tue, 25 Nov 2008 17:21:12 +0100 From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <200811250020.mAP0KpCs012533@spartan.eng.sun.com> To: don.cragun@sun.com Cc: PSARC-ext@sun.com, Cynthia.Eastham@sun.com Message-id: <492c25f8.7DaRjh7V5+8ZHQZX%Joerg.Schilling@fokus.fraunhofer.de> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=-0.7/5.0, scanned in 0.342sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200811250020.mAP0KpCs012533@spartan.eng.sun.com> User-Agent: nail 11.22 3/20/05 X-OriginalArrivalTime: 25 Nov 2008 16:21:12.0366 (UTC) FILETIME=[D508A4E0:01C94F19] Status: RO Content-Length: 348 Don Cragun wrote: > >I canot speak for _very_ recent Solaris versions but for a case from 2006, > >I would expect to see results before the latest (Build89) I checked. > > > >Build 89 comes with a pax man page from 2004 > > I don't know when the man page got out to OpenSolaris, but the pax > fixes went into Nevada build 54. From an ISO-9000 standpoint, an undocumented feature does not exist ;-) Status: RO Content-Length: 2904 > >As a result of not marking the archive, archivers that carefully implement add > >on features depending on the archive format will not unpack the sparse files. > > > >star and AT&T pax will ignore bit 17, other archivers may include this > >bit in the file type with unkown results. > > That is a bug in star and AT&T pax (as well as in Sun's current cpio > and pax). A bug report will be filed to correct Sun's cpio and pax > when this case is resolved. Your material was not clear. As it now seems that bit 17 is just added, the results are known. Bit 17 is bejond the POSIX standard and thus ignored by a conforming implementation to avoid problems. > >OK; how about marking the archive in the header area past the filename? > > As I'm sure you know, the data describing the contents of the file > immediately follow the header in a cpio archive. The standard does not > allow any new fields to be added to the cpio archive format. The standard does not allow to add mode bits beyond bit 16. I send a description on how to extend the cpio header in an earlier mail today. > >It seems that you are too US centric and thus do not see the problem of > >being unable to see number pairs in a possiblily extremely long data > >stream. > > As you know, the cpio header is a string of 76 octal digits followed > immediately by the pathname of the file (c_namesize bytes including a > trailing NUL byte) followed immediately by the contents of the file > (c_filesize bytes). This header is not easy for humans to read. This > header is not intended to be read by humans; it is intended to be read > by cpio and other archivers that want to extract data from cpio > archives. The same is true of the holes data record both in the > ustar/pax archive format extended headers and in the initial portion of > the file data in the cpio archive format. The cpio header is small and it is possible to parse it as a human if needed. I was talking about the hole list that may become really huge. It seems that you never did try to debug this..... I did this 2 years ago when it turned out that all GNU tar implementations published before, did create broken sparse archives for files > 2 GB. The GNU tar maintainers did not work on the problem before I identified the bug and I was trying to find the background and whether I could implement a workaround. This was an archive with aprox. a million holes. With today GNU tar archives I had real problems to manually scan the data. At that time, the archive did use the old GNU tar definition from the 1990 that allows to see the hole data in pairs. Jörg -- EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin js@cs.tu-berlin.de (uni) schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily From Joerg.Schilling@fokus.fraunhofer.de Tue Nov 25 10:27:33 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAPIRWoG029011 for ; Tue, 25 Nov 2008 10:27:32 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAPIRMfw005493; Tue, 25 Nov 2008 18:27:28 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAW0060RJ9M5800@nwk-avmta-1.sfbay.Sun.COM>; Tue, 25 Nov 2008 10:27:22 -0800 (PST) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAW005UVJ9MIA00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 25 Nov 2008 10:27:22 -0800 (PST) Received: from relay15i.sun.com (ip125.net129179-4.block1.us.syntegra.com [129.179.4.125]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mAPIIfop008095; Tue, 25 Nov 2008 18:27:21 +0000 (GMT) Received: from mmp11es.mmp.us.syntegra.com ([160.41.208.11] [160.41.208.11]) by relay15i.sun.com with ESMTP id BT-MMP-991857; Tue, 25 Nov 2008 18:27:21 +0000 (Z) Received: from relay15i.sun.com (relay15i.sun.com [129.179.4.125]) by mmp11es.mmp.us.syntegra.com with ESMTP id BT-MMP-4765954; Tue, 25 Nov 2008 18:27:21 +0000 (Z) Received: from iron02.fraunhofer.de ([153.96.1.56] [153.96.1.56]) by relay1ib.sun.com with ESMTP id BT-MMP-10373346; Tue, 25 Nov 2008 18:27:20 +0000 (Z) Received: from pluto.fokus.fraunhofer.de ([195.37.77.164]) by iron02.fraunhofer.de with ESMTP/TLS/DHE-RSA-AES256-SHA; Tue, 25 Nov 2008 19:27:20 +0100 Received: from EXCHSRV.fokus.fraunhofer.de (bohr [10.147.9.231]) by pluto.fokus.fraunhofer.de (8.13.7/8.13.7) with SMTP id mAPIRJ1M016603; Tue, 25 Nov 2008 19:27:19 +0100 (MET) Received: from rigel ([10.147.65.195]) by EXCHSRV.fokus.fraunhofer.de with Microsoft SMTPSVC(6.0.3790.3959); Tue, 25 Nov 2008 19:27:19 +0100 Date: Tue, 25 Nov 2008 19:27:18 +0100 From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <492bd24d.1NC3ClFDsNAYP83R%Joerg.Schilling@fokus.fraunhofer.de> To: James.D.Carlson@sun.com, don.cragun@sun.com Cc: PSARC-ext@sun.com, Cynthia.Eastham@sun.com Message-id: <492c4386.yse3oSB3TJtox+HJ%Joerg.Schilling@fokus.fraunhofer.de> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=0.0/5.0, scanned in 0.325sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200811242237.mAOMbnLM012156@spartan.eng.sun.com> <492bd24d.1NC3ClFDsNAYP83R%Joerg.Schilling@fokus.fraunhofer.de> User-Agent: nail 11.22 3/20/05 X-OriginalArrivalTime: 25 Nov 2008 18:27:19.0216 (UTC) FILETIME=[733A5300:01C94F2B] Status: RO Content-Length: 2386 Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) wrote: > > >If there are, then should we be deliberately incompatible? > > > > The star and the recent AT&T pax archivers encode sparse files using > > ustar/pax format archives. The project team has not seen any other > > attempt to encode sparse files using cpio format. So, there is no > > other known cpio format that handles this case. We are not being > > deliberately incompatible; nothing else handles this case. > > If you mark the cpio archives using the proposal from above and if you > use comma separated data_offset/data_size pairs to encode the hole list, > I would be willing to implement this in star too. Let me make a completely new proposal that does not touch any POSIX defined field and that allows to add further extensions in the future. It is based on extension ideas from David Korn and Glenn Fowler. If the archive type name is either "ascii_sparse" or "odc_sparse", the string "VSUN\0\0" is appended to all file names in the cpio header and the field c_namesize is filled with a number that is 6 bigger than expected for a standard cpio archive. If a sparse file is encountered, the following string is used instead: "VSUN\0S%jx\0\0", (uintmax_t)statbuf.st_size c_namesize is filled with a number that is of the apropriate size bigger than in a standard cpio archive. In case of a sparse file, the following hole list is used inside the file data cunk: "%jx\n%s", (uintmax_t)number_of_holes_in_list The "%s" string is replaced by "number_of_holes_in_list" entris of the following form: "%jx %jx\n", (uintmax_t)data_offset, (uintmax_t)data_size Sparse files that end in a hole are treated the same was as in star. Using hex numbers instead of decimal numbers is aligned with the AT&T format. Using hex numbers instead of decimal numbers reduces the size of the hole list by 20%. Using "data_size" instead of "hole_offset" reduces the hole list by another 20%. The field c_filesize contains a size that is equal to the compressed file size + the size of the hole list. Please comment! Jörg -- EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin js@cs.tu-berlin.de (uni) schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily From gdamore@sun.com Tue Nov 25 10:52:14 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAPIqEet000103 for ; Tue, 25 Nov 2008 10:52:14 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAPIqDG4024837; Tue, 25 Nov 2008 10:52:13 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAW00L0PKEZBE00@nwk-avmta-2.sfbay.sun.com>; Tue, 25 Nov 2008 10:52:11 -0800 (PST) Received: from sca-es-mail-2.sun.com ([192.18.43.133]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAW00AF8KEYZ5D0@nwk-avmta-2.sfbay.sun.com>; Tue, 25 Nov 2008 10:52:10 -0800 (PST) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAPIqAAH012769; Tue, 25 Nov 2008 10:52:10 -0800 (PST) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAW00C01J8VOZ00@fe-sfbay-10.sun.com> (original mail from gdamore@sun.com) ; Tue, 25 Nov 2008 10:52:10 -0800 (PST) Received: from [10.7.251.172] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAW00BEVKEPTQ00@fe-sfbay-10.sun.com>; Tue, 25 Nov 2008 10:52:01 -0800 (PST) Date: Tue, 25 Nov 2008 10:45:43 -0800 From: "Garrett D'Amore" Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <492c4386.yse3oSB3TJtox+HJ%Joerg.Schilling@fokus.fraunhofer.de> Sender: Garrett.Damore@sun.com To: Joerg Schilling Cc: James.D.Carlson@sun.com, don.cragun@sun.com, PSARC-ext@sun.com, Cynthia.Eastham@sun.com Message-id: <492C47D7.6080507@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 References: <200811242237.mAOMbnLM012156@spartan.eng.sun.com> <492bd24d.1NC3ClFDsNAYP83R%Joerg.Schilling@fokus.fraunhofer.de> <492c4386.yse3oSB3TJtox+HJ%Joerg.Schilling@fokus.fraunhofer.de> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 3800 I have since learned, from the project team, that this problem is primarily centered around solving issues with Live Upgrade. They are in a bind on timing, because they are coming up on a key deliverable date. Redesign of the file format, and resetting the testing that has already been done, would be detrimental to some key deliverables around Live Upgrade and Solaris 10. Therefore, I recommend an alternate proposal, which I think might satisfy most of the participants here. 1) Relegate the ascii_sparse and odc_sparse arguments, and the associated file formats to "undocumented" status. That is, they won't be documented in the man pages. This way nobody should have to worry too much about dealing with these file formats portably. (Neither AT&T, nor GNU, nor star, ever needs deal with them.) These arguments and file formats will have "Project Private" binding. 2) File a contract for their use by the LU (or any other projects) that need them. (At the same time, we should be giving advice to these projects that they should try to convert to use pax if possible.) 3) Anyone else looking for sparse file support should be strongly urged to use the existing pax (or star, if you prefer. I don't want to get into a debate about Sun pax vs. star vs GNU tar. Its not this case.) This would allow the project to go forward with their existing code, with minimal disruption to their plans, and minimal disruption to "documented" formats that external parties (star, GNU tar, AT&T) have to be prepared to accept. -- Garrett Joerg Schilling wrote: > Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) wrote: > > >>>> If there are, then should we be deliberately incompatible? >>>> >>> The star and the recent AT&T pax archivers encode sparse files using >>> ustar/pax format archives. The project team has not seen any other >>> attempt to encode sparse files using cpio format. So, there is no >>> other known cpio format that handles this case. We are not being >>> deliberately incompatible; nothing else handles this case. >>> >> If you mark the cpio archives using the proposal from above and if you >> use comma separated data_offset/data_size pairs to encode the hole list, >> I would be willing to implement this in star too. >> > > Let me make a completely new proposal that does not touch any POSIX defined > field and that allows to add further extensions in the future. It is based on > extension ideas from David Korn and Glenn Fowler. > > If the archive type name is either "ascii_sparse" or "odc_sparse", the string > > "VSUN\0\0" is appended to all file names in the cpio header and the field > c_namesize is filled with a number that is 6 bigger than expected for a standard > cpio archive. > > If a sparse file is encountered, the following string is used instead: > > "VSUN\0S%jx\0\0", (uintmax_t)statbuf.st_size > > c_namesize is filled with a number that is of the apropriate size bigger than > in a standard cpio archive. > > In case of a sparse file, the following hole list is used inside the file data > cunk: > > "%jx\n%s", (uintmax_t)number_of_holes_in_list > > The "%s" string is replaced by "number_of_holes_in_list" entris of the > following form: > > "%jx %jx\n", (uintmax_t)data_offset, (uintmax_t)data_size > > Sparse files that end in a hole are treated the same was as in star. > > > Using hex numbers instead of decimal numbers is aligned with the AT&T format. > Using hex numbers instead of decimal numbers reduces the size of the hole list > by 20%. Using "data_size" instead of "hole_offset" reduces the hole list by > another 20%. > > The field c_filesize contains a size that is equal to the compressed file size > + the size of the hole list. > > Please comment! > > Jörg > > From Darren.Moffat@sun.com Tue Nov 25 11:00:40 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAPJ0e0r001066 for ; Tue, 25 Nov 2008 11:00:40 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAPJ0We5031782; Tue, 25 Nov 2008 12:00:39 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAW00A35KT29L00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 25 Nov 2008 11:00:38 -0800 (PST) Received: from gmp-eb-inf-2.sun.com ([192.18.6.24]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAW0054FKT0I890@nwk-avmta-1.sfbay.Sun.COM>; Tue, 25 Nov 2008 11:00:37 -0800 (PST) Received: from fe-emea-10.sun.com (gmp-eb-lb-2-fe2.eu.sun.com [192.18.6.11]) by gmp-eb-inf-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAPJ0aCE018672; Tue, 25 Nov 2008 19:00:36 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAW00F01KMZR400@fe-emea-10.sun.com> (original mail from Darren.Moffat@Sun.COM); Tue, 25 Nov 2008 19:00:36 +0000 (GMT) Received: from [129.156.173.21] by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAW00EWSKSZB4B0@fe-emea-10.sun.com>; Tue, 25 Nov 2008 19:00:36 +0000 (GMT) Date: Tue, 25 Nov 2008 19:00:35 +0000 From: Darren J Moffat Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <492C47D7.6080507@sun.com> Sender: Darren.Moffat@sun.com To: "Garrett D'Amore" Cc: Joerg Schilling , James.D.Carlson@sun.com, don.cragun@sun.com, PSARC-ext@sun.com, Cynthia.Eastham@sun.com Message-id: <492C4B53.1080409@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200811242237.mAOMbnLM012156@spartan.eng.sun.com> <492bd24d.1NC3ClFDsNAYP83R%Joerg.Schilling@fokus.fraunhofer.de> <492c4386.yse3oSB3TJtox+HJ%Joerg.Schilling@fokus.fraunhofer.de> <492C47D7.6080507@sun.com> User-Agent: Thunderbird 2.0.0.16 (X11/20080922) Status: RO Content-Length: 1819 Garrett D'Amore wrote: > I have since learned, from the project team, that this problem is > primarily centered around solving issues with Live Upgrade. They are in > a bind on timing, because they are coming up on a key deliverable date. > Redesign of the file format, and resetting the testing that has already > been done, would be detrimental to some key deliverables around Live > Upgrade and Solaris 10. > > Therefore, I recommend an alternate proposal, which I think might > satisfy most of the participants here. > > 1) Relegate the ascii_sparse and odc_sparse arguments, and the > associated file formats to "undocumented" status. That is, they won't > be documented in the man pages. This way nobody should have to worry > too much about dealing with these file formats portably. (Neither AT&T, > nor GNU, nor star, ever needs deal with them.) These arguments and file > formats will have "Project Private" binding. > > 2) File a contract for their use by the LU (or any other projects) that > need them. (At the same time, we should be giving advice to these > projects that they should try to convert to use pax if possible.) > > 3) Anyone else looking for sparse file support should be strongly urged > to use the existing pax (or star, if you prefer. I don't want to get > into a debate about Sun pax vs. star vs GNU tar. Its not this case.) > > This would allow the project to go forward with their existing code, > with minimal disruption to their plans, and minimal disruption to > "documented" formats that external parties (star, GNU tar, AT&T) have to > be prepared to accept. I agree with this proposed direction. Particularly sinve Live Upgrade is not in OpenSolaris which uses the newly designed that shares no code Snap upgrade instead. -- Darren J Moffat From carlsonj@phorcys.east.sun.com Tue Nov 25 11:07:28 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAPJ7SpE001590 for ; Tue, 25 Nov 2008 11:07:28 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAPJ7Odf012987; Tue, 25 Nov 2008 11:07:27 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAW0025TL4EYQ00@brm-avmta-1.central.sun.com>; Tue, 25 Nov 2008 12:07:26 -0700 (MST) Received: from phorcys.east.sun.com ([129.148.174.143]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAW000UAL4ESG10@brm-avmta-1.central.sun.com>; Tue, 25 Nov 2008 12:07:26 -0700 (MST) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id mAPJ7Q58022056; Tue, 25 Nov 2008 14:07:26 -0500 (EST) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id mAPJ7Q6r022053; Tue, 25 Nov 2008 14:07:26 -0500 (EST) Date: Tue, 25 Nov 2008 14:07:26 -0500 From: James Carlson Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <492C47D7.6080507@sun.com> To: "Garrett D'Amore" Cc: PSARC-ext@sun.com, don.cragun@sun.com, Cynthia.Eastham@sun.com Message-id: <18732.19694.183169.134167@gargle.gargle.HOWL> MIME-version: 1.0 X-Mailer: VM 7.01 under Emacs 21.3.1 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200811242237.mAOMbnLM012156@spartan.eng.sun.com> <492bd24d.1NC3ClFDsNAYP83R%Joerg.Schilling@fokus.fraunhofer.de> <492c4386.yse3oSB3TJtox+HJ%Joerg.Schilling@fokus.fraunhofer.de> <492C47D7.6080507@sun.com> Status: RO Content-Length: 1484 Garrett D'Amore writes: > I have since learned, from the project team, that this problem is > primarily centered around solving issues with Live Upgrade. They are in > a bind on timing, because they are coming up on a key deliverable date. > Redesign of the file format, and resetting the testing that has already > been done, would be detrimental to some key deliverables around Live > Upgrade and Solaris 10. I'd be mostly happy with that or with the original proposal to make these new things Committed option-arguments. I hope the project team has considered this issue carefully, but I think one of the unfortunate artifacts here is that (unless there are some fancy tricks involved), doing this will require that there be ON patches released for both S8 and S9 to upgrade cpio in order to allow those special LU packages (using the new -H argument) to continue to be used to upgrade to modern S10 releases. Don't forget that SUNWlu{u,r,cfg} must all install and run on release N-2 in order to upgrade to release N. As an outsider, it seems to me that it must almost certainly be simpler to rewrite the handful of places that currently use cpio to use pax instead than to do three backports and separate patch releases. But what do I know ... -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From Andrew.Gabriel@sun.com Tue Nov 25 11:47:24 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAPJlO33002473 for ; Tue, 25 Nov 2008 11:47:24 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAPJlM9p006946; Tue, 25 Nov 2008 11:47:23 -0800 (PST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAW00601MYYSF00@brm-avmta-1.central.sun.com>; Tue, 25 Nov 2008 12:47:22 -0700 (MST) Received: from gmp-eb-inf-2.sun.com ([192.18.6.24]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAW000N5MYXRS60@brm-avmta-1.central.sun.com>; Tue, 25 Nov 2008 12:47:22 -0700 (MST) Received: from fe-emea-09.sun.com (gmp-eb-lb-1-fe3.eu.sun.com [192.18.6.10]) by gmp-eb-inf-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAPJlL8F020824; Tue, 25 Nov 2008 19:47:21 +0000 (GMT) Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAW00B01MV2LC00@fe-emea-09.sun.com> (original mail from Andrew.Gabriel@Sun.COM); Tue, 25 Nov 2008 19:47:21 +0000 (GMT) Received: from [10.47.98.207] ([212.183.134.67]) by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAW00G2JMYR2280@fe-emea-09.sun.com>; Tue, 25 Nov 2008 19:47:21 +0000 (GMT) Date: Tue, 25 Nov 2008 19:47:09 +0000 From: Andrew Gabriel Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <492C47D7.6080507@sun.com> Sender: Andrew.Gabriel@sun.com To: "Garrett D'Amore" Cc: Joerg Schilling , James.D.Carlson@sun.com, don.cragun@sun.com, PSARC-ext@sun.com, Cynthia.Eastham@sun.com Message-id: <492C563D.1060403@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200811242237.mAOMbnLM012156@spartan.eng.sun.com> <492bd24d.1NC3ClFDsNAYP83R%Joerg.Schilling@fokus.fraunhofer.de> <492c4386.yse3oSB3TJtox+HJ%Joerg.Schilling@fokus.fraunhofer.de> <492C47D7.6080507@sun.com> User-Agent: Thunderbird 2.0.0.18 (Windows/20081105) Status: RO Content-Length: 683 Garrett D'Amore wrote: > 3) Anyone else looking for sparse file support should be strongly > urged to use the existing pax (or star, if you prefer. I don't want > to get into a debate about Sun pax vs. star vs GNU tar. Its not this > case.) I think that's a shame. The standard sysadmin archivers are tar and cpio. pax has never come close to being adopted, and given how long it's been around, I think I can safely say it never will. It would be good to see sparse support in cpio. At least, I hope it will do so in copy (-p) mode where this archive format discussion is irrelevant, and the fact it doesn't copy the original file faithfully is simply a bug. -- Andrew From Joerg.Schilling@fokus.fraunhofer.de Tue Nov 25 12:59:34 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAPKxWvp019970 for ; Tue, 25 Nov 2008 12:59:33 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id mAPKxH3Q014690; Wed, 26 Nov 2008 04:59:25 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAW0050HQB0WF00@nwk-avmta-2.sfbay.sun.com>; Tue, 25 Nov 2008 12:59:24 -0800 (PST) Received: from sca-ea-mail-1.sun.com ([192.18.43.24]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAW004PYQAUYV20@nwk-avmta-2.sfbay.sun.com>; Tue, 25 Nov 2008 12:59:19 -0800 (PST) Received: from relay24.sun.com (relay24.sun.com [192.12.251.74] (may be forged)) by sca-ea-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAPKwb2E022104; Tue, 25 Nov 2008 20:59:18 +0000 (GMT) Received: from mms25es.mms.us.syntegra.com ([150.143.232.90] [150.143.232.90]) by relay24i.sun.com with ESMTP id BT-MMP-282591; Tue, 25 Nov 2008 20:59:18 +0000 (Z) Received: from relay24.sun.com (relay24.sun.com [192.12.251.74]) by mms25es.mms.us.syntegra.com with ESMTP id BT-MMP-3918611; Tue, 25 Nov 2008 20:59:18 +0000 (Z) Received: from iron02.fraunhofer.de ([153.96.1.56] [153.96.1.56]) by relay24i.sun.com with ESMTP id BT-MMP-4932183; Tue, 25 Nov 2008 20:59:17 +0000 (Z) Received: from pluto.fokus.fraunhofer.de ([195.37.77.164]) by iron02.fraunhofer.de with ESMTP/TLS/DHE-RSA-AES256-SHA; Tue, 25 Nov 2008 21:59:16 +0100 Received: from EXCHSRV.fokus.fraunhofer.de (bohr [10.147.9.231]) by pluto.fokus.fraunhofer.de (8.13.7/8.13.7) with SMTP id mAPKxFdP020113; Tue, 25 Nov 2008 21:59:15 +0100 (MET) Received: from rigel ([10.147.65.195]) by EXCHSRV.fokus.fraunhofer.de with Microsoft SMTPSVC(6.0.3790.3959); Tue, 25 Nov 2008 21:59:16 +0100 Date: Tue, 25 Nov 2008 21:59:12 +0100 From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <492C47D7.6080507@sun.com> To: gdamore@sun.com Cc: PSARC-ext@sun.com, James.D.Carlson@sun.com, don.cragun@sun.com, Cynthia.Eastham@sun.com Message-id: <492c6720.F4uBIUBsSAqOjKzu%Joerg.Schilling@fokus.fraunhofer.de> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=-2.6/5.0, scanned in 0.339sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200811242237.mAOMbnLM012156@spartan.eng.sun.com> <492bd24d.1NC3ClFDsNAYP83R%Joerg.Schilling@fokus.fraunhofer.de> <492c4386.yse3oSB3TJtox+HJ%Joerg.Schilling@fokus.fraunhofer.de> <492C47D7.6080507@sun.com> User-Agent: nail 11.22 3/20/05 X-OriginalArrivalTime: 25 Nov 2008 20:59:16.0261 (UTC) FILETIME=[AD68FD50:01C94F40] Status: RO Content-Length: 3111 "Garrett D'Amore" wrote: > I have since learned, from the project team, that this problem is > primarily centered around solving issues with Live Upgrade. They are in > a bind on timing, because they are coming up on a key deliverable date. > Redesign of the file format, and resetting the testing that has already > been done, would be detrimental to some key deliverables around Live > Upgrade and Solaris 10. If I understand the background correctly, I would guess that there only seems to be a need for cpio -p. Is there a need to grant the ability to unpack cpio archives that use the feature for more than one hour? BTW: I expect the effort to change an existing implementation to my proposal to less than 5% of the effort already spent. The changes needed for my proposal are marginal compared to the complete effort for a correct implementation. > Therefore, I recommend an alternate proposal, which I think might > satisfy most of the participants here. > > 1) Relegate the ascii_sparse and odc_sparse arguments, and the > associated file formats to "undocumented" status. That is, they won't > be documented in the man pages. This way nobody should have to worry > too much about dealing with these file formats portably. (Neither AT&T, > nor GNU, nor star, ever needs deal with them.) These arguments and file > formats will have "Project Private" binding. If the result would be that possihle a cpio replacement does not need to support the same archive format, I have no problems with this idea. > 2) File a contract for their use by the LU (or any other projects) that > need them. (At the same time, we should be giving advice to these > projects that they should try to convert to use pax if possible.) > > 3) Anyone else looking for sparse file support should be strongly urged > to use the existing pax (or star, if you prefer. I don't want to get > into a debate about Sun pax vs. star vs GNU tar. Its not this case.) > > This would allow the project to go forward with their existing code, > with minimal disruption to their plans, and minimal disruption to > "documented" formats that external parties (star, GNU tar, AT&T) have to > be prepared to accept. I would be happy to see a single OpenSource implementation in future that has the advantage to grant the same features regardless of the CLI being used. pax was a result of the POSIX tar-wars that started around 1990. pax was never really accepted by the community and the sysadmins that mostly prefer tar. cpio seems to be preferred by people with an AT&T based UNIX background. Solaris currently has three archiver implementations and every idea needs to be implemented and tested three times. In the long term, it pays off to make a change to a single star based solution. Jörg -- EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin js@cs.tu-berlin.de (uni) schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily From cynthia.eastham@sun.com Tue Nov 25 15:32:08 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAPNW7ah000618 for ; Tue, 25 Nov 2008 15:32:08 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAPNVuoY003712; Tue, 25 Nov 2008 23:32:04 GMT Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAW00E0LXDEIL00@nwk-avmta-2.sfbay.sun.com>; Tue, 25 Nov 2008 15:32:02 -0800 (PST) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAW00CA4XDE2D30@nwk-avmta-2.sfbay.sun.com>; Tue, 25 Nov 2008 15:32:02 -0800 (PST) Received: from [10.7.250.74] (punchin-client-10-7-250-74.SFBay.Sun.COM [10.7.250.74]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id mAPNW19V151193; Tue, 25 Nov 2008 15:32:01 -0800 (PST) Date: Tue, 25 Nov 2008 15:32:01 -0800 From: Cynthia Eastham Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <492C47D7.6080507@sun.com> To: "Garrett D'Amore" Cc: Joerg Schilling , James.D.Carlson@sun.com, don.cragun@sun.com, PSARC-ext@sun.com Message-id: <492C8AF1.7090208@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 References: <200811242237.mAOMbnLM012156@spartan.eng.sun.com> <492bd24d.1NC3ClFDsNAYP83R%Joerg.Schilling@fokus.fraunhofer.de> <492c4386.yse3oSB3TJtox+HJ%Joerg.Schilling@fokus.fraunhofer.de> <492C47D7.6080507@sun.com> User-Agent: Thunderbird 2.0.0.6 (X11/20071023) Status: RO Content-Length: 4459 Garrett D'Amore wrote: > I have since learned, from the project team, that this problem is > primarily centered around solving issues with Live Upgrade. They are in > a bind on timing, because they are coming up on a key deliverable date. > Redesign of the file format, and resetting the testing that has already > been done, would be detrimental to some key deliverables around Live > Upgrade and Solaris 10. > > Therefore, I recommend an alternate proposal, which I think might > satisfy most of the participants here. > > 1) Relegate the ascii_sparse and odc_sparse arguments, and the > associated file formats to "undocumented" status. That is, they won't > be documented in the man pages. This way nobody should have to worry > too much about dealing with these file formats portably. (Neither AT&T, > nor GNU, nor star, ever needs deal with them.) These arguments and file > formats will have "Project Private" binding. > > 2) File a contract for their use by the LU (or any other projects) that > need them. (At the same time, we should be giving advice to these > projects that they should try to convert to use pax if possible.) > > 3) Anyone else looking for sparse file support should be strongly urged > to use the existing pax (or star, if you prefer. I don't want to get > into a debate about Sun pax vs. star vs GNU tar. Its not this case.) > > This would allow the project to go forward with their existing code, > with minimal disruption to their plans, and minimal disruption to > "documented" formats that external parties (star, GNU tar, AT&T) have to > be prepared to accept. > > -- Garrett > > Joerg Schilling wrote: >> Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) wrote: >> >> >>>>> If there are, then should we be deliberately incompatible? >>>>> >>>> The star and the recent AT&T pax archivers encode sparse files using >>>> ustar/pax format archives. The project team has not seen any other >>>> attempt to encode sparse files using cpio format. So, there is no >>>> other known cpio format that handles this case. We are not being >>>> deliberately incompatible; nothing else handles this case. >>>> >>> If you mark the cpio archives using the proposal from above and if you >>> use comma separated data_offset/data_size pairs to encode the hole list, >>> I would be willing to implement this in star too. >>> >> >> Let me make a completely new proposal that does not touch any POSIX >> defined field and that allows to add further extensions in the future. >> It is based on >> extension ideas from David Korn and Glenn Fowler. >> >> If the archive type name is either "ascii_sparse" or "odc_sparse", the >> string >> >> "VSUN\0\0" is appended to all file names in the cpio header and the >> field c_namesize is filled with a number that is 6 bigger than >> expected for a standard >> cpio archive. >> >> If a sparse file is encountered, the following string is used instead: >> >> "VSUN\0S%jx\0\0", (uintmax_t)statbuf.st_size >> >> c_namesize is filled with a number that is of the apropriate size >> bigger than >> in a standard cpio archive. >> >> In case of a sparse file, the following hole list is used inside the >> file data cunk: >> >> "%jx\n%s", (uintmax_t)number_of_holes_in_list >> >> The "%s" string is replaced by "number_of_holes_in_list" entris of the >> following form: >> >> "%jx %jx\n", (uintmax_t)data_offset, (uintmax_t)data_size >> >> Sparse files that end in a hole are treated the same was as in star. >> >> >> Using hex numbers instead of decimal numbers is aligned with the AT&T >> format. >> Using hex numbers instead of decimal numbers reduces the size of the >> hole list by 20%. Using "data_size" instead of "hole_offset" reduces >> the hole list by another 20%. >> >> The field c_filesize contains a size that is equal to the compressed >> file size + the size of the hole list. >> >> Please comment! >> >> Jörg >> >> > I've talked with Don Cragun, who is off-line today for medical reasons, about Garrett's proposal as well as Joerg's proposal. Due to extreme time constraints, we are happy to make the proposed interfaces undocumented and Project Private. I'll work with the LU team to draw up a contract. In the future, but not part of this case, after discussing it with AT&T, my intent will be to provide Committed interfaces for handling sparse files along the lines of Joerg's proposal. Cindy From cynthia.eastham@sun.com Tue Nov 25 15:40:41 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAPNeefK013464 for ; Tue, 25 Nov 2008 15:40:41 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAPNeXZc010335; Tue, 25 Nov 2008 23:40:38 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAW0000HXRP9300@nwk-avmta-1.sfbay.Sun.COM>; Tue, 25 Nov 2008 15:40:37 -0800 (PST) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.59]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAW002Q8XROYUB0@nwk-avmta-1.sfbay.Sun.COM>; Tue, 25 Nov 2008 15:40:36 -0800 (PST) Received: from [10.7.250.74] (punchin-client-10-7-250-74.SFBay.Sun.COM [10.7.250.74]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id mAPNeZ2X153221; Tue, 25 Nov 2008 15:40:35 -0800 (PST) Date: Tue, 25 Nov 2008 15:40:35 -0800 From: Cynthia Eastham Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <492C8AF1.7090208@sun.com> To: "Garrett D'Amore" Cc: Joerg Schilling , James.D.Carlson@sun.com, don.cragun@sun.com, PSARC-ext@sun.com Message-id: <492C8CF3.6030603@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200811242237.mAOMbnLM012156@spartan.eng.sun.com> <492bd24d.1NC3ClFDsNAYP83R%Joerg.Schilling@fokus.fraunhofer.de> <492c4386.yse3oSB3TJtox+HJ%Joerg.Schilling@fokus.fraunhofer.de> <492C47D7.6080507@sun.com> <492C8AF1.7090208@sun.com> User-Agent: Thunderbird 2.0.0.6 (X11/20071023) Status: RO Content-Length: 645 Cynthia Eastham wrote: > > I've talked with Don Cragun, who is off-line today for medical reasons, > about Garrett's proposal as well as Joerg's proposal. Due to extreme > time constraints, we are happy to make the proposed interfaces > undocumented and Project Private. I'll work with the LU team to draw up > a contract. I should clarify that we'll make the interfaces originally proposed in this case undocumented and Project Private. > > In the future, but not part of this case, after discussing it with AT&T, > my intent will be to provide Committed interfaces for handling sparse > files along the lines of Joerg's proposal. From gdamore@Sun.COM Tue Nov 25 16:23:34 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAQ0NXS1012298 for ; Tue, 25 Nov 2008 16:23:34 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id mAQ0NTBC010471; Wed, 26 Nov 2008 08:23:31 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAW00H0HZR5VG00@nwk-avmta-2.sfbay.sun.com>; Tue, 25 Nov 2008 16:23:29 -0800 (PST) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAW00C1NZR02F70@nwk-avmta-2.sfbay.sun.com>; Tue, 25 Nov 2008 16:23:29 -0800 (PST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAQ0NOIE012467; Tue, 25 Nov 2008 16:23:24 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAW00D01ZNTIK00@fe-sfbay-09.sun.com> (original mail from gdamore@sun.com) ; Tue, 25 Nov 2008 16:23:24 -0800 (PST) Received: from [10.7.251.172] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAW00J4SZQXGND0@fe-sfbay-09.sun.com>; Tue, 25 Nov 2008 16:23:22 -0800 (PST) Date: Tue, 25 Nov 2008 16:17:04 -0800 From: "Garrett D'Amore" Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <492C8AF1.7090208@sun.com> Sender: Garrett.Damore@Sun.COM To: Cynthia Eastham Cc: Joerg Schilling , James.D.Carlson@Sun.COM, don.cragun@Sun.COM, PSARC-ext@Sun.COM Message-id: <492C9580.402@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 References: <200811242237.mAOMbnLM012156@spartan.eng.sun.com> <492bd24d.1NC3ClFDsNAYP83R%Joerg.Schilling@fokus.fraunhofer.de> <492c4386.yse3oSB3TJtox+HJ%Joerg.Schilling@fokus.fraunhofer.de> <492C47D7.6080507@sun.com> <492C8AF1.7090208@sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 4683 Cynthia Eastham wrote: > Garrett D'Amore wrote: >> I have since learned, from the project team, that this problem is >> primarily centered around solving issues with Live Upgrade. They are >> in a bind on timing, because they are coming up on a key deliverable >> date. Redesign of the file format, and resetting the testing that >> has already been done, would be detrimental to some key deliverables >> around Live Upgrade and Solaris 10. >> >> Therefore, I recommend an alternate proposal, which I think might >> satisfy most of the participants here. >> >> 1) Relegate the ascii_sparse and odc_sparse arguments, and the >> associated file formats to "undocumented" status. That is, they >> won't be documented in the man pages. This way nobody should have to >> worry too much about dealing with these file formats portably. >> (Neither AT&T, nor GNU, nor star, ever needs deal with them.) These >> arguments and file formats will have "Project Private" binding. >> >> 2) File a contract for their use by the LU (or any other projects) >> that need them. (At the same time, we should be giving advice to >> these projects that they should try to convert to use pax if possible.) >> >> 3) Anyone else looking for sparse file support should be strongly >> urged to use the existing pax (or star, if you prefer. I don't want >> to get into a debate about Sun pax vs. star vs GNU tar. Its not this >> case.) >> >> This would allow the project to go forward with their existing code, >> with minimal disruption to their plans, and minimal disruption to >> "documented" formats that external parties (star, GNU tar, AT&T) have >> to be prepared to accept. >> >> -- Garrett >> >> Joerg Schilling wrote: >>> Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) wrote: >>> >>> >>>>>> If there are, then should we be deliberately incompatible? >>>>>> >>>>> The star and the recent AT&T pax archivers encode sparse files using >>>>> ustar/pax format archives. The project team has not seen any other >>>>> attempt to encode sparse files using cpio format. So, there is no >>>>> other known cpio format that handles this case. We are not being >>>>> deliberately incompatible; nothing else handles this case. >>>>> >>>> If you mark the cpio archives using the proposal from above and if you >>>> use comma separated data_offset/data_size pairs to encode the hole >>>> list, >>>> I would be willing to implement this in star too. >>>> >>> >>> Let me make a completely new proposal that does not touch any POSIX >>> defined field and that allows to add further extensions in the >>> future. It is based on >>> extension ideas from David Korn and Glenn Fowler. >>> >>> If the archive type name is either "ascii_sparse" or "odc_sparse", >>> the string >>> >>> "VSUN\0\0" is appended to all file names in the cpio header and the >>> field c_namesize is filled with a number that is 6 bigger than >>> expected for a standard >>> cpio archive. >>> >>> If a sparse file is encountered, the following string is used instead: >>> >>> "VSUN\0S%jx\0\0", (uintmax_t)statbuf.st_size >>> >>> c_namesize is filled with a number that is of the apropriate size >>> bigger than >>> in a standard cpio archive. >>> >>> In case of a sparse file, the following hole list is used inside the >>> file data cunk: >>> >>> "%jx\n%s", (uintmax_t)number_of_holes_in_list >>> >>> The "%s" string is replaced by "number_of_holes_in_list" entris of >>> the following form: >>> >>> "%jx %jx\n", (uintmax_t)data_offset, (uintmax_t)data_size >>> >>> Sparse files that end in a hole are treated the same was as in star. >>> >>> >>> Using hex numbers instead of decimal numbers is aligned with the >>> AT&T format. >>> Using hex numbers instead of decimal numbers reduces the size of the >>> hole list by 20%. Using "data_size" instead of "hole_offset" reduces >>> the hole list by another 20%. >>> >>> The field c_filesize contains a size that is equal to the compressed >>> file size + the size of the hole list. >>> >>> Please comment! >>> >>> Jörg >>> >>> >> > > I've talked with Don Cragun, who is off-line today for medical > reasons, about Garrett's proposal as well as Joerg's proposal. Due to > extreme time constraints, we are happy to make the proposed interfaces > undocumented and Project Private. I'll work with the LU team to draw > up a contract. > > In the future, but not part of this case, after discussing it with > AT&T, my intent will be to provide Committed interfaces for handling > sparse files along the lines of Joerg's proposal. This sounds like an excellent way forward. Thanks Cynthia. -- Garrett > > Cindy From Joerg.Schilling@fokus.fraunhofer.de Wed Nov 26 02:35:16 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAQAZFga026541 for ; Wed, 26 Nov 2008 02:35:15 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAQAZD1M009395; Wed, 26 Nov 2008 02:35:14 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAX00D0VS2Q6E00@nwk-avmta-2.sfbay.sun.com>; Wed, 26 Nov 2008 02:35:14 -0800 (PST) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAX0076RS2PJ180@nwk-avmta-2.sfbay.sun.com>; Wed, 26 Nov 2008 02:35:13 -0800 (PST) Received: from relay18i.sun.com (ip128.net129179-4.block1.us.syntegra.com [129.179.4.128]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mAQAO4QN003501; Wed, 26 Nov 2008 10:35:13 +0000 (GMT) Received: from mmp11es.mmp.us.syntegra.com ([160.41.208.11] [160.41.208.11]) by relay18i.sun.com with ESMTP id BT-MMP-1030395; Wed, 26 Nov 2008 10:35:12 +0000 (Z) Received: from relay13i.sun.com (relay13i.sun.com [129.179.4.123]) by mmp11es.mmp.us.syntegra.com with ESMTP id BT-MMP-5241631; Wed, 26 Nov 2008 10:35:12 +0000 (Z) Received: from iron01.fraunhofer.de ([153.96.1.54] [153.96.1.54]) by relay1i.sun.com with ESMTP id BT-MMP-9994741; Wed, 26 Nov 2008 10:35:12 +0000 (Z) Received: from pluto.fokus.fraunhofer.de ([195.37.77.164]) by iron01.fraunhofer.de with ESMTP/TLS/DHE-RSA-AES256-SHA; Wed, 26 Nov 2008 11:35:00 +0100 Received: from EXCHSRV.fokus.fraunhofer.de (bohr [10.147.9.231]) by pluto.fokus.fraunhofer.de (8.13.7/8.13.7) with SMTP id mAQAYwvR022246; Wed, 26 Nov 2008 11:34:58 +0100 (MET) Received: from rigel ([10.147.65.195]) by EXCHSRV.fokus.fraunhofer.de with Microsoft SMTPSVC(6.0.3790.3959); Wed, 26 Nov 2008 11:34:58 +0100 Date: Wed, 26 Nov 2008 11:34:58 +0100 From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <492C8AF1.7090208@sun.com> To: gdamore@sun.com, cynthia.eastham@sun.com Cc: PSARC-ext@sun.com, James.D.Carlson@sun.com, don.cragun@sun.com Message-id: <492d2652.n7nCGlch9hlPYTrB%Joerg.Schilling@fokus.fraunhofer.de> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Antispam: No, score=0.0/5.0, scanned in 0.276sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200811242237.mAOMbnLM012156@spartan.eng.sun.com> <492bd24d.1NC3ClFDsNAYP83R%Joerg.Schilling@fokus.fraunhofer.de> <492c4386.yse3oSB3TJtox+HJ%Joerg.Schilling@fokus.fraunhofer.de> <492C47D7.6080507@sun.com> <492C8AF1.7090208@sun.com> User-Agent: nail 11.22 3/20/05 X-OriginalArrivalTime: 26 Nov 2008 10:34:58.0229 (UTC) FILETIME=[A118A650:01C94FB2] Status: RO Content-Length: 879 Cynthia Eastham wrote: > I've talked with Don Cragun, who is off-line today for medical reasons, > about Garrett's proposal as well as Joerg's proposal. Due to extreme > time constraints, we are happy to make the proposed interfaces > undocumented and Project Private. I'll work with the LU team to draw up > a contract. > > In the future, but not part of this case, after discussing it with AT&T, > my intent will be to provide Committed interfaces for handling sparse > files along the lines of Joerg's proposal. This looks like a good solution. Jörg -- EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin js@cs.tu-berlin.de (uni) schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily From dwc@spartan.eng.sun.com Wed Nov 26 08:10:37 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAQGAa8k007224 for ; Wed, 26 Nov 2008 08:10:37 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAQGAN0W019764; Wed, 26 Nov 2008 16:10:33 GMT Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAY00B057LK5200@nwk-avmta-2.sfbay.sun.com>; Wed, 26 Nov 2008 08:10:32 -0800 (PST) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAY008LJ7LJL030@nwk-avmta-2.sfbay.sun.com>; Wed, 26 Nov 2008 08:10:31 -0800 (PST) Received: from spartan.eng.sun.com (spartan.SFBay.Sun.COM [129.146.226.64]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAQGATor000140; Wed, 26 Nov 2008 08:10:29 -0800 (PST) Received: from spartan.eng.sun.com (localhost [127.0.0.1]) by spartan.eng.sun.com (8.13.7+Sun/8.13.7) with ESMTP id mAQG30gL019599; Wed, 26 Nov 2008 08:03:00 -0800 (PST) Received: (from dwc@localhost) by spartan.eng.sun.com (8.13.7+Sun/8.13.7/Submit) id mAQG30dw019598; Wed, 26 Nov 2008 08:03:00 -0800 (PST) Date: Wed, 26 Nov 2008 08:03:00 -0800 (PST) From: Don Cragun Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] To: Darren.Moffat@sun.com, James.D.Carlson@sun.com, gdamore@sun.com Cc: PSARC-ext@sun.com, cynthia.eastham@sun.com Message-id: <200811261603.mAQG30dw019598@spartan.eng.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 1260 >Date: Tue, 25 Nov 2008 15:32:01 -0800 >From: Cynthia Eastham > ... ... ... > >I've talked with Don Cragun, who is off-line today for medical reasons, >about Garrett's proposal as well as Joerg's proposal. Due to extreme >time constraints, we are happy to make the proposed interfaces >undocumented and Project Private. I'll work with the LU team to draw up >a contract. > >In the future, but not part of this case, after discussing it with AT&T, >my intent will be to provide Committed interfaces for handling sparse >files along the lines of Joerg's proposal. > >Cindy Garrett, James, & Darren, As PSARC members who have commented on this case, do any of you have any objection to changing the status of this case back to closed approved automatic with the following changes to the original case proposal: 1. -H option option-arguments ascii_sparse and odc_sparse will be changed from committed to contracted project private interfaces 2. These option-arguments will not be documented on any man page 3. A contract will be created between this project team and the live upgrade project team to use these option-arguments. There is time pressure to implement this in the next Solaris 10 update. Thanks, Don From gdamore@sun.com Wed Nov 26 08:17:00 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAQGGxUp006884 for ; Wed, 26 Nov 2008 08:16:59 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAQGGoJp024566; Wed, 26 Nov 2008 16:16:57 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAY00M057W7RG00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 26 Nov 2008 08:16:55 -0800 (PST) Received: from sca-es-mail-1.sun.com ([192.18.43.132]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAY00G7P7W6N140@nwk-avmta-1.sfbay.Sun.COM>; Wed, 26 Nov 2008 08:16:54 -0800 (PST) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAQGGsLh026082; Wed, 26 Nov 2008 08:16:54 -0800 (PST) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAY00I017OQYR00@fe-sfbay-10.sun.com> (original mail from gdamore@sun.com) ; Wed, 26 Nov 2008 08:16:54 -0800 (PST) Received: from [10.7.251.172] by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAY00AU07W0T0A0@fe-sfbay-10.sun.com>; Wed, 26 Nov 2008 08:16:48 -0800 (PST) Date: Wed, 26 Nov 2008 08:10:27 -0800 From: "Garrett D'Amore" Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <200811261603.mAQG30dw019598@spartan.eng.sun.com> Sender: Garrett.Damore@sun.com To: Don Cragun Cc: Darren.Moffat@sun.com, James.D.Carlson@sun.com, PSARC-ext@sun.com, Cynthia.Eastham@sun.com Message-id: <492D74F3.5060508@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200811261603.mAQG30dw019598@spartan.eng.sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080616) Status: RO Content-Length: 1514 Don Cragun wrote: >> Date: Tue, 25 Nov 2008 15:32:01 -0800 >> From: Cynthia Eastham >> >> > ... ... ... > >> I've talked with Don Cragun, who is off-line today for medical reasons, >> about Garrett's proposal as well as Joerg's proposal. Due to extreme >> time constraints, we are happy to make the proposed interfaces >> undocumented and Project Private. I'll work with the LU team to draw up >> a contract. >> >> In the future, but not part of this case, after discussing it with AT&T, >> my intent will be to provide Committed interfaces for handling sparse >> files along the lines of Joerg's proposal. >> >> Cindy >> > > Garrett, James, & Darren, > As PSARC members who have commented on this case, do any of you > have any objection to changing the status of this case back to closed > approved automatic with the following changes to the original case > proposal: > 1. -H option option-arguments ascii_sparse and odc_sparse will be > changed from committed to contracted project private interfaces > 2. These option-arguments will not be documented on any man page > 3. A contract will be created between this project team and the live > upgrade project team to use these option-arguments. > There is time pressure to implement this in the next Solaris 10 > update. > Given the reduced scope exposure (project private), resetting back to automatic approval seems quite reasonable. So I'm OK with it. -- Garrett > Thanks, > Don > From Darren.Moffat@sun.com Wed Nov 26 08:34:54 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAQGYspc007467 for ; Wed, 26 Nov 2008 08:34:54 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAQGYmhu001911; Wed, 26 Nov 2008 09:34:53 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAY0012J8Q4SX00@nwk-avmta-1.sfbay.Sun.COM>; Wed, 26 Nov 2008 08:34:52 -0800 (PST) Received: from gmp-eb-inf-2.sun.com ([192.18.6.24]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAY00GR68Q2N650@nwk-avmta-1.sfbay.Sun.COM>; Wed, 26 Nov 2008 08:34:51 -0800 (PST) Received: from fe-emea-09.sun.com (gmp-eb-lb-1-fe3.eu.sun.com [192.18.6.10]) by gmp-eb-inf-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAQGYoiu026662; Wed, 26 Nov 2008 16:34:50 +0000 (GMT) Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAY005016HNVC00@fe-emea-09.sun.com> (original mail from Darren.Moffat@Sun.COM); Wed, 26 Nov 2008 16:34:50 +0000 (GMT) Received: from [129.156.173.21] by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAY00IMP8P5CT70@fe-emea-09.sun.com>; Wed, 26 Nov 2008 16:34:18 +0000 (GMT) Date: Wed, 26 Nov 2008 16:34:17 +0000 From: Darren J Moffat Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <200811261603.mAQG30dw019598@spartan.eng.sun.com> Sender: Darren.Moffat@sun.com To: Don Cragun Cc: James.D.Carlson@sun.com, gdamore@sun.com, PSARC-ext@sun.com, Cynthia.Eastham@sun.com Message-id: <492D7A89.7010608@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200811261603.mAQG30dw019598@spartan.eng.sun.com> User-Agent: Thunderbird 2.0.0.17 (X11/20081104) Status: RO Content-Length: 733 Don Cragun wrote: > Garrett, James, & Darren, > As PSARC members who have commented on this case, do any of you > have any objection to changing the status of this case back to closed > approved automatic with the following changes to the original case > proposal: > 1. -H option option-arguments ascii_sparse and odc_sparse will be > changed from committed to contracted project private interfaces > 2. These option-arguments will not be documented on any man page > 3. A contract will be created between this project team and the live > upgrade project team to use these option-arguments. > There is time pressure to implement this in the next Solaris 10 > update. I'm happy with that proposal. -- Darren J Moffat From carlsonj@phorcys.east.sun.com Wed Nov 26 09:13:54 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAQHDs2Y008049 for ; Wed, 26 Nov 2008 09:13:54 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAQHDlEj034659; Wed, 26 Nov 2008 10:13:52 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAY00E2PAJ3SX00@nwk-avmta-2.sfbay.sun.com>; Wed, 26 Nov 2008 09:13:51 -0800 (PST) Received: from phorcys.east.sun.com ([129.148.174.143]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAY008HYAJ2KX90@nwk-avmta-2.sfbay.sun.com>; Wed, 26 Nov 2008 09:13:50 -0800 (PST) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id mAQHDoAX025433; Wed, 26 Nov 2008 12:13:50 -0500 (EST) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id mAQHDorF025430; Wed, 26 Nov 2008 12:13:50 -0500 (EST) Date: Wed, 26 Nov 2008 12:13:50 -0500 From: James Carlson Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <492D7A89.7010608@Sun.COM> To: Darren J Moffat Cc: Don Cragun , PSARC-ext@sun.com, gdamore@sun.com, Cynthia.Eastham@sun.com Message-id: <18733.33742.20260.712005@gargle.gargle.HOWL> MIME-version: 1.0 X-Mailer: VM 7.01 under Emacs 21.3.1 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200811261603.mAQG30dw019598@spartan.eng.sun.com> <492D7A89.7010608@Sun.COM> Status: RO Content-Length: 1017 Darren J Moffat writes: > Don Cragun wrote: > > Garrett, James, & Darren, > > As PSARC members who have commented on this case, do any of you > > have any objection to changing the status of this case back to closed > > approved automatic with the following changes to the original case > > proposal: > > 1. -H option option-arguments ascii_sparse and odc_sparse will be > > changed from committed to contracted project private interfaces > > 2. These option-arguments will not be documented on any man page > > 3. A contract will be created between this project team and the live > > upgrade project team to use these option-arguments. > > There is time pressure to implement this in the next Solaris 10 > > update. > > I'm happy with that proposal. No problem for me, either. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From dwc@spartan.eng.sun.com Wed Nov 26 12:23:25 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAQKNPQN018147 for ; Wed, 26 Nov 2008 12:23:25 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mAQKNOmM035919 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 26 Nov 2008 13:23:25 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KAY0030DJAZRY00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 26 Nov 2008 12:23:23 -0800 (PST) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KAY00LQMJAYX270@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 26 Nov 2008 12:23:22 -0800 (PST) Received: from spartan.eng.sun.com (spartan.SFBay.Sun.COM [129.146.226.64]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAQKNGPd057310; Wed, 26 Nov 2008 12:23:16 -0800 (PST) Received: from spartan.eng.sun.com (localhost [127.0.0.1]) by spartan.eng.sun.com (8.13.7+Sun/8.13.7) with ESMTP id mAQK8Ifl020396; Wed, 26 Nov 2008 12:08:18 -0800 (PST) Received: (from dwc@localhost) by spartan.eng.sun.com (8.13.7+Sun/8.13.7/Submit) id mAQK8HB4020395; Wed, 26 Nov 2008 12:08:18 -0800 (PST) Date: Wed, 26 Nov 2008 12:08:18 -0800 (PST) From: Don Cragun Subject: Re: Add sparse file support to cpio [PSARC/2008/727 Self Review] To: PSARC-ext@sun.com Cc: Cynthia.Eastham@sun.com Message-id: <200811262008.mAQK8HB4020395@spartan.eng.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 1262 >Date: Wed, 26 Nov 2008 12:13:50 -0500 >From: James Carlson > >Darren J Moffat writes: >> Don Cragun wrote: >> > Garrett, James, & Darren, >> > As PSARC members who have commented on this case, do any of you >> > have any objection to changing the status of this case back to closed >> > approved automatic with the following changes to the original case >> > proposal: >> > 1. -H option option-arguments ascii_sparse and odc_sparse will be >> > changed from committed to contracted project private interfaces >> > 2. These option-arguments will not be documented on any man page >> > 3. A contract will be created between this project team and the live >> > upgrade project team to use these option-arguments. >> > There is time pressure to implement this in the next Solaris 10 >> > update. >> >> I'm happy with that proposal. > >No problem for me, either. > OK. This case has been moved back to closed approved automatic status. A contract will be put in place next week. Thanks, Don >-- >James Carlson, Solaris Networking >Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 >MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From dwc@spartan.eng.sun.com Wed Dec 3 09:25:14 2008 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mB3HPEdm004265 for ; Wed, 3 Dec 2008 09:25:14 -0800 (PST) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mB3HPCTM037207 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 3 Dec 2008 10:25:14 -0700 (MST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KBB00H399Q1YJ00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 03 Dec 2008 10:25:13 -0700 (MST) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KBB006J19PZ1890@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 03 Dec 2008 10:25:11 -0700 (MST) Received: from spartan.eng.sun.com (spartan.SFBay.Sun.COM [129.146.226.64]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mB3HPAcH038720; Wed, 03 Dec 2008 09:25:10 -0800 (PST) Received: from spartan.eng.sun.com (localhost [127.0.0.1]) by spartan.eng.sun.com (8.13.7+Sun/8.13.7) with ESMTP id mB3HHfU3018329; Wed, 03 Dec 2008 09:17:41 -0800 (PST) Received: (from dwc@localhost) by spartan.eng.sun.com (8.13.7+Sun/8.13.7/Submit) id mB3HHfCF018328; Wed, 03 Dec 2008 09:17:41 -0800 (PST) Date: Wed, 03 Dec 2008 09:17:41 -0800 (PST) From: Don Cragun Subject: Signatures needed for contract: Add sparse file support to cpio [PSARC/2008/727 Self Review] To: Jon.Bowman@sun.com, Tim.Sparlin@sun.com Cc: Cynthia.Eastham@sun.com, PSARC-ext@sun.com Message-id: <200812031717.mB3HHfCF018328@spartan.eng.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 4877 Tim & Jon, I need signatures from both of you as project managers to approve the contract below allowing Live Upgrade to use the project private sparse file support that is being added to cpio. Please respond to this message saying that you approve the contract, or let me know what changes need to be made before you can approve the contract. Cheers, Don @(#)contract 1.8 @(#) /shared/sac/arc/ARC-Templates/contract [1.8 06/12/06] CONTRACT ALLOWING/REQUIRING SPECIAL ARRANGEMENTS FOR INTERFACES 0. Number: PSARC/2008/727-01 1. This contract is between a SUPPLIER of INTERFACES and a CONSUMER of those INTERFACES, both of whom are entities within Sun Microsystems, Incorporated. 2. The SUPPLIER (definer and/or implementor) is identified by the following: Product or Bundle: Solaris Consolidation: ON Department or Group: Solaris Interfaces Bugster Product/Category/SubCategory: solaris/utility/archiver Responsible Manager: Tim Sparlin 3. The CONSUMER is identified by the following: Product or Bundle: Live Upgrade Consolidation: Install Department or Group: Live Upgrade Bugster Product/Category/SubCategory: solaris/solaris_install/live_upgrade Responsible Manager: Jon Bowman 4. The INTERFACES are: cpio -H odc_sparse 5. The ARC controlling these INTERFACES is: PSARC 6. The CASE describing (Exporting) these INTERFACES is: PSARC/2008/727 7. The following SPECIAL ARRANGEMENTS are made which modify the rules imposed by the stability levels listed in section 4 above: _N_ 7a. Although the stability level doesn't normally restrict it, SUPPLIER promises to only modify INTERFACES in an incompatible way as follows: _N_ 7b. Although the stability level doesn't normally allow it, CONSUMER will expose INTERFACES to a PARTNER, which is external to Sun, namely: Name of Company: Name of Department or Group within Company: Responsible Manager: _N_ 7c. Although the stability level doesn't normally allow it, CONSUMER will import INTERFACES from a separate consolidation. _Y_ 7d. If SUPPLIER decides to change (including replace or remove) any portion of the INTERFACES, SUPPLIER will notify CONSUMER of the proposed new version, no later than the application for ARC approval of the new version. If SUPPLIER and CONSUMER are contained in the same consolidation, they have the option of arranging for simultaneous conversion to the new interfaces. If this is not possible, or if they are not in the same consolidation, then SUPPLIER will either make best effort to work with CONSUMER so that CONSUMER can detect which version of INTERFACES is being supplied, or else SUPPLIER will make best effort to supply both old and new versions of INTERFACES. If SUPPLIER cannot make both versions of INTERFACES available, and SUPPLIER and CONSUMER cannot devise a method whereby CONSUMER can detect which version of INTERFACES is being supplied, and the old version of CONSUMER will not run with the new version of SUPPLIER, then either the EOL process must be followed by SUPPLIER, or else a major release of SUPPLIER will be required, or the change will not be allowed. 8. If CONSUMER requires changes in INTERFACES, SUPPLIER will make best effort to accommodate such changes, which shall then be treated in accordance with paragraph 7 above. 9. Notwithstanding paragraphs 7 and 8, a change to any portion of the INTERFACES shall be regarded as a completely new set of INTERFACES which require both ARC approval and execution of a new contract. 10. SUPPLIER and CONSUMER agree that evolution of INTERFACES shall be handled as follows: [In particular, include whether the SUPPLIER will inform the CONSUMER or obtain approval of the change from the CONSUMER.] 11. SUPPLIER and CONSUMER agree that INTERFACES will be supported as follows: 12. SUPPLIER and CONSUMER agree that INTERFACES will be documented as follows: 13. SUPPLIER and CONSUMER agree that changes to the INTERFACES will be tested as follows: 14. SUPPLIER and CONSUMER agree that this contract can be terminated as follows: 15. This contract is not valid until "signed" via agreement from the SUPPLIER and CONSUMER, and approved by the ARC CASE referenced by this contract. E-mail agreement to the contract should be archived in the mail archive of CASE; verbal agreement to the contract should be noted in the meeting minutes. This contract remains valid until superseded or invalidated. For SUPPLIER: Date: For CONSUMER: Date: For ARC: Date: A copy of this contract shall be deposited in the CASE directory as "contract-" or in a "contracts" subdirectory. 16. (Not to be filled in until superseded or invalidated.) This contract was superseded or invalidated by CASE: For ARC: Date: From tim.sparlin@sun.com Wed Dec 3 09:39:50 2008 Return-Path: Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by spartan.eng.sun.com (8.13.7+Sun/8.13.7) with ESMTP id mB3Hdng1018532 for ; Wed, 3 Dec 2008 09:39:49 -0800 (PST) Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mB3HdnBP013979 for ; Wed, 3 Dec 2008 09:39:49 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mB3HdmSL015194; Wed, 3 Dec 2008 09:39:48 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KBB00M0DAECCI00@nwk-avmta-2.sfbay.sun.com>; Wed, 03 Dec 2008 09:39:48 -0800 (PST) Received: from jurassic-x4600.sfbay.sun.com ([129.146.17.63]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KBB00EDDAEB6LA0@nwk-avmta-2.sfbay.sun.com>; Wed, 03 Dec 2008 09:39:47 -0800 (PST) Received: from [129.146.228.175] (cmousse-pc.SFBay.Sun.COM [129.146.228.175]) by jurassic-x4600.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id mB3HdlJE422106; Wed, 03 Dec 2008 09:39:47 -0800 (PST) Date: Wed, 03 Dec 2008 09:40:36 -0800 From: Tim Sparlin Subject: Re: Signatures needed for contract: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <200812031717.mB3HHfCF018328@spartan.eng.sun.com> To: Don Cragun Cc: Jon.Bowman@sun.com, Cynthia.Eastham@sun.com, Tim Sparlin Message-id: <4936C494.6000304@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200812031717.mB3HHfCF018328@spartan.eng.sun.com> User-Agent: Thunderbird 2.0.0.19pre (X11/20081201) Content-Length: 5159 Status: RO I approve this contract - tim sparlin Don Cragun wrote: > Tim & Jon, > I need signatures from both of you as project managers to > approve the contract below allowing Live Upgrade to use the project > private sparse file support that is being added to cpio. Please > respond to this message saying that you approve the contract, or let me > know what changes need to be made before you can approve the contract. > > Cheers, > Don > > @(#)contract 1.8 @(#) /shared/sac/arc/ARC-Templates/contract [1.8 06/12/06] > > CONTRACT ALLOWING/REQUIRING SPECIAL ARRANGEMENTS FOR INTERFACES > > 0. Number: PSARC/2008/727-01 > > 1. This contract is between > a SUPPLIER of INTERFACES and > a CONSUMER of those INTERFACES, > both of whom are entities within Sun Microsystems, Incorporated. > > 2. The SUPPLIER (definer and/or implementor) is identified by the following: > Product or Bundle: Solaris > Consolidation: ON > Department or Group: Solaris Interfaces > Bugster Product/Category/SubCategory: solaris/utility/archiver > Responsible Manager: Tim Sparlin > > 3. The CONSUMER is identified by the following: > Product or Bundle: Live Upgrade > Consolidation: Install > Department or Group: Live Upgrade > Bugster Product/Category/SubCategory: solaris/solaris_install/live_upgrade > Responsible Manager: Jon Bowman > > 4. The INTERFACES are: > cpio -H odc_sparse > > 5. The ARC controlling these INTERFACES is: PSARC > > 6. The CASE describing (Exporting) these INTERFACES is: PSARC/2008/727 > > 7. The following SPECIAL ARRANGEMENTS are made which modify the rules > imposed by the stability levels listed in section 4 above: > > _N_ 7a. Although the stability level doesn't normally restrict it, > SUPPLIER promises to only modify INTERFACES in an incompatible > way as follows: > > _N_ 7b. Although the stability level doesn't normally allow it, CONSUMER will > expose INTERFACES to a PARTNER, which is external to Sun, namely: > Name of Company: > Name of Department or Group within Company: > Responsible Manager: > > _N_ 7c. Although the stability level doesn't normally allow it, CONSUMER will > import INTERFACES from a separate consolidation. > > _Y_ 7d. If SUPPLIER decides to change (including replace or remove) any > portion of the INTERFACES, SUPPLIER will notify CONSUMER of the > proposed new version, no later than the application for ARC > approval of the new version. > If SUPPLIER and CONSUMER are contained in the same consolidation, > they have the option of arranging for simultaneous conversion > to the new interfaces. If this is not possible, or if they are > not in the same consolidation, then SUPPLIER will either make best > effort to work with CONSUMER so that CONSUMER can detect which > version of INTERFACES is being supplied, or else SUPPLIER will > make best effort to supply both old and new versions of > INTERFACES. > If SUPPLIER cannot make both versions of INTERFACES available, > and SUPPLIER and CONSUMER cannot devise a method whereby > CONSUMER can detect which version of INTERFACES is being > supplied, and the old version of CONSUMER will not run with the > new version of SUPPLIER, then either the EOL process must be > followed by SUPPLIER, or else a major release of SUPPLIER will > be required, or the change will not be allowed. > > 8. If CONSUMER requires changes in INTERFACES, SUPPLIER will make > best effort to accommodate such changes, which shall then be > treated in accordance with paragraph 7 above. > > 9. Notwithstanding paragraphs 7 and 8, a change to any portion > of the INTERFACES shall be regarded as a completely new set of > INTERFACES which require both ARC approval and execution of > a new contract. > > 10. SUPPLIER and CONSUMER agree that evolution of INTERFACES shall be > handled as follows: > [In particular, include whether the SUPPLIER will inform the > CONSUMER or obtain approval of the change from the CONSUMER.] > > 11. SUPPLIER and CONSUMER agree that INTERFACES will be supported as > follows: > > 12. SUPPLIER and CONSUMER agree that INTERFACES will be documented as > follows: > > 13. SUPPLIER and CONSUMER agree that changes to the INTERFACES will be > tested as follows: > > 14. SUPPLIER and CONSUMER agree that this contract can be terminated as > follows: > > 15. This contract is not valid until "signed" via agreement from the > SUPPLIER and CONSUMER, and approved by the ARC CASE referenced by > this contract. E-mail agreement to the contract should be archived > in the mail archive of CASE; verbal agreement to the contract > should be noted in the meeting minutes. This contract remains > valid until superseded or invalidated. > > For SUPPLIER: Date: > For CONSUMER: Date: > For ARC: Date: > > A copy of this contract shall be deposited in the CASE directory as > "contract-" or in a "contracts" subdirectory. > > 16. (Not to be filled in until superseded or invalidated.) > This contract was superseded or invalidated by CASE: > For ARC: Date: > From Jon.Bowman@sun.com Wed Dec 3 10:03:34 2008 Return-Path: Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by spartan.eng.sun.com (8.13.7+Sun/8.13.7) with ESMTP id mB3I3XAK018654 for ; Wed, 3 Dec 2008 10:03:33 -0800 (PST) Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mB3I3X2g008540 for ; Wed, 3 Dec 2008 10:03:33 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id mB3I3WrP001854 for <@sunmail2sca.sfbay.sun.com:don.cragun@sun.com>; Wed, 3 Dec 2008 11:03:33 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KBB0002VBHWNY00@nwk-avmta-2.sfbay.sun.com> for don.cragun@sun.com (ORCPT don.cragun@sun.com); Wed, 03 Dec 2008 10:03:32 -0800 (PST) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KBB00E18BHV6IC0@nwk-avmta-2.sfbay.sun.com> for don.cragun@sun.com (ORCPT don.cragun@sun.com); Wed, 03 Dec 2008 10:03:31 -0800 (PST) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mB3I3UbS011126 for ; Wed, 03 Dec 2008 18:03:31 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KBB00I01ABVG600@mail-amer.sun.com> (original mail from Jon.Bowman@Sun.COM) for don.cragun@sun.com (ORCPT don.cragun@sun.com); Wed, 03 Dec 2008 11:03:16 -0700 (MST) Received: from boulder-dhcp-227.Central.Sun.COM ([172.20.24.227]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KBB0042JBH7PN00@mail-amer.sun.com>; Wed, 03 Dec 2008 11:03:07 -0700 (MST) Date: Wed, 03 Dec 2008 11:03:07 -0700 From: Jon Bowman Subject: Re: Signatures needed for contract: Add sparse file support to cpio [PSARC/2008/727 Self Review] In-reply-to: <4936C494.6000304@sun.com> Sender: Jon.Bowman@sun.com To: Tim Sparlin Cc: Don Cragun , Cynthia.Eastham@sun.com Message-id: MIME-version: 1.0 X-Mailer: Apple Mail (2.929.2) Content-type: multipart/alternative; boundary="Boundary_(ID_5VUXhpXpQzxvjuOYUKMfng)" X-PMX-Version: 5.4.1.325704 References: <200812031717.mB3HHfCF018328@spartan.eng.sun.com> <4936C494.6000304@sun.com> Content-Length: 22133 Status: RO --Boundary_(ID_5VUXhpXpQzxvjuOYUKMfng) Content-type: text/plain; delsp=yes; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT I approve this contract - Jon Bowman On Dec 3, 2008, at 10:40 AM, Tim Sparlin wrote: > > I approve this contract - tim sparlin > > > Don Cragun wrote: >> Tim & Jon, >> I need signatures from both of you as project managers to >> approve the contract below allowing Live Upgrade to use the project >> private sparse file support that is being added to cpio. Please >> respond to this message saying that you approve the contract, or >> let me >> know what changes need to be made before you can approve the >> contract. >> >> Cheers, >> Don >> >> @(#)contract 1.8 @(#) /shared/sac/arc/ARC-Templates/contract [1.8 >> 06/12/06] >> >> CONTRACT ALLOWING/REQUIRING SPECIAL ARRANGEMENTS FOR INTERFACES >> >> 0. Number: PSARC/2008/727-01 >> >> 1. This contract is between >> a SUPPLIER of INTERFACES and >> a CONSUMER of those INTERFACES, >> both of whom are entities within Sun Microsystems, Incorporated. >> >> 2. The SUPPLIER (definer and/or implementor) is identified by the >> following: >> Product or Bundle: Solaris >> Consolidation: ON >> Department or Group: Solaris Interfaces >> Bugster Product/Category/SubCategory: solaris/utility/archiver >> Responsible Manager: Tim Sparlin >> >> 3. The CONSUMER is identified by the following: >> Product or Bundle: Live Upgrade >> Consolidation: Install >> Department or Group: Live Upgrade >> Bugster Product/Category/SubCategory: solaris/solaris_install/ >> live_upgrade >> Responsible Manager: Jon Bowman >> >> 4. The INTERFACES are: >> cpio -H odc_sparse >> >> 5. The ARC controlling these INTERFACES is: PSARC >> >> 6. The CASE describing (Exporting) these INTERFACES is: PSARC/ >> 2008/727 >> >> 7. The following SPECIAL ARRANGEMENTS are made which modify the >> rules >> imposed by the stability levels listed in section 4 above: >> _N_ 7a. Although the stability level doesn't normally restrict it, >> SUPPLIER promises to only modify INTERFACES in an incompatible >> way as follows: >> >> _N_ 7b. Although the stability level doesn't normally allow it, >> CONSUMER will >> expose INTERFACES to a PARTNER, which is external to Sun, >> namely: >> Name of Company: >> Name of Department or Group within Company: >> Responsible Manager: >> >> _N_ 7c. Although the stability level doesn't normally allow it, >> CONSUMER will >> import INTERFACES from a separate consolidation. >> >> _Y_ 7d. If SUPPLIER decides to change (including replace or remove) >> any >> portion of the INTERFACES, SUPPLIER will notify CONSUMER of the >> proposed new version, no later than the application for ARC >> approval of the new version. >> If SUPPLIER and CONSUMER are contained in the same consolidation, >> they have the option of arranging for simultaneous conversion >> to the new interfaces. If this is not possible, or if they are >> not in the same consolidation, then SUPPLIER will either make best >> effort to work with CONSUMER so that CONSUMER can detect which >> version of INTERFACES is being supplied, or else SUPPLIER will >> make best effort to supply both old and new versions of >> INTERFACES. >> If SUPPLIER cannot make both versions of INTERFACES available, >> and SUPPLIER and CONSUMER cannot devise a method whereby >> CONSUMER can detect which version of INTERFACES is being >> supplied, and the old version of CONSUMER will not run with the >> new version of SUPPLIER, then either the EOL process must be >> followed by SUPPLIER, or else a major release of SUPPLIER will >> be required, or the change will not be allowed. >> >> 8. If CONSUMER requires changes in INTERFACES, SUPPLIER will make >> best effort to accommodate such changes, which shall then be >> treated in accordance with paragraph 7 above. >> >> 9. Notwithstanding paragraphs 7 and 8, a change to any portion >> of the INTERFACES shall be regarded as a completely new set of >> INTERFACES which require both ARC approval and execution of >> a new contract. >> >> 10. SUPPLIER and CONSUMER agree that evolution of INTERFACES shall be >> handled as follows: >> [In particular, include whether the SUPPLIER will inform the >> CONSUMER or obtain approval of the change from the CONSUMER.] >> >> 11. SUPPLIER and CONSUMER agree that INTERFACES will be supported as >> follows: >> >> 12. SUPPLIER and CONSUMER agree that INTERFACES will be documented as >> follows: >> >> 13. SUPPLIER and CONSUMER agree that changes to the INTERFACES will >> be >> tested as follows: >> >> 14. SUPPLIER and CONSUMER agree that this contract can be >> terminated as >> follows: >> >> 15. This contract is not valid until "signed" via agreement from the >> SUPPLIER and CONSUMER, and approved by the ARC CASE referenced by >> this contract. E-mail agreement to the contract should be >> archived >> in the mail archive of CASE; verbal agreement to the contract >> should be noted in the meeting minutes. This contract remains >> valid until superseded or invalidated. >> >> For SUPPLIER: Date: >> For CONSUMER: Date: >> For ARC: Date: >> >> A copy of this contract shall be deposited in the CASE directory >> as >> "contract-" or in a "contracts" subdirectory. >> >> 16. (Not to be filled in until superseded or invalidated.) >> This contract was superseded or invalidated by CASE: >> For ARC: Date: >> > Jon Bowman Solaris RPE - Install Technology Manager Sun Microsystems, Inc. 500 Eldorado Blvd Broomfield, CO 80021 US Phone x60609/+1 303-223-6127 Mobile +1 303-803-0460 Fax +1 303-223-6127 Email Jon.Bowman@Sun.COM --Boundary_(ID_5VUXhpXpQzxvjuOYUKMfng) Content-type: text/html; charset=US-ASCII Content-transfer-encoding: QUOTED-PRINTABLE I approve this contract - J= on Bowman




<= div>On Dec 3, 2008, at 10:40 AM, Tim Sparlin wrote:


I a= pprove this contract - tim sparlin


Don Cragun wrote:
Tim & Jon,
= =09I need signatures from both of you as project managers to
approve the contract below a= llowing Live Upgrade to use the project
private sparse file support that is being added to cpio.=  Please
respond to th= is message saying that you approve the contract, or let me
know what changes need to be made bef= ore you can approve the contract.

=09Cheers,
=09Don

@(#)contract=091.8 @(= #) /shared/sac/arc/ARC-Templates/contract [1.8 06/12/06]

=09CONTRACT ALLOWING/REQUIRING SPECIAL ARRANGEMENTS FOR INTERFACES
=

0.  Number: PSARC/2008/727-01

1. &n= bsp;This contract is between
=09= a SUPPLIER of INTERFACES and
=09= a CONSUMER of those INTERFACES,
   both of whom are entities within Sun Microsys= tems, Incorporated.

2.  The SUPPLIER (definer an= d/or implementor) is identified by the following:
   Product or Bundle: Solaris<= br>
   Consolid= ation: ON
  &nbs= p;Department or Group: Solaris Interfaces
   Bugster Product/Category/SubCategor= y: solaris/utility/archiver
   Responsible Manager: Tim Sparlin
=

= 3.  The CONSUMER is identified by the following:
   Product or Bundle: Live= Upgrade
   = ;Consolidation: Install
&n= bsp;  Department or Group: Live Upgrade
   Bugster Product/Category/Su= bCategory: solaris/solaris_install/live_upgrade
   Responsible Manager: Jon Bowm= an

4.  The INTERFACES are:
=09cpio -H odc_sparse

5.  The AR= C controlling these INTERFACES is: PSARC

6.  The= CASE describing (Exporting) these INTERFACES is: PSARC/2008/727
<= /blockquote>

7.  The following SPECIAL ARRANGEMENTS are made whic= h modify the rules
 &= nbsp; imposed by the stability levels listed in section 4 above:=
_N_ 7a. Although the stab= ility level doesn't normally restrict it,
       SUPPLIER pr= omises to only modify INTERFACES in an incompatible
<= blockquote type=3D"cite">=09way as follows:

_N_ 7b. Althou= gh the stability level doesn't normally allow it, CONSUMER will
     &= nbsp; expose INTERFACES to a PARTNER, which is external to Sun, = namely:
=09=09Name of Company:
=09=09Name of Department or Group within Co= mpany:
=09=09Responsible Manager:
=

_N_ 7c. Although the stability level doesn't normally al= low it, CONSUMER will
&nbs= p;      import INTERFACES from a separa= te consolidation.

_Y_ 7d. If SUPPLIER decides to chan= ge (including replace or remove) any
= =09portion of the INTERFACES, SUPPLIER will notify CONSUMER of= the
=09proposed new version, no= later than the application for ARC
= =09approval of the new version.
= =09If SUPPLIER and CONSUMER are contained in the same consolid= ation,
=09they have the option o= f arranging for simultaneous conversion
=09to the new interfaces.  If this is not possible, or i= f they are
=09not in the same co= nsolidation, then SUPPLIER will either make best
=09effort to work with CONSUMER so that CONSUMER can= detect which
=09version of IN= TERFACES is being supplied, or else SUPPLIER will
=09make best effort to supply both old and new vers= ions of
=09INTERFACES.
=09If SUPPLIER cannot make both versions = of INTERFACES available,
=09and = SUPPLIER and CONSUMER cannot devise a method whereby
=
=09CONSUMER can detect which version of INTERFAC= ES is being
=09supplied, and= the old version of CONSUMER will not run with the
=09new version of SUPPLIER, then either the EOL pr= ocess must be
=09followed by S= UPPLIER, or else a major release of SUPPLIER will
=09be required, or the change will not be allowed.<= br>

8. If CONSUMER requires changes in INTERFACES, SUPPLI= ER will make
  b= est effort to accommodate such changes, which shall then be
  treated in accordance w= ith paragraph 7 above.

=
9. Notwithstanding paragraphs = 7 and 8, a change to any portion
  of the INTERFACES shall be regarded as a compl= etely new set of
 &nb= sp;INTERFACES which require both ARC approval and execution of
  a new contract.
<= /blockquote>

10. SUPPLIER and CONSUMER agree that evolution of INTERFA= CES shall be
  &= nbsp;handled as follows:
&= nbsp;  [In particular, include whether the SUPPLIER will in= form the
   = ;CONSUMER or obtain approval of the change from the CONSUMER.]

11. SUPPLIER and CONSUMER agree that INTERFACES will be sup= ported as
  &nbs= p;follows:

12. SUPPLIER and CONSUMER agree that INTER= FACES will be documented as
   follows:

13. SUPPLIER and CONSUME= R agree that changes to the INTERFACES will be
   tested as follows:

14. SUPPLIER and CONSUMER agree that this contract can be termin= ated as
   = follows:

<= blockquote type=3D"cite">15. This contract is not valid until "signed= " via agreement from the
&= nbsp;  SUPPLIER and CONSUMER, and approved by the ARC CASE = referenced by
  =  this contract.  E-mail agreement to the contract should be= archived
  &nbs= p;in the mail archive of CASE; verbal agreement to the contract
   should be not= ed in the meeting minutes.  This contract remains
   valid until superseded= or invalidated.

For SUPPLIER:=09=09=09Date:
For CONSUMER:=09=09=09Date:
For ARC:=09=09=09Date:

   A copy of this c= ontract shall be deposited in the CASE directory as
<= blockquote type=3D"cite">    "contract-<digits>" or= in a "contracts" subdirectory.

16. (Not to be filled= in until superseded or invalidated.)
   This contract was superseded or invalid= ated by CASE:
  =  For ARC:=09= =09= =09Date:
 
=


Jon Bowman 
Solaris RPE - Install Technology Manager
Sun Microsy= stems, Inc.
500 Eldorado Blvd
Broomfield, CO 80021 US
Ph= one x60609/+1 303-223-6127
Mobile +1 303-803-0460
Fax +1 303-22= 3-6127
Email Jon.Bowman@Sun.= COM
=
--Boundary_(ID_5VUXhpXpQzxvjuOYUKMfng)--