From sacadmin Thu Apr 17 11:01:57 2008 Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m3HI1vsx001615; Thu, 17 Apr 2008 11:01:57 -0700 (PDT) Received: (from dr161460@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id m3HI1vS7001611; Thu, 17 Apr 2008 11:01:57 -0700 (PDT) Date: Thu, 17 Apr 2008 11:01:57 -0700 (PDT) From: Dean Roehrich Message-Id: <200804171801.m3HI1vS7001611@sac.sfbay.sun.com> To: PSARC-record@sac.sfbay.sun.com Subject: SAM-QFS Tar Format Update [PSARC/2008/267 FastTrack timeout 04/30/2008] Status: RO Content-Length: 562 Template Version: @(#)sac_nextcase 1.66 04/17/08 SMI This information is Copyright 2008 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: SAM-QFS Tar Format Update 1.2. Name of Document Author/Supplier: Author: Eric Diven 1.3 Date of This Document: 17 April, 2008 4. Technical Description See the case directory for more detail 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: unknown 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From sacadmin Tue Apr 22 23:32:45 2008 Received: from dm-eng-02.sfbay.sun.com (dm-eng-02.SFBay.Sun.COM [129.146.11.32]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m3N6WjIo020518; Tue, 22 Apr 2008 23:32:45 -0700 (PDT) Received: from marduk.eng.sun.com (marduk.SFBay.Sun.COM [129.146.108.224]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m3N6WjbA055784; Tue, 22 Apr 2008 23:32:45 -0700 (PDT) Received: from marduk.eng.sun.com (localhost [127.0.0.1]) by marduk.eng.sun.com (8.13.6+Sun/8.12.11) with ESMTP id m3N6WVIV022372; Tue, 22 Apr 2008 23:32:31 -0700 (PDT) Received: (from gww@localhost) by marduk.eng.sun.com (8.13.6+Sun/8.12.11/Submit) id m3N6WVde022371; Tue, 22 Apr 2008 23:32:31 -0700 (PDT) Date: Tue, 22 Apr 2008 23:32:31 -0700 (PDT) From: Gary Winiger Message-Id: <200804230632.m3N6WVde022371@marduk.eng.sun.com> To: dr161460@sac.sfbay.sun.com, psarc@sac.sfbay.sun.com Subject: Re: SAM-QFS Tar Format Update [PSARC/2008/267 FastTrack timeout 04/30/2008] Status: RO Content-Length: 990 > 4. Technical Description > See the case directory for more detail Nothing was ever mailed to the ARC mailing list. It's usually good form to send mail to the ARC when starting a fast track with more than see case directory. Did sac mail swallow it? What's the release binding? What's the interface taxonomy? > 6. Resources and Schedule > 6.4. Steering Committee requested information > 6.4.1. Consolidation C-team Name: > unknown > 6.5. ARC review type: FastTrack > 6.6. ARC Exposure: open If this is an open case, why do the case materials start with: ** XXX XXXXXXXXXXXX: Internal Only ** I'm confused by the statement that the tar format is changing from the current GNU based format to posix pax. Is this project using GNU tar? Will the tar/pax man page be updated to discuss SAM-QFS? Does archives(4) need to be updated? Does SAM-QFS deliver its own tar(1)? What is missing from pax ustar / tar that is needed to support SAM-QFS? Gary.. From sacadmin Wed Apr 23 07:11:03 2008 Received: from kickball-mn.Central.Sun.COM (kickball-mn.Central.Sun.COM [10.1.170.217]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m3NEB2UC002536 for ; Wed, 23 Apr 2008 07:11:02 -0700 (PDT) Received: from kickball-mn.Central.Sun.COM (localhost [127.0.0.1]) by kickball-mn.Central.Sun.COM (8.13.6+Sun/8.13.6) with ESMTP id m3NEB1rK016644; Wed, 23 Apr 2008 09:11:01 -0500 (CDT) Received: (from roehrich@localhost) by kickball-mn.Central.Sun.COM (8.13.6+Sun/8.13.6/Submit) id m3NEB1kc016643; Wed, 23 Apr 2008 09:11:01 -0500 (CDT) Date: Wed, 23 Apr 2008 09:11:01 -0500 From: Dean Roehrich To: Gary Winiger Cc: psarc@sac.sfbay.sun.com Subject: Re: SAM-QFS Tar Format Update [PSARC/2008/267 FastTrack timeout 04/30/2008] Message-ID: <20080423141101.GA16427@kickball-mn.Central.Sun.COM> References: <200804230632.m3N6WVde022371@marduk.eng.sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200804230632.m3N6WVde022371@marduk.eng.sun.com> User-Agent: Mutt/1.5.9i Status: RO Content-Length: 2074 On Tue, Apr 22, 2008 at 11:32:31PM -0700, Gary Winiger wrote: > > 4. Technical Description > > See the case directory for more detail > > Nothing was ever mailed to the ARC mailing list. > It's usually good form to send mail to the ARC when starting > a fast track with more than see case directory. > Did sac mail swallow it? I used the defaults given by sac_nextcase, which did not include any email addresses with which I'm familiar, but it did include a batch I hadn't heard about before...I decided to see what would happen, this being my first case. Those email addrs are: PSARC-record@sac.sfbay.sun.com one-pager-list@sac.sfbay one-pager-log@sac.sfbay sac-bar@sac.sfbay I suppose I should have stripped off all the email addrs it supplied in the To and Cc lines and just put in psarc-ext@sun.com? Actually, I think maybe it would work better if I don't let sac_nextcase send the email. I should construct the email myself, and put useful info into it. > What's the release binding? What's the interface taxonomy? The release binding is minor. I will find Rick later today and get his help with the taxonomy question. > > 6. Resources and Schedule > > 6.4. Steering Committee requested information > > 6.4.1. Consolidation C-team Name: > > unknown > > 6.5. ARC review type: FastTrack > > 6.6. ARC Exposure: open > > If this is an open case, why do the case materials start with: > ** XXX XXXXXXXXXXXX: Internal Only ** I will ask the team to produce another doc without that notice. I'll remove that version of the doc. > I'm confused by the statement that the tar format is > changing from the current GNU based format to posix pax. > Is this project using GNU tar? Will the tar/pax man page > be updated to discuss SAM-QFS? Does archives(4) need to > be updated? > Does SAM-QFS deliver its own tar(1)? > What is missing from pax ustar / tar that is needed to support > SAM-QFS? The product has two tools that understand tar format: 'arcopy' and 'stager'. These read and write tar headers directly. Dean From roehrich@kickball-mn.Central.Sun.COM Wed Apr 30 12:58:22 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m3UJwMeq022818 for ; Wed, 30 Apr 2008 12:58:22 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m3UJwMgA022357 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Wed, 30 Apr 2008 12:58:22 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K0500705M59RB00@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 30 Apr 2008 12:58:21 -0700 (PDT) Received: from dm-central-02.central.sun.com ([129.147.62.5]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K0500MAZM5942E0@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 30 Apr 2008 12:58:21 -0700 (PDT) Received: from kickball-mn.Central.Sun.COM (kickball-mn.Central.Sun.COM [10.1.170.217]) by dm-central-02.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m3UJwKTq063707 for ; Wed, 30 Apr 2008 13:58:21 -0600 (MDT) Received: from kickball-mn.Central.Sun.COM (localhost [127.0.0.1]) by kickball-mn.Central.Sun.COM (8.13.6+Sun/8.13.6) with ESMTP id m3UJwKE8002919 for ; Wed, 30 Apr 2008 14:58:20 -0500 (CDT) Received: (from roehrich@localhost) by kickball-mn.Central.Sun.COM (8.13.6+Sun/8.13.6/Submit) id m3UJwKkp002918 for psarc-ext@sun.com; Wed, 30 Apr 2008 14:58:20 -0500 (CDT) Date: Wed, 30 Apr 2008 14:58:20 -0500 From: Dean Roehrich Subject: SAM-QFS Tar Format Update [PSARC/2008/267 FastTrack timeout 05/07/2008] To: psarc-ext@sun.com Message-id: <20080430195820.GA2911@kickball-mn.Central.Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 User-Agent: Mutt/1.5.9i Status: RO Content-Length: 3405 I am sponsoring this case for Eric Diven. Timeout is set for May 7, 2008. This note replaces my earlier attempt to start this case. I've updated the functional spec, and removed my earlier "one-pager" file from the materials directory. All email in the 'mail' file prior to this is superceded by this note. Release binding is Minor. Template Version: @(#)sac_nextcase 1.66 04/17/08 SMI This information is Copyright 2008 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: SAM-QFS Tar Format Update 1.2. Name of Document Author/Supplier: Author: Eric Diven 1.3 Date of This Document: 29 April, 2008 4. Technical Description Proposal: Allow SAM-QFS stager and arcopy tools to read and write a pax compliant tar format. Detail: The SAM-QFS daemons 'sam-stagerd' and 'sam-arcopy' are currently reading and writing a tape format that is a slightly modified version of a format used by a very old version of GNU tar. The older GNU tar format was modified for SAM-QFS because it didn't satify the project's needs for file size and long file names. The SAM-QFS team would like to modify these daemons to read and write a pax compliant tar format, as described in [1]. The default behavior will call for 'sam-arcopy' to continue writing the old-style tar format. The new format can be selected by setting a new 'tar_format = posix' option in the existing SAM-QFS config file, /etc/opt/SUNWsamfs/defaults.conf. The 'sam-stagerd' daemon will read the tape by validating the magic in the tar header. If the old GNU magic of "ustar[sp][sp][null]" is found then the old format is read. If the new magic of "ustar[null]00" is found then the new format will be read. If an old version of SAM-QFS attempts to read a new pax format tar header it will declare the file 'damaged', which is an existing SAM-QFS concept. No legacy patch is planned, and this incompatibility will be documented for customers. The SAM-QFS 'sls -D' command and option, a SAM-QFS-aware version of ls(1), will be modified to report the tape format used for the given file, if that file is currently on tape. It will make this decision based on an available bit in the on-disk inode, which is currently zeroed on existing SAM-QFS filesystems. A new versioned library, libpax_xhdr.so, will be added to SAM-QFS. This library is created by abstracting the tar-specific bits out of 'stager' and 'arcopy'. Ultimately, this change puts the SAM-QFS team in a position where they could retire their old version of the GNU tar command, called star. This star command is used only in disaster-recovery scenarios, and is rarely used. Deliverables: /opt/SUNWsamfs/sbin/sam-stagerd /opt/SUNWsamfs/sbin/sam-arcopy /opt/SUNWsamfs/bin/sls /opt/SUNWsamfs/lib/libpax_xhdr.so.1 /opt/SUNWsamfs/lib/libpax_xhdr.so -> /opt/SUNWsamfs/lib/libpax_xhdr.so.1 Exported Interfaces: sam-stagerd Committed Commandline syntax sam-arcopy Committed Commandline syntax sls Committed Commandline syntax libpax_xhdr Committed Imported Interfaces: libiconv References: [1] http://www.opengroup.org/onlinepubs/009695399/utilities/pax.html 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: N/A 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From don.cragun@sun.com Tue May 6 10:40:06 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m46He54B028828 for ; Tue, 6 May 2008 10:40:05 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m46He15M010353 for <@sunmail2sca.sfbay.sun.com:psarc-ext@Sun.COM>; Tue, 6 May 2008 18:40:04 +0100 (BST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K0G00F2NJQQ9300@brm-avmta-1.central.sun.com> for psarc-ext@Sun.COM (ORCPT psarc-ext@Sun.COM); Tue, 06 May 2008 11:40:02 -0600 (MDT) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K0G003GNJQMNAE0@brm-avmta-1.central.sun.com> for psarc-ext@Sun.COM (ORCPT psarc-ext@Sun.COM); Tue, 06 May 2008 11:39:58 -0600 (MDT) Received: from spartan.eng.sun.com (spartan.SFBay.Sun.COM [129.146.226.64]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m46HdwPt033432; Tue, 06 May 2008 10:39:58 -0700 (PDT) Received: from spartan (spartan [129.146.226.64]) by spartan.eng.sun.com (8.13.7+Sun/8.13.7) with SMTP id m46HWSmf002756; Tue, 06 May 2008 10:32:28 -0700 (PDT) Date: Tue, 06 May 2008 10:32:28 -0700 (PDT) From: Don Cragun Subject: Re: SAM-QFS Tar Format Update [PSARC/2008/267 FastTrack timeout 05/07/2008] To: eric.diven@sun.com Cc: psarc-ext@sun.com Reply-to: Don Cragun Message-id: <200805061732.m46HWSmf002756@spartan.eng.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.6.2 SunOS 5.10 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-MD5: pS8c/XixJx83LxUmWDBpzg== X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 3066 >Detail: > > The SAM-QFS daemons 'sam-stagerd' and 'sam-arcopy' are currently > reading and writing a tape format that is a slightly modified version > of a format used by a very old version of GNU tar. The older GNU tar > format was modified for SAM-QFS because it didn't satify the project's > needs for file size and long file names. > > The SAM-QFS team would like to modify these daemons to read and write > a pax compliant tar format, as described in [1]. > > The default behavior will call for 'sam-arcopy' to continue writing > the old-style tar format. The new format can be selected by setting a > new 'tar_format = posix' option in the existing SAM-QFS config file, > /etc/opt/SUNWsamfs/defaults.conf. Why isn't this 'tar_format = pax'? The POSIX standard specifies three archive formats: cpio, ustar, and pax. Saying 'tar_format = posix' doesn't make it clear that you mean pax format. I assume you mean pax format because the ustar format has a 12 octet field for size (giving a maxiximum file size of 999,999,999,999 bytes), and prefix and name fields of 155 and 100 octets, respectively (giving a maximum pathname length of 256 bytes; and you can get this maximum only if the final component of the pathname is exactly 100 bytes long and the pathname of all of its parent directories from the current working directory is exactly 155 bytes long). > > The 'sam-stagerd' daemon will read the tape by validating the magic in > the tar header. If the old GNU magic of "ustar[sp][sp][null]" is > found then the old format is read. If the new magic of > "ustar[null]00" is found then the new format will be read. In the standard, ustar and pax archive formats have magic as a 6 octet field. Am I correct in assuming that you mean magic will be set to "ustar" (with a trailing null byte) and version will be set to "00" (with no trailing null) as specified for the ustar and pax archive formats? > > If an old version of SAM-QFS attempts to read a new pax format tar > header it will declare the file 'damaged', which is an existing > SAM-QFS concept. No legacy patch is planned, and this incompatibility > will be documented for customers. > > The SAM-QFS 'sls -D' command and option, a SAM-QFS-aware version of > ls(1), will be modified to report the tape format used for the given > file, if that file is currently on tape. It will make this decision > based on an available bit in the on-disk inode, which is currently > zeroed on existing SAM-QFS filesystems. > > A new versioned library, libpax_xhdr.so, will be added to SAM-QFS. > This library is created by abstracting the tar-specific bits out of > 'stager' and 'arcopy'. Rather than factor tar, pax, stager, and arcopy internals into a new library, why not just invoke the pax utility to read and write pax archives? If you use pax instead of rewriting stager and arcopy, you will be able to read several forms of cpio archive formats, standard ustar format, and standard pax format when you read archives without specifying the format. The pax utility autodetects these formats. From roehrich@kickball-mn.Central.Sun.COM Tue May 6 13:39:03 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m46Kd26n005757 for ; Tue, 6 May 2008 13:39:02 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m46Kcq7Q026818 for <@sunmail2sca.sfbay.sun.com:psarc-ext@Sun.COM>; Wed, 7 May 2008 04:39:01 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K0G00F0HS0ZG100@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@Sun.COM (ORCPT psarc-ext@Sun.COM); Tue, 06 May 2008 13:38:59 -0700 (PDT) Received: from dm-central-02.central.sun.com ([129.147.62.5]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K0G00GC6S0XY2E0@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@Sun.COM (ORCPT psarc-ext@Sun.COM); Tue, 06 May 2008 13:38:57 -0700 (PDT) Received: from kickball-mn.Central.Sun.COM (kickball-mn.Central.Sun.COM [10.1.170.217]) by dm-central-02.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m46Kcui2056321; Tue, 06 May 2008 14:38:56 -0600 (MDT) Received: from kickball-mn.Central.Sun.COM (localhost [127.0.0.1]) by kickball-mn.Central.Sun.COM (8.13.6+Sun/8.13.6) with ESMTP id m46Kcut7019913; Tue, 06 May 2008 15:38:56 -0500 (CDT) Received: (from roehrich@localhost) by kickball-mn.Central.Sun.COM (8.13.6+Sun/8.13.6/Submit) id m46KcuoB019912; Tue, 06 May 2008 15:38:56 -0500 (CDT) Date: Tue, 06 May 2008 15:38:56 -0500 From: Dean Roehrich Subject: Re: SAM-QFS Tar Format Update [PSARC/2008/267 FastTrack timeout 05/07/2008] In-reply-to: <200805061732.m46HWSmf002756@spartan.eng.sun.com> To: Don Cragun Cc: Eric.Diven@sun.com, psarc-ext@sun.com Message-id: <20080506203856.GA19762@kickball-mn.Central.Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200805061732.m46HWSmf002756@spartan.eng.sun.com> User-Agent: Mutt/1.5.9i Status: RO Content-Length: 3471 On Tue, May 06, 2008 at 10:32:28AM -0700, Don Cragun wrote: > >Detail: > > > > The SAM-QFS daemons 'sam-stagerd' and 'sam-arcopy' are currently > > reading and writing a tape format that is a slightly modified version > > of a format used by a very old version of GNU tar. The older GNU tar > > format was modified for SAM-QFS because it didn't satify the project's > > needs for file size and long file names. > > > > The SAM-QFS team would like to modify these daemons to read and write > > a pax compliant tar format, as described in [1]. > > > > The default behavior will call for 'sam-arcopy' to continue writing > > the old-style tar format. The new format can be selected by setting a > > new 'tar_format = posix' option in the existing SAM-QFS config file, > > /etc/opt/SUNWsamfs/defaults.conf. > > Why isn't this 'tar_format = pax'? The POSIX standard specifies three The project team agreed to this change. > > The 'sam-stagerd' daemon will read the tape by validating the magic in > > the tar header. If the old GNU magic of "ustar[sp][sp][null]" is > > found then the old format is read. If the new magic of > > "ustar[null]00" is found then the new format will be read. > > In the standard, ustar and pax archive formats have magic as a 6 octet > field. Am I correct in assuming that you mean magic will be set to > "ustar" (with a trailing null byte) and version will be set to "00" > (with no trailing null) as specified for the ustar and pax archive > formats? The functional spec has been clarified to say: The pax format requires that the magic is set to "ustar[null]" and the version to "00" (which is not null terminated). I've updated the case with the new spec. > > If an old version of SAM-QFS attempts to read a new pax format tar > > header it will declare the file 'damaged', which is an existing > > SAM-QFS concept. No legacy patch is planned, and this incompatibility > > will be documented for customers. > > > > The SAM-QFS 'sls -D' command and option, a SAM-QFS-aware version of > > ls(1), will be modified to report the tape format used for the given > > file, if that file is currently on tape. It will make this decision > > based on an available bit in the on-disk inode, which is currently > > zeroed on existing SAM-QFS filesystems. > > > > A new versioned library, libpax_xhdr.so, will be added to SAM-QFS. > > This library is created by abstracting the tar-specific bits out of > > 'stager' and 'arcopy'. > > Rather than factor tar, pax, stager, and arcopy internals into a new > library, why not just invoke the pax utility to read and write pax > archives? The sam-arcopy and sam-stagerd daemons are able to use filesystem-specific I/O operations to read and write the files to/from the filesystem. This means files "invisibly" move between the disk and tape, which is a necessary feature of HSM products such as SAM-QFS to make the disk appear "bottomless". The sam-arcopy and sam-stagerd daemons are highly threaded to optimize their particular activity profile, and they use direct I/O. That activity profile also makes it expensive to fork+exec a tar command for every file request, because they jump around a tape rather than reading/writing it from start to finish. In summary: the current design is that these daemons have the bare-bones bits necessary to read and write the archives. The project team doesn't want to redesign these daemons, they just want to add the new header format. Dean From don.cragun@sun.com Tue May 6 14:08:17 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m46L8G8R006917 for ; Tue, 6 May 2008 14:08:16 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m46L88L1008899 for <@sunmail2sca.sfbay.sun.com:psarc-ext@Sun.COM>; Wed, 7 May 2008 05:08:15 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K0G0050BTDOVH00@brm-avmta-1.central.sun.com> for psarc-ext@Sun.COM (ORCPT psarc-ext@Sun.COM); Tue, 06 May 2008 15:08:12 -0600 (MDT) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K0G00M5TTDMSM60@brm-avmta-1.central.sun.com> for psarc-ext@Sun.COM (ORCPT psarc-ext@Sun.COM); Tue, 06 May 2008 15:08:10 -0600 (MDT) Received: from spartan.eng.sun.com (spartan.SFBay.Sun.COM [129.146.226.64]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m46L89wi042498; Tue, 06 May 2008 14:08:09 -0700 (PDT) Received: from spartan (spartan [129.146.226.64]) by spartan.eng.sun.com (8.13.7+Sun/8.13.7) with SMTP id m46L0ePM003259; Tue, 06 May 2008 14:00:40 -0700 (PDT) Date: Tue, 06 May 2008 14:00:40 -0700 (PDT) From: Don Cragun Subject: Re: SAM-QFS Tar Format Update [PSARC/2008/267 FastTrack timeout 05/07/2008] To: Dean.Roehrich@sun.com Cc: Eric.Diven@sun.com, psarc-ext@sun.com Reply-to: Don Cragun Message-id: <200805062100.m46L0ePM003259@spartan.eng.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.6.2 SunOS 5.10 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-MD5: Iu21aGGTOOgZmk6iOn33jA== X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 3194 >Date: Tue, 06 May 2008 15:38:56 -0500 >From: Dean Roehrich > >On Tue, May 06, 2008 at 10:32:28AM -0700, Don Cragun wrote: >> >Detail: ... ... ... >> > The default behavior will call for 'sam-arcopy' to continue writing >> > the old-style tar format. The new format can be selected by setting a >> > new 'tar_format = posix' option in the existing SAM-QFS config file, >> > /etc/opt/SUNWsamfs/defaults.conf. >> >> Why isn't this 'tar_format = pax'? The POSIX standard specifies three > >The project team agreed to this change. OK. > ... ... ... >> >> In the standard, ustar and pax archive formats have magic as a 6 octet >> field. Am I correct in assuming that you mean magic will be set to >> "ustar" (with a trailing null byte) and version will be set to "00" >> (with no trailing null) as specified for the ustar and pax archive >> formats? > >The functional spec has been clarified to say: > > The pax format requires that the magic is set to "ustar[null]" and the > version to "00" (which is not null terminated). > > >I've updated the case with the new spec. OK. > > >> > If an old version of SAM-QFS attempts to read a new pax format tar >> > header it will declare the file 'damaged', which is an existing >> > SAM-QFS concept. No legacy patch is planned, and this incompatibility >> > will be documented for customers. >> > >> > The SAM-QFS 'sls -D' command and option, a SAM-QFS-aware version of >> > ls(1), will be modified to report the tape format used for the given >> > file, if that file is currently on tape. It will make this decision >> > based on an available bit in the on-disk inode, which is currently >> > zeroed on existing SAM-QFS filesystems. >> > >> > A new versioned library, libpax_xhdr.so, will be added to SAM-QFS. >> > This library is created by abstracting the tar-specific bits out of >> > 'stager' and 'arcopy'. >> >> Rather than factor tar, pax, stager, and arcopy internals into a new >> library, why not just invoke the pax utility to read and write pax >> archives? > >The sam-arcopy and sam-stagerd daemons are able to use filesystem-specific I/O >operations to read and write the files to/from the filesystem. This means >files "invisibly" move between the disk and tape, which is a necessary feature >of HSM products such as SAM-QFS to make the disk appear "bottomless". > >The sam-arcopy and sam-stagerd daemons are highly threaded to optimize their >particular activity profile, and they use direct I/O. That activity profile >also makes it expensive to fork+exec a tar command for every file request, >because they jump around a tape rather than reading/writing it from start to >finish. Tapes much have changed a lot since I stopped using them regularly. I don't remember any drives (other than DECtapes) where you could skip around on the tape while writing files to it. You could do it on reads, but it was highly inefficient. > >In summary: the current design is that these daemons have the bare-bones bits >necessary to read and write the archives. The project team doesn't want to >redesign these daemons, they just want to add the new header format. OK. Thanks for the info. - Don > >Dean From roehrich@kickball-mn.Central.Sun.COM Wed May 7 10:37:11 2008 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m47HbBrj011103 for ; Wed, 7 May 2008 10:37:11 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m47Hb7HW021277 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Wed, 7 May 2008 10:37:11 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K0I0000NE9Y0B00@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 07 May 2008 11:37:10 -0600 (MDT) Received: from dm-central-01.central.sun.com ([129.147.62.4]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K0I00HU2E9Y79A0@brm-avmta-1.central.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Wed, 07 May 2008 11:37:10 -0600 (MDT) Received: from kickball-mn.Central.Sun.COM (kickball-mn.Central.Sun.COM [10.1.170.217]) by dm-central-01.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m47Hb9nl018713; Wed, 07 May 2008 11:37:09 -0600 (MDT) Received: from kickball-mn.Central.Sun.COM (localhost [127.0.0.1]) by kickball-mn.Central.Sun.COM (8.13.6+Sun/8.13.6) with ESMTP id m47Hb9uX029649; Wed, 07 May 2008 12:37:09 -0500 (CDT) Received: (from roehrich@localhost) by kickball-mn.Central.Sun.COM (8.13.6+Sun/8.13.6/Submit) id m47Hb8A2029648; Wed, 07 May 2008 12:37:08 -0500 (CDT) Date: Wed, 07 May 2008 12:37:08 -0500 From: Dean Roehrich Subject: Re: SAM-QFS Tar Format Update [PSARC/2008/267 FastTrack timeout 05/07/2008] In-reply-to: <20080430195820.GA2911@kickball-mn.Central.Sun.COM> To: psarc-ext@sun.com Cc: Eric.Diven@sun.com, roman.saucedo@sun.com, ted.pogue@sun.com, mary.matejcek@sun.com Message-id: <20080507173708.GA29616@kickball-mn.Central.Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <20080430195820.GA2911@kickball-mn.Central.Sun.COM> User-Agent: Mutt/1.5.9i Status: RO Content-Length: 54 This case was approved at PSARC business today. Dean From Joerg.Schilling@fokus.fraunhofer.de Fri May 9 03:33:00 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m49AWwj1011341 for ; Fri, 9 May 2008 03:32:59 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m49AWwDJ027435; Fri, 9 May 2008 03:32:58 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0K0L00901JYY1U00@nwk-avmta-1.sfbay.Sun.COM>; Fri, 09 May 2008 03:32:58 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0K0L00BYNJYXBQA0@nwk-avmta-1.sfbay.Sun.COM>; Fri, 09 May 2008 03:32:57 -0700 (PDT) Received: from relay21.sun.com (relay21.sun.com [192.12.251.24] (may be forged)) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m49ARSKC003261; Fri, 09 May 2008 10:32:57 +0000 (GMT) Received: from mms23es.sun.com ([150.143.232.54] [150.143.232.54]) by relay21i.sun.com with ESMTP id BT-MMP-927617; Fri, 09 May 2008 10:32:57 +0000 (Z) Received: from relay23.sun.com (relay23.sun.com [192.12.251.54]) by mms23es.sun.com with ESMTP id BT-MMP-37760961; Fri, 09 May 2008 10:32:56 +0000 (Z) Received: from mailgw1.fraunhofer.de ([153.96.1.17] [153.96.1.17]) by relay23i.sun.com with ESMTP id BT-MMP-21429013; Fri, 09 May 2008 10:32:56 +0000 (Z) Received: from mailgw1.fraunhofer.de (localhost [127.0.0.1]) by mailgw1.fraunhofer.de[host mailgw26] (8.14.2+/8.14.2) with ESMTP id m49ASjWj023401; Fri, 09 May 2008 12:28:45 +0200 (CEST) Received: from pluto.fokus.fraunhofer.de (pluto.fokus.fraunhofer.de [195.37.77.164]) by mailgw1.fraunhofer.de (8.14.2+/8.14.2) with ESMTP id m49ASi01023395 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 09 May 2008 12:28:45 +0200 (CEST) Received: from EXCHSRV.fokus.fraunhofer.de (bohr [10.147.9.231]) by pluto.fokus.fraunhofer.de (8.13.7/8.13.7) with SMTP id m49ASiYt028625; Fri, 09 May 2008 12:28:44 +0200 (MEST) Received: from rigel ([10.147.65.195]) by EXCHSRV.fokus.fraunhofer.de with Microsoft SMTPSVC(6.0.3790.3959); Fri, 09 May 2008 12:28:44 +0200 Date: Fri, 09 May 2008 12:28:44 +0200 From: Joerg.Schilling@fokus.fraunhofer.de (Joerg Schilling) Subject: Re: SAM-QFS Tar Format Update [PSARC/2008/267 FastTrack timeout 05/07/2008] In-reply-to: <20080430195820.GA2911@kickball-mn.Central.Sun.COM> To: psarc-ext@sun.com, Dean.Roehrich@sun.com Message-id: <4824275c.yLak5s9/HdXv8xxa%Joerg.Schilling@fokus.fraunhofer.de> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 8BIT X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Fraunhofer-Email-Policy: accepted X-Antispam: No, score=0.0/5.0, scanned in 0.349sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <20080430195820.GA2911@kickball-mn.Central.Sun.COM> User-Agent: nail 11.22 3/20/05 X-OriginalArrivalTime: 09 May 2008 10:28:44.0840 (UTC) FILETIME=[75824280:01C8B1BF] Status: RO Content-Length: 2549 Dean Roehrich wrote: > Detail: > > The SAM-QFS daemons 'sam-stagerd' and 'sam-arcopy' are currently > reading and writing a tape format that is a slightly modified version > of a format used by a very old version of GNU tar. The older GNU tar > format was modified for SAM-QFS because it didn't satify the project's > needs for file size and long file names. > > The SAM-QFS team would like to modify these daemons to read and write > a pax compliant tar format, as described in [1]. > > The default behavior will call for 'sam-arcopy' to continue writing > the old-style tar format. The new format can be selected by setting a > new 'tar_format = posix' option in the existing SAM-QFS config file, > /etc/opt/SUNWsamfs/defaults.conf. Be very careful when copying POSIX.1-2001 implementation details from newer GNU tar implementations. 1) The current POSIX archive format (ustar + POSIX.1-2001 extended headers) is called "pax" and not "posix". 2) GNU tar's archive format extensions are not "stable" as they are frequently changed without notice in an self-incompatible way (see e.g. below for GNU tar sparse extensions). 3) GNU tar in many implementation details does not completely follow the POSIX "pax" standard. GNU tar sparse extensions did for some time completely disregard the POSIX rules for extended tar headers. After I noticed this and made a bug report, the format later has been changed in an incompatible way. The new format is still sub-optimal and in addition hard to implement inside a double buffered tar implementaion such as star(1). 4) Some GNU tar extensions are implemented in a way that could easily eat up a nearly unlimited amount of memory given the "right" constraints. 5) GNU tar's multi-volume extensions often fail to read multi-volume archives created by GNU tar. They cannot be implemented in double buffered tar implementations such aass star(1). 6) GNU tar's archive format is not documented in a way sufficient way. If you are looking for POSIX.1-2001 extensions that are defined in a POSIX.1-2001 compliant way and that stay stable/compatibly-evolving, have a look at: http://cdrecord.berlios.de/old/private/man/star/star.4.html Jörg -- EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin js@cs.tu-berlin.de (uni) schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily