From sacadmin Tue Jun 7 08:25:27 2005 Received: from eastmail1bur.East.Sun.COM (eastmail1bur.East.Sun.COM [129.148.9.49]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j57FPRs0008847 for ; Tue, 7 Jun 2005 08:25:27 -0700 (PDT) Received: from [129.148.226.11] (sr1-unsh01-01.East.Sun.COM [129.148.226.11]) by eastmail1bur.East.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id j57FP2Ft028807; Tue, 7 Jun 2005 11:25:02 -0400 (EDT) Message-ID: <42A5BC4E.5000609@sun.com> Date: Tue, 07 Jun 2005 11:25:02 -0400 From: Brian Utterback User-Agent: Mozilla Thunderbird 1.0.4 (X11/20050603) X-Accept-Language: en-us, en MIME-Version: 1.0 To: psarc@sac.sfbay.sun.com CC: frank.hofmann@sun.com Subject: 2005/361 PCFS timestamp handling cleanup Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 4003 I am sponsoring this case for Frank Hofmann. The timer expires on 06/14/2005. The requested binding is patch. Thanks. Summary: This proposal requests, for Patch binding, * to modify PCFS timestamp handling so that post-epoch times (which are possible in the FAT filesystem) are clamped to the end-of-epoch if a 32bit application inquires about it * to simplify PCFS timestamp handling by removing the broken code in it that attempts to handle daylight saving time adjustments. The Problem Timestamps on FAT filesystems cover a range (the "FAT epoch") between 01/Jan/1980, 00:00:00, and 31/12/2127, 23:59:59, and are represented as date/time bitfields. They must therefore be converted to the UNIX time_t scheme of "seconds since 01/Jan/1970, 00:00:00 UT" before use. In addition to that, FAT timestamps are recorded in local time (!) of the recording system, but the recorder's timezone is not stored in the filesystem metadata anywhere. With the current PCFS code, this causes several problems: a) If we're running a 64bit kernel, we dutifully return a 64bit time_t post-2038 (UNIX epoch) if the FAT timestamp specifies such. This causes stat() syscalls done by 32bit applications to fail with EOVERFLOW. It breaks all our file utils, including touch(1), so that such a file cannot even be "fixed" by touch'ing it. b) If we're running a 32bit kernel, the calculation of the UNIX time corresponding to a given FAT timestamp overflows and hence for post-UNIX-epoch FAT times, essentially a random value is returned. This causes no failure but prevents applications from detecting that something is odd about a file's access/mod time. As a best-guess attempt, PCFS accepts timezone information via mount argument, "struct pcfs_args", which contains two fields, "timezone" (which specifies seconds west of UT), and "daylight" (which specifies whether daylight savings time adjustments are supposed to be made). The daylight savings time adjustments done by pcfs are based on incomplete (only covering US/EMEA/AUS timezones) and completely outdated (not having been updated with DST rule changes made after 1985). PCFS in all its lifetime never dealt with this correctly. The Proposed Solution * If a FAT timestamp post-epoch is encountered when reading filesystem metadata, PCFS will - detect whether the caller is a 32bit application by using get_udatamodel(), and clamp the time_t to INT32_MAX. - return the full/correct 64bit timestamp if the application is 64bit. This allows 32bit application to detect "out of range" timestamps the same way as this works on other filesystems (ufs and udf, for example) - by testing for the epoch. * If a 64bit application requests writing a timestamp beyond the FAT epoch (see above), which is possible by explicit utimes(2) system calls, PCFS will log a kern.notice message about the attempt and set the FAT timestamp to the FAT epoch. * The code for dealing with daylight savings time adjustments in the conversion between UNIX time_t and FAT timestamps will be removed. Local time adjustment will be done exclusively based on the "seconds west of UT" information. Given the abovementioned restriction (no way of discovering the timezone in which the medium was created, and all such attempts therefore necessarily being "best guess"), the timezone correction via "seconds west of UT" is more than sufficient. MAN PAGE CHANGES The man page for pcfs(7fs) will be changed to reflect that the "daylight" member of "struct pcfs" has no effect. Manpage diffs are available. -- blu Remember when SOX compliant meant they were both the same color? ---------------------------------------------------------------------- Brian Utterback - OP/N1 RPE, Sun Microsystems, Inc. Ph:877-259-7345, Em:brian.utterback-at-ess-you-enn-dot-kom From sacadmin Tue Jun 7 08:38:02 2005 Received: from phorcys.East.Sun.COM (phorcys.East.Sun.COM [129.148.174.143]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j57Fc1s0009370 for ; Tue, 7 Jun 2005 08:38:02 -0700 (PDT) Received: from phorcys.East.Sun.COM (localhost [127.0.0.1]) by phorcys.East.Sun.COM (8.13.4+Sun/8.13.4) with ESMTP id j57Fbcr8003283; Tue, 7 Jun 2005 11:37:38 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.East.Sun.COM (8.13.4+Sun/8.13.4/Submit) id j57FbblY003280; Tue, 7 Jun 2005 11:37:37 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17061.48956.353342.774427@gargle.gargle.HOWL> Date: Tue, 7 Jun 2005 11:37:32 -0400 From: James Carlson To: Brian Utterback Cc: psarc@sac.sfbay.sun.com, Frank.Hofmann@Sun.COM Subject: Re: 2005/361 PCFS timestamp handling cleanup In-Reply-To: Brian Utterback's message of 7 June 2005 11:25:02 References: <42A5BC4E.5000609@sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 799 Brian Utterback writes: > I am sponsoring this case for Frank Hofmann. The timer expires on > 06/14/2005. The requested binding is patch. Thanks. Updates to PCFS! This is nice to see. > * If a 64bit application requests writing a timestamp beyond the > FAT epoch (see above), which is possible by explicit utimes(2) > system calls, PCFS will log a kern.notice message about the > attempt and set the FAT timestamp to the FAT epoch. Does writing a timestamp before the FAT epoch ("touch -t 7001010000") result in an error, conversion to 1980, or something else? -- James Carlson, KISS Network Sun Microsystems / 1 Network Drive 71.234W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.497N Fax +1 781 442 1677 From sacadmin Tue Jun 7 08:43:12 2005 Received: from phys-mayi-1 (phys-mayi-1-ipmp1.Germany.Sun.COM [129.157.128.114]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j57FhBs0009991 for ; Tue, 7 Jun 2005 08:43:11 -0700 (PDT) Received: from conversion-daemon.mayi-mail1.germany.sun.com by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IHQ00N010X414@mayi-mail1.germany.sun.com> (original mail from frankho@mayi-mail1.germany.sun.com) for psarc@sac.sfbay.sun.com; Tue, 07 Jun 2005 17:42:47 +0200 (MEST) Received: from estale (estale.UK.Sun.COM [129.156.173.199]) by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with SMTP id <0IHQ00A040ZAO8@mayi-mail1.germany.sun.com>; Tue, 07 Jun 2005 17:42:47 +0200 (MEST) Date: Tue, 07 Jun 2005 16:42:46 +0100 (BST) From: Frank Hofmann - Solaris Sustaining Subject: Re: 2005/361 PCFS timestamp handling cleanup To: Brian.Utterback@Sun.COM, James.D.Carlson@Sun.COM Cc: psarc@sac.sfbay.sun.com, Frank.Hofmann@Sun.COM Reply-to: Frank Hofmann - Solaris Sustaining Message-id: <0IHQ00A060ZAO8@mayi-mail1.germany.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_16 SunOS 5.11 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-MD5: sB0foZZizL/SYuk/+0FUsQ== Status: RO Content-Length: 730 > Brian Utterback writes: > > I am sponsoring this case for Frank Hofmann. The timer expires on > > 06/14/2005. The requested binding is patch. Thanks. > > Updates to PCFS! This is nice to see. It's about time, isn't it ? > > > * If a 64bit application requests writing a timestamp beyond the > > FAT epoch (see above), which is possible by explicit utimes(2) > > system calls, PCFS will log a kern.notice message about the > > attempt and set the FAT timestamp to the FAT epoch. > > Does writing a timestamp before the FAT epoch ("touch -t 7001010000") > result in an error, conversion to 1980, or something else? It'll clamp on both sides, i.e. a pre-FATepoch time will result in conversion to 1980. FrankH. From sacadmin Tue Jun 7 16:03:58 2005 Received: from spartan.SFBay.Sun.COM (spartan [129.146.226.64]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j57N3ws0020238 for ; Tue, 7 Jun 2005 16:03:58 -0700 (PDT) Received: from spartan.SFBay.Sun.COM (spartan.SFBay.Sun.COM [129.146.226.64]) by spartan.SFBay.Sun.COM (8.12.10+Sun/8.12.10) with SMTP id j57N3VQi020829; Tue, 7 Jun 2005 16:03:31 -0700 (PDT) Message-Id: <200506072303.j57N3VQi020829@spartan.SFBay.Sun.COM> Date: Tue, 7 Jun 2005 16:03:31 -0700 (PDT) From: Don Cragun Reply-To: Don Cragun Subject: Re: 2005/361 PCFS timestamp handling cleanup To: frank.hofmann@Sun.COM Cc: psarc@sac.sfbay.sun.com, brian.utterback@Sun.COM MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: uZR2I2jatXqBx2i+DiZtfQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.5.5 SunOS 5.9 sun4u sparc Status: RO Content-Length: 1118 Frank, I think this case is headed in the wrong direction. We should never tell applications that we succeeded in setting a timestamp if we cannot set the timestamp to the requested value. If an application asks the file system to set a timestamp it can't handle it should fail with errno set to EOVERFLOW or ERANGE depending on the problem. If the file system contains a timestamp that can't be presented correctly through a 32-bit interface, the interface should fail with errno set to EOVERFLOW. If you believe that there are really going to be files with timestamps in the near future that can't be handled by our file handling utilities (as opposed to the theoretical possibility that someone can create files with these timestamps), you should file bugs to convert those file handling utilities to use 64-bit time processing. You should not rewrite file systems to (arbitrarily and without error) convert requested time stamps to the beginning or end or a 32-bit time_t or the begining or end of the range of dates a file system or some subset of applications might be presumed to be able to handle. Don From sacadmin Tue Jun 7 16:33:07 2005 Received: from eastmail2bur.East.Sun.COM (eastmail2bur.East.Sun.COM [129.148.13.40]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j57NX7s0021536 for ; Tue, 7 Jun 2005 16:33:07 -0700 (PDT) Received: from thunk.east.sun.com (thunk.East.Sun.COM [129.148.174.66]) by eastmail2bur.East.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id j57NWZC0019956; Tue, 7 Jun 2005 19:32:35 -0400 (EDT) Received: from 127.0.0.1 (localhost [127.0.0.1]) by thunk.east.sun.com (8.13.3+Sun/8.13.3) with ESMTP id j57NWZTb001417; Tue, 7 Jun 2005 19:32:35 -0400 (EDT) Subject: Re: 2005/361 PCFS timestamp handling cleanup From: Bill Sommerfeld To: Don Cragun Cc: Frank.Hofmann@sun.com, psarc@sac.sfbay.sun.com, Brian.Utterback@sun.com In-Reply-To: <200506072303.j57N3VQi020829@spartan.SFBay.Sun.COM> References: <200506072303.j57N3VQi020829@spartan.SFBay.Sun.COM> Content-Type: text/plain; charset=iso-8859-1 Message-Id: <1118187154.28403.349.camel@thunk> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6.309 Date: Tue, 07 Jun 2005 19:32:34 -0400 Content-Transfer-Encoding: 7bit Status: RO Content-Length: 942 > If you believe that there are really going to be files with > timestamps in the near future that can't be handled by our file > handling utilities (as opposed to the theoretical possibility that > someone can create files with these timestamps). I believe the motivation of this case is that pcfs filesystems "in the wild" routinely include horribly broken timestamps like this. > you should file bugs > to convert those file handling utilities to use 64-bit time processing. Unfortunately, it looks like we blew it when we defined the transitional lf64 interfaces; even "struct stat64" only includes a 32-bit time_t in the timestamps and we don't ship a 32-bit version of localtime/gmtime which can cope with 64-bit time_t's, so there appears to be no way for a program written to our 32-bit ABIs to cope with files with out-of-range timestamps. This appears to be blowing the scope of the case way out of proportion. - Bill From sacadmin Tue Jun 7 16:50:59 2005 Received: from phys-aus08-2 (phys-aus08-2.Central.Sun.COM [129.153.131.89]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j57Nows0022348 for ; Tue, 7 Jun 2005 16:50:59 -0700 (PDT) Received: from conversion-daemon.aus08-mail1.central.sun.com by aus08-mail1.central.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IHQ00C01MWSJM@aus08-mail1.central.sun.com> (original mail from David.Robinson@sun.com) for psarc@sac.sfbay.sun.com; Tue, 07 Jun 2005 18:50:33 -0500 (CDT) Received: from [129.153.128.60] (jetsun.Central.Sun.COM [129.153.128.60]) by aus08-mail1.central.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with ESMTP id <0IHQ00ENKNK7KH@aus08-mail1.central.sun.com>; Tue, 07 Jun 2005 18:50:33 -0500 (CDT) Date: Tue, 07 Jun 2005 18:50:31 -0500 From: David Robinson Subject: Re: 2005/361 PCFS timestamp handling cleanup In-reply-to: <200506072303.j57N3VQi020829@spartan.SFBay.Sun.COM> To: Don Cragun Cc: Frank.Hofmann@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Message-id: <42A632C7.3040805@sun.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-Accept-Language: en-us, en User-Agent: Mozilla Thunderbird 1.0.2 (X11/20050329) References: <200506072303.j57N3VQi020829@spartan.SFBay.Sun.COM> Status: RO Content-Length: 1586 Don Cragun wrote: > I think this case is headed in the wrong direction. > We should never tell applications that we succeeded in setting > a timestamp if we cannot set the timestamp to the requested value. If > an application asks the file system to set a timestamp it can't handle > it should fail with errno set to EOVERFLOW or ERANGE depending on the > problem. If the file system contains a timestamp that can't be > presented correctly through a 32-bit interface, the interface should > fail with errno set to EOVERFLOW. > If you believe that there are really going to be files with > timestamps in the near future that can't be handled by our file > handling utilities (as opposed to the theoretical possibility that > someone can create files with these timestamps), you should file bugs > to convert those file handling utilities to use 64-bit time processing. > You should not rewrite file systems to (arbitrarily and without error) > convert requested time stamps to the beginning or end or a 32-bit > time_t or the begining or end of the range of dates a file system or > some subset of applications might be presumed to be able to handle. As a data point, the NFSv2 and v3 protocols define a timestamp as an unsigned integer since 1970. By default stat of any file with a negative timestamp will return EOVERFLOW. It can be overridden by setting the /etc/system tuneable nfs_allow_preepoch_time. Only a few questions or complaints have ever been raised about this behavior. Although in hindsight this should have been a mount option instead of a tuneable. -David From sacadmin Tue Jun 7 17:01:44 2005 Received: from eastmail1bur.East.Sun.COM (eastmail1bur.East.Sun.COM [129.148.9.49]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j5801hs0024149 for ; Tue, 7 Jun 2005 17:01:43 -0700 (PDT) Received: from thunk.east.sun.com (thunk.East.Sun.COM [129.148.174.66]) by eastmail1bur.East.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id j5801FFt014051; Tue, 7 Jun 2005 20:01:15 -0400 (EDT) Received: from 127.0.0.1 (localhost [127.0.0.1]) by thunk.east.sun.com (8.13.3+Sun/8.13.3) with ESMTP id j5801EjL001497; Tue, 7 Jun 2005 20:01:14 -0400 (EDT) Subject: Re: 2005/361 PCFS timestamp handling cleanup From: Bill Sommerfeld To: David Robinson Cc: Don Cragun , Frank.Hofmann@sun.com, psarc@sac.sfbay.sun.com, Brian.Utterback@sun.com In-Reply-To: <42A632C7.3040805@sun.com> References: <200506072303.j57N3VQi020829@spartan.SFBay.Sun.COM> <42A632C7.3040805@sun.com> Content-Type: text/plain Message-Id: <1118188873.28403.354.camel@thunk> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6.309 Date: Tue, 07 Jun 2005 20:01:14 -0400 Content-Transfer-Encoding: 7bit Status: RO Content-Length: 372 On Tue, 2005-06-07 at 19:50, David Robinson wrote: > Only a few questions or complaints have ever been raised about this > behavior. Although in hindsight this should have been a mount option > instead of a tuneable. pcfs filesystems are substantially more likely to be written to by embedded devices (such as cameras) which may have a poor idea of what decade it is. From sacadmin Wed Jun 8 03:20:53 2005 Received: from phys-mayi-1 (phys-mayi-1-ipmp1.Germany.Sun.COM [129.157.128.114]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58AKqs0022901 for ; Wed, 8 Jun 2005 03:20:53 -0700 (PDT) Received: from conversion-daemon.mayi-mail1.germany.sun.com by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IHR00201GPAGV@mayi-mail1.germany.sun.com> (original mail from frankho@mayi-mail1.germany.sun.com) for psarc@sac.sfbay.sun.com; Wed, 08 Jun 2005 12:20:27 +0200 (MEST) Received: from estale (estale.UK.Sun.COM [129.156.173.199]) by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with SMTP id <0IHR00CQ4GQ2B9@mayi-mail1.germany.sun.com>; Wed, 08 Jun 2005 12:20:27 +0200 (MEST) Date: Wed, 08 Jun 2005 11:20:26 +0100 (BST) From: Frank Hofmann - Solaris Sustaining Subject: Re: 2005/361 PCFS timestamp handling cleanup To: Don.Cragun@Sun.COM, sommerfeld@sun.com Cc: Frank.Hofmann@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Reply-to: Frank Hofmann - Solaris Sustaining Message-id: <0IHR00CQ9GQ2B9@mayi-mail1.germany.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_16 SunOS 5.11 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-MD5: ThaCNQVp1D3AeB41AzKpOQ== Status: RO Content-Length: 5283 > > If you believe that there are really going to be files with > > timestamps in the near future that can't be handled by our file > > handling utilities (as opposed to the theoretical possibility that > > someone can create files with these timestamps). > > I believe the motivation of this case is that pcfs filesystems "in the > wild" routinely include horribly broken timestamps like this. This has actually happened - see #6246725 (which I closed as "not a bug" using essentially the same arguments as Don does), that's an incident report about encountering exactly such a broken pcfs filesystem. > > > you should file bugs > > to convert those file handling utilities to use 64-bit time > processing. I've tried that path as well - see #6248065. Cynthia Eastham gives some reasons there why this suggestion is only of long-term interest. There are other reasons why "convert all /usr/bin to 64bit" isn't being done: a) It's not just "some utilities". Essentially every file utility uses stat(2), including things like touch/rm/mv/cp/ls/dd/find/. b) it has quite a negative impact on performance in standard benchmarks when running all-64bit /usr/bin. c) even doing that wouldn't help at all on a 32bit system (which we still support on x86) - touch(1) is one of the affected utilities, i.e. one cannot even "fix" the broken timestamp on a file, and a 64bit touch(1) wouldn't help with that on a 32bit system. I.e. if 64bit file utils are the answer, then the next question is obvious: What do do on 32bit systems ? ALL our file handling utilities are affected by this, i.e. they're 32bit applications even if we run a 64bit kernel. The main problem is that if a broken timestamp is encountered on a 64bit kernel, they fail without the user/admin having a chance of fixing the problem. The far simpler case therefore is to let pcfs lie to 32bit applications and tell the truth only to 64bit ones. Apart from somebody doing deliberate tests for 32/64bit differences (i.e.: let a 64bit app set a post-epoch timestamp and if that succeeds let a 32bit app stat that file to see whether it gets EOVERFLOW) this will not hurt anybody - a significant contrast to the current behaviour. And, as said, 64bit applications *will* see the full available timestamp range, so there's no standards issue for them anyway. To state that clearly: * This case isn't about making pcfs non-standard wrt. to timestamp handling. 64bit stat() and utimes() will NOT change behaviour. * This case is about changing 32bit timestamp handling in pcfs. Which at the moment is completely broken on 32bit kernels (arithmetic overflow), but DUE TO THAT BROKENNESS actually works (!) on 32bit kernels in the sense that file utilities can stat files with such timestamps. I.e. The case supposes to sanction the existing behaviour, 32bit applications that currently "work" on 32bit kernels shall continue to "work" on 64bit kernels. EVEN if they only work right now because the 32bit pcfs module has a bug. * The alternative (fixing pcfs so that 32bit apps will always see an EOVERFLOW error if encountering an invalid timestamp) is to deliberately break things that currently just work from the perspective of a user (example: "ls -l" on a 32/64bit kernel, see #6246725). With no chance of fixing it, since, as Brian mentioned, no lf64 interfaces to deal with 64bit time exist. * The only way that I see to achieve these two objectives: 1. 32bit apps behave identically no matter whether the kernel is 64bit or 32bit 2. no negative surprises to the user, i.e. don't deliberately break things (32bit app on 32bit kernel) that "works" right now (even if it only works due to a bug), is to introduce clamping for pcfs timestamps if the calling context is 32bit, i.e. for the case that somebody runs a 32bit app on a 64bit kernel. The whole argumentation wrt. to what pcfs should do is based on the simple assumption: "What works on a 32bit kernel shall also work on a 64bit kernel". Fixing the 32bit bug (arithmetic overflow in timestamp calculation, wrapping around) in the "standards" way (returning EOVERFLOW to 32bit apps for those timestamps) will have a usability regression, one that is UNFIXABLE on 32bit kernels. Fixing the 32bit bug in the proposed way (returning the epoch) will have NO such usability regression. Everything that works before the change will work the same way after the change - except that instead of seeing a "semi-random" (32bit wrapped around) timestamp they'll now see a decisive value - the epoch. > > Unfortunately, it looks like we blew it when we defined the transitional > lf64 interfaces; even "struct stat64" only includes a 32-bit time_t in > the timestamps and we don't ship a 32-bit version of localtime/gmtime > which can cope with 64-bit time_t's, so there appears to be no way for a > program written to our 32-bit ABIs to cope with files with out-of-range > timestamps. > > This appears to be blowing the scope of the case way out of proportion. Yes. "create 64bit /usr/bin/..." is not the scope of this request. This request is strictly limited to allowing for a defined, consistent behaviour of pcfs wrt. to handling timestamps for 32bit applications. FrankH. From sacadmin Wed Jun 8 03:29:17 2005 Received: from sunnl.Holland.Sun.COM (sunnl.Holland.Sun.COM [129.159.201.1]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58ATGs0023091 for ; Wed, 8 Jun 2005 03:29:17 -0700 (PDT) Received: from vaticaan.Holland.Sun.COM (vaticaan [129.159.201.10]) by sunnl.Holland.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.3beta1412) with ESMTP id j58ASlGg019752; Wed, 8 Jun 2005 12:28:47 +0200 (MEST) Received: from holland (casper@room101 [129.159.201.52]) by vaticaan.Holland.Sun.COM (8.12.10+Sun/8.12.9) with ESMTP id j58ASlDk020107; Wed, 8 Jun 2005 12:28:47 +0200 (MEST) Message-Id: <200506081028.j58ASlDk020107@vaticaan.Holland.Sun.COM> From: Casper.Dik@Sun.COM To: Frank Hofmann - Solaris Sustaining cc: Don.Cragun@Sun.COM, sommerfeld@Sun.COM, Frank.Hofmann@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Subject: Re: 2005/361 PCFS timestamp handling cleanup In-Reply-To: <0IHR00CQ9GQ2B9@mayi-mail1.germany.sun.com> References: <0IHR00CQ9GQ2B9@mayi-mail1.germany.sun.com> Date: Wed, 08 Jun 2005 12:28:47 +0200 Sender: casper@holland.sun.com Status: RO Content-Length: 1003 >This has actually happened - see #6246725 (which I closed as "not a bug" >using essentially the same arguments as Don does), that's an incident >report about encountering exactly such a broken pcfs filesystem. Note that this problem is exacerbated by the fact that there are two common PCFS layouts: one which uses the extension fields and one which do not; the latter shold not but may put junk there. >ALL our file handling utilities are affected by this, i.e. they're 32bit >applications even if we run a 64bit kernel. The main problem is that if >a broken timestamp is encountered on a 64bit kernel, they fail without >the user/admin having a chance of fixing the problem. Or rev libc.so and the ELF files to have a "Extended Fundamental Types Rev2" objects which will give the proper answer and have properly sized fields for everything (i.e., be more like the 64 bit libc) Considering that 32 bit systems are still shipping today, we will need to support this for many yers to come. Casper From sacadmin Wed Jun 8 05:28:06 2005 Received: from phys-mayi-1 (phys-mayi-1-ipmp1.Germany.Sun.COM [129.157.128.114]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58CS6s0028506 for ; Wed, 8 Jun 2005 05:28:06 -0700 (PDT) Received: from conversion-daemon.mayi-mail1.germany.sun.com by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IHR00I01MGWHN@mayi-mail1.germany.sun.com> (original mail from frankho@mayi-mail1.germany.sun.com) for psarc@sac.sfbay.sun.com; Wed, 08 Jun 2005 14:27:40 +0200 (MEST) Received: from estale (estale.UK.Sun.COM [129.156.173.199]) by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with SMTP id <0IHR005QZMM376@mayi-mail1.germany.sun.com>; Wed, 08 Jun 2005 14:27:40 +0200 (MEST) Date: Wed, 08 Jun 2005 13:27:39 +0100 (BST) From: Frank Hofmann - Solaris Sustaining Subject: Re: 2005/361 PCFS timestamp handling cleanup To: Casper.Dik@Sun.COM Cc: Don.Cragun@Sun.COM, sommerfeld@sun.com, Frank.Hofmann@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Reply-to: Frank Hofmann - Solaris Sustaining Message-id: <0IHR005R0MM376@mayi-mail1.germany.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_16 SunOS 5.11 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-MD5: af6/zd+PHBIpI3YkWsbKDg== Status: RO Content-Length: 2186 > >This has actually happened - see #6246725 (which I closed as "not a bug" > >using essentially the same arguments as Don does), that's an incident > >report about encountering exactly such a broken pcfs filesystem. > > Note that this problem is exacerbated by the fact that there > are two common PCFS layouts: one which uses the extension fields > and one which do not; the latter shold not but may put junk there. Microsoft's FAT specification says wrt. to timestamps (specifically, access time as it is one of the optional fields): "DIR_WrtTime and DIR_WrtDate must be supported, however. If the other date and time fields are not supported, they should be set to 0 on file create and ignored on other file operations." It then goes on to define what the valid ranges for time/date are. I would imply that "invalid" means "not supported". The problem is "should be set to 0", not "shall be set to 0". A FAT implementation that just keeps junk in non-mandatory timestamp fields is still conforming to the specification. As the spec says we should "ignore on file operations" this then, it also means we should not fail access if we find junk there. Unfortunately we can't distinguish between "junk" and "valid" for any timestamp except "last write" because there's no way to find out whether a medium was written by a FAT implementation that supports all timestamp fields or not. Which gives another very good argument that we should not block access to the file (i.e. EOVERFLOW/EINVAL on stat) if we find a timestamp that is out of the range that we can handle. To follow the FAT spec, we should "ignore" such timestamps - and the proposed way of doing that is to return a value for the timestamp that we can handle. I'm keeping a copy of the FAT specification within SWAN, if anybody wants to verify the above read page 25 of: http://www-ber.germany/~frankho/docs/fat32/fatgen103.pdf The above also means that the suggestion from the proposal to log a kern.notice message if an invalid FAT timestamp is found may only apply to the "last write time/date" - as these are mandatory. No notification for invalid/unparseable access timestamps is needed. FrankH. From sacadmin Wed Jun 8 08:34:12 2005 Received: from sfbaymail1sca.SFBay.Sun.COM (sfbaymail1sca.SFBay.Sun.COM [129.145.154.35]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58FYCs0007274 for ; Wed, 8 Jun 2005 08:34:12 -0700 (PDT) Received: from Spencer-Sheplers-Computer.local (vpn-129-150-35-47.Central.Sun.COM [129.150.35.47]) by sfbaymail1sca.SFBay.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id j58FXhPu028787; Wed, 8 Jun 2005 08:33:43 -0700 (PDT) Received: by Spencer-Sheplers-Computer.local (Postfix, from userid 76367) id 8BDF33EA303; Wed, 8 Jun 2005 10:34:02 -0500 (CDT) Date: Wed, 8 Jun 2005 10:34:02 -0500 From: Spencer Shepler To: Frank Hofmann - Solaris Sustaining Cc: Don.Cragun@sun.com, sommerfeld@sun.com, Frank.Hofmann@sun.com, psarc@sac.sfbay.sun.com, Brian.Utterback@sun.com Subject: Re: 2005/361 PCFS timestamp handling cleanup Message-ID: <20050608153402.GD4300@Spencer-Sheplers-Computer.local> Reply-To: spencer.shepler@sun.com References: <0IHR00CQ9GQ2B9@mayi-mail1.germany.sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <0IHR00CQ9GQ2B9@mayi-mail1.germany.sun.com> User-Agent: Mutt/1.4.1i Status: RO Content-Length: 2374 On Wed, Frank Hofmann - Solaris Sustaining wrote: > > > > If you believe that there are really going to be files with > > > timestamps in the near future that can't be handled by our file > > > handling utilities (as opposed to the theoretical possibility that > > > someone can create files with these timestamps). > > > > I believe the motivation of this case is that pcfs filesystems "in the > > wild" routinely include horribly broken timestamps like this. > > This has actually happened - see #6246725 (which I closed as "not a bug" > using essentially the same arguments as Don does), that's an incident > report about encountering exactly such a broken pcfs filesystem. > > > > > > you should file bugs > > > to convert those file handling utilities to use 64-bit time > > processing. > > I've tried that path as well - see #6248065. Cynthia Eastham gives > some reasons there why this suggestion is only of long-term interest. > There are other reasons why "convert all /usr/bin to 64bit" isn't being > done: > > a) It's not just "some utilities". Essentially every file utility uses > stat(2), including things like touch/rm/mv/cp/ls/dd/find/. > b) it has quite a negative impact on performance in standard benchmarks > when running all-64bit /usr/bin. > c) even doing that wouldn't help at all on a 32bit system (which we still > support on x86) - touch(1) is one of the affected utilities, i.e. one > cannot even "fix" the broken timestamp on a file, and a 64bit touch(1) > wouldn't help with that on a 32bit system. > I.e. if 64bit file utils are the answer, then the next question is > obvious: What do do on 32bit systems ? So on a general note of dealing with filesystems that have 64bit attributes (nodeids, sizes, timestamps, directory sizes), it seems like the time is now that applications need to change to at least use the lf64(5) interfaces. NFSv3 introduced 64bit nodeids and things have generally worked because most server filesystems haven't used them that much (to this point). However, NFSv4 has continued this trend and I imagine that with the introduction of ZFS and larger NFS servers in general that Solaris clients will begin to see larger attribute values. If 64bit apps are too expensive in general, then at least lf64(5) capabilities should be used. And, yes, this is on a tangent... Spencer From sacadmin Wed Jun 8 08:52:29 2005 Received: from sunnl.Holland.Sun.COM (sunnl.Holland.Sun.COM [129.159.201.1]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58FqSs0008484 for ; Wed, 8 Jun 2005 08:52:29 -0700 (PDT) Received: from vaticaan.Holland.Sun.COM (vaticaan [129.159.201.10]) by sunnl.Holland.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.3beta1412) with ESMTP id j58FpjGg009906; Wed, 8 Jun 2005 17:51:45 +0200 (MEST) Received: from holland (casper@room101 [129.159.201.52]) by vaticaan.Holland.Sun.COM (8.12.10+Sun/8.12.9) with ESMTP id j58FpjDk005304; Wed, 8 Jun 2005 17:51:45 +0200 (MEST) Message-Id: <200506081551.j58FpjDk005304@vaticaan.Holland.Sun.COM> From: Casper.Dik@Sun.COM To: Spencer.Shepler@Sun.COM cc: Frank Hofmann - Solaris Sustaining , Don.Cragun@Sun.COM, sommerfeld@Sun.COM, Frank.Hofmann@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Subject: Re: 2005/361 PCFS timestamp handling cleanup In-Reply-To: <20050608153402.GD4300@Spencer-Sheplers-Computer.local> References: <0IHR00CQ9GQ2B9@mayi-mail1.germany.sun.com> <20050608153402.GD4300@Spencer-Sheplers-Computer.local> Date: Wed, 08 Jun 2005 17:51:45 +0200 Sender: casper@holland.sun.com Status: RO Content-Length: 707 >So on a general note of dealing with filesystems that have 64bit >attributes (nodeids, sizes, timestamps, directory sizes), it seems like >the time is now that applications need to change to at least use >the lf64(5) interfaces. NFSv3 introduced 64bit nodeids and things >have generally worked because most server filesystems haven't used >them that much (to this point). However, NFSv4 has continued this >trend and I imagine that with the introduction of ZFS and larger >NFS servers in general that Solaris clients will begin to see larger >attribute values. If 64bit apps are too expensive in general, then >at least lf64(5) capabilities should be used. But as noted these are incomplete. Casper From sacadmin Wed Jun 8 09:08:36 2005 Received: from phys-mayi-1 (phys-mayi-1-ipmp1.Germany.Sun.COM [129.157.128.114]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58G8Zs0010639 for ; Wed, 8 Jun 2005 09:08:36 -0700 (PDT) Received: from conversion-daemon.mayi-mail1.germany.sun.com by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IHR00201WJTTW@mayi-mail1.germany.sun.com> (original mail from frankho@mayi-mail1.germany.sun.com) for psarc@sac.sfbay.sun.com; Wed, 08 Jun 2005 18:08:10 +0200 (MEST) Received: from estale (estale.UK.Sun.COM [129.156.173.199]) by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with SMTP id <0IHR00F5OWTK9O@mayi-mail1.germany.sun.com>; Wed, 08 Jun 2005 18:08:09 +0200 (MEST) Date: Wed, 08 Jun 2005 17:08:08 +0100 (BST) From: Frank Hofmann - Solaris Sustaining Subject: Re: 2005/361 PCFS timestamp handling cleanup To: Spencer.Shepler@Sun.COM, Casper.Dik@Sun.COM Cc: Frank.Hofmann@Sun.COM, Don.Cragun@Sun.COM, sommerfeld@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Reply-to: Frank Hofmann - Solaris Sustaining Message-id: <0IHR00F5QWTK9O@mayi-mail1.germany.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_16 SunOS 5.11 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-MD5: zHHlp4ep5jneFISLIq2nXQ== Status: RO Content-Length: 1403 > >So on a general note of dealing with filesystems that have 64bit > >attributes (nodeids, sizes, timestamps, directory sizes), it seems like > >the time is now that applications need to change to at least use > >the lf64(5) interfaces. NFSv3 introduced 64bit nodeids and things > >have generally worked because most server filesystems haven't used > >them that much (to this point). However, NFSv4 has continued this > >trend and I imagine that with the introduction of ZFS and larger > >NFS servers in general that Solaris clients will begin to see larger > >attribute values. If 64bit apps are too expensive in general, then > >at least lf64(5) capabilities should be used. > > But as noted these are incomplete. And to add to it, they're already being used by the existing 32bit utilities. But making a program largefile-capable doesn't make it large-timestamp/devid capable. lf64(5) does specify a way for a 32bit app to use 64bit size_t/offset_t, but misses to specify a way for 32bit apps to deal with 64bit time_t or dev_t. The interface just isn't there. The biggest issue is stat(). Even if the 32bit application does not need or want to know about 64bit time_t/dev_t, it wants to know about 64bit size_t/offset_t - but stat() can only succeed or fail as a whole. But how else but stat() can you query e.g. for whether the queried entity is a file, directory, device, ... ? FrankH. From sacadmin Wed Jun 8 11:54:33 2005 Received: from spartan.SFBay.Sun.COM (spartan [129.146.226.64]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58IsXs0024862 for ; Wed, 8 Jun 2005 11:54:33 -0700 (PDT) Received: from spartan.SFBay.Sun.COM (spartan.SFBay.Sun.COM [129.146.226.64]) by spartan.SFBay.Sun.COM (8.12.10+Sun/8.12.10) with SMTP id j58Is7Qi022110; Wed, 8 Jun 2005 11:54:07 -0700 (PDT) Message-Id: <200506081854.j58Is7Qi022110@spartan.SFBay.Sun.COM> Date: Wed, 8 Jun 2005 11:54:07 -0700 (PDT) From: Don Cragun Reply-To: Don Cragun Subject: Re: 2005/361 PCFS timestamp handling cleanup To: frank.hofmann@Sun.COM Cc: psarc@sac.sfbay.sun.com, brian.utterback@Sun.COM MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: zVFvRnp46wW8xf6kbkjxAA== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.5.5 SunOS 5.9 sun4u sparc Status: RO Content-Length: 6211 I'm going to try to respond to several issues in this message rather than responding individually to several messages from Bill, David, Frank, Casper, and Spencer. I am also making further comments below on the original case. *** Topic 1: The Epoch By definition, the "Epoch" on UNIX Systems is January 1, 1970 at 0:00:00 UCT. It is not a range and "post-UNIX-Epoch" times are any time after the Epoch. What this case is describing as post-Epoch times are times when the number of seconds since the Epoch overflows a 32-bit signed time_t. I will use the term "overflow time" to refer to this for the rest of this message. The overflow time is January 19, 2038 at 03:14:08 UCT. (There is also an overflow time at December 13, 1901 at 20:45:52 for negative time_t values, but I don't think that matters for this case. For purposes of this discussion, there is no overflow time for applications using 64-bit signed time_t values.) *** Topic 2: What happens when an application requests an overflow time? This case proposes that a 32-bit application will have the date reset to January 19, 2038 at 3:14:07 UCT and that a 64-bit application will see the true time stamp from the PCFS file. The Large File Summit (LFS) made a rule early on: Never have the OS lie to an application; give true data or give an error. We have pretty much adhered to this concept in PSARC. I do not believe this case has presented sufficient reason to break the rule now. Telling a lie to a 32-bit application while telling the truth to a 64-bit application is a recipe for data corruption. (However, see #3 below.) Even if it may be attractive now to violate the rule, the closer we get to the overflow date the more confusing it will become. If any of these 32-bit apps are still around when we cross the overflow date, it would be ludicrous. The case materials say that 32-bit kernels do not detect the overflow and give back a random number. I agree that this is a bug in the 32-bit kernel. I do not agree that the way to fix it is to unconditionally map over 90 years of FAT filesystem time stamps to a single point one second before the overflow time if and only if a 32-bit app asks the question. *** Topic 3: Should the NFSv2 and v3 tunable nfs_allow_preepoch_time be a mount option or a tunable and what do we do with 32-bit kernels on x86? The NFSv2 and v3 protocols say that valid times are from the POSIX Epoch up to February 7, 2106 at 06:28:16 UCT. The standards say that if you try to store February 7, 2105 at midnight in a 32-bit signed time_t with the POSIX Epoch as zero, you are required to get an EOVERFLOW. The intent is also that if you try to pass a year 1950 timestamp from a UFS file through NFSv2 or v3 that should also give you an EOVERFLOW. Since we have lots of UFS filesystems that are accessed through NFS, we need a way to keep these two errors (which usually cancel each other out by the time an application sees the timestamp) from making UFS and/or NFS unusable. I agree with Frank that this probably should have been a mount option rather than a system tunable, but the important point to me is that it can be disabled when it is appropriate to do so. I have absolutely no objection to adding a PCFS mount option to clamp timestamps as suggested by this case on 32-bit kernels. On 32-bit kernels it is clearly necessary sometimes. (At least all applications on the machine will be getting the same l\bi\be\b"translation of the truth".) I object to having any kernel give different applications running on the same machine different timestamps for a given file and reporting success to all of the applications involved. I would be relatively happy with the same PCFS mount option on 64-bit kernels as suggested above for 32-bit kernels as long as the mount option causes the same translation of timestamps to be given to all applications accessing any particular file. *** Topic 4: We should provide a way for 32-bit applications to process extended time ranges. This has been discussed ad nausea before. The LFS discussed it and decided very early in the process that 64-bit apps were coming and a way to handle large files was needed immediately (especially by the database ISVs). The Wyoming project decided a few months later that the wave of the future is 64-bit apps and explicitly decided not to provide any means for a 32-bit application to process overflow times (or large dev_t's, etc.) other than to convert to a 64-bit programming model. If this project team disagrees with that position, I believe they need to suggest a project to change that; but that is independent of this case and should not be used as a reason for the system to lie to a subset of applications about time stamps. For this reason, I also object to having the system convert pre-PCFS-Epoch time stamps to the PCFS-Epoch rather than report an EOVERFLOW error if someone uses utimes() to attempt to set the timestamp on a PCFS file to a time before 1980. I can live with a mount option that does this, but it shouldn't be the default. *** Topic 5: Should all of our file utilities be converted to be 64-bit apps? No, or more correctly, not yet (see footnote 1). As Cindy was quoted as saying, it would be too big of a performance hit. But, with PCFS here and ZFS almost here it seems perfectly reasonable to me to expect that a few utilities should be available (perhaps with different names or in a different bin directory) in 64-bit processing form. I believe we should at least provide 64-bit equivalents for ls, rm, and touch. If someone believes that there are lots of PCs and/or cameras out there putting the wrong time stamps on files, I would also welcome an RFE to provide a utility (perhaps named touchback) that would reset timestamps on a set of files forwards or backwards by a given time offset (in years, days, hours, minutes, and seconds) rather than to a single specified time the way touch does. But, that is not this project. *** *** Footnote 1: It will be crucial that we do convert all file handling utilities to be 64-bit apps at least seven years prior to overflow time (given our current EOL and support policies), but we do not need to do so yet. Don From sacadmin Wed Jun 8 12:12:05 2005 Received: from eastmail1bur.East.Sun.COM (eastmail1bur.East.Sun.COM [129.148.9.49]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58JC4s0025524 for ; Wed, 8 Jun 2005 12:12:05 -0700 (PDT) Received: from thunk.east.sun.com (thunk.East.Sun.COM [129.148.174.66]) by eastmail1bur.East.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id j58JBbFt019247; Wed, 8 Jun 2005 15:11:37 -0400 (EDT) Received: from 127.0.0.1 (localhost [127.0.0.1]) by thunk.east.sun.com (8.13.3+Sun/8.13.3) with ESMTP id j58JBajX004746; Wed, 8 Jun 2005 15:11:36 -0400 (EDT) Subject: Re: 2005/361 PCFS timestamp handling cleanup From: Bill Sommerfeld To: Don Cragun Cc: Frank.Hofmann@sun.com, psarc@sac.sfbay.sun.com, Brian.Utterback@sun.com In-Reply-To: <200506081854.j58Is7Qi022110@spartan.SFBay.Sun.COM> References: <200506081854.j58Is7Qi022110@spartan.SFBay.Sun.COM> Content-Type: text/plain; charset=iso-8859-1 Message-Id: <1118257896.4073.5.camel@thunk> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6.309 Date: Wed, 08 Jun 2005 15:11:36 -0400 Content-Transfer-Encoding: 7bit Status: RO Content-Length: 524 On Wed, 2005-06-08 at 14:54, Don Cragun wrote: > The Wyoming project decided a few months later that > the wave of the future is 64-bit apps and explicitly decided not to > provide any means for a 32-bit application to process overflow times > (or large dev_t's, etc.) other than to convert to a 64-bit programming > model. If this project team disagrees with that position, I believe > they need to suggest a project to change that. For the record, I believe this position is dangerously short-sighted. - Bill From sacadmin Wed Jun 8 12:17:34 2005 Received: from sunnl.Holland.Sun.COM (sunnl.Holland.Sun.COM [129.159.201.1]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58JHXs0025852 for ; Wed, 8 Jun 2005 12:17:33 -0700 (PDT) Received: from vaticaan.Holland.Sun.COM (vaticaan [129.159.201.10]) by sunnl.Holland.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.3beta1412) with ESMTP id j58JGqGg028424; Wed, 8 Jun 2005 21:16:53 +0200 (MEST) Received: from holland (casper@room101 [129.159.201.52]) by vaticaan.Holland.Sun.COM (8.12.10+Sun/8.12.9) with ESMTP id j58JGqDk016765; Wed, 8 Jun 2005 21:16:52 +0200 (MEST) Message-Id: <200506081916.j58JGqDk016765@vaticaan.Holland.Sun.COM> From: Casper.Dik@sun.com To: Bill Sommerfeld cc: Don Cragun , Frank.Hofmann@sun.com, psarc@sac.sfbay.sun.com, Brian.Utterback@sun.com Subject: Re: 2005/361 PCFS timestamp handling cleanup In-Reply-To: <1118257896.4073.5.camel@thunk> References: <200506081854.j58Is7Qi022110@spartan.SFBay.Sun.COM> <1118257896.4073.5.camel@thunk> Date: Wed, 08 Jun 2005 21:16:52 +0200 Sender: casper@holland.sun.com Status: RO Content-Length: 734 >On Wed, 2005-06-08 at 14:54, Don Cragun wrote: >> The Wyoming project decided a few months later that >> the wave of the future is 64-bit apps and explicitly decided not to >> provide any means for a 32-bit application to process overflow times >> (or large dev_t's, etc.) other than to convert to a 64-bit programming >> model. If this project team disagrees with that position, I believe >> they need to suggest a project to change that. > >For the record, I believe this position is dangerously short-sighted. > > - Bill Which position? The Wyoming one or the other one? "We can only open 256 files with stdio" is another one. I strongly believe we should rev the object file format and go for 32 bit ABI II. Casper From sacadmin Wed Jun 8 12:26:14 2005 Received: from eastmail2bur.East.Sun.COM (eastmail2bur.East.Sun.COM [129.148.13.40]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58JQEs0026047 for ; Wed, 8 Jun 2005 12:26:14 -0700 (PDT) Received: from thunk.east.sun.com (thunk.East.Sun.COM [129.148.174.66]) by eastmail2bur.East.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id j58JPeC0001462; Wed, 8 Jun 2005 15:25:40 -0400 (EDT) Received: from 127.0.0.1 (localhost [127.0.0.1]) by thunk.east.sun.com (8.13.3+Sun/8.13.3) with ESMTP id j58JPdOS004867; Wed, 8 Jun 2005 15:25:39 -0400 (EDT) Subject: Re: 2005/361 PCFS timestamp handling cleanup From: Bill Sommerfeld To: Casper.Dik@sun.com Cc: Don Cragun , Frank.Hofmann@sun.com, psarc@sac.sfbay.sun.com, Brian.Utterback@sun.com In-Reply-To: <200506081916.j58JGqDk016765@vaticaan.Holland.Sun.COM> References: <200506081854.j58Is7Qi022110@spartan.SFBay.Sun.COM> <1118257896.4073.5.camel@thunk> <200506081916.j58JGqDk016765@vaticaan.Holland.Sun.COM> Content-Type: text/plain Message-Id: <1118258739.4073.11.camel@thunk> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6.309 Date: Wed, 08 Jun 2005 15:25:39 -0400 Content-Transfer-Encoding: 7bit Status: RO Content-Length: 1244 On Wed, 2005-06-08 at 15:16, Casper.Dik@sun.com wrote: > >On Wed, 2005-06-08 at 14:54, Don Cragun wrote: > >> The Wyoming project decided a few months later that > >> the wave of the future is 64-bit apps and explicitly decided not to > >> provide any means for a 32-bit application to process overflow times > >> (or large dev_t's, etc.) other than to convert to a 64-bit programming > >> model. If this project team disagrees with that position, I believe > >> they need to suggest a project to change that. > > > >For the record, I believe this position is dangerously short-sighted. > > > > - Bill > > Which position? The Wyoming one or the other one? Wyoming. (failed to correctly edit down quoted text). > "We can only open 256 files with stdio" is another one. Yup. > I strongly believe we should rev the object file format and go > for 32 bit ABI II. And start a multi-decade transition process for 32-bit ABI I. In the meantime, given that the is a need to interoperate with consumer devices which get timestamps horribly wrong in pcfs on a routine basis, this case seems to be approximately the right short-term answer. "It won't let me copy my pictures off the camera" is always the wrong answer. - Bill From sacadmin Wed Jun 8 12:29:23 2005 Received: from sunnl.Holland.Sun.COM (sunnl.Holland.Sun.COM [129.159.201.1]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58JTMs0026066 for ; Wed, 8 Jun 2005 12:29:22 -0700 (PDT) Received: from vaticaan.Holland.Sun.COM (vaticaan [129.159.201.10]) by sunnl.Holland.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.3beta1412) with ESMTP id j58JSsGg004027; Wed, 8 Jun 2005 21:28:54 +0200 (MEST) Received: from holland (casper@room101 [129.159.201.52]) by vaticaan.Holland.Sun.COM (8.12.10+Sun/8.12.9) with ESMTP id j58JSrDk024348; Wed, 8 Jun 2005 21:28:53 +0200 (MEST) Message-Id: <200506081928.j58JSrDk024348@vaticaan.Holland.Sun.COM> From: Casper.Dik@sun.com To: Bill Sommerfeld cc: Don Cragun , Frank.Hofmann@sun.com, psarc@sac.sfbay.sun.com, Brian.Utterback@sun.com Subject: Re: 2005/361 PCFS timestamp handling cleanup In-Reply-To: <1118258739.4073.11.camel@thunk> References: <200506081854.j58Is7Qi022110@spartan.SFBay.Sun.COM> <1118257896.4073.5.camel@thunk> <200506081916.j58JGqDk016765@vaticaan.Holland.Sun.COM> <1118258739.4073.11.camel@thunk> Date: Wed, 08 Jun 2005 21:28:53 +0200 Sender: casper@holland.sun.com Status: RO Content-Length: 733 >> I strongly believe we should rev the object file format and go >> for 32 bit ABI II. > >And start a multi-decade transition process for 32-bit ABI I. Not necessarily; but 3 copies of all libraries is a bit taxing. (My initial thought about this was to have three types of objects: ABI32-I ABI32-II ABI32-neutral (not using any of the extended types) where we could try to make as much stuff ABI32-neutral. >In the meantime, given that the is a need to interoperate with consumer >devices which get timestamps horribly wrong in pcfs on a routine basis, >this case seems to be approximately the right short-term answer. > >"It won't let me copy my pictures off the camera" is always the wrong >answer. Absolutely. Casper From sacadmin Wed Jun 8 12:57:40 2005 Received: from spartan.SFBay.Sun.COM (spartan [129.146.226.64]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58Jves0029518 for ; Wed, 8 Jun 2005 12:57:40 -0700 (PDT) Received: from spartan.SFBay.Sun.COM (spartan.SFBay.Sun.COM [129.146.226.64]) by spartan.SFBay.Sun.COM (8.12.10+Sun/8.12.10) with SMTP id j58JvEQi022232; Wed, 8 Jun 2005 12:57:14 -0700 (PDT) Message-Id: <200506081957.j58JvEQi022232@spartan.SFBay.Sun.COM> Date: Wed, 8 Jun 2005 12:57:14 -0700 (PDT) From: Don Cragun Reply-To: Don Cragun Subject: Re: 2005/361 PCFS timestamp handling cleanup To: Casper.Dik@Sun.COM Cc: Frank.Hofmann@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: sEvLLn5K7tFtLpqenTC45w== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.5.5 SunOS 5.9 sun4u sparc Status: RO Content-Length: 1078 >Date: Wed, 08 Jun 2005 21:28:53 +0200 >From: Casper.Dik@sun.com > >(My initial thought about this was to have three types of >objects: > > ABI32-I > ABI32-II > ABI32-neutral (not using any of the extended types) > >where we could try to make as much stuff ABI32-neutral. The number of applications running in the ABI32-neutral pot is probably zero. By definition, applications have access to stdin, stdout, and stderr when they enter main() and you're implying a different FILE struct between ABI32-I and ABI32-II... > >>In the meantime, given that the is a need to interoperate with consumer >>devices which get timestamps horribly wrong in pcfs on a routine basis, >>this case seems to be approximately the right short-term answer. >> >>"It won't let me copy my pictures off the camera" is always the wrong >>answer. > >Absolutely. And if you have your source on a PC with horribly wrong timestamps and make says everything is always up-to-date that is also always the wrong answer. Unfortunately, you can't fix both of these at these problems at the same time. - Don From sacadmin Wed Jun 8 13:01:41 2005 Received: from sunnl.Holland.Sun.COM (sunnl.Holland.Sun.COM [129.159.201.1]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58K1es0029966 for ; Wed, 8 Jun 2005 13:01:40 -0700 (PDT) Received: from vaticaan.Holland.Sun.COM (vaticaan [129.159.201.10]) by sunnl.Holland.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.3beta1412) with ESMTP id j58K1BGg012458; Wed, 8 Jun 2005 22:01:11 +0200 (MEST) Received: from holland (casper@room101 [129.159.201.52]) by vaticaan.Holland.Sun.COM (8.12.10+Sun/8.12.9) with ESMTP id j58K1BDk014819; Wed, 8 Jun 2005 22:01:11 +0200 (MEST) Message-Id: <200506082001.j58K1BDk014819@vaticaan.Holland.Sun.COM> From: Casper.Dik@Sun.COM To: Don Cragun cc: Frank.Hofmann@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Subject: Re: 2005/361 PCFS timestamp handling cleanup In-Reply-To: <200506081957.j58JvEQi022232@spartan.SFBay.Sun.COM> References: <200506081957.j58JvEQi022232@spartan.SFBay.Sun.COM> Date: Wed, 08 Jun 2005 22:01:11 +0200 Sender: casper@holland.sun.com Status: RO Content-Length: 479 >The number of applications running in the ABI32-neutral pot is probably >zero. By definition, applications have access to stdin, stdout, and >stderr when they enter main() and you're implying a different FILE >struct between ABI32-I and ABI32-II... That could still be made to work as long as they use some functional; as long the interfaces are used with pointers onlu. So something using: printf() fprintf(stderr could be made to work (hm, perhaps not stderr) Casper From sacadmin Wed Jun 8 13:06:45 2005 Received: from phorcys.East.Sun.COM (phorcys.East.Sun.COM [129.148.174.143]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58K6is0000066 for ; Wed, 8 Jun 2005 13:06:45 -0700 (PDT) Received: from phorcys.East.Sun.COM (localhost [127.0.0.1]) by phorcys.East.Sun.COM (8.13.4+Sun/8.13.4) with ESMTP id j58K6Jcw013479; Wed, 8 Jun 2005 16:06:19 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.East.Sun.COM (8.13.4+Sun/8.13.4/Submit) id j58K6JRB013476; Wed, 8 Jun 2005 16:06:19 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17063.20410.694930.784334@gargle.gargle.HOWL> Date: Wed, 8 Jun 2005 16:06:18 -0400 From: James Carlson To: Don Cragun Cc: Casper.Dik@Sun.COM, Frank.Hofmann@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Subject: Re: 2005/361 PCFS timestamp handling cleanup In-Reply-To: Don Cragun's message of 8 June 2005 12:57:14 References: <200506081957.j58JvEQi022232@spartan.SFBay.Sun.COM> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 926 Don Cragun writes: > >Absolutely. > > And if you have your source on a PC with horribly wrong timestamps and > make says everything is always up-to-date that is also always the wrong > answer. Unfortunately, you can't fix both of these at these problems > at the same time. Running 'make' on pcfs ... is that really a useful scenario? And how does breaking access to these files help? What happens if 'make' reports that it can't stat anything? That hardly seems better than silent failure. If we must make pcfs fail like this -- and I'm not sure I agree that we should do that; the original proposal seemed decent enough -- then can we at least make it a non-default mount option? "-o hurtme" -- James Carlson, KISS Network Sun Microsystems / 1 Network Drive 71.234W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.497N Fax +1 781 442 1677 From sacadmin Wed Jun 8 13:14:22 2005 Received: from eastmail1bur.East.Sun.COM (eastmail1bur.East.Sun.COM [129.148.9.49]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58KELs0001027 for ; Wed, 8 Jun 2005 13:14:22 -0700 (PDT) Received: from thunk.east.sun.com (thunk.East.Sun.COM [129.148.174.66]) by eastmail1bur.East.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id j58KDnFt008023; Wed, 8 Jun 2005 16:13:49 -0400 (EDT) Received: from 127.0.0.1 (localhost [127.0.0.1]) by thunk.east.sun.com (8.13.3+Sun/8.13.3) with ESMTP id j58KDmPs005030; Wed, 8 Jun 2005 16:13:49 -0400 (EDT) Subject: Re: 2005/361 PCFS timestamp handling cleanup From: Bill Sommerfeld To: Don Cragun Cc: Casper.Dik@sun.com, Frank.Hofmann@sun.com, psarc@sac.sfbay.sun.com, Brian.Utterback@sun.com In-Reply-To: <200506081957.j58JvEQi022232@spartan.SFBay.Sun.COM> References: <200506081957.j58JvEQi022232@spartan.SFBay.Sun.COM> Content-Type: text/plain; charset=iso-8859-1 Message-Id: <1118261628.4073.15.camel@thunk> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6.309 Date: Wed, 08 Jun 2005 16:13:48 -0400 Content-Transfer-Encoding: 7bit Status: RO Content-Length: 456 On Wed, 2005-06-08 at 15:57, Don Cragun wrote: > And if you have your source on a PC with horribly wrong timestamps and > make says everything is always up-to-date that is also always the wrong > answer. Unfortunately, you can't fix both of these at these problems > at the same time. if we were to change pcfs in the way you propose, you'll go from requiring a "make clean" to get a correct build to not being able to see the source code at all. From sacadmin Wed Jun 8 13:26:06 2005 Received: from jurassic.eng.sun.com (jurassic [129.146.106.105] (may be forged)) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58KQ6s0002052 for ; Wed, 8 Jun 2005 13:26:06 -0700 (PDT) Received: from 129.146.108.198 (braveheart.SFBay.Sun.COM [129.146.108.198]) by jurassic.eng.sun.com (8.13.4+Sun/8.13.4) with ESMTP id j58KPfex377809; Wed, 8 Jun 2005 13:25:41 -0700 (PDT) Subject: Re: 2005/361 PCFS timestamp handling cleanup From: Darren J Moffat To: Don Cragun Cc: frank.hofmann@Sun.COM, psarc@sac.sfbay.sun.com, brian.utterback@Sun.COM In-Reply-To: <200506081854.j58Is7Qi022110@spartan.SFBay.Sun.COM> References: <200506081854.j58Is7Qi022110@spartan.SFBay.Sun.COM> Content-Type: text/plain Organization: Sun Microsystems, Inc. Message-Id: <1118262317.3416.2.camel@braveheart> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6.316 Date: Wed, 08 Jun 2005 13:25:17 -0700 Content-Transfer-Encoding: 7bit Status: RO Content-Length: 616 On Wed, 2005-06-08 at 11:54, Don Cragun wrote: > *** Topic 5: Should all of our file utilities be converted to be 64-bit > apps? > No, or more correctly, not yet (see footnote 1). As Cindy was quoted > as saying, it would be too big of a performance hit. But, with PCFS > here and ZFS almost here it seems perfectly reasonable to me to expect > that a few utilities should be available (perhaps with different names > or in a different bin directory) in 64-bit processing form. I believe > we should at least provide 64-bit equivalents for ls, rm, and touch. /usr/bin/64/ls already exists. -- Darren J Moffat From sacadmin Wed Jun 8 14:05:17 2005 Received: from spartan.SFBay.Sun.COM (spartan [129.146.226.64]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58L5Hs0006485 for ; Wed, 8 Jun 2005 14:05:17 -0700 (PDT) Received: from spartan.SFBay.Sun.COM (spartan.SFBay.Sun.COM [129.146.226.64]) by spartan.SFBay.Sun.COM (8.12.10+Sun/8.12.10) with SMTP id j58L4qQi022349; Wed, 8 Jun 2005 14:04:52 -0700 (PDT) Message-Id: <200506082104.j58L4qQi022349@spartan.SFBay.Sun.COM> Date: Wed, 8 Jun 2005 14:04:52 -0700 (PDT) From: Don Cragun Reply-To: Don Cragun Subject: Re: 2005/361 PCFS timestamp handling cleanup To: Don.Cragun@Sun.COM, james.d.carlson@Sun.COM Cc: Casper.Dik@Sun.COM, Frank.Hofmann@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: WhYTaCCNzOI5hOmDfDmk7w== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.5.5 SunOS 5.9 sun4u sparc Status: RO Content-Length: 1973 >Date: Wed, 8 Jun 2005 16:06:18 -0400 >From: James Carlson > >Don Cragun writes: >> >Absolutely. >> >> And if you have your source on a PC with horribly wrong timestamps and >> make says everything is always up-to-date that is also always the wrong >> answer. Unfortunately, you can't fix both of these at these problems >> at the same time. > >Running 'make' on pcfs ... is that really a useful scenario? Maybe, maybe not. It is an example the compressing 90 years worth of timestamps to a single second seems wrong. > >And how does breaking access to these files help? What happens if >'make' reports that it can't stat anything? That hardly seems better >than silent failure. Having make say that it can't stat files sounds better to me than saying that all files are up-to-date when none of the binaries are rebuilt after the source changes. This isn't just silent failure; it can be silent corruption. > >If we must make pcfs fail like this -- and I'm not sure I agree that >we should do that; the original proposal seemed decent enough -- then >can we at least make it a non-default mount option? "-o hurtme" For 32-bit systems, I completely agree that the mount option needs to map (or compress) timestamps by default. For 64-bit systems, I believe the option name should be the same as for 32-bit systems; I'm just not sure that it should have the same default setting. As more and more apps are built in 64-bit mode, compressing timestamps becomes less and less desirable. The part I really object to is telling application foo32 that the timestamp on a file is 1/19/2038@03:14:07 while telling application foo64 that the same timestamp on the same file is a different date somewhere in the range from 1/19/2038@03:14:08 through 12/31/2127@23:59:59. This is a disaster waiting to happen. Always doing the mapping (even when the person who mounts the PCFS filesystem knows the timestamps are correct) is perverse. - Don From sacadmin Wed Jun 8 15:03:24 2005 Received: from jurassic.eng.sun.com (jurassic [129.146.108.31] (may be forged)) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58M3Os0019324 for ; Wed, 8 Jun 2005 15:03:24 -0700 (PDT) Received: from jurassic.eng.sun.com (localhost [127.0.0.1]) by jurassic.eng.sun.com (8.13.4+Sun/8.13.4) with ESMTP id j58M2xJP663480 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Wed, 8 Jun 2005 15:02:59 -0700 (PDT) Received: (from tpm@localhost) by jurassic.eng.sun.com (8.13.4+Sun/8.13.4/Submit) id j58M2xdx663467; Wed, 8 Jun 2005 15:02:59 -0700 (PDT) Date: Wed, 8 Jun 2005 15:02:58 -0700 From: Tim Marsland To: Casper.Dik@sun.com Cc: Bill Sommerfeld , Don Cragun , Frank.Hofmann@sun.com, psarc@sac.sfbay.sun.com, Brian.Utterback@sun.com Subject: Re: 2005/361 PCFS timestamp handling cleanup Message-ID: <20050608220258.GJ836756@eng.sun.com> Reply-To: Tim Marsland References: <200506081854.j58Is7Qi022110@spartan.SFBay.Sun.COM> <1118257896.4073.5.camel@thunk> <200506081916.j58JGqDk016765@vaticaan.Holland.Sun.COM> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200506081916.j58JGqDk016765@vaticaan.Holland.Sun.COM> User-Agent: Mutt/1.4.2.1i Organization: Operating Platforms Group; Sun Microsystems, Inc. Status: RO Content-Length: 1910 * Casper.Dik@sun.com [2005-06-08 12:18]: > > >On Wed, 2005-06-08 at 14:54, Don Cragun wrote: > >> The Wyoming project decided a few months later that > >> the wave of the future is 64-bit apps and explicitly decided not to > >> provide any means for a 32-bit application to process overflow times > >> (or large dev_t's, etc.) other than to convert to a 64-bit programming > >> model. That was then, and it was entirely a statement of what -that- project would and wouldn't do given the time/resources available, and the market (1998) context the project found itself in, and not intended as some edict forever etched in stone. Sheesh. I would also note that this is the Linux position more generally, so I'm intrigued to hear where this position is actually a disadvantage to us (beyond the 64-bit time computation noted below) > >> If this project team disagrees with that position, I believe > >> they need to suggest a project to change that. > > > >For the record, I believe this position is dangerously short-sighted. I believe we should look to the response provided by our competitors. I think AIX/HP-UX offered APIs that allow 64-bit time computations (the canonical example being mortgage calculations that often go beyond y2038) to be done in a 32-bit application. However, I find it hard to believe they have offered two versions of every API that contains a time_t .. > Which position? The Wyoming one or the other one? > > "We can only open 256 files with stdio" is another one. > > I strongly believe we should rev the object file format and go > for 32 bit ABI II. Do we have a full list of things we would fix in that environment, if we had it? My list includes requiring the SSE registers for floating point (and the v9 FP definition too), as well as FILE, dev_t/off_t/time_t. It certainly needs to be a compelling list to undertake such an effort. tim From sacadmin Wed Jun 8 15:22:33 2005 Received: from spartan.SFBay.Sun.COM (spartan [129.146.226.64]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58MMXs0019874 for ; Wed, 8 Jun 2005 15:22:33 -0700 (PDT) Received: from spartan.SFBay.Sun.COM (spartan.SFBay.Sun.COM [129.146.226.64]) by spartan.SFBay.Sun.COM (8.12.10+Sun/8.12.10) with SMTP id j58MM7Qi022539; Wed, 8 Jun 2005 15:22:07 -0700 (PDT) Message-Id: <200506082222.j58MM7Qi022539@spartan.SFBay.Sun.COM> Date: Wed, 8 Jun 2005 15:22:07 -0700 (PDT) From: Don Cragun Reply-To: Don Cragun Subject: Re: 2005/361 PCFS timestamp handling cleanup To: sommerfeld@Sun.COM Cc: Frank.Hofmann@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: OttipQElAF0rgAQePckg+A== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.5.5 SunOS 5.9 sun4u sparc Status: RO Content-Length: 2739 >From: Bill Sommerfeld >Date: Wed, 08 Jun 2005 16:13:48 -0400 > >On Wed, 2005-06-08 at 15:57, Don Cragun wrote: >> And if you have your source on a PC with horribly wrong timestamps and >> make says everything is always up-to-date that is also always the wrong >> answer. Unfortunately, you can't fix both of these at these problems >> at the same time. > >if we were to change pcfs in the way you propose, you'll go from >requiring a "make clean" to get a correct build to not being able to see >the source code at all. A make clean doesn't correct the problem, it shifts the problem. After you do the make clean, every subsequent make will be a full build. The problem is that all of the source files have dates set more than 30 years in the future. If you are going to use dates on files for anything (make, backup plans, determining the order in which files were changed), you need correct dates. Mapping 90+ years worth of timestamps for use by 32-bit applications to a single second that is now more than 32 years in the future while 64-bit applications see a different timestamp doesn't solve the problem; it just shifts the problem. As Darren pointed out, we already have /usr/bin/64/ls (or /usr/bin/sparcv9/ls depending on your hardware). I already suggested that we should add 64-bit versions of touch to reset dates on filesystems imported from systems that don't have a clock set anywhere close to "real" world time. Are you proposing that ZFS also ALWAYS map out-of-range timestamps for files to one of the overflow times when 32-bit applications are making *stat() calls? Is it really impossible for everyone else to imagine that a PCFS filesystem could actually intend to have file timestamps beyond 2037 and that the creator of those files didn't want the timestamps to be translated to another arbitrary point in time when accessed by some, but not all, applications? Do we have any reason to believe that ALL PCFS filesystems come from systems with no concept of correct time? The fact that PCFS has a 1980 (Windows) Epoch and UFS and ZFS have a 1970 (UNIX) Epoch is not a bug and is not the issue here. The issue is how you handle conversion of file timestamps from any filesystem to a signed time_t on a Solaris system for a 32-bit application. On UFS it is not an issue (file timestamps have the same Epoch and range as a 32-bit application's signed 32-bit time_t; the value is sign extended for use as a 64-bit application's signed 64-bit time_t). What is being done here as a fast-track for PCFS may well set precedent for ZFS or any other filesystem whose timestamp range, signedness, or Epoch is different from UFS. I believe this is an extremely dangerous precedent. - Don From sacadmin Wed Jun 8 15:35:23 2005 Received: from eastmail1bur.East.Sun.COM (eastmail1bur.East.Sun.COM [129.148.9.49]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j58MZNs0020219 for ; Wed, 8 Jun 2005 15:35:23 -0700 (PDT) Received: from thunk.east.sun.com (thunk.East.Sun.COM [129.148.174.66]) by eastmail1bur.East.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id j58MYoFt016043; Wed, 8 Jun 2005 18:34:50 -0400 (EDT) Received: from 127.0.0.1 (localhost [127.0.0.1]) by thunk.east.sun.com (8.13.3+Sun/8.13.3) with ESMTP id j58MYorW005853; Wed, 8 Jun 2005 18:34:50 -0400 (EDT) Subject: Re: 2005/361 PCFS timestamp handling cleanup From: Bill Sommerfeld To: Don Cragun Cc: James.D.Carlson@sun.com, Casper.Dik@sun.com, Frank.Hofmann@sun.com, psarc@sac.sfbay.sun.com, Brian.Utterback@sun.com In-Reply-To: <200506082104.j58L4qQi022349@spartan.SFBay.Sun.COM> References: <200506082104.j58L4qQi022349@spartan.SFBay.Sun.COM> Content-Type: text/plain Message-Id: <1118270089.4073.107.camel@thunk> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6.309 Date: Wed, 08 Jun 2005 18:34:50 -0400 Content-Transfer-Encoding: 7bit Status: RO Content-Length: 721 On Wed, 2005-06-08 at 17:04, Don Cragun wrote: > Having make say that it can't stat files sounds better to me than > saying that all files are up-to-date when none of the binaries are > rebuilt after the source changes. This isn't just silent failure; it > can be silent corruption. Failure to do an incremental build correctly is a common, expected, and grudgingly tolerated behavior endemic to make which can be triggered by any number of causes besides timestamp compression. You can work around the problem with a fresh checkout or a make clean/make clobber. By comparison, system behavior which effectively denied access to the source files would be substantially more likely to induce violence. - Bill From sacadmin Wed Jun 8 20:37:42 2005 Received: from phorcys.East.Sun.COM (phorcys.East.Sun.COM [129.148.174.143]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j593bfs0007902 for ; Wed, 8 Jun 2005 20:37:42 -0700 (PDT) Received: from phorcys.East.Sun.COM (localhost [127.0.0.1]) by phorcys.East.Sun.COM (8.13.4+Sun/8.13.4) with ESMTP id j593bGEk000417; Wed, 8 Jun 2005 23:37:16 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.East.Sun.COM (8.13.4+Sun/8.13.4/Submit) id j593bGQY000414; Wed, 8 Jun 2005 23:37:16 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17063.47468.445694.286040@gargle.gargle.HOWL> Date: Wed, 8 Jun 2005 23:37:16 -0400 From: James Carlson To: Don Cragun Cc: sommerfeld@Sun.COM, Frank.Hofmann@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Subject: Re: 2005/361 PCFS timestamp handling cleanup In-Reply-To: Don Cragun's message of 8 June 2005 15:22:07 References: <200506082222.j58MM7Qi022539@spartan.SFBay.Sun.COM> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 3029 Don Cragun writes: > >if we were to change pcfs in the way you propose, you'll go from > >requiring a "make clean" to get a correct build to not being able to see > >the source code at all. > > A make clean doesn't correct the problem, it shifts the problem. After > you do the make clean, every subsequent make will be a full build. Suboptimal, to be sure. But far better than being completely unable to read your data at all. As I remember hearing someone say, customers tend to form unnatural attachments to their data. > As Darren pointed out, we already have /usr/bin/64/ls (or > /usr/bin/sparcv9/ls depending on your hardware). I already suggested > that we should add 64-bit versions of touch to reset dates on > filesystems imported from systems that don't have a clock set anywhere > close to "real" world time. That doesn't work unless the 32-bit world suddenly goes away. It's unfortunately not going away any time soon. > Are you proposing that ZFS also ALWAYS map out-of-range timestamps > for files to one of the overflow times when 32-bit applications are > making *stat() calls? No, of course not. That's a different situation. We're talking about PCFS here, and the ways in which multiple implementations (and perhaps some ancient lack of foresight) have hemmed us in. > Is it really impossible for everyone else to imagine that a PCFS > filesystem could actually intend to have file timestamps beyond 2037 > and that the creator of those files didn't want the timestamps to be > translated to another arbitrary point in time when accessed by some, > but not all, applications? Do we have any reason to believe that ALL > PCFS filesystems come from systems with no concept of correct time? So ... this hypothetical user would be displeased if the timestamp showed up as some arbitrary value, but would end up being relatively happy if stat() returned -1, and he just couldn't read his files at all. Why is that? > What is being done here as a fast-track for PCFS may well set precedent > for ZFS or any other filesystem whose timestamp range, signedness, or > Epoch is different from UFS. I believe this is an extremely dangerous > precedent. I see it very differently. I see it as a regrettable kludge that makes the best of a bad situation. Existing systems produce PCFS files with bad timestamps that happen also to be beyond the UNIX epoch. Existing Solaris interfaces can't deal with this. We're thus stuck with denying access to the user's data (very bad), forcing the user to recompile and switch to an interface we haven't invented yet (extremely bad), or lying to the user by clamping the timestamp to an arbitrary but predictable value (bad). I fail to see why the integrity of the file timestamp is more important than the contents of the file itself. -- James Carlson, KISS Network Sun Microsystems / 1 Network Drive 71.234W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.497N Fax +1 781 442 1677 From sacadmin Thu Jun 9 00:44:56 2005 Received: from phys-gadget-1 (phys-gadget-1.UK.Sun.COM [129.156.85.171]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j597its0020294 for ; Thu, 9 Jun 2005 00:44:56 -0700 (PDT) Received: from conversion-daemon.gadget-mail1.uk.sun.com by gadget-mail1.uk.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IHT0010140IOV@gadget-mail1.uk.sun.com> (original mail from steve.mckinty@sun.com) for psarc@sac.sfbay.sun.com; Thu, 09 Jun 2005 08:44:30 +0100 (BST) Received: from [129.157.192.78] (hardy.France.Sun.COM [129.157.192.78]) by gadget-mail1.uk.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with ESMTPA id <0IHT00F0T465V9@gadget-mail1.uk.sun.com>; Thu, 09 Jun 2005 08:44:30 +0100 (BST) Date: Thu, 09 Jun 2005 09:44:29 +0200 From: Steve McKinty Subject: Re: 2005/361 PCFS timestamp handling cleanup In-reply-to: <200506082222.j58MM7Qi022539@spartan.SFBay.Sun.COM> To: Don Cragun Cc: sommerfeld@sun.com, Frank.Hofmann@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Message-id: <42A7F35D.8070507@sun.com> Organization: Sun Microsystems MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-Accept-Language: en-us, en User-Agent: Mozilla Thunderbird 1.0 (X11/20041208) References: <200506082222.j58MM7Qi022539@spartan.SFBay.Sun.COM> Status: RO Content-Length: 4273 Don Cragun wrote: >>From: Bill Sommerfeld >>Date: Wed, 08 Jun 2005 16:13:48 -0400 >> >>On Wed, 2005-06-08 at 15:57, Don Cragun wrote: >> >>>And if you have your source on a PC with horribly wrong timestamps and >>>make says everything is always up-to-date that is also always the wrong >>>answer. Unfortunately, you can't fix both of these at these problems >>>at the same time. >> >>if we were to change pcfs in the way you propose, you'll go from >>requiring a "make clean" to get a correct build to not being able to see >>the source code at all. > > > A make clean doesn't correct the problem, it shifts the problem. After > you do the make clean, every subsequent make will be a full build. The > problem is that all of the source files have dates set more than 30 > years in the future. If you are going to use dates on files for > anything (make, backup plans, determining the order in which files were > changed), you need correct dates. Mapping 90+ years worth of > timestamps for use by 32-bit applications to a single second that is > now more than 32 years in the future while 64-bit applications see a > different timestamp doesn't solve the problem; it just shifts the > problem. If you are going to lie to applications, might it not be better to lie convincingly, and return a time_t of 0 (Unix Epoch) instead of INT32_MAX? At least that would mean that: - It is obviously wrong, no FAT filesystem can have that time. - It is consistently wrong, it won't seem "more plausible" as we approach 2038. - Even a semi-experienced Unix user will recognise a date of Jan 1970 as meaning "somethings's screwy here". - I think the 'make' issue is somewhat spurious (will people really run make on future-dated filesystems & expect reasonable results?) but at least a timestamp of zero will mean that, should a file be 'touched' or edited, make will behave as you might expect afterwards. - Granted, the files might be missed by backup utilities looking for data changed after a certain date, but in my experience such utilities get pretty upset with future-dated files as well. Just 0.02 from the gallery... Steve > > As Darren pointed out, we already have /usr/bin/64/ls (or > /usr/bin/sparcv9/ls depending on your hardware). I already suggested > that we should add 64-bit versions of touch to reset dates on > filesystems imported from systems that don't have a clock set anywhere > close to "real" world time. > > Are you proposing that ZFS also ALWAYS map out-of-range timestamps > for files to one of the overflow times when 32-bit applications are > making *stat() calls? > > Is it really impossible for everyone else to imagine that a PCFS > filesystem could actually intend to have file timestamps beyond 2037 > and that the creator of those files didn't want the timestamps to be > translated to another arbitrary point in time when accessed by some, > but not all, applications? Do we have any reason to believe that ALL > PCFS filesystems come from systems with no concept of correct time? > > The fact that PCFS has a 1980 (Windows) Epoch and UFS and ZFS have a > 1970 (UNIX) Epoch is not a bug and is not the issue here. The issue is > how you handle conversion of file timestamps from any filesystem to a > signed time_t on a Solaris system for a 32-bit application. On UFS it > is not an issue (file timestamps have the same Epoch and range as a > 32-bit application's signed 32-bit time_t; the value is sign extended > for use as a 64-bit application's signed 64-bit time_t). > > What is being done here as a fast-track for PCFS may well set precedent > for ZFS or any other filesystem whose timestamp range, signedness, or > Epoch is different from UFS. I believe this is an extremely dangerous > precedent. > > - Don > -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From sacadmin Thu Jun 9 00:51:16 2005 Received: from sunnl.Holland.Sun.COM (sunnl.Holland.Sun.COM [129.159.201.1]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j597pFs0020312 for ; Thu, 9 Jun 2005 00:51:15 -0700 (PDT) Received: from vaticaan.Holland.Sun.COM (vaticaan [129.159.201.10]) by sunnl.Holland.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.3beta1412) with ESMTP id j597oaGg005518; Thu, 9 Jun 2005 09:50:36 +0200 (MEST) Received: from holland (casper@room101 [129.159.201.52]) by vaticaan.Holland.Sun.COM (8.12.10+Sun/8.12.9) with ESMTP id j597oaDk006413; Thu, 9 Jun 2005 09:50:36 +0200 (MEST) Message-Id: <200506090750.j597oaDk006413@vaticaan.Holland.Sun.COM> From: Casper.Dik@Sun.COM To: Steve McKinty cc: Don Cragun , sommerfeld@Sun.COM, Frank.Hofmann@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Subject: Re: 2005/361 PCFS timestamp handling cleanup In-Reply-To: <42A7F35D.8070507@sun.com> References: <200506082222.j58MM7Qi022539@spartan.SFBay.Sun.COM> <42A7F35D.8070507@sun.com> Date: Thu, 09 Jun 2005 09:50:36 +0200 Sender: casper@holland.sun.com Status: RO Content-Length: 415 >If you are going to lie to applications, might it not be better to >lie convincingly, and return a time_t of 0 (Unix Epoch) instead >of INT32_MAX? There are several timestamps which are optional; it's those fields which are problematic, in most cases. Surely, when we find unreasonable timestamps we could try and use one of the other times? And if the time is far in the future, why not return "now"? Casper From sacadmin Thu Jun 9 04:30:51 2005 Received: from phys-mayi-1 (phys-mayi-1-ipmp1.Germany.Sun.COM [129.157.128.114]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j59BUos0000397 for ; Thu, 9 Jun 2005 04:30:51 -0700 (PDT) Received: from conversion-daemon.mayi-mail1.germany.sun.com by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IHT00401EM2IX@mayi-mail1.germany.sun.com> (original mail from frankho@mayi-mail1.germany.sun.com) for psarc@sac.SFBay.sun.com; Thu, 09 Jun 2005 13:30:24 +0200 (MEST) Received: from estale (estale.UK.Sun.COM [129.156.173.199]) by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with SMTP id <0IHT004N0EMNQG@mayi-mail1.germany.sun.com>; Thu, 09 Jun 2005 13:30:24 +0200 (MEST) Date: Thu, 09 Jun 2005 12:30:24 +0100 (BST) From: Frank Hofmann - Solaris Sustaining Subject: Re: 2005/361 PCFS timestamp handling cleanup To: Frank.Hofmann@Sun.COM, Don.Cragun@Sun.COM Cc: psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Reply-to: Frank Hofmann - Solaris Sustaining Message-id: <0IHT004N1EMOQG@mayi-mail1.germany.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_16 SunOS 5.11 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-MD5: IH5octgeGtncAO9rNyJ13g== Status: RO Content-Length: 4659 [ ... ] > *** Topic 3: Should the NFSv2 and v3 tunable nfs_allow_preepoch_time > be a mount option or a tunable and what do we do with 32-bit kernels > on x86? [ ... ] > out by the time an application sees the timestamp) from making UFS > and/or NFS unusable. I agree with Frank that this probably should > have been a mount option rather than a system tunable, but the That suggestion was made by Bill, not me. > important point to me is that it can be disabled when it is appropriate > to do so. > > I have absolutely no objection to adding a PCFS mount option to clamp > timestamps as suggested by this case on 32-bit kernels. On 32-bit > kernels it is clearly necessary sometimes. (At least all applications Why is this necessity 32bit-kernel specific ? I can perform the timestamp arithmetics in 64bit even on a 32bit kernel and return EOVERFLOW (or whatever error, see below ...) if the conversion to UNIX 32bit time_t exceeds the range of the data type. The *only* reason why it is necessary NOT to do that is to make things work. And that doesn't depend on whether I'm running a 32bit or a 64bit kernel. > on the machine will be getting the same l\bi\be\b"translation of the > truth".) I object to having any kernel give different applications > running on the same machine different timestamps for a given file and > reporting success to all of the applications involved. I would be > relatively happy with the same PCFS mount option on 64-bit kernels as > suggested above for 32-bit kernels as long as the mount option causes > the same translation of timestamps to be given to all applications > accessing any particular file. I agree into making this a mount option, or rather a pair of options "clamptime/noclamptime" that switches between the two behaviours: a) both 32bit and 64bit applications never see timestamps outside of the range "FAT epoch" (01/01/1980 00:00:00) and "end of 32bit time_t" (01/19/2038, 03:14:07). This means: * The attempt to set a timestamp to outside that range will cause an error return to the application - for both 32bit/64bit. This error CANNOT (!) be EOVERFLOW - see utime(2)/utimes(2), which don't include EOVERFLOW in the possible list of error codes ... * Finding a timestamp on the medium that is post-32bit-time_t will report 01/19/2038, 03:14:07, again for both 32bit/64bit. This will be the behaviour with the "clamptime" option. b) The full range of FAT time will be made accessible to applications. This means: * A 64bit application will see timestamps post-"end of 32bit time_t" as they are recorded on the medium, and will be able to set timestamps to any value within the range possible for FAT (01/01/1980 ... 31/12/2127). The attempt to set a time to outside that range will result in an error - see above, can unfortunately NOT be EOVERFLOW ... * A 32bit application will get EOVERFLOW on stat(2) if a timestamp on the medium is post-"end of 32bit time_t". It will get the same error as the 64bit app if it attempts to set a time outside of the possible range (applies only to pre-1980 times for 32bit apps, of course). This will be the behaviour with the "noclamptime" option. I do insist, though, that behaviour a) ("clamptime" or whatever) becomes the default behaviour. I do not agree with breaking things that work now (32bit app on 32bit kernel), even if they only work due to a (32bit-specific) bug. I'll rather propagate the bug to 64bit in order to let 32/64bit apps see the same, and WORK the same, indifferent of the underlaying kernel. I.e.: If somebody *WANTS* post-32bit-time_t timestamps on PCFS, then let them ASK for it ("-o noclamptime"), and mention clearly in the pcfs manpages that using this will make files with such timestamps inaccessible to 32bit applications. > > *** Topic 4: We should provide a way for 32-bit applications to process > extended time ranges. [ ... ] Beyond the scope of this case - I don't want to follow up on this here. > > *** Topic 5: Should all of our file utilities be converted to be 64-bit > apps? [ ... ] Again, beyond the scope of this case. > > *** *** Footnote 1: It will be crucial that we do convert all file > handling utilities to be 64-bit apps at least seven years prior to > overflow time (given our current EOL and support policies), but we do > not need to do so yet. Note on this: Once this happens, the default behaviour of PCFS will then be switched from "allow 32bit sloppyness" to "insist on 64bit correctness", i.e. in the terms I used above, make the default option "noclamptime". > > Don > FrankH. From sacadmin Thu Jun 9 06:46:38 2005 Received: from phys-mayi-1 (phys-mayi-1-ipmp1.Germany.Sun.COM [129.157.128.114]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j59Dkcs0006492 for ; Thu, 9 Jun 2005 06:46:38 -0700 (PDT) Received: from conversion-daemon.mayi-mail1.germany.sun.com by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IHT00I01KTC2O@mayi-mail1.germany.sun.com> (original mail from frankho@mayi-mail1.germany.sun.com) for psarc@sac.SFBay.sun.com; Thu, 09 Jun 2005 15:46:11 +0200 (MEST) Received: from estale (estale.UK.Sun.COM [129.156.173.199]) by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with SMTP id <0IHT00FGGKWV2R@mayi-mail1.germany.sun.com> for psarc@sac.SFBay.sun.com; Thu, 09 Jun 2005 15:46:08 +0200 (MEST) Date: Thu, 09 Jun 2005 14:46:07 +0100 (BST) From: Frank Hofmann - Solaris Sustaining Subject: Re: 2005/361 PCFS timestamp handling cleanup To: psarc@sac.sfbay.sun.com Reply-to: Frank Hofmann - Solaris Sustaining Message-id: <0IHT00FGJKWW2R@mayi-mail1.germany.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_16 SunOS 5.11 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-MD5: IiVqZL9Yqo/paLdhj48XRg== Status: RO Content-Length: 999 Hello, somewhat relevant here: Why do the manpages for utimes(2)/utime(2) NOT specify EOVERFLOW as possible error code ? The manpages for these two functions don't list anything that could be used to flag the condition "beyond what the implementation can do" unless one counts EIO as being a suitable indicator for this (EINVAL is listed but given a very specific meaning). I would consider it proper and correct do give EOVERFLOW if somebody e.g. attempted to set a file timestamp to a value past the range the fs supports. ufs, for example, does exactly that, from ufs_setattr(): /* * Change file access or modified times. */ if (mask & (AT_ATIME|AT_MTIME)) { /* Check that the time value is within ufs range */ if (((mask & AT_ATIME) && TIMESPEC_OVERFLOW(&vap->va_atime)) || ((mask & AT_MTIME) && TIMESPEC_OVERFLOW(&vap->va_mtime))) { error = EOVERFLOW; [ ... ] Is that the right thing to do (if so, where does it state that ?), or is that beyond specs ? FrankH. From sacadmin Thu Jun 9 06:55:56 2005 Received: from phys-mayi-1 (phys-mayi-1-ipmp1.Germany.Sun.COM [129.157.128.114]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j59Dtus0008171 for ; Thu, 9 Jun 2005 06:55:56 -0700 (PDT) Received: from conversion-daemon.mayi-mail1.germany.sun.com by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IHT00M01L7PH0@mayi-mail1.germany.sun.com> (original mail from frankho@mayi-mail1.germany.sun.com) for psarc@sac.SFBay.sun.com; Thu, 09 Jun 2005 15:55:30 +0200 (MEST) Received: from estale (estale.UK.Sun.COM [129.156.173.199]) by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with SMTP id <0IHT00MN1LCHO2@mayi-mail1.germany.sun.com> for psarc@sac.SFBay.sun.com; Thu, 09 Jun 2005 15:55:30 +0200 (MEST) Date: Thu, 09 Jun 2005 14:55:30 +0100 (BST) From: Frank Hofmann - Solaris Sustaining Subject: Re: 2005/361 PCFS timestamp handling cleanup To: psarc@sac.sfbay.sun.com Cc: Frank.Hofmann@Sun.COM Reply-to: Frank Hofmann - Solaris Sustaining Message-id: <0IHT00MN2LCIO2@mayi-mail1.germany.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_16 SunOS 5.11 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-MD5: 2X/+CUX2A53yCk5iQeTmfQ== Status: RO Content-Length: 922 Hello, before everyone jumps on standards boundary conditions again, let me raise attention for another item in the proposal that I had originally suspected would raise comments. I don't want to sneak that through because "time handling standards" have overshadowed this case and we've been off-topic frequently. The second proposal in the case is: * to simplify PCFS timestamp handling by removing the broken code in it that attempts to handle daylight saving time adjustments. I.e. I'll retire the "daylight" part of: struct pcfs_args { int timezone; /* seconds west of Greenwich */ int daylight; /* type of dst correction */ int flags; }; The reasons for doing that have been given in the case submission. Does anybody have to add to these, argue some points of it, or is this part of the proposal unanonimously/uncommentedly accepted ? Best Regards, FrankH. From sacadmin Thu Jun 9 13:24:00 2005 Received: from spartan.SFBay.Sun.COM (spartan [129.146.226.64]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j59KO0s0005025 for ; Thu, 9 Jun 2005 13:24:00 -0700 (PDT) Received: from spartan.SFBay.Sun.COM (spartan.SFBay.Sun.COM [129.146.226.64]) by spartan.SFBay.Sun.COM (8.12.10+Sun/8.12.10) with SMTP id j59KNYQi023900; Thu, 9 Jun 2005 13:23:34 -0700 (PDT) Message-Id: <200506092023.j59KNYQi023900@spartan.SFBay.Sun.COM> Date: Thu, 9 Jun 2005 13:23:34 -0700 (PDT) From: Don Cragun Reply-To: Don Cragun Subject: Re: 2005/361 PCFS timestamp handling cleanup To: james.d.carlson@Sun.COM Cc: Frank.Hofmann@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: ua6wNk5q/1hftgkil7Ztbg== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.5.5 SunOS 5.9 sun4u sparc Status: RO Content-Length: 7349 >Date: Wed, 8 Jun 2005 23:37:16 -0400 >From: James Carlson > >Don Cragun writes: >> >if we were to change pcfs in the way you propose, you'll go from >> >requiring a "make clean" to get a correct build to not being able to see >> >the source code at all. >> >> A make clean doesn't correct the problem, it shifts the problem. After >> you do the make clean, every subsequent make will be a full build. > >Suboptimal, to be sure. But far better than being completely unable >to read your data at all. Which is why I said earlier that a few utilities (in addition to ls) should be made available for use by everyone on all 64-bit kernels. > >As I remember hearing someone say, customers tend to form unnatural >attachments to their data. The attachment isn't unnatural at all. Some people (including me) even believe that the timestamp on a file is part of their data. (I timestamps to determine what files I have changed since the last time I ran a backup.) > >> As Darren pointed out, we already have /usr/bin/64/ls (or >> /usr/bin/sparcv9/ls depending on your hardware). I already suggested >> that we should add 64-bit versions of touch to reset dates on >> filesystems imported from systems that don't have a clock set anywhere >> close to "real" world time. > >That doesn't work unless the 32-bit world suddenly goes away. It's >unfortunately not going away any time soon. This is why I agreee that we need a mount option to munge timestamps and why the option to perform munging should be on by default on 32-bit systems. > >> Are you proposing that ZFS also ALWAYS map out-of-range timestamps >> for files to one of the overflow times when 32-bit applications are >> making *stat() calls? > >No, of course not. That's a different situation. We're talking about >PCFS here, and the ways in which multiple implementations (and perhaps >some ancient lack of foresight) have hemmed us in. How is it different? Why won't it be cited as a precedent? A PCFS filesystem can contain over 2.5 billion different timestamps that won't fit into a 32-bit signed time_t with the Epoch that Solaris (and all UNIX and all POSIX) systems use. There is nothing invalid about these time stamps (although some of them may currently be an indication of poor system administration.) A ZFS filesystem can contain over 18 quintillion different timestamps that won't fit into a 32-bit signed time_t. If PSARC's decision is that all timestamps on a PCFS filesystem that won't fit in a signed 32-bit time_t should be changed to Tue Jan 19 03:14:07 2038 when requested by a 32-bit application AND that all attempts to set a timestamp on a PCFS filesystem file to a time before Tue Jan 1 00:00:00 1980 are to be changed to set the timestamp to Tue Jan 1 00:00:00 1980 rather than return an error saying PCFS doesn't support that timestamp, then why shouldn't I believe that the committee believes that all timestamps on a ZFS filesystem that won't fit in a signed 32-bit time_t should be changed to Tue Jan 19 03:14:07 2038 when requested by a 32-bit application AND why shouldn't I believe that if a 64-bit application tries to set the timestamp on a UFS filesystem file to Mon Jan 1 00:00:00 1900 that PSARC doesn't believe that utimes() should change the timestamp to Fri Dec 13 20:45:52 1901 rather than return an error saying UFS doesn't support that timestamp? > >> Is it really impossible for everyone else to imagine that a PCFS >> filesystem could actually intend to have file timestamps beyond 2037 >> and that the creator of those files didn't want the timestamps to be >> translated to another arbitrary point in time when accessed by some, >> but not all, applications? Do we have any reason to believe that ALL >> PCFS filesystems come from systems with no concept of correct time? > >So ... this hypothetical user would be displeased if the timestamp >showed up as some arbitrary value, but would end up being relatively >happy if stat() returned -1, and he just couldn't read his files at >all. > >Why is that? The user won't be happy. There are several ways to allow the user to become happy again (before, during, or after mounting that a filesystem from a poorly administered system that may have a problem with timestamps). These include, but are clearly not limited to: 1. Do what this case is suggesting as a mount option (rather than as unconditional behavior). 2. Add or replace a few of our file utilities with 64-bit versions to make it easy to clean up timestamps on files. 3. Have PCFSmount check the timestamp on the root of the filesystem being mounted and give the person mounting the filesystem an option to reset all timestamps if the root timestamp is more than a year in the future. 4. Implement ABI-II as described by Casper earlier (all time_t's are 64-bit even in 32-bit applications among other sweeping changes). I just believe the decision to unconditionally modify timestamps for one set of applications while leaving them alone for others is a horrendous choice. > >> What is being done here as a fast-track for PCFS may well set precedent >> for ZFS or any other filesystem whose timestamp range, signedness, or >> Epoch is different from UFS. I believe this is an extremely dangerous >> precedent. > >I see it very differently. I see it as a regrettable kludge that >makes the best of a bad situation. > >Existing systems produce PCFS files with bad timestamps that happen >also to be beyond the UNIX epoch. Existing Solaris interfaces can't >deal with this. We're thus stuck with denying access to the user's >data (very bad), forcing the user to recompile and switch to an >interface we haven't invented yet (extremely bad), or lying to the >user by clamping the timestamp to an arbitrary but predictable value >(bad). No, we're not stuck. We can give users the tools they need to correct bad timestamps and all of the other problems this case "solves" go away for 32 years. At that point, UFS will be dead anyway (except as a possible read-only archive format), but PCFS will still have another 90 years of life before the date passes the range of values that can be stored in a PCFS timestamp. And, please stop misusing "UNIX epoch" like this. The UNIX epoch is Thu Jan 1 00:00:00 1970 UCT. All the times we're talking about are "beyond the UNIX epoch". The fact that PCFS cannot store the UNIX epoch as a timestamp is not a problem. Problem 1 is that PCFS timestamps include lots of values that we can't store in a 32-bit signed time_t with the UNIX epoch as the zero point. ZFS timestamps have exactly the same problem. Problem 2 is that (apparently) a lot of systems on which PCFS filesystems are created have clocks that are set more than 30 years into the future and that some of these filesystems have been mounted on Solaris systems. We assume that no one mounting a ZFS filesystem will run into this problem because all Solaris/UNIX/POSIX/Linux system admins know they need to set the system clock. :-} > >I fail to see why the integrity of the file timestamp is more >important than the contents of the file itself. The timestamp is not MORE important that the contents. But for some applications, the timestamp IS the only thing about the file that matters (as in "find dir -newer file"). - Don ... ... ... From sacadmin Thu Jun 9 14:53:19 2005 Received: from spartan.SFBay.Sun.COM (spartan [129.146.226.64]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j59LrJs0015835 for ; Thu, 9 Jun 2005 14:53:19 -0700 (PDT) Received: from spartan.SFBay.Sun.COM (spartan.SFBay.Sun.COM [129.146.226.64]) by spartan.SFBay.Sun.COM (8.12.10+Sun/8.12.10) with SMTP id j59LqrQi023962; Thu, 9 Jun 2005 14:52:53 -0700 (PDT) Message-Id: <200506092152.j59LqrQi023962@spartan.SFBay.Sun.COM> Date: Thu, 9 Jun 2005 14:52:53 -0700 (PDT) From: Don Cragun Reply-To: Don Cragun Subject: Re: 2005/361 PCFS timestamp handling cleanup To: Frank.Hofmann@Sun.COM Cc: psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: hFs2cCQnV0T6wigtgeyw7A== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.5.5 SunOS 5.9 sun4u sparc Status: RO Content-Length: 7610 >Date: Thu, 09 Jun 2005 12:30:24 +0100 (BST) >From: Frank Hofmann - Solaris Sustaining > >[ ... ] >> *** Topic 3: Should the NFSv2 and v3 tunable nfs_allow_preepoch_time >> be a mount option or a tunable and what do we do with 32-bit kernels >> on x86? >[ ... ] >> out by the time an application sees the timestamp) from making UFS >> and/or NFS unusable. I agree with Frank that this probably should >> have been a mount option rather than a system tunable, but the > >That suggestion was made by Bill, not me. Sorry. I got lost switching back and forth between messages. > >> important point to me is that it can be disabled when it is appropriate >> to do so. >> >> I have absolutely no objection to adding a PCFS mount option to clamp >> timestamps as suggested by this case on 32-bit kernels. On 32-bit >> kernels it is clearly necessary sometimes. (At least all applications > >Why is this necessity 32bit-kernel specific ? Because we currently have no 64-bit applications on 32-bit kernels that can can easily be used to clean up a bad timestamp problem. > >I can perform the timestamp arithmetics in 64bit even on a 32bit kernel >and return EOVERFLOW (or whatever error, see below ...) if the conversion >to UNIX 32bit time_t exceeds the range of the data type. Yes, but only by reading the raw PCFS filesystem and finding the timestamps. On a 32-bit kernel there is no libc interface that will actually give you the PCFS timestamp if it doesn't fit in a 32-bit signed time_t with 1/1/1970 0:00:00 UCT as the zero point. > >The *only* reason why it is necessary NOT to do that is to make things >work. And that doesn't depend on whether I'm running a 32bit or a 64bit >kernel. > >> on the machine will be getting the same l\bi\be\b"translation of the >> truth".) I object to having any kernel give different applications >> running on the same machine different timestamps for a given file and >> reporting success to all of the applications involved. I would be >> relatively happy with the same PCFS mount option on 64-bit kernels as >> suggested above for 32-bit kernels as long as the mount option causes >> the same translation of timestamps to be given to all applications >> accessing any particular file. > >I agree into making this a mount option, or rather a pair of options >"clamptime/noclamptime" that switches between the two behaviours: > >a) both 32bit and 64bit applications never see timestamps outside of > the range "FAT epoch" (01/01/1980 00:00:00) and "end of 32bit time_t" > (01/19/2038, 03:14:07). This means: > * The attempt to set a timestamp to outside that range will > cause an error return to the application - for both 32bit/64bit. > This error CANNOT (!) be EOVERFLOW - see utime(2)/utimes(2), > which don't include EOVERFLOW in the possible list of error > codes ... It can and should be EOVERFLOW. It isn't listed in SUSv3 (and therefore in our man page) because you'd never have run into the problem when the standards haven't fully addressed timestamp overflows yet. (I believe EOVERFLOW is correctly specified everywhere where it should be used due to file size problems.) Nonetheless, the description of EOVERFLOW from the standards is: "Value too large to be stored in data type. An operation was attempted which would generate a value that is outside the range of values that can be represented in the relevant data type or that are allowed for a given data item." If utime() or utimes() is used to try to set a timestamp of 01/01/1970 00:00:00 on a PCFS file, EOVERFLOW is the correct error code. (You may also notice that none of our man pages specify EFAULT as a possible error. The standards do this intentionally. The expected behavior is that a "system call" will give an EFAULT, but a similar function implemented purely as a library routine would give memory fault, core dumped. If the condition is caught and reported, EFAULT is THE errno value to use even if it isn't explicitly listed on the man page.) > * Finding a timestamp on the medium that is post-32bit-time_t > will report 01/19/2038, 03:14:07, again for both 32bit/64bit. > This will be the behaviour with the "clamptime" option. > >b) The full range of FAT time will be made accessible to applications. > This means: > * A 64bit application will see timestamps post-"end of 32bit > time_t" as they are recorded on the medium, and will be able > to set timestamps to any value within the range possible for > FAT (01/01/1980 ... 31/12/2127). The attempt to set a time > to outside that range will result in an error - see above, > can unfortunately NOT be EOVERFLOW ... As above the error can, and should, be EOVERFLOW for this case. > * A 32bit application will get EOVERFLOW on stat(2) if a > timestamp on the medium is post-"end of 32bit time_t". > It will get the same error as the 64bit app if it attempts to > set a time outside of the possible range (applies only to > pre-1980 times for 32bit apps, of course). > This will be the behaviour with the "noclamptime" option. > >I do insist, though, that behaviour a) ("clamptime" or whatever) >becomes the default behaviour. I do not agree with breaking things >that work now (32bit app on 32bit kernel), even if they only work >due to a (32bit-specific) bug. > > >I'll rather propagate the bug to 64bit in order to let 32/64bit apps >see the same, and WORK the same, indifferent of the underlaying kernel. > If PCFS was the only filesystem that had timestamps that can't be viewed by 32-bit apps, I would agree with you. But, since ZFS is "coming real soon" to a Solaris system near you, this won't be true for long. When ZFS arrives we get into the situation where we get different behavior based on filesystem type for exactly the same actions. I don't like that inconsistency. > >I.e.: If somebody *WANTS* post-32bit-time_t timestamps on PCFS, then >let them ASK for it ("-o noclamptime"), and mention clearly in the >pcfs manpages that using this will make files with such timestamps >inaccessible to 32bit applications. > >> >> *** Topic 4: We should provide a way for 32-bit applications to process >> extended time ranges. >[ ... ] > >Beyond the scope of this case - I don't want to follow up on this here. Agreed. > >> >> *** Topic 5: Should all of our file utilities be converted to be 64-bit >> apps? >[ ... ] > >Again, beyond the scope of this case. Agreed. > >> >> *** *** Footnote 1: It will be crucial that we do convert all file >> handling utilities to be 64-bit apps at least seven years prior to >> overflow time (given our current EOL and support policies), but we do >> not need to do so yet. > >Note on this: Once this happens, the default behaviour of PCFS will >then be switched from "allow 32bit sloppyness" to "insist on 64bit >correctness", i.e. in the terms I used above, make the default option >"noclamptime". Good. Maybe the introduction of ZFS will make this happen sooner. I would hope that we could reconsider the default PCFS mount option when a "significant" subset of the file handling utilities have been converted. (Although I'm not prepared to suggest exactly which utilities would need to be included in that set at this time. I firmly believe that any such set would at least have to include touch, cp/ln/mv, and rm as well as the ls that is already available. I'm not convinced yet that all file handling utilities, e.g. editors and compilation systems, are necessary to change this mount option default.) - Don > >> >> Don >> > >FrankH. > From sacadmin Thu Jun 9 15:32:28 2005 Received: from spartan.SFBay.Sun.COM (spartan [129.146.226.64]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j59MWIs0020336 for ; Thu, 9 Jun 2005 15:32:27 -0700 (PDT) Received: from spartan.SFBay.Sun.COM (spartan.SFBay.Sun.COM [129.146.226.64]) by spartan.SFBay.Sun.COM (8.12.10+Sun/8.12.10) with SMTP id j59MVqQi023994; Thu, 9 Jun 2005 15:31:52 -0700 (PDT) Message-Id: <200506092231.j59MVqQi023994@spartan.SFBay.Sun.COM> Date: Thu, 9 Jun 2005 15:31:52 -0700 (PDT) From: Don Cragun Reply-To: Don Cragun Subject: Re: 2005/361 PCFS timestamp handling cleanup To: Frank.Hofmann@Sun.COM Cc: psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: 3NF8/1Gz5/eFC1dBuB9gTA== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.5.5 SunOS 5.9 sun4u sparc Status: RO Content-Length: 223 Frank, As a postscript to my last message, I also want the case materials to explicitly state that this case does not set precedent for the way any other filesystem should handle extended range timestamps. Thanks, Don From sacadmin Fri Jun 10 02:42:29 2005 Received: from phys-mayi-1 (phys-mayi-1-ipmp1.Germany.Sun.COM [129.157.128.114]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j5A9gSs0024967 for ; Fri, 10 Jun 2005 02:42:28 -0700 (PDT) Received: from conversion-daemon.mayi-mail1.germany.sun.com by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IHV0000147FD4@mayi-mail1.germany.sun.com> (original mail from frankho@mayi-mail1.germany.sun.com) for psarc@sac.SFBay.sun.com; Fri, 10 Jun 2005 11:42:02 +0200 (MEST) Received: from estale (estale.UK.Sun.COM [129.156.173.199]) by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with SMTP id <0IHV0027B49Z9L@mayi-mail1.germany.sun.com>; Fri, 10 Jun 2005 11:42:00 +0200 (MEST) Date: Fri, 10 Jun 2005 10:42:00 +0100 (BST) From: Frank Hofmann - Solaris Sustaining Subject: Re: 2005/361 PCFS timestamp handling cleanup To: James.D.Carlson@Sun.COM, Don.Cragun@Sun.COM Cc: Frank.Hofmann@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Reply-to: Frank Hofmann - Solaris Sustaining Message-id: <0IHV0027C4A09L@mayi-mail1.germany.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_16 SunOS 5.11 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-MD5: lOCxBnd19QpHggqlKBwjBw== Status: RO Content-Length: 3468 [ ... ] > The user won't be happy. There are several ways to allow the user to > become happy again (before, during, or after mounting that a filesystem from a > poorly administered system that may have a problem with timestamps). > > These include, but are clearly not limited to: > 1. Do what this case is suggesting as a mount option (rather than as > unconditional behavior). There's nothing wrong with making the behaviour somewhat configurable, and mount options are clearly preferrable to kernel tunables. I agree with this. > 2. Add or replace a few of our file utilities with 64-bit versions to > make it easy to clean up timestamps on files. At least touch(1). But, as said, that doesn't solve the problem if you're running a 32bit kernel - you cannot run 64bit utilities then. > 3. Have PCFSmount check the timestamp on the root of the filesystem > being mounted and give the person mounting the filesystem an option > to reset all timestamps if the root timestamp is more than a year > in the future. Hmm, maybe that comes as a surprise but PCFS/FAT, believe it or not, does NOT have a timestamp on root. Doing 3. is technically impossible. See BugID #4633448. > 4. Implement ABI-II as described by Casper earlier (all time_t's are > 64-bit even in 32-bit applications among other sweeping changes). Far beyond the scope of this fasttrack. Far beyond the scope of any fasttrack ... > I just believe the decision to unconditionally modify timestamps for > one set of applications while leaving them alone for others is a > horrendous choice. The decision to deliberately break things (the ability to stat a file) fully conscious of the fact that a still-supported environment (32bit kernel) does not allow for any mechanism to fix this breakage is even more horrendous, IMHO. But we've argued this point long enough; For usability, a solution where both 32/64bit applications see the same clamped timestamps is equivalent to one where 32bit sees the clamped one and 64bit the "real" one. So, any objections to "clamptime/noclamptime" as a pair of mount options to control this, with "clamptime" being the default behaviour on both 32bit and 64bit kernel, to be changed in the future on the 64bit kernel once we have a decent set of 64bit file utils ? [ ... ] > No, we're not stuck. We can give users the tools they need to correct > bad timestamps and all of the other problems this case "solves" go away > for 32 years. At that point, UFS will be dead anyway (except as a We can give the users these tools on 64bit, or rather we can change the existing tools on a 64bit system to be able to deal with this. What we cannot do is to enable the existing tools on a 32bit system to deal with this (unless you wish to go the "ABI II" way). I think a "touchback" special, non-standard, utility is a very bad idea. It'll have about the same effect as telling people "use fsdb to fix this". > possible read-only archive format), but PCFS will still have another 90 > years of life before the date passes the range of values that can be > stored in a PCFS timestamp. Yes, PCFS in the embedded area (consumer electronic gadgets, aka mobile devices, cameras, ...) has a long life ahead of it. After all, FAT32 can handle 8 TB of data. The time by which you plug that memory card into your digicam is still a while away, so I'm surely looking forward to becoming a well-paid Y2038 consultant before I retire :) FrankH. From sacadmin Fri Jun 10 03:04:31 2005 Received: from phys-mayi-1 (phys-mayi-1-ipmp1.Germany.Sun.COM [129.157.128.114]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j5AA4Us0026748 for ; Fri, 10 Jun 2005 03:04:31 -0700 (PDT) Received: from conversion-daemon.mayi-mail1.germany.sun.com by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IHV00A015A0O8@mayi-mail1.germany.sun.com> (original mail from frankho@mayi-mail1.germany.sun.com) for psarc@sac.SFBay.sun.com; Fri, 10 Jun 2005 12:04:05 +0200 (MEST) Received: from estale (estale.UK.Sun.COM [129.156.173.199]) by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with SMTP id <0IHV002OI5AS9L@mayi-mail1.germany.sun.com>; Fri, 10 Jun 2005 12:04:04 +0200 (MEST) Date: Fri, 10 Jun 2005 11:04:04 +0100 (BST) From: Frank Hofmann - Solaris Sustaining Subject: Re: 2005/361 PCFS timestamp handling cleanup To: Frank.Hofmann@Sun.COM, Don.Cragun@Sun.COM Cc: psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Reply-to: Frank Hofmann - Solaris Sustaining Message-id: <0IHV002OL5AS9L@mayi-mail1.germany.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_16 SunOS 5.11 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-MD5: FTmh7maznTBJLKkFu5wmTg== Status: RO Content-Length: 3820 [ ... ] > >> I have absolutely no objection to adding a PCFS mount option to clamp > >> timestamps as suggested by this case on 32-bit kernels. On 32-bit > >> kernels it is clearly necessary sometimes. (At least all applications > > > >Why is this necessity 32bit-kernel specific ? > > Because we currently have no 64-bit applications on 32-bit kernels that > can can easily be used to clean up a bad timestamp problem. We don't have *any* such applications. We don't even have the 64bit applications on a 64bit kernel (which is a problem that is 'trivially' solveable by creating the 64bit utils, of course); the 32bit kernel is only peculiar in the sense that we cannot even make the applications that people expect to do that job (i.e. the standard fileutils) be able to to it. > > > >I can perform the timestamp arithmetics in 64bit even on a 32bit kernel > >and return EOVERFLOW (or whatever error, see below ...) if the conversion > >to UNIX 32bit time_t exceeds the range of the data type. > > Yes, but only by reading the raw PCFS filesystem and finding the > timestamps. On a 32-bit kernel there is no libc interface that will > actually give you the PCFS timestamp if it doesn't fit in a 32-bit > signed time_t with 1/1/1970 0:00:00 UCT as the zero point. The "reading the raw PCFS filesystem" is exactly what the kernel module does. That's what I meant. There's no problem at all making usr/kernel/fs/pcfs do the timestamp arithmetics in 64bit, whether the module is 32bit or not. In fact it's the only correct thing to do in any case. The 32bit kernel module would well be able to return EOVERFLOW to the applications. [ ... ] > > It can and should be EOVERFLOW. It isn't listed in SUSv3 (and I trust you on that. I just wondered why the manpages don't mention that. It may be similar to EFAULT, as you say, but then utimes(2) for example explicitly mentions EFAULT. Over the years, I got the impression that error returns from syscalls not listed in the manpages for these syscalls are actually bugs if they occur. Anyway, talking precedents, ufs sets one for returning EOVERFLOW on utimes(2), so I'm ok to follow that. [ ... ] > (You may also notice that none of our man pages specify EFAULT as a > possible error. The standards do this intentionally. The expected $ grep EFAULT /usr/share/man/man2/* | cut -d':' -f1 | sort -u | wc -l 84 $ uname -a SunOS estale 5.11 snv_16 sun4u sparc SUNW,Sun-Fire 84 manpages from section 2 refer to EFAULT on a current system. [ ... ] > >Note on this: Once this happens, the default behaviour of PCFS will > >then be switched from "allow 32bit sloppyness" to "insist on 64bit > >correctness", i.e. in the terms I used above, make the default option > >"noclamptime". > > Good. Maybe the introduction of ZFS will make this happen sooner. It'll become a necessity. Think of somebody deliberately using utimes(2) to create 64bit timestamps in order to denial-of-service attack the file utilities. You can't touch/rm/mv such a file right now. You only can "ls -l" it if you remember to use /usr/bin/64/ls. > > I would hope that we could reconsider the default PCFS mount option > when a "significant" subset of the file handling utilities have been > converted. (Although I'm not prepared to suggest exactly which > utilities would need to be included in that set at this time. I firmly > believe that any such set would at least have to include touch, > cp/ln/mv, and rm as well as the ls that is already available. I'm not > convinced yet that all file handling utilities, e.g. editors and > compilation systems, are necessary to change this mount option > default.) I fully agree with you on that. But again, sorting out which file utils shall be changed in which way is surely not the objective of this specific case. Thanks, FrankH. From sacadmin Fri Jun 10 03:09:17 2005 Received: from phys-mayi-1 (phys-mayi-1-ipmp1.Germany.Sun.COM [129.157.128.114]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j5AA9Gs0026767 for ; Fri, 10 Jun 2005 03:09:16 -0700 (PDT) Received: from conversion-daemon.mayi-mail1.germany.sun.com by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IHV00E015HLM4@mayi-mail1.germany.sun.com> (original mail from frankho@mayi-mail1.germany.sun.com) for psarc@sac.SFBay.sun.com; Fri, 10 Jun 2005 12:08:50 +0200 (MEST) Received: from estale (estale.UK.Sun.COM [129.156.173.199]) by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with SMTP id <0IHV00CN05IPGL@mayi-mail1.germany.sun.com>; Fri, 10 Jun 2005 12:08:50 +0200 (MEST) Date: Fri, 10 Jun 2005 11:08:50 +0100 (BST) From: Frank Hofmann - Solaris Sustaining Subject: Re: 2005/361 PCFS timestamp handling cleanup To: Don.Cragun@Sun.COM Cc: psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Reply-to: Frank Hofmann - Solaris Sustaining Message-id: <0IHV00CN25IQGL@mayi-mail1.germany.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_16 SunOS 5.11 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-MD5: Usc0Qhwz6NdyxY96v+CNMQ== Status: RO Content-Length: 635 > Frank, > As a postscript to my last message, I also want the case > materials to explicitly state that this case does not set precedent for > the way any other filesystem should handle extended range timestamps. > > Thanks, > Don > Do we also want to state that in the case a precedent (for how this situation is to be dealt with) *IS* accepted in some future case by the committee, that PCFS will be expected to match/follow this ? That decides on the stability classification of "clamptime/noclamptime" as interfaces. These need to be documented, and I read your comment that we need a note saying "may change" ? FrankH. From sacadmin Fri Jun 10 03:57:43 2005 Received: from phorcys.East.Sun.COM (phorcys.East.Sun.COM [129.148.174.143]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j5AAvhs0029524 for ; Fri, 10 Jun 2005 03:57:43 -0700 (PDT) Received: from phorcys.East.Sun.COM (localhost [127.0.0.1]) by phorcys.East.Sun.COM (8.13.4+Sun/8.13.4) with ESMTP id j5AAvENF005989; Fri, 10 Jun 2005 06:57:14 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.East.Sun.COM (8.13.4+Sun/8.13.4/Submit) id j5AAvEP8005986; Fri, 10 Jun 2005 06:57:14 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17065.29194.813188.300322@gargle.gargle.HOWL> Date: Fri, 10 Jun 2005 06:57:14 -0400 From: James Carlson To: Frank Hofmann - Solaris Sustaining Cc: Don.Cragun@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Subject: Re: 2005/361 PCFS timestamp handling cleanup In-Reply-To: Frank Hofmann - Solaris Sustaining's message of 10 June 2005 11:08:50 References: <0IHV00CN25IQGL@mayi-mail1.germany.sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 768 Frank Hofmann - Solaris Sustaining writes: > > As a postscript to my last message, I also want the case > > materials to explicitly state that this case does not set precedent for > > the way any other filesystem should handle extended range timestamps. > > Do we also want to state that in the case a precedent (for how this > situation is to be dealt with) *IS* accepted in some future case by > the committee, that PCFS will be expected to match/follow this ? I don't think that's necessary. That's essentially what the case already is. -- James Carlson, KISS Network Sun Microsystems / 1 Network Drive 71.234W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.497N Fax +1 781 442 1677 From sacadmin Fri Jun 10 08:03:03 2005 Received: from engmail1mpk.Eng.Sun.COM (engmail1mpk [129.146.11.21]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j5AF33s0010979 for ; Fri, 10 Jun 2005 08:03:03 -0700 (PDT) Received: from marduk.eng.sun.com (marduk.SFBay.Sun.COM [129.146.108.224]) by engmail1mpk.Eng.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id j5AF2Y80019146; Fri, 10 Jun 2005 08:02:34 -0700 (PDT) Received: from marduk.eng.sun.com (localhost [127.0.0.1]) by marduk.eng.sun.com (8.12.11+Sun/8.12.11) with ESMTP id j5AF1M36002892; Fri, 10 Jun 2005 08:01:22 -0700 (PDT) Received: (from gww@localhost) by marduk.eng.sun.com (8.12.11+Sun/8.12.11/Submit) id j5AF1MVi002891; Fri, 10 Jun 2005 08:01:22 -0700 (PDT) Date: Fri, 10 Jun 2005 08:01:22 -0700 (PDT) From: Gary Winiger Message-Id: <200506101501.j5AF1MVi002891@marduk.eng.sun.com> To: frankho@mayi-mail1.germany.sun.com, james.d.carlson@sun.com Subject: Re: 2005/361 PCFS timestamp handling cleanup Cc: Don.Cragun@sun.com, psarc@sac.sfbay.sun.com, Brian.Utterback@sun.com X-Sun-Charset: US-ASCII Status: RO Content-Length: 194 > I don't think that's necessary. That's essentially what the case > already is. When we're ``all done here,'' I'd like to see a final spec (so we can point others to it if needed). Gary.. From sacadmin Fri Jun 10 09:08:14 2005 Received: from spartan.SFBay.Sun.COM (spartan [129.146.226.64]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j5AG8Es0015179 for ; Fri, 10 Jun 2005 09:08:14 -0700 (PDT) Received: from spartan.SFBay.Sun.COM (spartan.SFBay.Sun.COM [129.146.226.64]) by spartan.SFBay.Sun.COM (8.12.10+Sun/8.12.10) with SMTP id j5AG7kQi025231; Fri, 10 Jun 2005 09:07:46 -0700 (PDT) Message-Id: <200506101607.j5AG7kQi025231@spartan.SFBay.Sun.COM> Date: Fri, 10 Jun 2005 09:07:46 -0700 (PDT) From: Don Cragun Reply-To: Don Cragun Subject: Re: 2005/361 PCFS timestamp handling cleanup To: frankho@mayi-mail1.germany.sun.com Cc: psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: a+hxJTgF55mnLXTj7MUCqw== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.5.5 SunOS 5.9 sun4u sparc Status: RO Content-Length: 4903 >Date: Fri, 10 Jun 2005 10:42:00 +0100 (BST) >From: Frank Hofmann - Solaris Sustaining > ... ... ... >> These include, but are clearly not limited to: >> 1. Do what this case is suggesting as a mount option (rather than as >> unconditional behavior). > >There's nothing wrong with making the behaviour somewhat configurable, >and mount options are clearly preferrable to kernel tunables. >I agree with this. Great. > ... ... ... > >> 3. Have PCFSmount check the timestamp on the root of the filesystem >> being mounted and give the person mounting the filesystem an option >> to reset all timestamps if the root timestamp is more than a year >> in the future. > >Hmm, maybe that comes as a surprise but PCFS/FAT, believe it or not, >does NOT have a timestamp on root. Doing 3. is technically impossible. >See BugID #4633448. It surprised me, but not much. ;-( Especially with floppy disks, CDs and DVDs; users on PCs physically install and remove the media with no concept of doing a mount or unmount unless they are physically installing or removing the drive at the same time. > >> 4. Implement ABI-II as described by Casper earlier (all time_t's are >> 64-bit even in 32-bit applications among other sweeping changes). > >Far beyond the scope of this fasttrack. Far beyond the scope of >any fasttrack ... Agreed. But, if some other project did implement ABI-II, there would be no need for this case. ;-) > >> I just believe the decision to unconditionally modify timestamps for >> one set of applications while leaving them alone for others is a >> horrendous choice. > >The decision to deliberately break things (the ability to stat a file) >fully conscious of the fact that a still-supported environment (32bit >kernel) does not allow for any mechanism to fix this breakage is even >more horrendous, IMHO. But we've argued this point long enough; For >usability, a solution where both 32/64bit applications see the same >clamped timestamps is equivalent to one where 32bit sees the clamped >one and 64bit the "real" one. > >So, any objections to "clamptime/noclamptime" as a pair of mount options >to control this, with "clamptime" being the default behaviour on both >32bit and 64bit kernel, to be changed in the future on the 64bit kernel >once we have a decent set of 64bit file utils ? I can live with this solution. > >[ ... ] >> No, we're not stuck. We can give users the tools they need to correct >> bad timestamps and all of the other problems this case "solves" go away >> for 32 years. At that point, UFS will be dead anyway (except as a > >We can give the users these tools on 64bit, or rather we can change >the existing tools on a 64bit system to be able to deal with this. >What we cannot do is to enable the existing tools on a 32bit system >to deal with this (unless you wish to go the "ABI II" way). I think PCFS specific utilities could also be written to do this on 32-bit (and, if desired, on 64-bit) systems, but I agree that that is not this project and would be of limited value. >a "touchback" special, non-standard, utility is a very bad idea. >It'll have about the same effect as telling people "use fsdb to fix >this". With my Canon camera's software, the timestamp on files on the compact flash card (and and maybe data in the picture files themselves) controls where those images are downloaded onto my PC and can be displayed on prints. If I found that I had incorrectly set the time on my camera before taking a bunch of pictures, I would GREATLY appreciate a utility like this to adjust the timestamps on those pictures to when they were actually taken. I would much rather do that in one invocation of a nonstandard utility giving it the difference between what time the camera thought it was and what time it actually was than to calculate the correct timestamps by hand and invoke fsdb or touch to fix each picture file individually. If we believe that lots of PCFS filesystems come from systems with unset clocks, this would seem to be a user friendly way clean things up. However, that is clearly not this project. > >> possible read-only archive format), but PCFS will still have another 90 >> years of life before the date passes the range of values that can be >> stored in a PCFS timestamp. > >Yes, PCFS in the embedded area (consumer electronic gadgets, aka >mobile devices, cameras, ...) has a long life ahead of it. After >all, FAT32 can handle 8 TB of data. The time by which you plug >that memory card into your digicam is still a while away, so I'm >surely looking forward to becoming a well-paid Y2038 consultant >before I retire :) I was deeply involved in the Y2K efforts, but unless Bush and congress screw up Social Security even more radically than I expect, I will be retired well before I could make much money as a Y2038 consultant. ;-} - Don > >FrankH. From sacadmin Fri Jun 10 09:09:14 2005 Received: from spartan.SFBay.Sun.COM (spartan [129.146.226.64]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j5AG9Ds0015193 for ; Fri, 10 Jun 2005 09:09:13 -0700 (PDT) Received: from spartan.SFBay.Sun.COM (spartan.SFBay.Sun.COM [129.146.226.64]) by spartan.SFBay.Sun.COM (8.12.10+Sun/8.12.10) with SMTP id j5AG8kQi025234; Fri, 10 Jun 2005 09:08:46 -0700 (PDT) Message-Id: <200506101608.j5AG8kQi025234@spartan.SFBay.Sun.COM> Date: Fri, 10 Jun 2005 09:08:46 -0700 (PDT) From: Don Cragun Reply-To: Don Cragun Subject: Re: 2005/361 PCFS timestamp handling cleanup To: Frank.Hofmann@Sun.COM Cc: psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: CqkWqCr7FbOgNkEmWFa1GA== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.5.5 SunOS 5.9 sun4u sparc Status: RO Content-Length: 3412 >Date: Fri, 10 Jun 2005 11:04:04 +0100 (BST) >From: Frank Hofmann - Solaris Sustaining > ... ... ... >> >> It can and should be EOVERFLOW. It isn't listed in SUSv3 (and > >I trust you on that. I just wondered why the manpages don't >mention that. It may be similar to EFAULT, as you say, but >then utimes(2) for example explicitly mentions EFAULT. Over >the years, I got the impression that error returns from syscalls >not listed in the manpages for these syscalls are actually bugs >if they occur. Anyway, talking precedents, ufs sets one for >returning EOVERFLOW on utimes(2), so I'm ok to follow that. > >[ ... ] >> (You may also notice that none of our man pages specify EFAULT as a >> possible error. The standards do this intentionally. The expected > >$ grep EFAULT /usr/share/man/man2/* | cut -d':' -f1 | sort -u | wc -l > 84 >$ uname -a >SunOS estale 5.11 snv_16 sun4u sparc SUNW,Sun-Fire > >84 manpages from section 2 refer to EFAULT on a current system. Sorry. I meant The Austin Group when I talked about "our man pages" and didn't give a pointer when I switched streams. Our (Sun) man pages are kind of schizoid on this topic. Every once in a while, Sun's man pages are aligned with POSIX/SUS (The Austin Group) man pages. When that happens, sometimes the errors list is stripped down to what is in the underlying standard and sometimes updated to actually document the errors Sun returns. Actually documenting what we do is clearly better for programmers using our manuals, but it takes a lot more work to keep the documentation up-to-date than many kernel and library maintainers are willing to do when they fix a bug that adds or eliminates an error condition. (Note that I am describing what I see in Sun man pages, not what I believe should happen. Don't shoot the messenger.) The standards state (in this case from the System Interfaces volume of IEEE Std 1003.1-2001(2004 edition) AKA Single UNIX Specification, Version 3 (2004 edition) AKA ISO/IEC 9945-2:2002 with Technical Corrigenda 1:2005 applied from Subclause 2.3 (Error Numbers) on page 21, lines 835-841: Implementations may support additional errors not included in this list, may generate errors included in this list under circumstances other than those described here, or may contain extensions or limitations that prevent some errors from occurring. The ERRORS section on each reference page specifies whether an error shall be returned, or whether it may be returned. Implementations shall not generate a different error number from the ones described here for error conditions described in this volume of IEEE Std 1003.1-2001, but may generate additional errors unless explicitly disallowed for a particular function. Sun's Intro(2) man page says: Each system call description attempts to list all possible error numbers. The following is a complete list of the error numbers and their names as defined in . It would be nice if the "attempts to list" were instead "lists" and all of our man pages were absolutely correct, but ... (And, don't misinterpret what I'm saying here. Most of our tech pubs people do an amazing job given the information they get from engineers. Furthermore, despite the problems with Sun man pages, the fact that they are so good makes some of our competitors envious.) > ... ... ... > >Thanks, >FrankH. - Don From sacadmin Fri Jun 10 12:07:11 2005 Received: from sentry.sfbay.sun.com (sentry [129.146.157.28]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j5AJ7Bs0027058 for ; Fri, 10 Jun 2005 12:07:11 -0700 (PDT) Received: from [192.9.61.225] (punchin-client-192-9-61-225.SFBay.Sun.COM [192.9.61.225]) by sentry.sfbay.sun.com (8.12.11+Sun/8.12.11) with ESMTP id j5AJ4ohg011623; Fri, 10 Jun 2005 12:04:51 -0700 (PDT) Message-ID: <42A9E4CA.7080003@sun.com> Date: Fri, 10 Jun 2005 12:06:50 -0700 From: Mike Ditto User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.7) Gecko/20041221 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Frank Hofmann - Solaris Sustaining CC: James.D.Carlson@sun.com, Don.Cragun@sun.com, Frank.Hofmann@sun.com, psarc@sac.sfbay.sun.com, Brian.Utterback@sun.com Subject: Re: 2005/361 PCFS timestamp handling cleanup References: <0IHV0027C4A09L@mayi-mail1.germany.sun.com> In-Reply-To: <0IHV0027C4A09L@mayi-mail1.germany.sun.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 466 >>2. Add or replace a few of our file utilities with 64-bit versions to >> make it easy to clean up timestamps on files. > > At least touch(1). But, as said, that doesn't solve the problem if > you're running a 32bit kernel - you cannot run 64bit utilities then. The fact that touch won't touch a file it can't stat is simply a bug in touch. Fixing it doesn't require a 64bit version or large file awareness or anything special at all. -=] Mike [=- From sacadmin Fri Jun 10 12:53:36 2005 Received: from spartan.SFBay.Sun.COM (spartan [129.146.226.64]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j5AJras0028755 for ; Fri, 10 Jun 2005 12:53:36 -0700 (PDT) Received: from spartan.SFBay.Sun.COM (spartan.SFBay.Sun.COM [129.146.226.64]) by spartan.SFBay.Sun.COM (8.12.10+Sun/8.12.10) with SMTP id j5AJr8Qi028527; Fri, 10 Jun 2005 12:53:08 -0700 (PDT) Message-Id: <200506101953.j5AJr8Qi028527@spartan.SFBay.Sun.COM> Date: Fri, 10 Jun 2005 12:53:08 -0700 (PDT) From: Don Cragun Reply-To: Don Cragun Subject: Re: 2005/361 PCFS timestamp handling cleanup To: Michael.Ditto@Sun.COM Cc: Frank.Hofmann@Sun.COM, psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: aVm/4mId0CSb7x/NKAPzuw== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.5.5 SunOS 5.9 sun4u sparc Status: RO Content-Length: 906 >Date: Fri, 10 Jun 2005 12:06:50 -0700 >From: Mike Ditto > ... ... ... > >The fact that touch won't touch a file it can't stat is simply a bug in >touch. Fixing it doesn't require a 64bit version or large file awareness >or anything special at all. Not true unless you build knowledge of all filesystems directly into the touch utility. You can't implement the -a option or the -m option (unless both happen to be specified) without some way to get the other timestamp that is currently on the file. It is not possible to implement the -r option if you can't stat(ref_file) to get its timestamps. Furthermore, attempts to set a PCFS file timestamp to a time in the range Jan 19 03:14:08 2038 through Dec 31 23:59:59 2127 is perfectly legal, but can't be done with the 32-bit libc times() or utimes() syscalls available to a 32-bit touch utility. - Don > > -=] Mike [=- From sacadmin Fri Jun 10 13:13:31 2005 Received: from sentry.sfbay.sun.com (sentry [129.146.157.28]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j5AKDUs0001554 for ; Fri, 10 Jun 2005 13:13:30 -0700 (PDT) Received: from [192.9.61.225] (punchin-client-192-9-61-225.SFBay.Sun.COM [192.9.61.225]) by sentry.sfbay.sun.com (8.12.11+Sun/8.12.11) with ESMTP id j5AKB9O9011752; Fri, 10 Jun 2005 13:11:11 -0700 (PDT) Message-ID: <42A9F453.9050507@sun.com> Date: Fri, 10 Jun 2005 13:13:07 -0700 From: Mike Ditto User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.7) Gecko/20041221 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Don Cragun CC: Frank.Hofmann@sun.com, psarc@sac.sfbay.sun.com, Brian.Utterback@sun.com Subject: Re: 2005/361 PCFS timestamp handling cleanup References: <200506101953.j5AJr8Qi028527@spartan.SFBay.Sun.COM> In-Reply-To: <200506101953.j5AJr8Qi028527@spartan.SFBay.Sun.COM> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 1606 Don Cragun wrote: >>The fact that touch won't touch a file it can't stat is simply a bug in >>touch. Fixing it doesn't require a 64bit version or large file awareness >>or anything special at all. > > Not true unless you build knowledge of all filesystems directly into > the touch utility. It's true for the cases that matter here. > You can't implement the -a option or the -m option (unless both happen > to be specified) without some way to get the other timestamp that is > currently on the file. -a or -m alone fundamentally require a stat() call due to the nature of the utimes() interface, and if that stat() fails, it's perfectly acceptable and correct for touch to fail, saying that it can't stat the file. What we need here, though, is to let the user do "touch foo" or "touch (a particular date) foo" to repair incorrect/corrupt time stamps on a file. That should just work. touch has no business complaining about the failure of a system call that it should never have made in the first place, and it MUST do what it was instructed to do (perform a call to utimes()) and report the success or failure of that operation. > Furthermore, > attempts to set a PCFS file timestamp to a time in the range Jan 19 > 03:14:08 2038 through Dec 31 23:59:59 2127 is perfectly legal, but > can't be done with the 32-bit libc times() or utimes() syscalls > available to a 32-bit touch utility. Yes, "legal" in some sense, but impossible regardless of the underlying file system, and thus not relevant to this case, and not important to real users for the next 20 or 30 years. -=] Mike [=- From sacadmin Wed Jun 22 09:42:14 2005 Received: from phys-mayi-1 (phys-mayi-1-ipmp1.Germany.Sun.COM [129.157.128.114]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j5MGgDEu017361 for ; Wed, 22 Jun 2005 09:42:13 -0700 (PDT) Received: from conversion-daemon.mayi-mail1.germany.sun.com by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IIH00G01VM6EL@mayi-mail1.germany.sun.com> (original mail from Frank.Hofmann@sun.com) for psarc@sac.sfbay.sun.com; Wed, 22 Jun 2005 18:42:12 +0200 (MEST) Received: from estale (estale.UK.Sun.COM [129.156.173.199]) by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with SMTP id <0IIH005DRVQBZ0@mayi-mail1.germany.sun.com>; Wed, 22 Jun 2005 18:42:12 +0200 (MEST) Date: Wed, 22 Jun 2005 17:42:12 +0100 (BST) From: Frank Hofmann - Solaris Sustaining Subject: Re: 2005/361 PCFS timestamp handling cleanup To: psarc@sac.sfbay.sun.com Cc: Brian.Utterback@Sun.COM Reply-to: Frank Hofmann - Solaris Sustaining Message-id: <0IIH005DSVQCZ0@mayi-mail1.germany.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_16 SunOS 5.11 sun4u sparc Content-type: multipart/mixed; boundary="Boundary_(ID_rp4lvPP8xNqeXIcNBsWRQA)" Status: RO Content-Length: 8749 --Boundary_(ID_rp4lvPP8xNqeXIcNBsWRQA) Content-type: TEXT/plain; charset=us-ascii Content-MD5: Yz+uHKbFTzIIfwNkW6ADBA== Hi, attached is the revised spec for this case. FrankH. --Boundary_(ID_rp4lvPP8xNqeXIcNBsWRQA) Content-type: TEXT/plain; name=fasttrack.pcfs-timestamps; charset=us-ascii; x-unix-mode=0600 Content-description: fasttrack.pcfs-timestamps Content-MD5: oi1q8gZTbQHCRR8Qh/k0TQ== Case: 2005/xxx PCFS timestamp handling cleanup Author: Frank Hofmann Summary: This proposal requests, for Patch binding, * to simplify PCFS timestamp handling by removing the broken code in it that attempts to handle daylight saving time adjustments. * to modify PCFS timestamp handling to exhibit a defined behaviour at the attempt to set/query a timestamp that is outside the range of time PCFS timestamps can cover, or outside the range of time 32bit UNIX time_t can cover. The Problem Timestamps on FAT filesystems cover the range between 01/Jan/1980, 00:00:00, and 31/12/2127, 23:59:59, and are represented as date/time bitfields, recorded in local time (of the recording timezone). In addition to that, FAT knows multiple types of timestamps but apart from the "last written" time all fields are optional. This causes two problems: First, this results in a problem "how to represent local time". The fact that FAT specifies time to be recorded in local time but fails to specify a way for the recording implementation to store information about the recording timezone on the media makes it impossible for receiving/reading systems to accurately report FAT timestamps. The best possible way of dealing with this is to provide a way for the "user" (the person initiating access to a FAT medium on the receiving system) to tell the filesystem what time correction to apply. PCFS allows this via argument to the mount(2) system call, "struct pcfs_args", which contains two fields, "timezone" (which specifies seconds west of UT) and "daylight" (which specifies whether DST adjustments shall be made). The use of the "daylight" parameter is: a) redundant (a corresponding offset could be added to the "timezone" parameter to get the same effect") b) inconsistent (does it apply to the creation of new files, i.e. PCFS being the recording implementation, or does it apply to the access to existing files, i.e. PCFS being the receiving implementation) c) incomplete (there is no shared timezone database between all recording/receiving implementations of FAT, and DST rules have changed over the lifetime of FAT and continue to change even today). Second, "time_t and FAT don't match". There's a disparity between UNIX time_t (32bit as well as 64bit) and FAT time/date - they cannot be unambiguously mapped onto each other for the whole range of timestamps that either type (32bit time_t, 64bit time_t, FAT timestamp) can cover. With the current PCFS code, this causes several problems: a) If we're running a 64bit kernel, we dutifully return a 64bit time_t post-2038 (UNIX epoch) if the FAT timestamp specifies such. This causes stat() syscalls done by 32bit applications to fail with EOVERFLOW. It breaks all our file utils, including touch(1), so that such a file cannot even be "fixed" by touch'ing it. b) If we're running a 32bit kernel, the calculation of the UNIX time corresponding to a given FAT timestamp overflows and hence for post-UNIX-epoch FAT times, essentially a random value is returned. This causes no failure but prevents applications from detecting that something is odd about a file's access/mod time. c) the optional timestamp fields may, in case they contain uninitialized instead of valid data, cause unexpected failure on access when running a 64bit kernel. The Proposed Solution * DST handling The code for dealing with daylight savings time adjustments in the conversion between UNIX time_t and FAT timestamps will be removed. Local time adjustment will be done exclusively based on the "seconds west of UT" information. Given the abovementioned restriction (no way of discovering the timezone in which the medium was created, and all such attempts therefore necessarily being "best guess"), the timezone correction via "seconds west of UT" is more than sufficient. * FAT/UNIX time disparity handling For dealing with the disparity of UNIX time and FAT time, a pair of PCFS mount options "clamptime" / "noclamptime" will be introduced. The behaviour will be as follows: Definitions used below ====================== "FAT 0" is the earliest date/time representable in timestamps on a FAT filesystem - 01/01/1980, 00:00:00. "FAT end" is the last date/time representable in timestamps on a FAT filesystem - 31/12/2127 23:59:59. "N/A" means that the implementation cannot return or the user cannot supply a value in this range due to limitations in the internal representation (data types used). "UNIX 0" is the time represented by (time_t)0. "UNIX32 end" is the time represented by (time_t)INT32_MAX. "(unambiguous)" means that a 1:1 mapping between FAT time and UNIX time_t exists, keeping sizeof(time_t) in mind. time therefore splits into four regions: v UNIX 0 v FAT 0 v UNIX32 end v FAT end ... R1 | R2 | R3 | R4 ... R1 and R4 are unbounded on lower/upper end. Behaviour ========= The following table will apply if the "clamptime" mount option is in effect: 32bit set time 32bit get time 64bit set time 64bit get time ----------------------------------------------------------------------- R1 EOVERFLOW N/A EOVERFLOW N/A R2 (unambiguous) (unambiguous) (unambiguous) (unambiguous) R3 N/A UNIX32 end EOVERFLOW UNIX32 end R4 N/A N/A EOVERFLOW N/A The following table will apply if the "noclamptime" mount option is in effect: 32bit set time 32bit get time 64bit set time 64bit get time ----------------------------------------------------------------------- R1 EOVERFLOW N/A EOVERFLOW N/A R2 (unambiguous) (unambiguous) (unambiguous) (unambiguous) R3 N/A EOVERFLOW (unambiguous) (unambiguous) R4 N/A N/A EOVERFLOW N/A The difference clamptime/noclamptime will be only in R3. For "clamptime", 32bit and 64bit applications will behave identically on system calls setting/querying timestamps on a PCFS filesystem, the range of timestamps that PCFS will allow to set/query will be limited to [ FAT 0 ... UNIX32 end ]. For "noclamptime", 32bit applications will fail with EOVERFLOW errors when attempting to query a timestamp whose FAT representation is after "UNIX32 end", while 64bit applications will have access to the full range [ FAT 0 ... FAT end ] of timestamps that FAT supports. The attempt to set a timestamp on PCFS to a value outside of the range that PCFS supports (R1 and R4 above) will return EOVERFLOW to the application in any case, regardless of whether the app is 32bit (possible only for R1) or 64bit (possible for R1 and R4). Default behaviour and reasoning =============================== Default behaviour at time of implementation will be "clamptime". The reason to do this is to retain the current user experience of "32bit apps on 32bit kernels work" (never see EOVERFLOW on stat). From a pure standards point of view, this positive user experience is only due to a bug (we're not returning EOVERFLOW on a 32bit kernel ever because timestamp numerics in PCFS are broken), but there are good reasons to make this the default behaviour nonetheless: a) functionality regressions (something that is perceived to work breaks by this change) are undesirable. It does not matter that "it worked only due to a bug", user perception "it worked" must be preserved. b) Solaris, as of this writing, does not supply 64bit file utilities throughout. I.e. even fixing a "broken" timestamp (by touching the file) is not possible. 64bit file utilities would not solve this problem, we still support 32bit kernels on x86 platforms. The time to revisit the decision on which of (no)clamptime shall be the default behaviour will need to be revisited once we EOF the 32bit kernel, or provide a full set of 64bit file utilities, whatever earlier. MAN PAGE CHANGES The man page for pcfs(7fs) will be changed to reflect that the "daylight" member of "struct pcfs" has no effect. The man page for mount_pcfs(1M) will be changed to describe the new mount options "clamptime" and "noclamptime" including the choice of "clamptime" as default. --Boundary_(ID_rp4lvPP8xNqeXIcNBsWRQA)-- From sacadmin Wed Jun 22 16:19:10 2005 Received: from thunk.east.sun.com (thunk.East.Sun.COM [129.148.174.66]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j5MNJ9Eu001484 for ; Wed, 22 Jun 2005 16:19:09 -0700 (PDT) Received: from 127.0.0.1 (localhost [127.0.0.1]) by thunk.east.sun.com (8.13.4+Sun/8.13.4) with ESMTP id j5MNJ8S8109021; Wed, 22 Jun 2005 19:19:08 -0400 (EDT) Subject: Re: 2005/361 PCFS timestamp handling cleanup From: Bill Sommerfeld To: Frank Hofmann - Solaris Sustaining Cc: psarc@sac.sfbay.sun.com, Brian.Utterback@sun.com In-Reply-To: <0IIH005DSVQCZ0@mayi-mail1.germany.sun.com> References: <0IIH005DSVQCZ0@mayi-mail1.germany.sun.com> Content-Type: text/plain Message-Id: <1119482348.107243.279.camel@thunk> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6.316 Date: Wed, 22 Jun 2005 19:19:08 -0400 Content-Transfer-Encoding: 7bit Status: RO Content-Length: 690 nit: > The time to revisit the decision on which of (no)clamptime > shall be the default behaviour will need to be revisited once > we EOF the 32bit kernel, or provide a full set of 64bit file > utilities, whatever earlier. I'm sure we could think of other reasons to reexamine this case, but I don't see it as a reason to hold this up. one other thing: is it really appropriate for 64-bit settime to return EOVERFLOW in R3 (between UNIX32 end and FAT end) when clamptime is in effect? that settime is unambiguous. [I don't feel strongly about this; any such timestamps will be "wrong" until after the UNIX32 time_t range is over] - Bill From sacadmin Thu Jun 23 00:45:46 2005 Received: from phys-mayi-1 (phys-mayi-1-ipmp1.Germany.Sun.COM [129.157.128.114]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j5N7jjEu029364 for ; Thu, 23 Jun 2005 00:45:46 -0700 (PDT) Received: from conversion-daemon.mayi-mail1.germany.sun.com by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IIJ00F011J4FI@mayi-mail1.germany.sun.com> (original mail from Frank.Hofmann@sun.com) for psarc@sac.sfbay.sun.com; Thu, 23 Jun 2005 09:45:44 +0200 (MEST) Received: from estale (estale.UK.Sun.COM [129.156.173.199]) by mayi-mail1.germany.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with SMTP id <0IIJ009B31K87K@mayi-mail1.germany.sun.com>; Thu, 23 Jun 2005 09:45:44 +0200 (MEST) Date: Thu, 23 Jun 2005 08:45:44 +0100 (BST) From: Frank Hofmann - Solaris Sustaining Subject: Re: 2005/361 PCFS timestamp handling cleanup To: Frank.Hofmann@Sun.COM, sommerfeld@sun.com Cc: psarc@sac.sfbay.sun.com, Brian.Utterback@Sun.COM Reply-to: Frank Hofmann - Solaris Sustaining Message-id: <0IIJ009B51K87K@mayi-mail1.germany.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_16 SunOS 5.11 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-MD5: zPpMRvyPimTAC6kqJ+dV4Q== Status: RO Content-Length: 1883 > nit: > > The time to revisit the decision on which of (no)clamptime > > shall be the default behaviour will need to be revisited once > > we EOF the 32bit kernel, or provide a full set of 64bit file > > utilities, whatever earlier. > > I'm sure we could think of other reasons to reexamine this case, but I > don't see it as a reason to hold this up. I thought it worth mentioning given the "precedence" potential. > > one other thing: is it really appropriate for 64-bit settime to return > EOVERFLOW in R3 (between UNIX32 end and FAT end) when clamptime is in > effect? that settime is unambiguous. > [I don't feel strongly about this; any such timestamps will be "wrong" > until after the UNIX32 time_t range is over] We've discussed this piece before; for "clamptime", it's best to let 32bit and 64bit applications behave the same (and therefore give 64bit apps EOVERFLOW in the same cases as we give it to 32bit apps - after all, it's called "clamptime", so let's clamp it ...). Reasoning: - if I let a 64bit app set time past (time_t)INT32_MAX but silently truncate to (time_t)INT32_MAX before putting the timestamp onto PCFS then I'll lie to the 64bit app about success setting the timestamp. - if I let a 64bit app set a time past (time_t)INT32_MAX and really put this value into the PCFS timestamp, but return (time_t)INT32_MAX on reading it, then I read something else but what I've written. - if I let the 64bit app read and write a 64bit timestamp then the 32bit and 64bit app will see different timestamps (due to letting the 32bit app have success on reading it, getting (time_t)INT32_MAX) I.e. if I want 32bit apps to never see EOVERFLOW on reading a timestamp and at the same time never want 32bit and 64bit apps see different timestamps then I must restrict the range of values for 64bit apps. FrankH. From sacadmin Wed Jun 29 09:21:46 2005 Received: from eastmail1bur.East.Sun.COM (eastmail1bur.East.Sun.COM [129.148.9.49]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j5TGLjEu017032 for ; Wed, 29 Jun 2005 09:21:45 -0700 (PDT) Received: from [129.148.226.13] (sr1-unsh01-03.East.Sun.COM [129.148.226.13]) by eastmail1bur.East.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id j5TGLhFt005187; Wed, 29 Jun 2005 12:21:44 -0400 (EDT) Message-ID: <42C2CA97.8050006@sun.com> Date: Wed, 29 Jun 2005 12:21:43 -0400 From: Brian Utterback User-Agent: Mozilla Thunderbird 1.0.5 (X11/20050628) X-Accept-Language: en-us, en MIME-Version: 1.0 To: psarc@sac.sfbay.sun.com CC: Frank Hofmann - Solaris Sustaining Subject: Re: 2005/361 PCFS timestamp handling cleanup References: <0IIJ009B51K87K@mayi-mail1.germany.sun.com> In-Reply-To: <0IIJ009B51K87K@mayi-mail1.germany.sun.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 508 The timer has expired on this fasttrack. There were no further comments after the revised spec apart from a request for clarification from Bill, who explicitly said he did not want to hold this up. As such, I am marking it "closed approved fasttrack" -- blu Remember when SOX compliant meant they were both the same color? ---------------------------------------------------------------------- Brian Utterback - OP/N1 RPE, Sun Microsystems, Inc. Ph:877-259-7345, Em:brian.utterback-at-ess-you-enn-dot-kom