From blake@Central Tue Mar 5 05:43:57 1991 From: blake@Central (Blake L.) To: G.@opus.Eng.Sun.COM Subject: 64 bits in 5.0 Resend---forgot the $#&#*! domain name! ----- Begin Included Message ----- To: G. harpy S. srk Subject: 5.0 64-bit proposal Cc: ehc Here is what I have written so far about putting 64 bit interfaces into 5.0 to support large file systems. Comments, please! ------------------------------------------------------------------ Introduction ------------ 5.0 requires changes in order to support file systems larger than two gigabytes. This size limit arises because the kernel represents file offsets as signed 32-bit integers. The DDI/DKI provides limited support for 64-bit file offsets, but further steps are needed before this feature will be of any use. The proposed changes fall into three areas: extending kernel support for 64-bit offsets, enabling device drivers to use these offsets, and adding a hook for a loadable system call that will provide the user-level interface to the extended offsets. The primary goal is to allow the creation of file systems as large as one terabyte. This feature is important for Sun's server business, as the 2GB limit is becoming increasingly restrictive in large configurations. A secondary goal is to make the necessary changes in a way that will be compatible with the expected evolution of SunOS towards more general use of 64-bit interfaces. It would be a mistake, for instance, to imitate the method used by Medusa in 4.1.1 since that implementation clearly represents a blind alley that would interfere with future development. Note that it is not a goal of this proposal to make 64-bit offsets visible at user level, nor to we propose to make all kernel data structures use them. These are worthy projects, but they involve a level of effort that is significantly greater than what we propose here. Our concern is just to support large file systems, for which purpose it suffices to make 64-bit file offsets usable with the raw disk interface. Current 64-bit support in 5.0 ----------------------------- With a view towards future support of large devices, the DDI/DKI defines two 64-bit types, lloff_t and lldaddr_t, to represent file offsets and disk addresses. The _uio_offset member in struct uio is of type lloff_t, and lldaddr_t appears in struct buf and struct uio. However, at present these quantities are in effect 32-bit values with an extra 32 bits of space reserved for future use. There are three obstacles to making use of the full 64 bits: 1) The kernel code has been written to ignore the upper 32 bits of all 64-bit quantities. 2) Related quantities in other structures have not been widened to 64 bits. For instance, file offsets in the "file" and "page" structures remain at 32 bits, and struct buf uses both lldaddr_t and the 32-bit daddr_t. 3) No mechanism has been established for device drivers to indicate whether or not they understand 64-bit offsets and disk addresses. Proposed kernel changes ---------------- 1) To enable use of 64-bit file offsets, we must expand the f_offset member of struct file to 64-bits. (It may also be desirable to expand the vnode offsets used in the VM system to 64 bits, but that is beyond the scope of this proposal.) 2) Kernel code that operates on f_offset and uio_offset file offsets must be rewritten to use all 64 bits instead of ignoring the upper half. Since the 5.0 C compiler is unlikely to support a built-in 64-bit type ("long long"), this will require using functions that implement 64-bit arithmetic and relational operations. A library of such functions is already included in Medusa. The number of required changes is small and the impact on performance should be negligible. 3) For consistency, the offset parameter of the vop_seek vnode operation should also be widened to 64 bits, although this is not a requirement for large file system support, since the specfs seek operation is a no-op. 4) The kernel must be able to decide on a per-device basis whether offsets greater than 2GB are allowed or cause an error. Drivers must have a mechanism for indicating that they understand 64-bit offsets, and there must be a place to record this information. For want of a better alternative, it has been suggested that we use a bit in the f_flag/uio_fmode flags for this purpose. 5) Sun-supplied disk drivers should support 64-bit file offsets. Device driver interface ----------------------- It is important that we continue to support device drivers that are written with 32-bit file offsets in mind. We cannot safely pass 64-bit offsets to such drivers since they will behave incorrectly when the offset exceeds 2GB. Thus, drivers need a method of indicating that they support extended offsets. The DDI "property" mechanism provides a reasonable solution---the kernel can call ddi_getprop to inquire whether the driver understands 64-bit offsets. This requires no changes to existing 32-bit drivers. The kernel can query the driver for this property once per open, and record the result in a bit in the f_flag member of struct file. Subsequent read/write/seek operations will use this bit to determine whether or not to return an error when the offset exceeds 2GB. System call hook ---------------- File system utilities such as mkfs require an interface that allows them to seek to arbitrary locations on the raw disk. For this purpose, we propose to provide a new system call, llseek, which is identical in semantics to lseek with the exception that it takes a 64-bit offset instead of 32. If we wish to make large file system support part of an unbundled product, this should be a loadable system call. In any case, we need to reserve a slot for it in the sysent table. ----- End Included Message ----- From S.@datsun Mon Mar 11 15:08:50 1991 From: S.@datsun (Bill S.) To: blake@Central, G.@Eng, harpy@rmtc.Central.Sun.COM, srk@Eng Subject: Re: 5.0 64-bit proposal (resend) Cc: ehc@Central 2) Kernel code that operates on f_offset and uio_offset file offsets must be rewritten to use all 64 bits instead of ignoring the upper half. Since the 5.0 C compiler is unlikely to support a built-in 64-bit type ("long long"), this will require using functions that implement 64-bit arithmetic and relational operations. A library of such functions is already included in Medusa. The number of required changes is small and the impact on performance should be negligible. Is this for sure? There's no long long support in the 5.0 C compiler? 3) For consistency, the offset parameter of the vop_seek vnode operation should also be widened to 64 bits, although this is not a requirement for large file system support, since the specfs seek operation is a no-op. Seems like all offset parameters to all vop functions should be changed to be consistent. From blake@Central Wed Mar 27 23:58:40 1991 From: blake@Central (Blake L.) To: G.@Eng Subject: 64-bit file offset proposal (PERISHABLE!) Here is a revised version of my proposal for putting 64-bit file offsets into 5.0. Since time is drawing short, I'd appreciate it if you could get this onto the PSARC agenda sometime soon. I don't think that the proposal will excite much controversy, but then one never can tell. The main obstacle that I see before me is finding a suitable train to climb aboard. ---------------------------------------------------------- Introduction ------------ 5.0 requires changes in order to support file systems larger than two gigabytes. This size limit arises because the kernel represents file offsets as signed 32-bit integers. The DDI/DKI provides limited support for 64-bit file offsets, but further steps are needed before this feature will be of any use. The proposed changes are in the following areas: 1) extended kernel support for 64-bit offsets; 2) enabling device drivers to use these offsets; 3) a new llseek system call to provide user-level access to the extended offsets; 4) changes to file system utilities to support large file systems. In addition, there are several open bugs concerning integer overflow when the size of a file system or individual file is too large. These should be fixed as part of the effort outlined here. The primary goal of this proposal is to allow file systems as large as one terabyte to be created and used with no special restrictions. This feature is important for Sun's server business, as the current 2GB limit is becoming increasingly restrictive in large configurations. A secondary goal is to make the necessary changes in a way that is compatible with the expected evolution of SunOS towards more general use of 64-bit interfaces. It would be a mistake, for instance, to imitate the method used by Medusa in 4.1.1, since that implementation represents a blind alley that would hinder future development. It is not a goal of this proposal to make the use of 64-bit file offsets universal either at user level or within kernel data structures. Both the level of effort and the risk of such extensive changes are significantly greater than what is reasonable in the 5.0 time-frame. Our concern here is just to support large file systems, for which purpose it suffices to make 64-bit file offsets usable with the raw disk interface. Current 64-bit support in 5.0 ----------------------------- With a view towards future support of large devices, the DDI/DKI defines two 64-bit types, lloff_t and lldaddr_t, to represent file offsets and disk addresses. The uio_loffset member in struct uio is of type lloff_t (uio_offset is an alias for the lower 32-bits of this extended offset), and lldaddr_t appears in struct buf and struct uio. However, at present these quantities are in effect 32-bit values with an extra 32 bits of space reserved for future use. There are three obstacles to making use of the full 64 bits: 1) The kernel code has been written to ignore the upper 32 bits of all 64-bit quantities. 2) Related quantities in other structures have not been widened to 64 bits. For instance, file offsets in the "file" and "page" structures remain at 32 bits, and struct buf uses both lldaddr_t and the 32-bit daddr_t. 3) No mechanism has been established for device drivers to indicate whether or not they understand 64-bit offsets and disk addresses. Proposed kernel changes ---------------- 1) To enable use of 64-bit file offsets, we must expand the f_offset member of struct file to 64 bits. There are other 32-bit offsets that are candidates for widening to 64 bits, particularly, the l_start and l_len members of struct flock, and the vnode offsets used in the VM system. However, we consider these changes to be beyond the scope of the present proposal. 2) The offset parameter of the vop_seek should also be widened to 64-bits. For consistency, we may also wish to widen the offset parameters of other vnode operations such as vop_frlock. We would ingnore the upper 32 bits of these parameters for now. 3) The vop_seek operation for all file system types except specfs will check for and reject offsets greater than 2GB. These file systems types will continue to use 32-bit offsets for other vnode operations. The vnode operations for specfs will be rewritten to use 64-bit offsets. 4) The generic read/write/lseek code will use the 64-bit f_offset file offset. 5) The kernel must be able to decide on a per-device basis whether uio offsets greater than 2GB are allowed or cause an error. Drivers must have a mechanism for indicating that they understand 64-bit offsets, and there must be a place to record this information. For want of a better alternative, we propose to use a bit in the f_flag/uio_fmode flags for this purpose. 6) The kernel code that deals with device I/O, e.g., physio, will use 64-bit uio offsets. It will check for 2GB overflow when calling 32-bit drivers. We will need to ensure that file systems that use only 32 bits of the offset leave the upper 32 bits set to zero. 7) Sun-supplied disk drivers should support 64-bit uio offsets. It now appears that the 5.0 C compiler will support a built-in 64-bit "long long" type. All 64-bit operations will be implemented using long long. Device driver interface ----------------------- It is important that we continue to support device drivers written with 32-bit file offsets in mind. We cannot safely pass 64-bit offsets to such drivers since they will behave incorrectly when the offset exceeds 2GB. Thus, drivers need a method of indicating that they support extended offsets. The DDI "property" mechanism provides a reasonable solution---the kernel can call ddi_getprop to inquire whether the driver understands 64-bit offsets. This requires no changes to existing 32-bit drivers. The kernel can query the driver for this property once per open, and record the result in a bit in the f_flag member of struct file. Subsequent read/write/seek operations will use this bit to determine whether or not to return an error when the offset exceeds 2GB. User-level interface -------------------- A new system call, llseek, will supply the user-level interface to the extended file offsets. llseek is identical in usage and semantics to lseek, with the exception that it takes a 64-bit offset parameter of 32. File system utilities that use to raw device interface to read and write file system structures and data must be rewritten to use the new llseek interface. We have identified the following utilities as requiring change: dump, fsck, mkfs, mkfs, clri, fsirand, edquota, quot, tunefs, fsdb, ff, fstyp, and labelit. The change is a straight- forward substitution of llseek for lseek. From S.@datsun Thu Apr 4 12:31:55 1991 From: S.@datsun (Bill S.) To: blake@Central Subject: 64-bit file offset proposal Cc: plat.sw.arc@opus PSARC reviewed your proposal yesterday and we had several questions. Under "Proposed kernel changes"... item 2 - we generally felt that all vop_* interfaces should be modified to pass 64-bit offsets, but we're curious as to the expected impact of this on the many existing filesystem types. item 5 - it wasn't clear to me that this information should be saved in the f_flag or uio_fmode fields. it seems like this is state that is private to specfs. specfs should handle checking whether a particular device can support 64-bit offsets and returning errors if it can not. obviously any part of the system that operates on devices without going through specfs would also need to maintain this state. I believe UFS falls into this category; are there others? it was also noted that if one part of the system *does* operate on devices through specfs, it might not have available to it a corresponding f_flag word with this saved state. item 7 - I take it that your proposal does *not* include modifying any disk drivers. other - you didn't specify how device drivers (or other parts of the kernel) using 64-bit offsets can print out such offsets (in error messages, for example), or whether they even need such a capability. Under "Device driver interace"... you need to specify which property will be used to indicate support for 64 bit offsets. Under "User level interface"... clearly you need to change more than just the calls to lseek(). you also need to change the data types of some variables in these programs so that they can store 64-bit offsets. From blake@Central Fri Apr 19 20:56:01 1991 From: blake@Central (Blake L.) To: S.@Eng Subject: Re: 64-bit file offset proposal Cc: plat.sw.arc@opus.Eng.Sun.COM Sorry for the late response. I've actually done most of the proposed changes at this point to make sure that everything works as advertised. > Under "Proposed kernel changes"... > > item 2 - we generally felt that all vop_* interfaces should be > modified to pass 64-bit offsets, but we're curious as to the expected > impact of this on the many existing filesystem types. The impact is small because we never let the offset get outside the 32-bit range. The read/write and seek routines guarantee this and therefore in the others, we can promptly convert the 64-bit value to 32 bits (I will put in assertions to make sure that we don't accidentally stray outside the legal range.) It is true that this requires changing the vnodeops for every file system type, but this is straightforward and only needs to be done once. Another global change is that everywhere that 32-bit uio offsets are used, we have to make sure that the upper 32 bits are set to zero, since physio() will use all 64 bits. > item 5 - it wasn't clear to me that this information should be saved > in the f_flag or uio_fmode fields. it seems like this is state that > is private to specfs. specfs should handle checking whether a particular > device can support 64-bit offsets and returning errors if it can not. > obviously any part of the system that operates on devices without going > through specfs would also need to maintain this state. I believe UFS > falls into this category; are there others? > > it was also noted that if one part of the system *does* operate on > devices through specfs, it might not have available to it a corresponding > f_flag word with this saved state. Yes, I agree, it should be private to specfs. However, taking the long view, we will probably eventually want a more general mechanism so that other file system types can also communicate safely with both 32- and 64-bit devices. > item 7 - I take it that your proposal does *not* include modifying any > disk drivers. There is no need to do so since we don't yet support any larger-than-2GB devices. On the other hand, I don't see any reason not to be prepared. I was planning to change the SCSI disk and tape drivers to use 64-bit uio offsets. This means a simple change to the read/write entry points, plus defining the "uio_loffset" property (see below). > other - you didn't specify how device drivers (or other parts of the > kernel) using 64-bit offsets can print out such offsets (in error > messages, for example), or whether they even need such a capability. I didn't see that as an important requirement---the programmer can always decompose the long long into its two halves with casts or other methods. David Keaton added long long format items, e.g., "%lld", to printf while he was at Prisma, and we now own this code. I can add this to the kernel printf if necessary, though I'm inclined to view it as an RFE. > Under "Device driver interface"... > > you need to specify which property will be used to indicate support > for 64 bit offsets. If no one objects, I will call it "uio_loffset", which is the name of the 64-bit offset member of struct uio. It is a boolean property. > Under "User level interface"... > > clearly you need to change more than just the calls to lseek(). you also > need to change the data types of some variables in these programs so that > they can store 64-bit offsets. Sure, but based on my 4.1 experience, it's not a big deal. The FS utilities already represent disk locations as block numbers rather than byte offsets. These just need to be converted to 64-bit byte offsets before calling llseek. From S.@datsun Wed Apr 24 16:44:56 1991 From: S.@datsun (Bill S.) To: blake@Central Subject: Re: 64-bit file offset proposal Cc: plat.sw.arc@opus.Eng.Sun.COM From blake@Central Fri Apr 19 20:56:08 1991 From: blake@Central (Blake L.) To: S.@Eng Subject: Re: 64-bit file offset proposal Cc: plat.sw.arc@opus.Eng.Sun.COM ... > item 5 - it wasn't clear to me that this information should be saved > in the f_flag or uio_fmode fields. it seems like this is state that > is private to specfs. specfs should handle checking whether a particular > device can support 64-bit offsets and returning errors if it can not. > obviously any part of the system that operates on devices without going > through specfs would also need to maintain this state. I believe UFS > falls into this category; are there others? > > it was also noted that if one part of the system *does* operate on > devices through specfs, it might not have available to it a corresponding > f_flag word with this saved state. Yes, I agree, it should be private to specfs. However, taking the long view, we will probably eventually want a more general mechanism so that other file system types can also communicate safely with both 32- and 64-bit devices. It seems like either the DDI ought to offer such an interface, or specfs ought to offer the interface. In any event, you didn't say what you're going to do instead of using f_flag. other than that, I don't think *I* have any more issues. other members of PSARC should jump in here so we can tell whether this is ready for approval or not. From blake@Central Wed Apr 24 18:58:06 1991 From: blake@Central (Blake L.) To: S.@datsun.Eng.Sun.COM Subject: Re: 64-bit file offset proposal Cc: plat.sw.arc@opus.Eng.Sun.COM > It seems like either the DDI ought to offer such an interface, or specfs > ought to offer the interface. In any event, you didn't say what you're > going to do instead of using f_flag. Ultimately, the information is stored in the device driver property lists and is obtained by calling ddi_get_prop(). The f_flag idea was just an efficiency hack to avoid making this call on every I/O operation. My new scheme caches the information in specfs. I have defined a new function offset_t spec_maxoffset(struct snode *sp) that returns the maximum valid file offset for the device associated with sp. A table indexed by hashed device number indicates which devices expect 64-bit uio offsets. If no entry exists for the device, one is created by calling ddi_get_prop() and if necessary, overwriting another entry. When the last reference to a device is removed in spec_close(), its entry is purged from the table. A small table should suffice. I have it set at 64 entries right now, which I think is generous. One open issue is what to do with negative offsets. The 5.0 code currently imposes no restrictions on these. In the same spirit, I could accept any 32-bit offset value and adjust the offset following each I/O to keep it in the range -2^31 to 2^31 - 1. But as far as I know, only two types of devices have occasion to use negative (32-bit) offsets: memory devices, and streams devices, where the offset overflows after 2 GB of serial I/O. I think that a better solution is to put these two classes of devices into the 64-bit category, and then prohibit all negative offsets (2^63 is large enough that we can ignore overflow: at 1 million bytes per second, we would not run into negative offsets for 300,000 years). Right now, I am implementing the "permissive" approach. What do you think is the right thing to do? From glenn@ivrel Wed Apr 24 19:16:37 1991 From: glenn@ivrel (Glenn C. Skinner) To: blake@Central, plat.sw.arc@opus Subject: Re: 64-bit file offset proposal From blake@Central Wed Apr 24 18:58:40 1991 Date: Wed, 24 Apr 91 19:55:36 MDT From: blake@Central (Blake L.) To: S.@datsun.Eng.Sun.COM Subject: Re: 64-bit file offset proposal Cc: plat.sw.arc@opus.Eng.Sun.COM My new scheme caches the information in specfs. I have defined a new function offset_t spec_maxoffset(struct snode *sp) that returns the maximum valid file offset for the device associated with sp. A table indexed by hashed device number indicates which devices expect 64-bit uio offsets. If no entry exists for the device, one is created by calling ddi_get_prop() and if necessary, overwriting another entry. When the last reference to a device is removed in spec_close(), its entry is purged from the table. A small table should suffice. I have it set at 64 entries right now, which I think is generous. This looks ok to me. I assume that you intend this function to become part of the DDI. One open issue is what to do with negative offsets. The 5.0 code currently imposes no restrictions on these. In the same spirit, I could accept any 32-bit offset value and adjust the offset following each I/O to keep it in the range -2^31 to 2^31 - 1. But as far as I know, only two types of devices have occasion to use negative (32-bit) offsets: memory devices, and streams devices, where the offset overflows after 2 GB of serial I/O. I think that a better solution is to put these two classes of devices into the 64-bit category, and then prohibit all negative offsets (2^63 is large enough that we can ignore overflow: at 1 million bytes per second, we would not run into negative offsets for 300,000 years). Right now, I am implementing the "permissive" approach. What do you think is the right thing to do? The primary reason for making this stuff be signed is to allow the offset type to do double duty as a displacement type. I don't see any reason to allow a quantity that's known to be an offset (as opposed to a displacement) be negative, so I'm inclined to favor your restrictive alternative. From S.@datsun Wed Apr 24 16:45:31 1991 Date: Wed, 24 Apr 91 16:39:30 PDT From: S.@datsun (Bill S.) To: blake@Central Subject: Re: 64-bit file offset proposal Cc: plat.sw.arc@opus.Eng.Sun.COM other than that, I don't think *I* have any more issues. other members of PSARC should jump in here so we can tell whether this is ready for approval or not. I'm ready to approve. -- Glenn From S.@datsun Wed Apr 24 23:22:46 1991 From: S.@datsun (Bill S.) To: blake@Central Subject: Re: 64-bit file offset proposal Cc: plat.sw.arc@opus.Eng.Sun.COM My new scheme caches the information in specfs. I have defined a new function offset_t spec_maxoffset(struct snode *sp) that returns the maximum valid file offset for the device associated with sp. A table indexed by hashed device number indicates which devices expect 64-bit uio offsets. If no entry exists for the device, one is created by calling ddi_get_prop() and if necessary, overwriting another entry. When the last reference to a device is removed in spec_close(), its entry is purged from the table. A small table should suffice. I have it set at 64 entries right now, which I think is generous. Why the need for the extra table instead of storing it in the (common) snode? One open issue is what to do with negative offsets. The 5.0 code currently imposes no restrictions on these. In the same spirit, I could accept any 32-bit offset value and adjust the offset following each I/O to keep it in the range -2^31 to 2^31 - 1. But as far as I know, only two types of devices have occasion to use negative (32-bit) offsets: memory devices, and streams devices, where the offset overflows after 2 GB of serial I/O. I think that a better solution is to put these two classes of devices into the 64-bit category, and then prohibit all negative offsets (2^63 is large enough that we can ignore overflow: at 1 million bytes per second, we would not run into negative offsets for 300,000 years). Right now, I am implementing the "permissive" approach. What do you think is the right thing to do? I think this is ok, but what will users get back from lseek? For the memory device they'll probably get back 32 bits that match what they expect (assuming they interpret it as unsigned). For streams maybe we don't care? From blake@Central Thu Apr 25 07:58:31 1991 From: blake@Central (Blake L.) To: Bill.S.@Eng Subject: Re: 64-bit file offset proposal Cc: plat.sw.arc@opus.Eng.Sun.COM > Why the need for the extra table instead of storing it in the (common) snode? No need. I'll put the information into the snode. > I think this is ok, but what will users get back from lseek? For the > memory device they'll probably get back 32 bits that match what they > expect (assuming they interpret it as unsigned). For streams maybe > we don't care? lseek gives back the least significant 32 bits of the offset, which will work as expected with the memory device. I am assuming that we don't care in the streams case---what you get back today is essentially the same thing. From S.@datsun Thu Jun 6 14:55:15 1991 Return-Path: Received: from Eng.Sun.COM (zigzag) by opus.Eng.Sun.COM (4.1/SMI-4.1) id AA13709; Thu, 6 Jun 91 14:55:11 PDT Received: from opus.Eng.Sun.COM by Eng.Sun.COM (4.1/SMI-4.1) id AA09381; Thu, 6 Jun 91 14:55:08 PDT Received: from datsun.Eng.Sun.COM by opus.Eng.Sun.COM (4.1/SMI-4.1) id AA13706; Thu, 6 Jun 91 14:55:05 PDT Received: by datsun.Eng.Sun.COM (4.1/SMI-4.1) id AA00537; Thu, 6 Jun 91 14:55:04 PDT Date: Thu, 6 Jun 91 14:55:04 PDT From: S.@datsun (Bill S.) Message-Id: <9106062155.AA00537@datsun.Eng.Sun.COM> To: plat.sw.arc@opus Subject: 64-bit file offset proposal Cc: blake@central Status: RO Here's the latest version of this proposal. It is a few weeks old at this point. A few people raised issues about this proposal. Those issues continue to be worked. I've included comments in the proposal to mark those issues. Also, please note the request at the end of this message that requires your response by 6/12/91. ----- >From blake@Central Wed May 15 17:23:24 1991 From: blake@Central (Blake L.) To: S.@Eng Subject: revised 64-bit file offset proposal Here is the updated version, with the changes that resulted from PSARC discussion: ----------------------------------------------------------------------- Introduction ------------ 5.0 requires changes in order to support file systems larger than two gigabytes. This size limit arises because the kernel represents file offsets as signed 32-bit integers. The DDI/DKI provides limited support for 64-bit file offsets, but further steps are needed before this feature will be of any use. The proposed changes are in the following areas: 1) extended kernel support for 64-bit offsets; 2) enabling device drivers to use these offsets; 3) a new llseek system call to provide user-level access to the extended offsets; 4) changes to file system utilities to support large file systems. In addition, there are several open bugs concerning integer overflow when the size of a file system or individual file is too large. These should be fixed as part of the effort outlined here. The primary goal of this proposal is to allow file systems as large as one terabyte to be created and used with no special restrictions. This feature is important for Sun's server business, as the current 2GB limit is becoming increasingly restrictive in large configurations. A secondary goal is to make the necessary changes in a way that is compatible with the expected evolution of SunOS towards more general use of 64-bit interfaces. It would be a mistake, for instance, to imitate the method used by Medusa in 4.1.1, since that implementation represents a blind alley that would hinder future development. It is not a goal of this proposal to make the use of 64-bit file offsets universal either at user level or within kernel data structures. Both the level of effort and the risk of such extensive changes are significantly greater than what is reasonable in the 5.0 time-frame. Our concern here is just to support large file systems, for which purpose it suffices to make 64-bit file offsets usable with the raw disk interface. Current 64-bit support in 5.0 ----------------------------- With a view towards future support of large devices, the DDI/DKI defines two 64-bit types, lloff_t and lldaddr_t, to represent file offsets and disk addresses. The uio_loffset member in struct uio is of type lloff_t (uio_offset is an alias for the lower 32-bits of this extended offset), and lldaddr_t appears in struct buf and struct uio. However, at present these quantities are in effect 32-bit values with an extra 32 bits of space reserved for future use. There are three obstacles to making use of the full 64 bits: 1) The kernel code has been written to ignore the upper 32 bits of all 64-bit quantities. 2) Related quantities in other structures have not been widened to 64 bits. For instance, file offsets in the "file" and "page" structures remain at 32 bits, and struct buf uses both lldaddr_t and the 32-bit daddr_t. 3) No mechanism has been established for device drivers to indicate whether or not they understand 64-bit offsets and disk addresses. Proposed kernel changes ---------------- 1) To enable use of 64-bit file offsets, we must expand the f_offset member of struct file to 64 bits. There are other 32-bit offsets that are candidates for widening to 64 bits, particularly, the l_start and l_len members of struct flock, and the vnode offsets used in the VM system. However, we consider these changes to be beyond the scope of the present proposal. [All offsets that are passed as parameters to any vnode functions will be converted to the 64-bit type. Within the vnode functions they will usually be converted immediately back to 32 bits. Thus, functions such as VOP_GETPAGE will be changed to take 64 bit offsets, but none of the vm system will be changed to use 64 bit offsets. - S.] 2) The offset parameter of the vop_seek vnode operation must also be widened to 64-bits. For consistency, we may wish to widen the offset parameters of other vnode operations, specifically vop_close, vop_frlock, and vop_space. We will ignore the upper 32 bits of these parameters for now. This requires changing the vnode operations for every file system type, but the changes are localized and largely mechanical. Since the read, write, and seek operations guarantee that the offset remains in the range 0 <= offset < 2GB, the other operations can immediately convert the offset to a 32-bit type on entry. 3) The vop_seek, vop_read, and vop_write operations for all file system types except specfs will check for and reject offsets greater than 2GB. These file systems types will continue to use 32-bit offsets for other vnode operations. Assertions will check that the offset is always in the legal range. The vnode operations for specfs will be rewritten as described in (6) below. 4) The generic read/write/lseek code will use the 64-bit f_offset file offset. lseek will return the least significant 32 bits of the offset. 5) The kernel must be able to decide on a per-device basis whether uio offsets greater than 2GB are allowed or cause an error, and drivers need a mechanism for indicating that they understand 64-bit offsets. The DDI property mechanism will be used to communicate this information between drivers and specfs. When specfs creates a "common" snode, it will query the driver for the boolean "uio_loffset" property. If the result is success, it will set a newly defined bit, SLOFFSET (0x40) in the s_flags member of the snode. [The exact property name and flag name are being discussed with the DDI and OBP groups. - S.] 6) The spec_seek, spec_read, and spec_write specfs operations will call a new function, offset_t spec_maxoffset(struct vnode *vp); to determine the maximum legal file offset for the vnode on which the operation is to be performed. The return value is determined as follows: a) If vp is a streams device, returns -1 b) Else if the SLOFFSET bit is set, returns 1TB - 1 c) Else returns 2GB - 1 When -1 is returned, any offset is allowed; otherwise, offsets are required to remain positive and less than or equal to the returned maximum. (The current specfs implementation puts no restrictions on offset values; the 4.1 policy was to disallow negative offsets except for streams devices and the memory device. A consequence of this change is that the memory device must use 64-bit offsets). 7) The kernel code that deals with device I/O, e.g., physio, will use 64-bit uio offsets. We will need to ensure that file systems that use only 32 bits of the offset leave the upper 32 bits set to zero. 8) The standard disk and tape drivers (sd.c, st.c) will define the uio_loffset property and use 64-bit offsets. The memory device driver will do so also. [The IPI disk driver will also be modified. - S.] It now appears that the 5.0 C compiler will support a built-in 64-bit "long long" type. All 64-bit operations will be implemented using long long. Device driver interface ----------------------- It is important that we continue to support device drivers written with 32-bit file offsets in mind. We cannot safely pass 64-bit offsets to such drivers since they will behave incorrectly when the offset exceeds 2GB. Thus, drivers need a method of indicating that they support extended offsets. The DDI property mechanism provides a reasonable solution---the kernel calls ddi_getprop to determine the value of the boolean "uio_loffset" property. Drivers that understand 64-bit offsets define this property to be true. By default, the property is false, and thus existing 32-bit drivers require no changes. specfs will query the driver for this property whenever it creates a "common" device snode. The result is recorded in the s_flags member of the snode using the newly-defined SLOFFSET bit. Subsequent operations on the device use this bit to determine the maximum valid file offset for the device. User-level interface -------------------- A new system call, llseek, will supply the user-level interface to the extended file offsets. llseek is identical in usage and semantics to lseek, with the exception that it takes a 64-bit offset parameter instead of a 32-bit one. File system utilities that use to raw device interface to read and write file system structures and data must be rewritten to use the new llseek interface. The following utilities require change: dump, fsck, mkfs, mkfs, clri, fsirand, edquota, quot, tunefs, fsdb, ff, fstyp, and labelit. llseek replaces lseek, and some variables correspondingly change type from off_t to the 64-bit offset_t. Since the utilities represent disk locations in terms of block numbers rather than byte offsets, the changes are localized and straightforward. ------ Blake, I don't think we ever decided which library should contain the llseek() interface. PSARC, I believe our default position would be to put this in a separate library. Let me know by 6/12 whether: - you believe llseek should be in libc. - you believe llseek should be in some other pre-existing library. - you have a suggested name for the new library to contain llseek. - you believe we should discuss this or any other issues related to this proposal in a PSARC meeting before deciding. Otherwise, if I don't hear any objections by 6/12, I will assume that we require llseek to be in a new library and that Blake can choose the name of that library. Also, unless I hear otherwise by 6/12, I will assume that this proposal is accepted conditional on PSARC receiving a final version of this "spec" which resolves all open issues. I will write an opinion document. Anyone with any comments to contribute to the opinion document should send them to me. Thanks. From sE.@alydar Thu Jun 6 16:53:42 1991 Return-Path: Received: from Eng.Sun.COM (zigzag) by opus.Eng.Sun.COM (4.1/SMI-4.1) id AA13884; Thu, 6 Jun 91 16:53:40 PDT Received: from opus.Eng.Sun.COM by Eng.Sun.COM (4.1/SMI-4.1) id AA14686; Thu, 6 Jun 91 16:53:32 PDT Received: from alydar.Eng.Sun.COM by opus.Eng.Sun.COM (4.1/SMI-4.1) id AA13881; Thu, 6 Jun 91 16:53:31 PDT Received: from window.Eng.Sun.COM by alydar.Eng.Sun.COM (4.1/SMI-4.1) id AA12741; Thu, 6 Jun 91 16:53:25 PDT Date: Thu, 6 Jun 91 16:53:25 PDT From: sE.@alydar (Steve E.) Message-Id: <9106062353.AA12741@alydar.Eng.Sun.COM> To: plat.sw.arc@opus, S.@datsun Subject: Re: 64-bit file offset proposal Cc: blake@central Status: RO > Blake, I don't think we ever decided which library should contain the > llseek() interface. PSARC, I believe our default position would be to > put this in a separate library. Let me know by 6/12 whether: lseek is in libc. Putting the long version there makes the most sense to me unless we are going to start a trend of putting this type stuff in liblc (long c). Having a default position of putting new stuff in separate libraries seems to run counter to what I understand to be the draw backs of lots of dynamic libraries on executable startup. Is this notion to avoid bumping minor numbers on libc? But, I'm probably missing some edict about library assignment. sE. From blake@Central Thu Jun 6 19:56:44 1991 Return-Path: Received: from Eng.Sun.COM (zigzag) by opus.Eng.Sun.COM (4.1/SMI-4.1) id AA14026; Thu, 6 Jun 91 19:56:41 PDT Received: from opus.Eng.Sun.COM by Eng.Sun.COM (4.1/SMI-4.1) id AA19280; Thu, 6 Jun 91 19:56:39 PDT Received: from Eng.Sun.COM (exodus) by opus.Eng.Sun.COM (4.1/SMI-4.1) id AA14023; Thu, 6 Jun 91 19:56:32 PDT Received: from Central.Sun.COM (central.Central.Sun.COM) by Eng.Sun.COM (4.1/SMI-4.1) id AA13803; Thu, 6 Jun 91 19:56:31 PDT Received: from rmtc.Central.Sun.COM by Central.Sun.COM (4.1/SMI-4.1) id AA00911; Thu, 6 Jun 91 21:56:29 CDT Received: by rmtc.Central.Sun.COM (4.1/SMI-4.1-900117) id AA18332; Thu, 6 Jun 91 20:56:29 MDT Date: Thu, 6 Jun 91 20:56:29 MDT From: blake@Central (Blake L.) Message-Id: <9106070256.AA18332@rmtc.Central.Sun.COM> To: plat.sw.arc@opus.Eng.Sun.COM Subject: Re: 64-bit file offset proposal Status: RO > lseek is in libc. Putting the long version there makes the most > sense to me unless we are going to start a trend of putting this > type stuff in liblc (long c). Yes, that was my thinking too. I intended to put llseek in libc and treat it like any other system call. Are we enjoined from doing this for some reason? From G.@speed Thu Jun 6 20:22:21 1991 Return-Path: Received: from Eng.Sun.COM (zigzag) by opus.Eng.Sun.COM (4.1/SMI-4.1) id AA14050; Thu, 6 Jun 91 20:22:19 PDT Received: from opus.Eng.Sun.COM by Eng.Sun.COM (4.1/SMI-4.1) id AA19589; Thu, 6 Jun 91 20:22:15 PDT Received: from speed.Eng.Sun.COM by opus.Eng.Sun.COM (4.1/SMI-4.1) id AA14047; Thu, 6 Jun 91 20:22:13 PDT Received: from opus.Eng.Sun.COM by speed.Eng.Sun.COM (4.1/SMI-4.1) id AA07662; Thu, 6 Jun 91 20:22:10 PDT Date: Thu, 6 Jun 91 20:22:10 PDT From: G.@speed (Rob G.) Message-Id: <9106070322.AA07662@speed.Eng.Sun.COM> To: blake@Central, plat.sw.arc@opus.Eng.Sun.COM Subject: Re: 64-bit file offset proposal Status: RO >> lseek is in libc. Putting the long version there makes the most >> sense to me unless we are going to start a trend of putting this >> type stuff in liblc (long c). > >Yes, that was my thinking too. I intended to put llseek in libc and >treat it like any other system call. Are we enjoined from doing this >for some reason? We have consciously deprecated the historical practice of making the C library a cesspool of functionality, thereby obscuring it's interface and leading to a programming environment hostile to outward portability. Thus, in 5.0, the RPC functionality is not present in the C library, neither is all the YP access code. Sun "value-added" stuff is encouraged to go elsewhere. To force "llseek" to follow this principle, however, seems to be an excessive application of the rule. However, my mind is sufficiently worn out at the moment that I'm struggling with my attempts to explain why. So, I'll just say "this seems wrong" and I'll try to explain better at a later time. From blake@Central Fri Jun 7 20:29:41 1991 Return-Path: Received: from speed.Eng.Sun.COM by opus.Eng.Sun.COM (4.1/SMI-4.1) id AA15369; Fri, 7 Jun 91 20:29:38 PDT Received: from Eng.Sun.COM (exodus) by speed.Eng.Sun.COM (4.1/SMI-4.1) id AA11074; Fri, 7 Jun 91 20:29:37 PDT Received: from Central.Sun.COM (central.Central.Sun.COM) by Eng.Sun.COM (4.1/SMI-4.1) id AA18625; Fri, 7 Jun 91 20:29:37 PDT Received: from rmtc.Central.Sun.COM by Central.Sun.COM (4.1/SMI-4.1) id AA18446; Fri, 7 Jun 91 22:29:35 CDT Received: by rmtc.Central.Sun.COM (4.1/SMI-4.1-900117) id AA06028; Fri, 7 Jun 91 21:29:33 MDT Date: Fri, 7 Jun 91 21:29:33 MDT From: blake@Central (Blake L.) Message-Id: <9106080329.AA06028@rmtc.Central.Sun.COM> To: G.@speed.Eng.Sun.COM Subject: Re: 64-bit file offset proposal Cc: plat.sw.arc@opus.Eng.Sun.COM Status: RO > So, I'll just say "this seems wrong" and I'll try to explain better > at a later time. Forgive me for being a bit thick, but I can't quite make out which seems wrong, putting llseek into libc or putting it someplace else. I applaud the "no more cesspools" principle, but since llseek is a just a system call that does nothing but trap into the kernel, I thought that it fit most naturally with all the other system calls that do the same. However, one library is as good as another---let me know what you prefer. From S.@datsun Wed Jun 12 17:05:28 1991 Return-Path: Received: from speed.Eng.Sun.COM by opus.Eng.Sun.COM (4.1/SMI-4.1) id AA19310; Wed, 12 Jun 91 17:05:24 PDT Received: from datsun.Eng.Sun.COM by speed.Eng.Sun.COM (4.1/SMI-4.1) id AA26322; Wed, 12 Jun 91 17:05:21 PDT Received: by datsun.Eng.Sun.COM (4.1/SMI-4.1) id AA24364; Wed, 12 Jun 91 17:05:20 PDT Date: Wed, 12 Jun 91 17:05:20 PDT From: S.@datsun (Bill S.) Message-Id: <9106130005.AA24364@datsun.Eng.Sun.COM> To: blake@Central, G.@speed, plat.sw.arc@opus.Eng.Sun.COM Subject: Re: 64-bit file offset proposal Status: RO We more or less decided today that llseek should be in libc. Rob will tell me why and I will include it in the opinion document I will write. (If I don't hear from Rob in time, I will make it up.) All we need now is for Blake to provide us with the resolution of the other open issues and for me to write the opinion document. From blake@Central Wed Jun 12 17:51:59 1991 Return-Path: Received: from Eng.Sun.COM (zigzag) by opus.Eng.Sun.COM (4.1/SMI-4.1) id AA19380; Wed, 12 Jun 91 17:51:57 PDT Received: from opus.Eng.Sun.COM by Eng.Sun.COM (4.1/SMI-4.1) id AA00273; Wed, 12 Jun 91 17:51:33 PDT Received: from Eng.Sun.COM (exodus) by opus.Eng.Sun.COM (4.1/SMI-4.1) id AA19377; Wed, 12 Jun 91 17:51:53 PDT Received: from Central.Sun.COM (central.Central.Sun.COM) by Eng.Sun.COM (4.1/SMI-4.1) id AA29288; Wed, 12 Jun 91 17:51:52 PDT Received: from rmtc.Central.Sun.COM by Central.Sun.COM (4.1/SMI-4.1) id AA19548; Wed, 12 Jun 91 19:51:50 CDT Received: from bongo.Central.Sun.COM.Central.Sun.COM by rmtc.Central.Sun.COM (4.1/SMI-4.1-900117) id AA00945; Wed, 12 Jun 91 18:51:49 MDT Date: Wed, 12 Jun 91 18:51:49 MDT From: blake@Central (Blake L.) Message-Id: <9106130051.AA00945@rmtc.Central.Sun.COM> To: Bill.S.@Eng Subject: Re: 64-bit file offset proposal Cc: plat.sw.arc@opus.Eng.Sun.COM Status: RO > All we need now is for Blake to provide us with the resolution of the > other open issues and for me to write the opinion document. I'm aware of three open issues, viz. 1) Steve Chessin objected to my symbol and property names 2) Matt Jacob suggested using a bit in the dev_ops structure instead of a property 3) Matt pointed out that the ipi driver should support 64-bit offsets Are there any others? From S.@datsun Wed Jun 12 23:58:44 1991 Return-Path: Received: from Eng.Sun.COM (zigzag) by opus.Eng.Sun.COM (4.1/SMI-4.1) id AA19795; Wed, 12 Jun 91 23:58:42 PDT Received: from opus.Eng.Sun.COM by Eng.Sun.COM (4.1/SMI-4.1) id AA04487; Wed, 12 Jun 91 23:58:20 PDT Received: from datsun.Eng.Sun.COM by opus.Eng.Sun.COM (4.1/SMI-4.1) id AA19792; Wed, 12 Jun 91 23:58:37 PDT Received: by datsun.Eng.Sun.COM (4.1/SMI-4.1) id AA25488; Wed, 12 Jun 91 23:58:35 PDT Date: Wed, 12 Jun 91 23:58:35 PDT From: S.@datsun (Bill S.) Message-Id: <9106130658.AA25488@datsun.Eng.Sun.COM> To: blake@Central Subject: Re: 64-bit file offset proposal Cc: plat.sw.arc@opus.Eng.Sun.COM Status: RO I'm aware of three open issues, viz. 1) Steve Chessin objected to my symbol and property names 2) Matt Jacob suggested using a bit in the dev_ops structure instead of a property 3) Matt pointed out that the ipi driver should support 64-bit offsets Are there any others? No. I'm assuming that the latter issue is closed (and in any event is non-architectural) and that you will deliver an IPI driver with the 64 bit support. From G.@speed Sun Jun 16 11:31:07 1991 Return-Path: Received: from speed.Eng.Sun.COM by opus.Eng.Sun.COM (4.1/SMI-4.1) id AA22908; Sun, 16 Jun 91 11:29:48 PDT Received: from opus.Eng.Sun.COM by speed.Eng.Sun.COM (4.1/SMI-4.1) id AA08098; Sun, 16 Jun 91 11:28:30 PDT Date: Sun, 16 Jun 91 11:28:30 PDT From: G.@speed (Rob G.) Message-Id: <9106161828.AA08098@speed.Eng.Sun.COM> To: blake@Central, G.@speed.Eng.Sun.COM Subject: Re: 64-bit file offset proposal Cc: plat.sw.arc@opus.Eng.Sun.COM Status: R >Forgive me for being a bit thick, but I can't quite make out which >seems wrong, putting llseek into libc or putting it someplace else. No problem -- sorry about the ambiguity. What seems wrong is *NOT* putting it in libc. >I applaud the "no more cesspools" principle, but since llseek is a >just a system call that does nothing but trap into the kernel, I >thought that it fit most naturally with all the other system calls >that do the same. I agree -- and what I'm trying to figure out whether there's a general principle at work here (and what that is) and if so, how to say that.