SUMMARY: This is a proposal to add support for getting information about holes in sparse files. This is primarily to enable efficient backups, but knowledge of holes in files could be useful to other applications. PROBLEM: Current Solaris backup utilities (e.g. tar, cpio) read files from beginning to end and write out the file in a backup format. When reading the holes in sparse files zeros are returned. So a file with just 1K bytes at a 1TB offset would occupy only of 1K bytes (plus metadata), but when backed up occupy 1TB (plus metadata). It is possible to check for zeros and compress the file but the whole file still needs to be read. PROPOSED SOLUTION: We are not proposing to change any existing backup utility or archive (backup) format. The format of existing archive files like tar and cpio is standardised to allow use between different operating systems and their different releases. The proposal here is to define a change to lseek to retrieve information about holes in sparse files. This allows for efficient new backup utilities to be produced (for example in ZFS). A micro/patchbinding is requested. lseek changes: ------------- Here's the gist of the proposed change to the lseek(2) and llseek(2) man pages: New modes: o If whence is SEEK_HOLE, the offset of the start of the next hole greater than or equal to the supplied offset is returned. However, the current file pointer is not changed. See text below for the definition of a hole. o If whence is SEEK_HOLE, the file pointer is set to the start of the next hole greater than or equal to the supplied offset. o If whence is SEEK_DATA, the file pointer is set to the start of the next non-hole file region greater than or equal to the supplied offset. New text: A "hole" is defined as a contiguous range of bytes in a file, all having the value of zero, but not all zeros in a file are guaranteed to be represented as holes returned with SEEK_HOLE. Filesystems are allowed to expose ranges of zeros via SEEK_HOLE but not required to. Applications may use SEEK_HOLE to optimise their behaviour for ranges of zeros, but must not depend on it to find all such ranges in a file. For every data region there is a hole at the end of it. This allows for easy programming and implies a virtual hole exists at the end of the file. For filesystems that do not supply information about holes, the file will be represented as one entire data region. New errors: ENXIO For SEEK_DATA there are no more data regions past the supplied offset. For SEEK_HOLE there are no more holes past the supplied offset. Stability level Stable new ioctls ---------- The lseek system call seems the logical user interface for such support. The kernel largely handles the lseek system call only calling the filesystem VOP_SEEK routine to do additional checks passing only the vnode, old offset and a pointer to the new offset. Changing the VOP_SEEK arguments would not be possible, so for SEEK_DATA and SEEK_HOLE a new VOP_IOCTL call is made by the kernel lseek system call handler. This ioctl is defined as: ioctl(int fd, int _FIO_SEEK_DATA, offset_t *offsetp); ioctl(int fd, int _FIO_SEEK_HOLE, offset_t *offsetp); Both _FIO_SEEK_HOLE and _FIO_SEEK_DATA will be defined as usual in sys/filio.h. The semantics for the ioctl command and are as defined in lseek/llseek. The offsetp argument point to an offset_t containing the input offset. On successful return the offset_t holds the resulting offset of the data or hole. It is possible to use this ioctl directly without using lseek. Error returns for this ioctl are as follows: EFAULT The supplied offsetp points to an illegal address for reading or writing. ENOTTY This ioctl is not supported by this filesystem. ENXIO For _FIO_SEEK_HOLE there are no more data regions past the supplied offset. For _FIO_SEEK_DATA there are no more holes past the supplied offset. Both ENOTTY and EFAULT are caught and handled by the kernel lseek system call handler and not passed back to user land. Stability level Contracted Consolidation Private This allows other file systems (e.g. Veritas) to play. new pathconf variable --------------------- A new pathconf variable _PC_MIN_HOLE_SIZE is defined. If a filesystem supports the reporting of holes (see lseek(2)) then pathconf/fpathconf will return a positive number. This number represents the minimum hole size returned in bytes and furthermore offsets of holes returned will be aligned to the the same value. A special value of 1 is returned if the filesystem chooses not to specify the minimum hole size but still reports holes. Stability level Stable. SUPPLEMENTAL INFORMATION 1. Programming example Using the new interfaces, a utility to efficiently copy a file could do the following. Note, this works for both filesystems that supply and do not supply information about holes. off_t off = 0, data, hole; for (;;) { data = lseek(fd, off, SEEK_DATA); if (data == -1) break; hole = lseek(fd, data, SEEK_HOLE); off = hole; } 2. Although there are no current users of this interface, it will highly desirable for ZFS. Other filesystems (e.g. UFS or QFS) should also support it.