From Tim.Haley.-.Sun.Microsystem@sun.com Mon Oct 1 20:27:56 2007 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l923Rufa011270 for ; Mon, 1 Oct 2007 20:27:56 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l923OkYG004422 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Tue, 2 Oct 2007 04:24:47 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JP900G05LHB5O00@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 01 Oct 2007 20:24:47 -0700 (PDT) Received: from spidey.Central.Sun.COM ([172.20.25.27]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JP9007M7LHBZW90@nwk-avmta-2.sfbay.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 01 Oct 2007 20:24:47 -0700 (PDT) Received: from spidey.Central.Sun.COM (localhost [127.0.0.1]) by spidey.Central.Sun.COM (8.14.1+Sun/8.14.1) with ESMTP id l923OkM6028575; Mon, 01 Oct 2007 21:24:46 -0600 (MDT) Received: (from timh@localhost) by spidey.Central.Sun.COM (8.14.1+Sun/8.14.1/Submit) id l923OkCx028573; Mon, 01 Oct 2007 21:24:46 -0600 (MDT) Date: Mon, 01 Oct 2007 21:24:46 -0600 (MDT) From: Tim Haley - Sun Microsystem Subject: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] To: PSARC-ext@sun.com Message-id: <200710020324.l923OkCx028573@spidey.Central.Sun.COM> Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 Status: RO Content-Length: 5417 I am sponsoring this fast-track for Alok Aggarwal. Requested binding is micro, timeout 10/08/2007. This information is Copyright 2007 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: lofi(7D) compression support 1.2. Name of Document Author/Supplier: Author: Alok Aggarwal 1.3 Date of This Document: 01 October, 2007 4. Technical Description clofi.psarc 1.2 07/09/28 SMI Template Version: @(#)sac_nextcase 1.2 09/28/07 SMI 1. Introduction 1.1 Project/Component Working Name: lofi(7D) compression support 1.2 Name of Document Author/Supplier Author: Alok Aggarwal 1.3 Date of this Document 28 September, 2007 4. Technical Description There are cases where certain files containing a file system image need to be compressed. A classic use case is that of a LiveCD. Compression support in lofi is required to make LiveCDs practical. It allows for more data to be packed on the media and improves performance by reducing head seek times. This project aims to extend lofi(7D) to support reading from a compressed file via on-the-fly decompression. This project will also add extensions to the lofiadm(1M) command to provide for the ability to compress a file in a suitable format and to decompress it entirely. Design Overview The basic design is to compress a file in fixed size segments and store the segments sequentially. The information about where each of the segments start is maintained in an index. This index along with a header that contains the compression algorithm used, segment size, number of segments and the uncompressed size of the last segment is stored at the start of the file. When a compressed file is mapped, the header containing the compression metadata is read in and stored in the lofi state structure. The compression metadata thus read is then used to decompress the requested data when a read request comes in. If the decompression fails for some reason, an EIO is returned. On disk format A graphic representation of the compression header stored at the start of the file can be found under materials/ondisk_format.png It should be noted that this header information is always stored in network byte order. This allows for endian neutral filesystems such as ZFS to be compressed on a little endian machine, and subsequently be uncompressed on a big endian machine; and vice-versa. On the fly decompression A graphic representation of reading from compressed versus regular files can be found under materials/compress_io.jpg Compression Algorithms The lofi(7D) driver will support compression algorithms present in the Solaris kernel. Currently, lzjb and gzip are the two algorithms available. The lzjb algorithm provides very fast compression as compared to gzip. gzip, even though it is slower than lzjb, is fast enough for the typical use case of a LiveCD. In addition, gzip provides a much better compression ratio as compared to lzjb (on the order of ~25% in some cases) and it allows for more data to be packed on to the media. These characteristics make gzip well suited for a LiveCD. Thus, gzip will be the only supported algorithm at this time. The changes being made as part of this project will allow for compression algorithms to be plugged into lofi with relative ease as more of these algorithms are made available in the Solaris kernel. Command Changes lofiadm(1M) will take the following additional arguments - -C Compress the file with a specified compression algorithm or with the default compression algorithm if one isn't specified. The only supported compression algorithm at this time is gzip which is also the default. -s The segment size to use to divide the file being compressed. If no segment size is specified, a default segment size of 128k is used. -U Uncompress a compressed file The output of lofiadm(1M) will also change to add another field to reflect if the mapped file(s) is a compressed file or a regular file. Examples: 1. Compress a file with gzip # lofiadm -C gzip /export/home/solaris.iso 2. Compress a file with gzip and compress it in 64k chunks # lofiadm -C gzip -s 64k /export/home/solaris.iso 3. Check if there are any compressed files that are mapped # lofiadm Block Device File Type /dev/lofi/1 /export/home/solaris.iso Compressed (gzip) /dev/lofi/2 /export/home/regular.iso Regular 4. Uncompress a compressed file # lofiadm -U /export/home/solaris.iso References PSARC/2007/492 - Project Indiana http://sac.eng/PSARC/2007/492/discussion.materials/Indiana.txt PSARC/1999/463 - lofi http://sac.sfbay/Archives/CaseLog/arc/PSARC/1999/463 On the Fly Decompression in BeleniX http://www.genunix.org/distributions/belenix_site/?q=compression gzip for ZFS update http://blogs.sun.com/ahl/entry/gzip_for_zfs_update 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From gww@eng.sun.com Mon Oct 1 22:13:55 2007 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l925DtZr013831 for ; Mon, 1 Oct 2007 22:13:55 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l9259qrx061231; Mon, 1 Oct 2007 23:09:52 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JP900L0JQDY8400@brm-avmta-1.central.sun.com>; Mon, 01 Oct 2007 23:10:46 -0600 (MDT) Received: from engmail3mpk.sfbay.Sun.COM ([129.146.11.26]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JP900HWHQDYPZ10@brm-avmta-1.central.sun.com>; Mon, 01 Oct 2007 23:10:46 -0600 (MDT) Received: from marduk.eng.sun.com (marduk.SFBay.Sun.COM [129.146.108.224]) by engmail3mpk.sfbay.Sun.COM (8.13.6+Sun/8.13.6/ENSMAIL,v2.2) with ESMTP id l925AjEf014414; Mon, 01 Oct 2007 22:10:45 -0700 (PDT) Received: from marduk.eng.sun.com (localhost [127.0.0.1]) by marduk.eng.sun.com (8.13.6+Sun/8.12.11) with ESMTP id l925CUqf000021; Mon, 01 Oct 2007 22:12:30 -0700 (PDT) Received: (from gww@localhost) by marduk.eng.sun.com (8.13.6+Sun/8.12.11/Submit) id l925CUqn000020; Mon, 01 Oct 2007 22:12:30 -0700 (PDT) Date: Mon, 01 Oct 2007 22:12:30 -0700 (PDT) From: Gary Winiger Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] To: psarc-ext@sun.com, Tim.Haley@sun.com Message-id: <200710020512.l925CUqn000020@marduk.eng.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 Status: RO Content-Length: 455 > I am sponsoring this fast-track for Alok Aggarwal. > Requested binding is micro, timeout 10/08/2007. Just checking that Micro (which today equates to Minor) is what is meant. That is there is no intent to backport to any marketing release in current distrobution. I can perhaps see why that might be. IMO, this is a case were Micro seems justified as it could leave data on a compressed image unrecoverable if the "patch" were removed. Gary.. From dwc@spartan.eng.sun.com Tue Oct 2 03:06:23 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92A6M8F027295 for ; Tue, 2 Oct 2007 03:06:23 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l92A3AvR025131; Tue, 2 Oct 2007 18:03:12 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPA00J2Z3XA9800@brm-avmta-1.central.sun.com>; Tue, 02 Oct 2007 04:03:10 -0600 (MDT) Received: from spartan.SFBay.Sun.COM ([129.146.226.64]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPA008Q43X9CQ60@brm-avmta-1.central.sun.com>; Tue, 02 Oct 2007 04:03:09 -0600 (MDT) Received: from spartan.SFBay.Sun.COM (localhost [127.0.0.1]) by spartan.SFBay.Sun.COM (8.13.6+Sun/8.13.6) with ESMTP id l92A39Sp014979; Tue, 02 Oct 2007 03:03:09 -0700 (PDT) Received: (from dwc@localhost) by spartan.SFBay.Sun.COM (8.13.6+Sun/8.13.6/Submit) id l92A393T014978; Tue, 02 Oct 2007 03:03:09 -0700 (PDT) Date: Tue, 02 Oct 2007 03:03:09 -0700 (PDT) From: Don Cragun Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] To: Alok.Aggarwal@sun.com Cc: PSARC-ext@sun.com Message-id: <200710021003.l92A393T014978@spartan.SFBay.Sun.COM> Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 Status: RO Content-Length: 1218 >Command Changes > lofiadm(1M) will take the following additional arguments - > > -C Compress the file with a specified compression algorithm > or with the default compression algorithm if one isn't > specified. The only supported compression algorithm at this > time is gzip which is also the default. > > -s The segment size to use to divide the file being compressed. > If no segment size is specified, a default segment size of > 128k is used. > Optional option arguments are not allowed by the utility syntax guidelines, and all of your examples show option arguments for both of these options even though they use the default values. Please drop the statements in both of these option descriptions that specify default values and specify that the option arguments are optional. Since the -s segment size option argument is not "just a number", please also clearly specify the formats accepted and the allowed range of segment sizes. - Don ... ... ... > > Examples: > > 1. Compress a file with gzip > > # lofiadm -C gzip /export/home/solaris.iso > > 2. Compress a file with gzip and compress it in 64k chunks > > # lofiadm -C gzip -s 64k /export/home/solaris.iso ... ... ... From dwc@spartan.eng.sun.com Tue Oct 2 03:26:05 2007 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92AQ5BJ028028 for ; Tue, 2 Oct 2007 03:26:05 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l92AMuwA024715; Tue, 2 Oct 2007 03:22:57 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPA00L054U7HO00@nwk-avmta-2.sfbay.sun.com>; Tue, 02 Oct 2007 03:22:55 -0700 (PDT) Received: from spartan.SFBay.Sun.COM ([129.146.226.64]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPA00J5D4U6OT10@nwk-avmta-2.sfbay.sun.com>; Tue, 02 Oct 2007 03:22:54 -0700 (PDT) Received: from spartan.SFBay.Sun.COM (localhost [127.0.0.1]) by spartan.SFBay.Sun.COM (8.13.6+Sun/8.13.6) with ESMTP id l92AMswu015020; Tue, 02 Oct 2007 03:22:54 -0700 (PDT) Received: (from dwc@localhost) by spartan.SFBay.Sun.COM (8.13.6+Sun/8.13.6/Submit) id l92AMsAW015019; Tue, 02 Oct 2007 03:22:54 -0700 (PDT) Date: Tue, 02 Oct 2007 03:22:54 -0700 (PDT) From: Don Cragun Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] To: Alok.Aggarwal@sun.com Cc: PSARC-ext@sun.com Message-id: <200710021022.l92AMsAW015019@spartan.SFBay.Sun.COM> Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 Status: RO Content-Length: 1948 >From psarc-intern-list-request@sun.com Tue Oct 2 03:04:11 2007 >Date: Tue, 02 Oct 2007 03:03:09 -0700 (PDT) >From: Don Cragun >Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout > 10/08/2007] >To: Alok.Aggarwal@sun.com >Cc: PSARC-ext@sun.com >MIME-version: 1.0 >Content-transfer-encoding: 7BIT >X-PMX-Version: 5.2.0.264296 >X-PMX-Version: 5.2.0.264296 > >>Command Changes >> lofiadm(1M) will take the following additional arguments - >> >> -C Compress the file with a specified compression algorithm >> or with the default compression algorithm if one isn't >> specified. The only supported compression algorithm at this >> time is gzip which is also the default. >> >> -s The segment size to use to divide the file being compressed. >> If no segment size is specified, a default segment size of >> 128k is used. >> > >Optional option arguments are not allowed by the utility syntax >guidelines, and all of your examples show option arguments for both of >these options even though they use the default values. Please drop the >statements in both of these option descriptions that specify default >values and specify that the option arguments are optional. I think the last sentence above could be misinterpreted. I mean: Please drop the statement in both of these option descriptions that specify default values and drop the statement in both of these option descriptions that talk about what happens if the option argument is not specified. > >Since the -s segment size option argument is not "just a number", >please also clearly specify the formats accepted and the allowed range >of segment sizes. > > - Don > > ... ... ... >> >> Examples: >> >> 1. Compress a file with gzip >> >> # lofiadm -C gzip /export/home/solaris.iso >> >> 2. Compress a file with gzip and compress it in 64k chunks >> >> # lofiadm -C gzip -s 64k /export/home/solaris.iso > ... ... ... > From Darren.Moffat@sun.com Tue Oct 2 05:21:23 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92CLMn2001791 for ; Tue, 2 Oct 2007 05:21:22 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l92CI3Wt013371; Tue, 2 Oct 2007 20:18:13 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPA00503A6AE400@nwk-avmta-2.sfbay.sun.com>; Tue, 02 Oct 2007 05:18:10 -0700 (PDT) Received: from gmp-eb-mail-2.sun.com ([192.18.6.24]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPA00J0LA69P340@nwk-avmta-2.sfbay.sun.com>; Tue, 02 Oct 2007 05:18:10 -0700 (PDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-2-fe1.eu.sun.com [192.18.6.10]) by gmp-eb-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l92CI95A008839; Tue, 02 Oct 2007 12:18:09 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPA0080190VNL00@fe-emea-10.sun.com> (original mail from Darren.Moffat@Sun.COM); Tue, 02 Oct 2007 13:18:09 +0100 (BST) Received: from [129.156.173.21] by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPA00GNGA5YW610@fe-emea-10.sun.com>; Tue, 02 Oct 2007 13:17:58 +0100 (BST) Date: Tue, 02 Oct 2007 13:17:58 +0100 From: Darren J Moffat Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] In-reply-to: <200710020324.l923OkCx028573@spidey.Central.Sun.COM> Sender: Darren.Moffat@sun.com To: Tim Haley - Sun Microsystem Cc: PSARC-ext@sun.com Message-id: <470236F6.1090509@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200710020324.l923OkCx028573@spidey.Central.Sun.COM> User-Agent: Thunderbird 2.0.0.4 (X11/20070731) Status: RO Content-Length: 2076 DJM-0 I don't quite understand how this is used. It seems that the process is like this: Create some image file that has a filesystem on it, lets make it UFS. # mkfile 1g /path/to/backingfile.img # lofiadm -C gzip /path/to/backingfile.img # lofiadm -a /path/to/backingfile.img /dev/lofi/1 # mkfs -F ufs /dev/rlofi/1 ... # mount -F ufs /dev/lofi/1 /mnt Is that correct ? In otherwords unlike the crypto case, and what your man page synopsis suggests, the -C/-s arguments aren't things you pass in when you do a mapping with -a but instead they are auxiliary functions. DJM-1 How does this interact with lofi crypto ? (http://opensolaris.org/os/community/arc/caselog/2007/001/) Which happens first crypto or compression (hint it needs to be compression otherwise it is waste of CPU resource) ? However if my understanding of how -C and -U work it seems that the compression would be happening after encryption because by design it isn't on the fly but actually a distinct admin action (which for building LiveCD images is fine). DJM-2 What is the guidance for changing the segment size ? DJM-3 What happens if I make an lofi mapping with the default segment size and later map the same file with a different size ? DJM-4 What is the minimum segment size ? DJM-5 How big is the header section ? The project to add encryption to lofi stalled on exactly the issue of the adding metadata to the image. Since this project is adding a header section it seems to make sense that the crypto and compression projects share a metadata section. DJM-6 How big is the index section ? DJM-7 In the ZFS use of gzip it is possible to specify the level with gzip-6 being the default. I think this case should do the same. DJM-8 The encrypted lofi case also proposed to add a new column to the output to indicate if the mapping was encrypted, since this case also adds a new column, coordination is needed (I think we are okay here though). Process issue: http://opensolaris.org/os/community/arc/caselog/2007/569/ gives 404 at the moment. -- Darren J Moffat From Alok.Aggarwal@Sun.COM Tue Oct 2 06:16:47 2007 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92DGlIh002763 for ; Tue, 2 Oct 2007 06:16:47 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l92DDasE012962; Tue, 2 Oct 2007 14:13:36 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPA0080RCQNI000@nwk-avmta-2.sfbay.sun.com>; Tue, 02 Oct 2007 06:13:35 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPA00JXLCQNP170@nwk-avmta-2.sfbay.sun.com>; Tue, 02 Oct 2007 06:13:35 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l92DDZgP001932; Tue, 02 Oct 2007 13:13:35 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPA00A01CKWKT00@mail-amer.sun.com> (original mail from Alok.Aggarwal@Sun.COM); Tue, 02 Oct 2007 07:13:35 -0600 (MDT) Received: from sr1-ubrm-44 ([129.147.9.134]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPA003N3CQD3K30@mail-amer.sun.com>; Tue, 02 Oct 2007 07:13:25 -0600 (MDT) Date: Tue, 02 Oct 2007 07:13:25 -0600 (MDT) From: Alok Aggarwal Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] In-reply-to: <200710021003.l92A393T014978@spartan.SFBay.Sun.COM> Sender: Alok.Aggarwal@Sun.COM X-X-Sender: aalok@sr1-ubrm-44 To: Don Cragun Cc: Alok.Aggarwal@Sun.COM, PSARC-ext@Sun.COM Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200710021003.l92A393T014978@spartan.SFBay.Sun.COM> Status: RO Content-Length: 1225 On Tue, 2 Oct 2007, Don Cragun wrote: >> Command Changes >> lofiadm(1M) will take the following additional arguments - >> >> -C Compress the file with a specified compression algorithm >> or with the default compression algorithm if one isn't >> specified. The only supported compression algorithm at this >> time is gzip which is also the default. >> >> -s The segment size to use to divide the file being compressed. >> If no segment size is specified, a default segment size of >> 128k is used. >> > > Optional option arguments are not allowed by the utility syntax > guidelines, and all of your examples show option arguments for both of > these options even though they use the default values. Please drop the > statements in both of these option descriptions that specify default > values and specify that the option arguments are optional. I'll clarify. > Since the -s segment size option argument is not "just a number", > please also clearly specify the formats accepted and the allowed range > of segment sizes. Since lofi(7D) works on block boundaries, any value that is a multiple of 512 bytes is theoretically an acceptable value for the segment size. I'll add verbiage around this. Alok From Alok.Aggarwal@sun.com Tue Oct 2 07:37:21 2007 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92EbKFv005141 for ; Tue, 2 Oct 2007 07:37:20 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l92EYA8m029992; Tue, 2 Oct 2007 07:34:12 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPA00E0HGH0M200@brm-avmta-1.central.sun.com>; Tue, 02 Oct 2007 08:34:12 -0600 (MDT) Received: from brmea-mail-3.sun.com ([192.18.98.34]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPA009MIGGZU270@brm-avmta-1.central.sun.com>; Tue, 02 Oct 2007 08:34:12 -0600 (MDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-3.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l92EYB2p014295; Tue, 02 Oct 2007 14:34:11 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPA00H01FARBM00@mail-amer.sun.com> (original mail from Alok.Aggarwal@Sun.COM); Tue, 02 Oct 2007 08:34:11 -0600 (MDT) Received: from sr1-ubrm-44 ([129.147.9.134]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPA008RRGGT0K90@mail-amer.sun.com>; Tue, 02 Oct 2007 08:34:05 -0600 (MDT) Date: Tue, 02 Oct 2007 08:34:04 -0600 (MDT) From: Alok Aggarwal Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] In-reply-to: Sender: Alok.Aggarwal@sun.com X-X-Sender: aalok@sr1-ubrm-44 To: Darren.Moffat@sun.com Cc: PSARC-ext@sun.com, Tim Haley - Sun Microsystem Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: Status: RO Content-Length: 3500 On Tue, 2 Oct 2007, Darren Moffat wrote: > DJM-0 I don't quite understand how this is used. > > It seems that the process is like this: > > Create some image file that has a filesystem on it, lets make it UFS. > > # mkfile 1g /path/to/backingfile.img > # lofiadm -C gzip /path/to/backingfile.img > # lofiadm -a /path/to/backingfile.img > /dev/lofi/1 > # mkfs -F ufs /dev/rlofi/1 > ... > # mount -F ufs /dev/lofi/1 /mnt > > Is that correct ? > > In otherwords unlike the crypto case, and what your man page synopsis > suggests, the -C/-s arguments aren't things you pass in when you do a mapping > with -a but instead they are auxiliary functions. Correct. > DJM-1 How does this interact with lofi crypto ? > > (http://opensolaris.org/os/community/arc/caselog/2007/001/) > > Which happens first crypto or compression (hint it needs to be compression > otherwise it is waste of CPU resource) ? > > However if my understanding of how -C and -U work it seems that the > compression would be happening after encryption because by design it isn't on > the fly but actually a distinct admin action (which for building LiveCD > images is fine). I need to think about this aspect some more, I'll come back to this. > DJM-2 What is the guidance for changing the segment size ? Theoretically, segment size can be a multiple of 512 bytes. For the LiveCD case we've seen that a segment size of 128k gives us the best boot time performance. For cases other than LiveCD, it would probably require some experimentation to arrive at an optimal segment size. > DJM-3 What happens if I make an lofi mapping with the default segment size > and later map the same file with a different size ? Segment size can only be specified at the time of compressing a file. That segment size is simply read in while mapping the file; you can't change it while mapping it in. > DJM-4 What is the minimum segment size ? That would be 512 bytes. > DJM-5 How big is the header section ? It is 12 bytes plus the length of compression algorithm used. So, for gzip, it would be 16 bytes (12 + 4). > The project to add encryption to lofi stalled on exactly the issue of the > adding metadata to the image. Since this project is adding a header section > it seems to make sense that the crypto and compression projects share a > metadata section. We should talk on this point, it does make sense to have a shared metadata section. > DJM-6 How big is the index section ? It depends on the size of the file being compressed and the segment size used. As an example, for a 100 Mb file with 128k segment size, the number of index entries would be 781 where each index entry is a uin64_t. This results in the index section to be 6284 bytes. > DJM-7 In the ZFS use of gzip it is possible to specify the level with gzip-6 > being the default. I think this case should do the same. Can you think of a use case for providing gzip-6? For the case of a LiveCD, we've found gzip-9 to be the most optimal in terms of providing the best compression ratio. gzip-9 is what is used in this project internally. > DJM-8 The encrypted lofi case also proposed to add a new column to the output > to indicate if the mapping was encrypted, since this case also adds a new > column, coordination is needed (I think we are okay here though). Yes, we should coordinate on this point too. > Process issue: > > http://opensolaris.org/os/community/arc/caselog/2007/569/ gives 404 at the > moment. I'll defer this to Tim. Alok From gww@eng.sun.com Tue Oct 2 07:47:59 2007 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92Elx6Z005512 for ; Tue, 2 Oct 2007 07:47:59 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l92EipMb017561; Tue, 2 Oct 2007 07:44:51 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPA0038JGYPDR00@nwk-avmta-1.sfbay.Sun.COM>; Tue, 02 Oct 2007 07:44:49 -0700 (PDT) Received: from engmail3mpk.sfbay.Sun.COM ([129.146.11.26]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPA00LP1GYH5C70@nwk-avmta-1.sfbay.Sun.COM>; Tue, 02 Oct 2007 07:44:42 -0700 (PDT) Received: from marduk.eng.sun.com (marduk.SFBay.Sun.COM [129.146.108.224]) by engmail3mpk.sfbay.Sun.COM (8.13.6+Sun/8.13.6/ENSMAIL,v2.2) with ESMTP id l92EieSu024073; Tue, 02 Oct 2007 07:44:40 -0700 (PDT) Received: from marduk.eng.sun.com (localhost [127.0.0.1]) by marduk.eng.sun.com (8.13.6+Sun/8.12.11) with ESMTP id l92EkP2A000477; Tue, 02 Oct 2007 07:46:25 -0700 (PDT) Received: (from gww@localhost) by marduk.eng.sun.com (8.13.6+Sun/8.12.11/Submit) id l92EkPT3000476; Tue, 02 Oct 2007 07:46:25 -0700 (PDT) Date: Tue, 02 Oct 2007 07:46:25 -0700 (PDT) From: Gary Winiger Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] To: Darren.Moffat@sun.com, Alok.Aggarwal@sun.com Cc: PSARC-ext@sun.com, Tim.Haley.-.Sun.Microsystem@sun.com Message-id: <200710021446.l92EkPT3000476@marduk.eng.sun.com> Content-transfer-encoding: 7BIT X-Sun-Charset: US-ASCII X-PMX-Version: 5.2.0.264296 Status: RO Content-Length: 720 > > DJM-1 How does this interact with lofi crypto ? > > > > (http://opensolaris.org/os/community/arc/caselog/2007/001/) > > > > Which happens first crypto or compression (hint it needs to be compression > > otherwise it is waste of CPU resource) ? > > > > However if my understanding of how -C and -U work it seems that the > > compression would be happening after encryption because by design it isn't on > > the fly but actually a distinct admin action (which for building LiveCD > > images is fine). > > I need to think about this aspect some more, I'll come > back to this. Presumably compress first encrypt second. Decrypt first uncompress second. Compressing noise doesn't seem like a good plan. Gary.. From Nicolas.Williams@sun.com Tue Oct 2 08:41:27 2007 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92FfRj0006522 for ; Tue, 2 Oct 2007 08:41:27 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l92FcCgu021388; Tue, 2 Oct 2007 08:38:18 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPA00I5GJFSXB00@brm-avmta-1.central.sun.com>; Tue, 02 Oct 2007 09:38:16 -0600 (MDT) Received: from binky.Central.Sun.COM ([129.153.128.104]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPA009G5JFQTUE0@brm-avmta-1.central.sun.com>; Tue, 02 Oct 2007 09:38:14 -0600 (MDT) Received: from binky.Central.Sun.COM (localhost [127.0.0.1]) by binky.Central.Sun.COM (8.14.1+Sun/8.14.1) with ESMTP id l92FcEbE017748; Tue, 02 Oct 2007 10:38:14 -0500 (CDT) Received: (from nw141292@localhost) by binky.Central.Sun.COM (8.14.1+Sun/8.14.1/Submit) id l92FcEZp017747; Tue, 02 Oct 2007 10:38:14 -0500 (CDT) Date: Tue, 02 Oct 2007 10:38:14 -0500 From: Nicolas Williams Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] In-reply-to: To: Alok Aggarwal Cc: Darren.Moffat@sun.com, PSARC-ext@sun.com, Tim Haley - Sun Microsystem Message-id: <20071002153813.GW11376@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.2.0.264296 References: X-Authentication-warning: binky.Central.Sun.COM: nw141292 set sender to Nicolas.Williams@sun.com using -f User-Agent: Mutt/1.5.7i Status: RO Content-Length: 3263 On Tue, Oct 02, 2007 at 08:34:04AM -0600, Alok Aggarwal wrote: > On Tue, 2 Oct 2007, Darren Moffat wrote: > >DJM-0 I don't quite understand how this is used. > > > >It seems that the process is like this: > > > >Create some image file that has a filesystem on it, lets make it UFS. > > > ># mkfile 1g /path/to/backingfile.img > ># lofiadm -C gzip /path/to/backingfile.img > ># lofiadm -a /path/to/backingfile.img > >/dev/lofi/1 > ># mkfs -F ufs /dev/rlofi/1 > >... > ># mount -F ufs /dev/lofi/1 /mnt > > > >Is that correct ? > > > >In otherwords unlike the crypto case, and what your man page synopsis > >suggests, the -C/-s arguments aren't things you pass in when you do a > >mapping with -a but instead they are auxiliary functions. > > Correct. Further, this is all read-only -- no on-the-fly compression takes place, right? If so then Darren's example can't be correct, but instead it should be this: # lofiadm -a /path/to/backingfile.img /dev/lofi/1 # mkfs -F ufs /dev/rlofi/1 # mount -F ufs /dev/lofi/1 /mnt ... # umount /mnt # lofiadm -d /dev/rlofi/1 # lofiadm -C gzip /path/to/backingfile.img # lofiadm -a /path/to/backingfile.img /dev/lofi/1 # mount -F ufs /dev/lofi/1 /mnt Also, if compressed lofi devices are read-only then the answer to DJM-1 is fairly obvious [to me, though perhaps also wrong]. See below. > >DJM-1 How does this interact with lofi crypto ? > > > >(http://opensolaris.org/os/community/arc/caselog/2007/001/) > > > >Which happens first crypto or compression (hint it needs to be compression > >otherwise it is waste of CPU resource) ? > > > >However if my understanding of how -C and -U work it seems that the > >compression would be happening after encryption because by design it isn't > >on the fly but actually a distinct admin action (which for building LiveCD > >images is fine). Since compression is a one-time task for a read-only lofi file, then so should encryption be a one-time task to be done along with (after) compression: # lofiadm -a /path/to/backingfile.img /dev/lofi/1 # mkfs -F ufs /dev/rlofi/1 # mount -F ufs /dev/lofi/1 /mnt ... # umount /mnt # lofiadm -c aes128-cbc -e /etc/.../backingfile.key -C gzip /path/to/backingfile.img # lofiadm -a -c aes128-cbc -k /etc/.../backingfile.key /path/to/backingfile.img ... OR, perhaps, since there'd now be a header where to store the name of the cipher used: ... # lofiadm -c aes128-cbc -e /etc/.../backingfile.key -C gzip /path/to/backingfile.img # lofiadm -a -k /etc/.../backingfile.key /path/to/backingfile.img ... Maybe even the -k option could be optional in lofiadm -a ... > >DJM-5 How big is the header section ? > > It is 12 bytes plus the length of compression algorithm used. > So, for gzip, it would be 16 bytes (12 + 4). > > >The project to add encryption to lofi stalled on exactly the issue of the > >adding metadata to the image. Since this project is adding a header > >section it seems to make sense that the crypto and compression projects > >share a metadata section. > > We should talk on this point, it does make sense to > have a shared metadata section. I agree. More persistent information stored in metadata == less for the sysadmin to remember to repeat in subsequent lofiadm incantantions == less trouble. Nico -- From Darren.Moffat@sun.com Tue Oct 2 08:50:25 2007 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92FoObb006900 for ; Tue, 2 Oct 2007 08:50:24 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l92Fl6Kg021419; Tue, 2 Oct 2007 16:47:15 +0100 (BST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPA00J1RJUNHM00@brm-avmta-1.central.sun.com>; Tue, 02 Oct 2007 09:47:11 -0600 (MDT) Received: from gmp-eb-mail-1.sun.com ([192.18.6.21]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPA009Z8JULTYD0@brm-avmta-1.central.sun.com>; Tue, 02 Oct 2007 09:47:10 -0600 (MDT) Received: from fe-emea-09.sun.com (gmp-eb-lb-2-fe2.eu.sun.com [192.18.6.11]) by gmp-eb-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l92Fl93w000728; Tue, 02 Oct 2007 15:47:09 +0000 (GMT) Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPA00C01JCYH100@fe-emea-09.sun.com> (original mail from Darren.Moffat@Sun.COM); Tue, 02 Oct 2007 16:47:09 +0100 (BST) Received: from [129.156.173.21] by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPA008L4JUKWL20@fe-emea-09.sun.com>; Tue, 02 Oct 2007 16:47:09 +0100 (BST) Date: Tue, 02 Oct 2007 16:47:08 +0100 From: Darren J Moffat Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] In-reply-to: <20071002153813.GW11376@Sun.COM> Sender: Darren.Moffat@sun.com To: Nicolas Williams Cc: Alok Aggarwal , PSARC-EXT@sun.com, Tim Haley - Sun Microsystem Message-id: <470267FC.3020205@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <20071002153813.GW11376@Sun.COM> User-Agent: Thunderbird 2.0.0.4 (X11/20070731) Status: RO Content-Length: 609 Nicolas Williams wrote: > Since compression is a one-time task for a read-only lofi file, then so > should encryption be a one-time task to be done along with (after) > compression: but that isn't how lofi encryption is designed to work, and making it work like that completely defeat the whole purpose of why do are doing encryption in lofi. It also creates a window where the data is stored on disk in the clear - which is exactly what we don't want. It would also mean we couldn't use lofi with encryption to swap on (which we need until we get a proper encrypting VM system). -- Darren J Moffat From Nicolas.Williams@sun.com Tue Oct 2 08:54:09 2007 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92Fs8Jp007568 for ; Tue, 2 Oct 2007 08:54:08 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l92Fo2pR030281; Tue, 2 Oct 2007 09:50:03 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPA00G09K0YRE00@nwk-avmta-2.sfbay.sun.com>; Tue, 02 Oct 2007 08:50:58 -0700 (PDT) Received: from binky.Central.Sun.COM ([129.153.128.104]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPA00EZGK0X9RB0@nwk-avmta-2.sfbay.sun.com>; Tue, 02 Oct 2007 08:50:57 -0700 (PDT) Received: from binky.Central.Sun.COM (localhost [127.0.0.1]) by binky.Central.Sun.COM (8.14.1+Sun/8.14.1) with ESMTP id l92FovAc017766; Tue, 02 Oct 2007 10:50:57 -0500 (CDT) Received: (from nw141292@localhost) by binky.Central.Sun.COM (8.14.1+Sun/8.14.1/Submit) id l92Fovnw017765; Tue, 02 Oct 2007 10:50:57 -0500 (CDT) Date: Tue, 02 Oct 2007 10:50:57 -0500 From: Nicolas Williams Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] In-reply-to: <470267FC.3020205@Sun.COM> To: Darren J Moffat Cc: Alok Aggarwal , PSARC-EXT@sun.com, Tim Haley - Sun Microsystem Message-id: <20071002155056.GY11376@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.2.0.264296 References: <20071002153813.GW11376@Sun.COM> <470267FC.3020205@Sun.COM> X-Authentication-warning: binky.Central.Sun.COM: nw141292 set sender to Nicolas.Williams@sun.com using -f User-Agent: Mutt/1.5.7i Status: RO Content-Length: 1068 On Tue, Oct 02, 2007 at 04:47:08PM +0100, Darren J Moffat wrote: > Nicolas Williams wrote: > >Since compression is a one-time task for a read-only lofi file, then so > >should encryption be a one-time task to be done along with (after) > >compression: > > but that isn't how lofi encryption is designed to work, and making it > work like that completely defeat the whole purpose of why do are doing > encryption in lofi. It also creates a window where the data is stored > on disk in the clear - which is exactly what we don't want. Yes, it doesn't seem worthwhile to support one-time compression + encryption in lofi. The crucial thing though is that compressed lofi images are read-only. > It would > also mean we couldn't use lofi with encryption to swap on (which we need > until we get a proper encrypting VM system). Well, you couldn't compress swap using this case because the compression step is a one-time step and lofiadm -a of compressed images results in read-only devices! Nico -- From Alok.Aggarwal@sun.com Tue Oct 2 09:14:45 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92GEibr010994 for ; Tue, 2 Oct 2007 09:14:44 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l92GBJeW021667; Wed, 3 Oct 2007 00:11:35 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPA00L2BKZB6I00@brm-avmta-1.central.sun.com>; Tue, 02 Oct 2007 10:11:35 -0600 (MDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPA00KUJKZ95O70@brm-avmta-1.central.sun.com>; Tue, 02 Oct 2007 10:11:33 -0600 (MDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l92GBXlx019466; Tue, 02 Oct 2007 16:11:33 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPA00701JKQFY00@mail-amer.sun.com> (original mail from Alok.Aggarwal@Sun.COM); Tue, 02 Oct 2007 10:11:33 -0600 (MDT) Received: from sr1-ubrm-44 ([129.147.9.134]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPA004QWKZ06G60@mail-amer.sun.com>; Tue, 02 Oct 2007 10:11:24 -0600 (MDT) Date: Tue, 02 Oct 2007 10:11:24 -0600 (MDT) From: Alok Aggarwal Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] In-reply-to: <20071002155056.GY11376@Sun.COM> Sender: Alok.Aggarwal@sun.com X-X-Sender: aalok@sr1-ubrm-44 To: Nicolas Williams Cc: Darren J Moffat , Alok Aggarwal , PSARC-EXT@sun.com, Tim Haley - Sun Microsystem Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <20071002153813.GW11376@Sun.COM> <470267FC.3020205@Sun.COM> <20071002155056.GY11376@Sun.COM> Status: RO Content-Length: 850 Darren and I talked about the areas of overlap between lofi encryption and lofi compression. Here's a summary of what we talked about - Compressed lofi images are read-only so it makes sense to allow either compression or encryption on an image but not both at the same time. So, lofi compression should be able to spot an encrypted image and prevent compression. Likewise, lofi encryption should be able to spot a compressed image and prevent encryption. This obviates the need for on-disk metadata to be stored by lofi encryption. In other words, there is a dependancy between PSARC/2007/001 and this case. The two project teams will work together to make sure that encrypted images can't be compressed and vice versa. Also, I need to make it clear in the documentation that compressed lofi images can be read-only and not read/write. Alok From gww@eng.sun.com Tue Oct 2 10:19:23 2007 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92HJMj0015572 for ; Tue, 2 Oct 2007 10:19:22 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l92HG0IF000416; Tue, 2 Oct 2007 18:16:13 +0100 (BST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPA0024LNYZH300@brm-avmta-1.central.sun.com>; Tue, 02 Oct 2007 11:16:11 -0600 (MDT) Received: from engmail3mpk.sfbay.Sun.COM ([129.146.11.26]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPA00MGPNYWAC40@brm-avmta-1.central.sun.com>; Tue, 02 Oct 2007 11:16:09 -0600 (MDT) Received: from marduk.eng.sun.com (marduk.SFBay.Sun.COM [129.146.108.224]) by engmail3mpk.sfbay.Sun.COM (8.13.6+Sun/8.13.6/ENSMAIL,v2.2) with ESMTP id l92HG79M014741; Tue, 02 Oct 2007 10:16:07 -0700 (PDT) Received: from marduk.eng.sun.com (localhost [127.0.0.1]) by marduk.eng.sun.com (8.13.6+Sun/8.12.11) with ESMTP id l92HHqD5000829; Tue, 02 Oct 2007 10:17:52 -0700 (PDT) Received: (from gww@localhost) by marduk.eng.sun.com (8.13.6+Sun/8.12.11/Submit) id l92HHqDR000828; Tue, 02 Oct 2007 10:17:52 -0700 (PDT) Date: Tue, 02 Oct 2007 10:17:52 -0700 (PDT) From: Gary Winiger Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] To: Nicolas.Williams@sun.com, Alok.Aggarwal@sun.com Cc: Darren.Moffat@sun.com, PSARC-EXT@sun.com, Tim.Haley@sun.com Message-id: <200710021717.l92HHqDR000828@marduk.eng.sun.com> Content-transfer-encoding: 7BIT X-Sun-Charset: US-ASCII X-PMX-Version: 5.2.0.264296 Status: RO Content-Length: 995 > Darren and I talked about the areas of overlap between > lofi encryption and lofi compression. Here's a summary > of what we talked about - > > Compressed lofi images are read-only so it makes sense > to allow either compression or encryption on an image > but not both at the same time. I can understand doing both may be ETOOHARD or ENOTWORTHIT, and I don't see why it isn't architecturally correct to compress then encrypt/decrypt then expand. > So, lofi compression should be able to spot an > encrypted image and prevent compression. Likewise, lofi > encryption should be able to spot a compressed image > and prevent encryption. > > This obviates the need for on-disk metadata to be stored > by lofi encryption. So ENOTWORTHIT? > Also, I need to make it clear in the documentation that > compressed lofi images can be read-only and not read/write. This seems to me to be a change from the original proposal. Have I missed something, or has the proposal changed? Gary.. From Darren.Moffat@sun.com Tue Oct 2 10:29:58 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92HTv3p015913 for ; Tue, 2 Oct 2007 10:29:58 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l92HQkt6022355; Wed, 3 Oct 2007 01:26:48 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPA00303OGN5600@brm-avmta-1.central.sun.com>; Tue, 02 Oct 2007 11:26:47 -0600 (MDT) Received: from gmp-eb-mail-1.sun.com ([192.18.6.21]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPA00MM6OGLA940@brm-avmta-1.central.sun.com>; Tue, 02 Oct 2007 11:26:46 -0600 (MDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-2-fe1.eu.sun.com [192.18.6.10]) by gmp-eb-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l92HQjTO008955; Tue, 02 Oct 2007 17:26:45 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPA00801OAEDC00@fe-emea-10.sun.com> (original mail from Darren.Moffat@Sun.COM); Tue, 02 Oct 2007 18:26:45 +0100 (BST) Received: from [129.156.173.21] by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPA00GTKOGGW620@fe-emea-10.sun.com>; Tue, 02 Oct 2007 18:26:41 +0100 (BST) Date: Tue, 02 Oct 2007 18:26:40 +0100 From: Darren J Moffat Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] In-reply-to: <200710021717.l92HHqDR000828@marduk.eng.sun.com> Sender: Darren.Moffat@sun.com To: Gary Winiger Cc: Nicolas.Williams@sun.com, Alok.Aggarwal@sun.com, PSARC-EXT@sun.com, Tim.Haley@sun.com Message-id: <47027F50.6010700@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200710021717.l92HHqDR000828@marduk.eng.sun.com> User-Agent: Thunderbird 2.0.0.4 (X11/20070731) Status: RO Content-Length: 689 Gary Winiger wrote: >> Darren and I talked about the areas of overlap between >> lofi encryption and lofi compression. Here's a summary >> of what we talked about - >> >> Compressed lofi images are read-only so it makes sense >> to allow either compression or encryption on an image >> but not both at the same time. > > I can understand doing both may be ETOOHARD or > ENOTWORTHIT, and I don't see why it isn't architecturally > correct to compress then encrypt/decrypt then expand. ENOTWORTHIT because we already have compression support in ZFS and crypto is being added in such away that we *will* be able to compress then encrypt (decrypt then decompress). -- Darren J Moffat From gww@eng.sun.com Tue Oct 2 10:31:35 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92HVY6q015982 for ; Tue, 2 Oct 2007 10:31:34 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l92HSCow022754; Wed, 3 Oct 2007 01:28:24 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPA0030BOJ87W00@brm-avmta-1.central.sun.com>; Tue, 02 Oct 2007 11:28:20 -0600 (MDT) Received: from engmail3mpk.sfbay.Sun.COM ([129.146.11.26]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPA00MW2OJ7AD30@brm-avmta-1.central.sun.com>; Tue, 02 Oct 2007 11:28:19 -0600 (MDT) Received: from marduk.eng.sun.com (marduk.SFBay.Sun.COM [129.146.108.224]) by engmail3mpk.sfbay.Sun.COM (8.13.6+Sun/8.13.6/ENSMAIL,v2.2) with ESMTP id l92HSHc3016060; Tue, 02 Oct 2007 10:28:17 -0700 (PDT) Received: from marduk.eng.sun.com (localhost [127.0.0.1]) by marduk.eng.sun.com (8.13.6+Sun/8.12.11) with ESMTP id l92HU28M000870; Tue, 02 Oct 2007 10:30:02 -0700 (PDT) Received: (from gww@localhost) by marduk.eng.sun.com (8.13.6+Sun/8.12.11/Submit) id l92HU2qr000869; Tue, 02 Oct 2007 10:30:02 -0700 (PDT) Date: Tue, 02 Oct 2007 10:30:02 -0700 (PDT) From: Gary Winiger Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] To: gww@eng.sun.com, Darren.Moffat@Sun.COM Cc: Nicolas.Williams@Sun.COM, Alok.Aggarwal@Sun.COM, PSARC-EXT@Sun.COM, Tim.Haley@Sun.COM Message-id: <200710021730.l92HU2qr000869@marduk.eng.sun.com> Content-transfer-encoding: 7BIT X-Sun-Charset: US-ASCII X-PMX-Version: 5.2.0.264296 Status: RO Content-Length: 209 > ENOTWORTHIT because we already have compression support in ZFS and > crypto is being added in such away that we *will* be able to compress > then encrypt (decrypt then decompress). Fine with me. Gary.. From Alok.Aggarwal@sun.com Tue Oct 2 11:06:56 2007 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92I6t53018265 for ; Tue, 2 Oct 2007 11:06:56 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l92I3QEW019903; Tue, 2 Oct 2007 19:03:46 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPA00003Q68WB00@nwk-avmta-2.sfbay.sun.com>; Tue, 02 Oct 2007 11:03:44 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPA00K25Q67TR50@nwk-avmta-2.sfbay.sun.com>; Tue, 02 Oct 2007 11:03:44 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l92I3hQ7016488; Tue, 02 Oct 2007 18:03:43 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPA00701P8E8C00@mail-amer.sun.com> (original mail from Alok.Aggarwal@Sun.COM); Tue, 02 Oct 2007 12:03:43 -0600 (MDT) Received: from sr1-ubrm-44 ([129.147.9.134]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPA00J66Q5XM8G0@mail-amer.sun.com>; Tue, 02 Oct 2007 12:03:33 -0600 (MDT) Date: Tue, 02 Oct 2007 12:03:33 -0600 (MDT) From: Alok Aggarwal Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] In-reply-to: <200710021717.l92HHqDR000828@marduk.eng.sun.com> Sender: Alok.Aggarwal@sun.com X-X-Sender: aalok@sr1-ubrm-44 To: Gary Winiger Cc: Nicolas.Williams@sun.com, Alok.Aggarwal@sun.com, Darren.Moffat@sun.com, PSARC-EXT@sun.com, Tim.Haley@sun.com Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200710021717.l92HHqDR000828@marduk.eng.sun.com> Status: RO Content-Length: 690 On Tue, 2 Oct 2007, Gary Winiger wrote: >> Also, I need to make it clear in the documentation that >> compressed lofi images can be read-only and not read/write. > > This seems to me to be a change from the original > proposal. Have I missed something, or has the proposal > changed? It was a case of me having missed specifying this in the case materials. Basically, this case provides "offline" compression if you will; compressing a file and mapping that file via lofi are two separate steps. If we were to provide read/write access to compressed files, we'd essentially have to provide on-the-fly compression. There doesn't seem to be a need to provide such a facility. Alok From Timothy.Haley@sun.com Tue Oct 2 14:47:00 2007 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92Ll0tb029714 for ; Tue, 2 Oct 2007 14:47:00 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l92Lgtu6008659 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Tue, 2 Oct 2007 15:42:56 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPB00L010D3F600@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Tue, 02 Oct 2007 14:43:51 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPB00BMV0D36N70@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Tue, 02 Oct 2007 14:43:51 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l92LhpE9013831 for ; Tue, 02 Oct 2007 21:43:51 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPA00E01Y7C7N00@mail-amer.sun.com> (original mail from Timothy.Haley@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Tue, 02 Oct 2007 15:43:51 -0600 (MDT) Received: from dhcp-ubrm05-50-203.central.sun.com ([129.147.50.203]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPB009E50D2H3B0@mail-amer.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Tue, 02 Oct 2007 15:43:50 -0600 (MDT) Date: Tue, 02 Oct 2007 15:43:50 -0600 From: Timothy Haley Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] In-reply-to: <470236F6.1090509@Sun.COM> Sender: Timothy.Haley@sun.com To: Darren J Moffat Cc: PSARC-ext@sun.com Message-id: <4702BB96.2060807@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200710020324.l923OkCx028573@spidey.Central.Sun.COM> <470236F6.1090509@Sun.COM> User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) Status: RO Content-Length: 317 Darren J Moffat wrote: > > Process issue: > > http://opensolaris.org/os/community/arc/caselog/2007/569/ gives 404 at > the moment. > John P told me this morning a nightly copy script failed with issues. He restarted it but it either failed again, or hasn't got to this case yet. -tim > -- > Darren J Moffat From Timothy.Haley@sun.com Tue Oct 2 14:48:43 2007 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l92Lmhqt029867 for ; Tue, 2 Oct 2007 14:48:43 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l92Licqt008824 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Tue, 2 Oct 2007 15:44:39 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPB00L070FYOL00@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 02 Oct 2007 14:45:34 -0700 (PDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPB00BGM0FX6R70@nwk-avmta-1.sfbay.Sun.COM> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 02 Oct 2007 14:45:33 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l92LjXHr016228 for ; Tue, 02 Oct 2007 21:45:33 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPA00M01ZZNNF00@mail-amer.sun.com> (original mail from Timothy.Haley@Sun.COM) for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Tue, 02 Oct 2007 15:45:33 -0600 (MDT) Received: from dhcp-ubrm05-50-203.central.sun.com ([129.147.50.203]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPB0075E0FFP4B0@mail-amer.sun.com>; Tue, 02 Oct 2007 15:45:15 -0600 (MDT) Date: Tue, 02 Oct 2007 15:45:14 -0600 From: Timothy Haley Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] In-reply-to: <200710020512.l925CUqn000020@marduk.eng.sun.com> Sender: Timothy.Haley@sun.com To: Gary Winiger Cc: PSARC-ext@sun.com Message-id: <4702BBEA.1030506@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200710020512.l925CUqn000020@marduk.eng.sun.com> User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) Status: RO Content-Length: 543 Gary Winiger wrote: >> I am sponsoring this fast-track for Alok Aggarwal. >> Requested binding is micro, timeout 10/08/2007. > > Just checking that Micro (which today equates to Minor) is what > is meant. That is there is no intent to backport to any marketing > release in current distrobution. > I can perhaps see why that might be. IMO, this is a case were > Micro seems justified as it could leave data on a compressed > image unrecoverable if the "patch" were removed. > > Gary.. Correct, there is no intent to backport. -tim From Darren.Moffat@sun.com Wed Oct 3 10:22:51 2007 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l93HMpx3002102 for ; Wed, 3 Oct 2007 10:22:51 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l93HJfgd028879 for <@sunmail2sca.sfbay.sun.com:PSARC-EXT@sun.com>; Wed, 3 Oct 2007 10:19:41 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPC00A0HISSYB00@brm-avmta-1.central.sun.com> for PSARC-EXT@sun.com (ORCPT PSARC-EXT@Sun.COM); Wed, 03 Oct 2007 11:19:40 -0600 (MDT) Received: from gmp-eb-mail-2.sun.com ([192.18.6.24]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPC001JTISRB870@brm-avmta-1.central.sun.com> for PSARC-EXT@sun.com (ORCPT PSARC-EXT@Sun.COM); Wed, 03 Oct 2007 11:19:39 -0600 (MDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-2-fe2.eu.sun.com [192.18.6.11]) by gmp-eb-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l93HJcUL007912 for ; Wed, 03 Oct 2007 17:19:38 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPC00F01ILXE400@fe-emea-10.sun.com> (original mail from Darren.Moffat@Sun.COM) for PSARC-EXT@Sun.COM (ORCPT PSARC-EXT@Sun.COM); Wed, 03 Oct 2007 18:19:38 +0100 (BST) Received: from [129.156.173.21] by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPC00CQ6ISPD120@fe-emea-10.sun.com> for PSARC-EXT@Sun.COM (ORCPT PSARC-EXT@Sun.COM); Wed, 03 Oct 2007 18:19:38 +0100 (BST) Date: Wed, 03 Oct 2007 18:19:37 +0100 From: Darren J Moffat Subject: [waiting need spec] PSARC/2007/569 lofi(7D) compression support Sender: Darren.Moffat@sun.com To: PSARC-EXT@sun.com Message-id: <4703CF29.2050300@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 User-Agent: Thunderbird 2.0.0.4 (X11/20070731) Status: RO Content-Length: 138 This case was marked as "waiting need spec" due to the required co-ordination with PSARC/2007/001 lofi(7D) crypto. -- Darren J Moffat From Alok.Aggarwal@sun.com Thu Oct 4 15:19:20 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l94MJJgD012989 for ; Thu, 4 Oct 2007 15:19:19 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l94MFp5V011453; Fri, 5 Oct 2007 06:16:08 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPE0090VR6TBF00@nwk-avmta-1.sfbay.Sun.COM>; Thu, 04 Oct 2007 15:16:05 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPE007TNR6SNT00@nwk-avmta-1.sfbay.Sun.COM>; Thu, 04 Oct 2007 15:16:04 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l94MG4wk004456; Thu, 04 Oct 2007 22:16:04 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPE00G01QHA3W00@mail-amer.sun.com> (original mail from Alok.Aggarwal@Sun.COM); Thu, 04 Oct 2007 16:16:04 -0600 (MDT) Received: from sr1-ubrm-44 ([129.147.9.134]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPE00DM4R6JWCC0@mail-amer.sun.com>; Thu, 04 Oct 2007 16:15:55 -0600 (MDT) Date: Thu, 04 Oct 2007 16:15:55 -0600 (MDT) From: Alok Aggarwal Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] In-reply-to: Sender: Alok.Aggarwal@sun.com X-X-Sender: aalok@sr1-ubrm-44 To: PSARC-EXT@sun.com Cc: caiman-discuss@opensolaris.org, Tim Haley - Sun Microsystem Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <20071002153813.GW11376@Sun.COM> <470267FC.3020205@Sun.COM> <20071002155056.GY11376@Sun.COM> Status: RO Content-Length: 11184 Here's an updated spec (diffs followed by the actual spec), Tim will be updating the case materials as well. Alok -- ------- clofi.psarc ------- --- /tmp/geta7790 Thu Oct 4 16:05:24 2007 +++ /tmp/getb7790 Thu Oct 4 16:05:24 2007 @@ -20,11 +20,12 @@ This project aims to extend lofi(7D) to support reading from a compressed file via on-the-fly decompression. This project will also add extensions to the lofiadm(1M) command to provide for the ability to compress a file in a suitable format - and to decompress it entirely. + and to decompress it entirely. The ability to decompress is provided + for debugging purposes. Design Overview The basic design is to compress a file in fixed size segments and store the segments sequentially. The information about where each @@ -38,10 +39,15 @@ structure. The compression metadata thus read is then used to decompress the requested data when a read request comes in. If the decompression fails for some reason, an EIO is returned. + It should be noted that a compressed file can only be + accessed in a read-only mode and no changes are allowed to it + once it is compressed. An attempt to access the file in read/write + or mount it in read/write mode will fail with an ENOTSUP. + On disk format A graphic representation of the compression header stored at the start of the file can be found under materials/ondisk_format.png @@ -54,10 +60,20 @@ On the fly decompression A graphic representation of reading from compressed versus regular files can be found under materials/compress_io.jpg +Interaction with lofi encryption + + PSARC/2007/001 proposes to add encryption to lofi(7D). Since + compressed lofi images are read-only, it makes sense to allow + either compression or encryption on a file but not both at + the same time. + + Depending upon when PSARC/2007/001 delivers, support will need + to be added to prevent compressing encrypted files and vice-versa. + Compression Algorithms The lofi(7D) driver will support compression algorithms present in the Solaris kernel. Currently, lzjb and gzip are the two algorithms available. @@ -65,13 +81,14 @@ The lzjb algorithm provides very fast compression as compared to gzip. gzip, even though it is slower than lzjb, is fast enough for the typical use case of a LiveCD. In addition, gzip provides a much better compression ratio as compared to lzjb (on the order of ~25% in some cases) and it allows for more data - to be packed on to the media. These characteristics make gzip - well suited for a LiveCD. Thus, gzip will be the only supported - algorithm at this time. + to be packed on to the media. These characteristics make gzip, + specifically gzip level 9, well suited for a LiveCD. Thus, gzip + will be the only supported algorithm at this time with levels 6 + and 9 being supported. gzip level 6 will be the default. The changes being made as part of this project will allow for compression algorithms to be plugged into lofi with relative ease as more of these algorithms are made available in the Solaris kernel. @@ -78,40 +95,44 @@ Command Changes lofiadm(1M) will take the following additional arguments - - -C Compress the file with a specified compression algorithm - or with the default compression algorithm if one isn't - specified. The only supported compression algorithm at this - time is gzip which is also the default. + -C gzip | gzip-N Compress the file with "gzip" compression + algorithm before mapping the file in. You can + specify the "gzip" level by using the value + "gzip-N" where N is 6 (fast) or 9 (best compression + ratio). Currently, "gzip" is equivalent to "gzip-6" + (which is also the default for zfs(1M) and gzip(1)). - -s The segment size to use to divide the file being compressed. - If no segment size is specified, a default segment size of - 128k is used. + -s segsize The segment size to use to divide the file being + compressed. Segment size must be an integer + multiple of 512. -U Uncompress a compressed file The output of lofiadm(1M) will also change to add another field to reflect if the mapped file(s) is a compressed file or a regular file. Examples: - 1. Compress a file with gzip + 1. Compress a file with gzip and add the compressed file as a block + device - # lofiadm -C gzip /export/home/solaris.iso + # lofiadm -a -C gzip /export/home/solaris.iso - 2. Compress a file with gzip and compress it in 64k chunks + 2. Compress a file with gzip in 64k chunks and add the compressed + file as a block device # lofiadm -C gzip -s 64k /export/home/solaris.iso 3. Check if there are any compressed files that are mapped # lofiadm - Block Device File Type + Block Device File Options /dev/lofi/1 /export/home/solaris.iso Compressed (gzip) - /dev/lofi/2 /export/home/regular.iso Regular + /dev/lofi/2 /export/home/regular.iso - 4. Uncompress a compressed file # lofiadm -U /export/home/solaris.iso --- clofi.psarc 1.3 07/10/04 SMI Template Version: @(#)sac_nextcase 1.3 10/04/07 SMI 1. Introduction 1.1 Project/Component Working Name: lofi(7D) compression support 1.2 Name of Document Author/Supplier Author: Alok Aggarwal 1.3 Date of this Document 28 September, 2007 4. Technical Description There are cases where certain files containing a file system image need to be compressed. A classic use case is that of a LiveCD. Compression support in lofi is required to make LiveCDs practical. It allows for more data to be packed on the media and improves performance by reducing head seek times. This project aims to extend lofi(7D) to support reading from a compressed file via on-the-fly decompression. This project will also add extensions to the lofiadm(1M) command to provide for the ability to compress a file in a suitable format and to decompress it entirely. The ability to decompress is provided for debugging purposes. Design Overview The basic design is to compress a file in fixed size segments and store the segments sequentially. The information about where each of the segments start is maintained in an index. This index along with a header that contains the compression algorithm used, segment size, number of segments and the uncompressed size of the last segment is stored at the start of the file. When a compressed file is mapped, the header containing the compression metadata is read in and stored in the lofi state structure. The compression metadata thus read is then used to decompress the requested data when a read request comes in. If the decompression fails for some reason, an EIO is returned. It should be noted that a compressed file can only be accessed in a read-only mode and no changes are allowed to it once it is compressed. An attempt to access the file in read/write or mount it in read/write mode will fail with an ENOTSUP. On disk format A graphic representation of the compression header stored at the start of the file can be found under materials/ondisk_format.png It should be noted that this header information is always stored in network byte order. This allows for endian neutral filesystems such as ZFS to be compressed on a little endian machine, and subsequently be uncompressed on a big endian machine; and vice-versa. On the fly decompression A graphic representation of reading from compressed versus regular files can be found under materials/compress_io.jpg Interaction with lofi encryption PSARC/2007/001 proposes to add encryption to lofi(7D). Since compressed lofi images are read-only, it makes sense to allow either compression or encryption on a file but not both at the same time. Depending upon when PSARC/2007/001 delivers, support will need to be added to prevent compressing encrypted files and vice-versa. Compression Algorithms The lofi(7D) driver will support compression algorithms present in the Solaris kernel. Currently, lzjb and gzip are the two algorithms available. The lzjb algorithm provides very fast compression as compared to gzip. gzip, even though it is slower than lzjb, is fast enough for the typical use case of a LiveCD. In addition, gzip provides a much better compression ratio as compared to lzjb (on the order of ~25% in some cases) and it allows for more data to be packed on to the media. These characteristics make gzip, specifically gzip level 9, well suited for a LiveCD. Thus, gzip will be the only supported algorithm at this time with levels 6 and 9 being supported. gzip level 6 will be the default. The changes being made as part of this project will allow for compression algorithms to be plugged into lofi with relative ease as more of these algorithms are made available in the Solaris kernel. Command Changes lofiadm(1M) will take the following additional arguments - -C gzip | gzip-N Compress the file with "gzip" compression algorithm before mapping the file in. You can specify the "gzip" level by using the value "gzip-N" where N is 6 (fast) or 9 (best compression ratio). Currently, "gzip" is equivalent to "gzip-6" (which is also the default for zfs(1M) and gzip(1)). -s segsize The segment size to use to divide the file being compressed. Segment size must be an integer multiple of 512. -U Uncompress a compressed file The output of lofiadm(1M) will also change to add another field to reflect if the mapped file(s) is a compressed file or a regular file. Examples: 1. Compress a file with gzip and add the compressed file as a block device # lofiadm -a -C gzip /export/home/solaris.iso 2. Compress a file with gzip in 64k chunks and add the compressed file as a block device # lofiadm -C gzip -s 64k /export/home/solaris.iso 3. Check if there are any compressed files that are mapped # lofiadm Block Device File Options /dev/lofi/1 /export/home/solaris.iso Compressed(gzip) /dev/lofi/2 /export/home/regular.iso - 4. Uncompress a compressed file # lofiadm -U /export/home/solaris.iso References PSARC/2007/492 - Project Indiana http://sac.eng/PSARC/2007/492/discussion.materials/Indiana.txt PSARC/1999/463 - lofi http://sac.sfbay/Archives/CaseLog/arc/PSARC/1999/463 On the Fly Decompression in BeleniX http://www.genunix.org/distributions/belenix_site/?q=compression gzip for ZFS update http://blogs.sun.com/ahl/entry/gzip_for_zfs_update 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack From Moinak.Ghosh@sun.com Thu Oct 4 21:57:45 2007 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l954virI019753 for ; Thu, 4 Oct 2007 21:57:44 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l954sT79024307; Fri, 5 Oct 2007 05:54:33 +0100 (BST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPF00L199MVSZ00@brm-avmta-1.central.sun.com>; Thu, 04 Oct 2007 22:54:31 -0600 (MDT) Received: from sineb-mail-1.sun.com ([192.18.19.6]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPF00IY39MOGS30@brm-avmta-1.central.sun.com>; Thu, 04 Oct 2007 22:54:26 -0600 (MDT) Received: from fe-apac-06.sun.com (fe-apac-06.sun.com [192.18.19.177] (may be forged)) by sineb-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l954sOnu025246; Fri, 05 Oct 2007 04:54:24 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) id <0JPF00B019LTT400@mail-apac.sun.com> (original mail from Moinak.Ghosh@Sun.COM); Fri, 05 Oct 2007 12:54:24 +0800 (SGT) Received: from [192.168.1.34] ([59.92.223.35]) by mail-apac.sun.com (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) with ESMTPSA id <0JPF00H7J9M8YDK8@mail-apac.sun.com>; Fri, 05 Oct 2007 12:54:15 +0800 (SGT) Date: Fri, 05 Oct 2007 10:30:04 +0530 From: Moinak Ghosh Subject: Re: [caiman-discuss] lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] In-reply-to: Sender: Moinak.Ghosh@sun.com To: Alok Aggarwal Cc: PSARC-EXT@sun.com, caiman-discuss@opensolaris.org, Tim Haley - Sun Microsystem Message-id: <4705C4D4.3030009@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <20071002153813.GW11376@Sun.COM> <470267FC.3020205@Sun.COM> <20071002155056.GY11376@Sun.COM> User-Agent: Thunderbird 2.0b2 (X11/20070227) Status: RO Content-Length: 1564 Alok Aggarwal wrote: > [...] Examples: > > - 1. Compress a file with gzip > + 1. Compress a file with gzip and add the compressed file as a block > + device > > - # lofiadm -C gzip /export/home/solaris.iso > + # lofiadm -a -C gzip /export/home/solaris.iso > > - 2. Compress a file with gzip and compress it in 64k chunks > + 2. Compress a file with gzip in 64k chunks and add the compressed > + file as a block device > > # lofiadm -C gzip -s 64k /export/home/solaris.iso > Shouldn't this be: lofiadm -a -C gzip -s 64k /export/home/solaris.iso ? On the other hand, what will be the name of the compressed file ? Would it remain as the original filename ? Can cause confusion. What happens to the compressed file when it is unmapped ? The process of compression is a very time consuming task, probably on the order of 30 - 50mins for 2GB worth of data, because it is chunked compression (though this can be improved by some parallelization and buffering techniques). So IMHO it is less confusing to do something like this: lofiadm -C gzip -s 64k /export/home/solaris.iso Produces solaris.iso. compressed file. lofiadm -a /export/home/solaris.iso. Maps the compressed file. lofiadm Outputs: Block Device File Options /dev/lofi/1 /export/home/solaris.iso. Compressed (gzip) /dev/lofi/2 /export/home/regular.iso. - Semantically similar to using gzip or any other compression utility for that matter. Regards, Moinak. From Alok.Aggarwal@sun.com Fri Oct 5 07:01:20 2007 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l95E1JDD027685 for ; Fri, 5 Oct 2007 07:01:19 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l95DvnLk014694; Fri, 5 Oct 2007 14:58:07 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPF00009YSV5R00@nwk-avmta-2.sfbay.sun.com>; Fri, 05 Oct 2007 06:58:07 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPF003UOYSUPOB0@nwk-avmta-2.sfbay.sun.com>; Fri, 05 Oct 2007 06:58:06 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l95Dw6jp007025; Fri, 05 Oct 2007 13:58:06 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPF00501YPO1R00@mail-amer.sun.com> (original mail from Alok.Aggarwal@Sun.COM); Fri, 05 Oct 2007 07:58:06 -0600 (MDT) Received: from sr1-ubrm-44 ([129.147.9.134]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPF00EWVYSQD3E0@mail-amer.sun.com>; Fri, 05 Oct 2007 07:58:02 -0600 (MDT) Date: Fri, 05 Oct 2007 07:58:02 -0600 (MDT) From: Alok Aggarwal Subject: Re: [caiman-discuss] lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] In-reply-to: <4705C4D4.3030009@Sun.COM> Sender: Alok.Aggarwal@sun.com X-X-Sender: aalok@sr1-ubrm-44 To: Moinak Ghosh Cc: PSARC-EXT@sun.com, caiman-discuss@opensolaris.org, Tim Haley - Sun Microsystem Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <20071002153813.GW11376@Sun.COM> <470267FC.3020205@Sun.COM> <20071002155056.GY11376@Sun.COM> <4705C4D4.3030009@Sun.COM> Status: RO Content-Length: 3006 On Fri, 5 Oct 2007, Moinak Ghosh wrote: > Alok Aggarwal wrote: >> [...] Examples: >> >> - 1. Compress a file with gzip >> + 1. Compress a file with gzip and add the compressed file as a block >> + device >> >> - # lofiadm -C gzip /export/home/solaris.iso >> + # lofiadm -a -C gzip /export/home/solaris.iso >> >> - 2. Compress a file with gzip and compress it in 64k chunks >> + 2. Compress a file with gzip in 64k chunks and add the compressed >> + file as a block device >> >> # lofiadm -C gzip -s 64k /export/home/solaris.iso >> > > Shouldn't this be: > > lofiadm -a -C gzip -s 64k /export/home/solaris.iso > ? It should be, it's a typo. > On the other hand, what will be the name of the compressed file ? > Would it remain as the original filename ? Can cause confusion. The name of the file remains the same. Why will it cause confusion? If you start adding extensions such as .gz to the compressed file, what's stopping a user from running gunzip(1) or your favorite decompression utility on that file and getting disappointed to see it fail. The key is that a file compressed with lofiadm is useful only within the context of lofi(7D) and not outside it. Not adding an extension just reinforces that point. > What happens to the compressed file when it is unmapped ? If it wasn't explicitly uncompressed by the user it stays compressed. > The process of compression is a very time consuming task, probably > on the order of 30 - 50mins for 2GB worth of data, because it is > chunked compression (though this can be improved by some > parallelization and buffering techniques). So IMHO it is less > confusing to do something like this: > > lofiadm -C gzip -s 64k /export/home/solaris.iso > Produces solaris.iso. compressed file. > > lofiadm -a /export/home/solaris.iso. > Maps the compressed file. > > lofiadm Outputs: > > Block Device File Options > /dev/lofi/1 /export/home/solaris.iso. Compressed (gzip) > /dev/lofi/2 /export/home/regular.iso. - > > > Semantically similar to using gzip or any other compression utility > for that matter. The original proposal had what you're suggesting, i.e. "compress this file with -C" and "map this compressed file with -a" as a two step process. As part of some offline discussions it was suggested that having these two actions as disjoint steps doesn't buy us anything. If a user compresses a file with lofiadm, in all certainity he's bound to map that compressed file as a logical next step so why not just enable him to do that in a single command invocation. Apropos making compression a separate step so the user doesn't wonder if the command is hung, the lofiadm output will change to printing something like this to stderr (so scripts don't break) - # lofiadm -a -C gzip -s 64k /export/home/solaris.iso Compressing solaris.iso Mapping solaris.iso /dev/lofi/1 I had meant to add this to the updated spec but apparently missed it. Alok From Darren.Moffat@Sun.COM Fri Oct 5 07:19:27 2007 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l95EJQt3027948 for ; Fri, 5 Oct 2007 07:19:26 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l95EGEXI022364; Fri, 5 Oct 2007 15:16:14 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPF00001ZN2ZW00@nwk-avmta-1.sfbay.Sun.COM>; Fri, 05 Oct 2007 07:16:14 -0700 (PDT) Received: from gmp-eb-mail-1.sun.com ([192.18.6.21]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPF00JPJZMZUID0@nwk-avmta-1.sfbay.Sun.COM>; Fri, 05 Oct 2007 07:16:12 -0700 (PDT) Received: from fe-emea-09.sun.com (gmp-eb-lb-2-fe2.eu.sun.com [192.18.6.11]) by gmp-eb-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l95EGBng008287; Fri, 05 Oct 2007 14:16:11 +0000 (GMT) Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPF00L01Y0AZJ00@fe-emea-09.sun.com> (original mail from Darren.Moffat@Sun.COM); Fri, 05 Oct 2007 15:16:11 +0100 (BST) Received: from [129.156.173.21] by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPF0052PZMNOK20@fe-emea-09.sun.com>; Fri, 05 Oct 2007 15:15:59 +0100 (BST) Date: Fri, 05 Oct 2007 15:15:59 +0100 From: Darren J Moffat Subject: Re: [caiman-discuss] lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] In-reply-to: Sender: Darren.Moffat@Sun.COM To: Alok Aggarwal Cc: Moinak Ghosh , PSARC-EXT@Sun.COM, caiman-discuss@opensolaris.org, Tim Haley - Sun Microsystem Message-id: <4706471F.1070600@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <20071002153813.GW11376@Sun.COM> <470267FC.3020205@Sun.COM> <20071002155056.GY11376@Sun.COM> <4705C4D4.3030009@Sun.COM> User-Agent: Thunderbird 2.0.0.4 (X11/20070731) Status: RO Content-Length: 2653 Alok Aggarwal wrote: > On Fri, 5 Oct 2007, Moinak Ghosh wrote: > >> Alok Aggarwal wrote: >>> [...] Examples: >>> >>> - 1. Compress a file with gzip >>> + 1. Compress a file with gzip and add the compressed file as a block >>> + device >>> >>> - # lofiadm -C gzip /export/home/solaris.iso >>> + # lofiadm -a -C gzip /export/home/solaris.iso >>> >>> - 2. Compress a file with gzip and compress it in 64k chunks >>> + 2. Compress a file with gzip in 64k chunks and add the compressed >>> + file as a block device >>> >>> # lofiadm -C gzip -s 64k /export/home/solaris.iso >>> >> Shouldn't this be: >> >> lofiadm -a -C gzip -s 64k /export/home/solaris.iso >> ? > > It should be, it's a typo. Now I'm really confused. I thought that the -C was done while there was no mapping in place, but -a adds a mapping. > The original proposal had what you're suggesting, i.e. > "compress this file with -C" and "map this compressed file > with -a" as a two step process. As part of some offline > discussions it was suggested that having these two actions > as disjoint steps doesn't buy us anything. So how long does the compress take ? If it is anything more than a few seconds I think it should be a separate step because -a has always been near instant. > If a user compresses a file with lofiadm, in all certainity > he's bound to map that compressed file as a logical next step > so why not just enable him to do that in a single command > invocation. I don't think that follows, in fact given that one of the main motivations for this project is building compressed LiveCD images I'd have expected that it wouldn't be getting mapped immediately afterwards. It also incorrectly leads the user into thinking that compression is on the fly which it is not. > Apropos making compression a separate step so the user doesn't > wonder if the command is hung, the lofiadm output will change > to printing something like this to stderr (so scripts don't > break) - > # lofiadm -a -C gzip -s 64k /export/home/solaris.iso > Compressing solaris.iso > Mapping solaris.iso > /dev/lofi/1 You really shouldn't do that the output of lofiadm -a is intended to be scriptable and even says so in the man page: The ability to specify a device is pro- vided for use in scripts that wish to re-establish a particular set of asso- ciations. The other problem with -C being part of the mapping is it is now different from the -U option to uncompress since you wouldn't do lofiadm -d -U -- Darren J Moffat From Alok.Aggarwal@sun.com Fri Oct 5 13:06:08 2007 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l95K68E6008250 for ; Fri, 5 Oct 2007 13:06:08 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l95K2hKL020202; Fri, 5 Oct 2007 21:02:56 +0100 (BST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPG00E30FOT0S00@brm-avmta-1.central.sun.com>; Fri, 05 Oct 2007 14:02:53 -0600 (MDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPG002DNFOSN880@brm-avmta-1.central.sun.com>; Fri, 05 Oct 2007 14:02:52 -0600 (MDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l95K2qDr005153; Fri, 05 Oct 2007 20:02:52 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPG00601ENS6N00@mail-amer.sun.com> (original mail from Alok.Aggarwal@Sun.COM); Fri, 05 Oct 2007 14:02:52 -0600 (MDT) Received: from sr1-ubrm-44 ([129.147.9.134]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPG00J5VFOHT290@mail-amer.sun.com>; Fri, 05 Oct 2007 14:02:41 -0600 (MDT) Date: Fri, 05 Oct 2007 14:02:41 -0600 (MDT) From: Alok Aggarwal Subject: Re: [caiman-discuss] lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/08/2007] In-reply-to: <4706471F.1070600@Sun.COM> Sender: Alok.Aggarwal@sun.com X-X-Sender: aalok@sr1-ubrm-44 To: Darren J Moffat Cc: Moinak Ghosh , PSARC-ext@sun.com, caiman-discuss@opensolaris.org, Tim Haley - Sun Microsystem Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <20071002153813.GW11376@Sun.COM> <470267FC.3020205@Sun.COM> <20071002155056.GY11376@Sun.COM> <4705C4D4.3030009@Sun.COM> <4706471F.1070600@Sun.COM> Status: RO Content-Length: 3399 On Fri, 5 Oct 2007, Darren J Moffat wrote: > Alok Aggarwal wrote: >> On Fri, 5 Oct 2007, Moinak Ghosh wrote: >> >>> Alok Aggarwal wrote: >>>> [...] Examples: >>>> >>>> - 1. Compress a file with gzip >>>> + 1. Compress a file with gzip and add the compressed file as a block >>>> + device >>>> >>>> - # lofiadm -C gzip /export/home/solaris.iso >>>> + # lofiadm -a -C gzip /export/home/solaris.iso >>>> >>>> - 2. Compress a file with gzip and compress it in 64k chunks >>>> + 2. Compress a file with gzip in 64k chunks and add the compressed >>>> + file as a block device >>>> >>>> # lofiadm -C gzip -s 64k /export/home/solaris.iso >>>> >>> Shouldn't this be: >>> >>> lofiadm -a -C gzip -s 64k /export/home/solaris.iso >>> ? >> >> It should be, it's a typo. > > Now I'm really confused. I thought that the -C was done while there was no > mapping in place, but -a adds a mapping. > >> The original proposal had what you're suggesting, i.e. >> "compress this file with -C" and "map this compressed file >> with -a" as a two step process. As part of some offline >> discussions it was suggested that having these two actions >> as disjoint steps doesn't buy us anything. > > So how long does the compress take ? If it is anything more than a few > seconds I think it should be a separate step because -a has always been near > instant. Yes, it takes a while to compress and depends on the size of the file being compressed. >> If a user compresses a file with lofiadm, in all certainity >> he's bound to map that compressed file as a logical next step >> so why not just enable him to do that in a single command >> invocation. > > I don't think that follows, in fact given that one of the main motivations > for this project is building compressed LiveCD images I'd have expected that > it wouldn't be getting mapped immediately afterwards. > > It also incorrectly leads the user into thinking that compression is on the > fly which it is not. How does supporting both flavors of the command line sound? That is - # lofiadm -C gzip -s 64k /export/home/solaris.iso <-- compress the file Or, optionally, # lofiadm -a -C gzip -s 64k /export/home/solaris.iso <-- compress the file and add it >> Apropos making compression a separate step so the user doesn't >> wonder if the command is hung, the lofiadm output will change >> to printing something like this to stderr (so scripts don't >> break) - > > >> # lofiadm -a -C gzip -s 64k /export/home/solaris.iso >> Compressing solaris.iso >> Mapping solaris.iso >> /dev/lofi/1 > > You really shouldn't do that the output of lofiadm -a is intended to be > scriptable and even says so in the man page: > > The ability to specify a device is pro- > vided for use in scripts that wish to > re-establish a particular set of asso- > ciations. Will the scripts still break if the new messages "Compressing file" and "Mapping file" are sent to *stderr* though? "/dev/lofi/1" still goes to stdout so there should be no difference from the scripts' perspective. > The other problem with -C being part of the mapping is it is now different > from the -U option to uncompress since you wouldn't do lofiadm -d -U Supporting both forms of the command line should take care of this. Alok From Timothy.Haley@sun.com Mon Oct 8 08:59:12 2007 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l98FxCJj007089 for ; Mon, 8 Oct 2007 08:59:12 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l98FtvwQ016040 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Mon, 8 Oct 2007 08:55:58 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPL00H0BO98DZ00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 08 Oct 2007 09:55:56 -0600 (MDT) Received: from brmea-mail-3.sun.com ([192.18.98.34]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPL00FYZO97YQ00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 08 Oct 2007 09:55:55 -0600 (MDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-3.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l98Ftt3R028570 for ; Mon, 08 Oct 2007 15:55:55 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPL00H01NHTRB00@mail-amer.sun.com> (original mail from Timothy.Haley@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 08 Oct 2007 09:55:55 -0600 (MDT) Received: from spidey.Central.Sun.COM ([172.20.25.27]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPL00HGWO91D410@mail-amer.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 08 Oct 2007 09:55:49 -0600 (MDT) Date: Mon, 08 Oct 2007 09:55:49 -0600 From: Tim Haley Subject: lofi(7D) compression support PSARC/2007/569 FastTrack [restart] Sender: Timothy.Haley@sun.com To: PSARC-ext@sun.com Message-id: <470A5305.3060203@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 User-Agent: Thunderbird 2.0.0.4 (X11/20070827) Status: RO Content-Length: 387 I've restarted the timer on this fast-track case; it's now set to 10/12/2007. Some new materials are present in the materials directory: -rw-r--r-- 1 timh staff 11011 Oct 4 16:43 lofiadm.1m.compress -rw-r--r-- 1 timh staff 498 Oct 4 16:43 lofi.7d.diff -rw-r--r-- 1 timh staff 5858 Oct 4 16:43 clofi.psarc Please review and comment. Thanks, -tim From Darren.Moffat@sun.com Mon Oct 8 09:09:46 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l98G9jhn007470 for ; Mon, 8 Oct 2007 09:09:46 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l98G6TUw013390 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Tue, 9 Oct 2007 00:06:31 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPL00N01OQU4S00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 08 Oct 2007 09:06:30 -0700 (PDT) Received: from gmp-eb-mail-1.sun.com ([192.18.6.21]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPL00MU9OQT7110@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 08 Oct 2007 09:06:30 -0700 (PDT) Received: from fe-emea-10.sun.com (gmp-eb-lb-2-fe1.eu.sun.com [192.18.6.10]) by gmp-eb-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l98G6T7S019253 for ; Mon, 08 Oct 2007 16:06:29 +0000 (GMT) Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPL00F01O1ZFY00@fe-emea-10.sun.com> (original mail from Darren.Moffat@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 08 Oct 2007 17:06:29 +0100 (BST) Received: from [129.156.173.21] by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPL00BQLOQB3X20@fe-emea-10.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 08 Oct 2007 17:06:12 +0100 (BST) Date: Mon, 08 Oct 2007 17:06:11 +0100 From: Darren J Moffat Subject: Re: lofi(7D) compression support PSARC/2007/569 FastTrack [restart] In-reply-to: <470A5305.3060203@sun.com> Sender: Darren.Moffat@sun.com To: Tim Haley Cc: PSARC-ext@sun.com, Alok Aggarwal Message-id: <470A5573.8020104@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <470A5305.3060203@sun.com> User-Agent: Thunderbird 2.0.0.4 (X11/20070731) Status: RO Content-Length: 621 I feel strongly that -C and -a are mutually exclusive options, as are -U and -d. -a maps a file as a device node - lofi's main job, it does not modify the file in any way (an upper layer filesystem or application may). -C/-U actually transforms a file in a destructive manner by doing a copy/modify/rename. The compress could take a long time and take a non trivial amount of cpu/memory and diskspace. So much that a progress indication would be good, however such a progress indication would be incompatible with the current output of -a since it is explicitly listed as being scriptable. -- Darren J Moffat From Alok.Aggarwal@sun.com Wed Oct 10 05:52:28 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l9ACqMdg003491 for ; Wed, 10 Oct 2007 05:52:27 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l9ACmr71028259 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 10 Oct 2007 20:49:07 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPP00D014XSVK00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Oct 2007 05:49:04 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPP00DC04XSAM00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Oct 2007 05:49:04 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l9ACn4uN012351 for ; Wed, 10 Oct 2007 12:49:04 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPP00D014QP7V00@mail-amer.sun.com> (original mail from Alok.Aggarwal@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Oct 2007 06:49:04 -0600 (MDT) Received: from sr1-ubrm-44 ([129.147.9.134]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPP007FZ4XPU3C0@mail-amer.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Oct 2007 06:49:01 -0600 (MDT) Date: Wed, 10 Oct 2007 06:49:01 -0600 (MDT) From: Alok Aggarwal Subject: Re: lofi(7D) compression support PSARC/2007/569 FastTrack [restart] In-reply-to: <470A5573.8020104@Sun.COM> Sender: Alok.Aggarwal@sun.com X-X-Sender: aalok@sr1-ubrm-44 To: Darren J Moffat Cc: Tim Haley , PSARC-ext@sun.com Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <470A5305.3060203@sun.com> <470A5573.8020104@Sun.COM> Status: RO Content-Length: 1028 On Mon, 8 Oct 2007, Darren J Moffat wrote: > I feel strongly that -C and -a are mutually exclusive options, as are -U and > -d. > > -a maps a file as a device node - lofi's main job, it does not modify the > file in any way (an upper layer filesystem or application may). > > -C/-U actually transforms a file in a destructive manner by doing a > copy/modify/rename. This is a good point. > The compress could take a long time and take a non trivial amount of > cpu/memory and diskspace. So much that a progress indication would be good, > however such a progress indication would be incompatible with the current > output of -a since it is explicitly listed as being scriptable. I'll make the change to where -a won't be allowed with -C. That is, the command line specification would be - /usr/sbin/lofiadm -C [algorithm] [-s segment_size] file The following won't be allowed - /usr/sbin/lofiadm -a [-C [algorithm] [-s segment_size]] file The case materials will be updated shortly to reflect this change. Alok From Darren.Moffat@sun.com Wed Oct 10 06:19:30 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l9ADJTnr004254 for ; Wed, 10 Oct 2007 06:19:30 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l9ADGCuL011231 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 10 Oct 2007 21:16:14 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPP00G0J671ZC00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Oct 2007 06:16:13 -0700 (PDT) Received: from gmp-eb-mail-1.sun.com ([192.18.6.21]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPP00DDM66ZAF20@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Oct 2007 06:16:12 -0700 (PDT) Received: from fe-emea-09.sun.com (gmp-eb-lb-2-fe2.eu.sun.com [192.18.6.11]) by gmp-eb-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id l9ADGBQx022758 for ; Wed, 10 Oct 2007 13:16:11 +0000 (GMT) Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPP001015KYU100@fe-emea-09.sun.com> (original mail from Darren.Moffat@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Oct 2007 14:16:11 +0100 (BST) Received: from [129.156.173.21] by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPP004SR66C2N10@fe-emea-09.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Oct 2007 14:15:49 +0100 (BST) Date: Wed, 10 Oct 2007 14:15:48 +0100 From: Darren J Moffat Subject: Re: lofi(7D) compression support PSARC/2007/569 FastTrack [restart] In-reply-to: Sender: Darren.Moffat@sun.com To: Alok Aggarwal Cc: Tim Haley , PSARC-ext@sun.com Message-id: <470CD084.1080209@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <470A5305.3060203@sun.com> <470A5573.8020104@Sun.COM> User-Agent: Thunderbird 2.0.0.4 (X11/20070731) Status: RO Content-Length: 281 I believe I have one other question unanswered which is about the size of the header/metadata. It wasn't clear to me from the case material how big the header area is - though this is less architectural and more personal interest/lower level design review. -- Darren J Moffat From Alok.Aggarwal@sun.com Wed Oct 10 06:38:32 2007 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l9ADcWWM004450 for ; Wed, 10 Oct 2007 06:38:32 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l9ADYCbU056325 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 10 Oct 2007 07:34:13 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JPP00I0T72SSI00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Oct 2007 06:35:16 -0700 (PDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JPP00DTE72QAK30@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Oct 2007 06:35:15 -0700 (PDT) Received: from fe-amer-09.sun.com ([192.18.109.79]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l9ADZ67G018874 for ; Wed, 10 Oct 2007 13:35:06 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JPP005016PX5I00@mail-amer.sun.com> (original mail from Alok.Aggarwal@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Oct 2007 07:35:06 -0600 (MDT) Received: from sr1-ubrm-44 ([129.147.9.134]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JPP00MF472HS5D0@mail-amer.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 10 Oct 2007 07:35:05 -0600 (MDT) Date: Wed, 10 Oct 2007 07:35:05 -0600 (MDT) From: Alok Aggarwal Subject: Re: lofi(7D) compression support PSARC/2007/569 FastTrack [restart] In-reply-to: <470CD084.1080209@Sun.COM> Sender: Alok.Aggarwal@sun.com X-X-Sender: aalok@sr1-ubrm-44 To: Darren J Moffat Cc: Tim Haley , PSARC-ext@sun.com Message-id: MIME-version: 1.0 Content-type: TEXT/PLAIN; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <470A5305.3060203@sun.com> <470A5573.8020104@Sun.COM> <470CD084.1080209@Sun.COM> Status: RO Content-Length: 763 On Wed, 10 Oct 2007, Darren J Moffat wrote: > I believe I have one other question unanswered which is about the size of the > header/metadata. It wasn't clear to me from the case material how big the > header area is - though this is less architectural and more personal > interest/lower level design review. So, here are the sizes of the various fields mentioned in materials/ondisk_format.png - Signature - strlen(name_of_algorithm) Segment Size - uint32_t (4 bytes) No. of Segments - uint32_t (4 bytes) Size of Last Segment - uint32_t (4 bytes) This is followed by an "Index" where each index entry is a uint64_t (8 bytes). The number of index entries depends on the size of the file in question and the segment size used to compress that file. Alok From Timothy.Haley@sun.com Wed Oct 17 16:14:04 2007 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l9HNE4G0028211 for ; Wed, 17 Oct 2007 16:14:04 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id l9HNAcoO027542 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 17 Oct 2007 16:10:42 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JQ200G0NWDTSR00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Oct 2007 17:10:41 -0600 (MDT) Received: from brmea-mail-3.sun.com ([192.18.98.34]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JQ2004RGWDSV4D0@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Oct 2007 17:10:40 -0600 (MDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-3.sun.com (8.13.6+Sun/8.12.9) with ESMTP id l9HNAevW025840 for ; Wed, 17 Oct 2007 23:10:40 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JQ200301W70VF00@mail-amer.sun.com> (original mail from Timothy.Haley@Sun.COM) for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Oct 2007 17:10:40 -0600 (MDT) Received: from spidey.Central.Sun.COM ([172.20.25.27]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JQ2008HWWDS6170@mail-amer.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed, 17 Oct 2007 17:10:40 -0600 (MDT) Date: Wed, 17 Oct 2007 17:10:35 -0600 From: Tim Haley Subject: Re: lofi(7D) compression support [PSARC/2007/569 FastTrack timeout 10/12/2007] In-reply-to: <200710020324.l923OkCx028573@spidey.Central.Sun.COM> Sender: Timothy.Haley@sun.com To: PSARC-ext@sun.com Message-id: <4716966B.7020107@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.2.0.264296 References: <200710020324.l923OkCx028573@spidey.Central.Sun.COM> User-Agent: Thunderbird 2.0.0.6 (X11/20070924) Status: RO Content-Length: 59 This case was approved during today's PSARC meeting. -tim