From sacadmin Fri Jun 18 14:38:12 2010 Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o5ILcCjW017213; Fri, 18 Jun 2010 14:38:12 -0700 (PDT) Received: (from ab196087@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id o5ILcCve017209; Fri, 18 Jun 2010 14:38:12 -0700 (PDT) Date: Fri, 18 Jun 2010 14:38:12 -0700 (PDT) From: Ali Bahrami Message-Id: <201006182138.o5ILcCve017209@sac.sfbay.sun.com> To: PSARC-record@sac.sfbay.sun.com Subject: Support for archives larger than 2 GB [PSARC/2010/224 Self Review] Status: RO Content-Length: 608 Template Version: @(#)sac_nextcase 1.70 03/30/10 SMI This information is Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. 1. Introduction 1.1. Project/Component Working Name: Support for archives larger than 2 GB 1.2. Name of Document Author/Supplier: Author: Ali Bahrami 1.3 Date of This Document: 18 June, 2010 4. Technical Description See the case directory for more detail 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: on 6.5. ARC review type: Automatic 6.6. ARC Exposure: open From Ali.Bahrami@oracle.com Fri Jun 18 14:48:26 2010 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o5ILmQef017275 for ; Fri, 18 Jun 2010 14:48:26 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.4) with ESMTP id o5ILmNgV006297 for <@sunmail2sca.sfbay.sun.com:psarc-ext@sun.com>; Fri, 18 Jun 2010 15:48:24 -0600 (MDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0L4800007CKO6Y00@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Fri, 18 Jun 2010 14:48:24 -0700 (PDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0L48007KACKNTKC0@nwk-avmta-2.sfbay.sun.com> for psarc-ext@sun.com (ORCPT psarc-ext@sun.com); Fri, 18 Jun 2010 14:48:23 -0700 (PDT) Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id o5ILmNqT010475 for ; Fri, 18 Jun 2010 21:48:23 +0000 (GMT) Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5ILXStr012533 for ; Fri, 18 Jun 2010 21:48:21 +0000 (GMT) Received: from abhmt012.oracle.com by acsmt353.oracle.com with ESMTP id 340196981276897603; Fri, 18 Jun 2010 14:46:43 -0700 Received: from [172.20.25.67] (/10.85.25.67) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 18 Jun 2010 14:46:43 -0700 Date: Fri, 18 Jun 2010 15:45:04 -0600 From: Ali Bahrami Subject: PSARC 2010/224 Support for archives larger than 2 GB To: psarc-ext@sun.com Message-id: <4C1BE8E0.6060506@Oracle.COM> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090207.4C1BE9A6.020C:SCFMA4539814,ss=1,fgs=0 User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9.1.9) Gecko/20100524 Lightning/1.0b1 Thunderbird/3.0.4 Status: RO Content-Length: 5156 I am sponsoring the following self-reviewed case for myself. It improves the ability of ar to handle ELF archives with a size in the range between 2GB to 4GB, and introduces the ability for ELF archives to be larger than 4GB. I propose to: - Deliver a 64-bit version of the ar command - Implement a 64-bit archive symbol table, capable of supporting archives larger than the current 32-bit limit of 4GB. - Add a new option to the ar command (-S) to force the use of the 64-bit symbol table format. I believe this qualifies for self review: - The operational details of the proposed 64-bit ar command is consistent with the operation of other Solaris linker tools. - The proposed 64-bit archive symbol table format is a natural extension of the existing 32-bit format. It also follows existing practice: SGI employed the same extension in their IRIX64 Unix over a decade ago. - The 64-bit symbol table is transparent to the user. No new libelf functions are required, and user programs do not need to be rebuilt. The following supporting documents can be found in the case materials: - Original and New versions of the ar(1) manpage, as well as diffs (ar.1.orig, ar.1.new, ar.1.pdf, ar.1.diffs). - Original and New versions of the ar.h(3HEAD) manpage, as well as diffs (ar.h.3head.orig, ar.h.3head.new, ar.h.3head.pdf, ar.h.3head.diffs). - The SGI/MIPS 64-bit ELF Object File Specification (Irix64ELFObjectFileSpecificationDraft2.5.pdf). The pages relevant to this case are 96-98. ----- Release Binding: Patch/Micro 64-bit ar command: Committed 64-bit archive symbol table format: Committed ar -S option: Committed ----- /usr/bin/ar is currently delivered as a 32-bit executable. A standard 32-bit executable is limited to reading and writing files that are 2GB in size or less. This limit has historically been of little concern. However, objects continue to grow, both in size, and in number, and so archives are always trending towards larger size. In 2004, a fix was implemented to ar that allowed it to produce archives larger than 2GB: 4987898 The archive utility ar gives 'could not allocate memory' on Solaris x86 platform By employing the 32-bit large file support available under Solaris, this fix allowed the creation of archives larger than 2GB. However, ar is still unable to read archives larger than 2GB, giving the current situation in which ar can produce a large archive that it cannot read: % ar t big.a ar: cannot open big.a Value too large for defined data type This occurs because ar relies on libelf for reading archives, and the 32-bit libelf is not largefile capable. Looking past this limit, a 32-bit ar is limited to a 4GB address space, and so cannot produce archives larger than that. The fix for 4987898 bought us 6 years, but is clearly insufficient for the long term. There is currently an open CR against this issue for an important customer with archives that are larger than 2GB. There are indications that the 4GB boundary will soon become an issue as well. The proper fix for all of the above issues is to deliver a 64-bit version of /usr/bin/ar (/usr/bin/{amd64,sparcv9}/ar), as is done for the majority of Solaris linker tools (ld, elfdump, etc) found under /usr/src/cmd/sgs. As with the other tools, the 32-bit version of ar will exec the 64-bit version when a 64-bit kernel is running. By itself, a 64-bit ar is not sufficient to produce archives larger than 4GB. The existing archive symbol table format, as described by ar.h.(3HEAD) is 32-bit limited, and cannot be used for archives that are larger than 4GB. It is therefore necessary to add support for an alternative archive symbol table format that employs 64-bit offsets. Fortunately, there is existing precedent in this area: SGI employed a 64-bit symbol table in IRIX64 archives over a decade ago. Due to common System V Unix roots, IRIX used the same 32-bit symbol table format as Solaris. Their 64-bit solution was to retain the layout and operation of the 32-bit format, employing 64-bit integer words rather than 32-bit words. 32-bit symbol tables are identified by the special member name "/". The 64-bit version is named "/SYM64/". The /SYM64/ symbol table format is the obvious extension to our 32-bit archive symbol tables, and has the benefit of already being well known, and supported by other ar implementations, notably the GNU ar. A further benefit is that this solution is transparent to the end user: - No changes or additions are needed to the libelf APIs - User code does not need to be recompiled. For maximum portability, the ar default will be to produce the /SYM64/ symbol table only when the archive is larger than 4GB. A new ar option (-S) will be provided to force the large format when building smaller archives. This feature is primarily to support linker testing, as the smaller format is the better choice where possible.