From Thu Apr 3 09:10:55 2008 Date: Thu, 3 Apr 2008 09:10:55 -0700 (PDT) From: Postmaster Subject: Message from mail server Content-Length: 95 Mime-Version: 1.0 Status: RO X-IMAP: 1207239055 12 Delete. This is a system message. --END+PSEUDO-- From sacadmin Sun Mar 30 22:20:44 2008 Received: from chaz.sfbay.sun.com (chaz.SFBay.Sun.COM [129.146.228.120]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m2V5Kikt015694; Sun, 30 Mar 2008 22:20:44 -0700 (PDT) Received: from chaz.sfbay.sun.com (localhost [127.0.0.1]) by chaz.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m2V5JJ1O006816; Sun, 30 Mar 2008 22:19:19 -0700 (PDT) Received: (from rie@localhost) by chaz.sfbay.sun.com (8.14.2+Sun/8.14.2/Submit) id m2V5JJOl006813; Sun, 30 Mar 2008 22:19:19 -0700 (PDT) Date: Sun, 30 Mar 2008 22:19:19 -0700 (PDT) From: Rod Evans Message-Id: <200803310519.m2V5JJOl006813@chaz.sfbay.sun.com> To: PSARC-record@sac.sfbay.sun.com Subject: mmapfd(2) - mmap file descriptor [PSARC/2008/226 FastTrack timeout 04/07/2008] Content-Length: 573 Status: RO X-Status: $$$$ X-UID: 0000000001 Template Version: @(#)sac_nextcase 1.64 07/13/07 SMI This information is Copyright 2008 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: mmapfd(2) - mmap file descriptor 1.2. Name of Document Author/Supplier: Author: Micheal Corcoran 1.3 Date of This Document: 30 March, 2008 4. Technical Description See the case directory for more detail 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: OSNet 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From Rod.Evans@sun.com Sun Mar 30 22:22:22 2008 Received: from jurassic-x4600.sfbay.sun.com (cretaceous.SFBay.Sun.COM [129.146.17.63]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m2V5MLvq015738 for ; Sun, 30 Mar 2008 22:22:21 -0700 (PDT) Received: from Macintosh-2.local (punchin-rie.SFBay.Sun.COM [10.7.251.250]) by jurassic-x4600.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m2V5MLwS101441; Sun, 30 Mar 2008 22:22:21 -0700 (PDT) Message-ID: <47F0750D.4040809@sun.com> Date: Sun, 30 Mar 2008 22:22:21 -0700 From: Rod Evans User-Agent: Thunderbird 2.0.0.12 (Macintosh/20080213) MIME-Version: 1.0 To: psarc-ext@sac.sfbay.sun.com CC: Michael Corcoran , rod.evans@sun.com Subject: 2008/226: mmapfd(2) - mmap file descriptor Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Length: 11809 Status: RO X-Status: $$$$ X-UID: 0000000002 I'm sponsoring the following case for Mike Corcoran. Time out 04/07/08. The case introduces a new system call, mmapfd(2). This call is primarily targeted for use by ld.so.1(1), and provides for the efficient mapping of ELF files (and 4.x AOUT files). Release Binding: Patch/Micro mmapfd(2): Consolidation Private -------------------------------------------------------------------------- 1. Introduction 1.1. Project/Component Working Name: mmapfd: mmap file descriptor 1.2. Name of Document Author/Supplier: Michael Corcoran 1.3. Date of This Document: 03/24/08 1.4. Name of Major Document Customer(s)/Consumer(s): 1.4.1. The PAC or CPT you expect to review your project: Solaris PAC 1.4.2. The ARC(s) you expect to review your project: PSARC 1.4.3. The Director/VP who is "Sponsoring" this project: William.Franklin@sun.com 1.4.4. The name of your business unit: Software 1.5. Email Aliases: 1.5.1. Responsible Manager: Darrin.Johnson@sun.com 1.5.2. Responsible Engineer: Michael.Corcoran@sun.com 2. Project Summary 2.1. Project Description: mmapfd is a new system call targeted for use in mapping files that need to be interpreted. The runtime linker (ld.so.1) will make use of this system call to map dynamic objects. Both ELF and AOUT (4.x) file formats are supported. Under the covers, the OS can optimize the placement of these interpreted dynamic objects. 4. Technical Description: 4.1. Details: mmapfd is a new system call which can interpret and map ELF and AOUT (4.x) objects. This system call allows the interpretation and mapping of ELF and AOUT files to be carried out completely by the kernel rather than by ld.so.1. mmapfd also provides for mapping a whole file, without interpretation in a read only mode. mmapfd returns a description of the mappings that have been used to represent the associated file. This data is used by ld.so.1 to continue processing the file - searching for dependencies, symbols and performing relocations. The data provides for individual operations to be carried out for each mapping, ie. munmap(2), mprotect(2), etc. All of the mapping capabilities that ld.so.1 provides today, such as the addition of padding (for dbx) and fixed address object mapping, are available with mmapfd. In the past, there have been many requests for ld.so.1 to handle different platforms in different manners by passing new flags or by trying to have ld.so.1 discover what to do for a given platform. By creating a single mmapfd interface, these special flags and behaviors can be removed from ld.so.1, resulting in a much cleaner ld.so.1 and more flexibility for different platforms to perform optimizations. For example, one optimization is for the kernel to use large pages where applicable. Another optimization might be to use the same virtual address for the same object among different processes. These optimizations can be achieved more easily as the kernel will now interpret the program headers of the associated file, and thus can deduce segment size and segment alignment requirements, together with the files used throughout the system. Future projects which can build off of the new mmapfd system call include: - Having elfexec and mmapfd share a common set of routines to do all mapping of ELF files. This will provide a centralized location for interpreting all ELF objects which will be easier to support and provide a consistent behavior for all applications. - Having the kernel do hardware capability checking, thus relieving ld.so.1 from having to do this work, while mapping in an ELF file. Once again, elfexec and mmapfd can share common interfaces so there is no duplication of code and consistent behaviors. - Allow DTrace better access to ELF information. DTrace engineers have been looking to access DTrace information from within an ELF file within critical regions. This DTrace information can be gathered as an extension to the ELF file processing already undertaken by mmapfd. - Allow new file types to be interpreted in the future via this central interface. 4.2. Bug/RFE Number(s): 6502792: Same dynamic libraries should be mapped at the same virtual addresses in different processes 6561987 data vac_conflict faults on lipthread libthread libs in s10 5. Reference Documents: Linker and Libraries Guide http://docs.sfbay.sun.com/app/docs/doc/819-0690 CR 6502792 Same dynamic libraries should be mapped at the same virtual addresses in different processes http://monaco.sfbay/detail.jsf?cr=6502792 Originally a request to add new linker support for shared contexts. Lots of info about why this was desired CR 6561987 data vac_conflict faults on lipthread libthread libs in s10. http://monaco.sfbay/detail.jsf?cr=6561987 We can reduce vac_conflicts with mmapfd since we can map libraries at the same virtual color throughout the kernel to prevent vac conflicts. 6. Resources and Schedule: 6.4. Product Approval Committee requested information: 6.4.1. Consolidation or Component Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open -------------------------------------------------------------------------- System Calls mmapfd(2) NAME mmapfd - map a file descriptor in the appropriate manner. SYNOPSIS #include int mmapfd(int fd, uint_t flags, mmapfd_result_t *storage, uint_t *elements, void *arg) DESCRIPTION The mmapfd() function establishes a set of mappings between a process's address space and a file. By default, mmapfd maps the whole file as a single, private, read-only mapping. The MMFD_INTERPRET flag instructs mmapfd to attempt to interpret the file and map it according to the rules for that file format. Currently only the following ELF and AOUT formats are supported. ET_EXEC and AOUT executables Result in one or more mappings whose size, alignment and protections are as described by the files program header information. The address of each mapping is explicitly defined by the files program headers. ET_DYN and AOUT shared objects Result in one or more mappings whose size, alignment and protections are as described by the files program header information. The base address of the initial mapping is obtained by mapfd(). The address of adjacent mappings are based off of this base address as explicitly defined by the files program headers. ET_REL and ET_CORE Result in a single, read-only mapping. The base address of this mapping is obtained by mmapfd(). mmapfd will not map over any currently used mappings within the process except for the case of an ELF file for which a previous reservation has been made via /dev/null. PARAMETERS fd The open file descriptor for the file to be mapped. flags Indicates that the default behavior of mmapfd should be modified accordingly. Available flags are MMFD_INTERPRET and MMFD_PADDING. storage A pointer to the mmapfd_result_t array where the mapping data will be copied out after a successful mapping of fd. elements A pointer to the number of mmapfd_result_t elements pointed to by storage. On return, elements contains the number of mappings required to fully map the requested object. If the original value of elements was too small, an error will be returned, and elements will be modified to contain the number of mappings necessary. arg A pointer to additional information that might be associated with the specific request. Presently, only the MMFD_PADDING request uses this argument. In this case, args should be a pointer to size_t that indicates how much padding is requested. This amount of padding is added before the first mapping and immediately after the last mapping. FLAGS MMFD_INTERPRET Interpret the contents of the file descriptor instead of just mapping a single image. Can only be used with ELF and AOUT files. MMFD_PADDING When mapping in the file descriptor, padding of the amount pointed to by arg is requested before the lowest mapping and after the highest mapping. TYPES USED typedef struct { caddr_t mr_addr; /* mapping address */ size_t mr_msize; /* mapping size */ size_t mr_fsize; /* file size */ size_t mr_offset; /* offset into file */ int mr_prot; /* the protections provided */ uint_t mr_flags; /* info on the mapping */ } mmapfd_result_t; Values for mr_flags include: MFD_ELF_HDR 0x1 /* the ELF header is mapped at mr_addr */ MFD_AOUT_HDR 0x2 /* the AOUT header is mapped at mr_addr */ MFD_PADDING 0x4 /* this mapping represents requested padding */ RETURN VALUES -1 indicates an error occurred and errno will hold the reason. No data will be copied to storage. On success, 0 will be returned and elements will contain how many program headers were mapped for fd. The data for these elements will be copied to storage such that the first members of the storage array will contain valid mapping data. ERROR VALUES E2BIG Elements was not large enough to hold the number of loadable segments in fd. elements will be modified to contain the number of segments required. EBADF fd was not a valid open file descriptor EPERM fd was not open for reading EINVAL MMFD_INTERPRET was specified and fd is not a valid file type to be interpreted. MMFD_PADDING was specified and arg is NULL. flags contains and invalid flag. EACCES The file system containing the fd to be interpreted does not provide for execute access. ENOMEM Insufficient memory is available to hold the program headers. EADDRINUSE The mapping requirements overlap an object that is already used by the process. EFAULT storage or args, points to an invalid address. ENOTSUP The current user data model does not match the fd to be interpreted. Thus a 32-bit process that tried to use mmapfd to interpret a 64-bit object would return ENOTSUP. fd is an ELF file whose type can not be interpreted. SEE ALSO ld.so.1(1), mmap(2), attributes(5) Linker and Libraries Guide -------------------------------------------------------------------------- -- Rod From Darren.Moffat@Sun.COM Mon Mar 31 02:30:21 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m2V9ULSt021657 for ; Mon, 31 Mar 2008 02:30:21 -0700 (PDT) Received: from gmp-eb-inf-2.sun.com (gmp-eb-inf-2.EU.Sun.COM [192.18.6.24]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m2V9UKmt019047 for ; Mon, 31 Mar 2008 02:30:21 -0700 (PDT) Received: from fe-emea-09.sun.com (gmp-eb-lb-2-fe3.eu.sun.com [192.18.6.12]) by gmp-eb-inf-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m2V9UFGi009054 for ; Mon, 31 Mar 2008 09:30:15 GMT Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JYL007017KSTP00@fe-emea-09.sun.com> (original mail from Darren.Moffat@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Mon, 31 Mar 2008 10:30:15 +0100 (BST) Received: from [129.156.173.199] by fe-emea-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JYL00M6Y922M310@fe-emea-09.sun.com>; Mon, 31 Mar 2008 10:30:02 +0100 (BST) Date: Mon, 31 Mar 2008 10:30:01 +0100 From: Darren J Moffat Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor In-reply-to: <47F0750D.4040809@sun.com> Sender: Darren.Moffat@Sun.COM To: Rod Evans Cc: psarc-ext@sac.sfbay.sun.com, Michael Corcoran Message-id: <47F0AF19.7050609@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <47F0750D.4040809@sun.com> User-Agent: Thunderbird 2.0.0.9 (X11/20080225) Content-Length: 431 Status: RO X-Status: $$$$ X-UID: 0000000003 I find the function name a bit strange. So strange in fact it made me look at the mmap(2) man page to check that it wasn't taking a char* rather than an int for the file since I was sure it took and fd. Since mmap(2) already takes an fd. The mmap(2) versus mmapfd(2) makes me think of fopen(3C) versus fdopen(3C). As for a better name it looks to me more like mmapv(2), eg like read(2) versus readv(2). -- Darren J Moffat From roland.mainz@nrubsig.org Mon Mar 31 04:37:57 2008 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m2VBbvcw024098 for ; Mon, 31 Mar 2008 04:37:57 -0700 (PDT) Received: from sca-ea-mail-1.sun.com (sca-ea-mail-1.Sun.COM [192.18.43.24]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m2VBbvqm055793 for ; Mon, 31 Mar 2008 04:37:57 -0700 (PDT) Received: from relay24.sun.com (ip192-12-251-74.block6.us.syntegra.com [192.12.251.74]) by sca-ea-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m2VBSG14020655 for ; Mon, 31 Mar 2008 11:37:57 GMT Received: from mms23es.sun.com ([150.143.232.54] [150.143.232.54]) by relay24i.sun.com with ESMTP id BT-MMP-1597866 for psarc-ext@sac.sfbay.sun.com; Mon, 31 Mar 2008 11:37:48 Z Received: from relay21.sun.com (relay21.sun.com [192.12.251.24]) by mms23es.sun.com with ESMTP id BT-MMP-649200 for psarc-ext@sac.sfbay.sun.com; Mon, 31 Mar 2008 11:37:48 Z Received: from mail-in-10.arcor-online.net ([151.189.21.50] [151.189.21.50]) by relay21i.sun.com with ESMTP id BT-MMP-8100596 for psarc-ext@sac.sfbay.sun.com; Mon, 31 Mar 2008 11:37:47 Z Received: from mail-in-15-z2.arcor-online.net (mail-in-15-z2.arcor-online.net [151.189.8.32]) by mail-in-10.arcor-online.net (Postfix) with ESMTP id 84FE21F509F; Mon, 31 Mar 2008 13:37:46 +0200 (CEST) Received: from mail-in-13.arcor-online.net (mail-in-13.arcor-online.net [151.189.21.53]) by mail-in-15-z2.arcor-online.net (Postfix) with ESMTP id 681B372415F; Mon, 31 Mar 2008 13:37:46 +0200 (CEST) Received: from jupiterb48.nrubsig.org (dslb-084-059-024-183.pools.arcor-ip.net [84.59.24.183]) by mail-in-13.arcor-online.net (Postfix) with ESMTP id 8E25922D167; Mon, 31 Mar 2008 13:37:45 +0200 (CEST) Received: from nrubsig.org (localhost [127.0.0.1]) by jupiterb48.nrubsig.org (8.13.8+Sun/8.13.8) with ESMTP id m2VBbg5u002373; Mon, 31 Mar 2008 13:37:43 +0200 (CEST) Sender: gisburn@jupiterb48.nrubsig.org Message-ID: <47F0CD06.25D53234@nrubsig.org> Date: Mon, 31 Mar 2008 13:37:42 +0200 From: Roland Mainz X-Brightmail-Tracker: AAAAAA== X-Mailer: Mozilla 4.76 [en] (X11; U; SunOS 5.11 sun4u) X-Accept-Language: en To: Rod Evans CC: psarc-ext@sac.sfbay.sun.com, Michael Corcoran Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor References: <47F0750D.4040809@sun.com> X-Virus-Scanned: ClamAV 0.92.1/6495/Mon Mar 31 04:43:39 2008 on mail-in-13.arcor-online.net X-Virus-Status: Clean X-Antispam: No, score=-2.6/5.0, scanned in 0.120sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Length: 6119 Status: RO X-Status: $$$$ X-UID: 0000000004 Rod Evans wrote: > > I'm sponsoring the following case for Mike Corcoran. Time out 04/07/08. > > The case introduces a new system call, mmapfd(2). This call is primarily > targeted for use by ld.so.1(1), and provides for the efficient mapping of > ELF files (and 4.x AOUT files). > > Release Binding: Patch/Micro > mmapfd(2): Consolidation Private > > -------------------------------------------------------------------------- > > 1. Introduction > 1.1. Project/Component Working Name: > mmapfd: mmap file descriptor [snip] > 4. Technical Description: > 4.1. Details: > mmapfd is a new system call which can interpret and map ELF and AOUT > (4.x) objects. This system call allows the interpretation and mapping > of ELF and AOUT files to be carried out completely by the kernel rather > than by ld.so.1. Erm... the call seems to be ELF+AOUT-specifc - why does it have such a generic name ? The call can't handle other types of executables (e.g. "javaexec" or "shbinexec") nor does it seem to be usefull to map normal files... ... what renaming the call to |mmapexecfd()| (= "memory map of executable fd") ? > mmapfd also provides for mapping a whole file, without interpretation > in a read only mode. What does that mean ? Can these data+MMU mappings be shared between processes ? [snip] > System Calls mmapfd(2) > > NAME > > mmapfd - map a file descriptor in the appropriate manner. > > SYNOPSIS > > #include > > int > mmapfd(int fd, uint_t flags, mmapfd_result_t *storage, > uint_t *elements, void *arg) Uhm... how do I unmap the mapping done by |mmapfd()| ? > DESCRIPTION > > The mmapfd() function establishes a set of mappings between a process's > address space and a file. By default, mmapfd maps the whole file as a single, > private, read-only mapping. The MMFD_INTERPRET flag instructs mmapfd to > attempt to interpret the file and map it according to the rules for that file > format. Currently only the following ELF and AOUT formats are supported. What will happen if the file is executable but uses an unsupported format (e.g. "javaexec", "shbinexec" etc.) ? > ET_EXEC and AOUT executables > Result in one or more mappings whose size, alignment and protections > are as described by the files program header information. The address > of each mapping is explicitly defined by the files program headers. > > ET_DYN and AOUT shared objects > Result in one or more mappings whose size, alignment and protections > are as described by the files program header information. The base > address of the initial mapping is obtained by mapfd(). The address of > adjacent mappings are based off of this base address as explicitly > defined by the files program headers. > > ET_REL and ET_CORE > Result in a single, read-only mapping. The base address of this > mapping is obtained by mmapfd(). > > mmapfd will not map over any currently used mappings within the process > except for the case of an ELF file for which a previous reservation has been > made via /dev/null. > > PARAMETERS > > fd The open file descriptor for the file to be mapped. > > flags Indicates that the default behavior of mmapfd should be modified > accordingly. Available flags are MMFD_INTERPRET and MMFD_PADDING. > > storage > A pointer to the mmapfd_result_t array where the mapping > data will be copied out after a successful mapping of fd. > > elements > A pointer to the number of mmapfd_result_t elements pointed to by > storage. On return, elements contains the number of mappings required > to fully map the requested object. If the original value of > elements was too small, an error will be returned, and elements > will be modified to contain the number of mappings necessary. > > arg A pointer to additional information that might be associated with the > specific request. Presently, only the MMFD_PADDING request uses this > argument. In this case, args should be a pointer to size_t that > indicates how much padding is requested. This amount of padding is > added before the first mapping and immediately after the last mapping. > > FLAGS > > MMFD_INTERPRET > Interpret the contents of the file descriptor instead of just mapping a > single image. Can only be used with ELF and AOUT files. > > MMFD_PADDING > When mapping in the file descriptor, padding of the amount pointed to by > arg is requested before the lowest mapping and after the highest > mapping. > > TYPES USED > > typedef struct { > caddr_t mr_addr; /* mapping address */ > size_t mr_msize; /* mapping size */ > size_t mr_fsize; /* file size */ > size_t mr_offset; /* offset into file */ > int mr_prot; /* the protections provided */ Why is this |signed int| ? > uint_t mr_flags; /* info on the mapping */ Please change this to |uint64_t| (e.g. it may be nice to have more flags available by default). > } mmapfd_result_t; > > Values for mr_flags include: > > MFD_ELF_HDR 0x1 /* the ELF header is mapped at mr_addr */ > MFD_AOUT_HDR 0x2 /* the AOUT header is mapped at mr_addr */ What about reserving the first four bits for |MFD_*_HDR| flags (|MFD_ELF_HDR|, |MFD_AOUT_HDR| and two bits reserved for future |MFD_*_HDR| flags) ? Finally... how can I _force_ the mapping to use something like 64k pages by default ? ---- Bye, Roland -- __ . . __ (o.\ \/ /.o) roland.mainz@nrubsig.org \__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer /O /==\ O\ TEL +49 641 7950090 (;O/ \/ \O;) From Nicolas.Williams@sun.com Mon Mar 31 08:30:22 2008 Received: from binky.Central.Sun.COM (binky.Central.Sun.COM [129.153.128.104]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m2VFUMJ0029350 for ; Mon, 31 Mar 2008 08:30:22 -0700 (PDT) Received: from binky.Central.Sun.COM (localhost [127.0.0.1]) by binky.Central.Sun.COM (8.14.1+Sun/8.14.1) with ESMTP id m2VFULRA028210; Mon, 31 Mar 2008 10:30:21 -0500 (CDT) Received: (from nw141292@localhost) by binky.Central.Sun.COM (8.14.1+Sun/8.14.1/Submit) id m2VFULJq028209; Mon, 31 Mar 2008 10:30:21 -0500 (CDT) X-Authentication-Warning: binky.Central.Sun.COM: nw141292 set sender to Nicolas.Williams@sun.com using -f Date: Mon, 31 Mar 2008 10:30:20 -0500 From: Nicolas Williams To: Roland Mainz Cc: Rod Evans , psarc-ext@sac.sfbay.sun.com, Michael Corcoran Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor Message-ID: <20080331153020.GG16998@Sun.COM> Mail-Followup-To: Roland Mainz , Rod Evans , psarc-ext@sac.sfbay.sun.com, Michael Corcoran References: <47F0750D.4040809@sun.com> <47F0CD06.25D53234@nrubsig.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <47F0CD06.25D53234@nrubsig.org> User-Agent: Mutt/1.5.7i Content-Length: 1670 Status: RO X-Status: $$$$ X-UID: 0000000005 On Mon, Mar 31, 2008 at 01:37:42PM +0200, Roland Mainz wrote: > Rod Evans wrote: > > mmapfd(int fd, uint_t flags, mmapfd_result_t *storage, > > uint_t *elements, void *arg) > > Uhm... how do I unmap the mapping done by |mmapfd()| ? I thought that was clear (each mmapfd_result_t's mr_addr can be used as an argument to munmap(), mprotect(), ...). > > uint_t mr_flags; /* info on the mapping */ > > Please change this to |uint64_t| (e.g. it may be nice to have more flags > available by default). Same thing with the flags argument, no? > > } mmapfd_result_t; > > > > Values for mr_flags include: > > > > MFD_ELF_HDR 0x1 /* the ELF header is mapped at mr_addr */ > > MFD_AOUT_HDR 0x2 /* the AOUT header is mapped at mr_addr */ > > What about reserving the first four bits for |MFD_*_HDR| flags > (|MFD_ELF_HDR|, |MFD_AOUT_HDR| and two bits reserved for future > |MFD_*_HDR| flags) ? Why? Isn't the ARC going to be the authority for this namespace anyways? My interpretation is that all currently unused flag bits are reserved. Or did you mean that file format shouldn't be a flag but an integer value? My comments: I'm curious about the 'arg' argument. It's interpretation depends on the flags, but presumably there could be many flags in use in one call that make use of 'arg'. My guess is that arg would have to point to a concatenation (with padding) in memory of each flag's arg in (descending?) numeric order of the flags. But this is nothing that can't be settled when we come to that bridge. I agree with Darren's comment about the system call name. It's odd. Nico -- From Michael.Corcoran@Sun.COM Mon Mar 31 11:46:41 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m2VIkfOe005393 for ; Mon, 31 Mar 2008 11:46:41 -0700 (PDT) Received: from brmea-mail-4.sun.com (brmea-mail-4.Sun.COM [192.18.98.36]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m2VIkfJG048488 for ; Mon, 31 Mar 2008 11:46:41 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m2VIkf3r028704 for ; Mon, 31 Mar 2008 18:46:41 GMT Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JYL00801W4XHB00@mail-amer.sun.com> (original mail from Michael.Corcoran@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Mon, 31 Mar 2008 12:46:41 -0600 (MDT) Received: from musetta ([129.153.88.82]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JYL005YEYTQKC70@mail-amer.sun.com>; Mon, 31 Mar 2008 12:46:38 -0600 (MDT) Date: Mon, 31 Mar 2008 11:46:34 -0700 From: Michael Corcoran Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor In-reply-to: <47F0AF19.7050609@Sun.COM> Sender: Michael.Corcoran@Sun.COM To: Darren J Moffat Cc: Michael.Corcoran@Sun.COM, Rod Evans , psarc-ext@sac.sfbay.sun.com Message-id: <1206989194.1298.186.camel@musetta> MIME-version: 1.0 X-Mailer: Evolution 2.6.2 Content-type: text/plain Content-transfer-encoding: 7BIT References: <47F0750D.4040809@sun.com> <47F0AF19.7050609@Sun.COM> Content-Length: 873 Status: RO X-Status: $$$$ X-UID: 0000000006 On Mon, 2008-03-31 at 10:30 +0100, Darren J Moffat wrote: > I find the function name a bit strange. So strange in fact it made me > look at the mmap(2) man page to check that it wasn't taking a char* > rather than an int for the file since I was sure it took and fd. Since > mmap(2) already takes an fd. > > The mmap(2) versus mmapfd(2) makes me think of fopen(3C) versus fdopen(3C). > > As for a better name it looks to me more like mmapv(2), eg like read(2) > versus readv(2). A good name for this syscall has been elusive :) George has always wanted to call it "mapme(2)" but that seemed a bit odd. I agree that mmapfd(2) doesn't seem like it differentiates itself from mmap(2) enough. mmapv(2) does seem to follow readv(2) more closely so maybe that will be a better name. I'm open for other suggestions as well. Thanks, Mike > > -- > Darren J Moffat From edward.pilatowicz@sun.com Mon Mar 31 11:52:11 2008 Received: from jurassic-x4600.sfbay.sun.com (cretaceous.SFBay.Sun.COM [129.146.17.63]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m2VIqBYM005537 for ; Mon, 31 Mar 2008 11:52:11 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com (localhost [127.0.0.1]) by jurassic-x4600.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m2VIqBNH181035 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 31 Mar 2008 11:52:11 -0700 (PDT) Received: (from edp@localhost) by jurassic-x4600.sfbay.sun.com (8.14.2+Sun/8.14.2/Submit) id m2VIqB7Q181034; Mon, 31 Mar 2008 11:52:11 -0700 (PDT) X-Authentication-Warning: jurassic-x4600.sfbay.sun.com: edp set sender to edward.pilatowicz@sun.com using -f Date: Mon, 31 Mar 2008 11:52:11 -0700 From: Edward Pilatowicz To: Michael Corcoran Cc: Darren J Moffat , Rod Evans , psarc-ext@sac.sfbay.sun.com Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor Message-ID: <20080331185211.GE161409@eng.sun.com> References: <47F0750D.4040809@sun.com> <47F0AF19.7050609@Sun.COM> <1206989194.1298.186.camel@musetta> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1206989194.1298.186.camel@musetta> User-Agent: Mutt/1.4.2.1i Content-Length: 1112 Status: RO X-Status: $$$$ X-UID: 0000000007 On Mon, Mar 31, 2008 at 11:46:34AM -0700, Michael Corcoran wrote: > On Mon, 2008-03-31 at 10:30 +0100, Darren J Moffat wrote: > > I find the function name a bit strange. So strange in fact it made me > > look at the mmap(2) man page to check that it wasn't taking a char* > > rather than an int for the file since I was sure it took and fd. Since > > mmap(2) already takes an fd. > > > > The mmap(2) versus mmapfd(2) makes me think of fopen(3C) versus fdopen(3C). > > > > As for a better name it looks to me more like mmapv(2), eg like read(2) > > versus readv(2). > > A good name for this syscall has been elusive :) George has always > wanted to call it "mapme(2)" but that seemed a bit odd. I agree that > mmapfd(2) doesn't seem like it differentiates itself from mmap(2) > enough. mmapv(2) does seem to follow readv(2) more closely so maybe > that will be a better name. I'm open for other suggestions as well. > given that this system call interprets and maps different kinds of objects (and the psarc case says it could be extended to understand other object types as well) how about mmapobj(). ed From Michael.Corcoran@Sun.COM Mon Mar 31 12:44:36 2008 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m2VJiaXC007928 for ; Mon, 31 Mar 2008 12:44:36 -0700 (PDT) Received: from brmea-mail-4.sun.com (brmea-mail-4.Sun.COM [192.18.98.36]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m2VJiZnF002135 for ; Mon, 31 Mar 2008 12:44:35 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m2VJiZGe025318 for ; Mon, 31 Mar 2008 19:44:35 GMT Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JYM007010IC8900@mail-amer.sun.com> (original mail from Michael.Corcoran@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Mon, 31 Mar 2008 13:44:35 -0600 (MDT) Received: from musetta ([129.153.88.82]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JYM00DP91GYKSB0@mail-amer.sun.com>; Mon, 31 Mar 2008 13:43:46 -0600 (MDT) Date: Mon, 31 Mar 2008 12:43:42 -0700 From: Michael Corcoran Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor In-reply-to: <47F0CD06.25D53234@nrubsig.org> Sender: Michael.Corcoran@Sun.COM To: Roland Mainz Cc: Michael.Corcoran@Sun.COM, Rod Evans , psarc-ext@sac.sfbay.sun.com Message-id: <1206992622.1298.222.camel@musetta> MIME-version: 1.0 X-Mailer: Evolution 2.6.2 Content-type: text/plain Content-transfer-encoding: 7BIT References: <47F0750D.4040809@sun.com> <47F0CD06.25D53234@nrubsig.org> Content-Length: 9303 Status: RO X-Status: $$$$ X-UID: 0000000008 On Mon, 2008-03-31 at 13:37 +0200, Roland Mainz wrote: > Rod Evans wrote: > > > > I'm sponsoring the following case for Mike Corcoran. Time out 04/07/08. > > > > The case introduces a new system call, mmapfd(2). This call is primarily > > targeted for use by ld.so.1(1), and provides for the efficient mapping of > > ELF files (and 4.x AOUT files). > > > > Release Binding: Patch/Micro > > mmapfd(2): Consolidation Private > > > > -------------------------------------------------------------------------- > > > > 1. Introduction > > 1.1. Project/Component Working Name: > > mmapfd: mmap file descriptor > [snip] > > 4. Technical Description: > > 4.1. Details: > > mmapfd is a new system call which can interpret and map ELF and AOUT > > (4.x) objects. This system call allows the interpretation and mapping > > of ELF and AOUT files to be carried out completely by the kernel rather > > than by ld.so.1. > > Erm... the call seems to be ELF+AOUT-specifc - why does it have such a > generic name ? It has a generic name for future expansion. > The call can't handle other types of executables (e.g. > "javaexec" or "shbinexec") Not yet, but that doesn't mean that it won't. If there is a file format that needs to be interpreted in order to be mapped, this new syscall is a good way of going about doing that. > nor does it seem to be usefull to map normal > files... Also not yet. You can map a whole file read only and we've talked about allowing mapping read/write ... but I have questions about why someone would use this interface over mmap(2) when dealing with files which don't need interpretation. mmapfd(2) only deals with whole files while mmap(2) lets you map a range within a file. The burning question is "Why would someone use mmapfd(2) over mmap(2) for mapping a file?" I don't have a good answer for that yet. I went down the path of implementing just that but right now it seems like a feature no one would use and thus I don't want to commit to an interface for that. A future phase of this project can address that issue. > ... what renaming the call to |mmapexecfd()| (= "memory map of > executable fd") ? > I like Darren's suggestion of mmapv(2) better so far. Why limit us to only being able to interpret executable files? What if there's some other non-executable file type that would naturally use this interface? > > mmapfd also provides for mapping a whole file, without interpretation > > in a read only mode. > > What does that mean ? Can these data+MMU mappings be shared between > processes ? > It means that you can map a file read-only without doing any interpretation of the file. Thus if you passed in an ELF file without the MMFD_INTERPRET flag, the file would get mapped as a single read-only segment. The data+MMU mappings would not be shared between processes. > [snip] > > System Calls mmapfd(2) > > > > NAME > > > > mmapfd - map a file descriptor in the appropriate manner. > > > > SYNOPSIS > > > > #include > > > > int > > mmapfd(int fd, uint_t flags, mmapfd_result_t *storage, > > uint_t *elements, void *arg) > > Uhm... how do I unmap the mapping done by |mmapfd()| ? > Nico answered this in a subsequent post and you would use munmap(2) on each element in the "storage" array to unmap that segment. I agree that this should be explicitly pointed out somewhere since it's not clear. > > DESCRIPTION > > > > The mmapfd() function establishes a set of mappings between a process's > > address space and a file. By default, mmapfd maps the whole file as a single, > > private, read-only mapping. The MMFD_INTERPRET flag instructs mmapfd to > > attempt to interpret the file and map it according to the rules for that file > > format. Currently only the following ELF and AOUT formats are supported. > > What will happen if the file is executable but uses an unsupported > format (e.g. "javaexec", "shbinexec" etc.) ? > If the MMFD_INTERPRET flag is specified and it's not ELF or AOUT (eg. "javaexec", "shbinexec", ...) then EINVAL will be returned as specified later in the man page. If the MMFD_INTERPRET flag is not set, then the file will get mapped as a single read-only segment. > > ET_EXEC and AOUT executables > > Result in one or more mappings whose size, alignment and protections > > are as described by the files program header information. The address > > of each mapping is explicitly defined by the files program headers. > > > > ET_DYN and AOUT shared objects > > Result in one or more mappings whose size, alignment and protections > > are as described by the files program header information. The base > > address of the initial mapping is obtained by mapfd(). The address of > > adjacent mappings are based off of this base address as explicitly > > defined by the files program headers. > > > > ET_REL and ET_CORE > > Result in a single, read-only mapping. The base address of this > > mapping is obtained by mmapfd(). > > > > mmapfd will not map over any currently used mappings within the process > > except for the case of an ELF file for which a previous reservation has been > > made via /dev/null. > > > > PARAMETERS > > > > fd The open file descriptor for the file to be mapped. > > > > flags Indicates that the default behavior of mmapfd should be modified > > accordingly. Available flags are MMFD_INTERPRET and MMFD_PADDING. > > > > storage > > A pointer to the mmapfd_result_t array where the mapping > > data will be copied out after a successful mapping of fd. > > > > elements > > A pointer to the number of mmapfd_result_t elements pointed to by > > storage. On return, elements contains the number of mappings required > > to fully map the requested object. If the original value of > > elements was too small, an error will be returned, and elements > > will be modified to contain the number of mappings necessary. > > > > arg A pointer to additional information that might be associated with the > > specific request. Presently, only the MMFD_PADDING request uses this > > argument. In this case, args should be a pointer to size_t that > > indicates how much padding is requested. This amount of padding is > > added before the first mapping and immediately after the last mapping. > > > > FLAGS > > > > MMFD_INTERPRET > > Interpret the contents of the file descriptor instead of just mapping a > > single image. Can only be used with ELF and AOUT files. > > > > MMFD_PADDING > > When mapping in the file descriptor, padding of the amount pointed to by > > arg is requested before the lowest mapping and after the highest > > mapping. > > > > TYPES USED > > > > typedef struct { > > caddr_t mr_addr; /* mapping address */ > > size_t mr_msize; /* mapping size */ > > size_t mr_fsize; /* file size */ > > size_t mr_offset; /* offset into file */ > > int mr_prot; /* the protections provided */ > > Why is this |signed int| ? I agree it should be uint_t or even uchar_t. Both appear to be commonly used throughout the kernel. uint_t is my preference here. > > > uint_t mr_flags; /* info on the mapping */ > > Please change this to |uint64_t| (e.g. it may be nice to have more flags > available by default). > I go back and forth on this. uint64_t implies that 32 flags will not be enough. 32 seems like a lot but since this interface should be around for a long time, maybe 64 would be better since it will take a long time to use all 64 :) > > } mmapfd_result_t; > > > > Values for mr_flags include: > > > > MFD_ELF_HDR 0x1 /* the ELF header is mapped at mr_addr */ > > MFD_AOUT_HDR 0x2 /* the AOUT header is mapped at mr_addr */ > > What about reserving the first four bits for |MFD_*_HDR| flags > (|MFD_ELF_HDR|, |MFD_AOUT_HDR| and two bits reserved for future > |MFD_*_HDR| flags) ? > I can see the vanity reason for doing this, it's nice to clump things in groups of 4 with everything in the group being related, but at the same time, why the first 4 flags and not the first 8 or 16 or ... I think densely packing the bits is easiest. I'm willing to move MFD_PADDING first in the list aka 0x1 so that future header types will follow numerically from the current header types. I'll make this change since it is aesthetically more pleasing. > Finally... how can I _force_ the mapping to use something like 64k pages > by default ? Using mpss.so.1 or ppgsz are ways to get the heap segment to be a specific size. Other than that, there is no control for the page size used by the other segments and the kernel will pick the best size for the given platform. Thanks for the comments, Mike > > ---- > > Bye, > Roland > From blakej@cagey.sfbay.sun.com Mon Mar 31 13:33:22 2008 Received: from cagey.sfbay.sun.com (cagey.SFBay.Sun.COM [129.146.228.133]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m2VKXMi2009342 for ; Mon, 31 Mar 2008 13:33:22 -0700 (PDT) Received: from cagey (localhost [127.0.0.1]) by cagey.sfbay.sun.com (8.14.1+Sun/8.14.1) with ESMTP id m2VKPqje016223; Mon, 31 Mar 2008 13:25:52 -0700 (PDT) To: Roland Mainz cc: Michael Corcoran , Rod Evans , psarc-ext@sac.sfbay.sun.com Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor In-Reply-To: Your message of "Mon, 31 Mar 2008 12:43:42 PDT." <1206992622.1298.222.camel@musetta> From: Blake Jones Reply-To: Blake Jones Date: Mon, 31 Mar 2008 13:25:52 -0700 Message-ID: <16222.1206995152@cagey> Sender: blakej@cagey.sfbay.sun.com Content-Length: 1443 Status: RO X-Status: $$$$ X-UID: 0000000009 >> Finally... how can I _force_ the mapping to use something like 64k >> pages by default ? > > Using mpss.so.1 or ppgsz are ways to get the heap segment to be a > specific size. Or, if you have the source code for the relevant program, call memcntl(MC_HAT_ADVISE), which is what mpss.so.1 and ppgsz end up calling. (And note that memcntl(2) says that MC_HAT_ADVISE "advise[s] [the] system how a region of user-mapped memory will be accessed"; it doesn't actually *force* the system to use 64k pages. In practice, of course, we generally do follow that advice.) > Other than that, there is no control for the page size used by the > other segments and the kernel will pick the best size for the given > platform. To elaborate on this: part of the idea behind mmapfd() is that it gives the kernel the flexibility to make global-scope decisions about where and how a file should be mapped. This includes the appropriate page size to use (which might vary depending on what platform you're running on, or even potentially which CPU on a heterogeneous machine), what virtual address to start the mapping at, how much padding to leave between mappings, etc. Applications shouldn't have to know how the MMU is designed in order to perform well, and we're trying to discourage them from doing so. The auto-MPSS project already makes some decisions about what page size to use; mmapfd() is just another step in that general direction. Blake From carlsonj@phorcys.east.sun.com Mon Mar 31 13:47:54 2008 Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m2VKlsv2010071 for ; Mon, 31 Mar 2008 13:47:54 -0700 (PDT) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m2VKeNf8001793; Mon, 31 Mar 2008 16:40:23 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.2+Sun/8.14.2/Submit) id m2VKeNek001788; Mon, 31 Mar 2008 16:40:23 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18417.19493.631606.468310@gargle.gargle.HOWL> Date: Mon, 31 Mar 2008 16:40:05 -0400 From: James Carlson To: Michael Corcoran Cc: Roland Mainz , Rod Evans , psarc-ext@sac.sfbay.sun.com Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor In-Reply-To: <1206992622.1298.222.camel@musetta> References: <47F0750D.4040809@sun.com> <47F0CD06.25D53234@nrubsig.org> <1206992622.1298.222.camel@musetta> X-Mailer: VM 7.01 under Emacs 21.3.1 Content-Length: 1223 Status: RO X-Status: $$$$ X-UID: 0000000010 Michael Corcoran writes: > > ... what renaming the call to |mmapexecfd()| (= "memory map of > > executable fd") ? > > > I like Darren's suggestion of mmapv(2) better so far. Why limit us to > only being able to interpret executable files? What if there's some > other non-executable file type that would naturally use this interface? (Everyone loves a good naming project, right? ;-) The implication of "v" in those existing interfaces is that the caller is in control of the vector, and that's not true for this new case, so it's not quite like readv/writev. I think I like `mmapobj' better, with "object" referring to a generic name for a thing, rather than meaning "object file." If we later want to extend it to other files that have embedded structure (say, OLE), we could do it. The only problem we'd have is that some of those other structured types allow _nested_ elements, and it's a little unclear how that'd work out in a nice flat address space ... but I guess I'm not worried. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From Michael.Corcoran@Sun.COM Mon Mar 31 13:48:22 2008 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m2VKmMRQ010088 for ; Mon, 31 Mar 2008 13:48:22 -0700 (PDT) Received: from brmea-mail-2.sun.com (brmea-mail-2.Sun.COM [192.18.98.43]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m2VKmLDH049444 for ; Mon, 31 Mar 2008 13:48:21 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m2VKmLeV022607 for ; Mon, 31 Mar 2008 20:48:21 GMT Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JYM00K012HGAN00@mail-amer.sun.com> (original mail from Michael.Corcoran@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Mon, 31 Mar 2008 14:48:21 -0600 (MDT) Received: from musetta ([129.153.88.82]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JYM00MWN4GA3J10@mail-amer.sun.com>; Mon, 31 Mar 2008 14:48:11 -0600 (MDT) Date: Mon, 31 Mar 2008 13:48:06 -0700 From: Michael Corcoran Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor In-reply-to: <20080331153020.GG16998@Sun.COM> Sender: Michael.Corcoran@Sun.COM To: Nicolas Williams Cc: Michael.Corcoran@Sun.COM, Roland Mainz , Rod Evans , psarc-ext@sac.sfbay.sun.com Message-id: <1206996486.1298.228.camel@musetta> MIME-version: 1.0 X-Mailer: Evolution 2.6.2 Content-type: text/plain Content-transfer-encoding: 7BIT References: <47F0750D.4040809@sun.com> <47F0CD06.25D53234@nrubsig.org> <20080331153020.GG16998@Sun.COM> Content-Length: 2118 Status: RO X-Status: $$$$ X-UID: 0000000011 On Mon, 2008-03-31 at 10:30 -0500, Nicolas Williams wrote: > On Mon, Mar 31, 2008 at 01:37:42PM +0200, Roland Mainz wrote: > > Rod Evans wrote: > > > mmapfd(int fd, uint_t flags, mmapfd_result_t *storage, > > > uint_t *elements, void *arg) > > > > Uhm... how do I unmap the mapping done by |mmapfd()| ? > > I thought that was clear (each mmapfd_result_t's mr_addr can be used as > an argument to munmap(), mprotect(), ...). > Correct. > > > uint_t mr_flags; /* info on the mapping */ > > > > Please change this to |uint64_t| (e.g. it may be nice to have more flags > > available by default). > > Same thing with the flags argument, no? > Same argument could be made. > > > } mmapfd_result_t; > > > > > > Values for mr_flags include: > > > > > > MFD_ELF_HDR 0x1 /* the ELF header is mapped at mr_addr */ > > > MFD_AOUT_HDR 0x2 /* the AOUT header is mapped at mr_addr */ > > > > What about reserving the first four bits for |MFD_*_HDR| flags > > (|MFD_ELF_HDR|, |MFD_AOUT_HDR| and two bits reserved for future > > |MFD_*_HDR| flags) ? > > Why? Isn't the ARC going to be the authority for this namespace > anyways? My interpretation is that all currently unused flag bits are > reserved. Or did you mean that file format shouldn't be a flag but an > integer value? > > My comments: > > I'm curious about the 'arg' argument. It's interpretation depends on > the flags, but presumably there could be many flags in use in one call > that make use of 'arg'. My guess is that arg would have to point to a > concatenation (with padding) in memory of each flag's arg in > (descending?) numeric order of the flags. But this is nothing that > can't be settled when we come to that bridge. > Exactly. There are probably a few different ways that we could expand the "arg" argument in the future to handle data needed for new flags. We made it a void * (pointer to size_t) for just this reason. > I agree with Darren's comment about the system call name. It's odd. I agree. We'll be changing it to something new. --Mike > Nico From gww@eng.sun.com Wed Apr 2 12:20:27 2008 Received: from dm-eng-02.sfbay.sun.com (dm-eng-02.SFBay.Sun.COM [129.146.11.32]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m32JKQ3t015873 for ; Wed, 2 Apr 2008 12:20:26 -0700 (PDT) Received: from marduk.eng.sun.com (marduk.SFBay.Sun.COM [129.146.108.224]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m32JKQFJ034264; Wed, 2 Apr 2008 12:20:26 -0700 (PDT) Received: from marduk.eng.sun.com (localhost [127.0.0.1]) by marduk.eng.sun.com (8.13.6+Sun/8.12.11) with ESMTP id m32KL19O024079; Wed, 2 Apr 2008 12:21:01 -0800 (PST) Received: (from gww@localhost) by marduk.eng.sun.com (8.13.6+Sun/8.12.11/Submit) id m32KL0i2024078; Wed, 2 Apr 2008 12:21:00 -0800 (PST) Date: Wed, 2 Apr 2008 12:21:00 -0800 (PST) From: Gary Winiger Message-Id: <200804022021.m32KL0i2024078@marduk.eng.sun.com> To: Rod.Evans@sun.com, psarc-ext@sac.sfbay.sun.com Cc: Michael.Corcoran@sun.com Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor Content-Length: 139 Status: RO X-Status: $$$$ X-UID: 0000000012 At today's PSARC meeting, it was asked to extend the timer to 9Apr. As the case owner was present, I volunteered and have done so. Gary.. From gww@eng.sun.com Thu Apr 3 16:31:48 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m33NVl7S007827 for ; Thu, 3 Apr 2008 16:31:47 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m33NVh7R009806; Fri, 4 Apr 2008 07:31:46 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JYR00K0FW0WHF00@nwk-avmta-2.sfbay.sun.com>; Thu, 03 Apr 2008 16:31:44 -0700 (PDT) Received: from dm-eng-02.sfbay.sun.com ([129.146.11.32]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JYR00EQTW0WOS80@nwk-avmta-2.sfbay.sun.com>; Thu, 03 Apr 2008 16:31:44 -0700 (PDT) Received: from marduk.eng.sun.com (marduk.SFBay.Sun.COM [129.146.108.224]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m33NVimG015610; Thu, 03 Apr 2008 16:31:44 -0700 (PDT) Received: from marduk.eng.sun.com (localhost [127.0.0.1]) by marduk.eng.sun.com (8.13.6+Sun/8.12.11) with ESMTP id m340WKuf026076; Thu, 03 Apr 2008 16:32:20 -0800 (PST) Received: (from gww@localhost) by marduk.eng.sun.com (8.13.6+Sun/8.12.11/Submit) id m340WKpU026075; Thu, 03 Apr 2008 16:32:20 -0800 (PST) Date: Thu, 03 Apr 2008 16:32:20 -0800 (PST) From: Gary Winiger Subject: 2008/226 mmapfd and 2008/195 validated execution To: psarc-ext@sun.com Cc: valex-core@sun.com Message-id: <200804040032.m340WKpU026075@marduk.eng.sun.com> Content-transfer-encoding: 7BIT X-Sun-Charset: US-ASCII X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 439 When I extended the timer yesterday it was for two reasons. I should have mentioned them. First was the interface name that had not converged. The second was that there seemed to be some possible interactions between mmapfd and validated execution and I wanted to ask both project teams to take a quick look at the other project to see if some alignment might be needed? Please consider this a request to both projects. Thankx, Gary.. From Michael.Corcoran@sun.com Thu Apr 3 17:29:59 2008 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m340TwfN008837 for ; Thu, 3 Apr 2008 17:29:59 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id m340TtRL000726; Fri, 4 Apr 2008 08:29:57 +0800 (SGT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JYR00K07YPV2200@brm-avmta-1.central.sun.com>; Thu, 03 Apr 2008 18:29:55 -0600 (MDT) Received: from brmea-mail-1.sun.com ([192.18.98.31]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JYR00LI2YPUNQE0@brm-avmta-1.central.sun.com>; Thu, 03 Apr 2008 18:29:54 -0600 (MDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m340Tspe029774; Fri, 04 Apr 2008 00:29:54 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JYR00301YGZAM00@mail-amer.sun.com> (original mail from Michael.Corcoran@Sun.COM); Thu, 03 Apr 2008 18:29:54 -0600 (MDT) Received: from musetta ([129.153.88.82]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JYR00J9SYPU8NC0@mail-amer.sun.com>; Thu, 03 Apr 2008 18:29:54 -0600 (MDT) Date: Thu, 03 Apr 2008 17:29:50 -0700 From: Michael Corcoran Subject: Re: 2008/226 mmapfd and 2008/195 validated execution In-reply-to: <200804040032.m340WKpU026075@marduk.eng.sun.com> Sender: Michael.Corcoran@sun.com To: Gary Winiger Cc: Michael.Corcoran@sun.com, PSARC-ext@sun.com, valex-core@sun.com Message-id: <1207268990.6990.67.camel@musetta> MIME-version: 1.0 X-Mailer: Evolution 2.6.2 Content-type: text/plain Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200804040032.m340WKpU026075@marduk.eng.sun.com> Status: RO Content-Length: 1161 Thanks Gary. Reading through the 1.1 Design doc for validated execution, it's clear that there is a bit of interaction between these two projects. At the very least mmapfd(2) or more likely mmapobj(2) will need to be added to Section 2 along with the following block diagram. I don't see anything in mmapfd that would conflict with valex but valex would have to add some logic to mmapfd to call signedexec_validate(). Since other binaries, such as Java will not be mapped via mmapfd, the mmap modifications for PROT_EXEC still seem necessary. I was hoping that they could be eliminated, but that does not appear to be the case. --Mike On Thu, 2008-04-03 at 16:32 -0800, Gary Winiger wrote: > When I extended the timer yesterday it was for two reasons. I should have > mentioned them. First was the interface name that had not converged. > The second was that there seemed to be some possible interactions between > mmapfd and validated execution and I wanted to ask both project teams to > take a quick look at the other project to see if some alignment might > be needed? > > Please consider this a request to both projects. > > Thankx, > Gary.. > From johnz@domus.sfbay.sun.com Thu Apr 3 17:38:56 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m340ctQ0008883 for ; Thu, 3 Apr 2008 17:38:55 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m340clS9016966; Fri, 4 Apr 2008 01:38:54 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JYR00001Z4S5200@nwk-avmta-2.sfbay.sun.com>; Thu, 03 Apr 2008 17:38:52 -0700 (PDT) Received: from dm-sfbay-01.sfbay.sun.com ([129.145.155.118]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JYR00ECPZ4SOUC0@nwk-avmta-2.sfbay.sun.com>; Thu, 03 Apr 2008 17:38:52 -0700 (PDT) Received: from domus.sfbay.sun.com (domus.SFBay.Sun.COM [10.6.64.11]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m340cq77032919; Thu, 03 Apr 2008 17:38:52 -0700 (PDT) Received: (from johnz@localhost) by domus.sfbay.sun.com (Trusted Solaris (8.11.7)/8.11.6) id m340cps12268; Thu, 03 Apr 2008 17:38:51 -0700 (PDT) Date: Thu, 03 Apr 2008 17:38:51 -0700 (PDT) From: John Zolnowsky x69422/408-404-5064 Subject: Re: 2008/226 mmapfd and 2008/195 validated execution To: psarc-ext@sun.com, gww@eng.sun.com Cc: valex-core@sun.com Message-id: <200804040038.m340cps12268@domus.sfbay.sun.com> Content-transfer-encoding: 7BIT X-Sun-Charset: US-ASCII X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 1417 > Date: Thu, 03 Apr 2008 16:32:20 -0800 (PST) > From: Gary Winiger ... > The second was that there seemed to be some possible interactions between > mmapfd and validated execution and I wanted to ask both project teams to > take a quick look at the other project to see if some alignment might > be needed? > > Please consider this a request to both projects. The execution-time validation of libraries being introduced into process images is expected to be triggered by rtld's idiomatic use of mmap(..., PROT_EXEC, ...). Since mmapfd() would subsume multiple mmap() calls into a single call producing essentially the same effects on the process image, it would appear that mmapfd() would provide the same opportunity for triggering the validation of the mapped object. The general nature of mmapfd() mapping represents a possible solution to a concern being discussed in 2008/195. The issue is that interpreters other than rtld often have the equivalent of libraries, for example, perl's .pm and .pl or the shell "source" or "." commands. These extended forms of library are presently introduced into the process "execution" using general interfaces (open(), read()), precluding any reliable triggering for validition of the object. As much as mmapfd() provides a generalized mechanism for accessing these forms of libraries, it would serve as a enabler for validated execution. -JZ From bart.smaalders@sun.com Thu Apr 3 20:45:56 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m343juCR012969 for ; Thu, 3 Apr 2008 20:45:56 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m343jsEo017752; Thu, 3 Apr 2008 20:45:56 -0700 (PDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JYS00C0J7SJR000@brm-avmta-1.central.sun.com>; Thu, 03 Apr 2008 21:45:55 -0600 (MDT) Received: from zion.sfbay.sun.com ([129.146.17.75]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JYS002IA7SIVI60@brm-avmta-1.central.sun.com>; Thu, 03 Apr 2008 21:45:54 -0600 (MDT) Received: from [129.146.228.109] (cyber.SFBay.Sun.COM [129.146.228.109]) by zion.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m343jroF029767; Fri, 04 Apr 2008 03:45:53 +0000 (GMT) Date: Thu, 03 Apr 2008 20:45:53 -0700 From: Bart Smaalders Subject: Re: 2008/226 mmapfd and 2008/195 validated execution In-reply-to: <200804040038.m340cps12268@domus.sfbay.sun.com> To: John Zolnowsky x69422/408-404-5064 Cc: PSARC-ext@sun.com, gww@eng.sun.com, valex-core@sun.com Message-id: <47F5A471.10408@Sun.COM> Organization: Sun Microsystems MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200804040038.m340cps12268@domus.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.9 (X11/20080201) Status: RO Content-Length: 1031 John Zolnowsky x69422/408-404-5064 wrote: > The general nature of mmapfd() mapping represents a possible solution > to a concern being discussed in 2008/195. The issue is that > interpreters other than rtld often have the equivalent of libraries, > for example, perl's .pm and .pl or the shell "source" or "." commands. > These extended forms of library are presently introduced into the > process "execution" using general interfaces (open(), read()), > precluding any reliable triggering for validition of the object. As > much as mmapfd() provides a generalized mechanism for accessing these > forms of libraries, it would serve as a enabler for validated > execution. cat /etc/file | sh ??? It seems to be that validated execution is somewhat missing the point by focusing on "execution". I'll take my comments to 2008/195 when I get a chance.. - Bart -- Bart Smaalders Solaris Kernel Performance barts@cyber.eng.sun.com http://blogs.sun.com/barts "You will contribute more with mercurial than with thunderbird." From Nicolas.Williams@sun.com Fri Apr 4 08:49:41 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m34FneNF028503 for ; Fri, 4 Apr 2008 08:49:40 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m34FnbaV029836; Fri, 4 Apr 2008 16:49:37 +0100 (BST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JYT00C995AOR400@nwk-avmta-1.sfbay.Sun.COM>; Fri, 04 Apr 2008 08:49:36 -0700 (PDT) Received: from binky.Central.Sun.COM ([129.153.128.104]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JYT00M4S5AM2K80@nwk-avmta-1.sfbay.Sun.COM>; Fri, 04 Apr 2008 08:49:34 -0700 (PDT) Received: from binky.Central.Sun.COM (localhost [127.0.0.1]) by binky.Central.Sun.COM (8.14.1+Sun/8.14.1) with ESMTP id m34FnXNE003785; Fri, 04 Apr 2008 10:49:33 -0500 (CDT) Received: (from nw141292@localhost) by binky.Central.Sun.COM (8.14.1+Sun/8.14.1/Submit) id m34FnXs8003784; Fri, 04 Apr 2008 10:49:33 -0500 (CDT) Date: Fri, 04 Apr 2008 10:49:33 -0500 From: Nicolas Williams Subject: Re: 2008/226 mmapfd and 2008/195 validated execution In-reply-to: <1207268990.6990.67.camel@musetta> To: Michael Corcoran Cc: Gary Winiger , PSARC-ext@sun.com, valex-core@sun.com Reply-to: valex-discuss@opensolaris.org Mail-followup-to: Michael Corcoran , Gary Winiger , PSARC-ext@sun.com, valex-core@sun.com Message-id: <20080404154933.GR16998@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200804040032.m340WKpU026075@marduk.eng.sun.com> <1207268990.6990.67.camel@musetta> X-Authentication-warning: binky.Central.Sun.COM: nw141292 set sender to Nicolas.Williams@sun.com using -f User-Agent: Mutt/1.5.7i Status: RO Content-Length: 1375 On Thu, Apr 03, 2008 at 05:29:50PM -0700, Michael Corcoran wrote: > Thanks Gary. > > Reading through the 1.1 Design doc for validated execution, it's clear > that there is a bit of interaction between these two projects. At the > very least mmapfd(2) or more likely mmapobj(2) will need to be added to > Section 2 along with the following block diagram. > > I don't see anything in mmapfd that would conflict with valex but valex > would have to add some logic to mmapfd to call signedexec_validate(). > Since other binaries, such as Java will not be mapped via mmapfd, the > mmap modifications for PROT_EXEC still seem necessary. I was hoping > that they could be eliminated, but that does not appear to be the case. Slightly OT (reply-to set -- please honor it): Please see the discussion on the valex-discuss list this week. IMO mmap() and mmapobj() shouldn't be validating file signatures because either the caller should have done so already when it called open(). That the valex project proposes to do so for some calls to mmap() is indicative of a problem: finding all user-land code that should have config files, interpretable code, etc... verified, and modifying it to do so, does not scale. Darren Moffat and I propose a file attribute to deal with this that would, incidentally, make the need for mmap() and mmapobj() to validate files go away. Nico -- From Nicolas.Williams@sun.com Fri Apr 4 08:53:28 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m34FrSPw028758 for ; Fri, 4 Apr 2008 08:53:28 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m34FrLoe004873; Fri, 4 Apr 2008 08:53:27 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JYT00D355H21Q00@nwk-avmta-1.sfbay.Sun.COM>; Fri, 04 Apr 2008 08:53:26 -0700 (PDT) Received: from binky.Central.Sun.COM ([129.153.128.104]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JYT00MCA5GZ2C90@nwk-avmta-1.sfbay.Sun.COM>; Fri, 04 Apr 2008 08:53:24 -0700 (PDT) Received: from binky.Central.Sun.COM (localhost [127.0.0.1]) by binky.Central.Sun.COM (8.14.1+Sun/8.14.1) with ESMTP id m34FrNF7003794; Fri, 04 Apr 2008 10:53:23 -0500 (CDT) Received: (from nw141292@localhost) by binky.Central.Sun.COM (8.14.1+Sun/8.14.1/Submit) id m34FrNoe003793; Fri, 04 Apr 2008 10:53:23 -0500 (CDT) Date: Fri, 04 Apr 2008 10:53:23 -0500 From: Nicolas Williams Subject: Re: 2008/226 mmapfd and 2008/195 validated execution In-reply-to: <47F5A471.10408@Sun.COM> To: Bart Smaalders Cc: John Zolnowsky x69422/408-404-5064 , PSARC-ext@sun.com, gww@eng.sun.com, valex-core@sun.com Mail-followup-to: Bart Smaalders , John Zolnowsky x69422/408-404-5064 , PSARC-ext@sun.com, gww@eng.sun.com, valex-core@sun.com Message-id: <20080404155323.GS16998@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Content-disposition: inline X-PMX-Version: 5.4.1.325704 References: <200804040038.m340cps12268@domus.sfbay.sun.com> <47F5A471.10408@Sun.COM> X-Authentication-warning: binky.Central.Sun.COM: nw141292 set sender to Nicolas.Williams@sun.com using -f User-Agent: Mutt/1.5.7i Status: RO Content-Length: 1341 On Thu, Apr 03, 2008 at 08:45:53PM -0700, Bart Smaalders wrote: > John Zolnowsky x69422/408-404-5064 wrote: > > >The general nature of mmapfd() mapping represents a possible solution > >to a concern being discussed in 2008/195. The issue is that > >interpreters other than rtld often have the equivalent of libraries, > >for example, perl's .pm and .pl or the shell "source" or "." commands. > >These extended forms of library are presently introduced into the > >process "execution" using general interfaces (open(), read()), > >precluding any reliable triggering for validition of the object. As > >much as mmapfd() provides a generalized mechanism for accessing these > >forms of libraries, it would serve as a enabler for validated > >execution. > > cat /etc/file | sh ??? > > It seems to be that validated execution is somewhat missing the point > by focusing on "execution". Darren M. noticed this a while back and commented on this on the valex-discuss list, with no resolution. This week there's a more active discussion of the issue on that list. > I'll take my comments to 2008/195 when I get a chance.. Perhaps it's time to take the valex-discuss thread onto the 2008/195 case. With 2008/195 a full case I thought it'd be better to discuss the matter in the project list and then bring issues to the meeting. Nico -- From gww@sac.sfbay.sun.com Fri Apr 4 13:22:44 2008 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m34KMhN1008013 for ; Fri, 4 Apr 2008 13:22:44 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m34KMePf014042; Fri, 4 Apr 2008 21:22:42 +0100 (BST) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JYT00I09HXSF000@brm-avmta-1.central.sun.com>; Fri, 04 Apr 2008 14:22:40 -0600 (MDT) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JYT00AXLHXRVV40@brm-avmta-1.central.sun.com>; Fri, 04 Apr 2008 14:22:40 -0600 (MDT) Received: from sac.sfbay.sun.com (new-sac.SFBay.Sun.COM [129.146.175.65]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m34KMcck032234; Fri, 04 Apr 2008 13:22:38 -0700 (PDT) Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m34KMcDC008010; Fri, 04 Apr 2008 13:22:38 -0700 (PDT) Received: (from gww@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id m34KMcGk008009; Fri, 04 Apr 2008 13:22:38 -0700 (PDT) Date: Fri, 04 Apr 2008 13:22:38 -0700 (PDT) From: Gary Winiger Subject: Re: 2008/226 mmapfd and 2008/195 validated execution To: Nicolas.Williams@sun.com, bart.smaalders@sun.com Cc: John.Zolnowsky@sun.com, PSARC-ext@sun.com, gww@eng.sun.com, valex-core@sun.com Message-id: <200804042022.m34KMcGk008009@sac.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 1633 > On Thu, Apr 03, 2008 at 08:45:53PM -0700, Bart Smaalders wrote: > > John Zolnowsky x69422/408-404-5064 wrote: > > > > >The general nature of mmapfd() mapping represents a possible solution > > >to a concern being discussed in 2008/195. The issue is that > > >interpreters other than rtld often have the equivalent of libraries, > > >for example, perl's .pm and .pl or the shell "source" or "." commands. > > >These extended forms of library are presently introduced into the > > >process "execution" using general interfaces (open(), read()), > > >precluding any reliable triggering for validition of the object. As > > >much as mmapfd() provides a generalized mechanism for accessing these > > >forms of libraries, it would serve as a enabler for validated > > >execution. The point of my asking for more time on the mmapfd case was to ensure both teams say the other case. IMO, that has been satisfied and both projects can proceed. > > cat /etc/file | sh ??? > > > > It seems to be that validated execution is somewhat missing the point > > by focusing on "execution". > > Darren M. noticed this a while back and commented on this on the > valex-discuss list, with no resolution. This week there's a more active > discussion of the issue on that list. For this further point, IMO this is gw-0 Case boundaries relative to things like "cat foo | sh" and is proper to discuss as part of the full review. If you have architectural points, please bring them to the review. Gary.. P.S. I'm happy to proxy issue to the issues file for those who may not have modify access rights if they are mailed privately to me. From roland.mainz@nrubsig.org Sat Apr 5 18:59:09 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m361x9Lo026720 for ; Sat, 5 Apr 2008 18:59:09 -0700 (PDT) Received: from sca-ea-mail-1.sun.com (sca-ea-mail-1.Sun.COM [192.18.43.24]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m361x971018392 for ; Sat, 5 Apr 2008 18:59:09 -0700 (PDT) Received: from relay24.sun.com (ip192-12-251-74.block6.us.syntegra.com [192.12.251.74]) by sca-ea-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m361x8V1007414 for ; Sun, 6 Apr 2008 01:59:08 GMT Received: from mms23es.sun.com ([150.143.232.54] [150.143.232.54]) by relay24i.sun.com with ESMTP id BT-MMP-2069700 for psarc-ext@sac.sfbay.sun.com; Sun, 6 Apr 2008 01:59:08 Z Received: from relay21.sun.com (relay21.sun.com [192.12.251.24]) by mms23es.sun.com with ESMTP id BT-MMP-2546342 for psarc-ext@sac.sfbay.sun.com; Sun, 6 Apr 2008 01:59:08 Z Received: from mail-in-14.arcor-online.net ([151.189.21.54] [151.189.21.54]) by relay21i.sun.com with ESMTP id BT-MMP-4244662 for psarc-ext@sac.sfbay.sun.com; Sun, 6 Apr 2008 01:59:07 Z Received: from mail-in-14-z2.arcor-online.net (mail-in-14-z2.arcor-online.net [151.189.8.31]) by mail-in-14.arcor-online.net (Postfix) with ESMTP id 48316187D40; Sun, 6 Apr 2008 03:59:06 +0200 (CEST) Received: from mail-in-01.arcor-online.net (mail-in-01.arcor-online.net [151.189.21.41]) by mail-in-14-z2.arcor-online.net (Postfix) with ESMTP id 36381100B9; Sun, 6 Apr 2008 03:59:06 +0200 (CEST) Received: from jupiterb48.nrubsig.org (dslb-084-059-001-044.pools.arcor-ip.net [84.59.1.44]) by mail-in-01.arcor-online.net (Postfix) with ESMTP id AC26E104BFB; Sun, 6 Apr 2008 03:59:05 +0200 (CEST) Received: from nrubsig.org (localhost [127.0.0.1]) by jupiterb48.nrubsig.org (8.13.8+Sun/8.13.8) with ESMTP id m361x2Ww001220; Sun, 6 Apr 2008 03:59:03 +0200 (CEST) Sender: gisburn@jupiterb48.nrubsig.org Message-ID: <47F82E66.99E057E@nrubsig.org> Date: Sun, 06 Apr 2008 03:59:02 +0200 From: Roland Mainz X-Brightmail-Tracker: AAAAAA== X-Mailer: Mozilla 4.76 [en] (X11; U; SunOS 5.11 sun4u) X-Accept-Language: en To: Michael Corcoran CC: Rod Evans , psarc-ext@sac.sfbay.sun.com Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor References: <47F0750D.4040809@sun.com> <47F0CD06.25D53234@nrubsig.org> <1206992622.1298.222.camel@musetta> X-Virus-Scanned: ClamAV 0.92.1/6628/Sun Apr 6 03:14:00 2008 on mail-in-01.arcor-online.net X-Virus-Status: Clean X-Antispam: No, score=-2.6/5.0, scanned in 0.134sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Status: RO Content-Length: 6925 Michael Corcoran wrote: > On Mon, 2008-03-31 at 13:37 +0200, Roland Mainz wrote: > > Rod Evans wrote: > > > I'm sponsoring the following case for Mike Corcoran. Time out 04/07/08. > > > > > > The case introduces a new system call, mmapfd(2). This call is primarily > > > targeted for use by ld.so.1(1), and provides for the efficient mapping of > > > ELF files (and 4.x AOUT files). > > > > > > Release Binding: Patch/Micro > > > mmapfd(2): Consolidation Private > > > > > > -------------------------------------------------------------------------- > > > > > > 1. Introduction > > > 1.1. Project/Component Working Name: > > > mmapfd: mmap file descriptor > > [snip] > > > 4. Technical Description: > > > 4.1. Details: > > > mmapfd is a new system call which can interpret and map ELF and AOUT > > > (4.x) objects. This system call allows the interpretation and mapping > > > of ELF and AOUT files to be carried out completely by the kernel rather > > > than by ld.so.1. > > > > Erm... the call seems to be ELF+AOUT-specifc - why does it have such a > > generic name ? > > It has a generic name for future expansion. Yeah... but |mmapfd()| sounds it's for generic usage and what's described here is IMO very specific and therefore a longer (and more specific) name may be nice... > > The call can't handle other types of executables (e.g. > > "javaexec" or "shbinexec") > > Not yet, but that doesn't mean that it won't. If there is a file format > that needs to be interpreted in order to be mapped, this new syscall is > a good way of going about doing that. What is the exact definition of "interpreted" in this case - and why is the kernel needed for this ? Just for sharing the interpreted data between processes, e.g. to avoid that multiple processes waste memory with multiple copies of the same (interpreted) data ? [snip] > > ... what renaming the call to |mmapexecfd()| (= "memory map of > > executable fd") ? > > I like Darren's suggestion of mmapv(2) better so far. Why limit us to > only being able to interpret executable files? What if there's some > other non-executable file type that would naturally use this interface? Mhhh... Ok... but |mmapv()| somehow sounds like a vector version of |mmap()| (e.g. something which accepts an array of |mmap()| options and handles them all in one single syscall) ... ... what about |mmapintp()| ("mmap interpreted") ? > > > mmapfd also provides for mapping a whole file, without interpretation > > > in a read only mode. > > > > What does that mean ? Can these data+MMU mappings be shared between > > processes ? > > It means that you can map a file read-only without doing any > interpretation of the file. Thus if you passed in an ELF file without > the MMFD_INTERPRET flag, the file would get mapped as a single read-only > segment. Ok... > The data+MMU mappings would not be shared between processes. Why ? > > [snip] > > > System Calls mmapfd(2) > > > > > > NAME > > > > > > mmapfd - map a file descriptor in the appropriate manner. > > > > > > SYNOPSIS > > > > > > #include > > > > > > int > > > mmapfd(int fd, uint_t flags, mmapfd_result_t *storage, > > > uint_t *elements, void *arg) > > > > Uhm... how do I unmap the mapping done by |mmapfd()| ? > > > Nico answered this in a subsequent post and you would use munmap(2) on > each element in the "storage" array to unmap that segment. I agree that > this should be explicitly pointed out somewhere since it's not clear. What about having a helper function to unmap the storage array in one step (IMO it would be nice and it would maintain the Unix "tradition" of symmetric function call pairs (e.g. |open()|+|close()|, |malloc()|+|free()| etc.) ? [snip] > > > TYPES USED > > > > > > typedef struct { > > > caddr_t mr_addr; /* mapping address */ > > > size_t mr_msize; /* mapping size */ > > > size_t mr_fsize; /* file size */ > > > size_t mr_offset; /* offset into file */ > > > int mr_prot; /* the protections provided */ > > > > Why is this |signed int| ? > > I agree it should be uint_t or even uchar_t. Both appear to be commonly > used throughout the kernel. uint_t is my preference here. Ok... > > > uint_t mr_flags; /* info on the mapping */ > > > > Please change this to |uint64_t| (e.g. it may be nice to have more flags > > available by default). > > > I go back and forth on this. uint64_t implies that 32 flags will not be > enough. 32 seems like a lot Yeah... but after some time the number of flags gets used-up... > but since this interface should be around > for a long time, maybe 64 would be better since it will take a long time > to use all 64 :) IMO 64 is better (and IMO each system/library call used to create a resource should take a "flags" option that the call can be changed/extended easily) ... > > > } mmapfd_result_t; > > > > > > Values for mr_flags include: > > > > > > MFD_ELF_HDR 0x1 /* the ELF header is mapped at mr_addr */ > > > MFD_AOUT_HDR 0x2 /* the AOUT header is mapped at mr_addr */ > > > > What about reserving the first four bits for |MFD_*_HDR| flags > > (|MFD_ELF_HDR|, |MFD_AOUT_HDR| and two bits reserved for future > > |MFD_*_HDR| flags) ? > > I can see the vanity reason for doing this, it's nice to clump things in > groups of 4 with everything in the group being related, but at the same > time, why the first 4 flags and not the first 8 or 16 or ... I think > densely packing the bits is easiest. > I'm willing to move MFD_PADDING > first in the list aka 0x1 so that future header types will follow > numerically from the current header types. I'll make this change since > it is aesthetically more pleasing. Ok... > > Finally... how can I _force_ the mapping to use something like 64k pages > > by default ? > > Using mpss.so.1 or ppgsz are ways to get the heap segment to be a > specific size. Other than that, there is no control for the page size > used by the other segments and the kernel will pick the best size for > the given platform. Uufortunately (at least tested with B72) the kernel seems to do a bad job in this case (e.g. the stack won't be mapped with 64k pages unless the size significantly exceeds 64k in size and code doens't get mapped wth 64k pages at all), even on platforms (like the T2000 we had on the Oct 2007 Summit) where I would expect extensive use of 64k pages... ;-( ... IMO a manual control would be nice... ---- Bye, Roland -- __ . . __ (o.\ \/ /.o) roland.mainz@nrubsig.org \__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer /O /==\ O\ TEL +49 641 7950090 (;O/ \/ \O;) From roland.mainz@nrubsig.org Sat Apr 5 19:02:50 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m3622oEg026864 for ; Sat, 5 Apr 2008 19:02:50 -0700 (PDT) Received: from brmea-mail-1.sun.com (brmea-mail-1.Sun.COM [192.18.98.31]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m3622oCp020178 for ; Sat, 5 Apr 2008 19:02:50 -0700 (PDT) Received: from relay43i.sun.com ([192.5.209.74]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m362245V014778 for ; Sun, 6 Apr 2008 02:02:49 GMT Received: from mms48es.sun.com ([160.41.221.231] [160.41.221.231]) by relay43i.sun.com with ESMTP id BT-MMP-114924 for psarc-ext@sac.sfbay.sun.com; Sun, 6 Apr 2008 02:02:49 Z Received: from relay43i.sun.com (relay43i.sun.com [192.5.209.74]) by mms48es.sun.com with ESMTP id BT-MMP-391185 for psarc-ext@sac.sfbay.sun.com; Sun, 6 Apr 2008 02:02:49 Z Received: from mail-in-14.arcor-online.net ([151.189.21.54] [151.189.21.54]) by relay4i.sun.com with ESMTP id BT-MMP-5648724 for psarc-ext@sac.sfbay.sun.com; Sun, 6 Apr 2008 02:02:48 Z Received: from mail-in-06-z2.arcor-online.net (mail-in-06-z2.arcor-online.net [151.189.8.18]) by mail-in-14.arcor-online.net (Postfix) with ESMTP id 60579187CD8; Sun, 6 Apr 2008 04:02:48 +0200 (CEST) Received: from mail-in-13.arcor-online.net (mail-in-13.arcor-online.net [151.189.21.53]) by mail-in-06-z2.arcor-online.net (Postfix) with ESMTP id 51BEE5BD69; Sun, 6 Apr 2008 04:02:48 +0200 (CEST) Received: from jupiterb48.nrubsig.org (dslb-084-059-001-044.pools.arcor-ip.net [84.59.1.44]) by mail-in-13.arcor-online.net (Postfix) with ESMTP id 2E9AF25D2E1; Sun, 6 Apr 2008 04:02:48 +0200 (CEST) Received: from nrubsig.org (localhost [127.0.0.1]) by jupiterb48.nrubsig.org (8.13.8+Sun/8.13.8) with ESMTP id m3622ktx001223; Sun, 6 Apr 2008 04:02:47 +0200 (CEST) Sender: gisburn@jupiterb48.nrubsig.org Message-ID: <47F82F46.BDE98FF8@nrubsig.org> Date: Sun, 06 Apr 2008 04:02:46 +0200 From: Roland Mainz X-Brightmail-Tracker: AAAAAA== X-Mailer: Mozilla 4.76 [en] (X11; U; SunOS 5.11 sun4u) X-Accept-Language: en To: Blake Jones CC: Michael Corcoran , Rod Evans , psarc-ext@sac.sfbay.sun.com Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor References: <16222.1206995152@cagey> X-Virus-Scanned: ClamAV 0.92.1/6628/Sun Apr 6 03:14:00 2008 on mail-in-13.arcor-online.net X-Virus-Status: Clean X-Antispam: No, score=-2.6/5.0, scanned in 0.114sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Status: RO Content-Length: 1349 Blake Jones wrote: [snip] > To elaborate on this: part of the idea behind mmapfd() is that it gives > the kernel the flexibility to make global-scope decisions about where > and how a file should be mapped. This includes the appropriate page > size to use (which might vary depending on what platform you're running > on, or even potentially which CPU on a heterogeneous machine), what > virtual address to start the mapping at, how much padding to leave > between mappings, etc. Applications shouldn't have to know how the MMU > is designed in order to perform well, and we're trying to discourage > them from doing so. The auto-MPSS project already makes some decisions > about what page size to use; mmapfd() is just another step in that > general direction. The problem is that (at least in the B72 installation on the Oct 2007 Summit's T2000 machine) the auto-MPSS stuff performs (IMO) poorly or does nothing - for example the only way to get the stack mapped with 64k pages was to use mpss.so.1 or memcntl(2) and the code didn't seem to use 64k pages at all (and libraries like libast are larger than 64k and ciuld benefit from something like that...). ---- Bye, Roland -- __ . . __ (o.\ \/ /.o) roland.mainz@nrubsig.org \__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer /O /==\ O\ TEL +49 641 7950090 (;O/ \/ \O;) From roland.mainz@nrubsig.org Sat Apr 5 19:11:13 2008 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m362BDYV027140 for ; Sat, 5 Apr 2008 19:11:13 -0700 (PDT) Received: from brmea-mail-2.sun.com (brmea-mail-2.Sun.COM [192.18.98.43]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m362BCGc055786 for ; Sat, 5 Apr 2008 19:11:13 -0700 (PDT) Received: from relay14i.sun.com (ip124.net129179-4.block1.us.syntegra.com [129.179.4.124]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m362BCg2014718 for ; Sun, 6 Apr 2008 02:11:12 GMT Received: from mmp13es.sun.com ([160.41.209.23] [160.41.209.23]) by relay14i.sun.com with ESMTP id BT-MMP-429565 for psarc-ext@sac.sfbay.sun.com; Sun, 6 Apr 2008 02:11:12 Z Received: from relay17i.sun.com (relay17i.sun.com [129.179.4.127]) by mmp13es.sun.com with ESMTP id BT-MMP-170463 for psarc-ext@sac.sfbay.sun.com; Sun, 6 Apr 2008 02:11:11 Z Received: from mail-in-05.arcor-online.net ([151.189.21.45] [151.189.21.45]) by relay1i.sun.com with ESMTP id BT-MMP-5246578 for psarc-ext@sac.sfbay.sun.com; Sun, 6 Apr 2008 02:11:11 Z Received: from mail-in-20-z2.arcor-online.net (mail-in-20-z2.arcor-online.net [151.189.8.85]) by mail-in-05.arcor-online.net (Postfix) with ESMTP id 7DB11183A41; Sun, 6 Apr 2008 04:11:10 +0200 (CEST) Received: from mail-in-01.arcor-online.net (mail-in-01.arcor-online.net [151.189.21.41]) by mail-in-20-z2.arcor-online.net (Postfix) with ESMTP id 71944107897; Sun, 6 Apr 2008 04:11:10 +0200 (CEST) Received: from jupiterb48.nrubsig.org (dslb-084-059-001-044.pools.arcor-ip.net [84.59.1.44]) by mail-in-01.arcor-online.net (Postfix) with ESMTP id 4C292104BDF; Sun, 6 Apr 2008 04:11:10 +0200 (CEST) Received: from nrubsig.org (localhost [127.0.0.1]) by jupiterb48.nrubsig.org (8.13.8+Sun/8.13.8) with ESMTP id m362B979001228; Sun, 6 Apr 2008 04:11:09 +0200 (CEST) Sender: gisburn@jupiterb48.nrubsig.org Message-ID: <47F8313D.6A8EAF3E@nrubsig.org> Date: Sun, 06 Apr 2008 04:11:09 +0200 From: Roland Mainz X-Brightmail-Tracker: AAAAAA== X-Mailer: Mozilla 4.76 [en] (X11; U; SunOS 5.11 sun4u) X-Accept-Language: en To: Edward Pilatowicz CC: Michael Corcoran , Rod Evans , psarc-ext@sac.sfbay.sun.com Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor References: <47F0750D.4040809@sun.com> <47F0AF19.7050609@Sun.COM> <1206989194.1298.186.camel@musetta> <20080331185211.GE161409@eng.sun.com> X-Virus-Scanned: ClamAV 0.92.1/6628/Sun Apr 6 03:14:00 2008 on mail-in-01.arcor-online.net X-Virus-Status: Clean X-Antispam: No, score=-2.6/5.0, scanned in 0.067sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Status: RO Content-Length: 1416 Edward Pilatowicz wrote: > On Mon, Mar 31, 2008 at 11:46:34AM -0700, Michael Corcoran wrote: > > On Mon, 2008-03-31 at 10:30 +0100, Darren J Moffat wrote: > > > I find the function name a bit strange. So strange in fact it made me > > > look at the mmap(2) man page to check that it wasn't taking a char* > > > rather than an int for the file since I was sure it took and fd. Since > > > mmap(2) already takes an fd. > > > > > > The mmap(2) versus mmapfd(2) makes me think of fopen(3C) versus fdopen(3C). > > > > > > As for a better name it looks to me more like mmapv(2), eg like read(2) > > > versus readv(2). > > > > A good name for this syscall has been elusive :) George has always > > wanted to call it "mapme(2)" but that seemed a bit odd. I agree that > > mmapfd(2) doesn't seem like it differentiates itself from mmap(2) > > enough. mmapv(2) does seem to follow readv(2) more closely so maybe > > that will be a better name. I'm open for other suggestions as well. > > > > given that this system call interprets and maps different kinds of > objects (and the psarc case says it could be extended to understand > other object types as well) how about mmapobj(). What about |mmapintp()| - "memory-map interpreted" ? ---- Bye, Roland -- __ . . __ (o.\ \/ /.o) roland.mainz@nrubsig.org \__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer /O /==\ O\ TEL +49 641 7950090 (;O/ \/ \O;) From roland.mainz@nrubsig.org Sat Apr 5 19:24:19 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m362OII0028393 for ; Sat, 5 Apr 2008 19:24:18 -0700 (PDT) Received: from sca-ea-mail-2.sun.com (sca-ea-mail-2.Sun.COM [192.18.43.25]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m362OIgY026099 for ; Sat, 5 Apr 2008 19:24:18 -0700 (PDT) Received: from relay16i.sun.com (ip126.net129179-4.block1.us.syntegra.com [129.179.4.126]) by sca-ea-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m362OIFP013845 for ; Sun, 6 Apr 2008 02:24:18 GMT Received: from mmp14es.sun.com ([160.41.209.24] [160.41.209.24]) by relay16i.sun.com with ESMTP id BT-MMP-449346 for psarc-ext@sac.sfbay.sun.com; Sun, 6 Apr 2008 02:24:18 Z Received: from relay11i.sun.com (relay11i.sun.com [129.179.4.121]) by mmp14es.sun.com with ESMTP id BT-MMP-174858 for psarc-ext@sac.sfbay.sun.com; Sun, 6 Apr 2008 02:24:17 Z Received: from mail-in-12.arcor-online.net ([151.189.21.52] [151.189.21.52]) by relay1i.sun.com with ESMTP id BT-MMP-5001095 for psarc-ext@sac.sfbay.sun.com; Sun, 6 Apr 2008 02:24:17 Z Received: from mail-in-14-z2.arcor-online.net (mail-in-14-z2.arcor-online.net [151.189.8.31]) by mail-in-12.arcor-online.net (Postfix) with ESMTP id 8F1B84C75B; Sun, 6 Apr 2008 04:24:16 +0200 (CEST) Received: from mail-in-02.arcor-online.net (mail-in-02.arcor-online.net [151.189.21.42]) by mail-in-14-z2.arcor-online.net (Postfix) with ESMTP id 7C52A100BE; Sun, 6 Apr 2008 04:24:16 +0200 (CEST) Received: from jupiterb48.nrubsig.org (dslb-084-059-001-044.pools.arcor-ip.net [84.59.1.44]) by mail-in-02.arcor-online.net (Postfix) with ESMTP id 5822036E862; Sun, 6 Apr 2008 04:24:16 +0200 (CEST) Received: from nrubsig.org (localhost [127.0.0.1]) by jupiterb48.nrubsig.org (8.13.8+Sun/8.13.8) with ESMTP id m362OElZ001245; Sun, 6 Apr 2008 04:24:14 +0200 (CEST) Sender: gisburn@jupiterb48.nrubsig.org Message-ID: <47F8344E.369E6C@nrubsig.org> Date: Sun, 06 Apr 2008 04:24:14 +0200 From: Roland Mainz X-Brightmail-Tracker: AAAAAA== X-Mailer: Mozilla 4.76 [en] (X11; U; SunOS 5.11 sun4u) X-Accept-Language: en To: Nicolas Williams CC: Rod Evans , psarc-ext@sac.sfbay.sun.com, Michael Corcoran Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor References: <47F0750D.4040809@sun.com> <47F0CD06.25D53234@nrubsig.org> <20080331153020.GG16998@Sun.COM> X-Virus-Scanned: ClamAV 0.92.1/6628/Sun Apr 6 03:14:00 2008 on mail-in-02.arcor-online.net X-Virus-Status: Clean X-Antispam: No, score=-2.6/5.0, scanned in 0.072sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Status: RO Content-Length: 2186 Nicolas Williams wrote: > On Mon, Mar 31, 2008 at 01:37:42PM +0200, Roland Mainz wrote: > > Rod Evans wrote: > > > mmapfd(int fd, uint_t flags, mmapfd_result_t *storage, > > > uint_t *elements, void *arg) > > > > Uhm... how do I unmap the mapping done by |mmapfd()| ? > > I thought that was clear (each mmapfd_result_t's mr_addr can be used as > an argument to munmap(), mprotect(), ...). Erm... I appears I missed this when I was looking at 3AM on the manual page... sorry... ;-( > > > uint_t mr_flags; /* info on the mapping */ > > > > Please change this to |uint64_t| (e.g. it may be nice to have more flags > > available by default). > > Same thing with the flags argument, no? Erm... depends... I wasn't sure what may happen when this happens on a 32bit ABI... > > > } mmapfd_result_t; > > > > > > Values for mr_flags include: > > > > > > MFD_ELF_HDR 0x1 /* the ELF header is mapped at mr_addr */ > > > MFD_AOUT_HDR 0x2 /* the AOUT header is mapped at mr_addr */ > > > > What about reserving the first four bits for |MFD_*_HDR| flags > > (|MFD_ELF_HDR|, |MFD_AOUT_HDR| and two bits reserved for future > > |MFD_*_HDR| flags) ? > > Why? Isn't the ARC going to be the authority for this namespace > anyways? Right (originally I was thinking (at 3am, sleep-deprivated and without any coffee) about aesthetic (anyone remember the X11 toolkits and their wild mix of flag lists which grew over time ?) and other platforms which may include this system call in the future but this was another half-backed thought since "binary interoperabilty" is not needed in this case and ARC is the authority... :-) ) ... > My interpretation is that all currently unused flag bits are > reserved. Or did you mean that file format shouldn't be a flag but an > integer value? My idea was to pack similar flags together, mainly for aesthetic reasons (and maybe easier debuging since all *_HDR flags would sit in one nibble/byte) ... ---- Bye, Roland -- __ . . __ (o.\ \/ /.o) roland.mainz@nrubsig.org \__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer /O /==\ O\ TEL +49 641 7950090 (;O/ \/ \O;) From roland.mainz@nrubsig.org Sat Apr 5 19:27:47 2008 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m362RlJw028892 for ; Sat, 5 Apr 2008 19:27:47 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id m362RkON000708; Sat, 5 Apr 2008 19:27:46 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JYV00L05TIA2Z00@nwk-avmta-2.sfbay.sun.com>; Sat, 05 Apr 2008 19:27:46 -0700 (PDT) Received: from sca-ea-mail-2.sun.com ([192.18.43.25]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JYV00M05TI9RCF0@nwk-avmta-2.sfbay.sun.com>; Sat, 05 Apr 2008 19:27:45 -0700 (PDT) Received: from relay18i.sun.com (ip128.net129179-4.block1.us.syntegra.com [129.179.4.128]) by sca-ea-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m362RjSL014399; Sun, 06 Apr 2008 02:27:45 +0000 (GMT) Received: from mmp14es.sun.com ([160.41.209.24] [160.41.209.24]) by relay18i.sun.com with ESMTP id BT-MMP-466915; Sun, 06 Apr 2008 02:27:44 +0000 (Z) Received: from relay18i.sun.com (relay18i.sun.com [129.179.4.128]) by mmp14es.sun.com with ESMTP id BT-MMP-176443; Sun, 06 Apr 2008 02:27:44 +0000 (Z) Received: from mail-in-03.arcor-online.net ([151.189.21.43] [151.189.21.43]) by relay1i.sun.com with ESMTP id BT-MMP-4692171; Sun, 06 Apr 2008 02:27:44 +0000 (Z) Received: from mail-in-10-z2.arcor-online.net (mail-in-10-z2.arcor-online.net [151.189.8.27]) by mail-in-03.arcor-online.net (Postfix) with ESMTP id B9B112CB2A9; Sun, 06 Apr 2008 04:27:43 +0200 (CEST) Received: from mail-in-07.arcor-online.net (mail-in-07.arcor-online.net [151.189.21.47]) by mail-in-10-z2.arcor-online.net (Postfix) with ESMTP id A687F23D40B; Sun, 06 Apr 2008 04:27:43 +0200 (CEST) Received: from jupiterb48.nrubsig.org (dslb-084-059-001-044.pools.arcor-ip.net [84.59.1.44]) by mail-in-07.arcor-online.net (Postfix) with ESMTP id 860CF292B61; Sun, 06 Apr 2008 04:27:43 +0200 (CEST) Received: from nrubsig.org (localhost [127.0.0.1]) by jupiterb48.nrubsig.org (8.13.8+Sun/8.13.8) with ESMTP id m362RfPO001248; Sun, 06 Apr 2008 04:27:42 +0200 (CEST) Date: Sun, 06 Apr 2008 04:27:41 +0200 From: Roland Mainz Subject: Re: 2008/226 mmapfd and 2008/195 validated execution Sender: gisburn@jupiterb48.nrubsig.org To: Bart Smaalders Cc: John Zolnowsky x69422/408-404-5064 , PSARC-ext@sun.com, valex-core@sun.com, gww@eng.sun.com Message-id: <47F8351D.7F125B14@nrubsig.org> MIME-version: 1.0 X-Mailer: Mozilla 4.76 [en] (X11; U; SunOS 5.11 sun4u) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-Accept-Language: en X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Virus-Scanned: ClamAV 0.92.1/6628/Sun Apr 6 03:14:00 2008 on mail-in-07.arcor-online.net X-Virus-Status: Clean X-Antispam: No, score=-2.6/5.0, scanned in 0.060sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ References: <200804040038.m340cps12268@domus.sfbay.sun.com> <47F5A471.10408@Sun.COM> Status: RO Content-Length: 1273 Bart Smaalders wrote: > John Zolnowsky x69422/408-404-5064 wrote: > > The general nature of mmapfd() mapping represents a possible solution > > to a concern being discussed in 2008/195. The issue is that > > interpreters other than rtld often have the equivalent of libraries, > > for example, perl's .pm and .pl or the shell "source" or "." commands. > > These extended forms of library are presently introduced into the > > process "execution" using general interfaces (open(), read()), > > precluding any reliable triggering for validition of the object. As > > much as mmapfd() provides a generalized mechanism for accessing these > > forms of libraries, it would serve as a enabler for validated > > execution. > > cat /etc/file | sh ??? Or worse: "eval" ... ... another issue is that shells like "ksh" support function libraries (via FPATH) which are loaded on demand... how should this be handled ? > It seems to be that validated execution is somewhat missing the point > by focusing on "execution". > > I'll take my comments to 2008/195 when I get a chance.. /me, too... ---- Bye, Roland -- __ . . __ (o.\ \/ /.o) roland.mainz@nrubsig.org \__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer /O /==\ O\ TEL +49 641 7950090 (;O/ \/ \O;) From Blake.Jones@sun.com Mon Apr 7 11:50:42 2008 Received: from jurassic-x4600.sfbay.sun.com (cretaceous.SFBay.Sun.COM [129.146.17.63]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m37IogX8025213 for ; Mon, 7 Apr 2008 11:50:42 -0700 (PDT) Received: from jurassic-x4600.sfbay.sun.com (localhost [127.0.0.1]) by jurassic-x4600.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m37Iogve858461; Mon, 7 Apr 2008 11:50:42 -0700 (PDT) To: Roland Mainz cc: Michael Corcoran , Rod Evans , psarc-ext@sac.sfbay.sun.com Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor In-Reply-To: Your message of "Sun, 06 Apr 2008 04:02:46 +0200." <47F82F46.BDE98FF8@nrubsig.org> From: Blake Jones Reply-To: Blake Jones Date: Mon, 07 Apr 2008 11:50:42 -0700 Message-ID: <858460.1207594242@jurassic-x4600.sfbay.sun.com> Sender: Blake.Jones@sun.com Status: RO Content-Length: 666 >> To elaborate on this: part of the idea behind mmapfd() is that it >> gives the kernel the flexibility to make global-scope decisions about >> where and how a file should be mapped. [...] > > The problem is that (at least in the B72 installation on the Oct 2007 > Summit's T2000 machine) the auto-MPSS stuff performs (IMO) poorly or > does nothing That's an implementation issue which is the focus of ongoing work. mmapfd() is a new interface which is another step toward making the kernel's automatic decisions better. You will still be able to use the memcntl(MC_HAT_ADVISE) (and mpss.so.1) interfaces with mmapfd(), just as you can with mmap() today. Blake From Michael.Corcoran@Sun.COM Mon Apr 7 13:17:43 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m37KHhXG028709 for ; Mon, 7 Apr 2008 13:17:43 -0700 (PDT) Received: from brmea-mail-1.sun.com (brmea-mail-1.Sun.COM [192.18.98.31]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m37KHg33009102 for ; Mon, 7 Apr 2008 13:17:43 -0700 (PDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m37KHgwf005091 for ; Mon, 7 Apr 2008 20:17:42 GMT Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JYZ00C010PWI200@mail-amer.sun.com> (original mail from Michael.Corcoran@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Mon, 07 Apr 2008 14:17:42 -0600 (MDT) Received: from musetta ([129.153.88.82]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JYZ00HDB1OXZW70@mail-amer.sun.com>; Mon, 07 Apr 2008 14:17:22 -0600 (MDT) Date: Mon, 07 Apr 2008 13:17:17 -0700 From: Michael Corcoran Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor In-reply-to: <47F82E66.99E057E@nrubsig.org> Sender: Michael.Corcoran@Sun.COM To: Roland Mainz Cc: Michael.Corcoran@Sun.COM, Rod Evans , psarc-ext@sac.sfbay.sun.com Message-id: <1207599437.6990.181.camel@musetta> MIME-version: 1.0 X-Mailer: Evolution 2.6.2 Content-type: text/plain Content-transfer-encoding: 7BIT References: <47F0750D.4040809@sun.com> <47F0CD06.25D53234@nrubsig.org> <1206992622.1298.222.camel@musetta> <47F82E66.99E057E@nrubsig.org> Status: RO Content-Length: 9438 On Sun, 2008-04-06 at 03:59 +0200, Roland Mainz wrote: > Michael Corcoran wrote: > > On Mon, 2008-03-31 at 13:37 +0200, Roland Mainz wrote: > > > Rod Evans wrote: > > > > I'm sponsoring the following case for Mike Corcoran. Time out 04/07/08. > > > > > > > > The case introduces a new system call, mmapfd(2). This call is primarily > > > > targeted for use by ld.so.1(1), and provides for the efficient mapping of > > > > ELF files (and 4.x AOUT files). > > > > > > > > Release Binding: Patch/Micro > > > > mmapfd(2): Consolidation Private > > > > > > > > -------------------------------------------------------------------------- > > > > > > > > 1. Introduction > > > > 1.1. Project/Component Working Name: > > > > mmapfd: mmap file descriptor > > > [snip] > > > > 4. Technical Description: > > > > 4.1. Details: > > > > mmapfd is a new system call which can interpret and map ELF and AOUT > > > > (4.x) objects. This system call allows the interpretation and mapping > > > > of ELF and AOUT files to be carried out completely by the kernel rather > > > > than by ld.so.1. > > > > > > Erm... the call seems to be ELF+AOUT-specifc - why does it have such a > > > generic name ? > > > > It has a generic name for future expansion. > > Yeah... but |mmapfd()| sounds it's for generic usage and what's > described here is IMO very specific and therefore a longer (and more > specific) name may be nice... > We're working on a more specific name as I'm sure you've seen from other emails on the subject. I think mmapobj() is probably what we will go with. > > > The call can't handle other types of executables (e.g. > > > "javaexec" or "shbinexec") > > > > Not yet, but that doesn't mean that it won't. If there is a file format > > that needs to be interpreted in order to be mapped, this new syscall is > > a good way of going about doing that. > > What is the exact definition of "interpreted" in this case - and why is > the kernel needed for this ? Just for sharing the interpreted data > between processes, e.g. to avoid that multiple processes waste memory > with multiple copies of the same (interpreted) data ? > "interpreted" means that given a file, based upon the type of the file, it will be mapped accordingly. Thus for ELF files, it will see the ELF magic numbers at the very beginning of the file and thus read (interpret) the headers to create all the mappings needed to map that given ELF file. The same would hold true for any other sort of file that would need some form of interpretation to map it. The kernel isn't necessarily needed to do this, but it is useful for it to do so for a few reasons. First, the kernel already does this interpretation via elfexec when exec()'ing a process. So if the kernel already knows how to read ELF files, for example, then why require the linker to do so as well. In addition, this will allow the kernel to decide how best to map the file, including what addresses to use and what page sizes to use. In the past, the linker kept getting requests for new flags to do "special things" for given mappings, possibly only for a given platform, and with mmapfd() the kernel will be in control and will do the right thing for a given platform. So the whole point is really to centralize control and have the kernel make decisions based upon the knowledge that it has of the system. > [snip] > > > ... what renaming the call to |mmapexecfd()| (= "memory map of > > > executable fd") ? > > > > I like Darren's suggestion of mmapv(2) better so far. Why limit us to > > only being able to interpret executable files? What if there's some > > other non-executable file type that would naturally use this interface? > > Mhhh... Ok... but |mmapv()| somehow sounds like a vector version of > |mmap()| (e.g. something which accepts an array of |mmap()| options and > handles them all in one single syscall) ... > ... what about |mmapintp()| ("mmap interpreted") ? > Agreed that mmapv wasn't quite right as pointed out in a previous email. The only issue with mmapintp is if we want to map a file without any form of interpretation. I'm still not quite sure why we'd want to do this since one could always use mmap for this, but "interpreted" is a bit limiting whereas mmapobj is a bit more generic since we're mapping an object and returning the info about the mappings which were made for the object. > > > > mmapfd also provides for mapping a whole file, without interpretation > > > > in a read only mode. > > > > > > What does that mean ? Can these data+MMU mappings be shared between > > > processes ? > > > > It means that you can map a file read-only without doing any > > interpretation of the file. Thus if you passed in an ELF file without > > the MMFD_INTERPRET flag, the file would get mapped as a single read-only > > segment. > > Ok... > > > The data+MMU mappings would not be shared between processes. > > Why ? > > > > [snip] > > > > System Calls mmapfd(2) > > > > > > > > NAME > > > > > > > > mmapfd - map a file descriptor in the appropriate manner. > > > > > > > > SYNOPSIS > > > > > > > > #include > > > > > > > > int > > > > mmapfd(int fd, uint_t flags, mmapfd_result_t *storage, > > > > uint_t *elements, void *arg) > > > > > > Uhm... how do I unmap the mapping done by |mmapfd()| ? > > > > > Nico answered this in a subsequent post and you would use munmap(2) on > > each element in the "storage" array to unmap that segment. I agree that > > this should be explicitly pointed out somewhere since it's not clear. > > What about having a helper function to unmap the storage array in one > step (IMO it would be nice and it would maintain the Unix "tradition" of > symmetric function call pairs (e.g. |open()|+|close()|, > |malloc()|+|free()| etc.) ? > I disagree here. If we had a helper function which took an array of mmapfd_result_t's for unmapping, then we'd need to have another function for mprotect ... The main difference between mmap and mmapfd is that mmapfd can map and return multiple address ranges while mmap will only map and return a single address range. > [snip] > > > > TYPES USED > > > > > > > > typedef struct { > > > > caddr_t mr_addr; /* mapping address */ > > > > size_t mr_msize; /* mapping size */ > > > > size_t mr_fsize; /* file size */ > > > > size_t mr_offset; /* offset into file */ > > > > int mr_prot; /* the protections provided */ > > > > > > Why is this |signed int| ? > > > > I agree it should be uint_t or even uchar_t. Both appear to be commonly > > used throughout the kernel. uint_t is my preference here. > > Ok... > > > > > uint_t mr_flags; /* info on the mapping */ > > > > > > Please change this to |uint64_t| (e.g. it may be nice to have more flags > > > available by default). > > > > > I go back and forth on this. uint64_t implies that 32 flags will not be > > enough. 32 seems like a lot > > Yeah... but after some time the number of flags gets used-up... > > > but since this interface should be around > > for a long time, maybe 64 would be better since it will take a long time > > to use all 64 :) > > IMO 64 is better (and IMO each system/library call used to create a > resource should take a "flags" option that the call can be > changed/extended easily) ... > > > > > } mmapfd_result_t; > > > > > > > > Values for mr_flags include: > > > > > > > > MFD_ELF_HDR 0x1 /* the ELF header is mapped at mr_addr */ > > > > MFD_AOUT_HDR 0x2 /* the AOUT header is mapped at mr_addr */ > > > > > > What about reserving the first four bits for |MFD_*_HDR| flags > > > (|MFD_ELF_HDR|, |MFD_AOUT_HDR| and two bits reserved for future > > > |MFD_*_HDR| flags) ? > > > > I can see the vanity reason for doing this, it's nice to clump things in > > groups of 4 with everything in the group being related, but at the same > > time, why the first 4 flags and not the first 8 or 16 or ... I think > > densely packing the bits is easiest. > > I'm willing to move MFD_PADDING > > first in the list aka 0x1 so that future header types will follow > > numerically from the current header types. I'll make this change since > > it is aesthetically more pleasing. > > Ok... > > > > Finally... how can I _force_ the mapping to use something like 64k pages > > > by default ? > > > > Using mpss.so.1 or ppgsz are ways to get the heap segment to be a > > specific size. Other than that, there is no control for the page size > > used by the other segments and the kernel will pick the best size for > > the given platform. > > Uufortunately (at least tested with B72) the kernel seems to do a bad > job in this case (e.g. the stack won't be mapped with 64k pages unless > the size significantly exceeds 64k in size and code doens't get mapped > wth 64k pages at all), even on platforms (like the T2000 we had on the > Oct 2007 Summit) where I would expect extensive use of 64k pages... ;-( > ... IMO a manual control would be nice... > That would be a different project. We're working on these sort of issues but it's orthogonal to this discussion. Thanks, Mike > ---- > > Bye, > Roland > From Rod.Evans@sun.com Tue Apr 8 13:08:39 2008 Received: from jurassic-x4600.sfbay.sun.com (cretaceous.SFBay.Sun.COM [129.146.17.63]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m38K8dAp008460 for ; Tue, 8 Apr 2008 13:08:39 -0700 (PDT) Received: from [129.146.228.120] (chaz.SFBay.Sun.COM [129.146.228.120]) by jurassic-x4600.sfbay.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m38K8dGn181746; Tue, 8 Apr 2008 13:08:39 -0700 (PDT) Message-ID: <47FBD06E.6070309@sun.com> Date: Tue, 08 Apr 2008 13:07:10 -0700 From: Rod Evans Reply-To: Rod.Evans@sun.com Organization: Sun Microsystems Inc. User-Agent: Thunderbird 2.0.0.6 (X11/20071119) MIME-Version: 1.0 To: psarc-ext@sac.sfbay.sun.com CC: Rod Evans , Michael Corcoran Subject: Re: 2008/226: mmapfd(2) - mmap file descriptor Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 702 The following minor changes are being made to this case material to capture the comments received so far. An updated man page has been added to the materials directory. i. change name from mmapfd(2) to mmapobj(2). ii. change data structure name, and associated flags to follow mmapobj convention. iii. change values for mr_flags field of mmapobj_result_t (the present flags are mutually exclusive). #define MOBJ_PADDING 0x1 #define MOBJ_HDR_ELF 0x2 #define MOBJ_HDR_AOUT 0x3 iv. change mr_prot field of mmapobj_result_t from an int to uint_t. v. various other small man page updates. -- Rod.