From jw137282@sac.sfbay.sun.com Fri Mar 6 11:39:14 2009 Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n26JdDRA026596 for ; Fri, 6 Mar 2009 11:39:14 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n26Jd5Sd006836; Sat, 7 Mar 2009 03:39:12 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KG300J1SNXB0N00@nwk-avmta-1.sfbay.Sun.COM>; Fri, 06 Mar 2009 11:39:11 -0800 (PST) Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KG3008BJNXAVBF0@nwk-avmta-1.sfbay.Sun.COM>; Fri, 06 Mar 2009 11:39:10 -0800 (PST) Received: from sac.sfbay.sun.com (sac.SFBay.Sun.COM [129.146.226.132]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n26Jd8pn034285; Fri, 06 Mar 2009 11:39:08 -0800 (PST) Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n26Jd76U026591; Fri, 06 Mar 2009 11:39:07 -0800 (PST) Received: (from jw137282@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id n26Jd7fa026587; Fri, 06 Mar 2009 11:39:07 -0800 (PST) Date: Fri, 06 Mar 2009 11:39:07 -0800 (PST) From: James Walker Subject: aget [PSARC/2009/159 FastTrack timeout 03/13/2009] To: PSARC-ext@sun.com Cc: Si-Wei.Liu@sun.com Message-id: <200903061939.n26Jd7fa026587@sac.sfbay.sun.com> Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 Status: RO Content-Length: 11968 I'm sponsoring this familiarity case for Si-Wei Louis Liu. The requested release binding is minor. The man page has been posted in the materials directory. Note. aget currently does not support IPv6, but the project team will work to support it when resources become available and work with the community in the mean time. Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI This information is Copyright 2009 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: aget 1.2. Name of Document Author/Supplier: Author: Louis Liu 1.3 Date of This Document: 06 March, 2009 4. Technical Description FCL--FOSS Check List 1.0 Project Information 1.1 Name of project/component Aget 1.2 Author of document Si-wei Louis Liu 2.0 Project Summary 2.1 Project Description Aget is a program for mutli-threaded HTTP downloading in the text mode. It fetches HTTP URLs in a manner similar to wget, but segments the retrieval into multiple parts to increase download speed, while each parts could be resumed automatically once download failure occurs. Aget can be many times as fast as wget in some circumstances. Please be noted that aget does not currently support IPv6 and there are no resources available to add support at this time. Si-wei Liu will work with the upstream community to provide IPv6 support. Aget-0.4 will be integrated into the SFW consolidation as part of this proposal, and will be installed as SUNWaget. 2.2 Release binding What is is the release binding? (see http://opensolaris.org/os/community/arc/policies/release-taxonomy/) [ ] Major [X] Minor [ ] Patch or Micro [ ] Unknown -- ARC review required 2.3 Type of project Is this case a Linux Familiarity project? [X] Yes [ ] No 2.4 Originating Community 2.4.1 Community Name aget - Adnan Sancak, Murat Balaban [1] 2.4.2 Community Involvement Indicate Sun's involvement in the community [ ] Maintainer [ ] Contributor [X] Monitoring Will the project team work with the upstream community to resolve architectural issues of interest to Sun? [X] Yes [ ] No - briefly explain Will we or are we forking from the community? [ ] Yes - ARC review required prior to forking [X] No 3.0 Technical Description 3.1 Installation & Sharable 3.1.1S Solaris Installation - section only required for Solaris Software Does this project follow the Install Locations best practice? [X] Yes [ ] No - ARC review required Does this project install into /usr under [sbin|bin|lib|include|man|share]? [X] Yes [ ] No or N/A Does this project install into /opt? [ ] Yes - explain below [X] No or N/A Does this project install into a different directory structure? [ ] Yes - ARC review required [X] No or N/A Do any of the components of this project conflict with anything under /usr? (see http://opensolaris.org/os/community/arc/caselog/2007/047/ for details) [ ] Yes - explain below [X] No If conflicts exist then will this project install under /usr/gnu? [ ] Yes [ ] No - ARC review required [X] N/A Is this project installing into /usr/sfw? [ ] Yes - ARC review required [X] No 3.1.1W Windows Installation - section only required for Windows Software (see http://sac.sfbay/WSARC/2002/494 for details) Does this project install software into a :\Program Files\Sun\ or :\Sun\ directory? [ ] Yes [ ] No - ARC review required Does the project use the Windows registry? [ ] Yes [ ] No - ARC review required Does the project use HKEY_LOCAL_MACHINE\SOFTWARE\Sun Microsystems\\ for the registry key? [ ] Yes [ ] No - ARC review required Is the project's stored location HKEY_LOCAL_MACHINE\SOFTWARE\Sun Microsystems\\\Path? [ ] Yes [ ] No - ARC review required 3.1.2 Share and Sharable Does the module include any components that are used or shared by other projects? [ ] Yes [X] No If yes are these components packaged to be shared with the other FOSS? [ ] Yes [ ] No - ARC review required [X] N/A Are these components already in the Solaris WOS? [ ] Yes [X] No - continue with next section (section 3.2) If yes are these newer versions being delivered? [ ] Yes [ ] No - ARC review required If yes are the newer versions replacing the existing versions? [ ] Yes [ ] No - ARC review required 3.2 Exported Libraries Are libraries being delivered by this project? [ ] Yes [X] No - continue with next section (section 3.3) Are 64-bit versions of the libraries being delivered? [ ] Yes [ ] No - ARC review required Are static versions of the libraries being delivered? [ ] Yes - ARC review required [ ] No 3.3 Services and the /etc Directory (see http://opensolaris.org/os/community/arc/policies/SMF-policy/) Does the project integrate anything into /etc/init.d or /etc/rc?.d? [ ] Yes - ARC review required [X] No Does the project integrate any new entries into /etc/inittab or /etc/inetd.conf? [ ] Yes - ARC review required [X] No Does the project integrate any private non-public files into /etc/default or /etc/ configuration files? [ ] Yes - ARC review required [X] No Does the service manifests method context grant rights above that of the noaccess user and basic privilege set? [ ] Yes - ARC review required [X] No 3.4 Security 3.4.1 Secure By Default Are there any network services provided by this project? [ ] Yes [X] No - continue with the next section (section 3.4.2) Are network services enabled by default? [ ] Yes - ARC review required [ ] No [ ] N/A Are network services automatically enabled by the project during installation? [ ] Yes - ARC review required [ ] No [ ] N/A Are inbound network communications denied by default? [ ] Yes [ ] No - ARC review required [ ] N/A Is inbound data checked to prevent content-based attacks? [ ] Yes [ ] No - ARC review required [ ] N/A Is the outbound receiver authenticated? [ ] Yes [ ] No - ARC review required [ ] N/A Is the receiver authenticated prior to receiving any sensitive outbound communication? [ ] Yes [ ] No - ARC review required [ ] N/A 3.4.2 Authorization Are there any setuid/setgid privileged binaries in the project? [ ] Yes - ARC review required [X] No - continue with next section (section 3.4.3) If yes then are the setuid/setgid privileges handled by the use of roles? [ ] Yes [ ] No - ARC review required 3.4.3 Auditing (see http://opensolaris.org/os/community/arc/policies/audit-policy/ for details) (see http://opensolaris.org/os/community/arc/caselog/2003/397 for details) Does this component contain administrative or security enforcing software? [ ] Yes - ARC review required [X] No - continue to next section (section 3.4.4) (see http://opensolaris.org/os/community/arc/caselog/2003/397 for details) Do the components create audit logs detailing what took place including what event took place, who was involved, when the event took place? [ ] Yes - ARC contract and Audit project team review required [ ] No - ARC review required 3.4.4 Authentication Do the components contain any authentication code? [ ] Yes [X] No - continue to next section (section 3.4.5) If yes do the components use PAM (plugable authentication modules) for authentication? [ ] Yes [ ] No - ARC review required If yes is a single PAM session maintained during authentication? [ ] Yes [ ] No - ARC review required If yes are the components sufficiently privileged to allow the requested operations (authentication, password change, process credential manipulation, audit state initialization)? [ ] Yes - briefly describe below [ ] No - ARC review required 3.4.5 Passwords Do any of the components for the project deal with passwords? [ ] Yes [X] No - continue to next section (section 3.4.6) If yes are these passwords entered via the CLI or environment? [ ] Yes - ARC review required [ ] No Are passwords stored within the file system for the component? [ ] Yes [ ] No - continue to next section (section 3.4.6) If yes are the permissions on the file such to protect exposing the password(s)? [ ] Yes [ ] No - ARC review required 3.4.6 General Security Questions (see http://opensolaris.org/os/community/arc/bestpractices/security-questions/ for details) Are there any network protocols used by this project? [X] Yes [ ] No - continue with the next section (section 3.5) Do the components use standard network protocols? [X] Yes [ ] No - ARC review required Do network services for the project make decisions based upon user, host or service identities? [ ] Yes - explain below [X] No [ ] N/A Do the components make use of secret information during authentication and/or authorization? [ ] Yes - explain below [X] No [ ] N/A 3.5 Networking Do the components access the network? [X] Yes [ ] No - continue with the next section (section 3.6) If yes do the components support IPv6? [ ] Yes [X] No - ARC review required 3.6 Core Solaris Components Do the components of this project compete with or duplicate core Solaris components? [ ] Yes - ARC review required [X] No Examples of Core Solaris Components include but are not limited to: Secure By Default Authorizations PAM -- Plugable Authentication Module Privilege PRM -- Process Rights Management -- Privilege Audit xVm -- Virtualization zones / Solaris Containers PRM -- Process Rights Management RBAC -- Role Based Access Control TX / Trusted Extensions ZFS SMF -- Service Management Facility FMA -- Fault Management Architecture SCF -- Smart Card Facility IPsec 4.0 Interfaces 4.1 Exported Interfaces Interface Name Classification Comments --------------------------- -------------- ------------------------ SUNWaget Uncommitted Package name /usr/bin/aget Uncommitted Command 4.2 Imported Interfaces Interface Name Classification Comments --------------------------- -------------- -------------------------- SUNWcsl Committed Core Solaris Libraries Appendix A - References [1] http://www.enderunix.org/aget/ OSR ID# 10898 RFE ID# 6806771 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: SFW 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From carlsonj@phorcys.east.sun.com Fri Mar 6 12:51:05 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n26Kp5kj023052 for ; Fri, 6 Mar 2009 12:51:05 -0800 (PST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n26Kp3gg010335; Fri, 6 Mar 2009 12:51:04 -0800 (PST) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KG300M07R93VZ00@nwk-avmta-2.sfbay.sun.com>; Fri, 06 Mar 2009 12:51:03 -0800 (PST) Received: from phorcys.east.sun.com ([129.148.174.143]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KG300MS7R927Y00@nwk-avmta-2.sfbay.sun.com>; Fri, 06 Mar 2009 12:51:02 -0800 (PST) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n26KoteF013308; Fri, 06 Mar 2009 15:50:55 -0500 (EST) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n26KotKb013305; Fri, 06 Mar 2009 15:50:55 -0500 (EST) Date: Fri, 06 Mar 2009 15:50:55 -0500 From: James Carlson Subject: Re: aget [PSARC/2009/159 FastTrack timeout 03/13/2009] In-reply-to: <200903061939.n26Jd7fa026587@sac.sfbay.sun.com> To: James Walker Cc: PSARC-ext@sun.com, Si-Wei.Liu@sun.com Message-id: <18865.36015.85909.667649@gargle.gargle.HOWL> MIME-version: 1.0 X-Mailer: VM 7.01 under Emacs 21.3.1 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200903061939.n26Jd7fa026587@sac.sfbay.sun.com> Status: RO Content-Length: 681 James Walker writes: > Aget is a program for mutli-threaded HTTP downloading in the > text mode. It fetches HTTP URLs in a manner similar to wget, but > segments the retrieval into multiple parts to increase download > speed, while each parts could be resumed automatically once > download failure occurs. Aget can be many times as fast as wget > in some circumstances. Why aget and not puf? Have you compared them? -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From Si-Wei.Liu@sun.com Sun Mar 8 21:53:40 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n294re0J015390 for ; Sun, 8 Mar 2009 21:53:40 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n294rc8r015029 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Mon, 9 Mar 2009 04:53:39 GMT Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KG800C012XE0300@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 08 Mar 2009 21:53:38 -0700 (PDT) Received: from sineb-mail-2.sun.com ([192.18.19.7]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KG800HAK2XDYRD0@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Sun, 08 Mar 2009 21:53:38 -0700 (PDT) Received: from fe-apac-06.sun.com (fe-apac-06.sun.com [192.18.19.177] (may be forged)) by sineb-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n294raq5019086 for ; Mon, 09 Mar 2009 04:53:36 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23 2008)) id <0KG800J002WM4H00@mail-apac.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 09 Mar 2009 12:53:36 +0800 (SGT) Received: from [129.158.215.29] ([unknown] [129.158.215.29]) by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23 2008)) with ESMTPSA id <0KG800KKW2X7KYC0@mail-apac.sun.com>; Mon, 09 Mar 2009 12:53:36 +0800 (SGT) Date: Mon, 09 Mar 2009 12:51:42 +0800 From: Si-wei Louis Liu Subject: Re: aget [PSARC/2009/159 FastTrack timeout 03/13/2009] In-reply-to: <18865.36015.85909.667649@gargle.gargle.HOWL> Sender: Si-Wei.Liu@sun.com To: James Carlson Cc: James Walker , PSARC-ext@sun.com Message-id: <49B4A05E.7050305@Sun.COM> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_cCVPfqG1SVoQqyBdGA//OA)" X-PMX-Version: 5.4.1.325704 References: <200903061939.n26Jd7fa026587@sac.sfbay.sun.com> <18865.36015.85909.667649@gargle.gargle.HOWL> User-Agent: Thunderbird 2.0.0.18 (X11/20081201) Status: RO Content-Length: 3351 This is a multi-part message in MIME format. --Boundary_(ID_cCVPfqG1SVoQqyBdGA//OA) Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT I don't think puf is totally functioning equal to aget. ;-) puf, just as its name, fetches bunch of URLs in parallel, more like another wget with parallelism. However, it cannot simply download a large file (i.e. a kernel archive like, an .iso image, etc.) in parallel, esp. over a not so fast network. Aget can fill this gap by dividing the large file downloaded into multiple parts, each of which is handled by a pthread, and facilitates the falling over from download failures. Moreover, aget is based on BSD(-like) license, and its been ported to OpenBSD, NetBSD, FreeBSD, and Linux, etc. long before. This is another reason for selecting aget as multi-threaded HTTP file downloader on Solaris. Thanks, -Louis On 2009/03/07, 04:50, James Carlson wrote: > James Walker writes: > >> Aget is a program for mutli-threaded HTTP downloading in the >> text mode. It fetches HTTP URLs in a manner similar to wget, but >> segments the retrieval into multiple parts to increase download >> speed, while each parts could be resumed automatically once >> download failure occurs. Aget can be many times as fast as wget >> in some circumstances. >> > > Why aget and not puf? Have you compared them? > > --Boundary_(ID_cCVPfqG1SVoQqyBdGA//OA) Content-type: text/html; charset=ISO-8859-1 Content-transfer-encoding: 7BIT I don't think puf is totally functioning equal to aget. ;-)

puf, just as its name, fetches bunch of URLs in parallel, more like another wget with parallelism. However, it cannot simply download a large file (i.e. a kernel archive like, an .iso image, etc.) in parallel, esp. over a not so fast network. Aget can fill this gap by dividing the large file downloaded into multiple parts, each of which is handled by a pthread, and facilitates the falling over from download failures.

Moreover, aget is based on BSD(-like) license, and its been ported to OpenBSD, NetBSD, FreeBSD, and Linux, etc. long before. This is another reason for selecting aget as multi-threaded HTTP file downloader on Solaris.

Thanks,
-Louis

On 2009/03/07, 04:50, James Carlson wrote: 
James Walker writes:
  
      Aget is a program for mutli-threaded HTTP downloading in the 
      text mode. It fetches HTTP URLs in a manner similar to wget, but 
      segments the retrieval into multiple parts to increase download
      speed, while each parts could be resumed automatically once 
      download failure occurs. Aget can be many times as fast as wget
      in some circumstances.
    

Why aget and not puf?  Have you compared them?

--Boundary_(ID_cCVPfqG1SVoQqyBdGA//OA)-- From carlsonj@phorcys.east.sun.com Mon Mar 9 06:07:40 2009 Received: from sunmail2sca.sfbay.sun.com (sunmail2sca.SFBay.Sun.COM [129.145.155.234]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n29D7eRj007214 for ; Mon, 9 Mar 2009 06:07:40 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail2sca.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n29D7Y3E012356; Mon, 9 Mar 2009 06:07:39 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KG800G05PSQ1K00@nwk-avmta-1.sfbay.Sun.COM>; Mon, 09 Mar 2009 06:07:38 -0700 (PDT) Received: from phorcys.east.sun.com ([129.148.174.143]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KG800BGBPSPDDE0@nwk-avmta-1.sfbay.Sun.COM>; Mon, 09 Mar 2009 06:07:38 -0700 (PDT) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n29D7RYc017137; Mon, 09 Mar 2009 09:07:27 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n29D7ROD017134; Mon, 09 Mar 2009 09:07:27 -0400 (EDT) Date: Mon, 09 Mar 2009 09:07:27 -0400 From: James Carlson Subject: Re: aget [PSARC/2009/159 FastTrack timeout 03/13/2009] In-reply-to: <49B4A05E.7050305@Sun.COM> To: Si-wei Louis Liu Cc: James Walker , PSARC-ext@sun.com Message-id: <18869.5263.540062.951878@gargle.gargle.HOWL> MIME-version: 1.0 X-Mailer: VM 7.01 under Emacs 21.3.1 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200903061939.n26Jd7fa026587@sac.sfbay.sun.com> <18865.36015.85909.667649@gargle.gargle.HOWL> <49B4A05E.7050305@Sun.COM> Status: RO Content-Length: 1431 Si-wei Louis Liu writes: > I don't think puf is totally functioning equal to aget. ;-) I wasn't necessarily suggesting they were; just surprised to see the somewhat obscure aget proposed with no mention of 'puf.' > puf, just as its name, fetches bunch of URLs in parallel, more like > another wget with parallelism. However, it cannot simply download a > large file (i.e. a kernel archive like, an .iso image, etc.) in > parallel, esp. over a not so fast network. Aget can fill this gap by > dividing the large file downloaded into multiple parts, each of which is > handled by a pthread, and facilitates the falling over from download > failures. OK ... though there are likely a few controversial issues buried in there. > Moreover, aget is based on BSD(-like) license, and its been ported to > OpenBSD, NetBSD, FreeBSD, and Linux, etc. long before. This is another > reason for selecting aget as multi-threaded HTTP file downloader on Solaris. I don't think the license really matters, at least in this review. The unfortunate thing here is the apparent long-standing disagreement between the aget and wget folks on multiple streams, leading to duplicate tools. *sigh* In any event, +1. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From Si-Wei.Liu@sun.com Mon Mar 9 08:37:26 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n29FbQfO001075 for ; Mon, 9 Mar 2009 08:37:26 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n29FbHqV004244 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Mon, 9 Mar 2009 15:37:25 GMT Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KG80091JWQAYT00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 09 Mar 2009 09:37:22 -0600 (MDT) Received: from gmp-eb-inf-1.sun.com ([192.18.6.21]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KG8007KMWQ9D020@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 09 Mar 2009 09:37:21 -0600 (MDT) Received: from fe-emea-09.sun.com (gmp-eb-lb-2-fe3.eu.sun.com [192.18.6.12]) by gmp-eb-inf-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n29FbKqM026361 for ; Mon, 09 Mar 2009 15:37:20 +0000 (GMT) Received: from conversion-daemon.fe-emea-09.sun.com by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23 2008)) id <0KG800500VCWV000@fe-emea-09.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 09 Mar 2009 15:37:20 +0000 (GMT) Received: from [129.150.144.31] ([unknown] [129.150.144.31]) by fe-emea-09.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23 2008)) with ESMTPSA id <0KG8001ZQWPZP2D0@fe-emea-09.sun.com>; Mon, 09 Mar 2009 15:37:14 +0000 (GMT) Date: Mon, 09 Mar 2009 23:37:14 +0800 From: Siwei Liu - Sun Microsystems - Beijing China Subject: Re: aget [PSARC/2009/159 FastTrack timeout 03/13/2009] In-reply-to: <18869.5263.540062.951878@gargle.gargle.HOWL> Sender: Si-Wei.Liu@sun.com To: James Carlson Cc: James Walker , PSARC-ext@sun.com Message-id: <49B537AA.7080204@Sun.COM> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200903061939.n26Jd7fa026587@sac.sfbay.sun.com> <18865.36015.85909.667649@gargle.gargle.HOWL> <49B4A05E.7050305@Sun.COM> <18869.5263.540062.951878@gargle.gargle.HOWL> User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) Status: RO Content-Length: 2217 James Carlson wrote: > Si-wei Louis Liu writes: > >> I don't think puf is totally functioning equal to aget. ;-) >> > > I wasn't necessarily suggesting they were; just surprised to see the > somewhat obscure aget proposed with no mention of 'puf.' > okay, that's fine. > >> puf, just as its name, fetches bunch of URLs in parallel, more like >> another wget with parallelism. However, it cannot simply download a >> large file (i.e. a kernel archive like, an .iso image, etc.) in >> parallel, esp. over a not so fast network. Aget can fill this gap by >> dividing the large file downloaded into multiple parts, each of which is >> handled by a pthread, and facilitates the falling over from download >> failures. >> > > OK ... though there are likely a few controversial issues buried in > there. > did you mean potential impact of over-stressing the network or servers? I'd like to hear your concerns and comments here. If so, we could apply a patch against aget to limit the connections/pthread to a safe value. Or any other similiar mechanism to minimize the side effect. > >> Moreover, aget is based on BSD(-like) license, and its been ported to >> OpenBSD, NetBSD, FreeBSD, and Linux, etc. long before. This is another >> reason for selecting aget as multi-threaded HTTP file downloader on Solaris. >> > > I don't think the license really matters, at least in this review. > But OSR review was biting me ever... sometimes it's easy to seek a pretty nice software to port, but legals might tell you there are often trademark issues or security concerns, *potentially*. Anyway, apparently I was not that dog in its day. :-( > The unfortunate thing here is the apparent long-standing disagreement > between the aget and wget folks on multiple streams, leading to > duplicate tools. *sigh* > Better late than never. :-) PS, any chance for peer to peer software to integrate into Solaris at present? I ask this just for my own interest. I think Transmission is the one for bittorrent network for the time being. How's the strategy of Solaris over the multiple stream software for now? > In any event, +1. > > Thanks for your review, James. Regards, -Louis From carlsonj@phorcys.east.sun.com Mon Mar 9 08:55:40 2009 Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n29FtevM001276 for ; Mon, 9 Mar 2009 08:55:40 -0700 (PDT) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n29FtUu9014202; Mon, 9 Mar 2009 08:55:38 -0700 (PDT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KG800L3VXKPPX00@nwk-avmta-1.sfbay.Sun.COM>; Mon, 09 Mar 2009 08:55:37 -0700 (PDT) Received: from phorcys.east.sun.com ([129.148.174.143]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KG800F0PXKBN630@nwk-avmta-1.sfbay.Sun.COM>; Mon, 09 Mar 2009 08:55:24 -0700 (PDT) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n29FtDvW017862; Mon, 09 Mar 2009 11:55:13 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n29FtDVM017859; Mon, 09 Mar 2009 11:55:13 -0400 (EDT) Date: Mon, 09 Mar 2009 11:55:13 -0400 From: James Carlson Subject: Re: aget [PSARC/2009/159 FastTrack timeout 03/13/2009] In-reply-to: <49B537AA.7080204@Sun.COM> To: Siwei Liu - Sun Microsystems - Beijing China Cc: James Walker , PSARC-ext@sun.com Message-id: <18869.15329.534836.255399@gargle.gargle.HOWL> MIME-version: 1.0 X-Mailer: VM 7.01 under Emacs 21.3.1 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200903061939.n26Jd7fa026587@sac.sfbay.sun.com> <18865.36015.85909.667649@gargle.gargle.HOWL> <49B4A05E.7050305@Sun.COM> <18869.5263.540062.951878@gargle.gargle.HOWL> <49B537AA.7080204@Sun.COM> Status: RO Content-Length: 2539 Siwei Liu - Sun Microsystems - Beijing China writes: > James Carlson wrote: > > OK ... though there are likely a few controversial issues buried in > > there. > > > did you mean potential impact of over-stressing the network or servers? > I'd like to hear your concerns and comments here. That's one of them. The other is that when TCP is working correctly, there shouldn't be any need for something like this. TCP's designed to fill the pipe. > If so, we could apply a patch against aget to limit the > connections/pthread to a safe value. Or any other similiar mechanism to > minimize the side effect. There's not much point; the malicious user can always launch a flurry of processes to get around any limit at the application UI level. > > The unfortunate thing here is the apparent long-standing disagreement > > between the aget and wget folks on multiple streams, leading to > > duplicate tools. *sigh* > > > Better late than never. :-) > PS, any chance for peer to peer software to integrate into Solaris at > present? I ask this just for my own interest. I think Transmission is > the one for bittorrent network for the time being. How's the strategy of > Solaris over the multiple stream software for now? This likely isn't the right list to discuss that. I suggest networking-discuss@opensolaris.org instead, or perhaps some internal list instead. If you're asking about the architectural review committee's position on that software, I don't think we have one. As an ARC member, I can't see an obvious problem with including it, but I'd have to see a project in front of me for review to comment further. If you're asking about which one of the implementations is "best" or how someone would choose among them, then that sounds like something for the Networking Community to discuss. The ARC typically doesn't get involved in those discussions until there's a decision (that is, a project) to be reviewed. (Though I suppose that a project -could- list alternatives for an inception review ...) If you're asking about resources (personnel) or strategy for Sun's products, then that's something best asked internally, and it has nothing to do with the ARC, and perhaps not much to do with OpenSolaris. At a guess, it's something for your management and/or for the Solaris PAC. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From Si-Wei.Liu@sun.com Mon Mar 9 19:37:56 2009 Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2A2btFJ016710 for ; Mon, 9 Mar 2009 19:37:55 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n2A2bUfH000974 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Tue, 10 Mar 2009 02:37:54 GMT Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KG90060RRANMV00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 09 Mar 2009 20:37:35 -0600 (MDT) Received: from sineb-mail-2.sun.com ([192.18.19.7]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KG9004IARALSL70@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon, 09 Mar 2009 20:37:34 -0600 (MDT) Received: from fe-apac-06.sun.com (fe-apac-06.sun.com [192.18.19.177] (may be forged)) by sineb-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n2A2bXLh013264 for ; Tue, 10 Mar 2009 02:37:33 +0000 (GMT) Received: from conversion-daemon.mail-apac.sun.com by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23 2008)) id <0KG900600R7BBY00@mail-apac.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Tue, 10 Mar 2009 10:37:32 +0800 (SGT) Received: from [129.158.215.28] ([unknown] [129.158.215.28]) by mail-apac.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23 2008)) with ESMTPSA id <0KG9002ICRAJ3E10@mail-apac.sun.com>; Tue, 10 Mar 2009 10:37:32 +0800 (SGT) Date: Tue, 10 Mar 2009 10:36:48 +0800 From: Si-wei Louis Liu Subject: Re: aget [PSARC/2009/159 FastTrack timeout 03/13/2009] In-reply-to: <18869.15329.534836.255399@gargle.gargle.HOWL> Sender: Si-Wei.Liu@sun.com To: James Carlson Cc: James Walker , PSARC-ext@sun.com Message-id: <49B5D240.1000508@Sun.COM> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_ER3h3c4YBFWh8RKzCwIroA)" X-PMX-Version: 5.4.1.325704 References: <200903061939.n26Jd7fa026587@sac.sfbay.sun.com> <18865.36015.85909.667649@gargle.gargle.HOWL> <49B4A05E.7050305@Sun.COM> <18869.5263.540062.951878@gargle.gargle.HOWL> <49B537AA.7080204@Sun.COM> <18869.15329.534836.255399@gargle.gargle.HOWL> User-Agent: Thunderbird 2.0.0.17 (X11/20081023) Status: RO Content-Length: 7162 This is a multi-part message in MIME format. --Boundary_(ID_ER3h3c4YBFWh8RKzCwIroA) Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT Thank you for the constructive reply. I appreciate it very much. -Louis James Carlson wrote: > Siwei Liu - Sun Microsystems - Beijing China writes: > >> James Carlson wrote: >> >>> OK ... though there are likely a few controversial issues buried in >>> there. >>> >>> >> did you mean potential impact of over-stressing the network or servers? >> I'd like to hear your concerns and comments here. >> > > That's one of them. The other is that when TCP is working correctly, > there shouldn't be any need for something like this. TCP's designed > to fill the pipe. > > >> If so, we could apply a patch against aget to limit the >> connections/pthread to a safe value. Or any other similiar mechanism to >> minimize the side effect. >> > > There's not much point; the malicious user can always launch a flurry > of processes to get around any limit at the application UI level. > > >>> The unfortunate thing here is the apparent long-standing disagreement >>> between the aget and wget folks on multiple streams, leading to >>> duplicate tools. *sigh* >>> >>> >> Better late than never. :-) >> PS, any chance for peer to peer software to integrate into Solaris at >> present? I ask this just for my own interest. I think Transmission is >> the one for bittorrent network for the time being. How's the strategy of >> Solaris over the multiple stream software for now? >> > > This likely isn't the right list to discuss that. I suggest > networking-discuss@opensolaris.org instead, or perhaps some internal > list instead. > > If you're asking about the architectural review committee's position > on that software, I don't think we have one. As an ARC member, I > can't see an obvious problem with including it, but I'd have to see a > project in front of me for review to comment further. > > If you're asking about which one of the implementations is "best" or > how someone would choose among them, then that sounds like something > for the Networking Community to discuss. The ARC typically doesn't > get involved in those discussions until there's a decision (that is, a > project) to be reviewed. (Though I suppose that a project -could- > list alternatives for an inception review ...) > > If you're asking about resources (personnel) or strategy for Sun's > products, then that's something best asked internally, and it has > nothing to do with the ARC, and perhaps not much to do with > OpenSolaris. At a guess, it's something for your management and/or > for the Solaris PAC. > > -- Best Regards ______ /_____/\ Si-Wei Liu /____ \\ \ Sun Microsystems China (ERI) /_____\ \\ / Email: Si-Wei.Liu@Sun.COM /_____/ \/ / / Direct: (86-10)6267-3670 /_____/ / \//\ SWAN: 51670 \_____\//\ / / \_____/ / /\ / \_____/ \\ \ \_____\ \\ 7/F, Tower A, Tsinghua Science Park \_____\/ Beijing 100084, P.R.China --Boundary_(ID_ER3h3c4YBFWh8RKzCwIroA) Content-type: text/html; charset=ISO-8859-1 Content-transfer-encoding: 7BIT Thank you for the constructive reply. I appreciate it very much.

-Louis


James Carlson wrote: 
Siwei Liu - Sun Microsystems - Beijing China writes:
  
James Carlson wrote:
    
OK ... though there are likely a few controversial issues buried in
there.
  
      
did you mean potential impact of over-stressing the network or servers? 
I'd like to hear your concerns and comments here.
    

That's one of them.  The other is that when TCP is working correctly,
there shouldn't be any need for something like this.  TCP's designed
to fill the pipe.

  
If so, we could apply a patch against aget to limit the 
connections/pthread to a safe value. Or any other similiar mechanism to 
minimize the side effect.
    

There's not much point; the malicious user can always launch a flurry
of processes to get around any limit at the application UI level.

  
The unfortunate thing here is the apparent long-standing disagreement
between the aget and wget folks on multiple streams, leading to
duplicate tools.  *sigh*
  
      
Better late than never. :-)
PS, any chance for peer to peer software to integrate into Solaris at 
present? I ask this just for my own interest. I think Transmission is 
the one for bittorrent network for the time being. How's the strategy of 
Solaris over the multiple stream software for now?
    

This likely isn't the right list to discuss that.  I suggest
networking-discuss@opensolaris.org instead, or perhaps some internal
list instead.

If you're asking about the architectural review committee's position
on that software, I don't think we have one.  As an ARC member, I
can't see an obvious problem with including it, but I'd have to see a
project in front of me for review to comment further.

If you're asking about which one of the implementations is "best" or
how someone would choose among them, then that sounds like something
for the Networking Community to discuss.  The ARC typically doesn't
get involved in those discussions until there's a decision (that is, a
project) to be reviewed.  (Though I suppose that a project -could-
list alternatives for an inception review ...)

If you're asking about resources (personnel) or strategy for Sun's
products, then that's something best asked internally, and it has
nothing to do with the ARC, and perhaps not much to do with
OpenSolaris.  At a guess, it's something for your management and/or
for the Solaris PAC.

  


-- 
Best Regards
      ______
     /_____/\            Si-Wei Liu
    /____ \\ \           Sun Microsystems China (ERI)
   /_____\ \\ /          Email: Si-Wei.Liu@Sun.COM
  /_____/ \/ / /         Direct: (86-10)6267-3670
 /_____/ /   \//\        SWAN: 51670
 \_____\//\   / /
  \_____/ / /\ /
   \_____/ \\ \
    \_____\ \\           7/F, Tower A, Tsinghua Science Park
     \_____\/            Beijing 100084, P.R.China
--Boundary_(ID_ER3h3c4YBFWh8RKzCwIroA)-- From James.Walker@sun.com Tue Mar 24 10:45:45 2009 Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n2OHjiE3016171 for ; Tue, 24 Mar 2009 10:45:45 -0700 (PDT) Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n2OHjexM008395 for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Tue, 24 Mar 2009 11:45:44 -0600 (MDT) Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KH000D4NUO6LJ00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Tue, 24 Mar 2009 11:45:43 -0600 (MDT) Received: from brmea-mail-2.sun.com ([192.18.98.43]) by brm-avmta-1.central.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KH00092BUO2IF60@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Tue, 24 Mar 2009 11:45:38 -0600 (MDT) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n2OHjcrI009198 for ; Tue, 24 Mar 2009 17:45:38 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) id <0KH000700TAOKG00@mail-amer.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Tue, 24 Mar 2009 11:45:38 -0600 (MDT) Received: from [172.20.25.153] ([unknown] [172.20.25.153]) by mail-amer.sun.com (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009)) with ESMTPSA id <0KH00030AUNJ0TE0@mail-amer.sun.com> for PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Tue, 24 Mar 2009 11:45:19 -0600 (MDT) Date: Tue, 24 Mar 2009 11:50:01 -0600 From: Jim Walker Subject: Re: aget [PSARC/2009/159 FastTrack timeout 03/13/2009] In-reply-to: <200903061939.n26Jd7fa026587@sac.sfbay.sun.com> Sender: James.Walker@sun.com To: PSARC-ext@sun.com Cc: Si-Wei.Liu@sun.com Reply-to: James.Walker@sun.com Message-id: <49C91D49.7080505@sun.com> Organization: Sun Microsystems, Inc. MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT X-PMX-Version: 5.4.1.325704 References: <200903061939.n26Jd7fa026587@sac.sfbay.sun.com> User-Agent: Thunderbird 2.0.0.14 (X11/20080728) Status: RO Content-Length: 106 This case received a +1 and the timeout has expired. I am marking this case closed approved. Cheers, Jim