From sacadmin Wed Apr 5 03:08:51 2006 Received: from cathy.eng.sun.com (cathy.SFBay.Sun.COM [129.146.228.63]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k35A8oIQ012446; Wed, 5 Apr 2006 03:08:51 -0700 (PDT) Received: from cathy.eng.sun.com (localhost [127.0.0.1]) by cathy.eng.sun.com (8.13.5+Sun/8.13.5) with ESMTP id k35A8nd1224403; Wed, 5 Apr 2006 03:08:49 -0700 (PDT) Received: (from bonwick@localhost) by cathy.eng.sun.com (8.13.5+Sun/8.13.5/Submit) id k35A8nHB224396; Wed, 5 Apr 2006 03:08:49 -0700 (PDT) Date: Wed, 5 Apr 2006 03:08:49 -0700 (PDT) From: Jeff Bonwick Message-Id: <200604051008.k35A8nHB224396@cathy.eng.sun.com> To: PSARC@sac.sfbay.sun.com Cc: jeff.bonwick@sun.com Subject: ZFS Hot Spares [PSARC/2006/223 Timeout: 04/11/2006] Status: RO Content-Length: 12887 Subject: PSARC FastTrack [04/11/2006]: ZFS Hot Spares Template Version: @(#)sac_nextcase 1.56 10/26/05 SMI Copyright 2006 Sun Microsystems, Inc. 1. Introduction 1.1. Project/Component Working Name: ZFS Hot Spares 1.2. Name of Document Author/Supplier: Author: Eric Schrock 1.3 Date of This Document: 05 April, 2006 4. Technical Description This case requests patch binding. As with the rest of zpool(1M), the subcommands and their options are Evolving, and the output is Unstable. Previous discussions with the community on this topic can be found at the following threads: http://www.opensolaris.org/jive/thread.jspa?threadID=7303 http://www.opensolaris.org/jive/thread.jspa?threadID=7406 A. DESCRIPTION ZFS, as an integrated volume manager and filesystem, has the ability to replace disks within an active pool. This allows administrators to replace failing or faulted drives to keep the system functioning with the required level of replication. Most other volume managers also support the ability to perform this replacement automatically through the use of "hot spares". This case will add this functionality to ZFS. This case will increment the on-disk version number in accordance to PSARC 2006/206, as the resulting labels introduce a new pool state that older pools will not understand, and exported pools containing hot spares will not be importable on earlier versions. B. POOL MANAGEMENT Hot spares are stored with each pool, although they can be overlapped between different pools. This allows administrators to reserve system-wide hot spares, as well as per-pool hot spares according to their policies. 1. Creating a pool with hot spares A pool can be created with hot spares by using the new 'spare' vdev: # zpool create test mirror c0d0 c1d0 spare c2d0 c3d0 This will create a pool with a single mirror and two spares. Only a single 'spare' vdev can be specified, though it can appear anywhere within the command line. The resulting pool looks like the following: # zpool status pool: test state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 mirror ONLINE 0 0 0 c0d0 ONLINE 0 0 0 c1d0 ONLINE 0 0 0 spares c2d0 AVAILABLE c3d0 AVAILABLE errors: No known data errors To create multiple pools that share the same spares, simply specify the same spares to each command: # zpool create test mirror c0d0 c1d0 spare c2d0 c3d0 # zpool create test mirror c4d0 c5d0 spare c2d0 c3d0 2. Adding hot spares to a pool Hot spares can be added to a pool in the same manner by using 'zpool add': # zpool add test spare c4d0 c5d0 This will add two disks to the set of available spares in the pool. 3. Removing hot spares from a pool Hot spares can be removed from a pool with the new 'zpool remove' subcommand. This subcommand suggests the ability to remove arbitrary devices, and certainly is a feature that will be supported in a future release, but currently this will only allow removing hot spares. For example: # zpool remove test c2d0 If the hot spare is currently spared in, then the command will print an error and exit. 4. Activating a hot spare Hot spares can be used for replacement just like any other device using 'zpool replace'. Both mirrored and RAID-Z devices can be replaced with a hot spare. Even unreplicated devices can be replaced with a hot spare through predictive failure analysis, but the usefulness of such a configuration is questionable. If ZFS detects that the device is a hot spare within the same pool, then it will create a 'spare' vdev instead of a 'replacing' vdev: # zpool replace test c0d0 c2d0 # zpool status ... config: NAME STATE READ WRITE CKSUM test DEGRADED 0 0 0 mirror DEGRADED 0 0 0 spare DEGRADED 0 0 0 c0d0 FAULTED 0 0 0 c2d0 ONLINE 0 0 0 35.5K resilvered c1d0 ONLINE 0 0 0 spares c2d0 INUSE by current pool c3d0 AVAILABLE The difference between a 'replacing' and 'spare' vdev is that the former automatically removes the original drive once the replace completes. With spares, the vdev remains until the original device is removed from the system, at which point the hot spare is returned to the pool of available spares. Note that in this example we have replaced an online device. Under normal circumstances, the device in question would be faulted or the administrator would have proactively offlined the device. The state of a 'spare' vdev is always DEGRADED. When a spare is in use, its status is displaed as INUSE instead of AVAILABLE. If the device is spared in the current pool, it will display "by current pool". If the spare is shared between multiple pools, and it is in use in another pool, it will display "by pool 'foo'". 5. Deactivating a hot spare There are 3 ways in which a hot spare can be deactivated: cancelling the hot spare, replacing the original drive, or permanently swapping in the hot spare. To cancel a hot spare attempt, the user can simply 'zpool detach' the hot spare in question, at which point it will be returned to the set of available spares, the original drive will remain in its current position (faulted or not): # zpool detach test c2d0 # zpool status ... config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 mirror ONLINE 0 0 0 c0d0 ONLINE 0 0 0 35.5K resilvered c1d0 ONLINE 0 0 0 spares c2d0 AVAILABLE c3d0 AVAILABLE If the original device is replaced, then the spare is automatically removed once the replace completes: # zpool replace test c0d0 c4d0 # zpool status ... config: NAME STATE READ WRITE CKSUM test DEGRADED 0 0 0 mirror DEGRADED 0 0 0 spare DEGRADED 0 0 0 replacing ONLINE 0 0 0 c0d0 FAULTED 0 0 0 c4d0 ONLINE 0 0 0 38K resilvered c2d0 ONLINE 0 0 0 38K resilvered c1d0 ONLINE 0 0 0 spares c2d0 INUSE by current pool c3d0 AVAILABLE # zpool status ... config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 mirror ONLINE 0 0 0 c4d0 ONLINE 0 0 0 35.5K resilvered c1d0 ONLINE 0 0 0 spares c2d0 AVAILABLE c3d0 AVAILABLE If the user instead wants the hot spare to permanently assume the place of the original device, the original device can be removed with 'zpool detach'. At this point the hot spare will become a functioning device, and automatically be removed from the list of available hot spares: # zpool detach test c0d0 # zpool status ... config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 mirror ONLINE 0 0 0 c2d0 ONLINE 0 0 0 35K resilvered c1d0 ONLINE 0 0 0 spares c3d0 AVAILABLE 6. Device usage and spare health A hot spare is considered 'in use' for the purpose of libdiskmgt and zpool(1M) if it is labelled as a spare and is currently in one or more pool's list of active spares. If a spare is part of an exported pool, it is not considered in use, due largely to the fact that distinguishing this case from a recently destroyed pool is difficult and not solvable in the general case. Normally, hot spares are displayed as AVAILABLE. If the drive becomes inaccessible for some reason, or the contents overwritten such that it cannot be identified as the original spare, then it will be displayed as FAULTED. A check will be performed periodically (albeit infrequently) to verify spare health. C. AUTOMATED REPLACEMENT In order to perform automated replacement, a ZFS FMA agent will be added that subscribes to 'fault.zfs.vdev.*' faults. When a fault is received, the agent will examine the pool to see if it has any available hot spares. If so, it will perform a 'zpool replace' with an available spare. The initial algorithm for this will be 'first come, first serve', which may not be ideal for all circumstances (such as when not all spares are the same size). It is anticipated that these circumstances will be rare, and that the algorithm can be improved in the future. If a hot spare that is currently spared in fails, the agent will detach the spare (thereby cancelling the replacment), and attempt to replace the drive with another hot spare. This is currently limited by the fact that the ZFS diagnosis engine only emits faults when a device has disappeared from the system. When the DE is enhanced to proactively fault drives based on error rates, then the agent will automaticaly leverage this feature. In addition, note that there is no automated response capable of bringing the original drive back online. The user must explicitly take one of the actions described above. A future enhancement will allow ZFS to subscribe to hotplug events and automatically replace the affected drive when it is replaced on the system. D. MANPAGE DIFFS *** zpool.orig Tue Apr 4 11:29:52 2006 --- zpool.new Tue Apr 4 11:40:17 2006 *************** *** 15,24 **** --- 15,26 ---- zpool destroy [-f] pool zpool add [-fn] pool vdev ... + zpool remove pool vdev + zpool list [-H] [-o field[,field]*] [pool] ... zpool iostat [-v] [pool] ... [interval [count]] zpool status [-xv] [pool] ... *************** *** 122,133 **** --- 124,141 ---- before data integrity is compromised. The minimum number of devices in a raidz group is 2. The recommended number is between 3 and 9. + spare + A special pseudo-vdev which keeps track of available + hot spares for a pool. For more information, see the + "Hot Spares" section, below. + + SunOS 5.10 Last change: 28 Mar 2006 2 *************** *** 175,184 **** --- 183,215 ---- more devices have failed, but the data is still available due to a redundant configuration. A faulted pool has one or more failed devices, and there is insufficient redundancy to replicate the missing data. + Hot Spares + ZFS allows devices to be associated with pools as "hot + spares". These devices are not actively used in the pool, + but when an active device fails it will be automatically + replaced by a hot spare. To create a pool with hot spares, + simply specify a 'spare' vdev with any number of devices: + + # zpool create pool mirror c0d0 c1d0 \ + spare c2d0 c3d0 + + Spares can be shared across multiple pools, and can be added + with 'zpool add' and removed with 'zpool remove'. Once a + spare replacement is initiated, it will create a new 'spare' + vdev within the configuration that will remain until the + original device is replaced, at which point the hot spare + will again become available should another device fail. + + An in-progress spare replacment can be cancelled by + detaching the hot spare. If the original faulted device is + detached, then the hot spare will assume its place in the + configuration, and be removed from the spare list of all + active pools. + Alternate Root Pools The "zpool create -R" and "zpool import -R" commands allow users to create and import a pool with a different root path. By default, whenever a pool is created or imported on a system, it is permanently added so that it is available *************** *** 353,362 **** --- 384,402 ---- pool creation can still fail due to insuffi- cient privileges or device sharing. + zpool remove pool vdev + + Removes the given vdev from the pool. This command + currently only supports removing hot spares. Devices + which are part of a mirror can be removed using + 'zpool detach'. RAID-Z and top-level vdevs cannot be + removed from a pool. + + zpool list [-H] [-o field[,field*]] [pool] ... Lists the given pools along with a health status and space usage. When given no arguments, all pools in the 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack From sacadmin Wed Apr 5 07:20:54 2006 Received: from localhost.east.sun.com (punchin-sommerfeld.East.Sun.COM [129.148.19.3]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k35EKsIQ017483 for ; Wed, 5 Apr 2006 07:20:54 -0700 (PDT) Received: from localhost.east.sun.com (localhost [127.0.0.1]) by localhost.east.sun.com (8.13.5+Sun/8.13.5) with ESMTP id k35EJrN9023499; Wed, 5 Apr 2006 14:19:53 GMT Received: (from sommerfeld@localhost) by localhost.east.sun.com (8.13.5+Sun/8.13.5/Submit) id k35EJr4o023498; Wed, 5 Apr 2006 10:19:53 -0400 (EDT) X-Authentication-Warning: localhost.east.sun.com: sommerfeld set sender to sommerfeld@sun.com using -f Subject: Re: ZFS Hot Spares [PSARC/2006/223 Timeout: 04/11/2006] From: Bill Sommerfeld To: Jeff Bonwick Cc: PSARC@sac.sfbay.sun.com, Jeff.Bonwick@sun.com In-Reply-To: <200604051008.k35A8nHB224396@cathy.eng.sun.com> References: <200604051008.k35A8nHB224396@cathy.eng.sun.com> Content-Type: text/plain Content-Transfer-Encoding: 7bit Message-Id: <1144246792.23144.243.camel@localhost> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6.334 Date: Wed, 05 Apr 2006 10:19:52 -0400 Status: RO Content-Length: 418 A big picture update question: The original ZFS one-pager mentioned the concept of "hot space", where a reservation scheme was used to allow a pool to contract if a disk was lost. Several arguments were made as to why this was better than hot spares; I won't repeat them here. Does this case represent a change in direction, a stop-gap until "hot space" can be made to work, or something in between? - Bill From sacadmin Wed Apr 5 12:46:54 2006 Received: from jurassic.eng.sun.com (jurassic.SFBay.Sun.COM [129.146.106.105]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k35JksIQ029845 for ; Wed, 5 Apr 2006 12:46:54 -0700 (PDT) Received: from hawaiian-sun (vpn-129-150-12-95.SFBay.Sun.COM [129.150.12.95]) by jurassic.eng.sun.com (8.13.5+Sun/8.13.6) with SMTP id k35JkluB841093; Wed, 5 Apr 2006 12:46:53 -0700 (PDT) Message-Id: <200604051946.k35JkluB841093@jurassic.eng.sun.com> Date: Wed, 5 Apr 2006 09:46:25 -1000 (HST) From: Joseph Kowalski Reply-To: Joseph Kowalski Subject: Re: ZFS Hot Spares [PSARC/2006/223 Timeout: 04/11/2006] To: PSARC@sac.sfbay.sun.com, bonwick@cathy.eng.sun.com Cc: jeff.bonwick@sun.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: 85l3Ixyr089dhp8TOtwCIw== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_20 SunOS 5.11 i86pc i386 Status: RO Content-Length: 305 Could you drop a full copy of the proposed zpool man page into the case directory (and let us know when you have done so). There are some things I'm having a bit of trouble figuring out from the diffs. Also, it seems strange that Eric (the listed author) isn't on the distribution. - thanks, - jek3 From sacadmin Thu Apr 6 15:20:15 2006 Received: from zion.eng.sun.com (zion.SFBay.Sun.COM [129.146.17.75]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k36MKFIQ010636 for ; Thu, 6 Apr 2006 15:20:15 -0700 (PDT) Received: from zion.eng.sun.com (localhost [127.0.0.1]) by zion.eng.sun.com (8.13.3+Sun/8.13.3) with ESMTP id k36MKFfN019298; Thu, 6 Apr 2006 15:20:15 -0700 (PDT) Received: (from eschrock@localhost) by zion.eng.sun.com (8.13.3+Sun/8.13.3/Submit) id k36MKFK1019297; Thu, 6 Apr 2006 15:20:15 -0700 (PDT) Date: Thu, 6 Apr 2006 15:20:15 -0700 From: Eric Schrock To: Bill Sommerfeld Cc: PSARC@sac.sfbay.sun.com, Jeff Bonwick Subject: Re: ZFS Hot Spares [PSARC/2006/223 Timeout: 04/11/2006] Message-ID: <20060406222014.GA19110@eng.sun.com> References: <200603300627.k2U6R48N188202@cathy.eng.sun.com> <20060331192110.GE11507@eng.sun.com> <20060331210908.GG11507@eng.sun.com> <6b0a2c6933d97732756b3ca3d5634223@sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1144246792.23144.243.camel@localhost> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 1021 [Please make sure to CC myself on future mails for this case - I was inadvertently left off the original mail] There were a couple reasons for not doing hot space. One of the reasons is that while using hot space, while making use of those drives, you're actually inducing wear & tear as well. This may not actually be desirable. The main reason focuses around the difficulty of rewriting blocks in a COW filesystems with snapshots. Jeff can probably elaborate on more of the implementation specifics. - Eric > > A big picture update question: > > The original ZFS one-pager mentioned the concept of "hot space", where a > reservation scheme was used to allow a pool to contract if a disk was > lost. Several arguments were made as to why this was better than hot > spares; I won't repeat them here. > > Does this case represent a change in direction, a stop-gap until "hot > space" can be made to work, or something in between? -- Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock From sacadmin Thu Apr 6 17:31:55 2006 Received: from zion.eng.sun.com (zion.SFBay.Sun.COM [129.146.17.75]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k370VtIQ015059 for ; Thu, 6 Apr 2006 17:31:55 -0700 (PDT) Received: from zion.eng.sun.com (localhost [127.0.0.1]) by zion.eng.sun.com (8.13.3+Sun/8.13.3) with ESMTP id k370Vt1g022142; Thu, 6 Apr 2006 17:31:55 -0700 (PDT) Received: (from eschrock@localhost) by zion.eng.sun.com (8.13.3+Sun/8.13.3/Submit) id k370Vtj5022141; Thu, 6 Apr 2006 17:31:55 -0700 (PDT) Date: Thu, 6 Apr 2006 17:31:54 -0700 From: Eric Schrock To: Joseph Kowalski Cc: PSARC@sac.sfbay.sun.com, Jeff Bonwick Subject: Re: ZFS Hot Spares [PSARC/2006/223 Timeout: 04/11/2006] Message-ID: <20060407003154.GB19792@eng.sun.com> References: <200603300627.k2U6R48N188202@cathy.eng.sun.com> <20060331192110.GE11507@eng.sun.com> <20060331210908.GG11507@eng.sun.com> <6b0a2c6933d97732756b3ca3d5634223@sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200604051946.k35JkluB841093@jurassic.eng.sun.com> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 561 > Could you drop a full copy of the proposed zpool man page into the > case directory (and let us know when you have done so). There are > some things I'm having a bit of trouble figuring out from the diffs. You can find a copy of 'zpool.1m' in the materials directory. > Also, it seems strange that Eric (the listed author) isn't on the > distribution. Yes, please CC me on any future mail regarding this case. This was an oversight when the original mail was sent. - Eric -- Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock From sacadmin Sun Apr 9 15:42:30 2006 Received: from jurassic.eng.sun.com (jurassic.SFBay.Sun.COM [129.146.106.105]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k39MgUIQ026932 for ; Sun, 9 Apr 2006 15:42:30 -0700 (PDT) Received: from hawaiian-sun (vpn-129-150-12-95.SFBay.Sun.COM [129.150.12.95]) by jurassic.eng.sun.com (8.13.5+Sun/8.13.6) with SMTP id k39MgLlU712890; Sun, 9 Apr 2006 15:42:26 -0700 (PDT) Message-Id: <200604092242.k39MgLlU712890@jurassic.eng.sun.com> Date: Sun, 9 Apr 2006 12:41:50 -1000 (HST) From: Joseph Kowalski Reply-To: Joseph Kowalski Subject: Re: ZFS Hot Spares [PSARC/2006/223 Timeout: 04/11/2006] To: Joseph.Kowalski@eng.sun.com, eric.schrock@sun.com Cc: PSARC@sac.sfbay.sun.com, bonwick@cathy.eng.sun.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: uIQt7DM/a6NvVoccirm9aw== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_20 SunOS 5.11 i86pc i386 Status: RO Content-Length: 236 I'm not sure if this case was being held open due to my request or what. Anyway, I've now assured myself that what I'd like to comment on was approved many, many cases ago. Oh well. I'm done. Thanks for the full man page. - jek3 From sacadmin Wed Apr 19 09:20:15 2006 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k3JGKEIQ026062 for ; Wed, 19 Apr 2006 09:20:15 -0700 (PDT) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.149.247.22]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id k3JGK0Yv025873 for <@sunmail3.sfbay.sun.com:psarc@Sun.COM>; Thu, 20 Apr 2006 00:20:13 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0IXZ0060Z9DJHU00@nwk-avmta-2.sfbay.sun.com> for psarc@Sun.COM (ORCPT psarc@sun.com); Wed, 19 Apr 2006 09:20:07 -0700 (PDT) Received: from sfbaymail1sca.SFBay.Sun.COM ([129.145.154.35]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0IXZ002EK9DJLY60@nwk-avmta-2.sfbay.sun.com> for psarc@Sun.COM (ORCPT psarc@sun.com); Wed, 19 Apr 2006 09:20:07 -0700 (PDT) Received: from phys-mpk-2 (phys-mpk-2.SFBay.Sun.COM [129.146.11.82]) by sfbaymail1sca.SFBay.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k3JGK7oc028104 for ; Wed, 19 Apr 2006 09:20:07 -0700 (PDT) Received: from conversion-daemon.mpk-mail1.sfbay.sun.com by mpk-mail1.sfbay.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IXZ0090190PZJ@mpk-mail1.sfbay.sun.com> (original mail from Sherri.Shieh@Sun.COM) for psarc@sun.com; Wed, 19 Apr 2006 09:20:06 -0700 (PDT) Received: from [129.146.11.202] (sr1-umpk-18.SFBay.Sun.COM [129.146.11.202]) by mpk-mail1.sfbay.sun.com (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with ESMTP id <0IXZ00MQU9DIZF@mpk-mail1.sfbay.sun.com>; Wed, 19 Apr 2006 09:20:06 -0700 (PDT) Date: Wed, 19 Apr 2006 09:20:06 -0700 From: Sherri Shieh Subject: PSARC Fast track: ZFS Hot Spares (2006/223) To: psarc@sun.com, Jeff Bonwick , Eric Schrock Message-id: <44466336.70405@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii; format=flowed Content-transfer-encoding: 7BIT X-Accept-Language: en-us, en X-PMX-Version: 5.1.2.240295 User-Agent: Mozilla/5.0 (X11; U; SunOS sun4u; en-US; rv:1.7) Gecko/20050530 Status: RO Content-Length: 337 This case was approved last week. I have marked it closed approved. - Sherri -- ========================================================= Sherri Shieh Sun Microsystems, Inc. Program Manager Email: sherri.shieh@sun.com Systems Architecture Phone: 650-786-5245/x85245 ===========================================================