From sacadmin Wed Jul 20 17:10:10 2005 Received: from thunk.east.sun.com (thunk.East.Sun.COM [129.148.174.66]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j6L0A9Eu007514 for ; Wed, 20 Jul 2005 17:10:10 -0700 (PDT) Received: from 127.0.0.1 (localhost [127.0.0.1]) by thunk.east.sun.com (8.13.4+Sun/8.13.4) with ESMTP id j6L0A8ND101310; Wed, 20 Jul 2005 20:10:09 -0400 (EDT) Subject: Re: SVM default interlace and resync buffer increase [PSARC/2005/427 Timeout: 07/27/2005] From: Bill Sommerfeld To: Brian Utterback Cc: PSARC@sac.sfbay.sun.com, Truong.Q.Nguyen@sun.com In-Reply-To: <200507201915.j6KJFmZg033453@sr1-unsh01-03.East.Sun.COM> References: <200507201915.j6KJFmZg033453@sr1-unsh01-03.East.Sun.COM> Content-Type: text/plain Message-Id: <1121904608.100966.145.camel@thunk> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6.316 Date: Wed, 20 Jul 2005 20:10:08 -0400 Content-Transfer-Encoding: 7bit Status: RO Content-Length: 1228 On Wed, 2005-07-20 at 15:15, Brian Utterback wrote: > Summary: This proposal requests a change to the default interlace value for SVM > stripes and RAID5 metadevices and the mirror resync buffer size. The > new default interlace and resync buffer value of 512KB would result in > 15 to 50+ percent increase in throughput for I/O greater 16KB and half > the resynchronization time for mirror initialization and component > failure/recovery. On a v40z I set up recently, I noticed substantially *reduced* svm mirror initialization performance when I set the resynch buffer size to the maximum value, compared with the small default in nevada as of when I installed it. (the documentation seemed to imply performance could only increase with a larger buffer size....) I haven't had time to dig into this and the box is now in production. how much measurement of speed vs. resynch buffer size have you done across different hardware? maybe this is overengineering but can we somehow make this self-tuning? mirror resynchs typically take at least a few minutes; seems like you could do a slow-start-like trick (spend a couple seconds copying at each power-of-two-size and then back off if it slows down appreciably?) From sacadmin Thu Jul 21 13:59:17 2005 Received: from phys-giza-1 (phys-giza-1.Central.Sun.COM [129.147.4.102]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j6LKxGEu021123 for ; Thu, 21 Jul 2005 13:59:16 -0700 (PDT) Received: from conversion-daemon.giza-mail1.Central.Sun.COM by giza-mail1.Central.Sun.COM (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) id <0IJZ00101WLU0B@giza-mail1.Central.Sun.COM> (original mail from Truong.Q.Nguyen@Sun.COM) for PSARC@sac.sfbay.sun.com; Thu, 21 Jul 2005 14:59:16 -0600 (MDT) Received: from [172.20.25.177] (caolila.Central.Sun.COM [172.20.25.177]) by giza-mail1.Central.Sun.COM (iPlanet Messaging Server 5.2 HotFix 1.24 (built Dec 19 2003)) with ESMTP id <0IJZ009A8WYOI1@giza-mail1.Central.Sun.COM>; Thu, 21 Jul 2005 14:59:12 -0600 (MDT) Date: Thu, 21 Jul 2005 14:47:04 -0600 From: Tony Nguyen Subject: Re: SVM default interlace and resync buffer increase [PSARC/2005/427 Timeout: 07/27/2005] In-reply-to: <1121904608.100966.145.camel@thunk> To: Bill Sommerfeld Cc: Brian Utterback , PSARC@sac.sfbay.sun.com, Sanjay Nadkarni Reply-to: Truong.Q.Nguyen@Sun.COM Message-id: <42E009C8.7030808@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii; format=flowed Content-transfer-encoding: 7BIT X-Accept-Language: en-us, en User-Agent: Mozilla/5.0 (X11; U; SunOS sun4u; en-US; rv:1.7) Gecko/20041221 References: <200507201915.j6KJFmZg033453@sr1-unsh01-03.East.Sun.COM> <1121904608.100966.145.camel@thunk> Status: RO Content-Length: 2076 Bill, See comments inline. Bill Sommerfeld wrote: > On Wed, 2005-07-20 at 15:15, Brian Utterback wrote: > >>Summary: This proposal requests a change to the default interlace value for SVM >> stripes and RAID5 metadevices and the mirror resync buffer size. The >> new default interlace and resync buffer value of 512KB would result in >> 15 to 50+ percent increase in throughput for I/O greater 16KB and half >> the resynchronization time for mirror initialization and component >> failure/recovery. > > > On a v40z I set up recently, I noticed substantially *reduced* svm > mirror initialization performance when I set the resynch buffer size to > the maximum value, compared with the small default in nevada as of when > I installed it. > (the documentation seemed to imply performance could only increase with > a larger buffer size....) > > I haven't had time to dig into this and the box is now in production. This sounds strange. Can you get me both SVM config (metastat output) and disks related information? In the mean time, I'll see if I can get a v40z and try to reproduce this. > > how much measurement of speed vs. resynch buffer size have you done > across different hardware? I hadn't done testing with across many different hardware. However, the correlation between the reduced resync time and increased resync buffer size on my test machine, an x86 with unrecalled hardware specifics, were quite consistent. > maybe this is overengineering but can we somehow make this self-tuning? > mirror resynchs typically take at least a few minutes; seems like you > could do a slow-start-like trick (spend a couple seconds copying at each > power-of-two-size and then back off if it slows down appreciably?) > I agree and this is what our team has been talking about. The other possible enhancement is to utilize the drive buffer to enable larger resync regions and IO. However, we just don't have the resource the staff such projects now. This fasttrack is intended to be a solution that will give us the best bang for the buck. Thanks, tony From sacadmin Mon Jul 25 15:39:05 2005 Received: from eastmail1bur.East.Sun.COM (eastmail1bur.East.Sun.COM [129.148.9.49]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j6PMd5Eu019441 for ; Mon, 25 Jul 2005 15:39:05 -0700 (PDT) Received: from thunk.east.sun.com (thunk.East.Sun.COM [129.148.174.66]) by eastmail1bur.East.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id j6PMd4Ft011158; Mon, 25 Jul 2005 18:39:04 -0400 (EDT) Received: from 127.0.0.1 (localhost [127.0.0.1]) by thunk.east.sun.com (8.13.4+Sun/8.13.4) with ESMTP id j6PMd3e4005913; Mon, 25 Jul 2005 18:39:03 -0400 (EDT) Subject: Re: SVM default interlace and resync buffer increase [PSARC/2005/427 Timeout: 07/27/2005] From: Bill Sommerfeld To: Truong.Q.Nguyen@sun.com Cc: Brian Utterback , PSARC@sac.sfbay.sun.com, Sanjay Nadkarni In-Reply-To: <42E009C8.7030808@Sun.COM> References: <200507201915.j6KJFmZg033453@sr1-unsh01-03.East.Sun.COM> <1121904608.100966.145.camel@thunk> <42E009C8.7030808@Sun.COM> Content-Type: text/plain Message-Id: <1122331143.2863.667.camel@thunk> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6.319 Date: Mon, 25 Jul 2005 18:39:03 -0400 Content-Transfer-Encoding: 7bit Status: RO Content-Length: 1066 On Thu, 2005-07-21 at 16:47, Tony Nguyen wrote: > This sounds strange. Can you get me both SVM config (metastat output) > and disks related information? In the mean time, I'll see if I can get a > v40z and try to reproduce this. Tony has gotten back to me with test results. Turns out the performance regression I noticed which prompted my comment only happens at a buffer size which is larger than the proposed new default. I've asked him to file a man page bug asking for a bit less certainty in the following section of metasync(1m): buffer_size Specifies the size (number of 512-byte disk blocks) of the internal copy buffer for the mirror resync. The size defaults to 128 512-byte disk blocks (64 Kbytes). It can be no more than 2048 blocks. For best perfor- mance (quickest completion of the resync), 2048 blocks is the recommended size. (that paragraph will have to be edited to report the new default anyway). I have no problem with the case as proposed. - Bill From sacadmin Fri Jul 29 05:15:34 2005 Received: from eastmail1bur.East.Sun.COM (eastmail1bur.East.Sun.COM [129.148.9.49]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j6TCFYEu029778 for ; Fri, 29 Jul 2005 05:15:34 -0700 (PDT) Received: from [129.148.226.13] (sr1-unsh01-03.East.Sun.COM [129.148.226.13]) by eastmail1bur.East.Sun.COM (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id j6TCFXFt012430; Fri, 29 Jul 2005 08:15:33 -0400 (EDT) Message-ID: <42EA1DE4.5020801@sun.com> Date: Fri, 29 Jul 2005 08:15:32 -0400 From: Brian Utterback User-Agent: Mozilla Thunderbird 1.0.6 (X11/20050715) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Brian Utterback CC: PSARC@sac.sfbay.sun.com, Truong.Q.Nguyen@sun.com Subject: Re: SVM default interlace and resync buffer increase [PSARC/2005/427 Timeout: 07/27/2005] References: <200507201915.j6KJFmZg033453@sr1-unsh01-03.East.Sun.COM> In-Reply-To: <200507201915.j6KJFmZg033453@sr1-unsh01-03.East.Sun.COM> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 365 The timer has expired on this case. The discussion closed, so I am marking it "closed approved fasttrack". -- blu Remember when SOX compliant meant they were both the same color? ---------------------------------------------------------------------- Brian Utterback - OP/N1 RPE, Sun Microsystems, Inc. Ph:877-259-7345, Em:brian.utterback-at-ess-you-enn-dot-kom