From sherrym@too-cool.local Wed Jun  3 16:18:31 2009
Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165])
	by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n53NIU4U019464
	for <psarc-ext@sac.sfbay.sun.com>; Wed, 3 Jun 2009 16:18:31 -0700 (PDT)
Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74])
	by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n53NILXH014351
	for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Jun 2009 00:18:29 +0100 (BST)
Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by
 nwk-avmta-1.sfbay.Sun.COM
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 id <0KKO00M1BRET4Y00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com
 (ORCPT PSARC-ext@sun.com); Wed, 03 Jun 2009 16:18:29 -0700 (PDT)
Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31])
 by nwk-avmta-1.sfbay.Sun.COM
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 with ESMTP id <0KKO003HHRESUU40@nwk-avmta-1.sfbay.Sun.COM> for
 PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed,
 03 Jun 2009 16:18:28 -0700 (PDT)
Received: from too-cool.local (too-cool.SFBay.Sun.COM [129.146.226.229])
	by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2)
 with ESMTP id n53NISS1032973; Wed, 03 Jun 2009 16:18:28 -0700 (PDT)
Received: from too-cool.local (too-cool [127.0.0.1])
	by too-cool.local (8.14.3+Sun/8.14.3) with ESMTP id n53NF6au017831; Wed,
 03 Jun 2009 16:15:06 -0700 (PDT)
Received: (from sherrym@localhost)	by too-cool.local (8.14.3+Sun/8.14.3/Submit)
 id n53NF6gd017828; Wed, 03 Jun 2009 16:15:06 -0700 (PDT)
Date: Wed, 03 Jun 2009 16:15:06 -0700 (PDT)
From: Sherry Moore <sherrym@too-cool.local>
Subject: LatencyTOP for OpenSolaris [PSARC/2009/339 FastTrack timeout
 06/10/2009]
To: PSARC-ext@sun.com
Cc: Krishnendu.Sadhukhan@sun.com
Message-id: <200906032315.n53NF6gd017828@too-cool.local>
Content-transfer-encoding: 7BIT
X-PMX-Version: 5.4.1.325704
Status: RO
Content-Length: 10305


Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI
This information is Copyright 2009 Sun Microsystems
1. Introduction
    1.1. Project/Component Working Name:
	 LatencyTOP for OpenSolaris
    1.2. Name of Document Author/Supplier:
	 Author:  Krishnendu Sadhukhan
    1.3  Date of This Document:
	03 June, 2009
4. Technical Description
1. Introduction
   1.1. Project/Component Working Name:
	LatencyTOP for OpenSolaris

   1.2. Name of Document Author/Supplier:
	krishnendu.sadhukhan@sun.com

   1.3. Date of This Document:
	05/20/09
	
	1.3.1. Date this project was conceived:
		05/20/09

   1.4. Name of Major Document Customer(s)/Consumer(s):
	1.4.1. The PAC or CPT you expect to review your project:
		Solaris PAC
	1.4.2. The ARC(s) you expect to review your project:
		PSARC
	1.4.3. The Director/VP who is "Sponsoring" this project:
		greg.lavendar@sun.com
	1.4.4. The name of your business unit:
		Software

   1.5. Email Aliases:
    	1.5.1. Responsible Manager:	darrin.johnson@sun.com
    	1.5.2. Responsible Engineer:	krishnendu.sadhukhan@sun.com
    	1.5.3. Marketing Manager:	mike.mulkey@sun.com
	1.5.4. Interest List:		latencytop-dev@opensolaris.org

2. Project Summary
   2.1. Project Description:
	LatencyTOP is an observability tool that can be used to identify
	latencies in application and system software. Latency occurs
	when a process/thread can not run and goes to sleep due to
	unavailability of some resource. Developers can use this tool
	to identify latency holes in their applications; system
	programmers can equally use this tool to identify latencies in
	system processes.
	This tool was initially developed for Linux. Intel and Sun are
	jointly collaborating on developing this tool for OpenSolaris.
	The first version of the tool is already available for use on
	OpenSolaris website.

   2.2. Risks and Assumptions:
	LatencyTOP uses probes from the proc, sched and lockstat DTrace
	provider. So, the OpenSolaris version in which LatencyTOP is run
	must have these probes available.

3. Business Summary
   3.1. Problem Area:
	Latency affects performance of applications and systems. This tool
	can be used to locate latencies, which when removed improves
	performance of applications and systems.

   3.2. Market/Requester:

   3.3. Business Justification:
	LatencyTOP is being jointly developed by Sun and Intel through
	the OpenSolaris community.

   3.4. Competitive Analysis:
	LatencyTOP was originally developed for Linux. Making this tool
	available in OpenSolaris will grow the user base of OpenSolaris
	and will provide OpenSolaris with competitive advantage over
	Linux.

   3.5. Opportunity Window/Exposure:
	The first version of the tool is already available on OpenSolaris
	for download and use.

   3.6. How will you know when you are done?:
	Verson 1.0 of the tool is already complete.

4. Technical Description:
    4.1. Details:
	The original LatencyTOP project is hosted on

		http://www.latencytop.org

	The OpenSolaris port of this project is hosted on

		http://opensolaris.org/os/project/latencytop

    4.2. Bug/RFE Number(s):
	6825817 Integrate latencyTOP into OpenSolaris
	6847419 man page for LatencyTOP
    
    4.3. In Scope:
	LatencyTOP traces two types of latencies :
		i) an LWP goes to sleep state because it is waiting for
	some resource to be available for it to run again
		ii) an LWP spinning in order to acquire a synchronization
	object

    4.4. Out of Scope:
	LatencyTOP is only an observability tool; it does not allow the
	user to accomplish anything other than getting latency statistics.

	LatencyTOP does not detect busy loop inside user application.
	Neither does it detect delay that is not caused by waiting,
	e.g. process not running because a higher priority process takes a
	lot of CPU time.
    
    4.5. Interfaces:

	Minor binding only.

	INTERFACES				COMMITTMENT LEVEL
	==========				=================
	/usr/bin/i86/latencytop			committed
	/usr/bin/amd64/latencytop		committed
	/bin/i86/latencytop			committed
	/bin/amd64/latencytop			committed
	/usr/bin/latencytop (hard link)		committed
	/bin/latencytop (hard link)		committed

	LatencyTOP has a UI based on libcurses.

    4.6. Doc Impact:
	A new man(1M) page will be required (see Appendix). 
	The OpenSolaris system administration needs to include this tool.

    4.7. Admin/Config Impact:
	None
 
    4.8. HA Impact:
	None

    4.9. I18N/L10N Impact:
	None
 
    4.10. Packaging & Delivery:
	A new package called SUNWlatencytop will be introduced.

    4.11. Security Impact:
	The user must have DTrace privilege to run LatencyTOP.
 
    4.12. Dependencies:
	LatencyTOP uses the Solaris DTrace APIs, specifically the following
	DTrace providers: sched, proc and lockstat.

5. Reference Documents:
	http://monaco.sfbay/detail.jsf?cr=6825817
	http://opensolaris.org/os/project/latencytop

6. Resources and Schedule:
   6.1. Projected Availability:
	LatencyTOP is currently available through OpenSolaris.

   6.2. Cost of Effort:
	This project is being jointly developed by Sun and Intel through
	OpenSolaris. One Sun engineer is required to sponsor the effort
	and facilitate the integration into OpenSolaris.

   6.3. Cost of Capital Resources:
	None

   6.4. Product Approval Committee requested information:
   	6.4.1. Consolidation or Component Name:	sfw
	6.4.3. Type of CPT Review and Approval expected:
		   FastTrack

        6.4.4. Project Boundary Conditions:	None
	6.4.5. Is this a necessary project for OEM agreements:	No
	6.4.6. Notes:	N/A
	6.4.7. Target RTI Date/Release: onnv_120
	6.4.8. Target Code Design Review Date:	06/20/2009
	6.4.9. Update approval addition:	No

   6.5. ARC review type:
		   FastTrack
   6.6. ARC Exposure:
		   open
       6.6.1. Rationale: N/A

7. Prototype Availability:
   7.1. Prototype Availability:
	Version 1.0 of LatencyTOP is already available on OpenSolaris.

   7.2. Prototype Cost:
	N/A

8. Appendix

System Administration Commands                       latencytop(1M)

NAME
     latencytop - report statistics related to latencies in the
     system and in applications

SYNOPSIS
     latencytop [-o log file] [-k log-level] [-t interval] [-f]
		[-s] [-l log interval] [-h]

DESCRIPTION
     LatencyTOP is an observability tool that reports statistics
     about latencies in the system and in applications. The tool
     reports  statistics about where and what  kind of latencies
     are  happening in the  system and in the  applications that
     are running on the system. The  statistics then can be used
     to  improve  performance  throughput  of  applications  and
     system by removing the latencies.

     The tool analyzes system activity periodically and displays
     the data in the output window. Two  types of  latencies are
     tracked - an  LWP  going in and out  of  sleep and  an  LWP
     spinning order to acquire a synchronization object. The tool
     uses the Solaris DTrace framework to collect the statistics
     corresponding to  these two  scenarios of inactivity of the
     system and application LWPs.

     The output window is divided into two sections - upper part
     displays the system-wide  statistics  while  the lower part
     displays  statistics  about  individual processes. The user
     can navigate  the list of  processes (using the < and the >
     keys) and  select the one they are  interested in,  and the
     tool will  display  statistics about that selected  process
     in the lower part of the window; if the t or T key is 
     pressed, the tool  displays  the LWP-specific view of that
     selected process. Thus, the  t or T key can be used to 
     toggle  between the  process-view and the thread-view.

     During execution, a user can force a refresh of the analysis
     by pressing the r or R key.  The interval  time is restored
     to the default or to a specified value (if -t was used). To
     quit  the  application, the user must press the q or Q key.


OPTIONS
     The following options are supported:

     -o [log file]

         Specifies the log file where output will be written. The
	 default log file is /var/log/latencytop.log.

     -k [log level]

         Specifies  the level of  logging  in the log file. Valid
         values are: 0 = none (default), 1 = unknown, and 2 = all;

     -t [interval]

         Specifies the interval, in seconds, at  which  the  tool
         collects statistics from the system. The possible values
         are between 1 and 60; the default is 5 seconds.

     -f

	 Filter large interruptible latencies (e.g. sleep).

     -s

         Monitors the sched (PID=0) process for any latency.

     -l [log-interval]

         Writes data to the log file every log-interval seconds;
	 log-interval must be > 60.

     -h

         Displays the command's usage.


EXAMPLES
     Example 1 Running the tool

     The following command launches the tool with default values
     for options.

       % latencytop

     Example 2 Setting the Interval

     The following command sets the interval to two seconds.

       % latencytop -t 2

     Example 3 Setting the log file

     The following command sets the log file to /tmp/latencytop.log.

       % latencytop -o /tmp/latencytop.log

     Example 4 Setting the log level

     The following command sets the log level to "all".

       % latencytop -l 2


EXIT STATUS
     0

         Successful operation.

     1

         An error occurred.


ATTRIBUTES
     See attributes(5) for descriptions of the  following  attri-
     butes:

     ____________________________________________________________
    |       ATTRIBUTE TYPE        |       ATTRIBUTE VALUE       |
    |_____________________________|_____________________________|
    | Architecture                | x86, SPARC                  |
    |_____________________________|_____________________________|
    | Availability                | SUNWlatencytop              |
    |_____________________________|_____________________________|
    | Interface Stability         | Volatile                    |
    |_____________________________|_____________________________|

SEE ALSO
     kstat(1M), dtrace(1M)

     Among  non-SunOS  man  pages,  xscreensaver(1),   from   the
     OpenWindows man pages.

USAGE
     You must have DTrace privileges to run LatencyTOP. 

From sherry.moore@sun.com Wed Jun  3 16:56:32 2009
Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165])
	by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n53NuW28020233
	for <psarc-ext@sac.sfbay.sun.com>; Wed, 3 Jun 2009 16:56:32 -0700 (PDT)
Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74])
	by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n53NuOip002823
	for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Jun 2009 00:56:31 +0100 (BST)
Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by
 nwk-avmta-1.sfbay.Sun.COM
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 id <0KKO0060BT66ZS00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com
 (ORCPT PSARC-ext@sun.com); Wed, 03 Jun 2009 16:56:30 -0700 (PDT)
Received: from dm-sfbay-02.sfbay.sun.com ([129.146.11.31])
 by nwk-avmta-1.sfbay.Sun.COM
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 with ESMTP id <0KKO003M8T65UQ60@nwk-avmta-1.sfbay.Sun.COM> for
 PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed,
 03 Jun 2009 16:56:29 -0700 (PDT)
Received: from too-cool.local (too-cool.SFBay.Sun.COM [129.146.226.229])
	by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2)
 with ESMTP id n53NuSol051524; Wed, 03 Jun 2009 16:56:28 -0700 (PDT)
Received: from too-cool.local (too-cool [127.0.0.1])
	by too-cool.local (8.14.3+Sun/8.14.3) with ESMTP id n53Nr7Yk017968; Wed,
 03 Jun 2009 16:53:07 -0700 (PDT)
Received: (from sherrym@localhost)	by too-cool.local (8.14.3+Sun/8.14.3/Submit)
 id n53Nr7us017967; Wed, 03 Jun 2009 16:53:07 -0700 (PDT)
Date: Wed, 03 Jun 2009 16:53:07 -0700
From: Sherry Moore <sherry.moore@sun.com>
Subject: LatencyTOP for OpenSolaris [PSARC/2009/339 FastTrack timeout
	06/10/2009]
To: PSARC-ext@sun.com
Cc: Krishnendu.Sadhukhan@sun.com
Message-id: <20090603235307.GC1089@sun.com>
MIME-version: 1.0
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Content-disposition: inline
X-PMX-Version: 5.4.1.325704
X-Authentication-warning: too-cool.local: sherrym set sender to
 sherry.moore@sun.com using -f
User-Agent: Mutt/1.5.19 (2009-01-05)
Status: RO
Content-Length: 10197

I am sponsoring this case for Krishnendu Sadhukhan.  Minor binding only.
Timer expires on June 10th, 2009.

Thanks,
Sherry

Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI
This information is Copyright 2009 Sun Microsystems
1. Introduction
   1.1. Project/Component Working Name:
	 LatencyTOP for OpenSolaris
   1.2. Name of Document Author/Supplier:
	 Author:  Krishnendu Sadhukhan

   1.3. Date of This Document:
	05/20/09
	
	1.3.1. Date this project was conceived:
		05/20/09

   1.4. Name of Major Document Customer(s)/Consumer(s):
	1.4.1. The PAC or CPT you expect to review your project:
		Solaris PAC
	1.4.2. The ARC(s) you expect to review your project:
		PSARC
	1.4.3. The Director/VP who is "Sponsoring" this project:
		greg.lavendar@sun.com
	1.4.4. The name of your business unit:
		Software

   1.5. Email Aliases:
    	1.5.1. Responsible Manager:	darrin.johnson@sun.com
    	1.5.2. Responsible Engineer:	krishnendu.sadhukhan@sun.com
    	1.5.3. Marketing Manager:	mike.mulkey@sun.com
	1.5.4. Interest List:		latencytop-dev@opensolaris.org

2. Project Summary
   2.1. Project Description:
	LatencyTOP is an observability tool that can be used to identify
	latencies in application and system software. Latency occurs
	when a process/thread can not run and goes to sleep due to
	unavailability of some resource. Developers can use this tool
	to identify latency holes in their applications; system
	programmers can equally use this tool to identify latencies in
	system processes.
	This tool was initially developed for Linux. Intel and Sun are
	jointly collaborating on developing this tool for OpenSolaris.
	The first version of the tool is already available for use on
	OpenSolaris website.

   2.2. Risks and Assumptions:
	LatencyTOP uses probes from the proc, sched and lockstat DTrace
	provider. So, the OpenSolaris version in which LatencyTOP is run
	must have these probes available.

3. Business Summary
   3.1. Problem Area:
	Latency affects performance of applications and systems. This tool
	can be used to locate latencies, which when removed improves
	performance of applications and systems.

   3.2. Market/Requester:

   3.3. Business Justification:
	LatencyTOP is being jointly developed by Sun and Intel through
	the OpenSolaris community.

   3.4. Competitive Analysis:
	LatencyTOP was originally developed for Linux. Making this tool
	available in OpenSolaris will grow the user base of OpenSolaris
	and will provide OpenSolaris with competitive advantage over
	Linux.

   3.5. Opportunity Window/Exposure:
	The first version of the tool is already available on OpenSolaris
	for download and use.

   3.6. How will you know when you are done?:
	Verson 1.0 of the tool is already complete.

4. Technical Description:
    4.1. Details:
	The original LatencyTOP project is hosted on

		http://www.latencytop.org

	The OpenSolaris port of this project is hosted on

		http://opensolaris.org/os/project/latencytop

    4.2. Bug/RFE Number(s):
	6825817 Integrate latencyTOP into OpenSolaris
	6847419 man page for LatencyTOP
    
    4.3. In Scope:
	LatencyTOP traces two types of latencies :
		i) an LWP goes to sleep state because it is waiting for
	some resource to be available for it to run again
		ii) an LWP spinning in order to acquire a synchronization
	object

    4.4. Out of Scope:
	LatencyTOP is only an observability tool; it does not allow the
	user to accomplish anything other than getting latency statistics.

	LatencyTOP does not detect busy loop inside user application.
	Neither does it detect delay that is not caused by waiting,
	e.g. process not running because a higher priority process takes a
	lot of CPU time.
    
    4.5. Interfaces:

	Minor binding only.

	INTERFACES				COMMITTMENT LEVEL
	==========				=================
	/usr/bin/i86/latencytop			committed
	/usr/bin/amd64/latencytop		committed
	/bin/i86/latencytop			committed
	/bin/amd64/latencytop			committed
	/usr/bin/latencytop (hard link)		committed
	/bin/latencytop (hard link)		committed

	LatencyTOP has a UI based on libcurses.

    4.6. Doc Impact:
	A new man(1M) page will be required (see Appendix). 
	The OpenSolaris system administration needs to include this tool.

    4.7. Admin/Config Impact:
	None
 
    4.8. HA Impact:
	None

    4.9. I18N/L10N Impact:
	None
 
    4.10. Packaging & Delivery:
	A new package called SUNWlatencytop will be introduced.

    4.11. Security Impact:
	The user must have DTrace privilege to run LatencyTOP.
 
    4.12. Dependencies:
	LatencyTOP uses the Solaris DTrace APIs, specifically the following
	DTrace providers: sched, proc and lockstat.

5. Reference Documents:
	http://monaco.sfbay/detail.jsf?cr=6825817
	http://opensolaris.org/os/project/latencytop

6. Resources and Schedule:
   6.1. Projected Availability:
	LatencyTOP is currently available through OpenSolaris.

   6.2. Cost of Effort:
	This project is being jointly developed by Sun and Intel through
	OpenSolaris. One Sun engineer is required to sponsor the effort
	and facilitate the integration into OpenSolaris.

   6.3. Cost of Capital Resources:
	None

   6.4. Product Approval Committee requested information:
   	6.4.1. Consolidation or Component Name:	sfw
	6.4.3. Type of CPT Review and Approval expected:
		   FastTrack

        6.4.4. Project Boundary Conditions:	None
	6.4.5. Is this a necessary project for OEM agreements:	No
	6.4.6. Notes:	N/A
	6.4.7. Target RTI Date/Release: onnv_120
	6.4.8. Target Code Design Review Date:	06/20/2009
	6.4.9. Update approval addition:	No

   6.5. ARC review type:
		   FastTrack
   6.6. ARC Exposure:
		   open
       6.6.1. Rationale: N/A

7. Prototype Availability:
   7.1. Prototype Availability:
	Version 1.0 of LatencyTOP is already available on OpenSolaris.

   7.2. Prototype Cost:
	N/A

8. Appendix

System Administration Commands                       latencytop(1M)

NAME
     latencytop - report statistics related to latencies in the
     system and in applications

SYNOPSIS
     latencytop [-o log file] [-k log-level] [-t interval] [-f]
		[-s] [-l log interval] [-h]

DESCRIPTION
     LatencyTOP is an observability tool that reports statistics
     about latencies in the system and in applications. The tool
     reports  statistics about where and what  kind of latencies
     are  happening in the  system and in the  applications that
     are running on the system. The  statistics then can be used
     to  improve  performance  throughput  of  applications  and
     system by removing the latencies.

     The tool analyzes system activity periodically and displays
     the data in the output window. Two  types of  latencies are
     tracked - an  LWP  going in and out  of  sleep and  an  LWP
     spinning order to acquire a synchronization object. The tool
     uses the Solaris DTrace framework to collect the statistics
     corresponding to  these two  scenarios of inactivity of the
     system and application LWPs.

     The output window is divided into two sections - upper part
     displays the system-wide  statistics  while  the lower part
     displays  statistics  about  individual processes. The user
     can navigate  the list of  processes (using the < and the >
     keys) and  select the one they are  interested in,  and the
     tool will  display  statistics about that selected  process
     in the lower part of the window; if the t or T key is 
     pressed, the tool  displays  the LWP-specific view of that
     selected process. Thus, the  t or T key can be used to 
     toggle  between the  process-view and the thread-view.

     During execution, a user can force a refresh of the analysis
     by pressing the r or R key.  The interval  time is restored
     to the default or to a specified value (if -t was used). To
     quit  the  application, the user must press the q or Q key.


OPTIONS
     The following options are supported:

     -o [log file]

         Specifies the log file where output will be written. The
	 default log file is /var/log/latencytop.log.

     -k [log level]

         Specifies  the level of  logging  in the log file. Valid
         values are: 0 = none (default), 1 = unknown, and 2 = all;

     -t [interval]

         Specifies the interval, in seconds, at  which  the  tool
         collects statistics from the system. The possible values
         are between 1 and 60; the default is 5 seconds.

     -f

	 Filter large interruptible latencies (e.g. sleep).

     -s

         Monitors the sched (PID=0) process for any latency.

     -l [log-interval]

         Writes data to the log file every log-interval seconds;
	 log-interval must be > 60.

     -h

         Displays the command's usage.


EXAMPLES
     Example 1 Running the tool

     The following command launches the tool with default values
     for options.

       % latencytop

     Example 2 Setting the Interval

     The following command sets the interval to two seconds.

       % latencytop -t 2

     Example 3 Setting the log file

     The following command sets the log file to /tmp/latencytop.log.

       % latencytop -o /tmp/latencytop.log

     Example 4 Setting the log level

     The following command sets the log level to "all".

       % latencytop -l 2


EXIT STATUS
     0

         Successful operation.

     1

         An error occurred.


ATTRIBUTES
     See attributes(5) for descriptions of the  following  attri-
     butes:

     ____________________________________________________________
    |       ATTRIBUTE TYPE        |       ATTRIBUTE VALUE       |
    |_____________________________|_____________________________|
    | Architecture                | x86, SPARC                  |
    |_____________________________|_____________________________|
    | Availability                | SUNWlatencytop              |
    |_____________________________|_____________________________|
    | Interface Stability         | Volatile                    |
    |_____________________________|_____________________________|

SEE ALSO
     kstat(1M), dtrace(1M)

     Among  non-SunOS  man  pages,  xscreensaver(1),   from   the
     OpenWindows man pages.

USAGE
     You must have DTrace privileges to run LatencyTOP. 

From ro@techfak.uni-bielefeld.de Thu Jun  4 03:08:36 2009
Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245])
	by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n54A8XgG026281
	for <psarc-ext@sac.sfbay.sun.com>; Thu, 4 Jun 2009 03:08:36 -0700 (PDT)
Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74])
	by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n54A8UxX054894;
	Thu, 4 Jun 2009 04:08:32 -0600 (MDT)
Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by
 nwk-avmta-1.sfbay.Sun.COM
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 id <0KKP00D0LLI6QR00@nwk-avmta-1.sfbay.Sun.COM>; Thu,
 04 Jun 2009 03:08:30 -0700 (PDT)
Received: from sca-ea-mail-1.sun.com ([192.18.43.24])
 by nwk-avmta-1.sfbay.Sun.COM
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 with ESMTP id <0KKP00BCXLI4P3F0@nwk-avmta-1.sfbay.Sun.COM>; Thu,
 04 Jun 2009 03:08:29 -0700 (PDT)
Received: from relay44i.sun.com ([192.5.209.118])
	by sca-ea-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n54A8PuG025828;
 Thu, 04 Jun 2009 10:08:28 +0000 (GMT)
Received: from mms48es.mms.us.syntegra.com ([160.41.221.230] [160.41.221.230])
 by relay44i.sun.com with ESMTP id BT-MMP-235956; Thu,
 04 Jun 2009 10:08:25 +0000 (Z)
Received: from relay42i.sun.com (relay42i.sun.com [192.5.209.72])
 by mms48es.mms.us.syntegra.com with ESMTP id BT-MMP-18287426; Thu,
 04 Jun 2009 10:08:25 +0000 (Z)
Received: from smarthost.TechFak.Uni-Bielefeld.DE
 ([129.70.137.17] [129.70.137.17]) by relay4i.sun.com with ESMTP id
 BT-MMP-22551122; Thu, 04 Jun 2009 10:08:25 +0000 (Z)
Received: from manam.TechFak.Uni-Bielefeld.DE
 (manam.TechFak.Uni-Bielefeld.DE [129.70.137.47])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)	by smarthost.TechFak.Uni-Bielefeld.DE
 (Postfix) with ESMTP id BE87A482DF; Thu, 04 Jun 2009 12:08:24 +0200 (CEST)
Date: Thu, 04 Jun 2009 12:08:20 +0200
From: Rainer Orth <ro@techfak.uni-bielefeld.de>
Subject: Re: LatencyTOP for OpenSolaris [PSARC/2009/339 FastTrack timeout
	06/10/2009]
In-reply-to: Sherry Moore's message of "Wed, 03 Jun 2009 16:53:07 -0700"
Sender: ro@techfak.uni-bielefeld.de
To: Sherry Moore <sherry.moore@sun.com>
Cc: PSARC-ext@sun.com, Krishnendu.Sadhukhan@sun.com
Message-id: <yddprdk5n1n.fsf@manam.TechFak.Uni-Bielefeld.DE>
MIME-version: 1.0
X-Mailer: Gnus v5.6.44/Emacs 19.34
Content-type: text/plain; charset=US-ASCII
Content-transfer-encoding: 7BIT
X-PMX-Version: 5.4.1.325704
X-Brightmail-Tracker: AAAAAA==
X-Antispam: No, score=0.0/5.0, scanned in 0.121sec at (localhost [127.0.0.1])
	by smf-spamd v1.3.1 - http://smfs.sf.net/
Lines: 44
References: <20090603235307.GC1089@sun.com>
Status: RO
Content-Length: 1550

Sherry Moore <sherry.moore@sun.com> writes:

>    2.2. Risks and Assumptions:
> 	LatencyTOP uses probes from the proc, sched and lockstat DTrace
> 	provider. So, the OpenSolaris version in which LatencyTOP is run
> 	must have these probes available.
[...]
>     4.5. Interfaces:
> 
> 	Minor binding only.
> 
> 	INTERFACES				COMMITTMENT LEVEL
> 	==========				=================
> 	/usr/bin/i86/latencytop			committed
> 	/usr/bin/amd64/latencytop		committed
> 	/bin/i86/latencytop			committed
> 	/bin/amd64/latencytop			committed

Seems like a strange duplication, given that /bin is just a symlink to
/usr/bin.

> 	/usr/bin/latencytop (hard link)		committed
> 	/bin/latencytop (hard link)		committed
[...]
> ATTRIBUTES
>      See attributes(5) for descriptions of the  following  attri-
>      butes:
> 
>      ____________________________________________________________
>     |       ATTRIBUTE TYPE        |       ATTRIBUTE VALUE       |
>     |_____________________________|_____________________________|
>     | Architecture                | x86, SPARC                  |
>     |_____________________________|_____________________________|

The interface list above only lists the x86 version, while SPARC is
mentioned here.  Since this seems to depend only on generic DTrace probes,
it should be available on SPARC as well (and if not, there needs to be a
strong justification for the omission).

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Faculty of Technology, Bielefeld University

From carlsonj@phorcys.east.sun.com Thu Jun  4 06:07:31 2009
Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245])
	by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n54D7Uxe027994
	for <psarc-ext@sac.sfbay.sun.com>; Thu, 4 Jun 2009 06:07:30 -0700 (PDT)
Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11])
	by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n54D7U6A025807
	for <@sunmail2sca.sfbay.sun.com:PSARC-ext@Sun.COM>; Thu, 4 Jun 2009 07:07:30 -0600 (MDT)
Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by
 brm-avmta-1.central.sun.com
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 id <0KKP00L0JTSH8R00@brm-avmta-1.central.sun.com> for PSARC-ext@Sun.COM
 (ORCPT PSARC-ext@Sun.COM); Thu, 04 Jun 2009 07:07:29 -0600 (MDT)
Received: from dm-east-01.east.sun.com ([129.148.9.192])
 by brm-avmta-1.central.sun.com
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 with ESMTP id <0KKP00C3OTSGJ960@brm-avmta-1.central.sun.com> for
 PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Thu,
 04 Jun 2009 07:07:28 -0600 (MDT)
Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143])
	by dm-east-01.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2)
 with ESMTP id n54D7QK7011920; Thu, 04 Jun 2009 09:07:26 -0400 (EDT)
Received: from phorcys.east.sun.com (phorcys.local [127.0.0.1])
	by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n54D6HS4003310; Thu,
 04 Jun 2009 09:06:17 -0400 (EDT)
Received: (from carlsonj@localhost)
	by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n54D6H5p003307; Thu,
 04 Jun 2009 09:06:17 -0400 (EDT)
Date: Thu, 04 Jun 2009 09:06:17 -0400
From: James Carlson <james.d.carlson@sun.com>
Subject: Re: LatencyTOP for OpenSolaris [PSARC/2009/339 FastTrack timeout
	06/10/2009]
In-reply-to: <20090603235307.GC1089@sun.com>
To: Sherry Moore <Sherry.Moore@sun.com>
Cc: PSARC-ext@sun.com, Krishnendu.Sadhukhan@sun.com
Message-id: <18983.50889.251377.447085@gargle.gargle.HOWL>
MIME-version: 1.0
X-Mailer: VM 7.01 under Emacs 21.3.1
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
X-PMX-Version: 5.4.1.325704
References: <20090603235307.GC1089@sun.com>
Status: RO
Content-Length: 877

Sherry Moore writes:
> 	/usr/bin/i86/latencytop			committed
> 	/usr/bin/amd64/latencytop		committed

I assume that lack of SPARC here is just an oversight, right?

Are you really planning to document the per-architecture binaries?
That's what "Committed" means.  I don't think that's right, as I don't
see what someone could do usefully do with them.  I would have
expected this instead:

	/usr/bin/latencytop		Committed
	/usr/bin/i86/latencytop		Project Private
	/usr/bin/amd64/latencytop	Project Private
	/usr/bin/sparcv9/latencytop	Project Private

>      Among  non-SunOS  man  pages,  xscreensaver(1),   from   the
>      OpenWindows man pages.

?

-- 
James Carlson, Solaris Networking              <james.d.carlson@sun.com>
Sun Microsystems / 35 Network Drive        71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

From Krishnendu.Sadhukhan@sun.com Thu Jun  4 09:31:21 2009
Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19])
	by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n54GVKUh029701
	for <psarc-ext@sac.sfbay.sun.com>; Thu, 4 Jun 2009 09:31:20 -0700 (PDT)
Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74])
	by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n54GV9s4025845
	for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 5 Jun 2009 00:31:19 +0800 (SGT)
Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by
 nwk-avmta-1.sfbay.Sun.COM
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 id <0KKQ00F093851800@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com
 (ORCPT PSARC-ext@sun.com); Thu, 04 Jun 2009 09:31:17 -0700 (PDT)
Received: from sca-es-mail-2.sun.com ([192.18.43.133])
 by nwk-avmta-1.sfbay.Sun.COM
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 with ESMTP id <0KKQ00GPL384PYE0@nwk-avmta-1.sfbay.Sun.COM> for
 PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu,
 04 Jun 2009 09:31:16 -0700 (PDT)
Received: from fe-sfbay-10.sun.com ([192.18.43.129])
	by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n54GVG90002382	for
 <PSARC-ext@sun.com>; Thu, 04 Jun 2009 09:31:16 -0700 (PDT)
Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com
 (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009))
 id <0KKQ0040032JCS00@fe-sfbay-10.sun.com> for PSARC-ext@sun.com
 (ORCPT PSARC-ext@sun.com); Thu, 04 Jun 2009 09:31:16 -0700 (PDT)
Received: from [192.168.1.100] ([unknown] [71.141.125.151])
 by fe-sfbay-10.sun.com
 (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009))
 with ESMTPSA id <0KKQ00H8G37T4N40@fe-sfbay-10.sun.com>; Thu,
 04 Jun 2009 09:31:15 -0700 (PDT)
Date: Thu, 04 Jun 2009 09:31:05 -0700
From: Krishnendu Sadhukhan <Krishnendu.Sadhukhan@sun.com>
Subject: Re: LatencyTOP for OpenSolaris [PSARC/2009/339 FastTrack timeout
 06/10/2009]
In-reply-to: <18983.50889.251377.447085@gargle.gargle.HOWL>
Sender: Krishnendu.Sadhukhan@sun.com
To: James Carlson <James.D.Carlson@sun.com>
Cc: Sherry Moore <Sherry.Moore@sun.com>, PSARC-ext@sun.com,
        "Zhu, Lejun" <lejun.zhu@intel.com>
Message-id: <4A27F6C9.1030803@sun.com>
Organization: Sun Microsystems
MIME-version: 1.0
Content-type: multipart/mixed; boundary="Boundary_(ID_EgbHaSxpcF6fL8nq+U5dJg)"
X-PMX-Version: 5.4.1.325704
References: <20090603235307.GC1089@sun.com>
 <18983.50889.251377.447085@gargle.gargle.HOWL>
User-Agent: Thunderbird 2.0.0.17 (X11/20081023)
Status: RO
Content-Length: 11229

This is a multi-part message in MIME format.

--Boundary_(ID_EgbHaSxpcF6fL8nq+U5dJg)
Content-type: text/plain; CHARSET=US-ASCII; format=flowed
Content-transfer-encoding: 7BIT

Yes, it was an oversight that the sparc version was omitted.
The update one pager is attached.

Thanks,
Krish

James Carlson wrote:
> Sherry Moore writes:
>   
>> 	/usr/bin/i86/latencytop			committed
>> 	/usr/bin/amd64/latencytop		committed
>>     
>
> I assume that lack of SPARC here is just an oversight, right?
>
> Are you really planning to document the per-architecture binaries?
> That's what "Committed" means.  I don't think that's right, as I don't
> see what someone could do usefully do with them.  I would have
> expected this instead:
>
> 	/usr/bin/latencytop		Committed
> 	/usr/bin/i86/latencytop		Project Private
> 	/usr/bin/amd64/latencytop	Project Private
> 	/usr/bin/sparcv9/latencytop	Project Private
>
>   
>>      Among  non-SunOS  man  pages,  xscreensaver(1),   from   the
>>      OpenWindows man pages.
>>     
>
> ?
>
>   


--Boundary_(ID_EgbHaSxpcF6fL8nq+U5dJg)
Content-type: text/plain; name=one_pager.txt
Content-transfer-encoding: 7BIT
Content-disposition: inline; filename=one_pager.txt

Template Version: @(#)onepager.txt 1.35 07/11/07 SMI
Copyright 2007 Sun Microsystems

1. Introduction
   1.1. Project/Component Working Name:
	LatencyTOP for OpenSolaris

   1.2. Name of Document Author/Supplier:
	krishnendu.sadhukhan@sun.com

   1.3. Date of This Document:
	05/20/09
	
	1.3.1. Date this project was conceived:
		05/20/09

   1.4. Name of Major Document Customer(s)/Consumer(s):
	1.4.1. The PAC or CPT you expect to review your project:
		Solaris PAC
	1.4.2. The ARC(s) you expect to review your project:
		PSARC
	1.4.3. The Director/VP who is "Sponsoring" this project:
		greg.lavendar@sun.com
	1.4.4. The name of your business unit:
		Software

   1.5. Email Aliases:
    	1.5.1. Responsible Manager:	darrin.johnson@sun.com
    	1.5.2. Responsible Engineer:	krishnendu.sadhukhan@sun.com
    	1.5.3. Marketing Manager:	mike.mulkey@sun.com
	1.5.4. Interest List:		latencytop-dev@opensolaris.org

2. Project Summary
   2.1. Project Description:
	LatencyTOP is an observability tool that can be used to identify
	latencies in application and system software. Latency occurs
	when a process/thread can not run and goes to sleep due to
	unavailability of some resource. Developers can use this tool
	to identify latency holes in their applications; system
	programmers can equally use this tool to identify latencies in
	system processes.
	This tool was initially developed for Linux. Intel and Sun are
	jointly collaborating on developing this tool for OpenSolaris.
	The first version of the tool is already available for use on
	OpenSolaris website.

   2.2. Risks and Assumptions:
	LatencyTOP uses probes from the proc, sched and lockstat DTrace
	provider. So, the OpenSolaris version in which LatencyTOP is run
	must have these probes available.

3. Business Summary
   3.1. Problem Area:
	Latency affects performance of applications and systems. This tool
	can be used to locate latencies, which when removed improves
	performance of applications and systems.

   3.2. Market/Requester:

   3.3. Business Justification:
	LatencyTOP is being jointly developed by Sun and Intel through
	the OpenSolaris community.

   3.4. Competitive Analysis:
	LatencyTOP was originally developed for Linux. Making this tool
	available in OpenSolaris will grow the user base of OpenSolaris
	and will provide OpenSolaris with competitive advantage over
	Linux.

   3.5. Opportunity Window/Exposure:
	The first version of the tool is already available on OpenSolaris
	for download and use.

   3.6. How will you know when you are done?:
	Verson 1.0 of the tool is already complete.

4. Technical Description:
    4.1. Details:
	The original LatencyTOP project is hosted on

		http://www.latencytop.org

	The OpenSolaris port of this project is hosted on

		http://opensolaris.org/os/project/latencytop

    4.2. Bug/RFE Number(s):
	6825817 Integrate latencyTOP into OpenSolaris
	6847419 man page for LatencyTOP
    
    4.3. In Scope:
	LatencyTOP traces two types of latencies :
		i) an LWP goes to sleep state because it is waiting for
	some resource to be available for it to run again
		ii) an LWP spinning in order to acquire a synchronization
	object

    4.4. Out of Scope:
	LatencyTOP is only an observability tool; it does not allow the
	user to accomplish anything other than getting latency statistics.

	LatencyTOP does not detect busy loop inside user application.
	Neither does it detect delay that is not caused by waiting,
	e.g. process not running because a higher priority process takes a
	lot of CPU time.
    
    4.5. Interfaces:

	Minor binding only.

	INTERFACES				COMMITTMENT LEVEL
	==========				=================
	/usr/bin/latencytop			committed
	/usr/bin/i86/latencytop			Project Private
	/usr/bin/amd64/latencytop		Project Private
	/usr/bin/sparcv9/latencytop		Project Private

	LatencyTOP has a UI based on libcurses.

    4.6. Doc Impact:
	A new man(1M) page will be required (see Appendix). 
	The OpenSolaris system administration needs to include this tool.

    4.7. Admin/Config Impact:
	None
 
    4.8. HA Impact:
	None

    4.9. I18N/L10N Impact:
	None
 
    4.10. Packaging & Delivery:
	A new package called SUNWlatencytop will be introduced.

    4.11. Security Impact:
	The user must have DTrace privilege to run LatencyTOP.
 
    4.12. Dependencies:
	LatencyTOP uses the Solaris DTrace APIs, specifically the following
	DTrace providers: sched, proc and lockstat.

5. Reference Documents:
	http://monaco.sfbay/detail.jsf?cr=6825817
	http://opensolaris.org/os/project/latencytop

6. Resources and Schedule:
   6.1. Projected Availability:
	LatencyTOP is currently available through OpenSolaris.

   6.2. Cost of Effort:
	This project is being jointly developed by Sun and Intel through
	OpenSolaris. One Sun engineer is required to sponsor the effort
	and facilitate the integration into OpenSolaris.

   6.3. Cost of Capital Resources:
	None

   6.4. Product Approval Committee requested information:
   	6.4.1. Consolidation or Component Name:	sfw
	6.4.3. Type of CPT Review and Approval expected:
		   FastTrack

        6.4.4. Project Boundary Conditions:	None
	6.4.5. Is this a necessary project for OEM agreements:	No
	6.4.6. Notes:	N/A
	6.4.7. Target RTI Date/Release: onnv_120
	6.4.8. Target Code Design Review Date:	06/20/2009
	6.4.9. Update approval addition:	No

   6.5. ARC review type:
		   FastTrack
   6.6. ARC Exposure:
		   open
       6.6.1. Rationale: N/A

7. Prototype Availability:
   7.1. Prototype Availability:
	Version 1.0 of LatencyTOP is already available on OpenSolaris.

   7.2. Prototype Cost:
	N/A

8. Appendix

System Administration Commands                       latencytop(1M)

NAME
     latencytop - report statistics related to latencies in the
     system and in applications

SYNOPSIS
     latencytop [-o log file] [-k log-level] [-t interval] [-f]
		[-s] [-l log interval] [-h]

DESCRIPTION
     LatencyTOP is an observability tool that reports statistics
     about latencies in the system and in applications. The tool
     reports  statistics about where and what  kind of latencies
     are  happening in the  system and in the  applications that
     are running on the system. The  statistics then can be used
     to  improve  performance  throughput  of  applications  and
     system by removing the latencies.

     The tool analyzes system activity periodically and displays
     the data in the output window. Two  types of  latencies are
     tracked - an  LWP  going in and out  of  sleep and  an  LWP
     spinning order to acquire a synchronization object. The tool
     uses the Solaris DTrace framework to collect the statistics
     corresponding to  these two  scenarios of inactivity of the
     system and application LWPs.

     The output window is divided into two sections - upper part
     displays the system-wide  statistics  while  the lower part
     displays  statistics  about  individual processes. The user
     can navigate  the list of  processes (using the < and the >
     keys) and  select the one they are  interested in,  and the
     tool will  display  statistics about that selected  process
     in the lower part of the window; if the t or T key is 
     pressed, the tool  displays  the LWP-specific view of that
     selected process. Thus, the  t or T key can be used to 
     toggle  between the  process-view and the thread-view.

     During execution, a user can force a refresh of the analysis
     by pressing the r or R key.  The interval  time is restored
     to the default or to a specified value (if -t was used). To
     quit  the  application, the user must press the q or Q key.


OPTIONS
     The following options are supported:

     -o [log file]

         Specifies the log file where output will be written. The
	 default log file is /var/log/latencytop.log.

     -k [log level]

         Specifies  the level of  logging  in the log file. Valid
         values are: 0 = none (default), 1 = unknown, and 2 = all;

     -t [interval]

         Specifies the interval, in seconds, at  which  the  tool
         collects statistics from the system. The possible values
         are between 1 and 60; the default is 5 seconds.

     -f

	 Filter large interruptible latencies (e.g. sleep).

     -s

         Monitors the sched (PID=0) process for any latency.

     -l [log-interval]

         Writes data to the log file every log-interval seconds;
	 log-interval must be > 60.

     -h

         Displays the command's usage.


EXAMPLES
     Example 1 Running the tool

     The following command launches the tool with default values
     for options.

       % latencytop

     Example 2 Setting the Interval

     The following command sets the interval to two seconds.

       % latencytop -t 2

     Example 3 Setting the log file

     The following command sets the log file to /tmp/latencytop.log.

       % latencytop -o /tmp/latencytop.log

     Example 4 Setting the log level

     The following command sets the log level to "all".

       % latencytop -l 2


EXIT STATUS
     0

         Successful operation.

     1

         An error occurred.


ATTRIBUTES
     See attributes(5) for descriptions of the  following  attri-
     butes:

     ____________________________________________________________
    |       ATTRIBUTE TYPE        |       ATTRIBUTE VALUE       |
    |_____________________________|_____________________________|
    | Architecture                | x86, SPARC                  |
    |_____________________________|_____________________________|
    | Availability                | SUNWlatencytop              |
    |_____________________________|_____________________________|
    | Interface Stability         | Volatile                    |
    |_____________________________|_____________________________|

SEE ALSO
     kstat(1M), dtrace(1M)

     Among  non-SunOS  man  pages,  xscreensaver(1),   from   the
     OpenWindows man pages.

USAGE
     You must have DTrace privileges to run LatencyTOP. 

--Boundary_(ID_EgbHaSxpcF6fL8nq+U5dJg)--

From Alan.Coopersmith@sun.com Thu Jun  4 10:13:52 2009
Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245])
	by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n54HDqRd000657
	for <psarc-ext@sac.sfbay.sun.com>; Thu, 4 Jun 2009 10:13:52 -0700 (PDT)
Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74])
	by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n54HDoVD001952
	for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Thu, 4 Jun 2009 11:13:51 -0600 (MDT)
Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by
 nwk-avmta-1.sfbay.Sun.COM
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 id <0KKQ00M19573GJ00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@sun.com
 (ORCPT PSARC-ext@sun.com); Thu, 04 Jun 2009 10:13:51 -0700 (PDT)
Received: from sca-es-mail-2.sun.com ([192.18.43.133])
 by nwk-avmta-1.sfbay.Sun.COM
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 with ESMTP id <0KKQ00HP7571SR20@nwk-avmta-1.sfbay.Sun.COM> for
 PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu,
 04 Jun 2009 10:13:49 -0700 (PDT)
Received: from fe-sfbay-10.sun.com ([192.18.43.129])
	by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n54HDntB008203	for
 <PSARC-ext@sun.com>; Thu, 04 Jun 2009 10:13:49 -0700 (PDT)
Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com
 (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009))
 id <0KKQ004003TG8H00@fe-sfbay-10.sun.com> for PSARC-ext@sun.com
 (ORCPT PSARC-ext@sun.com); Thu, 04 Jun 2009 10:13:49 -0700 (PDT)
Received: from [10.6.102.27] ([unknown] [10.6.102.27])
 by fe-sfbay-10.sun.com (Sun Java(tm) System Messaging Server 7u2-7.02 64bit
 (built Apr 16 2009)) with ESMTPSA id <0KKQ00KXG56W5G40@fe-sfbay-10.sun.com>;
 Thu, 04 Jun 2009 10:13:44 -0700 (PDT)
Date: Thu, 04 Jun 2009 10:13:44 -0700
From: Alan Coopersmith <Alan.Coopersmith@sun.com>
Subject: Re: LatencyTOP for OpenSolaris [PSARC/2009/339 FastTrack timeout
 06/10/2009]
In-reply-to: <4A27F6C9.1030803@sun.com>
Sender: Alan.Coopersmith@sun.com
To: Krishnendu Sadhukhan <Krishnendu.Sadhukhan@sun.com>
Cc: James Carlson <James.D.Carlson@sun.com>,
        Sherry Moore <Sherry.Moore@sun.com>, PSARC-ext@sun.com,
        "Zhu, Lejun" <lejun.zhu@intel.com>
Message-id: <4A2800C8.3040709@sun.com>
MIME-version: 1.0
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT
X-PMX-Version: 5.4.1.325704
X-Enigmail-Version: 0.95.1
References: <20090603235307.GC1089@sun.com>
 <18983.50889.251377.447085@gargle.gargle.HOWL> <4A27F6C9.1030803@sun.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090323)
Status: RO
Content-Length: 725

Krishnendu Sadhukhan wrote:
> Yes, it was an oversight that the sparc version was omitted.
> The update one pager is attached.

The updated materials still contains this line in the man page SEE ALSO
section that Jim Carlson asked about -- the line  is both wrong
(xscreensaver is not from OpenWindows, OpenWindows isn't shipped any
 more), and doesn't seem to have any relevance to this case:

>>>      Among  non-SunOS  man  pages,  xscreensaver(1),   from   the
>>>      OpenWindows man pages.

Is this a copy/paste mistake from another man page, and if so, which
man page do I need to file a bug against?

-- 
	-Alan Coopersmith-           alan.coopersmith@sun.com
	 Sun Microsystems, Inc. - X Window System Engineering


From carlsonj@phorcys.east.sun.com Thu Jun  4 10:28:35 2009
Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52])
	by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n54HSZSZ001068
	for <psarc-ext@sac.sfbay.sun.com>; Thu, 4 Jun 2009 10:28:35 -0700 (PDT)
Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74])
	by sunmail3mpk.sfbay.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n54HSWMH004177
	for <@sunmail2sca.sfbay.sun.com:PSARC-ext@Sun.COM>; Thu, 4 Jun 2009 10:28:34 -0700 (PDT)
Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by
 nwk-avmta-1.sfbay.Sun.COM
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 id <0KKQ0020N5VL6Z00@nwk-avmta-1.sfbay.Sun.COM> for PSARC-ext@Sun.COM
 (ORCPT PSARC-ext@Sun.COM); Thu, 04 Jun 2009 10:28:33 -0700 (PDT)
Received: from dm-east-01.east.sun.com ([129.148.9.192])
 by nwk-avmta-1.sfbay.Sun.COM
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 with ESMTP id <0KKQ00H725VLSZ40@nwk-avmta-1.sfbay.Sun.COM> for
 PSARC-ext@Sun.COM (ORCPT PSARC-ext@Sun.COM); Thu,
 04 Jun 2009 10:28:33 -0700 (PDT)
Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143])
	by dm-east-01.east.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2)
 with ESMTP id n54HSUEf024545; Thu, 04 Jun 2009 13:28:30 -0400 (EDT)
Received: from phorcys.east.sun.com (phorcys.local [127.0.0.1])
	by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n54HRM8O004732; Thu,
 04 Jun 2009 13:27:22 -0400 (EDT)
Received: (from carlsonj@localhost)
	by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id n54HRM15004729; Thu,
 04 Jun 2009 13:27:22 -0400 (EDT)
Date: Thu, 04 Jun 2009 13:27:22 -0400
From: James Carlson <james.d.carlson@sun.com>
Subject: Re: LatencyTOP for OpenSolaris [PSARC/2009/339 FastTrack timeout
 06/10/2009]
In-reply-to: <4A27F6C9.1030803@sun.com>
To: Krishnendu Sadhukhan <krishnendu.sadhukhan@sun.com>
Cc: Sherry Moore <Sherry.Moore@sun.com>, PSARC-ext@sun.com,
        "Zhu, Lejun" <lejun.zhu@intel.com>
Message-id: <18984.1018.179782.650942@gargle.gargle.HOWL>
MIME-version: 1.0
X-Mailer: VM 7.01 under Emacs 21.3.1
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
X-PMX-Version: 5.4.1.325704
References: <20090603235307.GC1089@sun.com>
 <18983.50889.251377.447085@gargle.gargle.HOWL> <4A27F6C9.1030803@sun.com>
Status: RO
Content-Length: 434

Krishnendu Sadhukhan writes:
> Yes, it was an oversight that the sparc version was omitted.
> The update one pager is attached.

I'm still not sure what xscreensaver has to do with this project, but
+1 anyway.

-- 
James Carlson, Solaris Networking              <james.d.carlson@sun.com>
Sun Microsystems / 35 Network Drive        71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

From Krishnendu.Sadhukhan@sun.com Thu Jun  4 10:40:29 2009
Received: from sunmail4.singapore.sun.com (sunmail4.Singapore.Sun.COM [129.158.71.19])
	by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n54HeSUc001334
	for <psarc-ext@sac.sfbay.sun.com>; Thu, 4 Jun 2009 10:40:28 -0700 (PDT)
Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11])
	by sunmail4.singapore.sun.com (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id n54HeKac005873
	for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Fri, 5 Jun 2009 01:40:27 +0800 (SGT)
Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by
 brm-avmta-1.central.sun.com
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 id <0KKQ001156FE8O00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com
 (ORCPT PSARC-ext@sun.com); Thu, 04 Jun 2009 11:40:26 -0600 (MDT)
Received: from sca-es-mail-2.sun.com ([192.18.43.133])
 by brm-avmta-1.central.sun.com
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 with ESMTP id <0KKQ0082L6FC3AE0@brm-avmta-1.central.sun.com> for
 PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Thu,
 04 Jun 2009 11:40:25 -0600 (MDT)
Received: from fe-sfbay-10.sun.com ([192.18.43.129])
	by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n54HeO1m011525	for
 <PSARC-ext@sun.com>; Thu, 04 Jun 2009 10:40:24 -0700 (PDT)
Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com
 (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009))
 id <0KKQ004003TG8H00@fe-sfbay-10.sun.com> for PSARC-ext@sun.com
 (ORCPT PSARC-ext@sun.com); Thu, 04 Jun 2009 10:40:21 -0700 (PDT)
Received: from [192.168.1.100] ([unknown] [71.141.125.151])
 by fe-sfbay-10.sun.com
 (Sun Java(tm) System Messaging Server 7u2-7.02 64bit (built Apr 16 2009))
 with ESMTPSA id <0KKQ007J16F1UL00@fe-sfbay-10.sun.com>; Thu,
 04 Jun 2009 10:40:14 -0700 (PDT)
Date: Thu, 04 Jun 2009 10:40:13 -0700
From: Krishnendu Sadhukhan <Krishnendu.Sadhukhan@sun.com>
Subject: Re: LatencyTOP for OpenSolaris [PSARC/2009/339 FastTrack timeout
 06/10/2009]
In-reply-to: <4A2800C8.3040709@sun.com>
Sender: Krishnendu.Sadhukhan@sun.com
To: Alan Coopersmith <Alan.Coopersmith@sun.com>
Cc: James Carlson <James.D.Carlson@sun.com>,
        Sherry Moore <Sherry.Moore@sun.com>, PSARC-ext@sun.com,
        "Zhu, Lejun" <lejun.zhu@intel.com>
Message-id: <4A2806FD.5070402@sun.com>
Organization: Sun Microsystems
MIME-version: 1.0
Content-type: multipart/mixed; boundary="Boundary_(ID_yYkvhf2vXf5NC1UrsroQxg)"
X-PMX-Version: 5.4.1.325704
References: <20090603235307.GC1089@sun.com>
 <18983.50889.251377.447085@gargle.gargle.HOWL> <4A27F6C9.1030803@sun.com>
 <4A2800C8.3040709@sun.com>
User-Agent: Thunderbird 2.0.0.17 (X11/20081023)
Status: RO
Content-Length: 11250

This is a multi-part message in MIME format.

--Boundary_(ID_yYkvhf2vXf5NC1UrsroQxg)
Content-type: text/plain; CHARSET=US-ASCII; format=flowed
Content-transfer-encoding: 7BIT

Alan Coopersmith wrote:
> Krishnendu Sadhukhan wrote:
>   
>> Yes, it was an oversight that the sparc version was omitted.
>> The update one pager is attached.
>>     
>
> The updated materials still contains this line in the man page SEE ALSO
> section that Jim Carlson asked about -- the line  is both wrong
> (xscreensaver is not from OpenWindows, OpenWindows isn't shipped any
>  more), and doesn't seem to have any relevance to this case:
>
>   
>>>>      Among  non-SunOS  man  pages,  xscreensaver(1),   from   the
>>>>      OpenWindows man pages.
>>>>         
>
>   
Sorry, I forgot to delete that. It is no more in the attached one pager.
> Is this a copy/paste mistake from another man page, and if so, which
> man page do I need to file a bug against?
>
>   
I basically followed the man page for powertop, but it was not just a 
blind copy/paste - I adapted the man page to suit latencytop, but 
thought that line was a valid one, so I kept it.

-Krish

--Boundary_(ID_yYkvhf2vXf5NC1UrsroQxg)
Content-type: text/plain; name=one_pager.txt
Content-transfer-encoding: 7BIT
Content-disposition: inline; filename=one_pager.txt

Template Version: @(#)onepager.txt 1.35 07/11/07 SMI
Copyright 2007 Sun Microsystems

1. Introduction
   1.1. Project/Component Working Name:
	LatencyTOP for OpenSolaris

   1.2. Name of Document Author/Supplier:
	krishnendu.sadhukhan@sun.com

   1.3. Date of This Document:
	05/20/09
	
	1.3.1. Date this project was conceived:
		05/20/09

   1.4. Name of Major Document Customer(s)/Consumer(s):
	1.4.1. The PAC or CPT you expect to review your project:
		Solaris PAC
	1.4.2. The ARC(s) you expect to review your project:
		PSARC
	1.4.3. The Director/VP who is "Sponsoring" this project:
		greg.lavendar@sun.com
	1.4.4. The name of your business unit:
		Software

   1.5. Email Aliases:
    	1.5.1. Responsible Manager:	darrin.johnson@sun.com
    	1.5.2. Responsible Engineer:	krishnendu.sadhukhan@sun.com
    	1.5.3. Marketing Manager:	mike.mulkey@sun.com
	1.5.4. Interest List:		latencytop-dev@opensolaris.org

2. Project Summary
   2.1. Project Description:
	LatencyTOP is an observability tool that can be used to identify
	latencies in application and system software. Latency occurs
	when a process/thread can not run and goes to sleep due to
	unavailability of some resource. Developers can use this tool
	to identify latency holes in their applications; system
	programmers can equally use this tool to identify latencies in
	system processes.
	This tool was initially developed for Linux. Intel and Sun are
	jointly collaborating on developing this tool for OpenSolaris.
	The first version of the tool is already available for use on
	OpenSolaris website.

   2.2. Risks and Assumptions:
	LatencyTOP uses probes from the proc, sched and lockstat DTrace
	provider. So, the OpenSolaris version in which LatencyTOP is run
	must have these probes available.

3. Business Summary
   3.1. Problem Area:
	Latency affects performance of applications and systems. This tool
	can be used to locate latencies, which when removed improves
	performance of applications and systems.

   3.2. Market/Requester:

   3.3. Business Justification:
	LatencyTOP is being jointly developed by Sun and Intel through
	the OpenSolaris community.

   3.4. Competitive Analysis:
	LatencyTOP was originally developed for Linux. Making this tool
	available in OpenSolaris will grow the user base of OpenSolaris
	and will provide OpenSolaris with competitive advantage over
	Linux.

   3.5. Opportunity Window/Exposure:
	The first version of the tool is already available on OpenSolaris
	for download and use.

   3.6. How will you know when you are done?:
	Verson 1.0 of the tool is already complete.

4. Technical Description:
    4.1. Details:
	The original LatencyTOP project is hosted on

		http://www.latencytop.org

	The OpenSolaris port of this project is hosted on

		http://opensolaris.org/os/project/latencytop

    4.2. Bug/RFE Number(s):
	6825817 Integrate latencyTOP into OpenSolaris
	6847419 man page for LatencyTOP
    
    4.3. In Scope:
	LatencyTOP traces two types of latencies :
		i) an LWP goes to sleep state because it is waiting for
	some resource to be available for it to run again
		ii) an LWP spinning in order to acquire a synchronization
	object

    4.4. Out of Scope:
	LatencyTOP is only an observability tool; it does not allow the
	user to accomplish anything other than getting latency statistics.

	LatencyTOP does not detect busy loop inside user application.
	Neither does it detect delay that is not caused by waiting,
	e.g. process not running because a higher priority process takes a
	lot of CPU time.
    
    4.5. Interfaces:

	Minor binding only.

	INTERFACES				COMMITTMENT LEVEL
	==========				=================
	/usr/bin/latencytop			committed
	/usr/bin/i86/latencytop			Project Private
	/usr/bin/amd64/latencytop		Project Private
	/usr/bin/sparcv9/latencytop		Project Private

	LatencyTOP has a UI based on libcurses.

    4.6. Doc Impact:
	A new man(1M) page will be required (see Appendix). 
	The OpenSolaris system administration needs to include this tool.

    4.7. Admin/Config Impact:
	None
 
    4.8. HA Impact:
	None

    4.9. I18N/L10N Impact:
	None
 
    4.10. Packaging & Delivery:
	A new package called SUNWlatencytop will be introduced.

    4.11. Security Impact:
	The user must have DTrace privilege to run LatencyTOP.
 
    4.12. Dependencies:
	LatencyTOP uses the Solaris DTrace APIs, specifically the following
	DTrace providers: sched, proc and lockstat.

5. Reference Documents:
	http://monaco.sfbay/detail.jsf?cr=6825817
	http://opensolaris.org/os/project/latencytop

6. Resources and Schedule:
   6.1. Projected Availability:
	LatencyTOP is currently available through OpenSolaris.

   6.2. Cost of Effort:
	This project is being jointly developed by Sun and Intel through
	OpenSolaris. One Sun engineer is required to sponsor the effort
	and facilitate the integration into OpenSolaris.

   6.3. Cost of Capital Resources:
	None

   6.4. Product Approval Committee requested information:
   	6.4.1. Consolidation or Component Name:	sfw
	6.4.3. Type of CPT Review and Approval expected:
		   FastTrack

        6.4.4. Project Boundary Conditions:	None
	6.4.5. Is this a necessary project for OEM agreements:	No
	6.4.6. Notes:	N/A
	6.4.7. Target RTI Date/Release: onnv_120
	6.4.8. Target Code Design Review Date:	06/20/2009
	6.4.9. Update approval addition:	No

   6.5. ARC review type:
		   FastTrack
   6.6. ARC Exposure:
		   open
       6.6.1. Rationale: N/A

7. Prototype Availability:
   7.1. Prototype Availability:
	Version 1.0 of LatencyTOP is already available on OpenSolaris.

   7.2. Prototype Cost:
	N/A

8. Appendix

System Administration Commands                       latencytop(1M)

NAME
     latencytop - report statistics related to latencies in the
     system and in applications

SYNOPSIS
     latencytop [-o log file] [-k log-level] [-t interval] [-f]
		[-s] [-l log interval] [-h]

DESCRIPTION
     LatencyTOP is an observability tool that reports statistics
     about latencies in the system and in applications. The tool
     reports  statistics about where and what  kind of latencies
     are  happening in the  system and in the  applications that
     are running on the system. The  statistics then can be used
     to  improve  performance  throughput  of  applications  and
     system by removing the latencies.

     The tool analyzes system activity periodically and displays
     the data in the output window. Two  types of  latencies are
     tracked - an  LWP  going in and out  of  sleep and  an  LWP
     spinning order to acquire a synchronization object. The tool
     uses the Solaris DTrace framework to collect the statistics
     corresponding to  these two  scenarios of inactivity of the
     system and application LWPs.

     The output window is divided into two sections - upper part
     displays the system-wide  statistics  while  the lower part
     displays  statistics  about  individual processes. The user
     can navigate  the list of  processes (using the < and the >
     keys) and  select the one they are  interested in,  and the
     tool will  display  statistics about that selected  process
     in the lower part of the window; if the t or T key is 
     pressed, the tool  displays  the LWP-specific view of that
     selected process. Thus, the  t or T key can be used to 
     toggle  between the  process-view and the thread-view.

     During execution, a user can force a refresh of the analysis
     by pressing the r or R key.  The interval  time is restored
     to the default or to a specified value (if -t was used). To
     quit  the  application, the user must press the q or Q key.


OPTIONS
     The following options are supported:

     -o [log file]

         Specifies the log file where output will be written. The
	 default log file is /var/log/latencytop.log.

     -k [log level]

         Specifies  the level of  logging  in the log file. Valid
         values are: 0 = none (default), 1 = unknown, and 2 = all;

     -t [interval]

         Specifies the interval, in seconds, at  which  the  tool
         collects statistics from the system. The possible values
         are between 1 and 60; the default is 5 seconds.

     -f

	 Filter large interruptible latencies (e.g. sleep).

     -s

         Monitors the sched (PID=0) process for any latency.

     -l [log-interval]

         Writes data to the log file every log-interval seconds;
	 log-interval must be > 60.

     -h

         Displays the command's usage.


EXAMPLES
     Example 1 Running the tool

     The following command launches the tool with default values
     for options.

       % latencytop

     Example 2 Setting the Interval

     The following command sets the interval to two seconds.

       % latencytop -t 2

     Example 3 Setting the log file

     The following command sets the log file to /tmp/latencytop.log.

       % latencytop -o /tmp/latencytop.log

     Example 4 Setting the log level

     The following command sets the log level to "all".

       % latencytop -l 2


EXIT STATUS
     0

         Successful operation.

     1

         An error occurred.


ATTRIBUTES
     See attributes(5) for descriptions of the  following  attri-
     butes:

     ____________________________________________________________
    |       ATTRIBUTE TYPE        |       ATTRIBUTE VALUE       |
    |_____________________________|_____________________________|
    | Architecture                | x86, SPARC                  |
    |_____________________________|_____________________________|
    | Availability                | SUNWlatencytop              |
    |_____________________________|_____________________________|
    | Interface Stability         | Volatile                    |
    |_____________________________|_____________________________|

SEE ALSO
     kstat(1M), dtrace(1M)

USAGE
     You must have DTrace privileges to run LatencyTOP. 

--Boundary_(ID_yYkvhf2vXf5NC1UrsroQxg)--

From sherry.moore@sun.com Wed Sep 23 09:13:32 2009
Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165])
	by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n8NGDUqZ004151
	for <psarc-ext@sac.sfbay.sun.com>; Wed, 23 Sep 2009 09:13:30 -0700 (PDT)
Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11])
	by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n8NGDN4i027823
	for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Wed, 23 Sep 2009 17:13:29 +0100 (BST)
Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by
 brm-avmta-1.central.sun.com
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 id <0KQF00C0JMEB0M00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com
 (ORCPT PSARC-ext@sun.com); Wed, 23 Sep 2009 10:13:23 -0600 (MDT)
Received: from dm-sfbay-01.sfbay.sun.com ([129.145.155.118])
 by brm-avmta-1.central.sun.com
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 with ESMTP id <0KQF004ZFME9SA50@brm-avmta-1.central.sun.com> for
 PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Wed,
 23 Sep 2009 10:13:21 -0600 (MDT)
Received: from too-cool.sfbay.sun.com
 (too-cool.SFBay.Sun.COM [129.146.226.229])	by dm-sfbay-01.sfbay.sun.com
 (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n8NGDLsS041130; Wed,
 23 Sep 2009 09:13:21 -0700 (PDT)
Received: from too-cool.eng.sun.com (too-cool [127.0.0.1])
	by too-cool.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id n8NGFcTr002433;
 Wed, 23 Sep 2009 09:15:38 -0700 (PDT)
Received: (from sherrym@localhost)
	by too-cool.eng.sun.com (8.14.3+Sun/8.14.3/Submit) id n8NGFcEE002432; Wed,
 23 Sep 2009 09:15:38 -0700 (PDT)
Date: Wed, 23 Sep 2009 09:15:38 -0700
From: Sherry Moore <sherry.moore@sun.com>
Subject: Re: LatencyTOP for OpenSolaris [PSARC/2009/339 FastTrack timeout
	06/10/2009]
In-reply-to: <20090603235307.GC1089@sun.com>
To: Sherry Moore <sherry.moore@sun.com>
Cc: PSARC-ext@sun.com, Krishnendu.Sadhukhan@sun.com
Message-id: <20090923161537.GA2334@sun.com>
MIME-version: 1.0
Content-type: multipart/mixed; boundary="Boundary_(ID_p7vcnKXvqfKr4eNUz3jQDQ)"
Content-disposition: inline
X-PMX-Version: 5.4.1.325704
References: <20090603235307.GC1089@sun.com>
X-Authentication-warning: too-cool.eng.sun.com: sherrym set sender to
 sherry.moore@sun.com using -f
User-Agent: Mutt/1.5.19 (2009-01-05)
Status: RO
Content-Length: 9106


--Boundary_(ID_p7vcnKXvqfKr4eNUz3jQDQ)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Content-disposition: inline

The project team would like the latest man page reflected in the case
archive.  man-page.txt and man-page-diff.txt can be found in the
materials directory.

They have also been attached for easy viewing.

Sherry
--
Sherry Moore, Solaris Core Kernel	http://blogs.sun.com/sherrym

--Boundary_(ID_p7vcnKXvqfKr4eNUz3jQDQ)
Content-type: text/plain; NAME=man-page.txt; charset=us-ascii
Content-transfer-encoding: 7BIT
Content-disposition: attachment; filename=man-page.txt

System Administration Commands                       latencytop(1M)

NAME
     latencytop - report statistics related to latencies in the
     system and in applications

SYNOPSIS
     latencytop [-t interval] [-o log-file] [-k log-level] [c config-file]
		[-f (no)feature,...] [-l log-interval] [-h]

DESCRIPTION
     LatencyTOP is an observability tool that reports statistics
     about latencies in the system and in applications. The tool
     reports  statistics about where and what  kind of latencies
     are  happening in the  system and in the  applications that
     are running on the system. The  statistics then can be used
     to  improve  performance  throughput  of  applications  and
     system by removing the latencies.

     The tool analyzes system activity periodically and displays
     the data in the output window. Two  types of  latencies are
     tracked - an  LWP  going in and out  of  sleep and  an  LWP
     spinning order to acquire a synchronization object. The tool
     uses the Solaris DTrace framework to collect the statistics
     corresponding to  these two  scenarios of inactivity of the
     system and application LWPs.

     The output window is divided into two sections - upper part
     displays the system-wide  statistics  while  the lower part
     displays  statistics  about  individual processes. The user
     can navigate  the list of  processes (using the < and the >
     keys) and  select the one they are  interested in,  and the
     tool will  display  statistics about that selected  process
     in the lower part of the window; if the t or T key is 
     pressed, the tool  displays  the LWP-specific view of that
     selected process. Thus, the  t or T key can be used to 
     toggle  between the  process-view and the thread-view.

     During execution, a user can force a refresh of the analysis
     by pressing the r or R key.  The interval  time is restored
     to the default or to a specified value (if -t was used). To
     quit  the  application, the user must press the q or Q key.


OPTIONS
     The following options are supported:

     -o [log-file]

         Specifies the log file where output will be written. The
	 default log file is /var/log/latencytop.log.

     -k [log-level]

         Specifies  the level of  logging  in the log file. Valid
         values are: 0 = none (default), 1 = unknown, and 2 = all;

     -t [interval]

         Specifies the interval, in seconds, at  which  the  tool
         collects statistics from the system. The possible values
         are between 1 and 60; the default is 5 seconds.

     -f, --feature [no]feature1,[no]feature2,...
	
	 Enables/disables features in LatencyTOP. Features can only
	 be one of the following :

    		[no]filter: Filter large interruptible latencies,
			e.g. sleep. Default: off

    		[no]sched: Monitors sched (PID=0). Default: off

    		[no]sobj: Monitors synchronize objects. Default: on

    		[no]low: Lower overhead by sampling small latencies.
			Enabling this feature will lower CPU utilization
			by estimating small latencies statistically.
			Use it for heavy workloads such as a very busy
			web server. Default: off

     -l [log-interval]

         Writes data to the log file every log-interval seconds;
	 log-interval must be > 60.

     -h

         Displays the command's usage.


EXAMPLES
     Example 1 Running the tool

     The following command launches the tool with default values
     for options.

       % latencytop

     Example 2 Setting the Interval

     The following command sets the interval to two seconds.

       % latencytop -t 2

     Example 3 Setting the log file

     The following command sets the log file to /tmp/latencytop.log.

       % latencytop -o /tmp/latencytop.log

     Example 4 Setting the log level

     The following command sets the log level to "all".

       % latencytop -l 2

     Example 4 Enabling tracing of latencies caused by synchronization
	objects.

     The following command enables the tracing of latencies caused by
	synchronization objects.

       % latencytop -f sobj


EXIT STATUS
     0

         Successful operation.

     1

         An error occurred.


ATTRIBUTES
     See attributes(5) for descriptions of the  following  attri-
     butes:

     ____________________________________________________________
    |       ATTRIBUTE TYPE        |       ATTRIBUTE VALUE       |
    |_____________________________|_____________________________|
    | Architecture                | x86, SPARC                  |
    |_____________________________|_____________________________|
    | Availability                | SUNWlatencytop              |
    |_____________________________|_____________________________|
    | Interface Stability         | Volatile                    |
    |_____________________________|_____________________________|

SEE ALSO
     kstat(1M), dtrace(1M)

USAGE
     You must have DTrace privileges to run LatencyTOP. 

--Boundary_(ID_p7vcnKXvqfKr4eNUz3jQDQ)
Content-type: text/plain; NAME=man-page-diff.txt; charset=us-ascii
Content-transfer-encoding: 7BIT
Content-disposition: attachment; filename=man-page-diff.txt

*** /tmp/old-man-page.txt	Wed Sep 23 08:55:24 2009
--- /tmp/man-page.txt	Wed Sep 23 08:54:18 2009
***************
*** 5,12 ****
       system and in applications
  
  SYNOPSIS
!      latencytop [-o log file] [-k log-level] [-t interval] [-f]
! 		[-s] [-l log interval] [-h]
  
  DESCRIPTION
       LatencyTOP is an observability tool that reports statistics
--- 5,12 ----
       system and in applications
  
  SYNOPSIS
!      latencytop [-t interval] [-o log-file] [-k log-level] [c config-file]
! 		[-f (no)feature,...] [-l log-interval] [-h]
  
  DESCRIPTION
       LatencyTOP is an observability tool that reports statistics
***************
*** 45,56 ****
  OPTIONS
       The following options are supported:
  
!      -o [log file]
  
           Specifies the log file where output will be written. The
  	 default log file is /var/log/latencytop.log.
  
!      -k [log level]
  
           Specifies  the level of  logging  in the log file. Valid
           values are: 0 = none (default), 1 = unknown, and 2 = all;
--- 45,56 ----
  OPTIONS
       The following options are supported:
  
!      -o [log-file]
  
           Specifies the log file where output will be written. The
  	 default log file is /var/log/latencytop.log.
  
!      -k [log-level]
  
           Specifies  the level of  logging  in the log file. Valid
           values are: 0 = none (default), 1 = unknown, and 2 = all;
***************
*** 61,74 ****
           collects statistics from the system. The possible values
           are between 1 and 60; the default is 5 seconds.
  
!      -f
  
! 	 Filter large interruptible latencies (e.g. sleep).
  
!      -s
  
!          Monitors the sched (PID=0) process for any latency.
  
       -l [log-interval]
  
           Writes data to the log file every log-interval seconds;
--- 61,84 ----
           collects statistics from the system. The possible values
           are between 1 and 60; the default is 5 seconds.
  
!      -f, --feature [no]feature1,[no]feature2,...
! 	
! 	 Enables/disables features in LatencyTOP. Features can only
! 	 be one of the following :
  
!     		[no]filter: Filter large interruptible latencies,
! 			e.g. sleep. Default: off
  
!     		[no]sched: Monitors sched (PID=0). Default: off
  
!     		[no]sobj: Monitors synchronize objects. Default: on
  
+     		[no]low: Lower overhead by sampling small latencies.
+ 			Enabling this feature will lower CPU utilization
+ 			by estimating small latencies statistically.
+ 			Use it for heavy workloads such as a very busy
+ 			web server. Default: off
+ 
       -l [log-interval]
  
           Writes data to the log file every log-interval seconds;
***************
*** 105,111 ****
--- 115,129 ----
  
         % latencytop -l 2
  
+      Example 4 Enabling tracing of latencies caused by synchronization
+ 	objects.
  
+      The following command enables the tracing of latencies caused by
+ 	synchronization objects.
+ 
+        % latencytop -f sobj
+ 
+ 
  EXIT STATUS
       0
  
***************
*** 133,140 ****
  SEE ALSO
       kstat(1M), dtrace(1M)
  
-      Among  non-SunOS  man  pages,  xscreensaver(1),   from   the
-      OpenWindows man pages.
- 
  USAGE
       You must have DTrace privileges to run LatencyTOP. 
--- 151,155 ----

--Boundary_(ID_p7vcnKXvqfKr4eNUz3jQDQ)--

From Krishnendu.Sadhukhan@Sun.COM Mon Feb  8 20:53:52 2010
Received: from sunmail3mpk.sfbay.sun.com (sunmail3mpk.SFBay.Sun.COM [129.146.11.52])
	by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id o194rqEm015307
	for <psarc-ext@sac.sfbay.sun.com>; Mon, 8 Feb 2010 20:53:52 -0800 (PST)
Received: from brm-avmta-1.central.sun.com (brm-avmta-1.Central.Sun.COM [129.147.4.11])
	by sunmail3mpk.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.4) with ESMTP id o194rpdJ010418
	for <@sunmail2sca.sfbay.sun.com:PSARC-ext@sun.com>; Mon, 8 Feb 2010 20:53:51 -0800 (PST)
Received: from pmxchannel-daemon.brm-avmta-1.central.sun.com by
 brm-avmta-1.central.sun.com
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 id <0KXK00E015LRBA00@brm-avmta-1.central.sun.com> for PSARC-ext@sun.com
 (ORCPT PSARC-ext@sun.com); Mon, 08 Feb 2010 21:53:51 -0700 (MST)
Received: from sca-es-mail-1.sun.com ([192.18.43.132])
 by brm-avmta-1.central.sun.com
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 with ESMTP id <0KXK007MN5LQ1QD0@brm-avmta-1.central.sun.com> for
 PSARC-ext@sun.com (ORCPT PSARC-ext@sun.com); Mon,
 08 Feb 2010 21:53:51 -0700 (MST)
Received: from fe-sfbay-10.sun.com ([192.18.43.129])
	by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id o194roOX006762	for
 <PSARC-ext@sun.com>; Mon, 08 Feb 2010 20:53:50 -0800 (PST)
Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com
 (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul  2 2009))
 id <0KXK002005LQVX00@fe-sfbay-10.sun.com> for PSARC-ext@sun.com
 (ORCPT PSARC-ext@sun.com); Mon, 08 Feb 2010 20:53:50 -0800 (PST)
Received: from [192.168.1.101] ([unknown] [71.141.106.112])
 by fe-sfbay-10.sun.com
 (Sun Java(tm) System Messaging Server 7u2-7.04 64bit (built Jul  2 2009))
 with ESMTPSA id <0KXK00DW25LPD040@fe-sfbay-10.sun.com> for PSARC-ext@sun.com
 (ORCPT PSARC-ext@sun.com); Mon, 08 Feb 2010 20:53:50 -0800 (PST)
Date: Mon, 08 Feb 2010 20:53:52 -0800
From: Krishnendu Sadhukhan <Krishnendu.Sadhukhan@Sun.COM>
Subject: LatencyTOP for OpenSolaris [PSARC/2009/339 FastTrack timeout
	06/10/2009]
Sender: Krishnendu.Sadhukhan@Sun.COM
To: PSARC-ext@Sun.COM
Message-id: <4B70EA60.7040506@sun.com>
MIME-version: 1.0
Content-type: text/plain; CHARSET=US-ASCII; format=flowed
Content-transfer-encoding: 7BIT
X-PMX-Version: 5.4.1.325704
User-Agent: Thunderbird 2.0.0.21 (X11/20090323)
Status: RO
Content-Length: 2232

The project team is adding one new command line option, "-s", to the 
latencytop tool. Below are the output from the current and the proposed 
command, respectively. We would like to archive this enhancement in the 
case directory.

Thanks,
Krish

----------------------------------------------------------------------------------------------------------------------------------------------------------
# latencytop -h
Usage: latencytop [option(s)]
Options:
   -h, --help
       Print this help.
   -t, --interval TIME
       Set refresh interval to TIME. Valid range [1...60] seconds, 
default = 5
   -o, --output-log-file FILE
       Output kernel log to FILE. Default = /var/log/latencytop.log
   -k, --kernel-log-level LEVEL
       Set kernel log level to LEVEL.
       0(default) = None, 1 = Unmapped, 2 = Mapped, 3 = All.
   -f, --feature [no]feature1,[no]feature2,...
       Enable/disable features in LatencyTOP.
       [no]filter:
       Filter large interruptible latencies, e.g. sleep.
       [no]sched:
       Monitors sched (PID=0).
       [no]sobj:
       Monitors synchronization objects.
       [no]low:
       Lower overhead by sampling small latencies.
   -l, --log-period TIME
       Write and restart log every TIME seconds, TIME >= 60

#latencytop -h
Usage: latencytop [option(s)]
Options:
   -h, --help
       Print this help.
   -t, --interval TIME
       Set refresh interval to TIME. Valid range [1...60] seconds, 
default = 5
   -o, --output-log-file FILE
       Output kernel log to FILE. Default = /var/log/latencytop.log
   -k, --kernel-log-level LEVEL
       Set kernel log level to LEVEL.
       0(default) = None, 1 = Unmapped, 2 = Mapped, 3 = All.
   -f, --feature [no]feature1,[no]feature2,...
       Enable/disable features in LatencyTOP.
       [no]filter:
       Filter large interruptible latencies, e.g. sleep.
       [no]sched:
       Monitors sched (PID=0).
       [no]sobj:
       Monitors synchronization objects.
       [no]low:
       Lower overhead by sampling small latencies.
   -l, --log-period TIME
       Write and restart log every TIME seconds, TIME >= 60
   -s --select [ pid=<pid> | pgid=<pgid> ]
       Monitor only the given process or processes in the given process 
group.

