Subject: PSARC/2008/098 Lockstat: Changing spin count to spin time
To:      PSARC-record@sac.sfbay.sun.com
Cc:      Steve.Sistare@sun.com Chris.kiick@sun.com
Bcc:     one-pager-list@sac.sfbay one-pager-log@sac.sfbay sac-bar@sac.sfbay


I am sponsoring this fasttrack for Steve Sistare and the timeout
is set to expire 02/19/2008. The proposal is attached below.
The project team is asking for a release binding of patch/micro.

This project proposes to change the spin count metric provided
by lockstat(1M) to a more appropriate and useful metric of spin time.


---- Cut here ----

Lockstat(1M): Changing spin count to spin time.

Certain kernel lock probes return a value called the "spin count",
which is the number of iterations that are taken through the spin loop
before the lock is acquired.

This metric has been used in the past to diagnose certain performance
problems due to lock contention . However it has been of limited
usefulness.  The metric of "spin count" does not provide an absolute
measure of overhead and cannot be used to compare between systems.
With the addition of exponential backoff optimizations, it becomes
even cruder because the number of spins is no longer linearly
proportional to the amount of time spent acquiring the lock.  The
dtrace guide itself says:  "The spin count has little meaning on its
own, but can be used to compare spin times."

This fast-track proposes that the spin count metric should be
replaced with a direct measurement of the spin time.  This will be a
measure of time spent busy-waiting to acquire a lock, using high
resolution time (in nanoseconds).  This provides an accurate measure
of lock contention that will be very useful to performance analysts;
in particular, it allows them to see how much %sys time is due to
lock contention.  Nanoseconds can also be compared between OS
versions, CPU speeds and platforms.  It will also be more consistent
with other dtrace probes and commands (like plockstat) that report
times instead of spin counts.

Updating the spin metric will require three changes:
    1) Change the mutex and disp_lock code to gather high-resolution 
       timing data using gethrtime() instead of counting loop iterations.
    2) Update the lockstat command to use the column header "nsec" 
       instead of "spin" for the spin values.
    3) Modify the documentation - the lockstat man page and dtrace guide - 
       to reflect this change.

These changes have already been tested.  Note that change #1
does not introduce any performance hit from the overhead of calling
gethrtime() in the mutex spin code - this has been tested.

Modified Interfaces:

1. man lockstat(1M)

  The existing text says:

     spin or nsec            Average number of times caller  spun
                             trying  to  get the lock, or average
                             duration   of    the    events    in
                             nanoseconds,  as appropriate for the
                             event...

  The proposed change:

    nsec                    Average duration of the events in nanoseconds,
                            as  appropriate for  the  event...

  Stability: Unstable

    The output of lockstat is not a stable interface and may be modified.
    The materials for the original lockstat case PSARC/1997/086 state:
      "The lockstat command and interfaces are all classified as Unstable"
    The man page makes this clear:
      "The format  and  content  of lockstat output reflect the current
      Solaris kernel implementation and are therefore  subject  to
      change in future releases."

2. Solaris Dynamic Tracing Guide, Chapter 18, lockstat Provider

  The probe lockstat:::adaptive-spin arg1 was defined as spin count,
  but will now defined to be spin time in nanoseconds.

  The probe lockstat:::spin-spin arg1 was defined as spin count,
  and is now defined to be spin time in nanoseconds.

  The existing text for both says:

    "arg1 ... contains the spin count: the number of iterations that
    were taken through the spin loop before the lock was acquired.
    The spin count has little meaning on its own, but can be used to
    compare spin times."

  The proposed change:

    "arg1 ... contains the spin time in nanoseconds:  the elapsed
    time spent in the spin loop before the lock was acquired."

  Stability: Evolving

    The dtrace guide marks the Arguments in the lockstat provider as
    Evolving stability.  Evolving interfaces can change but an effort must
    be made to ensure binary compatibility. Since this is only a change
    in units, there is no issue with binary compatibility.

References:
    CR6624951 replace lockstat spin metric with busy-wait time
    CR6658169 dtrace documentation update for lock spin metric change
    CR6657854 man page and documentation changes for lockstat spin metric