#pragma ident	"@(#)flowadm.1m.txt	1.22	08/05/28 SMI"


System Administration Commands			      flowadm(1M)



NAME
     flowadm - administer bandwidth resource control and priority
               for protocols, services, Containers and Virtual machines

SYNOPSIS
     flowadm show-flow [-pP] [-S] [-s [-i interval]] [-l link] [flow]

     flowadm add-flow [-t] [-R root-dir] -l link 
         -a attr=value[,...] -p prop=value[,...] flow
     flowadm remove-flow [-t] [-R root-dir] {-l link | flow}

     flowadm set-flowprop [-t] [-R root-dir] -p prop=value[,...] flow
     flowadm reset-flowprop [-t] [-R root-dir] [-p prop,...] flow
     flowadm show-flowprop [-cP] [-l link] [-o field,...] [-p prop,...] [flow]


     flowadm show-usage [-d] -f <filename>
         [[-s <time>] [-e  <time>] -p ] [-l <link>] [<flow>]


DESCRIPTION
     The flowadm command is used to create, modify, remove and show
     networking bandwidth and associated resources for a type of traffic on 
     a particular link.

CONCEPTS

     The flowadm command allows users to manage networking bandwidth
     resource for a transport, service, or a subnet. The
     service is specified as a combination of transport and local port.
     The subnet is specified by it IP address and subnet mask.
     The command can be used on any type of data link, including
     physical links, virtual NICs, and link aggregations.

     A flow is defined as a set of attributes based on Layer3 and
     4 headers which can be used to identify a protocol, service, or a
     virtual machine. When a flow is identified based on flow
     attributes, separate kernel resources including layer 2, 3, and 4
     queues, their processing threads, etc are uniquely created for it
     such that other traffic has minimal or zero impact on it. The 
     processing of the packets for a given flow can be restricted to
     a given CPU or processor set such that all the resources 
     necessary for processing the packets of the flow are dedicated to 
     it.
     Inboud and outbound packet are matched to flows in a very
     fast and scalable way, so that limits can be enforced with minimal
     performance impact.

     The flowadm command can be used to identify a flow without
     imposing any bandwidth resource control. This would result in 
     the traffic type getting its own resources and queues so that it is
     isolated from rest of the networking traffic for more obserbale
     and deterministic behavior.


  SUBCOMMANDS
     The following subcommands are supported:

     flowadm show-flow [-pP] [-S] [-s [-i interval]] [-o field,...] 
         [-l link | flow]

         Show flow configuration information (the  default)  or
         statistics,  either for all flows, all flows on a link
         or for the specified name.

	 -o field, --output=field

             A case sensitive, comma-separated list of output
             fields to display.  The field name must be one of the
             fields listed below, or a special value all, to
             display all fields.  For each flow found, the following 
             fields can be displayed:

             LINK

                  The name of the link the flow is on.

             LOCAL_IP

                   Local IP address of the flow.


    	     TRANSPORT

                  The name of the layer for protocol to be used.

	     IP_VERSION

                   Version of the IP protocol for the flow

             LOCAL_PORT

                    Local port of service for flow.

             DSFIELD

                    Differentiated services value for flow.

             DSFIELD_MASK

                    Mask used with DSFIELD value to stat the
                    bits of intrestest in the differentiated
                    services field of the IP header
   
             MAXBW

                    Maximum bandwidth allowed for the flow in Mbps.


         -p, --parseable
 
             Display using a stable machine-parseable format.


         -P, --persistent

            Display persistent flow property information.
 

	 -S, --continuous

             Continuously display network utilization by flow in a manner 
             similar to the way that prstat(1m) displays cpu utilization
             by process.


         -s, --statistics
 
             Displays flow statistics.
 
 
         -i interval, --interval=interval
 
             Used with the -s option to specify an  interval,  in
             seconds, at which statistics should be displayed. If
             this option is not specified, statistics  will  only
             be displayed once.


         -l link, --link=link | flow

             Display information for all flows on the named link or information
             for the named flow.


     flowadm add-flow [-t] [-R root-dir] -l link 
         -a attr=value[,...] -p prop=value[,...] flow

	 Adds a flow to the system identified by its flow attributes and
         properties.
	 

	 As part of identifying a particular flow, its bandwidth
	 resource can be limited and its relative priority to other 
         traffic can be specified. If no bandwidth limit or priority is
	 specified, the traffic still gets its unique layer 2, 3, and 4
	 queues and processing threads including NIC hardware resources
	 (when supported) so that the selected traffic can be separated
	 from others and can flow with minimal impact from other traffic.

         -t, --temporary

             The changes are temporary and will not persist across reboots.
	     Persistence is the dedault.


         -R root-dir, --root-dir=root-dir

             Specifies an alternate root  directory  where  flowadm should 
             apply persistent creation.


         -l <link>, --link=<link>

             Specify the link to which the flow will be added.


         -a attr=value,..., --attr attr=value,...
 
             A comma-separated list of attributes to be set  to  the
             specified values.


         -p prop=value,..., --prop prop=value,...
 
             A comma-separated list of properties to be set  to  the
             specified values.


     flowadm remove-flow [-t] [-R root-dir] {-l link | flow}

	 Remove an existing flows identified by its link or name. 

         -t, --temporary

             The changes are temporary and will not persist across reboots.


         -R root-dir, --root-dir=root-dir

             Specifies an alternate root  directory  where  flowadm should 
             apply persistent removal.


         -l <link>, --link=<link> | flow

             If a link is specified, remove all flows from that link.  If
             a single single is specified, remove only that flow.


     flowadm set-flowprop [-t] [-R root-dir] -p prop=value[,...] flow

	 Set values of one or more properties on the flow specifed by 
         name.  The complete list of properties can be retrieved using 
         show-flow

         -t, --temporary

             The changes are temporary and will not persist across reboots.


         -R root-dir, --root-dir=root-dir

             Specifies an alternate root  directory  where  flowadm should 
             apply persistent creation.

         -p prop=value,..., --prop prop=value,...
 
             A comma-separated list of properties to be set  to  the
             specified values.


     flowadm reset-flowprop [-t] [-R root-dir] -p prop, ... flow

         Resets one or more properties to their default values on the
         specified flow. If no properties are specified, all pro-
         perties are reset. See show-flowprop for  a  description of 
         properties which includes the default values

         -t, --temporary

             Specifies that the resets are  temporary.  Temporary
             resets last until the next reboot.

         -R root-dir, --root-dir=root-dir

             Specifies an alternate root  directory  where  flowadm
             should apply persistent creation.

         -p prop, ..., --prop=prop, ...

             A comma-separated list of properties to reset.


     flowadm show-flowprop [-cP] [-l link] [-p prop, ...] [flow]

         Show the current or persistent values  of  one  or  more
         properties,  either for all flows, flows on a specified link,
         or for the specified flow.
         By default, current values are shown. If no  properties are 
         specified, all available flow properties are displayed. For  
         each  property,  the  following  fields are displayed:
 
         FLOW        The name of the flow.

         PROPERTY    The name of the property.

         VALUE       The current (or persistent) property  value.
                     The  value  is  shown  as "--", if it is not
                     set, and "?", if the value is unknown.  Per-
                     sistent values that are not set or have been
                     reset will be shown as "--" and will use the
                     system DEFAULT value (if any).
 
         DEFAULT     The default value of the  property.  If  the
                     property  has  no  default  value,  "--"  is
                     shown.

         POSSIBLE    A comma-separated list  of  the  values  the
                     property  may  have.  If  the  values span a
                     numeric range, min - max  may  be  shown  as
                     shorthand.   If   the  possible  values  are
                     unknown or unbounded, "--" is shown.

         Flow properties are documented in the FLOW PROPERTIES section.

         -c, --parseable

             Display using a stable machine-parseable format.

         -P, --persistent

             Display persistent flow property information

         -p prop, ..., --prop=prop, ...

             A comma-separated list of properties to show.

     flowadm show-usage [-d] -f <filename> 
         [[-s <time>] [-e <time>] -p][-l <link> | <flow> ]
 
      
         Show the historical network flow usage from a stored extended
         accounting file.  Configuration and enabling of net accounting 
         through acctadm(1m) is required.  The default output will be
         the summary of flow usage for the entire period of time in
         which extended accounting was enabled.
  
         -d
 
         Display the dates for which there is logging information. The
	 date is in the format DD/MM/YYYY.

         -f <filename>
 
         Read extended accounting records of network flow usage from the
         <filename>. 
 
         -s <time>, -e <time>
 
         Start and stop times for data display.  Time is in the format
	 YYYY.MM.DD,hh:mm:ss.

         -p
 
         When specified with -s and/or -e, outputs flow usage data in a
	 form suitable for input to gnuplot.
 
         -l <link>

         If specified, only print the network flow usage from the named
         link.  Otherwise, display network usage from all links.

         <flow>
 
         If specified, only print the network flow usage from the named
         flow.  Otherwise, display network usage from all flows.
 
	  
  FLOW ATTRIBUTES

     The flow attribute that identify a flow is a comma separated list
     of one or more keyword,value pair from below

     ip_version={4|6}
             Version of IP protocol.
	     If unspecified, ip_version will be inferred from the local_ip
	     attribute. In the absence of local_ip and ip_version attributes,
	     the default ip_version will be considered as IPV4. 

      local_ip[/prefix_len]
 	      Identifies a network flow by the local IP address.
 	      <value> must be a IPv4 address in dotted-decimal notation
 	      or an IPv6 address in colon-separated notation.
	      prefix_len is optional.
	      If specified, it describes the netmask for a subnet address, 
	      following the same notation convention of ifconfig(1M) and
	      route(1M) addresses.
	      If unspecified, given IP address will be considered as
	      host address for which default prefix length for a
	      IPv4 address is /32 and for IPv6 it is /128.
		

     transport={tcp|udp|icmpv6|icmp}
	     Identifies a layer 4 protocol to be used. Is typically used
	     in combination with local_port to identify the service which 
 	     needs special attention.

     local_port
	     Identifies a service specified by the local port.


     dsfield
	     Identifies a value in the 8-bit differentiated services
	     field (as defined in RFC 2474).

     dsfield_mask
	     Specifies a mask to be used along with the dsfield value.
	     The mask is used to state the bits of interest in the
	     differentiated services field when comparing with the
	     dsfield value. A '0' in a bit position indicates that
	     the bit value needs to be ignored and a '1' indicates
	     otherwise. Thus, the mask can range from 1 to 255. When
	     a mask is not specified, the value is defaulted to 255,
	     i.e. check all the bits in the dsfield. A mask of 0 is
	     invalid.

     The following four types of combinations of attributes are supported:

	[ip_version={4|6}],local_ip[/prefixlen]=addess
	transport={tcp|udp|icmp|icmpv6}
	transport={tcp|udp},local_port=port
	dsfield=val,dsfield_mask=mask

	On a given link, the combinations above are mutually exclusive.
	An attempt to create flows of different types will fail.


 RESTRICTIONS
      Individual flow restrictions

          Restrictions on individual flows do not require knowledge of other
          flows that have been added to the link.

          Duplicate attributes
	       An attribute may only be listed once for each flow.

	       Not Valid:
	       #flowadm add-flow -l vnic1 -a local_port=80,local_port=8080
	                           httpflow

           transport and local_port
 	       TCP and UDP flows can be specified with a local 
 	       port.  An ICMPv6 flow which specifies a port will 
 	       not be allowed.

 	       If a local port is specifed, the transport must
 	       be either TCP or UDP.
 
 	       Valid:
 	       #flowadm add-flow -l e1000g0 -a transport=udp udpflow
 	       #flowadm add-flow -l e1000g0 -a transport=tcp,local_port=80
                                 udp80flow

 	       Not valid:
 	       #flowadm add-flow -l e1000g0 -a local_port=25 flow25
 	       #flowadm add-flow -l e1000g0 -a transport=icmpv6,local_port=16 \
		    flow16
  
      Flow restrictions per link

          Disjoint flows
 
 	  Two flows are considered disjoint if they specify a different
          value for the same attribute.  Disjoint flows are allowed.

	      #flowadm add-flow -l bge0 -a transport=tcp flow1
 	      #flowadm add-flow -l bge0 -a transport=udp flow2

          Enclosed flows
 
          
          A flow is considered to be enclosed by another flow if all of the 
          flow's packets belong  to the other flow.  The properties of the 
          flow which is more specific will be applied.  In the example, 
          all packets that match flow2 will also match flow1.  As flow2 is more 
          specific, its resource constraint (maxbw=200) will be used.

 	      #flowadm add-flow -l bge0 -a local_ip=129.146.108.0/24  \
		-p maxbw=300 flow1
 	      #flowadm add-flow -l bge0 -p local_ip=129.146.108.1 \
		-p maxbw=200 flow2
       

	  CPU bindings and fanout

	  A flow on a primary MAC client (i.e. bge0, e1000g1 etc), can have
	  a CPU list or fanout specified as per the following:

		o  If the underlying link has a CPU list specified, the
		   flow's CPU list must be equal to or a subset of the link's
		   CPU list.

		o  If the underlying link has a fanout, a flow can
		   have a fanout less than or equal to that of the link's
		   fanout. Moreover, a flow cannot have a CPU list
		   specified in this case.

		o  If the underlying link has a CPU list specified, a
		   flow can have a fanout that is less than or equal to
		   the number of CPUs in the link's CPU list.

		o  If the underlying link doesn't have a fanout or CPU
		   list, a flow can specify any fanout or CPU list.

	  A flow on a MAC client other than the primary (say a VNIC), cannot
	  have a fanout or CPU list.


      Flow restrictions per zone

          Flow names

 	        Within a zone, no 2 flows may have the same name.  Conversely,
                after adding the flow with the link specified, the link will
                not be required for display, modifcation or deletion of the 
                flow.

 FLOW PROPERTIES

     The following flow properties are supported. Note  that
     the ability to set a given property to a given value depends
     on the driver and hardware.
 

     maxbw   Sets the full duplex bandwidth for the flow. The
             bandwidth is specified as an integer with one of the
             scale suffixes(K, M, or G for Kbps, Mbps, and Gbps).
             If no units are specified, the input value will be 
             read as Mbps.  Default is no bandwidth limit.

 
     priority
             Sets the relative priority for the  flow. The value may
             be given as one of the tokens  high, medium, or low. 
             Default is medium. 

             

     cpus    Bind the processing of packets for a given flow to
             a processor or a set of processors. The value can be
             a comma separated list of one or more processor id.
             If the list consists of more than one processor,
             the processing will spread out to all the processors
             although connection to processor affinity and packet
             ordering for packets of any individual connection will
             be maintained.
 
             The processor or set of processors are not exclusively
             reserved  for the network device. Only the kernel
             threads and interrupt  associated with processing of
             the device are bound to the processor or the set of
             processor specified. In case it is desired that
             processors be dedicated to the device, psrset(1M) can
             be used to create a processor_set and then
             specifiying the processors from  the processor_set to
             bind the device to.
 
             If the flow was already bound to processor or set
             of processors  due to a previous operation, the binding
             will be removed and the  new set of processors will be
             used instead.

	     Processor (or sets) can be specified only for a flow that
	     is on a physical NIC (see RESTRICTIONS for more on this).

             The cpu binding property cannot be used with the fanout 
             property described below.

             Default is no cpu binding, which is to say that the 
             processing of packets for a flow is not bound to any 
             specific processor or processor set.


     fanout

             Specifies number of processors to be dedicated to packet 
             processing for a flow.

             This is similar but to the cpus but does not specify which
             cpus are to be used for packet processing, only the number.
             The specific cpus are determined internally and are not 
             persistent across reboot; only the number of cpus

             The fanout property cannot be used with the cpus property 
             described above.

             Default is no fanout.


EXAMPLES
	Example 1 Create a policy around mission critical port 443 traffic
	which is https service.


	To create a policy around inbound https traffic on a https server
	so that https gets it dedicated NIC hardware and kernel TCP/IP
	resources. The name specified is https-1 which is used to
	later modify of delete the policy.

	# flowadm add-flow -l bge0 -a transport=TCP local_port=443 https-1
        # flowadm show-flow -l bge0
        FLOW        LINK      SPEED    ATTR
        https-1     bge0      ---      transport=TCP
                                       local_port=443


	Example 2 Modify an existing policy to add bandwidth resource control


	To modify https-1 policy to add bandwidth control, give it a 
	high priority and bind the processing of the flow to CPU# 9
	
	# flowadm set-flowprop -p maxbw=500Mbps,priority=high,cpus=9 https-1
        # flowadm show-flow https-1
        FLOW        LINK      SPEED    ATTR
        https-1     bge0      500Mbps  transport=TCP
                                       local_port=443
        # flowadm show-flowprop https-1
        FLOW        PROPERTY    VALUE     DEFAULT      POSSIBLE
        https-1     maxbw       500       --           1000
        https-1     cpus        9         --           --
        https-1     priority    HIGH      NORMAL       LOW,NORMAL,HIGH



	Example 3 Limit the bandwidth usage of UDP protocol


	To create a policy for UDP protocol so that it can not consume more
	than 100Mbps of available bandwidth. The flow is named limit-udp-1.

	# flowadm add-flow -l bge0 -a transport=UDP -p maxbw=100Mbps
	 priority=low limit-udp-1


	Example 4 Show flow usage


	Flow usage statistics can be stored using the extended accounting
	facility acctadm(1m).

	# acctadm -e extended -f /var/log/net.log net

	# acctadm net
	    Network accounting: active
	    Network accounting file: /var/log/net.log
	    Tracked Network resources: extended
	    Untracked Network resources: none

	The historical data which was saved can be retrieved in summary
	form using the show-usage option of flowadm.


	Example 5 Set a policy for EF PHB (DSCP value of 101110 from
	RFC 2598) with bandwidth of 500 Mbps and high priority. The dsfield
	value for this flow will be 46 (101110) with the dsfield_mask
	being 252 (since we want to ignore the 2 least significant bits).


	# flowadm add-flow -l bge0 -a dsfield=46,dsfield_mask=252 
	 -p maxbw=500Mbps,priority=high efphb-flow


	Display summary information

	# flowadm show-usage -f /var/log/net.log
        FLOW      DURATION  IPACKETS RBYTES      OPACKETS OBYTES     BANDWIDTH
        flowtcp   100       1031     546908      0        0          43.76 Kbps
        flowudp   0         0        0           0        0           0.00 Mbps


	Display dates for which logging information is available

	# flowadm show-usage -d -f /var/log/net.log
	02/19/2008


	Display logging information for 'flowtcp' starting at
	02/19/2008,10:38:46 and ending at 02/19/2008,10:40:06

	# flowadm show-usage -s 02/19/2008,10:39:06 -e 02/19/2008,10:40:06
	    -f /var/log/net.log flowtcp
        FLOW      TIME       IPACKETS RBYTES      OPACKETS OBYTES     BANDWIDTH
        flowtcp   10:39:06   1        1546         4       6539       3.23 Kbps
	flowtcp   10:39:26   2        3586         5       9922       5.40 Kbps
	flowtcp   10:39:46   1        240          1       216       182.40 bps
	flowtcp   10:40:06   0        0            0       0           0.00 bps


	Output the same information as above as a plotfile

	# flowadm show-usage -s 02/19/2008,12:28:04 -e 02/19/2008,12:32:04
	    -p -f /var/log/net.log flowtcp
	# Time tcp-flow
	10:39:06 3.23
	10:39:26 5.40
	10:39:46 0.18
	10:40:06 0.00


EXIT STATUS
     The following exit	values are returned:
     0	      All actions were performed successfully.

     >0	      An error occurred.



ATTRIBUTES
     See attributes(5) for descriptions	of the	following  attri-
     butes:
     ____________________________________________________________
    |	    ATTRIBUTE TYPE	  |	  ATTRIBUTE VALUE	|
    |_____________________________|_____________________________|
    |_____________________________|_____________________________|
    | Interface	Stability	  | Evolving			|
    |_____________________________|_____________________________|

SEE ALSO
     dladm(1M), attributes(5), dlpi(7P), ifconfig(1M), route(1M),  acctadm(1m)