#pragma ident "@(#)flowadm.1m.txt 1.32 08/08/14 SMI" System Administration Commands flowadm(1M) NAME flowadm - administer bandwidth resource control and priority for protocols, services, Containers and Virtual machines SYNOPSIS flowadm show-flow [-pP] [-S] [-s [-i interval]] [-l link] [-o field,...] [flow] flowadm add-flow [-t] [-R root-dir] -l link -a attr=value[,...] -p prop=value[,...] flow flowadm remove-flow [-t] [-R root-dir] {-l link | flow} flowadm set-flowprop [-t] [-R root-dir] -p prop=value[,...] flow flowadm reset-flowprop [-t] [-R root-dir] [-p prop,...] flow flowadm show-flowprop [-cP] [-l link] [-o field,...] [-p prop,...] [flow] flowadm show-usage [-d] -f [[-s time] [-e time] -p ] [-l link] [flow] DESCRIPTION The flowadm command is used to create, modify, remove and show networking bandwidth and associated resources for a type of traffic on a particular link. CONCEPTS The flowadm command allows users to manage networking bandwidth resource for a transport, service, or a subnet. The service is specified as a combination of transport and local port. The subnet is specified by its IP address and subnet mask. The command can be used on any type of data link, including physical links, virtual NICs, and link aggregations. A flow is defined as a set of attributes based on Layer3 and 4 headers which can be used to identify a protocol, service, or a virtual machine. When a flow is identified based on flow attributes, separate kernel resources including layer 2, 3, and 4 queues, their processing threads, etc are uniquely created for it such that other traffic has minimal or zero impact on it. The processing of the packets for a given flow can be restricted to a given CPU or processor set such that all the resources necessary for processing the packets of the flow are dedicated to it. Inboud and outbound packet are matched to flows in a very fast and scalable way, so that limits can be enforced with minimal performance impact. The flowadm command can be used to identify a flow without imposing any bandwidth resource control. This would result in the traffic type getting its own resources and queues so that it is isolated from rest of the networking traffic for more obserbale and deterministic behavior. SUBCOMMANDS The following subcommands are supported: flowadm show-flow [-pP] [-S] [-s [-i interval]] [-o field,...] [-l link] [flow] Show flow configuration information (the default) or statistics, either for all flows, all flows on a link or for the specified name. -o field, --output=field A case-insensitive, comma-separated list of output fields to display. The field name must be one of the fields listed below, or a special value all, to display all fields. For each flow found, the following fields can be displayed: LINK The name of the link the flow is on. LOCAL_IP Local IP address of the flow. TRANSPORT The name of the layer for protocol to be used. IP_VERSION Version of the IP protocol for the flow LOCAL_PORT Local port of service for flow. DSFIELD Differentiated services value for flow. DSFIELD_MASK Mask used with DSFIELD value to stat the bits of intrestest in the differentiated services field of the IP header MAXBW Maximum bandwidth allowed for the flow in Mbps. -p, --parseable Display using a stable machine-parseable format. -P, --persistent Display persistent flow property information. -S, --continuous Continuously display network utilization by flow in a manner similar to the way that prstat(1m) displays cpu utilization by process. -s, --statistics Displays flow statistics. -i interval, --interval=interval Used with the -s option to specify an interval, in seconds, at which statistics should be displayed. If this option is not specified, statistics will only be displayed once. -l link, --link=link | flow Display information for all flows on the named link or information for the named flow. flowadm add-flow [-t] [-R root-dir] -l link -a attr=value[,...] -p prop=value[,...] flow Adds a flow to the system identified by its flow attributes and properties. As part of identifying a particular flow, its bandwidth resource can be limited and its relative priority to other traffic can be specified. If no bandwidth limit or priority is specified, the traffic still gets its unique layer 2, 3, and 4 queues and processing threads including NIC hardware resources (when supported) so that the selected traffic can be separated from others and can flow with minimal impact from other traffic. -t, --temporary The changes are temporary and will not persist across reboots. Persistence is the dedault. -R root-dir, --root-dir=root-dir Specifies an alternate root directory where flowadm should apply persistent creation. -l , --link= Specify the link to which the flow will be added. -a attr=value,..., --attr attr=value,... A comma-separated list of attributes to be set to the specified values. -p prop=value,..., --prop prop=value,... A comma-separated list of properties to be set to the specified values. flowadm remove-flow [-t] [-R root-dir] {-l link | flow} Remove an existing flows identified by its link or name. -t, --temporary The changes are temporary and will not persist across reboots. -R root-dir, --root-dir=root-dir Specifies an alternate root directory where flowadm should apply persistent removal. -l , --link= | flow If a link is specified, remove all flows from that link. If a single flow is specified, remove only that flow. flowadm set-flowprop [-t] [-R root-dir] -p prop=value[,...] flow Set values of one or more properties on the flow specifed by name. The complete list of properties can be retrieved using show-flow -t, --temporary The changes are temporary and will not persist across reboots. -R root-dir, --root-dir=root-dir Specifies an alternate root directory where flowadm should apply persistent creation. -p prop=value,..., --prop prop=value,... A comma-separated list of properties to be set to the specified values. flowadm reset-flowprop [-t] [-R root-dir] [-p prop, ...] flow Resets one or more properties to their default values on the specified flow. If no properties are specified, all pro- perties are reset. See show-flowprop for a description of properties which includes the default values -t, --temporary Specifies that the resets are temporary. Temporary resets last until the next reboot. -R root-dir, --root-dir=root-dir Specifies an alternate root directory where flowadm should apply persistent creation. -p prop, ..., --prop=prop, ... A comma-separated list of properties to reset. flowadm show-flowprop [-cP] [-l link] [-p prop, ...] [flow] Show the current or persistent values of one or more properties, either for all flows, flows on a specified link, or for the specified flow. By default, current values are shown. If no properties are specified, all available flow properties are displayed. For each property, the following fields are displayed: FLOW The name of the flow. PROPERTY The name of the property. VALUE The current (or persistent) property value. The value is shown as "--", if it is not set, and "?", if the value is unknown. Per- sistent values that are not set or have been reset will be shown as "--" and will use the system DEFAULT value (if any). DEFAULT The default value of the property. If the property has no default value, "--" is shown. POSSIBLE A comma-separated list of the values the property may have. If the values span a numeric range, min - max may be shown as shorthand. If the possible values are unknown or unbounded, "--" is shown. Flow properties are documented in the FLOW PROPERTIES section. -c, --parseable Display using a stable machine-parseable format. -P, --persistent Display persistent flow property information -p prop, ..., --prop=prop, ... A comma-separated list of properties to show. flowadm show-usage [-d] -f filename [[-s time] [-e time] -p][-l link ] [flow] Show the historical network flow usage from a stored extended accounting file. Configuration and enabling of net accounting through acctadm(1m) is required. The default output will be the summary of flow usage for the entire period of time in which extended accounting was enabled. -d Display the dates for which there is logging information. The date is in the format DD/MM/YYYY. -f filename Read extended accounting records of network flow usage from the filename. -s time, -e time Start and stop times for data display. Time is in the format YYYY.MM.DD,hh:mm:ss. -p When specified with -s and/or -e, outputs flow usage data in a form suitable for input to gnuplot. -l link If specified, only print the network flow usage from the named link. Otherwise, display network usage from all links. flow If specified, only print the network flow usage from the named flow. Otherwise, display network usage from all flows. FLOW ATTRIBUTES The flow attribute that identify a flow is a comma separated list of one or more keyword,value pair from below local_ip[/prefix_len] Identifies a network flow by the local IP address. must be a IPv4 address in dotted-decimal notation or an IPv6 address in colon-separated notation. prefix_len is optional. If specified, it describes the netmask for a subnet address, following the same notation convention of ifconfig(1M) and route(1M) addresses. If unspecified, given IP address will be considered as host address for which default prefix length for a IPv4 address is /32 and for IPv6 it is /128. remote_ip[/prefix_len] Identifies a network flow by the remote IP address. The syntax is the same as local_ip attribute's transport={tcp|udp|sctp|icmp|icmpv6} Identifies a layer 4 protocol to be used. Is typically used in combination with local_port to identify the service which needs special attention. local_port Identifies a service specified by the local port. dsfield[:dsfield_mask] Identifies the 8-bit differentiated services field (as defined in RFC 2474). The optional dsfield_mask is used to state the bits of interest in the differentiated services field when comparing with the dsfield value. A '0' in a bit position indicates that the bit value needs to be ignored and a '1' indicates otherwise. The mask can range from 0x01 to 0xff. If dsfield_mask is not specified, the default mask 0xff is used. Both dsfield value and mask must be in hexadecimal. The following five types of combinations of attributes are supported: local_ip[/prefixlen]=address remote_ip[/prefixlen]=address transport={tcp|udp|sctp|icmp|icmpv6} transport={tcp|udp|sctp},local_port=port dsfield=val[:dsfield_mask] On a given link, the combinations above are mutually exclusive. An attempt to create flows of different types will fail. RESTRICTIONS Individual flow restrictions Restrictions on individual flows do not require knowledge of other flows that have been added to the link. Duplicate attributes An attribute may only be listed once for each flow. Not Valid: #flowadm add-flow -l vnic1 -a local_port=80,local_port=8080 httpflow transport and local_port TCP, UDP, or SCTP flows can be specified with a local port. ICMP or ICMPv6 flow which specifies a port will not be allowed. If a local port is specifed, the transport must be either TCP, UDP or SCTP. Valid: #flowadm add-flow -l e1000g0 -a transport=udp udpflow #flowadm add-flow -l e1000g0 -a transport=tcp,local_port=80 udp80flow Not valid: #flowadm add-flow -l e1000g0 -a local_port=25 flow25 #flowadm add-flow -l e1000g0 -a transport=icmpv6,local_port=16 \ flow16 Flow restrictions per link CPU bindings and fanout A flow on a primary MAC client (i.e. bge0, e1000g1 etc), can have a CPU list or fanout specified as per the following: o If the underlying link has a CPU list specified, the flow's CPU list must be equal to or a subset of the link's CPU list. o If the underlying link has a fanout, a flow can have a fanout less than or equal to that of the link's fanout. Moreover, a flow cannot have a CPU list specified in this case. o If the underlying link has a CPU list specified, a flow can have a fanout that is less than or equal to the number of CPUs in the link's CPU list. o If the underlying link doesn't have a fanout or CPU list, a flow can specify any fanout or CPU list. Flow restrictions per zone Flow names Within a zone, no 2 flows may have the same name. Conversely, after adding the flow with the link specified, the link will not be required for display, modifcation or deletion of the flow. FLOW PROPERTIES The following flow properties are supported. Note that the ability to set a given property to a given value depends on the driver and hardware. maxbw Sets the full duplex bandwidth for the flow. The bandwidth is specified as an integer with one of the scale suffixes(K, M, or G for Kbps, Mbps, and Gbps). If no units are specified, the input value will be read as Mbps. Default is no bandwidth limit. priority Sets the relative priority for the flow. The value may be given as one of the tokens high, medium, or low. Default is medium. cpus Bind the processing of packets for a given flow to a processor or a set of processors. The value can be a comma separated list of one or more processor id. If the list consists of more than one processor, the processing will spread out to all the processors although connection to processor affinity and packet ordering for packets of any individual connection will be maintained. The processor or set of processors are not exclusively reserved for the network device. Only the kernel threads and interrupt associated with processing of the device are bound to the processor or the set of processor specified. In case it is desired that processors be dedicated to the device, psrset(1M) can be used to create a processor_set and then specifiying the processors from the processor_set to bind the device to. If the flow was already bound to processor or set of processors due to a previous operation, the binding will be removed and the new set of processors will be used instead. Processor (or sets) can be specified only for a flow that is on a physical NIC (see RESTRICTIONS for more on this). The cpu binding property cannot be used with the fanout property described below. Default is no cpu binding, which is to say that the processing of packets for a flow is not bound to any specific processor or processor set. fanout Specifies the number of processors to be dedicated to packet processing for a flow. This is similar to the cpus but does not specify which cpus are to be used for packet processing, only the number. The specific cpus are determined internally and are not persistent across reboot; only the number of cpus persists. The fanout property cannot be used with the cpus property described above. Default is no fanout. EXAMPLES Example 1 Create a policy around mission critical port 443 traffic which is https service. To create a policy around inbound https traffic on a https server so that https gets it dedicated NIC hardware and kernel TCP/IP resources. The name specified is https-1 which is used to later modify of delete the policy. # flowadm add-flow -l bge0 -a transport=TCP,local_port=443 https-1 # flowadm show-flow -l bge0 FLOW LINK SPEED ATTR https-1 bge0 --- transport=TCP local_port=443 Example 2 Modify an existing policy to add bandwidth resource control To modify https-1 policy to add bandwidth control, give it a high priority and bind the processing of the flow to CPU# 9 # flowadm set-flowprop -p maxbw=500Mbps,priority=high,cpus=9 https-1 # flowadm show-flow https-1 FLOW LINK SPEED ATTR https-1 bge0 500Mbps transport=TCP local_port=443 # flowadm show-flowprop https-1 FLOW PROPERTY VALUE DEFAULT POSSIBLE https-1 maxbw 500 -- 1000 https-1 cpus 9 -- -- https-1 priority HIGH NORMAL LOW,NORMAL,HIGH Example 3 Limit the bandwidth usage of UDP protocol To create a policy for UDP protocol so that it can not consume more than 100Mbps of available bandwidth. The flow is named limit-udp-1. # flowadm add-flow -l bge0 -a transport=UDP -p maxbw=100Mbps priority=low limit-udp-1 Example 4 Show flow usage Flow usage statistics can be stored using the extended accounting facility acctadm(1m). # acctadm -e extended -f /var/log/net.log net # acctadm net Network accounting: active Network accounting file: /var/log/net.log Tracked Network resources: extended Untracked Network resources: none The historical data which was saved can be retrieved in summary form using the show-usage option of flowadm. Example 5 Set a policy for EF PHB (DSCP value of 101110 from RFC 2598) with bandwidth of 500 Mbps and high priority. The dsfield value for this flow will be 0x2e (101110) with the dsfield_mask being 0xfc (since we want to ignore the 2 least significant bits). # flowadm add-flow -l bge0 -a dsfield=0x2e:0xfc -p maxbw=500Mbps,priority=high efphb-flow Display summary information # flowadm show-usage -f /var/log/net.log FLOW DURATION IPACKETS RBYTES OPACKETS OBYTES BANDWIDTH flowtcp 100 1031 546908 0 0 43.76 Kbps flowudp 0 0 0 0 0 0.00 Mbps Display dates for which logging information is available # flowadm show-usage -d -f /var/log/net.log 02/19/2008 Display logging information for 'flowtcp' starting at 02/19/2008,10:38:46 and ending at 02/19/2008,10:40:06 # flowadm show-usage -s 02/19/2008,10:39:06 -e 02/19/2008,10:40:06 -f /var/log/net.log flowtcp FLOW TIME IPACKETS RBYTES OPACKETS OBYTES BANDWIDTH flowtcp 10:39:06 1 1546 4 6539 3.23 Kbps flowtcp 10:39:26 2 3586 5 9922 5.40 Kbps flowtcp 10:39:46 1 240 1 216 182.40 bps flowtcp 10:40:06 0 0 0 0 0.00 bps Output the same information as above as a plotfile # flowadm show-usage -s 02/19/2008,12:28:04 -e 02/19/2008,12:32:04 -p -f /var/log/net.log flowtcp # Time tcp-flow 10:39:06 3.23 10:39:26 5.40 10:39:46 0.18 10:40:06 0.00 EXIT STATUS The following exit values are returned: 0 All actions were performed successfully. >0 An error occurred. ATTRIBUTES See attributes(5) for descriptions of the following attri- butes: ____________________________________________________________ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | |_____________________________|_____________________________| |_____________________________|_____________________________| | Interface Stability | Evolving | |_____________________________|_____________________________| SEE ALSO dladm(1M), attributes(5), dlpi(7P), ifconfig(1M), route(1M), acctadm(1m)