--- old/./arp.1m.txt Fri Dec 19 04:18:42 2008 +++ new/./arp.1m.txt Fri Dec 19 04:18:42 2008 @@ -118,11 +118,13 @@ -d Delete an entry for the host called hostname. + + Note that ARP entries for IPMP data and test addresses + are managed by the kernel, and thus cannot be deleted. - SunOS 5.11 Last change: 25 Jul 2006 2 @@ -167,7 +169,26 @@ using arp -s. This is useful in certain SLIP confi- gurations. + Non-temporary proxy ARP entries for an IPMP group are + automatically managed by the kernel. Specifically, if + the hardware address in an entry matches the hardware + address of an IP interface in an IPMP group, and the + IP address is not local to the system, this will be + regarded as an IPMP proxy ARP entry. This entry will + have its hardware address automatically adjusted in + order to keep the IP address reachable so long as the + IPMP group has not entirely failed. + ARP entries must be consistent across an IPMP group. + Therefore, ARP entries cannot be associated with + individual underlying IP interfaces in an IPMP group, + and must instead be associated with the corresponding + IPMP IP interface. + + Note that ARP entries for IPMP data and test addresses + are managed by the kernel, and thus cannot be changed. + + ATTRIBUTES See attributes(5) for descriptions of the following attri- butes: --- old/./arp.7p.txt Fri Dec 19 04:18:44 2008 +++ new/./arp.7p.txt Fri Dec 19 04:18:44 2008 @@ -116,7 +116,7 @@ The address family for the [x]arp_pa sockaddr must be AF_INET. The ATF_COM flag bits ([x]arp_flags) cannot be - altered. ATF_USETRAILER is not implemented on Solaris and + altered. ATF_USETRAILERS is not implemented on Solaris and is retained for compatibility only. ATF_PERM makes the entry permanent (disables aging) if the ioctl() request succeeds. ATF_PUBL specifies that the system should respond @@ -171,6 +171,31 @@ interface. The information returned is very similar to that returned via routing sockets on an RTM_IFINFO message. + The ARP ioctls have several additional restrictions and + enhancements when used in conjunction with IPMP: + + * The ARP mappings for IPMP data and test addresses are + managed by the kernel, and thus cannot be changed + through any ARP ioctls, though they may be retrieved + using *SIOCGARP* or *SIOCGXARP*. + + * ARP mappings for a given IPMP group must be consistent + across the group. Therefore, ARP mappings cannot be + associated with individual underlying IP interfaces in + an IPMP group, and must instead be associated with the + corresponding IPMP IP interface. + + * Proxy ARP mappings for an IPMP group are automatically + managed by the kernel. Specifically, if the hardware + address in a *SIOCSARP* or *SIOCSXARP* request matches + the hardware address of an IP interface in an IPMP + group, and the IP address is not local to the system, + the kernel will regard this as a IPMP Proxy ARP entry. + This IPMP Proxy ARP entry will have its hardware + address automatically adjusted in order to keep the IP + address reachable (provided the IPMP group has not + entirely failed). + ARP performs duplicate address detection for local addresses. When a logical interface is brought up (IFF_UP) or any time the hardware link goes up (IFF_RUNNING), ARP --- old/./dhcpagent.txt Fri Dec 19 04:18:45 2008 +++ new/./dhcpagent.txt Fri Dec 19 04:18:45 2008 @@ -73,22 +73,23 @@ downed link is brought back up. - For IPv4, if the configured interface is found to be - unplumbed, marked down, or to have a different IP address, - subnet mask, or broadcast address from those obtained from - DHCP, the interface is abandoned by DHCP control. + For IPv4, if the configured interface is found to be + unplumbed, or to have a different IP address, subnet mask, + or broadcast address from those obtained from DHCP, the + interface is abandoned from DHCP control. - For IPv6, dhcpagent automatically plumbs and unplumbs logi- - cal interfaces as necessary for the IPv6 addresses supplied - by the server. The IPv6 prefix length (netmask) is not set - by the DHCPv6 protocol, but is instead set by in.ndpd(1M) - using prefix information obtained by Router Advertisements. - If any of the logical interfaces created by dhcpagent is - unplumbed, marked down, or configured with a different IP - address, it will be abandoned by DHCP control. If the link- - local interface is unplumbed, then all addresses configured - by DHCP on that physical interface will be removed. + For IPv6, dhcpagent automatically plumbs and unplumbs + logical interfaces as necessary for the IPv6 addresses + supplied by the server. The IPv6 prefix length (netmask) is + not set by the DHCPv6 protocol, but is instead set by + *in.ndpd(1M)* using prefix information obtained by Router + Advertisements. If any of the logical interfaces created by + dhcpagent is unplumbed, or configured with a different IP + address, it will be abandoned from DHCP control. If the + link-local interface is unplumbed, then all addresses + configured by DHCP on that physical interface will be + removed. In addition to DHCP, dhcpagent also supports BOOTP (IPv4 @@ -741,11 +742,11 @@ IPv6 (DHCPv6). Cisco Systems. July 2003. NOTES - The dhcpagent daemon can be used on IPv4 logical interfaces, - just as with physical interfaces. When used on a logical - interface, the daemon automatically constructs a Client ID - value based on the DUID and IAID values, according to RFC - 4361. The /etc/default/dhcpclient CLIENT_ID value, if any, + DHCP can be performed on IPv4 logical interfaces just as + with physical interfaces. When used on a logical interface, + the daemon automatically constructs a Client ID value based + on the DUID and IAID values, according to RFC 4361. The + */etc/default/dhcpclient* *CLIENT_ID* value, if any, overrides this automatic identifier. @@ -757,6 +758,30 @@ interfaces. + DHCP can be performed on IPMP IP interfaces to acquire and + maintain IPMP data addresses. Because an IPMP IP interface + has no hardware address, the daemon automatically constructs + a Client ID using the same approach described above for IPv4 + logical interfaces. In addition, the lack of a hardware + address means the daemon must set the "broadcast" flag in + all *DISCOVER* and *REQUEST* messages on IPMP IP interfaces. + Some DHCP servers may refuse such requests. + + + DHCP can be performed on IP interfaces that are part of an + IPMP group (to acquire and maintain test addresses). The + daemon will automatically set the *NOFAILOVER* and + *DEPRECATED* flags on each test address. Additionally, the + daemon will not add or remove default routes in this case. + Note that the actual DHCP packet exchange may be performed + over any active IP interface in the IPMP group. It is + strongly recommended that test addresses have infinite + leases. Otherwise, an extended network outage detectable + only by probes may cause test address leases to expire, + causing *in.mpathd(1M)* to revert to link-based failure + detection and trigger an erroneous repair. + + With DHCPv6, the link-local interface must be configured using /etc/hostname6.hme0 in order for DHCPv6 to run on hme0 at boot time. The logical interfaces for each address are --- old/./getsockopt.3socket.txt Fri Dec 19 04:18:47 2008 +++ new/./getsockopt.3socket.txt Fri Dec 19 04:18:47 2008 @@ -25,14 +25,17 @@ most "socket" level. - When manipulating socket options, the level at which the - option resides and the name of the option must be specified. - To manipulate options at the "socket" level, level is speci- - fied as SOL_SOCKET. To manipulate options at any other - level, level is the protocol number of the protocol that - controls the option. For example, to indicate that an option - is to be interpreted by the TCP protocol, level is set to - the TCP protocol number. See getprotobyname(3SOCKET). + The *level* argument specifies the protocol level at which + the option resides. To manipulate options at the socket + level, specify the *level* argument as *SOL_SOCKET*. To + manipulate options at the protocol level, supply the + appropriate protocol number for the protocol controlling the + option. For example, to indicate that an option will be + interpreted by TCP, set *level* to the protocol number of + TCP, as defined in the ** header, or as + determined by using *getprotobyname(3SOCKET)*. Some socket + protocol families may also define additional levels, such as + *SOL_ROUTE*. Only socket-level options are described here. The parameters optval and optlen are used to access option --- old/./if_mpadm.txt Fri Dec 19 04:18:48 2008 +++ new/./if_mpadm.txt Fri Dec 19 04:18:48 2008 @@ -3,92 +3,79 @@ NAME - if_mpadm - change operational status of interfaces within a - multipathing group + if_mpadm - administer interfaces in an IP multipathing group SYNOPSIS - /usr/sbin/if_mpadm -d interface_name + if_mpadm -d | -r ifname - - /usr/sbin/if_mpadm -r interface_name - - DESCRIPTION - Use the if_mpadm utility to change the operational status of - interfaces that are part of an IP multipathing group. If the - interface is operational, you can use if_mpadm -d to detach - or off-line the interface. If the interface is off-lined, - use if_mpadm -r to revert it to its original state. + The *if_mpadm* utility administers IP interfaces that are + part of an IP multipathing (IPMP) group. Currently, + administration is limited to offlining IP interfaces and + undoing previous offline operations. + When an IP interface is taken offline, all IP data traffic + that was flowing over the IP interface is moved to another + IP interface in the IPMP group. In addition, all *UP* IP + addresses hosted on the IP interface are brought down, + causing *in.mpathd(1M)* to stop probe-based failure + detection on the IP interface. As a result, an offline IP + interface will not be used for any inbound or outbound IP + traffic. Only IP interfaces that are in an IPMP group may + be brought offline. If the IP interface is the last + functioning interface in the IPMP group, the offline + operation will fail. + + When an offline operation is undone, any IP addresses hosted + on that IP interface are brought *UP* and will be considered + by *in.mpathd* for probe-based failure detection. In + addition, provided the IP interface is otherwise active (see + *in.mpathd(1M)*), it will again be used to send and receive + IP data traffic for the IPMP group. Note that not all + offline operations can be undone. For instance, *in.mpathd* + may have offlined an IP interface because its hardware + address was not unique within its IPMP group. The + *ipmpstat* utility can be used to determine why an IP + interface is offline, identify which IP interfaces in a + group are being used for inbound and outbound IP traffic, + and more; see *ipmpstat(1M)*. - When a network interface is off-lined, all network access - fails over to a different interface in the IP multipathing - group. Any addresses that do not failover are brought down. - Network access includes unicast, broadcast, and multicast - for IPv4 and unicast and multicast for IPv6. Addresses - marked with IFF_NOFAILOVER do not failover. They are marked - down. After an interface is off-lined, the system will not - use the interface for any outbound or inbound traffic, and - the interface can be safely removed from the system without - any loss of network access. - - - The if_mpadm utility can be applied only to interfaces that - are part of an IP multipathing group. - OPTIONS - The if_mpadm utility supports the following options: + The *if_mpadm* utility supports the following options: - -d interface_name Detach or off-line the interface speci- - fied by interface_name. + -d ifname Offline the IP interface specified by + *ifname*. If *ifname* is not in an + IPMP group, or the offline would cause + the IPMP group to lose network + connectivity, the operation will fail. + -r ifname Undo a previous offline of the IP + interface specified by *ifname*. If + *ifname* is not offline, the operation + will fail. - -r interface_name Reattach or undo the previous detach - or off-line operation on the interface - specified by interface_name. Unless the - -d option was used to detach or off- - line the interface, this option will - fail. - EXAMPLES - Example 1 Detaching an Interface + Example 1 Offlining an IP Interface + The following command offlines IP interface *under0*, + causing any IP packets that were being sent and received + through it to be handled by another IP interface in its + group. + example% if_mpadm -d under0 -SunOS 5.11 Last change: 3 Sep 2002 1 + Example 2 Undoing a Previous Offline Operation + Use the following command to undo the previous operation: + example% if_mpadm -r under0 -System Administration Commands if_mpadm(1M) - - - Use the following command to off-line or detach the inter- - face. All network access will failover from hme0 to other - interfaces in the same IP multipathing group. If no other - interfaces are in the same group, the operation will fail. - - - example% if_mpadm -d hme0 - - - - Example 2 Reattaching an Off-line Interface - - - Use the following command to undo the previous operation. - Network access will failback to hme0. - - - example% if_mpdadm -r hme0 - - - ATTRIBUTES See attributes(5) for descriptions of the following attri- butes: @@ -108,88 +95,19 @@ ifconfig(1M), in.mpathd(1M), attributes(5) DIAGNOSTICS - off-line failed as there is no other functional interface - available in the multipathing group for failing over the - network access. + cannot offline: no other functioning interfaces are in its + IPMP group. + Description: + This message means that offlining the IP interface would + leave the IPMP group without network connectivity. - This message means that other interfaces in the group - are failed over already or the multipathing configura- - tion was not suitable for completing a failover. + cannot offline: not a physical interface or not in an IPMP + group - - - - - -SunOS 5.11 Last change: 3 Sep 2002 2 - - - - - - -System Administration Commands if_mpadm(1M) - - - - off-line cannot be undone because multipathing configuration - is not consistent across all the interfaces in the group. Description: - - This message means that some interfaces in the IP mul- - tipathing group are not configured consistently with - other interfaces in the group, for example, one of the - interfaces in the group does not have an IFF_NOFAILOVER - address. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -SunOS 5.11 Last change: 3 Sep 2002 3 - - - + This means that the IP interface is not an underlying + interface in an IPMP group, and therefore is not + eligible to be offlined. --- old/./if_nameindex.txt Fri Dec 19 04:18:50 2008 +++ new/./if_nameindex.txt Fri Dec 19 04:18:50 2008 @@ -86,8 +86,8 @@ and errno would be set to the proper value, for example, ENOMEM. - - *if_nameindex() The if_nameindex() function returns an + + if_nameindex() The if_nameindex() function returns an array of if_nameindex structures, one structure per interface. The if_nameindex structure holds the @@ -97,9 +97,14 @@ struct if_nameindex unsigned int if_index; /* 1, 2, ... */ - char *if_name; /* null terminated name: "eri0", ... */ + char *if_name; /* "net0", ... */ }; + While any IPMP IP interfaces are + returned by *if_nameindex()*, the + underlying IP interfaces that comprise + each IPMP group are not returned. + The end of the array of structures is indicated by a structure with an if_index of 0 and an if_name of NULL. --- old/./if_tcp.txt Fri Dec 19 04:18:51 2008 +++ new/./if_tcp.txt Fri Dec 19 04:18:51 2008 @@ -31,6 +31,10 @@ destinations on the network of the interface is installed automatically when an interface's address is set. + Note IPMP IP interfaces cannot be created using the + procedure described above. Instead, ifconfig(1M) must be + used to create IPMP IP interfaces. + IOCTLS The following ioctl() calls may be used to manipulate IP network interfaces. Unless specified otherwise, the request @@ -100,40 +104,53 @@ SIOCGLIFFLAGS Get interface flags. - SIOCGLIFCONF Get interface configuration list. This - request takes an lifconf structure (see + SIOCGLIFCONF Get interface configuration list. This + request takes a *lifconf* structure (see + below) as a value-result parameter. The + *lifc_family* field can be set to + *AF_UNSPEC* to retrieve both *AF_INET* + and *AF_INET6* interfaces. The + *lifc_len* field should be set to the + size of the buffer pointed to by + *lifc_buf*. -SunOS 5.10 Last change: 14 Nov 2008 2 + The *lifc_flags* field should usually be + set to zero, but callers that need low- + level knowledge of the underlying IP + interfaces that comprise an IPMP group + can set it to *LIFC_UNDER_IPMP* to + request that those interfaces be + included in the result. + + Upon success, *lifc_len* will contain + the length, in bytes, of the array of + *lifreq* structures pointed to by + *lifc_req*. For each *lifreq* structure, + the *lifr_name* and *lifr_addr* fields + will be valid. -Protocols if_tcp(7P) - below) as a value-result parameter. The - lifc_family field can be set to - AF_UNSPEC to retrieve both AF_INET and - AF_INET6 interfaces. The lifc_flags - field should be set to zero. The - lifc_len field should be set to the size - of the buffer pointed to by lifc_buf. - Upon success, lifc_len will contain the - length, in bytes, of the array of lifreq - structures pointed to by lifc_req. For - each lifreq structure, the lifr_name and - lifr_addr fields will be valid. - SIOCGLIFNUM Get number of interfaces. This request returns an integer which is the number of interface descriptions (struct lifreq) that will be returned by the SIOCGLIFCONF ioctl; that is, it gives an - indication of how large lifc_len has to - be. This request takes an lifnum struc- - ture (see below) as a value-result - parameter. The lifn_family field should - be set to AF_UNSPEC to count both - AF_INET and AF_INET6 interfaces. The - lifn_flags field should be initially set - to zero. + indication of how large lifc_len has + to be. + This request takes a *struct lifnum* + (see below) as a value-result parameter. + The *lifn_family* field can be set to + *AF_UNSPEC* to count both *AF_INET* and + *AF_INET6* interfaces. The *lifn_flags* + field should usually be set to zero, but + callers that need low-level knowledge of + the underlying IP interfaces that + comprise an IPMP group can set it to + *LIFC_UNDER_IPMP* to request that those + interfaces be included in the count. + + SIOCSLIFMTU Set the maximum transmission unit (MTU) size for interface. Place the request in the lifru_mtu field. The MTU can not @@ -176,21 +193,11 @@ SIOCLIFADDIF Add a new logical interface on a physi- cal interface using an unused logical - unit number. If the physical interface - is part of an IP multipathing group, the - logical interface may be added to a dif- - ferent physical interface in the same - group. Upon return, the lifr_name field - contains the name of the actual logical - interface created. + interface number. SIOCLIFREMOVEIF Remove a logical interface by specifying its IP address or logical interface - name. When the IP address is specified - and the interface is part of an IP mul- - tipathing group, the logical interface - is removed from the physical interface - in the group which holds the IP address. + name. SIOCSLIFTOKEN Set the address token used to form IPv6 link-local addresses and for stateless @@ -318,6 +325,7 @@ }; The structure used by SIOCGLIFNUM has the form: + struct lifnum { sa_family_t lifn_family; int lifn_flags; /* req. specf. interfaces */ @@ -401,8 +409,12 @@ success, ifc_len will contain the length, in bytes, of the array of ifreq struc- tures pointed to by ifc_req. For each - ifreq structure, the ifr_name and - ifr_addr fields will be valid. + ifreq structure, the ifr_name and + *ifr_addr* fields will be valid. While + any IPMP IP interfaces will be included + in the array, the underlying IP + interfaces that comprise those IPMP + groups will not be. SunOS 5.10 Last change: 14 Nov 2008 8 @@ -413,7 +425,11 @@ interface descriptions (struct ifreq) that will be returned by the SIOCGIFCONF ioctl; that is, it gives an indication of - how large ifc_len has to be. + how large *ifc_len* has to be. While any + IPMP IP interfaces will be included in + the array, the underlying IP interfaces + that comprise those IPMP groups will not + be. SIOCSIFMTU Set the maximum transmission unit (MTU) size for interface. Place the request in @@ -468,7 +484,7 @@ flags listed below (with the leading IFF_ prefix removed). See the ifconfig(1M) manpage for a definition of each flag. - #define IFF_UP 0x0000000001 /* Interface is up */ + #define IFF_UP 0x0000000001 /* Address is up */ #define IFF_BROADCAST 0x0000000002 /* Broadcast address valid */ #define IFF_DEBUG 0x0000000004 /* Turn on debugging */ #define IFF_LOOPBACK 0x0000000008 /* Loopback net */ @@ -490,7 +506,7 @@ #define IFF_NOXMIT 0x0000010000 /* Do not transmit pkts */ #define IFF_NOLOCAL 0x0000020000 /* No address - just on-link subnet */ - #define IFF_DEPRECATED 0x0000040000 /* Interface addr. deprecated */ + #define IFF_DEPRECATED 0x0000040000 /* Address is deprecated */ #define IFF_ADDRCONF 0x0000080000 /* Addr. from stateless addrconf */ #define IFF_ROUTER 0x0000100000 /* Router on interface */ @@ -500,17 +516,12 @@ #define IFF_IPV4 0x0001000000 /* IPv4 interface */ #define IFF_IPV6 0x0002000000 /* IPv6 interface */ - #define IFF_NOFAILOVER 0x0008000000 /* No failover on NIC fail. */ - #define IFF_FAILED 0x0010000000 /* NIC has failed */ + #define IFF_NOFAILOVER 0x0008000000 /* in.mpathd test address */ + #define IFF_FAILED 0x0010000000 /* Interface has failed */ - #define IFF_STANDBY 0x0020000000 /* Standby NIC-use on fail. */ - #define IFF_INACTIVE 0x0040000000 /* Stndby active or not? */ - #define IFF_OFFLINE 0x0080000000 /* NIC offlined */ - -SunOS 5.10 Last change: 14 Nov 2008 10 - -Protocols if_tcp(7P) - + #define IFF_STANDBY 0x0020000000 /* Interface is a hot-spare */ + #define IFF_INACTIVE 0x0040000000 /* Functioning but not used */ + #define IFF_OFFLINE 0x0080000000 /* Interface is offline */ #define IFF_XRESOLV 0x0100000000 /* IPv6 external resolver */ #define IFF_COS_ENABLED 0x0200000000 /* If CoS marking is supported */ @@ -520,6 +531,7 @@ #define IFF_VIRTUAL 0x2000000000 /* Cannot send/receive pkts */ #define IFF_DUPLICATE 0x4000000000 /* Local address in use */ + #define IFF_IPMP 0x8000000000 /* IPMP IP interface */ ERRORS EPERM Calling process has insufficient privileges. --- old/./ifconfig.txt Fri Dec 19 04:18:53 2008 +++ new/./ifconfig.txt Fri Dec 19 04:18:53 2008 @@ -1,6 +1,3 @@ - - - System Administration Commands ifconfig(1M) @@ -17,7 +14,7 @@ [broadcast address] [deprecated | -deprecated] [preferred | -preferred] [destination dest_address] [ether [address]] [failover | -failover] [group - [name | ""]] [index if_index] [metric n] [modlist] + [name | ""]] [index if_index] [ipmp] [metric n] [modlist] [modinsert mod_name@pos] [modremove mod_name@pos] [mtu n] [netmask mask] [plumb] [unplumb] [private | -private] [nud | -nud] [set [address] [/netmask]] @@ -81,12 +78,8 @@ addif address Create the next unused logical interface on the speci- - fied physical interface. If the physical interface is - part of a multipathing group, the logical interface can - be added to a different physical interface in the same - group. + fied physical interface. - all-zones Make the interface available to every shared-IP zone on @@ -140,6 +133,7 @@ -arp Disable the use of the ARP on a physical interface. + ARP cannot be disabled on an IPMP IP interface. auth_algs authentication algorithm @@ -479,7 +473,7 @@ ether [ address ] If no address is given and the user is root or has suf- - ficient privileges to open the underlying device, then + ficient privileges to open the underlying datalink, then display the current Ethernet address information. Otherwise, if the user is root or has sufficient @@ -490,54 +484,52 @@ over InfiniBand) interfaces, the address will be 20 bytes of colon-separated hex numbers between 0 and FF. - Some, though not all, Ethernet interface cards have + Some, though not all, Ethernet interface cards have their own addresses. To use cards that do not have their - own addresses, refer to section 3.2.3(4) of the IEEE + own addresses, refer to section 3.2.3(4) of the IEEE 802.3 specification for a definition of the locally - administered address space. The use of multipathing - groups should be restricted to those cards with their - own addresses (see MULTIPATHING GROUPS). + administered address space. Note that all IP interfaces + in an IPMP group must have unique hardware addresses; + see *in.mpathd(1M)*. -failover - Mark the logical interface as a non-failover interface. - Addresses assigned to non-failover logical interfaces - will not failover when the interface fails. Status - display shows NOFAILOVER as part of flags. + Set *NOFAILOVER* on the logical interface. This makes + the associated address available for use by *in.mpathd* + to perform probe-based failure detection for the + associated physical IP interface. As a side effect, + *DEPRECATED* will also be set on the logical interface. + This operation is not permitted on an IPMP IP interface. failover - Mark the logical interface as a failover interface. An - address assigned to such an interface will failover when - the interface fails. Status display does not show - NOFAILOVER as part of flags. + Clear *NOFAILOVER* on the logical interface. This is + the default. These logical interfaces are subject to + migration when brought up (see IP MULTIPATHING GROUPS). - group [ name |""] + group [ name | "" ] - Insert the logical interface in the multipathing group - specified by name. To delete an interface from a group, + When applied to a physical interface, it places the + interface into the named group. If the group does not + exist, it will be created, along with one or more IPMP + IP interfaces (for IPv4, IPv6, or both). Any *UP* + addresses that are not also marked *NOFAILOVER* are + subject to migration to the IPMP IP interface (see IP + MULTIPATHING GROUPS). Specifying a group name of *""* + removes the physical IP interface from the group. + When applied to a physical IPMP IP interface, it renames + the IPMP group to have the new name. If the name + already exists, or a name of *""* is specified, it + fails. Renaming IPMP groups is discouraged. Instead, + the IPMP IP interface should be given a meaningful name + when it is created via the *ipmp* subcommand, which the + system will also use as the IPMP group name. -SunOS 5.11 Last change: 21 Jan 2007 8 - - - - - - -System Administration Commands ifconfig(1M) - - - - use a null string "". When invoked on the logical inter- - face with id zero, the status display shows the group - name. - - index n Change the interface index for the interface. The value @@ -547,6 +539,15 @@ interface on the system. + ipmp + + Create an IPMP IP interface with the specified name. An + interface must be separately created for use by IPv4 and + IPv6. The *address_family* parameter controls whether + the command applies to IPv4 or IPv6 (IPv4 if + unspecified). All IPMP IP interfaces have the *IPMP* + flag set. + metric n Set the routing metric of the interface to n; if no @@ -739,18 +740,21 @@ plumb - Open the device associated with the physical interface - name and set up the streams needed for IP to use the - device. When used with a logical interface name, this - command is used to create a specific named logical - interface. An interface must be separately plumbed for - use by IPv4 and IPv6. The address_family parameter con- - trols whether the ifconfig command applies to IPv4 or - IPv6. + For a physical IP interface, open the datalink + associated with the physical interface name and set up + the plumbing needed for IP to use the datalink. When + used with a logical interface name, this command is used + to create a specific named logical interface on an + existing physical IP interface. - Before an interface has been plumbed, the interface will - not show up in the output of the ifconfig -a command. + An interface must be separately plumbed for IPv4 and + IPv6 according to the *address_family* parameter (IPv4 + if unspecified). Before an interface has been plumbed, + it will not be shown by *ifconfig -a*. + Note that IPMP IP interfaces are not tied to a specific + datalink and are instead created with the *ipmp* + subcommand. private @@ -766,23 +770,23 @@ removeif address Remove the logical interface on the physical interface - specified that matches the address specified. When the - interface is part of a multipathing group, the logical - interface will be removed from the physical interface in - the group that holds the address. + specified that matches the address specified. - router Enable IP forwarding on the interface. When enabled, the - interface is marked ROUTER, and IP packets can be for- - warded to and from the interface. + interface is marked *ROUTER*, and IP packets can be for- + warded to and from the interface. Enabling *ROUTER* on + any IP interface in an IPMP group applies the flag to + all IP interfaces in that IPMP group. -router - Disable IP forwarding on the interface. IP packets are - not forwarded to and from the interface. + Disable IP forwarding on the interface. IP packets are + not forwarded to and from the interface. Disabling + *ROUTER* on any IP interface in an IPMP group disables + it on all IP interfaces in that IPMP group. @@ -805,24 +809,19 @@ standby - Marks the physical interface as a standby interface. If - the interface is marked STANDBY and is part of the mul- - tipathing group, the interface will not be selected to - send out packets unless some other interface in the - group has failed and the network access has been failed - over to this standby interface. + Mark the physical IP interface as a *STANDBY* interface. + If an interface is marked *STANDBY* and is part of an + IPMP group, the interface will not be used for data + traffic unless another interface in the IPMP group + becomes unusable. When a *STANDBY* interface is + functional but not being used for data traffic, it will + also be marked *INACTIVE*. This operation is not + permitted on an IPMP IP interface. - The status display shows "STANDBY, INACTIVE" indicating - that that the interface is a standby and is also inac- - tive. IFF_INACTIVE will be cleared when some other - interface belonging to the same multipathing group fails - over to this interface. Once a failback happens, the - status display will return to INACTIVE. - -standby - Turns off standby on this interface. + Clear *STANDBY* on this interface. This is the default. subnet @@ -894,20 +893,26 @@ unplumb - Close the device associated with this physical interface - name and any streams that ifconfig set up for IP to use - the device. When used with a logical interface name, the - logical interface is removed from the system. After this - command is executed, the device name will no longer - appear in the output of ifconfig -a. + For a physical or IPMP interface, remove all associated + logical IP interfaces and tear down any plumbing needed + for IP to use the interface. For an IPMP IP interface, + this command will fail if the group is not empty. For a + logical interface, the logical interface is removed. + An interface must be separately unplumbed for IPv4 and + IPv6 according to the *address_family* parameter (IPv4 + if unspecified). Upon success, the interface name will + no longer appear in the output of *ifconfig -a*. + up - Mark a logical interface "up". This happens automati- - cally when assigning the first address to a logical - interface. The up option enables an interface after an - ifconfig down, which reinitializes the hardware. + Mark a logical interface *UP*. As a result, the IP + module will accept packets destined to the associated + address (unless the address is zero), along with any + associated multicast and broadcast IP addresses. + Similarly, the IP module will allow packets to be sent + with the associated address as a source address. usesrc [ name | none ] @@ -973,17 +978,13 @@ (route subcommand), in that order. The use of the usesrc option is mutually exclusive of - the IP multipathing ifconfig options, group and standby. - That is, if an interface is already part of a IP mul- - tipathing group or specified as a standby interface, - then it cannot be specified with a usesrc option, and - vice-versa. For more details on IP multipathing, see - in.mpathd(1M) and the . + the IPMP *group* and *standby* subcommands. That is, if + an interface is already part of a IPMP group or + specified as a *STANDBY* interface, then it cannot be + specified with a usesrc option, and vice-versa. - - SunOS 5.11 Last change: 21 Jan 2007 15 @@ -1238,7 +1239,9 @@ This interface supports some form of Class of Service (CoS) marking. An example is the 802.1D user priority - marking supported on VLAN interfaces. + marking supported on VLAN interfaces. For IPMP IP + interfaces, this will only be set if all interfaces in + the group have CoS set. DEPRECATED @@ -1245,35 +1248,19 @@ This address is deprecated. This address will not be used as a source address for outbound packets unless - - - -SunOS 5.11 Last change: 21 Jan 2007 19 - - - - - - -System Administration Commands ifconfig(1M) - - - there are no other addresses on this interface or an - application has explicitly bound to this address. An - IPv6 deprecated address will eventually be deleted when - not used, whereas an IPv4 deprecated address is often - used with IP network multipathing IPv4 test addresses, - which are determined by the setting of the NOFAILOVER - flag. Further, the DEPRECATED flag is part of the stan- - dard mechanism for renumbering in IPv6. + application has explicitly bound to this address. An + IPv6 deprecated address is part of the standard + mechanism for renumbering in IPv6 and will eventually be + deleted when not used. For both IPv4 and IPv6, + *DEPRECATED* is also set on all *NOFAILOVER* addresses, + though this may change in a future release. + DHCPRUNNING - DHCP + The logical interface is managed by *dhcpagent(1M)*. - DHCP is used to manage this address. - DUPLICATE The logical interface has been disabled because the IP @@ -1292,10 +1279,14 @@ FAILED - The interface has failed. New addresses cannot be - created on this interface. If this interface is part of - an IP network multipathing group, a failover will occur - to another interface in the group, if possible + The *in.mpathd* daemon has determined that the interface + has failed. *FAILED* interfaces will not be used to + send or receive IP data traffic. If this is set on a + physical IP interface in an IPMP group, IP data traffic + will continue to flow over other usable IP interfaces in + the IPMP group. If this is set on an IPMP IP interface, + the entire group has failed and no data traffic can be + sent or received over any interfaces in that group. FIXEDMTU @@ -1309,40 +1300,25 @@ INACTIVE - Indicates that the interface is not currently being used - for regular traffic by the system. New addresses cannot + The physical interface is functioning but is not used to + send or receive data traffic according to administrative + policy. This flag is initially set by the *standby* + subcommand and is subsequently controlled by + *in.mpathd*. It also set when *FAILBACK=no* mode is + enabled (see *in.mpathd(1M)*) to indicate that the IP + interface has repaired but is not being used. + IPMP -SunOS 5.11 Last change: 21 Jan 2007 20 + Indicates that this is an IPMP IP interface. - - - - -System Administration Commands ifconfig(1M) - - - - be created on this interface. The flag is set automati- - cally on standby interfaces. It can also be set when the - system detects that a failed interface has been repaired - and FAILBACK=no is configured in /etc/default/mpathd. - The flag is cleared when the interface fails or when a - failover to that interface occurs. - - LOOPBACK Indicates that this is the loopback interface. - MIP - - Indicates that mobile IP controls this interface. - - MULTI_BCAST Indicates that the broadcast address is used for multi- @@ -1366,9 +1342,9 @@ NOFAILOVER - This address will not failover if the interface fails. - IP network multipathing test addresses must be marked - nofailover. + The address associated with this logical interface is + available to *in.mpathd* for probe-based failure + detection of the associated physical IP interface. NOLOCAL @@ -1419,11 +1395,9 @@ OFFLINE - Indicates that the interface has been offlined. New - addresses cannot be created on this interface. Inter- - faces in an IP network multipathing group are offlined - prior to removal and replacement using dynamic reconfi- - guration. + The interface is offline and thus cannot send or receive + IP data traffic. This is only set on IP interfaces in + an IPMP group. See *if_mpadm(1M)* and *cfgadm(1M)*. POINTOPOINT @@ -1473,20 +1447,19 @@ RUNNING - Indicates that the required resources for an interface + Indicates that the required resources for an i nterface are allocated. For some interfaces this also indicates - that the link is up. + that the link is up. For IPMP IP interfaces, *RUNNING* + is set as long as one IP interface in the group is + active. STANDBY - Indicates that this is a standby interface to be used on - failures. Only interfaces in an IP network multipathing - group should be designated as standby interfaces. If - this interface is part of a IP network multipathing - group, the interface will not be selected to send out - packets unless some other interface in the group fails - over to it. + Indicates that this physical interface will not be used + for data traffic unless another interface in the IPMP + group becomes unusable. The *INACTIVE* and *FAILED* + flags indicate whether it is actively being used. TEMPORARY @@ -1504,14 +1477,15 @@ UP - Indicates that the interface is up, that is, all the - routing entries and the like for this interface have - been set up. + Indicates that the logical interface (and the associated + physical interface) is up. The IP module will accept + packets destined to UP addresses (unless the address is + zero), along with any associated multicast and broadcast + IP addresses. Similarly, the IP module will allow + packets to be sent with an UP address as a source + address. - - - SunOS 5.11 Last change: 21 Jan 2007 23 @@ -1529,7 +1503,7 @@ hardware. It is not possible to transmit or receive packets through a virtual interface. These interfaces are useful for configuring local addresses that can be - used on multiple interfaces. (See also the -usesrc + used on multiple interfaces. (See also the *usesrc* option.) @@ -1600,31 +1574,84 @@ eri0 has been plumbed. - To delete a logical interface, use the -unplumb or -removeif - options. For example, + To delete a logical interface, use the *unplumb* or + *removeif* options. For example, example% ifconfig eri0:1 down unplumb + will delete the logical interface *eri0:1*. +IP MULTIPATHING GROUPS + Physical interfaces that share the same IP broadcast domain + _must_ be collected into a single IP Multipathing (IPMP) + group using the *group* subcommand. Each IPMP group has an + associated IPMP IP interface, which can either be explicitly + created (the preferred method) by using the *ipmp* + subcommand or implicitly created by *ifconfig* in response + to placing an IP interface into a new IPMP group. + Implicitly-created IPMP interfaces will be named ipmp_N_ + where _N_ is the lowest integer that doesn't conflict with + an existing IP interface name or IPMP group name. - will delete the logical interface eri0:1. + Each IPMP IP interface is created with a matching IPMP group + name, though it can be changed using the *group* subcommand. + Each IPMP IP interface hosts a set of highly-available IP + addresses. These addresses will remain reachable so long as + at least one interface in the group is active, where + "active" is defined as having at least one UP address and + having *INACTIVE*, *FAILED*, and *OFFLINE* clear. IP + addresses hosted on the IPMP IP interface may either be + configured statically or configured through DHCP via the + *dhcp* subcommand. -MULTIPATHING GROUPS - Physical interfaces that share the same IP broadcast domain - can be collected into a multipathing group using the group - keyword. Interfaces assigned to the same multipathing group - are treated as equivalent and outgoing traffic is spread - across the interfaces on a per-IP-destination basis. In - addition, individual interfaces in a multipathing group are - monitored for failures; the addresses associated with failed - interfaces are automatically transferred to other function- - ing interfaces within the group. + Interfaces assigned to the same IPMP group are treated as + equivalent and monitored for failure by *in.mpathd*. + Provided that active interfaces in the group remain, IP + interface failures (and any subsequent repairs) are handled + transparently to sockets-based applications. IPMP is also + integrated with the Dynamic Reconfiguration framework (see + *cfgadm(1M)*), which enables network adapters to be replaced + transparently to sockets-based applications. + + The IP module automatically load-spreads all outbound + traffic across all active interfaces in an IPMP group. + Similarly, all *UP* addresses hosted on the IPMP IP + interface and will be distributed across the active + interfaces to promote inbound load-spreading. The + *ipmpstat(1M)* utility allows many aspects of the IPMP + subsystem to be observed, including the current binding of + IP data addresses to IP interfaces. + When an interface is placed into an IPMP group, any *UP* + logical interfaces are "migrated" to the IPMP IP interface + for use by the group, unless: - For more details on IP multipathing, see in.mpathd(1M) and - the . See netstat(1M) for per-IP-destination information. + * The logical interface is marked *NOFAILOVER* + * The logical interface hosts an IPv6 link-local address. + * The logical interface hosts an IPv4 0.0.0.0 address. + Likewise, once an interface is in a group, if changes are + made to a logical interface such that it is *UP* and not + exempted by one of the conditions above, it will also + migrate to the associated IPMP IP interface. Logical + interfaces never migrate back, even if the physical + interface that contributed the address is removed from the + group. + + Each interface placed into an IPMP group may be optionally + configured with a "test" address that *in.mpathd* will use + for probe-based failure detection; see *in.mpathd(1M)*. + These addresses must be marked *NOFAILOVER* (using the + *-failover* subcommand) prior to being marked *UP*. Test + addresses may also be acquired through DHCP via the *dhcp* + subcommand. + + For more background on IPMP, please see the "IPMP + Administrative Overview" and "IPMP Configuration Tasks" + chapters of the administrator documentation. + + CONFIGURING IPV6 INTERFACES When an IPv6 physical interface is plumbed and configured "up" with ifconfig, it is automatically assigned an IPv6 @@ -2097,7 +2124,7 @@ The following command, using the none keyword, undoes the - effect of the preceding ifconfig usersrc command. + effect of the preceding *ifconfig* *usesrc* command. example% ifconfig qfe2 usesrc none @@ -2329,7 +2356,7 @@ SEE ALSO dhcpinfo(1), dhcpagent(1M), in.mpathd(1M), in.ndpd(1M), - in.routed(1M), ipsecconf(1M), ndd(1M), netstat(1M), + in.routed(1M), ipmpstat(1M), ipsecconf(1M), netstat(1M), zoneadm(1M), zonecfg(1M), ethers(3SOCKET), gethostbyname(3NSL), getnetbyname(3SOCKET), hosts(4), inet_type(4), ndpd.conf(4), netmasks(4), networks(4), --- old/./in.mpathd.1m.txt Fri Dec 19 04:18:56 2008 +++ new/./in.mpathd.1m.txt Fri Dec 19 04:18:55 2008 @@ -3,8 +3,7 @@ NAME - in.mpathd - daemon for network adapter (NIC) failure detec- - tion, recovery, automatic failover and failback + in.mpathd - IP multipathing daemon SYNOPSIS /usr/lib/inet/in.mpathd @@ -11,28 +10,30 @@ DESCRIPTION - The in.mpathd daemon performs Network Interface Card (NIC) - failure and repair detection. In the event of a NIC failure, - it causes IP network access from the failed NIC to failover - to a standby NIC, if available, or to any another opera- - tional NIC that has been configured as part of the same net- - work multipathing group. Once the failed NIC is repaired, - all network access is restored to the repaired NIC. + The *in.mpathd* daemon performs failure and repair detection + for IP interfaces that have been placed into an IPMP group + (or optionally, for all IP interfaces on the system). It + also controls which IP interfaces in an IPMP group are + "active" (being used by the system to send or receive IP + data traffic) in a manner which is consistent with the + administrator's configured policy. - The in.mpathd daemon can detect NIC failure and repair - through two methods: by monitoring the IFF_RUNNING flag for - each NIC (link-based failure detection), and by sending and - receiving ICMP echo requests and replies on each NIC - (probe-based failure detection). Link-based failure detec- - tion requires no explicit configuration and thus is always - enabled (provided the NIC driver supports the feature); - probe-based failure detection must be enabled through the - configuration of one or more test addresses (described - below), but has the benefit of testing the entire NIC send - and receive path. + The *in.mpathd* daemon can detect IP interface failure and + repair through two methods: by monitoring the *IFF_RUNNING* + flag for each IP interface (link-based failure detection), + and by sending and receiving ICMP probes on each IP + interface (probe-based failure detection). Link-based + failure detection is instantaneous and is always enabled + (provided the network driver supports the feature); + probe-based failure detection must be enabled through the + configuration of one or more test addresses (described + below), but tests the entire IP interface send and receive + path. The *ipmpstat(1M)* utility can be used to check which + failure detection methods are enabled. + If only link-based failure detection is enabled, then the health of the interface is determined solely from the state of the IFF_RUNNING flag. Otherwise, the interface is con- @@ -42,118 +43,98 @@ configured with the same failure detection methods. - As mentioned above, in order to perform probe-based failure - detection in.mpathd needs a special test address on each NIC - for the purpose of sending and receiving probes on the NIC. - Use the ifconfig command -failover option to configure these - test addresses. See ifconfig(1M). The test address must - belong to a subnet that is known to the hosts and routers on - the link. + As mentioned above, to perform probe-based failure detection + *in.mpathd* requires a test address on each IP interface for + the purpose of sending and receiving probes. Each address + must be marked *NOFAILOVER* (see *ifconfig(1M)*) and + *in.mpathd* will be limited to probing targets on the same + subnet. Each address may be configured statically or + acquired via DHCP. To find targets, *in.mpathd* first + consults the routing table for routes on the same subnet, + and uses the specified next-hop. If no routes match, it + sends all-hosts ICMP probes and selects a subset of the + systems that respond. Thus, for probe-based failure + detection to operate, there must be at least one neighbor on + each subnet that responds to ICMP echo request probes. The + *ipmpstat(1M)* utility can be used to display both the + current probe target information and the status of sent + probes. + + Both IPv4 and IPv6 are supported. If an IP interface is + plumbed for IPv4 and an IPv4 test address is configured then + *in.mpathd* will start sending ICMPv4 probes over that IP + interface. Similarly, if an IP interface is plumbed for + IPv6 and an IPv6 test address is configured then *in.mpathd* + will start sending ICMPv6 probes over that IP interface. + However, note that *in.mpathd* will ignore IPv6 test + addresses that are not link-local. If both IPv4 and IPv6 + are plumbed, it is sufficient to configure only one of the + two, that is, either an IPv4 test address or an IPv6 test + address. If both IPv4 and IPv6 test addresses are + configured, *in.pathd* probes using both ICMPv4 and ICMPv6. - The in.mpathd daemon can detect NIC failure and repair by - two methods, by sending and receiving ICMP echo requests and - replies on each NIC, and by monitoring the IFF_RUNNING flag + As mentioned above, *in.mpathd* also controls which IP + interfaces in an IPMP group are "active" (used by the system + to send and receive IP data traffic). Specifically, + *in.mpathd* tracks the administrative configuration of each + IPMP group and attempts to keep the number of active IP + interfaces in each group consistent with that configuration. + Therefore, if an active IP interface fails, *in.mpathd* will + activate an *INACTIVE* interface in the group, provided one + exists (it will prefer *INACTIVE* interfaces that are also + marked *STANDBY*). Likewise, if an IP interface repairs and + the resulting repair leaves the IPMP group with more active + interfaces than the administrative configuration specifies, + *in.mpathd* will deactivate one of the interfaces + (preferably one marked *STANDBY*), except when the + *FAILBACK* variable is used, as described below. Similar + adjustments will be made by *in.mpathd* when offlining IP + interfaces (for instance, in response to *if_mpadm(1M)*). -SunOS 5.11 Last change: 8 Sep 2006 1 - - - - - - -System Administration Commands in.mpathd(1M) - - - - for each NIC. The link state on some models of NIC is indi- - cated by the IFF_RUNNING flag, allowing for faster failure - detection when the link goes down. The in.mpathd daemon con- - siders a NIC to have failed if either of the above two - methods indicates failure. A NIC is considered to be - repaired only if both methods indicate the NIC is repaired. - - - The in.mpathd daemon sends the ICMP echo request probes to - on-link routers. If no routers are available, it sends the - probes to neighboring hosts. Thus, for network failure - detection and repair, there must be at least one neighbor on - each link that responds to ICMP echo request probes. - - - in.mpathd works on both IPv4 and IPv6. If IPv4 is plumbed on - a NIC, an IPv4 test address is configured on theNIC, and the - NIC is configured as part of a network multipathing group, - then in.mpathd will start sending ICMP probes on the NIC - using IPv4. - - - In the case of IPv6, the link-local address must be config- - ured as the test address. The in.mpathd daemon will not - accept a non-link-local address as a test address. If the - NIC is part of a multipathing group, and the test address - has been configured, then in.mpathd will probe the NIC for - failures using IPv6. - - - Even if both the IPv4 and IPv6 protocol streams are plumbed, - it is sufficient to configure only one of the two, that is, - either an IPv4 test address or an IPv6 test address on a - NIC. If only an IPv4 test address is configured, it probes - using only ICMPv4. If only an IPv6 test address is config- - ured, it probes using only ICMPv6. If both type test - addresses are configured, it probes using both ICMPv4 and - ICMPv6. - - The in.mpathd daemon accesses three variable values in /etc/default/mpathd: FAILURE_DETECTION_TIME, FAILBACK and TRACK_INTERFACES_ONLY_WITH_GROUPS. - The FAILURE_DETECTION_TIME variable specifies the NIC - failure detection time for the ICMP echo request probe - method of detecting NIC failure. The shorter the failure - detection time, the greater the volume of probe traffic. The - default value of FAILURE_DETECTION_TIME is 10 seconds. This - means that NIC failure will be detected by in.mpathd within - 10 seconds. NIC failures detected by the IFF_RUNNING flag + The *FAILURE_DETECTION_TIME* variable specifies the + probe-based failure detection time. The shorter the failure + detection time, the more probe traffic. The default value + of *FAILURE_DETECTION_TIME* is 10 seconds. This means that + IP interface failure will be detected by *in.mpathd* within + 10 seconds. The IP interface repair detection time is + always twice the value of *FAILURE_DETECTION_TIME*. Note + that failures and repairs detected by link-based failure + detection are acted on immediately, though *in.mpathd* may + ignore link state changes if it suspects that the link state + is flapping due to defective hardware; see DIAGNOSTICS. + By default, *in.mpathd* limits failure and repair detection + to IP interfaces that are configured as part of a named IPMP + group. Setting *TRACK_INTERFACES_ONLY_WITH_GROUPS* to *no* + enables failure and repair detection on all IP interfaces, + even if they are not part of a named IPMP group. IP + interfaces that are tracked but not part of a named IPMP + group are considered to be part of the "anonymous" IPMP + group. In addition to having no name, this IPMP group is + special in that its IP interfaces are not equivalent and + thus cannot take over for one another in the event of an IP + interface failure. That is, the anonymous IPMP group can + only be used for failure and repair detection, and provides + no high-availability or load-spreading. -SunOS 5.11 Last change: 8 Sep 2006 2 + As described above, when *in.mpathd* detects that an IP + interface has repaired, it activates it so that it will + again be used to send and receive IP data traffic. However, + if *FAILBACK* is set to *no*, then the IP interface will + only be activated if no other active IP interfaces in the + group remain. However, the interface may subsequently be + activated if another IP interface in the group fails. - - - - -System Administration Commands in.mpathd(1M) - - - - being cleared are acted on as soon as the in.mpathd daemon - notices the change in the flag. The NIC repair detection - time cannot be configured; however, it is defined as double - the value of FAILURE_DETECTION_TIME. - - - By default, in.mpathd does failure detection only on NICs - that are configured as part of a multipathing group. You can - set TRACK_INTERFACES_ONLY_WITH_GROUPS to no to enable - failure detection by in.mpathd on all NICs, even if they are - not part of a multipathing group. However, in.mpathd cannot - do failover from a failed NIC if it is not part of a mul- - tipathing group. - - - The in.mpathd daemon will restore network traffic back to - the previously failed NIC, after it has detected a NIC - repair. To disable this, set the value of FAILBACK to no in - /etc/default/mpathd. - FILES /etc/default/mpathd Contains default values used by the in.mpathd daemon. @@ -173,22 +154,39 @@ SEE ALSO - ifconfig(1M), attributes(5), icmp(7P), icmp6(7P), + ifconfig(1M), ipmpstat(1M), if_mpadm(1M), icmp(7P), icmp6(7P) DIAGNOSTICS - Test address address is not unique; disabling probe based - failure detection on interface_name + IP interface *interface_name* has a hardware address which + is not unique in group *group_name*; offlining Description: + For probe-based failure detection, load-spreading, and + other code IPMP features to work properly, each IP + interface in an IPMP group must have a unique hardware + address. If this requirement is not met, *in.mpathd* + will automatically offline all but one of the IP + interfaces with duplicate hardware addresses. - For in.mpathd to perform probe-based failure detection, - each test address in the group must be unique. Since the - IPv6 test address is a link-local address derived from - the MAC address, each IP interface in the group must + IP interface *interface_name* now has a unique hardware + address in group *group_name*; onlining + Description: + The previously-detected duplicate hardware address is + now unique, and therefore *in.mpathd* has brought + *interface_name* back online. + Test address *address* is not unique in group; disabling + probe-based failure detection on *interface_name* + Description: + + + For in.mpathd to perform probe-based failure detection, + each test address in the group must be unique. + + SunOS 5.11 Last change: 8 Sep 2006 3 @@ -200,10 +198,16 @@ - have a unique MAC address. + No test address configured on interface *interface_name*; + disabling probe-based failure detection on it + Description: + For *in.mpathd* to perform probe-based failure detection + on an IP interface, it must be configured with a test + address: IPv4, IPv6, or both. + NIC interface_name of group group_name is not plumbed for IPv[4|6] and may affect failover capability Description: @@ -212,34 +216,23 @@ All NICs in a multipathing group must be homogeneously plumbed. For example, if a NIC is plumbed for IPv4, then all NICs in the group must be plumbed for IPv4. The - streams modules pushed on all NICs must be identical. + STREAMS modules pushed on all NICs must also be identical. - - No test address configured on interface interface_name disa- - bling probe-based failure detection on it - Description: - - - In order for in.mpathd to perform probe-based failure - detection on a NIC, it must be configured with a test - address: IPv4, IPv6, or both. - - - The link has come up on interface_name more than 2 times in - the last minute; disabling failback until it stabilizes. + the last minute; disabling repair until it stabilizes. Description: - In order to prevent interfaces with intermittent - hardware, such as a bad cable, from causing repeated - failovers and failbacks, in.mpathd does not failback to - interfaces with frequently fluctuating link states. + To limit the impact of interfaces with intermittent + hardware (such as a bad cable), *in.mpathd* will not + consider an IP interface with a frequently changing link + state as repaired until the link state stabilizes. - Invalid failure detection time assuming default 10000 + Invalid failure detection time of *time*, assuming default + of 10000 ms Description: @@ -248,8 +241,8 @@ - Too small failure detection time of time assuming minimum - 100 + Too small failure detection time of *time*, assuming minimum + of 100 ms Description: @@ -315,7 +308,7 @@ - NIC failure detected on interface_name + IP interface failure detected on interface_name Description: @@ -332,48 +325,28 @@ - in.mpathd has detected NIC failure on interface_name, - and has set the IFF_FAILED flag on NIC interface_name. + *in.mpathd* has detected a failure on *interface_name*, + and has set the *IFF_FAILED* flag on *interface_name*, + ensuring that it will not be used for IP data traffic. - - Successfully failed over from NIC interface_name1 to NIC - interface_name2 + IP interface repair detected on *interface_name* Description: - in.mpathd has caused the network traffic to failover - from NIC interface_name1 to NIC interface_name2, which - is part of the multipathing group. + *in.mpathd* has detected a repair on *interface_name*, + and has cleared the *IFF_FAILED* flag. Depending on the + administrative configuration, the *interface_name* may + again be used for IP data traffic. - - NIC repair detected on interface_name - Description: - - - in.mpathd has detected that NIC interface_name is - repaired and operational. If the IFF_FAILED flag on the - NIC was previously set, it will be reset. - - - - Successfully failed back to NIC interface_name - Description: - - - in.mpathd has restored network traffic back to NIC - interface_name, which is now repaired and operational. - - - The link has gone down on interface_name Description: - in.mpathd has detected that the IFF_RUNNING flag for NIC - interface_name has been cleared, indicating the link has - gone down. + *in.mpathd* has detected that the *IFF_RUNNING* flag for + *interface_name* has been cleared, indicating the link + has gone down. @@ -381,8 +354,8 @@ Description: - in.mpathd has detected that the IFF_RUNNING flag for NIC - interface_name has been set, indicating the link has + *in.mpathd* has detected that the *IFF_RUNNING* flag for + *interface_name* has been set, indicating the link has come up. --- /dev/null Fri Dec 19 04:18:57 2008 +++ new/./ipmpstat.1m.txt Fri Dec 19 04:18:56 2008 @@ -0,0 +1,399 @@ +System Administration Commands ipmpstat(1M) + +NAME + ipmpstat - display IPMP subsystem status + +SYNOPSIS + + ipmpstat [-n] [-o _field_ [-P]] -a|-g|-i|-p|-t + +DESCRIPTION + + The *ipmpstat* command concisely displays information about + the IPMP subsystem. It supports five different output + modes, each of which provides a different view of the IPMP + subsystem (address, group, interface, probe, and target), + described below. At most one output mode may be specified + per invocation, and the displayed information is guaranteed + to be self-consistent. It also provides a parsable output + format which may be used by scripts to examine the state of + the IPMP subsystem. Only basic privileges are needed to + invoke ipmpstat, with the exception of probe mode which + requires all privileges. + +OPTIONS + + -a + + Display IPMP data address information ("address" + output mode). + + -g + + Display IPMP group information ("group" output mode). + + -i + + Display IP interface information ("interface" output + mode). + + -n + + Display IP addresses numerically, rather than + attempting to resolve them to hostnames. This option + may be used in any output mode. + + + -o _field_[,...] + + Display only the specified output fields, in order. + The list of field names is case-insensitive and + comma-separated. The field names that are supported + depend on the selected output mode, described below. + The special field name "all" may be used to print all + fields for a given output mode. + + -p + + Display IPMP probe information ("probe" output mode). + + -t + + Display IPMP target information ("target" output mode). + + -P + + Display using a machine-parsable format, described + below. If this option is specified, an explicit list + of fields must be specified using the *-o* option. + + +OUTPUT MODES + + Address Mode + + Address mode displays the state of all IPMP data + addresses on the system. The following output fields are + supported: + + ADDRESS The hostname (or IP address) associated with + the information. Note that because duplicate + down addresses may exist, the address must be + taken together with the GROUP to form a + unique identity. For a given IPMP group, if + duplicate addresses exist, at most one will + be displayed, and an up address will always + take precedence. + + STATE The state of the address. Either *up* if the + address is IFF_UP (see ifconfig(1M)), or + *down* if the address is not IFF_UP. + + GROUP The IPMP IP interface hosting the address. + + INBOUND The underlying IP interface that will receive + packets for this address. This may change in + response to external events such as IP + interface failure. If this field is empty, + then the system will not accept IP packets + sent to this address (e.g., because the + address is down or because there are no + active IP interfaces left in the IPMP group). + + OUTBOUND The underlying IP interfaces that will send + packets using this source address. This may + change in response to external events such as + IP interface failure. If this field is + empty, then the system will not send packets + with this address as a source (e.g., because + the address is down or because there are no + active IP interfaces left in the IPMP group). + + If *-o* is not specified, all output fields are displayed. + + Group Mode + + Group mode displays the state of all IPMP groups on the + system. The following output fields are supported: + + GROUP The IPMP IP interface name associated with + the information. For the anonymous group + (see *in.mpathd(1M)*), this field will be + empty. + + GROUPNAME The IPMP group name. For the anonymous + group, this field will be empty. + + STATE The state of the group: + + *ok* All interfaces in the group are + usable. + *degraded* Some (but not all) interfaces in + the group are usable. + *failed* No interfaces in the group are + usable. + + FDT The probe-based failure detection time. If + probe-based failure detection is disabled, + this field will be empty. + + INTERFACES The list of underlying IP interfaces in the + group. The list is divided into three parts: + + 1. Active interfaces are listed first and not + enclosed in any brackets or parenthesis. + Active interfaces are those being used by + the system to send or receive data traffic. + + 2. *INACTIVE* interfaces are listed next and + enclosed in parenthesis. *INACTIVE* + interfaces are those that are functioning, + but not being used according to + administrative policy. + + 3. Unusable interfaces are listed last and + enclosed in brackets. Unusable interfaces + are those that cannot be used at all in + their present configuration (e.g., + *FAILED* or *OFFLINE*). + + If *-o* is not specified, all output fields are displayed. + + Interface Mode + + Interface mode displays the state of all IP interfaces + that are tracked by *in.mpathd* on the system. The + following output fields are supported: + + INTERFACE The IP interface name associated with the + information. + + ACTIVE Either *yes* or *no*, depending on if the IP + interface is being used by the system for + IP data traffic. + + GROUP The IPMP IP interface associated with the IP + interface. For IP interfaces in the + anonymous group (see *in.mpathd(1M)*), this + field will be empty. + + FLAGS Assorted information about the IP interface: + + i Unusable due to being *INACTIVE*. + + s Marked *STANDBY*. + + m Nominated to send/receive IPv4 multicast + for its IPMP group. + + b Nominated to receive IPv4 broadcast for + its IPMP group. + + M Nominated to send/receive IPv6 multicast + for its IPMP group. + + d Unusable due to being *down*. + + h Unusable due to being brought *OFFLINE* by + *in.mpathd* because of a duplicate + hardware address + + LINK The state of link-based failure detection: + + *up* The link is up. + + *down* The link is down. + + *unknown* The network driver does not + detect link state changes. + + PROBE The state of probe-based failure detection: + + *ok* Probes detect no problems. + + *failed* Probes detect failure. + + *unknown* Probes cannot be sent since no + suitable probe targets are known. + + *disabled* Probes have been disabled because + a unique IP test address has not + been configured. + + STATE The overall state of the interface: + + *ok* The interface is online and + functioning properly based on the + configured failure detection + methods. + + *failed* The interface is online but has a + link state of *down* or a probe + state of *failed*. + + *offline* The interface is offline. + + *unknown* The interface is online but may + or may not be functioning because + the configured failure detection + methods are in *unknown* states. + + If *-o* is not specified, all output fields are displayed. + + Probe Mode + + Probe mode displays information about the probes being + sent by *in.mpathd*. Unlike other output modes, this + mode runs until explicitly terminated using *^C*. The + following output fields are supported: + + TIME The time the probe was sent, relative to when + *ipmpstat* was started. If the probe was + sent prior to starting *ipmpstat*, the time + will be negative. + + PROBE An identifier representing the probe. The + identifier will start at zero and will + monotonically increment for each probe sent + by *in.mpathd* over a given interface. To + enable more detailed analysis by packet + monitoring tools, this identifier matches the + *icmp_seq* field of the ICMP probe packet. + + INTERFACE The IP interface the probe was sent on. + + TARGET The hostname (or IP address) of the target + the probe was sent to. + + NETRTT The network round-trip-time for the probe. + This is the time between when the IP module + sends the probe and when the IP module + receives the ack. If *in.mpathd* has + concluded that the probe has been lost, this + field will be empty. + + RTT The total round-trip-time for the probe. + This is the time between when *in.mpathd* + starts executing the code to send the probe, + and when it completes processing the ack. If + *in.mpathd* has concluded that the probe has + been lost, this field will be empty. Spikes + in the total round-trip time that are not + present in the network round-trip time + indicate that the local system itself is + overloaded. + + RTTAVG The average round-trip-time to *TARGET* over + *INTERFACE*. This aids identification of + slow targets. If there is insufficient data + to calculate the average, this field will be + empty. + + RTTDEV The standard deviation for the + round-trip-time to *TARGET* over *INTERFACE*. + This aids identification of jittery targets. + If there is insufficient data to calculate + the standard deviation, this field will be + empty. + + If *-o* is not specified, all fields except for *RTTAVG* + and *RTTDEV* are displayed. + + Target Mode + + Target mode displays IPMP probe target information. The + following output fields are supported: + + INTERFACE The IP interface name associated with the + information. + + MODE The probe target discovery mode: + + *routes* Probe targets found via the + routing table. + + *multicast* Probe targets found via multicast + ICMP probes. + + *disabled* Probe-based failure detection is + disabled. + + TESTADDR The hostname (or IP address) that will be + used for sending and receiving probes. If a + unique test address has not been configured, + this field will be empty. Note that if an IP + interface is configured with both IPv4 and + IPv6 test addresses, probe target information + will be displayed separately for each test + address. + + TARGETS A space-separated list of probe target + hostnames (or IP addresses), in firing order. + If no probe targets could be found, this + field will be empty. + + If *-o* is not specified, all output fields are displayed. + +OUTPUT FORMAT + + By default, ipmpstat uses a human-friendly tabular format + for its output modes, where each row contains one or more + fields of information about a given object, which is in turn + uniquely identified by one or more of those fields. In this + format, a header identifying the fields is displayed above + the table (and after each screenful of information), fields + are separated by whitespace, empty fields are represented by + *--*, and other visual aids are used. + + Machine-parsable format also uses a tabular format, but is + designed to be efficient to programmatically parse. + Specifically, machine-parsable format differs from + human-friendly format in the following ways: + + * No headers are displayed. + + * Fields with empty values yield no output, rather than + *--*. + + * Fields are separated by a single *:*, rather than + variable amounts of whitespace. + + * If multiple fields are requested, and a literal *:* or + *\* occur in a field's value, they are escaped by + prefixing them with *\*. + +EXAMPLES + + Use the machine-parsable output format to create a *ksh* + function that outputs the failure detection time of a given + IPMP IP interface: + + getfdt() { + ipmpstat -gP -o group,fdt | while IFS=: read group fdt; do + [[ "$group" = "$1" ]] && { echo "$fdt"; return; } + done + } + +ATTRIBUTES + + See attributes(5) for descriptions of the following + attributes: + + /usr/sbin + + ____________________________________________________________ + | ATTRIBUTE TYPE | ATTRIBUTE VALUE | + |_____________________________|_____________________________| + | Availability | SUNWcsu | + |_____________________________|_____________________________| + | Interface Stability | Committed | + |_____________________________|_____________________________| + | Machine-Parsable Format | Committed | + |_____________________________|_____________________________| + | Human-Friendly Format | Not-an-Interface | + |_____________________________|_____________________________| + + +SEE ALSO + + in.mpathd(1M), ifconfig(1M), if_mpadm(1M) --- old/./route.1m.txt Fri Dec 19 04:18:57 2008 +++ new/./route.1m.txt Fri Dec 19 04:18:57 2008 @@ -100,7 +100,7 @@ are applied only to the list of saved routes to be used at startup, not to the network routing tables. In addition, certain checks, - such as the existance of network interfaces + such as the existence of network interfaces used with -ifp, are skipped. This can be use- ful from within JumpStart scripts, where the root directory of the system being modified @@ -345,7 +345,7 @@ o IP address following the gateway address . This is typically specified in decimal dot notation as for - inet_addr(3SOCKET) rather than in symbollic form. + inet_addr(3SOCKET) rather than in symbolic form. o IP address following the -netmask qualifier. @@ -398,9 +398,14 @@ - specify the interface by name. For example, -ifp lo0 associ- - ates the route with the lo0 interface. + specify the interface by name. For example, *-ifp lo0* + associates the route with the *lo0* interface. If the named + interface is an underlying interface in an IPMP group, then + requests to add a route will automatically be translated to + the corresponding IPMP IP interface, and requests to delete + or change a route on an underlying interface will fail. + Routing Flags Routes have associated flags that influence operation of the protocols when sending to destinations matched by the --- old/./route.7p.txt Fri Dec 19 04:18:58 2008 +++ new/./route.7p.txt Fri Dec 19 04:18:58 2008 @@ -167,6 +167,43 @@ for further input. + By default, underlying IP interfaces in an IPMP group are + not visible to routing sockets. As such, routing sockets + will not receive events related to underlying IP interfaces + in an IPMP group. For consistency, when an IP interface is + placed into an IPMP group, *RTM_DELADDR* messages will be + generated for each of the *IFF_UP* addresses that are not + migrated to the corresponding IPMP IP interface, and an + *RTM_IFINFO* message will be sent indicating that the + interface is now down. Similarly, when an underlying + interface is removed from an IPMP group, an *RTM_IFINFO* + message will be sent indicating that the interface is again + up, and *RTM_NEWADDR* messages will be generated for each + *IFF_UP* address found on the interface. + + + The *RT_AWARE* socket option at the *SOL_ROUTE* level allows + an application to indicate its awareness of certain + features, which controls routing socket behavior. The + supported values are: + + *RTAW_DEFAULT* Default awareness. + + *RTAW_UNDER_IPMP* IPMP underlying interface awareness. + When this is enabled, underlying IP + interfaces in an IPMP group will remain + visible to the routing socket and + events related to them will continue to + be generated. + + + An *RTM_ADD* request tied to an underlying IP interface in + an IPMP group will be translated to an *RTM_ADD* request for + its corresponding IPMP IP interface. All routing socket + requests other than *RTM_ADD* and *RTM_GET* will fail when + issued on an underlying IP interface in an IPMP group. + + If a route is in use when it is deleted, the routing entry is marked down and removed from the routing table, but the resources associated with it are not reclaimed until all --- old/./socket.h.txt Fri Dec 19 04:18:59 2008 +++ new/./socket.h.txt Fri Dec 19 04:18:59 2008 @@ -92,7 +92,7 @@ The IPv4 data formats generally use the same values for data passed back in cmsghdr as for setsockopt() to enable the feature. The IPv4 data formats are listed below with the - assocated payload for each. + associated payload for each. IPPROTO_IP IP_RECVDSTADDR @@ -147,7 +147,7 @@ The IPv6 data formats use different values for enabling the option and for passing the value back to the application. - The IPv6 data formats are listed below with the assocated + The IPv6 data formats are listed below with the associated payload for each. IPPROTO_IPV6 @@ -299,12 +299,15 @@ - The header defines the following macro for - use as the level argument of setsockopt() and getsockopt(). + The ** header defines the following macros for + use as the *level* argument of *setsockopt()* and + getsockopt()*. - SOL_SOCKET Options to be accessed at socket level, not - protocol level. + *SOL_SOCKET* Options to be accessed at the socket level, + not the protocol level. + *SOL_ROUTE* Options to be accessed at the routing socket + level, not the protocol level. The header defines the following macros for