Release binding is "minor" and stability of the interfaces remains unchanged (Evolving). Problem statement ================= IP Network Multipathing (IPMP; PSARC 1999/225) allows administrators to group network interfaces so that the system is resilient in the event of a network problem. When this happens, the interface is marked "failed" and all the addresses hosted on this interface are failed over to another, working, interface. Currently, IPMP does not allow the administrator to add an address to a "failed", "standby" or "offline" interface, when using the SIOCLIFADDIF ioctl or "ifconfig addif" ([PSARC 1997/184] and [PSARC 2001/665]). The administrator has no way to avoid this failure mode of SIOCLIFADDIF, because the failed status is outside his control and can change asynchronously, for example upon link failure. This problem was first reported in [4503508], and causes specific problems with Zones as reported in [5048068]. Solution ======== The proposed solution is to have the SIOCLIFADDIF ioctl create the new address on a working interface in the IPMP group, if the specified interface has one of FAILED, INACTIVE or OFFLINE flags. Upon return, the lifr_name field of the 'struct lifreq' argument contains the name of the actual logical interface created. Also, the SIOCLIFREMOVEIF ioctl will remove the specified address from the IPMP group even if it's not hosted on the specified interface. As a consequence, the ifconfig(1M) user can see the following change of behavior: Before - # ifconfig ce1 addif 1.2.3.4/24 up ifconfig: addif: SIOCLIFADDIF: ce1: Network is down # ifconfig ce0 addif 1.2.3.4/24 up Created new logical interface ce0:1 # ifconfig ce1 removeif 1.2.3.4 ifconfig: removeif: SIOCLIFREMOVEIF: ce1: Cannot assign requested address After - # ifconfig ce1 addif 1.2.3.4/24 up Created new logical interface ce0:2 # ifconfig ce1 removeif 1.2.3.4 # The impact of this change on scripting is expected to be positive: scripts would be broken by the current behavior and will be fixed by the change. The submitter can't imagine how a script could rely on the "Network is down" failure mode to work correctly. It's also worth noting that the ENETDOWN return code for SIOCLIFADDIF is not documented in if_tcp(7P). The "ifconfig plumb" subcommand will also start working in this case: # ifconfig ce1:5 plumb 1.2.3.4/24 up will create the interface on ce0, but not necessarily with the same unit number (not necessarily ce0:5). There is strictly no change in behavior when the specified physical interface is not part of an IPMP group. Restrictions/Limitations ------------------------ This solution doesn't remove the failure for IPMP test addresses added to a failed/standby/offline interface. Doing so with ifconfig will produce the following error message: ifconfig: can't set -failover on failed/standby/offlined interface ce1 (The same operation currently gives the "Network is down" message.) This restriction is caused by the non-atomicity of the operation: ifconfig must first create the logical interface using SIOCLIFADDIF, then set the IFF_NOFAILOVER flag using SIOCSLIFFLAGS. So the first ioctl will add the address to a working interface, which is not the expected result. Solving this would involve introducing a new API for atomic creation of a logical interface with specific flags; the cost and impact of doing this seem to outweigh the benefits. "ifconfig addif" still requires the specified physical interface to be plumbed. For example, if the administrator has removed the physical card using DR, the "ifconfig addif" will fail as before with "no such interface" (ENXIO). The fact that the alias number may be different when a specific number is plumbed and the interface is down is currently unavailable. It may cause scripts to fail, if those scripts rely on issuing multiple 'ifconfig' commands against the same alias. However, those scripts would already fail on the current implementation, as the initial plumbing itself would be prohibited on the failed interface. It's expected that a future project to unify the IPMP address management mechanisms will resolve this issue without harming the interfaces introduced in this project. Alternatives ------------ Another solution would be to allow the creation of the interface on the "failed"/"standby"/"offline" interface, and let the IPMP daemon (in.mpathd) move the address to a working interface. This was considered and rejected because a lot of changes to in.mpathd would be required to make the failover happen quickly. Also, the fact that ifconfig shows the actual interface created seemed better for the user experience. As pointed out in [5048068], the proper solution to this problem would be to host IPMP data addresses on a virtual network interface [PSARC 2004/447]. However, the solution proposed here has very limited impact on the APIs and the implementation, and it's not an impediment to the development of a better solution. Man page diffs -------------- *** if_tcp.7p Mon Aug 2 16:04:29 2004 --- if_tcp.7p.new Mon Aug 2 16:38:07 2004 *************** *** 221,231 **** SIOCLIFADDIF Add a new logical interface on a physical interface ! using an unused logical unit number. SIOCLIFREMOVEIF Remove a logical interface by specifying its IP ! address or logical interface name. SIOCSLIFTOKEN Set the address token used to form IPv6 link-local --- 221,240 ---- SIOCLIFADDIF Add a new logical interface on a physical interface ! using an unused logical unit number. If the physical ! interface is part of an IP multipathing group, the ! logical interface can be added to a different physical ! interface in the same group. Upon return, the ! lifr_name field contains the name of the actual ! logical interface created. SIOCLIFREMOVEIF Remove a logical interface by specifying its IP ! address or logical interface name. When the IP address ! is specified and the interface is part of an IP ! multipathing group, the logical interface will be ! removed from the physical interface in the group which ! holds the IP address. SIOCSLIFTOKEN Set the address token used to form IPv6 link-local *** ifconfig.1m Mon Aug 2 16:27:56 2004 --- ifconfig.1m.new Mon Aug 2 16:27:12 2004 *************** *** 103,109 **** addif address Create the next unused logical interface on the speci- ! fied physical interface. arp Enable the use of the Address Resolution Protocol ("ARP") in mapping between network level addresses and --- 103,112 ---- addif address Create the next unused logical interface on the speci- ! fied physical interface. If the physical interface is ! part of a multipathing group, the logical interface ! may be added to a different physical interface in the ! same group. arp Enable the use of the Address Resolution Protocol ("ARP") in mapping between network level addresses and *************** *** 612,618 **** removeif address Remove the logical interface on the physical interface ! specified that matches the address specified. router Enable IP forwarding on the interface. When enabled, --- 615,624 ---- removeif address Remove the logical interface on the physical interface ! specified that matches the address specified. When the ! interface is part of a multipathing group, the logical ! interface will be removed from the physical interface ! in the group which holds the address. router Enable IP forwarding on the interface. When enabled, References ========== [4503508] semantics of adding logical interfaces to standby interfaces needs work http://dtsw.eng.sun.com/cgi-bin/bugtraq_showbug?bugid=4503508 [5048068] zones don't boot if one of its interfaces has failed http://dtsw.eng.sun.com/cgi-bin/bugtraq_showbug?bugid=5048068 [PSARC 1997/184] IPv6 Protocol Stack and Utilities http://sac.sfbay.sun.com/PSARC/1997/184/ [PSARC 2001/665] EEXIST For SIOCLIFADDIF http://sac.sfbay.sun.com/PSARC/2001/665/ [PSARC 2004/447] Virtual IP Source Address Selection http://sac.sfbay.sun.com/PSARC/2004/447/