From sacadmin Tue Aug 3 14:22:26 2004 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Date: Tue, 3 Aug 2004 17:22:03 -0400 From: James Carlson To: psarc@sac.sfbay.sun.com cc: Blaise.Sanouillet@sun.com, Peter.Memishian@sun.com, thirumalai.srinivasan@sun.com Subject: 2004/596 Adding Aliases To Failed Interfaces Content-Length: 9091 I'm sponsoring this fast-track request for Blaise Sanouillet. The timer is set for 08/10/2004. It's a little on the long side, but boils down to making a case that currently returns an error instead do something sensible, though imperfect. Release binding is "minor" and stability of the interfaces remains unchanged (Evolving). Problem statement ================= IP Network Multipathing (IPMP; PSARC 1999/225) allows administrators to group network interfaces so that the system is resilient in the event of a network problem. When this happens, the interface is marked "failed" and all the addresses hosted on this interface are failed over to another, working, interface. Currently, IPMP does not allow the administrator to add an address to a "failed", "standby" or "offline" interface, when using the SIOCLIFADDIF ioctl or "ifconfig addif" ([PSARC 1997/184] and [PSARC 2001/665]). The administrator has no way to avoid this failure mode of SIOCLIFADDIF, because the failed status is outside his control and can change asynchronously, for example upon link failure. This problem was first reported in [4503508], and causes specific problems with Zones as reported in [5048068]. Solution ======== The proposed solution is to have the SIOCLIFADDIF ioctl create the new address on a working interface in the IPMP group, if the specified interface has one of FAILED, INACTIVE or OFFLINE flags. Upon return, the lifr_name field of the 'struct lifreq' argument contains the name of the actual logical interface created. Also, the SIOCLIFREMOVEIF ioctl will remove the specified address from the IPMP group even if it's not hosted on the specified interface. As a consequence, the ifconfig(1M) user can see the following change of behavior: Before - # ifconfig ce1 addif 1.2.3.4/24 up ifconfig: addif: SIOCLIFADDIF: ce1: Network is down # ifconfig ce0 addif 1.2.3.4/24 up Created new logical interface ce0:1 # ifconfig ce1 removeif 1.2.3.4 ifconfig: removeif: SIOCLIFREMOVEIF: ce1: Cannot assign requested address After - # ifconfig ce1 addif 1.2.3.4/24 up Created new logical interface ce0:2 # ifconfig ce1 removeif 1.2.3.4 # The impact of this change on scripting is expected to be positive: scripts would be broken by the current behavior and will be fixed by the change. The submitter can't imagine how a script could rely on the "Network is down" failure mode to work correctly. It's also worth noting that the ENETDOWN return code for SIOCLIFADDIF is not documented in if_tcp(7P). The "ifconfig plumb" subcommand will also start working in this case: # ifconfig ce1:5 plumb 1.2.3.4/24 up will create the interface on ce0, but not necessarily with the same unit number (not necessarily ce0:5). There is strictly no change in behavior when the specified physical interface is not part of an IPMP group. Restrictions/Limitations ------------------------ This solution doesn't remove the failure for IPMP test addresses added to a failed/standby/offline interface. Doing so with ifconfig will produce the following error message: ifconfig: can't set -failover on failed/standby/offlined interface ce1 (The same operation currently gives the "Network is down" message.) This restriction is caused by the non-atomicity of the operation: ifconfig must first create the logical interface using SIOCLIFADDIF, then set the IFF_NOFAILOVER flag using SIOCSLIFFLAGS. So the first ioctl will add the address to a working interface, which is not the expected result. Solving this would involve introducing a new API for atomic creation of a logical interface with specific flags; the cost and impact of doing this seem to outweigh the benefits. "ifconfig addif" still requires the specified physical interface to be plumbed. For example, if the administrator has removed the physical card using DR, the "ifconfig addif" will fail as before with "no such interface" (ENXIO). The fact that the alias number may be different when a specific number is plumbed and the interface is down is currently unavailable. It may cause scripts to fail, if those scripts rely on issuing multiple 'ifconfig' commands against the same alias. However, those scripts would already fail on the current implementation, as the initial plumbing itself would be prohibited on the failed interface. It's expected that a future project to unify the IPMP address management mechanisms will resolve this issue without harming the interfaces introduced in this project. Alternatives ------------ Another solution would be to allow the creation of the interface on the "failed"/"standby"/"offline" interface, and let the IPMP daemon (in.mpathd) move the address to a working interface. This was considered and rejected because a lot of changes to in.mpathd would be required to make the failover happen quickly. Also, the fact that ifconfig shows the actual interface created seemed better for the user experience. As pointed out in [5048068], the proper solution to this problem would be to host IPMP data addresses on a virtual network interface [PSARC 2004/447]. However, the solution proposed here has very limited impact on the APIs and the implementation, and it's not an impediment to the development of a better solution. Man page diffs -------------- *** if_tcp.7p Mon Aug 2 16:04:29 2004 --- if_tcp.7p.new Mon Aug 2 16:38:07 2004 *************** *** 221,231 **** SIOCLIFADDIF Add a new logical interface on a physical interface ! using an unused logical unit number. SIOCLIFREMOVEIF Remove a logical interface by specifying its IP ! address or logical interface name. SIOCSLIFTOKEN Set the address token used to form IPv6 link-local --- 221,240 ---- SIOCLIFADDIF Add a new logical interface on a physical interface ! using an unused logical unit number. If the physical ! interface is part of an IP multipathing group, the ! logical interface can be added to a different physical ! interface in the same group. Upon return, the ! lifr_name field contains the name of the actual ! logical interface created. SIOCLIFREMOVEIF Remove a logical interface by specifying its IP ! address or logical interface name. When the IP address ! is specified and the interface is part of an IP ! multipathing group, the logical interface will be ! removed from the physical interface in the group which ! holds the IP address. SIOCSLIFTOKEN Set the address token used to form IPv6 link-local *** ifconfig.1m Mon Aug 2 16:27:56 2004 --- ifconfig.1m.new Mon Aug 2 16:27:12 2004 *************** *** 103,109 **** addif address Create the next unused logical interface on the speci- ! fied physical interface. arp Enable the use of the Address Resolution Protocol ("ARP") in mapping between network level addresses and --- 103,112 ---- addif address Create the next unused logical interface on the speci- ! fied physical interface. If the physical interface is ! part of a multipathing group, the logical interface ! may be added to a different physical interface in the ! same group. arp Enable the use of the Address Resolution Protocol ("ARP") in mapping between network level addresses and *************** *** 612,618 **** removeif address Remove the logical interface on the physical interface ! specified that matches the address specified. router Enable IP forwarding on the interface. When enabled, --- 615,624 ---- removeif address Remove the logical interface on the physical interface ! specified that matches the address specified. When the ! interface is part of a multipathing group, the logical ! interface will be removed from the physical interface ! in the group which holds the address. router Enable IP forwarding on the interface. When enabled, References ========== [4503508] semantics of adding logical interfaces to standby interfaces needs work http://dtsw.eng.sun.com/cgi-bin/bugtraq_showbug?bugid=4503508 [5048068] zones don't boot if one of its interfaces has failed http://dtsw.eng.sun.com/cgi-bin/bugtraq_showbug?bugid=5048068 [PSARC 1997/184] IPv6 Protocol Stack and Utilities http://sac.sfbay.sun.com/PSARC/1997/184/ [PSARC 2001/665] EEXIST For SIOCLIFADDIF http://sac.sfbay.sun.com/PSARC/2001/665/ [PSARC 2004/447] Virtual IP Source Address Selection http://sac.sfbay.sun.com/PSARC/2004/447/ From sacadmin Wed Aug 4 09:46:18 2004 Date: Wed, 4 Aug 2004 09:18:09 -0700 (PDT) From: Erik Nordmark Subject: Re: Fw: 2004/596 Adding Aliases To Failed Interfaces To: Blaise Sanouillet - Sun Microsystems Cc: snt-interest@sun.com, psarc@sac.sfbay.sun.com MIME-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII Content-Length: 811 > After - > # ifconfig ce1 addif 1.2.3.4/24 up > Created new logical interface ce0:2 > # ifconfig ce1 removeif 1.2.3.4 > # Makes sense. > The "ifconfig plumb" subcommand will also start working in this > case: > > # ifconfig ce1:5 plumb 1.2.3.4/24 up > > will create the interface on ce0, but not necessarily with the same > unit number (not necessarily ce0:5). So how would this work: ifconfig ce1:5 plumb 1.2.3.4 ifconfig ce1:5 netmask 255.255.255.0 ifconfig ce1:5 up I think if you are going to reinterpret an explicit LUN you need to do this for all ifconfig commands (which is probably infeasible). "addif" doesn't have that issue since it returns a dynamic LUN thus for things to work today all the ifconfig commands would have to be issued at once. Erik From sacadmin Wed Aug 4 10:06:09 2004 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Date: Wed, 4 Aug 2004 13:05:47 -0400 From: James Carlson To: Erik Nordmark Cc: Blaise Sanouillet - Sun Microsystems , snt-interest@Sun.COM, psarc@sac.sfbay.sun.com Subject: Re: Fw: 2004/596 Adding Aliases To Failed Interfaces Content-Length: 1219 Erik Nordmark writes: > > will create the interface on ce0, but not necessarily with the same > > unit number (not necessarily ce0:5). > > So how would this work: > ifconfig ce1:5 plumb 1.2.3.4 > ifconfig ce1:5 netmask 255.255.255.0 > ifconfig ce1:5 up Previously, the "plumb" would have just failed, and the subsequent commands would thus also fail. After the change, the "plumb" succeeds, but ends up putting the alias on another interface, so the subsequent commands still fail. > I think if you are going to reinterpret an explicit LUN you need to > do this for all ifconfig commands (which is probably infeasible). Essentially, it already does this, but it just doesn't tell you that it does this. If I plumb up "ce1:5", and ce1 fails, that alias will be transferred over to some other interface, and will get a new alias number in the process. There's not much you can do about it. This case just makes it "as if" the plumbing and the fail-over were done in one operation. -- James Carlson, IP Systems Group Sun Microsystems / 1 Network Drive 71.234W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.497N Fax +1 781 442 1677 From sacadmin Thu Aug 12 04:56:08 2004 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Date: Thu, 12 Aug 2004 07:55:49 -0400 From: James Carlson To: psarc@sac.sfbay.sun.com cc: Blaise.Sanouillet@sun.com, Peter.Memishian@sun.com, thirumalai.srinivasan@sun.com Subject: 2004/596 Adding Aliases To Failed Interfaces Content-Length: 383 This fast-track request was approved during ARC business at yesterday's meeting. A final version of the specification is in the case directory as 'spec.txt'. -- James Carlson, IP Systems Group Sun Microsystems / 1 Network Drive 71.234W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.497N Fax +1 781 442 1677