From sacadmin Thu May 26 04:37:11 2005 Received: from phorcys.East.Sun.COM (phorcys.East.Sun.COM [129.148.174.143]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j4QBbBs0007532 for ; Thu, 26 May 2005 04:37:11 -0700 (PDT) Received: from phorcys.East.Sun.COM (localhost [127.0.0.1]) by phorcys.East.Sun.COM (8.13.4+Sun/8.13.4) with ESMTP id j4QBat0F012079; Thu, 26 May 2005 07:36:55 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.East.Sun.COM (8.13.4+Sun/8.13.4/Submit) id j4QBatBW012076; Thu, 26 May 2005 07:36:55 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17045.46295.902373.456594@gargle.gargle.HOWL> Date: Thu, 26 May 2005 07:36:55 -0400 From: James Carlson To: psarc@sac.sfbay.sun.com cc: ipmp-team@sun.com Subject: 2005/341 IPMP Inactive Symmetry X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 5737 I'm sponsoring this fast-track request for Ethindra Ramamurthy. The timeout is set to 06/02/2005. BACKGROUND "IP Multipathing" (IPMP; PSARC 1999/225) uses several flags on IP interfaces to control usage and track dynamic state. IPMP also has an optional feature that allows "failback" (automatic re-enabling of interfaces after repair) to be disabled. By default "failback" is enabled. IPMP also has a currently-unrelated feature that allows an interface to be marked as "standby" by the administrator. IPMP treats "standby" interfaces specially, and will mark them "inactive" when not in use by the system. This "IFF_INACTIVE" flag causes the kernel to avoid using the interface for regular traffic. See "IP Multipathing Updates" (PSARC 1999/637). PROBLEM When a failure is detected, a spare interface will be configured and used by IPMP. If failback is disabled and the interface is repaired, the configuration should stay the same, and the previously-failed interface should still not be used by system for regular traffic. However, this doesn't actually work. Instead, although the interface configuration is (correctly) not changed by IPMP when the interface is repaired, the kernel itself will automatically begin using the repaired interface, regardless of the setting of the "failback" option. SOLUTION A "patch/micro" release binding is asserted. The existing IPMP-specific IFF_INACTIVE flag will be used to mark interfaces that have failed and later repaired. Setting of IFF_FAILED or IFF_INACTIVE will be mutually exclusive. When an interface fails, IFF_FAILED is set. Once the interface has been repaired, IFF_FAILED will be removed and, if failback is disabled, IFF_INACTIVE will be set to indicate that the system should not use this interface for regular traffic, even though it is now operational (it will be used to probe targets). A new Project Private ioctl SIOCSIPMPFAILBACK is introduced, requiring PRIV_SYS_NET_CONFIG privilege. Based on the user configuration of FAILBACK in /etc/default/mpathd, IPMP daemon in.mpathd will call this ioctl to inform the kernel, whether failback is enabled. The stability level of the IFF_INACTIVE flag is not at all clear. "IPMP Clarifications" (PSARC 2002/755) redefined a number of the original 1999/225 IFF_* flags as Project Private. However, it missed the IFF_INACTIVE flag from 1999/637, even though that case was cited as a reference. Even more confusing is that these flags are exposed to users by /usr/sbin/ifconfig output, and are at least partially documented in ifconfig(1M). They're thus probably still Evolving. Nobody would reasonably depend on the IFF_INACTIVE bit without also looking at IFF_STANDBY, and in any event IFF_FAILED is much more important and IFF_UP is even more important still. Also these flags (IFF_FAILED, IFF_INACTIVE, IFF_STANDBY) are not really visible to normal applications anyway, because they're all above the 32-bit boundary (above which neither routing sockets nor the standard SIOCGIF* ioctls can express). For the above reason, this change can be allowed in a patch. As a practical matter, though, the IFF_INACTIVE flag can never be set by the user and is thus always controlled by the kernel. The only thing the user can do is observe the flag. The likelihood that anyone ever used this IPMP-specific flag in an application seems fairly remote, and the chance that such an application would be confused by the new (and logically upward-compatible) usage appears to be vanishingly small. Despite the apparent Evolving stability, the project team asserts that the patch/micro release binding for the change in usage is safe. DETAILS Old system with FAILBACK=no in active-standby mode (just showing physical interface with the flags involved in the operation, when test address is set on their respective logical interface): hme0 hme1 -> failure of hme0 hme0 hme1 -> repair of hme0 hme0 hme1 New system with FAILBACK=no (active-standby): hme0 hme1 -> failure of hme0 hme0 hme1 -> repair of hme0 hme0 hme1 ifconfig(1M): - INACTIVE Only set on standby interfaces, this flag - indicates no failover has occurred to the - interface. New addresses cannot be created - on this interface. This flag is cleared if a - failover occurs to the interface. + INACTIVE This flag indicates that the interface is + not currently used for regular traffic by + the system. New addresses cannot be created + on this interface. It is set automatically + on standby interfaces. It is also set when + the system detects that a failed interface + has been repaired and FAILBACK=no is config- + ured in /etc/default/mpathd. It is cleared + when interface fails or when a failover to + that interface occurs. The System Administrator's Guide will also be updated (as part of the FAILBACK=no option) to describe the new behavior. + The default is yes. When set to no, failback is disabled. + When system detects that a failed interface has repaired, + interface is marked INACTIVE and not used for regular + traffic +Note - Setting of flags IFF_FAILED or IFF_INACTIVE on an interface +is mutually exclusive. From sacadmin Wed Jun 1 10:13:16 2005 Received: from phorcys.East.Sun.COM (phorcys.East.Sun.COM [129.148.174.143]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id j51HDFs0016605 for ; Wed, 1 Jun 2005 10:13:16 -0700 (PDT) Received: from phorcys.East.Sun.COM (localhost [127.0.0.1]) by phorcys.East.Sun.COM (8.13.4+Sun/8.13.4) with ESMTP id j51HCukR006051; Wed, 1 Jun 2005 13:12:56 -0400 (EDT) Received: (from carlsonj@localhost) by phorcys.East.Sun.COM (8.13.4+Sun/8.13.4/Submit) id j51HCuoR006048; Wed, 1 Jun 2005 13:12:56 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17053.60568.326660.578181@gargle.gargle.HOWL> Date: Wed, 1 Jun 2005 13:12:56 -0400 From: James Carlson To: psarc@sac.sfbay.sun.com cc: ipmp-team@sun.com Subject: 2005/341 IPMP Inactive Symmetry X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 301 This fast-track request was approved during ARC business at today's meeting. -- James Carlson, KISS Network Sun Microsystems / 1 Network Drive 71.234W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.497N Fax +1 781 442 1677