From sacadmin Tue Jan 29 09:51:21 2008 Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0THpLmx020611; Tue, 29 Jan 2008 09:51:21 -0800 (PST) Received: (from carlsonj@localhost) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8/Submit) id m0THpLEZ020603; Tue, 29 Jan 2008 09:51:21 -0800 (PST) Date: Tue, 29 Jan 2008 09:51:21 -0800 (PST) From: James Carlson Message-Id: <200801291751.m0THpLEZ020603@sac.sfbay.sun.com> To: PSARC-record@sac.sfbay.sun.com Subject: Default Route For Zones [PSARC/2008/057 FastTrack timeout 02/05/2008] Status: RO Content-Length: 558 Template Version: @(#)sac_nextcase 1.64 07/13/07 SMI This information is Copyright 2008 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: Default Route For Zones 1.2. Name of Document Author/Supplier: Author: Glenn Faden 1.3 Date of This Document: 29 January, 2008 4. Technical Description See the case directory for more detail 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From carlsonj@phorcys.east.sun.com Tue Jan 29 10:05:10 2008 Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0TI5AIb021517 for ; Tue, 29 Jan 2008 10:05:10 -0800 (PST) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m0TI59TM009536; Tue, 29 Jan 2008 13:05:09 -0500 (EST) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.2+Sun/8.14.2/Submit) id m0TI59YS009533; Tue, 29 Jan 2008 13:05:09 -0500 (EST) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18335.27349.654381.581913@gargle.gargle.HOWL> Date: Tue, 29 Jan 2008 13:05:09 -0500 From: James Carlson To: psarc-ext@sac.sfbay.sun.com cc: Glenn Faden Subject: 2008/057 Default Route For Zones X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 3396 I'm sponsoring this fast-track request for Glenn Faden. The timer is set to 02/05/2008. The requested release binding is "patch/micro" (as it will be needed for an S10 Update), and the stability level for the new zonecfg keyword is "Committed." Zones may be configured with shared or exclusive IP stacks. The shared stack configuration has network interfaces configured by the global zone, not by the non-global zone itself. Zones with shared IP stacks are assigned logical interfaces associated with the physical interface specified in the zone's configuration file. Therefore, the physical interface must exist (be plumbed) in the global zone. Customers who are using zones for cross-domain solutions associate each zone with a unique physical network. They want to restrict network traffic such that each network interface is used by only a single zone, and the global zone is isolated from any of these networks. Currently such network configurations are difficult to manage because there is no facility to specify the default route for each network. Existing mechanisms such as persistent routes and /etc/defaultrouters don't work because they require that the interface is up when the routes are applied. However, these customers do not want to bring up the zone's network interfaces in the global zone. They must wait until the zone is booted and then apply its default route, but there is no mechanism for synchronizing these events. This project proposes that the default route for a zone's interface that uses a shared IP stack can be optionally specified using a new default router property, defrouter, that is associated with the network resource (net), via zonecfg(1M). Currently, the network resource has two properties, "physical" and "address." The latter can only be used with the shared IP stack. The default router property is optional for the shared IP stack, and not permitted with exclusive IP stack. The value for the default router is a hostname or an IP address. Host names are resolved in the context of the global zone. The router address is specified as part of the "net" resource type, so that the installed routes may be associated with a specific interface via the route(1M) "-ifp" option. For zones using a shared IP stack, the interface specified in the "physical" property must be plumbed in the global zone prior to booting the non-global zone. However, if the interface is not intended to be used by the global zone, it should be configured in the global zone via ifconfig(1M) into the down state. In this project, the specification of the default route in the zone's configuration file is needed for the zone to send packets off the assigned subnet. When specified, the default route for the interface is interpreted by zoneadmd, after it has brought up the specified logical interface and applied its address and netmask. If a default route is specified, zoneadmd then calls the route(1M) command to establish the specified route. If the exit status of the route command is non-zero, it is checked for the value EEXIST, which indicates that the route already exists. If any other exist status result is returned, a non-fatal warning message is generated. (The exit status values of the route command are not documented; the dependency is Consolidation Private.) This change is backward compatible because the new "defrouter" keyword is optional. From erik.nordmark@sun.com Tue Jan 29 21:57:09 2008 Received: from jurassic.eng.sun.com (jurassic-226-b [129.146.226.130]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0U5v9TR009870 for ; Tue, 29 Jan 2008 21:57:09 -0800 (PST) Received: from [10.7.251.248] (punchin-nordmark.SFBay.Sun.COM [10.7.251.248]) by jurassic.eng.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0U5v7RD898930 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 29 Jan 2008 21:57:08 -0800 (PST) Message-ID: <47A011B3.5060008@sun.com> Date: Tue, 29 Jan 2008 21:57:07 -0800 From: Erik Nordmark User-Agent: Thunderbird 2.0.0.4 (X11/20070723) MIME-Version: 1.0 To: James Carlson CC: psarc-ext@sac.sfbay.sun.com, Glenn Faden Subject: Re: 2008/057 Default Route For Zones References: <18335.27349.654381.581913@gargle.gargle.HOWL> In-Reply-To: <18335.27349.654381.581913@gargle.gargle.HOWL> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 2230 James Carlson wrote: > Customers who are using zones for cross-domain solutions associate each > zone with a unique physical network. They want to restrict network > traffic such that each network interface is used by only a single zone, > and the global zone is isolated from any of these networks. Currently > such network configurations are difficult to manage because there is no > facility to specify the default route for each network. Existing > mechanisms such as persistent routes and /etc/defaultrouters don't work > because they require that the interface is up when the routes are > applied. However, these customers do not want to bring up the zone's > network interfaces in the global zone. They must wait until the zone is > booted and then apply its default route, but there is no mechanism for > synchronizing these events. > > This project proposes that the default route for a zone's interface that > uses a shared IP stack can be optionally specified using a new default > router property, defrouter, that is associated with the network > resource (net), via zonecfg(1M). Currently, the network resource has two > properties, "physical" and "address." The latter can only be used with the > shared IP stack. The default router property is optional for the shared > IP stack, and not permitted with exclusive IP stack. The value for the > default router is a hostname or an IP address. Host names are resolved > in the context of the global zone. When we talked about this earlier my thinking was limited to the case when the ngz has a unique subnet (could be on a unique physical, or on a shared physical). In that case it works to specify a default router for the zone. However, if the ngz's shares a subnet with some other zone, then the current logic in the kernel isn't capable of supporting a different default route for different zones. This is because the kernel check is whether the gateway field in the default route is on the same subnet as one of the zone's IP addresses. Sorry for not catching this earlier. Would it make sense to somehow restrict this property to the case when the ngz has IP address(es) that do not have a common subnet with any other zone on the system? Erik From Glenn.Faden@sun.com Tue Jan 29 22:37:56 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02 [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0U6buhx010401 for ; Tue, 29 Jan 2008 22:37:56 -0800 (PST) Received: from sca-es-mail-2.sun.com (sca-es-mail-2.Sun.COM [192.18.43.133]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0U6buHP041396 for ; Tue, 29 Jan 2008 22:37:56 -0800 (PST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m0U6bp7W016100 for ; Tue, 29 Jan 2008 22:37:51 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JVG009012D6KE00@fe-sfbay-09.sun.com> (original mail from Glenn.Faden@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Tue, 29 Jan 2008 22:37:50 -0800 (PST) Received: from rampartgf.local ([129.150.18.193]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JVG00EA12F25IF0@fe-sfbay-09.sun.com>; Tue, 29 Jan 2008 22:37:50 -0800 (PST) Date: Tue, 29 Jan 2008 22:37:51 -0800 From: Glenn Faden Subject: Re: 2008/057 Default Route For Zones In-reply-to: <47A011B3.5060008@sun.com> Sender: Glenn.Faden@sun.com To: Erik Nordmark Cc: James Carlson , psarc-ext@sac.sfbay.sun.com, Glenn Faden Message-id: <47A01B3F.50103@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) Status: RO Content-Length: 2455 Erik Nordmark wrote: > > When we talked about this earlier my thinking was limited to the case > when the ngz has a unique subnet (could be on a unique physical, or on > a shared physical). > In that case it works to specify a default router for the zone. > > However, if the ngz's shares a subnet with some other zone, then the > current logic in the kernel isn't capable of supporting a different > default route for different zones. This is because the kernel check is > whether the gateway field in the default route is on the same subnet > as one of the zone's IP addresses. > > Sorry for not catching this earlier. > > Would it make sense to somehow restrict this property to the case when > the ngz has IP address(es) that do not have a common subnet with any > other zone on the system? > > I have tried to avoid making this feature only apply to Trusted Extensions, although that is the customer base that is requesting it. For TX customers each zone has its own subnet and its own interface. The primary use of TX is to provide cross-domain services for physically separate networks. Furthermore, the labeling policy provides the separation of routes because each subnet and router is assigned a unique sensitivity label. While it may be less useful for standard Solaris customers, we shouldn't prevent them from taking advantage of this feature. We just need to be clear that it is intended for zones with separate subnets. I think it is a bad idea for try to verify that items specified via zonecfg XML files are valid based on the state of the system at the time the configuration is being generated. If any subnet verification were to occur it should be at zone boot time. In practice, I don't feel this is necessary, and adds considerable complexity. If it were to be done, I think the logic should be in the route command itself. For the record, I think much of the current verification of possible zone configuration conflicts that is done today in Solaris is counter-productive and is an obstacle to deploying large numbers of zones. The only useful verification is between the currently booting zone and other active zones. Anyone who has tried to boot 1000 zones into the ready state will appreciate what I am saying. It took 3 days of execution time just to verify the zonecfg XML device and network specifications in S10u4 beta! This became a showstopper which delayed the release of S10u4. --Glenn From carlsonj@phorcys.east.sun.com Wed Jan 30 06:27:50 2008 Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UERoUx019926 for ; Wed, 30 Jan 2008 06:27:50 -0800 (PST) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m0UERJ2Z013280; Wed, 30 Jan 2008 09:27:26 -0500 (EST) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.2+Sun/8.14.2/Submit) id m0UER76R013273; Wed, 30 Jan 2008 09:27:07 -0500 (EST) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18336.35131.636630.805135@gargle.gargle.HOWL> Date: Wed, 30 Jan 2008 09:27:07 -0500 From: James Carlson To: Erik Nordmark , Glenn Faden Cc: psarc-ext@sac.sfbay.sun.com, Glenn Faden Subject: Re: 2008/057 Default Route For Zones In-Reply-To: <47A011B3.5060008@sun.com> References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 2484 Erik Nordmark writes: > However, if the ngz's shares a subnet with some other zone, then the > current logic in the kernel isn't capable of supporting a different > default route for different zones. This is because the kernel check is > whether the gateway field in the default route is on the same subnet as > one of the zone's IP addresses. There's another problem buried here, which is that the proposed feature doesn't delete the static routes when the zone is shut down. It doesn't delete them because it's trying to cover for the EEXIST case. I think the right thing to do is to remember when the "route add" attempt fails, and conditionally remove the route on zone shutdown. That potentially leaves a zone relying on a duplicate out in the cold, but since it's a misconfiguration anyway, it doesn't seem like a big problem. Needless to say, there are a few sharp edges here for the unwary user. The original design goal of Zones was simply for consolidation on shared networks, not separated ones, and stepping outside of that has been a fairly persistent source of pain. > Would it make sense to somehow restrict this property to the case when > the ngz has IP address(es) that do not have a common subnet with any > other zone on the system? If the "restriction" is in the documentation provided for the feature, then I would agree. Otherwise, I have to agree with Glenn that static checks against the configured zones don't necessarily do the right thing, as users are under no obligation to boot and run all installed zones all of the time. Glenn Faden writes: > I think it is a bad idea for try to verify that items specified via > zonecfg XML files are valid based on the state of the system at the time > the configuration is being generated. If any subnet verification were to > occur it should be at zone boot time. In practice, I don't feel this is > necessary, and adds considerable complexity. If it were to be done, I > think the logic should be in the route command itself. I'm not sure it really belongs there, either. For what it's worth, I've gotten several private messages from users who've been waiting for this feature. If nothing else, it's tapping pent-up demand. I just hope they can make good use of it. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From edward.pilatowicz@sun.com Wed Jan 30 09:37:29 2008 Received: from dm-eng-02.sfbay.sun.com (dm-eng-02 [129.146.11.32]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UHbTYp024918 for ; Wed, 30 Jan 2008 09:37:29 -0800 (PST) Received: from mcescher.eng.sun.com (mcescher.SFBay.Sun.COM [129.146.224.55]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0UHbSfq057198; Wed, 30 Jan 2008 09:37:28 -0800 (PST) Received: from mcescher.eng.sun.com (localhost [127.0.0.1]) by mcescher.eng.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m0UHbSPm786955; Wed, 30 Jan 2008 09:37:28 -0800 (PST) Received: (from edp@localhost) by mcescher.eng.sun.com (8.14.2+Sun/8.14.2/Submit) id m0UHbSww786912; Wed, 30 Jan 2008 09:37:28 -0800 (PST) X-Authentication-Warning: mcescher.eng.sun.com: edp set sender to edward.pilatowicz@sun.com using -f Date: Wed, 30 Jan 2008 09:37:28 -0800 From: Edward Pilatowicz To: James Carlson Cc: Erik Nordmark , Glenn Faden , Glenn Faden , psarc-ext@sac.sfbay.sun.com Subject: Re: 2008/057 Default Route For Zones Message-ID: <20080130173728.GB732888@eng.sun.com> References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <18336.35131.636630.805135@gargle.gargle.HOWL> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <18336.35131.636630.805135@gargle.gargle.HOWL> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 1374 On Wed, Jan 30, 2008 at 09:27:07AM -0500, James Carlson wrote: > Erik Nordmark writes: > > However, if the ngz's shares a subnet with some other zone, then the > > current logic in the kernel isn't capable of supporting a different > > default route for different zones. This is because the kernel check is > > whether the gateway field in the default route is on the same subnet as > > one of the zone's IP addresses. > > There's another problem buried here, which is that the proposed > feature doesn't delete the static routes when the zone is shut down. > It doesn't delete them because it's trying to cover for the EEXIST > case. > > I think the right thing to do is to remember when the "route add" > attempt fails, and conditionally remove the route on zone shutdown. > That potentially leaves a zone relying on a duplicate out in the cold, > but since it's a misconfiguration anyway, it doesn't seem like a big > problem. > this seems sloppy. i think that when zones are shutdown they should remote any default routes that they installed. and if it's a misconfiguration to boot two zones with the same default route then we shouldn't allow multiple zones to boot into this state. instead when the user boots subsequent zones that would have overlapping default routes we should generate an error message telling the user that the configuration is incorrect. ed From Glenn.Faden@sun.com Wed Jan 30 11:14:55 2008 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01 [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UJEtVb010860 for ; Wed, 30 Jan 2008 11:14:55 -0800 (PST) Received: from sca-es-mail-1.sun.com (sca-es-mail-1.Sun.COM [192.18.43.132]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0UJEtdN056238 for ; Wed, 30 Jan 2008 11:14:55 -0800 (PST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m0UJEoOb001386 for ; Wed, 30 Jan 2008 11:14:50 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JVG00201ZZ6VQ00@fe-sfbay-09.sun.com> (original mail from Glenn.Faden@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Wed, 30 Jan 2008 11:14:49 -0800 (PST) Received: from rampartgf.local ([129.150.18.193]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JVH009YP1GFPGG0@fe-sfbay-09.sun.com>; Wed, 30 Jan 2008 11:14:40 -0800 (PST) Date: Wed, 30 Jan 2008 11:14:42 -0800 From: Glenn Faden Subject: Re: 2008/057 Default Route For Zones In-reply-to: <20080130173728.GB732888@eng.sun.com> Sender: Glenn.Faden@sun.com To: Edward Pilatowicz Cc: James Carlson , Erik Nordmark , Glenn Faden , psarc-ext@sac.sfbay.sun.com Message-id: <47A0CCA2.4030301@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <18336.35131.636630.805135@gargle.gargle.HOWL> <20080130173728.GB732888@eng.sun.com> User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) Status: RO Content-Length: 2198 Edward Pilatowicz wrote: > On Wed, Jan 30, 2008 at 09:27:07AM -0500, James Carlson wrote: > >> Erik Nordmark writes: >> >>> However, if the ngz's shares a subnet with some other zone, then the >>> current logic in the kernel isn't capable of supporting a different >>> default route for different zones. This is because the kernel check is >>> whether the gateway field in the default route is on the same subnet as >>> one of the zone's IP addresses. >>> >> There's another problem buried here, which is that the proposed >> feature doesn't delete the static routes when the zone is shut down. >> It doesn't delete them because it's trying to cover for the EEXIST >> case. >> >> I think the right thing to do is to remember when the "route add" >> attempt fails, and conditionally remove the route on zone shutdown. >> That potentially leaves a zone relying on a duplicate out in the cold, >> but since it's a misconfiguration anyway, it doesn't seem like a big >> problem. >> >> > > this seems sloppy. i think that when zones are shutdown they > should remote any default routes that they installed. > Currently the code in zoneadmd doesn't consult the zone configuration file when removing the network interfaces. Instead, it examines all the plumbed interfaces and removes those that belong to the zone. There wasn't a convenient place in the code to remove the route, but I'm not sure it can be done safely. > and if it's a misconfiguration to boot two zones with the same > default route then we shouldn't allow multiple zones to boot into > this state. instead when the user boots subsequent zones that would > have overlapping default routes we should generate an error message > telling the user that the configuration is incorrect. > If we're going to remove the route when the zone is halted, then we have to prevent the another zone from using the same route. However, I think that adding this complexity will not improve the customer experience. For example, I'm afraid that there may be race conditions between a zone that is booting and one that is shutting down with the same default route. The simpler solution seemed adequate to me. --Glenn From carlsonj@phorcys.east.sun.com Wed Jan 30 11:30:45 2008 Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UJUiZO011852 for ; Wed, 30 Jan 2008 11:30:44 -0800 (PST) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m0UJUiXN015212; Wed, 30 Jan 2008 14:30:44 -0500 (EST) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.2+Sun/8.14.2/Submit) id m0UJUigt015209; Wed, 30 Jan 2008 14:30:44 -0500 (EST) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18336.53348.123514.290889@gargle.gargle.HOWL> Date: Wed, 30 Jan 2008 14:30:44 -0500 From: James Carlson To: Glenn Faden Cc: Edward Pilatowicz , Erik Nordmark , Glenn Faden , psarc-ext@sac.sfbay.sun.com Subject: Re: 2008/057 Default Route For Zones In-Reply-To: <47A0CCA2.4030301@sun.com> References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <18336.35131.636630.805135@gargle.gargle.HOWL> <20080130173728.GB732888@eng.sun.com> <47A0CCA2.4030301@sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 1221 Glenn Faden writes: > If we're going to remove the route when the zone is halted, then we have > to prevent the another zone from using the same route. However, I think > that adding this complexity will not improve the customer experience. > For example, I'm afraid that there may be race conditions between a zone > that is booting and one that is shutting down with the same default > route. The simpler solution seemed adequate to me. I don't think it is. Try this: - set up a default route to 10.0.0.1 for a zone. - boot it. - shut it down. - change that default route to 10.0.0.2. - boot it again. - examine the kernel's forwarding (routing) table. You should end up with two default routes; one to the new destination, and another to the old destination. The system caches these things. ("-nostatic" might do the trick, but I'm not positive.) As for the overlap case, I'm happy with just saying in the documentation that this configuration is a mistake: don't do it. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From erik.nordmark@sun.com Wed Jan 30 12:18:37 2008 Received: from jurassic.eng.sun.com (jurassic-226-b [129.146.226.130]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UKIb6X017817 for ; Wed, 30 Jan 2008 12:18:37 -0800 (PST) Received: from [10.7.251.248] (punchin-nordmark.SFBay.Sun.COM [10.7.251.248]) by jurassic.eng.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UKIZM0926966 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 30 Jan 2008 12:18:35 -0800 (PST) Message-ID: <47A0DB9A.7030606@sun.com> Date: Wed, 30 Jan 2008 12:18:34 -0800 From: Erik Nordmark User-Agent: Thunderbird 2.0.0.4 (X11/20070723) MIME-Version: 1.0 To: James Carlson CC: Glenn Faden , psarc-ext@sac.sfbay.sun.com, Glenn Faden Subject: Re: 2008/057 Default Route For Zones References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <18336.35131.636630.805135@gargle.gargle.HOWL> In-Reply-To: <18336.35131.636630.805135@gargle.gargle.HOWL> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 1677 James Carlson wrote: > Needless to say, there are a few sharp edges here for the unwary user. > The original design goal of Zones was simply for consolidation on > shared networks, not separated ones, and stepping outside of that has > been a fairly persistent source of pain. Yes, there was a reason that I spent some time to build the exclusive-IP zone support; to many cases of trying to push the shared-IP zones outside of their original design center. But it will take some time before the customers that need this flexibility can move over to using exclusive-IP zones; what is important is that we work actively to remove any obstacles to using exclusive-IP zones. (I believe we have some work to do for TX here.) >> Would it make sense to somehow restrict this property to the case when >> the ngz has IP address(es) that do not have a common subnet with any >> other zone on the system? > > If the "restriction" is in the documentation provided for the feature, > then I would agree. Otherwise, I have to agree with Glenn that static > checks against the configured zones don't necessarily do the right > thing, as users are under no obligation to boot and run all installed > zones all of the time. We do know that there is a number of users that will benefit from this - those that use different subnets for different zones (whether or not they use TX). But what is hard to predict is to what extent other users will end up trying to use this feature and call us when it doesn't work. I wouldn't mind seeing the draft documentation changes (and commitment from the docs/man page folks to ship those) before we ship the implementation. Erik From carlsonj@phorcys.east.sun.com Wed Jan 30 12:28:04 2008 Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UKS4sX017893 for ; Wed, 30 Jan 2008 12:28:04 -0800 (PST) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m0UKS4hm015597; Wed, 30 Jan 2008 15:28:04 -0500 (EST) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.2+Sun/8.14.2/Submit) id m0UKS3Va015594; Wed, 30 Jan 2008 15:28:03 -0500 (EST) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18336.56787.187799.250005@gargle.gargle.HOWL> Date: Wed, 30 Jan 2008 15:28:03 -0500 From: James Carlson To: Erik Nordmark Cc: Glenn Faden , psarc-ext@sac.sfbay.sun.com, Glenn Faden Subject: Re: 2008/057 Default Route For Zones In-Reply-To: <47A0DB9A.7030606@sun.com> References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <18336.35131.636630.805135@gargle.gargle.HOWL> <47A0DB9A.7030606@sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 1045 Erik Nordmark writes: > But what is hard to predict is to what extent other users will end up > trying to use this feature and call us when it doesn't work. Based on history here, I can guarantee that users will attempt to use the system in ways we haven't predicted and even ways we explicitly said it wouldn't work, and still call us when it hits the fan. No problem there. ;-} > I wouldn't mind seeing the draft documentation changes (and commitment > from the docs/man page folks to ship those) before we ship the > implementation. The draft documentation request seems quite fair here. Glenn? (I don't see how docs commitment is an architectural matter, though. That coordination -- making sure that resources are in place and associated documentation changes are committed -- is more of a C-team matter.) -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From Glenn.Faden@sun.com Wed Jan 30 12:32:31 2008 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01 [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UKWUcq017926 for ; Wed, 30 Jan 2008 12:32:30 -0800 (PST) Received: from sca-es-mail-2.sun.com (sca-es-mail-2.Sun.COM [192.18.43.133]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0UKWU7i048737 for ; Wed, 30 Jan 2008 12:32:30 -0800 (PST) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m0UKWPVY003444 for ; Wed, 30 Jan 2008 12:32:25 -0800 (PST) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JVH00G0151GCB00@fe-sfbay-10.sun.com> (original mail from Glenn.Faden@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Wed, 30 Jan 2008 12:32:25 -0800 (PST) Received: from rampartgf.local ([129.150.18.193]) by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JVH00DM15215R10@fe-sfbay-10.sun.com>; Wed, 30 Jan 2008 12:32:25 -0800 (PST) Date: Wed, 30 Jan 2008 12:32:28 -0800 From: Glenn Faden Subject: Re: 2008/057 Default Route For Zones In-reply-to: <47A0DB9A.7030606@sun.com> Sender: Glenn.Faden@sun.com To: Erik Nordmark Cc: James Carlson , psarc-ext@sac.sfbay.sun.com, Glenn Faden Message-id: <47A0DEDC.202@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <18336.35131.636630.805135@gargle.gargle.HOWL> <47A0DB9A.7030606@sun.com> User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) Status: RO Content-Length: 686 Erik Nordmark wrote: > > We do know that there is a number of users that will benefit from this > - those that use different subnets for different zones (whether or not > they use TX). > > But what is hard to predict is to what extent other users will end up > trying to use this feature and call us when it doesn't work. > > I wouldn't mind seeing the draft documentation changes (and commitment > from the docs/man page folks to ship those) before we ship the > implementation. I'll certainly include you in the review of the final man page update. I've already filed a bug with the suggested changes, but this may be modified based on issues raised in this thread. --Glenn From Ken.Powell@sun.com Wed Jan 30 12:35:41 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02 [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UKZfSU017941 for ; Wed, 30 Jan 2008 12:35:41 -0800 (PST) Received: from brmea-mail-3.sun.com (brmea-mail-3.Sun.COM [192.18.98.34]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0UKZfjr056284 for ; Wed, 30 Jan 2008 12:35:41 -0800 (PST) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-3.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m0UKZfuZ008163 for ; Wed, 30 Jan 2008 20:35:41 GMT Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JVH00E0142BQV00@mail-amer.sun.com> (original mail from Ken.Powell@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Wed, 30 Jan 2008 13:35:41 -0700 (MST) Received: from [129.148.174.109] by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JVH006V357BLEA0@mail-amer.sun.com>; Wed, 30 Jan 2008 13:35:36 -0700 (MST) Date: Wed, 30 Jan 2008 15:35:44 -0500 From: Ken Powell Subject: Re: 2008/057 Default Route For Zones In-reply-to: <47A01B3F.50103@sun.com> Sender: Ken.Powell@sun.com To: Glenn Faden Cc: Erik Nordmark , James Carlson , psarc-ext@sac.sfbay.sun.com, Glenn Faden Message-id: <47A0DFA0.7040908@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> User-Agent: Thunderbird 2.0.0.6 (X11/20070825) Status: RO Content-Length: 587 Glenn Faden wrote: > I have tried to avoid making this feature only apply to Trusted > Extensions, although that is the customer base that is requesting it. > For TX customers each zone has its own subnet and its own interface. The customer I know about has a more complex situation. They have multiple physically independent networks, but some of the networks carry information for multiple compartments. This indicates multiple zones will need to map to the same subnet. I believe in their case, all the non-global zones on a given subnet will have the same default route. Ken From edward.pilatowicz@sun.com Wed Jan 30 13:04:23 2008 Received: from dm-eng-02.sfbay.sun.com (dm-eng-02 [129.146.11.32]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UL4MEc018951 for ; Wed, 30 Jan 2008 13:04:22 -0800 (PST) Received: from mcescher.eng.sun.com (mcescher.SFBay.Sun.COM [129.146.224.55]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0UL4MeK006106; Wed, 30 Jan 2008 13:04:22 -0800 (PST) Received: from mcescher.eng.sun.com (localhost [127.0.0.1]) by mcescher.eng.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m0UL4MU4610076; Wed, 30 Jan 2008 13:04:22 -0800 (PST) Received: (from edp@localhost) by mcescher.eng.sun.com (8.14.2+Sun/8.14.2/Submit) id m0UL4MdH610020; Wed, 30 Jan 2008 13:04:22 -0800 (PST) X-Authentication-Warning: mcescher.eng.sun.com: edp set sender to edward.pilatowicz@sun.com using -f Date: Wed, 30 Jan 2008 13:04:21 -0800 From: Edward Pilatowicz To: Ken Powell Cc: Glenn Faden , Erik Nordmark , Glenn Faden , psarc-ext@sac.sfbay.sun.com, James Carlson Subject: Re: 2008/057 Default Route For Zones Message-ID: <20080130210421.GD732888@eng.sun.com> References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <47A0DFA0.7040908@sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <47A0DFA0.7040908@sun.com> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 999 i can compleatly imagine people trying to do what you describe below. which is why i requested that the tools fail if a user tries to setup this configuration. it's not just enough to document this limitation. On Wed, Jan 30, 2008 at 03:35:44PM -0500, Ken Powell wrote: > Glenn Faden wrote: > > I have tried to avoid making this feature only apply to Trusted > > Extensions, although that is the customer base that is requesting it. > > For TX customers each zone has its own subnet and its own interface. > The customer I know about has a more complex situation. They have > multiple physically independent networks, but some of the networks carry > information for multiple compartments. This indicates multiple zones > will need to map to the same subnet. I believe in their case, all the > non-global zones on a given subnet will have the same default route. > > Ken > > _______________________________________________ > opensolaris-arc mailing list > opensolaris-arc@opensolaris.org From erik.nordmark@sun.com Wed Jan 30 13:06:20 2008 Received: from jurassic.eng.sun.com (jurassic-226-b [129.146.226.130]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UL6KsL018986 for ; Wed, 30 Jan 2008 13:06:20 -0800 (PST) Received: from [10.7.251.248] (punchin-nordmark.SFBay.Sun.COM [10.7.251.248]) by jurassic.eng.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UL6JlC928176 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 30 Jan 2008 13:06:20 -0800 (PST) Message-ID: <47A0E6CB.2000301@sun.com> Date: Wed, 30 Jan 2008 13:06:19 -0800 From: Erik Nordmark User-Agent: Thunderbird 2.0.0.4 (X11/20070723) MIME-Version: 1.0 To: James Carlson CC: Glenn Faden , psarc-ext@sac.sfbay.sun.com, Glenn Faden Subject: Re: 2008/057 Default Route For Zones References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <18336.35131.636630.805135@gargle.gargle.HOWL> <47A0DB9A.7030606@sun.com> <18336.56787.187799.250005@gargle.gargle.HOWL> In-Reply-To: <18336.56787.187799.250005@gargle.gargle.HOWL> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 581 James Carlson wrote: >> I wouldn't mind seeing the draft documentation changes (and commitment >> from the docs/man page folks to ship those) before we ship the >> implementation. > > The draft documentation request seems quite fair here. Glenn? > > (I don't see how docs commitment is an architectural matter, though. > That coordination -- making sure that resources are in place and > associated documentation changes are committed -- is more of a C-team > matter.) Agreed (which is why I said "before we ship the implementation" i.e. prior to integration.) Erik From edward.pilatowicz@sun.com Wed Jan 30 13:08:03 2008 Received: from dm-eng-02.sfbay.sun.com (dm-eng-02 [129.146.11.32]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UL83T0019162 for ; Wed, 30 Jan 2008 13:08:03 -0800 (PST) Received: from mcescher.eng.sun.com (mcescher.SFBay.Sun.COM [129.146.224.55]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0UL83Xo008036; Wed, 30 Jan 2008 13:08:03 -0800 (PST) Received: from mcescher.eng.sun.com (localhost [127.0.0.1]) by mcescher.eng.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m0UL83TS624445; Wed, 30 Jan 2008 13:08:03 -0800 (PST) Received: (from edp@localhost) by mcescher.eng.sun.com (8.14.2+Sun/8.14.2/Submit) id m0UL82x9624421; Wed, 30 Jan 2008 13:08:02 -0800 (PST) X-Authentication-Warning: mcescher.eng.sun.com: edp set sender to edward.pilatowicz@sun.com using -f Date: Wed, 30 Jan 2008 13:08:02 -0800 From: Edward Pilatowicz To: James Carlson Cc: Glenn Faden , Erik Nordmark , Glenn Faden , psarc-ext@sac.sfbay.sun.com Subject: Re: 2008/057 Default Route For Zones Message-ID: <20080130210802.GE732888@eng.sun.com> References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <18336.35131.636630.805135@gargle.gargle.HOWL> <20080130173728.GB732888@eng.sun.com> <47A0CCA2.4030301@sun.com> <18336.53348.123514.290889@gargle.gargle.HOWL> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <18336.53348.123514.290889@gargle.gargle.HOWL> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 1440 On Wed, Jan 30, 2008 at 02:30:44PM -0500, James Carlson wrote: > Glenn Faden writes: > > If we're going to remove the route when the zone is halted, then we have > > to prevent the another zone from using the same route. However, I think > > that adding this complexity will not improve the customer experience. > > For example, I'm afraid that there may be race conditions between a zone > > that is booting and one that is shutting down with the same default > > route. The simpler solution seemed adequate to me. > > I don't think it is. Try this: > > - set up a default route to 10.0.0.1 for a zone. > - boot it. > - shut it down. > - change that default route to 10.0.0.2. > - boot it again. > - examine the kernel's forwarding (routing) table. > > You should end up with two default routes; one to the new destination, > and another to the old destination. The system caches these things. > ("-nostatic" might do the trick, but I'm not positive.) > this is really bad. i really think that a zone should cleanup any routes it added to the system when it's shutdown. we don't want zones leaving configuration turds lying around the system after zones have been shutdown. if a zone adds a route to the system successfully it needs to record this fact in the zone state, a zone attribute would be perfect for this. then at zone shutdown the framework can check to see if there are any routes it needs to tear down. ed From erik.nordmark@sun.com Wed Jan 30 13:10:16 2008 Received: from jurassic.eng.sun.com (jurassic-226-b [129.146.226.130]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0ULAGbw019205 for ; Wed, 30 Jan 2008 13:10:16 -0800 (PST) Received: from [10.7.251.248] (punchin-nordmark.SFBay.Sun.COM [10.7.251.248]) by jurassic.eng.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0ULAFcE928233 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 30 Jan 2008 13:10:15 -0800 (PST) Message-ID: <47A0E7B7.5080102@sun.com> Date: Wed, 30 Jan 2008 13:10:15 -0800 From: Erik Nordmark User-Agent: Thunderbird 2.0.0.4 (X11/20070723) MIME-Version: 1.0 To: Glenn Faden CC: James Carlson , psarc-ext@sac.sfbay.sun.com, Glenn Faden Subject: Re: 2008/057 Default Route For Zones References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <18336.35131.636630.805135@gargle.gargle.HOWL> <47A0DB9A.7030606@sun.com> <47A0DEDC.202@sun.com> In-Reply-To: <47A0DEDC.202@sun.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 383 Glenn Faden wrote: > I'll certainly include you in the review of the final man page update. > I've already filed a bug with the suggested changes, but this may be > modified based on issues raised in this thread. In addition to man pages I think we should also update the zone book. E.g., http://docs.sun.com/app/docs/doc/819-2450/6n4o5mdik?l=en&a=view&q=exclusive-ip Erik From Glenn.Faden@sun.com Wed Jan 30 13:56:02 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02 [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0ULu2DA022315 for ; Wed, 30 Jan 2008 13:56:02 -0800 (PST) Received: from sca-es-mail-1.sun.com (sca-es-mail-1.Sun.COM [192.18.43.132]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0ULu23q035491 for ; Wed, 30 Jan 2008 13:56:02 -0800 (PST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m0ULtv2G021125 for ; Wed, 30 Jan 2008 13:55:57 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JVH00F018T4DJ00@fe-sfbay-09.sun.com> (original mail from Glenn.Faden@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Wed, 30 Jan 2008 13:55:57 -0800 (PST) Received: from rampartgf.local ([129.150.18.193]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JVH003478X8F5F0@fe-sfbay-09.sun.com>; Wed, 30 Jan 2008 13:55:56 -0800 (PST) Date: Wed, 30 Jan 2008 13:55:59 -0800 From: Glenn Faden Subject: Re: 2008/057 Default Route For Zones In-reply-to: <20080130210421.GD732888@eng.sun.com> Sender: Glenn.Faden@sun.com To: Edward Pilatowicz Cc: Ken Powell , Erik Nordmark , Glenn Faden , psarc-ext@sac.sfbay.sun.com, James Carlson Message-id: <47A0F26F.2010803@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <47A0DFA0.7040908@sun.com> <20080130210421.GD732888@eng.sun.com> User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) Status: RO Content-Length: 1498 Edward Pilatowicz wrote: > i can compleatly imagine people trying to do what you describe > below. which is why i requested that the tools fail if a user > tries to setup this configuration. it's not just enough to > document this limitation. > I can't see how punishing the customer is a winning strategy. The proposed implementation supports the scenario that Ken Powell has described. Your proposal doesn't. FYI, these systems are currently being used in critical government installations and the customer has requested an enhancement (which has been escalated) to simplify the administrative complexity. Their current workaround is untenable. --Glenn > On Wed, Jan 30, 2008 at 03:35:44PM -0500, Ken Powell wrote: > >> Glenn Faden wrote: >> >>> I have tried to avoid making this feature only apply to Trusted >>> Extensions, although that is the customer base that is requesting it. >>> For TX customers each zone has its own subnet and its own interface. >>> >> The customer I know about has a more complex situation. They have >> multiple physically independent networks, but some of the networks carry >> information for multiple compartments. This indicates multiple zones >> will need to map to the same subnet. I believe in their case, all the >> non-global zones on a given subnet will have the same default route. >> >> Ken >> >> _______________________________________________ >> opensolaris-arc mailing list >> opensolaris-arc@opensolaris.org >> From erik.nordmark@sun.com Wed Jan 30 14:13:01 2008 Received: from jurassic.eng.sun.com (jurassic-226-b [129.146.226.130]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UMD1Ua023848 for ; Wed, 30 Jan 2008 14:13:01 -0800 (PST) Received: from [10.7.251.248] (punchin-nordmark.SFBay.Sun.COM [10.7.251.248]) by jurassic.eng.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UMCxEr929153 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 30 Jan 2008 14:13:00 -0800 (PST) Message-ID: <47A0F66B.6040300@sun.com> Date: Wed, 30 Jan 2008 14:12:59 -0800 From: Erik Nordmark User-Agent: Thunderbird 2.0.0.4 (X11/20070723) MIME-Version: 1.0 To: Glenn Faden CC: Edward Pilatowicz , Ken Powell , Glenn Faden , psarc-ext@sac.sfbay.sun.com, James Carlson Subject: Re: 2008/057 Default Route For Zones References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <47A0DFA0.7040908@sun.com> <20080130210421.GD732888@eng.sun.com> <47A0F26F.2010803@sun.com> In-Reply-To: <47A0F26F.2010803@sun.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 934 Glenn Faden wrote: > I can't see how punishing the customer is a winning strategy. The > proposed implementation supports the scenario that Ken Powell has > described. Your proposal doesn't. FYI, these systems are currently being > used in critical government installations and the customer has requested > an enhancement (which has been escalated) to simplify the administrative > complexity. Their current workaround is untenable. Glenn, I don't think Ed is trying to punish the customer. This part of the system is extremely complex and brittle - it isn't built to be able to do per-zones routes. Hence being concerned about customers creating invalid configurations is a concern to be taken very seriously. Perhaps we should go back to the drawing port and design per zones routes i.e., the ability for route(1m) to explicitly specify a zone name and have that cause ire_zoneid to be set in the kernel. Erik From ceri@submonkey.net Wed Jan 30 14:14:26 2008 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01 [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UMEQZn023863 for ; Wed, 30 Jan 2008 14:14:26 -0800 (PST) Received: from brmea-mail-1.sun.com (brmea-mail-1.Sun.COM [192.18.98.31]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0UMEPXL046414 for ; Wed, 30 Jan 2008 14:14:26 -0800 (PST) Received: from relay23.sun.com (relay23.sun.com [192.12.251.54] (may be forged)) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m0UM4g48000392 for ; Wed, 30 Jan 2008 22:14:25 GMT Received: from mms24es.sun.com ([150.143.232.74] [150.143.232.74]) by relay23i.sun.com with ESMTP id BT-MMP-2488767 for psarc-ext@sac.sfbay.sun.com; Wed, 30 Jan 2008 22:14:25 Z Received: from relay25.sun.com (relay25.sun.com [192.12.251.14]) by mms24es.sun.com with ESMTP id BT-MMP-217990 for psarc-ext@sac.sfbay.sun.com; Wed, 30 Jan 2008 22:14:23 Z Received: from shrike.submonkey.net ([81.106.128.65] [81.106.128.65]) by relay25i.sun.com with ESMTP id BT-MMP-14567584 for psarc-ext@sac.sfbay.sun.com; Wed, 30 Jan 2008 22:14:22 Z Received: from ceri by shrike.submonkey.net with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1JKLCH-000HVs-2n; Wed, 30 Jan 2008 22:14:21 +0000 Date: Wed, 30 Jan 2008 22:14:21 +0000 From: Ceri Davies To: Edward Pilatowicz Cc: James Carlson , Glenn Faden , Erik Nordmark , Glenn Faden , psarc-ext@sac.sfbay.sun.com Subject: Re: 2008/057 Default Route For Zones Message-ID: <20080130221420.GY6064@submonkey.net> References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <18336.35131.636630.805135@gargle.gargle.HOWL> <20080130173728.GB732888@eng.sun.com> In-Reply-To: <20080130173728.GB732888@eng.sun.com> X-Brightmail-Tracker: AAAAAA== X-PGP: finger ceri@FreeBSD.org User-Agent: Mutt/1.5.17 (2007-11-01) Sender: Ceri Davies X-Antispam: No, score=-2.6/5.0, scanned in 2.052sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="TXIPBuAs4GDcsx9K" Content-Disposition: inline Status: RO Content-Length: 3097 --TXIPBuAs4GDcsx9K Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Wed, Jan 30, 2008 at 09:37:28AM -0800, Edward Pilatowicz wrote: > On Wed, Jan 30, 2008 at 09:27:07AM -0500, James Carlson wrote: > > Erik Nordmark writes: > > > However, if the ngz's shares a subnet with some other zone, then the > > > current logic in the kernel isn't capable of supporting a different > > > default route for different zones. This is because the kernel check is > > > whether the gateway field in the default route is on the same subnet = as > > > one of the zone's IP addresses. > > > > There's another problem buried here, which is that the proposed > > feature doesn't delete the static routes when the zone is shut down. > > It doesn't delete them because it's trying to cover for the EEXIST > > case. > > > > I think the right thing to do is to remember when the "route add" > > attempt fails, and conditionally remove the route on zone shutdown. > > That potentially leaves a zone relying on a duplicate out in the cold, > > but since it's a misconfiguration anyway, it doesn't seem like a big > > problem. >=20 > this seems sloppy. i think that when zones are shutdown they > should remote any default routes that they installed. >=20 > and if it's a misconfiguration to boot two zones with the same > default route then we shouldn't allow multiple zones to boot into > this state. instead when the user boots subsequent zones that would > have overlapping default routes we should generate an error message > telling the user that the configuration is incorrect. That sounds like it would cause a regression for us. We currently have dedicated interfaces for some zones; these zones live on the same subnet, one which is not intended to be used by either other local zones or the global zone. Since they are on the same subnet, they have the same default route and there isn't a problem. We use VCS Zone agents to online the zones and postonline triggers to bring the routes online; deleting the routes when the zone is shut down isn't really a problem in our situation because we don't care if it's there. Basically what I'm saying is that not having the route removed on zone shutdown is a much smaller problem than not being able to share a default route between zones which would basically disrupt our entire infrastructure. I don't expect you to do what we want, but just wanted to throw this out as a data point; in short, we are already rolling our own solution to the "routes for zones" problem and would rather that it not be fixed if that is going to introduce new limitations elsewhere. Ceri --=20 That must be wonderful! I don't understand it at all. -- Moliere --TXIPBuAs4GDcsx9K Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFHoPa8ocfcwTS3JF8RArMbAJ9udQWSpFJxYzKLr1AvFK/TKOfxAwCeL0SR 1s5g4p1NEdBUIeF4Q3+a4pg= =IapU -----END PGP SIGNATURE----- --TXIPBuAs4GDcsx9K-- From Glenn.Faden@sun.com Wed Jan 30 14:33:03 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02 [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UMX3To024183 for ; Wed, 30 Jan 2008 14:33:03 -0800 (PST) Received: from sca-es-mail-1.sun.com (sca-es-mail-1.Sun.COM [192.18.43.132]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0UMX3bT058674 for ; Wed, 30 Jan 2008 14:33:03 -0800 (PST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m0UMWwHl025480 for ; Wed, 30 Jan 2008 14:32:58 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JVH00E01AEU0000@fe-sfbay-09.sun.com> (original mail from Glenn.Faden@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Wed, 30 Jan 2008 14:32:58 -0800 (PST) Received: from rampartgf.local ([129.150.18.193]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JVH00GI7AMXFK10@fe-sfbay-09.sun.com>; Wed, 30 Jan 2008 14:32:58 -0800 (PST) Date: Wed, 30 Jan 2008 14:33:01 -0800 From: Glenn Faden Subject: Re: 2008/057 Default Route For Zones In-reply-to: <47A0F66B.6040300@sun.com> Sender: Glenn.Faden@sun.com To: Erik Nordmark Cc: Edward Pilatowicz , Ken Powell , Glenn Faden , psarc-ext@sac.sfbay.sun.com, James Carlson Message-id: <47A0FB1D.6030708@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <47A0DFA0.7040908@sun.com> <20080130210421.GD732888@eng.sun.com> <47A0F26F.2010803@sun.com> <47A0F66B.6040300@sun.com> User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) Status: RO Content-Length: 644 Erik Nordmark wrote: > > This part of the system is extremely complex and brittle - it isn't > built to be able to do per-zones routes. Hence being concerned about > customers creating invalid configurations is a concern to be taken > very seriously. > > Perhaps we should go back to the drawing port and design per zones > routes i.e., the ability for route(1m) to explicitly specify a zone > name and have that cause ire_zoneid to be set in the kernel. Perhaps, but the proposed solution meets the requirements of customers that have asked for this feature, and I'd like to get it integrated and backported in time for S10U6. --Glenn From edward.pilatowicz@sun.com Wed Jan 30 15:07:08 2008 Received: from dm-eng-01.sfbay.sun.com (dm-eng-01 [129.145.155.198]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UN78SH026824 for ; Wed, 30 Jan 2008 15:07:08 -0800 (PST) Received: from mcescher.eng.sun.com (mcescher.SFBay.Sun.COM [129.146.224.55]) by dm-eng-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0UN77Db015979; Wed, 30 Jan 2008 15:07:07 -0800 (PST) Received: from mcescher.eng.sun.com (localhost [127.0.0.1]) by mcescher.eng.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m0UN77o4708064; Wed, 30 Jan 2008 15:07:07 -0800 (PST) Received: (from edp@localhost) by mcescher.eng.sun.com (8.14.2+Sun/8.14.2/Submit) id m0UN77Ik708047; Wed, 30 Jan 2008 15:07:07 -0800 (PST) X-Authentication-Warning: mcescher.eng.sun.com: edp set sender to edward.pilatowicz@sun.com using -f Date: Wed, 30 Jan 2008 15:07:06 -0800 From: Edward Pilatowicz To: Ceri Davies Cc: James Carlson , Glenn Faden , Erik Nordmark , Glenn Faden , psarc-ext@sac.sfbay.sun.com Subject: Re: 2008/057 Default Route For Zones Message-ID: <20080130230706.GG732888@eng.sun.com> References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <18336.35131.636630.805135@gargle.gargle.HOWL> <20080130173728.GB732888@eng.sun.com> <20080130221420.GY6064@submonkey.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080130221420.GY6064@submonkey.net> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 4082 On Wed, Jan 30, 2008 at 10:14:21PM +0000, Ceri Davies wrote: > On Wed, Jan 30, 2008 at 09:37:28AM -0800, Edward Pilatowicz wrote: > > On Wed, Jan 30, 2008 at 09:27:07AM -0500, James Carlson wrote: > > > Erik Nordmark writes: > > > > However, if the ngz's shares a subnet with some other zone, then the > > > > current logic in the kernel isn't capable of supporting a different > > > > default route for different zones. This is because the kernel check is > > > > whether the gateway field in the default route is on the same subnet as > > > > one of the zone's IP addresses. > > > > > > There's another problem buried here, which is that the proposed > > > feature doesn't delete the static routes when the zone is shut down. > > > It doesn't delete them because it's trying to cover for the EEXIST > > > case. > > > > > > I think the right thing to do is to remember when the "route add" > > > attempt fails, and conditionally remove the route on zone shutdown. > > > That potentially leaves a zone relying on a duplicate out in the cold, > > > but since it's a misconfiguration anyway, it doesn't seem like a big > > > problem. > > > > this seems sloppy. i think that when zones are shutdown they > > should remote any default routes that they installed. > > > > and if it's a misconfiguration to boot two zones with the same > > default route then we shouldn't allow multiple zones to boot into > > this state. instead when the user boots subsequent zones that would > > have overlapping default routes we should generate an error message > > telling the user that the configuration is incorrect. > > That sounds like it would cause a regression for us. > > We currently have dedicated interfaces for some zones; these zones live > on the same subnet, one which is not intended to be used by either other > local zones or the global zone. Since they are on the same subnet, they > have the same default route and there isn't a problem. We use VCS Zone > agents to online the zones and postonline triggers to bring the routes > online; deleting the routes when the zone is shut down isn't really a > problem in our situation because we don't care if it's there. > > Basically what I'm saying is that not having the route removed on zone > shutdown is a much smaller problem than not being able to share a > default route between zones which would basically disrupt our entire > infrastructure. I don't expect you to do what we want, but just wanted > to throw this out as a data point; in short, we are already rolling our > own solution to the "routes for zones" problem and would rather that it > not be fixed if that is going to introduce new limitations elsewhere. > this would not be a regression for you. since you're already doing this you've obviously worked around the problem. your workaound should continue to work regardless of the changes introduced by this proposal. (if it wouldn't i'd be curious to see why.) the way this would impact you is if: - you tried to remove your workaround and use the new defrouter zonecfg method - and that method did not allow multiple zones to use the same route. having new functionality that doesn't fullfuill your requirements is hardly a regression. (otherwise i'd ask the zfs team to fix my desktop problems with their next RFE.) james carlson said he was ok with allowing the user to create these types of configurations, but then just documenting that they are unsupported. i disagree. already we've had two people (you and ken) say that you plan to use this configuration that would be documented as unsupported. if there's such a strong requirement for this configuration then this proposal should either address it or prevent it. you can always continue to use your current workaround. also, i'm not saying that we can't shouldn't support this type of functionality. but if we want to support it then i'm saying we should do it correctly and associate the routes with the zones that depend on them such that when all the zones which use the same default route go away, so does that default route. ed From ceri@submonkey.net Wed Jan 30 15:25:09 2008 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01 [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0UNP9pk027534 for ; Wed, 30 Jan 2008 15:25:09 -0800 (PST) Received: from sca-ea-mail-1.sun.com (sca-ea-mail-1.Sun.COM [192.18.43.24]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0UNP9c8027239 for ; Wed, 30 Jan 2008 15:25:09 -0800 (PST) Received: from relay18i.sun.com (ip128.net129179-4.block1.us.syntegra.com [129.179.4.128]) by sca-ea-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m0UNP8gp022583 for ; Wed, 30 Jan 2008 23:25:08 GMT Received: from mmp13es.sun.com ([160.41.209.23] [160.41.209.23]) by relay18i.sun.com with ESMTP id BT-MMP-149750 for psarc-ext@sac.sfbay.sun.com; Wed, 30 Jan 2008 23:25:05 Z Received: from relay11i.sun.com (relay11i.sun.com [129.179.4.121]) by mmp13es.sun.com with ESMTP id BT-MMP-99718 for psarc-ext@sac.sfbay.sun.com; Wed, 30 Jan 2008 23:25:05 Z Received: from shrike.submonkey.net ([81.106.128.65] [81.106.128.65]) by relay1i.sun.com with ESMTP id BT-MMP-995764 for psarc-ext@sac.sfbay.sun.com; Wed, 30 Jan 2008 23:25:04 Z Received: from ceri by shrike.submonkey.net with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1JKMIM-0008aI-Lc; Wed, 30 Jan 2008 23:24:42 +0000 Date: Wed, 30 Jan 2008 23:24:42 +0000 From: Ceri Davies To: Edward Pilatowicz Cc: James Carlson , Glenn Faden , Erik Nordmark , Glenn Faden , psarc-ext@sac.sfbay.sun.com Subject: Re: 2008/057 Default Route For Zones Message-ID: <20080130232442.GC6064@submonkey.net> References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <18336.35131.636630.805135@gargle.gargle.HOWL> <20080130173728.GB732888@eng.sun.com> <20080130221420.GY6064@submonkey.net> <20080130230706.GG732888@eng.sun.com> In-Reply-To: <20080130230706.GG732888@eng.sun.com> X-Brightmail-Tracker: AAAAAA== X-PGP: finger ceri@FreeBSD.org User-Agent: Mutt/1.5.17 (2007-11-01) Sender: Ceri Davies X-Antispam: No, score=-2.6/5.0, scanned in 0.225sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="zsAhXfiBV62A5hVr" Content-Disposition: inline Status: RO Content-Length: 4128 --zsAhXfiBV62A5hVr Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Wed, Jan 30, 2008 at 03:07:06PM -0800, Edward Pilatowicz wrote: > On Wed, Jan 30, 2008 at 10:14:21PM +0000, Ceri Davies wrote: > > On Wed, Jan 30, 2008 at 09:37:28AM -0800, Edward Pilatowicz wrote: > > > On Wed, Jan 30, 2008 at 09:27:07AM -0500, James Carlson wrote: > > > > Erik Nordmark writes: > > > > > However, if the ngz's shares a subnet with some other zone, then = the > > > > > current logic in the kernel isn't capable of supporting a differe= nt > > > > > default route for different zones. This is because the kernel che= ck is > > > > > whether the gateway field in the default route is on the same sub= net as > > > > > one of the zone's IP addresses. > > > > > > > > There's another problem buried here, which is that the proposed > > > > feature doesn't delete the static routes when the zone is shut down. > > > > It doesn't delete them because it's trying to cover for the EEXIST > > > > case. > > > > > > > > I think the right thing to do is to remember when the "route add" > > > > attempt fails, and conditionally remove the route on zone shutdown. > > > > That potentially leaves a zone relying on a duplicate out in the co= ld, > > > > but since it's a misconfiguration anyway, it doesn't seem like a big > > > > problem. > > > > > > this seems sloppy. i think that when zones are shutdown they > > > should remote any default routes that they installed. > > > > > > and if it's a misconfiguration to boot two zones with the same > > > default route then we shouldn't allow multiple zones to boot into > > > this state. instead when the user boots subsequent zones that would > > > have overlapping default routes we should generate an error message > > > telling the user that the configuration is incorrect. > > > > That sounds like it would cause a regression for us. > > > > We currently have dedicated interfaces for some zones; these zones live > > on the same subnet, one which is not intended to be used by either other > > local zones or the global zone. Since they are on the same subnet, they > > have the same default route and there isn't a problem. We use VCS Zone > > agents to online the zones and postonline triggers to bring the routes > > online; deleting the routes when the zone is shut down isn't really a > > problem in our situation because we don't care if it's there. > > > > Basically what I'm saying is that not having the route removed on zone > > shutdown is a much smaller problem than not being able to share a > > default route between zones which would basically disrupt our entire > > infrastructure. I don't expect you to do what we want, but just wanted > > to throw this out as a data point; in short, we are already rolling our > > own solution to the "routes for zones" problem and would rather that it > > not be fixed if that is going to introduce new limitations elsewhere. > > >=20 > this would not be a regression for you. >=20 > since you're already doing this you've obviously worked around the proble= m. > your workaound should continue to work regardless of the changes > introduced by this proposal. (if it wouldn't i'd be curious to see why.) While you're obviously correct, I prefer to use the provided mechanisms where they exist. I agree with everything you've said, I would just prefer that the second clause below weren't the case: > the way this would impact you is if: > - you tried to remove your workaround and use the new defrouter zonecfg > method > - and that method did not allow multiple zones to use the same route. Just a statement of preference, that is all. Ceri --=20 That must be wonderful! I don't understand it at all. -- Moliere --zsAhXfiBV62A5hVr Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFHoQc6ocfcwTS3JF8RAvBwAKCD0nSFjVDO2mXzKBLUbEp3/71inQCglJVI PbSxQMbCFkndB0SZtjO0JIo= =Qw6y -----END PGP SIGNATURE----- --zsAhXfiBV62A5hVr-- From erik.nordmark@sun.com Wed Jan 30 17:37:54 2008 Received: from jurassic.eng.sun.com (jurassic-226-a [129.146.226.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0V1bsDa001299 for ; Wed, 30 Jan 2008 17:37:54 -0800 (PST) Received: from [10.7.251.248] (punchin-nordmark.SFBay.Sun.COM [10.7.251.248]) by jurassic.eng.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0V1brih933684 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 30 Jan 2008 17:37:53 -0800 (PST) Message-ID: <47A12671.1030205@sun.com> Date: Wed, 30 Jan 2008 17:37:53 -0800 From: Erik Nordmark User-Agent: Thunderbird 2.0.0.4 (X11/20070723) MIME-Version: 1.0 To: Glenn Faden CC: Edward Pilatowicz , Ken Powell , Glenn Faden , psarc-ext@sac.sfbay.sun.com, James Carlson Subject: Re: 2008/057 Default Route For Zones References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <47A0DFA0.7040908@sun.com> <20080130210421.GD732888@eng.sun.com> <47A0F26F.2010803@sun.com> <47A0F66B.6040300@sun.com> <47A0FB1D.6030708@sun.com> In-Reply-To: <47A0FB1D.6030708@sun.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 884 Glenn Faden wrote: > Erik Nordmark wrote: >> >> This part of the system is extremely complex and brittle - it isn't >> built to be able to do per-zones routes. Hence being concerned about >> customers creating invalid configurations is a concern to be taken >> very seriously. >> >> Perhaps we should go back to the drawing port and design per zones >> routes i.e., the ability for route(1m) to explicitly specify a zone >> name and have that cause ire_zoneid to be set in the kernel. > Perhaps, but the proposed solution meets the requirements of customers > that have asked for this feature, and I'd like to get it integrated and > backported in time for S10U6. I'd encourage you at least reflect upon Ed's concerns before charging full steam ahead - it is good to know whether there is a cliff in front of you before you decide to dismiss the advise of others. Erik From Jarrett.Lu@sun.com Wed Jan 30 19:58:54 2008 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01 [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0V3wsmn004266 for ; Wed, 30 Jan 2008 19:58:54 -0800 (PST) Received: from sca-es-mail-2.sun.com (sca-es-mail-2.Sun.COM [192.18.43.133]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0V3wsiw008628 for ; Wed, 30 Jan 2008 19:58:54 -0800 (PST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m0V3wnJZ008954 for ; Wed, 30 Jan 2008 19:58:49 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JVH00501PPPY300@fe-sfbay-09.sun.com> (original mail from Jarrett.Lu@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Wed, 30 Jan 2008 19:58:49 -0800 (PST) Received: from [192.168.1.115] ([67.169.89.221]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JVH00JOTPPXYZC0@fe-sfbay-09.sun.com>; Wed, 30 Jan 2008 19:58:49 -0800 (PST) Date: Wed, 30 Jan 2008 19:58:35 -0800 From: Jarrett Lu Subject: Re: 2008/057 Default Route For Zones In-reply-to: <47A12671.1030205@sun.com> Sender: Jarrett.Lu@sun.com To: Erik Nordmark Cc: Glenn Faden , Edward Pilatowicz , Ken Powell , Glenn Faden , psarc-ext@sac.sfbay.sun.com, James Carlson Message-id: <47A1476B.70201@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <47A0DFA0.7040908@sun.com> <20080130210421.GD732888@eng.sun.com> <47A0F26F.2010803@sun.com> <47A0F66B.6040300@sun.com> <47A0FB1D.6030708@sun.com> <47A12671.1030205@sun.com> User-Agent: Thunderbird 2.0.0.6 (X11/20071203) Status: RO Content-Length: 2089 Erik Nordmark wrote: > Glenn Faden wrote: >> Erik Nordmark wrote: >>> >>> This part of the system is extremely complex and brittle - it isn't >>> built to be able to do per-zones routes. Hence being concerned about >>> customers creating invalid configurations is a concern to be taken >>> very seriously. >>> >>> Perhaps we should go back to the drawing port and design per zones >>> routes i.e., the ability for route(1m) to explicitly specify a zone >>> name and have that cause ire_zoneid to be set in the kernel. >> Perhaps, but the proposed solution meets the requirements of >> customers that have asked for this feature, and I'd like to get it >> integrated and backported in time for S10U6. > > I'd encourage you at least reflect upon Ed's concerns before charging > full steam ahead - it is good to know whether there is a cliff in > front of you before you decide to dismiss the advise of others. So far, I don't see a cliff yet. Ed's concern is mostly about not cleaning up default routes after zones are halted. While I think Ed has a point, I don't see zones not cleaning up their default routes upon halting as a disaster. We already have situations that default/static routes don't get cleaned up after they are no longer needed. Your earlier suggestion about adding zoneid to route(1M) is interesting. When ire_zoneid is set explicitly by "zoneid", it's clear who set the ire entry, who can use the entry, and who should clean up the entry. It also implies that the routing table could have many almost identical entries that only differ in ire_zoneid. This may have performance implications when finding a matching entry. Anyway, we haven't thought about the design enough. My suggestion, which an architecture purest may not like :-), is to let Glenn go forward as he planned. We have a separate effort to enhance zone routing in shared IP stack. The interface proposed in this case looks forward compatible with your "route add default " suggestion. We do have some paying customers needing immediate relief badly. Jarrett > > Erik From carlsonj@phorcys.east.sun.com Thu Jan 31 05:34:04 2008 Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0VDY46T016513 for ; Thu, 31 Jan 2008 05:34:04 -0800 (PST) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m0VDY38v018079; Thu, 31 Jan 2008 08:34:03 -0500 (EST) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.2+Sun/8.14.2/Submit) id m0VDY3Ve018076; Thu, 31 Jan 2008 08:34:03 -0500 (EST) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18337.52811.612279.691598@gargle.gargle.HOWL> Date: Thu, 31 Jan 2008 08:34:03 -0500 From: James Carlson To: Jarrett Lu Cc: Erik Nordmark , Glenn Faden , Edward Pilatowicz , psarc-ext@sac.sfbay.sun.com, Ken Powell Subject: Re: 2008/057 Default Route For Zones In-Reply-To: <47A1476B.70201@sun.com> References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <47A0DFA0.7040908@sun.com> <20080130210421.GD732888@eng.sun.com> <47A0F26F.2010803@sun.com> <47A0F66B.6040300@sun.com> <47A0FB1D.6030708@sun.com> <47A12671.1030205@sun.com> <47A1476B.70201@sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 2111 Jarrett Lu writes: > So far, I don't see a cliff yet. Ed's concern is mostly about not > cleaning up > default routes after zones are halted. While I think Ed has a point, I don't > see zones not cleaning up their default routes upon halting as a disaster. > We already have situations that default/static routes don't get cleaned up > after they are no longer needed. I agree with Ed that failing to remove the routes is sloppy. It means that a simple reconfiguration ("oh, the router is .2 here and not .1!") using the documented interfaces results in a broken system -- one with two routes, one good and one bad, leaving the administrator to clean up after it. That's not a great idea. It's possible that this would be acceptable for TX, where all zones behave as a collective and where administration practices might be more constrained, but I can't see it being that way otherwise. A somewhat simple solution could be implemented in zoneadmd. You'd have to keep track of the routes you add as you add them, and note the duplicates. Then, when the zones shut down, delete the route when the last zone using it goes away. That has the benefit of working for both TX and regular cases, of allowing the configuration that Ceri Davies wanted, and not leaving any mess around afterwards. > Your earlier suggestion about adding zoneid to route(1M) is interesting. > When ire_zoneid is set explicitly by "zoneid", it's clear who set the ire > entry, who can use the entry, and who should clean up the entry. It also > implies that the routing table could have many almost identical entries > that only differ in ire_zoneid. This may have performance implications > when finding a matching entry. Anyway, we haven't thought about > the design enough. That's possible, but seems like deeper and riskier surgery to me, particularly for something intended to be released as a patch. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From Glenn.Faden@sun.com Thu Jan 31 07:03:47 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02 [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0VF3kUl020626 for ; Thu, 31 Jan 2008 07:03:47 -0800 (PST) Received: from sca-es-mail-2.sun.com (sca-es-mail-2.Sun.COM [192.18.43.133]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0VF3kqO051966 for ; Thu, 31 Jan 2008 07:03:46 -0800 (PST) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m0VF3fpL022443 for ; Thu, 31 Jan 2008 07:03:41 -0800 (PST) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JVI00301KDJSR00@fe-sfbay-10.sun.com> (original mail from Glenn.Faden@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Thu, 31 Jan 2008 07:03:41 -0800 (PST) Received: from rampartgf.local ([129.150.18.193]) by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JVI0040DKI5N0A0@fe-sfbay-10.sun.com>; Thu, 31 Jan 2008 07:03:41 -0800 (PST) Date: Thu, 31 Jan 2008 07:03:47 -0800 From: Glenn Faden Subject: Re: 2008/057 Default Route For Zones In-reply-to: <18337.52811.612279.691598@gargle.gargle.HOWL> Sender: Glenn.Faden@sun.com To: James Carlson Cc: Jarrett Lu , Erik Nordmark , Edward Pilatowicz , psarc-ext@sac.sfbay.sun.com, Ken Powell Message-id: <47A1E353.3080103@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <47A0DFA0.7040908@sun.com> <20080130210421.GD732888@eng.sun.com> <47A0F26F.2010803@sun.com> <47A0F66B.6040300@sun.com> <47A0FB1D.6030708@sun.com> <47A12671.1030205@sun.com> <47A1476B.70201@sun.com> <18337.52811.612279.691598@gargle.gargle.HOWL> User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) Status: RO Content-Length: 2450 James Carlson wrote: > Jarrett Lu writes: > >> So far, I don't see a cliff yet. Ed's concern is mostly about not >> cleaning up >> default routes after zones are halted. While I think Ed has a point, I don't >> see zones not cleaning up their default routes upon halting as a disaster. >> We already have situations that default/static routes don't get cleaned up >> after they are no longer needed. >> > > I agree with Ed that failing to remove the routes is sloppy. It means > that a simple reconfiguration ("oh, the router is .2 here and not > .1!") using the documented interfaces results in a broken system -- > one with two routes, one good and one bad, leaving the administrator > to clean up after it. > > That's not a great idea. It's possible that this would be acceptable > for TX, where all zones behave as a collective and where > administration practices might be more constrained, but I can't see it > being that way otherwise. > > A somewhat simple solution could be implemented in zoneadmd. You'd > have to keep track of the routes you add as you add them, and note the > duplicates. Then, when the zones shut down, delete the route when the > last zone using it goes away. > How can you tell you are the last zone? It seems that some kind of shared reference counter would be required which is synchronized among all instances of zoneadmd, for each default route. Unless someone can explain how to do this better, I will amend my proposal as follows: 1. Each zoneadmd will keep track of routes that it was successful in establishing. It will remove such routes when it halts a zone. 2. If the attempt to create the route fails with EEXIST, a warning message is printed stating that the route is was previously established and may be removed by another zone.That means that duplicate (shared) routes are permitted, but discouraged, since they will fail if the first zone to set this route is halted. It should be noted that this information will be lost if zoneadmd dies and is restarted. That shouldn't be happening, but there is logic in zoneadm to restart a failed daemon. This approach has the opposite problem that it cleans up too aggressively instead of leaving obsolete routes. I still think it is an improvement over the current lack of support for defualt routes for zones. A follow on project could implement reference counters to determine when routes are no longer in use. --Glenn From ceri@submonkey.net Thu Jan 31 12:27:02 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02 [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0VKR25H002763 for ; Thu, 31 Jan 2008 12:27:02 -0800 (PST) Received: from brmea-mail-4.sun.com (brmea-mail-4.Sun.COM [192.18.98.36]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0VKR19N054135 for ; Thu, 31 Jan 2008 12:27:01 -0800 (PST) Received: from relay41i.sun.com ([192.5.209.70]) by brmea-mail-4.sun.com (8.13.6+Sun/8.12.9) with ESMTP id m0VKHROb013315 for ; Thu, 31 Jan 2008 20:27:01 GMT Received: from mms48es.sun.com ([160.41.221.231] [160.41.221.231]) by relay41i.sun.com with ESMTP id BT-MMP-1198879 for psarc-ext@sac.sfbay.sun.com; Thu, 31 Jan 2008 20:27:01 Z Received: from relay42i.sun.com (relay42i.sun.com [192.5.209.72]) by mms48es.sun.com with ESMTP id BT-MMP-1537202 for psarc-ext@sac.sfbay.sun.com; Thu, 31 Jan 2008 20:27:00 Z Received: from shrike.submonkey.net ([81.106.128.65] [81.106.128.65]) by relay4i.sun.com with ESMTP id BT-MMP-2731871 for psarc-ext@sac.sfbay.sun.com; Thu, 31 Jan 2008 20:27:00 Z Received: from ceri by shrike.submonkey.net with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1JKfzv-000IWm-5G; Thu, 31 Jan 2008 20:26:59 +0000 Date: Thu, 31 Jan 2008 20:26:59 +0000 From: Ceri Davies To: Glenn Faden Cc: James Carlson , Jarrett Lu , Erik Nordmark , psarc-ext@sac.sfbay.sun.com, Ken Powell , Edward Pilatowicz Subject: Re: 2008/057 Default Route For Zones Message-ID: <20080131202658.GD6064@submonkey.net> References: <47A01B3F.50103@sun.com> <47A0DFA0.7040908@sun.com> <20080130210421.GD732888@eng.sun.com> <47A0F26F.2010803@sun.com> <47A0F66B.6040300@sun.com> <47A0FB1D.6030708@sun.com> <47A12671.1030205@sun.com> <47A1476B.70201@sun.com> <18337.52811.612279.691598@gargle.gargle.HOWL> <47A1E353.3080103@sun.com> In-Reply-To: <47A1E353.3080103@sun.com> X-Brightmail-Tracker: AAAAAA== X-PGP: finger ceri@FreeBSD.org User-Agent: Mutt/1.5.17 (2007-11-01) Sender: Ceri Davies X-Antispam: No, score=-2.6/5.0, scanned in 0.071sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="bAwSoJxbKYwy34Oe" Content-Disposition: inline Status: RO Content-Length: 1834 --bAwSoJxbKYwy34Oe Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Thu, Jan 31, 2008 at 07:03:47AM -0800, Glenn Faden wrote: >=20 > Unless someone can explain how to do this better, I will amend my=20 > proposal as follows: >=20 >=20 > 1. Each zoneadmd will keep track of routes that it was successful in=20 > establishing. It will remove such routes when it halts a zone. >=20 > 2. If the attempt to create the route fails with EEXIST, a warning=20 > message is printed stating that the route is was previously established= =20 > and may be removed by another zone.That means that duplicate (shared)=20 > routes are permitted, but discouraged, since they will fail if the first= =20 > zone to set this route is halted. >=20 > It should be noted that this information will be lost if zoneadmd dies=20 > and is restarted. That shouldn't be happening, but there is logic in=20 > zoneadm to restart a failed daemon. >=20 > This approach has the opposite problem that it cleans up too=20 > aggressively instead of leaving obsolete routes. I'm a little concerned that this would cause problems for users who are treating zones as administrative containers and assume that rebooting zoneX will have no effect on zoneY. It sounds as if that would no longer be the case, and that worries me: more and more the more I think about it actually. Ceri --=20 That must be wonderful! I don't understand it at all. -- Moliere --bAwSoJxbKYwy34Oe Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFHoi8SocfcwTS3JF8RAvoGAJ0beKpKWM+HYldyWVBGvCZnzcz8vgCfWKkc 3xTad52zGzxjw8S07nXCPR8= =f/8B -----END PGP SIGNATURE----- --bAwSoJxbKYwy34Oe-- From Jarrett.Lu@sun.com Thu Jan 31 12:40:29 2008 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01 [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0VKeTQT002972 for ; Thu, 31 Jan 2008 12:40:29 -0800 (PST) Received: from sca-es-mail-2.sun.com (sca-es-mail-2.Sun.COM [192.18.43.133]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0VKeTgA062582 for ; Thu, 31 Jan 2008 12:40:29 -0800 (PST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m0VKeOXj000788 for ; Thu, 31 Jan 2008 12:40:24 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JVI00B01ZZYUU00@fe-sfbay-09.sun.com> (original mail from Jarrett.Lu@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Thu, 31 Jan 2008 12:40:24 -0800 (PST) Received: from [192.168.1.116] ([67.169.89.221]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JVJ00A7A039GD10@fe-sfbay-09.sun.com>; Thu, 31 Jan 2008 12:40:21 -0800 (PST) Date: Thu, 31 Jan 2008 12:40:09 -0800 From: Jarrett Lu Subject: Re: 2008/057 Default Route For Zones In-reply-to: <18337.52811.612279.691598@gargle.gargle.HOWL> Sender: Jarrett.Lu@sun.com To: James Carlson Cc: Erik Nordmark , Glenn Faden , Edward Pilatowicz , psarc-ext@sac.sfbay.sun.com, Ken Powell Message-id: <47A23229.1080309@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <47A0DFA0.7040908@sun.com> <20080130210421.GD732888@eng.sun.com> <47A0F26F.2010803@sun.com> <47A0F66B.6040300@sun.com> <47A0FB1D.6030708@sun.com> <47A12671.1030205@sun.com> <47A1476B.70201@sun.com> <18337.52811.612279.691598@gargle.gargle.HOWL> User-Agent: Thunderbird 2.0.0.6 (X11/20071203) Status: RO Content-Length: 2618 James Carlson wrote: > Jarrett Lu writes: > >> So far, I don't see a cliff yet. Ed's concern is mostly about not >> cleaning up >> default routes after zones are halted. While I think Ed has a point, I don't >> see zones not cleaning up their default routes upon halting as a disaster. >> We already have situations that default/static routes don't get cleaned up >> after they are no longer needed. >> > > I agree with Ed that failing to remove the routes is sloppy. It means > that a simple reconfiguration ("oh, the router is .2 here and not > .1!") using the documented interfaces results in a broken system -- > one with two routes, one good and one bad, leaving the administrator > to clean up after it. > > That's not a great idea. It's possible that this would be acceptable > for TX, where all zones behave as a collective and where > administration practices might be more constrained, but I can't see it > being that way otherwise. > > A somewhat simple solution could be implemented in zoneadmd. You'd > have to keep track of the routes you add as you add them, and note the > duplicates. Then, when the zones shut down, delete the route when the > last zone using it goes away. > This has a better chance of working if default routes can *only* be added by zonecfg/zoneadmd. If someone manually adds a default route, without knowledge of zoneadmd, we are back to a non working system again. To manage a shared resource correctly, we need to have a reference count. The best place to keep the count is in kernel, I believe. > That has the benefit of working for both TX and regular cases, of > allowing the configuration that Ceri Davies wanted, and not leaving > any mess around afterwards. > > >> Your earlier suggestion about adding zoneid to route(1M) is interesting. >> When ire_zoneid is set explicitly by "zoneid", it's clear who set the ire >> entry, who can use the entry, and who should clean up the entry. It also >> implies that the routing table could have many almost identical entries >> that only differ in ire_zoneid. This may have performance implications >> when finding a matching entry. Anyway, we haven't thought about >> the design enough. >> > > That's possible, but seems like deeper and riskier surgery to me, > particularly for something intended to be released as a patch. > > That's a good point. Many TX customers require a lengthy and expensive certification on the OS before they can deploy it. Sun certifies Update Releases, but probably no plan to certify onnv any time soon. We do need a solution that fits Update Release model. Jarrett From Jarrett.Lu@sun.com Thu Jan 31 12:43:57 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02 [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m0VKhvR7003028 for ; Thu, 31 Jan 2008 12:43:57 -0800 (PST) Received: from sca-es-mail-1.sun.com (sca-es-mail-1.Sun.COM [192.18.43.132]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m0VKhuhM063737 for ; Thu, 31 Jan 2008 12:43:56 -0800 (PST) Received: from fe-sfbay-10.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m0VKhp2X020112 for ; Thu, 31 Jan 2008 12:43:51 -0800 (PST) Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JVJ00B01041NK00@fe-sfbay-10.sun.com> (original mail from Jarrett.Lu@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Thu, 31 Jan 2008 12:43:51 -0800 (PST) Received: from [192.168.1.116] ([67.169.89.221]) by fe-sfbay-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JVJ00HYF092QBB0@fe-sfbay-10.sun.com>; Thu, 31 Jan 2008 12:43:50 -0800 (PST) Date: Thu, 31 Jan 2008 12:43:38 -0800 From: Jarrett Lu Subject: Re: 2008/057 Default Route For Zones In-reply-to: <47A1E353.3080103@sun.com> Sender: Jarrett.Lu@sun.com To: Glenn Faden Cc: James Carlson , Erik Nordmark , Edward Pilatowicz , psarc-ext@sac.sfbay.sun.com, Ken Powell Message-id: <47A232FA.40303@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <47A0DFA0.7040908@sun.com> <20080130210421.GD732888@eng.sun.com> <47A0F26F.2010803@sun.com> <47A0F66B.6040300@sun.com> <47A0FB1D.6030708@sun.com> <47A12671.1030205@sun.com> <47A1476B.70201@sun.com> <18337.52811.612279.691598@gargle.gargle.HOWL> <47A1E353.3080103@sun.com> User-Agent: Thunderbird 2.0.0.6 (X11/20071203) Status: RO Content-Length: 2764 Glenn Faden wrote: > James Carlson wrote: >> Jarrett Lu writes: >> >>> So far, I don't see a cliff yet. Ed's concern is mostly about not >>> cleaning up >>> default routes after zones are halted. While I think Ed has a point, >>> I don't >>> see zones not cleaning up their default routes upon halting as a >>> disaster. >>> We already have situations that default/static routes don't get >>> cleaned up >>> after they are no longer needed. >>> >> >> I agree with Ed that failing to remove the routes is sloppy. It means >> that a simple reconfiguration ("oh, the router is .2 here and not >> .1!") using the documented interfaces results in a broken system -- >> one with two routes, one good and one bad, leaving the administrator >> to clean up after it. >> >> That's not a great idea. It's possible that this would be acceptable >> for TX, where all zones behave as a collective and where >> administration practices might be more constrained, but I can't see it >> being that way otherwise. >> >> A somewhat simple solution could be implemented in zoneadmd. You'd >> have to keep track of the routes you add as you add them, and note the >> duplicates. Then, when the zones shut down, delete the route when the >> last zone using it goes away. >> > > How can you tell you are the last zone? It seems that some kind of > shared reference counter would be required which is synchronized among > all instances of zoneadmd, for each default route. > > Unless someone can explain how to do this better, I will amend my > proposal as follows: > > > 1. Each zoneadmd will keep track of routes that it was successful in > establishing. It will remove such routes when it halts a zone. > > 2. If the attempt to create the route fails with EEXIST, a warning > message is printed stating that the route is was previously > established and may be removed by another zone.That means that > duplicate (shared) routes are permitted, but discouraged, since they > will fail if the first zone to set this route is halted. > > It should be noted that this information will be lost if zoneadmd dies > and is restarted. That shouldn't be happening, but there is logic in > zoneadm to restart a failed daemon. > > This approach has the opposite problem that it cleans up too > aggressively instead of leaving obsolete routes. I still think it is > an improvement over the current lack of support for defualt routes for > zones. A follow on project could implement reference counters to > determine when routes are no longer in use. I agree with Ceri in that this approach can break other zones without any warning. Leaving obsolete around is a much lesser evil than deleting a route that other zones are currently using. Jarrett > > --Glenn From Glenn.Faden@sun.com Mon Feb 4 12:19:47 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02 [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m14KJlZ3022324 for ; Mon, 4 Feb 2008 12:19:47 -0800 (PST) Received: from sca-es-mail-2.sun.com (sca-es-mail-2.Sun.COM [192.18.43.133]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m14KJlf6034872 for ; Mon, 4 Feb 2008 12:19:47 -0800 (PST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id m14KJfbd022722 for ; Mon, 4 Feb 2008 12:19:41 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0JVQ00601DJEBR00@fe-sfbay-09.sun.com> (original mail from Glenn.Faden@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Mon, 04 Feb 2008 12:19:41 -0800 (PST) Received: from rampartgf.local ([129.150.18.183]) by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0JVQ0056GDSSZL30@fe-sfbay-09.sun.com>; Mon, 04 Feb 2008 12:19:41 -0800 (PST) Date: Mon, 04 Feb 2008 12:19:52 -0800 From: Glenn Faden Subject: Re: 2008/057 Default Route For Zones In-reply-to: <47A232FA.40303@sun.com> Sender: Glenn.Faden@sun.com To: Jarrett Lu Cc: James Carlson , Erik Nordmark , Edward Pilatowicz , psarc-ext@sac.sfbay.sun.com, Ken Powell Message-id: <47A77368.6040808@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <18335.27349.654381.581913@gargle.gargle.HOWL> <47A011B3.5060008@sun.com> <47A01B3F.50103@sun.com> <47A0DFA0.7040908@sun.com> <20080130210421.GD732888@eng.sun.com> <47A0F26F.2010803@sun.com> <47A0F66B.6040300@sun.com> <47A0FB1D.6030708@sun.com> <47A12671.1030205@sun.com> <47A1476B.70201@sun.com> <18337.52811.612279.691598@gargle.gargle.HOWL> <47A1E353.3080103@sun.com> <47A232FA.40303@sun.com> User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) Status: RO Content-Length: 3710 After much discussion we have reached consensus that the original proposal is satisfactory. We have received feedback on this alias, other OpenSolaris forums, and private correspondence encouraging us to provide this feature without additional restrictions. We will provide documentation in the appropriate administrative chapters about the appropriate uses and limitations associated with applying default zone routes. Although it is agreed that the proposal could be improved by keeping reference counts for which zones are using each route, such an enhancement might be quite complicated to develop and test and isn't required in order for customers to take advantage of the default routes. The project team has already initiated a project to provide the ability to remove per-zone default routes when the zone is halted. --Glenn Jarrett Lu wrote: > Glenn Faden wrote: >> James Carlson wrote: >>> Jarrett Lu writes: >>> >>>> So far, I don't see a cliff yet. Ed's concern is mostly about not >>>> cleaning up >>>> default routes after zones are halted. While I think Ed has a >>>> point, I don't >>>> see zones not cleaning up their default routes upon halting as a >>>> disaster. >>>> We already have situations that default/static routes don't get >>>> cleaned up >>>> after they are no longer needed. >>>> >>> >>> I agree with Ed that failing to remove the routes is sloppy. It means >>> that a simple reconfiguration ("oh, the router is .2 here and not >>> .1!") using the documented interfaces results in a broken system -- >>> one with two routes, one good and one bad, leaving the administrator >>> to clean up after it. >>> >>> That's not a great idea. It's possible that this would be acceptable >>> for TX, where all zones behave as a collective and where >>> administration practices might be more constrained, but I can't see it >>> being that way otherwise. >>> >>> A somewhat simple solution could be implemented in zoneadmd. You'd >>> have to keep track of the routes you add as you add them, and note the >>> duplicates. Then, when the zones shut down, delete the route when the >>> last zone using it goes away. >>> >> >> How can you tell you are the last zone? It seems that some kind of >> shared reference counter would be required which is synchronized >> among all instances of zoneadmd, for each default route. >> >> Unless someone can explain how to do this better, I will amend my >> proposal as follows: >> >> >> 1. Each zoneadmd will keep track of routes that it was successful in >> establishing. It will remove such routes when it halts a zone. >> >> 2. If the attempt to create the route fails with EEXIST, a warning >> message is printed stating that the route is was previously >> established and may be removed by another zone.That means that >> duplicate (shared) routes are permitted, but discouraged, since they >> will fail if the first zone to set this route is halted. >> >> It should be noted that this information will be lost if zoneadmd >> dies and is restarted. That shouldn't be happening, but there is >> logic in zoneadm to restart a failed daemon. >> >> This approach has the opposite problem that it cleans up too >> aggressively instead of leaving obsolete routes. I still think it is >> an improvement over the current lack of support for defualt routes >> for zones. A follow on project could implement reference counters to >> determine when routes are no longer in use. > > I agree with Ceri in that this approach can break other zones without > any warning. > Leaving obsolete around is a much lesser evil than deleting a route > that other zones > are currently using. > > Jarrett > >> >> --Glenn > From edward.pilatowicz@sun.com Tue Feb 5 03:41:34 2008 Received: from dm-eng-01.sfbay.sun.com (dm-eng-01 [129.145.155.198]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m15BfYcm020707 for ; Tue, 5 Feb 2008 03:41:34 -0800 (PST) Received: from mcescher.eng.sun.com (mcescher.SFBay.Sun.COM [129.146.224.55]) by dm-eng-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m15BfYja016911; Tue, 5 Feb 2008 03:41:34 -0800 (PST) Received: from mcescher.eng.sun.com (localhost [127.0.0.1]) by mcescher.eng.sun.com (8.14.2+Sun/8.14.2) with ESMTP id m15BfYP8975264; Tue, 5 Feb 2008 03:41:34 -0800 (PST) Received: (from edp@localhost) by mcescher.eng.sun.com (8.14.2+Sun/8.14.2/Submit) id m15BfYjk975263; Tue, 5 Feb 2008 03:41:34 -0800 (PST) X-Authentication-Warning: mcescher.eng.sun.com: edp set sender to edward.pilatowicz@sun.com using -f Date: Tue, 5 Feb 2008 03:41:34 -0800 From: Edward Pilatowicz To: Glenn Faden Cc: James Carlson , Jarrett Lu , Erik Nordmark , psarc-ext@sac.sfbay.sun.com, Ken Powell Subject: Re: 2008/057 Default Route For Zones Message-ID: <20080205114134.GE975231@eng.sun.com> References: <47A01B3F.50103@sun.com> <47A0DFA0.7040908@sun.com> <20080130210421.GD732888@eng.sun.com> <47A0F26F.2010803@sun.com> <47A0F66B.6040300@sun.com> <47A0FB1D.6030708@sun.com> <47A12671.1030205@sun.com> <47A1476B.70201@sun.com> <18337.52811.612279.691598@gargle.gargle.HOWL> <47A1E353.3080103@sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <47A1E353.3080103@sun.com> User-Agent: Mutt/1.4.2.1i Status: RO Content-Length: 3279 On Thu, Jan 31, 2008 at 07:03:47AM -0800, Glenn Faden wrote: > James Carlson wrote: > >Jarrett Lu writes: > > > >>So far, I don't see a cliff yet. Ed's concern is mostly about not > >>cleaning up > >>default routes after zones are halted. While I think Ed has a point, I > >>don't > >>see zones not cleaning up their default routes upon halting as a disaster. > >>We already have situations that default/static routes don't get cleaned up > >>after they are no longer needed. > >> > > > >I agree with Ed that failing to remove the routes is sloppy. It means > >that a simple reconfiguration ("oh, the router is .2 here and not > >.1!") using the documented interfaces results in a broken system -- > >one with two routes, one good and one bad, leaving the administrator > >to clean up after it. > > > >That's not a great idea. It's possible that this would be acceptable > >for TX, where all zones behave as a collective and where > >administration practices might be more constrained, but I can't see it > >being that way otherwise. > > > >A somewhat simple solution could be implemented in zoneadmd. You'd > >have to keep track of the routes you add as you add them, and note the > >duplicates. Then, when the zones shut down, delete the route when the > >last zone using it goes away. > > > > How can you tell you are the last zone? It seems that some kind of > shared reference counter would be required which is synchronized among > all instances of zoneadmd, for each default route. > > Unless someone can explain how to do this better, I will amend my > proposal as follows: > > 1. Each zoneadmd will keep track of routes that it was successful in > establishing. It will remove such routes when it halts a zone. > > 2. If the attempt to create the route fails with EEXIST, a warning > message is printed stating that the route is was previously established > and may be removed by another zone.That means that duplicate (shared) > routes are permitted, but discouraged, since they will fail if the first > zone to set this route is halted. > > It should be noted that this information will be lost if zoneadmd dies > and is restarted. That shouldn't be happening, but there is logic in > zoneadm to restart a failed daemon. > > This approach has the opposite problem that it cleans up too > aggressively instead of leaving obsolete routes. I still think it is an > improvement over the current lack of support for defualt routes for > zones. A follow on project could implement reference counters to > determine when routes are no longer in use. > the state wrt how routing was configured should be saved as a zone property. that way if zoneadmd dies and is restarted it can check this information. i would also be possible for zoneadmd to query this attribute for all zones on the system, and using ipc, to have all the zoneamd copies coordinate to essentially implement reference counting which zones are using which routes. this also doesn't address the other issue i raised. two zones on the same subnet with different default routes should be a configuration error, since there are no guarantess that once both routes are added to the system the zone will actually use the default route that it was configured. each zone could use both default routes. ed From gww@eng.sun.com Wed Feb 20 16:52:16 2008 Received: from dm-eng-02.sfbay.sun.com (dm-eng-02 [129.146.11.32]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id m1L0qGJv019052 for ; Wed, 20 Feb 2008 16:52:16 -0800 (PST) Received: from marduk.eng.sun.com (marduk.SFBay.Sun.COM [129.146.108.224]) by dm-eng-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id m1L0qGn0029355; Wed, 20 Feb 2008 16:52:16 -0800 (PST) Received: from marduk.eng.sun.com (localhost [127.0.0.1]) by marduk.eng.sun.com (8.13.6+Sun/8.12.11) with ESMTP id m1L0qc0w020637; Wed, 20 Feb 2008 16:52:38 -0800 (PST) Received: (from gww@localhost) by marduk.eng.sun.com (8.13.6+Sun/8.12.11/Submit) id m1L0qcK7020636; Wed, 20 Feb 2008 16:52:38 -0800 (PST) Date: Wed, 20 Feb 2008 16:52:38 -0800 (PST) From: Gary Winiger Message-Id: <200802210052.m1L0qcK7020636@marduk.eng.sun.com> To: james.d.carlson@sun.com, psarc-ext@sac.sfbay.sun.com Cc: gfaden@sun.com Subject: Re: 2008/057 Default Route For Zones Status: RO Content-Length: 447 > I'm sponsoring this fast-track request for Glenn Faden. The timer is > set to 02/05/2008. The requested release binding is "patch/micro" (as > it will be needed for an S10 Update), and the stability level for the > new zonecfg keyword is "Committed." Since Jim is on sabbatical, I thought I'd act as proxy. This case was approved as submitted at today's PSARC meeting. Issues of removing routes will be dealt with in other cases. Gary..