zonecfg(1m) enhancements for exclusive-IP zones Requested release binding: Minor Summary: ------- This case proposes a solution for 6944327 need to support address and defrouter resources for exclusive-IP zones Problem Description -------------------- Typical zone deployments that exist today use shared-IP zones to run applications and services like Apache or Weblogic in the contained environment provided by the shared-IP zone. In these use-cases, the Administrator in the global zone has full control over the networking resources used by the non-global zone. In the common case, networking is simply configured by specifying the IP interface, IP addresses and, optionally, the default routers from zonecfg(1m). The configuration resources thus supplied are then applied for the non-global zone when it is booted, and the non-global zone itself may not modify any of these configuration parameters. However, there is no such simple configuration mechanism for the simple networking use-case in place for exclusive IP zones, which must be configured through sysidcfg, ifconfig, and an assortment of other methods, all of which are not controllable from the global zone, and may not be managed through zonecfg(1m). The addition of many new virtualization and resource management features to Solaris (PSARC 2006/357, PSARC 2005/132 and associated cases) makes Exclusive-IP zones a cleaner and more powerful Zone model than shared-IP zones. Thus, bridging any gaps in the configuration methods for the simple use cases between shared- and exlusive-IP zones is important to ease the transition of shared-IP customer configurations to exlusive-IP. In addition, there are many ongoing projects in the "Zones Networking" effort [ZONES-NET] to facilitate the consolidation of existing Solaris 10 host installations as Solaris 10 Containers. These efforts would leverage from the ability to specify networking resource values for address and default router uniformly for zones using zonecfg(1m). This case proposes support in zonecfg(1m) for IP configuration through the support of 'allowed-address' and 'defrouter' properties in the 'net' resource for exclusive-IP zones. When the global-zone specifies the 'allowed-address' for an interface via zonecfg(1m), the non-global zone may not use any other IP addresses for that interface. The address information provided via zonecfg(1m) will be used to set up Layer-3 protection [PSARC/2009/436] for the non-global zone during zone-boot to filter out all other addresses for the selected interface. For instance, when zonecfg(1m) has been used in the global-zone to set one or more IPv4 addresses on an interface, an attempt to set an IPv4 or IPv6 address on the interface that is outside the globally defined set will encounter the EPERM failure. Thus ifconfig(1m), ipadm(1m), and associated ioctls will receive this error if they are used within the non-global zone to set addresses that are not in the set that is permitted from the global-zone, and attempts by the non-global zone to turn on forwarding on the interface will also encounter EPERM. Attempts to boot a zone that has already been configured for IP, or has previously customized values for Layer-3 protection [PSARC/2009/436] will fail. Implementation Overview: ------------------------ A brief overview of the implementation is provided here. Note that all interfaces between zoneadmd(1m) and the kernel/zonecfg(1m), are Private interfaces, subject to change in the future. When the non-global zone is booted, zoneadmd(1m) will store the information specified by the 'allowed-address' and 'defrouter' properties as nvlists in the kernel following a mechanism similar to that in use today for other zonecfg(1m) resources such as the "physical" datalink. At the same time, the zone boot process will also ensure 'ip-nospoof' protection for the datalink with the specified addresses used as input to the 'allowed-ips' property. On the first boot of an installed non-global native (ipkg) zone, and on every boot of a solaris10 branded zone [PSARC 2010/111], the stored nvlist information for 'allowed-address' will be retrieved by the ipmgmtd daemon which will create the interface (persistently, for native ipkg zones), and apply the IP addresses non-persistently before any other IP configuration is applied. On subsequent boots of native (ipkg) zones, any IP interface with persistent configuration in the ipadm data-store will be recreated using IP address information from the kernel's nvlists set up by zoneadmd. The 'defrouter' information set up by the global zone will be applied after the 'allowed-address' information has been configured, so that the behavior of the IP configuration scheme matches that of other address acquisition protocols like DHCP. Routes added by ipmgmtd will be identifiable in the 'netstat -r' output by the 'Z' flag which will indicate that the associated route was based on information configured in the global zone. The GLDv3 layer communicates the set of "allowed-ips" to the IP layer through DLPI notifications. The notification sent is a DL_NOTIFY_IND message of (Project Private) type DL_NOTE_ALLOWED_IPS, sent to the IP clients that have registered for this notification. The payload of the DL_NOTE_ALLOWED_IPS message is the mac_protect_t data-structure introduced by PSARC 2009/436. Receipt of this notification enables the IP layer to track the current set of permitted IP addresses per interface, so that the IP layer can return informative error diagnostics if an attempt is made to enable forwarding or set addresses on the interface that are outside the permitted set. IP interfaces with 'allowed-ips' constraints will be identifiable by the "L3PROTECT" flag in the output of ifconfig, and by a 'Z' flag in the output of 'ipadm show-if'. All configuration state, including datalink properties like protection, allowed-ips and zone, and associated nvlist information, that is created during zone boot will be reset to the default values when the zone is halted. Interfaces added by this case: ------------------------------ This case adds the following: - a new RTF_ZONE flag which will be passed down in the routing socket message sent by the ipmgmtd daemon in the non-global zone to add default route information set up via zonecfg(1m). The output of 'netstat -r' will display 'Z' when the route was added by ipmgmtd in this manner. The output of the 'route get' command will also display the "ZONE" string as part of the 'flags' field for the route. - a read-only interface flag IFF_L3PROTECT which will be set by the kernel on the IP interface when the 'allowed-ips' link property has been set for the interface. The IFF_L3PROTECT flag will be indicated by a 'Z' in the CURRENT column of the output from 'ipadm show-if' Interface Classification Comments _______________________________________________________________________ netstat(1m) -r output Uncommitted (unchanged) RTF_ZONE Committed route(1m) Uncommitted (unchanged) IFF_L3PROTECT Committed , read-only The output of 'ipadm show-addr' will be modified to display a TYPE of 'from-gz' when the address has been configured based on IP address information allocated to the non-global zone from the global zone. Relationship to other configuration mechanisms ---------------------------------------------- sysidcfg: The Solaris Install team is currently working in conjunction with the Network Configuration team [INSTALL] on the implementation of a framework to replace sysidcfg(4) that will provide configuration profiles to be processed by startup scripts when a system boots. However this configuration information is only applied on the first boot of the zone, and unlike zonecfg, may not be easily modified in the global zone across reboot. Moreover, there is no mechanism in place to prevent the non-global zone from modifying the configuration provided via profiles. As mentioned earlier, the global-zone is the authoritative source of IP configuration information for the "secure container" use-case. This implies that attempts to use boot profiles in the non-global zone to add addresses other than those mandated in the global-zone will fail with EINVAL erors. Note that the profiles defined in [INSTALL] may still be applied to interfaces for which 'net' resource properties have not been defined in the global-zone, i.e., this proposal is fully backward-compatible. svc:/network/physical:{default, nwam} Configuration specified via zonecfg(1m) in the global-zone overrides any information specified via /etc/hostname.intf, ipadm(1m) or nwam. Thus if the IP address information specified by these other methods is not a subset of the globally managed information, attempts to apply it within the non-global zone will encounter an error. DHCP: It is assumed that when IP resources for an IP interface have been set up in the global-zone using zonecfg(1m), then that interface has been marked for static IP configuration only, and attempts to send out outgoing DHCP DISCOVER packets will be dropped in GLDv3. Thus attempts to start DHCP on these interfaces in the non-global zone will time out. Examples -------- root@gz# zonecfg -z tz1 zonecfg:tz1> info zonename: tz1 zonepath: /rpool/zones/tz1 brand: ipkg autoboot: false bootargs: pool: limitpriv: scheduling-class: ip-type: exclusive hostid: net: allowed-address: 11.1.1.1/24 physical: vnic0 defrouter: 11.1.1.2 net: allowed-address: taddr2 physical: vnic0 defrouter not specified net: address not specified physical: vnic1 defrouter not specified In this example, the zone 'tz1' has been assigned 2 interfaces: vnic0 and vnic1. The vnic1 interface does not have any IP address or default router assigned to it, so that it may be configured in any desired way from within the zone using ifconfig(1m), ipadm(1m), NWAM etc. The vnic0 interface has 2 addresses assigned: 11.1.1.1/24 (with default router 11.1.1.2 on the 11.1.1.0/24 subnet) and the address 'taddr2' whose numeric value and netmask will be resolved when the zone boots. Thus, if, at the time of zone boot, the global-zone is also configured as: root@gz# egrep 'hosts|netmask' /etc/nsswitch.conf hosts: files nis dns netmasks: files nis root@gz# grep taddr2 /etc/hosts 12.2.3.4 taddr2 root@gz# grep 12 /etc/netmasks 12.2.0.0 255.255.0.0 tz1 will be booted as root@gz# zoneadm -z tz1 boot root@gz# zlogin tz1 ipadm show-addr ADDROBJ TYPE STATE ADDR vnic0/_a from-gz ok 11.1.1.1/24 vnic0/_b from-gz ok 12.2.3.4/16 lo0/v4 static ok 127.0.0.1/8 lo0/v6 static ok ::1/128 root@gz# zlogin tz1 ipadm show-if IFNAME STATE CURRENT PERSISTENT vnic0 ok bm-------Z4- -4- lo0 ok -m-v------46 --- root@gz# zlogin tz1 netstat -rn Routing Table: IPv4 Destination Gateway Flags Ref Use Interface -------------------- -------------------- ----- ----- ---------- --------- default 11.1.1.2 UGZ 2 2 vnic0 11.1.1.0 11.1.1.1 U 3 2 vnic0 12.2.0.0 12.2.3.4 U 2 0 vnic0 127.0.0.1 127.0.0.1 UH 2 0 lo0 Routing Table: IPv6 Destination/Mask Gateway Flags Ref Use If --------------------------- ---------------------- ----- --- ------- ----- ::1 ::1 UH 2 0 lo0 roott@gz# zlogin tz1 route get default route to: default destination: default mask: default gateway: 11.1.1.2 interface: vnic0 flags: recvpipe sendpipe ssthresh rtt,ms rttvar,ms hopcount mtu expire Proposed man page changes: -------------------------- --- zonecfg.1m.orig Thu Apr 22 14:41:33 2010 +++ zonecfg.1m.new Fri Jun 11 18:54:58 2010 @@ -625,7 +625,7 @@ The directory path. - net: address, physical, defrouter + net: address, allowed-address, physical, defrouter The network address and physical interface name of the network interface. The network address is one of: @@ -673,13 +673,33 @@ to booting the non-global zone. However, if the inter- face is not used by the global zone, it should be con- figured down in the global zone, and the default router - for the interface should be specified here. + for the interface should be specified here. The + 'allowed-address' property cannot be set for a shared-IP + zone For an exclusive-IP zone, the physical property must be - set and the address and default router properties cannot - be set. + set and the 'address' property may not be set. Optionally, + the set of IP addresses that the exclusive-IP zone can use + may be constrained by specifying the 'allowed-address' + property. If 'allowed-address' has not been specified, then the + exclusive-IP zone may use any IP address on the associated 'physical' + interface for the net resource. Otherwise, when 'allowed-address' + is specified, the exclusive-IP zone cannot use IP addresses that + are not in any of the 'allowed-address' list for the 'physical'. + In addition, when the 'allowed-address' has been specified, + the 'defrouter' property may also be optionally specified. The + interface specifed for the physical property must not be in + use in the global zone. If an allowed-address and default + router are specified via zonecfg(1m), these will be applied to + the interface when it is enabled via ipadm(1m) in the + non-global exclusive-IP zone, typically during zone boot. + The non-global exclusive-IP zone will not be able to apply any other + addresses to that interface nor transmit packets with a different + source address for the specified IP Version. A default router + set up via zonecfg(1m) may not be persistently deleted from + within the non-global exclusive-IP zone using the -p flag with + route(1m). - device: match Device name to match. @@ -1817,7 +1837,7 @@ svcadm(1M), sysidtool(1M), zfs(1M), zoneadm(1M), priv_str_to_set(3C), kstat(3KSTAT), vfstab(4), attri- butes(5), brands(5), fnmatch(5), lx(5), privileges(5), - resource_controls(5), zones(5) + resource_controls(5), zones(5), route(1m) System Administration Guide: Solaris Containers-Resource --- dladm.1m.orig Fri May 7 10:10:28 2010 +++ dladm.1m.new Fri Jun 11 08:20:35 2010 @@ -3746,17 +3746,25 @@ IP address anti-spoof. This protection type works in conjunction with the link property allowed-ips. - allowed-ips is a list containing IPv4 addresses. + allowed-ips is a list containing IP (IPv4 or IPv6) addresses. This list is empty by default. The unspecified address (all-zeros) is implicitly in this list. - An outbound IPv4 packet may pass if its source - address is in allowed-ips. + An outbound IP packet may pass if its source + address is in allowed-ips An outbound ARP packet may pass if its sender proto- col address is in allowed-ips. + When a datalink has been protected by setting allowed-ips + to a set of one or more IP addresses, any attempts to + configure IP addresses that are not in this set will + fail with an EPERM error being returned to the user. + Moreover, the interface may not be used for forwarding + IP packets, and attempts to set the ipadm(1m) forwarding + property on the interface will encounter EPERM error. + restricted This protection restricts outgoing packet types to @@ -4572,7 +4580,7 @@ SEE ALSO acctadm(1M), autopush(1M), ifconfig(1M), ipsecconf(1M), ndd(1M), vrrpadm(1M), psrset(1M), wpad(1M), zonecfg(1M), - attributes(5), ieee802.3(5), dlpi(7P) + attributes(5), ieee802.3(5), dlpi(7P), ipadm(1m) --- netstat.orig Wed May 19 17:02:56 2010 +++ netstat.new Fri Jun 11 06:21:08 2010 @@ -740,6 +740,9 @@ I Indirect routes (gateway not directly reachable) esta- blished with the -indirect option. + Z (non-global exclusive-IP zone only) The route was statically added + on boot based on routing information configured using + zonecfg(1m) in the global zone. If the -a option is specified, there will be routing entries @@ -881,7 +884,7 @@ iostat(1M), kstat(1M), mibiisa(1M), savecore(1M), vmstat(1M), hosts(4), inet_type(4), networks(4), proto- cols(4), services(4), attributes(5), dhcp(5), kstat(7D), - inet(7P), inet6(7P) + inet(7P), inet6(7P), zonecfg(1m) Droms, R., RFC 2131, Dynamic Host Configuration Protocol, --- ipadm.1m.orig Tue May 25 09:12:35 2010 +++ ipadm.1m.new Fri Jun 11 08:30:45 2010 @@ -321,10 +321,11 @@ a VRRP interface is in accept mode (~IFF_NOACCEPT) + Z Layer-3 protection of IP addresses for the + interface has been administratively enforced. - SunOS 5.11 Last change: 29 Mar 2010 5 @@ -891,9 +892,14 @@ TYPE Type of the address object. It will be one of: - static, dhcp, or addrconf. It corresponds to the - type of the address object specified by the -T - option of create-addr. + from-gz, static, dhcp, or addrconf. The static, + dhcp and addrconf types correspond to the type of the + address object specified by the -T option of + create-addr. The "from-gz" type will only be displayed + in non-global zones, and indicates that the address + was configured based on the 'allowed-address' property + configured for the non-global exclusive-IP zone from + the global zone. STATE @@ -2315,7 +2321,21 @@ address objects configured on that interface are enabled also. + The following command creates persistent configuration for the + net0 interface in a non-global exclusive-IP zone + so that the net0 interface will be configured with the set of + addresses made available through the 'allowed-address' resource + from the global zone on the next reboot. + # ipadm create-if net0 + + The net0 interface may also be configured with the available set + of 'allowed-address' values in the non-global exclusive-IP zone + without a reboot by executing the following commands: + + # ipadm disable-if -t net0 + # ipadm enable-if -t net0 + ATTRIBUTES See attributes(5) for descriptions of the following attri- butes: --- ifconfig.1m.orig Tue May 25 11:05:48 2010 +++ ifconfig.1m.new Tue May 25 11:07:31 2010 @@ -1599,7 +1599,13 @@ used on multiple interfaces. (See also the usesrc option.) + L3PROTECT + Indicates that Layer-3 protection has been enforced on + the physical interface using the 'allowed-ips' link + property in dladm(1m) + + LOGICAL INTERFACES Solaris TCP/IP allows multiple logical interfaces to be associated with a physical network interface. This allows a References: ---------- [PSARC/2002/174] Virtualization and Namespace Isolation in Solaris, aka "Zones" [PSARC/2003/621] Zone Administration Updates [PSARC/2008/057] Default Route For Zones [PSARC/2009/436] Anti-spoofing Link Protection [PSARC/2010/111] exclusive IP for s10c [INSTALL] http://solaris-networking.sfbay/index.php/Network_Configuration:install [ZONES-NET] http://solaris-networking.sfbay/index.php/Zones_Networking