From sacadmin Wed Nov 19 19:15:33 2008 Received: from tethys.sfbay.sun.com (tethys.SFBay.Sun.COM [129.146.226.92]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAK3FXIe000772; Wed, 19 Nov 2008 19:15:33 -0800 (PST) Received: from tethys.sfbay.sun.com (localhost [127.0.0.1]) by tethys.sfbay.sun.com (8.14.3+Sun/8.14.3) with ESMTP id mAK3AjFD001724; Wed, 19 Nov 2008 19:10:45 -0800 (PST) Received: (from jg@localhost) by tethys.sfbay.sun.com (8.14.3+Sun/8.14.3/Submit) id mAK3Aj9x001720; Wed, 19 Nov 2008 19:10:45 -0800 (PST) Date: Wed, 19 Nov 2008 19:10:45 -0800 (PST) From: Jerry Gilliam Message-Id: <200811200310.mAK3Aj9x001720@tethys.sfbay.sun.com> To: PSARC-record@sac.sfbay.sun.com Subject: Device Remap [PSARC/2008/723 FastTrack timeout 11/26/2008] Status: RO Content-Length: 542 Template Version: @(#)sac_nextcase %I% %G% SMI This information is Copyright 2008 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: Device Remap 1.2. Name of Document Author/Supplier: Author: Sree Vemuri 1.3 Date of This Document: 19 November, 2008 4. Technical Description See the case directory for more detail 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack 6.6. ARC Exposure: open From jg@jurassic.sfbay.Sun.COM Wed Nov 19 19:19:34 2008 Received: from jurassic.eng.sun.com (jurassic-226-a.SFBay.Sun.COM [129.146.226.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAK3JY6q000925 for ; Wed, 19 Nov 2008 19:19:34 -0800 (PST) Received: from tethys (tethys.SFBay.Sun.COM [129.146.226.92]) by jurassic.eng.sun.com (8.13.8+Sun/8.13.8) with SMTP id mAK3JXXx178036; Wed, 19 Nov 2008 19:19:34 -0800 (PST) Message-Id: <200811200319.mAK3JXXx178036@jurassic.eng.sun.com> Date: Wed, 19 Nov 2008 19:14:46 -0800 (PST) From: Jerry Gilliam Reply-To: Jerry Gilliam Subject: Device Remap [PSARC/2008/723 FastTrack timeout 11/26/2008] To: psarc-ext@sac.sfbay.sun.com, sree.vemuri@sun.com Cc: batoka-debug@sun.com, John.Johnson@sun.com, Fadi.Salem@sun.com, Ken.Brucker@sun.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: fzt+VZxyo6iscmc/iwg1pA== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_94 SunOS 5.11 sun4u sparc Status: RO Content-Length: 6058 I'm sponsoring the following fast-track for Sree Vemuri, with timeout of one week, 11/26/2008. The project requests patch/minor binding, with the path name of the command committed and the output of the command not an interface. ------- 1. Introduction 1.1. Project/Component Working Name: Device Remap Script for SPARC Enterprise T5440 Servers 1.2. Name of Document Author/Supplier: Sree Vemuri 1.3. Date of This Document: 10/09/2008 1.4. Name of Major Document Customer(s)/Consumer(s): 1.4.1. The PAC or CPT you expect to review your project: Solaris PAC 1.4.2. The ARC(s) you expect to review your project: PSARC 1.4.3. The Director/VP who is "Sponsoring" this project: quresh.dhoon@sun.com 1.4.4. The name of your business unit: Niagara Rock SW Engineering 1.5. Email Aliases: 1.5.1. Responsible Manager: Fadi Salem 1.5.2. Responsible Engineer: Sree Vemuri 1.5.3. Interest List: John Johnson 2. Project Summary 2.1. Project Description: Certain multi-node sun4v platforms (Batoka or Maramba) have an integrated PCI topology that causes the IO device paths to change in a CMP node failover condition. In the event a CMP node fails, the system can be booted to OBP with the remaining CMP nodes. But, with the current architecture Solaris fails to boot as the PCIE device paths have changed. The device paths are hard coded in the /etc/path_to_inst file and the symlinks under /dev. When the same device is found at a different I/O path after power-cycle it is treated as a new device. Customer can not easily recover from the loss of I/O. A change to the boot path requires Solaris re-install. This project creates a process that allows the system administrator to modify the Solaris idea of the device paths to match that of the HW. 2.2. Risks and Assumptions: The solaris script should pose no risk. The script assumes /etc/path_to_inst doesn't change format. 3. Business Summary 3.1. Problem Area: Current architecture prevents operating system boot if the PCIe device paths have changed due to a node failover condition. This case is attempting to solve a long standing problem where I/O topology changes necessitate an operating system reinstall. 4. Technical Description: 4.1. Details: The T5440 FW describes the possible device path aliases and current configuration in the MD (see FWARC 2008/349). For example: When cpu0 exists, we use its path (/pci@400) to access IO; the other paths via other CPUs are listed as aliases: node ioalias pcie0 { current = "/pci@400"; aliases = "/pci@500/pci@0/pci@1/pci@0/pci@1" + "/pci@600/pci@0/pci@1" + "/pci@700/pci@0/pci@1/pci@0/pci@1/pci@0/pci@1"; back -> ioaliases; }; This project involves a perl script that combines the current Solaris device paths and the ioalias information from the FW to change /etc/path_to_inst and the /dev symlinks to reflect the new HW configuration. Note that since the device path of the root file system may be affected, the suggested procedure will be to boot the failsafe archive (which is on a ramdisk that is immune to device path changes) and run the script from there. 5. Reference Documents: [1] FWARC 2008/349 New MD Nodes to Support I/O Reconfiguration for Failover Conditions http://sac.sfbay.sun.com/Archives/CaseLog/arc/FWARC/2008/349/onepager.txt System Administration Commands device_remap(1M) NAME device_remap - administration program for the Solaris I/O remapping feature SYNOPSIS /usr/platform/sun4v/sbin/device_remap [-v | -d dir] DESCRIPTION Certain multi-node sun4v platforms, for example T5440 and T5240 servers have an integrated PCI topology that cause the IO device paths to change in a CPU node failover condition. The Device Remapping Script for SPARC Enterprise T5440 servers remaps the device paths in /etc/path_to_inst file and the symlinks under /dev. OPTIONS The following options are supported: -v Prints the path_to_inst and /dev/symlink changes -d dir Operate on the /etc/path_to_inst and /dev on the root image at USAGE The primary function of device_remap is to remap the device paths in /etc/path_to_inst file and the symlinks under /dev in a CPU node failover condition. After adding CPU node/s or removing CPU node/s, boot the system to OBP prompt and use the following procedure: 1. Boot an install miniroot, either with "boot net -s" or "boot -F failsafe". 2. Mount the root disk as /mnt. 3. Change directory to the mounted root disk. cd /mnt 4. Run device_remap script. /mnt/usr/platform/sun4v/sbin/device_remap 5. Reboot the system. All the error messages are self-explanatory, except for the error message "missing ioaliases node" which means the FW on the system doesn't support device remapping. EXAMPLES Example 1 Prints the path_to_inst and /dev/symlink changes # device_remap -v Example 2 Changes directory to dir before doing any changes # device_remap -d dir ATTRIBUTES See attributes(5) for descriptions of the following attri- butes: ____________________________________________________________ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | |_____________________________|_____________________________| | Availability | SUNWkvm | |_____________________________|_____________________________| | Interface Stability | Committed | |_____________________________|_____________________________| SEE ALSO boot(1M) NOTES From Peter.Dennis@Sun.COM Thu Nov 20 02:53:07 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAKAr7p1014527 for ; Thu, 20 Nov 2008 02:53:07 -0800 (PST) Received: from gmp-eb-inf-2.sun.com (gmp-eb-inf-2.EU.Sun.COM [192.18.6.24]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAKAr6qg020430 for ; Thu, 20 Nov 2008 02:53:06 -0800 (PST) Received: from fe-emea-10.sun.com (gmp-eb-lb-2-fe2.eu.sun.com [192.18.6.11]) by gmp-eb-inf-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAKAr0K6019162 for ; Thu, 20 Nov 2008 10:53:00 GMT Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAM00101NKH6C00@fe-emea-10.sun.com> (original mail from Peter.Dennis@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Thu, 20 Nov 2008 10:53:00 +0000 (GMT) Received: from [129.156.173.66] by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAM00D1LOVQXQB0@fe-emea-10.sun.com>; Thu, 20 Nov 2008 10:52:39 +0000 (GMT) Date: Thu, 20 Nov 2008 10:52:38 +0000 From: Peter Dennis - Sustaining Engineer Subject: Re: Device Remap [PSARC/2008/723 FastTrack timeout 11/26/2008] In-reply-to: <200811200319.mAK3JXXx178036@jurassic.eng.sun.com> Sender: Peter.Dennis@Sun.COM To: Jerry Gilliam Cc: psarc-ext@sac.sfbay.sun.com, Sree.Vemuri@Sun.COM, Ken.Brucker@Sun.COM, Fadi.Salem@Sun.COM, batoka-debug@Sun.COM, John.Johnson@Sun.COM Message-id: <49254176.4080406@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <200811200319.mAK3JXXx178036@jurassic.eng.sun.com> User-Agent: Thunderbird 2.0.0.16 (X11/20080908) Status: RO Content-Length: 7276 Just some nits: -d dir - should this be -R instead to be consistent with other commands that change files in a mounted root disk (ie update_grub -R ) -v - does this mean show what would happen but do nothing or actually be verbose about what changes are being applied ? Is the example in the man page correct: > 3. Change directory to the mounted root disk. > cd /mnt > > 4. Run device_remap script. > /mnt/usr/platform/sun4v/sbin/device_remap Should it be /mnt/usr/platform/sun4v/sbin/device_remap -d /mnt ? Will the device_remap be included in the failsafe image, if so then the command becomes: /usr/platform/sun4v/sbin/device_remap -d /mnt Pete Jerry Gilliam wrote: > > I'm sponsoring the following fast-track for Sree Vemuri, with > timeout of one week, 11/26/2008. > > The project requests patch/minor binding, with the path name > of the command committed and the output of the command > not an interface. > > ------- > > > 1. Introduction > 1.1. Project/Component Working Name: > Device Remap Script for SPARC Enterprise T5440 Servers > > 1.2. Name of Document Author/Supplier: > Sree Vemuri > > 1.3. Date of This Document: > 10/09/2008 > > 1.4. Name of Major Document Customer(s)/Consumer(s): > 1.4.1. The PAC or CPT you expect to review your project: > Solaris PAC > 1.4.2. The ARC(s) you expect to review your project: > PSARC > 1.4.3. The Director/VP who is "Sponsoring" this project: > quresh.dhoon@sun.com > 1.4.4. The name of your business unit: > Niagara Rock SW Engineering > > 1.5. Email Aliases: > 1.5.1. Responsible Manager: Fadi Salem > 1.5.2. Responsible Engineer: Sree Vemuri > 1.5.3. Interest List: John Johnson > > 2. Project Summary > 2.1. Project Description: > > Certain multi-node sun4v platforms (Batoka or Maramba) have an > integrated > PCI topology that causes the IO device paths to change in a CMP node > failover condition. In the event a CMP node fails, the system can be > booted to OBP with the remaining CMP nodes. But, with the current > architecture Solaris fails to boot as the PCIE device paths have > changed. The device paths are hard coded in the /etc/path_to_inst > file and the symlinks under /dev. When the same device is found > at a different I/O path after power-cycle it is treated as a new > device. Customer can not easily recover from the loss of I/O. > A change to the boot path requires Solaris re-install. > > This project creates a process that allows the system administrator > to modify the Solaris idea of the device paths to match that of the HW. > > 2.2. Risks and Assumptions: > > The solaris script should pose no risk. > The script assumes /etc/path_to_inst doesn't change format. > > 3. Business Summary > 3.1. Problem Area: > > Current architecture prevents operating system boot if the PCIe > device paths have changed due to a node failover condition. > > This case is attempting to solve a long standing problem where I/O > topology changes necessitate an operating system reinstall. > > 4. Technical Description: > 4.1. Details: > > The T5440 FW describes the possible device path aliases and current > configuration in the MD (see FWARC 2008/349). For example: > > When cpu0 exists, we use its path (/pci@400) to access IO; the other > paths via other CPUs are listed as aliases: > > node ioalias pcie0 { > current = "/pci@400"; > aliases = "/pci@500/pci@0/pci@1/pci@0/pci@1" + > "/pci@600/pci@0/pci@1" + > "/pci@700/pci@0/pci@1/pci@0/pci@1/pci@0/pci@1"; > back -> ioaliases; > }; > > This project involves a perl script that combines the current Solaris > device paths and the ioalias information from the FW to change > /etc/path_to_inst and the /dev symlinks to reflect the new HW > configuration. > > Note that since the device path of the root file system may be affected, > the suggested procedure will be to boot the failsafe archive (which > is on a ramdisk that is immune to device path changes) and run the > script from there. > > 5. Reference Documents: > > [1] FWARC 2008/349 New MD Nodes to Support I/O Reconfiguration for > Failover Conditions > http://sac.sfbay.sun.com/Archives/CaseLog/arc/FWARC/2008/349/onepager.txt > > > > > > System Administration Commands device_remap(1M) > > NAME > device_remap - administration program for the Solaris I/O > remapping feature > > SYNOPSIS > /usr/platform/sun4v/sbin/device_remap > [-v | -d dir] > > DESCRIPTION > Certain multi-node sun4v platforms, for example T5440 and > T5240 servers have an integrated PCI topology that cause the > IO device paths to change in a CPU node failover condition. > The Device Remapping Script for SPARC Enterprise T5440 > servers remaps the device paths in /etc/path_to_inst > file and the symlinks under /dev. > > OPTIONS > The following options are supported: > > -v > > Prints the path_to_inst and /dev/symlink changes > > -d dir > > Operate on the /etc/path_to_inst and /dev on the > root image at > > > USAGE > > The primary function of device_remap is to remap the > device paths in /etc/path_to_inst file and the symlinks under > /dev in a CPU node failover condition. > > After adding CPU node/s or removing CPU node/s, boot the system > to OBP prompt and use the following procedure: > > 1. Boot an install miniroot, either with "boot net -s" or > "boot -F failsafe". > > 2. Mount the root disk as /mnt. > > 3. Change directory to the mounted root disk. > cd /mnt > > 4. Run device_remap script. > /mnt/usr/platform/sun4v/sbin/device_remap > > 5. Reboot the system. > > All the error messages are self-explanatory, except for the error > message "missing ioaliases node" which means the FW on the system > doesn't support device remapping. > > EXAMPLES > Example 1 > > Prints the path_to_inst and /dev/symlink changes > > # device_remap -v > > Example 2 > > Changes directory to dir before doing any changes > > # device_remap -d dir > > > ATTRIBUTES > See attributes(5) for descriptions of the following attri- > butes: > > ____________________________________________________________ > | ATTRIBUTE TYPE | ATTRIBUTE VALUE | > |_____________________________|_____________________________| > | Availability | SUNWkvm | > |_____________________________|_____________________________| > | Interface Stability | Committed | > |_____________________________|_____________________________| > > SEE ALSO > boot(1M) > > NOTES > > _______________________________________________ > opensolaris-arc mailing list > opensolaris-arc@opensolaris.org From Brian.Ruthven@Sun.COM Thu Nov 20 03:25:58 2008 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAKBPw9w015050 for ; Thu, 20 Nov 2008 03:25:58 -0800 (PST) Received: from gmp-eb-inf-2.sun.com (gmp-eb-inf-2.EU.Sun.COM [192.18.6.24]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAKBPvJr025895 for ; Thu, 20 Nov 2008 03:25:58 -0800 (PST) Received: from fe-emea-10.sun.com (gmp-eb-lb-1-fe3.eu.sun.com [192.18.6.10]) by gmp-eb-inf-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAKBPqOe025570 for ; Thu, 20 Nov 2008 11:25:52 GMT Received: from conversion-daemon.fe-emea-10.sun.com by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAM00K01P2JYS00@fe-emea-10.sun.com> (original mail from Brian.Ruthven@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Thu, 20 Nov 2008 11:25:52 +0000 (GMT) Received: from [129.156.173.239] by fe-emea-10.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAM00GSXQEV3490@fe-emea-10.sun.com>; Thu, 20 Nov 2008 11:25:43 +0000 (GMT) Date: Thu, 20 Nov 2008 11:25:43 +0000 From: Brian Ruthven - Sun UK Subject: Re: Device Remap [PSARC/2008/723 FastTrack timeout 11/26/2008] In-reply-to: <200811200319.mAK3JXXx178036@jurassic.eng.sun.com> Sender: Brian.Ruthven@Sun.COM To: Jerry Gilliam Cc: psarc-ext@sac.sfbay.sun.com, Sree.Vemuri@Sun.COM, batoka-debug@Sun.COM, John.Johnson@Sun.COM, Fadi.Salem@Sun.COM, Ken.Brucker@Sun.COM Message-id: <49254937.5060702@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=us-ascii Content-transfer-encoding: 7BIT References: <200811200319.mAK3JXXx178036@jurassic.eng.sun.com> User-Agent: Thunderbird 2.0.0.9 (X11/20071119) Status: RO Content-Length: 8267 Some thoughts (mostly nits): What happens if /usr is mounted from a separate device, and this device is inaccesible until device_remap is run (i.e. chicken/egg problem)? Or does this simply fall into the "boot the failsafe archive" procedure you mention? Is this specific to the T5440, or more general to the sun4v architecture? i.e. should it go in /usr/platform/SUNW,SPARC-Enterprise-T5440/sbin (or similar) rather than /usr/platform/sun4v/sbin? The man page seems to indicate this is specific to the T5440: DESCRIPTION ... The Device Remapping Script for SPARC Enterprise T5440 servers remaps the device paths in /etc/path_to_inst file and the symlinks under /dev. [ nit: remaps them to what? to "match the hardware" I suspect. Same at the top of the usage section. ] What happens if I run this on a non-T5440 system, e.g. T2000? Also in the USAGE section: 1. Boot an install miniroot, either with "boot net -s" or "boot -F failsafe". I don't think the failsafe is an install miniroot is it? Does that matter? Would this be better as something like: 1. Boot either the failsafe miniroot using "boot -F failsafe", or an install miniroot using "boot net -s" or similar. Why is step 3 necessary - "cd /mnt" when the script is run from /mnt anyway. Is the cwd significant? [ This picks up on Pete's comments about the -d option ] In step 5, does it matter how the system is rebooted? Must init be used, or is "reboot" sufficient? I've even come across sysadmins who think reboot == pull the power, but that's a different problem, and not one PSARC can solve ;-) Thanks, Brian Jerry Gilliam wrote: > I'm sponsoring the following fast-track for Sree Vemuri, with > timeout of one week, 11/26/2008. > > The project requests patch/minor binding, with the path name > of the command committed and the output of the command > not an interface. > > ------- > > > 1. Introduction > 1.1. Project/Component Working Name: > Device Remap Script for SPARC Enterprise T5440 Servers > > 1.2. Name of Document Author/Supplier: > Sree Vemuri > > 1.3. Date of This Document: > 10/09/2008 > > 1.4. Name of Major Document Customer(s)/Consumer(s): > 1.4.1. The PAC or CPT you expect to review your project: > Solaris PAC > 1.4.2. The ARC(s) you expect to review your project: > PSARC > 1.4.3. The Director/VP who is "Sponsoring" this project: > quresh.dhoon@sun.com > 1.4.4. The name of your business unit: > Niagara Rock SW Engineering > > 1.5. Email Aliases: > 1.5.1. Responsible Manager: Fadi Salem > 1.5.2. Responsible Engineer: Sree Vemuri > 1.5.3. Interest List: John Johnson > > 2. Project Summary > 2.1. Project Description: > > Certain multi-node sun4v platforms (Batoka or Maramba) have an > integrated > PCI topology that causes the IO device paths to change in a CMP node > failover condition. In the event a CMP node fails, the system can be > booted to OBP with the remaining CMP nodes. But, with the current > architecture Solaris fails to boot as the PCIE device paths have > changed. The device paths are hard coded in the /etc/path_to_inst > file and the symlinks under /dev. When the same device is found > at a different I/O path after power-cycle it is treated as a new > device. Customer can not easily recover from the loss of I/O. > A change to the boot path requires Solaris re-install. > > This project creates a process that allows the system administrator > to modify the Solaris idea of the device paths to match that of the HW. > > 2.2. Risks and Assumptions: > > The solaris script should pose no risk. > The script assumes /etc/path_to_inst doesn't change format. > > 3. Business Summary > 3.1. Problem Area: > > Current architecture prevents operating system boot if the PCIe > device paths have changed due to a node failover condition. > > This case is attempting to solve a long standing problem where I/O > topology changes necessitate an operating system reinstall. > > 4. Technical Description: > 4.1. Details: > > The T5440 FW describes the possible device path aliases and current > configuration in the MD (see FWARC 2008/349). For example: > > When cpu0 exists, we use its path (/pci@400) to access IO; the other > paths via other CPUs are listed as aliases: > > node ioalias pcie0 { > current = "/pci@400"; > aliases = "/pci@500/pci@0/pci@1/pci@0/pci@1" + > "/pci@600/pci@0/pci@1" + > "/pci@700/pci@0/pci@1/pci@0/pci@1/pci@0/pci@1"; > back -> ioaliases; > }; > > This project involves a perl script that combines the current Solaris > device paths and the ioalias information from the FW to change > /etc/path_to_inst and the /dev symlinks to reflect the new HW > configuration. > > Note that since the device path of the root file system may be affected, > the suggested procedure will be to boot the failsafe archive (which > is on a ramdisk that is immune to device path changes) and run the > script from there. > > 5. Reference Documents: > > [1] FWARC 2008/349 New MD Nodes to Support I/O Reconfiguration for > Failover Conditions > http://sac.sfbay.sun.com/Archives/CaseLog/arc/FWARC/2008/349/onepager.txt > > > > > > System Administration Commands device_remap(1M) > > NAME > device_remap - administration program for the Solaris I/O > remapping feature > > SYNOPSIS > /usr/platform/sun4v/sbin/device_remap > [-v | -d dir] > > DESCRIPTION > Certain multi-node sun4v platforms, for example T5440 and > T5240 servers have an integrated PCI topology that cause the > IO device paths to change in a CPU node failover condition. > The Device Remapping Script for SPARC Enterprise T5440 > servers remaps the device paths in /etc/path_to_inst > file and the symlinks under /dev. > > OPTIONS > The following options are supported: > > -v > > Prints the path_to_inst and /dev/symlink changes > > -d dir > > Operate on the /etc/path_to_inst and /dev on the > root image at > > > USAGE > > The primary function of device_remap is to remap the > device paths in /etc/path_to_inst file and the symlinks under > /dev in a CPU node failover condition. > > After adding CPU node/s or removing CPU node/s, boot the system > to OBP prompt and use the following procedure: > > 1. Boot an install miniroot, either with "boot net -s" or > "boot -F failsafe". > > 2. Mount the root disk as /mnt. > > 3. Change directory to the mounted root disk. > cd /mnt > > 4. Run device_remap script. > /mnt/usr/platform/sun4v/sbin/device_remap > > 5. Reboot the system. > > All the error messages are self-explanatory, except for the error > message "missing ioaliases node" which means the FW on the system > doesn't support device remapping. > > EXAMPLES > Example 1 > > Prints the path_to_inst and /dev/symlink changes > > # device_remap -v > > Example 2 > > Changes directory to dir before doing any changes > > # device_remap -d dir > > > ATTRIBUTES > See attributes(5) for descriptions of the following attri- > butes: > > ____________________________________________________________ > | ATTRIBUTE TYPE | ATTRIBUTE VALUE | > |_____________________________|_____________________________| > | Availability | SUNWkvm | > |_____________________________|_____________________________| > | Interface Stability | Committed | > |_____________________________|_____________________________| > > SEE ALSO > boot(1M) > > NOTES > > -- Brian Ruthven Sun Microsystems UK Solaris Revenue Product Engineering Tel: +44 (0)1252 422 312 Sparc House, Guillemont Park, Camberley, GU17 9QG From carlsonj@phorcys.east.sun.com Thu Nov 20 05:08:46 2008 Received: from phorcys.east.sun.com (phorcys.East.Sun.COM [129.148.174.143]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAKD8jbM017084 for ; Thu, 20 Nov 2008 05:08:46 -0800 (PST) Received: from phorcys.east.sun.com (localhost [127.0.0.1]) by phorcys.east.sun.com (8.14.3+Sun/8.14.3) with ESMTP id mAKD8iGk006404; Thu, 20 Nov 2008 08:08:44 -0500 (EST) Received: (from carlsonj@localhost) by phorcys.east.sun.com (8.14.3+Sun/8.14.3/Submit) id mAKD8iXN006401; Thu, 20 Nov 2008 08:08:44 -0500 (EST) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <18725.24924.59368.428455@gargle.gargle.HOWL> Date: Thu, 20 Nov 2008 08:08:44 -0500 From: James Carlson To: Jerry Gilliam Cc: psarc-ext@sac.sfbay.sun.com, Sree.Vemuri@sun.com, Ken.Brucker@sun.com, Fadi.Salem@sun.com, batoka-debug@sun.com, John.Johnson@sun.com Subject: Re: Device Remap [PSARC/2008/723 FastTrack timeout 11/26/2008] In-Reply-To: <200811200319.mAK3JXXx178036@jurassic.eng.sun.com> References: <200811200319.mAK3JXXx178036@jurassic.eng.sun.com> X-Mailer: VM 7.01 under Emacs 21.3.1 Status: RO Content-Length: 886 Jerry Gilliam writes: > The solaris script should pose no risk. > The script assumes /etc/path_to_inst doesn't change format. That assumption is actually an imported interface. I believe that path_to_inst is Uncommitted. As long as this script is delivered from the same consolidation as the other parsers of that file (ON), that shouldn't be a problem. (The generation of the 'failsafe' archive is shrouded in cross-consolidation mystery, but I suspect that hiding this script away in /usr/platform should be sufficient to get it included. That's something for the project team to check.) +1, assuming that all of the above is no problem. -- James Carlson, Solaris Networking Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 From Sree.Vemuri@Sun.COM Thu Nov 20 12:33:09 2008 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAKKX97S023170 for ; Thu, 20 Nov 2008 12:33:09 -0800 (PST) Received: from sca-es-mail-1.sun.com (sca-es-mail-1.Sun.COM [192.18.43.132]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAKKX8fI024759 for ; Thu, 20 Nov 2008 12:33:08 -0800 (PST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAKKX3XM023369 for ; Thu, 20 Nov 2008 12:33:03 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAN00L01FPBY300@fe-sfbay-09.sun.com> (original mail from Sree.Vemuri@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Thu, 20 Nov 2008 12:33:03 -0800 (PST) Received: from [129.146.90.66] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAN0002AFQY4U00@fe-sfbay-09.sun.com>; Thu, 20 Nov 2008 12:32:59 -0800 (PST) Date: Thu, 20 Nov 2008 12:26:09 -0800 From: Sree Vemuri Subject: Re: Device Remap [PSARC/2008/723 FastTrack timeout 11/26/2008] In-reply-to: <49254176.4080406@sun.com> Sender: Sree.Vemuri@Sun.COM To: Peter Dennis - Sustaining Engineer Cc: Jerry Gilliam , psarc-ext@sac.sfbay.sun.com, Ken.Brucker@Sun.COM, Fadi.Salem@Sun.COM, batoka-debug@Sun.COM, John.Johnson@Sun.COM Message-id: <4925C7E1.5090309@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT References: <200811200319.mAK3JXXx178036@jurassic.eng.sun.com> <49254176.4080406@sun.com> User-Agent: Thunderbird 2.0.0.18pre (X11/20081102) Status: RO Content-Length: 1378 Peter Dennis - Sustaining Engineer wrote: > Just some nits: > > -d dir > - should this be -R instead to be consistent with > other commands that change files in a mounted > root disk (ie update_grub -R ) > I can change this to -R > -v > - does this mean show what would happen but do > nothing or actually be verbose about what changes > are being applied ? It is a verbose about the changes that have been applied. > > Is the example in the man page correct: > > > 3. Change directory to the mounted root disk. > > cd /mnt > > > > 4. Run device_remap script. > > /mnt/usr/platform/sun4v/sbin/device_remap > > Should it be /mnt/usr/platform/sun4v/sbin/device_remap -d /mnt ? If -d option is not specified, the script will run on the current directory which is /mnt in this case. > Will the device_remap be included in the failsafe image, if > so then the command becomes: > /usr/platform/sun4v/sbin/device_remap -d /mnt > This script can be included in the failsafe image. Will check and make the change. -thanks sree > > Pete > > Jerry Gilliam wrote: >> >> I'm sponsoring the following fast-track for Sree Vemuri, with >> timeout of one week, 11/26/2008. >> >> The project requests patch/minor binding, with the path name >> of the command committed and the output of the command >> not an interface. >> From Sree.Vemuri@Sun.COM Thu Nov 20 14:36:16 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mAKMaFRp027226 for ; Thu, 20 Nov 2008 14:36:15 -0800 (PST) Received: from sca-es-mail-2.sun.com (sca-es-mail-2.Sun.COM [192.18.43.133]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mAKMaFBp054684 for ; Thu, 20 Nov 2008 14:36:15 -0800 (PST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mAKMaAML011081 for ; Thu, 20 Nov 2008 14:36:10 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAN00L01KX9J000@fe-sfbay-09.sun.com> (original mail from Sree.Vemuri@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Thu, 20 Nov 2008 14:36:10 -0800 (PST) Received: from [129.146.90.66] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAN00CZGLG34J00@fe-sfbay-09.sun.com>; Thu, 20 Nov 2008 14:36:03 -0800 (PST) Date: Thu, 20 Nov 2008 14:29:13 -0800 From: Sree Vemuri Subject: Re: Device Remap [PSARC/2008/723 FastTrack timeout 11/26/2008] In-reply-to: <49254937.5060702@sun.com> Sender: Sree.Vemuri@Sun.COM To: Brian Ruthven - Sun UK Cc: Jerry Gilliam , psarc-ext@sac.sfbay.sun.com, batoka-debug@Sun.COM, John.Johnson@Sun.COM, Fadi.Salem@Sun.COM, Ken.Brucker@Sun.COM Message-id: <4925E4B9.9070901@sun.com> MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=us-ascii Content-transfer-encoding: 7BIT References: <200811200319.mAK3JXXx178036@jurassic.eng.sun.com> <49254937.5060702@sun.com> User-Agent: Thunderbird 2.0.0.18pre (X11/20081102) Status: RO Content-Length: 2808 Brian Ruthven - Sun UK wrote: > > Some thoughts (mostly nits): > > What happens if /usr is mounted from a separate device, and this > device is inaccesible until device_remap is run (i.e. chicken/egg > problem)? Or does this simply fall into the "boot the failsafe > archive" procedure you mention? > When a cpu node fails for some reason, the system will power off. When the system is powered back up, you have a system with fewer CPU nodes and the IO path of the devices on the failed CPU node would have reconfigured to a different path. If you boot up with the previous /dev links and /etc/path_to_inst files, those IO devices will not be accessible. So, boot the system in failsafe mode and run device_remap script which will update /dev and /etc/path_to_inst to match the new the path. > Is this specific to the T5440, or more general to the sun4v > architecture? i.e. should it go in > /usr/platform/SUNW,SPARC-Enterprise-T5440/sbin (or similar) rather > than /usr/platform/sun4v/sbin? > This script is not specific to T5440, it will be used by Rock. It can be used on a system which supports plx reconfig and the FW supports the ioalias MD property (FWARC 2008/349). It can be used on Maramba. Will remove T5440 from the description. > The man page seems to indicate this is specific to the T5440: > > DESCRIPTION > ... > The Device Remapping Script for SPARC Enterprise T5440 > servers remaps the device paths in /etc/path_to_inst > file and the symlinks under /dev. > > [ nit: remaps them to what? to "match the hardware" I suspect. Same at > the top of the usage section. ] > Yes, it remaps to match the hardware. Will make the change. Thanks. > What happens if I run this on a non-T5440 system, e.g. T2000? You will see the error message which is explained in the man page. "missing ioaliases node" > > Also in the USAGE section: > > 1. Boot an install miniroot, either with "boot net -s" or > "boot -F failsafe". > > > I don't think the failsafe is an install miniroot is it? Does that > matter? Would this be better as something like: > > 1. Boot either the failsafe miniroot using "boot -F failsafe", or an > install miniroot using "boot net -s" or similar. > > Will make this change. > Why is step 3 necessary - "cd /mnt" when the script is run from /mnt > anyway. Is the cwd significant? [ This picks up on Pete's comments > about the -d option ] > You do not have to specify -d option if step 3 is followed. > In step 5, does it matter how the system is rebooted? Must init be > used, or is "reboot" sufficient? I've even come across sysadmins who > think reboot == pull the power, but that's a different problem, and > not one PSARC can solve ;-) > No. You can use any boot command as long as you are booting from the disk you have run the script on. -thanks sree From Sree.Vemuri@Sun.COM Fri Nov 21 10:07:18 2008 Received: from dm-sfbay-02.sfbay.sun.com (dm-sfbay-02.SFBay.Sun.COM [129.146.11.31]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mALI7ImD012351 for ; Fri, 21 Nov 2008 10:07:18 -0800 (PST) Received: from sca-es-mail-1.sun.com (sca-es-mail-1.Sun.COM [192.18.43.132]) by dm-sfbay-02.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mALI7HAY061402 for ; Fri, 21 Nov 2008 10:07:17 -0800 (PST) Received: from fe-sfbay-09.sun.com ([192.18.43.129]) by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id mALI7CCK016758 for ; Fri, 21 Nov 2008 10:07:12 -0800 (PST) Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAP00M013724K00@fe-sfbay-09.sun.com> (original mail from Sree.Vemuri@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Fri, 21 Nov 2008 10:07:12 -0800 (PST) Received: from [129.146.90.66] by fe-sfbay-09.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAP00IVE3NQ4RB0@fe-sfbay-09.sun.com>; Fri, 21 Nov 2008 10:07:02 -0800 (PST) Date: Fri, 21 Nov 2008 10:00:12 -0800 From: Sree Vemuri Subject: Re: Device Remap [PSARC/2008/723 FastTrack timeout 11/26/2008] In-reply-to: <18725.24924.59368.428455@gargle.gargle.HOWL> Sender: Sree.Vemuri@Sun.COM To: James Carlson Cc: Jerry Gilliam , psarc-ext@sac.sfbay.sun.com, Ken.Brucker@Sun.COM, Fadi.Salem@Sun.COM, batoka-debug@Sun.COM, John.Johnson@Sun.COM Message-id: <4926F72C.9080807@sun.com> MIME-version: 1.0 Content-type: multipart/mixed; boundary="Boundary_(ID_yd5xS7S6xo93zckZ44NRJA)" References: <200811200319.mAK3JXXx178036@jurassic.eng.sun.com> <18725.24924.59368.428455@gargle.gargle.HOWL> User-Agent: Thunderbird 2.0.0.18pre (X11/20081102) Status: RO Content-Length: 3758 This is a multi-part message in MIME format. --Boundary_(ID_yd5xS7S6xo93zckZ44NRJA) Content-type: text/plain; format=flowed; charset=us-ascii Content-transfer-encoding: 7BIT Thanks for all the review comments. Attached is the updated man page. -thanks sree James Carlson wrote: > Jerry Gilliam writes: > >> The solaris script should pose no risk. >> The script assumes /etc/path_to_inst doesn't change format. >> > > That assumption is actually an imported interface. I believe that > path_to_inst is Uncommitted. As long as this script is delivered from > the same consolidation as the other parsers of that file (ON), that > shouldn't be a problem. > > (The generation of the 'failsafe' archive is shrouded in > cross-consolidation mystery, but I suspect that hiding this script > away in /usr/platform should be sufficient to get it included. That's > something for the project team to check.) > > +1, assuming that all of the above is no problem. > > --Boundary_(ID_yd5xS7S6xo93zckZ44NRJA) Content-type: text/plain; name=device_remap-man.txt Content-transfer-encoding: 7BIT Content-disposition: inline; filename=device_remap-man.txt System Administration Commands device_remap(1M) NAME device_remap - administration program for the Solaris I/O remapping feature SYNOPSIS /usr/platform/sun4v/sbin/device_remap [-v | -R dir] DESCRIPTION Certain multi-node sun4v platforms, for example T5440 and T5240 servers have an integrated PCI topology that cause the IO device paths to change in a CPU node failover condition. The Device Remapping Script remaps the device paths in /etc/path_to_inst file and the symlinks under /dev to match the hardware. OPTIONS The following options are supported: -v Prints the path_to_inst and /dev/symlink changes -R dir Operate on the /etc/path_to_inst and /dev on the root image at USAGE The primary function of device_remap is to remap the device paths in /etc/path_to_inst file and the symlinks under /dev in a CPU node failover condition to match the hardware. After adding CPU node/s or removing CPU node/s, boot the system to OBP prompt and use the following procedure: 1. Boot either the failsafe miniroot using "boot -F failsafe", or an install miniroot using "boot net -s" or similar. 2. Mount the root disk as /mnt. 3. Change directory to the mounted root disk. cd /mnt 4. Run device_remap script. /mnt/usr/platform/sun4v/sbin/device_remap 5. Boot the system from disk. All the error messages are self-explanatory, except for the error message "missing ioaliases node" which means the FW on the system doesn't support device remapping. EXAMPLES Example 1 Prints the path_to_inst and /dev/symlink changes # device_remap -v Example 2 Changes directory to dir before doing any changes # device_remap -R dir ATTRIBUTES See attributes(5) for descriptions of the following attri- butes: ____________________________________________________________ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | |_____________________________|_____________________________| | Availability | SUNWkvm | |_____________________________|_____________________________| | Interface Stability | Committed | |_____________________________|_____________________________| SEE ALSO boot(1M) NOTES SunOS 5.10 Last change: 09 Oct 2008 1 --Boundary_(ID_yd5xS7S6xo93zckZ44NRJA)-- From Ken.Brucker@Sun.COM Fri Nov 21 11:21:03 2008 Received: from dm-sfbay-01.sfbay.sun.com (dm-sfbay-01.SFBay.Sun.COM [129.145.155.118]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mALJL34L016170 for ; Fri, 21 Nov 2008 11:21:03 -0800 (PST) Received: from brmea-mail-1.sun.com (brmea-mail-1.Sun.COM [192.18.98.31]) by dm-sfbay-01.sfbay.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id mALJL25G058839 for ; Fri, 21 Nov 2008 11:21:03 -0800 (PST) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id mALJL2Kk006306 for ; Fri, 21 Nov 2008 19:21:02 GMT Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KAP00A01684Z300@mail-amer.sun.com> (original mail from Ken.Brucker@Sun.COM) for psarc-ext@sac.sfbay.sun.com; Fri, 21 Nov 2008 12:21:02 -0700 (MST) Received: from [192.168.1.100] ([72.235.158.45]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KAP006Q572R4F80@mail-amer.sun.com>; Fri, 21 Nov 2008 12:20:53 -0700 (MST) Date: Fri, 21 Nov 2008 09:20:50 -1000 From: Ken Brucker Subject: Re: Device Remap [PSARC/2008/723 FastTrack timeout 11/26/2008] In-reply-to: <4926F72C.9080807@sun.com> Sender: Ken.Brucker@Sun.COM To: Sree Vemuri Cc: James Carlson , Jerry Gilliam , psarc-ext@sac.sfbay.sun.com, Fadi.Salem@Sun.COM, batoka-debug@Sun.COM, John.Johnson@Sun.COM Message-id: <73055D20-A606-4590-8351-4D4A0C3C9BE1@Sun.COM> MIME-version: 1.0 X-Mailer: Apple Mail (2.929.2) Content-type: multipart/alternative; boundary="Boundary_(ID_sH6Hh6rIjmtRRMPEJp7WRA)" References: <200811200319.mAK3JXXx178036@jurassic.eng.sun.com> <18725.24924.59368.428455@gargle.gargle.HOWL> <4926F72C.9080807@sun.com> Status: RO Content-Length: 2713 --Boundary_(ID_sH6Hh6rIjmtRRMPEJp7WRA) Content-type: text/plain; delsp=yes; format=flowed; charset=US-ASCII Content-transfer-encoding: 7BIT On Nov 21, 2008, at 8:00 AM, Sree Vemuri wrote: > DESCRIPTION > Certain multi-node sun4v platforms, for example T5440 and > T5240 servers have an integrated PCI topology that cause the > IO device paths to change in a CPU node failover condition. > The Device Remapping Script remaps the device paths in > /etc/path_to_inst file and the symlinks under /dev to match > the hardware. With respect to this portion of the page, how does a customer determine if they are on one of these "certain multi-node sun4v platforms"? Executing the script to determine this doesn't seem like an ideal solution. -- Ken --Boundary_(ID_sH6Hh6rIjmtRRMPEJp7WRA) Content-type: text/html; charset=US-ASCII Content-transfer-encoding: QUOTED-PRINTABLE
On Nov 21, 20= 08, at 8:00 AM, Sree Vemuri wrote:

DESCRIPTION
   = ; Certain multi-node  sun4v  platforms, for  exam= ple T5440 and
    T5240 servers have an integr= ated PCI topology that cause the 
    IO device paths to change i= n a CPU node  failover condition.
    The=  Device  Remapping  Script  remaps  the &nb= sp;device paths in
    /etc/path_to_inst file = and the symlinks under /dev to  match
    = ;the hardware.

With respect to = this portion of the page, how does a customer determine if they are o= n one of these "certain multi-node sun4v platforms"?  Executing = the script to determine this doesn't seem like an ideal solution.

-- Ken
--Boundary_(ID_sH6Hh6rIjmtRRMPEJp7WRA)-- From jg@jurassic.sfbay.Sun.COM Tue Dec 2 14:22:59 2008 Received: from jurassic.eng.sun.com (jurassic-226-b.SFBay.Sun.COM [129.146.226.130]) by sac.sfbay.sun.com (8.13.8+Sun/8.13.8) with ESMTP id mB2MMxX0027446 for ; Tue, 2 Dec 2008 14:22:59 -0800 (PST) Received: from tethys (tethys.SFBay.Sun.COM [129.146.226.92]) by jurassic.eng.sun.com (8.13.8+Sun/8.13.8) with SMTP id mB2MMwxV603088; Tue, 2 Dec 2008 14:22:58 -0800 (PST) Message-Id: <200812022222.mB2MMwxV603088@jurassic.eng.sun.com> Date: Tue, 2 Dec 2008 14:18:03 -0800 (PST) From: Jerry Gilliam Reply-To: Jerry Gilliam Subject: Re: Device Remap [PSARC/2008/723 FastTrack timeout 11/26/2008] To: psarc-ext@sac.sfbay.sun.com, Sree.Vemuri@sun.com Cc: batoka-debug@sun.com, John.Johnson@sun.com, Fadi.Salem@sun.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: VZhBTP67Y21fCH8ipXFgEQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.7_94 SunOS 5.11 sun4u sparc Status: RO Content-Length: 109 Timed out, closed and approved, with the approved materials in the case directory (final.spec). thx, -jg