Tag - upgrade - blog'o thnet

After writing about how to patch (or upgrade) a running system playing with a mirrored OpenSolaris SVM, here is a little step-by-step how to on upgrading (or patching, etc.) a live system using the Live Upgrade feature.

Before installing or running Live Upgrade, you are required to install a limited set of patch revisions. Make sure you have the most recently updated patch list by consulting sunsolve.sun.com. Search for the info doc 72099 on the SunSolve web site (you must have a registered Sun support customer account to be able to view this document).

Note: In the following procedure, we will assume that all we want (and need) to upgrade to is provided via a one large DVD ISO image.

If all seems OK, you must begin to update the current running system with the appropriate lu packages, i.e. those provided for the targeted OS revision. You can either use the provided tools:

# /cdrom/cdrom0/Solaris_11/Tools/Installers/liveupgrade20

Or do it yourself:

# pkgrm SUNWluu SUNWlur
# pkgadd -d /cdrom/cdrom0/Solaris_11/Product SUNWlur SUNWluu

Since the current OS is totally installed on the first slice of the first disk (c1d0s0), and that the slice six (c1d0s6) is exactly the same size as the first one, we will use it for the second ABE device for our purpose and create the corresponding Boot Environment.

# lucreate -c snv_38 -n snv_39 -m /:/dev/dsk/c1d0s6:ufs
/* If the snv_38 BE already exists, just create the new one for snv_39. */
# lucreate -n snv_39 -m /:/dev/dsk/c1d0s6:ufs
Discovering physical storage devices
Discovering logical storage devices
Cross referencing storage devices with boot environment configurations
Determining types of file systems supported
Validating file system requests
Preparing logical storage devices
Preparing physical storage devices
Configuring physical storage devices
Configuring logical storage devices
Analyzing system configuration.
Comparing source boot environment <snv_38> file systems with the file
system(s) you specified for the new boot environment. Determining which
file systems should be in the new boot environment.
Updating boot environment description database on all BEs.
Searching /dev for possible boot environment filesystem devices

Updating system configuration files.
The device </dev/dsk/c1d0s6> is not a root device for any boot environment.
Creating configuration for boot environment <snv_39>.
Source boot environment is <snv_38>.
Creating boot environment <snv_39>.
Checking for GRUB menu on boot environment <snv_39>.
The boot environment <snv_39> does not contain the GRUB menu.
Creating file systems on boot environment <snv_39>.
Creating <ufs> file system for </> on </dev/dsk/c1d0s6>.
Mounting file systems for boot environment <snv_39>.
Calculating required sizes of file systems for boot environment <snv_39>.
Populating file systems on boot environment <snv_39>.
Checking selection integrity.
Integrity check OK.
Populating contents of mount point </>.
Copying.
Creating shared file system mount points.
Creating compare databases for boot environment <snv_39>.
Creating compare database for file system </>.
Updating compare databases on boot environment <snv_39>.
Making boot environment <snv_39> bootable.
Updating bootenv.rc on ABE <snv_39>.
Population of boot environment <snv_39> successful.
Creation of boot environment <snv_39> successful.

Verify the correct attribution of the different file systems, in particular between those which are cloned (required by a Solaris installation, such as /, /var, /usr, and /opt) and those which are shared (such as /export).

# lufslist -n snv_38
               boot environment name: snv_38
               This boot environment is currently active.
               This boot environment will be active on next system boot.

Filesystem           fstype    device size Mounted on     Mount Options
-------------------- -------- ------------ -------------- --------------
/dev/dsk/c1d0s1      swap       4301821440 -              -
/dev/dsk/c1d0s0      ufs        8595417600 /              -
/dev/dsk/c1d0s7      ufs       58407713280 /export        -
#
# lufslist -n snv_39
               boot environment name: snv_39

Filesystem           fstype    device size Mounted on     Mount Options
-------------------- -------- ------------ -------------- --------------
/dev/dsk/c1d0s1      swap       4301821440 -              -
/dev/dsk/c1d0s6      ufs        8595417600 /              -
/dev/dsk/c1d0s7      ufs       58407713280 /export        -

You then just need to upgrade the second BE using the installation media of the desired release or revision.

# luupgrade -u -n snv_39 -s /cdrom/cdrom0

Install media is CD/DVD. </cdrom/cdrom0>.
Waiting for CD/DVD media </cdrom/cdrom0> ...
Copying failsafe multiboot from media.
Uncompressing miniroot
Creating miniroot device
miniroot filesystem is <ufs>
Mounting miniroot at </cdrom/cdrom0/Solaris_11/Tools/Boot>
Validating the contents of the media </cdrom/cdrom0>.
The media is a standard Solaris media.
The media contains an operating system upgrade image.
The media contains <Solaris> version <11>.
Constructing upgrade profile to use.
Locating the operating system upgrade program.
Checking for existence of previously scheduled Live Upgrade requests.
Creating upgrade profile for BE <snv_39>.
Checking for GRUB menu on ABE <snv_39>.
Checking for x86 boot partition on ABE.
Determining packages to install or upgrade for BE <snv_39>.
Performing the operating system upgrade of the BE <snv_39>.
CAUTION: Interrupting this process may leave the boot environment unstable
or unbootable.
Upgrading Solaris: 100% completed
Installation of the packages from this media is complete.
Deleted empty GRUB menu on ABE <snv_39>.
Adding operating system patches to the BE <snv_39>.
The operating system patch installation is complete.
ABE boot partition backing deleted.
Configuring failsafe for system.
Failsafe configuration is complete.
INFORMATION: The file </var/sadm/system/logs/upgrade_log> on boot
environment <snv_39> contains a log of the upgrade operation.
INFORMATION: The file </var/sadm/system/data/upgrade_cleanup> on boot
environment <snv_39> contains a log of cleanup operations required.
INFORMATION: Review the files listed above. Remember that all of the files
are located on boot environment <snv_39>. Before you activate boot
environment <snv_39>, determine if any additional system maintenance is
required or if additional media of the software distribution must be
installed.
The Solaris upgrade of the boot environment <snv_39> is complete.
Installing failsafe
Failsafe install is complete.

If something went wrong during the upgrade of the new Boot Environment snv_39, you can always restart with a very fresh one using the lumake -n snv_39 command. If all went smooth, you can now check and compare the newly created BE:

# lucompare -t snv_39 -o /tmp/lucompare.snv_39
# lumount -n snv_39
/.alt.snv_39
# mount -p | grep snv_39
/dev/dsk/c1d0s6 - /.alt.snv_39 ufs - no rw,intr,largefiles,logging,xattr,onerror=panic
# luumount -n snv_39
#
# lustatus
Boot Environment           Is       Active Active    Can    Copy
Name                       Complete Now    On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
snv_38                     yes      yes    yes       no     -
snv_39                     yes      no     no        yes    -

Since the overall upgrade is OK, you just need to activate the fresh BE snv_39, export your data zpool and perform a clean reboot, otherwise the new environment will not be activated. Do not use the uadmin, halt, or reboot commands!

# luactivate snv_39
# lustatus
Boot Environment           Is       Active Active    Can    Copy
Name                       Complete Now    On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
snv_38                     yes      yes    no        no     -
snv_39                     yes      no     yes       no     -
#
# zpool export datazp
# shutdown -y -g 0 -i 6

Et voilà! After the reboot, you must see something similar to:

# uname -a
SunOS unic 5.11 snv_39 i86pc i386 i86pc
#
# cat /etc/release
                            Solaris Nevada snv_39 X86
           Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                              Assembled 01 May 2006

Last, please find some invaluable documentation on the subject below:

Aim of this memo

The main purpose of this technical note is to demonstrate how to patch a running (live) system currently mirrored using SVM, minimizing the downtime as far as possible.

The idea is simple: detach one side of the mirror, apply the cluster patch against it and reboot on it. If all seems OK, re-encapsulate the system. This can achieve similar goal currently found in the Live Upgrade feature of the Solaris OS (see live_upgrade(5)), with less complexity and different requirement (LVM RAID-1 vs. spare disk, or free slice).

Using this solution, the downtime can go between 10 to 30 minutes of service unavailability (depending on the hardware POST) and a maximum of two reboots are required, whatever is the number of patches to apply.

Here it is

Here is a system encapsulated using SDS 4.x or SVM 1.x, and the associated SVM encapsulation configuration:

# metastat -p
d3 -m d13 d23 1
d13 1 1 c0t0d0s3
d23 1 1 c0t1d0s3
d1 -m d11 d21 1
d11 1 1 c0t0d0s1
d21 1 1 c0t1d0s1
d0 -m d10 d20 1
d10 1 1 c0t0d0s0
d20 1 1 c0t1d0s0
#
# cat /etc/vfstab
#device         device          mount   FS      fsck    mount   mount
#to mount       to fsck         point   type    pass    at boot options
#
fd      -       /dev/fd fd      -       no      -
/proc   -       /proc   proc    -       no      -
/dev/md/dsk/d3  -       -       swap    -       no      -
/dev/md/dsk/d0  /dev/md/rdsk/d0 /       ufs     1       no      -
/dev/md/dsk/d1  /dev/md/rdsk/d1 /var    ufs     1       no      -
swap    -       /tmp    tmpfs   -       yes     -

Run an explorer and generate a cluster patch, based on tools provided by the OSE for example, if you are luckily enough to have one included with your support plan (or just pick one provided at SunSolve).

Then, be sure to be able to boot on the two disks, just in case:

# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t0d0s0
# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t1d0s0

The next step is to voluntarily detach one side of the mirror: take the first one for the sake of simplicity (i.e. c0t0d0). Indeed, in this case we are pretty sure that its alias name at the OBP is disk.

Note: You can always create it at the OBP (using the usual set of commands, such as show-disks, devalias, etc.) if you want. That is just a matter of personal preferences.

# lockfs -af /* Just to minimize the fs inconsistencies at next fsck(1m). */
#
# metadetach d0 d10
# metadetach d1 d11
# metadetach d3 d13
#
# metaclear d10
# metaclear d11
# metaclear d13

Check and repair the file systems if necessary, since we will boot on them the next time:

# fsck /dev/dsk/c0t0d0s0
# fsck /dev/dsk/c0t0d0s1

Next steps include mounting the recently detached file systems and prepare the first disk to boot without SVM encapsulation:

# mkdir /mirror
# mount /dev/dsk/c0t0d0s0 /mirror
# mount /dev/dsk/c0t0d0s1 /mirror/var
#
# cat << EOF > /mirror/etc/vfstab
#device         device          mount   FS      fsck    mount   mount
#to mount       to fsck         point   type    pass    at boot options
#
fd      -       /dev/fd fd      -       no      -
/proc   -       /proc   proc    -       no      -
/dev/dsk/c0t0d0s3       -       -       swap    -       no      -
/dev/dsk/c0t0d0s0       /dev/rdsk/c0t0d0s0      /       ufs     1       no      -
/dev/dsk/c0t0d0s1       /dev/rdsk/c0t0d0s1      /var    ufs     1       no      -
swap    -       /tmp    tmpfs   -       yes     -
EOF
#
# cp /mirror/etc/system /mirror/etc/system.orig
# sed -e 's;rootdev:/pseudo/md@0:0,0,blk;*rootdev:/pseudo/md@0:0,0,blk;' \
   /mirror/etc/system.orig > /mirror/etc/system

Last, install patches against the first disk, clean things up a little and reboot if the install procedure went all smooth:

# ./install_all_patches -R /mirror
#
# umount /mirror/var
# umount /mirror
# rmdir /mirror
#
# shutdown -y -g 0 -i 6

After rebooting, carefully review the behavior of the very freshly patched system. If all seems well, don't forget to re-encapsulate the second disk. Here is a quick and easy way to this:

/* Recreate the metadb. */
# metadb -d c0t0d0s4 c0t1d0s4
# metadb -a -c3 -f c0t0d0s4 c0t1d0s4
#
/* Clean the system metadevices always present. */
# metaclear d0
# metaclear d1
# metaclear d3
# metaclear d20
# metaclear d21
# metaclear d23
#
/* Re-create them as part of a mirror. */
# metainit -f d10 1 1 c0t0d0s0
# metainit d0 -m d10
# metainit -f d11 1 1 c0t0d0s1
# metainit d1 -m d11
# metainit -f d13 1 1 c0t0d0s3
# metainit d3 -m d13
#
/* Be able to boot on the new metadevices. */
# metaroot d0
#
/* Reboot, and create the second side of the mirror. */
# shutdown -y -g 0 -i 6
[...]
# metainit d20 1 1 c0t1d0s0
# metattach d0 d20
# metainit d21 1 1 c0t1d0s1
# metattach d1 d21
# metainit d23 1 1 c0t1d0s3
# metattach d3 d23

For a little more detailed explanation about encapsulating the system using SVM on Sun Solaris, please refer to the dedicated entry in this blog.

Last, it must be mentioned that this documentation was written by our OSE, and that this procedure was officially marked as supported by Sun Microsystems.