blog'o thnet

In a previous discussion about the GRUB boot archive and how it can be regenerated, I mentioned that it will not be as easy as it can be when the root file system use the md driver. I will now show two different methods to do the same thing when the root file system is build upon a SVM mirror (RAID-1):

1. Unmirror the root file system only.
2. Unmirror the entire system, i.e. all devices.

Note: Although this test case was done using Solaris 10 8/07 under a virtual machine build upon VirtualBox on latest Solaris Express Community Edition, the instructions must be valid for Solaris 10 1/06 and later.

Initial setup

As we can see, the system use only a root file system, and a swap device. Both are encapsulated with SVM.

# df -k -F ufs
Filesystem     kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d0 6147798 3455578 2630743      57%  /
# swap -l
swapfile             dev  swaplo blocks   free
/dev/md/dsk/d1      85,1       8 4194288 4194288
# metastat -c d0 d1
d0               m  6.0GB d10 d20
    d10          s  6.0GB c0d0s0
    d20          s  6.0GB c1d1s0
d1               m  2.0GB d11 d21
    d11          s  2.0GB c0d0s1
    d21          s  2.0GB c1d1s1

Unmirror the root file system only

The idea is to boot on the GRUB Failsafe mode, select the first side of the mirror, and modify the system and vfstab configuration files to use the correct device path. For the system file, this means to actually remove the rootdev:/pseudo/md@0:0,0,bl entry, not just comment it. For the vfstab file, this means replacing the root file system metadevice path /dev/md/[r]dsk/d0 by the first underlying device path, i.e. /dev/[r]dsk/c0d0s0. Last, regenerate the boot archive on the alternate root path.

[...]
Booting to milestone "milestone/single-user:default".
Configuring devices.
Searching for installed OS instances...
/dev/dsk/c0d0s0 is under md control, skipping.
/dev/dsk/c1d1s0 is under md control, skipping.
No installed OS instance found.

Starting shell.
# fsck /dev/rdsk/c0d0s0
# mount -F ufs /dev/dsk/c0d0s0 /a
# cp /a/etc/system /a/etc/system.bckp
# cp /a/etc/vfstab /a/etc/vfstab.bckp
# TERM=vt100 vi /a/etc/system
# TERM=vt100 vi /a/etc/vfstab
# bootadm update-archive -R /a
# umount /a
# fsck /dev/rdsk/c0d0s0
# reboot

Then, boot into milestone/multi-user:default level and detach the second half of the mirror, since the first half correspond to the valid and updated underlying device. Next, restore the original configuration files which refers to the encapsulated metadevices, and reboot.

# df -k -F ufs
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0d0s0      6147798 3458810 2627511    57%    /
# swap -l
swapfile             dev  swaplo blocks   free
/dev/md/dsk/d1      85,1       8 4194288 4194288
# metastat -c d0
d0               m  6.0GB d10 d20
    d10          s  6.0GB c0d0s0
    d20          s  6.0GB c1d1s0
# metadetach d0 d20
d0: submirror d20 is detached
# metastat -c d0
d0               m  6.0GB d10
    d10          s  6.0GB c0d0s0
# cp /etc/system.orig /etc/system
# cp /etc/vfstab.orig /etc/vfstab
# shutdown -y -i 6 -g 0

After the reboot, just reattach the second half of the mirror, and wait for complete synchronization to be fully redundant again.

# df -k -F ufs
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d0       6147798 3458714 2627607    57%    /
# swap -l
swapfile             dev  swaplo blocks   free
/dev/md/dsk/d1      85,1       8 4194288 4194288
# metattach d0 d20
d0: submirror d20 is attached
# metastat -c d0
d0               m  6.0GB d10 d20 (resync-29%)
    d10          s  6.0GB c0d0s0
    d20          s  6.0GB c1d1s0

Unmirror the entire system, i.e. all devices

The idea is exactly the same as for unmirroring the root file system only, but adapting the vfstab file to change the swap entry, too. (So, I didn't reproduce the code listing here.)

Then, boot into milestone/single-user:default level modifying the corresponding GRUB entry as follow: kernel /platform/i86pc/multiboot -s. Completely delete all the metadevices and metadb configurations to clear SVM settings. Last, continue into milestone/multi-user:default level to boot unmirrored.

# metaclear -f -r d0 d1
# metadb -f -d  c1d0s4 c1d0s4
# ^D

Now, the system must be fully encapsulate by SVM again. Please refer to online Sun Documentation, or some past entries on this subject, depending on the system's architecture: SPARC systems, or x86 platforms.

Here is a little step-by-step guide to create a soft mirror from the root file system, known as an encapsulation of the system's disk. This will provide full protection against one disk failure, and complete redundancy. In the same time, this will have the effect to speed read requests (since there exists multiple backing devices hosting the same data), but write performance is generally degraded. First, know your running system, particularly on which disk it is currently installed and which other device is available for the second mirror side.

# df -hF ufs
Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c1d0s0        7.9G   5.2G   2.6G    67%    /
# swap -lh
swapfile             dev    swaplo   blocks     free
/dev/dsk/c1d0s1     102,65       4K     4.0G     4.0G
#
# echo | format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c1d0 
          /pci@0,0/pci-ide@8/ide@0/cmdk@0,0
       1. c2d0 
          /pci@0,0/pci-ide@8/ide@1/cmdk@0,0
[...]

Well, we will use the c2d0 as the second submirror. So, we need to default to one Solaris partition that uses the whole disk and make it bootable (we are using GRUB in this case). The slice for the second submirror must have a slice tag of root and the root slice must be slice 0 (so, we will duplicate the label's content from the boot disk to the mirror disk).

# fdisk -B /dev/rdsk/c2d0p0
# fdisk /dev/rdsk/c2d0p0
             Total disk size is 36483 cylinders
             Cylinder size is 16065 (512 byte) blocks

                                               Cylinders
      Partition   Status    Type          Start   End   Length    %
      =========   ======    ============  =====   ===   ======   ===
          1       Active    Solaris2          1  36482    36482    100

SELECT ONE OF THE FOLLOWING:
   1. Create a partition
   2. Specify the active partition
   3. Delete a partition
   4. Change between Solaris and Solaris2 Partition IDs
   5. Exit (update disk configuration and exit)
   6. Cancel (exit without updating disk configuration)
Enter Selection:
#
# prtvtoc /dev/rdsk/c1d0s2 | fmthard -s - /dev/rdsk/c2d0s2
fmthard:  New volume table of contents now in place.
#
# /sbin/installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c2d0s0
stage1 written to partition 0 sector 0 (abs 16065)
stage2 written to partition 0, 260 sectors starting at 50 (abs 16115)

Create replicas of the metadevice state database:

# metadb -a -c 3 -f c1d0s4 c2d0s4
# metadb
        flags           first blk       block count
     a        u         16              8192            /dev/dsk/c1d0s4
     a        u         8208            8192            /dev/dsk/c1d0s4
     a        u         16400           8192            /dev/dsk/c1d0s4
     a        u         16              8192            /dev/dsk/c2d0s4
     a        u         8208            8192            /dev/dsk/c2d0s4
     a        u         16400           8192            /dev/dsk/c2d0s4

Flag -f is needed because it is the first invocation/creation of metadb(1m).

Set up the RAID-0 metadevices (stripe or concatenation volumes) corresponding to the / file system and the swap space, and automatically configure system files (/etc/vfstab and /etc/system) for the root metadevice.

# metainit -f d10 1 1 c1d0s0
d10: Concat/Stripe is setup
# metainit -f d11 1 1 c1d0s1
d11: Concat/Stripe is setup
# metainit d20 1 1 c2d0s0
d20: Concat/Stripe is setup
# metainit d21 1 1 c2d0s1
d21: Concat/Stripe is setup
# metainit d0 -m d10
d0: Mirror is setup
# metainit d1 -m d11
d1: Mirror is setup
#
# cp /etc/vfstab /etc/vfstab.beforesvm
# sed -e 's@/dev/dsk/c1d0s1@/dev/md/dsk/d1@' /etc/vfstab.beforesvm > /etc/vfstab
# metaroot d0
# diff /etc/vfstab /etc/vfstab.beforesvm
6,7c6,7
< /dev/md/dsk/d1   -                 -   swap   -   no   -
< /dev/md/dsk/d0   /dev/md/rdsk/d0   /   ufs    1   no   -
---
> /dev/dsk/c1d0s1  -                 -   swap   -   no   -
> /dev/dsk/c1d0s0  /dev/rdsk/c1d0s0  /   ufs    1   no   -

Flag -f is needed because the file systems created on the slice we want to initialize a new metadevice are currently mounted (in use).

Reboot on the metadevices: the operating system will now boot encapsulated, on a one-side mirror. Last, attach the second part of the mirror and adapt the system dump configuration.

# lockfs -af && shutdown -y -g 0 -i 6
[...]
# metattach d0 d20
d0: submirror d20 is attached
# metattach d1 d21
d1: submirror d21 is attached
#
# metastat -p
d1 -m /dev/md/rdsk/d11 /dev/md/rdsk/d21 1
d11 1 1 /dev/rdsk/c1d0s1
d21 1 1 /dev/rdsk/c2d0s1
d0 -m /dev/md/rdsk/d10 /dev/md/rdsk/d20 1
d10 1 1 /dev/rdsk/c1d0s0
d20 1 1 /dev/rdsk/c2d0s0
# metastat | grep %
    Resync in progress: 41 % done
    Resync in progress: 46 % done
#
# rmdir /var/crash/*
# mkdir /var/crash/`hostname`
# chmod 700 /var/crash/`hostname`
# dumpadm -s /var/crash/`hostname` -d /dev/md/dsk/d1
      Dump content: kernel pages
       Dump device: /dev/md/dsk/d1 (swap)
Savecore directory: /var/crash/bento
  Savecore enabled: yes

Last, define the alternative boot path in the menu.lst GRUB configuration file: the Solaris/BSD slice 0 on the first fdisk partition on the second BIOS disk.

cat << EOF >> /boot/grub/menu.lst
title Solaris Nevada snv_65 X86 (Alternate Boot Path)
root (hd1,0,a)
kernel$ /platform/i86pc/kernel/$ISADIR/unix
module$ /platform/i86pc/$ISADIR/boot_archive
EOF
#
# bootadm list-menu 
The location for the active GRUB menu is: /boot/grub/menu.lst
default 0
timeout 10
0 Solaris Nevada snv_65 X86
1 Solaris failsafe
2 Solaris Nevada snv_65 X86 (Alternate Boot Path)

For further (and deeper) information on this subject, please refer to the excellent Sun Microsystems Documentation on Solaris Volume Manager, and particularly x86: Creating a RAID-1 Volume From the root (/) File System.

1. c0t0d0s2 represents the first system disk (boot)
2. c0t1d0s2 represents the second disk (mirror)

Duplicate the label's content from the boot disk to the mirror disk:

# prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t1d0s2

Create replicas of the metadevice state database:

# metadb -a -c3 -f c0t0d0s4 c0t1d0s4
# metadb

Option -f is needed because it is the first invocation/creation of metadb(1m).

Creation of metadevices:

# metainit -f d10 1 1 c0t0d0s0
# metainit -f d11 1 1 c0t0d0s1
# metainit -f d13 1 1 c0t0d0s3
# metainit -f d16 1 1 c0t0d0s6
#
# metainit d20 1 1 c0t1d0s0
# metainit d21 1 1 c0t1d0s1
# metainit d23 1 1 c0t1d0s3
# metainit d26 1 1 c0t1d0s6

Option -f is needed because the file systems created on the slice we want to initialize a new metadevice are already mounted.

Create the first part of the mirror:

# metainit d0 -m d10
# metainit d1 -m d11
# metainit d3 -m d13
# metainit d6 -m d16
#
# cp /etc/vfstab /etc/vfstab.beforesvm
# metaroot d0

Don't forget to edit /etc/vfstab in order to reflect the other metadevices:

• s@/dev/dsk/cXtYdZsN@/dev/md/dsk/dN@
• s@/dev/rdsk/cXtYdZsN@/dev/md/rdsk/dN@

Install the boot block code on the alternate boot disk and set it in the OpenBoot Prom (OBP):

# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t1d0s0
# eeprom boot-device="disk disk1 net"   /* Or just "disk disk1". */

Reboot on the new metadevices (the operating system will now boot encapsulated):

# shutdown -y -g 0 -i 6

Attach the second part of the mirror:

# metattach d0 d20
# metattach d1 d21
# metattach d3 d23
# metattach d6 d26

Verify all:

# metastat -p
# metastat | grep \%

Modify the system dump configuration:

# mkdir /var/crash/`hostname`
# chmod 700 /var/crash/`hostname`
# dumpadm -s /var/crash/`hostname`
# dumpadm -d /dev/md/dsk/d1