blog'o thnet

Here, the final aim was to provide data access redundancy through SAN storage hosted on remote sites across Wide Area Network (WAN) links. After some relatively long and painful tries to mimic software mirroring as found on HP-UX platform using Logical Volume Management (LVM), i.e. at the logical volume level, I finally give up deciding this functionality will definitely not fit my need. Why? Here are my comments.

1. It is not possible to provide clear and manageable storage multipath when the need to distinguish between the multiple sites is important, ala mirror across controllers found on Veritas VxVM on Sun Solaris system, for example. So, managing many physical volumes along with lots of logical volumes is very complicated.
2. There is no exact mapping capability between logical volume storage on a given physical volume.
3. The need to have a disk-based log, i.e. a persistent log. Yes, one can always provide the option --corelog at the creation time to the logical volume initial build and have an in-memory log , i.e. a non-persistent log, but this requires the entire copies (mirrors) be resynchronized upon reboot. Not really viable on multi-TB environments.
4. A write-intensive workload on a file system living on a logical volume mirror will suffer high latency: the overhead is important, and the time to do mostly-write jobs grow dramatically. It is really hard to get high level statistics, only low level metrics seems consistent: sd SCSI devices and dm- device mapper components for each paths entries. Not from the multipath devices standpoint, which is the more interesting from the end user and SA point of view.
5. You can't extend a logical volume, which is really a no-go per-se. On that point, the Red Hat support answered that this functionality may be added in a future release, the current state may eventually be a Request For Enhancement (RFE), if a proper business justification is provided. One must break the logical volume mirror copy, then rebuild it completely. Not realistic when the logical volume is made of a lot of physical extents across multiple physical volumes.
6. A LVM configuration can be totally blocked by itself, and not usable at all. The fact is, LVM use persistent storage blocks to keep track of its own metadata. The metadata size is set at physical volume creation time only, and can't be change afterward. This size is statically defined as 255 physical volume blocks, and can be adjust from the LVM configuration file. The problem is, when this circular buffer space (stored in ASCII) fills up--such as when there are a lot of logical volumes in a mirrored environment--it is not possible to do anything more with LVM. So you can't add more logical volume, can't add more logical volume copies,... and can't delete them trying to reestablish a proper LVM configuration. Well, here are the answers given by the Red Hat support to two keys questions in this situation:
   • How to size the metadata, i.e. if we need to change it from the default value, how can we determine the new proper and appropriate size, and from which information?
     I am afraid but Metadata size can only be defined at the time of PV creation and there is no real formula for calculating the size in advance. By changing the default value of 255 you can get a higher space value. For general LVM setup (with less LV's and VG's) default size works fine however in cases where high number of LV's are required a custom value will be required.
   • We just want to delete all LV copies, which means to return to the initial situation and have 0 copy for all LV, i.e. only one LV per-se, in order to be able to change LVM configuration again (we can't do anything on our production server right now)?
     I discussed this situation with peer engineers and also referenced a similar previous case. From the notes of the same the workaround is to use the backup file (/etc/lvm/backup) and restore the PV's. I agree that this really not a production environment method however seems the only workaround.

So, the production RDBMS Oracle server is finally now being evacuate to an other machine. Hum... Hope to see better enterprise experience using the mdadm package to handle RAID software, instead of mirror (RAID-1) LVM. Maybe more about that in an other blog entry?

Here is a little step-by-step guide to create a soft mirror from the root file system, known as an encapsulation of the system's disk. This will provide full protection against one disk failure, and complete redundancy. In the same time, this will have the effect to speed read requests (since there exists multiple backing devices hosting the same data), but write performance is generally degraded. First, know your running system, particularly on which disk it is currently installed and which other device is available for the second mirror side.

# df -hF ufs
Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c1d0s0        7.9G   5.2G   2.6G    67%    /
# swap -lh
swapfile             dev    swaplo   blocks     free
/dev/dsk/c1d0s1     102,65       4K     4.0G     4.0G
#
# echo | format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c1d0 
          /pci@0,0/pci-ide@8/ide@0/cmdk@0,0
       1. c2d0 
          /pci@0,0/pci-ide@8/ide@1/cmdk@0,0
[...]

Well, we will use the c2d0 as the second submirror. So, we need to default to one Solaris partition that uses the whole disk and make it bootable (we are using GRUB in this case). The slice for the second submirror must have a slice tag of root and the root slice must be slice 0 (so, we will duplicate the label's content from the boot disk to the mirror disk).

# fdisk -B /dev/rdsk/c2d0p0
# fdisk /dev/rdsk/c2d0p0
             Total disk size is 36483 cylinders
             Cylinder size is 16065 (512 byte) blocks

                                               Cylinders
      Partition   Status    Type          Start   End   Length    %
      =========   ======    ============  =====   ===   ======   ===
          1       Active    Solaris2          1  36482    36482    100

SELECT ONE OF THE FOLLOWING:
   1. Create a partition
   2. Specify the active partition
   3. Delete a partition
   4. Change between Solaris and Solaris2 Partition IDs
   5. Exit (update disk configuration and exit)
   6. Cancel (exit without updating disk configuration)
Enter Selection:
#
# prtvtoc /dev/rdsk/c1d0s2 | fmthard -s - /dev/rdsk/c2d0s2
fmthard:  New volume table of contents now in place.
#
# /sbin/installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c2d0s0
stage1 written to partition 0 sector 0 (abs 16065)
stage2 written to partition 0, 260 sectors starting at 50 (abs 16115)

Create replicas of the metadevice state database:

# metadb -a -c 3 -f c1d0s4 c2d0s4
# metadb
        flags           first blk       block count
     a        u         16              8192            /dev/dsk/c1d0s4
     a        u         8208            8192            /dev/dsk/c1d0s4
     a        u         16400           8192            /dev/dsk/c1d0s4
     a        u         16              8192            /dev/dsk/c2d0s4
     a        u         8208            8192            /dev/dsk/c2d0s4
     a        u         16400           8192            /dev/dsk/c2d0s4

Flag -f is needed because it is the first invocation/creation of metadb(1m).

Set up the RAID-0 metadevices (stripe or concatenation volumes) corresponding to the / file system and the swap space, and automatically configure system files (/etc/vfstab and /etc/system) for the root metadevice.

# metainit -f d10 1 1 c1d0s0
d10: Concat/Stripe is setup
# metainit -f d11 1 1 c1d0s1
d11: Concat/Stripe is setup
# metainit d20 1 1 c2d0s0
d20: Concat/Stripe is setup
# metainit d21 1 1 c2d0s1
d21: Concat/Stripe is setup
# metainit d0 -m d10
d0: Mirror is setup
# metainit d1 -m d11
d1: Mirror is setup
#
# cp /etc/vfstab /etc/vfstab.beforesvm
# sed -e 's@/dev/dsk/c1d0s1@/dev/md/dsk/d1@' /etc/vfstab.beforesvm > /etc/vfstab
# metaroot d0
# diff /etc/vfstab /etc/vfstab.beforesvm
6,7c6,7
< /dev/md/dsk/d1   -                 -   swap   -   no   -
< /dev/md/dsk/d0   /dev/md/rdsk/d0   /   ufs    1   no   -
---
> /dev/dsk/c1d0s1  -                 -   swap   -   no   -
> /dev/dsk/c1d0s0  /dev/rdsk/c1d0s0  /   ufs    1   no   -

Flag -f is needed because the file systems created on the slice we want to initialize a new metadevice are currently mounted (in use).

Reboot on the metadevices: the operating system will now boot encapsulated, on a one-side mirror. Last, attach the second part of the mirror and adapt the system dump configuration.

# lockfs -af && shutdown -y -g 0 -i 6
[...]
# metattach d0 d20
d0: submirror d20 is attached
# metattach d1 d21
d1: submirror d21 is attached
#
# metastat -p
d1 -m /dev/md/rdsk/d11 /dev/md/rdsk/d21 1
d11 1 1 /dev/rdsk/c1d0s1
d21 1 1 /dev/rdsk/c2d0s1
d0 -m /dev/md/rdsk/d10 /dev/md/rdsk/d20 1
d10 1 1 /dev/rdsk/c1d0s0
d20 1 1 /dev/rdsk/c2d0s0
# metastat | grep %
    Resync in progress: 41 % done
    Resync in progress: 46 % done
#
# rmdir /var/crash/*
# mkdir /var/crash/`hostname`
# chmod 700 /var/crash/`hostname`
# dumpadm -s /var/crash/`hostname` -d /dev/md/dsk/d1
      Dump content: kernel pages
       Dump device: /dev/md/dsk/d1 (swap)
Savecore directory: /var/crash/bento
  Savecore enabled: yes

Last, define the alternative boot path in the menu.lst GRUB configuration file: the Solaris/BSD slice 0 on the first fdisk partition on the second BIOS disk.

cat << EOF >> /boot/grub/menu.lst
title Solaris Nevada snv_65 X86 (Alternate Boot Path)
root (hd1,0,a)
kernel$ /platform/i86pc/kernel/$ISADIR/unix
module$ /platform/i86pc/$ISADIR/boot_archive
EOF
#
# bootadm list-menu 
The location for the active GRUB menu is: /boot/grub/menu.lst
default 0
timeout 10
0 Solaris Nevada snv_65 X86
1 Solaris failsafe
2 Solaris Nevada snv_65 X86 (Alternate Boot Path)

For further (and deeper) information on this subject, please refer to the excellent Sun Microsystems Documentation on Solaris Volume Manager, and particularly x86: Creating a RAID-1 Volume From the root (/) File System.

Because of the nature of services provided by the ThNET Project, i already want to keep the I/O very reliable and used some old hardware RAID technology to do the job. Since there is no full hardware support due to some provider legal binary restrictions, the solution wasn't perfect and i always take a lot of time rebuilding the RAID array because of some obscure problem when problems on disks occurred.

I didn't want to do the same thing with the new infrastructure server, and decided to build the RAID-1 solution on top of the gmirror(8) software, a GEOM framework based tool provided under recent FreeBSD releases.

So, based on the work of others (see the end of this entry for references), here are the steps i follow to switch from classical to mirror solution for the main server of the project.

First, make sure that the second disk is treated as a new, fresh one:

# dd if=/dev/zero of=/dev/ad10 bs=512 count=79

Put a GEOM label onto it and force load the gmirror.ko kernel module:

# gmirror label -v -n -b round-robin gm0 /dev/ad10
# gmirror load

Then write a PC (BIOS) MBR, place a new BSD label, initialize it and create custom partitions:

# fdisk -v -B -I /dev/mirror/gm0
# bsdlabel -w -B /dev/mirror/gm0s1
# bsdlabel -e /dev/mirror/gm0s1
# cat << EOF > /etc/bsdlabel.gm0s1
# /dev/mirror/gm0s1:
8 partitions:
#         size   offset   fstype   [fsize bsize bps/cpg]
# a: Will be mounted as `/'.
# d: Will be mounted as `/var'.
# e: Will be mounted as `/tmp'.
# f: Will be mounted as `/usr'.
# g: Will be mounted as `/home'.
a:      512M       16   4.2BSD     2048 16384       8
b:     2048M        *     swap
c: 586099332        0   unused        0     0         # "raw" part, don't edit
d:     2048M        *   4.2BSD     2048 16384   28528
e:     4096M        *   4.2BSD     2048 16384   28528
f:    12288M        *   4.2BSD     2048 16384   28528
g:         *        *   4.2BSD     2048 16384   28528
EOF
# bsdlabel -R /dev/mirror/gm0s1 /etc/bsdlabel.gm0s1

Make new file systems on the corresponding partitions (note: generally speaking, it seems better not to put soft-updates on the root partition):

# newfs /dev/mirror/gm0s1a
# newfs -U /dev/mirror/gm0s1d
# newfs -U /dev/mirror/gm0s1e
# newfs -U /dev/mirror/gm0s1f
# newfs -U /dev/mirror/gm0s1g

Populate the content the of the second disk using dump(8) and restore(8) for example, or use some backup if this may be applicable for you:

# mkdir /tmp/gm0s1 && mount /mnt/da0
# 
# mount /dev/mirror/gm0s1a /tmp/gm0s1
# gzip -dc /mnt/da0/dump/2006-04-10.*/bento.thilelli.net.root.dump.gz | \
(cd /tmp/gm0s1 && restore -rf -)
# 
# mount /dev/mirror/gm0s1d /tmp/gm0s1/var
# gzip -dc /mnt/da0/dump/2006-04-10.*/bento.thilelli.net.var.dump.gz | \
(cd /tmp/gm0s1/var && restore -rf -)
# 
# mount /dev/mirror/gm0s1f /tmp/gm0s1/usr
# gzip -dc /mnt/da0/dump/2006-04-10.*/bento.thilelli.net.usr.dump.gz | \
(cd /tmp/gm0s1/usr && restore -rf -)
# 
# mount /dev/mirror/gm0s1g /tmp/gm0s1/home
# gzip -dc /mnt/da0/dump/2006-04-10.*/bento.thilelli.net.home.dump.gz | \
(cd /tmp/gm0s1/home && restore -rf -)
# 
# mount /dev/mirror/gm0s1e /tmp/gm0s1/tmp
# chmod 1777 /tmp/gm0s1/tmp

Prepare the new file system table, force the load of the GEOM mirror at boot time (necessary for the root mount) and be sure to boot on the second disk at the next reboot:

# cp -p /tmp/gm0s1/etc/fstab /tmp/gm0s1/etc/fstab.orig
/*
 * sed -e 's/dev\/ad8/dev\/mirror\/gm0/g' < /tmp/gm0s1/etc/fstab.orig \
 *  > /tmp/gm0s1/etc/fstab
 */
# cat << EOF > /tmp/gm0s1/etc/fstab
# Device                Mountpoint      FStype  Options                         Dump    Pass#
/dev/mirror/gm0s1b      none            swap    sw                              0       0
/dev/mirror/gm0s1a      /               ufs     rw                              1       1
/dev/mirror/gm0s1e      /tmp            ufs     rw,noatime,nosuid,nodev         2       2
/dev/mirror/gm0s1f      /usr            ufs     rw                              2       2
/dev/mirror/gm0s1d      /var            ufs     rw,noexec                       2       2
/dev/mirror/gm0s1g      /home           ufs     rw,userquota,nosuid,nodev       2       2
/dev/acd0               /cdrom          cd9660  ro,noauto                       0       0
/dev/da0s1              /mnt/da0        ufs     rw,noauto,nosuid,nodev          0       0
/dev/da1s1              /mnt/da1        msdosfs rw,noauto                       0       0
EOF
# echo geom_mirror_load=\"YES\" >> /tmp/gm0s1/boot/loader.conf
# echo "1:ad(1,a)/boot/loader" > /boot.config

Unmount the second side of the mirror and reboot:

# umount /tmp/gm0s1/tmp
# umount /tmp/gm0s1/home
# umount /tmp/gm0s1/usr
# umount /tmp/gm0s1/var
# umount /tmp/gm0s1
# 
# sync && shutdown -r now

After rebooting on the second disk (the GEOMified one), switch the mirror to auto-synchronization and add the first disk, which is now immediately synchronized with the second disk's content:

# gmirror configure -a gm0
# gmirror insert gm0 /dev/ad8
# 
# gmirror list
Geom name: gm0
State: DEGRADED
Components: 2
Balance: round-robin
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 770137303
Providers:
1. Name: mirror/gm0
Mediasize: 300090727936 (279G)
Sectorsize: 512
Mode: r7w6e7
Consumers:
1. Name: ad10
Mediasize: 300090728448 (279G)
Sectorsize: 512
Mode: r1w1e1
State: ACTIVE
Priority: 0
Flags: NONE
GenID: 0
SyncID: 1
ID: 2706535066
2. Name: ad8
Mediasize: 300090728448 (279G)
Sectorsize: 512
Mode: r1w1e1
State: SYNCHRONIZING
Priority: 0
Flags: DIRTY, SYNCHRONIZING
GenID: 0
SyncID: 1
Synchronized: 71%
ID: 2682952005
# 
# gmirror status
Name    Status  Components
mirror/gm0  DEGRADED  ad10
              ad8 (71%)

During all of these steps, some kernel messages may be seen on the console or in the /var/log/messages system logs file, as shown below:

# tail -f /var/log/messages
GEOM_MIRROR: Device gm0: provider ad8 detected.
GEOM_MIRROR: Device gm0: rebuilding provider ad8.
GEOM_MIRROR: Device gm0: rebuilding provider ad8 finished.
GEOM_MIRROR: Device gm0: provider ad8 activated.

Last, please find some invaluable documentation on the subject, with a special note for the BSD DevCenter one since, although less secure for the data than the others, bypassed the need for duplicating the data from one disk to the other, much less as found when using SVM on Solaris from Sun Microsystems.

• BSD DevCenter: Using Software RAID-1 with FreeBSD
• FreeBSD Material from Ralf S. Engelschall: FreeBSD System Disk Mirroring
• FreeBSD Handbook: RAID1 - Mirroring

Last week, the mirror array for the system broke one more time. After some investigation, It turns out to be a faulty Serillel adapter. Luckily i had one more of this in the stock... just in case (what a good idea, isn't it?).

On the other hand, the mirror array for the data (home directories) broke itself during the overall restore procedure. Still in this degraded mode for now, need more time to work on this... quickly :(

# grep DEGRADED /var/log/messages | tail -1
Jul 17 17:54:24 bento kernel: ar1: 117246MB <ATA RAID1 array> [14946/255/63] status: DEGRADED subdisks:

# atacontrol status ar1
ar1: ATA RAID1 subdisks: ad8 status: DEGRADED

Don't forget to make some good backups! There is no good reason not to do so. It is just a matter of time to use them.

1. c0t0d0s2 represents the first system disk (boot)
2. c0t1d0s2 represents the second disk (mirror)

Duplicate the label's content from the boot disk to the mirror disk:

# prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t1d0s2

Create replicas of the metadevice state database:

# metadb -a -c3 -f c0t0d0s4 c0t1d0s4
# metadb

Option -f is needed because it is the first invocation/creation of metadb(1m).

Creation of metadevices:

# metainit -f d10 1 1 c0t0d0s0
# metainit -f d11 1 1 c0t0d0s1
# metainit -f d13 1 1 c0t0d0s3
# metainit -f d16 1 1 c0t0d0s6
#
# metainit d20 1 1 c0t1d0s0
# metainit d21 1 1 c0t1d0s1
# metainit d23 1 1 c0t1d0s3
# metainit d26 1 1 c0t1d0s6

Option -f is needed because the file systems created on the slice we want to initialize a new metadevice are already mounted.

Create the first part of the mirror:

# metainit d0 -m d10
# metainit d1 -m d11
# metainit d3 -m d13
# metainit d6 -m d16
#
# cp /etc/vfstab /etc/vfstab.beforesvm
# metaroot d0

Don't forget to edit /etc/vfstab in order to reflect the other metadevices:

• s@/dev/dsk/cXtYdZsN@/dev/md/dsk/dN@
• s@/dev/rdsk/cXtYdZsN@/dev/md/rdsk/dN@

Install the boot block code on the alternate boot disk and set it in the OpenBoot Prom (OBP):

# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t1d0s0
# eeprom boot-device="disk disk1 net"   /* Or just "disk disk1". */

Reboot on the new metadevices (the operating system will now boot encapsulated):

# shutdown -y -g 0 -i 6

Attach the second part of the mirror:

# metattach d0 d20
# metattach d1 d21
# metattach d3 d23
# metattach d6 d26

Verify all:

# metastat -p
# metastat | grep \%

Modify the system dump configuration:

# mkdir /var/crash/`hostname`
# chmod 700 /var/crash/`hostname`
# dumpadm -s /var/crash/`hostname`
# dumpadm -d /dev/md/dsk/d1