Because of the nature of services provided by the ThNET Project, i already want to keep the I/O very reliable and used some old hardware RAID technology to do the job. Since there is no full hardware support due to some provider legal binary restrictions, the solution wasn't perfect and i always take a lot of time rebuilding the RAID array because of some obscure problem when problems on disks occurred.

I didn't want to do the same thing with the new infrastructure server, and decided to build the RAID-1 solution on top of the gmirror(8) software, a GEOM framework based tool provided under recent FreeBSD releases.

So, based on the work of others (see the end of this entry for references), here are the steps i follow to switch from classical to mirror solution for the main server of the project.

First, make sure that the second disk is treated as a new, fresh one:

# dd if=/dev/zero of=/dev/ad10 bs=512 count=79

Put a GEOM label onto it and force load the gmirror.ko kernel module:

# gmirror label -v -n -b round-robin gm0 /dev/ad10
# gmirror load

Then write a PC (BIOS) MBR, place a new BSD label, initialize it and create custom partitions:

# fdisk -v -B -I /dev/mirror/gm0
# bsdlabel -w -B /dev/mirror/gm0s1
# bsdlabel -e /dev/mirror/gm0s1
# cat << EOF > /etc/bsdlabel.gm0s1
# /dev/mirror/gm0s1:
8 partitions:
#         size   offset   fstype   [fsize bsize bps/cpg]
# a: Will be mounted as `/'.
# d: Will be mounted as `/var'.
# e: Will be mounted as `/tmp'.
# f: Will be mounted as `/usr'.
# g: Will be mounted as `/home'.
a:      512M       16   4.2BSD     2048 16384       8
b:     2048M        *     swap
c: 586099332        0   unused        0     0         # "raw" part, don't edit
d:     2048M        *   4.2BSD     2048 16384   28528
e:     4096M        *   4.2BSD     2048 16384   28528
f:    12288M        *   4.2BSD     2048 16384   28528
g:         *        *   4.2BSD     2048 16384   28528
EOF
# bsdlabel -R /dev/mirror/gm0s1 /etc/bsdlabel.gm0s1

Make new file systems on the corresponding partitions (note: generally speaking, it seems better not to put soft-updates on the root partition):

# newfs /dev/mirror/gm0s1a
# newfs -U /dev/mirror/gm0s1d
# newfs -U /dev/mirror/gm0s1e
# newfs -U /dev/mirror/gm0s1f
# newfs -U /dev/mirror/gm0s1g

Populate the content the of the second disk using dump(8) and restore(8) for example, or use some backup if this may be applicable for you:

# mkdir /tmp/gm0s1 && mount /mnt/da0
# 
# mount /dev/mirror/gm0s1a /tmp/gm0s1
# gzip -dc /mnt/da0/dump/2006-04-10.*/bento.thilelli.net.root.dump.gz | \
(cd /tmp/gm0s1 && restore -rf -)
# 
# mount /dev/mirror/gm0s1d /tmp/gm0s1/var
# gzip -dc /mnt/da0/dump/2006-04-10.*/bento.thilelli.net.var.dump.gz | \
(cd /tmp/gm0s1/var && restore -rf -)
# 
# mount /dev/mirror/gm0s1f /tmp/gm0s1/usr
# gzip -dc /mnt/da0/dump/2006-04-10.*/bento.thilelli.net.usr.dump.gz | \
(cd /tmp/gm0s1/usr && restore -rf -)
# 
# mount /dev/mirror/gm0s1g /tmp/gm0s1/home
# gzip -dc /mnt/da0/dump/2006-04-10.*/bento.thilelli.net.home.dump.gz | \
(cd /tmp/gm0s1/home && restore -rf -)
# 
# mount /dev/mirror/gm0s1e /tmp/gm0s1/tmp
# chmod 1777 /tmp/gm0s1/tmp

Prepare the new file system table, force the load of the GEOM mirror at boot time (necessary for the root mount) and be sure to boot on the second disk at the next reboot:

# cp -p /tmp/gm0s1/etc/fstab /tmp/gm0s1/etc/fstab.orig
/*
 * sed -e 's/dev\/ad8/dev\/mirror\/gm0/g' < /tmp/gm0s1/etc/fstab.orig \
 *  > /tmp/gm0s1/etc/fstab
 */
# cat << EOF > /tmp/gm0s1/etc/fstab
# Device                Mountpoint      FStype  Options                         Dump    Pass#
/dev/mirror/gm0s1b      none            swap    sw                              0       0
/dev/mirror/gm0s1a      /               ufs     rw                              1       1
/dev/mirror/gm0s1e      /tmp            ufs     rw,noatime,nosuid,nodev         2       2
/dev/mirror/gm0s1f      /usr            ufs     rw                              2       2
/dev/mirror/gm0s1d      /var            ufs     rw,noexec                       2       2
/dev/mirror/gm0s1g      /home           ufs     rw,userquota,nosuid,nodev       2       2
/dev/acd0               /cdrom          cd9660  ro,noauto                       0       0
/dev/da0s1              /mnt/da0        ufs     rw,noauto,nosuid,nodev          0       0
/dev/da1s1              /mnt/da1        msdosfs rw,noauto                       0       0
EOF
# echo geom_mirror_load=\"YES\" >> /tmp/gm0s1/boot/loader.conf
# echo "1:ad(1,a)/boot/loader" > /boot.config

Unmount the second side of the mirror and reboot:

# umount /tmp/gm0s1/tmp
# umount /tmp/gm0s1/home
# umount /tmp/gm0s1/usr
# umount /tmp/gm0s1/var
# umount /tmp/gm0s1
# 
# sync && shutdown -r now

After rebooting on the second disk (the GEOMified one), switch the mirror to auto-synchronization and add the first disk, which is now immediately synchronized with the second disk's content:

# gmirror configure -a gm0
# gmirror insert gm0 /dev/ad8
# 
# gmirror list
Geom name: gm0
State: DEGRADED
Components: 2
Balance: round-robin
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 770137303
Providers:
1. Name: mirror/gm0
Mediasize: 300090727936 (279G)
Sectorsize: 512
Mode: r7w6e7
Consumers:
1. Name: ad10
Mediasize: 300090728448 (279G)
Sectorsize: 512
Mode: r1w1e1
State: ACTIVE
Priority: 0
Flags: NONE
GenID: 0
SyncID: 1
ID: 2706535066
2. Name: ad8
Mediasize: 300090728448 (279G)
Sectorsize: 512
Mode: r1w1e1
State: SYNCHRONIZING
Priority: 0
Flags: DIRTY, SYNCHRONIZING
GenID: 0
SyncID: 1
Synchronized: 71%
ID: 2682952005
# 
# gmirror status
Name    Status  Components
mirror/gm0  DEGRADED  ad10
              ad8 (71%)

During all of these steps, some kernel messages may be seen on the console or in the /var/log/messages system logs file, as shown below:

# tail -f /var/log/messages
GEOM_MIRROR: Device gm0: provider ad8 detected.
GEOM_MIRROR: Device gm0: rebuilding provider ad8.
GEOM_MIRROR: Device gm0: rebuilding provider ad8 finished.
GEOM_MIRROR: Device gm0: provider ad8 activated.

Last, please find some invaluable documentation on the subject, with a special note for the BSD DevCenter one since, although less secure for the data than the others, bypassed the need for duplicating the data from one disk to the other, much less as found when using SVM on Solaris from Sun Microsystems.