Because of the nature of services provided by the ThNET Project, i already want to keep the I/O very reliable and used some old hardware RAID technology to do the job. Since there is no full hardware support due to some provider legal binary restrictions, the solution wasn't perfect and i always take a lot of time rebuilding the RAID array because of some obscure problem when problems on disks occurred.
I didn't want to do the same thing with the new infrastructure server,
and decided to build the RAID-1 solution on top of the gmirror(8)
software, a GEOM framework based tool provided under recent FreeBSD
releases.
So, based on the work of others (see the end of this entry for references), here are the steps i follow to switch from classical to mirror solution for the main server of the project.
First, make sure that the second disk is treated as a new, fresh one:
# dd if=/dev/zero of=/dev/ad10 bs=512 count=79
Put a GEOM label onto it and force load the gmirror.ko kernel module:
# gmirror label -v -n -b round-robin gm0 /dev/ad10
# gmirror load
Then write a PC (BIOS) MBR, place a new BSD label, initialize it and create custom partitions:
# fdisk -v -B -I /dev/mirror/gm0
# bsdlabel -w -B /dev/mirror/gm0s1
# bsdlabel -e /dev/mirror/gm0s1
# cat << EOF > /etc/bsdlabel.gm0s1
# /dev/mirror/gm0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
# a: Will be mounted as `/'.
# d: Will be mounted as `/var'.
# e: Will be mounted as `/tmp'.
# f: Will be mounted as `/usr'.
# g: Will be mounted as `/home'.
a: 512M 16 4.2BSD 2048 16384 8
b: 2048M * swap
c: 586099332 0 unused 0 0 # "raw" part, don't edit
d: 2048M * 4.2BSD 2048 16384 28528
e: 4096M * 4.2BSD 2048 16384 28528
f: 12288M * 4.2BSD 2048 16384 28528
g: * * 4.2BSD 2048 16384 28528
EOF
# bsdlabel -R /dev/mirror/gm0s1 /etc/bsdlabel.gm0s1
Make new file systems on the corresponding partitions (note: generally speaking, it seems better not to put soft-updates on the root partition):
# newfs /dev/mirror/gm0s1a
# newfs -U /dev/mirror/gm0s1d
# newfs -U /dev/mirror/gm0s1e
# newfs -U /dev/mirror/gm0s1f
# newfs -U /dev/mirror/gm0s1g
Populate the content the of the second disk using dump(8) and
restore(8) for example, or use some backup if this may be applicable
for you:
# mkdir /tmp/gm0s1 && mount /mnt/da0
#
# mount /dev/mirror/gm0s1a /tmp/gm0s1
# gzip -dc /mnt/da0/dump/2006-04-10.*/bento.thilelli.net.root.dump.gz | \
(cd /tmp/gm0s1 && restore -rf -)
#
# mount /dev/mirror/gm0s1d /tmp/gm0s1/var
# gzip -dc /mnt/da0/dump/2006-04-10.*/bento.thilelli.net.var.dump.gz | \
(cd /tmp/gm0s1/var && restore -rf -)
#
# mount /dev/mirror/gm0s1f /tmp/gm0s1/usr
# gzip -dc /mnt/da0/dump/2006-04-10.*/bento.thilelli.net.usr.dump.gz | \
(cd /tmp/gm0s1/usr && restore -rf -)
#
# mount /dev/mirror/gm0s1g /tmp/gm0s1/home
# gzip -dc /mnt/da0/dump/2006-04-10.*/bento.thilelli.net.home.dump.gz | \
(cd /tmp/gm0s1/home && restore -rf -)
#
# mount /dev/mirror/gm0s1e /tmp/gm0s1/tmp
# chmod 1777 /tmp/gm0s1/tmp
Prepare the new file system table, force the load of the GEOM mirror at boot time (necessary for the root mount) and be sure to boot on the second disk at the next reboot:
# cp -p /tmp/gm0s1/etc/fstab /tmp/gm0s1/etc/fstab.orig
/*
* sed -e 's/dev\/ad8/dev\/mirror\/gm0/g' < /tmp/gm0s1/etc/fstab.orig \
* > /tmp/gm0s1/etc/fstab
*/
# cat << EOF > /tmp/gm0s1/etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/mirror/gm0s1b none swap sw 0 0
/dev/mirror/gm0s1a / ufs rw 1 1
/dev/mirror/gm0s1e /tmp ufs rw,noatime,nosuid,nodev 2 2
/dev/mirror/gm0s1f /usr ufs rw 2 2
/dev/mirror/gm0s1d /var ufs rw,noexec 2 2
/dev/mirror/gm0s1g /home ufs rw,userquota,nosuid,nodev 2 2
/dev/acd0 /cdrom cd9660 ro,noauto 0 0
/dev/da0s1 /mnt/da0 ufs rw,noauto,nosuid,nodev 0 0
/dev/da1s1 /mnt/da1 msdosfs rw,noauto 0 0
EOF
# echo geom_mirror_load=\"YES\" >> /tmp/gm0s1/boot/loader.conf
# echo "1:ad(1,a)/boot/loader" > /boot.config
Unmount the second side of the mirror and reboot:
# umount /tmp/gm0s1/tmp
# umount /tmp/gm0s1/home
# umount /tmp/gm0s1/usr
# umount /tmp/gm0s1/var
# umount /tmp/gm0s1
#
# sync && shutdown -r now
After rebooting on the second disk (the GEOMified one), switch the mirror to auto-synchronization and add the first disk, which is now immediately synchronized with the second disk's content:
# gmirror configure -a gm0
# gmirror insert gm0 /dev/ad8
#
# gmirror list
Geom name: gm0
State: DEGRADED
Components: 2
Balance: round-robin
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 770137303
Providers:
1. Name: mirror/gm0
Mediasize: 300090727936 (279G)
Sectorsize: 512
Mode: r7w6e7
Consumers:
1. Name: ad10
Mediasize: 300090728448 (279G)
Sectorsize: 512
Mode: r1w1e1
State: ACTIVE
Priority: 0
Flags: NONE
GenID: 0
SyncID: 1
ID: 2706535066
2. Name: ad8
Mediasize: 300090728448 (279G)
Sectorsize: 512
Mode: r1w1e1
State: SYNCHRONIZING
Priority: 0
Flags: DIRTY, SYNCHRONIZING
GenID: 0
SyncID: 1
Synchronized: 71%
ID: 2682952005
#
# gmirror status
Name Status Components
mirror/gm0 DEGRADED ad10
ad8 (71%)
During all of these steps, some kernel messages may be seen on the
console or in the /var/log/messages system logs file, as shown below:
# tail -f /var/log/messages
GEOM_MIRROR: Device gm0: provider ad8 detected.
GEOM_MIRROR: Device gm0: rebuilding provider ad8.
GEOM_MIRROR: Device gm0: rebuilding provider ad8 finished.
GEOM_MIRROR: Device gm0: provider ad8 activated.
Last, please find some invaluable documentation on the subject, with a special note for the BSD DevCenter one since, although less secure for the data than the others, bypassed the need for duplicating the data from one disk to the other, much less as found when using SVM on Solaris from Sun Microsystems.