Because of the nature of services provided by the ThNET Project, i already want to keep the I/O very reliable and used some old hardware RAID technology to do the job. Since there is no full hardware support due to some provider legal binary restrictions, the solution wasn't perfect and i always take a lot of time rebuilding the RAID array because of some obscure problem when problems on disks occurred.
I didn't want to do the same thing with the new infrastructure server,
and decided to build the RAID-1 solution on top of the
software, a GEOM framework based tool provided under recent FreeBSD
So, based on the work of others (see the end of this entry for references), here are the steps i follow to switch from classical to mirror solution for the main server of the project.
First, make sure that the second disk is treated as a new, fresh one:
# dd if=/dev/zero of=/dev/ad10 bs=512 count=79
Put a GEOM label onto it and force load the
gmirror.ko kernel module:
# gmirror label -v -n -b round-robin gm0 /dev/ad10 # gmirror load
Then write a PC (BIOS) MBR, place a new BSD label, initialize it and create custom partitions:
# fdisk -v -B -I /dev/mirror/gm0 # bsdlabel -w -B /dev/mirror/gm0s1 # bsdlabel -e /dev/mirror/gm0s1 # cat << EOF > /etc/bsdlabel.gm0s1 # /dev/mirror/gm0s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] # a: Will be mounted as `/'. # d: Will be mounted as `/var'. # e: Will be mounted as `/tmp'. # f: Will be mounted as `/usr'. # g: Will be mounted as `/home'. a: 512M 16 4.2BSD 2048 16384 8 b: 2048M * swap c: 586099332 0 unused 0 0 # "raw" part, don't edit d: 2048M * 4.2BSD 2048 16384 28528 e: 4096M * 4.2BSD 2048 16384 28528 f: 12288M * 4.2BSD 2048 16384 28528 g: * * 4.2BSD 2048 16384 28528 EOF # bsdlabel -R /dev/mirror/gm0s1 /etc/bsdlabel.gm0s1
Make new file systems on the corresponding partitions (note: generally speaking, it seems better not to put soft-updates on the root partition):
# newfs /dev/mirror/gm0s1a # newfs -U /dev/mirror/gm0s1d # newfs -U /dev/mirror/gm0s1e # newfs -U /dev/mirror/gm0s1f # newfs -U /dev/mirror/gm0s1g
Populate the content the of the second disk using
restore(8) for example, or use some backup if this may be applicable
# mkdir /tmp/gm0s1 && mount /mnt/da0 # # mount /dev/mirror/gm0s1a /tmp/gm0s1 # gzip -dc /mnt/da0/dump/2006-04-10.*/bento.thilelli.net.root.dump.gz | \ (cd /tmp/gm0s1 && restore -rf -) # # mount /dev/mirror/gm0s1d /tmp/gm0s1/var # gzip -dc /mnt/da0/dump/2006-04-10.*/bento.thilelli.net.var.dump.gz | \ (cd /tmp/gm0s1/var && restore -rf -) # # mount /dev/mirror/gm0s1f /tmp/gm0s1/usr # gzip -dc /mnt/da0/dump/2006-04-10.*/bento.thilelli.net.usr.dump.gz | \ (cd /tmp/gm0s1/usr && restore -rf -) # # mount /dev/mirror/gm0s1g /tmp/gm0s1/home # gzip -dc /mnt/da0/dump/2006-04-10.*/bento.thilelli.net.home.dump.gz | \ (cd /tmp/gm0s1/home && restore -rf -) # # mount /dev/mirror/gm0s1e /tmp/gm0s1/tmp # chmod 1777 /tmp/gm0s1/tmp
Prepare the new file system table, force the load of the GEOM mirror at boot time (necessary for the root mount) and be sure to boot on the second disk at the next reboot:
# cp -p /tmp/gm0s1/etc/fstab /tmp/gm0s1/etc/fstab.orig /* * sed -e 's/dev\/ad8/dev\/mirror\/gm0/g' < /tmp/gm0s1/etc/fstab.orig \ * > /tmp/gm0s1/etc/fstab */ # cat << EOF > /tmp/gm0s1/etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/mirror/gm0s1b none swap sw 0 0 /dev/mirror/gm0s1a / ufs rw 1 1 /dev/mirror/gm0s1e /tmp ufs rw,noatime,nosuid,nodev 2 2 /dev/mirror/gm0s1f /usr ufs rw 2 2 /dev/mirror/gm0s1d /var ufs rw,noexec 2 2 /dev/mirror/gm0s1g /home ufs rw,userquota,nosuid,nodev 2 2 /dev/acd0 /cdrom cd9660 ro,noauto 0 0 /dev/da0s1 /mnt/da0 ufs rw,noauto,nosuid,nodev 0 0 /dev/da1s1 /mnt/da1 msdosfs rw,noauto 0 0 EOF # echo geom_mirror_load=\"YES\" >> /tmp/gm0s1/boot/loader.conf # echo "1:ad(1,a)/boot/loader" > /boot.config
Unmount the second side of the mirror and reboot:
# umount /tmp/gm0s1/tmp # umount /tmp/gm0s1/home # umount /tmp/gm0s1/usr # umount /tmp/gm0s1/var # umount /tmp/gm0s1 # # sync && shutdown -r now
After rebooting on the second disk (the GEOMified one), switch the mirror to auto-synchronization and add the first disk, which is now immediately synchronized with the second disk's content:
# gmirror configure -a gm0 # gmirror insert gm0 /dev/ad8 # # gmirror list Geom name: gm0 State: DEGRADED Components: 2 Balance: round-robin Slice: 4096 Flags: NONE GenID: 0 SyncID: 1 ID: 770137303 Providers: 1. Name: mirror/gm0 Mediasize: 300090727936 (279G) Sectorsize: 512 Mode: r7w6e7 Consumers: 1. Name: ad10 Mediasize: 300090728448 (279G) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Priority: 0 Flags: NONE GenID: 0 SyncID: 1 ID: 2706535066 2. Name: ad8 Mediasize: 300090728448 (279G) Sectorsize: 512 Mode: r1w1e1 State: SYNCHRONIZING Priority: 0 Flags: DIRTY, SYNCHRONIZING GenID: 0 SyncID: 1 Synchronized: 71% ID: 2682952005 # # gmirror status Name Status Components mirror/gm0 DEGRADED ad10 ad8 (71%)
During all of these steps, some kernel messages may be seen on the
console or in the
/var/log/messages system logs file, as shown below:
# tail -f /var/log/messages GEOM_MIRROR: Device gm0: provider ad8 detected. GEOM_MIRROR: Device gm0: rebuilding provider ad8. GEOM_MIRROR: Device gm0: rebuilding provider ad8 finished. GEOM_MIRROR: Device gm0: provider ad8 activated.
Last, please find some invaluable documentation on the subject, with a special note for the BSD DevCenter one since, although less secure for the data than the others, bypassed the need for duplicating the data from one disk to the other, much less as found when using SVM on Solaris from Sun Microsystems.