For some months, I used to encrypt the SWAP
(which is a ZFS volume) and thus have an encrypted
/tmp. This worked
fine with Solaris 11 Express, but I encountered a strange behavior in
Solaris 11 EA which leads to have the SWAP device to... well, just
Here is what I found after two boots; and on several machines:
# swap -l No swap devices configured # zfs list -t volume NAME USED AVAIL REFER MOUNTPOINT rpool/dump 32.8G 240G 31.8G - # grep swap /etc/vfstab swap - /tmp tmpfs - yes - /dev/zvol/dsk/rpool/swap - - swap - no encrypted
rpool/swap dataset disappeared. I am sure not to have
destroyed it, in particular since this appears on multiple servers.
Nevertheless, I found this in the history of the
# zpool history | grep destroy [...] 2011-10-05.10:22:49 zfs destroy rpool/swap # last reboot | head -2 reboot system boot Wed Oct 5 10:23 reboot system down Wed Oct 5 10:20
So, this problem seems to be related to some actions at boot time. What have the logs of SMF services to say about that?
# find /var/svc/log -print | xargs grep -i swap /var/svc/log/system-filesystem-usr:default.log:cannot create 'rpool/swap': pool must be upgraded to set this property or value /var/svc/log/system-filesystem-usr:default.log:cannot open 'rpool/swap': dataset does not exist /var/svc/log/system-filesystem-usr:default.log:cannot create 'rpool/swap': pool must be upgraded to set this property or value /var/svc/log/system-filesystem-usr:default.log:cannot open 'rpool/swap': dataset does not exist # tail -3 /var/svc/log/system-filesystem-usr:default.log [ Oct 5 12:00:05 Executing start method ("/lib/svc/method/fs-usr"). ] cannot create 'rpool/swap': pool must be upgraded to set this property or value [ Oct 5 12:00:13 Method "start" exited with status 0. ]
Ouch, what happened here? The message is interesting, but is a little misleading: it is on fresh Solaris 11 EA installations, and so the pools and datasets are all up to date:
# zpool upgrade && zfs upgrade This system is currently running ZFS pool version 33. All pools are formatted using this version. This system is currently running ZFS filesystem version 5. All filesystems are formatted with the current version.
So, it seems that the
rpool/swap device is re-created at boot time,
and for some reason it doesn't work as expected. Here is an attempt to
discover where the device is re-created and why it does fail.
# find /lib/svc/method -print | xargs grep -i sbin/swapadd /lib/svc/method/fs-usr:/usr/sbin/swapadd -1 /lib/svc/method/nfs-client: /usr/sbin/swapadd /lib/svc/method/fs-local:/usr/sbin/swapadd >/dev/null 2>&1 # grep "zfs destroy" /usr/sbin/swapadd zfs destroy $zvol > /dev/null 2>&1 # sed -n '/zfs create/,/\$zvol/p' /usr/sbin/swapadd zfs create -V $volsize -o volblocksize=`/usr/bin/pagesize` \ -o primarycache=$primarycache -o secondarycache=$secondarycache \ -o encryption=$encryption -o keysource=raw,file:///dev/random $zvol
So, the re-creation at boot time of the
rpool/swap appears only when
using an encrypted volume. And after a bit of digging, here what I
found. At the first boot, here is the command used to create the
zfs create -V 4G -o volblocksize=8192 -o primarycache=metadata -o secondarycache=all -o encryption=on -o keysource=raw,file:///dev/random rpool/swap
But on a second boot, here is the slightly different command used this time:
zfs create -V 4G -o volblocksize=8192 -o primarycache=metadata -o secondarycache=all -o encryption=aes-128-ctr -o keysource=raw,file:///dev/random rpool/swap
This is because the arguments passed to the command is backed-up and
restored from the settings just before the deletion of the volume. As
mentioned in the
zfs(1m) manual page, only the following encryption
algorithm are supported... and so the one which is sets is not valid
(the error message saying that the pool must be upgraded to set this
property or value is a little more clear by now).
encryption=off | on | aes-128-ccm | aes-192-ccm | aes-256-ccm | aes-128-gcm | aes-192-gcm | aes-256-gcm
The question is, how can this happen? Where does this algorithm com
from? The answer is simple: it seems that this is the
which alters some properties of the
# swap -d /dev/zvol/dsk/rpool/swap # zfs destroy rpool/swap # zfs create -V 4G -o volblocksize=8192 -o primarycache=metadata -o secondarycache=all -o encryption=on -o keysource=raw,file:///dev/random rpool/swap # zfs list -H -o type,volsize,volblocksize,encryption rpool/swap volume 4G 8K on # swap -1 -a /dev/zvol/dsk/rpool/swap # zfs list -H -o type,volsize,volblocksize,encryption rpool/swap volume 4G 1M aes-128-ctr
Not only is the algorithm changed to something not supported (yet?), but
volblocksize property is touched as well. This was not the case on
Solaris 11 Express 2010.11.
Hope someone can help me on this side, and that this is a known bug which is already (or will be quickly) addressed, in particular for the Solaris 11 GA. I already posted a comment on the blog of Darren Moffat, just in case this can help a bit.