blog'o thnet

If you are not able to start the Oracle Cluster Synchronization Services Daemon (OCCSD) in a non-global zone on Solaris 10, I bet you are running Oracle 10.2.0.3 and higher. In this case, you will see something similar in the the /var/adm/messages file--but nothing is coming up:

Apr 26 10:39:51 zonename oracle: [ID 702911 user.error] Oracle Cluster Synchronization Service starting by user request.
Apr 26 10:39:52 zonename root: [ID 702911 user.error] Cluster Ready Services completed waiting on dependencies.

Trying to trace the ocssd.bin process during start-up give you something similar to:

[...]
12564:   0.0803 setrlimit(RLIMIT_CORE, 0xFFFFFFFF7FFFF900)      = 0
12564:   0.0804 priocntlsys(1, 0xFFFFFFFF7FFFF694, 6, 0xFFFFFFFF7FFFF768, 0) Err#1 EPERM [proc_priocntl]
12564:   0.0810 fstat(2, 0xFFFFFFFF7FFFE870)                    = 0
12564:   0.0811 brk(0x100229E80)                                = 0
12564:   0.0813 brk(0x10022DE80)                                = 0
12564:   0.0815 fstat(2, 0xFFFFFFFF7FFFE740)                    = 0
12564:   0.0816 ioctl(2, TCGETA, 0xFFFFFFFF7FFFE7AC)            Err#25 ENOTTY
12564:   0.0818 write(2, " s e t p r i o r i t y :".., 52)      = 52
12564:   0.0821 _exit(100)

So, in this case you just hit a privilege restriction, which did not apply before with older release of Oracle. As clearly mentioned in the output of truss, the proc_priocntl is not available in the non-global zone for use by Oracle. A clean solution, available only with Solaris 10 11/06 (U3) and later, is to use the limitpriv configuration property to extend the basic privileges provided by the zone framework.

As stated in the privileges(5) man page:

PRIV_PROC_PRIOCNTL Allow a process to elevate its priority above its current level. Allow a process to change its scheduling class to any scheduling class, including the RT class.

Interestingly, this seems to be exactly the case for the Oracle Cluster Synchronization Services Daemon:

# zonecfg -z zonename set limitpriv=default,proc_priocntl
# zoneadm -z zonename reboot
# zlogin zonename "ps -o class,args -p `pgrep ocssd.bin`"
 CLS COMMAND
  RT /soft/oracle/10.2.0/asm_1/bin/ocssd.bin

Ok, that's fine right now.

As a little follow-up to Fake The hostid Of A Solaris Zone, and regarding the discussion on the capacity to change the hostid of a Solaris non-global zone, it is interesting to mention these (updated) informations:

1. The LD_PRELOAD trick proposed before is not a proper option, and is really ugly (and intrusive if you didn't unset it before continuing the execution of a program).
2. When using Solaris 8 or Solaris 9 Containers, there is a feature called Host ID Emulation from the zonecfg utility which can do exactly that.
3. Before the introduction of the privileges in a non-global zone with Solaris 11/06 (a.k.a. Solaris Update 3), you must run the DTrace zhostid script (daemon) within the global zone. It is not mandatory to run it from the global zone anymore. Using the appropriate dtrace_user privilege only, you can run it directly from the non-global zone:

   # zonecfg -z ngzone set limitpriv=default,dtrace_user
   # zoneadm -z ngzone boot
   # zlogin ngzone
   [Connected to zone 'ngzone' pts/5]
   Last login: Sat Oct  4 18:57:17 on pts/5
   Sun Microsystems Inc.   SunOS 5.11      snv_99  November 2008
   # /sbin/zonename 
   ngzone
   # /usr/bin/hostid
   837d47dd
   # ./zhostid &
   [1] 21506
   # /usr/bin/hostid
   20a82f32
   # ^D
   [Connection to zone 'ngzone' pts/5 closed]
   

Although not very versed in BrandZ technologies, I tried the lx brand in the past, with success. But the fact is, the just supported release of the lx brand in the Solaris 8/07 release will not really help us, since it can only provide a 2.4 Linux kernel working environment which is now a bit outdated (yes, I know the most recent Linux distributions will be supported via the Xen open-source hypervisor ). Interestingly, two recent news will certainly (hopefully?) change this in the future.

The first one is the availability of an experimental Linux 2.6 support in OpenSolaris. The work has been done by a summer intern (!) in the Solaris kernel group, Evan Hoke. What a hard work done in so little time, really impressive. I just hope that the community will follow and will enhance the proposed experimental branded zone extension.

The second news is really exciting too. This is the upcoming availability of a SPARC-only brand designed to emulate the Solaris 8 kernel. Although not yet ready for use, this news is really, really amazing since this enable a more soft-upgrade to Solaris 10 by enabling the use of all the great features provided to us with this version (DTrace, ZFS, FMA, SMF, etc.) on the global zone, while running a Solaris 8 branded zone... on top of the most recent hardware (say Sun Fire V215, T1000, T2000, etc.). Wow. I just can't wait to try this out, in particular since it will be available for the last supported Solaris 8/07 release.

The aim of this little test case is to present new features of zones, and its tightly integration with the ZFS powerful file system. In order to do so, we will:

1. Create a specific ZFS namespace for non-global zone(s).
2. Create a new zone.
3. Move it to a new zonepath.
4. Configure it automatically, using a sysidcfg file.
5. Clone it.
6. Duplicate the first zone using the clone.

Be sure to have a valid hostname and IP address for the two non-global zones:

# getent hosts | egrep 'beastie|watchie'
192.168.1.1   beastie.thilelli.net beastie
192.168.1.2   watchie.thilelli.net watchie

Create a valid ZFS dedicated namespace:

# zfs create -o compression=on \
   -o mountpoint=/export/zone \
   -o canmount=off pool0/zone
# zfs list -r pool0/zone
NAME         USED  AVAIL  REFER  MOUNTPOINT
pool0/zone  24.5K   228G  24.5K  /export/zone
# zfs get compression,mountpoint,canmount pool0/zone
NAME        PROPERTY     VALUE         SOURCE
pool0/zone  compression  on            local
pool0/zone  mountpoint   /export/zone  local
pool0/zone  canmount     off           local
# zfs mount | grep zone
#

Note: Since build 48 of Nevada, some new ZFS features were added. Create time properties and canmount property are two of them. Please refer to this excellent blog entry from Eric Schrock's weblog for more information on these putbacks.

Configure the different zone's informations:

# zonecfg -z beastie 'create; set autoboot=true; \
   set zonepath=/export/zone/badbeastie; add net; \
   set address=192.168.1.1/24; set physical=nge0; \
   end; verify; commit; exit'
# zonecfg -z watchie 'create; set autoboot=true; \
   set zonepath=/export/zone/watchie; add net; \
   set address=192.168.1.2/24; set physical=nge0; \
   end; verify; commit; exit'
# zoneadm list -vc
  ID NAME             STATUS         PATH
   0 global           running        /
   - beastie          configured     /export/zone/badbeastie
   - watchie          configured     /export/zone/watchie

Then, fire the zoneadm command:

# zoneadm -z beastie install
A ZFS file system has been created for this zone.
Preparing to install zone <beastie>.
Creating list of files to copy from the global zone.
[...]

Instead of configure it manually at first boot, create a configuration file which will do this task automatically for you, and start the zone:

# cat << EOF > /export/zone/badbeastie/root/etc/sysidcfg
system_locale=C
timezone=Europe/Paris
terminal=vt100
security_policy=NONE
root_password=xxxxxxxxxxxxx
timeserver=localhost
name_service=NONE
nfs4_domain=dynamic
network_interface=primary {
  hostname=beastie.thilelli.net
  ip_address=192.168.1.1
  netmask=255.255.255.0
  protocol_ipv6=no
  default_route=192.168.1.254
}
EOF
#
# zoneadm -z beastie boot && zlogin -C beastie
[Connected to zone 'beastie' console]
SunOS Release 5.11 Version snv_48 64-bit
Copyright 1983-2006 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: beastie
Loading smf(5) service descriptions: 119/119
Oct  4 04:30:22 svc.startd[3003]: svc:/system/dbus:default:
 Method "/lib/svc/method/svc-dbus start" failed with exit status 95.
Oct  4 04:30:22 svc.startd[3003]: system/dbus:default failed fatally:
 transitioned to maintenance (see 'svcs -xv' for details)
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair
rebooting system due to change(s) in /etc/default/init

[NOTICE: Zone rebooting]
SunOS Release 5.11 Version snv_48 64-bit
Copyright 1983-2006 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: beastie.thilelli.net
Oct  4 13:30:40 svc.startd[3757]: svc:/system/dbus:default:
 Method "/lib/svc/method/svc-dbus start" failed with exit status 95.
Oct  4 13:30:40 svc.startd[3757]: system/dbus:default failed fatally:
 transitioned to maintenance (see 'svcs -xv' for details)

beastie.thilelli.net console login: ~.
[Connection to zone 'beastie' console closed]

Well done. Now, change the zone path to something more appropriate. Eventually, adapt the ZFS name accordingly:

# zoneadm -z beastie halt
# zoneadm -z beastie move /export/zone/beastie
# zoneadm list -vc
  ID NAME             STATUS         PATH
   0 global           running        /
   - beastie          installed      /export/zone/beastie
   - watchie          configured     /export/zone/watchie
#
# zfs list -r pool0/zone
NAME                    USED  AVAIL  REFER  MOUNTPOINT
pool0/zone              248M   227G  24.5K  /export/zone
pool0/zone/badbeastie   248M   227G   248M  /export/zone/beastie
# zfs rename pool0/zone/badbeastie pool0/zone/beastie
# zfs list -r pool0/zone
NAME                 USED  AVAIL  REFER  MOUNTPOINT
pool0/zone           248M   227G  24.5K  /export/zone
pool0/zone/beastie   248M   227G   248M  /export/zone/beastie
#
# zoneadm -z beastie boot

Wow... Very interesting feature, isn't it?!

Now, lets try the cloning feature bundle with the new zoneadm command. Do some specific non-global zone tuning before; then do:

# zlogin beastie svcadm disable system/dbus
# zoneadm -z beastie halt
# zoneadm -z watchie clone beastie
Cloning snapshot pool0/zone/beastie@SUNWzone1
Instead of copying, a ZFS clone has been created for this zone.
# zoneadm list -vc
  ID NAME             STATUS         PATH
   0 global           running        /
   - beastie          installed      /export/zone/beastie
   - watchie          installed      /export/zone/watchie
#
# zfs list -r pool0/zone/beastie
NAME                           USED  AVAIL  REFER  MOUNTPOINT
pool0/zone/beastie             251M   227G   248M  /export/zone/beastie
pool0/zone/beastie@SUNWzone1  3.77M      -   248M  -
#
# sed -e 's/beastie/watchie/' \
   -e 's/ip_address=192.168.1.1/ip_address=192.168.1.2/'
   /export/zone/beastie/root/etc/sysidcfg > \
   /export/zone/watchie/root/etc/sysidcfg

You can now enjoy the first boot of the newly created zone:

# zoneadm -z watchie boot && zlogin -C watchie 
[Connected to zone 'watchie' console]
Hostname: watchie
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair

watchie.thilelli.net console login: ~.
[Connection to zone 'watchie' console closed]

Awesome features and technologies i think! Really.

Last, note that Ben Rockwood already has a well written blog entry on this very same subject... he was the first to publish it though ;)

Note: It seems there is a little bug in the snv_48 SX:CR release which prevents the expected automatically ZFS file system creation or cloning from happening properly; the action fails with an error similar to this one:

cannot create ZFS dataset <zfs_name>: 'sharenfs' must be a string

This bug is already closed and fixed, and will be available in the next Solaris Express, see Bug ID: 6468554 for more information on this one.