blog'o thnet

Here is a little but annoying bug I faced recently on a production system running Solaris 10 11/06: a process can not be killed.

The KILL signal can't be ignored by a process. The associated handler must be the default--taking the signal into account, and honor it--and it is the case (the incriminated process id is 26632):

# psig 26632 | grep KILL
KILL    default

Nevertheless, the process seems not to stop as expected:

# echo $$
2913
#
# kill -s KILL 26632
#
# dtrace -n 'fbt::sigtoproc:entry /pid == 2913/ {trace(arg2); trace(execname);}'
dtrace: description 'fbt::sigtoproc:entry ' matched 1 probe
CPU     ID                    FUNCTION:NAME
 18  16387                  sigtoproc:entry                 9  tcsh
#
# ps -ef | awk '$2 ~ /26632/ {print $0}'
nonpriv 26632     1   0   Nov 13 ?         0:01 /usr/sbin/in.ftpd -a

This not the correct behavior since a pending signal is not honored:

# pflags 26632
26632:  /usr/sbin/in.ftpd -a
       data model = _ILP32  flags = ORPHAN|MSACCT|MSFORK
       sigpend = 0x00006100,0x00000000
/1:    flags = 0

More, the process seems in a non-coherent system state: it is now nearly impossible to trace it using, truss, the proctools,...

# truss -alef -p 26632
truss: unanticipated system error: 26632
#
# pstack 26632
pstack: cannot examine 26632: unanticipated system error
#
# pfiles 26632
pfiles: unanticipated system error: 26632
#
# pldd 26632
pldd: cannot examine 26632: unanticipated system error

... or helped by DTace:

# dtrace -n 'profile-1000hz /pid ==26632/ { @[stack()] = count() }'
dtrace: description 'profile-1000hz ' matched 1 probe
^C
#
# dtrace -n 'profile-1000hz /pid ==26632/ { @[ustack()] = count() }'
dtrace: description 'profile-1000hz ' matched 1 probe
^C

Follow some information gathered through mdb in kernel debugging mode:

> ::status
debugging live kernel (64-bit) on socrate
operating system: 5.10 Generic_118833-36 (sun4u)
>
> ::showrev
Hostname: socrate
Release: 5.10
Kernel architecture: sun4u
Application architecture: sparcv9
Kernel version: SunOS 5.10 sun4u Generic_118833-36
Platform: SUNW,Sun-Fire-V490
>
> ::pgrep in.ftpd
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R  26632      1    261    261  30501 0x5a024b00 00000600144b0c28 in.ftpd
>
> 00000600144b0c28::kill
mdb: command is not supported by current target
>
> 00000600144b0c28::thread
            ADDR    STATE    FLG PFLG SFLG   PRI  EPRI PIL             INTR
00000600144b0c28 inval/2000 157e 5778    0     0     0   6                2
>
> 00000600144b0c28::walk thread | ::findstack
stack pointer for thread 30002aba360: 2a1022e7ec1
[ 000002a1022e7ec1 cv_wait+0x38() ]
  000002a1022e7f71 page_lock_es+0x204()
  000002a1022e8021 pvn_vplist_dirty+0x2a4()
  000002a1022e8101 nfs_putpages+0x124()
  000002a1022e81c1 nfs3_putpage+0xcc()
  000002a1022e8271 fop_putpage+0x1c()
  000002a1022e8321 nfs_purge_caches+0xe4()
  000002a1022e83d1 nfs_attr_cache+0x20c()
  000002a1022e8481 nfs3_getattr_otw+0x1b8()
  000002a1022e85f1 nfs3_validate_caches+0x4c()
  000002a1022e8731 nfs3_getpage+0xa4()
  000002a1022e8861 fop_getpage+0x44()
  000002a1022e8931 segmap_getmapflt+0x588()
  000002a1022e8a41 snf_segmap+0x13c()
  000002a1022e8bc1 sosendfile64+0x298()
  000002a1022e8d21 sendvec64+0xf8()
  000002a1022e8f61 sendfilev+0x178()
  000002a1022e92e1 syscall_trap32+0xcc()

We can see the use of the sendfilev() syscall: it appears to be a known bug around this (you must have a registered Sun support customer account to be able to view the second document).

The good news is the root cause is already fix in the development branch of the operating system. The bad news is, after opening a call to the Sun support team, that no patch will be released in a near future to correct this: the incorporation of the fix is currently planned for the next major Solaris 10 Update (Update 5) which is scheduled for summer 2008. However, the fix is already available via the Solaris Express program, if that is applicable to your environment. As a last note, and if the non-killable process owns a resource necessary to another program, there seems no other option than to plan a reboot of the system.

After creating a new boot environment (BE) named beastie, see below, to upgrade a system running Solaris 8 to Solaris 10 11/06 (as I do many times in the past without a hiccup), I encounter a problem when I tried to apply the appropriate Recommended cluster patch to the new BE with this message:

# luupgrade -t -n beastie -s /var/tmp/10_Recommended

Validating the contents of the media .
The media contains 76 software patches that can be added.
All 76 patches will be added because you did not specify any specific
patches to add.
Mounting the BE .
ERROR: The boot environment  supports non-global
zones.The current boot environment does not support non-global zones.
Releases prior to Solaris 10 cannot be used to maintain Solaris 10 and
later releases that include support for non-global zones. You may only
execute the specified operation on a system with Solaris 10 (or later)
installed.

I can't find any reference to a known bug or problem after looking for this against SunSolve, Sun Support, and Googling. Has anyone already seen this error, and solved it The Right Way? As for me, I needed to boot from the BE and apply the cluster patch: this was a pain since this bundle include the -36 kernel patch which is known to be relatively disruptive, since it need two reboot to apply the entire cluster patch (it contains a new version for the kernel).

Once again, well known and great developer Casper Dik from Sun Microsystems give us an interesting answer on the fact that there is not 32-bit emulation in the Solaris OS when running a 64-bit kernel, and why most installed binaries seems to be compiled as 32-bit executables, even if you boot on a 64-bit platform, as in the following example:

# isainfo -kv
64-bit amd64 kernel modules
# file /usr/bin/ls
/usr/bin/ls:    ELF 32-bit LSB executable 80386 Version 1 [FPU], dynamically linked, \
 not stripped, no debugging information available

Here is the explanation, shown as a little Q&A:

Q: Once I boot it in 64bit mode, i'd have to run emulation libraries to run 32bit bins right?

A: No; you run the exact same binaries and libraries under 32 and 64 bit. It's not emulation; it's basically two syscall entry tables one for 32 bit and one for 64 bit, mostly sharing the same code except where pointer sizes matter.

Q: Since it will be a very light server (only bare binaries to run my req.s) they probably end up most likely all being 64bit?

A: No, most binaries are 32 bit only those that need to be 64 bit are 64 bit.

Q: so you are saying that i can run both 32bit and 64bit code simultaneously, natively with the solaris kernel? That's pretty damn cool

A: Correct. There are several reasons for an OS which generally comes in binary distributions to do so:

• maintain complete binary compatibility with old applications (and yes, closed source does matter, even to a lot of Linux customers).
• allows a single distribution to work on both 32 and 64 bit systems of the same architecture.

As a last note, it is interesting to see that a full install of SXCE snv_70 provide 1188 32-bit files and only 82 64-bit files in the following paths /bin, /usr/bin, /usr/openwin/bin, /usr/ucb, /usr/sfw/bin, and /opt/SUNWspro/bin. The 64-bit files are mostly stored in a specific architecture subdirectory; i.e. amd64 in this case, while some of them are provided as both 32-bit and 64-bit incarnations:

# file /usr/bin/pfiles /usr/bin/amd64/pfiles
/usr/bin/pfiles:        ELF 32-bit LSB executable 80386 Version 1 [FPU], \
 dynamically linked, not stripped, no debugging information available
/usr/bin/amd64/pfiles:  ELF 64-bit LSB executable AMD64 Version 1 [SSE FXSR FPU], \
 dynamically linked, not stripped, no debugging information available

Update #1: 2008-04-07

Be sure to consult the excellent 64-bit FAQ for Solaris.

The long awaited Solaris 10 8/07 release, a.k.a. Update 4 is now available, and ready to download. This release was postponed many times due to some concerns about better testing and validating advanced features such as the new Deferred-Activation Patching, or the support for the upcoming AMD quad-core next generation processor. Interestingly, this release re-introduce the Documentation DVD.

As always, you must consult the excellent What's New in the Solaris 10 8/07 Release web pages. The following points are those I prefer:

• ZFS and Solaris iSCSI improvements
• x86: Fault Management For Next Generation AMD Opteron Processors
• Solaris Live Upgrade: Upgrading the Solaris OS When Non-Global Zones Are Installed
• Deferred-Activation Patching
• Single Hosts File
• Support for iSCSI Target Devices
• Multi-level CMT Scheduling Optimizations

Last, it is interesting to mention that the release number corresponds to the build date of the release, not the general availability (GA)date. It is the case for the 8/07 release, made public at the beginning of September 2007. And it was the case in the past with the 11/06 release which was made publicly available at the beginning of December, although it was build during mid-November 2006:

# cat /etc/release
                       Solaris 10 11/06 s10s_u3wos_10 SPARC
           Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                           Assembled 14 November 2006

# cat /etc/release
                       Solaris 10 8/07 s10x_u4wos_12b X86
           Copyright 2007 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                           Assembled 16 August 2007

Update #1: 2007-09-10

I just forgot to update the preferred list with this particular point: the lx Branded Zones: Solaris Containers for Linux Applications. Let me explain why in a next blog entry.

Update #2: 2007-09-11

Sadly, the impressive work currently being done in the OpenSolaris community (and already available with the Solaris Express Community Edition) on the performance on NFS with ZFS problem--particularly the boot time problem--is not yet back-ported to the official release tree, including the use of evolution from sharemgr(1m) plus in-kernel sharetab(4) facility. I hope for the next Update.

You are a big corporate Sun customer? You are particularly interested in running the Solaris operating system on various and heterogeneous x64/x86 platforms including non-Sun servers (more and more hardware providers are certifying their servers to run the Solaris 10 OS these days)?

You are already aware that H-P can be a SPOC (Single Point Of Contact) for both their hardware and the support for the Red Hat Enterprise Linux system, and will enjoy the very same mechanism for the Solaris platform?

Well. It is now possible... and a little more in fact: this quiet the opposite. I mean, you can ask Sun to be the SPOC for running Solaris on Sun and non-Sun systems. I feel strongly having Sun to be the very first entry point is a must, in particular since they know really well (today) what can be done on Solaris on x64/x86 platforms, seeing the rise of work which is currently achieve through the OpenSolaris project. Not only Solaris (and its open source counterparts distributions) is becoming more and more relevant in the x64/x86 AMD/Intel world, but they are leading innovations on these platforms.

So, enjoy the not-so-recent announcement about the new SunSpectrum Enterprise Service Plan for enterprise customers.