Here is a little but annoying bug I faced recently on a production system running Solaris 10 11/06: a process can not be killed.
The KILL signal can't be ignored by a process. The associated handler
must be the default--taking the signal into account, and honor it--and
it is the case (the incriminated process id is 26632):
# psig 26632 | grep KILL
KILL default
Nevertheless, the process seems not to stop as expected:
# echo $$
2913
#
# kill -s KILL 26632
#
# dtrace -n 'fbt::sigtoproc:entry /pid == 2913/ {trace(arg2); trace(execname);}'
dtrace: description 'fbt::sigtoproc:entry ' matched 1 probe
CPU ID FUNCTION:NAME
18 16387 sigtoproc:entry 9 tcsh
#
# ps -ef | awk '$2 ~ /26632/ {print $0}'
nonpriv 26632 1 0 Nov 13 ? 0:01 /usr/sbin/in.ftpd -a
This not the correct behavior since a pending signal is not honored:
# pflags 26632
26632: /usr/sbin/in.ftpd -a
data model = _ILP32 flags = ORPHAN|MSACCT|MSFORK
sigpend = 0x00006100,0x00000000
/1: flags = 0
More, the process seems in a non-coherent system state: it is now nearly
impossible to trace it using, truss, the proctools,...
# truss -alef -p 26632
truss: unanticipated system error: 26632
#
# pstack 26632
pstack: cannot examine 26632: unanticipated system error
#
# pfiles 26632
pfiles: unanticipated system error: 26632
#
# pldd 26632
pldd: cannot examine 26632: unanticipated system error
... or helped by DTace:
# dtrace -n 'profile-1000hz /pid ==26632/ { @[stack()] = count() }'
dtrace: description 'profile-1000hz ' matched 1 probe
^C
#
# dtrace -n 'profile-1000hz /pid ==26632/ { @[ustack()] = count() }'
dtrace: description 'profile-1000hz ' matched 1 probe
^C
Follow some information gathered through mdb in kernel debugging mode:
> ::status
debugging live kernel (64-bit) on socrate
operating system: 5.10 Generic_118833-36 (sun4u)
>
> ::showrev
Hostname: socrate
Release: 5.10
Kernel architecture: sun4u
Application architecture: sparcv9
Kernel version: SunOS 5.10 sun4u Generic_118833-36
Platform: SUNW,Sun-Fire-V490
>
> ::pgrep in.ftpd
S PID PPID PGID SID UID FLAGS ADDR NAME
R 26632 1 261 261 30501 0x5a024b00 00000600144b0c28 in.ftpd
>
> 00000600144b0c28::kill
mdb: command is not supported by current target
>
> 00000600144b0c28::thread
ADDR STATE FLG PFLG SFLG PRI EPRI PIL INTR
00000600144b0c28 inval/2000 157e 5778 0 0 0 6 2
>
> 00000600144b0c28::walk thread | ::findstack
stack pointer for thread 30002aba360: 2a1022e7ec1
[ 000002a1022e7ec1 cv_wait+0x38() ]
000002a1022e7f71 page_lock_es+0x204()
000002a1022e8021 pvn_vplist_dirty+0x2a4()
000002a1022e8101 nfs_putpages+0x124()
000002a1022e81c1 nfs3_putpage+0xcc()
000002a1022e8271 fop_putpage+0x1c()
000002a1022e8321 nfs_purge_caches+0xe4()
000002a1022e83d1 nfs_attr_cache+0x20c()
000002a1022e8481 nfs3_getattr_otw+0x1b8()
000002a1022e85f1 nfs3_validate_caches+0x4c()
000002a1022e8731 nfs3_getpage+0xa4()
000002a1022e8861 fop_getpage+0x44()
000002a1022e8931 segmap_getmapflt+0x588()
000002a1022e8a41 snf_segmap+0x13c()
000002a1022e8bc1 sosendfile64+0x298()
000002a1022e8d21 sendvec64+0xf8()
000002a1022e8f61 sendfilev+0x178()
000002a1022e92e1 syscall_trap32+0xcc()
We can see the use of the sendfilev() syscall: it appears to be a
known
bug
around
this
(you must have a registered Sun support customer account to be able to
view the second document).
The good news is the root cause is already fix in the development branch of the operating system. The bad news is, after opening a call to the Sun support team, that no patch will be released in a near future to correct this: the incorporation of the fix is currently planned for the next major Solaris 10 Update (Update 5) which is scheduled for summer 2008. However, the fix is already available via the Solaris Express program, if that is applicable to your environment. As a last note, and if the non-killable process owns a resource necessary to another program, there seems no other option than to plan a reboot of the system.