blog'o thnet

Oracle On The Future Of OpenSolaris, Finally

Julien Gabel — Sun, 28 Feb 2010 14:51:00 +0100

After the official and long-awaited public information from Oracle on the merge with Sun Microsystems, some great news came on both hardware and software portfolios, in particular the x86 ans SPARC ecosystems, and around the Solaris operating system.

The main unknown was about the OpenSolaris community, and distribution model. And until very recently, some important voices around the community stayed without answer, in particular from Ben Rockwood, or Peter Tribble.

Well, until recently. In fact, the OpenSolaris Annual Meeting (held on IRC through the #opensolaris-meeting canal last 26 February) brought some answers very shortly, which currently begin to spread through the community. I hope this will quiet some recent misunderstanding on the support model of the OpenSolaris distribution.

Oracle Commitment To Sun Technologies

Julien Gabel — Tue, 01 Dec 2009 12:07:00 +0100

Here are more information about Oracle commitment to Sun business, and the Solaris operating system in particular:

There is the Oracle Metalink #742060.1 on the new position of the Solaris 10 x86 platform which is raised up at a higher level than before.
The licensing mode for the UltraSPARC T2+ is more favorable than for other processor chips, such as IBM POWER5+ and POWER6 notably.
The Database Smart Flash Cache technology from the Exadata V2 Database Machine can only be used with Oracle Database 11g Release 2 on Solaris or Oracle Enterprise Linux operating systems only.

Lastly, Oracle and Sun Overview and FAQ for customers and partners is a must read for all persons interested in the future and Oracle's investment in all the current technologies from Sun Microsystems.

Oracle Database 11g Release 2 For Solaris

Julien Gabel — Thu, 26 Nov 2009 14:41:00 +0100

As a follow-up to the preceding entry on the upcoming availability on the Oracle Database 11g Release 2 on both the Solaris SPARC and x86 releases, we can see that this is a reality as of today. Both architectures are readily available for download.

As a system administrator I think this very interesting and encouraging, not only because of the availability of one of the more robust RDBMS system on Solaris platforms, but because this is some actions taken after words from Oracle which seems to fit together. And so, the interest in Solaris as an OS of choice is more reinforced now.

Upcoming Oracle RDBMS And Solaris News

Julien Gabel — Wed, 07 Oct 2009 20:12:00 +0200

Following the recent news about the future of Sun from Larry Ellison itself, we now can hope about more things to come on Solaris, both on SPARC and x86 platforms. In particular, this quote is particularly encouraging:

We're a big supporter of Linux, but the fact is that Solaris just a much more mature OS, its just a fact. We became a big supporter of Linux years ago because it ran on smaller and cheaper X86 processors and Solaris did not, we had no choice. [...] So we are a supporter of Linux, but Solaris is a more mature operating system designed for bigger systems. We support both.

In the very same vein, I just heard from two different sources these upcoming changes from Oracle:

The just release Oracle Database 11g Release 2, currently available only for Linux (and released on 1 September 2009), will be available soon--one to two months--for both Solaris SPARC and x86, at the same time.
Secondly, Solaris x86 will be raised to Tier 2 platform from Tier 3 currently.

Well, pretty good news in fact! Seems that Solaris will be a serious and growing competitor in the (near) future!

Finding The Process Responsible For Crashing A System

Julien Gabel — Sun, 22 Mar 2009 15:27:00 +0100

Recently, we encounter a wave of suicide on most of the nodes which formed some Oracle RAC cluster on lots of Sun M5000 domains platforms. Although the logs found on Oracle RAC were interesting, they didn't help us to determine precisely the origin of the crashes. Since the domains panic'ed, we were able to briefly analyze the cores generated at crash time to get the process which initiated the panics. Here is how to do so.

First, be sure to have proper and usable core on persistent storage:

# cd /var/crash/nodename
# file *.0
unix.0:         ELF 64-bit MSB executable SPARCV9 Version 1, UltraSPARC1 Extensions Required, statically linked, not stripped, no debugging information available
vmcore.0:       SunOS 5.10 Generic_127111-11 64-bit SPARC crash dump from 'nodename'

Then, extract useful information using MDB dcmds such as ::status, ::showrev and ::panicinfo which give us the exact panic message and provide us the message and thread responsible for the system crash:

# mdb -k unix.0 vmcore.0
Loading modules: [ unix krtld genunix specfs dtrace ufs sd mpt px ssd fcp fctl md ip qlc hook neti sctp arp usba nca zfs random logindmux ptm cpc sppp crypto wrsmd fcip nfs ipc ]
> ::status
debugging crash dump vmcore.0 (64-bit) from nodename
operating system: 5.10 Generic_127111-11 (sun4u)
panic message: forced crash dump initiated at user request
dump content: kernel pages only
> ::showrev
Hostname: nodename
Release: 5.10
Kernel architecture: sun4u
Application architecture: sparcv9
Kernel version: SunOS 5.10 sun4u Generic_127111-11
Platform: SUNW,SPARC-Enterprise
> ::panicinfo
             cpu                0
          thread      300171c7300
         message forced crash dump initiated at user request
          tstate       4400001606
              g1                b
              g2                0
              g3          11c13e0
              g4              6e0
              g5         88000000
              g6                0
              g7      300171c7300
              o0          1208020
              o1      2a10176b9e8
              o2                1
              o3                0
              o4 fffffffffffffff5
              o5             1000
              o6      2a10176b0b1
              o7          10626a4
              pc          1044d8c
             npc          1044d90
               y                0

Well. Now what we have the exact thread number (thread ID), we can find the corresponding UNIX process helped by the following script:

# cat /var/tmp/findstack.vmcore.sh
#!/usr/bin/env sh

echo "::ps" | mdb -k unix.0 vmcore.0 | \
 nawk '$8 !~ /ADDR/ {print $8" "$NF}' > /tmp/.core.$$

cat /dev/null > /tmp/core.$$

while read ps; do
  echo "process name: `echo ${ps} | nawk '{print $2}'`" >> /tmp/core.$$
  echo ${ps} | nawk '{print $1"::walk thread | ::findstack"}' | \
   mdb unix.0 vmcore.0 >> /tmp/core.$$
  echo >> /tmp/core.$$
done < /tmp/.core.$$

\rm /tmp/.core.$$

exit 0

Now, just find the lines for the guilty process in the output file. In our case, it is the oprocd.bin process:

# vi /tmp/core.*
[...]
process name: oprocd.bin
stack pointer for thread 300171c7300: 2a10176b0b1
  000002a10176b161 kadmin+0x4a4()
  000002a10176b221 uadmin+0x11c()
  000002a10176b2e1 syscall_trap+0xac()
[...]

This process is locked in memory to monitor the cluster and provide I/O fencing. oprocd.bin performs its check, stops running, and if the wake up is beyond the expected time, then it resets the processor and reboots the node. An oprocd.bin failure results in Oracle Clusterware restarting the node. Please read the Oracle Clusterware and Oracle Real Application Clusters documentation for more information.

Although the incident is always under investigation, it seems the nodes were impacted by the additional second that was added at the end of 2008...

Problem Querying ru. Name Servers

Julien Gabel — Wed, 25 Feb 2009 19:21:00 +0100

In a previous position, I encounter a rather strange problem I would like to share here. It has to do with DNS resolution. From the internal network of a company it was not possible to get the IP address of the www.banks2ifrs.ru. host name.

We determined that the name servers which manage the banks2ifrs.ru. domain are ns3.nic.ru. and ns4.nic.ru.. In the same time, we saw that for the fbk.ru. domain name, the managing name servers are ns3.nic.ru. and gw.fbk.ru.. Interestingly, it was possible to resolve www.fbk.ru. using gw.fbk.ru., but not from ns3.nic.ru.. More, we noted that the reverse resolution for the name servers ns3.nic.ru. and ns4.nic.ru. are not correct, translated to ns3.ripn.net. and ns4.ripn.net., respectively. So, it is worth to mention that nothing was wrong when querying the name server gw.fbk.ru..

# host -t a ns3.nic.ru.
ns3.nic.ru has address 194.85.61.20
# host -t a ns4.nic.ru.
ns4.nic.ru has address 194.226.96.8
# host -t ptr 194.85.61.20
20.61.85.194.in-addr.arpa domain name pointer ns3.ripn.net.
# host -t ptr 194.226.96.8
8.96.226.194.in-addr.arpa domain name pointer ns4.ripn.net.

The error from the DNS query seems to be related to an incomplete answer from the server (the truncated flag was set to 1 in the network trace) when the query is made over UDP. In this case, an automatic fallback over TCP must be used, certainly prohibited from the company's network security policy. This may say that the answer is larger than 512 bytes long, too. So, we tried to advertise different sizes of the UDP message buffer, but without being confident that this message went through network devices properly. Nonetheless it would seem curious to get an answer larger that 120 bytes long.

Last, we can note that the complexity of the network layout (DMZ, firewalls, NAT, etc.) may badly interact and hamper DNS queries, at least in certain circumstances.

After more investigation from the network team, they decided to permit TCP DNS queries. And it worked. It worked letting the internal DNS servers doing their job themselves...

# dig +trace -t a www.banks2ifrs.ru.
; <<>> DiG 9.3.4-P1 <<>> +trace www.banks2ifrs.ru.
;; global options:  printcmd
.                       449798  IN   NS   L.ROOT-SERVERS.NET.
.                       449798  IN   NS   M.ROOT-SERVERS.NET.
.                       449798  IN   NS   A.ROOT-SERVERS.NET.
.                       449798  IN   NS   B.ROOT-SERVERS.NET.
.                       449798  IN   NS   C.ROOT-SERVERS.NET.
.                       449798  IN   NS   D.ROOT-SERVERS.NET.
.                       449798  IN   NS   E.ROOT-SERVERS.NET.
.                       449798  IN   NS   F.ROOT-SERVERS.NET.
.                       449798  IN   NS   G.ROOT-SERVERS.NET.
.                       449798  IN   NS   H.ROOT-SERVERS.NET.
.                       449798  IN   NS   I.ROOT-SERVERS.NET.
.                       449798  IN   NS   J.ROOT-SERVERS.NET.
.                       449798  IN   NS   K.ROOT-SERVERS.NET.
;; Received 512 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms

ru.                   172800  IN   NS   ns.ripn.net.
ru.                   172800  IN   NS   ns2.nic.fr.
ru.                   172800  IN   NS   ns2.ripn.net.
ru.                   172800  IN   NS   ns5.msk-ix.net.
ru.                   172800  IN   NS   ns9.ripn.net.
ru.                   172800  IN   NS   sunic.sunet.se.
;; Received 297 bytes from 199.7.83.42#53(L.ROOT-SERVERS.NET) in 125 ms

banks2ifrs.ru.   345600  IN   NS   ns4.nic.ru.
banks2ifrs.ru.   345600  IN   NS   ns3.nic.ru.
;; Received 107 bytes from 194.85.105.17#53(ns.ripn.net) in 66 ms

www.banks2ifrs.ru.   86400   IN   A     83.222.6.194
banks2ifrs.ru.            86400   IN   NS   ns4.nic.ru.
banks2ifrs.ru.            86400   IN   NS   ns3.nic.ru.
;; Received 91 bytes from 194.226.96.8#53(ns4.nic.ru) in 65 ms

... and it worked when querying directly the name servers responsible for the wanted domain:

# dig @ns4.nic.ru. -t a www.banks2ifrs.ru.
; <<>> DiG 9.3.4-P1 <<>> @ns4.nic.ru. -t a www.banks2ifrs.ru.
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1530
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 0

;; QUESTION SECTION:
;www.banks2ifrs.ru.             IN   A

;; ANSWER SECTION:
www.banks2ifrs.ru.      86400   IN   A   83.222.6.194

;; AUTHORITY SECTION:
banks2ifrs.ru.          86400   IN   NS   ns4.nic.ru.
banks2ifrs.ru.          86400   IN   NS   ns3.nic.ru.

;; Query time: 65 msec
;; SERVER: 194.226.96.8#53(194.226.96.8)
;; WHEN: Tue Apr 15 21:05:12 2008
;; MSG SIZE  rcvd: 91

Note: The size of the answer is 91 bytes long, so nothing wrong from this side.

I think we will never know what was going wrong here, even if the heart of the problem seems related specifically only to the two same name servers.

Quick Reference Guides For Network Technologies

Julien Gabel — Mon, 05 Jan 2009 08:06:00 +0100

I am always looking for some good IT cheat sheets, and for a lots of thing. Here is one of my favorite web site around network technologies. Interestingly, this site propose some regularly updated quick reference guides for the network, and are classified under the following categories (at the time of this writing):

Protocols
Applications
Reference
Syntax
Technologies
Miscellaneous

You can find them all at the Cheat Sheets Library section.

As the author wrote on his blog, if you notice an error or would like to see a new cheat sheet on a specific topic, don't hesitate and drop him a line.

More News To Come About Shrinking A zpool

Julien Gabel — Thu, 25 Dec 2008 19:14:00 +0100

As a little update to an older post on this subject, and although this post from Matthew Ahrens is about the new scrub code recently introduced in OpenSolaris build 94--and was in fact a priority before the launch of the Sun Storage 7000 Unified Storage Systems (a.k.a. Amber Road)--it is interesting to note that some of the new code will be usable to remove a disk from a ZFS pool.

As Matthew wrote:

This work lays a bunch of infrastructure that will be used by the upcoming device removal feature.

GRUB Boot Archive With SVM, A Better Approach

Julien Gabel — Wed, 03 Dec 2008 10:25:00 +0100

In a previous discussion about the GRUB boot archive and how it can be regenerated in Failsafe mode, I mentioned that it will not be as easy as it can be when the root file system use the md driver. I previously show a method to do this which necessitate to unmirror one or more file systems when the root file system is build upon a SVM mirror. This was not very optimal since a lot of of manipulations are involved, which may lead to human error(s), and may seems to be a little complicated.

This method was build on Performing System Recovery from the Solaris Volume Manager official documentation, which show up last month on the Sun-Managers mailing list.

Note: Although this test case was done using Solaris 10 10/08 under a virtual machine build upon VirtualBox on latest OpenSolaris release, the instructions must be valid for Solaris 10 1/06 and later.

Initial setup

As we saw before, the system use only a root file system, and a swap device. Both are encapsulated with SVM:

# df -k -F ufs
Filesystem     kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d0 6147798 3455578 2630743      57%  /
# swap -l
swapfile             dev  swaplo blocks   free
/dev/md/dsk/d1      85,1       8 4194288 4194288
# metastat -c d0 d1
d0               m  6.0GB d10 d20
    d10          s  6.0GB c0d0s0
    d20          s  6.0GB c1d1s0
d1               m  2.0GB d11 d21
    d11          s  2.0GB c0d0s1
    d21          s  2.0GB c1d1s1

Regenerate the GRUB boot archive

The idea is to boot on the GRUB Failsafe mode, get the md configuration from local root file system, and load manually the md module, hence properly configured. The main advantage is to be fully self hosted from the Failsafe mode, and not have to manipulate SVM more than necessary, especially when breaking the mirror, loosing redundancy for a time.

[...]
Booting to milestone "milestone/single-user:default".
Configuring devices.
Searching for installed OS instances...
/dev/dsk/c0d0s0 is under md control, skipping.
/dev/dsk/c1d1s0 is under md control, skipping.
No installed OS instance found.

Starting shell.
# mount -F ufs -o ro /dev/dsk/c0d0s0 /a
# cp -p /a/kernel/drv/md.conf /kernel/drv
# umount /a
# update_drv -f md
devfsadm: mkdir failed for /dev 0x1ed: Read-only file system
# metainit -r
# metasync d0
# fsck /dev/md/rdsk/d0
# mount -F ufs /dev/md/dsk/d0 /a
# bootadm update-archive -R /a
# umount /a
# reboot

Really interesting!

System V IPC Now Managed By Resource Controls

Julien Gabel — Sat, 01 Nov 2008 15:08:00 +0100

When it comes to Solaris 10, all IPC facilities are either automatically configured or can be controlled by resource controls. In the same time, they get new default values, when applicable.

As an example, we will assume that we need to change the limit on number of shared memory segments that can be created, and that the new default (128) is not enough either. Before Solaris 10, you've had to set the shmsys:shminfo_shmmni tunable parameter in the /etc/system kernel configuration file, which is a system wide limit, and required a reboot. This parameter is now marked as Obsolete or Have Been Removed, and its use is clearly deprecated.

To increase the corresponding limit up to 256 shared memory segments, we now have to deal with the project.max-shm-ids resource control which is controlled at the project level. The idea is to set the appropriate resource control to a project, then execute a program in the context of this project. One method to achieve this is to create a project at one side (using the project(4) database), and to populate the extended user attributes to do the association between this project and a user account (using the user_attr(4) database) in order to put the new project as the default project for the user. Or it is possible not to create an extended user attribute with this project at all, but use its characteristics explicitly through the newtask(1) command (and the login(1), cron(1M), and su(1M) programs, or the setproject(3PROJECT) function). But the simplest method, and the less intrusive one, is certainly to directly put the project as the default one for a user account. Here is how to do so.

By default, no error message is logged against the syslog daemon for resource controls. To be able to see an appropriate message in the messages log file, you must first enable globally the syslog action for the wanted resource control (the default level is notice).

# rctladm -e syslog project.max-shm-ids
# rctladm -l project.max-shm-ids
project.max-shm-ids   syslog=notice   [ no-basic deny count ]

When the limit on the number of shared memory segments is reached, one message similar to the following is write to the log file:

# grep rctl /var/adm/messages
/var/adm/messages:Oct 21 16:47:29 hostname genunix: [ID 883052 kern.notice] privileged rctl project.max-shm-ids (value 128) exceeded by project 3

Here is the definition of the new project, and its configuration.

# getent project user.username
user.username:1000:Project To Increase The Limit Of SHM Segments:::project.max-shm-ids=(priv,256,deny)
#
# projects -l user.username
user.username
      projid : 1000
      comment: "Project To Increase The Limit Of SHM Segments"
      users  : (none)
      groups : (none)
      attribs: project.max-shm-ids=(priv,256,deny)

When a project name begin with the pattern user., the project will automatically be set as the default one for the corresponding user, without the need to populate the extended user attributes database. Check that the project is set as the default project for the account username.

# id -p username
uid=100(username) gid=100(groupname) projid=1000(user.username)
#
# projects -d username
user.username

After a login phase using the username identity, the programs progname is launched. We can confirm the use of shared memory segments under the context of the project user.username, and we can consult the programs statistics report.

# ipcs -mJ
IPC status from  as of Wed Oct 29 11:39:59 CET 2008
T         ID KEY        MODE    OWNER     GROUP       PROJECT
Shared Memory:
m 1409286255   0 --rw-rw-rw- username groupname user.username
m  469762152   0 --rw-rw-rw- username groupname user.username
m         56   0 --rw-rw-rw- username groupname user.username
#
# prstat -n5 -cJ
   PID USERNAME  SIZE   RSS STATE PRI NICE    TIME  CPU PROCESS/NLWP
  3704 username  373M  284M cpu24   2   10 0:07:37 2.1% progname/26
  6785 username  285M  196M sleep  29   10 0:04:13 1.1% progname/26
  4480 username  785M  697M sleep  29   10 0:11:40 1.1% progname/26
  5836 username  293M  204M sleep  29   10 0:06:31 1.0% progname/26
  7635 username  277M  188M sleep  29   10 0:01:00 0.9% progname/26
PROJID    NPROC  SWAP   RSS MEMORY      TIME  CPU PROJECT
  1000       26 6472M 6333M    26%   3:57:24  23% user.username
     1       17   41M   87M   0.4%   2:39:58 0.0% user.root
     0       43  184M  267M   1.1%   4:07:25 0.0% system
     3        4 5856K   11M   0.0%   0:00:00 0.0% default
Total: 90 processes, 916 lwps, load averages: 4.41, 2.36, 1.04

Last, we can verify the new setting for one progname instance. For example for PID 3704:

# prctl -n project.max-shm-ids 3704
process: 3704: bin/progname 54 80 -Xmx192m
NAME    PRIVILEGE       VALUE    FLAG   ACTION      RECIPIENT
project.max-shm-ids
        privileged        256       -   deny                -
        system          16.8M     max   deny                -

The resource management facility can do much more than just tuning IPC settings, such as managing CPU usage, and physical memory control. It is a more fine-grained facility than what is in place before Solaris 10, and did not required a reboot anymore.

As a last word, we can note that there are command line tools to help creating and managing projects and extended user attributes for locally stored databases: respectively projadd(1M), projmod(1M), and useradd(1M), usermod(1M). But since the information sources was hosted in NIS and LDAP network directories, we did not use them for this test case though.

Solaris vs. RHEL Costs And Features Comparisons

Julien Gabel — Sun, 26 Oct 2008 13:25:00 +0100

Clearly, the costs involved in running Solaris and RHEL platforms are not well understood, and generally favors GNU/Linux environments. This is (most of the time) untrue, since this tend to be based on personal user experience, which is in fact far different from running lots of systems in high demand production data centers.

Here some interesting readings on these subjects--costs and features analysis--from:

Jim Laurent, OS Ambassador within Sun and evangelist for the Solaris OS
Solaris 10 costs less than Red Hat Enterprise Linux (and does more)
Paul Murphy, an IT consultant specializing in Unix and related technologies
Cost comparison: Solaris/SPARC vs Linux/x86
Marc Andreessen, creator of the Mosaic web browser, and co-founder of Netscape Communications
Solaris is a better Linux than Linux
Bill Vass, President and Chief Operating Officer of Sun Federal
UPDATED Solaris vs. RHEL Comparison - to include Microsoft Windows

YMMV for sure, but I personally think that Solaris costs are overestimated, and its features are mostly unknown, or at least underused... but this is a very large and hot topic nowadays, I know.

Update #1 (2008-11-14): Go to read interesting comment update from Jim Laurent.

Update #2 (2008-12-02): Go to read Joerg Moellenkamp's entry about similar points.

Update #3 (2008-12-03): Go to read this article appearing in Computerworld.com.

Discrepancies Between df And du Outputs

Julien Gabel — Sat, 18 Oct 2008 13:03:00 +0200

As a SA, it not uncommon to have regularly requests about big differences between the du and df outputs on a UFS file system. (For ZFS specific considerations, please see the ZFS FAQ.)

The du utility reports the sum of space allocated to all files in the file hierarchy rooted in the directory plus the space allocated to the directory itself. The df utility reports the amount of disk space occupied by a mounted file system.

When a file is remove from the file system, i.e. is unlinked (the hard link count goes to zero), the space belonging to this file is accounted against the du tool, but is not visible to the df utility until all references to it (open file descriptors) are closed. In order to find the guilty process, one can follow the information found in the SunManagers Frequently Asked Questions. Here is an example of such finding, but using a slightly different method to get the process currently holding the open descriptor to the deleted file.

Find the file which has been unlinked through the procfs interface:

# find /proc/*/fd \( -type f -a ! -size 0 -a -links 0 \) -print | xargs \ls -li
 415975 --w-------   0 user  group  2125803025 Oct 15 23:59 /proc/1252/fd/3

Eventually, get more detail about it:

# pargs -c 1252
1252:   rvd.basic -reliability 5 -listen tcp:9876 -logfile /path/to/log/rvd_9876.l
argv[0]: rvd.basic
argv[1]: -reliability
argv[2]: 5
argv[3]: -listen
argv[4]: tcp:9876
argv[5]: -logfile
argv[6]: /path/to/log/rvd_9876.log

Check to see if you can understand what is the content of the unlinked file:

# tail /proc/1252/fd/3
-------------------------------------------------------------------------------
2008-10-15 23:59:32.002116 - [MSG] BBG_Transmitter_class.cc, line 792 (thread 25087:4)
[4060] Sent a heartbeat
-------------------------------------------------------------------------------
BBG_Transmitter_class.cc: [4111] No activity detected. Send a Heartbeat message
-------------------------------------------------------------------------------
2008-10-15 23:59:32.134829 - [MSG] BBG_Transmitter_class.cc, line 1138 (thread 25087:4)
[4065] Heartbeat acknowledged by Bloomberg

You can correlate the size of the removed, but always referenced, file to the space accounted from the du and df tools:

# df -k /path/to
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d5       6017990 5874592   83219    99%    /path/to
# du -sk /path/to
3791632 /data
# echo "(5874616-3791632)*1024" | bc
2132975616

So, we now found the ~2GB log file which was always opened (used) by a process. Now, there are two solutions to be able to get back the freed space:

Truncate the unlinked file (quick workaround).
Simply restart properly the corresponding program (better option).

Use the solution which fits the best your need in your environment.

Anatomy Of An Attack

Julien Gabel — Sat, 11 Oct 2008 21:53:00 +0200

Well, although I didn't generally give credibility by speaking about public FUD, I will just let you know about really great, and point by point explanations on the recent InfoWorld (and New York Times) publication from Paul Krill Is Sun Solaris on its deathbed?

As Jim Grisanzio stated recently on the advocacy-discuss mailing list:

[...] More importantly, though, is that the original article not only fell flat but it was actually aggressively rejected by many in the open source community. That's an interesting shift out there. And a good one, too.

Well, I can't agree more with Jim on his points.

Update #1 (2008-10-28): Don't forget to read the interesting inputs from Joerg Moellenkamp.

Fake The hostid Of A Solaris Zone, Updated

Julien Gabel — Mon, 06 Oct 2008 15:00:00 +0200

As a little follow-up to Fake The hostid Of A Solaris Zone, and regarding the discussion on the capacity to change the hostid of a Solaris non-global zone, it is interesting to mention these (updated) informations:

The LD_PRELOAD trick proposed before is not a proper option, and is really ugly (and intrusive if you didn't unset it before continuing the execution of a program).
When using Solaris 8 or Solaris 9 Containers, there is a feature called Host ID Emulation from the zonecfg utility which can do exactly that.

Before the introduction of the privileges in a non-global zone with Solaris 11/06 (a.k.a. Solaris Update 3), you must run the DTrace zhostid script (daemon) within the global zone. It is not mandatory to run it from the global zone anymore. Using the appropriate dtrace_user privilege only, you can run it directly from the non-global zone:

# zonecfg -z ngzone set limitpriv=default,dtrace_user
# zoneadm -z ngzone boot
# zlogin ngzone
[Connected to zone 'ngzone' pts/5]
Last login: Sat Oct  4 18:57:17 on pts/5
Sun Microsystems Inc.   SunOS 5.11      snv_99  November 2008
# /sbin/zonename 
ngzone
# /usr/bin/hostid
837d47dd
# ./zhostid &
[1] 21506
# /usr/bin/hostid
20a82f32
# ^D
[Connection to zone 'ngzone' pts/5 closed]

Sun's Free Proficiency Assessment System

Julien Gabel — Mon, 29 Sep 2008 08:21:00 +0200

Free. Online. Right now. As Peter Tribble mentioned in his blog, it is clearly not perfect and some questions (and answers) seems a little obscure sometimes. Nonetheless, as I have never had any (yes, any) formal training, I though interesting to try these tests, and here are the results for pre-assessment for:

UNIX Essentials Featuring the Solaris 10 OS, I scored 37 out of 42.
Sun Certified System Administrator for the Solaris 10 OS (Part 1), I scored 45 out of 48.
Certified System Administrator for the Solaris 10 OS (Part 2), I scored 45 out of 48.

Maybe, is it time to pass to official Solaris Operating System certifications? ;)

Sun Certified Solaris Associate (SCSAS)
Sun Certified System Administrator (SCSA)
Sun Certified Network Administrator (SCNA)
Sun Certified Security Administrator (SCSECA)

Well... if time permit!

About GNU/Linux Software Mirroring And LVM

Julien Gabel — Mon, 22 Sep 2008 09:43:00 +0200

Here, the final aim was to provide data access redundancy through SAN storage hosted on remote sites across Wide Area Network (WAN) links. After some relatively long and painful tries to mimic software mirroring as found on HP-UX platform using Logical Volume Management (LVM), i.e. at the logical volume level, I finally give up deciding this functionality will definitely not fit my need. Why? Here are my comments.

It is not possible to provide clear and manageable storage multipath when the need to distinguish between the multiple sites is important, ala mirror across controllers found on Veritas VxVM on Sun Solaris system, for example. So, managing many physical volumes along with lots of logical volumes is very complicated.
There is no exact mapping capability between logical volume storage on a given physical volume.
The need to have a disk-based log, i.e. a persistent log. Yes, one can always provide the option --corelog at the creation time to the logical volume initial build and have an in-memory log , i.e. a non-persistent log, but this requires the entire copies (mirrors) be resynchronized upon reboot. Not really viable on multi-TB environments.
A write-intensive workload on a file system living on a logical volume mirror will suffer high latency: the overhead is important, and the time to do mostly-write jobs grow dramatically. It is really hard to get high level statistics, only low level metrics seems consistent: sd SCSI devices and dm- device mapper components for each paths entries. Not from the multipath devices standpoint, which is the more interesting from the end user and SA point of view.
You can't extend a logical volume, which is really a no-go per-se. On that point, the Red Hat support answered that this functionality may be added in a future release, the current state may eventually be a Request For Enhancement (RFE), if a proper business justification is provided. One must break the logical volume mirror copy, then rebuild it completely. Not realistic when the logical volume is made of a lot of physical extents across multiple physical volumes.
A LVM configuration can be totally blocked by itself, and not usable at all. The fact is, LVM use persistent storage blocks to keep track of its own metadata. The metadata size is set at physical volume creation time only, and can't be change afterward. This size is statically defined as 255 physical volume blocks, and can be adjust from the LVM configuration file. The problem is, when this circular buffer space (stored in ASCII) fills up--such as when there are a lot of logical volumes in a mirrored environment--it is not possible to do anything more with LVM. So you can't add more logical volume, can't add more logical volume copies,... and can't delete them trying to reestablish a proper LVM configuration. Well, here are the answers given by the Red Hat support to two keys questions in this situation:
- How to size the metadata, i.e. if we need to change it from the default value, how can we determine the new proper and appropriate size, and from which information?
  I am afraid but Metadata size can only be defined at the time of PV creation and there is no real formula for calculating the size in advance. By changing the default value of 255 you can get a higher space value. For general LVM setup (with less LV's and VG's) default size works fine however in cases where high number of LV's are required a custom value will be required.
- We just want to delete all LV copies, which means to return to the initial situation and have 0 copy for all LV, i.e. only one LV per-se, in order to be able to change LVM configuration again (we can't do anything on our production server right now)?
  I discussed this situation with peer engineers and also referenced a similar previous case. From the notes of the same the workaround is to use the backup file (/etc/lvm/backup) and restore the PV's. I agree that this really not a production environment method however seems the only workaround.

So, the production RDBMS Oracle server is finally now being evacuate to an other machine. Hum... Hope to see better enterprise experience using the mdadm package to handle RAID software, instead of mirror (RAID-1) LVM. Maybe more about that in an other blog entry?

Announcing Solaris 10 10/08

Julien Gabel — Tue, 16 Sep 2008 19:16:00 +0200

Although this seems a little bit confident, the long-awaited Update 6 to the Solaris 10 operating system release is just behind the door. This release (scheduled to be available in mid-October) will includes virtualization enhancements including the ability for a Solaris Container to automatically update its environment when moved from one system to another, Logical Domains support for dynamically reconfigurable disk and network I/O, and paravirtualization support when Solaris 10 is used as a guest OS in Xen-based environments such as Sun xVM Server. Solaris 10 10/08 also includes support for the latest systems from Sun and other vendors, such as those based on the Intel Xeon Processor 7400 Series.

This release will be the very first Solaris release to be able to boot from ZFS and use it as their root file system, such as what can be found on OpenSolaris or Solaris Express Community Release today.

Check the What’s New web page for Solaris, and consult the Solaris Media Gallery videos for more information.

Update #1 (2008-10-14): Don't forget to consult the must read What's New in Solaris 10 10/08? from the San Antonio OpenSolaris User Group.

Update #2 (2008-10-31): Get yours, and go reading the official What's New in the Solaris 10 10/08 Release page.

Comparison: EMC PowerPath vs. GNU/Linux dm-multipath

Julien Gabel — Fri, 16 May 2008 21:10:00 +0200

I will present some notes about the use of multipath solutions on Red Hat systems: EMC PowerPath and GNU/Linux dm-multipath. Along those notes, keep in mind that they were based on tests done when pressure was very high to put new systems in production, so lack of time resulted in less complete tests than expected. These tests were done more than a year ago, and so before the release of RHEL4 Update 5 and some of RHBA related to both LVM and dm-multipath technologies.

Keep in mind that without purchasing an appropriate EMC license, PowerPath can only be used in failover mode (active-passive mode). Multiple paths accesses are not supported in this case: no round-robin, and no I/O load balancer for example.

EMC PowerPath

Advantages

Not specific to the SAN Host Bus Adapter (HBA).
Support for multiple and heterogeneous SAN storage provider.
Support for most UNIX and Unix-like platforms.
Without a valid license, can only work in degraded mode (failover).
Is not sensible to a change in the SCSI LUN renumbering. Adapt accordingly the corresponding multiple sd devices (different paths to a given device) with its multipath definition of the emcpower device.
Provide easily the ID of the SAN storage.

Drawbacks

Not integrated with the operating system (which generally has its own solution).
The need to force a RPM re-installation in case of a kernel upgrade on RHEL systems (due to the fact that kernel modules are stored in a path containing the exact major and minor versions of the installed (booted) kernel.
Non-automatic update procedure.

GNU/Linux device-mapper-multipath

Advantages

Not specific to the SAN Host Bus Adapter (HBA).
Support for multiple and heterogeneous SAN storage provider.
Well integrated with the operating system.
Automatic update using RHN (you must be a licensed and registered user in this case).
No additional license cost.

Drawbacks

Only available on GNU/Linux systems.
Configuration (files and keywords) very tedious and difficult.
Without the use of LVM (Logical Volume Management), it has not the ability to follow SCSI LUN renumbering! Even in this case, be sure not to have blacklisted the newly discovered SCSI devices (sd).

Last, please find some interesting documentation on the subject below:

PHP APC Extension Bug With Optimized Open Source Software Stack

Julien Gabel — Fri, 02 May 2008 15:57:00 +0200

To easily manage LDAP accounts (and general LDAP entries in fact), we have created a Solaris Zone and installed the excellent Cool Stack bundle to host the LAM (LDAP Account Manager) management web tool. But after upgrading the Cool Stack to version 1.2 we encountered a very annoying problem mostly with freezing web pages, and generally ending up in restarting the Apache web server provided by the Cool Stack. After some troubleshooting, we discover that this behavior was introduced by a bug in the APC-3.0.14 module bundled with the updated php-5.2.4 scripting software in this version of the Cool Stack.

Luckily, the bug was already fixed and a new version of the APC extension of PHP is available for download (in fact, just replace to original apc.so module by the new one). All the Cool Stack related problems, associated fixes and instructions are listed on the Cool Stack 1.2 Patches page: be sure to keep in sync' if you are a Cool Stack consumer.

memconf And AMD Athlon 64 X2 Dual Core Processor

Julien Gabel — Mon, 28 Apr 2008 18:25:00 +0200

The last update to the excellent memconf utility (V2.5 22-Feb-2008) support properly recent Solaris Express releases, and my recent change from the stock AMD Opteron Processor 148 to an AMD Athlon 64 X2 Dual Core Processor 3800+. (I mostly did that change just to be able to access two run queues separately, not to gain more power per se.)

So, here is the new and appropriate memconf report:

# memconf -d
memconf:  V2.5 22-Feb-2008 http://www.4schmidts.com/unix.html
hostname: unic
manufacturer: Sun Microsystems, Inc.
model:    Sun Ultra 20 Workstation (AMD Athlon(tm) 64 X2 Dual Core \
 Processor 3800+ Socket 939 2010MHz)
Sun Family Part Number: A63
Solaris Express Community Edition snv_87 X86, 64-bit kernel, SunOS 5.11
1 AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ Socket 939 2010MHz cpu
diagbanner = Sun Ultra 20 Workstation
cpubanner = AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ Socket 939 2010MHz
model = Sun Ultra 20 Workstation
machine = i86pc
platform = i86pc
perl version: 5.008004
CPU Units:
==== Processor Sockets ====================================
Version                          Location Tag
-------------------------------- --------------------------
AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ Socket 939
AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ Socket 939
Memory Units:
Type    Status Set Device Locator      Bank Locator
------- ------ --- ------------------- --------------------
unknown in use 0   A0                  Bank0/1
unknown in use 0   A1                  Bank2/3
unknown in use 0   A2                  Bank4/5
unknown in use 0   A3                  Bank6/7
total memory = 2048MB (2GB)

You can check and compare with the previous report on my blog.