blog'o thnet

System V IPC Now Managed By Resource Controls

Julien Gabel — Sat, 01 Nov 2008 15:08:00 +0100

When it comes to Solaris 10, all IPC facilities are either automatically configured or can be controlled by resource controls. In the same time, they get new default values, when applicable.

As an example, we will assume that we need to change the limit on number of shared memory segments that can be created, and that the new default (128) is not enough either. Before Solaris 10, you've had to set the shmsys:shminfo_shmmni tunable parameter in the /etc/system kernel configuration file, which is a system wide limit, and required a reboot. This parameter is now marked as Obsolete or Have Been Removed, and its use is clearly deprecated.

To increase the corresponding limit up to 256 shared memory segments, we now have to deal with the project.max-shm-ids resource control which is controlled at the project level. The idea is to set the appropriate resource control to a project, then execute a program in the context of this project. One method to achieve this is to create a project at one side (using the project(4) database), and to populate the extended user attributes to do the association between this project and a user account (using the user_attr(4) database) in order to put the new project as the default project for the user. Or it is possible not to create an extended user attribute with this project at all, but use its characteristics explicitly through the newtask(1) command (and the login(1), cron(1M), and su(1M) programs, or the setproject(3PROJECT) function). But the simplest method, and the less intrusive one, is certainly to directly put the project as the default one for a user account. Here is how to do so.

By default, no error message is logged against the syslog daemon for resource controls. To be able to see an appropriate message in the messages log file, you must first enable globally the syslog action for the wanted resource control (the default level is notice).

# rctladm -e syslog project.max-shm-ids
# rctladm -l project.max-shm-ids
project.max-shm-ids   syslog=notice   [ no-basic deny count ]

When the limit on the number of shared memory segments is reached, one message similar to the following is write to the log file:

# grep rctl /var/adm/messages
/var/adm/messages:Oct 21 16:47:29 hostname genunix: [ID 883052 kern.notice] privileged rctl project.max-shm-ids (value 128) exceeded by project 3

Here is the definition of the new project, and its configuration.

# getent project user.username
user.username:1000:Project To Increase The Limit Of SHM Segments:::project.max-shm-ids=(priv,256,deny)
#
# projects -l user.username
user.username
      projid : 1000
      comment: "Project To Increase The Limit Of SHM Segments"
      users  : (none)
      groups : (none)
      attribs: project.max-shm-ids=(priv,256,deny)

When a project name begin with the pattern user., the project will automatically be set as the default one for the corresponding user, without the need to populate the extended user attributes database. Check that the project is set as the default project for the account username.

# id -p username
uid=100(username) gid=100(groupname) projid=1000(user.username)
#
# projects -d username
user.username

After a login phase using the username identity, the programs progname is launched. We can confirm the use of shared memory segments under the context of the project user.username, and we can consult the programs statistics report.

# ipcs -mJ
IPC status from  as of Wed Oct 29 11:39:59 CET 2008
T         ID KEY        MODE    OWNER     GROUP       PROJECT
Shared Memory:
m 1409286255   0 --rw-rw-rw- username groupname user.username
m  469762152   0 --rw-rw-rw- username groupname user.username
m         56   0 --rw-rw-rw- username groupname user.username
#
# prstat -n5 -cJ
   PID USERNAME  SIZE   RSS STATE PRI NICE    TIME  CPU PROCESS/NLWP
  3704 username  373M  284M cpu24   2   10 0:07:37 2.1% progname/26
  6785 username  285M  196M sleep  29   10 0:04:13 1.1% progname/26
  4480 username  785M  697M sleep  29   10 0:11:40 1.1% progname/26
  5836 username  293M  204M sleep  29   10 0:06:31 1.0% progname/26
  7635 username  277M  188M sleep  29   10 0:01:00 0.9% progname/26
PROJID    NPROC  SWAP   RSS MEMORY      TIME  CPU PROJECT
  1000       26 6472M 6333M    26%   3:57:24  23% user.username
     1       17   41M   87M   0.4%   2:39:58 0.0% user.root
     0       43  184M  267M   1.1%   4:07:25 0.0% system
     3        4 5856K   11M   0.0%   0:00:00 0.0% default
Total: 90 processes, 916 lwps, load averages: 4.41, 2.36, 1.04

Last, we can verify the new setting for one progname instance. For example for PID 3704:

# prctl -n project.max-shm-ids 3704
process: 3704: bin/progname 54 80 -Xmx192m
NAME    PRIVILEGE       VALUE    FLAG   ACTION      RECIPIENT
project.max-shm-ids
        privileged        256       -   deny                -
        system          16.8M     max   deny                -

The resource management facility can do much more than just tuning IPC settings, such as managing CPU usage, and physical memory control. It is a more fine-grained facility than what is in place before Solaris 10, and did not required a reboot anymore.

As a last word, we can note that there are command line tools to help creating and managing projects and extended user attributes for locally stored databases: respectively projadd(1M), projmod(1M), and useradd(1M), usermod(1M). But since the information sources was hosted in NIS and LDAP network directories, we did not use them for this test case though.

Solaris vs. RHEL Costs And Features Comparisons

Julien Gabel — Sun, 26 Oct 2008 13:25:00 +0100

Clearly, the costs involved in running Solaris and RHEL platforms are not well understood, and generally favors GNU/Linux environments. This is (most of the time) untrue, since this tend to be based on personal user experience, which is in fact far different from running lots of systems in high demand production datacenters.

Here some interesting readings on these subjects--costs and features analysis--from:

Jim Laurent, OS Ambassador within Sun and evangelist for the Solaris OS
Solaris 10 costs less than Red Hat Enterprise Linux (and does more)
Paul Murphy, an IT consultant specializing in Unix and related technologies
Cost comparison: Solaris/SPARC vs Linux/x86
Marc Andreessen, creator of the Mosaic web browser, and co-founder of Netscape Communications
Solaris is a better Linux than Linux
Bill Vass, President and Chief Operating Officer of Sun Federal
UPDATED Solaris vs. RHEL Comparison - to include Microsoft Windows

YMMV for sure, but I personally think that Solaris costs are overestimated, and its features are mostly unknown, or at least underused... but this is a very large and hot topic nowadays, I know.

Discrepancies Between df And du Outputs

Julien Gabel — Sat, 18 Oct 2008 13:03:00 +0200

As a SA, it not uncommon to have regularly requests about big differences between the du and df outputs on a UFS file system. (For ZFS specific considerations, please see the ZFS FAQ.)

The du utility reports the sum of space allocated to all files in the file hierarchy rooted in the directory plus the space allocated to the directory itself. The df utility reports the amount of disk space occupied by a mounted file system.

When a file is remove from the file system, i.e. is unlinked (the hard link count goes to zero), the space belonging to this file is accounted against the du tool, but is not visible to the df utility until all references to it (open file descriptors) are closed. In order to find the guilty process, one can follow the information found in the SunManagers Frequently Asked Questions. Here is an example of such finding, but using a slightly different method to get the process currently holding the open descriptor to the deleted file.

Find the file which has been unlinked through the procfs interface:

# find /proc/*/fd \( -type f -a ! -size 0 -a -links 0 \) -print | xargs \ls -li
 415975 --w-------   0 user  group  2125803025 Oct 15 23:59 /proc/1252/fd/3

Eventually, get more detail about it:

# pargs -c 1252
1252:   rvd.basic -reliability 5 -listen tcp:9876 -logfile /path/to/log/rvd_9876.l
argv[0]: rvd.basic
argv[1]: -reliability
argv[2]: 5
argv[3]: -listen
argv[4]: tcp:9876
argv[5]: -logfile
argv[6]: /path/to/log/rvd_9876.log

Check to see if you can understand what is the content of the unlinked file:

# tail /proc/1252/fd/3
-------------------------------------------------------------------------------
2008-10-15 23:59:32.002116 - [MSG] BBG_Transmitter_class.cc, line 792 (thread 25087:4)
[4060] Sent a heartbeat
-------------------------------------------------------------------------------
BBG_Transmitter_class.cc: [4111] No activity detected. Send a Heartbeat message
-------------------------------------------------------------------------------
2008-10-15 23:59:32.134829 - [MSG] BBG_Transmitter_class.cc, line 1138 (thread 25087:4)
[4065] Heartbeat acknowledged by Bloomberg

You can correlate the size of the removed, but always referenced, file to the space accounted from the du and df tools:

# df -k /path/to
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d5       6017990 5874592   83219    99%    /path/to
# du -sk /path/to
3791632 /data
# echo "(5874616-3791632)*1024" | bc
2132975616

So, we now found the ~2GB log file which was always opened (used) by a process. Now, there are two solutions to be able to get back the freed space:

Truncate the unlinked file (quick workaround).
Simply restart properly the corresponding program (better option).

Use the solution which fits the best your need in your environment.

Anatomy Of An Attack

Julien Gabel — Sat, 11 Oct 2008 21:53:00 +0200

Well, although I didn't generally give credibility by speaking about public FUD, I will just let you know about really great, and point by point explanations on the recent InfoWorld (and New York Times) publication from Paul Krill Is Sun Solaris on its deathbed?

As Jim Grisanzio stated recently on the advocacy-discuss mailing list:

[...] More importantly, though, is that the original article not only fell flat but it was actually aggressively rejected by many in the open source community. That's an interesting shift out there. And a good one, too.

Well, I can't agree more with Jim on his points.

Update #1 (2008-10-28): Don't forget to read the interesting inputs from Joerg Moellenkamp.

Fake The hostid Of A Solaris Zone, Updated

Julien Gabel — Mon, 06 Oct 2008 15:00:00 +0200

As a little follow-up to Fake The hostid Of A Solaris Zone, and regarding the discussion on the capacity to change the hostid of a Solaris non-global zone, it is interesting to mention these (updated) informations:

The LD_PRELOAD trick proposed before is not a proper option, and is really ugly (and intrusive if you didn't unset it before continuing the execution of a program).
When using Solaris 8 or Solaris 9 Containers, there is a feature called Host ID Emulation from the zonecfg utility which can do exactly that.

Before the introduction of the privileges in a non-global zone with Solaris 11/06 (a.k.a. Solaris Update 3), you must run the DTrace zhostid script (daemon) within the global zone. It is not mandatory to run it from the global zone anymore. Using the appropriate dtrace_user privilege only, you can run it directly from the non-global zone:

# zonecfg -z ngzone set limitpriv=default,dtrace_user
# zoneadm -z ngzone boot
# zlogin ngzone
[Connected to zone 'ngzone' pts/5]
Last login: Sat Oct  4 18:57:17 on pts/5
Sun Microsystems Inc.   SunOS 5.11      snv_99  November 2008
# /sbin/zonename 
ngzone
# /usr/bin/hostid
837d47dd
# ./zhostid &
[1] 21506
# /usr/bin/hostid
20a82f32
# ^D
[Connection to zone 'ngzone' pts/5 closed]

Sun's Free Proficiency Assessment System

Julien Gabel — Mon, 29 Sep 2008 08:21:00 +0200

Free. Online. Right now. As Peter Tribble mentioned in his blog, it is clearly not perfect and some questions (and answers) seems a little obscure sometimes. Nonetheless, as I have never had any (yes, any) formal training, I though interesting to try these tests, and here are the results for pre-assessment for:

UNIX Essentials Featuring the Solaris 10 OS, I scored 37 out of 42.
Sun Certified System Administrator for the Solaris 10 OS (Part 1), I scored 45 out of 48.
Certified System Administrator for the Solaris 10 OS (Part 2), I scored 45 out of 48.

Maybe, is it time to pass to official Solaris Operating System certifications? ;)

Sun Certified Solaris Associate (SCSAS)
Sun Certified System Administrator (SCSA)
Sun Certified Network Administrator (SCNA)
Sun Certified Security Administrator (SCSECA)

Well... if time permit!

About GNU/Linux Software Mirroring And LVM

Julien Gabel — Mon, 22 Sep 2008 09:43:00 +0200

Here, the final aim was to provide data access redundancy through SAN storage hosted on remote sites across Wide Area Network (WAN) links. After some relatively long and painful tries to mimic software mirroring as found on HP-UX platform using Logical Volume Management (LVM), i.e. at the logical volume level, I finally give up deciding this functionality will definitely not fi my need. Why? Here are my comments.

It is not possible to provide clear and manageable storage multipath when the need to distinguish between the multiple sites is important, ala mirror across controllers found on Veritas VxVM on Sun Solaris system, for example. So, managing many physical volumes along with lots of logical volumes is very complicated.
There is no exact mapping capability between logical volume storage on a given physical volume.
The need to have a disk-based log, i.e. a persistent log. Yes, one can always provide the option --corelog at the creation time to the logical volume initial build and have an in-memory log , i.e. a non-persistent log, but this requires the entire copies (mirrors) be resynchronized upon reboot. Not really viable on multi-TB environments.
A write-intensive workload on a file system living on a logical volume mirror will suffer high latency: the overhead is important, and the time to do mostly-write jobs grow dramatically. It is really hard to get high level statistics, only low level metrics seems consistent: sd SCSI devices and dm- device mapper components for each paths entries. Not from the multipath devices standpoint, which is the more interesting from the end user and SA point of view.
You can't extend a logical volume, which is really a no-go per-se. On that point, the Red Hat support answered that this functionality may be added in a future release, the current state may eventually be a Request For Enhancement (RFE), if a proper business justification is provided. One must break the logical volume mirror copy, then rebuild it completely. Not realistic when the logical volume is made of a lot of physical extents across multiple physical volumes.
A LVM configuration can be totally blocked by itself, and not usable at all. The fact is, LVM use persistent storage blocks to keep track of its own metadata. The metadata size is set at physical volume creation time only, and can't be change afterward. This size is statically defined as 255 physical volume blocks, and can be adjust from the LVM configuration file. The problem is, when this circular buffer space (stored in ASCII) fills up--such as when there are a lot of logical volumes in a mirrored environment--it is not possible to do anything more with LVM. So you can't add more logical volume, can't add more logical volume copies,... and can't delete them trying to reestablish a proper LVM configuration. Well, here are the answers given by the Red Hat support to two keys questions in this situation:
- How to size the metadata, i.e. if we need to change it from the default value, how can we determine the new proper and appropriate size, and from which information?
  I am afraid but Metadata size can only be defined at the time of PV creation and there is no real formula for calculating the size in advance. By changing the default value of 255 you can get a higher space value. For general LVM setup (with less LV's and VG's) default size works fine however in cases where high number of LV's are required a custom value will be required.
- We just want to delete all LV copies, which means to return to the initial situation and have 0 copy for all LV, i.e. only one LV per-se, in order to be able to change LVM configuration again (we can't do anything on our production server right now)?
  I discussed this situation with peer engineers and also referenced a similar previous case. From the notes of the same the workaround is to use the backup file (/etc/lvm/backup) and restore the PV's. I agree that this really not a production environment method however seems the only workaround.

So, the production RDBMS Oracle server is finally now being evacuate to an other machine. Hum... Hope to see better enterprise experience using the mdadm package to handle RAID software, instead of mirror (RAID-1) LVM. Maybe more about that in an other blog entry?

Announcing Solaris 10 10/08

Julien Gabel — Tue, 16 Sep 2008 19:16:00 +0200

Although this seems a little bit confident, the long-awaited Update 6 to the Solaris 10 operating system release is just behind the door. This release (scheduled to be available in mid-October) will includes virtualization enhancements including the ability for a Solaris Container to automatically update its environment when moved from one system to another, Logical Domains support for dynamically reconfigurable disk and network I/O, and paravirtualization support when Solaris 10 is used as a guest OS in Xen-based environments such as Sun xVM Server. Solaris 10 10/08 also includes support for the latest systems from Sun and other vendors, such as those based on the Intel Xeon Processor 7400 Series.

This release will be the very first Solaris release to be able to boot from ZFS and use it as their root file system, such as what can be found on OpenSolaris or Solaris Express Community Release today.

Check the What’s New web page for Solaris, and consult the Solaris Media Gallery videos for more information.

Update #1 (2008-10-14): Don't forget to consult the must read What's New in Solaris 10 10/08? from the San Antonio OpenSolaris User Group.

Update #2 (2008-10-31): Get yours, and go reading the official What's New in the Solaris 10 10/08 Release page.

Comparison: EMC PowerPath vs. GNU/Linux dm-multipath

Julien Gabel — Fri, 16 May 2008 21:10:00 +0200

I will present some notes about the use of multipath solutions on Red Hat systems: EMC PowerPath and GNU/Linux dm-multipath. Along those notes, keep in mind that they were based on tests done when pressure was very high to put new systems in production, so lack of time resulted in less complete tests than expected. These tests were done more than a year ago, and so before the release of RHEL4 Update 5 and some of RHBA related to both LVM and dm-multipath technologies.

Keep in mind that without purchasing an appropriate EMC license, PowerPath can only be used in failover mode (active-passive mode). Multiple paths accesses are not supported in this case: no round-robin, and no I/O load balancer for example.

EMC PowerPath

Advantages

Not specific to the SAN Host Bus Adapter (HBA).
Support for multiple and heterogeneous SAN storage provider.
Support for most UNIX and Unix-like platforms.
Without a valid license, can only work in degraded mode (failover).
Is not sensible to a change in the SCSI LUN renumbering. Adapt accordingly the corresponding multiple sd devices (different paths to a given device) with its multipath definition of the emcpower device.
Provide easily the ID of the SAN storage.

Drawbacks

Not integrated with the operating system (which generally has its own solution).
The need to force a RPM re-installation in case of a kernel upgrade on RHEL systems (due to the fact that kernel modules are stored in a path containing the exact major and minor versions of the installed (booted) kernel.
Non-automatic update procedure.

GNU/Linux device-mapper-multipath

Advantages

Not specific to the SAN Host Bus Adapter (HBA).
Support for multiple and heterogeneous SAN storage provider.
Well integrated with the operating system.
Automatic update using RHN (you must be a licensed and registered user in this case).
No additional license cost.

Drawbacks

Only available on GNU/Linux systems.
Configuration (files and keywords) very tedious and difficult.
Without the use of LVM (Logical Volume Management), it has not the ability to follow SCSI LUN renumbering! Even in this case, be sure not to have blacklisted the newly discovered SCSI devices (sd).

Last, please find some interesting documentation on the subject below:

PHP APC Extension Bug With Optimized Open Source Software Stack

Julien Gabel — Fri, 02 May 2008 15:57:00 +0200

To easily manage LDAP accounts (and general LDAP entries in fact), we have created a Solaris Zone and installed the excellent Cool Stack bundle to host the LAM (LDAP Account Manager) management web tool. But after upgrading the Cool Stack to version 1.2 we encountered a very annoying problem mostly with freezing web pages, and generally ending up in restarting the Apache web server provided by the Cool Stack. After some troubleshooting, we discover that this behavior was introduced by a bug in the APC-3.0.14 module bundled with the updated php-5.2.4 scripting software in this version of the Cool Stack.

Luckily, the bug was already fixed and a new version of the APC extension of PHP is available for download (in fact, just replace to original apc.so module by the new one). All the Cool Stack related problems, associated fixes and instructions are listed on the Cool Stack 1.2 Patches page: be sure to keep in sync' if you are a Cool Stack consumer.

memconf And AMD Athlon 64 X2 Dual Core Processor

Julien Gabel — Mon, 28 Apr 2008 18:25:00 +0200

The last update to the excellent memconf utility (V2.5 22-Feb-2008) support properly recent Solaris Express releases, and my recent change from the stock AMD Opteron Processor 148 to an AMD Athlon 64 X2 Dual Core Processor 3800+. (I mostly did that change just to be able to access two run queues separately, not to gain more power per se.)

So, here is the new and appropriate memconf report:

# memconf -d
memconf:  V2.5 22-Feb-2008 http://www.4schmidts.com/unix.html
hostname: unic
manufacturer: Sun Microsystems, Inc.
model:    Sun Ultra 20 Workstation (AMD Athlon(tm) 64 X2 Dual Core \
 Processor 3800+ Socket 939 2010MHz)
Sun Family Part Number: A63
Solaris Express Community Edition snv_87 X86, 64-bit kernel, SunOS 5.11
1 AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ Socket 939 2010MHz cpu
diagbanner = Sun Ultra 20 Workstation
cpubanner = AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ Socket 939 2010MHz
model = Sun Ultra 20 Workstation
machine = i86pc
platform = i86pc
perl version: 5.008004
CPU Units:
==== Processor Sockets ====================================
Version                          Location Tag
-------------------------------- --------------------------
AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ Socket 939
AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ Socket 939
Memory Units:
Type    Status Set Device Locator      Bank Locator
------- ------ --- ------------------- --------------------
unknown in use 0   A0                  Bank0/1
unknown in use 0   A1                  Bank2/3
unknown in use 0   A2                  Bank4/5
unknown in use 0   A3                  Bank6/7
total memory = 2048MB (2GB)

You can check and compare with the previous report on my blog.

Change The DST Time Zone Definition For Test Purpose

Julien Gabel — Sun, 06 Apr 2008 19:13:00 +0200

Because we continue to encounter some specialized software vendor which can't tell if there is a problem with the Daylight Saving Time change for their application, the need to test the time adjustment beforehand arise sometimes. In this case, the first thing which comes in mind is to change the system clock, without modifying the timezone. Although this can do the job, it doesn't test the DST adjustment properly, and affect the overall operating environment, system-wide.

To do things cleaner, we can try to modify directly the timezone in use. This will test the real DST automatic adjustment, while not changing the system clock (and not impacting other software, or services). Say we want to change DST for Europe/Paris time zone one day before the official date for summer 2008. First, obtain the current setting:

# zdump -v `rtc` | \
   awk '$6 ~ /'"`date '+%Y'`"'/ && $12 !~ /'"`date '+%H:%M:%S'`"'/ {print $0}'
Europe/Paris  Sun Mar 30 00:59:59 2008 UTC = Sun Mar 30 01:59:59 2008 CET isdst=0
Europe/Paris  Sun Mar 30 01:00:00 2008 UTC = Sun Mar 30 03:00:00 2008 CEST isdst=1
Europe/Paris  Sun Oct 26 00:59:59 2008 UTC = Sun Oct 26 02:59:59 2008 CEST isdst=1
Europe/Paris  Sun Oct 26 01:00:00 2008 UTC = Sun Oct 26 02:00:00 2008 CET isdst=0

Then, get the time zone source file for the Europe/Paris geographic definition and adapt it writing an exception rule for the current year. Because the current definition for Europe/Paris is based on the EU rule, our modification will be based on this rule name.

# mkdir -p /tmp/zoneinfo/src
# cp /usr/share/lib/zoneinfo/src/europe /tmp/zoneinfo/src
# diff /usr/share/lib/zoneinfo/src/europe /tmp/zoneinfo/src/europe
1073a1074,1076
> # Test the DST adjustment one day in advance, for March 2008 only.
> # Rule        NAME    FROM    TO      TYPE    IN      ON      AT      SAVE    LETTER/S
> Rule          EU      2008    only    -       Mar     29       2:00   1:00    S

Last, compile the updated time zone definition, put it in place, and verify the new DST date.

# zic -d /tmp/zoneinfo /tmp/zoneinfo/src/europe
# cp /tmp/zoneinfo/Europe/Paris /usr/share/lib/zoneinfo/Europe
# zdump -v `rtc` | \
   awk '$6 ~ /'"`date '+%Y'`"'/ && $12 !~ /'"`date '+%H:%M:%S'`"'/ {print $0}'
Europe/Paris  Sat Mar 29 00:59:59 2008 UTC = Sat Mar 29 01:59:59 2008 CET isdst=0
Europe/Paris  Sat Mar 29 01:00:00 2008 UTC = Sat Mar 29 03:00:00 2008 CEST isdst=1
Europe/Paris  Sun Oct 26 00:59:59 2008 UTC = Sun Oct 26 02:59:59 2008 CEST isdst=1
Europe/Paris  Sun Oct 26 01:00:00 2008 UTC = Sun Oct 26 02:00:00 2008 CET isdst=0

Once the fake DST adjustment is validated at the software level, clean things up a little.

# zic /usr/share/lib/zoneinfo/src/europe
# rm -r /tmp/zoneinfo

Update A Corrupted GRUB Boot Archive, With SVM

Julien Gabel — Thu, 13 Mar 2008 12:25:00 +0100

In a previous discussion about the GRUB boot archive and how it can be regenerated, I mentioned that it will not be as easy as it can be when the root file system use the md driver. I will now show two different methods to do the same thing when the root file system is build upon a SVM mirror (RAID-1):

Unmirror the root file system only.
Unmirror the entire system, i.e. all devices.

Note: Although this test case was done using Solaris 10 8/07 under a virtual machine build upon VirtualBox on latest Solaris Express Community Edition, the instructions must be valid for Solaris 10 1/06 and later.

Initial setup

As we can see, the system use only a root file system, and a swap device. Both are encapsulated with SVM.

# df -k -F ufs
Filesystem     kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d0 6147798 3455578 2630743      57%  /
# swap -l
swapfile             dev  swaplo blocks   free
/dev/md/dsk/d1      85,1       8 4194288 4194288
# metastat -c d0 d1
d0               m  6.0GB d10 d20
    d10          s  6.0GB c0d0s0
    d20          s  6.0GB c1d1s0
d1               m  2.0GB d11 d21
    d11          s  2.0GB c0d0s1
    d21          s  2.0GB c1d1s1

Unmirror the root file system only

The idea is to boot on the GRUB failsafe mode, select the first side of the mirror, and modify the system and vfstab configuration files to use the correct device path. For the system file, this means to actually remove the rootdev:/pseudo/md@0:0,0,bl entry, not just comment it. For the vfstab file, this means replacing the root file system metadevice path /dev/md/[r]dsk/d0 by the first underlying device path, i.e. /dev/[r]dsk/c0d0s0. Last, regenerate the boot archive on the alternate root path.

[...]
Booting to milestone "milestone/single-user:default".
Configuring devices.
Searching for installed OS instances...
/dev/dsk/c0d0s0 is under md control, skipping.
/dev/dsk/c1d1s0 is under md control, skipping.
No installed OS instance found.

Starting shell.
# fsck /dev/rdsk/c0d0s0
# mount -F ufs /dev/dsk/c0d0s0 /a
# cp /a/etc/system /a/etc/system.bckp
# cp /a/etc/vfstab /a/etc/vfstab.bckp
# TERM=vt100 vi /a/etc/system
# TERM=vt100 vi /a/etc/vfstab
# bootadm update-archive -R /a
# umount /a
# fsck /dev/rdsk/c0d0s0
# reboot

Then, boot into milestone/multi-user:default level and detach the second half of the mirror, since the first half correspond to the valid and updated underlying device. Next, restore the original configuration files which refers to the encapsulated metadevices, and reboot.

# df -k -F ufs
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0d0s0      6147798 3458810 2627511    57%    /
# swap -l
swapfile             dev  swaplo blocks   free
/dev/md/dsk/d1      85,1       8 4194288 4194288
# metastat -c d0
d0               m  6.0GB d10 d20
    d10          s  6.0GB c0d0s0
    d20          s  6.0GB c1d1s0
# metadetach d0 d20
d0: submirror d20 is detached
# metastat -c d0
d0               m  6.0GB d10
    d10          s  6.0GB c0d0s0
# cp /etc/system.orig /etc/system
# cp /etc/vfstab.orig /etc/vfstab
# shutdown -y -i 6 -g 0

After the reboot, just reattach the second half of the mirror, and wait for complete synchronization to be fully redundant again.

# df -k -F ufs
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d0       6147798 3458714 2627607    57%    /
# swap -l
swapfile             dev  swaplo blocks   free
/dev/md/dsk/d1      85,1       8 4194288 4194288
# metattach d0 d20
d0: submirror d20 is attached
# metastat -c d0
d0               m  6.0GB d10 d20 (resync-29%)
    d10          s  6.0GB c0d0s0
    d20          s  6.0GB c1d1s0

Unmirror the entire system, i.e. all devices

The idea is exactly the same as for unmirroring the root file system only, but adapting the vfstab file to change the swap entry, too. (So, I didn't reproduce the code listing here.)

Then, boot into milestone/single-user:default level modifying the corresponding GRUB entry as follow: kernel /platform/i86pc/multiboot -s. Completely delete all the metadevices and metadb configurations to clear SVM settings. Last, continue into milestone/multi-user:default level to boot unmirrored.

# metaclear -f -r d0 d1
# metadb -f -d  c1d0s4 c1d0s4
# ^D

Now, the system must be fully encapsulate by SVM again. Please refer to online Sun Documentation, or some past entries on this subject, depending on the system's architecture: SPARC systems, or x86 platforms.

Update A Corrupted GRUB Boot Archive, Without SVM

Julien Gabel — Sun, 09 Mar 2008 18:32:00 +0100

Solaris 10 systems on x86 architecture use the GNU GRand Unified Bootloader (GRUB) which is the boot loader responsible for loading a boot archive into a system's memory. The boot archive is a collection of critical files (kernel modules and configuration files) that are required to boot the Solaris OS. As stated in the Sun documentation:

These files are needed during system startup before the root file system is mounted. Two boot archives are maintained on a system:

The boot archive that is used to boot the Solaris OS on a system. This boot archive is sometimes called the primary boot archive.

The boot archive that is used for recovery when the primary boot archive is damaged. This boot archive starts the system without mounting the root file system. On the GRUB menu, this boot archive is called failsafe. The archive's essential purpose is to regenerate the primary boot archive, which is usually used to boot the system.

The Solaris OS generally keeps the boot archive properly synchronized on its own. Sometimes, the boot archive gets corrupted--for example when (bad) patches are applied, or the the operating system crashed. In these cases, the boot archive must be regenerated. This is easily accomplished following the Sun documentations x86: How to Boot the Failsafe Archive for Recovery Purposes, and x86: How to Boot the Failsafe Archive to Forcibly Update a Corrupt Boot Archive. The main drawback is when the system is encapsulated under a SVM mirror (RAID-1) since the md driver is not managed under the failsafe mode. Please refer to this blog entry on this subject, if needed.

Create And Remove A Remote Printer Queue (CLI)

Julien Gabel — Wed, 05 Mar 2008 15:04:00 +0100

You can easily create and remove a remote printer queue using the BSD type spooler. You just have to fill the configuration file /tmp/lp.list properly, i.e. provide the local printer name, the remote LPD server, and the remote printer queue:

# cat << EOF > /tmp/lp.list
locname1 lpdserv1 remname1
locname2 lpdserv2 remname2
EOF

Then, just run the appropriate script depending of the desired behavior. Follow, an example when removing the two queues:

# cat << EOF > /tmp/lp.remove
#!/usr/bin/env sh

for lplocal in `awk '{print $1}' /tmp/lp.list`; do
  /usr/sbin/lpshut
  /usr/bin/cancel ${lplocal} -e 2> /dev/null
  /usr/sbin/lpadmin -x${lplocal}
  /usr/sbin/lpsched -v
  sleep 1
done

exit 0
EOF
# sh /tmp/lp.remove
scheduler stopped
scheduler is running
scheduler stopped
scheduler is running
# lpstat -olocname1
no system default destination
lpstat: "locname1" not a request id or a destination

And now, the creation:

# cat << EOF > /tmp/lp.create
#!/usr/bin/env sh

while read lp; do
  eval set -- `IFS=" "; printf '"%s" ' ${lp}`
  lplocal="$1"
  lpserver="$2"
  lpremote="$3"

  /usr/sbin/lpshut
  /usr/sbin/lpadmin -p${lplocal} -orm${lpserver} -orp${lpremote} \
   -mrmodel -v/dev/null -orc -ob3 -ocmrcmodel -osmrsmodel
  /usr/sbin/accept ${lplocal}
  /usr/bin/enable ${lplocal}
  /usr/sbin/lpsched -v
  sleep 1
done < /tmp/lp.list

exit 0
EOF
# sh /tmp/lp.create
scheduler stopped
destination "locname1" now accepting requests
printer "locname1" now enabled
scheduler is running
scheduler stopped
destination "locname2" now accepting requests
printer "locname2" now enabled
scheduler is running
# lpstat -olocname1
no system default destination
printer queue for locname1
                         Windows LPD Server
                   Printer \\lpdserv1
emname1
Owner       Status         Jobname          Job-Id    Size   Pages  Priority
----------------------------------------------------------------------------
hostname: locname1: ready and waiting
no entries

That's it!

LVM2 Simple Mirroring On RHEL4

Julien Gabel — Fri, 15 Feb 2008 13:25:00 +0100

When the need to evacuate all persistent SAN storage from EMC DMX1K to HP XP12K (HDS), three main solutions were envisaged. The first one was brute data copy (tar, cpio, etc.) but was not very practical with the size of the data (multi-terabytes) and the time involved in copying them. The two others were based on LVM technologies: mirroring, or moving.

Although the choice has been to use the online and transparent moving data technology (see pvmove for more information), it was interesting to note that Red Hat has backported support for the creation and manipulation of simple mirrors to their RHEL4 distribution. These functionalities were introduced with the RHBA-2006:0504-15 advisory issued on 2006-08-10, i.e. between RHEL4 Update 4 and RHEL4 Update 5 (and so available via RHN at this time). It is just too bad that the online help for LVM commands are not properly synchronized nor fully documented by the corresponding manual page: clearly, this doesn't help to use them in the best conditions (no, Google isn't always the better option when using these kinds of functionalities in big companies).

Deleting SCSI Device Paths For A Multipath SAN LUN

Julien Gabel — Sat, 09 Feb 2008 12:28:00 +0100

When releasing a multipath device under RHEL4, different SCSI devices corresponding to different paths must be cleared properly before removing the SAN LUN effectively. When the LUN was delete before to clean up the paths at the OS level, it is always possible to remove them afterwards. In the following example, it is assume that the freeing LVM manipulations were already done, and that the LUN is managed by EMC PowerPath.

First, get and verify the SCSI devices corresponding to the multipath LUN:

# grep "I/O error on device" /var/log/messages | tail -2
Feb  4 00:20:47 beastie kernel: Buffer I/O error on device sdo, \
 logical block 12960479
Feb  4 00:20:47 beastie kernel: Buffer I/O error on device sdp, \
 logical block 12960479
# powermt display dev=sdo
Bad dev value sdo, or not under Powerpath control.
# powermt display dev=sdp
Bad dev value sdp, or not under Powerpath control.

Then, get the appropriate scsi#:channel#:id#:lun# informations:

# find /sys/devices -name "*block" -print | \
 xargs \ls -l | awk -F\/ '$NF ~ /sdo$/ || $NF ~ /sdp$/ \
 {print "HBA: "$7"\tscsi#:channel#:id#:lun#: "$9}'
HBA: host0      scsi#:channel#:id#:lun#: 0:0:0:9
HBA: host0      scsi#:channel#:id#:lun#: 0:0:1:9

When the individual SCSI paths are known, remove them from the system:

# echo 1 > /sys/bus/scsi/devices/0\:0\:0\:9/delete
# echo 1 > /sys/bus/scsi/devices/0\:0\:1\:9/delete
# dmesg | grep "Synchronizing SCSI cache"
Synchronizing SCSI cache for disk sdp:
Synchronizing SCSI cache for disk sdo:

Tuning Is Evil

Julien Gabel — Sun, 23 Dec 2007 21:48:00 +0100

Each month, I hear many coworkers or specific application management teams asking about putting some system tunings in place, even on very recent operating system releases. All the time. Most of these settings comes from the Internet, are found in forum posts, or articles related to a subsystem, or in technical publications. And some of them comes from third party software providers, or editors. A very, very few settings are proposed or recommended by system administrators, or by knowledgeable people in tuning area.

The problem is that, most of the time, these tunings are related to another release of the operating system, are not updated to keep current with the Best Of Practices for a given OS release, or simply are not well understood and not applicable without affecting (badly) current running environments. More, already present tunings are reported as-is on upgraded and fresh installed systems without more thinking, or without be assured these are always applicable (or obsolete) and what are the new defaults (if not dynamic). One of the most representative example today of this is the new System V IPC facilities found from the GA Solaris 10, and later, operating system, where some Oracle DBAs always ask SA team for shared memory settings as found on Solaris 8 systems.

Although extract from the Solaris Internals and Performance FAQ for ZFS, here is a great excerpt we all must read carefully and try to keep in mind when modifying default behavior of a system:

Tuning is evil and should not be done...in general.

First, consider that the default values are set by the people who know most things about the effects of the tuning. If a better value exists, it would be the default. While alternative values might help a given workload, it could quite possibly degrade some other aspects of performance. Maybe, catastrophically so.

Over time, tuning recommendations might become stale at best or might lead to performance degradations. Customers are leery of changing a tuning that is in place and the net effect is a worse product than what it could be. Moreover, tuning enabled on a given system might spread to other systems, where it might not be warranted at all.

A Process Can Not Be Killed Upon Hanging In sendfilev()

Julien Gabel — Mon, 17 Dec 2007 14:30:00 +0100

Here is a little but annoying bug I faced recently on a production system running Solaris 10 11/06: a process can not be killed.

The KILL signal can't be ignored by a process. The associated handler must be the default--taking the signal into account, and honor it--and it is the case (the incriminated process id is 26632):

# psig 26632 | grep KILL
KILL    default

Nevertheless, the process seems not to stop as expected:

# echo $$
2913
#
# kill -s KILL 26632
#
# dtrace -n 'fbt::sigtoproc:entry /pid == 2913/ {trace(arg2); trace(execname);}'
dtrace: description 'fbt::sigtoproc:entry ' matched 1 probe
CPU     ID                    FUNCTION:NAME
 18  16387                  sigtoproc:entry                 9  tcsh
#
# ps -ef | awk '$2 ~ /26632/ {print $0}'
nonpriv 26632     1   0   Nov 13 ?         0:01 /usr/sbin/in.ftpd -a

This not the correct behavior since a pending signal is not honored:

# pflags 26632
26632:  /usr/sbin/in.ftpd -a
       data model = _ILP32  flags = ORPHAN|MSACCT|MSFORK
       sigpend = 0x00006100,0x00000000
/1:    flags = 0

More, the process seems in a non-coherent system state: it is now nearly impossible to trace it using, truss, the proctools,...

# truss -alef -p 26632
truss: unanticipated system error: 26632
#
# pstack 26632
pstack: cannot examine 26632: unanticipated system error
#
# pfiles 26632
pfiles: unanticipated system error: 26632
#
# pldd 26632
pldd: cannot examine 26632: unanticipated system error

... or helped by DTace:

# dtrace -n 'profile-1000hz /pid ==26632/ { @[stack()] = count() }'
dtrace: description 'profile-1000hz ' matched 1 probe
^C
#
# dtrace -n 'profile-1000hz /pid ==26632/ { @[ustack()] = count() }'
dtrace: description 'profile-1000hz ' matched 1 probe
^C

Follow some information gathered through mdb in kernel debugging mode:

> ::status
debugging live kernel (64-bit) on socrate
operating system: 5.10 Generic_118833-36 (sun4u)
>
> ::showrev
Hostname: socrate
Release: 5.10
Kernel architecture: sun4u
Application architecture: sparcv9
Kernel version: SunOS 5.10 sun4u Generic_118833-36
Platform: SUNW,Sun-Fire-V490
>
> ::pgrep in.ftpd
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R  26632      1    261    261  30501 0x5a024b00 00000600144b0c28 in.ftpd
>
> 00000600144b0c28::kill
mdb: command is not supported by current target
>
> 00000600144b0c28::thread
            ADDR    STATE    FLG PFLG SFLG   PRI  EPRI PIL             INTR
00000600144b0c28 inval/2000 157e 5778    0     0     0   6                2
>
> 00000600144b0c28::walk thread | ::findstack
stack pointer for thread 30002aba360: 2a1022e7ec1
[ 000002a1022e7ec1 cv_wait+0x38() ]
  000002a1022e7f71 page_lock_es+0x204()
  000002a1022e8021 pvn_vplist_dirty+0x2a4()
  000002a1022e8101 nfs_putpages+0x124()
  000002a1022e81c1 nfs3_putpage+0xcc()
  000002a1022e8271 fop_putpage+0x1c()
  000002a1022e8321 nfs_purge_caches+0xe4()
  000002a1022e83d1 nfs_attr_cache+0x20c()
  000002a1022e8481 nfs3_getattr_otw+0x1b8()
  000002a1022e85f1 nfs3_validate_caches+0x4c()
  000002a1022e8731 nfs3_getpage+0xa4()
  000002a1022e8861 fop_getpage+0x44()
  000002a1022e8931 segmap_getmapflt+0x588()
  000002a1022e8a41 snf_segmap+0x13c()
  000002a1022e8bc1 sosendfile64+0x298()
  000002a1022e8d21 sendvec64+0xf8()
  000002a1022e8f61 sendfilev+0x178()
  000002a1022e92e1 syscall_trap32+0xcc()

We can see the use of the sendfilev() syscall: it appears to be a known bug around this (you must have a registered Sun support customer account to be able to view the second document).

The good news is the root cause is already fix in the development branch of the operating system. The bad news is, after opening a call to the Sun support team, that no patch will be released in a near future to correct this: the incorporation of the fix is currently planned for the next major Solaris 10 Update (Update 5) which is scheduled for summer 2008. However, the fix is already available via the Solaris Express program, if that is applicable to your environment. As a last note, and if the non-killable process owns a resource necessary to another program, there seems no other option than to plan a reboot of the system.

Nifty Tool For Querying Heterogeneous SCSI Devices

Julien Gabel — Sat, 01 Dec 2007 17:45:00 +0100

Lasse Østerild remind us about the EMC inq tool, which is able to query SCSI buses to find a large range of devices, of many sort. This great utility support non-EMC targets, and is freely available (just be aware that the latest link seems not to be updated frequently, so check the latest version yourself in the list).

Here are two examples taken respectively from a Sun Fire V490 UltraSPARC system running Solaris 9...

# ./inq.sol64
Inquiry utility, Version V7.3-845 (Rev 2.0)      (SIL Version V6.4.2.0
(Edit Level 845)
Copyright (C) by EMC Corporation, all rights reserved.
For help type inq -h.
---------------------------------------------------------------------------------------
DEVICE                            :VEND     :PROD             :REV  :SERNUM  :CAP(kb)
---------------------------------------------------------------------------------------
/dev/rdsk/c0t0d0s2                :TSSTcorp :DVD-ROM TS-H352C :SI00 :        :  -----
/dev/rdsk/c1t0d0s2                :FUJITSU  :MAX3147FCSUN146G :1103:0639G02A :143369664
/dev/rdsk/c1t3d0s2                :FUJITSU  :MAX3147FCSUN146G :1103:0638G02A :143369664
/dev/rdsk/c2t50060E8004F2F520d0s2 :HP       :OPEN-V*2         :5007 :500F2F5 :103683840
/dev/rdsk/c3t50060E8004F2F510d6s2 :HP       :OPEN-V*7         :5007 :500F2F5 :362893440
/dev/vx/rdmp/XP12K_SQC_0s2        :HP       :OPEN-V*3         :5007 :500F2F5 :155525760
/dev/vx/rdmp/XP12K_SQC_6s2        :HP       :OPEN-V*7         :5007 :500F2F5 :362893440
/dev/vx/rdmp/c1t0d0s2             :FUJITSU  :MAX3147FCSUN146G :1103:0639G02A :143369664
/dev/vx/rdmp/c1t3d0s2             :FUJITSU  :MAX3147FCSUN146G :1103:0638G02A :143369664
# 
# ./inq.sol64 -hba
Inquiry utility, Version V7.3-845 (Rev 2.0)      (SIL Version V6.4.2.0
(Edit Level 845)
Copyright (C) by EMC Corporation, all rights reserved.
For help type inq -h.
---------------------------------------------------
HBA name:           QLogic Corp.-2200-0
host WWN:           200000144F415386
vendor name:        QLogic Corp.
model:              2200
firmware version:   2.1.144
driver version:     20060630-2.16
serial number:      Unknown
vendor code:        0x144f
HBA type:           Fibre Channel
port count:         1

port number:       1
    port WWN:           210000144F415386
    Port OS name:       /dev/cfg/c1
    port type:          LPORT
    port speed:         1GBIT
    supported speed:    1GBIT
    port state:         ONLINE
    port FCID:          0x1
---------------------------------------------------
HBA name:           Sun Microsystems, Inc.-LP10000-S-1
host WWN:           20000000C957A8E8
vendor name:        Sun Microsystems, Inc.
model:              LP10000-S
firmware version:   1.91a5
driver version:     1.11i (2006.07.11.10.53)
serial number:      0999BG0-0635000219
vendor code:        0xc9
HBA type:           Fibre Channel
port count:         1

port number:       1
    port WWN:           10000000C957A8E8
    Port OS name:       /dev/cfg/c2
    port type:          NPORT
    port speed:         2GBIT
    supported speed:    2GBIT
    port state:         ONLINE
    port FCID:          0x10900
[...]

... and a HP DL585G2 AMD64 system running RHEL4U5. Both are connected to a remote SAN served by a HP XP12K (HDS refurbished) storage system:

# ./inq.LinuxAMD64 -f_powerpath -f_hds
Inquiry utility, Version V7.3-845 (Rev 2.0)      (SIL Version V6.4.2.0
(Edit Level 845)
Copyright (C) by EMC Corporation, all rights reserved.
For help type inq -h.
----------------------------------------------------------------------
DEVICE         :VEND    :PROD            :REV   :SER NUM    :CAP(kb)
----------------------------------------------------------------------
/dev/emcpowere :HP      :OPEN-V*6        :5007  :50 0F2F5   :311051520
/dev/emcpowerf :HP      :OPEN-V*6        :5007  :50 0F2F5   :311051520
/dev/emcpowerg :HP      :OPEN-V*10       :5007  :50 0F2F5   :307785600
/dev/emcpowerh :HP      :OPEN-V*10       :5007  :50 0F2F5   :307785600
/dev/emcpowerj :HP      :OPEN-V*6        :5007  :50 0F2F5   :307203840
/dev/emcpowerk :HP      :OPEN-V*4        :5007  :50 0F2F5   :206085120

Although this tool works great with an OpenSolaris distribution (say, the Solaris Express family), there appear not to have a x86 declination which is a pity knowing the growing Solaris/OpenSolaris community in the marketplace today. Well, maybe for a next inq release, at least I hope.