blog'o thnet

As a SA, it not uncommon to have regularly requests about big differences between the du and df outputs on a UFS file system. (For ZFS specific considerations, please see the ZFS FAQ.)

The du utility reports the sum of space allocated to all files in the file hierarchy rooted in the directory plus the space allocated to the directory itself. The df utility reports the amount of disk space occupied by a mounted file system.

When a file is remove from the file system, i.e. is unlinked (the hard link count goes to zero), the space belonging to this file is accounted against the du tool, but is not visible to the df utility until all references to it (open file descriptors) are closed. In order to find the guilty process, one can follow the information found in the SunManagers Frequently Asked Questions. Here is an example of such finding, but using a slightly different method to get the process currently holding the open descriptor to the deleted file.

Find the file which has been unlinked through the procfs interface:

# find /proc/*/fd \( -type f -a ! -size 0 -a -links 0 \) -print | xargs \ls -li
 415975 --w-------   0 user  group  2125803025 Oct 15 23:59 /proc/1252/fd/3

Eventually, get more detail about it:

# pargs -c 1252
1252:   rvd.basic -reliability 5 -listen tcp:9876 -logfile /path/to/log/rvd_9876.l
argv[0]: rvd.basic
argv[1]: -reliability
argv[2]: 5
argv[3]: -listen
argv[4]: tcp:9876
argv[5]: -logfile
argv[6]: /path/to/log/rvd_9876.log

Check to see if you can understand what is the content of the unlinked file:

# tail /proc/1252/fd/3
-------------------------------------------------------------------------------
2008-10-15 23:59:32.002116 - [MSG] BBG_Transmitter_class.cc, line 792 (thread 25087:4)
[4060] Sent a heartbeat
-------------------------------------------------------------------------------
BBG_Transmitter_class.cc: [4111] No activity detected. Send a Heartbeat message
-------------------------------------------------------------------------------
2008-10-15 23:59:32.134829 - [MSG] BBG_Transmitter_class.cc, line 1138 (thread 25087:4)
[4065] Heartbeat acknowledged by Bloomberg

You can correlate the size of the removed, but always referenced, file to the space accounted from the du and df tools:

# df -k /path/to
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d5       6017990 5874592   83219    99%    /path/to
# du -sk /path/to
3791632 /data
# echo "(5874616-3791632)*1024" | bc
2132975616

So, we now found the ~2GB log file which was always opened (used) by a process. Now, there are two solutions to be able to get back the freed space:

1. Truncate the unlinked file (quick workaround).
2. Simply restart properly the corresponding program (better option).

Use the solution which fits the best your need in your environment.

As a little follow-up to Fake The hostid Of A Solaris Zone, and regarding the discussion on the capacity to change the hostid of a Solaris non-global zone, it is interesting to mention these (updated) informations:

1. The LD_PRELOAD trick proposed before is not a proper option, and is really ugly (and intrusive if you didn't unset it before continuing the execution of a program).
2. When using Solaris 8 or Solaris 9 Containers, there is a feature called Host ID Emulation from the zonecfg utility which can do exactly that.
3. Before the introduction of the privileges in a non-global zone with Solaris 11/06 (a.k.a. Solaris Update 3), you must run the DTrace zhostid script (daemon) within the global zone. It is not mandatory to run it from the global zone anymore. Using the appropriate dtrace_user privilege only, you can run it directly from the non-global zone:

   # zonecfg -z ngzone set limitpriv=default,dtrace_user
   # zoneadm -z ngzone boot
   # zlogin ngzone
   [Connected to zone 'ngzone' pts/5]
   Last login: Sat Oct  4 18:57:17 on pts/5
   Sun Microsystems Inc.   SunOS 5.11      snv_99  November 2008
   # /sbin/zonename 
   ngzone
   # /usr/bin/hostid
   837d47dd
   # ./zhostid &
   [1] 21506
   # /usr/bin/hostid
   20a82f32
   # ^D
   [Connection to zone 'ngzone' pts/5 closed]
   

Although this seems a little bit confident, the long-awaited Update 6 to the Solaris 10 operating system release is just behind the door. This release (scheduled to be available in mid-October) will includes virtualization enhancements including the ability for a Solaris Container to automatically update its environment when moved from one system to another, Logical Domains support for dynamically reconfigurable disk and network I/O, and paravirtualization support when Solaris 10 is used as a guest OS in Xen-based environments such as Sun xVM Server. Solaris 10 10/08 also includes support for the latest systems from Sun and other vendors, such as those based on the Intel Xeon Processor 7400 Series.

This release will be the very first Solaris release to be able to boot from ZFS and use it as their root file system, such as what can be found on OpenSolaris or Solaris Express Community Release today.

Check the What’s New web page for Solaris, and consult the Solaris Media Gallery videos for more information.

Update #1 (2008-10-14): Don't forget to consult the must read What's New in Solaris 10 10/08? from the San Antonio OpenSolaris User Group.

Update #2 (2008-10-31): Get yours, and go reading the official What's New in the Solaris 10 10/08 Release page.

Because we continue to encounter some specialized software vendor which can't tell if there is a problem with the Daylight Saving Time change for their application, the need to test the time adjustment beforehand arise sometimes. In this case, the first thing which comes in mind is to change the system clock, without modifying the timezone. Although this can do the job, it doesn't test the DST adjustment properly, and affect the overall operating environment, system-wide.

To do things cleaner, we can try to modify directly the timezone in use. This will test the real DST automatic adjustment, while not changing the system clock (and not impacting other software, or services). Say we want to change DST for Europe/Paris time zone one day before the official date for summer 2008. First, obtain the current setting:

# zdump -v `rtc` | \
   awk '$6 ~ /'"`date '+%Y'`"'/ && $12 !~ /'"`date '+%H:%M:%S'`"'/ {print $0}'
Europe/Paris  Sun Mar 30 00:59:59 2008 UTC = Sun Mar 30 01:59:59 2008 CET isdst=0
Europe/Paris  Sun Mar 30 01:00:00 2008 UTC = Sun Mar 30 03:00:00 2008 CEST isdst=1
Europe/Paris  Sun Oct 26 00:59:59 2008 UTC = Sun Oct 26 02:59:59 2008 CEST isdst=1
Europe/Paris  Sun Oct 26 01:00:00 2008 UTC = Sun Oct 26 02:00:00 2008 CET isdst=0

Then, get the time zone source file for the Europe/Paris geographic definition and adapt it writing an exception rule for the current year. Because the current definition for Europe/Paris is based on the EU rule, our modification will be based on this rule name.

# mkdir -p /tmp/zoneinfo/src
# cp /usr/share/lib/zoneinfo/src/europe /tmp/zoneinfo/src
# diff /usr/share/lib/zoneinfo/src/europe /tmp/zoneinfo/src/europe
1073a1074,1076
> # Test the DST adjustment one day in advance, for March 2008 only.
> # Rule        NAME    FROM    TO      TYPE    IN      ON      AT      SAVE    LETTER/S
> Rule          EU      2008    only    -       Mar     29       2:00   1:00    S

Last, compile the updated time zone definition, put it in place, and verify the new DST date.

# zic -d /tmp/zoneinfo /tmp/zoneinfo/src/europe
# cp /tmp/zoneinfo/Europe/Paris /usr/share/lib/zoneinfo/Europe
# zdump -v `rtc` | \
   awk '$6 ~ /'"`date '+%Y'`"'/ && $12 !~ /'"`date '+%H:%M:%S'`"'/ {print $0}'
Europe/Paris  Sat Mar 29 00:59:59 2008 UTC = Sat Mar 29 01:59:59 2008 CET isdst=0
Europe/Paris  Sat Mar 29 01:00:00 2008 UTC = Sat Mar 29 03:00:00 2008 CEST isdst=1
Europe/Paris  Sun Oct 26 00:59:59 2008 UTC = Sun Oct 26 02:59:59 2008 CEST isdst=1
Europe/Paris  Sun Oct 26 01:00:00 2008 UTC = Sun Oct 26 02:00:00 2008 CET isdst=0

Once the fake DST adjustment is validated at the software level, clean things up a little.

# zic /usr/share/lib/zoneinfo/src/europe
# rm -r /tmp/zoneinfo