blog'o thnet

To content | To menu | To search

Wednesday 7 March 2007

Stale NFS File Handle

How to decode this cryptic NFS write error a Solaris client system, found in the /var/adm/messages log file?

Feb 26 11:54:26 clssunp120 nfs: [ID 626546 kern.notice] NFS write error on \
 host nfs1_prd: Stale NFS file handle.
Feb 26 11:54:26 clssunp120 nfs: [ID 702911 kern.notice] (file handle: 40260001 \
 ffffffff a0000 1dc24 d5 a0000 2 0)

The problem is that the file handle specification is used and interpreted internally by the NFS client subsystem, and evolved with each release of the operating system. Luckily, we need only two fields (on the eight printed in the record file). The first field is the file system id (or device id) and is generally the first number of the file handle. The second field of interest is the inode number, which is found at the four, or five reference in the file handle (please note that the inode is reported in hexadecimal format).

So, consider the following file handle:

40260001 ffffffff a0000 1dc24 d5 a0000 2 0

We can now know which file system is the culprit:

# grep 40260001 /etc/mnttab
nfssrv:/t/tools/SunOS/isa /tools/isa nfs rw,intr,soft,dev=40260001 1172569690

Translate the inode number to decimal, to be used in conjunction with the find(1) utility:

# echo "ibase=16; `echo 1dc24 | tr [:lower:] [:upper:]`" | bc
121892

Last, the file name a process is currently trying to reference can be found easily with:

# find /tools/isa -mount -inum 121892 -print 2> /dev/null

Search for the info doc 73152 on the SunSolve web site for more information in this subject (you must have a registered Sun support customer account to be able to view this document).

Thursday 11 January 2007

NFS and ZFS plus ZIL Interesting Notes

I recently learn about NFS on ZFS interaction problem reading the great blog of Ben Rockwood. Although not directly related to what he encountered, this recent great post about how NFS behaves with ZFS backend, particularly on the performance comparison front, says a lot of things about why you might see poor performance using this two technologies together.

To go deeper on this front, you can read more about the ZIL purpose on Eric Kustarz's weblog, and follow closely this ZFS thread on the OpenSolaris website.

Wednesday 31 August 2005

Use the NIS and NFS Infrastructure on Red Hat Advanced Server 2.1

Here are the steps to be able to use the current NIS and NFS infrastructure from a Linux server.

NIS

Be sure to resolve the NIS servers (slave and/or master) for the int domain name:

# egrep "nasty|bigup" /etc/hosts
192.168.4.74           nasty
192.168.4.23           bigup

Configure the NIS client:

# cat << EOF >> /etc/yp.conf
domain int server nasty
domain int server nasty
EOF
# grep NIS /etc/sysconfig/authconfig /etc/sysconfig/network
/etc/sysconfig/authconfig:USENIS=yes
/etc/sysconfig/network:NISDOMAIN=int

NFS

The NFS part is relatively simple since the autofs maps is looked up in the NIS maps (already managed by the corresponding boot script's service).

So, it is just needed to modify the automountd service to add some arguments that must be passed to the program. This is a necessary step to be able to automount the correct remote path using our customized autofs server. Here is how to do so.

Check the configuration of the run-level informations for the autofs service:

# chkconfig --list autofs
autofs          0:off   1:off   2:off   3:on    4:on    5:on    6:off

Modify the initial service configuration and reload it:

# diff -u /etc/init.d/autofs.orig /etc/init.d/autofs
--- /etc/init.d/autofs.orig     Thu Aug 25 13:12:38 2005
+++ /etc/init.d/autofs  Thu Aug 25 17:25:47 2005
@@ -67,7 +67,10 @@
 # We can add local options here
 # e.g. localoptions='rsize=8192,wsize=8192'
 #
-localoptions=''
+localoptions="-DOSNAME=`uname -s` \
+              -DCPU=x86 \
+              -DNATISA=32 \
+              -DOSREL=`uname -r | awk -F\. '{print $1\".\"$2}'`"
 
 # Daemon options
 # e.g. --timeout 60
# service autofs restart

Verify if all is ok:

# service autofs status
Configured Mount Points:
------------------------
/usr/sbin/automount /Soft yp auto.soft -ro,hard,bg,intr -DOSNAME=Linux        -DCPU=x86        -DNATISA=32        -DOSREL=2.4
/usr/sbin/automount /NTFS yp auto.nt  -DOSNAME=Linux        -DCPU=x86        -DNATISA=32        -DOSREL=2.4
/usr/sbin/automount /Home yp auto.home -rw,hard,bg,intr -DOSNAME=Linux        -DCPU=x86        -DNATISA=32        -DOSREL=2.4
/usr/sbin/automount /Apps yp auto.apps -ro,hard,bg,intr -DOSNAME=Linux        -DCPU=x86        -DNATISA=32        -DOSREL=2.4
/usr/sbin/automount /- yp auto.direct  -DOSNAME=Linux        -DCPU=x86        -DNATISA=32        -DOSREL=2.4

Active Mount Points:
--------------------
/usr/sbin/automount /Soft yp auto.soft -ro,hard,bg,intr -DOSNAME=Linux -DCPU=x86 -DNATISA=32 -DOSREL=2.4
/usr/sbin/automount /NTFS yp auto.nt -DOSNAME=Linux -DCPU=x86 -DNATISA=32 -DOSREL=2.4
/usr/sbin/automount /Home yp auto.home -rw,hard,bg,intr -DOSNAME=Linux -DCPU=x86 -DNATISA=32 -DOSREL=2.4
/usr/sbin/automount /Apps yp auto.apps -ro,hard,bg,intr -DOSNAME=Linux -DCPU=x86 -DNATISA=32 -DOSREL=2.4
/usr/sbin/automount /- yp auto.direct -DOSNAME=Linux -DCPU=x86 -DNATISA=32 -DOSREL=2.4

Monday 29 August 2005

How to Set Up the NFS Server, for the Use of the Automounter Capability

  1. soul is the hostname of the master NIS server for a given domain
  2. kitchen is the hostname of the slave NIS server for a given domain
  3. bento is the hostname of the NFS server
  4. nfs is an alias (CNAME) for the NFS server

NFS configuration, logged on bento

Just add the wanted share in the export list on server bento. On Sun Solaris, it is done as follow:

# cat << EOF >> /etc/dfs/dfstab
share -F nfs -o rw -d "Applications repository, NFS shared" /export/bento/apps
EOF

The file system /export/bento/apps must already be available.

Start the nfsd service (and associated daemons: mountd, statd and lockd) if necessary:

# /etc/init.d/nfs.server start

Or simply export the new share:

# shareall

NIS configuration, logged on soul

Because all of the automounted NFS clients (in particular Solaris, AIX and GNU/Linux) know how to use automount maps shared via the NIS protocol, the centralized auto_apps (SYSV style) map resides on the master NIS server, soul.

Since this is a new map to add, there is a little more to do here.

Create the map in the DIR (in NIS parlance) directory:

# touch /etc/ivyp/auto_apps
# ci -i /etc/ivyp/auto_apps
/*
 * Description message: "Automounter map for the applications repository,
 * NFS shared.".
 */
# co -l /etc/yp/auto_apps
# cat << EOF > /etc/ivyp/auto_apps
* -rw,hard,bg,intr nfs:/export/bento/apps/${OSNAME}/${CPU}/${NATISA}/${OSREL}/?
EOF
# ci -u /etc/ivyp/auto_apps

The RCS commands may (and must) be used through the ivv.sh wrapper script.

Teach the customized NIS infrastructure (based on GNU tools, not the native ones) about the new map:

# diff -u /var/yp/GNUmakefile.2005-08-25 /var/yp/GNUmakefile
--- /var/yp/GNUmakefile.2005-08-25      Thu Aug 25 13:18:31 2005
+++ /var/yp/GNUmakefile Thu Aug 25 14:30:04 2005
@@ -76,6 +76,7 @@
 AUTONT=auto_nt
 AUTOALT=auto_alt
 AUTODIRECT=auto_direct
+AUTOAPPS=auto_apps
 else
 AUTOMASTER=auto.master
 AUTOHOME=auto.home
@@ -83,6 +84,7 @@
 AUTONT=auto.nt
 AUTOALT=auto.alt
 AUTODIRECT=auto.direct
+AUTOAPPS=auto.apps
 endif
 GNUGREP=/usr/local/bin/ggrep
 MALIASES=$(YPDBDIR)/$(DOM)/mail.aliases
@@ -96,7 +98,7 @@
 
 all:: passwd group hosts ethers networks rpc services protocols \
        netgroup bootparams aliases publickey netid netmasks c2secure \
-       timezone auto.master auto.home auto.soft auto.nt auto.alt auto.direct \
+       timezone auto.master auto.home auto.soft auto.nt auto.alt auto.apps auto.direct \
        printcap dmmo printers.conf virtualip
 
 diams: dmmo_applis dmmo_sites dmmo_users ypservers
@@ -424,6 +426,22 @@
                echo "couldn't find $(DIR)/auto.alt"; \
        fi
 
+auto.apps.time:  $(DIR)/$(AUTOAPPS)
+       -@if [ -f $(DIR)/$(AUTOAPPS) ]; then \
+               sed -e "/^#/d" -e s/#.*$$// $(DIR)/$(AUTOAPPS) \
+               | $(MAKEDBM) - $(YPDBDIR)/$(DOM)/auto.apps; \
+               touch auto.apps.time; \
+               echo "updated auto.apps"; \
+               if [ ! $(NOPUSH) ]; then \
+                       $(YPPUSH) auto.apps; \
+                       echo "pushed auto.apps"; \
+               else \
+               : ; \
+               fi \
+       else \
+               echo "couldn't find $(DIR)/auto.apps"; \
+       fi
+
 printcap.time: $(DIR)/ypprintcap
        -@if [ -f $(DIR)/yp$(@:.time=) ]; then \
                sed < $(DIR)/yp$(@:.time=) \
@@ -686,6 +704,7 @@
 auto.soft: auto.soft.time
 auto.nt: auto.nt.time
 auto.alt: auto.alt.time
+auto.apps: auto.apps.time
 auto.direct: auto.direct.time
 printcap: printcap.time
 printers.conf: printers.conf.time
@@ -710,6 +729,7 @@
 $(DIR)/$(AUTOSOFT):
 $(DIR)/$(AUTONT):
 $(DIR)/$(AUTOALT):
+$(DIR)/$(AUTOAPPS):
 $(DIR)/$(AUTODIRECT):
 $(DIR)/ypprintcap:
 $(DIR)/printers.conf:

At this stage, the new map must be first manually transferred on the slave NIS server.

Create the corresponding map files (*.dir and *.pag ) only, using:

# cd /etc/ivyp
# gmake -C /var/yp NOPUSH="YES"

Log in on the slave server (a Sun system), for example kitchen :

# /usr/lib/netsvc/yp/ypxfr -h soul auto.apps
# ls -l /var/yp/`domainname`/auto.apps.*
-rw-r--r--   1 root     root           0 Aug 25 18:10 /var/yp/`domainname`/auto.apps.dir
-rw-r--r--   1 root     root        1024 Aug 25 18:10 /var/yp/`domainname`/auto.apps.pag

Note: You can know which hostname is(are) currently set up as slave server(s) using:

# cd /var/yp/`domainname`
# makedbm -u ypservers
YP_LAST_MODIFIED 1072719185
YP_MASTER_NAME soul
kitchen
timal

Then, on the master server, really push the map this time:

# cd /etc/ivyp
# gmake -C /var/yp

Now, the new mount point and sub-tree must be ready and available to all your properly configured clients, as found in the following online documents:

  1. Use the NIS and NFS Infrastructure on AIX 5L
  2. Use the NIS and NFS Infrastructure on Red Hat Advanced Server 2.1

Saturday 18 June 2005

Use the NIS and NFS Infrastructure on AIX 5L

Here are the steps to be able to use the current NIS and NFS infrastructure from an AIX server:

# cat /etc/resolv.conf  
domain          dev.example.com
nameserver      10.239.208.24
nameserver      10.251.140.96
search          dev.example.com int.example.com prod.example.com
#
# TERM=vt220 smitty
/*
 * Communications Applications and Services
 *  TCP/IP
 *   Further Configuration
 *    Name Resolution
 *     Hosts Table (/etc/hosts)
 *      Add a Host
 *       INTERNET ADDRESS (dotted decimal)               [10.254.234.22]
 *       HOST NAME                                       [neptune.dev.example.com]
 *       ALIAS(ES) (if any - separated by blank space)   [neptune]
 *       COMMENT (if any - for the host entry)           [NIS server for domain devex]
 *  NFS
 *   Network Information Service (NIS)
 *    Configure / Modify NIS
 *     Change NIS Domain Name of this Host
 *      Domain name of this host                        [devex]
 *     Configure this Host as a NIS Client
 *      NIS server - required if there are              [neptune]
 *   Network File System (NFS)
 *    Configure NFS on This System
 *     Start Automounter
 *      PARAMETERS to be used for the automount daemon  [-n]
 */

Launch the automountd at run-level #2:

# cat << EOF > /etc/rc.d/rc2.d/Sautomountd
#!/usr/bin/env ksh
#################################################################
# name: {K|S}automountd
# purpose: script that will start or stop the automountd service.
#################################################################

case "$1" in
start)
  /usr/sbin/automount -n
  ;;
stop)
  stopsrc -g autofs
  ;;
*)
  echo "Usage: $0 {start|stop}"
  exit 1
esac

exit 0
EOF
# ln /etc/rc.d/rc2.d/Sautomountd /etc/rc.d/rc2.d/Kautomountd
# chmod 754 /etc/rc.d/rc2.d/?automountd

In the same time, modify the automountd service to add some arguments that must be passed to the program. This is a necessary step to be able to automount the correct remote path using our customized autofs server. Here is how to do so:

# chssys -s automountd -a "-DOSNAME=`uname -s` -DCPU=`uname -p` -DNATISA=`bootinfo -K` -DOSREL=`uname -v`.`uname -r`"
# stopsrc -g autofs
# /usr/sbin/automount -n

Very important

To resolve information correctly, it was needed to explicitly specify the ordering of name resolution and hosts setting in /etc/netsvc.conf. This file corresponds to /etc/nsswitch.conf under Solaris, GNU/Linux or the BSDs for hosts name resolution. For example:

# cat << EOF >> /etc/netsvc.conf
hosts = local, nis, bind
EOF