blog'o thnet

To content | To menu | To search

Tag - SAN

Entries feed

Wednesday 18 July 2012

Update the HBA firmware on Oracle-branded HBAs

Updating the emlxs driver will no longer automatically update the HBA firmware on Oracle-branded HBAs. If an HBA firmware update is required on an Oracle-branded HBA, a WARNING message will be placed in the /var/adm/messages file, such as this one:

# grep emlx /var/adm/messages
[...]
Jul 18 02:37:11 beastie emlxs: [ID 349649 kern.info] [ 1.0340]emlxs0:WARNING:1540: Firmware update required. (A manual HBA reset or link reset (using luxadm or fcadm) is required.)
Jul 18 02:37:15 beastie emlxs: [ID 349649 kern.info] [ 1.0340]emlxs1:WARNING:1540: Firmware update required. (A manual HBA reset or link reset (using luxadm or fcadm) is required.)
[...]

If found, this message is stating that the emlxs driver has determined that the firmware kernel component needs to be updated. To perform this update, execute luxadm -e forcelip on Solaris 10 (or a fcadm force-lip on Solaris 11) against each emlxs instance that reports the message. As stated in the documentation:

This procedure, while disruptive, will ensure that both driver and firmware are current. The force lip will temporarily disrupt I/O on the port. The disruption and firmware upgrade takes approximately 30-60 seconds to complete as seen from the example messages below. The example shows an update is needed for emlxs instance 0 (emlxs0) and emlxs instance 1 (emlxs1), which happens to correlate to the c1 and c2 controllers in this case.

# fcinfo hba-port
HBA Port WWN: 10000000c9e43860
        OS Device Name: /dev/cfg/c1
        Manufacturer: Emulex
        Model: LPe12000-S
        Firmware Version: 1.00a12 (U3D1.00A12)
        FCode/BIOS Version: Boot:5.03a0 Fcode:3.01a1
        Serial Number: 0999BT0-1136000725
        Driver Name: emlxs
        Driver Version: 2.60k (2011.03.24.16.45)
        Type: N-port
        State: online
        Supported Speeds: 2Gb 4Gb 8Gb
        Current Speed: 8Gb
        Node WWN: 20000000c9e43860
HBA Port WWN: 10000000c9e435fe
        OS Device Name: /dev/cfg/c2
        Manufacturer: Emulex
        Model: LPe12000-S
        Firmware Version: 1.00a12 (U3D1.00A12)
        FCode/BIOS Version: Boot:5.03a0 Fcode:3.01a1
        Serial Number: 0999BT0-1136000724
        Driver Name: emlxs
        Driver Version: 2.60k (2011.03.24.16.45)
        Type: N-port
        State: online
        Supported Speeds: 2Gb 4Gb 8Gb
        Current Speed: 8Gb
        Node WWN: 20000000c9e435fe

In order not to interrupt the service, and because MPxIO (native multipathing I/O) is in use, each emlxs instance will be update one after each other.

# date
Wed Jul 18 09:34:11 CEST 2012

# luxadm -e forcelip /dev/cfg/c1

# grep emlx /var/adm/messages
[...]
Jul 18 09:35:48 beastie emlxs: [ID 349649 kern.info] [ 5.0334]emlxs0: NOTICE: 710: Link down.
Jul 18 09:35:53 beastie emlxs: [ID 349649 kern.info] [13.02C0]emlxs0: NOTICE: 200: Adapter initialization. (Firmware update needed. Updating. id=67 fw=6)
Jul 18 09:35:53 beastie emlxs: [ID 349649 kern.info] [ 3.0ECB]emlxs0: NOTICE:1520: Firmware download. (AWC file: KERN: old=1.00a11  new=1.10a8  Update.)
Jul 18 09:35:53 beastie emlxs: [ID 349649 kern.info] [ 3.0EEB]emlxs0: NOTICE:1520: Firmware download. (DWC file: TEST:             new=1.00a4  Update.)
Jul 18 09:35:53 beastie emlxs: [ID 349649 kern.info] [ 3.0EFF]emlxs0: NOTICE:1520: Firmware download. (DWC file: STUB: old=1.00a12  new=2.00a3  Update.)
Jul 18 09:35:53 beastie emlxs: [ID 349649 kern.info] [ 3.0F1D]emlxs0: NOTICE:1520: Firmware download. (DWC file: SLI2: old=1.00a12  new=2.00a3  Update.)
Jul 18 09:35:53 beastie emlxs: [ID 349649 kern.info] [ 3.0F2C]emlxs0: NOTICE:1520: Firmware download. (DWC file: SLI3: old=1.00a12  new=2.00a3  Update.)
Jul 18 09:36:01 beastie emlxs: [ID 349649 kern.info] [ 3.0143]emlxs0: NOTICE:1521: Firmware download complete. (Status good.)
Jul 18 09:36:06 beastie emlxs: [ID 349649 kern.info] [ 5.055E]emlxs0: NOTICE: 720: Link up. (8Gb, fabric, initiator)

# date
Wed Jul 18 09:39:51 CEST 2012

# luxadm -e forcelip /dev/cfg/c2

# grep emlx /var/adm/messages
[...]
Jul 18 09:41:35 beastie emlxs: [ID 349649 kern.info] [ 5.0334]emlxs1: NOTICE: 710: Link down.
Jul 18 09:41:40 beastie emlxs: [ID 349649 kern.info] [13.02C0]emlxs1: NOTICE: 200: Adapter initialization. (Firmware update needed. Updating. id=67 fw=6)
Jul 18 09:41:40 beastie emlxs: [ID 349649 kern.info] [ 3.0ECB]emlxs1: NOTICE:1520: Firmware download. (AWC file: KERN: old=1.00a11  new=1.10a8  Update.)
Jul 18 09:41:40 beastie emlxs: [ID 349649 kern.info] [ 3.0EEB]emlxs1: NOTICE:1520: Firmware download. (DWC file: TEST:             new=1.00a4  Update.)
Jul 18 09:41:40 beastie emlxs: [ID 349649 kern.info] [ 3.0EFF]emlxs1: NOTICE:1520: Firmware download. (DWC file: STUB: old=1.00a12  new=2.00a3  Update.)
Jul 18 09:41:40 beastie emlxs: [ID 349649 kern.info] [ 3.0F1D]emlxs1: NOTICE:1520: Firmware download. (DWC file: SLI2: old=1.00a12  new=2.00a3  Update.)
Jul 18 09:41:40 beastie emlxs: [ID 349649 kern.info] [ 3.0F2C]emlxs1: NOTICE:1520: Firmware download. (DWC file: SLI3: old=1.00a12  new=2.00a3  Update.)
Jul 18 09:41:48 beastie emlxs: [ID 349649 kern.info] [ 3.0143]emlxs1: NOTICE:1521: Firmware download complete. (Status good.)
Jul 18 09:41:53 beastie emlxs: [ID 349649 kern.info] [ 5.055E]emlxs1: NOTICE: 720: Link up. (8Gb, fabric, initiator)

That's it. Lastly, the documentation says:

At this point, the firmware upgrade is complete as indicated by the Status good message above. A reboot is not strictly necessary to begin using the new firmware. But the fcinfo hba-port command may still report the old firmware version. This is only a reporting defect that does not affect firmware operation and will be corrected in a later version of fcinfo. To correct the version shown by fcinfo, a second reboot is necessary. On systems capable of DR, you can perform dynamic reconfiguration on the HBA (via cfgadm unconfigure/configure) instead of rebooting.

For my part, I tried to unconfigure/configure each emlxs instance using cfgadm without a reboot, but this didn't work as expected on Solaris 10. The fcinfo utility still report the old firmware version, seems until the next reboot.

Sunday 1 May 2011

Switching From RDAC To MPIO For DSXX00 SAN Array

Here is a simple procedure switching from a RDAC/fcparray management mode to a MPIO multipath mode for SAN disks presented from an IBM DSXX00 array.

Verification of the current monopath configuration:

# manage_disk_drivers
1: DS4100: currently RDAC; supported: RDAC/fcparray, MPIO
2: DS4300: currently RDAC; supported: RDAC/fcparray, MPIO
3: DS4500: currently RDAC; supported: RDAC/fcparray, MPIO
4: DS4700/DS4200: currently RDAC; supported: RDAC/fcparray, MPIO
5: DS4800: currently RDAC; supported: RDAC/fcparray, MPIO

Listing of the disks from the array:

# fget_config -vA
---dar0---

User array name = 'CUSTOMERSOFT'
dac0 ACTIVE dac5 ACTIVE

Disk     DAC   LUN Logical Drive
utm            127
hdisk7   dac5    6 beastie1_oracle
hdisk14  dac5   13 beastie2_datavg
hdisk15  dac0   14 beastie3_datavg
hdisk2   dac0    1 beastie3_rootvg
hdisk3   dac0    2 beastie4_rootvg
hdisk4   dac5    3 beastie5_rootvg
hdisk5   dac5    4 beastie2_rootvg
hdisk6   dac0    5 bakup
hdisk8   dac0    7 customer1
hdisk9   dac0    8 customer3
hdisk10  dac5    9 customer6
hdisk11  dac0   10 customer14
hdisk12  dac5   11 beastie2_db2
hdisk13  dac0   12 beastie3_scheduler
hdisk16  dac0   15 customer9
hdisk17  dac0   16 customer8

Listing of the disks as seen from the operating system:

# lsdev -Cc disk | grep DS
hdisk2  Available 00-08-02 1814     DS4700 Disk Array Device
hdisk3  Available 00-08-02 1814     DS4700 Disk Array Device
hdisk4  Available 02-00-02 1814     DS4700 Disk Array Device
hdisk5  Available 02-00-02 1814     DS4700 Disk Array Device
hdisk6  Available 00-08-02 1814     DS4700 Disk Array Device
hdisk7  Available 02-00-02 1814     DS4700 Disk Array Device
hdisk8  Available 00-08-02 1814     DS4700 Disk Array Device
hdisk9  Available 00-08-02 1814     DS4700 Disk Array Device
hdisk10 Available 02-00-02 1814     DS4700 Disk Array Device
hdisk11 Available 00-08-02 1814     DS4700 Disk Array Device
hdisk12 Available 02-00-02 1814     DS4700 Disk Array Device
hdisk13 Available 00-08-02 1814     DS4700 Disk Array Device
hdisk14 Available 02-00-02 1814     DS4700 Disk Array Device
hdisk15 Available 00-08-02 1814     DS4700 Disk Array Device
hdisk16 Available 00-08-02 1814     DS4700 Disk Array Device
hdisk17 Available 00-08-02 1814     DS4700 Disk Array Device

Switch to a multipath management for the SAN volumes presented from the DS4700/DS4200 arrays:

# manage_disk_drivers -c 4
DS4700/DS4200 currently RDAC/fcparray
Change to alternate driver? [Y/N] Y
DS4700/DS4200 now managed by MPIO

It is necessary to perform a bosboot before rebooting the system in
order to incorporate this change into the boot image.

In order to change to the new driver, either a reboot or a full
unconfigure and reconfigure of all devices of the type changed
must be performed.

Reboot the system:

# bosboot –a
bosboot: Boot image is 39636 512 byte blocks.

# shutdown –Fr
[...]

Verification of the new multipath configuration:

# manage_disk_drivers
1: DS4100: currently RDAC; supported: RDAC/fcparray, MPIO
2: DS4300: currently RDAC; supported: RDAC/fcparray, MPIO
3: DS4500: currently RDAC; supported: RDAC/fcparray, MPIO
4: DS4700/DS4200: currently MPIO; supported: RDAC/fcparray, MPIO
5: DS4800: currently RDAC; supported: RDAC/fcparray, MPIO

Listing of the disks from the array:

# mpio_get_config -vA
Frame id 0:
    Storage Subsystem worldwide name: 60ab80016253400009786efca
    Controller count: 2
    Partition count: 1
    Partition 0:
    Storage Subsystem Name = 'CUSTOMERSOFT'
        hdisk      LUN #   Ownership          User Label
        hdisk0         1   A (preferred)      beastie3_rootvg
        hdisk1         2   A (preferred)      beastie4_rootvg
        hdisk2         3   B (preferred)      beastie5_rootvg
        hdisk3         4   B (preferred)      beastie2_rootvg
        hdisk4         5   A (preferred)      bakup
        hdisk5         6   B (preferred)      beastie1_oracle
        hdisk6        16   A (preferred)      customer8
        hdisk7         7   A (preferred)      customer1
        hdisk8         8   A (preferred)      customer3
        hdisk9         9   B (preferred)      customer6
        hdisk10       10   A (preferred)      customer14
        hdisk11       11   B (preferred)      beastie2_db2
        hdisk12       12   A (preferred)      beastie3_scheduler
        hdisk13       13   B (preferred)      beastie2_datavg
        hdisk14       14   A (preferred)      beastie3_datavg
        hdisk15       15   A (preferred)      customer9

Listing of the disks as seen from the operating system:

# lsdev -Cc disk | grep DS
hdisk0  Available 06-08-02 MPIO Other DS4K Array Disk
hdisk1  Available 06-08-02 MPIO Other DS4K Array Disk
hdisk2  Available 06-08-02 MPIO Other DS4K Array Disk
hdisk3  Available 06-08-02 MPIO Other DS4K Array Disk
hdisk4  Available 06-08-02 MPIO Other DS4K Array Disk
hdisk5  Available 06-08-02 MPIO Other DS4K Array Disk
hdisk6  Available 06-08-02 MPIO Other DS4K Array Disk
hdisk7  Available 06-08-02 MPIO Other DS4K Array Disk
hdisk8  Available 06-08-02 MPIO Other DS4K Array Disk
hdisk9  Available 06-08-02 MPIO Other DS4K Array Disk
hdisk10 Available 06-08-02 MPIO Other DS4K Array Disk
hdisk11 Available 06-08-02 MPIO Other DS4K Array Disk
hdisk12 Available 06-08-02 MPIO Other DS4K Array Disk
hdisk13 Available 06-08-02 MPIO Other DS4K Array Disk
hdisk14 Available 06-08-02 MPIO Other DS4K Array Disk
hdisk15 Available 06-08-02 MPIO Other DS4K Array Disk
hdisk16 Available 06-08-02 MPIO Other DS4K Array Disk

That's it.

Monday 16 August 2010

Looking For WWN On HP-UX System

Just as a complement to more older entries about SAN storage and how to get specific SAN information, here is the way I generally use when looking for World Wide Node Name, and World Wide Port Name (WWNN, and WWPN respectively) on a HP-UX system.

First, get the driver and instance name of all the HBA installed on the system:

# ioscan -knfC fc
Class     I  H/W Path        Driver   S/W State   H/W Type     Description
===========================================================================
fc        0  0/0/6/0/0/4/0   fcd      CLAIMED     INTERFACE    HP A9784-60002 PCI/PCI-X Fibre Channel FC/GigE Combo Adapter (FC Port 1)
                            /dev/fcd0
fc        1  0/0/10/0/0/4/0  fcd      CLAIMED     INTERFACE    HP A9784-60002 PCI/PCI-X Fibre Channel FC/GigE Combo Adapter (FC Port 1)
                            /dev/fcd1
fc        2  0/0/12/0/0/4/0  fcd      CLAIMED     INTERFACE    HP A9784-60002 PCI/PCI-X Fibre Channel FC/GigE Combo Adapter (FC Port 1)
                            /dev/fcd2
fc        3  0/0/14/0/0/4/0  fcd      CLAIMED     INTERFACE    HP A9784-60002 PCI/PCI-X Fibre Channel FC/GigE Combo Adapter (FC Port 1)
                            /dev/fcd3

Then, get the disk or tape hardware paths to be crossed with the FC information above:

# ioscan -knfC tape
Class     I  H/W Path        Driver   S/W State   H/W Type     Description
===========================================================================
tape      0  0/0/10/0/0/4/0.11.16.255.0.0.1  stape    NO_HW       DEVICE       HP      Ultrium 3-SCSI
                            /dev/rmt/0m             /dev/rmt/0mn            /dev/rmt/c40t0d1BEST    /dev/rmt/c40t0d1BESTn
                            /dev/rmt/0mb            /dev/rmt/0mnb           /dev/rmt/c40t0d1BESTb   /dev/rmt/c40t0d1BESTnb
tape      1  0/0/10/0/0/4/0.11.16.255.0.0.2  stape    NO_HW       DEVICE       HP      Ultrium 3-SCSI
                            /dev/rmt/1m             /dev/rmt/1mn            /dev/rmt/c40t0d2BEST    /dev/rmt/c40t0d2BESTn
                            /dev/rmt/1mb            /dev/rmt/1mnb           /dev/rmt/c40t0d2BESTb   /dev/rmt/c40t0d2BESTnb

Last, you just have to use the fcmsutil on the proper FC device:

# fcmsutil /dev/fcd1

                           Vendor ID is = 0x001077
                           Device ID is = 0x002312
            PCI Sub-system Vendor ID is = 0x00103c
                   PCI Sub-system ID is = 0x0012c7
                               PCI Mode = PCI-X 133 MHz
                       ISP Code version = 3.3.166
                       ISP Chip version = 3
                               Topology = PTTOPT_FABRIC
                             Link Speed = 2Gb
                     Local N_Port_id is = 0x0b0e00
                  Previous N_Port_id is = None
            N_Port Node World Wide Name = 0x50060b00005de769
            N_Port Port World Wide Name = 0x50060b00005de768
            Switch Port World Wide Name = 0x200e00051e5c1c17
            Switch Node World Wide Name = 0x100000051e5c1c17
                           Driver state = ONLINE
                       Hardware Path is = 0/0/10/0/0/4/0
                     Maximum Frame Size = 2048
         Driver-Firmware Dump Available = NO
         Driver-Firmware Dump Timestamp = N/A
                         Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.11.10 /ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Sep 14 2006,17:45:41

That's it.

Saturday 9 February 2008

Deleting SCSI Device Paths For A Multipath SAN LUN

When releasing a multipath device under RHEL4, different SCSI devices corresponding to different paths must be cleared properly before removing the SAN LUN effectively. When the LUN was delete before to clean up the paths at the OS level, it is always possible to remove them afterwards. In the following example, it is assume that the freeing LVM manipulations were already done, and that the LUN is managed by EMC PowerPath.

  1. First, get and verify the SCSI devices corresponding to the multipath LUN:
    # grep "I/O error on device" /var/log/messages | tail -2
    Feb  4 00:20:47 beastie kernel: Buffer I/O error on device sdo, \
     logical block 12960479
    Feb  4 00:20:47 beastie kernel: Buffer I/O error on device sdp, \
     logical block 12960479
    # powermt display dev=sdo
    Bad dev value sdo, or not under Powerpath control.
    # powermt display dev=sdp
    Bad dev value sdp, or not under Powerpath control.
    
  2. Then, get the appropriate scsi#:channel#:id#:lun# informations:
    # find /sys/devices -name "*block" -print | \
     xargs \ls -l | awk -F\/ '$NF ~ /sdo$/ || $NF ~ /sdp$/ \
     {print "HBA: "$7"\tscsi#:channel#:id#:lun#: "$9}'
    HBA: host0      scsi#:channel#:id#:lun#: 0:0:0:9
    HBA: host0      scsi#:channel#:id#:lun#: 0:0:1:9
    
  3. When the individual SCSI paths are known, remove them from the system:
    # echo 1 > /sys/bus/scsi/devices/0\:0\:0\:9/delete
    # echo 1 > /sys/bus/scsi/devices/0\:0\:1\:9/delete
    # dmesg | grep "Synchronizing SCSI cache"
    Synchronizing SCSI cache for disk sdp:
    Synchronizing SCSI cache for disk sdo:
    

Saturday 1 December 2007

Nifty Tool For Querying Heterogeneous SCSI Devices

Lasse Østerild remind us about the EMC inq tool, which is able to query SCSI buses to find a large range of devices, of many sort. This great utility support non-EMC targets, and is freely available (just be aware that the latest link seems not to be updated frequently, so check the latest version yourself in the list).

Here are two examples taken respectively from a Sun Fire V490 UltraSPARC system running Solaris 9...

# ./inq.sol64
Inquiry utility, Version V7.3-845 (Rev 2.0)      (SIL Version V6.4.2.0
(Edit Level 845)
Copyright (C) by EMC Corporation, all rights reserved.
For help type inq -h.
---------------------------------------------------------------------------------------
DEVICE                            :VEND     :PROD             :REV  :SERNUM  :CAP(kb)
---------------------------------------------------------------------------------------
/dev/rdsk/c0t0d0s2                :TSSTcorp :DVD-ROM TS-H352C :SI00 :        :  -----
/dev/rdsk/c1t0d0s2                :FUJITSU  :MAX3147FCSUN146G :1103:0639G02A :143369664
/dev/rdsk/c1t3d0s2                :FUJITSU  :MAX3147FCSUN146G :1103:0638G02A :143369664
/dev/rdsk/c2t50060E8004F2F520d0s2 :HP       :OPEN-V*2         :5007 :500F2F5 :103683840
/dev/rdsk/c3t50060E8004F2F510d6s2 :HP       :OPEN-V*7         :5007 :500F2F5 :362893440
/dev/vx/rdmp/XP12K_SQC_0s2        :HP       :OPEN-V*3         :5007 :500F2F5 :155525760
/dev/vx/rdmp/XP12K_SQC_6s2        :HP       :OPEN-V*7         :5007 :500F2F5 :362893440
/dev/vx/rdmp/c1t0d0s2             :FUJITSU  :MAX3147FCSUN146G :1103:0639G02A :143369664
/dev/vx/rdmp/c1t3d0s2             :FUJITSU  :MAX3147FCSUN146G :1103:0638G02A :143369664
# 
# ./inq.sol64 -hba
Inquiry utility, Version V7.3-845 (Rev 2.0)      (SIL Version V6.4.2.0
(Edit Level 845)
Copyright (C) by EMC Corporation, all rights reserved.
For help type inq -h.
---------------------------------------------------
HBA name:           QLogic Corp.-2200-0
host WWN:           200000144F415386
vendor name:        QLogic Corp.
model:              2200
firmware version:   2.1.144
driver version:     20060630-2.16
serial number:      Unknown
vendor code:        0x144f
HBA type:           Fibre Channel
port count:         1

port number:       1
    port WWN:           210000144F415386
    Port OS name:       /dev/cfg/c1
    port type:          LPORT
    port speed:         1GBIT
    supported speed:    1GBIT
    port state:         ONLINE
    port FCID:          0x1
---------------------------------------------------
HBA name:           Sun Microsystems, Inc.-LP10000-S-1
host WWN:           20000000C957A8E8
vendor name:        Sun Microsystems, Inc.
model:              LP10000-S
firmware version:   1.91a5
driver version:     1.11i (2006.07.11.10.53)
serial number:      0999BG0-0635000219
vendor code:        0xc9
HBA type:           Fibre Channel
port count:         1

port number:       1
    port WWN:           10000000C957A8E8
    Port OS name:       /dev/cfg/c2
    port type:          NPORT
    port speed:         2GBIT
    supported speed:    2GBIT
    port state:         ONLINE
    port FCID:          0x10900
[...]

... and a HP DL585G2 AMD64 system running RHEL4U5. Both are connected to a remote SAN served by a HP XP12K (HDS refurbished) storage system:

# ./inq.LinuxAMD64 -f_powerpath -f_hds
Inquiry utility, Version V7.3-845 (Rev 2.0)      (SIL Version V6.4.2.0
(Edit Level 845)
Copyright (C) by EMC Corporation, all rights reserved.
For help type inq -h.
----------------------------------------------------------------------
DEVICE         :VEND    :PROD            :REV   :SER NUM    :CAP(kb)
----------------------------------------------------------------------
/dev/emcpowere :HP      :OPEN-V*6        :5007  :50 0F2F5   :311051520
/dev/emcpowerf :HP      :OPEN-V*6        :5007  :50 0F2F5   :311051520
/dev/emcpowerg :HP      :OPEN-V*10       :5007  :50 0F2F5   :307785600
/dev/emcpowerh :HP      :OPEN-V*10       :5007  :50 0F2F5   :307785600
/dev/emcpowerj :HP      :OPEN-V*6        :5007  :50 0F2F5   :307203840
/dev/emcpowerk :HP      :OPEN-V*4        :5007  :50 0F2F5   :206085120

Although this tool works great with an OpenSolaris distribution (say, the Solaris Express family), there appear not to have a x86 declination which is a pity knowing the growing Solaris/OpenSolaris community in the marketplace today. Well, maybe for a next inq release, at least I hope.

Saturday 9 June 2007

Getting Emulex HBA Information on a GNU/Linux System

As a GNU/Linux environment behaves always a little differently as with other UNIX platforms, here is a little sample of what commands I find useful when working with our SAN administrators. In this example, the server is connected using an Emulex Fibre Channel HBA (Host Bus Adapter) and is based on an updated RHEL4U4 system, as can be seen below:

# cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant Update 4)
#
# uname -a
Linux test.example.com 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 \
 x86_64 x86_64 x86_64 GNU/Linux

Here are some informations about the HBA itself. You can see--respectively--the HBA description, the firmware revision level, the WWNN (World Wide Node Name), the WWPN (World Wide Port Name), the operating system driver version, the associated serial number, and the current speed and link state.

# lspci | grep -i emulex
05:0d.0 Fibre Channel: Emulex Corporation LP9802 Fibre Channel Host Adapter (rev 01)
#
# cat /sys/class/scsi_host/host0/fwrev
1.90A4 (H2D1.90A4)
#
# cat /sys/class/scsi_host/host0/node_name
0x20000000c94f7dd9
#
# cat /sys/class/scsi_host/host0/port_name
0x10000000c94f7dd9
#
# cat /sys/class/scsi_host/host0/lpfc_drvr_version
Emulex LightPulse Fibre Channel SCSI driver 8.0.16.27
#
# cat /sys/class/scsi_host/host0/serialnum
MS54376943
#
# cat /sys/class/scsi_host/host0/speed
2 Gigabit
#
# cat /sys/class/scsi_host/host0/state
Link Up - Ready:
   Fabric

Wednesday 9 November 2005

Complete Step-by-Step Guide to Create, Install and Configure a Fully Fonctional DLPAR

Build the DLPAR on the Hardware Management Console

Log in to the HMC using the Java WSM (Web-based System Manager) client. Then, follow the illustrated guide:

  1. Set the logical name of the partition:
    screenshot #01
  2. Because a load manager is not needed, say so to the creation wizard:
    screenshot #02
  3. Provide a name for the default profile:
    screenshot #03
  4. Time to set the memory allocation sizes:
    screenshot #04
  5. Select the kind of logical partition, which is shared in this case:
    screenshot #05
  6. Choose the desired entitlement capacity settings,...
    screenshot #06a
    ... the shared mode and the use of virtual CPU (or not):
    screenshot #06b
  7. Select the I/O unit(s) and its(their) attribution(s) from the available hardware on the p550:
    screenshot #07
  8. Don't choose I/O pools (its purpose is for Mainframe installation only).
  9. Answer that virtual I/O cards are required and create two Ethernet I/O modules on the two local virtual networks port's ID, as for this one:
    screenshot #09a
    Do the same thing for a SCSI I/O module, using the same port's ID for the local and remote card (on the Virtual I/O Server):
    screenshot #09b
    Be sure to set them to required since the virtual SCSI adapter will host the boot disk and update the number of virtual cards slots, if necessary:
    screenshot #09c
  10. The use of a power management partition is not needed here.
  11. Last, select the optional settings for the profil, e.g. activate the surveillance of the connections:
    screenshot #11

Important note:Don't forget to add a new virtual SCSI I/O module (of server type) on the VIOS in order to connect to the just created VIOC. If you made this modification dynamically on the running VIOS, please report the changes back to the corresponding profile or you will be in big trouble at the next VIOS reboot!

Manage and allocate the Virtual I/O Server resources

The new DLPAR will use one sort of VIOS resource: storage device(s).

Case #1: reuse an already existing virtual adapter

If we want to reuse the virtual adapter provided to the DLPAR, we need to clean things up a little. So, delete the corresponding virtual adapter and its virtual devices attached:

$ lsmap -vadapter vhost3
SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost3          U9113.550.65E3A0C-V3-C8                      0x00000006

VTD                   vhdbromonts00
LUN                   0x8100000000000000
Backing device        hdbromonts00
Physloc                

VTD                   vhdbromonts01
LUN                   0x8200000000000000
Backing device        hdbromonts01
Physloc
$
$ rmdev -dev vhost3 -recursive
vhdbromonts00 deleted
vhdbromonts01 deleted
vhost3 deleted

Verify that the virtual adapter doesn't exists anymore:

$ lsmap -vadapter vhost3
Device "vhost3" is not a Server Virtual SCSI Adapter (SVSA).

If necessary, remove the virtual devices: in this case the virtual devices are some logical volumes but they may be some real storage devices, as with SAN disks. For example:

# rmlv hdbromonts00
Warning, all data contained on logical volume hdbromonts00 will be destroyed.
rmlv: Do you wish to continue? y(es) n(o)? y
rmlv: Logical volume hdbromonts00 is removed.
#
# rmlv hdbromonts01
Warning, all data contained on logical volume hdbromonts01 will be destroyed.
rmlv: Do you wish to continue? y(es) n(o)? y
rmlv: Logical volume hdbromonts01 is removed.

Don't forget to delete and recreate the corresponding virtual adapter on the VIOS via the HMC console, otherwise this will cause problem during the installation phase since the VIOS server will not properly give access to the underlying storage disk(s) or device(s)!

Recreate the device tree on the VIOS:

$ cfgdev
$
$ lsdev -virtual | grep vhost3
vhost3          Available  Virtual SCSI Server Adapter

Case #2: configure a newly created virtual adapter

On the other side, if the virtual adapter is freshly created -- as in the previous example -- it is needed to inform the VIOS operating system about it:

$ cfgdev
$
$ lsdev -virtual | grep vhost6
vhost6          Available  Virtual SCSI Server Adapter
$
$ lsmap -vadapter vhost6
SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost6          U9113.550.65E3A0C-V3-C11                     0x00000000

VTD                   NO VIRTUAL TARGET DEVICE FOUND

Assuming we will use one local SCSI disk space to hold the OS and one SAN disk to host the application data, here are the required steps to configure the VIOS.

Create a logical volume for the OS and be sure to have an unused SAN disk:

$ mklv -lv beastielv rootvg 80
beastielv
$
$ lsdev | grep hdisk15
hdisk15    Available EMC Symmetrix FCP MPIO RaidS

Insert the wanted storage resources in the virtual adapter:

$  mkvdev -vdev beastielv -vadapter vhost6
vtscsi16 Available
$
$  mkvdev  -vdev hdisk15 -vadapter vhost6  
vtscsi17 Available
$
$ lsmap -vadapter vhost6                   
SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost6          U9113.550.65E3A0C-V3-C11                     0x00000000

VTD                   vtscsi16
LUN                   0x8100000000000000
Backing device        beastielv
Physloc                

VTD                   vtscsi17
LUN                   0x8200000000000000
Backing device        hdisk15
Physloc               U787B.001.DNW3897-P1-C4-T1-W5006048448930A41-L2000000000000

Create and configure the Network Install Manager

Log in to the NIM server and set the standalone machine configuration for beastie:

# TERM=vt220 smitty nim
/*
 * Perform NIM Administration Tasks
 *  Manage Machines
 *   Define a Machine
 *    Host Name of Machine                                    [beastie]
 * Perform NIM Software Installation and Maintenance Tasks
 *  Install and Update Software
 *   Install the Base Operating System on Standalone Clients
 *    (Choose: "beastie        machines       standalone")
 *    (Choose: "spot - Install a copy of a SPOT resource")
 *    (Chosse: "spot530     resources       spot")
 *    (Select: "LPP_SOURCE                                    [lpp_source530]")
 *    (Select: "ACCEPT new license agreements?                [yes]")
 *    (Select: "Initiate reboot and installation now?         [no]")
 *    (Select: "ACCEPT new license agreements?                [yes]")
 */

The only required prerequisite is to have a fully working name resolution.

Boot and automatically network install the DLPAR

Start the DLPAR console via the HMC administration station: activate it and boot it using the SMS mode.

Then, simply configure the network stack of the Initial Program Load to be able to netboot the partition and remotely install it:

/*
 *  Steps:
 *   2.  Setup Remote IPL (Initial Program Load)
 *    2.   Interpartition Logical LAN      U9113.550.65E3A0C-V8-C3-T1  72ee80008003
 *     1.  IP Parameters
 *      1.  Client IP Address                    [10.126.213.196]
 *      2.  Server IP Address                    [10.126.213.193]
 *      3.  Gateway IP Address                   [000.000.000.000]
 *      4.  Subnet Mask                          [255.255.255.000]
 *     2.  Adapter Configuration
 *      2.  Spanning Tree Enabled
 *       2.  No  <===
 *     3.  Ping Test
 *      1.  Execute Ping Test
 *   5.  Select Boot Options
 *    1.  Select Install/Boot Device
 *     7.  List all Devices
 *      2.        -      Virtual Ethernet
 *                       ( loc=U9113.550.65E3A0C-V2-C3-T1 )
 *      2.  Normal Mode Boot
 *      1.  Yes
 */

Notes:

  1. 10.126.213.191 represents the IP address of the DLPAR
  2. 10.126.213.193 represents the IP address of the NIM server (bootp and tftp servers)

Then select the default console device, the installation language (English) and follow the illustrated guide:

  1. In the proposed screen, choose to modify the default installation settings...
    consoleshot #01
  2. ... in the optional choice...
    consoleshot #02
  3. ... and for additional software:
    consoleshot #03
  4. Select the Server package:
    consoleshot #04
  5. Choose "Install with the settings listed above.".
  6. Verify the overall selection...
    consoleshot #06
  7. ... and happily wait for the installation to proceed:
    consoleshot #07

Configuration steps on the DLPAR

Well. The DLPAR beastie is now installed and basically configured but lacks a lot of tools and none of the network infrastructure is currently available from it (DNS, NIS, NFS, etc.).

Here is what can be done to render this machine (a little) more usable.

Internal IP address (high inter-partition bandwidth)

As with the Inter-Domain Network (IDN) found on an E10K from Sun Microsystems, it may be interesting to use an internal LAN to communicate between LPARs using a high-bandwidth network. So, just pick up an unused IP address from the chosen private range and apply the settings to the correct network interface (e.g. configured with the right network ID):

# chdev -l en0 -a netaddr='193.168.10.5' -a netmask='255.255.255.0' -a state=up

Allocate data storage space

Applications and users data are kept outside the OS space and reside in their own volume group. All the data are hosted on SAN disk as provided via the VIOS:

# lsdev | grep hdisk
hdisk0     Available       Virtual SCSI Disk Drive
hdisk1     Available       Virtual SCSI Disk Drive
#
# lsvg
rootvg
#
# lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            639         468         127..85..00..128..128
#
# mkvg -y beastievg hdisk1
beastievg
#
# lsvg -p beastievg
beastievg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk1            active            539         539         108..108..107..108..108

Seems ready to put real stuff here right now. Later, if new disk will be added through the VIOS, simply add the new virtual SCSI disk to the volume group:

# cfgmgr
#
# lsdev | grep hdisk   
hdisk0     Available       Virtual SCSI Disk Drive
hdisk1     Available       Virtual SCSI Disk Drive
hdisk2     Available       Virtual SCSI Disk Drive
#
# extendvg beastievg hdisk2
0516-1254 extendvg: Changing the PVID in the ODM.
#
# lsvg -p beastievg
beastievg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk1            active            539         539         108..108..107..108..108
hdisk2            active            539         539         108..108..107..108..108

Update the operating system to the latest Maintenance Level (e.g. 5.3ML3 at the time of this writing)

Log in to the NIM server and set the standalone machine configuration for beastie:

# TERM=vt220 smitty nim
/*
 * Perform NIM Software Installation and Maintenance Tasks
 *  Install and Update Software
 *   Update Installed Software to Latest Level (Update All)
 *    (Choose: "beastie        machines       standalone")
 *    (Choose: "53ML3          resources      lpp_source")
 */

Most of the time, these kind of software changes require the system to be rebooted in order for the changes to be made effective:

# shutdown -r now

Attaching the partition to the general network infrastructure (LDAP, NIS, NFS, DNS, etc.)

Please consult the following online documentation: Use the NIS and NFS Infrastructure on AIX 5L.

Modify the maximum number of processes allowed per user

Check for default setting:

# lsattr -D -l sys0 | grep maxuproc
maxuproc        128     Maximum number of PROCESSES allowed per user      True

Then change it accordingly, for example default x 4:

# chdev -l sys0 -a maxuproc=512
sys0 changed
#
# lsattr -El sys0 | grep maxuproc
maxuproc        512                Maximum number of PROCESSES allowed per user      True

Get network time

Assuming that the clock is given by the default gateway network device, we can set and configure the NTP client this way.

Get the default gateway IP address...

# netstat -rn | grep default
default          10.126.213.1      UG        1     2738  en1     -   -      -

... and hard-set the date using ntpdate(8):

# ntpdate 10.126.213.1
30 Sep 18:04:48 ntpdate[282628]: step time server 10.126.213.1 offset -25063.236234 sec

Then, configure the NTP deamon (xntpd) and start the service:

# cp /etc/ntp.conf /etc/ntp.conf.orig
#
# diff -c /etc/ntp.conf.orig /etc/ntp.conf
*** /etc/ntp.conf.orig  Fri Sep 30 18:05:17 2005
--- /etc/ntp.conf       Fri Sep 30 18:05:43 2005
***************
*** 36,41 ****
  #
  #   Broadcast client, no authentication.
  #
! broadcastclient
  driftfile /etc/ntp.drift
  tracefile /etc/ntp.trace
--- 36,42 ----
  #
  #   Broadcast client, no authentication.
  #
! #broadcastclient
! server 10.126.213.1
  driftfile /etc/ntp.drift
  tracefile /etc/ntp.trace
#
# chrctcp -S -a xntpd
0513-059 The xntpd Subsystem has been started. Subsystem PID is 336032.

After some time, be sure that the system get its time from the network device (verify the * in front of the remote node):

# ntpq -pn
     remote           refid      st t when poll reach   delay   offset    disp
==============================================================================
*10.126.213.1    10.126.192.132   3 u    9   64   37     0.85   13.601  877.12

Install and configure OpenSSH and the administrative tool sshd_adm

The aim of this part is to have a tuned configuration of OpenSSH for all clients, and have a specialized configuration for sshd_adm, a second OpenSSH installation which purpose is to be dedicated to administrators and administrative tasks.

Install the OpenSSL RPM package provided by the Linux toolbox for AIX:

# mount -n nim -v nfs /export/lpp_source /mnt
# rpm -ivh /mnt/cd_roms/AIX_Toolbox_for_Linux_Applications_for_Power_Systems_11.2004/RPMS/ppc/openssl-0.9.7d-1.aix5.1.ppc.rpm
openssl                     ##################################################

Then, install the OpenSSH RPM packages, found on the IBM web site (e.g. openssh-3.8.1p1_53.tar.Z for AIX 5.3):

# mkdir /tmp/_sm_inst.$$   /* Put the downloaded package here. */
# zcat /tmp/_sm_inst.$$/openssh-3.8.1p1_`uname -v``uname -r`.tar.Z |  (cd /tmp/_sm_inst.$$ && tar xf -)
# /usr/lib/instl/sm_inst installp_cmd -a -Q -d /tmp/_sm_inst.$$ -f openssh.* -c -N -g -X -G -Y
geninstall -I "a -cgNQqwXY -J"  -Z   -d /tmp/_sm_inst.176290 -f File 2>&1

File:
    I:openssh.base.client          3.8.0.5302
    I:openssh.base.server          3.8.0.5302
    I:openssh.license              3.8.0.5302
    I:openssh.man.en_US            3.8.0.5302
    I:openssh.msg.HU_HU            3.8.0.5302
    I:openssh.msg.RU_RU            3.8.0.5302
    I:openssh.msg.hu_HU            3.8.0.5302
    I:openssh.msg.ru_RU            3.8.0.5302
[...]

Don't forget to clean things up...

# rm -r /tmp/_sm_inst.$$
# umount /mnt

For the configuration steps of the two deamons, please consult the following online documentation: How to Add a New "sshd_adm" Service on AIX 5L.

Install some useful tools commonly used nowadays

The tcsh shell

Log in to the NIM server and set the standalone machine configuration for beastie:

# TERM=vt220 smitty nim
/*
 * Perform NIM Software Installation and Maintenance Tasks
 *  Install and Update Software
 *   Install Software
 *    (Choose: "beastie        machines       standalone")
 *    (Choose: "AIX_Toolbox_for_Linux     resources       lpp_source")
 *     (Select: "coreutils-5.0                            ALL")
 *     (Select: "grep-2.5.1                               ALL")
 *     (Select: "tcsh-6.11                                ALL")
 */

The zip and unzip utilities

Log in to the NIM server and set the standalone machine configuration for beastie:

# TERM=vt220 smitty nim
/*
 * Perform NIM Software Installation and Maintenance Tasks
 *  Install and Update Software
 *   Install Software
 *    (Choose: "beastie        machines       standalone")
 *    (Choose: "AIX_Toolbox_for_Linux     resources       lpp_source")
 *     (Select: "zip-2.3                                  ALL")
 *     (Select: "unzip-5.51                               ALL")
 */

The j2se toolkit

Install the Java 1.4.2 packages in mode 32-bit and 64-bit, found on the IBM web site:

# mkdir /tmp/_sm_inst.$$   /* Put the downloaded packages here. */
# /usr/lib/instl/sm_inst installp_cmd -a -Q -d /tmp/_sm_inst.$$ -f _all_latest -g -X -G -Y
geninstall -I "agQqwXY -J"  -Z   -d /tmp/_sm_inst.299180 -f File 2>&1

File:
    I:Java14.license               1.4.2.0
    I:Java14.sdk                   1.4.2.4
    I:Java14_64.license            1.4.2.0
    I:Java14_64.sdk                1.4.2.3
[...]

Don't forget to clean things up...

# rm -r /tmp/_sm_inst.$$

The sudo security program

Log in to the NIM server and set the standalone machine configuration for beastie:

# TERM=vt220 smitty nim
/*
 * Perform NIM Software Installation and Maintenance Tasks
 *  Install and Update Software
 *   Install Software
 *    (Choose: "beastie        machines       standalone")
 *    (Choose: "AIX_Toolbox_for_Linux     resources       lpp_source")
 *     (Select: "sudo-1.6.7p5                             ALL")
 */

Don't forget to add the hostname beastie to the correct list in the sudoers global site configuration file.

The perl parsing tool

The perl program was compiled from sources, and is available as a shared NFS resource for AIX at:

# df -k /Apps/perl
Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on
XX.XX.XX.XX:/export/nfssrv/apps/AIX/powerpc/64/5.3/perl    35349504  32800724    8%     7885     1% /Apps/perl
#
# /Apps/perl/5.8.7/bin/perl -v

This is perl, v5.8.7 built for aix

The Unix administrative tool lsof (LiSt Open Files)

Log in to the NIM server and set the standalone machine configuration for beastie:

# TERM=vt220 smitty nim
/*
 * Perform NIM Software Installation and Maintenance Tasks
 *  Install and Update Software
 *   Install Software
 *    (Choose: "beastie        machines       standalone")
 *    (Choose: "AIX_Toolbox_for_Linux     resources       lpp_source")
 *     (Select: "lsof-4.61                                ALL")
 */

Configure the OS to boot via a SAN disk (thanks to the VIOS)

So, the operating system is now at the required ML (e.g. 5.3ML3) to support a SAN boot disk via the VIOS. Assuming that all the steps to make available this disk to the VIOC are already done, here is how to fully put this partition onto the SAN.

Information about the corresponding SAN disk provided through the VIOS:

# lscfg -v -l hdisk3
  hdisk3           U9113.550.65E3A0C-V8-C11-T1-L840000000000  Virtual SCSI Disk Drive

Try to put it in the rootvg system volume group:

# extendvg rootvg hdisk3
0516-1254 extendvg: Changing the PVID in the ODM.
0516-1162 extendvg: The Physical Partition Size of 16 requires the creation of
        1078 partitions for hdisk3.  The limitation for volume group rootvg is
        1016 physical partitions per physical volume.  Use chvg command with -t
        option to attempt to change the maximum Physical Partitions per Physical
        volume for this volume group.
0516-792 extendvg: Unable to extend volume group.

As shown, a limitation was reached, especially because of the current settings of the rootvg system volume group. In fact, it is due to the original size of the virtual disk hosting the system file systems... and so the size of the corresponding logical volume on the VIOS, beastielv. Change the rootvg charateristic as proposed and extend the volume group one more time:

# chvg -t 2 rootvg
0516-1193 chvg: WARNING, once this operation is completed, volume group rootvg
        cannot be imported into AIX 430 or lower versions. Continue (y/n) ?
y
0516-1164 chvg: Volume group rootvg changed.  With given characteristics rootvg
        can include upto 16 physical volumes with 2032 physical partitions each.
#
# extendvg rootvg hdisk3

The next step is to evacuate the content of hdisk0 (logical volume via a virtual SCSI disk) to hdisk3 (dedicated SAN disk via a virtual SCSI disk):

# migratepv hdisk0 hdisk3
0516-1011 migratepv: Logical volume hd5 is labeled as a boot logical volume.
0516-1246 migratepv: If hd5 is the boot logical volume, please run 'chpv -c hdisk0'
        as root user to clear the boot record and avoid a potential boot
        off an old boot image that may reside on the disk from which this
        logical volume is moved/removed.
migratepv: boot logical volume hd5 migrated. Please remember to run
        bosboot, specifying /dev/hdisk3 as the target physical boot device.
        Also, run bootlist command to modify bootlist to include /dev/hdisk3.

As explain by the command migratepv, modifiy some boot settings is necessary here and can't be avoided. Clear the boot record on the `old' system disk:

# chpv -c hdisk0

Verify and create a boot image on the newly system disk:

# bosboot -vd hdisk3 && bosboot -ad hdisk3

bosboot: Boot image is 23320 512 byte blocks.

Alter the device boot list to include (new) and remove (old) disks:

# bootlist -m normal -o
hdisk0
# bootlist -m normal hdisk3
# bootlist -m normal -o
hdisk3
#
# ls -ilF /dev/ipldevice /dev/rhdisk3
 1727 crw-------   2 root     system       17,  3 Oct  4 13:01 /dev/ipldevice
 1727 crw-------   2 root     system       17,  3 Oct  4 13:01 /dev/rhdisk3

We are done with this by now. Just clean rootvg, remove hdisk0 and restart the partition:

# reducevg rootvg hdisk0
#
# rmdev -l hdisk0 -Rd
hdisk0 deleted
#
# shutdown -r now

Last, clean the VIOS virtual adapter configuratrion for the partition...

$ rmdev -dev vtscsi16
vtscsi16 deleted
$
$ lsmap -vadapter vhost6
SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost6          U9113.550.65E3A0C-V3-C11                     0x00000008

VTD                   vtscsi17
LUN                   0x8200000000000000
Backing device        hdisk15
Physloc               U787B.001.DNW3897-P1-C4-T1-W5006048448930A41-L2000000000000

VTD                   vtscsi18
LUN                   0x8300000000000000
Backing device        hdisk16
Physloc               U787B.001.DNW3897-P1-C4-T1-W5006048448930A41-L3000000000000

VTD                   vtscsi19
LUN                   0x8400000000000000
Backing device        hdisk23
Physloc               U787B.001.DNW3897-P1-C4-T1-W5006048448930A41-L12000000000000

VTD                   vtscsi20
LUN                   0x8500000000000000
Backing device        hdisk24
Physloc               U787B.001.DNW3897-P1-C4-T1-W5006048448930A41-L13000000000000

... and remove the corresponding logical volume:

$ rmlv beastielv
Warning, all data contained on logical volume beastielv will be destroyed.
rmlv: Do you wish to continue? y(es) n(o)? y
rmlv: Logical volume beastielv is removed.

Resize the /tmp file system and the swap space logical volume

Because the operating system was installed from scratch and the file system sizes were automatically adapted to the underlying SCSI virtual disk (in fact a logical volume provided by the VIOS on a locally attached SCSI disk), some default values may not be relevant for day to day use, especially on machine of server type.

In that sense, the size of the /tmp file system may be changed to a more sensible setting:

# df -k /tmp
Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd3            32768     32388    2%       10     1% /tmp
#
# chfs -a size=1024M /tmp
Filesystem size changed to 2097152
#
# df -k /tmp
Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd3          1048576   1048044    1%       10     1% /tmp

In the same time, the swap space may be enlarge to grow the available VM on the system:

# lsps -al
hd6
#
# lsps -as
Total Paging Space   Percent Used
      512MB               1%
#
# lsvg rootvg | grep "PP SIZE"
VG STATE:           active                   PP SIZE:        16 megabyte(s)
#
# echo "(2048-512)/16" | bc
96
#
# chps -s 96 -a y hd6
#
# lsps -as
Total Paging Space   Percent Used
      2048MB               1%

Install and configure the backup software tool (TSM)

Please consult the following documentation: How to Add a New "tsmsched" Service on AIX 5L.

Put this machine under the ldm management tool

Don't forget to add the new machine in the ldm local site utility in order to be able to manage it and obtain some useful information get on a daily basis.

# ldmadd beastie

TESTING beastie REMOTE ACCESS ...
sysadm: access denied on beastie ...
Configuring sysadm's ssh authorized_keys file ... beastie done.

Checking some prerequisites on beastie ... 
ok

Updating ldm agent files on beastie ...
[...]

Interesting additional reading

Saturday 6 August 2005

Details About SAN Disks and MPxIO Capabilities on a VIOS

Obtaining these sorts of particular and specific informations (such as MultiPath I/O status) from a Virtual I/O Server can be very easily achieved using the following one (long) line shell script, helped by the lsdev(1), lscfg(1) and lspath commands:

# for disk in `lsdev | grep hdisk | egrep  -v "SCSI Disk Drive|Raid1" | awk '{print $1}'`
> do
> lscfg -v -l ${disk} | egrep "${disk}|Manufacturer|Machine Type|ROS Level and ID|Serial Number|Part Number"
> echo "`lspath -H -l ${disk} | grep ${disk} | awk '{print\"\tMultiPath I/O (MPIO) status: \"$1\" on parent \"$3}'`"
> echo ""
> done

  hdisk3           U787B.001.DNW3897-P1-C3-T1-W5006048448930A41-L9000000000000  EMC Symmetrix FCP MPIO RaidS
        Manufacturer................EMC     
        Machine Type and Model......SYMMETRIX       
        ROS Level and ID............5670
        Serial Number...............9312A020
        Part Number.................000000000000510001000287
        MultiPath I/O (MPIO) status: Enabled on parent fscsi0
        MultiPath I/O (MPIO) status: Enabled on parent fscsi1

  hdisk4           U787B.001.DNW3897-P1-C3-T1-W5006048448930A41-LA000000000000  EMC Symmetrix FCP MPIO RaidS
        Manufacturer................EMC     
        Machine Type and Model......SYMMETRIX       
        ROS Level and ID............5670
        Serial Number...............9312E020
        Part Number.................000000000000510001000287
        MultiPath I/O (MPIO) status: Enabled on parent fscsi0
        MultiPath I/O (MPIO) status: Enabled on parent fscsi1
[...]

Pattern SCSI Disk Drive is excluded since it represents local SCSI disks, as well as pattern Raid1 because it is a view corresponding to parity disks (which are logical disks only used by SAN administrators).

Wednesday 6 July 2005

Export and Import a Volume Group... When Things Goes the Wrong Way

  1. nordika is the hostname of the LPAR... which is a VIOC too
  2. nordikavg is the name of volume group which resides on the SAN disks impacted by the export/reimport

Assuming that we want to migrate, on a VIOC, one or more currently SAN disks attached on a local fibre channel adapter to the same one or more SAN disks but now presented as SCSI storage media, seen -- this time -- through a VIOS.

Here is the logical steps to follow... when all things doesn't work as expected (real life example)!

Get the list of the physical and logical volumes corresponding to the volume group:

# lsvg -p nordikavg
nordikavg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk2            active            269         0           00..00..00..00..00
hdisk3            active            269         0           00..00..00..00..00
hdisk4            active            269         0           00..00..00..00..00
hdisk5            active            269         115         07..00..00..54..54
hdisk6            active            269         269         54..54..53..54..54
#
# lsvg -l nordikavg       
nordikavg:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
loglv01             jfs2log    1     1     1    open/syncd    N/A
fslv01              jfs2       480   480   2    open/syncd    /files/tables_oragl
fslv02              jfs2       80    80    2    open/syncd    /files/oracle
fslv03              jfs2       40    40    1    open/syncd    /files/tempo_oragl
fslv05              jfs2       40    40    1    open/syncd    /files/redologs_oragl
fslv06              jfs2       40    40    1    open/syncd    /files/system_oragl
fslv07              jfs2       280   280   2    open/syncd    /files/tmpcdinst

Unmount the already mounted file systems:

# umount /files/tables_oragl
# umount /files/oracle
# umount /files/tempo_oragl
# umount /files/redologs_oragl
# umount /files/system_oragl
# umount /files/tmpcdinst

Deactivate a volume group and export the definition of a volume group from a set of physical volumes:

# varyoffvg nordikavg
# exportvg nordikavg

Having verified that there is no physical volumes in the desired volume group using lspv, remove them from the devices list with the corresponding adapter:

# rmdev -l hdisk2 -Rd
# rmdev -l hdisk3 -Rd
# rmdev -l hdisk4 -Rd
# rmdev -l hdisk5 -Rd
# rmdev -l hdisk6 -Rd
#
# lsslot -c slot
# rmdev -l pci2 -Rd

We assume that the fibre channel adapter is now seen through a VIOS: it is not shown here how to dynamically move it from the LPAR to the VIOS and allocate the PVs to a particular VIOC, i.e. nordika in our case.

Make the new disks available to the OS and verify that the presented LUNs are the right ones:

# cfgmgr
# lscfg -l hdisk2
  hdisk2           U9113.550.65E3A0C-V5-C5-T1-L830000000000  Virtual SCSI Disk Drive
# lscfg -l hdisk3
  hdisk3           U9113.550.65E3A0C-V5-C5-T1-L840000000000  Virtual SCSI Disk Drive
# lscfg -l hdisk4
  hdisk4           U9113.550.65E3A0C-V5-C5-T1-L850000000000  Virtual SCSI Disk Drive
# lscfg -l hdisk5
  hdisk5           U9113.550.65E3A0C-V5-C5-T1-L860000000000  Virtual SCSI Disk Drive
# lscfg -l hdisk6
  hdisk6           U9113.550.65E3A0C-V5-C5-T1-L870000000000  Virtual SCSI Disk Drive

Generally, we just have to import the nordikavg volume group, activate it, mount the file systems on it and... enjoy. Since we encountered a problem during the import (the information between the VM and the ODM seems not synchronized accordingly), here are the steps we follow to recover the situation.

Reimport the volume group, redefine the set of PVs of the given VG in the device configuration database and activate it:

# importvg nordikavg               /* Ooops... something goes wrong here! */
# redefinevg -d hdisk2 nordikavg   /* One disk is sufficient to get the volume group information back. */
# varyonvg nordikavg

Ok, the PVs are back in the configuration but not the type of the LVs, according to:

# lsvg -l nordikavg
nordikavg:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
loglv01             ???        1     1     1    open/syncd    N/A
fslv01              ???        480   480   2    open/syncd    /files/tables_oragl
fslv02              ???        80    80    2    open/syncd    /files/oracle
fslv03              ???        40    40    1    open/syncd    /files/tempo_oragl
fslv05              ???        40    40    1    open/syncd    /files/redologs_oragl
fslv06              ???        40    40    1    open/syncd    /files/system_oragl
fslv07              ???        280   280   2    open/syncd    /files/tmpcdinst

Synchronize or rebuild the logical volume control block, the device configuration database and the volume group descriptor areas on the PVs:

# synclvodm -v -P nordikavg
synclvodm: Physical volume data updated.
synclvodm: Logical volume loglv01 updated.
synclvodm: Logical volume fslv01 updated.
synclvodm: Logical volume fslv02 updated.
synclvodm: Logical volume fslv03 updated.
synclvodm: Logical volume fslv05 updated.
synclvodm: Logical volume fslv06 updated.
synclvodm: Logical volume fslv07 updated.
#
# lsvg -l nordikavg
nordikavg:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
loglv01             jfs2log    1     1     1    open/syncd    N/A
fslv01              jfs2       480   480   2    open/syncd    /files/tables_oragl
fslv02              jfs2       80    80    2    open/syncd    /files/oracle
fslv03              jfs2       40    40    1    open/syncd    /files/tempo_oragl
fslv05              jfs2       40    40    1    open/syncd    /files/redologs_oragl
fslv06              jfs2       40    40    1    open/syncd    /files/system_oragl
fslv07              jfs2       280   280   2    open/syncd    /files/tmpcdinst

Create complete boot image and device (in order to keep the type of LVs persistent across reboot):

# bosboot -a

bosboot: Boot image is 23377 512 byte blocks.

Mount the file systems and... enjoy :)

# mount /files/tables_oragl
# mount /files/oracle
# mount /files/tempo_oragl
# mount /files/redologs_oragl
# mount /files/system_oragl
# mount /files/tmpcdinst