blog'o thnet

To content | To menu | To search

Tag - ODM

Entries feed

Thursday 21 June 2007

Altering LVM Configuration When a Disk is Not in ODM Anymore

If you remove a disk from the system using rmdev -dl hdiskX without having previously reduced the volume group to remove the disk from LVM, and thus have not updated properly the on-disk format information (called VGDA), you get a discrepancy between the ODM and the LVM configurations. Here is how to solve the issue (without any warranty though!).

What are the volume group informations:

# lsvg -p rootvg                
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            2157        1019        174..00..00..413..432
0516-304 : Unable to find device id 00ce4b6a01292201 in the Device
       Configuration Database.
00ce4b6a01292201  missing           2157        1019        174..71..00..342..432
# lspv
hdisk0          00ce4b6ade6da849                    rootvg          active
hdisk2          00ce4b6a01b09b83                    drakevg         active
hdisk3          00ce4b6afd175206                    drakevg         active
# lsdev -Cc disk
hdisk0 Available  Virtual SCSI Disk Drive
hdisk2 Available  Virtual SCSI Disk Drive
hdisk3 Available  Virtual SCSI Disk Drive

As we can notice, the disk is still in the LVM configuration but doesn't show up in the devices. To solve this issue, we need to cheat the ODM in order to be able to use LVM commands to change the LVM configuration, stored on the volume group disks. The idea is to reinsert a disk in the ODM configuration, remove the disk from LVM and then remove it from ODM. Here is how we do it. First, let's make a copy of the ODM files that we will change:

# cd /etc/objrepos/
# cp CuAt CuAt.before_cheat
# cp CuDv CuDv.before_cheat
# cp CuPath CuPath.before_cheat

Now, we will extract the hdisk0's definition from ODM and add it back as hdisk1's definition:

# odmget -q "name=hdisk0" CuAt
CuAt:
       name = "hdisk0"
       attribute = "unique_id"
       value = "3520200946033223609SYMMETRIX03EMCfcp05VDASD03AIXvscsi"
       type = "R"
       generic = ""
       rep = "n"
       nls_index = 0
CuAt:
       name = "hdisk0"
       attribute = "pvid"
       value = "00ce4b6ade6da8490000000000000000"
       type = "R"
       generic = "D"
       rep = "s"
       nls_index = 11
# odmget -q "name=hdisk0" CuDv
CuDv:
       name = "hdisk0"
       status = 1
       chgstatus = 2
       ddins = "scsidisk"
       location = ""
       parent = "vscsi0"
       connwhere = "810000000000"
       PdDvLn = "disk/vscsi/vdisk"
# odmget -q "name=hdisk0" CuPath
CuPath:
       name = "hdisk0"
       parent = "vscsi0"
       connection = "810000000000"
       alias = ""
       path_status = 1
       path_id = 0

Basically, we need to insert new entries in the three classes CuAt, CuDv and CuPath with hdisk0 changed to hdisk1. A few others attributes need to be changed. The most important one is the PVID, located in CuAt. We will use the value reported as missing by lsvg -p rootvg. Attribute unique_id also need to be changed. You can just change a few characters in the existing string, it just need to be unique in the system. The other attributes to change are connwhere in CuDv and connection in CuPath. Their value represent the LUN ID of the disk. Again, this value is not relevant, it just have to be unique. We can check the current LUN defined by running lscfg on all the disks defined:

# lscfg -vl hdisk*
 hdisk0           U9117.570.65E4B6A-V6-C2-T1-L810000000000  Virtual SCSI Disk Drive
 hdisk2           U9117.570.65E4B6A-V6-C3-T1-L810000000000  Virtual SCSI Disk Drive
 hdisk3           U9117.570.65E4B6A-V6-C3-T1-L820000000000  Virtual SCSI Disk Drive

LUN 81 is used on controller C2 and LUNs 81 and 82 on C3. Let's choose 85, which for sure will not collide with other devices. The following commands will generate the text files that will be used to cheat the ODM, according to what was just explained:

# mkdir /tmp/cheat
# cd /tmp/cheat
# odmget -q "name=hdisk0" CuAt | sed -e 's/hdisk0/hdisk1/g' \
   -e 's/00ce4b6ade6da849/00ce4b6a01292201/' \
   -e 's/609SYMMETRIX/719SYMMETRIX/' > hdisk1.CuAt
# odmget -q "name=hdisk0" CuDv | sed -e 's/hdisk0/hdisk1/' \
   -e 's/810000000000/850000000000/' > hdisk1.CuDv
# odmget -q "name=hdisk0" CuPath | sed -e 's/hdisk0/hdisk1/' \
   -e 's/810000000000/850000000000/' > hdisk1.CuPAth

Let's look at the generated files:

# cat hdisk1.CuAt
CuAt:
       name = "hdisk1"
       attribute = "unique_id"
       value = "3520200946033223719SYMMETRIX03EMCfcp05VDASD03AIXvscsi"
       type = "R"
       generic = ""
       rep = "n"
       nls_index = 0
CuAt:
       name = "hdisk1"
       attribute = "pvid"
       value = "00ce4b6a012922010000000000000000"
       type = "R"
       generic = "D"
       rep = "s"
       nls_index = 11
# cat hdisk1.CuDv
CuDv:
       name = "hdisk1"
       status = 1
       chgstatus = 2
       ddins = "scsidisk"
       location = ""
       parent = "vscsi0"
       connwhere = "850000000000"
       PdDvLn = "disk/vscsi/vdisk"
# cat hdisk1.CuPath
CuPath:
       name = "hdisk1"
       parent = "vscsi0"
       connection = "850000000000"
       alias = ""
       path_status = 1
       path_id = 0

So, we are ready to insert the data in the ODM:

# odmadd hdisk1.CuAt
# odmadd hdisk1.CuDv
# odmadd hdisk1.CuPath
# lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            2157        1019        174..00..00..413..432
hdisk1            missing           2157        1019        174..71..00..342..432

The disk is now back in ODM! Now, to remove the disk from the VGDA, we use the reducevg command:

# reducevg rootvg hdisk1
0516-016 ldeletepv: Cannot delete physical volume with allocated
       partitions. Use either migratepv to move the partitions or
       reducevg with the -d option to delete the partitions.
0516-884 reducevg: Unable to remove physical volume hdisk1.

We will use the -d flag to remove the physical partitions associated to each logical volumes and located hdisk1. A few lines have been remove to simplify listing...

# reducevg -d rootvg hdisk1
0516-914 rmlv: Warning, all data belonging to logical volume
       lv01 on physical volume hdisk1 will be destroyed.
rmlv: Do you wish to continue? y(es) n(o)?
y
0516-304 putlvodm: Unable to find device id 00ce4b6a012922010000000000000000 in the
       Device Configuration Database.
0516-896 reducevg: Warning, cannot remove physical volume hdisk1 from
       Device Configuration Database.
# lsvg -l rootvg
rootvg:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
hd5                 boot       2     2     1    closed/syncd  N/A
hd6                 paging     256   256   1    open/syncd    N/A
hd8                 jfs2log    1     1     1    open/syncd    N/A
hd4                 jfs2       7     7     1    open/syncd    /
hd2                 jfs2       384   384   1    open/syncd    /usr
hd9var              jfs2       64    64    1    open/syncd    /var
hd3                 jfs2       128   128   1    open/syncd    /tmp
hd1                 jfs2       2     2     1    open/syncd    /home
hd10opt             jfs2       32    32    1    open/syncd    /opt
fslv04              jfs2       256   256   1    open/syncd    /usr/sys/inst.images
loglv01             jfslog     1     1     1    closed/syncd  N/A
lv01                jfs        5     5     1    closed/syncd  /mkcd/cd_images
# lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            2157        1019        174..00..00..413..432

The disk has been deleted from the VGDA. What about ODM?

# lsdev -Cc disk
hdisk0 Available  Virtual SCSI Disk Drive
hdisk1 Available  Virtual SCSI Disk Drive
hdisk2 Available  Virtual SCSI Disk Drive
hdisk3 Available  Virtual SCSI Disk Drive
# rmdev -dl hdisk1
Method error (/etc/methods/ucfgdevice):
       0514-043 Error getting or assigning a minor number.

We probably forgot to cheat one ODM class... Never mind: let's remove the cheat we added to ODM and see what appends:

# odmdelete -o CuAt -q "name=hdisk1"
2 objects deleted
# lspv
hdisk0          00ce4b6ade6da849                    rootvg          active
hdisk2          00ce4b6a01b09b83                    drakevg         active
hdisk1          none                                None            
hdisk3          00ce4b6afd175206                    drakevg         active
# rmdev -dl hdisk1
Method error (/etc/methods/ucfgdevice):
       0514-043 Error getting or assigning a minor number.
# odmdelete -o CuDv -q "name=hdisk1"
1 objects deleted
# lspv
hdisk0          00ce4b6ade6da849                    rootvg          active
hdisk2          00ce4b6a01b09b83                    drakevg         active
hdisk3          00ce4b6afd175206                    drakevg         active
# lspath
Enabled hdisk0 vscsi0
Enabled hdisk2 vscsi0
Enabled hdisk2 vscsi1
Enabled hdisk3 vscsi1
Enabled hdisk3 vscsi0
Unknown hdisk1 vscsi0
# odmdelete -o CuPath -q "name=hdisk1"
1 objects deleted
# lspath
Enabled hdisk0 vscsi0
Enabled hdisk2 vscsi0
Enabled hdisk2 vscsi1
Enabled hdisk3 vscsi1
Enabled hdisk3 vscsi0

That's it! Use with care.

Side note: This entry was originally contributed by Patrice Lachance, which first wrote about this subject.

Monday 11 July 2005

Memo About Some Very Interesting CLI Tools

Boot disk configuration

After verifying there are two disks in the boot list...

# bootlist -m normal -o
hdisk0
hdisk1

... verify and create a boot image on the second mirrored boot disk:

# bosboot -vd hdisk1 && bosboot -ad hdisk1

How to know on which disk the OS has booted (bootblock used and kernel loaded):

# bootinfo -b
hdisk0

How to know on which mode the OS has booted (kernel in 32-bit or 64-bit):

# bootinfo -K
64

If there is some problem booting on one disk, be sure that the corresponding raw device are the same device as ipldevice:

# bootinfo -b
hdisk0
#
# ls -ilF /dev/ipldevice /dev/rhdisk0
 8231 crw-------   2 root     system       17,  0 Apr 21 14:37 /dev/ipldevice
 8231 crw-------   2 root     system       17,  0 Apr 21 14:37 /dev/rhdisk0

VM information vs. ODM information

Assuming the following mounted file system:

# mount
  node       mounted        mounted over    vfs       date        options      
-------- ---------------  ---------------  ------ ------------ --------------- 
[...]
         /dev/fslv07      /files/tmpcdinst jfs2   Jun 27 10:14 rw,log=/dev/loglv01

Here are the corresponding information found in the ODM:

# odmget -q "name=fslv07 and attribute=type" CuAt 

CuAt:
        name = "fslv07"
        attribute = "type"
        value = "jfs2"
        type = "R"
        generic = "DU"
        rep = "s"
        nls_index = 639

This can be compared with the status returned by the lsvg command:

# lsvg -l colombvg
colombvg:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
[...]
fslv07              jf2        280   280   2    open/syncd    /files/tmpcdinst

More on this particular subject in the story Export and Import a Volume Group... When Things Goes the Wrong Way.

Get the ODM volume group information for a given disk:

# lqueryvg -p hdisk0 -Avt
Max LVs:        256
PP Size:        27
Free PPs:       0
LV count:       12
PV count:       2
Total VGDAs:    3
Conc Allowed:   0
MAX PPs per PV  1016
MAX PVs:        32
Conc Autovaryo  0
Varied on Conc  0
Logical:        00ce3a0c00004c000000010364ba67bc.1   hd5 1  
                00ce3a0c00004c000000010364ba67bc.2   hd6 1  
                00ce3a0c00004c000000010364ba67bc.3   hd8 1  
                00ce3a0c00004c000000010364ba67bc.4   hd4 1  
                00ce3a0c00004c000000010364ba67bc.5   hd2 1  
                00ce3a0c00004c000000010364ba67bc.6   hd9var 1  
                00ce3a0c00004c000000010364ba67bc.7   hd3 1  
                00ce3a0c00004c000000010364ba67bc.8   hd1 1  
                00ce3a0c00004c000000010364ba67bc.9   hd10opt 1  
                00ce3a0c00004c000000010364ba67bc.10  loglv00 1  
                00ce3a0c00004c000000010364ba67bc.11  fslv00 1  
                00ce3a0c00004c000000010364ba67bc.12  fslv04 1  
Physical:       00ce3a0c64ba5da3                2   0  
                00ce3a0c8df2265d                1   0  
VGid:           00ce3a0c00004c000000010364ba67bc
Total PPs:      158
LTG size:       128
HOT SPARE:      0
AUTO SYNC:      0
VG PERMISSION:  0
SNAPSHOT VG:    0
IS_PRIMARY VG:  0
PSNFSTPP:       4352
VARYON MODE:    0
VG Type:        0
Max PPs:        32512

Operating system general status and information

Gather system configuration information:

# snap -r    /* Remove snap command output from the /tmp/ibmsupt directory. */
# snap -ac   /* Creates a compressed pax image (snap.pax.Z file) of all files
                in the /tmp/ibmsupt. */

This tool can be compared to the explorer (known as the SUNWexplo package) on Sun Solaris OE.

About starting services at boot time

List the content of the inittab file:

# lsitab -a   /* Use this command instead of `cat /etc/inittab`. */

Create a new file system

Create a new Enhanced Journaled File System in the the colombvg volume group with a size of 5 gigabytes in read-write mode, using the mount point /files/ddaeurd1/DATA and being automatically mounted at boot time:

# crfs -v jfs2 -g colombvg -a size=5G -m /files/ddaeurd1/DATA -p rw -A yes

Wednesday 6 July 2005

Export and Import a Volume Group... When Things Goes the Wrong Way

  1. nordika is the hostname of the LPAR... which is a VIOC too
  2. nordikavg is the name of volume group which resides on the SAN disks impacted by the export/reimport

Assuming that we want to migrate, on a VIOC, one or more currently SAN disks attached on a local fibre channel adapter to the same one or more SAN disks but now presented as SCSI storage media, seen -- this time -- through a VIOS.

Here is the logical steps to follow... when all things doesn't work as expected (real life example)!

Get the list of the physical and logical volumes corresponding to the volume group:

# lsvg -p nordikavg
nordikavg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk2            active            269         0           00..00..00..00..00
hdisk3            active            269         0           00..00..00..00..00
hdisk4            active            269         0           00..00..00..00..00
hdisk5            active            269         115         07..00..00..54..54
hdisk6            active            269         269         54..54..53..54..54
#
# lsvg -l nordikavg       
nordikavg:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
loglv01             jfs2log    1     1     1    open/syncd    N/A
fslv01              jfs2       480   480   2    open/syncd    /files/tables_oragl
fslv02              jfs2       80    80    2    open/syncd    /files/oracle
fslv03              jfs2       40    40    1    open/syncd    /files/tempo_oragl
fslv05              jfs2       40    40    1    open/syncd    /files/redologs_oragl
fslv06              jfs2       40    40    1    open/syncd    /files/system_oragl
fslv07              jfs2       280   280   2    open/syncd    /files/tmpcdinst

Unmount the already mounted file systems:

# umount /files/tables_oragl
# umount /files/oracle
# umount /files/tempo_oragl
# umount /files/redologs_oragl
# umount /files/system_oragl
# umount /files/tmpcdinst

Deactivate a volume group and export the definition of a volume group from a set of physical volumes:

# varyoffvg nordikavg
# exportvg nordikavg

Having verified that there is no physical volumes in the desired volume group using lspv, remove them from the devices list with the corresponding adapter:

# rmdev -l hdisk2 -Rd
# rmdev -l hdisk3 -Rd
# rmdev -l hdisk4 -Rd
# rmdev -l hdisk5 -Rd
# rmdev -l hdisk6 -Rd
#
# lsslot -c slot
# rmdev -l pci2 -Rd

We assume that the fibre channel adapter is now seen through a VIOS: it is not shown here how to dynamically move it from the LPAR to the VIOS and allocate the PVs to a particular VIOC, i.e. nordika in our case.

Make the new disks available to the OS and verify that the presented LUNs are the right ones:

# cfgmgr
# lscfg -l hdisk2
  hdisk2           U9113.550.65E3A0C-V5-C5-T1-L830000000000  Virtual SCSI Disk Drive
# lscfg -l hdisk3
  hdisk3           U9113.550.65E3A0C-V5-C5-T1-L840000000000  Virtual SCSI Disk Drive
# lscfg -l hdisk4
  hdisk4           U9113.550.65E3A0C-V5-C5-T1-L850000000000  Virtual SCSI Disk Drive
# lscfg -l hdisk5
  hdisk5           U9113.550.65E3A0C-V5-C5-T1-L860000000000  Virtual SCSI Disk Drive
# lscfg -l hdisk6
  hdisk6           U9113.550.65E3A0C-V5-C5-T1-L870000000000  Virtual SCSI Disk Drive

Generally, we just have to import the nordikavg volume group, activate it, mount the file systems on it and... enjoy. Since we encountered a problem during the import (the information between the VM and the ODM seems not synchronized accordingly), here are the steps we follow to recover the situation.

Reimport the volume group, redefine the set of PVs of the given VG in the device configuration database and activate it:

# importvg nordikavg               /* Ooops... something goes wrong here! */
# redefinevg -d hdisk2 nordikavg   /* One disk is sufficient to get the volume group information back. */
# varyonvg nordikavg

Ok, the PVs are back in the configuration but not the type of the LVs, according to:

# lsvg -l nordikavg
nordikavg:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
loglv01             ???        1     1     1    open/syncd    N/A
fslv01              ???        480   480   2    open/syncd    /files/tables_oragl
fslv02              ???        80    80    2    open/syncd    /files/oracle
fslv03              ???        40    40    1    open/syncd    /files/tempo_oragl
fslv05              ???        40    40    1    open/syncd    /files/redologs_oragl
fslv06              ???        40    40    1    open/syncd    /files/system_oragl
fslv07              ???        280   280   2    open/syncd    /files/tmpcdinst

Synchronize or rebuild the logical volume control block, the device configuration database and the volume group descriptor areas on the PVs:

# synclvodm -v -P nordikavg
synclvodm: Physical volume data updated.
synclvodm: Logical volume loglv01 updated.
synclvodm: Logical volume fslv01 updated.
synclvodm: Logical volume fslv02 updated.
synclvodm: Logical volume fslv03 updated.
synclvodm: Logical volume fslv05 updated.
synclvodm: Logical volume fslv06 updated.
synclvodm: Logical volume fslv07 updated.
#
# lsvg -l nordikavg
nordikavg:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
loglv01             jfs2log    1     1     1    open/syncd    N/A
fslv01              jfs2       480   480   2    open/syncd    /files/tables_oragl
fslv02              jfs2       80    80    2    open/syncd    /files/oracle
fslv03              jfs2       40    40    1    open/syncd    /files/tempo_oragl
fslv05              jfs2       40    40    1    open/syncd    /files/redologs_oragl
fslv06              jfs2       40    40    1    open/syncd    /files/system_oragl
fslv07              jfs2       280   280   2    open/syncd    /files/tmpcdinst

Create complete boot image and device (in order to keep the type of LVs persistent across reboot):

# bosboot -a

bosboot: Boot image is 23377 512 byte blocks.

Mount the file systems and... enjoy :)

# mount /files/tables_oragl
# mount /files/oracle
# mount /files/tempo_oragl
# mount /files/redologs_oragl
# mount /files/system_oragl
# mount /files/tmpcdinst