blog'o thnet

I will present some notes about the use of multipath solutions on Red Hat systems: EMC PowerPath and GNU/Linux dm-multipath. Along those notes, keep in mind that they were based on tests done when pressure was very high to put new systems in production, so lack of time resulted in less complete tests than expected. These tests were done more than a year ago, and so before the release of RHEL4 Update 5 and some of RHBA related to both LVM and dm-multipath technologies.

Keep in mind that without purchasing an appropriate EMC license, PowerPath can only be used in failover mode (active-passive mode). Multiple paths accesses are not supported in this case: no round-robin, and no I/O load balancer for example.

EMC PowerPath

Advantages

1. Not specific to the SAN Host Bus Adapter (HBA).
2. Support for multiple and heterogeneous SAN storage provider.
3. Support for most UNIX and Unix-like platforms.
4. Without a valid license, can only work in degraded mode (failover).
5. Is not sensible to a change in the SCSI LUN renumbering. Adapt accordingly the corresponding multiple sd devices (different paths to a given device) with its multipath definition of the emcpower device.
6. Provide easily the ID of the SAN storage.

Drawbacks

1. Not integrated with the operating system (which generally has its own solution).
2. The need to force a RPM re-installation in case of a kernel upgrade on RHEL systems (due to the fact that kernel modules are stored in a path containing the exact major and minor versions of the installed (booted) kernel.
3. Non-automatic update procedure.

GNU/Linux device-mapper-multipath

Advantages

1. Not specific to the SAN Host Bus Adapter (HBA).
2. Support for multiple and heterogeneous SAN storage provider.
3. Well integrated with the operating system.
4. Automatic update using RHN (you must be a licensed and registered user in this case).
5. No additional license cost.

Drawbacks

1. Only available on GNU/Linux systems.
2. Configuration (files and keywords) very tedious and difficult.
3. Without the use of LVM (Logical Volume Management), it has not the ability to follow SCSI LUN renumbering! Even in this case, be sure not to have blacklisted the newly discovered SCSI devices (sd).

Last, please find some interesting documentation on the subject below:

• How do I setup device-mapper multipathing in RHEL4?
• Setting Up Multipathing for Storage
• Enterprise Storage Quickstart on RHEL4

Once the corresponding partition has been defined on the managed system, log on to the HMC using an account having hmcsuperadmin authority. hscroot is such an account. Then, to install the VIOS partition using a previously generated backupios tar file, issue a command similar to the following:

$ installios \
   -s Server-9113-550-SN65E3R4F \
   -S uu.xx.yy.zz \
   -p vios01 \
   -r installation \
   -i vv.xx.yy.zz \
   -d nfssrv:/path/to/backupios/archive \
   -m 00:11:22:aa:bb:cc \
   -g ww.xx.yy.zz \
   -P 100 \
   -D full

Where:

• -s: Managed system
• -p: Partition name
• -r: Partition profile
• -d: Path to installation image(s) (/dev/cdrom or srv:/path/to/backup)
• -i: Client IP address
• -S: Client IP subnet mask
• -g: Client gateway
• -m: Client MAC address
• -P: Port speed (optional, 100 is the default (10, 100, or 1000))
• -D: Port duplex (optional, full is the default (full, or half))

Note that he profile named installation is very similar to the profile named normal: it just doesn't include all the extra-stuff necessary for our final pSeries configuration, i.e. SAN HBA, virtual LAN, etc. This is necessary not to install on SAN disks, or try to use a virtual Ethernet adapter during VIOS installation process. After rebooting on the fresh installed VIOS, connect to the console and check for:

1. Clean-up the content of the /etc/hosts file, in particular be sure that the FQDN and short name of the NIM server are mentioned properly.
2. Configure the IP address(es) on the physical interface(s), and the corresponding hostname--and don't forget that they will be modify latter in order to create SEA device!
3. Recreate the mirror in order to use the two first disks (with exact mapping), and be sure to have two copies the lg_dumplv logical volume (not really sure about this one, but it doesn't hurt anyway...).
4. Update the content of the /etc/resolv.conf file.
5. Be able to resolve hostnames using other network centralized mechanisms:

   # cat << EOF >> /etc/netsvc.conf
   hosts = local, nis, bind
   EOF
   

6. Don't forget to erase the installation NIM configuration found under /etc/niminfo and set it as a new NIM client for the current NIM server:

   # mv /etc/niminfo /etc/niminfo.orig
   # niminit -a name=vios01 \
    -a master=nim.example.com \
    -a pif_name=en0 \ # May be `en5' if the SEA was already configured.
    -a connect=nimsh
   

7. Change the padmin account password.

Last, here are some welcome tuning configuration steps:

• Update the VIOS installation software with the external bundle pack, if available.
• Reboot the VIOS using the profile named normal (whi include all the targeted hardware definitions).
• There are a few parameters to change on the fibre channel adapter and fscsi interface on top of it. The first one is dyntrk, which allow fabric reconfiguration without having to reboot the Virtual I/O Server. The second one is fs_err_recov, which will prevent the Virtual I/O Server to retry sending an operation on a disk if the disk become unavailable. We change it because the Virtual I/O Client will take care of accessing the disk using MPxIO and thus, will redirect the I/O operations to the second Virtual I/O Server. The last parameter we change is the one that controls the number of commands to queue to the physical adapter. A reboot is necessary in order to change these parameters:

  $ chdev -dev fscsi0 -attr dyntrk=yes -perm
  fscsi0 changed
  $ chdev -dev fscsi0 -attr fc_err_recov=fast_fail -perm
  fscsi0 changed
  $ chdev -dev fcs0 -attr num_cmd_elems=2048 -perm
  fcs0 changed
  

• We can safely change the software transmit queue size and descriptor queue size with the following commands. Since the adapter is in use, we change the settings in ODM only, and the new configuration will be use at next reboot:

  $ chdev -dev ent0 -attr tx_que_sz=16384 -perm
  ent0 changed
  $ chdev -dev ent1 -attr tx_que_sz=16384 -perm
  ent1 changed
  $ chdev -dev ent0 -attr txdesc_que_sz=1024 -perm
  ent0 changed
  $ chdev -dev ent1 -attr txdesc_que_sz=1024 -perm
  ent1 changed
  

• And be sure to force the speed and mode of the desired Ethernet interfaces:

  $ chdev -dev ent0 -attr media_speed=100_Full_Duplex -perm
  ent0 changed
  $ chdev -dev ent1 -attr media_speed=100_Full_Duplex -perm
  ent1 changed
  

• Now, we need to create the Shared Ethernet Adapter to be able to access the external network and bind the virtual adapter to the real one:

  $ chdev -dev en0 -attr state=detach
  en0 changed
  $ chdev -dev en1 -attr state=detach
  en1 changed
  $ mkvdev -sea ent0 -vadapter ent3 -default ent3 -defaultid 1
  ent5 Available
  en5
  et5
  $ mkvdev -sea ent1 -vadapter ent4 -default ent4 -defaultid 3
  ent6 Available
  en6
  et6
  $ mktcpip -hostname vios01 \
     -inetaddr vv.xx.yy.zz \
     -interface en5 \
     -netmask uu.xx.yy.zz \
     -gateway ww.xx.yy.zz \
     -nsrvaddr tt.xx.yy.zz \
     -nsrvdomain example.com \
     -start
  

• Don't forget to install the MPxIO driver provided by EMC on their FTP web site:

  # cd /mnt/EMC.Symmetrix
  # TERM=vt220 smitty installp
  # lslpp -al | grep 'EMC.Symmetrix' | sort -u
                               5.2.0.3  COMMITTED  EMC Symmetrix Fibre Channel
    EMC.Symmetrix.aix.rte      5.2.0.3  COMMITTED  EMC Symmetrix AIX Support
    EMC.Symmetrix.fcp.MPIO.rte
  

• Assuming that the clock is given by the default gateway network device, we can set and configure the NTP client this way:

  # ntpdate ww.xx.yy.zz
  # cp /etc/ntp.conf /etc/ntp.conf.orig
  # diff -c /etc/ntp.conf.orig /etc/ntp.conf
  *** /etc/ntp.conf.orig  Fri Sep 30 18:05:17 2005
  --- /etc/ntp.conf       Fri Sep 30 18:05:43 2005
  ***************
  *** 36,41 ****
    #
    #   Broadcast client, no authentication.
    #
  ! broadcastclient
    driftfile /etc/ntp.drift
    tracefile /etc/ntp.trace
  --- 36,42 ----
    #
    #   Broadcast client, no authentication.
    #
  ! #broadcastclient
  ! server ww.xx.yy.zz
    driftfile /etc/ntp.drift
    tracefile /etc/ntp.trace
  #
  # chrctcp -S -a xntpd
  

Side note: This entry was originally contributed by Patrice Lachance, which first wrote about this subject.