Comparison: EMC PowerPath vs. GNU/Linux dm-multipath

I will present some notes about the use of multipath solutions on Red Hat systems: EMC PowerPath and GNU/Linux dm-multipath. Along those notes, keep in mind that they were based on tests done when pressure was very high to put new systems in production, so lack of time resulted in less complete tests than expected. These tests were done more than a year ago, and so before the release of RHEL4 Update 5 and some of RHBA related to both LVM and dm-multipath technologies.

Keep in mind that without purchasing an appropriate EMC license, PowerPath can only be used in failover mode (active-passive mode). Multiple paths accesses are not supported in this case: no round-robin, and no I/O load balancer for example.

EMC PowerPath

Advantages

Not specific to the SAN Host Bus Adapter (HBA).
Support for multiple and heterogeneous SAN storage provider.
Support for most UNIX and Unix-like platforms.
Without a valid license, can only work in degraded mode (failover).
Is not sensible to a change in the SCSI LUN renumbering. Adapt accordingly the corresponding multiple sd devices (different paths to a given device) with its multipath definition of the emcpower device.
Provide easily the ID of the SAN storage.

Drawbacks

Not integrated with the operating system (which generally has its own solution).
The need to force a RPM re-installation in case of a kernel upgrade on RHEL systems (due to the fact that kernel modules are stored in a path containing the exact major and minor versions of the installed (booted) kernel.
Non-automatic update procedure.

GNU/Linux device-mapper-multipath

Advantages

Not specific to the SAN Host Bus Adapter (HBA).
Support for multiple and heterogeneous SAN storage provider.
Well integrated with the operating system.
Automatic update using RHN (you must be a licensed and registered user in this case).
No additional license cost.

Drawbacks

Only available on GNU/Linux systems.
Configuration (files and keywords) very tedious and difficult.
Without the use of LVM (Logical Volume Management), it has not the ability to follow SCSI LUN renumbering! Even in this case, be sure not to have blacklisted the newly discovered SCSI devices (sd).

Last, please find some interesting documentation on the subject below:

Comments

1. On Friday 16 May 2008, 21:04 by Ash

Nice comparison, thank you.
Is there any way to get Clariion/Symmetrix LUN numbers when using dm-multipath?

2. On Friday 16 May 2008, 21:07 by Julien Gabel

To be able to get the SCSI LUN ID--whatever is the SAN storage provider, you must know the appropriate sd devices which compose the multipath device (PowerPath, GNU/Linux dm-multipath, etc.). Then, you can identify the LUN ID using something like this:
# find /sys/devices -name "*block" -print | \
xargs \ls -l | awk -F\/ '$NF ~ /sdo$/ || $NF ~ /sdp$/ \
{print "HBA: "$7"\tscsi#:channel#:id#:lun#: "$9}'
HBA: host0 scsi#:channel#:id#:lun#: 0:0:0:9
HBA: host0 scsi#:channel#:id#:lun#: 0:0:1:9

If you are looking for more information on SCSI devices, you can also use the excellent EMC 'inq' little tool:
http://blog.thilelli.net/post/2007/...

3. On Saturday 7 June 2008, 15:20 by someone

One of the nice things I've found with multipathd is that you can alias devices based on their unique LUN ID (using the "alias" directive, I can end up with a /dev/mapper/SAN_LUN3 etc based on the unique ID of the LUN). This way you can be sure that the same LUN will have the same name across multiple hosts, between boots, after removing or adding LUNS etc. I haven't been able to replicate this using powerpath? Does anyone know if there is a way? It seems it just picks the emcpower<a,b,c,d...> names however it wants? i.e. if I attach 4 LUNS to one host then the same LUNs to another, in each case the LUNs will be different devices?

4. On Sunday 8 June 2008, 17:55 by Julien Gabel

As for EMC PowerPath, it can update its configuration on LUN name change if used with CLARiiON system. I don't know if it is possible for other storage provider (but this not important from PowerPath point o view). The final aim is to be able to properly discover multipath devices, which generally operate under LVM management: in this case, both solutions are usable nevertheless.

What really matter is that the multipath device follow automatically device name change (whatever the name is), and LUN renumbering. It doesn't really matter how is named the device if the multipath software can't recognize it (unless for human intervention in case of a software failure in order to reconfigure it properly). If the alias capability is the solution to do that with dm-multipath, that is a good news.

5. On Tuesday 16 December 2008, 15:00 by Christian Peper

thanks for the list! :)

Also note that RHEL 5.2 now has multipath-aware anaconda installer which means you should be able to install Red Hat out of the box on servers with 2 HBAs!

Regarding the comment about the emcpower devices and their numbering, you can use emcpadm to renumber devices. Better yet, use 'powermt save file=dmap.pp' on one host and 'powermt load file=dmap.pp' on the other to get exactly the same LUN mapping, which is required for cluster setups.

6. On Saturday 9 May 2009, 18:23 by Rui Seabra

Ha! Your list of drawbacks for the multipath version is a joke, right? :)

1. not a problem
2. not a problem
3. since LVM is really adviseable, not a problem as well.

Thank you very much for your comparison, it will really help me when the HBA cards for a couple of systems do arrive.

7. On Sunday 10 May 2009, 19:58 by Julien Gabel

Well, no it is not a joke... per-se :)

The fact is, I try to give both for and against multipath inputs. I think the two first points are valid. It doesn't mean they are for you, even if the very first point is just a fact.

As for the third point, I didn't encounter this kind of configuration file with other multipath and LVM technologies.

8. On Friday 11 September 2009, 12:56 by serfKong

I don't quite follow the probable impact of this on a system: 'The need to force a RPM re-installation in case of a kernel upgrade on RHEL systems (due to the fact that kernel modules are stored in a path containing the exact major and minor versions of the installed (booted) kernel.'

1) How difficult is it to force the re-installation of the software?

2) What effects on the configuration of the running system will this have - Would a reboot be required? (Surely a kernel upgrade would require this any way?!)

3) I'm from a Solaris background & am new to the Linux world: How often do you upgrade your kernel on a critical production system used as an Oracle dB server that almost never changes other than have more LUNs or HBAs added? Is this kernel update something you perform weekly, or per annum?

Best regards.

9. On Saturday 19 September 2009, 16:32 by Julien Gabel

It is not easy to answer concisely to these questions, however here are some inputs:

How difficult is it to force the re-installation of the software?

When you know it, it not as difficult. It is just not convenient, at 3am not to see your more than 7TB database not came up again after an OS upgrade because under GNU/Linux the modules are installed in a path which include the exact number and revision of the installed and running kernel. At last, it is not particularly elegant, and it must exists a better way.

What effects on the configuration of the running system will this have - Would a reboot be required? (Surely a kernel upgrade would require this any way?!)

The upgrade must be done using a running system. So, you have to quiesce your activity during this time, and reboot quickly after that. This must be part of a planned intervention window nevertheless. So, you must have to reboot in single-user, force a re-installation of the PowerPath RPM package (and have it available in single-user mode), then reboot in multi-user mode.

I'm from a Solaris background & am new to the Linux world: How often do you upgrade your kernel on a critical production system used as an Oracle dB server that almost never changes other than have more LUNs or HBAs added? Is this kernel update something you perform weekly, or per annum?

As you may know (if you read my blog), I am more well versed in Solaris too. The fact is, when you upgrade regularly for integrity, bug and security fixes, the kernel is sometimes impacted. When using Linux, the major revision of the kernel is the same, just the minor number is updated, but this is the base of the problem I described earlier. On Solaris, even if the kernel is impacted, the changing number for the generic kernel is never a problem for this type of intervention.

Depending on the number of times you have to upgrade, my point is you never have to encounter the problem I faced with Linux.

10. On Wednesday 26 January 2011, 17:46 by oczkov

One of the main differences between Linux native MPIO na PP is that EMC PowerPath is fully aware of what load balancing algorithm to use with a given storage device. It has a dedicated algorithms optimized for CLARiiON and Symmetrix arrays.

With MD-MPIO you have only round-robin, which may or may not be the best choice for active-passive or ALUA storage.

BTW: anybody found some I/O benchmarks showing what is the performance difference between PP and MD-MPIO? If it is just a few % points more for PP then is it worth to pay 1.1 up to 5.7 k USD per server?

Regards,

blog'o thnet