Per this post, error code 11 indicates that the 32-bit version of DISM is being used on a 64-bit system. This corresponds with installing a 32-bit MSI on a 64-bit system and using it to locate and launch DISM. Windows Installer does not allow you to refer to 64-bit locations from a 32-bit MSI. Heath Stewart's article Different Packages are Required for Different Processor Architectures touches on this, but mostly from the angle of installing to 32- or 64-bit locations. As it turns out, finding files there is just as hard.
In order to launch a 64-bit DISM from a 64-bit location, you will need some other code. It may be possible to locate the 64-bit system folder from 32-bit code, but I know some 64-bit locations can only be correctly queried by 64-bit code. As such I would suggest you write a 64-bit helper exe to find and launch the 64-bit DISM. Then you will need two variants of your custom action so that you only try to use the 64-bit wrapper on a 64-bit system (when VersionNT64
is defined) and use a 32-bit wrapper or direct call on a 32-bit system.
Alternately, if upgrading and using an exe is an option, InstallShield 2013 and later include support for installing Windows Features as part of the Suite project type, which will thus handle this work for you. (Disclaimer: I am paid to work on InstallShield.)