UPDATE: With regards to _MSIExecute Mutex. See towards bottom for technical info on checking the current Windows Installer status using QueryServiceStatusEx. Heath Stewart suggests another way (sample C++ code).
Also, some pre-existing, related answers:
First some comments on the issue at hand (virtuals updating and pending reboots):
Windows Updates In Progress: If you are dealing with the situation where Windows Updates are being performed in the background and MSI installations won't run, then it is possible that WiX Burn might hang whilst checking whether a system reboot is pending. I have seen issues like this before (see "technical issues" below).
Real Fix: If this is the case the only real fix is to let the Windows Updates finish and then reboot and persist a new virtual machine state and then install your package. This is the only sane fix - in my opinion. Not what you want to hear :-).
Dirty "Fix": I suppose you could also stop the Windows Update service to prevent Windows Updates from installing, but my guess is that your virtual will be infected with malware eventually if you do this, and then you have the horrible situation that you may accidentally persist malware in your virtual that is then resurrected regularly and not detected by security software (which often does not scan virtuals). This might be one of the worst malware vectors in existence due to the difficulty of erradicating it and discovering it (in the same league as malware on read-only media and malware checked into source control systems - there might not be an easy way to get rid of it). Well-meant advice (the kind that is never wanted - we all live in reality): please do not disable Windows Update on your virtual without a proper consideration of consequences. Corporate-wide I would never allow such a thing - only as exceptions for specific users who have testing needs beyond normal - and the virtuals would be forcibly malware checked regularly. Which reminds me to check what security software can scan offline virtuals properly. More research to do. Funny that, writing stackoverflow answers always teaches me what I don't do properly myself! Not good :-).
Technical Issues: As to the technicalities. I have seen issues before where WiX Burn hangs because it specifically tries to avoid installing on a system that is being updated:
Maybe check if that little VBScript in the linked answers (or equivalent COM call in whatever language you want) will tell you if the system is not ready for installation?
The Simple Hack?: Not verified, but maybe you can check the registry key / value (not sure whether this is a value or a key): HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer : InProgress
- it should be present to indicate an active Windows Installer session - as far as I know. I think this is better than trying to handle mutexes and other OS fundamentals.
Strangely I am not aware of any MSI API calls that will tell you whether there is an active installation session (mutex set). The only thing I can see is a Win32 function (i.e not COM automation): MsiBeginTransaction (a very recent addition to the MSI API, 4.5 up only).
There is also this: MSDN: _MSIExecute Mutex - suggesting to use QueryServiceStatusEx and check whether the value of dwControlsAccepted
is SERVICE_ACCEPT_SHUTDOWN
. I have never tried it. Frankly I would try to check the above registry key instead.