Perhaps you can try the CloseApplication feature from the Util schema:
http://wixtoolset.org/documentation/manual/v3/xsd/util/closeapplication.html
See here for an example code snippet: https://sourceforge.net/p/wix/mailman/message/20186650/
UPDATE: I ran some tests and this element works a little differently from what I expected. The first thing you need to add is the reference to the wixUtilExtension file. On the command line this is:
candle -ext WiXUtilExtension Test.wxs
light -ext WixUtilExtension Test.wixobj
In Visual Studio I think you simply add a project reference to WixUtilExtension.dll.
Then you simply add something like this to your wxs source file:
<util:CloseApplication Id="CloseNotepad" Target="notepad.exe"
CloseMessage="yes" RebootPrompt="no">
</util:CloseApplication>
And add this at the top of your wxs file:
<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"
xmlns:util="http://schemas.microsoft.com/wix/UtilExtension">
It looks like Wix takes care of the rest (custom actions and a custom table in your MSI with the list of processes to kill).
Here is my full test file:
<?xml version="1.0" encoding="utf-8"?>
<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"
xmlns:util="http://schemas.microsoft.com/wix/UtilExtension">
<Product Id="*" UpgradeCode="12345678-1234-1234-1234-111111111111"
Name="Example Product Name" Version="0.0.1" Manufacturer="Example Company Name" Language="1033">
<Package InstallerVersion="200" Compressed="yes" Comments="Windows Installer Package"/>
<Media Id="1" Cabinet="product.cab" EmbedCab="yes"/>
<Directory Id="TARGETDIR" Name="SourceDir">
<Component Id="ApplicationFiles" Guid="*">
</Component>
</Directory>
<Feature Id="DefaultFeature" Level="1">
<ComponentRef Id="ApplicationFiles"/>
</Feature>
<util:CloseApplication Id="CloseNotepad" Target="notepad.exe" CloseMessage="yes" RebootPrompt="no"></util:CloseApplication>
</Product>
</Wix>
Links: Some related or marginally related links for easy retrieval.