There is no support for anything to do this in Visual Studio setups. This is because all Visual Studio install custom actions are run almost at the end of the install after all the files are installed, so it's no use for checking anything "before" the install.
In fact many installers have no support for this because it's not needed. Windows will check to see if any in-use files (and their loaded Dlls) are going to be replaced, and will show the user a Files-In-Use dialog box asking what the user wants to do, which is to close down the app or not (which may require a reboot). Therefore the user has the choice of what to do, and is not required to shut down the app. So there is no need to do anything at all about any possible files-in-use situation. The user will already get a dialog about in-use files if there is an issue.
More thorough applications integrate with the Restart Manager such that they save their state, shut down automatically, and Windows will restart them after the update with their command line that means they restore state, the user loses no data and carries on.