False Positives: False positives from maleware scanners can be quite hard to deal with. To check using more than one malware scanner you can upload the release files individually as well as the complete setup to https://www.virustotal.com. This service runs many malware scanners on the submitted files so you can see what malware scanners flag which binary. There are a few other such anti-malware online scanners such as Kaspersky, Avira, etc...
Update: And then there is Process Explorer. Check this tweet chain for how to check your running application for malware hits per process and loaded file.
When you see the scope of the problem (how many files are flagged), you should work backwards to see how you could go about solving the problem. This can involve getting the files whitelisted by the malware vendor(s), eliminating them from your setup, or fixing technicalities that flag the files, etc... There are some options listed and elaborated below.
Fixes: There are both technical and practical fixes you can try. Don't expect it to be easy. The issue of false positives is a very serious deployment problem. The proposed fixes and workarounds below are in random order:
Compiler Settings: Sometimes you can actually choose different compiler settings to avoid the problem, but often you are not so lucky. I have seen this with files compiled with special Spectre / Meltdown mitigation settings. They were flagged as unknown by malware scanners.
Dangerous API-calls: You should also check what API-calls are made in the problem file(s) that could be known to cause security warnings (unusual and / or dangerous API-methods) - and remove them if you can. I have heard of cases where malware vendors want to refuse to whitelist your binary because what the binary does makes no sense to them (try calling a firmware update for an embedded system as part of your setup installation or some low-level call triggered by a security tool you are installing).
Eliminate Files: Removing certain components from your application can also help sometimes - especially if they are third-party components added to your application for convenience only. In other words your application works fine without them. Removing a problem can be much simpler than fixing it.
Vanilla Installer: Sometimes you can split problem components into a separate setup so your main setup installs without issues. This can help enormously with support issues or overall application approval in corporate settings. You can also make 2 full setups where one has all probable false-positive triggers removed - your "vanilla setup" that should install without drama in all cases.
Digital Signatures: Signing the file with a digital signature can help since a proper certificate "buys trust outright" in reputation-based score systems such as Microsoft SmartScreen. Note that this needs to be an EV-level certificate. Please check for updated information here as technology evolves. Certificate / signing technologies always seem to cause something unexpected.
Malware Scanner Whitelisting: Submit file for white listing. Then there is the formal approach with the malware vendors as explained by Bogdan Mitrache of Advanced Installer here: Antivirus Whitelisting Pains. You submit files to them for white-listing. The article explains real-world experience with binaries flagged as malware when delivering software. Mandatory reading.
Microsoft SmartScreen: Microsoft has their own way to submit files for analysis and white-listing: https://www.microsoft.com/en-us/wdsi/filesubmission. They state: "Microsoft security researchers analyze suspicious files to determine if they are threats, unwanted applications, or normal files. Submit files you think are malware or files that you believe have been incorrectly classified as malware.".
Unique Executable Per Customer: Sometimes a unique executable is used for every customer by auto-generating an installer for each sale. I would advise against this since the installer executable - even when signed - will be a "new encounter" for malware scanners. You could run into trouble you do not need. There is also an added risk for each generated installer executable to actually be infected by real malware, and there is also the QA-issue that every installer should be tested before release.
Signed Malware: Whatever you do, make sure the file in question isn't actually real malware! Obviously your own files can get infected. Test well. If you sign malware and deliver to your client the digital signature is proof positive that you delivered the malware to them. Not good. And then.
More on Digital Signatures: Some information and links to get your setup and / or files signed:
