Short Version: This is a version you can try first.
Run the this from a command prompt (kicks off administrative installation - maybe):
setup.exe /a
Extract to a location of your choice - if possible (more on extraction below).
Grab the MSI and open it with Orca and generate a transform as described below - setting the SecureCustomProperties in the Property table
Try to install silently using either the setup.exe or via msiexec.exe if you run the MSI on its own (pick one of the below command lines):
setup.exe /s /v"TRANSFORMS=\"C:\Transforms\1.mst\""
setup.exe /s /f1”c:\temp\my-answer-file.iss” /v"C:\Transforms\1.mst"
msiexec.exe /I "C:\Your.msi" /QN /L*V "C:\msilog.log" TRANSFORMS="C:\My.mst"
In point 4 the command lines are for Basic MSI, Installscript MSI and vanilla, extracted MSI respectively. For Installscript MSI a response file is needed. It might be found in the extract, or else it must be generated. See below.
Transform: I would use a transform. Transforms can change "anything" in an MSI, and hence it can easily set SecureCustomProperties as well. Essentially that property is a security measure (limiting what properties can be passed to the elevated installation process from the GUI sequence), and hence I guess they do not want it to be possible to set via the command line (my assumption).
Setup.exe vs MSI: You can try to deploy the setup.exe directly with a silent installation command, or you can try to extract the embedded MSI file along with its prerequisites from the setup.exe and deploy them separately. Very often the prerequisites are not needed to deploy - for example the .NET framework. An extracted MSI is also much easier to deal with since it supports standardized command lines, let me show a quick sample:
msiexec.exe /I "C:\Your.msi" /QN /L*V "C:\msilog.log" TRANSFORMS="C:\1031.mst;C:\My.mst"
Quick Parameter Explanation:
/I = run install sequence
/QN = run completely silently
/L*V "C:\msilog.log" = verbose logging
TRANSFORMS="C:\1031.mst;C:\My.mst" = Apply transforms 1031.mst and My.mst (see below).
File Extraction: Unfortunately a setup.exe can be lots of things ranging from legacy installers, to Installscript MSI to Basic MSI (both from Installshield) and various other possibilities exists from other vendors. They can even be Installshield Suite projects - which feature a totally different command line again. These are installers that can install any number of EXE and MSI files in sequence. I wrote up a similar answer the other day on the topic of how to extract files AND / OR installing silently: Create MSI from extracted setup files - please skim that answer. It focuses on extracing files, but also describes silent installation using the setup.exe.
Before trying anything else, try this:
setup.exe /a
See if you get a prompt to specify an output location. If you do, extract the files. You will need to extract the files to get hold of the MSI file in order to create the transform.
Suggestions: As stated you can extract the setup.exe or try to run it silently. I prefer extraction, but let me suggest some possible command lines for silent install via setup.exe. I don't know the details of your setup.exe, but let me just make a few guesses:
Basic MSI:
setup.exe /s /v"TRANSFORMS=\"C:\Transforms\1.mst\""
Installscript MSI:
Step 1: Record response file:
setup.exe /r /f1"c:\temp\my-answer-file.iss"
Step 2: Basic silent install (using the response file) & applying a transform:
setup.exe /s /f1"c:\temp\my-answer-file.iss" /v"C:\Transforms\1.mst"
Creating Transform: You can create a transform in Orca, or any other MSI deployment tool. Free Tools. Main Tools (free and commercial). In Orca you open an MSI, and then go Transforms => New Transform. You then change what you need to change - in your case the Property table and SecureCustomProperties. When you are ready: Transforms => Generate Transform... Now save the transform.
Installshield Help File:
There are several relevant sections in the Installshield help file.
Please study these if you need more tweaking of the installation parameters. All switches are documented here - these links are for
the 2018 edition of Installshield:
