The flags you mention all set properties for the MSI, which you can then reference in your custom action, converting to whatever format you need for your setup.exe.
For instance:
/i
corresponds to the value of Installed
being false (if the product isn't already installed).
/r
corresponds to a value of "Repair" on the WixUIInstallMode
property.
/x
corresponds to a value of "Remove" on the WixUIInstallMode
property.
/q
(and its modifiers like /qn
and /qb
) correspond to the UILEVEL
property, which has the following values found from this link.
UILevel:
INSTALLUILEVEL_NONE 2 Completely silent installation.
INSTALLUILEVEL_BASIC 3 Simple progress and error handling.
INSTALLUILEVEL_REDUCED 4 Authored UI, wizard dialogs suppressed.
INSTALLUILEVEL_FULL 5 Authored UI with wizards, progress, errors.
You can see this answer if you're unfamiliar with passing values and referencing them in a custom action.