OK, not sure why you refer to ClickOnce and WiX together - I am not aware of any connections here - they are different technologies - but maybe there are connections I am not aware of (something relating to AppX maybe).
The file you refer to: setup.bin
is installed via two MSI files on my system:
Microsoft.ClickOnce.BootStrapper.Msi.msi
(installed from source at C:\ProgramData\Microsoft\VisualStudio\Packages\Microsoft.ClickOnce.BootStrapper.Msi,version=15.0.27005.2)
Microsoft.ClickOnce.SignTool.Msi.msi
(installed from source at C:\ProgramData\Microsoft\VisualStudio\Packages\Microsoft.ClickOnce.SignTool.Msi,version=15.0.27005.2)
Did you go to Tools => Get Tools and Features => Individual Components
and select "ClickOnce Publishing"? I would try that first. There is no clear match between the actual MSI names and these "Individual Components" that I can see.
I believe "Microsoft Visual Studio 2017 Installer Projects", "WiX Toolset Visual Studio 2017 Extensions" and "IsWiX" (front end for WiX) can be installed from the Tools => Extension and Updates
. Or you can install the WiX 2017 extension from here.
You also obviously need to install the WiX toolkit itself to be able to make WiX installers. Here is the download location: http://wixtoolset.org/releases/ (and a WiX quick start suggestion answer).