python setup.py build_ext in Windows Azure pipeline

Question

I read many questions/answers here but didn't find a solution for my case.

I'm trying to establish Windows testing of my SW (Python + C). I'm using Azure in my GitHub project to test it. I need command line solution for *.bat file.

my batch script I call

pip install --upgrade setuptools # required by VS
python setup.py build_ext --inplace

and receive error:

error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

I tried to modify my script as following:

pip install --upgrade setuptools
"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installershell.exe" ^
  --add Microsoft.VisualStudio.Workload.NativeDesktop                            ^
  --includeOptional                                                              ^
  --includeRecommended                                                           ^
  --nocache                                                                      ^
  --quiet
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build\vcvars64.bat"
python setup.py build_clib

but it looks like it hangs during execution because VC installer requires GUI answers.

Could someone please give me an advice what should I modify in my script or provide a link for some solution?

In short form "How to install 'Microsoft C++ Build Tools' by command line without user involvement"?

Thank you

Answer 1

Ok, it looks like I gave up with "VS Build Tools" installation under Azure. I didn't find any workable way to do this.

What I did:

Part 1.

To solve my original issue ("python setup.py build_ext --inplace" under Windows)

setup.py modification required only:

if sys.platform in ['win32', 'cygwin']:
    os.environ["DISTUTILS_USE_SDK"] = "1"

I looked into setuptools sources and found it tried to "evaluate" build environment in Windows. It tries to find many diffrent variables and failed. Setting this particular environment variable prevent such internal logic and uses preset variables like "CC" to get a compiler name. Didn't investigate this logic deeply because I already lost ~week to push simple things to work on windows. (Python reduces time for programming? aha.. for "hello world" applications.) So, no need to install "VS Build Tools" in this case (to the question about valuable error messages in Python).

Part 2.

These are my failed attempts to install "VS Build Tools" under Azure environment. Perhaps it will helpful for someone in future.

Downloading:

curl -o webimage.exe        ^
  --retry 5 --retry-delay 5 ^
  -L https://download.visualstudio.microsoft.com/download/pr/9b3476ff-6d0a-4ff8-956d-270147f21cd4/ccfb9355f4f753315455542f966025f96de734292d3908c8c3717e9685b709f0/vs_BuildTools.exe

I'm not sure about the link itself. It worked today with no guarantee.

Print current VS configuration:

echo =============== VS config ===============
"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installer.exe" ^
  export ^
  --installPath "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise" ^
  --config "%CD%\vs_config.txt" ^
  --passive
type "%CD%\vs_config.txt"
del "%CD%\vs_config.txt"

This config remains the same before and after installation. Perhaps I printed it partially but I tired to investigate these tricky things. Set of components are ambiguous because I had no chance to make it work and reduce it.

vs_builttool.exe Installation

start /b /wait webimage.exe ^
    --add Microsoft.VisualStudio.Component.Roslyn.Compiler ^
    --add Microsoft.Component.MSBuild ^
    --add Microsoft.VisualStudio.Component.CoreBuildTools ^
    --add Microsoft.VisualStudio.Workload.MSBuildTools ^
    --add Microsoft.VisualStudio.Component.Windows10SDK ^
    --add Microsoft.VisualStudio.Component.VC.CoreBuildTools ^
    --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 ^
    --add Microsoft.VisualStudio.Component.VC.Redist.14.Latest ^
    --add Microsoft.VisualStudio.Component.Windows10SDK.18362 ^
    --add Microsoft.VisualStudio.Component.VC.CMake.Project ^
    --add Microsoft.VisualStudio.Component.TestTools.BuildTools ^
    --add Microsoft.VisualStudio.Component.VC.ATL ^
    --add Microsoft.VisualStudio.Component.VC.ATLMFC ^
    --add Microsoft.Net.Component.4.8.SDK ^
    --add Microsoft.Net.Component.4.6.1.TargetingPack ^
    --add Microsoft.VisualStudio.Component.VC.CLI.Support ^
    --add Microsoft.VisualStudio.Component.VC.ASAN ^
    --add Microsoft.VisualStudio.Component.VC.Modules.x86.x64 ^
    --add Microsoft.VisualStudio.Component.TextTemplating ^
    --add Microsoft.VisualStudio.Component.VC.CoreIde ^
    --add Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core ^
    --add Microsoft.VisualStudio.Component.VC.Llvm.ClangToolset ^
    --add Microsoft.VisualStudio.Component.VC.Llvm.Clang ^
    --add Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Llvm.Clang ^
    --add Microsoft.VisualStudio.Component.Windows10SDK.17763 ^
    --add Microsoft.VisualStudio.Component.Windows10SDK.17134 ^
    --add Microsoft.VisualStudio.Component.Windows10SDK.16299 ^
    --add Microsoft.VisualStudio.Component.VC.v141.x86.x64 ^
    --add Microsoft.Component.VC.Runtime.UCRTSDK ^
    --add Microsoft.VisualStudio.Component.VC.140 ^
    --add Microsoft.VisualStudio.Workload.VCTools ^
    --includeOptional --includeRecommended --nocache --wait --passive --quiet ^
    --installpath "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise"

del webimage.exe

Important options here are:

--wait

Prevent shell to run the process in parallel. You need to make sure it finishes before go further. I don't understand who needs such default behavior (default - non blocking shell command execution)

--passive 

Something like "--yes" in good utilities. No user interaction required with this option.

I hope this post will be helpful because I didn't find and answer to my original question before.

PS powershell attempt:

Start-Process -Wait -FilePath  "C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installer.exe" -ArgumentList "modify --add Microsoft.VisualStudio.Workload.NativeDesktop --add Microsoft.Component.MSBuild --add Microsoft.VisualStudio.Component.Windows10SDK --add Microsoft.VisualStudio.Component.VC.CoreBuildTools --passive --norestart --installpath ""C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise"""

Answer 2

You would better use VS Build Tool since you just want to use vc build tool. VS IDE is too big that it might hang for a long time. VS Build Tool is lightweight.

Also, --includeOptional will install other redundant components which will extend your installation time. It might cause some accidental problems during the process.

Besides, use --wait. It will wait for the Visual Studio installer to complete before executing the next command.

1) download VS Build Tool under https://visualstudio.microsoft.com/visual-cpp-build-tools/

2) use these command:

xxx\vs_buildtools.exe --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended --nocache --quiet --wait

In addition, no matter how, the Installer program needs administrator right to run, so the prompt box will appear, so no matter how you did, the prompt will appear and you have to click it.

Update 1

Thanks for your feedback and know that you have a bit right from the VM. And it seems that you can not get the buildtool.exe easily from Internet by GUI.

Obtaining buildtool.exe through scripts seems a bit complicated, so it is more convenient to use vs_installer.exe.

Try vs_installer.exe with modify switch:

"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installershell.exe" modify --installPath "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional"  --add Microsoft.VisualStudio.Workload.NativeDesktop --includeRecommended  --nocache --quiet

Note that you should run the bat file as Administrator, otherwise, it will turns out the Administrator privileges error.

It will first execute the modification and when it finishes, it will run the python file.