running a bat file though python in current process

Question

I am attempting to build a large system through a python script. I first need to set up the environment for Visual Studio. Having problems I decided to see if I could just set up and launch Visual Studio. I first set several environment variables and then call C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\vcvarsall.bat x64.

Once this finishes I call devenv /useenv. If I do these from the command prompt everything works fine and I can do what I need to do in VS. My python code for doing this is:

import os
vcdir=os.environ['ProgramFiles(x86)']
arch = 'x64'
command  = 'CALL "' +vcdir+'\\Microsoft Visual Studio 11.0\\VC\\vcvarsall.bat" '+arch
os.system(command)
command = "CALL devenv /useenv"
os.system(command)

If I run this, the bat file will run and when it tries the devenv command I get that it is not recognized. It looks like to bat file runs in a different subprocess than the one that the script is running in. I really need to get this running in my current process. My eventual goal is to do the entire build inside the python script and there will be many calls to devenv to do a major portion of the build.

Thank you.

@JakobBowyer: No, `subprocess` by itself will not solve this problem because it does not address the underlying issue. — martineau, Feb 05 '13 at 09:02
@MichaelB I have a similar problem as yours. I asked [this] (http://stackoverflow.com/questions/38790813/how-can-i-run-cl-using-x64/38791170#38791170) question and now I'm able to compile thanks to the answer. What I want to do now is execute this command from `os.system()`. Do you know how can I configure it, so that `os.system()` run the cmd in Visual Studio x64 Win64 Command Prompt (2010) and before running the cmd executes the vcvarsall.bat file ? Thanks — lads, Aug 05 '16 at 15:16

score 5 · Answer 1 · answered Oct 02 '14 at 19:26

I had the exact same problem as you. I was trying to run vcvarsall.bat as part of a build script written in python and I needed the environment created by vcvarsall. I found a way to do it. First create a wrapper script called setup_environment.bat:

call "C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\vcvarsall.bat" amd64
set > environment.txt

Then in your python script where you would have called vcvarsall.bat, run the wrapper script, then read the environment variables from the text file into your current environment:

    this_dir = os.path.dirname(os.path.realpath(__file__)) # path of the currently executing script
    os.system('call ' + this_dir + '\setup_environment.bat') # run the wrapper script, creates environment.txt
    f = open('environment.txt','r')
    lines = f.read().splitlines()
    f.close()
    os.remove('environment.txt')
    for line in lines:
        pair = line.split('=',1)
        os.environ[pair[0]] = pair[1]

score 5 · Answer 2 · answered Sep 11 '19 at 06:57

Here is a simple example, should work out of box:

import os
import platform


def init_vsvars():
    vswhere_path = r"%ProgramFiles(x86)%/Microsoft Visual Studio/Installer/vswhere.exe"
    vswhere_path = os.path.expandvars(vswhere_path)
    if not os.path.exists(vswhere_path):
        raise EnvironmentError("vswhere.exe not found at: %s", vswhere_path)

    vs_path = os.popen('"{}" -latest -property installationPath'.format(vswhere_path)).read().rstrip()
    vsvars_path = os.path.join(vs_path, "VC\\Auxiliary\\Build\\vcvars64.bat")

    output = os.popen('"{}" && set'.format(vsvars_path)).read()

    for line in output.splitlines():
        pair = line.split("=", 1)
        if(len(pair) >= 2):
            os.environ[pair[0]] = pair[1]

if "windows" in platform.system().lower():
    init_vsvars()
    os.system("where cl.exe")

Please note that environment variables don't have effect after python script exits.

steveha · Accepted Answer · 2013-02-04T23:46:54.130

1

What you are trying to do won't work. vcvarsall.bat sets environment variables, which only work inside a particular process. And there is no way for a Python process to run a CMD.exe process within the same process space.

I have several solutions for you.

One is to run vcvarsall.bat, then figure out all the environment variables it sets and make Python set them for you. You can use the command set > file.txt to save all the environment variables from a CMD shell, then write Python code to parse this file and set the environment variables. Probably too much work.

The other is to make a batch file that runs vcvarsall.bat, and then fires up a new Python interpreter from inside that batch file. The new Python interpreter will be in a new process, but it will be a child process under the CMD.exe process, and it will inherit all the environment variables.

Or, hmm. Maybe the best thing is just to write a batch file that runs vcvarsall.bat and then runs the devenv command. Yeah, that's probably simplest, and simple is good.

The important thing to note is that in the Windows batch file language (and DOS batch file before it), when you execute another batch file, variables set by the other batch file are set in the same environment. In *NIX you need to use a special command source to run a shell script in the same environment, but in batch that's just how it works. But you will never get variables to persist after a process has terminated.

edited Feb 04 '13 at 23:46

answered Feb 04 '13 at 23:41

steveha

74,789
21
92
117

It _is_ possible to make persistent changes to environment variables in batch files on Windows by using the `setx` command as mentioned in [this](http://superuser.com/a/524449/57398) answer. You might need to modify the `vcvarsall.bat` script to use it, though. – martineau Feb 05 '13 at 00:53
Hmm, I didn't know about `setx`, but I just looked it up. It appears to be a command for setting the appropriate registry entries to set the environment variables when starting a new `CMD.exe` shell. This will not, by itself, cause the variable values to be set in the copy of the environment that was passed to Python when Python first ran. And I am pretty sure that you don't want to make the `vcvarsall.bat` changes permanent; I believe that is supposed to set things up for a single session with Visual Studio. – steveha Feb 05 '13 at 03:48
By the way, I just checked, and there is a Python native equivalent of `setx` in the `_winreg` module: `SetValue()` or `SetValueEx()`. – steveha Feb 05 '13 at 03:50
`SetValue()` and `SetValueEx()` in `_winreg` have nothing to do with the `set` or `setx` batch commands. One big thing is they both affect the currently running shell process. It might be possible to to use one of the former to set the registry values used by new `cmd.exe` instances (which may be something `setx` also does). The registry keys that affect `cmd.exe` start-up are `HKLM\SOFTWARE\Microsoft\Command Processor\AutoRun` and `HKCU\Software\Microsoft\Command Processor\AutoRun`. Both are described in [this](http://technet.microsoft.com/en-us/library/bb490880.aspx) MS article. – martineau Feb 05 '13 at 08:49
@martineau, if I understand the documentation for `setx` correctly, it changes registry keys to cause the environment variables to persist; that's why I mentioned Python registry key functions. I'm sure `setx` also changes the environment. But the copy of the environment for the `CMD.exe` process will be lost when that process terminates. When a new process is created, it gets a copy of the parent process's environment as it was at the moment when the parent process created the child process; when the child process terminates, the parent process's environment variables are not changed. – steveha Feb 05 '13 at 08:57
I said one thing earlier that was wrong, variables set using `setx` are only available in future command windows vur not in the current one. – martineau Feb 05 '13 at 10:14

score 1 · Answer 4 · answered May 02 '18 at 13:10

Building on the answer by @derekswanson08 I came up with this which is a bit more fully-fledged. Uses wvwhere.exe which comes with VS 2017.

def init_vsvars():
    cprint("")
    cprint_header("Initializing vs vars")

    if "cygwin" in platform.system().lower():
        vswhere_path = "${ProgramFiles(x86)}/Microsoft Visual Studio/Installer/vswhere.exe"
    else:
        vswhere_path = r"%ProgramFiles(x86)%/Microsoft Visual Studio/Installer/vswhere.exe"
    vswhere_path = path.expandvars(vswhere_path)
    if not path.exists(vswhere_path):
        raise EnvironmentError("vswhere.exe not found at: %s", vswhere_path)

    vs_path = common.run_process(".", vswhere_path,
                                 ["-latest", "-property", "installationPath"])
    vs_path = vs_path.rstrip()

    vsvars_path = os.path.join(vs_path, "VC/Auxiliary/Build/vcvars64.bat")

    env_bat_file_path = "setup_build_environment_temp.bat"
    env_txt_file_path = "build_environment_temp.txt"
    with open(env_bat_file_path, "w") as env_bat_file:
        env_bat_file.write('call "%s"\n' % vsvars_path)
        env_bat_file.write("set > %s\n" % env_txt_file_path)

    os.system(env_bat_file_path)
    with open(env_txt_file_path, "r") as env_txt_file:
        lines = env_txt_file.read().splitlines()

    os.remove(env_bat_file_path)
    os.remove(env_txt_file_path)
    for line in lines:
        pair = line.split("=", 1)
        os.environ[pair[0]] = pair[1]

Mahdi Yusuf · Answer 5 · 2013-02-05T14:34:54.727

0

What you should be using is a module in the standard library called subprocess

I have linked you to an example in the standard library here.

Here is an example with your situation.

import shlex, subprocess
args = shlex.split(your_command)
p = subprocess.Popen(args)

That should execute it also return stderr if you need to know what happened with the call. your command might even be a third bat file that encompasses the two bat files so you get all the environment variables.

edited Feb 05 '13 at 14:34

answered Feb 04 '13 at 23:45

Mahdi Yusuf

19,931
26
72
101

I'm afraid this won't solve the problem of the environment variables set by the first command executed (`vcvarsall.bat`) persisting when the next one (`devenv`) is run. – martineau Feb 05 '13 at 00:58
@martineau so set them in the python script. – Mahdi Yusuf Feb 05 '13 at 01:40
Because doing so would essentially amount to converting the `vcvarsall.bat` batch file provided by Visual Studio into a Python script. – martineau Feb 05 '13 at 01:46
This answer will not work. @martineau told you, and now I am telling you. `subprocess` will start a new process, the `vcvarsall.bat` script will set environment variables inside that process, then the process will terminate and the environment variable settings will be lost. Then when the Python script runs the command to launch Visual Studio, none of the environment variables will be set. Probably the best solution is to make a batch file that first runs `vcvarsall.bat` and then runs the `devenv` command; if they are in the same batch file, then `devenv` will run in the same process. – steveha Feb 05 '13 at 03:57
@steveha: I tend to agree that creating a batch file to run the two commands and then executing it from Python sounds like the simplest solution. The new batch file could easily be created by Python and then deleted after it has run. – martineau Feb 05 '13 at 08:58
@steveha I have updated the answer you are correct I forgot about the fact that environments variable are lost once you leave the subprocess. – Mahdi Yusuf Feb 05 '13 at 14:36

score 0 · Answer 6 · answered Mar 07 '17 at 15:07

0

Another solution is to call vcvarsall.bat within the same os.system than the build command:

os.system("cd " + vcFolder + " & vcvarsall.bat amd64 & cd " + buildFolder + " & make")

answered Mar 07 '17 at 15:07

matthieu

343
2
7

running a bat file though python in current process

6 Answers6

Linked