python subprocess multiple commands with win path

Question

I'm trying to trigger the execution of a python script via conda.

I would then capture the output and report it to command prompt where this is executed.

This is basically the concept in the easiest way

wrap.py - wrapper inted to execute multiple times the following script

import subprocess
def wrap():
    while True:
        cmd1=r"call C:\\Users\\my_user\\anaconda3\\Scripts\\activate.bat"
        cmd2=r"cd C:\\myfolder\\mysubfolder"
        cmd3=r"C:\\Users\\my_user\\anaconda3\\python.exe C:\\myfolder\\mysubfolder\\test.py"
        proc = subprocess.run([cmd1,cmd2,cmd3])
if __name__ == '__main__':
    wrap()

test.py - script that has to be executed

def mytest():
   print("success")
if __name__ == '__main__':
    mytest()

since mytest prints success once, I would like the output of the wrapper (run on anaconda) to be

(base) C:\myfolder\mysubfolder> python wrap.py
success
success
success
...

I tried with

1 - subprocess.Popen

2 - using shell=True or not

3 - using a list ["first command","second command","third command"] or a single string "first;second;third"

4 - using or removing "r" in front of the string, here the blanks are breaking the game

5 - using single or double ""

6- in my_user the underscore is also resulting in an encoding error

I actually tried to replicate at least 20 different stackoverflow "solutions" but none of them really worked for me. I also read properly the subprocessing page of python documentation, but this didn't help.

Any hint is appreciated, I'm lost.

Answer 1

The syntax subprocess.run([cmd1, cmd2, cmd3]) means run cmd1 with cmd2 and cmd3 as command-line arguments to cmd1. You instead want to execute a single sequence of shell commands; several of the things you are trying to do here require the shell, so you do want shell=True, which dictates the use of a single string as input, rather than a list consisting of a command and its arguments.

(Windows has some finicky processing behind the scenes which makes it not completely impossible to use a list of strings as the first argument with shell=True; but this really isn't portable or obvious. Just don't.)

Regarding the requirement for shell=True here, commands like call and cd (and source or . in Bourne-family shells) are shell built-ins which do not exist as separate binaries; if you don't have shell=True you will simply get "command not found" or your local equivalent. (Under other circumstances, you should generally avoid shell=True when you can. But this is not one of those cases; here, it really is unavoidable without major code changes.)

If your shell is cmd I guess the command might look like

subprocess.run(
    r"call C:\Users\my_user\anaconda3\Scripts\activate.bat & C:\Users\my_user\anaconda3\python.exe C:\myfolder\mysubfolder\test.py",
    shell=True)

or equivalently the same without r before the string and with all backslashes doubled; the only difference between an r"..." string and a regular "..." string is how the former allows you to put in literal backslashes, whereas the latter requires you to escape them; in the former case, everything in the string is literal, whereas in the latter case, you can use symbolic notations like \n for a newline character, \t for tab, etc.

In Python, it doesn't really matter whether you use single or double quotes; you can switch between them freely, obviously as long as you use the same opening and closing quotes. If you need literal single quotes in the string, use double quotes so you don't have to backslash-escape the literal quote, and vice versa. There's also the triple-quoted string which accepts either quoting character, but is allowed to span multiple lines, i.e. contain literal newlines without quoting them.

If your preferred shell is sh or bash, the same syntax would look like

subprocess.run(r"""
    source C:\Users\my_user\anaconda3\Scripts\activate.bat &&
    C:\Users\my_user\anaconda3\python.exe C:\myfolder\mysubfolder\test.py""",
    shell=True)

I left out the cd in both cases because nothing in your code seems to require the subprocess to run in a particular directory. If you do actually have that requirement, you can add cwd=r'C:\myfolder\mysubfolder' after shell=True to run the entire subprocess in a separate directory.

There are situations where the facilities of subprocess.run() are insufficient, and you need to drop down to bare subprocess.Popen() and do the surrounding plumbing yourself; but this emphatically is not one of those scenarios. You should stay far away from Popen() if you can, especially if your understanding of subprocesses is not very sophisticated.