How to receive an argument in a python script while running it through subprocess in another python script?

Question

I am not so good at asking questions here so please let me know if any detail is missing.

I am trying to run a Python script ("o.py") multiple times in a loop from another script ("s.py"). o.py is in a subfolder inside the directory where s.py is located. I want to access a variable that I set in s.py (variable "l") in o.py. But since I am running o.py as a subprocess (I cannot run it any other way as I did not develop this part) I cannot import the variable from s.py to o.py because of some constraints. So as a work around I would like to run the script o.py from s.py with "l" as the input variable. I would like to know how can I accomplish this in both s.py and o.py files. I read some answers but everyone has mentioned the code for the subprocess function, but nothing for receiving the variable in the file being executed through the subprocess (in this case o.py).

Please note that I need to parallelize this process (multiprocessing) and every time, for each process the value of "l" should not change until the subprocess is finished.

Here is a sample of how s.py and o.py look like:

s.py

import multiprocessing as mp
import subprocess

def processToParallelize():
    subprocess.call(["python_path" , "o.py"], input = l, text=true)
    subprocess.call(["python_path" , "o.py"], input = l, text=true)
    subprocess.call(["python_path" , "o.py"], input = l, text=true)

if __name__ == '__main__':
        NUMBER_OF_TASKS =2
        pool = mp.Pool(NUMBER_OF_TASKS)

        pool.map(processToParallelize, range(4))

        pool.close()
        pool.join()

o.py

r = # this should be equal to "l"
x = someoperation(r)

The other questions that were asked related to this topic did not share the information about how to receive the input argument in the subprocess files. That was an important part which I needed to be resolved.

Does this answer your question? [Python subprocess arguments](https://stackoverflow.com/questions/11679936/python-subprocess-arguments) — Marco F., Jul 12 '23 at 15:06
That's not your actual code though is it? Show us what you're actually trying to run and tell us what does and does not happen. Also, why would you have 3 synchronous calls in a subprocess. Instead of trying to do that 4 times, why not do the subprocess call once and run 12 subprocesses? — DarkKnight, Jul 12 '23 at 15:20
Where did this `l` come from? Is it a global variable? Something passed to the pool worker function? — tdelaney, Jul 12 '23 at 16:10
I called the sub process thrice because the subsequent process uses data from the previous process. Also, I have limitation of using number of processors based on other users. — ArbitraryConstant, Jul 13 '23 at 09:20
"l" is a variable which changes the source of input data for the subprocesses, but it has to be changed only after the cycle of three subprocesses are finished. — ArbitraryConstant, Jul 13 '23 at 09:21

score 0 · Answer 1 · answered Jul 12 '23 at 15:39

It looks as though you're trying to pass information to o.py via the standard input stream (stdin). If that's the case then o.py needs to be consuming data from stdin.

Let's assume that o.py looks like this:

print('>>>>', input(), '<<<<')

i.e., it reads from stdin and prints the value between "chevrons".

Now our calling program could look something like this:

import subprocess
import multiprocessing

N = 5

PYTHON = '/Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11'

def process(n: int):
    subprocess.run([PYTHON, 'o.py'], input=str(n), text=True)

def main():
    with multiprocessing.Pool(N) as pool:
        pool.map(process, range(N))

if __name__ == '__main__':
    main()

Output:

>>>> 0 <<<<
>>>> 2 <<<<
>>>> 1 <<<<
>>>> 4 <<<<
>>>> 3 <<<<

The order of output will be indeterminate.

Note: subprocess.run rather than the outdated subprocess.call

score 0 · Accepted Answer · answered Jul 12 '23 at 16:13

0

Since your data is a simple string, use a command argument.

def processToParallelize():
    subprocess.call(["python_path" , "o.py"], args=(l,))

o.py

import sys
l = sys.argv[1]
print(l)

Its also common to use environment variables.

import sys

def processToParallelize():
    sys.environ["l"] = l
    subprocess.call(["python_path" , "o.py"])

o.py

import sys
l = sys.environ["l"]
print(l)

answered Jul 12 '23 at 16:13

tdelaney

73,364
6
83
116

Does the environment variable stay the same untill the process is finished or could it be modified by another process if it starts the execution while the first process is still running? – ArbitraryConstant Jul 13 '23 at 09:24
This solved my current problem, thanks!. However, in the future I might have to run the sub process o.py from a different route and I might not be able to pass an input argument in that case. So I was wondering if there is a way to access this variable while running things in parallel without importing the module? Because subprocess will be out of the picture in that case. – ArbitraryConstant Jul 13 '23 at 09:27
The environment variable is unique to that one process. No other processes see or change it. If that process goes on to create its own subprocesses, they will each get a copy of the variable but won't be able to change their parent's copy. – tdelaney Jul 13 '23 at 13:30
You could put its value in a file that can be read by the other processes. But if you have a special case where these methods don't work, you'd have to post a new question about that case. – tdelaney Jul 13 '23 at 13:33

How to receive an argument in a python script while running it through subprocess in another python script?

2 Answers2