Python execute all Python files in directory and subdirs

Question

I'm trying using this code for the city wise updated population data but the python file is not writing the output results in that folder

import os
import subprocess
import glob

root_dir = "path/to/root/directory"

for dirpath, dirnames, filenames in os.walk(root_dir):
    for text_file in glob.glob(os.path.join(dirpath, '*.txt')):
        os.remove(text_file)

    for filename in filenames:
        if filename.endswith(".py"):
            filepath = os.path.join(dirpath, filename)
            os.system("python" + filepath)
            
            or 

            subprocess.run(["python", filepath])

It is deleting the old text files but python file is not generating the updated data and in the sub process it showing in the command prompt but didn't write the text files or even creating new text files

but when I manually go to the folder and run the Python file it works fine

Can you try: `subprocess.run(["python", filepath], cwd=dirpath, shell=True)` — Corralien, Jan 21 '23 at 23:33
@Corralien Good call on the `shell=True`, but I think the main issue is that the file runs itself and causes an infinite loop. — Michael M., Jan 22 '23 at 00:28
Great it works now and getting the data in the same folder where python exe is placed — Sarfraz, Jan 22 '23 at 06:57
@Corralien how can I mark your comment as a solution of my question thank you — Sarfraz, Jan 22 '23 at 07:06
@Sarfraz. Glad to read it works. I posted my comment as answer :-) — Corralien, Jan 22 '23 at 07:57
The requirement to run a large number of Python scripts is weird, and quite possibly an [XY problem](https://en.wikipedia.org/wiki/XY_problem). If the scripts are all similar, a better solution is to write a single script which adapts its behavior based on the data you feed it; but I'm obviously reading a lot between the lines here. Perhaps see also [DRY Principle](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) — tripleee, Jan 22 '23 at 08:13

Michael M. · Answer 1 · 2023-01-21T23:57:40.527

The issue with the line os.system("python" + filepath) is that there is no space between "python" and the file, so it will attempt to run something like pythontest.py, an invalid command. The second issue is that your program might run itself, causing an infinite loop. To fix this, check if the file is the current file with the __file__ variable before you run it. Try this code:

import os
import subprocess
import glob

root_dir = os.getcwd()
for dirpath, dirnames, filenames in os.walk(root_dir):
    for text_file in glob.glob(os.path.join(dirpath, '*.txt')):
        os.remove(text_file)

    for filename in filenames:
        if filename.endswith(".py") and os.path.join(root_dir, filename) != __file__:
            filepath = os.path.join(dirpath, filename)
            subprocess.run(["python", filepath])

Note that I also changed root_dir to automatically get the current directory. You can change this if you want.

Also, thanks to MrChadMWoods for helping catch an edge case in this current file detection.

Is `os.getcwd()` preferred to `os.path.realpath(__file__)`? By my understanding, `os.getcwd()` does a bit more than returning the current working directory. It defaults to $PWD in your env. It's not where the script is located but where you were when you executed the script. On the other hand. `__file__` is the path of the module file. So you have to use this to be certain to have a path relative to your module instead of the current working directory that may change. — MrChadMWood, Jan 21 '23 at 23:51
@MrChadMWood You're completely right, thanks for the help! I've edited my answer to use a variation of that instead (and provided attribution to you, of course). — Michael M., Jan 21 '23 at 23:58

Corralien · Accepted Answer · 2023-01-22T08:19:27.493

1

You need to change the current working directory for your script to work:

Try:

subprocess.run(["python", filepath], cwd=dirpath)

edited Jan 22 '23 at 08:19

answered Jan 22 '23 at 07:56

Corralien

109,409
8
28
52

`shell=True` is actually wrong here; it will work on Windows for murky reasons, but not on other platforms. See also [Actual meaning of `shell=True` in `subprocess`](https://stackoverflow.com/questions/3172470/actual-meaning-of-shell-true-in-subprocess) – tripleee Jan 22 '23 at 08:07
1

Thanks @tripleee. I updated my answer according your comment. – Corralien Jan 22 '23 at 08:19

Python execute all Python files in directory and subdirs

2 Answers2