Import many python files in bash script

Question

I have a problem with the import file in the bash script.

My project directory looks like this:

project
├── folder1
│   └── file1.py
├── folder2
│   └── file2.py
├── data
│   └── example.txt
├── utils
│   ├── __init__.py
│   └── globals.py
└── run.sh

The globals.py file contains some config global variables that will be used in the whole project.

I want to write a bash file to run all .py files like this run.sh:

conda activate pyenv

# Step 1.1
python folder1/file1.py data/example.txt >> data/output_1_1.txt

#  Step 1.2
python folder2/file2.py data/output_1_1.txt >> data/output_1_2.txt

But then I got this error:

Traceback (most recent call last):
  File "folder2/file2.py", line 15, in <module>
    from utils import globals
ModuleNotFoundError: No module named 'utils'

This is my file2.py where I import utils.globals:

import sys

sys.path.append("../")  # Add "../" to utils folder path
from utils import globals

When I run each file individually, it works fine, but I don't know why it doesn't work when I run source run.sh.

@bigbounty I just updated my directory, I already have `__init__.py` in the `utils` folder. — huy, May 03 '21 at 06:00
@bigbounty I create a `__init__.py` file in every folder but it still doesn't work. — huy, May 03 '21 at 06:02

huy · Accepted Answer · 2021-05-03T08:34:06.850

As this answer points out it is not good to use a relative path when the project structure is complex.

So I change my project directory to this to make it a package like @bigbounty's suggested:

project
├── package
│   ├── folder1
│   │   └── file1.py
│   ├── folder2
│   │   └── file2.py
│   ├── data
│   │   └── example.txt
│   └── utils
│       ├── __init__.py
│       └── globals.py
└── run.sh

In the file2.py, change the import to this:

from package.utils import globals

The run.sh file:

conda activate pyenv

# Step 1.1
python -m package.folder1.file1 package/data/example.txt >> package/data/output_1_1.txt

#  Step 1.2
python -m package.folder2.file2 package/data/output_1_1.txt >> package/data/output_1_2.txt

To run the bash script, cd to the /project:

(base) user@user1:~/project$ source run.sh

Then it works as expected.

Indeed. Something I always try to point out is that, except in some very special cases, if you find yourself writing `sys.path.insert...` in your package you're probably doing something wrong =) — Iguananaut, May 03 '21 at 08:33
@Iguananaut I did not notice that before. Now the project contains more and more files which makes me realize this problem. — huy, May 03 '21 at 08:36

Import many python files in bash script

1 Answers1