Import python file from another folder that is not a child

Question

What I want

I'm using Visual Studio Code and Python 3.7.0 and I'm just trying to import another Python file, from another folder, into my python file.

Details

Here is my folder structure

root/
    dir1/
        data.txt
        task11.py
        task12.py
    dir2/
        data.txt
        task21.py
        task22.py
    Helpers/
        FileHelper/
            ReadHelper.py

So a short explanation:

I use the same function in every "task"-file
Instead of putting the function in every "task"-file, I've created a helper file where the function exists
I want to import the helper file "ReadHelper.py" into my task files

What I've tried

e.g. in the file task11.py:

from Helpers.FileHelper.ReadHelper import *

import os, sys
parentPath = os.path.abspath("../../")
if parentPath not in sys.path:
    sys.path.insert(0, parentPath)
from Helpers.FileHelper.ReadHelper import *

import os, sys
sys.path.append('../../')
from Helpers.FileHelper.ReadHelper import *

None of the above solutions works as I always end up with the error: ModuleNotFoundError: No module named 'Helpers'

I've also tried:

from ..Helpers.FileHelper.ReadHelper import *

But it ends up with the error: ValueError: attempted relative import beyond top-level package

So how can I import the file ReadHelper.py to my task files?

P.S

There are some similar questions to this but they are really old and the answers have not helped me.

Update 1

There is an option in Visual Studio code, If I run this command with this import from Helpers.FileHelper import ReadHelper then no errors are generated and the code executes perfectly.

One downside is that this interactive window is slow at starting and it cannot handle inputs.

I tried the answer of @Omni as well:

$> python -m root.dir1.task11

And it worked! but as he said, there is a downside: it is slow to type in the terminal.

So I tried to create a task in Visual Studio Code that could execute the above shell command for the file that I'm currently in, but did not succeed.

Do you know how to create a task in vscode to run the above command?

I've also tried to add __init__.py-files under every directory so they would be seen as packages Python3 tutorial - 6.4 Module Packages. But this didn't help and the same error occurred.

Update 2

I come up with a way to make it really easy to have a folder structure like this and get the imports to work correctly in the terminal.

Basically what I did was:

created a Python script
created a task in visual studio code

With this, I can now run my python files, with the imports, by only pressing cmd + shift + B.

Explanation

The visual studio task:

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "Run python file",
            "type": "shell",
            "command": "python3 /PATH_TO_ROOT_FOLDER/run_python_file.py ${file}",
            "group": {
                "kind": "build",
                "isDefault": true
            },
            "presentation": {
                "reveal": "always",
                "panel": "new",
                "focus": true
            }
        }
    ]
}

The part that we want to focus on is this one:

"command": "python3 /PATH_TO_ROOT_FOLDER/run_python_file.py ${file}",

This part runs the new python file I created at the root folder and passes the path, of the file which is active, as a parameter

The python script:

import os, sys

# This is a argument given trough a shell command
PATH_TO_MODULE_TO_RUN = sys.argv[1]

ROOT_FOLDER = "root/"

def run_module_gotten_from_shell():
    # Here I take only the part of the path that is needed
    relative_path_to_file = PATH_TO_MODULE_TO_RUN.split(ROOT_FOLDER)[1]
    
    # Creating the shell command I want to run
    shell_command = createShellCommand(relative_path_to_file)

    os.system(shell_command)


# Returning "python3 -m PATH.TO.MODULE"
def createShellCommand(relative_path_to_file):
    part1 = "python3"
    part2 = "-m"

    # Here I change the string "dir1/task11.py" => "dir1.task11"
    part3 = relative_path_to_file.replace("/", ".")[:-3]

    shell_command = "{:s} {:s} {:s}".format(part1, part2, part3)
    return shell_command

run_module_gotten_from_shell()

This python script gets as a parameter the path to the active file
Then it creates a shell command of the path (the shell command is like @kasper-keinänen 's answer)
Then it runs that shell command

With these modifications, I can run any file inside the root directory with imports from any file inside the root directory.

And I can do it by only pressing cmd + shift + B.

score 3 · Answer 1 · answered Dec 02 '19 at 21:57

3

Adding the full absolute path to the sys.path variable should make it work.

import sys
sys.path.append('/full/path/to/Helpers/FilesHelper/')

from ReadHelper import *

answered Dec 02 '19 at 21:57

jaouena

98
6

I thought of this, but as I would like to import many helpers from different repositories in the future, which makes this solution not that convenient. – OuuGiii Dec 03 '19 at 02:47
I cannot make comment under other messages but you say that making `__init__.py` files doesn't resolve your problem. Can you print the `sys.path` variable when you run a task file ? and what did you write on your `__init__.py` ? – jaouena Dec 03 '19 at 11:06
When I tried with `__init__.py` the files where empty. should I put anything inside them? and sys.path printed a list containing [PATH_TO_TASK_FILE, SOME_PYTHON_3.7_STUFF], do you want to know more of what is in the sys.path? – OuuGiii Dec 03 '19 at 13:44
you should read about the `__init__.py` files and the module name search logic in python. like [this subect](https://stackoverflow.com/questions/714063/importing-modules-from-parent-folder). sometimes you just have to change the directories organisation. – jaouena Dec 03 '19 at 14:09
It seems odd that I have to change my directories, but I really like the structure of my project so I will not change my directories. I also found a solution which I will publish as an answer soon. – OuuGiii Dec 03 '19 at 16:01

score 2 · Accepted Answer · answered Dec 03 '19 at 06:51

2

You could try running the script with the -m option that allows modules to be located using the Python module namespace docs.python.org.

If you run the task11.py script then:

$ python3 -m dir1.task11

And in the task11.py do the import like:

from Helpers.FileHelper.ReadHelper import *

answered Dec 03 '19 at 06:51

Kasper Keinänen

114
5

Nice, this solution is similar to @omni 's solution. This is still the best suggestion (that has worked) for me. – OuuGiii Dec 03 '19 at 16:06
I put this as the correct answer because it is a working solution for me, which is easy to understand, and it doesn't say I have to change my folder structure. I've also put an alternative solution, as an update, in my question. My solution makes it really easy to run python files by using vscode tasks, and the solution of this answer. – OuuGiii Dec 05 '19 at 00:08

score 2 · Answer 3 · answered Jun 16 '21 at 22:22

2

If you're only trying to do this in VSCode, and not during normal run time. You can add the path in the .vscode/settings.json

{
    "python.analysis.extraPaths": [
        "${workspaceFolder}/webapp"
    ],
}

NOTE: This does not solve standard Python importing. My use-case was specific to a monolithic project where all editor config files where in the root and thus I couldn't open 'webapp/' as a workspace itself.

answered Jun 16 '21 at 22:22

M.Vanderlee

2,847
2
19
16

Exactly what I was looking for as someone working with AWS Lambda layers. I should point out that if you are using Pylance, the path is already relative to the working directory, so put `webapp` instead of `${workspaceFolder}/webapp`. See https://github.com/microsoft/pylance-release/issues/29#issuecomment-653053357 – thenewpotato Jul 07 '21 at 16:16

score 1 · Answer 4 · answered Dec 02 '19 at 22:09

a) Execute the task modules as scripts within an environment that knows about the helper functions. That way the code in the taks modules does not have to know anything about the package structure present. It imitates the builtins of the python interpreter.

   # cli argument #1 is the task module to execute
   import sys  
   task_to_execute = sys.argv[1]

   from Helpers.FileHelper.ReadHelper import *
   exec(open(task_to_execute).read())

b) Use relative imports correctly. In order to do so, you have to execute the task code via (this might be a disadvantage of this solution).

   $> python -m root.dir1.task11.task11

score 0 · Answer 5 · answered Dec 02 '19 at 22:25

The problem is your file/folder structure. I would suggest creating a sort of 'control' file in your root folder, which can then work from the top-down to reference all your other modules.

So let's say you had a file in your root folder called MasterTask.py it could look like this:

from dir1.task11.task11 import *
from dir1.task12.task12 import *
from dir2.task21.task21 import *
from dir2.task22.task22 import *
from Helpers.FileHelper.ReadHelper import *

class Master:
#Do your task work here
    pass

One other option would be to move the Helpers folder into your Python37\Lib\site-packages folder, which would also allow the use of from Helpers.FileHelper.ReadHelper import * as is - assuming that you are not planning on this to be used on other machines other than your own.

This should have been the accepted answer, as it was posted before the currently accepted one and is every bit as simplified. — NL23codes, Jan 22 '20 at 18:56