10

I am making a package, and the modules within this package have code inside if __name__=='__main__': blocks for testing purposes. But my attempts to use relative imports in these modules causes errors.

I have read this thread and the billion others: Relative imports for the billionth time

Before you mark this as a duplicate, if what I want to do is not possible in Python3 then my question is why did it work in Python2 and what motivated the decision to make this such a hassle in Python3?

This is my sample Python project:

mypackage
- module1.py
- module2.py
- __init__.py

__init__.py and module2.py are empty

module1.py contains:

import module2

# module1 contents

if __name__=="__main__":
    # Some test cases for the contents of this module
    pass

This works fine in Python2. I am able to import module1 from other projects anywhere on my computer, and I'm also able to run module1 directly and have the code in the if block run.

However, this structure doesn't work in Python3. If I try import the module somewhere else it fails:

>>> from mypackage import module1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\_MyFiles\Programming\Python Modules\mypackage\module1.py", line 1, in <module>
    import module2
ModuleNotFoundError: No module named 'module2'

So I tried changing the first line to from . import module2, and that fixed it so I could import the module from anywhere successfully. But then when I try running module1 directly as a script I get this error:

Traceback (most recent call last):
  File "C:/_MyFiles/Programming/Python Modules/mypackage/module1.py", line 1, in <module>
    from . import module2
ImportError: cannot import name 'module2' from '__main__' (C:/_MyFiles/Programming/Python Projects/pgui/mypackage/module1.py)

I don't want to have to open a console and type python -m myfile every time I'm working on a module and want to run it directly as a script.

I want to be able to work on modules without adding their parent folder to PYTHONPATH by using relative imports like in Python2

Is there any better workaround or solution to these problems?

Machavity
  • 30,841
  • 27
  • 92
  • 100
pyjamas
  • 4,608
  • 5
  • 38
  • 70
  • 2
    "I don't want to have to open a console and type `python -m myfile` every time I'm working on a module and want to run it directly as a script." - as opposed to, what, opening a console and typing `python myfile.py`? How are you running your files currently? (Also it's `python -m packagename.modulename` for package submodules, not `python -m modulename`.) – user2357112 Feb 02 '19 at 06:40
  • 2
    I guess other peeps will tell you the proper python way, however, you could just use a` try ... except ImportError` to import it the way that works. Dive Into Python has an [example](https://www.diveinto.org/python3/your-first-python-program.html#importerror). – glumplum Feb 02 '19 at 06:47
  • 1
    @user2357112 Currently I'm running my files through the PyCharm IDE by clicking the run button in the GUI. I realize I could add -m to the configuration, but this seems like a hassle and I would prefer if my code worked when run normally, like in Python2. And I'd like to send this to someone to use without having to warn them and explain how to run it so as to avoid the gotchas of relative imports failing with __main__ blocks with a cryptic error. – pyjamas Feb 02 '19 at 06:59
  • @glumplum Thanks, that seems like a viable workaround if there's no 'proper' way. – pyjamas Feb 02 '19 at 07:11
  • Does this answer your question? [Python3 correct way to import relative or absolute?](https://stackoverflow.com/questions/28400690/python3-correct-way-to-import-relative-or-absolute) – Ani Menon Jul 03 '20 at 05:19
  • @AniMenon Nope, I've accepted Anoop's answer since nobody else has given any more, it seems like the answer to "How can I use relative importing in Python3 with an `if __name__='__main__' block?`" is just "You can't" – pyjamas Jul 03 '20 at 06:34
  • @Esostack Thats right, when you run a script you can't access something from its parent directory relatively. But if its part of a package and you are running it as a module then you can relatively import within the package. – Ani Menon Jul 03 '20 at 14:14
  • If you want to add file testing with a check for main and you also want to import the file to other modules, use the same check for main at the import level: if __name__ == '__main__': from mypackage import module1 else: from .mypackage import module1 – strawbot Dec 24 '20 at 09:31

4 Answers4

6

According to the Module documentation, for __main__ modules, you have to use absolute imports.

Note that relative imports are based on the name of the current module. Since the name of the main module is always "main", modules intended for use as the main module of a Python application must always use absolute imports.

So just change the import line in module1.py to:

from mypackage import module2

Everything else remains the same.

Anoop R Desai
  • 712
  • 5
  • 18
  • Is there a mistake with your code? `module2.py` as the first line of `__init__.py` gives an unresolved reference error. And removing that, I still get the same error as before in Python3. – pyjamas Feb 02 '19 at 06:18
  • 1
    Oh yes, I seem to be having the same issue now. I found a solution; will update my answer. Sorry about that. – Anoop R Desai Feb 02 '19 at 06:32
  • Thank you, but this still requires adding the parent folder of the package to PYTHONPATH or it fails. This means when I'm first writing the code I have to use relative imports, and then I have refactor all my import statements to be absolute once I decide to make it a module? It also means my code fails for anyone downloading my package as a zip unless they first add it it to their own PYTHONPATH. So while just not using relative imports does get the code working, it's not an answer to my OP about how to get relative imports working like they did in Python2. – pyjamas Feb 02 '19 at 07:00
  • Is adding the `__name__ == "__main__"` check only for testing? Ideally, as I see it, you would have just one `main` module, which would be the starting point for your code. For testing, you could use a different approach. That way you can use relative imports everywhere except in the `main` module. Also, if you are planning to distribute it as a zip, you don't want multiple entry points for the code (unless there is a specific reason to do so). – Anoop R Desai Feb 02 '19 at 10:12
  • Yes it's just for testing. It's a PyQt5 project with one module for the main GUI and other modules for various other widgets, dialogs and functions that are used in the main GUI. So for each module I have a `__name__ == "__main__"` block that creates test inputs and instantiates the widget alone so I don't have to open the main GUI and open it manually. What's the proper alternative? Should I just have a separate test file for each one like mydialog_test.py that uses relative imports and has the test code not in a __main__ block? – pyjamas Feb 02 '19 at 21:03
  • 1
    That's interesting. I haven't used PyQt5 but I had a similar requirement for one of my projects using Tkinter. We did have the exact same scenario where we used the main check for quick testing. I just dug that project up - looks like we had divided it into multiple packages and used absolute imports to import specific classes from different modules - something like `from package.module import class`. Now I'm unsure if there is a better way. – Anoop R Desai Feb 04 '19 at 09:51
2

A Python package isn't just a folder you stick your code in, and import behavior depends on more than just what folder you've stuck your code in.

When you run your file directly, you're not running it as part of a package. Package-level initialization doesn't run, and Python doesn't even recognize the package's existence. On Python 2, the existence of implicit relative imports meant that a bare import module2 would resolve to either an absolute import or an implicit relative import, hiding the problem, but the import structure is still broken. On Python 3, implicit relative imports are gone (for good reason), so the problem is immediately visible.

Running a submodule of a package directly by filename just doesn't work very well. These days, I believe the standard is to either use -m, or use a top-level entry point script that invokes the submodule's functionality.

There's sort of a way to get run-by-filename working anyway, but it's a lot of boilerplate. The designers of PEP 366 seem to have intended for a __package__ = 'appropriate.value' assignment to make relative imports work properly, but that's not actually enough, even if you fix the import path. You also have to initialize the parent package manually, or you'll get a "SystemError: Parent module 'foo' not loaded, cannot perform relative import" as soon as you try to run a relative import. The full boilerplate looks more like

import os.path
import sys
if __name__ == '__main__' and __package__ is None:
    __package__ = 'mypackage'
    right_import_root = os.path.abspath(__file__)
    for i in range(__package__.count('.') + 2):
        right_import_root = os.path.dirname(right_import_root)

    # sys.path[0] is usually the right sys.path entry to replace, but this
    # may need further refinement in the presence of anything else that messes
    # with sys.path
    sys.path[0] = right_import_root
    __import__(__package__)

This goes after stuff like future imports, but before any import that depends on your package.

I would wrap this boilerplate in a reusable function (using stack manipulation to access the caller's globals), except that if you try to put that function somewhere in your project, you won't be able to import the function until you've fixed your import situation, which you need the function to do. It might work as an installable dependency.

user2357112
  • 260,549
  • 28
  • 431
  • 505
2

I ended up in similar scenario and it troubled me a alot until I realised how module and package import is supposed to work.

Consider the following structure

mydir
- project
  - __init__.py
  - module1.py
  - module2.py

Contents of module1 and module2 looks like below

module1.py

print("moudule1")

module2.py

from . import module1

print("Module 2")

if __name__ == '__main__':
    print("Executed as script")

Now if i open a repl outside the package directory and try to make imports it works

Python 3.6.7 (default, Oct 22 2018, 11:32:17) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from package import module2
Module 1
Module 2
>>> sys.path
['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/home/rbhanot/.local/lib/python3.6/site-packages', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages']

Take a note at sys.path, as you can see it contains the current directory I am in as the first item, which means all my imports are going to be first searched in my current directory.

Now if i go into the package directory and then open a repl, and try making same imports see what happens

Python 3.6.7 (default, Oct 22 2018, 11:32:17) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from . import module2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'module2'
>>> import module2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/rbhanot/python-dotfiles/python3/modules-packages/mydir/package/module2.py", line 1, in <module>
    from . import module1
ImportError: attempted relative import with no known parent package
>>> import module1
Module 1
>>>

As you can see, the imports fail, the reason for failure is that when i try to import module from package python searches in sys.path to find any package with name package, since I could not find any , hence import fails. But importing the module1 works because it is found in the current directory.

Outside the package i can execute the script as 

python3 -m package.module2                                                                              2 ↵
Module 1
Module 2
Executed as script

Though I can execute the script but this is not how its supposed to be used. Remember packages are library of code that needs to shared across and should not have any code that is directly executable via command line. Packages and modules inside packages are meant to be just imported and then after importing you can write your scripts which execute via command line by putting __name__ clause in it.

Community
  • 1
  • 1
Rohit
  • 3,659
  • 3
  • 35
  • 57
1

My solution is to install my package with --ediable flag,

pip install -e .

and use absolute imports in your modules

for example:

import package.module2

# module1 contents

if __name__=="__main__":
    # Some test cases for the contents of this module
    pass

Use relative imports only in your init files.

This way you can work on your package, execute modules inside it to run tests or examples in the __main__ part.

zalavari
  • 819
  • 9
  • 21