119

Problem

PEP8 has a rule about putting imports at the top of a file:

Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants.

However, in certain cases, I might want to do something like:

import sys
sys.path.insert("..", 0)

import my_module

In this case, the pep8 command line utility flags my code:

E402 module level import not at top of file

What is the best way to achieve PEP8 compliance with sys.path modifications?

Why

I have this code because I'm following the project structure given in The Hitchhiker's Guide to Python.

That guide suggests that I have a my_module folder, separate from a tests folder, both of which are in the same directory. If I want to access my_module from tests, I think I need to add .. to the sys.path

Luke Taylor
  • 8,631
  • 8
  • 54
  • 92
  • 2
    Why don't you write a `setup.py` and actually *install* `my_module` for testing? – jonrsharpe Apr 24 '16 at 19:40
  • 1
    Because that's slightly less convenient. I suppose I *could* but I'd rather not. – Luke Taylor Apr 24 '16 at 19:42
  • 2
    For whom? If you ever want to actually *use* this project anywhere it's far and away the easiest way to get it up and running. – jonrsharpe Apr 24 '16 at 19:42
  • @jonrsharpe for me. This is more of a personal project for my use. I'll definitely use that if no other answers come up. – Luke Taylor Apr 24 '16 at 19:44
  • If you're not sharing it, why worry about following PEP-8 so closely? – jonrsharpe Apr 24 '16 at 19:45
  • 16
    From [PEP8](https://www.python.org/dev/peps/pep-0008/): "However, know when to be inconsistent -- sometimes style guide recommendations just aren't applicable. When in doubt, use your best judgment.". There are times when you have to break PEP8 compliance, and that is OK. – SethMMorton Apr 24 '16 at 19:47
  • 2
    @jonrsharpe It's a good habit to get into for future things that I will share. (I do see your point though, in that case I can use setup.py). I'll keep that in mind. – Luke Taylor Apr 24 '16 at 19:47

9 Answers9

128

If there are just a few imports, you can just ignore PEP8 on those import lines:

import sys
sys.path.insert("..", 0)
import my_module  # noqa: E402
astorga
  • 1,451
  • 1
  • 9
  • 8
  • 14
    I prefer to be more explicit, specifying the violated rule like `# noqa: E402` for example. ([source](http://flake8.pycqa.org/en/3.1.1/user/ignoring-errors.html#in-line-ignoring-errors)) – Max Goodridge Aug 30 '18 at 13:53
  • @MaxGoodridge indeed! Edited response to add the rule. – astorga Oct 16 '18 at 12:54
  • 6
    in case you have more than one import from the same path, i'ts shorter to put `# noqa: E402` on the `sys.path.insert...` line – Ohad Cohen Nov 30 '21 at 17:04
74

Often I have multiple files with tests in a subdirectory foo/tests of my project, while the modules I'm testing are in foo/src. To run the tests from foo/tests without import errors I create a file foo/tests/pathmagic.py that looks like this;

"""Path hack to make tests work."""

import os
import sys

bp = os.path.dirname(os.path.realpath('.')).split(os.sep)
modpath = os.sep.join(bp + ['src'])
sys.path.insert(0, modpath)

In every test file, I then use

import pathmagic  # noqa

as the first import. The "noqa" comment prevents pycodestyle/pep8 from complaining about an unused import.

Roland Smith
  • 42,427
  • 3
  • 64
  • 94
  • 2
    This is cool, but this still have 'imported but unused [F401]' problem. – Chung-Yen Hung Jul 19 '17 at 02:01
  • 2
    I guess creating a dummy function in that pathmagic module and calling it from the test module would solve that issue but I wish there was something cleaner... – Alexis.Rolland Oct 14 '17 at 03:49
  • 3
    @Chung-YenHung Keep in mind that pycodestyle/pep8 warnings are advisory rather than syntax errors or exceptions. You can choose to ignore them. I've updated my answer by adding a "noqa" comment after the import. – Roland Smith Oct 14 '17 at 07:04
  • One huge disadvantage of this, is turning your tests into a package so ``pathmagic`` can be importable. Most python test runners assume your tests are a collections of files not on ``sys.path``, and changing that can leads to issues, see how ``pytest`` deals with these issues https://docs.pytest.org/en/latest/goodpractices.html#choosing-a-test-layout-import-rules – Meitham Jul 06 '20 at 08:44
  • 1
    @Meitham Although my answer mentions how I used this to run tests back then, the question is not about running tests. (I have since moved to pytest for my tests.) In other situation this mechanism is still useful. – Roland Smith Jul 06 '20 at 17:33
  • @RolandSmith thanks for clarifying. I've landed in a project where their test is based on your answer, and moving the project to pytest. – Meitham Jul 06 '20 at 17:37
  • ```sys.path.insert(0,os.path.abspath(os.path.join(os.path.dirname(__file__), 'src')))``` – null Apr 27 '21 at 23:02
13

There is another workaround.

import sys
... all your other imports...

sys.path.insert("..", 0)
try:
    import my_module
except:
    raise
Peuchele
  • 326
  • 2
  • 5
  • 1
    I strongly believe this will not work. The reason being, there may be some modules imported under "... all your imports ..." which might require PYTHONPATH to be set first. – darkdefender27 Nov 01 '17 at 17:50
  • 4
    @darkdefender27 the idea is to put all imports that requires PYTHONPATH inside `try` body and everything else (that does not depend on it) up above. – astorga Oct 16 '18 at 12:52
  • or even simpler: `if 1: import module` – stason Aug 29 '20 at 05:03
8

This problem already has several solutions that work, but in the case that you have a number of non-initial imports, and don't want to annotate each with # noqa: E402 (or use one of the other proposals), the following works for a whole block:

import sys
sys.path.insert("..", 0)

if True:  # noqa: E402
    import my_module_1
    import my_module_2
    ...
shaunc
  • 5,317
  • 4
  • 43
  • 58
5

I've just struggled with a similar question, and I think I found a slightly nicer solution than the accepted answer.

Create a pathmagic module that does the actual sys.path manipulation, but make the change within a context manager:

"""Path hack to make tests work."""

import os
import sys

class context:
    def __enter__(self):
        bp = os.path.dirname(os.path.realpath('.')).split(os.sep)
        modpath = os.sep.join(bp + ['src'])
        sys.path.insert(0, modpath)

    def __exit__(self, *args):
        pass

Then, in your test files (or wherever you need this), you do:

import pathmagic

with pathmagic.context():
    import my_module
    # ...

This way you don't get any complaints from flake8/pycodestyle, you don't need special comments, and the structure seems to make sense.

For extra neatness, consider actually reverting the path in the __exit__ block, though this may cause problems with lazy imports (if you put the module code outside of the context), so maybe not worth the trouble.

EDIT: Just saw a much simpler trick in an answer to a different question: add assert pathmagic under your imports to avoid the noqa comment.

itsadok
  • 28,822
  • 30
  • 126
  • 171
  • All this really accomplished is getting rid of the next for the special comment at the expense of requiring the use of a context manager—a somewhat nebulous tradeoff in my opinion. As for cleaning-up in the `__exit__` block, to really do it properly would require just removing the added path (if it's still there) because restoring the entire previous value to what it was when the context was entered would also undo any other changes that might have been made to it by other code (executed in the same context) for some reason. – martineau Nov 30 '17 at 16:58
  • @martineau indeed, it's a matter of taste. I may be a little biased against special comments, because my current code base contains a little too many comments for the myriad static analysis tools and editors that the various teams are using. Also agreed on your second point. – itsadok Nov 30 '17 at 17:12
  • Problematic if `pathmagic.context()` can be called an arbitrary number of times (e.g. if running all tests). Reverting is problematic, too: any import at a later time (e.g. done on demand) can fail. – ivan_pozdeev Sep 21 '19 at 16:55
4

Did you already try the following:

import sys
from importlib import import_module

sys.path.insert("..", 0)

# import module
my_mod = import_module('my_module')

# get method or function from my_mod
my_method = getattr(my_mod , 'my_method')
Daniel Lee
  • 300
  • 4
  • 12
3

To comply with the pep8, you should include your project path to the python path in order to perform relative / absolute imports.

To do so, you can have a look at this answer: Permanently add a directory to PYTHONPATH

Community
  • 1
  • 1
Pierre Barre
  • 2,174
  • 1
  • 11
  • 23
  • 3
    I don't want to make this directory accessible globally to all my Python scripts, because that could cause conflicts. – Luke Taylor Apr 24 '16 at 19:41
  • 2
    You can use the https://docs.python.org/3/library/pkgutil.html package to use namespaces. If you think your solution is the best, you are not in obligation to follow the pep8. Pep8 is only advices and best practices, that don't mean you must follow every rule, everytime, everywhere. – Pierre Barre Apr 24 '16 at 19:42
  • 2
    PEP 8 actually says [A foolish consistency is the hobgoblin of little minds.](https://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds) – zondo Apr 24 '16 at 19:47
  • if only I didn't need to fight flake8 in git hooks and CI – rob Mar 28 '22 at 14:38
3

Given that the path to be added is relative to the script, this can be solved by using relative path.

from ..mypath import mymodule 
from ..mypath.mysubfolder import anothermodule

Then do not need to use sys.path.insert() anymore. Lint stoped complaining.

Bin
  • 3,645
  • 10
  • 33
  • 57
0

THE EASIEST SOLUTION: In the case of pytest would be to add the path in conftest.py. This ensures that package path is added before running any tests.

Ashish Yadav
  • 1