Python 3 modules and package relative import doesn't work?

Question

I have some difficulties constructing my project structure.

This is my project directory structure :

MusicDownloader/
   __init__.py
   main.py
   util.py
   chart/
      __init__.py
      chart_crawler.py
   test/
      __init__.py
      test_chart_crawler.py

These are codes :

1.main.py

from chart.chart_crawler import MelonChartCrawler

crawler = MelonChartCrawler()

2.test_chart_crawler.py

from ..chart.chart_crawler import MelonChartCrawler

def test_melon_chart_crawler():
  crawler = MelonChartCrawler()

3.chart_crawler.py

import sys
sys.path.append("/Users/Chois/Desktop/Programming/Project/WebScrape/MusicDownloader")
from .. import util

class MelonChartCrawler:
  def __init__(self):
    pass

4.util.py

def hi():
   print("hi")

In MusicDownloader, when I execute main.py by python main.py, it shows errors:

  File "main.py", line 1, in <module>
    from chart.chart_crawler import MelonChartCrawler
  File "/Users/Chois/Desktop/Programming/Project/WebScrape/MusicDownloader/chart/chart_crawler.py", line 4, in <module>
    from .. import util
ValueError: attempted relative import beyond top-level package

But when I execute my test code in test directory by py.test test_chart_crawler.py, it works

When I first faced with absolute, relative imports, it seems like very easy and intuitive. But it drives me crazy now. Need your helps. Thanks

Answer 1

The first problem is MusicDownloader not being a package. Add __init__.py to MusicDownloader along with main.py and your relative import ..chart should work. Relative imports work only inside packages, so you can't .. to non-package folder.

Editing my post to provide you with more accurate answer to your answer edit.

It's all about the __name__. Relative imports use __name__ of the module they are used in and the from .(.) part to form a full package/module name to import. Explaining in simple terms importer's __name__ is concatenated with from part, with dots showing how many components of name to ignore/remove, i.e.:

__name__='packageA.packageB.moduleA' of the file containing line: from .moduleB import something, leads to combined value for import packageA.packageB.moduleB, so roughly from packageA.packageB.moduleB import something(but not absolute import as it would be if typed like that directly).

__name__='packageA.packageB.moduleA' of the file containing line: from ..moduleC import something, leads to combined value for import packageA.moduleC, so roughly from packageA.moduleC import something(but not absolute import as it would be if typed like that directly).

Here if it's a moduleB(C) or a packageB(C) doesn't really matter. What's important is that we still have that packageA part which works as an 'anchor' for relative import in both cases. If there will be no packageA part, relative import won't be resolved, and we'll get an error like "Attempted relative import beyond toplevel package".

One more note here, when a module is run it gets a special __name__ value of __main__, which obviously prevents it from solving any relative imports.

Now regarding your case try adding print(__name__) as the very first line to every file and run your files in different scenarios and see how the output changes.

Namely if you run your main.py directly, you'll get:

__main__
chart.chart_crawler
Traceback (most recent call last):
  File "D:\MusicDownloader\main.py", line 2, in <module>
    from chart.chart_crawler import MelonChartCrawler
  File "D:\MusicDownloader\chart\chart_crawler.py", line 2, in <module>
    from .. import util
ValueError: Attempted relative import beyond toplevel package

What happened here is... main.py has no idea about MusicDownloader being a package (even after previous edit with adding __init__.py). In your chart_crawler.py: __name__='chart.chart_crawler' and when running relative import with from .. the combined value for package will need to remove two parts (one for every dot) as explained above, so the result will become '' as there're just two parts and no enclosing package. This leads to exception.

When you import a module the code inside it is run, so it's almost the same as executing it, but without the __name__ becoming __main__ and the enclosing package, if there's any, being 'noticed'.

So, the solution is to import main.py as part of the MusicDownloader package. To accomplish the described above, create a module, say named launcher.py on the same level of hierarchy as MusicDownloader folder (near it, not inside it near main.py) with the following code:

print(__name__)
from MusicDownloader import main

Now run launcher.py and see the changes. The output:

__main__
MusicDownloader.main
MusicDownloader.chart.chart_crawler
MusicDownloader.util

Here __main__ is the __name__ inside launcher.py. Inside chart_crawler.py: __name__='MusicDownloader.chart.chart_crawler' and when running relative import with from .. the combined value for package will need to remove two parts (one for every dot) as explained above, so the result will become 'MusicDownloader' with import becoming from MusicDownloader import util. And as we see on the next line when util.py is imported successfully it prints its __name__='MusicDownloader.util'.

So that's pretty much it - "it's all about that __name__".

P.S. One thing not mentioned is why the part with test package worked. It wasn't launched in common way, you used some additional module/program to lauch it and it probably imported it in some way, so it worked. To understand this it's best to see how that program works.

There's a note in official docs:

Note that relative imports are based on the name of the current module. Since the name of the main module is always "__main__", modules intended for use as the main module of a Python application must always use absolute imports.