2

Long time (very grateful) user of SO, but first time hitting something I can't seem to find a pre-existing answer for. So here goes my first plea for wisdom to the hive mind.

I hit this problem while implementing some Python multiprocessing routines in a code of mine. I tracked it all down to the following MWE (which actually has nothing to do with multiprocessing anymore):

import sys

if __name__ == '__main__':
    print(dir(sys.modules['__main__']))

If I put this in tmp.py, and run it from an ipython terminal (i.e. run tmp) a first time, it works fine and prints:

['__builtins__', '__doc__', '__file__', '__loader__', '__name__', '__nonzero__', '__package__', '__spec__', 'sys']

If I run it again in the same terminal, in prints:

['__builtins__', '__file__', '__name__', '__nonzero__', 'sys']

The fact that __spec__ is not available the second time around triggers the following error when multiprocessing.Pool tries to access it:

----> 6     print(sys.modules['__main__'].__spec__ is None)
AttributeError: module '__main__' has no attribute '__spec__'

There are a lot of questions about this error message out there, but none seem to apply to my case as far as I can tell. I am not using any IDE or anything fancy - just a regular OSX terminal. Most people seem to encounter this bug when using the multiprocessing package (like me), which is often blamed for it (e.g. here). But the above MWE makes it clear that something (i.e. the content of __spec__) is not constant between successive runs of a given script in an ipython terminal.

Questions: is this normal Python behavior ? Am I missing anything obvious ?

Edit (to clarify and anticipate possible replies): The suggestion to hard-code __spec__ = None to avoid the error, as suggested e.g. in this SO question, does work. But I am wary to do this as I do not understand the implications.

What is this __spec__ element ? Why is it not stable when I run the MWE above twice in a row ? What are the implications of setting it by hand ? Is there anyway that this can blow-up on a different OS ?

Running Python 3.8.5, ipython 7.20.0 on Mac OS 10.15.

fpavogt
  • 411
  • 5
  • 11
  • for checking any attribute of a module or object, try dir of it like `dir(sys.modules['__main__'])` it will give you allowed attribute/operation that you can apply – tard Mar 01 '21 at 15:34
  • @tard - thanks for the suggestion. I updated the MWE accordingly. – fpavogt Mar 01 '21 at 15:53
  • 1
    I have not found an answer, but I am having a hunch it has something to do with how ipython handles (some?) `None` values. The 3 values, which are culled after the second run are `__loader__`, `__spec__` and `__package__`. All these have `None` set as their values beforehand – Lala5th Oct 28 '21 at 13:49

0 Answers0