Multiprocessing errors

Question

I have some code that is basically a big for loop, running a lot of things on several groups of data and outputting them individually. As they don't rely on each other, I thought I could use multiprocessing to run some of the loops simultaneously to save time. I've looked into the multiprocessing module in python, and tried running a few examples to try and understand how it works, however every example I run gives the same error, and doesn't complete running. For example, the below from the multiprocessing documentation:

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

Returns the below error and doesn't output (I've removed my file paths for privacy reasons):

Process SpawnPoolWorker-1:
Traceback (most recent call last):
  File "*filepath*", line 315, in _bootstrap
    self.run()
  File "*filepath*", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "*filepath*", line 114, in worker
    task = get()
  File "*filepath*", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>
Process SpawnPoolWorker-2:
Traceback (most recent call last):
  File "*filepath*", line 315, in _bootstrap
    self.run()
  File "*filepath*", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "*filepath*", line 114, in worker
    task = get()
  File "*filepath*", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>
Process SpawnPoolWorker-3:
Traceback (most recent call last):
  File "*filepath*", line 315, in _bootstrap
    self.run()
  File "*filepath*", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "*filepath*", line 114, in worker
    task = get()
  File "*filepath*", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>

I've also tried running many different examples from places like python guides and Geeks for Geeks and get the same/similar errors on every example I've found.

I know others have had issues with this, and have found the following SO questions, but none of the answers have got me any closer to understanding what's going on and how to use this module:

I'm at a bit of a loss with this, as I can't start trying to figure out if/how this module could help in my example, if I can't get the examples online to work. I'm writing this in spyder.

@swaggg tried an example from their documentation (the primes one) and got the same error :( — Emi OB, Sep 17 '21 at 08:32
Is your code working correctly if you don't use multiprocessing? I mean, that you can run your function without getting exceptions on different data you try to map (not only first one ;) ) — Grysik, Sep 17 '21 at 09:20
It may be relevant to specify the operating system and the Python version. If you have any non-ASCII characters in the path leading to this file, it may be worth a test to run it from an ASCII-only location. Btw, if the path for the file is simply very long, that may also cause problems. — tevemadar, Sep 17 '21 at 09:43
@Grysik The function works on its own. I even tried saving it separately and importing it, and just calling the function works (i.e. f(5) returns 25). Running just the map bit doesn't raise an error, just returns `` — Emi OB, Sep 17 '21 at 09:44
The code example is fine. It must be something related to your environment. :thinking_face: — pbacterio, Sep 17 '21 at 09:52
@tevemadar C:\Users\firstname.lastname\Anaconda3\lib\multiprocessing\process.py is the file path that comes back in each of the *filepath*s, with a different end file each time. Neither my first or lastname in the file path contain any unusual charecters. — Emi OB, Sep 17 '21 at 09:56
ok, I've wrongly understood, I thought that you masked your specific function as `f` as minimal example. If possible try different environment. In my case - 3.8.6 win10 1909.18363.1801: works, 3.9.0 and 3.9.6 win10 21H1.19043.1165: works, 3.7.2 win10 21H1.19043.1165: hangs forever(!) — Grysik, Sep 17 '21 at 10:08
Is there a chance that your script is also called `process.py`, so it may conflict with `multiprocessing\process.py`? — tevemadar, Sep 17 '21 at 11:49
It sounds like you are running this in jupyter notebook or its ilk. In that case you must put function `f` in a file such as `workers.py` and then do `from workers import f`. See [Jupyter notebook never finishes processing using multiprocessing (Python 3)](https://stackoverflow.com/questions/47313732/jupyter-notebook-never-finishes-processing-using-multiprocessing-python-3). — Booboo, Sep 17 '21 at 14:47
@Booboo running in spyder. I've already tried this work around, and got the same error (Yet if I called f on its own, it worked). — Emi OB, Sep 17 '21 at 14:56
Try to run your code from a command line or Anaconda prompt, so `python test.py`. And if it miraculously runs well, add [spyder] tag to the question and also mention it in the description. — tevemadar, Sep 17 '21 at 15:17
This absolutely sounds to me like you're running in interactive mode, and the main file isn't properly importable. All the IDE's you mentioned use IPython interactive mode by default, which can cause this problem (child can't properly import main). In spyder, you must make sure to save the file before executing, and must execute the whole file (not just a cell). For the best compatibility, you can set the "run" settings to "execute in an external console". Jupyter imao is not well suited to developing multiprocessing code. — Aaron, Sep 18 '21 at 05:47
@Aaron running the whole file rather than just running selected lines works :) Thank you — Emi OB, Sep 20 '21 at 06:38

Multiprocessing errors

0 Answers0

Linked