python multiprocessing.pool() OSError: [Errno 22] Invalid argument

Question

I try to run this code:

import numpy as np

import multiprocessing
from functools import partial

CPUS = multiprocessing.cpu_count() # Anzahl der CPUs/Threads: 4
anzahl = CPUS-1

WORKERS = multiprocessing.Pool(processes=anzahl)

I tried to run it on 3 different computers: on 2 all works fine, but on 1 computer I get the following traceback error:

Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "L:\Python\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)

  File "L:\Python\lib\multiprocessing\spawn.py", line 125, in _main
    prepare(preparation_data)

  File "L:\Python\lib\multiprocessing\spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])

  File "L:\Python\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,

  File "L:\Python\lib\runpy.py", line 264, in run_path
    code, fname = _get_code_from_file(run_name, path_name)

  File "L:\Python\lib\runpy.py", line 234, in _get_code_from_file
    with io.open_code(decoded_path) as f:

OSError: [Errno 22] Invalid argument: 'L:\\Python\\<input>'

I don't know how to solve this problem. Can someone help me? My Python version: Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] on win32

*How* do you run this code? Is it from a script, the command line or an interactive session? ``multiprocessing`` must be able to import the ``__main__`` module, which is sometimes not possible if said module is not in a file. — MisterMiyagi, May 20 '20 at 19:43
I run the code from the Python Console in PyCharm (= I start the skript via the console. This works well with the other computers...). I also tried to run it from the PyCharm Terminal - also didn't work. — Larissa, Jun 02 '20 at 13:53

score 2 · Answer 1 · answered Jan 23 '21 at 15:59

2

A workaround to this (tested in PyCharm 2020.3.2) is to change the __file__ console variable to the name of an existing file before instantiating the Pool

answered Jan 23 '21 at 15:59

GydeonMylls

21
4

1

How do you change that variable? – Wouter Apr 01 '21 at 07:15
Just set the value of the variable like this: `__file__ = 'name_of_your_file.py'` – N. Maks Dec 13 '22 at 10:02

score 0 · Answer 2 · edited Apr 27 '21 at 12:14

Credit to those in the comments of how-to-create-multiprocessingpool-object

import collections
import multiprocessing
import time
from pprint import pprint

# this needs to be outside of if name == main or you get a pickle error
Scientist = collections.namedtuple('Scientist', ['name', 'field', 'born', 'nobel'])

# this might be ok inside if name == main but it might as well go here
scientists = (Scientist('Ada Lovelace', 'math', 1815, False), Scientist('Emmy Noether', 'math', 1882, False),
              Scientist('Marie Curie', 'math', 1867, True), Scientist('Tu Youyou', 'physics', 1930, True),
              Scientist('Ada Yonath', 'chemistry', 1939, True), Scientist('Vera Rubin', 'astronomy', 1928, False),
              Scientist('Sally Ride', 'physics', 1951, False))


def transform(x):
    print(f'Processing record {x.name}')
    time.sleep(1)
    result = {'name': x.name, 'age': 2021 - x.born}
    print(f'Done processing record {x.name}')
    return result


def run():
    start = time.time()

    pool = multiprocessing.Pool()
    result = pool.map(transform, scientists)

    # result = tuple(map(
    #     transform,
    #     scientists
    # ))

    end = time.time()

    print(f'\nTime to complete: {end - start:2f}s\n')
    pprint(result)


if __name__ == '__main__':
    run()
    # multiprocessing.freeze_support()
    # multiprocessing.set_start_method("spawn")

This thread talks about it as well: Can multiprocessing Process class be run from IDLE

score 0 · Answer 3 · answered Aug 13 '22 at 02:19

Running the Pool function inside of the if __name__ == '__main__': block solved the issue for me.

For example,

from multiprocessing import Pool

def mul2(x):
    return x*2

if __name__ == '__main__':
    p = Pool(4)

    results = p.map(mul2, range(4))
    p.close()
    p.join()

    print(results)

python multiprocessing.pool() OSError: [Errno 22] Invalid argument

3 Answers3