Python multiprocessing builtins.IOError: [Errno 22] Invalid argument

Question

Not sure why this error is appearing. I've searched around to no avail. I decided to take a punt at making my script run multi-threaded using the multiprocessing module, and if I remove that code the script runs fine.

So I ran the debugger and it also doesn't encounter any error with the multi-threading code, which seems a bit weird. But when I try to run the script just normally, it prints under 3.2.3:

Python 3.2.3 (default, Apr 11 2012, 07:12:16) [MSC v.1500 64 bit (AMD64)]
Type "help", "copyright", "credits" or "license" for more information.
[evaluate scratch.py]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python32\Lib\multiprocessing\forking.py", line 369, in main
    self = load(from_parent)
AttributeError: 'module' object has no attribute 'search_letters_in_words'
Traceback (most recent call last):
  File "C:\Program Files (x86)\Wing IDE 4.1\src\debug\tserver\_sandbox.py", line 122, in <module>
  File "C:\Python32\Lib\multiprocessing\process.py", line 132, in start
    self._popen = Popen(self)
  File "C:\Python32\Lib\multiprocessing\forking.py", line 269, in __init__
    to_child.close()
builtins.IOError: [Errno 22] Invalid argument

Edit: I switched over to 3.3 to see what happens and it unconsistently consistently throws one of these two tracebacks:

Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)]
Type "help", "copyright", "credits" or "license" for more information.
[evaluate scratch.py]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python33\Lib\multiprocessing\forking.py", line 344, in main
    self = load(from_parent)
AttributeError: 'module' object has no attribute 'search_letters_in_words'
Traceback (most recent call last):
  File "C:\Program Files (x86)\Wing IDE 4.1\src\debug\tserver\_sandbox.py", line 122, in <module>
  File "C:\Python33\Lib\multiprocessing\process.py", line 111, in start
    self._popen = Popen(self)
  File "C:\Python33\Lib\multiprocessing\forking.py", line 243, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python33\Lib\multiprocessing\forking.py", line 160, in dump
    ForkingPickler(file, protocol).dump(obj)
builtins.BrokenPipeError: [Errno 32] Broken pipe

Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)]
Type "help", "copyright", "credits" or "license" for more information.
[evaluate scratch.py]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python33\Lib\multiprocessing\forking.py", line 344, in main
    self = load(from_parent)
AttributeError: 'module' object has no attribute 'search_letters_in_words'

Edit #2 Added traceback when calling from the commandline:

D:\Python\PythonRepo>scratch.py > d:\download\error.txt
Traceback (most recent call last):
  File "D:\Python\PythonRepo\scratch.py", line 122, in <module>
    thread.start()
  File "C:\Python32\Lib\multiprocessing\process.py", line 131, in start
    from .forking import Popen
  File "C:\Python32\Lib\multiprocessing\forking.py", line 180, in <module>
    import _subprocess
ImportError: No module named '_subprocess'

Here's the multiprocessing code I've written so far. It could be (hah who am I kidding, probably is!)buggy but I'm not sure what's wrong as it looks correct (doesn't it always?).

wordList = pickle.load( open( r'd:\download\allwords.pickle', 'rb')) #a list
combos = make_letter_combinations(3) # a list
split = split_list_multi(combos) #2 item tuple with a dict and a number
if __name__ == '__main__':
    multiprocessing.freeze_support()
    jobs = []
    for num in range(split[1]):
        listLetters = split[0][str(num)] #a list
        thread = multiprocessing.Process(target=search_letters_in_words, args=(listLetters,wordList))
        jobs.append(thread)
        thread.start()
    for j in jobs:
        j.join()

Edit: Here's the search_letters_in_words_ function:

def search_letters_in_words(listOfLetters,wordlist):
    results = {}
    for letters in listOfLetters:
        results[letters] = [i for i in wordlist if letters in i]
    return results

If anyone could point out what I'm doing wrong I'd appreciate it!

unutbu · Accepted Answer · 2013-02-11T22:07:33.940

3

Try running the script from the command-line, not through Wing IDE:

python scripy.py

edited Feb 11 '13 at 22:07

answered Feb 11 '13 at 20:15

unutbu

842,883
184
1,785
1,677

Hi unutbu, that is actually a function, which works fine except when I try to multi-thread the script which is what is confusing me. – Jason White Feb 11 '13 at 20:44
Try running the script from the terminal, not from inside WingIDE. I'm not familiar with Windows (or WingIDE), but I know on Windows the multiprocessing module re-imports the script to emulate Unix `os.fork`. I'm not sure if that works well from inside WingIDE. Though maybe it works just fine. I just don't know. – unutbu Feb 11 '13 at 20:52
By the way, it looks like you are using multiprocessing to run functions in subprocesses. That is different than making your script multithreaded. [Threads are different than processes](http://stackoverflow.com/q/200469/190597). – unutbu Feb 11 '13 at 20:55
I googled around and it seemed that for python to utilize more than one core for parallel processing that I should use multiprocessing as the threading one doesn't guarantee that python will use all cores. Or something to that effect. I probably shouldn't be using multiprocess and multithreaded interchangebly. I ran the script from the command line which I've added to the post. – Jason White Feb 11 '13 at 21:01
1

Your problem (`AttributeError: 'module' object has no attribute ...`) looks a lot like [this SO question](http://stackoverflow.com/q/850424/190597). – unutbu Feb 11 '13 at 21:01
Ahha that link solved it! When you told me try from the commandline, I forgot to type python script.py heh. Forgot all about that command as I normally just run them from the ide. Thanks! – Jason White Feb 11 '13 at 21:09

score 1 · Answer 2 · answered Dec 19 '13 at 12:54

I was getting the error code for a different reason (with Py 3 and Win 7). Reading a filename from a .ini file with quotes broke a file open for a csv.DictWriter function. Removing the ""s fixed it:

...
[CSVFILE]
#CfgCsvNamePattern="C:\\temp\\PenSim-%%s.csv"
CfgCsvNamePattern=C:/temp/PenSim%%s.csv
...

Python multiprocessing builtins.IOError: [Errno 22] Invalid argument

2 Answers2