Python multiprocessing throws error with argparse and pyinstaller

Question

In my project, I'm using argprse to pass arguments and somewhere in script I'm using multiprocessing to do rest of the calculations. Script is working fine if I call it from command prompt for ex.

"python complete_script.py --arg1=xy --arg2=yz" .

But after converting it to exe using Pyinstaller using command "pyinstaller --onefile complete_script.py" it throws

error

" error: unrecognized arguments: --multiprocessing-fork 1448"

Any suggestions how could I make this work. Or any other alternative. My goal is to create an exe application which I can call in other system where Python is not installed.

Here are the details of my workstation:

Platform: Windows 10
Python : 2.7.13 <installed using Anaconda>
multiprocessing : 0.70a1
argparse:   1.1

Copied from comment:

def main():
     main_parser = argparse.ArgumentParser()
     < added up arguments here>
    all_inputs = main_parser.parse_args()
    wrap_function(all_inputs)


def wrap_function(all_inputs):
    <Some calculation here >
   distribute_function(<input array for multiprocessing>)

def distribute_function(<input array>):
    pool = Pool(process = cpu_count)
    jobs = [pool.apply_async(target_functions, args = (i,) for i in input_array)]
    pool.close()

I've had numerous issues using Anaconda distributions + pyinstaller. I'm not sure pyinstaller is tested against Anaconda. My advice, if you're going to freeze a python app, make your life easier and use canonical python, not third-party distributions — Overdrivr, Dec 13 '18 at 12:28
this was a really good solution for me : https://stackoverflow.com/questions/12818146/python-argparse-ignore-unrecognised-arguments use args, unknown = parser.parse_known_args() and it will quietly parse even with unknown args, and now multiprocessing works in pyinstaller just fine — gene tsai, Apr 06 '23 at 22:40

score 23 · Answer 1 · edited Jun 20 '20 at 09:12

23

(A bit late but it can be useful for someone else in the future...)

I had the same problem, after some research I found this multiprocessing pyInstaller recipe that states:

When using the multiprocessing module, you must call

multiprocessing.freeze_support()

straight after the if __name__ == '__main__': line of the main module.

Please read the Python library manual about multiprocessing.freeze_support for more information.

Adding that line of code solved the problem for me.

edited Jun 20 '20 at 09:12

Community

1
1

answered Feb 15 '18 at 10:36

sanzoghenzo

598
5
21

Awesome answer, you saved my day. – PythonLearner Jun 09 '20 at 14:13
Worth mentioning that `freeze_support()` is _only_ relevant on Windows. (This question is not tagged Windows, and while the OP mentions they _are_ using Windows, non-Windows users may find this question.) – mhucka Dec 10 '21 at 19:10

score 0 · Answer 2 · answered Sep 21 '17 at 05:31

I may be explaining the obvious, but you don't give us much information to work with.

python complete_script.py --arg1=xy --arg2=yz

This sort of call tells me that your parser is setup to accept at least these 2 arguments, ones flagged with '--arg1' and '--arg2'.

The error tells me that this parser (or maybe some other) is also seeing this string:

--multiprocessing-fork 1448

Possibly generated by the multiprocessing code. It would be good to see the usage part of the error, just to confirm which parser is complaining.

One of my first open source contributions to Python was to enhance the warnings about multiprocessing on Windows.

https://docs.python.org/2/library/multiprocessing.html#windows

Is your parser protected by a if __name__ block? Should this particular parser be called when run in a fork? You probably designed the parser to work when the program is called as a standalone script. But when happens when it is imported?

This might useful information to work for: ['code'] def main(): main_parser = argparse.ArgumentParser() < added up arguments here> all_inputs = main_parser.parse_args() wrap_function(all_inputs) def wrap_function(all_inputs): distribute_function() def distribute_function(): pool = Pool(process = cpu_count) jobs = [pool.apply_async(target_functions, args = (i,) for i in input_array)] pool.close() ['code'] — Prabhakar Mishra, Sep 25 '17 at 05:20
When is `main` called? On import or only by a `is __name__...` block? — hpaulj, Sep 27 '17 at 04:47

Python multiprocessing throws error with argparse and pyinstaller

2 Answers2

Linked