-1

I am trying o solve a task of multiprocessing in Python.

I can achieve his task with he help of multiprocess module but can't apply the same with multiprocessing module. I have to apply with multiprocessing module cause I need to use freeze_support().

First I thought that the issue is with my code and post this, which haven't received any answers yet. Then I thought to run a toy example of applying multiprocessing module to see whether I can figure out why multiprocessing doesn't run on my system. So I ran this code which I got from this tutorial.

import multiprocessing
import time


def cpu_bound(number):
    return sum(i * i for i in range(number))


def find_sums(numbers):
    with multiprocessing.Pool() as pool:
        pool.map(cpu_bound, numbers)

if __name__ == "__main__":
    numbers = [5_000_000 + x for x in range(20)]

start_time = time.time()
find_sums(numbers)
duration = time.time() - start_time
print(f"Duration {duration} seconds")

It seems that multiprocessing module doesn't run on my machine, cause this toy example won't run on my system either.

When I apply multiprocess module on this toy example, the results showed up in no time with utilizing all of my cores.

So I am wondering if the issue is with a specific configuration on my system or is there any equivalent of freeze_support() in multiprocess module?

By saying "this module won't run on my system", I mean that processes have spawned but the cpu won't go up and it seems that nothing is going on... after almost 10 minutes I manually terminate the run. But with multiprocess module the code completed in under 1 minute.

Thanks in advance.

PS: My machine has 4 cores.

Community
  • 1
  • 1
mpy
  • 622
  • 9
  • 23
  • 1
    try putting everything below `numbers = ...` into the `if` statement – warped May 03 '20 at 11:26
  • define "this toy example won't run on my system"? – njzk2 May 03 '20 at 11:26
  • How do you mean, "doesn't run on my machine"? Can you paste the error trace? – Lukasz Tracewski May 03 '20 at 11:26
  • @njzk2 I mean that processes have spawned but the cpu won't go up and it seems that nothing is going on... after almost 10 minutes I manually terminate the run. But with multiprocess module the code completed in under 1 minute. – mpy May 03 '20 at 11:37
  • @LukaszTracewski I edited my post to clarify it. – mpy May 03 '20 at 11:38
  • what does `os.cpu_count()` returns on your machine? – njzk2 May 03 '20 at 11:40
  • @njzk2 my machine has 4 cores. – mpy May 03 '20 at 11:41
  • 1
    You need to put all the bottom code block in the `if __name__` block... – Andras Deak -- Слава Україні May 03 '20 at 11:42
  • @mpy but is that actually what `os.cpu_count()` returns? – njzk2 May 03 '20 at 11:43
  • @njzk2 yes it is. – mpy May 03 '20 at 11:46
  • @AndrasDeak The issue has nothing to do with the "if __name__" block... I could not run the code from within the Spyder... I ran it perfectly from windows command prompt without any change in my code and closing this post was completely wrong. – mpy May 10 '20 at 11:59
  • 1
    Your code is broken on windows so it's a reasonable assumption that the problem was due to your inadequate use of copy-paste from the tutorial you linked. Closing your question was completely right. That being said since you now know the issue was something else (which you haven't mentioned for a week even though I left an explicit comment) I'll reopen it, simple as that. You should either delete the question or leave a self-answer explaining the issue. – Andras Deak -- Слава Україні May 10 '20 at 12:09
  • Just to be clear: did you actually fix the `if __name__` block and try again on spyder? – Andras Deak -- Слава Україні May 10 '20 at 12:12
  • 1
    Your code runs in 2.5 sec on my machine with spyder 4.1, python 3.8, ubuntu 16.04. – CKM May 10 '20 at 12:17
  • @CKM it's bound to work on ubuntu. The question used to be duped to https://stackoverflow.com/questions/20222534/python-multiprocessing-on-windows-if-name-main – Andras Deak -- Слава Україні May 10 '20 at 12:33
  • @AndrasDeak I really found the solution after one week of struggling. I was about to answer my own question but it was closed... and I have also tried move everything under the "if __name__" block, but that also didn't work. I would post the behavior of the Spyder on my system if I could video screen....Anyway thanks for reopening... – mpy May 11 '20 at 13:52
  • 1
    No worries, closing is not an end, it's a means to making questions answerable. I could've reopened the moment you verified that the `__name__` block wasn't it. Unfortunately spyder has quite a few surprising quirks, none for the better. – Andras Deak -- Слава Україні May 11 '20 at 14:53

1 Answers1

0

OK after almost one week of struggling, I found out that the issue was neither my system nor the "if __name__" block. The problem was with my IDE Spyder.

I ran the code from the windows command prompt and both the toy example and my own real code ran perfectly. I really don't understand why, however I managed to eventually run my script.

I would be happy if others had this similar issue and share their experience...

mpy
  • 622
  • 9
  • 23