Python 3 - Multiprocessing - Queue.get() does not respond

Question

I want to make a brute force attack and therefore need some speed... So I came up to use the multiprocessing library... However, in every tutorial I've found, something did not work.... Hm.. This one seems to work very well, except, that I whenever I call the get() function, idle seems to go to sleep and it doesn't respond at all. Am I just silly or what? I just copy pasted the example, so it should have worked....

import multiprocessing as mp
import random
import string

# Define an output queue
output = mp.Queue()

# define a example function
def rand_string(length, output):
    """ Generates a random string of numbers, lower- and uppercase chars. """
    rand_str = ''.join(random.choice(
                    string.ascii_lowercase
                    + string.ascii_uppercase
                    + string.digits)
               for i in range(length))
    output.put(rand_str)




# Setup a list of processes that we want to run
processes = [mp.Process(target=rand_string, args=(5, output)) for x in range(2)]

# Run processes
for p in processes:
    p.start()

# Exit the completed processes
for p in processes:
    p.join()

# Get process results from the output queue
results = [output.get() for p in processes]

print(results)

I have had issues trying to use multiprocessing from IDLE, suggest you try it from the command-line. — cdarke, Apr 27 '15 at 19:12
What platform are you running this on? It works fine for me on Linux. For Windows, you'd add `if __name__ == "__main__":` above the `processes = ..` line, and then indent everything to be under that if block. If it still doesn't work, you're probably seeing an issue with IDLE, as @cdarke mentioned. — dano, Apr 27 '15 at 20:16

score 1 · Answer 1 · answered Apr 28 '15 at 06:53

@dano hit it on the head! You don't have if __name__ == "__main__": so you have a "fork bomb". That is, each process is running the processes, and so on. You will also notice that I have moved the creation of the queue.

import multiprocessing as mp
import random
import string


# define a example function
def rand_string(length, output):
    """ Generates a random string of numbers, lower- and uppercase chars. """
    rand_str = ''.join(random.choice(
                string.ascii_lowercase
                + string.ascii_uppercase
                + string.digits)
    for i in range(length))
        output.put(rand_str)


 if __name__ == "__main__":
     # Define an output queue
     output = mp.Queue()

     # Setup a list of processes that we want to run
     processes = [mp.Process(target=rand_string, args=(5, output)) for x in    range(2)]

     # Run processes
    for p in processes:
        p.start()

    # Exit the completed processes
    for p in processes:
        p.join()

    # Get process results from the output queue
    results = [output.get() for p in processes]

    print(results)

What happens is that multiprocessing runs each child process as a module, so __name__ is only __main__ in the parent. If you don't have that then each child process will (attempt to) start two more processes, each of which will start two more, and so on. No wonder IDLE stops.

This will only show the first 2 results in the output (mp.Queue), you can use results = [output.get() for _ in range(output.qsize())] to get all the items in the Queue — Victor, Feb 19 '21 at 11:24

Python 3 - Multiprocessing - Queue.get() does not respond

1 Answers1

Linked