Python multiprocessing start a number of workers and start others when one finishes

Question

So i have something similar to this

import multiprocessing

class MyFancyClass(object):

    def __init__(self, name):
    self.name = name

    def do_something(self):
    proc_name = multiprocessing.current_process().name
    print 'Doing something fancy in %s for %s!' % (proc_name, self.name)


def worker(q):
    obj = q.get()
    obj.do_something()


if __name__ == '__main__':
        urls = [
    'http://www.python.org',
    'http://www.python.org/about/',
    'http://www.python.org/community/awards/'
    # etc..
]

    queue = multiprocessing.Queue()

    p = multiprocessing.Process(target=worker, args=(queue,))
    p.start()

    queue.put(MyFancyClass('Fancy Dan'))

    # Wait for the worker to finish
    queue.close()
    queue.join_thread()
    p.join()

What i want to do is have 4 or more "workers" start and process the urls and when one finishes start another one. What would be the best way to do this. I spend two days on this and can't figure it out.

Thank you Thank you.

You should use a process pool. Read more here https://docs.python.org/2/library/multiprocessing.html — AndreyF, Feb 21 '17 at 14:45
Could you give me an example of how to use it with a class ? — walentin, Feb 21 '17 at 14:49
https://www.pythonsheets.com/notes/python-concurrency.html --many of examples to use as templates for various parallel processing including multiprocessing — gregory, Feb 21 '17 at 15:08

Anfernee · Accepted Answer · 2017-02-21T18:44:10.113

2

Using concurrent.futures

import concurrent.futures
import multiprocessing

def do_something(name):
    proc_name = multiprocessing.current_process().name
    print 'Doing something fancy in %s for %s!' % (proc_name, name)

class MyFancyClass(object):

    def __init__(self, name):
    self.name = name


MyFancy = MyFancyClass("Name")

if __name__ == '__main__':
    urls = [
        'http://www.python.org',
        'http://www.python.org/about/',
        'http://www.python.org/community/awards/'
        # etc..
]

with concurrent.futures.ProcessPoolExecutor() as executor:
    results = executor.map(do_something, urls)

See concurrent.futures documentation for details.

edited Feb 21 '17 at 18:44

answered Feb 21 '17 at 15:25

Anfernee

1,445
1
15
26

NameError: name 'do_something' is not defined – walentin Feb 21 '17 at 15:35
1

You need to integrate the code above with yours. I'll update my answer. – Anfernee Feb 21 '17 at 15:43
PicklingError: Can't pickle : attribute lookup __builtin__.instancemethod failed – walentin Feb 21 '17 at 17:44
Define `do_something` as a function and not a class method to resolve this. Check http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-pythons-multiprocessing-pool-ma I've updated my answer – Anfernee Feb 21 '17 at 18:40
that will start just do_something. I need to start the hole class. – walentin Feb 21 '17 at 20:13
1

Take a look at the link in my last comment. It'll show you how to prepare your instance methods for pickling. – Anfernee Feb 21 '17 at 21:04

score 1 · Answer 2 · answered Feb 21 '17 at 14:47

1

No need to invent the wheel. ProcessPoolExecutor from concurrent.futures does exactly what you need.

answered Feb 21 '17 at 14:47

Roman Miroshnychenko

1,496
1
10
16

score 0 · Answer 3 · answered Feb 21 '17 at 14:54

0

Use Pool from multiprocessing:

Here is a short usage example that may fit your purpose:

from multiprocessing import Pool

def f(x,y):
    print x*y

p = Pool(5)
for i in range(100):
    p.apply_async(f,(i,i+1))
p.close()
p.join()

answered Feb 21 '17 at 14:54

AndreyF

1,798
1
14
25

Python multiprocessing start a number of workers and start others when one finishes

3 Answers3