0

I have a arbitrarily large set of cpu-intensive tasks which each take ~2 seconds to complete and I want to repeat each every 5-15 seconds on a multicore machine. What would be the best way to accomplish this in python? I am thinking that whenever a Process completes, the finished task should be pushed back onto the the list of tasks and then the free Process grabs the next task to run from the list. From my searching, I think this will require a combination of a shared queue and a thread for each process? I have am rather new to python multiprocessing so some guidance would really help me out.

paulkersey
  • 3
  • 1
  • 2
  • Could you be more specific about the nature of the tasks? Are they functions or external commands? Also, do you want them to be run every 5-15 seconds regardless of how busy the machine is, or only if there's a thread available? – Dan Jan 12 '14 at 04:40
  • You may find the answers to [Python thread pool similar to the multiprocessing Pool?](http://stackoverflow.com/q/3033952/78845) useful. – johnsyweb Jan 12 '14 at 04:40

1 Answers1

0

I can't guess as to the nature of your tasks (in particular, why you'd want to repeat the same ones), but this code will keep the processors busy with the same tasks over and over. I've left holes that you'll have to fill in. I don't really understand what you want about timing, but when you put the job back in the queue, you could include one data item that says how soon the job should be run again.

#!/usr/bin/python3

from multiprocessing import *
import time

def nic_process( q ):
    while True:
        job = q.get()
        # perform task
        print(job)

        # add a delay to make timing happy
        time.sleep(1)

        # Put the job back into the queue.
        q.put(job)


if __name__ == '__main__':
    job_q = Queue()
    proc_count = cpu_count()

    pool = [ Process(target=nic_process, args=(job_q,)) for p in range(proc_count) ]
    for p in pool:
        p.start()

    # Fill the job queue.
    job_q.put(['job1','params',0,999])
    job_q.put(['job2','params',1000,1999])
    job_q.put(['job3','params',2000,2999])
    # ...
mojo
  • 4,050
  • 17
  • 24
  • Yeah sorry about the vagueness. The tasks are functions that check URLs for changes. I want them to repeat at intervals but it doesn't have to be strictly timed so I figured a queue might be right for it. This sets me in the right direction though, thanks a bunch! – paulkersey Jan 12 '14 at 05:07