Python - multithreading / multiprocessing

Question

Background
I have a collection of Python scripts used to build and execute Verilog-AMS tesbenches. The overall design was built with threading in mind, as each major test case is its own testbench and I have all of the supporting files / data output separate for each instance. The only shared items will be the launcher script and my data extraction script. The problem that I'm faced with is that my Verilog-AMS simulator does not natively support multithreading and for my test cases it takes a substantial amount of time to complete.

Problem
The machine I'm running this on has 32GiB of RAM and 8 "cores" available for me to use and I may be able to access a machine with 32. I would like to take advantage of the available computing power and execute the simulations simultaneously. What would be the best approach?

I currently use subprocess.call to execute my simulation. I would like to execute up to n commands at once, with each one executing on a separate thread / as a separate process. Once a simulation has completed, the next one in the queue (if one exists) would execute.

I'm pretty new to Python and haven't really written a threaded application. I would like some advice on how I should proceed. I saw this question, and from that I think the multiprocessing module may be better suited to my needs.

What do you all recommend?

Answer 1

I had done some similar task in the past with Machine Learning and Data Mining. Using multiprocessing in your case may not be that difficult of a task. It depends on how tolerant you are keen on making the program, you can use a Threaded Pool pattern. My personal favourite is Producer - Consumer pattern using Queue, this design can handle a variety of complex task. Here is a sample toy program using multiprocessing:

import multiprocessing
from multiprocessing import Queue, Process
from Queue import Empty as QueueEmpty

# Assuming this text is very very very very large
text="Here I am writing some nonsense\nBut people will read\n..."

def read(q):
   """Read the text and put in a queue"""
   for line in text.split("\n"):
       q.put(line)

def work(qi, qo):
   """Put the line into the queue out"""
   while True:
        try:
            data = qi.get(timeout = 1) # Timeout after 1 second
            qo.put(data)
        except QueueEmpty:
            return # Exit when all work is done
        except:
            raise # Raise all other errors

def join(q):
    """Join all the output queue and write to a text file"""
    f = open("file.txt", w)
    while True:
         try:
            f.write(q.get(timeout=1))
         except QueueEmpty:
            f.close()
            return
         except:
            raise

def main():
   # Input queue
   qi = Queue()
   # Output queue
   qo = Queue()
   # Start the producer
   Process(target = read, args = (qi, )).start()
   # Start 8 consumers
   for i in range(8):
        Process(target = work, args = (qi, qo, )).start()
   # Final process to handle the queue out
   Process(target = join, args = (qo, )).start()

Type this from memory so if there is any error, please correct. :)