Asynchronously read and process an image in python

Question

Context

I often found myself in the following situation:

I have a list of image filenames I need to process
I read each image sequentially using for instance scipy.misc.imread
Then I do some kind of processing on each image and return a result
I save the result along the image filename into a Shelf

The problem is that simply reading the image takes a non negligible amount of time, sometime comparable or even longer than the image processing.

Question

So I was thinking that ideally I could read image n + 1 while processing image n. Or even better processing and reading multiple images at once in an automagically determined optimal way ?

I have read about multiprocessing, threads, twisted, gevent and the like but I can't figure out which one to use and how to implement this idea. Does anyone have a solution to this kind of issue ?

Minimal example

# generate a list of images
scipy.misc.imsave("lena.png", scipy.misc.lena())
files = ['lena.png'] * 100

# a simple image processing task
def process_image(im, threshold=128):
    label, n = scipy.ndimage.label(im > threshold)
    return n

# my current main loop
for f in files:
    im = scipy.misc.imread(f)
    print process_image(im)

Start off with a thread worker pool using `Queue`. Then once you have that working and understand it, move onto other methods if required. — Preet Kukreti, Sep 18 '12 at 09:39
Multithreading generally yields disappointing multicore performance scaling with Python; multiprocessing is the way to go (see Philip's answer). — timday, Sep 18 '12 at 20:17

score 15 · Accepted Answer · edited May 23 '17 at 11:45

Philip's answer is good, but will only create a couple of processes (one reading, one computing) which will hardly max out a modern >2 core system. Here's an alternative using multiprocessing.Pool (specifically, its map method) which creates processes which do both the reading and compute aspects, but which should make better use of all the cores you have available (assuming there are more files than cores).

#!/usr/bin/env python

import multiprocessing
import scipy
import scipy.misc
import scipy.ndimage

class Processor:
    def __init__(self,threshold):
        self._threshold=threshold

    def __call__(self,filename):
        im = scipy.misc.imread(filename)
        label,n = scipy.ndimage.label(im > self._threshold)
        return n

def main():
    scipy.misc.imsave("lena.png", scipy.misc.lena())
    files = ['lena.png'] * 100

    proc=Processor(128)
    pool=multiprocessing.Pool()
    results=pool.map(proc,files)

    print results

if __name__ == "__main__":
    main()

If I increase the number of images to 500, and use the processes=N argument to Pool, then I get

Processes   Runtime
   1         6.2s
   2         3.2s
   4         1.8s
   8         1.5s

on my quad-core hyperthreaded i7.

If you got into more realistic use-cases (ie actual different images), your processes might be spending more time waiting on the image data to load from storage (in my testing, they load virtually instantaneously from cached disk) and then it might be worth explicitly creating more processes than cores to get some more overlap of compute and load. Only your own scalability testing on a realistic load and HW can tell you what's actually best for you though.

score 14 · Answer 2 · edited Jun 08 '19 at 17:23

The multiprocessing package is pretty easy to use. Look at the Queues example for a guide. You'll be following the producer consumer model. You want one (or more) producer processes reading images, and one (or more) consumer processes doing the image processing.

Your example would look something like this:

from multiprocessing import Process, Queue
import scipy

def process_images(q):
    while not q.empty():
        im = q.get()
        # Do stuff with item from queue

def read_images(q, files):
    for f in files:
        q.put(scipy.misc.imread(f))

if __name__ == '__main__':
    q = Queue()

    producer = Process(target=read_images, args=(q, files))
    producer.start()
    consumer = Process(target=process_images, args=(q, ))
    consumer.start()

This is a bit simpler than your original idea. In this example the producer adds to the queue as fast as it can rather than just staying one ahead of the consumer. That might be a problem if the producer gets so far ahead that you don't have enough memory to hold the queue. If problems arise you can get deeper into the multiprocessing docs, but this should be enough to get you started.

Add a comma after `q` to turn the brackets into a tuple: `consumer = Process(target=process_images, args=(q,))`. — j08lue, Sep 21 '16 at 20:24

Asynchronously read and process an image in python

2 Answers2

Linked