Persisting Data from Threading

Question

I'm trying to write a 'market data engine' of sorts.

So far I have a queue of threads, each thread will urllib to google finance and re the stock details out the page. Each thread will poll the page ever few seconds.

From here, how can I persist the data in a way another class can just poll it, without the problem of 2 processes accessing the same resource at the same time? For example, if I get my threads to write to a dict that's constantly being updated, will I have trouble reading that same hash from another function?

Answer 1

You are correct that using a standard dict is not thread-safe (see here) and may cause you problems.

A very nice way to handle this is to use the Queue class in the queue module in the standard library. It is thread-safe. Have the worker threads send updates to the main thread via the queue and have the main thread alone update the dictionary.

You could also have the threads update a database, but that may or may not be overkill for what you're doing.

You might also want to take a look at something like eventlet. In fact, they have a web crawler example on their front page.