5

Premise:

I've created a mainwindow. One of the drop down menu's has an 'ProcessData' item. When it's selected, I create a QProgressDialog. I then do a lot of processing in the main loop and periodically update the label and percentage in the QProgressDialog.

My processing looks like: read a large amount of data from a file (numpy memmapped array), do some signal processing, write the output to a common h5py file. I iterate over the available input files, and all of the output is stored in a common h5py hdf5 file. The entire process takes about two minutes per input file and pins one CPU to 100%.

Goal:

How do I make this process non-blocking, so that the UI is still responsive? I'd still like my processing function to be able to update the QProgressDialog and it's associated label.

Can I extend this to process more than one dataset concurrently and retain the ability to update the progressbar info?

Can I write into h5py from more than one thread/process/etc.? Will I have to implement locking on the write operation?

Software Versions:

I use python 3.3+ with numpy/scipy/etc. UI is in PyQt4 4.11/ Qt 4.8, although I'd be interested in solutions that use python 3.4 (and therefore asyncio) or PyQt5.

troy.unrau
  • 1,142
  • 2
  • 12
  • 26
  • [this answer](http://stackoverflow.com/a/16633274/832621) shows how to achieve this using a `numpy.memmap` array – Saullo G. P. Castro Mar 19 '15 at 00:26
  • I'm already successfully loading the data using a memmap array to read the data from disk, then storing that in an h5py file. The question is about doing the process concurrently. – troy.unrau Mar 19 '15 at 16:12

1 Answers1

4

This is quite a complex problem to solve, and this format is not really suited to providing complete answers to all your questions. However, I'll attempt to put you on the right track.

How do I make this process non-blocking, so that the UI is still responsive? I'd still like my processing function to be able to update the QProgressDialog and it's associated label.

To make it non-blocking, you need to offload the processing into a Python thread or QThread. Better yet, offload it into a subprocess that communicates progress back to the main program via a thread in the main program.

I'll leave you to implement (or ask another question on) creating subprocesses or threads. However, you need to be aware that only the MainThread can access GUI methods. This means you need to emit a signal if using a QThread or use QApplication.postEvent() from a python thread (I've wrapped the latter up into a library for Python 2.7 here. Python 3 compatibility will come one day)

Can I extend this to process more than one dataset concurrently and retain the ability to update the progressbar info?

Yes. One example would be to spawn many subprocesses. Each subprocess can be configured to send messages back to an associated thread in the main process, which communicates the progress information to the GUI via the method described for the above point. How you display this progress information is up to you.

Can I write into h5py from more than one thread/process/etc.? Will I have to implement locking on the write operation?

You should not write to a hdf5 file from more than one thread at a time. You will need to implement locking. I think possibly even read access should be serialised.

A colleague of mine has produced something along these lines for Python 2.7 (see here and here), you are welcome to look at it or fork it if you wish.

three_pineapples
  • 11,579
  • 5
  • 38
  • 75
  • Due to the demise of mercurial hosting on BitBucket, it is likely the links at the end of my post are going to break soon."zlock" is now part of [zprocess](https://github.com/chrisjbillington/zprocess). h5lock has seen updates, but they are incorporated into a larger project [here](https://github.com/labscript-suite/labscript_utils/blob/master/h5_lock.py). – three_pineapples Apr 25 '20 at 01:56
  • Further update: Due to a package structure change, h5lock is now [here](https://github.com/labscript-suite/labscript-utils/blob/2c7deba0b5c091b44a877f1cfe058df9b7d612aa/labscript_utils/h5_lock.py) – three_pineapples Jun 30 '20 at 06:00