Parallel reading in hdf5

Question

My C++/C program read hundreds of compressed compound arrays from a hdf5 file sequentially and stores them in some vectors. I would like to improve its time performance. I wish I can read 3 or 4 of them in parallel, and then again next 3 or 4, etc.. I am totally new to multithreading or OpenMP or any parallel programming. My question is: - Is it possible to implement on hdf5/C/C++/Linux what I want? - If so, can you direct me to some info or tutorial for beginners? Thank you With respect Nyama

Answer 1

HDF5 technically has a thread-safe mode, but it serializes all library calls so there's no performance benefit (see the link). Depending on your application, you can use fork to create parallel processes instead of parallel threads. If you take this approach, you may need to use interprocess communication (IPC) to transfer the data back to the main process.

Note that whether any of these parallel reading approaches gives any benefit depends a lot on how the HDF5 files are stored on disk. If they're sitting on a standard 7200 RPM disk, you'll probably make things much slower by trying to do parallel reads because you'll start seeking all over the file instead of nicely streaming out contiguous chunks (assuming your disk is not already very fragmented). On the other hand, if the data are on a more advanced file server, on an SSD with a good controller, or on a RAID array, there's a better chance you'll see a benefit. I suggest first doing some profiling to see if the time is being spent doing real filesystem I/O (in which case you need better disk or to spread your data across multiple disks), decompression (multithreading or multiprocessing is more likely to be a big help if this is the bottleneck), or other operations.