h5py OSError: Unable to open file (File signature not found)

Question

I'm a bit confused about an error I'm receiving when using h5py. I'm trying to apply a python script to loop through sets of h5py files located in different directories. For example, the first set of h5py files is located at

Reduced/rho=0.75/2/Data/snapshots

When I run the python script from a sub directory of Reduced

Reduced/test_h5py

with the following python script

import h5py
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.colors as colors               
import cmocean
import os

de.logging_setup.rootlogger.setLevel('ERROR')


# Plot writes                                                                                                                          

path = '../rho=0.75/2/Data/snapshots'
for filename in os.listdir(path):
    with h5py.File(path+'/'+filename,'r') as file:

everything works fine, the script loops through the data and gives me an output. Now, the issue arises when I try to apply the same python script to other data i.e at the set of h5py files located at

Reduced/rho=0.75/4/Data/snapshots

Now, when I run the previous python script with just a modification in the path from 2 to 4

import h5py
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.colors as colors               
import cmocean
import os

de.logging_setup.rootlogger.setLevel('ERROR')


# Plot writes                                                                                                                          

path = '../rho=0.75/4/Data/snapshots'
for filename in os.listdir(path):
    with h5py.File(path+'/'+filename,'r') as file:

I get the following error

Traceback (most recent call last):
  File "newest_edit.py", line 17, in <module>
    with h5py.File(path+'/'+filename,'r') as file:
  File "/usr/local/lib/python3.5/site-packages/h5py/_hl/files.py", line 269, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
  File "/usr/local/lib/python3.5/site-packages/h5py/_hl/files.py", line 99, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 78, in h5py.h5f.open
OSError: Unable to open file (file signature not found)

Can anyone make sense of this? I thought it might be a os.listdir() error but after searching I didn't find anything to solve the issue. Thanks for your help.

EDIT:

I forgot to mention. When I run the python script in a Jupyter notebook and apply the script to individual h5py files it works just fine for both the 2 and 4 paths. The h5py files open and I can retrieve the data stored on them, so I can't imagine it would be corrupted files or not in hdf5 format.

Answer 1

I don't really know what I'm talking about here so sorry in advance if this doesn't make sense or help or just isn't related.

It's an extremely annoying error that no one has attempted to answer in over a year so I thought the tiny bit of knowledge I have might help.

I had a similar problem whilst working on a cluster. It turned out that h5py uses something called flock() and that this can sometimes have a negative effect on the filesystem performance.

Because of this flock() is sometimes not allowed. If you are working on someone elses filesystem then ask admin if they will alow it and if not then I can't find a way around it (I have not tried to find other libraries that don't use flock()).

If it is your own filesystem then you should be able to allow flock() but I'm not sure how to do this. This may help you if it's NFS

I would certainly be interested to hear how you got on?