Setting and reading dimscale in hdf5 files correctly in python

Question

I am trying to attach dimension scales to datasets I want to store in hdf5 files with python, but get an error when I try to print the attributes after setting them. The relevant code snippet reads as follows:

import h5py
import numpy as np

# create data and x-axis
my_data = np.random.randint(10, size=(100, 200))
x_axis  = np.linspace(0, 1, 100)

h5f = h5.File('my_file.h5','w')
h5f.create_dataset( 'data_1', data=my_data )
h5f['data_1'].dims[0].label = 'm'
h5f['data_1'].dims.create_scale( h5f['x_axis'], 'x' )

# the following line is creating the problems
h5f['data_1'].dims[0].attach_scale( h5f['x_axis'] )

# this is where the crash happens but only if the above line is included
for ii in h5f['data_1'].attrs.items():
    print ii

h5f.close()

The command print(h5.version.info) prints the following output:

Summary of the h5py configuration
---------------------------------

h5py    2.2.1
HDF5    1.8.11
Python  2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2]
sys.platform    linux2
sys.maxsize     9223372036854775807
numpy   1.8.2

The error message is the following:

Traceback (most recent call last):
  File "HDF_write_dimScales.py", line 16
    for ii in h5f['data_1'].attrs.items():
  File "/usr/lib/python2.7/dist-packages/h5py/_hl/base.py", line 347, in items
    return [(x, self.get(x)) for x in self]
  File "/usr/lib/python2.7/dist-packages/h5py/_hl/base.py", line 310, in get
    return self[name]
  File "/usr/lib/python2.7/dist-packages/h5py/_hl/attrs.py", line 55, in __getitem__
    rtdt = readtime_dtype(attr.dtype, [])
  File "h5a.pyx", line 318, in h5py.h5a.AttrID.dtype.__get__ (h5py/h5a.c:4285)
  File "h5t.pyx", line 337, in h5py.h5t.TypeID.py_dtype (h5py/h5t.c:3892)
TypeError: No NumPy equivalent for TypeVlenID exists

Any ideas or hints are appreciated.

score 1 · Accepted Answer · answered Nov 20 '15 at 18:02

It works with some slight adjusts for me on h5py 2.5.0. The problem may be to do with when you call create_scale. With h5py 2.5.0, I get a KeyError for h5f['x_axis'] in your create_scale() call. To get your example to work, I had to explicitly create the x_axis dataset first.

import h5py
import numpy as np

# create data and x-axis
my_data = np.random.randint(10, size=(100, 200))

# Use a context manager to ensure h5f is closed
with h5py.File('my_file.h5','w') as h5f:
    h5f.create_dataset( 'data_1', data=my_data )

    # Create the x_axis dataset directly in the HDF5 file
    h5f['x_axis']  = np.linspace(0, 1, 100)

    h5f['data_1'].dims[0].label = 'm'

    # Now we can create and attach the scale without problems
    h5f['data_1'].dims.create_scale( h5f['x_axis'], 'x' )
    h5f['data_1'].dims[0].attach_scale( h5f['x_axis'] )

    for ii in h5f['data_1'].attrs.items():
        print(ii)

# Output
#(u'DIMENSION_LABELS', array(['m', ''], dtype=object))
#(u'DIMENSION_LIST', array([array([<HDF5 object reference>], dtype=object),
#       array([], dtype=object)], dtype=object))

If you're still having problems, you may have to upgrade to h5py 2.5.0, which has better handling (although still not perfect) of VLEN types.

thanks for the answer, but it does not solve my problem. I guess I will check the update possibilities of the server I am working on (where I have no superuser/root access). — Alf, Nov 23 '15 at 08:48
@Alf If you have access to pip, you can install things in your home directory with `pip install --user h5py`. — Yossarian, Nov 23 '15 at 09:38
I just did this a few minutes before you posted it ;) It is working now (with my code above) and the output reads: — Alf, Nov 23 '15 at 10:24
the output reads: `(u'DIMENSION_LABELS', array(['m', ''], dtype=object)) (u'DIMENSION_LIST', array([array([], dtype=object), array([], dtype=object)], dtype=object)) ` — Alf, Nov 23 '15 at 10:30

score 0 · Answer 2 · edited May 23 '17 at 10:27

0

This is just a guess, but since the error references TypeVlenID, it may have something to do with an incomplete implementation of vlen in h5py (especially in our version of the module).

Inexplicable behavior when using vlen with h5py

Writing to compound dataset with variable length string via h5py (HDF5)

edited May 23 '17 at 10:27

Community

1
1

answered Nov 19 '15 at 20:29

hpaulj

221,503
14
230
353

hmm, thanks for pointing me to that problem, I will read up on that. – Alf Nov 23 '15 at 08:49

Setting and reading dimscale in hdf5 files correctly in python

2 Answers2