4

I was trying to input a list of numeric arrays into an HDF5 file using h5py. Consider for example:

f = h5py.File('tester.hdf5','w')
b = [[1,2][1,2,3]]

This throws an error. 

TypeError: Object dtype dtype('O') has no native HDF5 equivalent

So I am assuming HDF5 doesn't support this.

Like you can store a list of strings by using a special datatype, is there a way for list of numeric array as well.

H5py store list of list of strings

If not, what are the other suitable ways to store a list like this which I can access later from memory.

Thanks for the help in advance

Ankit Mishra
  • 43
  • 1
  • 4
  • I can `pickle.dump` it, I guess. But I am trying to fit a huge dataset in a single file. It would help if HDF5 can take it in, and thats where I got curious about this. – Ankit Mishra Jun 08 '17 at 10:13

1 Answers1

8

You could split your list of lists in seperate datasets and store them seperately:

import h5py
my_list = [[1,2],[1,2,3]]
f = h5py.File('tester.hdf5','w')
grp=f.create_group('list_of_lists')
for i,list in enumerate(my_list):
    grp.create_dataset(str(i),data=list)

After doing so, you can slice trough your datasets like before with a little variation:

In[1]: grp[str(0)][:].tolist()
Out[1]: [1, 2]
In[2]: grp[str(1)][:].tolist()
Out[2]: [1, 2, 3]
Franz
  • 623
  • 8
  • 14