3

I have a a number of large (13GB+ in size) h5 files, each h5 file has two datasets that were created with pandas: 

df.to_hdf('name_of_file_to_save', 'key_1',table=True) 
df.to_hdf('name_of_file_to_save', 'key_2', table=True) # saved to the same h5 file as above

I've seen a post here:

Concatenate two big pandas.HDFStore HDF5 files

on using odo to concatenate h5 files. What I want to do is for each h5 file that was created, each having key_1 and key_2, combine them so that all of the key_1 data are in one dataset in the new h5 file and all of the key_2 are in another dataset in the same new h5 file. All of key_1 have the same number of columns, the same applies to key_2

I've tried this:

from odo import odo
files = ['file1.h5','file2.h5','file3.h5','file4.h5']
for i in files:
    odo('hdfstore://path_to_here_h5_files_live/%s::key_1' % i,
        'hdfstore://path_store_new_large_h5::key_1')

Howeever I get an error: 

(tables/hdf5extension.c:7824)
tables.exceptions.HDF5ExtError: HDF5 error back trace

File "H5A.c", line 259, in H5Acreate2
  unable to create attribute
File "H5Aint.c", line 275, in H5A_create
  unable to create attribute in object header
File "H5Oattribute.c", line 347, in H5O_attr_create
  unable to create new attribute in header
File "H5Omessage.c", line 224, in H5O_msg_append_real
  unable to create new message
File "H5Omessage.c", line 1945, in H5O_msg_alloc
  unable to allocate space for message
File "H5Oalloc.c", line 1142, in H5O_alloc
  object header message is too large

End of HDF5 error back trace

Can't set attribute 'non_index_axes' in node:
/key_1 (Group) ''.
Closing remaining open
Community
  • 1
  • 1
atomsmasher
  • 725
  • 8
  • 20

1 Answers1

3

For this specific case it was a matter of having too many columns, which exceeded the memory limit allocated for that piece of information. The solution is to load the dataframe and transpose it.

atomsmasher
  • 725
  • 8
  • 20