0

picture of all the plant dataset h5 files in one directory

I have multiple .h5 files with plant datasets trained from a Siamese model. i m trying to combine all into one h5 file for evaluation. i can correctly evaluate each h5 file individually for the most part. but after view this link Combining hdf5 files and evaluating the combined h5 file i was hit with an error. any ideas on what might work combining the h5 files one on with out affecting the "shape"?

import h5py
import glob

with h5py.File('table_merge.h5',mode='w') as h5fw:
    row1 = 0
    for h5name in glob.glob('*.h5'):
        h5fr = h5py.File(h5name,'r') 
        dset1 = list(h5fr.keys())[0]
        arr_data = h5fr[dset1][:]
        dslen = arr_data.shape[0]
        cols = arr_data.shape[1]
        if row1 == 0: 
            h5fw.create_dataset('alldata', dtype="f",  shape=(dslen,cols), maxshape=(None, cols) )
        if row1+dslen <= len(h5fw['alldata']) :
            h5fw['alldata'][row1:row1+dslen,:] = arr_data[:]
        else :
            h5fw['alldata'].resize( (row1+dslen, cols) )
            h5fw['alldata'][row1:row1+dslen,:] = arr_data[:]
        row1 += dslen

the python code i ran above ran into a error :

Traceback (most recent call last):
  File "C:\Users\J.A.X\Desktop\model\model consept 2\h5 merge.py", line 9, in <module>
    arr_data = h5fr[dset1][:]
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "C:\Users\J.A.X\AppData\Local\Programs\Python\Python36\lib\site-packages\h5py\_hl\group.py", line 288, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "C:\Users\J.A.X\AppData\Local\Programs\Python\Python36\lib\site-packages\h5py\_hl\base.py", line 200, in _e
    name = name.encode('ascii')
AttributeError: 'slice' object has no attribute 'encode'
unknown name
  • 3
  • 2
  • The solution depends on the shape of the datasets in each file and how you are combining them. If you want to merge all the datasets into a single dataset, they need to have the same shape. If they have different shapes, that doesn't make sense. Check the dataset shape the same way as a numpy array (e.g. `f['dataset name'].shape`) . If the shapes are not the same, you can copy the datasets without merging. However, you have to resize the images so they are the same when you are training. Please add details about shapes in each file to write an answer. – kcw78 Dec 01 '22 at 20:46
  • Please include code you have written so far. It will help when writing an answer. Also, take a look at the answer to this question: [How can I combine multiple .h5 file?](https://stackoverflow.com/a/58223603/10462884). It shows multiple ways to copy and/or merge datasets. – kcw78 Dec 01 '22 at 21:35
  • Please provide enough code so others can better understand or reproduce the problem. – Community Dec 02 '22 at 10:22

0 Answers0