6

I want to read some HDF5 stream of several gigabytes. I also want to stay in native java for portability.

I have tried Java HDF Object Package and Java HDF5 Interface (JHI5) but theses are some JNI solutions (that I might reconsider if i don't find better options).

https://github.com/jamesmudd/jhdf is a native java library but does not support slicing or streaming so is not usable for big files.

Is there some more option in native Java ?

Edit : I found this : https://www.unidata.ucar.edu/software/netcdf-java/current/ It's supports slicing but not streaming.

bloub
  • 510
  • 1
  • 4
  • 21
  • 1
    There questions doesn't ask for a concrete solution but for a software/library recommendation. Please re-post it at http://softwarerecs.stackexchange.com – hc_dev Dec 03 '19 at 22:46

1 Answers1

4

As the author of jhdf thought I would add my answer. I am not aware of any other pure Java libraries attempting to read HDF5. This is the main reason I started writing one. Although jhdf currently doesn't support slicing or streaming I would certainly like to add it in the future, but that's probably still a while off. Having said that several gigabyte files should be OK with enough heap space so might still be worth a try. You would probably need an xmx about double the size of the dataset you want to open. I have opened several gigabyte datasets successfully using jhdf.

James Mudd
  • 1,816
  • 1
  • 20
  • 25
  • 1
    From my tests I need at least 3 times the size of the dataset into the Xmx (which I have not at all) . An other limitation of jhdf for my scenario is the only possibility to read a file or a bytes array. As I am having a stream and not a file, I am limited by the max array size and would have to wrote the file on the disk before opening it. As I was not able to do any streaming support I wrote a client that convert HDF5 in a custom format (using netcdf-java because it has slicing) that I can easily read in streaming (and compress as well as original hdf5 through the network using gzip). – bloub Dec 04 '19 at 16:57