Add datetime to HDF5 chunks with HDFql

Question

I have this code to write chunks of arrays to HDF5 and I want to add a timestamp attribute to each chunk or another field with a datetime

using AS.HDFql;

// declare variables
float[,,] traindata = new float[1, 100, 500];
int number;
int i;

// create an HDF5 file named 'Test.h5' and use (i.e. open) it
HDFql.Execute("create and use file Test.h5");

// create a chunked dataset named 'data' (within a group named 'grp') of data type float with three dimensions (the first dimension is extendible)
HDFql.Execute("create chunked(1, 100, 500) dataset grp/data as float(0 to unlimited, 100, 500)");

// register variable 'traindata'
number = HDFql.VariableRegister(traindata);

// loop 100000 times
for(i = 0; i < 100000; i++)
{
   // populate variable 'traindata'
   // (...)

   // insert (i.e. write) data stored in 'traindata' into the last position of 'data' (using a hyperslab selection)
   HDFql.Execute("insert into grp/data(-1:::) values from memory " + number);

   // alter (i.e. extend) first dimension of 'data' plus one unit
   HDFql.Execute("alter dimension grp/data to +1");
}

SOG · Accepted Answer · 2022-09-11T20:03:47.890

One way to have a timestamp associated to each chunk written in the HDF5 file is to have a compound dataset with two members: the first member stores the data itself while the second stores the timestamp (e.g. UNIX epoch time). Updating your code, this could look as follows:

// use HDFql namespace
using AS.HDFql;

// declare structure
[StructLayout(LayoutKind.Sequential, Pack=1)]
public struct Data
{
    public float [,]vals;
    public int timestamp;
}

// declare variables
Data []traindata = new Data[1];
int number;
int i;

// create an HDF5 file named 'Test.h5' and use (i.e. open) it
HDFql.Execute("create and use file Test.h5");

// create a chunked dataset named 'data' (within a group named 'grp') as an extendible compound (with two members 'vals' and 'timestamp')
HDFql.Execute("create chunked(1) dataset grp/data as compound(vals as float(100, 500), timestamp as unsigned int)(0 to unlimited)");

// register variable 'traindata'
number = HDFql.VariableRegister(traindata);

// loop 100000 times
for(i = 0; i < 100000; i++)
{
    // populate variable 'traindata' (i.e. update members 'vals' and 'timestamp' with values)
    // (...)

    // alter (i.e. extend) first dimension of dataset 'data' plus one unit
    HDFql.Execute("alter dimension grp/data to +1");

    // insert (i.e. write) data stored in 'traindata' into the last position of dataset 'data' (using a hyperslab selection)
    HDFql.Execute("insert into grp/data(-1:::) values from memory " + number);
}

Add datetime to HDF5 chunks with HDFql

1 Answers1