0

I need to read some huge database from HDF5 files and organize it in a nice way to make it easy to read and use.

I saw this post Python List as variable name and I'm trying to make a dictionary of dictionaries.

Basically I have a list of data sets and variables that I need to read form the HDF5 files. As an example I created this two lists:

dataset = [0,1,2,3]
var = ['a','b','c']

Now, there is legacy "home brewed" read_hdf5(dataset,var) function that reads the data from the HDF5 files and returns the appropriate array.

I can easily read from a specific dataset (say 0) at a time creating a dictionary like this:

data = {}
for type in var:
    data[type] = read_hdf5(0,type)

Which gives me a nice dictionary if all the data for each variable in dataset 0.

Now I wan to be able to implement a dictionary of dictionaries so I can be able to access the data like this: 

data[dataset][var]

That returns the array of data for the given set and variable

I tried the following but the only thing that the loop is doing is overwriting the last variable read:

for set in dataset:
    for type in var:
        data[set] = {'set':set, str(type): read_hdf5(set,type)}

Any ideas? Thank you!!!

Community
  • 1
  • 1
Jemme
  • 303
  • 2
  • 12
  • Instead of rolling your own, have you considered using [Pandas](http://pandas.pydata.org/)? It can [read HDF5 files](http://pandas.pydata.org/pandas-docs/stable/io.html#io-hdf5). – Carsten Oct 24 '14 at 11:17

1 Answers1

2

You have to create a new dict for each set before iterating on vars:

dataset = [0,1,2,3]
var = ['a', 'b', 'c']

data = {}
for set in datasets:
    data[set] = {}
    for type in var:
        data[set][type] = read_hdf5(set, type)

As a side note: set and type are builtin names so you'd better use something else.

bruno desthuilliers
  • 75,974
  • 6
  • 88
  • 118
  • Thanks! But it gives me back an Traceback (most recent call last) KeyError: 25. It seems to read the data for the firs set, then crashes. Any idea why? – Jemme Oct 24 '14 at 11:29
  • Never mind, it was a typo on my side. It works like charm!!! Thank you very much! – Jemme Oct 24 '14 at 12:01