Pandas: in memory sorting hdf5 files

Asked Jul 02 '14 at 08:22

Active Jul 02 '14 at 08:22

Viewed 911 times

I have the following problem:

I have a set several hdf5 files with similar data frames which I want to sort globally based on multiple columns.

My input is the file names and an ordered list of columns I want to use for sorting. The output should be a single hdf5 file containing all the sorted data.

Each file can contain millions of rows. I can afford loading a single file in memory but not the entire dataset.

Naively I would like first to copy all the data in a single hdf5 file (which is not difficult) and then find out a way to do in memory sorting of this huge file.

Is there a quick way to sort in memory a pandas datastructure stored in an hdf5 file based on multiple columns?

I have already seen ptrepack but it seems to allow you sorting only on a single column.

asked Jul 02 '14 at 08:22

Luca Fiaschi

3,145
7
31
44

Why do you actually need to sort the resulting file by more than one column? You can simply index in a query. – Jeff Jul 02 '14 at 11:35
Hi @Jeff, what do you mean exactly with index in a query? Thanks – Luca Fiaschi Aug 05 '14 at 06:41
why don't u show some code of what you are doing – Jeff Aug 05 '14 at 09:21
I am currently using a completely different strategy; I push all files one by one in a database table and then I retrieve the data I want sorted on the fly with pandas sql interface and an sql query. – Luca Fiaschi Aug 06 '14 at 10:46
if that works for you gr8 – Jeff Aug 06 '14 at 11:24

Pandas: in memory sorting hdf5 files

0 Answers0

Linked