Which python data type should I use to create a huge 2d array (7Mx7M) with fast random access?

Question

Which python data type should I use to create a huge 2d array (7Mx7M) with fast random access? I want to write each element once and read many times.

Thanks

See this so question http://stackoverflow.com/questions/1053928/python-numpy-very-large-matrices — Fredrik Pihl, May 20 '13 at 19:36

Joe · Accepted Answer · 2013-05-20T19:42:49.353

2

As such a large amount of data will probably not fit into memory, I would recommend PyTables which uses underlying HDF5 files and supports efficient slicing, ND array data, compression etc.

http://www.pytables.org/

edited May 20 '13 at 19:42

answered May 20 '13 at 19:37

Joe

6,497
4
29
55

score 1 · Answer 2 · answered May 20 '13 at 19:36

It would help to know more about your data, and what kind of access you need to provide. How fast is "fast enough" for you? Just to be clear, "7M" means 7,000,000 right?

As a quick answer without any of that information, I have had positive experiences working with redis and tokyo tyrant for fast read access to large amounts of data, either hundreds of megabytes or gigabytes.

Which python data type should I use to create a huge 2d array (7Mx7M) with fast random access?

2 Answers2