How to deal with large arrays in python?

Question

I have very large arrays which are needed to be stored on disk. How can I do that? I tried using h5py, I read the documentation and found the issue with it is its 2nd dimension can have maximum number of 1024. How can I deal with an array that have dimensions of lets say 10^5 x 10^5?

So what you are really looking is a *multidimensional array*, right? Check this answer: http://stackoverflow.com/a/1062629/742269 — Avión, Mar 22 '16 at 10:11

score 0 · Accepted Answer · answered Mar 22 '16 at 10:55

0

Assuming you read this part of the documentation. This does not say that there is a general maximum size of 1024 for the second dimension. It's just an example of using resizable datasets.

The per-axis limit for HDF5 (and h5py) is 2^64 elements, as can be read in the documentationon on resizable datasets.

answered Mar 22 '16 at 10:55

Hannes Ovrén

21,229
9
65
75

what is meaning of "per axis limit of 2^64 elements"? @Hannes Ovren – Kavan Mar 22 '16 at 11:20
It means that each dimension can have a size that is at most 2^64. That is a very very large number. – Hannes Ovrén Mar 22 '16 at 11:21
No. You can have approximately 10^19 rows if you want. – Hannes Ovrén Mar 22 '16 at 11:25
Yes. Every dimension actually. – Hannes Ovrén Mar 22 '16 at 11:30

How to deal with large arrays in python?

1 Answers1