Numpy For Large Data Sets

Question

For generating a probability density function of some cases, maybe 1 million observations are considered. When I work with numpy array, I was encountered by size limit 32.

Is it too few ?

In this case, how can we store more than 32 elements without distributing the elements into different columns and maybe arrays in arrays ?

import numpy
my_list = []
for i in range(0, 100):
    my_list.append(i)

np_arr = numpy.ndarray(np_arr) # ValueError: sequence too large; cannot be greater than 32

I have no idea what you're asking. What are "elements" here? — roganjosh, Dec 30 '18 at 23:40
Any interpretation I can make of the question suggests that no such limit exists — roganjosh, Dec 30 '18 at 23:42
How are you creating the array? The only limitation to Numpy array size is your hardware resources. You should have no problem with 1M elements. `np.zeros(1e6)` will created an array with a million zeros. Give it a try. — kcw78, Dec 31 '18 at 02:14
You did not specify how `np_arr` variable was defined in your code. — Xiaoyu Lu, Dec 31 '18 at 17:11
@Goktug: see https://stackoverflow.com/questions/15879315/what-is-the-difference-between-ndarray-and-array-in-numpy — Warren Weckesser, Dec 31 '18 at 21:01

score 2 · Answer 1 · answered Dec 31 '18 at 17:05

2

When you create an array with numpy.ndarray, the first argument is the shape of the array. Interpreting that list as a shape would indeed give a huge array. If you just want to turn the list into an array, you want numpy.array:

import numpy
my_list = []
for i in range(0, 100):
    my_list.append(i)

np_arr = numpy.array(my_list)

answered Dec 31 '18 at 17:05

RuthC

1,269
1
10
21

You don't need to create a list...just use arange; `np_arr = numpy.arange(100)` (for int32) or `np_arr = numpy.arange(100.0)` (for float64). – kcw78 Dec 31 '18 at 19:18
@kcw78 I believe the OP's error is coming up because they are using the wrong function to convert a list into an array. My answer aims to show the right way to make that conversion. – RuthC Dec 31 '18 at 19:32
Good point. It wasn't clear why the OP used a list to create an array. If he really needs to convert a list, then he should follow your steps. If he just needs an array with [0, 1, 2, 3, 4 ...100], he can use arange(). Will leave that for OP to figure out. – kcw78 Dec 31 '18 at 19:38
I assumed that the "real" code has a more complicated list and that `range` was just a convenient way to make a [mcve]. I could be completely wrong of course! – RuthC Dec 31 '18 at 19:55

Numpy For Large Data Sets

1 Answers1