3

New at Python and Numpy, trying to create 263-dimensional arrays. I need so much dimensions for Machine Learning model. Of course one way is using numpy.zeros or numpy.ones and writing code as below : 

x=np.zeros((1,1,1,1,1,1,1,1,1,1,1))   #and more 1,1,1,1

Is there an easier way to create arrays with many dimensions?

terminataor
  • 33
  • 1
  • 1
  • 6

4 Answers4

5

You don't need 263-dimensions. If every dimension had only size 2, you'd still have 2 ** 263 elements, which are: 14821387422376473014217086081112052205218558037201992197050570753012880593911808

You wouldn't be able to do anything with such a matrix : not even initializing on Google servers.

You either need an array with 263 values :

>>> np.zeros(263)
array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.])

or a matrix with 263 vectors of M elements (let's say 3):

>>> np.zeros((263, 3))
array([[ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       ...
       ...
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.]])

There are many advanced research centers that are perfectly happy with vanilla Numpy. Having to use less than 32 dimensions doesn't seem to bother them much for quantum mechanics or machine learning.

Eric Duminil
  • 52,989
  • 9
  • 71
  • 124
0

Let's start with the numpy documentation, help(np.zeros) gives

zeros(shape, dtype=float, order='C')

Return a new array of given shape and type, filled with zeros.

Parameters
----------
shape : int or sequence of ints
    Shape of the new array, e.g., ``(2, 3)`` or ``2``.
...
Returns
-------
out : ndarray
    Array of zeros with the given shape, dtype, and order.
...

The shape argument is just a list of the size of each dimension (but you probably knew that). There are lots of ways to easily create such a list in python, one quick way is 

 np.zeros(np.ones(263, dtype=int))

But, as others have mentioned, numpy has a somewhat arbitrary limitation of 32 dimensions. In my experience, you can get similar and more flexible behavior by keeping an index array showing which "dimension" each row belongs to.

user2699
  • 2,927
  • 14
  • 31
0

Most likely, for ML applications you don't actually want this:

shape = np.random.randint(1,10,(263,))
arr = np.zeros(shape)  # causes a ValueError anyway

You actually want something sparse

for i, value in enumerate(nonzero_values):
    arr[idx[i]] = value

idx in this case is a (num_samples, 263) array and nonzero_values is a (num_samples,) array.

ML algorithms usually work on these idx and value arrays (usually called X and Y) since the actual arrays would be enormous otherwise.

Sometimes you need a "one-hot" array of your dimensions, which will make idx.shape == (num_samples, shape.sum()), with idx containting only 0 or 1 values. But that's still smaller than any sort of high-dimetnsional array.

Daniel F
  • 13,620
  • 2
  • 29
  • 55
0

There is a new package called DimPy which can create multi-dimensional arrays in python very easily. To install use
pip install dimpy Use example

from dimpy import *
a=dim(4,5,6) # This is a 3 dimensional array of 4x5x6 elements. Use any number of dimensions within '( ) ' separated by comma
print(a)

By default every element will be zero. To change it use dfv(a, 'New value') To express it into numpy style array, use a=npary(a) See in more details here: https://www.respt.in/p/python-package-dimpy.html?m=1

Dwaipayan Deb
  • 11
  • 1