How do I multiply each element in an array with a large number without getting OverflowError: Python int too large to convert to C long?

Question

I'm writing a program where I want to multiply each number in an array (from numpy) with a big number (1.692287392646066e+41) and I do it like this:

x = array([ 16,  18,  19,  15,  27,  26,  13,  34,  38,  36,  43,  42,  48,
    50,  55,  57,  70,  67,  65,  85,  99,  94,  90, 112, 126, 130,
   142, 120])
weight = 1.692287392646066e+41

x[:] = [i*weight for i in x]

But then I recieve the error "OverflowError: Python int too large to convert to C long". The strange thing is that multiplying the numbers outside the for loop seems to work:

In [1]: x[0]*weight
Out[1]: 2.7076598282337055e+42

My question is then, is there any way for me to do this inside a for loop, since I don't know beforehand how long my array is? (I know it has 1 row.)

Answer 1

It makes no sense to make the multiplication in python and not in numpy. Not only is numpy faster, but it also works better.

What you are trying to do creates a python list, which it then tries to copy into the numpy array element wise.

Instead just use

x = x * weight

and you're fine.

EOL doubted that this works, so here's what I just did on my computer:

$ python
Python 2.7.3 (default, Dec 18 2014, 19:03:52)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from numpy import array
>>> x = array([ 16,  18,  19,  15,  27,  26,  13,  34,  38,  36,  43,  42,  48,
...     50,  55,  57,  70,  67,  65,  85,  99,  94,  90, 112, 126, 130,
...    142, 120])
>>> weight = 1.692287392646066e+41
>>> x = x * weight
>>> x
array([  2.70765983e+42,   3.04611731e+42,   3.21534605e+42,
         2.53843109e+42,   4.56917596e+42,   4.39994722e+42,
         2.19997361e+42,   5.75377713e+42,   6.43069209e+42,
         6.09223461e+42,   7.27683579e+42,   7.10760705e+42,
         8.12297948e+42,   8.46143696e+42,   9.30758066e+42,
         9.64603814e+42,   1.18460117e+43,   1.13383255e+43,
         1.09998681e+43,   1.43844428e+43,   1.67536452e+43,
         1.59075015e+43,   1.52305865e+43,   1.89536188e+43,
         2.13228211e+43,   2.19997361e+43,   2.40304810e+43,
         2.03074487e+43])
>>>

Answer 2

The following command with shallow copy (s[:]) will create a view of s for you that its elements are type of int so after assignment it tries to assign the elements created by [i*weight for i in s] to int container but as your result are C long it raise the following error :

OverflowError: Python int too large to convert to C long

If you just use [i*weight for i in s] you'll see the result as well :

>>> [i*weight for i in s]
[-1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60, -1.5608596215653603e+60]

Instead you can multiple directly :

>>> s = s*weight
>>> s
array([  2.70765983e+42,   3.04611731e+42,   3.21534605e+42,
         2.53843109e+42,   4.56917596e+42,   4.39994722e+42,
         2.19997361e+42,   5.75377713e+42,   6.43069209e+42,
         6.09223461e+42,   7.27683579e+42,   7.10760705e+42,
         8.12297948e+42,   8.46143696e+42,   9.30758066e+42,
         9.64603814e+42,   1.18460117e+43,   1.13383255e+43,
         1.09998681e+43,   1.43844428e+43,   1.67536452e+43,
         1.59075015e+43,   1.52305865e+43,   1.89536188e+43,
         2.13228211e+43,   2.19997361e+43,   2.40304810e+43,
         2.03074487e+43])

Answer 3

Please be reminded that the numpy.ndarray instance x, once created, will be assigned a unique type for the underlying data stored. This is quite unlike a Python native list, which is something like an array that contains references to arbitrary, duck-typed Python objects.

Subsequent in-place modification of the underlying array data (in this case a slice assignment, i.e. syntactic sugar that actually calls the __setitem__ method) will make attempts to cast values into the storage type. In this case, the array x has integers, but the float values computed at the RHS of the assignment are too large to be cast into integers, causing integer overflow.

It could have worked if instead you had created the array with a suitable float type. In pseudo-Python:

# Type for the storage of x explicitly set when creating it.
x = numpy.array([ whatever integer literals ... ], dtype=numpy.float64)
# Assignment succeeds.
x[:] = [ some list of floats ... ]

Answer 4

You can multiply the whole array with the factor in a single operation, there is no need to multiply the individual elements (that's the point of numpy):

>>> x = array([ 16,  18,  19,  15,  27,  26,  13,  34,  38,  36,  43,  42,  48,
...     50,  55,  57,  70,  67,  65,  85,  99,  94,  90, 112, 126, 130,
...    142, 120])
>>> weight = 1.692287392646066e+41
>>> x * weight
array([  2.70765983e+42,   3.04611731e+42,   3.21534605e+42,
         2.53843109e+42,   4.56917596e+42,   4.39994722e+42,
         2.19997361e+42,   5.75377713e+42,   6.43069209e+42,
         6.09223461e+42,   7.27683579e+42,   7.10760705e+42,
         8.12297948e+42,   8.46143696e+42,   9.30758066e+42,
         9.64603814e+42,   1.18460117e+43,   1.13383255e+43,
         1.09998681e+43,   1.43844428e+43,   1.67536452e+43,
         1.59075015e+43,   1.52305865e+43,   1.89536188e+43,
         2.13228211e+43,   2.19997361e+43,   2.40304810e+43,
         2.03074487e+43])