Why does built-in sum behave wrongly after "from numpy import *"?

Question

I have some code like:

import math, csv, sys, re, time, datetime, pickle, os, gzip
from numpy import *

x = [1, 2, 3, ... ]
y = sum(x)

The sum of the actual values in x is 2165496761, which is larger than the limit of 32bit integer. The reported y value is -2129470535, implying integer overflow.

Why did this happen? I thought the built-in sum was supposed to use Python's arbitrary-size integers?

_{See How to restore a builtin that I overwrote by accident? if you've accidentally done something like this at the REPL (interpreter prompt).}

Python can handle arbitrarily large numbers; what is your code? — Martijn Pieters, Oct 03 '12 at 19:49
x=[1,2,....] is a long list of integer numbers, add up to 2165496761. — notilas, Oct 03 '12 at 19:53
@notilas No, `import *` was the reason! You got bit by it once, now never do it again. — Lukas Graf, Oct 03 '12 at 20:01
Note that 64-bit builds of Python will require a 64-bit integer overflow to reproduce the problem. — Karl Knechtel, Jan 30 '23 at 07:41

score 14 · Answer 1 · edited Jan 30 '23 at 07:47

Doing from numpy import * causes the built-in sum function to be replaced with numpy.sum:

>>> sum(xrange(10**7))
49999995000000L
>>> from numpy import sum
>>> sum(xrange(10**7)) # assuming a 32-bit platform
-2014260032

To verify that numpy.sum is in use, try to check the type of the result:

>>> sum([721832253, 721832254, 721832254])
-2129470535
>>> type(sum([721832253, 721832254, 721832254]))
<type 'numpy.int32'>

To avoid this problem, don't use star import.

If you must use numpy.sum and want an arbitrary-sized integer result, specify a dtype for the result like so:

>>> sum([721832253, 721832254, 721832254],dtype=object)
2165496761L

or refer to the builtin sum explicitly (possibly giving it a more convenient binding):

>>> __builtins__.sum([721832253, 721832254, 721832254])
2165496761L

score 4 · Accepted Answer · answered Oct 03 '12 at 20:49

The reason why you get this invalid value is that you're using np.sum on a int32. Nothing prevents you from not using a np.int32 but a np.int64 or np.int128 dtype to represent your data. You could for example just use

x.view(np.int64).sum()

On a side note, please make sure that you never use from numpy import *. It's a terrible practice and a habit you must get rid of as soon as possible. When you use the from ... import *, you might be overwriting some Python built-ins which makes it very difficult to debug. Typical example, your overwriting of functions like sum or max...

I assume the issue here might apply in other areas, such as reduce(mul,[1,2,3,4,5...] ) ? How might this be fixed in problems doing products on large lists? — Mason Gardner, Oct 28 '16 at 04:56

Martijn Pieters · Answer 3 · 2012-10-03T19:58:45.487

1

Python handles large numbers with arbitrary precision:

>>> sum([721832253, 721832254, 721832254])
2165496761

Just sum them up!

To make sure you don't use numpy.sum, try __builtins__.sum() instead.

edited Oct 03 '12 at 19:58

answered Oct 03 '12 at 19:50

Martijn Pieters

1,048,767
296
4,058
3,343

mine returns -2129470535. Windows 64 machine – notilas Oct 03 '12 at 19:51
1

@notilas: Then your `x` is not what you expected it to be. Python integer support is platform independent. – Martijn Pieters Oct 03 '12 at 19:52
1

I have copied and pasted your text to my machine. "sum([721832253, 721832254, 721832254])", it returns a negative value. – notilas Oct 03 '12 at 19:54

Why does built-in sum behave wrongly after "from numpy import *"?

3 Answers3

Linked