Numpy overflow with np.divide(). RuntimeWarning: overflow encountered in ushort_scalars

Question

I am running the following code on 2 images:

ndvi = np.divide(img8 - img4, img8+img4)

invalid = (ndvi > 1).any()
if invalid:
    print("Stopping Execution")
    print(ndvi)

img8 and img4 are 2 images and have all positive values. ndvi is (img8-img4)/(img8+img4)

Hence, by definition, all elements of ndvi should be between -1 and 1. But I am getting some values>1

The dtypes of all the variables in this context are 'uint16'

When I check the index of the invalid values, and ran the individual code:

temp = (img8[88][118]-img4[88][118])/(img8[88][118]+img4[88][118])

I got the following warning:

<stdin>:1: RuntimeWarning: overflow encountered in ushort_scalars

The values are: img8[88][118] = 1462 img4[88][118] = 1652

The values themself are not large to result in an overflow, but when array sizes become large overflow happens.

Perhaps the values in `img8` and/or `img4` are not what you think they are. Print them — DeepSpace, Jul 14 '21 at 14:18
Also, just so you are aware, if a value in `img4` is greater than the corresponding value in `img8` then you *will* get negative values — DeepSpace, Jul 14 '21 at 14:19
@DeepSpace Yes, I understand. I meant it will be between -1 and 1. — Raghav Arora, Jul 14 '21 at 14:21
So, 1 particular invalid value was occuring at index (88,118). I printed the corresponding img values: img4[88][118]=1652 img8[88][118]=1462 So, ndvi[88][118] should be -0.06101 But it's reporting it to be 20.9845 — Raghav Arora, Jul 14 '21 at 14:22
Please print dtypes of everything (img8, img4, img4+img8, img8-img4) — Gulzar, Jul 14 '21 at 14:24
It **might** be due to the way `np.divide` works with nested/multi-dimensional arrays. Post a [mcve] — DeepSpace, Jul 14 '21 at 14:24
I think I found the error, when I do: `(img8[88][118]-img4[88][118])/(img8[88][118]+img4[88][118])` I get a warning: "RuntimeWarning: overflow encountered in ushort_scalars" But I fail to understand why this is an overflow — Raghav Arora, Jul 14 '21 at 14:26

Gulzar · Accepted Answer · 2021-07-14T19:30:15.800

When subtracting a large number from a small one and both are unitXX, you get an overflow (really, an underflow), and the result is the modulo-XX of the negative number, which is a large number.
This is because uint can't represent a negative number, and instead holds a large positive.

This modulo is actually adding maxint (==65535) to the negative

In this case, for the index you specified,

img8[88][118]-img4[88][118] == 1462 - 1652 == -190 == 65535 - 190 = 65345

Divide that by 1462 + 1652 == 3114 and get 20.984264611432241490044958253051

Solution:

Convert the dtypes to float before dividing, and usually work with floats for images, rather than uint.

Floats don't (normally) have the underflow issue for simple subtraction, as they are built to also represent negatives.

Also would work is using int rather than uint but stick to floats for your own sake.

Numpy overflow with np.divide(). RuntimeWarning: overflow encountered in ushort_scalars

1 Answers1