64

I am using numpy.log10 to calculate the log of an array of probability values. There are some zeros in the array, and I am trying to get around it using

result = numpy.where(prob > 0.0000000001, numpy.log10(prob), -10)

However, RuntimeWarning: divide by zero encountered in log10 still appeared and I am sure it is this line caused the warning.

Although my problem is solved, I am confused why this warning appeared again and again?

GeauxEric
  • 2,814
  • 6
  • 26
  • 33
  • 11
    `numpy.log10(prob)` is being evaluated before the `where` is being evaluated. – Bach Feb 06 '14 at 17:51
  • Note that you can use `numpy.seterr` eventually in combinations with `catch_warnings` to change the behaviour of numpy's division by zero. See [this](http://stackoverflow.com/q/15933741/510937) related question. – Bakuriu Feb 06 '14 at 18:04

6 Answers6

40

You can turn it off with seterr

numpy.seterr(divide = 'ignore')

and back on with

numpy.seterr(divide = 'warn')
john k
  • 6,268
  • 4
  • 55
  • 59
31

numpy.log10(prob) calculates the base 10 logarithm for all elements of prob, even the ones that aren't selected by the where. If you want, you can fill the zeros of prob with 10**-10 or some dummy value before taking the logarithm to get rid of the problem. (Make sure you don't compute prob > 0.0000000001 with dummy values, though.)

user2357112
  • 260,549
  • 28
  • 431
  • 505
  • 1
    One way to acomplish this is by using numpy.where twice: `prob_tmp = numpy.where(prob > 1.0e-10, prob, 1.0e-10)`, `result = numpy.where(prob > 1.0e-10, numpy.log10(prob_tmp), -10)` – feli_x Feb 14 '21 at 17:01
25

Just use the where argument in np.log10

import numpy as np
np.random.seed(0)

prob = np.random.randint(5, size=4) /4
print(prob)

result = np.where(prob > 0.0000000001, prob, -10)
# print(result)
np.log10(result, out=result, where=result > 0)
print(result)

Output

[1.   0.   0.75 0.75]
[  0.         -10.          -0.12493874  -0.12493874]
Markus Dutschke
  • 9,341
  • 4
  • 63
  • 58
8

I solved this by finding the lowest non-zero number in the array and replacing all zeroes by a number lower than the lowest :p

Resulting in a code that would look like:

def replaceZeroes(data):
  min_nonzero = np.min(data[np.nonzero(data)])
  data[data == 0] = min_nonzero
  return data

 ...

prob = replaceZeroes(prob)
result = numpy.where(prob > 0.0000000001, numpy.log10(prob), -10)

Note that all numbers get a tiny fraction added to them.

Ramon Balthazar
  • 3,907
  • 3
  • 25
  • 34
2

Just specify where to calculate log10 as follows:

 result = np.log10(prob,where=prob>0)

Here is a demo: demo

Shannon
  • 323
  • 1
  • 11
  • This answer was reviewed in the [Low Quality Queue](https://stackoverflow.com/help/review-low-quality). Here are some guidelines for How do [I write a good answer?](https://stackoverflow.com/help/how-to-answer). Code only answers are **not considered good answers**, and are likely to be downvoted and/or deleted because they are less useful to a community of learners. Please [edit] your answer to include an explanation of how and why the code solves the problem, when it should be used, what its limitations are, and if possible a link to relevant documentation. – ljmc Aug 30 '22 at 10:49
1

This solution worked for me, use numpy.sterr to turn warnings off followed by where

numpy.seterr(divide = 'ignore')
df_train['feature_log'] = np.where(df_train['feature']>0, np.log(df_train['feature']), 0)
E.Zolduoarrati
  • 1,539
  • 2
  • 9
  • 9