logistic regression not working in python

Question

i am trying to implement logistic regression in python using scipy.optimize and getting a error that i described below

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import scipy.optimize as sci

data=pd.read_csv("data.txt")
X=data.iloc[:,:-1]
y=data.iloc[:,-1]
admitted=data.loc[y==1]
not_admitted=data.loc[y==0]

plotting the data
plt.scatter(admitted.iloc[:,0],admitted.iloc[:,1],color='red',marker='X')
plt.scatter(not_admitted.iloc[:,0],not_admitted.iloc[:,1],color='green',marker='o')
plt.show()

X=np.c_[np.ones((X.shape[0],1)),X]
y=y[:,np.newaxis]
theta=np.zeros((X.shape[1],1))

def sigmoid(x):
    return 1/1+np.exp(-x)

def input(theta,x):
    return np.dot(x,theta)

def probablity(theta,x):
    return sigmoid(input(theta,x))

def cost_func(self,theta,x,y):
    m=x.shape[0]
    cost=-(1/m)*sum(y*np.log(probablity(theta,x))+(1-y)*np.log(1-probablity(theta,x)))
    return cost

def gradient(theta,x,y):
    m=x.shape[0]
    grad=(1/m)*np.dot(x.T,probablity(theta,x)-y)

def fit(self, x, y, theta):
    opt_weights=sci.fmin_tnc(func=cost_func,x0=theta,fprime=gradient,args=(x,y.flatten()))
    return opt_weights[0]

parameters = fit(X, y, theta)

print('The value of parameters are '+str(parameters))

i am getting the error that says

  RuntimeWarning: invalid value encountered in log
  total_cost = -(1 / m) * np.sum(y * np.log(probability(theta, x)) + (1 - y) * np.log(1 - probability(theta, x)))
  NIT   NF   F                       GTG
    0    1                    NAN   1.52331587E+04
tnc: fscale = 0.00810224
    0   66                    NAN   1.52331587E+04
tnc: Linear search failed

i do know that log can't be taken for negative values but i never got this error in octave can someone help in this

Remove `self` from function definition. You have a regular function not a method. — ayhan, Dec 03 '19 at 20:11
Your `fit` function takes 4 parameters: `self, x, y, theta` but you've only supplied three: `X, y, theta`. So your function call is equivalent to `fit(self=X, x=y, y=theta)` — G. Anderson, Dec 03 '19 at 20:13
thank you this solved the problem but after that i got another one, edited the problem in the question — Aakash Kaushik, Dec 03 '19 at 20:38

score 0 · Answer 1 · answered Dec 06 '19 at 20:07

That happens because you defined your sigmoid without any boundaries, for large. numbers or small number you will get +inf and -inf which lead to the problem you have. It can be referred to as a Quantization problem, or an estimation of sigmoid as well.

As it has been mentioned here , you can modify your sigmoid function like this ( I just copy that solution here), and your issue will be resolved:

def sigmoid(x):
    "Numerically-stable sigmoid function."
    if x >= 0:
        z = np.exp(-x)
        return 1 / (1 + z)
    else:
        z = np.exp(x)
        return z / (1 + z)

logistic regression not working in python

1 Answers1