Fitting a sigmoid curve (Python)

Question

I am trying to fit a sigmoid curve and a 3rd-degree polynomial to my data (cost vs revenue) and then find the point of inflection/diminishing return.

This is the code I have so far, the fit is not great. Any advice would be very helpful, thank you!

import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit

def sigmoid(x, a, b):
    y = 1 / (1 + np.exp(-b*(x-a)))
    return y

xdata = [ 404.91,  731.89,  804.23,    0.  ,  954.72,  954.72,  954.72, 744.54,  744.54,  498.5 ,  355.03,
         359.61,    0.  ,    0.  , 0.  ,  753.77, 1116.02,  557.07,  589.06,  761.86,  722.97, 162.69, 
         354.47,  474.  ,  306.83,  538.57,  134.26,  134.26, 134.26,  134.26,  134.26,  652.29, 1296.26,
         547.78,  845.22, 872.62,  881.59,  556.23,  500.2 ,  569.97,  679.46,  679.46, 623.08,  628.33,
         754.88, 2014.12, 1870.43, 1444.69,  826.05, 1071.03,  816.74]
ydata = [ 6986.97, 36591.27, 23702.95,  6380.01, 26873.68, 19398.27,24693.5 , 18435.52, 19066.1 ,  8534.14,  8534.14,  8534.14,
          2032.07,   567.26,  7544.64, 21051.07, 21051.07, 18592.84,18592.84, 18592.84, 19566.14,  4787.51,  7269.55,
         11596.66, 9083.43, 13260.51,  6280.95,  4112.17,  6004.46,  7613.15, 6436.83, 10726.22, 20430.67,  8265.88,
         15344.32, 30246.91,29928.96, 12215.02,  7776.27,  9714.94, 16642.3 , 29493.06,15496.04, 15496.04, 15496.04,
         33397.61, 33397.61, 33397.61, 22525.93, 22525.93, 48941.98]

#fit 3rd order polynomial
p = np.poly1d(np.polyfit(x, y, 3))
second_deriv = p.deriv(2)
inflection = -second_deriv[0]/second_deriv[1]
print("polynomial inflection point:", inflection)

#fitting a sigmoid curve
popt, pcov = curve_fit(sigmoid, xdata, ydata,  method='dogbox', p0=[1000, 0.6])
estimated_k, estimated_x0 = popt
print("sigmoid inflection point:", estimated_x0)

x = np.linspace(0, int(max(xdata)), len(ydata))
y = sigmoid(x, *popt)*max(ydata)

t = np.linspace(0, int(np.max(xdata)), int(np.max(xdata)))
plt.plot(xdata, ydata, 'o', label='data')
plt.plot(p(t), 'b-', label="polynomial")
plt.plot(x,y, label='sigmoid')
plt.legend(loc='best')
plt.show()

score 1 · Answer 1 · answered Dec 30 '20 at 16:00

The data is highly scattered. It is doudtful that consistent results be obtained depending on the chosen criteria of fitting (least mean square errors, or least mean square relative errors, or least mean absolute errors, or etc.). Also with usual softwares using iterative methods, guessing convenient initial values of parameters might be uncertain.

So, I am not standing here to answer to your question, but to compare your results with whose from another very different and unusual method of fitting. This method is not iterative and doesn't rerquires initial guess of parameters. For theory see pp.37-38 in https://fr.scribd.com/doc/14674814/Regressions-et-equations-integrales

asdf101 · Answer 2 · 2020-12-10T01:52:51.810

I couldn't really see a question in your post, so I interpreted this as "how do I get a better fit?". Please edit your question if this isn't the case.

There are three problems as far as I can see:

The first one is that a sigmoid is always between 0 and 1, so it will have a hard time fitting with those very high values (consider adding an extra argument to your sigmoid function to multiply the result with);

The second is that your p0 for 'b' is way too high so curve_fit may get stuck in some local optimum (using your data I found 0.003-ish to be a good number);

And finally, with those x-values, you get extremely small exponents in your sigmoid function (smaller than exp(-1000)) which will likely cause problems at some point. Try making those x-values smaller, or use something like mpmath.

all ok except for the last. `b` should be such that `b*x` will have reasonable values, i.e. the argument of the exponential is likely to be ok. — mikuszefski, Dec 14 '20 at 06:24

Fitting a sigmoid curve (Python)

2 Answers2

Linked