0

A set of points with coordinates x and y look like this. I want to construct a curve in the region below y = 0 of the form - a - np.exp(-(x - b)/c), where parameters a, b and c are found by the condition that 90% of the points below y = 0 are enclosed by this line and the function in question.

I've written the following code to do this, but the minimize function gives the initial guess as a result and I don't know what I'm missing.

from scipy.optimize import minimize
import numpy as np

def enclosed_points(params):
    a, b, c = params
    den = (y < 0).sum()                   # Calculate the number of points with y coordinate below y0
    func = - a - np.exp(-(x - b)/c)       # Calculate the value of the function for each x
    num = ((y < 0) & (y > func)).sum()    # Calculate the number of points with y coordinate
                                          # below y0 and above the function
    return np.abs(num/den - 0.9)          # Return the absolute value of the difference between
                                          # the ratio of num and den and the target number (0.9)

initial_guess = [0.1, 0.2, 1]             # Dummy initial guess
result = minimize(enclosed_points, initial_guess)

Edit. Here I have uploaded a random sample of the whole data in npy format.

WinterPanda
  • 3
  • 2
  • Could you provide your sample data? It's easier to catch the issue with sample data. – hajbabaeim Aug 06 '20 at 14:25
  • @mrhajbabaei You can download a random sample of the data from the link in the last line of the post. Thank you! It is not the full set of data because of high file size, but I can upload the full data if you need it. – WinterPanda Aug 06 '20 at 14:56

1 Answers1

0

Well, I tried some different methods and change some parts of your code:

def func(x, a, b, c):
    return - a - np.exp(-(x - b)/c)

def enclosed_points(params):
    a, b, c = params
    loss1 = y[np.argwhere( y > 0)]
    loss2 = loss1[np.argwhere( loss1 < func(x[np.argwhere( y > 0)], a, b, c) )]
    loss = ((loss2.sum() / len(y)) - 0.9)**2
    return loss

initial_guess = [-0.1, 0.2, 1]     
result = minimize(enclosed_points, initial_guess, method='SLSQP', options={'eps': 1e-2})

The loss1 and loss2 do the same work as your loss function, but I change abs to the power of 2 (because it's more common in my opinion) (also added "method='SLSQP', options={'eps': 1e-2}" to your minimizer based on another post on StackOverflow; try to read them carefully and get familiar with their issues about this minimizer too). However, I think the main issue is that your problem is non-convex, and minimize function tries to find a local minimum. Please see this post for a comprehensive description. In the end, I would say that with [-0.1, 0.2, 1] initial point I could find a solution (try different negative value for a, and may you can find another solutions :) )

Good luck

hajbabaeim
  • 156
  • 2
  • 10
  • Thank you very much. Your solution is not what I was looking for strictly speaking, but after some minor modifications I managed to adapt it. Thank you again. – WinterPanda Aug 06 '20 at 20:51