Scipy.optimize.minimize is not giving the minimum value even though it sees that value

Question

I am using scipy.optimize.minimize to find optimal parameters for my objective function.

My code :


import numpy as np
from scipy.optimize import minimize
from scipy.optimize import Bounds

bounds = Bounds([26,26,8,6,400,100,0,25,2],[36,38,28,28,1800,800,100,50,7])

energy_history = []
x_values = []

def objective(x):
    x_trail = x.reshape(1,-1)
    x_trail = sc_X.transform(x_trail)
    y_trail = regressorSVR.predict(x_trail)
    y_trail = y_trail.reshape(1,-1)
    y_trail = sc_Y.inverse_transform(y_trail)
    return y_trail[0]


def callback(x,y):
    fobj = objective(x)
    energy_history.append(fobj)
    x_values.append(x)

x0 = np.array([26,28,15,7,400,377,40,43,4.3])
res = minimize(objective, x0, method='trust-constr',
               options={'verbose': 1}, bounds=bounds,callback=callback)

optimal_values = res.x
energy = res.fun

With the initial values given, the minimized value(res.fun) that I get is -7.1. I am creating a list(energy_history) to see how it is reaching this value. I see some values which are less than -7.1 in that list, but still, why is -7.1 being returned as the minimal value.

There are multiple times where objective function reached a value of -21, but why is still -7 being returned as a minimum ?

Please provide [a Minimal, Complete, and Verifiable example](https://stackoverflow.com/help/mcve). This code does not even compile. By the way, why are you using `trust-constr` when you do not have any constraints? — SuperKogito, May 13 '19 at 10:59
Hi @SuperKogito , the objective function is an SVR regression model. I cannot show post the entire code with data and training steps. It will be very lengthy. That's why I have shown the results. I have boundaries for my input parameters, that's why I am using a constrained optimization technique. — chink, May 13 '19 at 11:42
I do not think that's the same thing. Imo `Bounds` are different from `Constraints` and that is why in `Scipy` you can define them separately. I suggest trying with a different `method` and seeing if there is a change in the behavior. — SuperKogito, May 13 '19 at 12:08
yes, `Bounds` are different from `Constraints` . https://stackoverflow.com/questions/19244527/scipy-optimize-how-to-restrict-argument-values Followed the answers of this question. — chink, May 13 '19 at 13:16
I think that the function is not convex so it is just luck if the local minimum is also global, in such case you may try multiple random initializations and hope... :) — Jirka, May 29 '19 at 22:53
but if it has seen a lesser value earlier than the value it's giving out as result, isn't the lesser value seen earlier should be returned as minimum? — chink, May 30 '19 at 05:52
Did you check the coordinates of those values? Could it be that they violate the bounds, and are therefore not included in the final output? — Dion, Jun 03 '19 at 17:49
they are objects of sklearn.preprocessing.StandardScaler class — chink, Jun 04 '19 at 11:49
Yes but, what they contain ? Could you show us ? Because we can't run the code — Hippolyte BRINGER, Jun 04 '19 at 12:01
Have you checked `x_values`? I would guess, that they are slighly violating the bounds. Hence, the objective function returns a better value than -7.1. However, it is not allowed and -7.1 remains the best possible allowed value. — LaurinO, Jun 04 '19 at 15:05
yes, x_values are going out of bounds, when -21 is reached. That's the reason -7.1 remains the best possible value. Thanks all. — chink, Jun 11 '19 at 05:53

score 3 · Accepted Answer · answered Jun 05 '19 at 14:13

If we take a look at the scipy.optimization documentation we can see that scipy.optimize.minimize is listed under local optimization. The main problem is that your problem is non-convex and thus scipy.optimize.minimize cannot guarantee the proper convergence. As it's also very much non-differentiable, many algorithms won't be suited at all.

scipy.optimize does provide some global optimization algorithms though that can be found on the documentation page under global optimization, namely basinhopping, brute, and differential_evolution. Look at this answer for some short explanation.

Basically you can try brute first, just to see any systematic problems. It's basically a brute force solution and will be slow, but find your minimum. The more sophisticated method would be using differential_evolution. Since your function isn't really smooth, basinhopping might not work, but it's still worth a shot and would probably converge the fastest.

Scipy.optimize.minimize is not giving the minimum value even though it sees that value

1 Answers1

Linked