Capture timeout exception with Mosek + Cvxpy

Question

We are solving our large scale MI optimization problems with Cvxpy and Mosek.

Often times it happens that Mosek consumes higher runtime then our stipulated timeout of two hours.

Is there a way to systematically capture those timeout exception?

Minimum reproducible example:

import cvxpy as cp
import numpy as np
import mosek

m = 15
n = 10
np.random.seed(1)
s0 = np.random.randn(m)
lamb0 = np.maximum(-s0, 0)
s0 = np.maximum(s0, 0)
x0 = np.random.randn(n)
A = np.random.randn(m, n)
b = A @ x0 + s0
c = -A.T @ lamb0

# Define and solve the CVXPY problem.
x = cp.Variable(n)
prob = cp.Problem(cp.Minimize(c.T@x),
                 [A @ x <= b])
# try:
prob.solve(cp.MOSEK, mosek_params={mosek.dparam.optimizer_max_time: 0.01})   # set verbose=True (to see actual error in solver logs)
# except Timeout exception
#     print('Timeout occured')

print(prob.value)

def execute_other_important_stuff():
    print("Hello world")

execute_other_important_stuff()  # Not executed currently

score 1 · Answer 1 · answered Sep 08 '21 at 16:59

1

I am not sure if this is a Cvxpy issue.

However, a general comment is that Mosek cannot check the time limit continuously so it will most likely go over time.

For instance for a SDP it has to compute eigenvalues of a potentially large matrix and the time limit cannot be checked before it is completed.

answered Sep 08 '21 at 16:59

ErlingMOSEK

391
1
7

Sorry for the confusion. By the sentence: **"Often times it happens that Mosek consumes higher runtime"** - we just meant that the optimization algorithm isn't completed in the stipulated timeout (and Mosek correctly identifies - **_Maximum amount of time 7200 is reached_**) – pqrz Sep 09 '21 at 02:11
The intent is just to systematically capture that exception (may be _try-except_ or something), i.e. not letting the program terminate by error out. – pqrz Sep 09 '21 at 02:15

score 1 · Answer 2 · answered Sep 08 '21 at 17:47

1

I have no other insights (other than what ErlingMOSEK suggested) on why it happened, but you can use signals library to force the .solve method to stop.

This is an independent way to make sure you can stop the function after whenever time you wise. See example in Timeout a function call

answered Sep 08 '21 at 17:47

Roim

2,986
2
10
25

Thanks @Roim! This would be the perfect approach, just that: `Total Function Time = Queue Wait Time + Actual optimization Time` and we want to apply timeout only one component of that i.e. `Actual optimization Time` – pqrz Sep 09 '21 at 02:20

score 1 · Accepted Answer · answered Sep 09 '21 at 06:18

1

When MOSEK terminates because of a timeout there will never be any exception - you set a timeout, so terminating at that point is a normal, not an abnormal, situation. I am not sure what "Error" you are referring to.

If you mean something like

Cannot unpack invalid solution: Solution(status=UNKNOWN

then it is not related to the timeout itself, but to the fact that there is no solution available (yet, so the only available "solution" has UNKNOWN status), and CVXPY handles this by throwing an exception. So the question is, has any solution at all to your problem been found after those 2 hours? If yes, it should be returned without problem. If not, you might see the above and I would guess the only way is to catch the ValueError CVXPY happens to throw.

If you were using the native Mosek interface then you could find out why it terminated from various response codes, but I CVXPY does not propagate them.

answered Sep 09 '21 at 06:18

Michal Adamaszek

898
5
7

Hi Michal, We are referring to the error which cvxpy throws on timeout: i.e. **cvxpy.error.SolverError: Solver 'MOSEK' failed. Try ....** (i.e. when we run the above _Minimum reproducible example_). – pqrz Sep 09 '21 at 09:18
The only problem with capturing generic Exceptions like `ValueError` or `cvxpy.error.SolverError` that they are generic and doesn't informs the actual specific reason for termination. – pqrz Sep 09 '21 at 09:18
If the bottleneck is - Cvxpy not having support for capturing those timeout interface, then we can also create a custom Cvxpy that does that? – pqrz Sep 09 '21 at 09:27
CVXPY does not propagate the reasons. CVXPY is not even aware you set some time limit via a Mosek parameter. If there is no solution to report back then SolverError seems like a good enough message from CVXPY perspective. If there was any solution to report CVXPY should have done that. What does the log say, does MOSEK really find no solution within the time limit? – Michal Adamaszek Sep 09 '21 at 09:27
Maybe ``problem.solver_stats.solve_time `` is still being set and you can check it in the exception handler. – Michal Adamaszek Sep 09 '21 at 09:30
Yes, Mosek is unable to find a solution (i.e. the B&B algorithm was running till last with `nan` in `BEST_INT_OBJ` column). Mosek's solver log says: **Maximum amount of time 7200 is reached**. – pqrz Sep 09 '21 at 09:34
Is there a way we can hack into Cvxpy (i.e. Custom Cvxpy) and add this timeout support from [Mosek's timeout response code](https://docs.mosek.com/9.2/opt-server/response-codes.html#mosek.rescode.trm_max_time) ? – pqrz Sep 09 '21 at 09:36
Apparently, they aren't populating `problem.solver_stats.solve_time` parameter. – pqrz Sep 09 '21 at 09:58
In https://github.com/cvxpy/cvxpy/blob/f722fadcef0d1c877eb63438c27d50c5077f2ed3/cvxpy/reductions/solvers/conic_solvers/mosek_conif.py#L235 you would have to get the termination code as in https://docs.mosek.com/latest/pythonapi/optimizer-task.html#mosek.task.optimize and then do what you want with it. – Michal Adamaszek Sep 09 '21 at 10:02
If you would write to Mosek support it will be easier to discuss there and also wouldn't mind looking at that big MIP to see if anything can be improved. – Michal Adamaszek Sep 09 '21 at 10:05

Capture timeout exception with Mosek + Cvxpy

3 Answers3