2

Im using GLPK with through Julia, and I need to optimize the same GLPK.Prob repeatedly. What changes between each optimization is that some combination of variables gets fixed to 0

simply put in pseudocode

lp = GLPK.Prob()
variables_to_block = [[1,2,3], [2,3], [6,7], [19,100,111]...]
for i in variables_to_block 
    block_vars(lp, i)
    simplex(lp)
    restore_vars(lp, i)
end

when I then run this, it looks like CPU1 acts as a scheduler, staying in the 9-11% range, and the load on CPU3 and CPU4 alternates between 0 and 100%, though never at the same time...the load on CPU2 stays at 0%

This can take a bit of time and I'd like to use all cores

There is however, a bit of a hassle to use the parallel features of Julia, particularly for lp-models because they involve pointers so (as far as I know) they cannot be copied easily between cores

Is there a way to either set the GLPK solver binary (or something) to automatically try to fully utilize all cores? either by compiling GLPK in such a way, or any other way

isebarn
  • 3,812
  • 5
  • 22
  • 38
  • Probably much better than following the multiprocessing-path: just update your problem and warm-start. Start [here](http://jump.readthedocs.io/en/latest/probmod.html). – sascha Oct 17 '16 at 11:05
  • I should have noted that Im using the GLPK module directly, so Im not using JuMP – isebarn Oct 17 '16 at 11:35
  • And it does not support warm-start? That would be sad. JuMP is very powerful and even by using MathProgBase alone some good stuff could be done. I'm very optimistic, that warm-starting would help much more than Mt here. But i might be wrong. (If this is academic work: use one of the commercial solvers which are multithreaded and free for use; in the academic setting; and in regards to open-source: i prefer cbc to glpk, but you already made a decision it seems) – sascha Oct 17 '16 at 11:45
  • Why are you using GLPK? and further, why are you using it directly, rather than via JuMP? – Frames Catherine White Oct 17 '16 at 15:55
  • @sascha are you suggesting that JuMP is faster than directly using the GLPK.jl interface? – isebarn Oct 17 '16 at 16:40
  • 1
    It probably is (possibly higher-quality of design; but that's just a hunch; the fastest way would be using MathProgBase), but that would be not my point. JuMP offers a nice way of warm-start capabilities if supported by the solver. You can easily switch the solver if needed and it's probably nicer to use in general. – sascha Oct 17 '16 at 16:41
  • @Oxinabox why am I using GLPK? because its a solver and because its free, and Im using it directly because Im looking to get as much speed as possible, and as far as I know, there is overhead involved with JuMP, my lp problems are simple, so simple that the entire problem is less than 20 lines of code – isebarn Oct 17 '16 at 16:42
  • @sascha I originally used JuMP but there was a ton of overhead involved, but I'll check out the link you provided earlier to make a quick comparison then I will comment again the results – isebarn Oct 17 '16 at 16:44
  • That's dependent on so much stuff and we don't know much about your problem. If you got everything ready in matrix-form, sure, use MathProgBase or some low-overhead lib. How big are the instances? How big is lib-formulation time compared to solving time and so on and on... Hard to help here. But the general consensus is: model-building is dominated by solving-time. – sascha Oct 17 '16 at 16:46
  • I understand. I've got chemical reactions as variables, chemical compounds as constraints. The sum of the production of the chemical compounds (sum of the constraints) must equal 0. The test model is e. coli, and my benchmark consists of minimizing/maximizing iteratively every single reaction to find theoretical bounds for reactions. This process takes roughly 3 times as long using JuMP. Model formulation takes 5 times longer using JuMP. – isebarn Oct 17 '16 at 16:55
  • Id like to add that Im not critizing JuMP, originally I just asked if there was a way to make GLPK automatically utilize all processors so that I woulnt have to explicitly write complicated parallel code in Julia that would be immensely difficult to test – isebarn Oct 17 '16 at 16:58
  • 2
    A simple device is to create say 4 batch files that each does 1/4th of the total number of scenarios and then run these in parallel. Here is a Windows [example](http://yetanothermathprogrammingconsultant.blogspot.com/2012/04/parallel-gams-jobs.html). Under Unix this even easier. – Erwin Kalvelagen Oct 17 '16 at 19:35
  • Just a systems biology perspective, you are doing a FVA. Since each LP is not dependent on the others, you can trivially paralellize them as suggested by @ErwinKalvelagen . Bear in mind that there are implementations that address this exactly https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-11-489 and even in your specific setup : https://opencobra.github.io/COBRA.jl/stable/cobratutorial.html#Flux-Variability-Analysis-(FVA)-1 – cladelpino Jan 25 '18 at 11:51
  • https://stackoverflow.com/users/4316542/cladelpino FVA is what I was doing. I was working on https://github.com/isebarn/CBM at the time – isebarn Jan 26 '18 at 11:19
  • https://stackoverflow.com/users/4316542/cladelpino if you're interested, the way I implemented this is located at https://github.com/isebarn/CBM/blob/master/src/core/multicore.jl – isebarn Jan 26 '18 at 11:21

1 Answers1

2

As far as I know, GLPK is not multithreaded. If you must have a multithreaded solver, then consider using a newer one such as Gurobi or MOSEK. They have free academic licenses.

If commercial solvers are anathema to your purposes, then perhaps try the Splitting Cone Solver. It is released for free under the MIT License.

Otherwise, the suggestions in the comments (e.g. batch proessing) may be your only recourse.

Kevin L. Keys
  • 997
  • 8
  • 21