1

I call cvxopt.glpk.ilp in Python 3.6.6, cvxopt==1.2.3 for a boolean optimization problem with about 500k boolean variables. It is solved in 1.5 hours, but it seems to run on just one core! How can I make it run on all or a specific set of cores?

The server with Linux Ubuntu x86_64 has 16 or 32 physical cores. My process affinity is 64 cores (I assume due to hyperthreading).

> grep ^cpu\\scores /proc/cpuinfo | uniq
16
> grep -c ^processor /proc/cpuinfo
64
> taskset -cp <PID>
pid <PID> current affinity list:  0-63

However top shows only 100% CPU for my process, and htop shows that only one core is 100% busy (some others are slightly loaded presumably by other users).

I set OMP_NUM_THREADS=32 and started my program again, but still one core. It's a bit difficult to restart the server itself. I don't have root access to the server.

I installed cvxopt from a company's internal repo which should be a mirror of PyPI. The following libs are installed in /usr/lib: liblapack, liblapack_atlas, libopenblas, libblas, libcblas, libatlas.

sascha
  • 32,238
  • 6
  • 68
  • 110
Jantar88
  • 137
  • 8

1 Answers1

4

Here some SO-user writes, that GLPK is not multithreaded. This is the solver used by default as cvxopt has no own MIP-solver.

As cvxopt only supports GLPK as open-source mixed-integer programming solver, you are out of luck.

Alternatively you can use CoinOR's Cbc, which is usually a much better solver than GLPK while still being open-source. This one also can be compiled with parallelization. See some benchmarks which also indicate that GLPK is really without parallel support.

But as there is no support in cvxopt, you will need some alternative access-point:

Those:

  • have very different modelling-styles (from completely low-level: cylp to very high-level: cvxpy)
  • i'm not sure if all those builds are compiled with enable-parallel (which is needed when compiling Cbx)

Furthermore: don't expect too much gain from multithreading. It's usually way worse than linear speedup (as for all combinatorial-optimization problems which are not based on brute-force).

(Imho the GIL does not matter as all those are C-extensions where the GIL is not in the way)

sascha
  • 32,238
  • 6
  • 68
  • 110