1

I'm debugging some simulations that do a lot of heavy linear algebra using numpy (on Ubuntu 20.04, python 3.7 via anaconda), and I'm getting nondeterministic results even when I set the same random seed. I think what's causing this is the parallel blas/lapack that does the linear algebra. I'm wondering of there's some flag I can set that makes numpy use single-threaded (and presumably deterministic) matrix operations.

I haven't looked deep enough to see where the nondeterminism arises, but the operations I'm using include dot, eigh, qr, solve, and possibly others.

I'd prefer to not recompile anything or install an alternate lapack/blas or anything like that.

mrip
  • 14,913
  • 4
  • 40
  • 58
  • [Python: How do you stop numpy from multithreading](https://stackoverflow.com/questions/17053671/python-how-do-you-stop-numpy-from-multithreading) – DarrylG Jul 15 '20 at 19:38
  • What exactly `nondeterminism` in your use case? Can you provide an example? – Quang Hoang Jul 15 '20 at 19:39
  • The parallelism in the blas/lapack occurs at a low level (called by the `dot`, `eigh` etc. I don't off hand see how that would affect `random` calls done in python code. It's not multithreading or processing your python code. `blas` level parallelism should be deterministic, true `parallel` that doesn't affect the results. – hpaulj Jul 15 '20 at 19:44
  • @QuangHoang nondeterminism means i run the same code twice and get different results. I don't have a reproducible example, because I haven't narrowed it down to where the nondeterminism starts. Basically I generate some random data and do a lot of matrix operations. – mrip Jul 15 '20 at 19:51
  • @hpaulj I'm not sure of the blas parallelism is deterministic. E.g. if the order in which results are accumulated depends on the way OS schedules threads then it can be non-deterministic. No, it wouldn't affect random, in fact random has nothing to do with it so probably I shouldn't have mentioned it. – mrip Jul 15 '20 at 19:54
  • I'm actually asking how **different** are different results that you getting. Did you set proper seed when you *generate some random data*? Are the output differ by large or small amount? In general, it's really a bad idea running single-threaded matrix operations. – Quang Hoang Jul 15 '20 at 19:55
  • I set a proper seed yes. The results aren't very different, but for debugging purposes I want them to be identical with the same random seed. I only want to use single-threading in debugging mode, in production of course I will use all the cores available. – mrip Jul 15 '20 at 19:59
  • When testing float results we usually recommend `np.allclose` rather than any sort of equality ('=='). Accumulation order differences should be within its tolerances. – hpaulj Jul 15 '20 at 20:29

0 Answers0