0

Does anybody know if I can safely compile multiple extensions concurrently with threading?

I realise this might not speed things up (although compilers are run in subprocesses, so maybe!), but I'm in a situation where GUI actions can start a simulation which may involve a compilation step, so I'd like to know if I need to prevent multiple simulations from compiling at the same time, or if this is fine

Michael Clerx
  • 2,928
  • 2
  • 33
  • 47

1 Answers1

1

Not only they're not thread-safe — they're not process-safe: you cannot call setup() multiple times in one process. See an example of errors from this.

To work around the limitation you have to run python setup.py or pip in a subprocess and then the question of thread-safety no longer applies.

phd
  • 82,685
  • 13
  • 120
  • 165
  • Thanks! That example looks like it's compiling - and installing - the same code twice, in the same location. That would cause errors you wouldn't get when compiling two separate packages at the same time (and with different paths) – Michael Clerx Mar 22 '21 at 17:16
  • @MichaelClerx There are two different calls to `setup()` so I'm sure you cannot do it in one process. – phd Mar 22 '21 at 18:13
  • Why are you sure you can't call `setup()` from two different threads? – Michael Clerx Mar 22 '21 at 19:47
  • Because it seems `distutils`/`setuptools` have a lot of global states. If one cannot call them 2 times from one thread I'm sure calling them from different threads would be even worse. – phd Mar 22 '21 at 19:55
  • Thanks! I also assumed that there'd be some issues, but I didn't see any mention in the docs or code about this, and noticed that compilation does happen in subprocesses already, so was wondering if perhaps it would be thread-safe & if someone in-the-know could tell me. The example you quoted doesn't convince me it's not thread-safe, as it's a much more obvious error (two eggs in the same path) – Michael Clerx Mar 23 '21 at 09:19