Is Tensorflow 2.0 thread-safe?
More specifically, is calling fit/predict or other methods on the same model from different threads safe in Tensorflow 2.0 (using Keras API)?
I couldn't find a clear answer from the documentation or from looking online.
I saw this question from 2017 which says that Keras (although the question mentioned a Theano backend) is thread-safe, but you have to call private method model._make_predict_function() before you call predict() (which I believe has been deprecated). However, I read this blog post from 2019 which says that it's not thread-safe.
I also found this question from 2018 that says that Tensorflow (pre-Keras) is thread-safe but you have to make sure you use the default graph explicity (which I believe is irrelevant for Tensorflow 2.* because of eager execution). When I looked up thread-safety in eager execution I saw this article in the documents that does mention thread-safety of eager execution, but it's in relation to Java.
And to make things more confusing, I saw an A3C implementation in Github with Keras from this year (2020) that was using locks before training the shared policy/value networks, hinting that Keras is not thread-safe and you have to acquire a lock before training a shared model. However, it looks to me like the implementation is flawed because each worker was creating and using it's own unique lock which defeats the purpose of having a lock. My conclusion is that either his code was running successfully regardless of the "lock" because Keras is thread-safe, or that he has a bug.
I did my own final test where I ran two threads fitting the same model to different outputs (for the same constant input) and tried calling predict during the training and it seems to be working, but I'm asking this question because I want to make sure. Are there any cases where Tensorflow 2.0/Keras are not thread safe?
