How are variables shared between concurrent `session.run(...)` calls in tensorflow?

Question

If you make two concurrent calls to the same session, sess.run(...), how are variables concurrently accessed in tensorflow?

Will each call see a snapshot of the variables as of the moment run was called, consistent throughout the call? Or will they see dynamic updates to the variables and only guarantee atomic updates to each variable?

I'm considering running test set evaluation on a separate CPU thread and want to verify that it's as trivial as running the inference op on a CPU device in parallel.

I'm having troubles figuring out exactly what guarantees are provided that make sessions "thread safe".

Answer 1

After doing some experimentation it appears that each call to sess.run(...) does indeed see a consistent point-in-time snapshot of the variables.

To test this I performed 2 big matrix multiply operations (taking about 10 sec each to complete), and updated a single, dependent, variable before, between, and after. In another thread I grabbed and printed that variable every 1/10th second to see if it picked up the change that occurred between operations while the first thread was still running. It did not, I only saw it's initial and final values. Therefore I conclude that variable changes are only visible outside of a specific call to sess.run(...) at the end of that run.