8

I've been using tensorflow for quite some time. Recently my scripts run very slow (> 80 sec) as to earlier (< 1 sec). I narrowed the issue down to import tensorflow, which alone is taking all the time (all other libs and ops are running << 1 sec).

I might have a trace, but I don't know what to do with it: When I keyboard-interrupt (Strg+C) the execution during the 80 sec import, usually this is what comes up:

Traceback (most recent call last):
  File "/.../py_env/tf_unet/lib/python3.5/site.py", line 703, in <module>
    main()
  File "/.../py_env/tf_unet/lib/python3.5/site.py", line 694, in main
    execsitecustomize()
  File "/.../py_env/tf_unet/lib/python3.5/site.py", line 548, in execsitecustomize
    import sitecustomize
  File "/usr/lib/python3.5/sitecustomize.py", line 3, in <module>
    import apport_python_hook
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 954, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 896, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1139, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1113, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1225, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1264, in _fill_cache
KeyboardInterrupt

Does this mean, something with "filling the caches" (_fill_cache) is wrong? Anyone experience with this? Can I fix this somehow?

What I've tried so far:

I broke down import tensorflow to only modules I need (from tensorflow import train / python_io / compat), with no improvement.

I found other people complaining about long import tensorflow speed here, here and in the corresponding SO question, but in the range of < 10 sec and referring to specific modules (tf.contrib or tf.learn). So not much to learn from there. Also I am using tensorflow 1.4.0 which apparently fixed the problems described there.

Just for reference, I am using this little piece of code to determine the speed:

from timeit import default_timer as timer
print('import tensorflow')
start = timer()
import tensorflow
end = timer()
print('Elapsed time: ' + str(end - start))
Honeybear
  • 2,928
  • 2
  • 28
  • 47
  • In my case I had serious slowness due to the fact that I had my TF virtual environment on a network drive. Moving it to local helped, not sure if that applies in your case. – mikkola Mar 01 '18 at 15:48
  • @mikkola: I didn't consider this option since I'm working on a cluster with distributed storage anyways. However we do have different raids and it turns out that one is running at the edge. Moving the tensorflow installation to a different raid actually hugely improved performance (import now @ 5-15 sec). Thanks! You want to turn that to an answer I can accept? – Honeybear Mar 02 '18 at 15:25

1 Answers1

7

This is probably not the only reason that can cause this, but in my experience certainly plays a role. I had serious slowness in importing Tensorflow due to the fact that I had my TF virtual environment on a network drive. Moving the virtual environment to a local hard drive helped quite a bit in this respect.

You could try doing something similar that applies in your environment.

mikkola
  • 3,376
  • 1
  • 19
  • 41