1

I'm seeing an issue where the execution of my python script can sometimes take longer to import libraries. This happens if I have not executed the script recently or if I run the script from a different server. After the first delay, the import times become much faster. I was wondering what is causing this slow import and if there is any way to prevent it?

import time
s_time = time.time()
import sys,re,os,logging,signal
from argparse  import ArgumentParser
print('Internal Import Time: {}'.format(time.time() - s_time))
s_time = time.time()
from backtrace import Backtrace,BacktraceSet
from report    import Report
from core      import Core
from burtapi   import BurtAPI
print('External Import Time: {}'.format(time.time() - s_time))

In this example backtrace,report,core, and burtapi are libraries I created.

[name@server1 tool]$ ./tool --python
Internal Import Time: 2.8281359672546387
External Import Time: 13.053943157196045
Enter/Paste your content. Ctrl-D to save it.
^CYou pressed Ctrl+C!
[name@server1 tool]$ ./tool --python
Internal Import Time: 0.12279081344604492
External Import Time: 0.6948020458221436
Enter/Paste your content. Ctrl-D to save it.
^CYou pressed Ctrl+C!

ssh to different server (with same storage mount)

[name@server2 tool]$ ./tool --python
Internal Import Time: 3.0217390060424805
External Import Time: 13.151482105255127
Enter/Paste your content. Ctrl-D to save it.
^CYou pressed Ctrl+C!
[name@server2 tool]$

./tool is a bash script that calls python3 /path/to/script.py

Takkun
  • 6,131
  • 16
  • 52
  • 69
  • It's probably the time needed by python to generate byte code (compile some libraries) before the first run. It is also what can be seen when you delete .pyc files. More about that here: https://stackoverflow.com/questions/2998215/if-python-is-interpreted-what-are-pyc-files Try deleting the .pyc files in the folders where you put the libraries you created and see if the same thing happens. – ma3oun Jun 08 '17 at 12:29
  • @ma3oun How does Python determine when it needs to recompile? The byte code is in __pycache__, but if I wait several hours or change servers it triggers another compile even though the byte code already exists. – Takkun Jun 08 '17 at 12:35

1 Answers1

1

There are more factors here but there are three major ones:

First - importing a module requires finding the modules on your system, which requires reading data from your disk.

Second - with that, if there are any changes done inside those modules, the interpreter would need to bite-compile them.

Last, but not least, when the interpreter starts importing them, the modules themselves might contain instructions/code that needs to be executed which could be performing a lot of operations. Say imagine one of those modules connects to a database or has to filter through a large hashtable...

Edit: I should point out that those are assumptions and it's hard to say what exactly is happening in your case but in most cases this is what's happening.

Alexander Ejbekov
  • 5,594
  • 1
  • 26
  • 26