1

I am trying to speed up a command line Python script I wrote. In the process of profiling, I noticed that the time to load the various libraries I am importing at the top is very variable. The modules I am currently importing are:

import os
import urllib2
import datetime
import dateutil.rrule as rr
import netCDF4
from pylab import *
import glob
import cookielib
import traceback
import base64
from scipy import interpolate
import dateutil.parser as dparser

And when I time just those imports from the command line, I get very variable results, from minimum of ~2.7 seconds ( which seems pretty long already just to load the above-mentioned modules) to ~8 seconds

$ time python -c "import mycode.py"

real    0m2.763s
user    0m0.239s
sys 0m0.104s

$ time python -c "import mycode.py"

real    0m8.852s
user    0m0.224s
sys 0m0.157s

So, 2 questions:

  • is there a way to reduce the time to perform those imports
  • if not, is there a way to ensure the time to import remains the lowest of the above numbers
Denis Barkats
  • 41
  • 1
  • 4
  • 1
    FYI -- "snippet" support is for JavaScript that can be run in a browser (see https://stackoverflow.blog/2014/09/16/introducing-runnable-javascript-css-and-html-code-snippets/ for an understanding of the context; it's basically an alternative to JSFiddle, and intended to be used only in situations where JSFiddle otherwise would be). Use the `{}` button to code-format code in other languages. – Charles Duffy May 25 '17 at 20:58
  • What's in `merra2Player`? `import merra2Player` may not just be running the imports shown. – user2357112 May 25 '17 at 21:00
  • 1
    ...with respect to the question, though -- the big question marks are pylab, scipy and netCDF4. Assuming this isn't an embedded system or otherwise a highly resource-constrained device, I'd almost place money that it's one of the three that's your primary culprit (since the others are builtins, and otherwise well-known). A good [mcve] would actually provide exact steps for someone else to reproduce the problem -- isolating the specific modules at fault. – Charles Duffy May 25 '17 at 21:00
  • 2
    Given how much bigger `real` is than `user` or `sys`, this may be due to other activity on your machine, or factors like what's in the disk cache. – user2357112 May 25 '17 at 21:02
  • Yup -- smells like block cache to me too. – Charles Duffy May 25 '17 at 21:03
  • Charles, thanks for the info. Ignore "merra2Player", it's just the name of the code which contains those imports. – Denis Barkats May 25 '17 at 21:03
  • @CharlesDuffy, a minimal and reproducible code would simply consist of putting the above imports in a file called mycode.py, and from the command line, running `time python -c "import mycode.py" `, wouldn't it ? – Denis Barkats May 25 '17 at 21:08
  • It's not "minimal" if it includes a bunch of modules that *don't* substantively increase time to load alongside the ones that do. Which is to say, once this is brought down to a MCVE it's likely to be a more specific question, like "why does scipy sometimes take so long to load?". – Charles Duffy May 25 '17 at 21:10
  • 1
    https://stackoverflow.com/questions/43989235/accurate-timing-for-imports-in-python/43995651?s=3|0.0000#43995651 – Jean-François Fabre May 25 '17 at 21:13
  • ok, thanks for the comments. I'll do some further tests and report here if I find something relevant. – Denis Barkats May 25 '17 at 21:19

0 Answers0