8

I have a CLI application that requires sympy. The speed of the CLI application matters - it's used a lot in a user feedback loop.

However, simply doing import sympy takes a full second. This gets incredibly annoying in a tight feedback loop. Is there anyway to 'preload' or optimize a module when a script is run again without a change to the module?

orlp
  • 112,504
  • 36
  • 218
  • 315
  • 1
    It seems the `sympy` devs are aware of this issue. There's a [old bug report](https://github.com/sympy/sympy/issues/3396), and there's also several comments in [`sympy/__init__.py`](https://github.com/sympy/sympy/blob/master/sympy/__init__.py) mentioning slow imports. Maybe file an issue again? (After making sure you're using the most recent version) – Lukas Graf Sep 15 '15 at 19:34
  • Rewrite your CLI application so that you can run it only once, yet feed it multiple different inputs...? – twalberg Sep 15 '15 at 20:35
  • @twalberg At that point it's not a CLI application anymore. To be more specific, my application is a language interpreter. – orlp Sep 15 '15 at 20:36
  • Now that Python 3.7 with `importtime` is out, I've opened a new sympy issue about it (https://github.com/sympy/sympy/issues/14854). – Nico Schlömer Jul 01 '18 at 15:27

3 Answers3

2

Obviously sympy does a lot when being imported. It could be initialization of internal data structures or similar. You could call this a flaw in the design of the sympy library.

Your only choice in this case would be to avoid redoing this initialization.

I assume that you find this behavior annoying because you intend to do it often. I propose to avoid doing it often. A way to achieve this could be to create a server which is started just once, imports sympy upon its startup, and then offers a service (via interprocess communication) which allows you to do whatever you want to do with sympy.

If this could be an option for you, I could elaborate on how to do this.

Alfe
  • 56,346
  • 20
  • 107
  • 159
  • I feel like that would be over-engineering it, and rather fragile to make cross-platform. Is there no way to 'snapshot' an interpreter state (after importing), and load the entire interpreter state from disk? – orlp Sep 15 '15 at 19:39
  • None that I know of. On my several years old laptop, btw, I need only a fraction of a second for importing `sympy`. – Alfe Sep 15 '15 at 19:59
  • *"load the entire interpreter state from disk"* - [ASLR](https://en.wikipedia.org/wiki/Address_space_layout_randomization) is just one of several reasons why that won't work. – Lukas Graf Sep 15 '15 at 20:11
  • @Alfe I'm interested to find out why, and see if I can reproduce that in some way. Could I invite you to join http://chat.stackoverflow.com/rooms/89736/sympy-import-mystery ? – orlp Sep 15 '15 at 20:12
  • Does your old laptop have SSD? – Mark Ransom Sep 15 '15 at 20:28
  • No SSD, no :) Lenovo T61. – Alfe Sep 15 '15 at 22:06
  • Inspite of this being an old post, I would appreciate if you can throw some light on your idea. We use C# to execute Python scripts (just started to work with Python and have some experience on C# already) and yes, in every request import loads and takes around 12 seconds. How can we do it just once and use the memory for further requests? – Sandy May 18 '18 at 00:51
  • I assume your (slow) code uses some way of starting your Python script from C# for each computation. I intend to replace that by starting the Python program once and let it wait in an input loop. For each input it would then start one computation, present the results as usual, and then it would not terminate but return to the input loop again. You might want to open your own Q with a [mcve] describing in more detail your situation, presenting your code, etc. also allowing us to go into more detail on how to do that. A comment seems unfit. – Alfe May 18 '18 at 06:29
2

I took a look at what happens when you run import sympy, and it imports all of sympy.

https://github.com/sympy/sympy/blob/master/sympy/__init__.py

If you are only using certain parts of sympy, then only import those parts that you need.

It would be nice if you could do this:

import sympy.sets

But (as you point out) that imports sympy and then sets.

One solution is to write your own importer. You can do this with the help of the imp module.

import imp
sets = imp.load_module("sets", open("sympy/sets/__init__.py"), "sympy/sets/__init__.py", ('.py', 'U', 1))

But, even that may not optimize enough. Taking a look at sympy/sets/__init__.py I see that it does this:

from .sets import (Set, Interval, Union, EmptySet, FiniteSet, ProductSet,
    Intersection, imageset, Complement, SymmetricDifference)
from .fancysets import TransformationSet, ImageSet, Range, ComplexRegion
from .contains import Contains
from .conditionset import ConditionSet

Maybe you can import only the sets module from simpy sets namespace?

import imp
sets = imp.load_module("sets", open("sympy/sets/set.py") "sympy/sets/set.py", ('.py', 'U', 1))
jrwren
  • 17,465
  • 8
  • 35
  • 56
  • This sadly doesn't work at all, as `import sympy.sets` alone would run `sympy/__init__.py` , which imports everything. – orlp Sep 15 '15 at 19:45
0

You should test if importing only the modules that you are using in the code improves the loading time.

IE:

from sympy import mod1, mod2, mod3

vs

import sympy

You should read these previous questions:

Python import X or from X import Y? (performance)

improving speed of Python module import

'import module' vs. 'from module import function'

Community
  • 1
  • 1
dot.Py
  • 5,007
  • 5
  • 31
  • 52