17

Python takes 17 times longer to load on my Windows 7 machine than Ubuntu 14.04 running on a VM (inside Windows on the same hardware). Anaconda3 distribution is used on Windows and Ubuntu default python3.4.

From a Bash prompt (Git bash on Windows):

$ time python3 -c "pass"

returns in 0.614s on Windows and 0.036s on Linux

When packages are loaded the situation gets worse:

$ time python3 -c "import matplotlib"

returns in 6.01s on Windows and 0.189s on Linux

Spyder takes a whopping 51s to load on Windows and 1.5s on Linux.

I have not had any luck determining why I have this performance problems. Does anyone have any ideas what I should try next?

edit:

Why is python so much slower on windows? has been suggested as a possible duplicate but my performance different is far greater and not explained simply by different library dependencies and compilers. This seems to me to be related to filesystem differences.

I had suspected antivirus on-access scans but disabled the antivirus just in case.

>>> sys.path

['', 'c:\\Anaconda3\\python34.zip', 'c:\\Anaconda3\\DLLs', 'c:\\Anaconda3\\lib', 'c:\\Anaconda3', 'c:\\Anaconda3\\lib\\site-packages', 'c:\\Anaconda3\\lib\\site-packages\\Sphinx-1.2.3-py3.4.egg', 'c:\\Anaconda3\\lib\\site-packages\\cryptography-0.8-py3.4-win-amd64.egg', 'c:\\Anaconda3\\lib\\site-packages\\nose-1.3.4-py3.4.egg', 'c:\\Anaconda3\\lib\\site-packages\\win32', 'c:\\Anaconda3\\lib\\site-packages\\win32\\lib', 'c:\\Anaconda3\\lib\\site-packages\\Pythonwin', 'c:\\Anaconda3\\lib\\site-packages\\setuptools-14.3-py3.4.egg']

Update:

A fresh install of Windows 8.1 Pro on the same PC solved the problem. After reinstalling all applications and Anaconda3 Python performance is the best I have seen. Unfortunately the root cause of this issue is still unknown.

Correction:

After my IT dept installed Sophos SafeGuard encryption software and MS Endpoint Protection the problem returned. Same slow start as before. Disabling the extra software did not solve the problem so we are trying tests on other machines to trace the problem.

Community
  • 1
  • 1
Mike
  • 2,637
  • 5
  • 29
  • 40
  • possible duplicate of [Why is python so much slower on windows?](http://stackoverflow.com/questions/10150881/why-is-python-so-much-slower-on-windows) – Bhargav Rao May 02 '15 at 01:24
  • I am not as much concerned about the performance of the Python language, but why it takes so long to initially load. I have not profiled code execution performance on both platforms but have not felt a significant difference after the interpreter starts. – Mike May 02 '15 at 01:35
  • how big is on your %path% variable? where is python in that? lots of programs just concatenate themselves into it, gradually turning the whole thing into a mess. now, i really dont think that would account for such a slowdown, but eliminating that as a suspect may still be worthwhile. an alternative is to provide the actual path to python.exe and see if that makes a difference. i've worked a lot with python on windows (nt, xp, and 7) and i've never seen this. ditto looking at python's sys.path when it comes out, does your module loading search go all over the place? – JL Peyret May 02 '15 at 01:44
  • you may also want to check your fragmentation status. again, unlikely, but... – JL Peyret May 02 '15 at 01:56
  • path variable is quite cluttered, but I get the same benchmark with an explicit path to c:\Anaconda3\python.exe – Mike May 02 '15 at 01:56
  • 4
    my results are 0.013s win & 0.040 linux for `python3 -c "pass"`, i.e. python works normally on win. i suppose that it's more depends to you environment, rather than python itself. Check your anaconda distribution and 3d party connected dlls, which can be not perfectly fast ans slow down python. To test that, you could install separately vanilla python3k and compare your checks. – Reishin May 02 '15 at 02:00
  • Good idea to install vanilla python for comparison. However, the time really slows down when libraries are loaded, particularly `matplotlib`. – Mike May 02 '15 at 02:02
  • Disk defragmenter reports 0% fragmented – Mike May 02 '15 at 02:03
  • I do have lots of mapped network drives for my corporate LAN but I don't see how they are being touched. – Mike May 02 '15 at 02:04
  • that big python34.zip at the start of your path... does that mean it needs to be searched for all \*.py files coming with std python? I am sure that's optimized for, but... what about a normal, non-Anaconda, install of Python elsewhere, with *only* a non-zip python **directory** in all its sys.path? and no eggs either. how long does the "python3 -c pass take there? i am pretty sure you can just install another python and remove it afterwards without much hassle. i.e. we know it's not your code, so it can be the OS or the particular Python installation. i.e. what Reishin said. – JL Peyret May 02 '15 at 04:06
  • nah, doubt the network drives have much to do with it. – JL Peyret May 02 '15 at 04:11
  • Mike - did you resolve this? I am also seeing insanely slow Windows starts. – Russ Jul 02 '15 at 00:12
  • No solution yet. I will be upgrading to Win 8.1 next week, hopefully this does something to help since I don't see the problem on my Win 8 laptop. I don't think it is the machine itself since Python loads fine in a VM. – Mike Jul 02 '15 at 06:03
  • I recently moved from a vanilla Python install to using Anaconda and can confirm the same slow down. It does only tend to be on the first invocation. There used to be `KnownDLL` tricks for improving these situations. – Martin Evans Jul 08 '15 at 09:02
  • @MartinEvans What OS and CPU are you running? I am on Win7 Pro with a dual Xeon. Don't have any trouble with Win8 on an i7. – Mike Jul 09 '15 at 07:55
  • It is a Win7(32bit) on Intel i5-3570k at 3.4Ghz / Quad core, 4Gb installed. – Martin Evans Jul 09 '15 at 09:17
  • Seems to have no clear pattern, had the result of 0,097s on a Win7 i5 4570 CPU@3.2GHz – mstuebner Jul 19 '15 at 20:25
  • I wiped the PC in question a few days ago and did a fresh install of Windows 8.1 Pro. After reinstalling all of my software and Anaconda3 Python no launches faster than I have seen on any of my computers. So I don't know what caused this problem in the first place, but now it is solved for me. – Mike Aug 06 '15 at 06:50
  • 1
    Correction: The problem reappeared after a my IT dept installed antivirus and encryption software. Working now to see if these are the culprits. – Mike Aug 27 '15 at 19:31
  • 1
    Sometimes antivirus is active even if you explicitly disable it (only uninstall fixes it), especially when some kind of filter driver gets installed. Have seen stuff like 50ms pauses for every registry key access due to antivirus at times. So if reinstalling AV + encryption does that, you probably have the reason. Btw. windows has some pretty good performance measurment tools (xperf/wpr/wpa) that could probably tell you exactly where the time is spent. – schlenk Aug 27 '15 at 19:42
  • @DSM Sure thing. This has problem has been nagging me for months and I try to keep the question up to date with my struggles. – Mike Aug 27 '15 at 19:45
  • WPR/WPA does not show any significant CPU or memory utilization during the slow startup period. There is a large spike in hard faults from the python.exe process, but only at the very start of the process. Disk activity is high for the entire startup period and the disk service time is about 200ms for python.exe which sounds excruciatingly slow. If antivirus was involved wouldn't I see a corresponding spike in disk usage from the AV process as well? – Mike Aug 28 '15 at 23:13
  • 1
    After uninstalling Sophos SafeGuard I see a huge improvement. `time python3 -c "import matplotlib"` now reports 20ms on Windows 8. This is faster than I was seeing in my Linux VM. – Mike Sep 24 '15 at 18:58

3 Answers3

2

The problem is solved by uninstalling Sophos SafeGuard. This is not really a satisfactory solution though since my company uses this filesystem encryption software on directories that I access daily. I have not seem any other performance problems except with Python (and apparently Ruby as well).

NOTE: Sophos SafeGuard is not antivirus software. It is an enterprise filesystem encryption system. The strange thing is that encryption is explicitly not enabled for local filesystems, such as where Python is installed.

Mike
  • 2,637
  • 5
  • 29
  • 40
2

May not be relevant to your case, but I found that running python in Windows with Sophos Safeguard and Mcafee Enteprise Antivirus that python startup times were an order of magnitude slower if python was being run as an elevated process. Switching it to run as a normal process made a dramatic difference for me.

ivanatpr
  • 1,862
  • 14
  • 18
  • Interesting, I have rarely needed to run python elevated and was easily seeing an order of magnitude slowdown with Safeguard running as a normal process. – Mike Dec 09 '15 at 07:49
1
  1. Perhaps a contributor to startup variance could be default loaded modules. You can use sys.modules to compare your two environments.

python -c "import sys;print(len(sys.modules))"

For me the answer is 

$ time py -2 -c "pass"

real    0m0.054s
user    0m0.000s
sys     0m0.000s

$ py -2 -c "import sys;print(len(sys.modules))"
44

$ time py -3 -c "pass"

real    0m0.063s
user    0m0.000s
sys     0m0.000s

$ py -3 -c "import sys;print(len(sys.modules))"
54

And you can use virtual envs to manipulate the default loaded modules. https://virtualenv.pypa.io/en/latest/

  1. Git bash for windows seems to be misbehaving for me with python. I don't see the version banner when I launch the interpreter. I would compare start times with a cmd prompt. Or even with a python launching python. E.g.

-

import subprocess
import time
start = time.time()
subprocess.check_call(["python", '-c ', 'pass'])
print time.time() - start
ubershmekel
  • 11,864
  • 10
  • 72
  • 89
  • I'm getting 0.047s for `time python -c "pass"` on my Lenovo Y410p running Win 10 but 0.585s for the same on my workstation running Win 8. The module counts for both machines are the same for Python 3. – Mike Aug 31 '15 at 06:56