How to troubleshoot very slow python code

Question

I have a python script that processes a large number of files.

For each file, the script goes line by line, searching for specific RegEx patters. If a pattern is found, the line is copied into the log file.

As sample input, I'm passing it a folder with 42 small files and 3 large files (~1500 lines each).

The scripts processes the first two large files very fast - it needs a few seconds for them. But when it reaches the third large file, it slows down, and it goes slower and slower.

In the middle of the third large file, it needs a whole second per line, and it keeps slowing down. If I don't stop it, the whole run takes an hour!

I added debugging code that prints out the line numbers - that's how I noticed that it keeps churning slower and slower, and it doesn't get stuck somewhere.

I have 20 years experience with c, and many other languages, but I'm a python beginner. What are steps that I can take to troubleshoot this script?

[The Python Profilers](https://docs.python.org/2/library/profile.html) — myaut, May 05 '15 at 14:50
Profiling will give most benefit, also [this](https://wiki.python.org/moin/PythonSpeed) and [this](https://wiki.python.org/moin/PythonSpeed/PerformanceTips) might help some. — Paul Rooney, May 05 '15 at 14:54
This might be a good question for http://codereview.stackexchange.com — Robᵩ, May 05 '15 at 15:17
Well, yeah. I assumed that he would post his actual code there. — Robᵩ, May 05 '15 at 15:19

score 2 · Accepted Answer · edited May 23 '17 at 12:03

2

If your code is a script you can run cProfile as shown in this answer

python -m cProfile myscript.py

I do not know if this gives you the granularity you wanted, otherwise have a look at The Python Profilers

As for the actual reason your code runs slow I suspect either catastrophic backtracking or that you open and append to your log file every time the pattern matches aka. Shlemiel The Painter

edited May 23 '17 at 12:03

Community

1
1

answered May 05 '15 at 14:55

Jonas

491
6
22

Reference to Shlemiel was very helpful. In certain rare circumstances, my script started concatenating lines together to no end... Fixed! – Ada Lovelace May 07 '15 at 14:34
Was this the cause of your slowdown? And were the profiling tools helpful / granular enough? – Jonas May 08 '15 at 06:41
I never got around to check the profiling. When I read your answer, the first thing I did was check the reference to Shlemiel. When I read the story, I said to myself: "I must be doing Shlemiel's algorithm somewhere". Followed by the "Duh" moment, when I found the bug. – Ada Lovelace May 08 '15 at 21:44

How to troubleshoot very slow python code

1 Answers1