python: disk-bound task, thread vs process

Question

I parse a big source code directory (100k files). I traverse every line in every file and do some simple regex matching. I tried threading this task to multiple threads but didn't get any speedup. Only multiprocessing managed to cut the time by 70%. I'm aware of the GIL death grip, but aren't threads supposed to help with IO bound access?

If the disk access is serial, how come several processes finish the job quicker?

because the only processing I do per line is updating a dict — susdu, Jun 23 '17 at 18:53
yes. I don't think regex matching lines a file takes more time than getting the file handle from the OS. — susdu, Jun 23 '17 at 19:17
Did you try profiling the program? Sounds like you need some hard data to make progress. You can try cProfile: https://stackoverflow.com/questions/582336/how-can-you-profile-a-script — Shmulik Asafi, Jun 23 '17 at 19:24

Warren Dew · Accepted Answer · 2017-07-06T16:46:27.083

1

Python "threads" permit independent threads of execution, but typically do not permit concurrency because of the global interpreter lock: only one thread can really be running at a time. This may be the reason why you only get a speedup with multiple processes, which do not share a global interpreter lock and thus can run concurrently.

edited Jul 06 '17 at 16:46

answered Jun 23 '17 at 20:38

Warren Dew

8,790
3
30
44

python: disk-bound task, thread vs process

1 Answers1

Linked