2

i have a question regards python locks and threading, i realise locks are used to prevent variables from being overwritten by another thread, is it normal to use locks to get around this issue as it then means you can only run a single thread concurrently, it also means creating acquire/release locks for every variable that maybe overwritten, which for my project runs into quite a few!

how are people doing this?, wrapping the variables in thread safe lists or creating unique variables based on the thread name perhaps?, or is everybody littering their code with lock acquire and releases?.

binhex
  • 374
  • 4
  • 13
  • 1
    Odd that this is closed as a duplicate, since the question referred to doesn't reference python at all. Yes there are many considerations that are common to most programming languages, but some languages are much more thread-friendly than others, and each has its own quirks (e.g. Python's GIL, as @nosklo told us). Answers focused on python are valuable for that reason. – LarsH Oct 17 '13 at 16:57
  • Nothing in the question as-stated is python specific. Further the question is broad and poorly posed, so the reason for closing is almost immaterial. – Nathaniel Ford Oct 17 '13 at 17:17

1 Answers1

4

The best idea is to just not use threads at all. Most Python implementations have a global interpreter lock which eliminates the advantages of using threads in first place. If you're using threading to wait for IO, you can get the same or better performance if you just use asynchronous IO instead. If you're using threading to perform computation (number crunching) across processors, the python global lock prevents it from working so you're better using multiple processes instead.

In contrast with having no upside, threading in python have lots of shortcomings and caveats, like you already found out. You still have to do data sharing control, and to deal with oddities related to threads receiving cpu attention in moments you don't control. All that for no benefit.

TL;DR just don't use threads

nosklo
  • 217,122
  • 57
  • 293
  • 297
  • ok thanks, so to summarise try and use multiprocessing instead of threading, as it sidesteps the requirement for a GIL, yes? – binhex Jun 18 '13 at 13:23
  • @Paul if you need multiple core number crunching, then yes. If you need to wait for input/output then just use asynchronous IO in a single thread/process. – nosklo Jul 23 '13 at 14:00
  • 1
    Its not true that threading has no upside. Use threading when you have a large amount of state to share between tasks. Use multiprocessing where you do not and the overheads for forking are high (e.g. when you need to share many kb of state between the processes). Here's a great video by a core python contributor https://www.youtube.com/watch?v=Bv25Dwe84g0 You *DO* need to be careful to avoid forking after creating a thread. I think number crunching usually happens outside the GIL (C libraries should typically release the GIL while doing their processing). – user48956 Aug 29 '17 at 05:08