do I need lock to protect mutli-thread race condition in my code

Question

Using Python 2.7 on Windows, and will use Jython which support true multi-threading. The method sendMessage is used to receive message from a specific client, and the client may send the same message to a few other clients (which is what parameter receivers is for, and receivers is a list). The method receiveMessage is used to receive message for a specific client, which are sent from other clients.

The question is whether I need any locks for method sendMessage and receiveMessage? I think there is no need, since even if a client X is receiving its message, it is perfect fine for another client Y to append to message pool to deliver message to client X. And I think for defaultdict/list, append/pop are both atomic and no need for protection.

Please feel free to correct me if I am wrong.

from collections import defaultdict

class Foo:
    def __init__(self):
        # key: receiver client ID, value: message
        self.messagePool = defaultdict(list)
    def sendMessage(self, receivers, message):
        # check valid for receivers
        for r in receivers:
            self.messagePool[r].append(message)
    def receiveMessage(self, clientID):
        result = []
        while len(self.messagePool[clientID]) > 0:
            result.append(self.messagePool[clientID].pop(0))

        return result

score 4 · Answer 1 · edited Sep 09 '16 at 01:17

4

I suggest to use Queue instead of list. It is designed for append\pop in threads with locking.

edited Sep 09 '16 at 01:17

rantanplan

7,283
1
24
45

answered Sep 06 '16 at 07:56

eri

3,133
1
23
35

Thanks eri, I accept your advice. Do you have any official document saying for a Python List, it does **not** "disigned for append\pop in treads with locking" – Lin Ma Sep 06 '16 at 16:19
Thanks eri, vote up for your reply. I read through this document, http://effbot.org/zone/thread-synchronization.htm, it seems no need to add lock if I add/remove elements from s list? – Lin Ma Sep 06 '16 at 16:26
1

I just say that Queue is best choice for queue realization. – eri Sep 06 '16 at 19:31
Thanks eri, I fully understand queue is better, my example is a bit misleading not using queue, but the whole purpose is not queue v.s. list. :) My example is about defaultdict and list. I may not using the right data structure here, but I am asking for defaultdict and list, if you have any comments on my original code, it will be great. – Lin Ma Sep 07 '16 at 05:38
1

lock is not needed for your code. there is another answer discrabing that opinion. but i think that my answer useful for people, who google this in future... – eri Sep 08 '16 at 20:41
Thanks eri, when you say "another answer discrabing that opinion", which answer do you mean? – Lin Ma Sep 09 '16 at 06:46

score 3 · Accepted Answer · edited May 23 '17 at 12:01

3

I think this question is already well-answered for CPython here and here (basically, you're safe because of GIL, although nothing in documentation (like on defaultdict or list) officially says about that). But I understand your concern about Jython, so let's solve it using some official source, like Jython source code. A pythonic list is a javaish PyList there with this kind of code:

public void append(PyObject o) {
    list_append(o);
}
final synchronized void list_append(PyObject o) {
...
}
public PyObject pop(int n) {
    return list_pop(n);
}
final synchronized PyObject list_pop(int n) {
...
}

And as we have these methods synchronized, we can be sure that list appends and pops are also thread-safe with Jython. Thus, your code seems to be safe wrt threading.

Although Queue suggestion is still valid one, it really is more appropriate for this use case.

edited May 23 '17 at 12:01

Community

1
1

answered Sep 06 '16 at 21:18

Roman Khimov

4,807
1
27
35

2

@LinMa: [Global Interpreter Lock](https://wiki.python.org/moin/GlobalInterpreterLock). – Roman Khimov Sep 07 '16 at 06:00
Thanks Roman, vote up for your reply. Last question is, do you think it is safe to avoid using lock if I am using Jython for operation like list add/remove, dictionary add/remove and queue push/pop? – Lin Ma Sep 08 '16 at 05:26
1

@LinMa: Well, it's the same source. [List add/remove](https://hg.python.org/jython/file/tip/src/org/python/core/PyList.java) are synchronized, dictionaries [use javaish ConcurrentHashMap](https://hg.python.org/jython/file/tip/src/org/python/core/PyDictionary.java#l55) internally, that is thread-safe and pythonic Queue is guaranteed to be thread-safe by [Python documentation](https://docs.python.org/2/library/queue.html), Jython [implements it](https://hg.python.org/jython/file/tip/lib-python/2.7/Queue.py) properly with a lock. – Roman Khimov Sep 08 '16 at 06:13
Thanks Roman, looks like everything (I mean single operation on containers in Jython) is thread-safe? Vote up for your reply. – Lin Ma Sep 09 '16 at 06:47
Thanks for the patience to answer Roman, mark your reply as answer. – Lin Ma Sep 11 '16 at 00:33

score 2 · Answer 3 · answered Sep 06 '16 at 07:42

2

Race conditions is about two or more threads changing some global states at the same time.

In your code for sendMessage, you are changing self.messagePool[r], which is a global object. Hence, self.messagePool[r] should be locked before appending a new item.

Same with your receiveMessage function.

list.append and list.pop are armotized O(1) and O(1) operations, so it rarely would cause any race condition. However, the risk is still there.

answered Sep 06 '16 at 07:42

Quan To

697
3
10

Thanks btquanto, I think in Python, `list.append` and `list.pop` are atomic operation (for atomic, I mean no thread context switch possible before completion of the operation), so I think no need for lock, any comments? – Lin Ma Sep 06 '16 at 07:48
1

Well, I think they don't take just one CPU instruction to complete. If the threads are running on two different CPUs, they can possibly (though unlikely) run at the same time. For example, what `list.pop` does is take the last element of the list, the change the list's final index. Imagine two threads popping the final element at the same time, then changing the final index twice. – Quan To Sep 06 '16 at 07:56
Thanks btquanto, vote up for your reply. I read through this document, http://effbot.org/zone/thread-synchronization.htm, it seems no need to add lock if I add/remove elements from s list? – Lin Ma Sep 06 '16 at 16:26

do I need lock to protect mutli-thread race condition in my code

3 Answers3