Python: Socket and threads?

Question

I have a design problem:

I have two threads, a heartbeat/control thread and a messagehandler thread.

Both are sharing the same socket, however the messageHandler thread only sends out messages and never receives. The heartbeat thread sends and receives (receives messages and reacts on heartbeats).

The problem is I'm not sure if this is safe. There is no mechanism, I myself, implemented to see if the socket is being used. So is sharing a socket over python automatically thread safe or not?

Also if it's not, the reason I put them in a separate thread is because the heartbeat is more important than the message handling. This means that if it gets flooded with messages, it still needs to do a heartbeat. So if I have to implement a bolt, is there away I can prioritize if my heartbeat/control thread needs to send a heartbeat?

score 21 · Answer 1 · answered Sep 27 '14 at 02:49

A better way instead of using a third thread is to use threading.Lock() to protect the socket resource because this removes the need of a third thread. You have lower overhead and less latency than if you had a third thread.

import threading

lock = threading.Lock()

def sendfunction(sock, data):
    with lock:
        sock.send(data)

You can call that from either of your threads but only one thread at a time will be allowed to call sock.send. When a thread reaches the lock that is already locked by another thread it will sleep until the other thread releases the lock then it will acquire the lock and the process repeats.

The threading module contains Lock, RLock, and Condition which are all very useful when dealing with multiple threads and you will find it well worth your time to become familiar with them and their usage.

You could incorporate the heartbeat into your message handling by checking the current time with the last time you sent a heartbeat before you process each message, and that would prevent being flooded with messages causing a heartbeat not to be sent. The problem is if your message handling code does not run then no heartbeats will be sent. You could alleviate this by having your message handling code get a dummy message on interval to allow it to check if it needs to send a heartbeat and just ignore the dummy message.

You should try to use threads sparingly (aim for a single thread) however in your case a thread would likely be okay since it is going to spend most of it's time sleeping. You should however not use a daemon thread because they do not properly shutdown. Although the damage might be nonexistent in your case if it did not properly shutdown it still might throw some type of fault (error message) which looks bad.

I do not agree with the multiple sockets method as I think it would actually complicate the situation. You will find a many types of network services/applications out there that incorporate the heartbeat and messages into a single socket byte stream.

score 10 · Accepted Answer · answered Jun 24 '12 at 11:59

10

Unfortunately,The socket shared by multi-thread is not thread safe.Think about buffer two threads operate on with no lock.

The normal way to implement is with two socket,just like what ftp does.cmd socket and msg socket.

If you wanna implement this by one socket,you can put different type msgs into different queues,and with a third thread consumes the queue and send them through the only socket.

In this way,you can control heartbeat msg priory to data msg.

answered Jun 24 '12 at 11:59

yancl

263
1
7

13

Reading on a thread and sending from another thread is thread safe? – Colateral Jan 18 '17 at 10:10
4

Can someone please address the question above by @Colateral? I have the same doubt. – rdp May 01 '18 at 18:36
I have the same doubt. – Jasha Jun 08 '21 at 11:59
I believe @yancl was talking about using a queue (possibly a deque?) between threads. A deque does atomic operations for appending values and popping values off. That means Thread A can be generating control messages and appending them to a control message deque. Thread B can be generating data messages and appending them to a data message deque. Thread C would then have the actual socket and would pop values from the two deque's and send them out on the socket. In that scenario there is still only 1 socket with 3 threads. Thread C could then prioritize messages from the control deque. – Eric Evans Sep 16 '21 at 18:22
As for reading and writing a socket from 2 different threads...you wouldn't be if there were deque's between the thread with the socket and your code that sends/receives. The thread with the socket would receive and push those messages into a deque that your reader code has the other end of. All sending/receiving directly with the socket would be in 1 thread. – Eric Evans Sep 16 '21 at 18:26

uselpa · Answer 3 · 2012-06-24T11:37:47.890

0

I don't know of a way to prioritize at the Python level.

So I'd suggest using 2 processes, not threads, and prioritize at the OS level. On Unix you can use os.nice() to do that.

You'd need to use 2 sockets then, and your sharing problem would be solved at the same time.

edited Jun 24 '12 at 11:37

answered Jun 24 '12 at 11:28

uselpa

18,732
2
34
52

score 0 · Answer 4 · answered Jun 24 '12 at 11:39

0

If both threads are client threads, it is a good idea to open two client sockets one to the server for heart beat and another for communication.

answered Jun 24 '12 at 11:39

Tariq Mehmood

259
1
5
15

Python: Socket and threads?

4 Answers4

Linked