0

I am creating a p2p network using python sockets where each node will have a listening thread and a client side thread that connects to other nodes and listens to other node connections respectively. I created a little buffering protocol on the listening side that will buffer the msgs until it gets all the bytes.

Now, the first issue I was getting was sometimes socket.sendall() would not send all the bytes and in that case there is no way for my listener (on other nodes) to know that this happens and so it would try to buffer msgs until it receives x amt of bytes and then call json.loads() on it, resulting in an exception. To resolve this, I decided to instead use socket.send() in a loop to ensure that all bytes will be sent as below.

while total_sent < msg_len:
    sent = recipient_node['socket'].send(byte_msg[total_sent:])
    if sent == 0: # disconnected?
        pass
    total_sent += sent

I get a new bug now which is 'Connection reset by peer' which the stack trace shows happens on the socket.send(). I understand that this is due to some GIL issues and inserting a time.sleep(.0001) or something before sends resolves this issue (kind of), but slows down the p2p network of course.

My question then is, is there a way I can ensure all bytes are delivered using sendall() or is there is a better method using send() and time.sleep(). I'm open to either.

kamui_34
  • 1
  • 1
  • Which question are you asking? - `how to ensure all bytes are sent/received?` or `What does connection reset by peer mean and how to fix it?` or `is there a better solution to my working solution`? – wwii Mar 08 '21 at 18:07
  • Does [Socket Programming Python: How to make sure entire message is received?](https://stackoverflow.com/questions/56813704/socket-programming-python-how-to-make-sure-entire-message-is-received) answer your question? Or, – wwii Mar 08 '21 at 18:11
  • 1
    Related: [How does the python socket.recv() method know that the end of the message has been reached?](https://stackoverflow.com/questions/41382127/how-does-the-python-socket-recv-method-know-that-the-end-of-the-message-has-be). – wwii Mar 08 '21 at 18:13
  • @wwii Sorry for the confusion. So at this point I know that the code snippet above will send all the bytes in however many iterations. I am now dealing with the connection is reset error. As I posted in the edit above, I saw somewhere that putting a time.sleep before the sends could resolve this issue, something to do with the GIL? I tested this on the vms and it seems to resolve the issue, however now of course the 1ms delay is very noticeable in the sends. I was wondering if there's a better solution. – kamui_34 Mar 08 '21 at 18:22
  • To be clear, I am having this connection reset by peer problem now by using send in a loop. Before I was using sendall, but decided to use send because if sendall doesn't send all the bytes (which happens sometimes), there is no way for my listener on each node to know how to parse the garbage data or clean it up to ensure it doesn't try to parse an incomplete json string. – kamui_34 Mar 08 '21 at 18:57
  • `connection reset by peer` means the program on the other side has closed the connection. There's not much you can do about it other than create a new TCP socket and try to connect again, and hope that he won't hang up on you again. Perhaps it is the receiving side that needs debugging? – Jeremy Friesner Mar 08 '21 at 23:06
  • @JeremyFriesner I only get this issue when I use the snippet above where I send bytes in a loop until all the bytes are sent. If I either just use sendall or use the time.sleep before I send I will not see this error. However, if I use sendall then because sometimes sendall will fail, I'm not able to reliably decode json strings on the listener side. – kamui_34 Mar 08 '21 at 23:27
  • Your receiver is likely not handling all cases correctly. It needs to be coded to do the right thing, regardless of how many bytes it gets handed by each call it makes to `recv()`; otherwise it will only work if you get "lucky" in that `recv()` passes to it the "right" number of bytes at once for it to work (which `recv()` is never guaranteed to do, so you can't rely on that) – Jeremy Friesner Mar 08 '21 at 23:32
  • @JeremyFriesner I'm not assuming that I get n bytes a time, whenever the sockets do send all the data of a msg (not necessarily at once), I do handle it properly because i have a msg header with a delimiter for each message. The problem is, sometimes the socket.sendall would not send all the data (it would throw an exception or something before all the data would be sent), in which case my listeners would receive partial data of the first msg and then once it got enough bytes from following msgs it would try to json.loads it. I'm not sure how to handle that case ... – kamui_34 Mar 08 '21 at 23:47

1 Answers1

0

Welp, I figured it out. I actually figured out the original problem I was having. Since I'm using select as my non blocking listener, it is basically one dedicated piece of code to handle EVERY socket connection and so recvs need not be from one node. Instead of a buffer string, I now use a buffer map that maps each client socket to its own buffer string so that it doesn't intermix messages from different sockets. I can't believe I didn't realize this was happening until now. Still weird that a lot of the times even though the buffer would be getting intermixed, it would still work majority of the time.

kamui_34
  • 1
  • 1