Python socket putting data of multiple send into one receive buffer

Question

I am trying to make a simple image-sharing app in Python such that images can be shared from clients to a server. I am doing that using socket and sending images in form of numpy arrays. My problem is that when the app loads, I have made such that 6 images or less(if less than 6 present for an account) are sent from server to the client. Each image is stored in the server with a unique name identifying it. This name is of variable length and is decided by the user and is sent before the image array is sent. Also, the shape of the image is sent to reproduce it from the bytes at the client. But as the name length is not fixed, I am reading 10 buffer size. But if the name is smaller, it is reading the shape of the are that I am sending later as well. How do I fix this? Here is the sending image code:

def send_multiple_imgs(send_socket, imgs):
    num = len(imgs)
    send_socket.send(str(num).encode())
    for img in imgs:
        print(img)
        send_socket.send(img.encode())
        send_img(send_socket,imgs[img])

part of send_img function:

def send_img(send_socket,img):
    send_socket.send(str(img.shape).encode())
    .....

The later part is not important, I think. Here is the receiving part:

def receive_multiple_img(recv_socket):
    num = int(recv_socket.recv(1).decode())
    imgs = {}
    for i in range(num):
        img_name = recv_socket.recv(10).decode()
        print(img_name)
        imgs[img_name] = recieve_image(recv_socket)

    return imgs

What is happening is, I have an image named 'ds' of shape (200,200,4), but the img_name reads: 'ds(200, 20' and then it messes up the further sending and receiving as well. How do I fix this? I am using TCP protocol:

s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)

I am new to networking in python. So, please consider if this is a silly question

Check out this answer which seems to be doing the same thing you want to do. https://stackoverflow.com/a/36258606/1076479 — Gil Hamilton, Nov 12 '19 at 22:44
Should expand on that... it's necessary to impose a *protocol* on the byte stream that TCP provides. That means, if you have more than one "atomic" thing to send, you usually want to provide a prefix containing a binary-encoded length for each variable-length piece of data you want to send. Then you need to carefully ensure that you send exactly that many bytes (say, with `.sendall`) and that you receive exactly that many (unfortunately, there is no corresponding `.recvall` but it's easy to create a function that does the same thing). See the `.struct` module for binary encoding/decoding. — Gil Hamilton, Nov 12 '19 at 22:48
@GilHamilton my question is why is the `send` putting both data into one buffer. I have made other socket programs but this has never happened. Everytime I have got just the length same as the sent string in return at `recv` — Shantanu Shinde, Nov 13 '19 at 00:33

Malt · Accepted Answer · 2019-11-14T15:33:18.933

TCP, unlike UDP, is a stream protocol. That means that a single send() doesn't correspond to a single read(). A call to send() simply places the data to be sent into the TCP send buffer, and a call to read() simply returns the bytes from the TCP receive buffer. The bytes in the buffer could have come from a single send() on the other side or a hundered.

This is a fairy common misunderstanding that leads to many bugs. Here's me explaining this again, and again, and again, and again.

If you want to send() several separate messages or pieces of data, the reader must be able to tell the messages apart. There are many ways to do that, such as using fixed-length messages, prefixing the message's length before each message, or using a delimiter.

thanks for the explanation. I solved the problem by sending acknowledgment after every recv of a send. so, now one send corresponds to one recv and now it is working — Shantanu Shinde, Nov 15 '19 at 07:25

Python socket putting data of multiple send into one receive buffer

1 Answers1

Linked