What is the proper way of sending a large amount of data over sockets in Python?

Question

Recently I wrote some code (client and server) to send an image - the client simply uploads the image to the server, just using the socket module: Sending image over sockets (ONLY) in Python, image can not be open.

However, the image sending part is now what I am concerned with. This is the original image I'm using:

In my server code (which receives the images), I have these lines:

myfile = open(basename % imgcounter, 'wb')
myfile.write(data)

data = sock.recv(40960000)
if not data:
     myfile.close()
     break
myfile.write(data)
myfile.close()

sock.sendall("GOT IMAGE")
sock.shutdown()

But I don't think this is the best way of doing it. I think I should instead implement the server such that it receives the data in chunks:

#!/usr/bin/env python

import random
import socket, select
from time import gmtime, strftime
from random import randint

imgcounter = 1
basename = "image%s.png"

HOST = '127.0.0.1'
PORT = 2905

connected_clients_sockets = []

server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_socket.bind((HOST, PORT))
server_socket.listen(10)

connected_clients_sockets.append(server_socket)

while True:

    read_sockets, write_sockets, error_sockets = select.select(connected_clients_sockets, [], [])

    for sock in read_sockets:

        if sock == server_socket:

            sockfd, client_address = server_socket.accept()
            connected_clients_sockets.append(sockfd)

        else:
            try:

                data = sock.recv(4096)
                txt = str(data)

                if data:

                    if data.startswith('SIZE'):
                        tmp = txt.split()
                        size = int(tmp[1])

                        print 'got size %s' % size

                        sock.sendall("GOT SIZE")

                    elif data.startswith('BYE'):
                        sock.shutdown()

                    else :

                        myfile = open(basename % imgcounter, 'wb')
                        myfile.write(data)

                        amount_received = 0
                        while amount_received < size:
                            data = sock.recv(4096)
                            amount_received += len(data)

                            print amount_received

                            if not data:
                                break
                            myfile.write(data)
                        myfile.close()

                        sock.sendall("GOT IMAGE")
                        sock.shutdown()
            except:
                sock.close()
                connected_clients_sockets.remove(sock)
                continue
        imgcounter += 1
server_socket.close()

But when I do this, the server prints:

got size 54674
4096
8192
12288
16384
20480
24576
28672
32768
36864
40960
45056
49152
50578

And then seems to hang, and the client hangs too. However, at the server's side I can see only a piece of the image the client wanted to send:

It seems like there are some bytes missing. What is the proper way of sending a huge amount of data (images, other type of file) using ONLY sockets?

you are using `select` but also an inner while-loop with `recv`. That makes no sense. — Daniel, Feb 25 '17 at 17:55
@Daniel: It's not the case. In my linked post, both client and server work, and I there (almost) exact code. — yak, Feb 25 '17 at 17:59
I can't read Python but if the data arrives in arbitrarily sized chunks (which it does) then you can't sensibly check that the data starts with "SIZE" because that may not be at the start surely? — Mark Setchell, Mar 01 '17 at 13:25
This question may help you : http://stackoverflow.com/questions/8994937/send-image-using-socket-programming-python — Naveen Holla, Mar 08 '17 at 09:13

score 25 · Accepted Answer · edited May 23 '17 at 12:18

I'm assuming that you have a particular reason for doing this with naked sockets, such as self-edification, which means that I won't answer by saying "You accidentally forgot to just use HTTP and Twisted", which perhaps you've heard before :-P. But really you should look at higher-level libraries at some point as they're a lot easier!

Define a protocol

If all you want is to send an image, then it can be simple:

Client -> server: 8 bytes: big endian, length of image.
Client -> server: length bytes: all image data.
(Client <- server: 1 byte, value 0: indicate transmission received - optional step you may not care if you're using TCP and just assume that it's reliable.)

Code it

server.py

import os
from socket import *
from struct import unpack


class ServerProtocol:

    def __init__(self):
        self.socket = None
        self.output_dir = '.'
        self.file_num = 1

    def listen(self, server_ip, server_port):
        self.socket = socket(AF_INET, SOCK_STREAM)
        self.socket.bind((server_ip, server_port))
        self.socket.listen(1)

    def handle_images(self):

        try:
            while True:
                (connection, addr) = self.socket.accept()
                try:
                    bs = connection.recv(8)
                    (length,) = unpack('>Q', bs)
                    data = b''
                    while len(data) < length:
                        # doing it in batches is generally better than trying
                        # to do it all in one go, so I believe.
                        to_read = length - len(data)
                        data += connection.recv(
                            4096 if to_read > 4096 else to_read)

                    # send our 0 ack
                    assert len(b'\00') == 1
                    connection.sendall(b'\00')
                finally:
                    connection.shutdown(SHUT_WR)
                    connection.close()

                with open(os.path.join(
                        self.output_dir, '%06d.jpg' % self.file_num), 'w'
                ) as fp:
                    fp.write(data)

                self.file_num += 1
        finally:
            self.close()

    def close(self):
        self.socket.close()
        self.socket = None

        # could handle a bad ack here, but we'll assume it's fine.

if __name__ == '__main__':
    sp = ServerProtocol()
    sp.listen('127.0.0.1', 55555)
    sp.handle_images()

client.py

from socket import *
from struct import pack


class ClientProtocol:

    def __init__(self):
        self.socket = None

    def connect(self, server_ip, server_port):
        self.socket = socket(AF_INET, SOCK_STREAM)
        self.socket.connect((server_ip, server_port))

    def close(self):
        self.socket.shutdown(SHUT_WR)
        self.socket.close()
        self.socket = None

    def send_image(self, image_data):

        # use struct to make sure we have a consistent endianness on the length
        length = pack('>Q', len(image_data))

        # sendall to make sure it blocks if there's back-pressure on the socket
        self.socket.sendall(length)
        self.socket.sendall(image_data)

        ack = self.socket.recv(1)

        # could handle a bad ack here, but we'll assume it's fine.

if __name__ == '__main__':
    cp = ClientProtocol()

    image_data = None
    with open('IMG_0077.jpg', 'r') as fp:
        image_data = fp.read()

    assert(len(image_data))
    cp.connect('127.0.0.1', 55555)
    cp.send_image(image_data)
    cp.close()

Thank you so much. But if I would like to do the same with the message padding, should it look like this: (server https://gist.github.com/anonymous/f349ab3ab963295baa6435ad9169e5da) and (client: https://gist.github.com/anonymous/a327124e0e82b748a04fbf468bd165fb)? I mean, your code is excellent, but I would rather use message padding. The problem is, I do not know if I implemented it right. — yak, Mar 01 '17 at 17:35
I don't quite understand what you mean by "I would rather use message padding". Could you explain what you want the protocol to look like exactly (maybe just update the question)? It should be easy to adapt my code to a slightly different protocol. Also there are a few problems I think with the code you've posted links to, but really if you want a code review you should do that on [codereview.stackexchange](https://codereview.stackexchange.com). Anyway, you asked what the "proper" way was, and I think I've answered that! — daphtdazz, Mar 01 '17 at 23:27
By message padding I meant what I did in linked codes - end messages with specific delimiters, not begin them with the size of the message as you did. The protocol I would like to implement is https://gist.github.com/anonymous/2dc52dc5288c85831dd370a85d255903. Thank you. — yak, Mar 02 '17 at 11:45
But the problem is you can't use a protocol that is based on delimiters to send arbitrary data, because what happens if the delimiter appears in the data? jpeg data is basically random. So you have to send a length number anyway, which is what you're doing. But if your algorithm is both partitioning the data based on a delimiter and using a length parameter depending on the last message, it is going to be a lot more complicated! Also, the protocol that you linked to is for a client to retrieve a file from the server, but this question is for a client to send a file to the server... — daphtdazz, Mar 03 '17 at 10:08

Vijendra1125 · Answer 2 · 2021-06-20T15:53:42.067

A simple way is to send data size as the first 4 bytes of your data and then read complete data in one shot. Use the below functions on both client and server-side to send and receive data.

def send_data(conn, data):
    serialized_data = pickle.dumps(data)
    conn.sendall(struct.pack('>I', len(serialized_data)))
    conn.sendall(serialized_data)


def receive_data(conn):
    data_size = struct.unpack('>I', conn.recv(4))[0]
    received_payload = b""
    reamining_payload_size = data_size
    while reamining_payload_size != 0:
        received_payload += conn.recv(reamining_payload_size)
        reamining_payload_size = data_size - len(received_payload)
    data = pickle.loads(received_payload)

    return data

you could find sample program at https://github.com/vijendra1125/Python-Socket-Programming.git

score 2 · Answer 3 · answered Mar 01 '17 at 15:04

The problem is you are not incrementing amount_received for the first chunk of the data received.

Fix below:

#!/usr/bin/env python

import random
import socket, select
from time import gmtime, strftime
from random import randint

imgcounter = 1
basename = "image%s.png"

HOST = '127.0.0.1'
PORT = 2905

connected_clients_sockets = []

server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_socket.bind((HOST, PORT))
server_socket.listen(10)

connected_clients_sockets.append(server_socket)

while True:

    read_sockets, write_sockets, error_sockets = select.select(connected_clients_sockets, [], [])

    for sock in read_sockets:

        if sock == server_socket:

            sockfd, client_address = server_socket.accept()
            connected_clients_sockets.append(sockfd)

        else:
            try:

                data = sock.recv(4096)
                txt = str(data)

                if data:

                    if data.startswith('SIZE'):
                        tmp = txt.split()
                        size = int(tmp[1])

                        print 'got size %s' % size

                        sock.sendall("GOT SIZE")

                    elif data.startswith('BYE'):
                        sock.shutdown()

                    else :

                        myfile = open(basename % imgcounter, 'wb')
                        myfile.write(data)

                        amount_received = len(data) #  The fix!
                        while amount_received < size:
                            data = sock.recv(4096)
                            amount_received += len(data)

                            print amount_received

                            if not data:
                                break
                            myfile.write(data)
                        myfile.close()

                        sock.sendall("GOT IMAGE")
                        sock.shutdown()
            except:
                sock.close()
                connected_clients_sockets.remove(sock)
                continue
        imgcounter += 1
server_socket.close()

What is the proper way of sending a large amount of data over sockets in Python?

3 Answers3

Define a protocol

Code it