Python3 Socket Server image data missing

Question

I am sending data (images and pictures) over a socket server from a laptop to a raspberry pi in python3.7, and it works fine when the client and the server are both on the same device (the laptop), however when the transfer occurs between the computer and the raspberry pi, the image loses about half the data. No matter how many times I try, or how many times I change the bytes read in, the same incomplete pictures are output:

Here is the input pics I tested with

and here are the pictures after I send them to the rpi and they get sent back:

Here is my server code:

#!/usr/bin/env python

import random
import socket, select
from time import gmtime, strftime
from random import randint

imgcounter = 1
basename = "image8.jpg"

HOST = '0.0.0.0'
PORT = 9999

connected_clients_sockets = []

server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_socket.bind((HOST, PORT))
server_socket.listen(10)

connected_clients_sockets.append(server_socket)

while True:
    read_sockets, write_sockets, error_sockets = select.select(connected_clients_sockets, [], [])

    for sock in read_sockets:

        if sock == server_socket:

            sockfd, client_address = server_socket.accept()
            connected_clients_sockets.append(sockfd)

        else:
            try:

                data = sock.recv(4096)
                try:
                    txt = data.decode(encoding = 'UTF-16',errors = 'strict')
                except:
                    txt = ""
                #txt = str(data)
                if data:
                    #print("recieved data",data.decode(encoding = 'UTF-16',errors = 'strict'))
                    if txt == 'SIZE 11907' :
                        print("correct")
                    if txt.startswith('SIZE'):
                        print('size?')
                        tmp = txt.split()
                        size = int(tmp[1])

                        print ('got size')

                        sock.sendall("GOT SIZE".encode(encoding = 'UTF-16',errors = 'strict'))
                        print ("done sending")
                    elif txt.startswith('BYE'):
                        print('bye')
                        sock.shutdown()

                    else :
                        print('image maybe')
                        myfile = open(basename, 'wb')
                        myfile.write(data)
                        print("saved")
                        data = sock.recv(40960000)
                        if not data:
                            myfile.close()
                            break
                        myfile.write(data)
                        myfile.close()
                        print("actually saved")
                        sock.sendall("GOT IMAGE".encode(encoding = 'UTF-16',errors = 'strict'))
                        sock.shutdown(socket.SHUT_RDWR)
            except Exception as inst:
 #               print('sock broke')
                print(inst)
                sock.close()
                connected_clients_sockets.remove(sock)
                continue
        imgcounter += 1
server_socket.close()

and then here is the client code:

#!/usr/bin/env python

import random
import socket, select
from time import gmtime, strftime
from random import randint

image = "download-1.jpg"

HOST = 'my_ip'
PORT = 9999

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = (HOST, PORT)
sock.connect(server_address)

try:

    # open image
    myfile = open(image, 'rb')
    bytess = myfile.read()
    print(bytess)
    size = len(bytess)
    message = "SIZE " + str(size)
    # send image size to server
    #sock.sendall("SIZE %s",size)
    sock.sendto(message.encode(encoding = 'UTF-16',errors = 'strict'),(HOST, PORT))

    answer = sock.recv(4096).decode(encoding = 'UTF-16',errors = 'strict')
    print ('answer = ',answer)

    # send image to server
    if answer == 'GOT SIZE':
        sock.sendto(bytess,(HOST, PORT))
        #sock.sendall(bytess)

        # check what server send
        answer = sock.recv(4096)
        print ('answer = ',answer)

        if answer == 'GOT IMAGE' :
            #sock.sendall("BYE BYE ")
            sock.sendto("BYE BYE ".encode(encoding = 'UTF-16',errors = 'strict'),(HOST, PORT))
            print ('Image successfully send to server')

    myfile.close()

finally:
    sock.close()

I have both scripts on both devices, and I first use the server on the rpi to transfer the picture there, and then I run the server on my local computer to transfer it back. Any idea why this happens?

You can't assume that you will get all the data in a single receive. Most likely you will only get ~1500 bytes at a time. You gave to loop. — user207421, Apr 26 '20 at 01:01
Does this answer your question? [Python Socket Receive Large Amount of Data](https://stackoverflow.com/questions/17667903/python-socket-receive-large-amount-of-data) — Steffen Ullrich, Apr 26 '20 at 06:25
I tried putting it in a loop like so: while not data: data = sock.recv(40960000), but had similar results — figbar, Apr 27 '20 at 02:23

score 1 · Accepted Answer · answered Oct 12 '20 at 06:07

Did you solve this yet? The only solution I could find was to reduce the buffer size. Using buffer size 1024 / 512 reduced the amount of missing data.

I used socket.sendfile() to send the file and received it in a loop this way:

buffer_size = 4096
recv_size = 0
with open(f"{downloads_path}/DataShare/{file_name}", 'wb') as file:
    while True:
        if file_size < buffer_size:
            buffer_size = file_size
        packet = self.socket.recv(buffer_size)
        file.write(packet)
        recv_size += buffer_size

        delta = file_size - recv_size
        buffer_size = delta if delta < buffer_size else buffer_size

        if recv_size == file_size or buffer_size == 0:
            break

I guess, receiving a large amount of data (4096 bytes) is the reason behind missing data as some data may get lost during transportation, and with smaller bytes to receive it's easy to not lose much.

Python3 Socket Server image data missing

1 Answers1