UnicodeDecodeError during file transfer between Linux and Windows using Python socket programming

Question

I am trying to send an image file from Raspberry Pi (the client) to the Laptop (the server). When I run client.py on Raspberry Pi (linux OS) and server.py on laptop (windows OS) connected in LAN, I get the following error message on laptop (server side).

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 5: invalid start byte

On the other hand, I don't get any error and file is transferred successfully, when I run both scripts(server.py and client.py) on the same windows laptop.

server.py code is given below:

import os
import socket

HOST = '192.168.2.80' #Private IP address of laptop
PORT = 3322
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((HOST, PORT))

print("STATUS_MSG: This-Is-Laptop")
print("STATUS_MSG: Awaiting-Connection-From-Client")
server.listen()

try:
    communication_socket, addrs_of_client = server.accept()
    print(f"STATUS_MSG: Connection-Established-To-Client-IP-{addrs_of_client}")
except:
    print("STATUS_MSG: Unable-To-Accept-Connection")
    exit(0) 

file_name = communication_socket.recv(1024).decode()
print(f"incoming file name = {file_name}")
file_size = communication_socket.recv(1024).decode()
print(f"incoming file size = {file_size}")

file = open("./recvt/" + file_name, "wb")
file_bytes = b""

done = False

while not done:
    data = communication_socket.recv(1024)
    if file_bytes[-5:] == b"<END>":
        done = True
    else:
        file_bytes += data

file.write(file_bytes)
file.close()
print("File Received Successfully")
communication_socket.close()
server.close()

client.py code is given below:

import os
import socket

HOST = '192.168.2.80' #IP of the server
PORT = 3322
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

try:
    client.connect((HOST, PORT))
    print(f"STATUS_MSG: Connected-Successfully-To-Server-IP-{HOST}")
except:
    print("STATUS_MSG: Unable-To-Connect-To-Server")
    exit(0) # to end the program

# Getting file details.
file_name = "image1.jpg"
file_size = os.path.getsize(file_name)

client.send(file_name.encode())
client.send(str(file_size).encode())

# Reading file and sending data
file = open(file_name, "rb")
data = file.read()
client.sendall(data)
client.send(b"<END>")

file.close()
client.close()

The output when both scripts run on windows laptop:

STATUS_MSG: This-Is-Laptop
STATUS_MSG: Awaiting-Connection-From-Client
STATUS_MSG: Connection-Established-To-Client-IP-('192.168.2.80', 58646)
incoming file name = image1.jpg
incoming file size = 81377
File Received Successfully

The output when the script client.py runs on raspberry pi and server.py on laptop.

STATUS_MSG: This-Is-Laptop
STATUS_MSG: Awaiting-Connection-From-Client
STATUS_MSG: Connection-Established-To-Client-IP-('192.168.2.197', 59062)
incoming file name = image1.jpg
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
Input In [2], in <cell line: 26>()
     24 file_name = communication_socket.recv(1024).decode()
     25 print(f"incoming file name = {file_name}")
---> 26 file_size = communication_socket.recv(1024).decode()
     27 print(f"incoming file size = {file_size}")
     29 file = open("./recvt/" + file_name, "wb")

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 5: invalid start byte

Please guide me how can I correct the encoding/decoding issue here as I want to extend this script to transfer multiple files in a loop back and forth between laptop(windows OS) and Raspberry Pi(raspbian OS). Thank you.

Does this answer your question? [Linux socket: How to make send() wait for recv()](https://stackoverflow.com/questions/19794764/linux-socket-how-to-make-send-wait-for-recv) — Sören, Oct 02 '22 at 15:00

Abhi747 · Accepted Answer · 2022-10-02T16:33:55.997

-1

file_name = communication_socket.recv(1024).decode()

here this call sockets tryes to fill 1024 bytes buffer. it will break only if (1) socket will change state from readable to writeable or (2) connection is closed.

client.send(file_name.encode())
client.send(str(file_size).encode())

# Reading file and sending data
file = open(file_name, "rb")
data = file.read()
client.sendall(data)

but here code is only sending data. so fist call of recv will end after 1024 bytes not after reciving just name of file.

reason for error: in that first 2048 bytes (1024 form each call) some bytes are from file also which is binary and can't be decoded in utf-8.

modifications: first thing you can send confermation one after reciving name and another after reciving size. or you can just use seperators between name, size and data.

EDIT: if you want to just transfer image/file you can use built in python's http server.python3 -m http.server {port} this will start http server at given port and serves files in current working directory

EDIT2:

if file_bytes[-5:] == b"<END>":

for same reason this will never happen, because <END> will come end of file contents not as a separate msg. in client you need to add confirmation after file ends

edited Oct 02 '22 at 16:33

answered Oct 02 '22 at 14:56

Abhi747

304
3
9

2

recv *can* return with fewer bytes than specified, even if the connection is still open. – Sören Oct 02 '22 at 15:17
@Sören yes, but only if sockets state changes (as mentioned in answer) – Abhi747 Oct 02 '22 at 15:36
You might want to elaborate on that point. Why does it matter if a socket is writable if the only thing we're doing is read from it? – Sören Oct 02 '22 at 15:44
@Sören readable state indicates data is being sent from other side and writeable state indicates other end is receiving data. now suppose u call recv and other end is sending data after that other end stopped sending data and called recv (i.e., it changes state from readable to writeable) then your recv call breaks although its buffer is not filled. terms are little bit confusing, but I hope it is clear – Abhi747 Oct 02 '22 at 15:57
1

Thank you very much for your help. I added the following lines to the code and it resolved the issue. `communication_socket.send("File Name Received".encode())` `communication_socket.send("File Size Received".encode())` – Ali Furqan Oct 02 '22 at 16:00
I am also trying to send acknowledgment from the server that the complete file is received using this line: `communication_socket.send("Complete File Received".encode())` I am adding this line after `file.close()` and before `communication_socket.close()` in server.py but now the program keeps on running and never stops. On the client side I added `print(f"SERVER_ACK: {client.recv(100).decode()}")` before `client.close()` and the client never receives "Complete File Received" acknowledgement from server and also program keeps on running. Can you please also help me on this. – Ali Furqan Oct 02 '22 at 16:16
2

Changing readable/writable is not considered. The client could have a thread sending and receiving at the same time. What matters is TCP is a byte stream and it will receive the bytes in the same order sent. There must be an agreement between sender and receiver to parse the byte stream. – Mark Tolonen Oct 02 '22 at 23:47
`recv(1024)` will return on a blocking socket when at least 1 or at most 1024 bytes are available or return 0 when the socket is closed. The key is “available”. Intermediate buffers can be full or network issues can cause delays. Buffer received data and extract complete messages. – Mark Tolonen Oct 02 '22 at 23:59

UnicodeDecodeError during file transfer between Linux and Windows using Python socket programming

1 Answers1