Losing Data in TCP socket connectivity

Question

I am working on socket programming.Here server sends random number of bytes to the machine who connected to server.This random number simulate that server is also receving data from somwhere else and data length can vary much and we are not sure how much.

I am running a server and client both on the same machine. I was expecting no data loss, but to my surprise I can see data loss is happening here too. I must be doing something wrong in my code. I have tried to find but I didn't found anything suspicious. I can just guess the length of data may be a problem, however still I am not sure.

From the server I am first sending the length of the message to the client and then sending the actual message so that I can be sure I have received complete message.

I am connecting to server 100 times and many times the data loss is happening.

Below is my server code:

import socket
import struct
import random as rand


sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# bind the socket to a specific address and port
server_address = ('localhost', 12345)
sock.bind(server_address)

sock.listen(1)

while True:
    message = "12345" * rand.randint(100000, 200000)
    connection, client_address = sock.accept()
    message_in_bytes=message.encode()
    length = len(message_in_bytes)
    print(client_address, "connected and message length is ",length)
    length_bytes = struct.pack("<I", length)
    connection.send(length_bytes + message_in_bytes)
    connection.close()

Below is client code:

import socket
import struct

def connect_server():
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_address = ('localhost', 12345)
    sock.connect(server_address)
    message_length_bytes=sock.recv(4)
    length_of_message = struct.unpack("<I",message_length_bytes)[0]
    whole_message=sock.recv(length_of_message)
    sock.close()
    if length_of_message == len(whole_message):
        return "success"
    else:
        return "failed"

stats={'success':0,'failed':0}

for _ in range(100):
    stats[connect_server()] +=1

print(stats)

Answer 1

send() is not guaranteed to send all the data, limited by the implementation of buffers in the networks stack. recv(n) is also not guaranteed to return n bytes. In both cases check the return value and send/receive the remaining data.

sendall() is an alternative that will loop and send all the data provided.

For receiving, buffer the data received until a complete message is received. socket.makefile() is a function that will wrap the socket in a file-like object, where I/O operations like read(n) will return n bytes unless the socket is closed, and readline() will read until a newline character.

Example:

server.py

import socket
import struct
import random as rand

sock = socket.socket()
server_address = ('localhost', 12345)
sock.bind(server_address)
sock.listen()

while True:
    connection, client_address = sock.accept()
    # closes socket when with is exited
    with connection:
        message = b'12345' * rand.randint(100000, 200000)
        length = len(message)
        print(f'Sending {length}-byte message...')
        length_bytes = struct.pack('<I', length)
        connection.sendall(length_bytes + message) # Use sendall

client.py

import socket
import struct

def connect_server():
    with socket.socket() as sock:
        server_address = ('localhost', 12345)
        sock.connect(server_address)
        with sock.makefile('rb') as rfile: # wrap in file-like object
            message_length_bytes = rfile.read(4)
            length_of_message = struct.unpack('<I', message_length_bytes)[0]
            whole_message = rfile.read(length_of_message)
            if length_of_message == len(whole_message):
                return 'success'
            else:
                return 'failed'

stats = {'success': 0, 'failed': 0}

for _ in range(100):
    stats[connect_server()] += 1

print(stats)

Output (server):

Sending 604035-byte message...
Sending 949255-byte message...
Sending 838075-byte message...
 ...
Sending 756820-byte message...
Sending 912215-byte message...
Sending 673175-byte message...

Output (client):

{'success': 100, 'failed': 0}

Answer 2

Then problem is in the sent length. Your message length may be up to 1 000 000 bytes. It will exceed the length of kernel buffer.
In that case send() return the length actually transferred to buffer and you must re-execute send() for the extra portion.
I did a test on my machine. The length of message was 542610 but send(0 returned 524288 (512KiB).
You must loop on send() until everything is transferred to kernel buffer.

rand = rand.randint( 100000,200000)
length_bytes = struct.pack("<I", rand * 5)
whole_message = length_bytes + ('12345' * rand).encode() 
actual_length = 0
while actual_length < len( whole_message:
  nb = send( whole_message[actual_length:])
  actual_length += nb
'''
Note: the length of kernel buffer may be different with your distrib.