Python TCP socket send receive large delay

Question

I used python socket to make a server on my Raspberry Pi 3 (Raspbian) and a client on my laptop (Windows 10). The server stream images to the laptop at a rate of 10fps, and can reach 15fps if I push it. The problem is when I want the laptop to send back a command based on the image, the frame rate drop sharply to 3fps. The process is like this:

Pi send img => Laptop receive img => Quick process => Send command based on process result => Pi receive command, print it => Pi send img => ...

The process time for each frame does not cause this (0.02s at most for each frame), so currently I am at a loss as to why the frame rate drop so much. The image is quite large, at around 200kB and the command is only a short string at 3B. The image is in matrix form and is pickled before sending, while the command is sent as is.

Can someone please explain to me why sending back such a short command would make the frame rate drop so much? And if possible, a solution for this problem. I tried making 2 servers, one dedicated to sending images and one for receiving command, but the result is the same.

Server:

import socket
import pickle
import time
import cv2
import numpy as np
from picamera.array import PiRGBArray
from picamera import PiCamera
from SendFrameInOO import PiImageServer

def main():
    # initialize the server and time stamp
    ImageServer = PiImageServer()
    ImageServer2 = PiImageServer()
    ImageServer.openServer('192.168.0.89', 50009)
    ImageServer2.openServer('192.168.0.89', 50002)

    # Initialize the camera object
    camera = PiCamera()
    camera.resolution = (320, 240)
    camera.framerate = 10 # it seems this cannot go higher than 10
                          # unless special measures are taken, which may
                          # reduce image quality
    camera.exposure_mode = 'sports' #reduce blur
    rawCapture = PiRGBArray(camera)

    # allow the camera to warmup
    time.sleep(1)

    # capture frames from the camera
    print('<INFO> Preparing to stream video...')
    timeStart = time.time()
    for frame in camera.capture_continuous(rawCapture, format="bgr",
                                           use_video_port = True):
        # grab the raw NumPy array representing the image, then initialize 
        # the timestamp and occupied/unoccupied text
        image = frame.array 
        imageData = pickle.dumps(image) 
        ImageServer.sendFrame(imageData) # send the frame data

        # receive command from laptop and print it
        command = ImageServer2.recvCommand()
        if command == 'BYE':
            print('BYE received, ending stream session...')
            break
        print(command)

        # clear the stream in preparation for the next one
        rawCapture.truncate(0) 

    print('<INFO> Video stream ended')
    ImageServer.closeServer()

    elapsedTime = time.time() - timeStart
    print('<INFO> Total elapsed time is: ', elapsedTime)

if __name__ == '__main__': main()

Client:

from SupFunctions.ServerClientFunc import PiImageClient
import time
import pickle
import cv2

def main():
    # Initialize
    result = 'STP'
    ImageClient = PiImageClient()
    ImageClient2 = PiImageClient()

    # Connect to server
    ImageClient.connectClient('192.168.0.89', 50009)
    ImageClient2.connectClient('192.168.0.89', 50002)
    print('<INFO> Connection established, preparing to receive frames...')
    timeStart = time.time()

    # Receiving and processing frames
    while(1):
        # Receive and unload a frame
        imageData = ImageClient.receiveFrame()
        image = pickle.loads(imageData)        

        cv2.imshow('Frame', image)
        key = cv2.waitKey(1) & 0xFF

        # Exit when q is pressed
        if key == ord('q'):
            ImageClient.sendCommand('BYE')
            break

        ImageClient2.sendCommand(result)

    ImageClient.closeClient()

    elapsedTime = time.time() - timeStart
    print('<INFO> Total elapsed time is: ', elapsedTime)
    print('Press any key to exit the program')
    #cv2.imshow('Picture from server', image)
    cv2.waitKey(0)  

if __name__ == '__main__': main()

PiImageServer and PiImageClient:

import socket
import pickle
import time

class PiImageClient:
    def __init__(self):
        self.s = None
        self.counter = 0

    def connectClient(self, serverIP, serverPort):
        self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.s.connect((serverIP, serverPort))

    def closeClient(self):
        self.s.close()

    def receiveOneImage(self):
        imageData = b''
        lenData = self.s.recv(8)
        length = pickle.loads(lenData) # should be 921764 for 640x480 images
        print('Data length is:', length)
        while len(imageData) < length:
            toRead = length-len(imageData)
            imageData += self.s.recv(4096 if toRead>4096 else toRead)
            #if len(imageData)%200000 <= 4096:
            #    print('Received: {} of {}'.format(len(imageData), length))
        return imageData

    def receiveFrame(self):        
        imageData = b''
        lenData = self.s.recv(8) 
        length = pickle.loads(lenData)
        print('Data length is:', length)
        '''length = 921764 # for 640x480 images
        length = 230563 # for 320x240 images'''
        while len(imageData) < length:
            toRead = length-len(imageData)
            imageData += self.s.recv(4096 if toRead>4096 else toRead)
            #if len(imageData)%200000 <= 4096:
            #    print('Received: {} of {}'.format(len(imageData), length))
        self.counter += 1
        if len(imageData) == length: 
            print('Successfully received frame {}'.format(self.counter))                
        return imageData

    def sendCommand(self, command):
        if len(command) != 3:
            print('<WARNING> Length of command string is different from 3')
        self.s.send(command.encode())
        print('Command {} sent'.format(command))


class PiImageServer:
    def __init__(self):
        self.s = None
        self.conn = None
        self.addr = None
        #self.currentTime = time.time()
        self.currentTime = time.asctime(time.localtime(time.time()))
        self.counter = 0

    def openServer(self, serverIP, serverPort):
        print('<INFO> Opening image server at {}:{}'.format(serverIP,
                                                            serverPort))
        self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.s.bind((serverIP, serverPort))
        self.s.listen(1)
        print('Waiting for client...')
        self.conn, self.addr = self.s.accept()
        print('Connected by', self.addr)

    def closeServer(self):
        print('<INFO> Closing server...')
        self.conn.close()
        self.s.close()
        #self.currentTime = time.time()
        self.currentTime = time.asctime(time.localtime(time.time()))
        print('Server closed at', self.currentTime)

    def sendOneImage(self, imageData):
        print('<INFO> Sending only one image...')
        imageDataLen = len(imageData)
        lenData = pickle.dumps(imageDataLen)
        print('Sending image length')
        self.conn.send(lenData)
        print('Sending image data')
        self.conn.send(imageData)

    def sendFrame(self, frameData):
        self.counter += 1
        print('Sending frame ', self.counter)
        frameDataLen = len(frameData)
        lenData = pickle.dumps(frameDataLen)        
        self.conn.send(lenData)        
        self.conn.send(frameData)

    def recvCommand(self):
        commandData = self.conn.recv(3)
        command = commandData.decode()
        return command

Answer 1

I believe the problem is two-fold. First, you are serializing all activity: The server is sending a complete image, then instead of continuing on to send the next image (which would better fit the definition of "streaming"), it is stopping, waiting for all bytes of the previous image to make themselves across the network to the client, then for the client to receive all bytes of the image, unpickle it, send a response and for the response to then make its way across the wire to the server.

Is there a reason you need them to be in lockstep like this? If not, try to parallelize the two sides. Have your server create a separate thread to listen for commands coming back (or simply use select to determine when the command socket has something to receive).

Second, you are likely being bitten by Nagle's algorithm (https://en.wikipedia.org/wiki/Nagle%27s_algorithm), which is intended to prevent sending numerous packets with small payloads (but lots of overhead) across the network. So, your client-side kernel has gotten your three-bytes of command data and has buffered it, waiting for you to provide more data before it sends the data to the server (it will eventually send it anyway, after a delay). To change that, you would want to use the TCP_NODELAY socket option on the client side (see https://stackoverflow.com/a/31827588/1076479).