Sending frame from Java Client to C++ Server via UDP

Question

I am trying to implement a UDP server with C++ to receive frames from the client. I am using this question as reference. Only difference is that I have a Java Client that sends the frames via udp:

Java client that sends image via UDP:

byte[] buffer = GetImageByte(FrameData,384,288,false); //GetImageByte is a function that returns a raw frame
DatagramSocket clientSocket = new DatagramSocket();
InetAddress IPAddress = InetAddress.getByName("127.0.0.1");
DatagramPacket packet = new DatagramPacket(buffer, buffer.length, IPAddress, 9999);
clientSocket.send(packet); 

C++ server that receives udp image:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h> 
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <iostream>

using namespace cv;
using namespace std;

void error(const char *msg)
{
    perror(msg);
    exit(1);
}

int main(int argc, char *argv[])
{
    int sockfd, newsockfd, portno;
    socklen_t clilen;
    char buffer[1024];
    struct sockaddr_in serv_addr, cli_addr;
    int n;
    int height = 384;
    int width= 288;


    sockfd = socket(AF_INET, SOCK_DGRAM, 0);
    serv_addr.sin_family = AF_INET;
    serv_addr.sin_addr.s_addr = INADDR_ANY;
    serv_addr.sin_port = htons(9999);


    Mat img = Mat::zeros( height,width, CV_8UC3);
    int imgSize = img.total()*img.elemSize();
    uchar sockData[imgSize];
    int bytes = 0; 
    for (int i = 0; i < imgSize; i += bytes) {
        if ((bytes = recv(sockfd, sockData +i, imgSize  - i, 0)) == -1)  {
            break;
        }
        
    }
    int ptr=0;
    for (int i = 0;  i < img.rows; i++) {
        for (int j = 0; j < img.cols; j++) {                                     
            img.at<cv::Vec3b>(i,j) = cv::Vec3b(sockData[ptr+ 0],sockData[ptr+1],sockData[ptr+2]);
            ptr=ptr+3;
    }
}


    Mat image(384,288,CV_8UC3,*buffer);
    close(newsockfd);
    close(sockfd);
    
    
    imshow( "Server", image);  
    waitKey(0);
    return 0; 
}

What I am trying to do is simply receive a frame and show it using OpenCV. The problem is that the code compiles successfully but no errors or any helpful message is shown in the console whatsoever. I tested sending the image to a UDP server in Python it works. I'd like to know how can I receive a frame via UDP using C++ and show it on screen using OpenCv?

Answer 1

After a brief consultation with OpenCV reference, the above appears to be working with 384x288 three channel image, which adds up to 324kb worth of data.

The Java client appears to be attempting to construct a single datagram packet, of this size, then send it all at once.

The chances of being able to send a single UDP 324kb packet, anywhere, are slim to none. The maximum size of a single UDP socket is unspecified and depends on the underlying network socket; 508 bytes is the generally accepted maximum size, with modern network protocols.

This is implied by your C++ code, which uses a loop to attempt to read the entire frame (which is also somewhat naive, for the reasons mentioned shortly, but let's set this aside for now). You will need to add appropriate logic to the sending code to break it up.

Once you do this you will run into the next problem: UDP gives you absolutely no guarantees, whatsoever, that any UDP packet you send will be received. UDP, by definition, lacks any kind of retransmission logic, and is not a guaranteed delivery transport layer. This is explained in every textbook that focuses on network programming.

Common media streaming platforms achieve more or less reliable streaming over UDP through a combination of sophisticated rate limiting, detecting and adapting to the available bandwidth, and using complicated logic that deals with occasional dropped packets; minimizing the impact of dropped packets with some level of redundancy (a single dropped frame in a Youtube stream will be hardly noticed, as well as a slight, temporary, degradation in the audio stream). The chances of successfully receiving every one of the packets that adds up to the 324kb image, when the Java sender simply blasts them out, as fast as it can, are not very promising.

So, you have lot of work cut out for you, here, inventing and implementing your own versions of all of these underlying algorithms, for implementing some kind of a throttled streaming mechanism on top of UDP. It will be frankly easier to switch to TCP, get it working with TCP, then attempt to implement all of this on top of UDP after the fact.

One final note:

    uchar sockData[imgSize];

This is not valid C++ either. Variable length arrays are non-standard C++, and are only a non-standard C++ extension that your compiler supports. Additionally, this will eat the aforementioned 324kb out of your stack, all in one gulp.

Although modern C++ implementations will typically have so much stack space available: getting into a habit of doing this will eventually lead to tears. Don't do this, use std::vector.