Sending OpenCV::Mat image to websocket Java client

Question

I've a c++ websocket server, and I want to ti send an openCV image (cv::Mat) to my Android client.

I understood that I should use base64 string, but I can't find out how to do it from my openCV frames.

I don't know how to convert a cv::Mat to a bytearray.

Thank you

Answer 1

Hi you can use this below code which works form me

C++ Client

• Here we will send BGR raw byte in to socket by accessing Mat data pointer.

• Before sending make sure that Mat is continues otherwise make it continues.

   int sendImage(Mat frame){
   int  imgSize = frame.total()*frame.elemSize();
   int  bytes=0;
   int clientSock;
   const char*        server_ip=ANDROID_IP;
   int        server_port=2000;
   struct  sockaddr_in serverAddr;
   socklen_t  serverAddrLen = sizeof(serverAddr);
  
      if ((clientSock = socket(PF_INET, SOCK_STREAM, 0)) < 0) {
          printf("\n--> socket() failed.");
          return -1;
      }
  
      serverAddr.sin_family = PF_INET;
      serverAddr.sin_addr.s_addr = inet_addr(server_ip);
      serverAddr.sin_port = htons(server_port);
  
      if (connect(clientSock, (sockaddr*)&serverAddr, serverAddrLen) < 0) {
              printf("\n--> connect() failed.");
              return -1;
      }
  
      frame = (frame.reshape(0,1)); // to make it continuous
  
      /* start sending images */
      if ((bytes = send(clientSock, frame.data, imgSize, 0)) < 0){
          printf("\n--> send() failed");
          return -1;
       }
  
      /* if something went wrong, restart the connection */
      if (bytes != imgSize) {
        cout << "\n-->  Connection closed " << endl;
        close(clientSock);
        return -1;
       }
  
  return 0;
  

  }

Java Server

• You should know size of image going to receive.
• Receives stream from socket and convert to byte array.
• Convert byte array BGR and create Bitmap.

Code for Receiving byte array from C++ Server

    public static byte imageByte[];
    int imageSize=921600;//expected image size 640X480X3

    InputStream in = server.getInputStream();
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    byte buffer[] = new byte[1024];
    int remainingBytes = imageSize; //
    while (remainingBytes > 0) {
    int bytesRead = in.read(buffer);
     if (bytesRead < 0) {
     throw new IOException("Unexpected end of data");
     }
     baos.write(buffer, 0, bytesRead);

    remainingBytes -= bytesRead;

    }
    in.close();


    imageByte = baos.toByteArray();   
    baos.close();

Code to Convert byte array to RGB bitmap image

    int nrOfPixels = imageByte.length / 3; // Three bytes per pixel.
            int pixels[] = new int[nrOfPixels];
            for(int i = 0; i < nrOfPixels; i++) {
               int r = imageByte[3*i];
               int g = imageByte[3*i + 1];
               int b = imageByte[3*i + 2];

               if (r < 0) 
                  r = r + 256; //Convert to positive

               if (g < 0) 
                  g = g + 256; //Convert to positive

               if (b < 0) 
                  b = b + 256; //Convert to positive

               pixels[i] = Color.rgb(b,g,r);
            }
         Bitmap bitmap = Bitmap.createBitmap(pixels, 640, 480, itmap.Config.ARGB_8888);

Answer 2

Check this answer in question Serializing OpenCV Mat_ . If it is not a problem for you to use boost, it can solve your problem. Probably you will need some additional JNI magic on client side.

Answer 3

You can take into account which of the data are important for you: cols (number of columns), rows (number of columns), data (which contains the pixel information), type (data type and channel number).

You have to vectorize your matrix, because it is not neccessarrily continuous and take into account the variations of the size of a pixel in the memory.

Suppose:

cv::Mat m;

Then to allocate:

int depth; // measured in bytes
switch (m.depth())
{
     // ... you might check for all of the possibilities
     case CV_16U:
          depth = 2;
}
char *array = new char[4 + 4 + 4 + m.cols * m.rows * m.channels() * depth]; // rows + cols + type + data

And than write the header information:

int *rows = array;
int *cols = &array[4];
int *type = &array[8];

*rows = m.rows;
*cols = m.cols;
*type = m.type;

And finally the data:

char *mPtr;
for (int i = 0; i < m.rows; i++)
{
    mPtr = m.ptr<char>(i); // data type doesn't matter
    for (int j = 0; j < m.cols; j++)
    {
         array[i * rows + j + 3 * 4] = mPtr[j];
    }
}

Hopefully no bugs in the code.