0

I have created an app to stream PC display to Android, and I have successfully created the code to get the PC Screen almost in real time. Now, I want to send this data (which is in Mat) to my Android device.

I have created Socket connections before and I know the basics of how to send data from PC to Android. I have tried saving each frame in png format, then sending it over Sockets, but it's quite slow and not much real time.

My question: What is the fastest approach to send OpenCV Mat from PC to Android in (near) real time?

On a side note: My PC code is written in C++ and Android code in Java. And when I see the to-be-streamed content of PC on my PC it's quite real time.

FadedCoder
  • 1,517
  • 1
  • 16
  • 36
  • What frame size? What sort of frame rate? Can you afford to drop frames when it can't keep up? Does it have to be lossless (as in compression)? How did you generate the png? To disk, or directly to memory? Did you play with the compression settings -- it's zlib underneath, and at higher compression levels that's unlikely to be fast enough. In general it's going to be a tradeoff between compression and speed. – Dan Mašek Apr 27 '16 at 13:41
  • @DanMašek The frame size can be the best (half HD?) that can be sent without lag. Frame rate should be above 20, and no, lossless isn't necessary, but will be good if it can be done. I generated to disk, (I am not currently sure how to write to memory with `imwrite`) and I kept compression level at 3/4 after some experimenting. – FadedCoder Apr 27 '16 at 15:48
  • To write directly to a memory buffer, use the [`imencode`](http://docs.opencv.org/3.0-beta/modules/imgcodecs/doc/reading_and_writing_images.html#imencode) function. (And there's also the reverse `imdecode`). – Dan Mašek Apr 27 '16 at 15:58
  • 1
    720p, that's gonna be 1280x720x3=2.7MiB/frame raw, at 20 FPS 54 MiB (or about 420 mBit). You could push that through a small gigabit LAN reasonably well, but my guess is that the android device is mobile, so that's already a bit too much. That means you need some serious compression ratio to get it to reasonable levels. Generic lossless compression of isolated frames (like PNG) won't cut it, especially at speeds that are close to what you're asking for. So something lossy, but quick, perhaps MJPEG. Even better some real video codec that can take advantage of similarities between frames. – Dan Mašek Apr 27 '16 at 16:34
  • I am gonna try out MJPEG soon, thanks, and I have found out level 3 to be best with PNG (Resolution - 540 x 960), the size was around 200 KB per frame without much loss in speed. – FadedCoder Apr 27 '16 at 16:46
  • 1
    If you have beefy PC with a nice GPU, you could try to compress h264 with [CUDA using openCV](http://docs.opencv.org/trunk/d0/d61/group__cudacodec.html#gsc.tab=0). This one let's you use callbacks so you don't need to encode into file, but can stream it directly. Alternately, hack your own in-memory encoder starting from the OpenCV code (e.g. [MJPEG](https://github.com/Itseez/opencv/blob/2f4e38c8313ff313de7c41141d56d945d91f47cf/modules/videoio/src/cap_mjpeg_encoder.cpp)) It seems you just need to change class BitStream to call a callback instead of writing to file. – Dan Mašek Apr 27 '16 at 16:46
  • Regarding PNG -- yeah, level 3 is last of the "fast" modes (without lazy matching). 200KB is quite nice, it really depends on what's in the video. – Dan Mašek Apr 27 '16 at 16:51

1 Answers1

0

After lots of trying, I could mostly solve my problem. Here is the code for the MJPEG server - Error while using QTcpSocket. (Mostly everything has been fixed except a small problem).

Community
  • 1
  • 1
FadedCoder
  • 1,517
  • 1
  • 16
  • 36