0

Description:

I have a Raspberry PI controlling a small vehicle, and it has a RealSense camera attached to it. What I need to do is to send a live stream from the camera to an HTML page/NodeJS server hosted on Google App Engine, so that the user can see the stream on his page. The stream will be used to manually control the vehicle, so low latency is very important.

What I attempted:

  1. My current solution is just a simple socket connection using Socket.IO, I send a frame through the socket, decode it and display it in the page. The problem is that this method is extremely slow, and from what I understood not a good way to stream to a remote client, that is why I need to find a different way.

  2. I tried using uv4l, when I run the line uv4l --driver uvc --device-id "realsense camera id" it says the camera is recognized, but then immediately stops without any error. When I try to open the stream with my IP and click "call" I get the error "invalid input device". Could not find any solution for this problem.

  3. I also thought about using webRTC, I tried to follow this example (which is the closest I found to what I need): https://dev.to/whitphx/python-webrtc-basics-with-aiortc-48id , but it uses a Python server and I want to use my GAE/NodeJS server, and I'm struggling to figure out how to convert this code to use a python client and a NodeJS server.

If anyone can provide some information or advice I'd really appreciate it.

Natanel Birarov
  • 1
  • 3
  • What does your current architecture look like? What language/technology are you using to capture the camera feed to then pipe it to Socket.IO? WebRTC is definitely the best route to go – Matt Davis Jan 10 '22 at 18:39
  • @MattDavis right now I'm using pyrealsense2 to capture frames from the RealSense camera on the vehicle, send them through the socket to the GAE app and display them in an HTML tag. I know WebRTC is supposed to be better, but like I said in the post, I'm struggling to figure out how to connect the Python client (vehicle) and the HTML page (user) through App Engine. All examples I found had the JS side taking the video and the Python side acting as the server – Natanel Birarov Jan 10 '22 at 18:44
  • See also: https://stackoverflow.com/questions/37457972/low-latency-2s-live-video-streaming-html5-solutions/37475943#37475943 – Brad Feb 16 '22 at 03:26

1 Answers1

0

If want to control the vehicle, the latency is extremely important. I think the latency is better if about 100ms, and should not greater than 400ms if the network is jitter for a while.

The latency is introduced by everywhere, from your encoder on Raspberry PI, transfer to media server, and H5 player. Especially the encoder and player.

The best solution is use UDP based protocol like WebRTC:

    Raspberry PI                                   PC Chrome H5
Camera --> Encoder ---UDP--> Media Server --UDP---> WebRTC Player

So recommend to use WebRTC to encode and send the frame to media server, and H5 WebRTC player. You could test this solution by replace the encoder with H5 WebRTC publisher, the latency is about 100ms, please see this wiki.The arch is bellow:

    Raspberry PI                                   PC Chrome H5
Camera --> WebRTC ---UDP--> Media Server --UDP---> WebRTC Player

Note: The WebRTC stack is complex, so you could build from H5 to H5, test the latency, then move the media server from intranet to internet and test the latency, next replace the H5 publisher by your Raspberry PI and test the latency.

If want to run the solution ASAP, FFmpeg is a better encoder, to encode the frame from camera and package it as RTMP packet, then publish to media server by RTMP, finally play by H5 WebRTC player, please read this wiki. The latency is larger than WebRTC encoder, I think it might be around 600ms, but it should be OK to run the demo. The arch is bellow:

    Raspberry PI                                   PC Chrome H5
Camera --> FFmpeg ---RTMP--> Media Server --UDP---> WebRTC Player
                    or SRT

Note that SRT is also realtime protocol, about 200~500ms latency.

Note that you could also run media server on Raspberry PI, and use WebRTC player to play the stream from it, when they are in the same WiFi. The latency should be the minimum, because it's intranet transport.

Henry Ecker
  • 34,399
  • 18
  • 41
  • 57
Winlin
  • 1,136
  • 6
  • 25