Record multiple RTSP streams into a single file

Question

I need to record 4 RTSP streams into a single file.

Streams must be placed into the video in this way:

 ---------- ---------- 
|          |          |
| STREAM 1 | STREAM 2 |
|          |          |
|----------|----------|
|          |          |
| STREAM 3 | STREAM 4 |
|          |          |
 ---------- ----------

I need to synchronize these live streams with about ~1 second accuracy. This is challenging because streams have variable framerate (FPS).

I have tried ffmpeg but streams are not synchronized. Here is the code:

ffmpeg \
  -i "rtsp://IP-ADDRESS/cam/realmonitor?channel=1&subtype=00" \
  -i "rtsp://IP-ADDRESS/live?real_stream" \
  -i "rtsp://IP-ADDRESS/live?real_stream" \
  -i "rtsp://IP-ADDRESS/live?real_stream" \
  -filter_complex " \
    nullsrc=size=1920x1080 [base]; \
    [0:v] scale=960x540 [video0]; \
    [1:v] scale=960x540 [video1]; \
    [2:v] scale=960x540 [video2]; \
    [3:v] scale=960x540 [video3]; \
    [base][video0] overlay=shortest=1:x=0:y=0 [tmp1]; \
    [tmp1][video1] overlay=shortest=0:x=960:y=0 [tmp2]; \
    [tmp2][video2] overlay=shortest=0:x=0:y=540 [tmp3]; \
    [tmp3][video3] overlay=shortest=0:x=960:y=540 [v]; \
    [0:a]amix=inputs=1[a]" \
  -map "[v]" -map "[a]" -c:v h264 videos/test-combine-cams.mp4

Is there a way to combine and synchronize streams in ffmpeg or using other utilities like: vlc, openRTSP, OpenCV?

@nathancy I am running this code on a server without UI. OpenCV is very good library, but I found that it doesn’t handle audio streams. — John, Dec 15 '19 at 14:31
Is the problem due to having multiple audio sources or the variable video frame rate? If it works with a single audio source, precombine them using separate invocation(s) of ffmpeg to a single simple format and combine with the video. — karmakaze, Dec 21 '19 at 18:33
@karmakaze We have a single audio source (STREAM 1). We tried to use ffmpeg to store each stream to a separate file, then combine them into a single file. But the problem is that some streams start to be recorded two-three secconds later and the final video is not sinchronised properly. Maybe it is possible to set timestamps to something like UNIX timestamp and then use these timestamps to sinchronise final video? — John, Jan 06 '20 at 11:54

shortcipher3 · Answer 1 · 2019-12-18T20:37:27.260

Have you tried gstreamer, it works with my rtsp streams.

gst-launch-1.0 -e rtspsrc location=rtsp_url1 ! rtph264depay ! h264parse ! decodebin ! videoconvert ! m.sink_0 \
               rtspsrc location=rtsp_url2 ! rtph264depay ! h264parse ! decodebin ! videoconvert ! m.sink_1 \
               rtspsrc location=rtsp_url3 ! rtph264depay ! h264parse ! decodebin ! videoconvert ! m.sink_2 \
               rtspsrc location=rtsp_url4 ! rtph264depay ! h264parse ! decodebin ! videoconvert ! m.sink_3 \
               videomixer name=m sink_1::xpos=1280 sink_2::ypos=720 sink_3::xpos=1280 sink_3::ypos=720 ! x264enc ! mp4mux ! filesink location=./out.mp4 sync=true

Of course you will need to add in your rtsp urls and adjust the videomixer xpos/ypos properties based on your video size (mine was 720p).

Before mixing you may want to run just one at a time to make sure you have all the dependencies installed correctly

gst-launch-1.0 rtspsrc location=rtsp_url1 ! rtph264depay ! h264parse ! decodebin ! x264enc ! mp4mux ! filesink location=./out.mp4 sync=true

I have not yet added the audio.

Record multiple RTSP streams into a single file

1 Answers1