Merge Conference Video and Audio call output using hstack ffmpeg

Question

having two videos and two audio files

Input #0, matroska,webm, from 'first.mkv':
  Metadata:
    encoder         : GStreamer matroskamux version 1.8.1.1
    creation_time   : 2017-10-16 14:13:15
  Duration: 00:06:01.24, start: 3.817000, bitrate: 1547 kb/s
    Stream #0:0(eng): Video: vp8, yuv420p, 640x480, SAR 1:1 DAR 4:3, 16.75 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      title           : Video
Input #1, matroska,webm, from 'second.mkv':
  Metadata:
    encoder         : GStreamer matroskamux version 1.8.1.1
    creation_time   : 2017-10-16 14:13:24
  Duration: 00:05:49.79, start: 13.509000, bitrate: 810 kb/s
    Stream #1:0(eng): Video: vp8, yuv420p, 640x480, SAR 1:1 DAR 4:3, 1k tbr, 1k tbn, 1k tbc (default)
    Metadata:
      title           : Video
Input #2, matroska,webm, from 'first.mka':
  Metadata:
    encoder         : GStreamer matroskamux version 1.8.1.1
    creation_time   : 2017-10-16 14:13:15
  Duration: 00:06:01.30, start: 3.786000, bitrate: 46 kb/s
    Stream #3:0(eng): Audio: opus, 48000 Hz, stereo, fltp (default)
    Metadata:
      title           : Audio
Input #3, matroska,webm, from 'second.mka':
  Metadata:
    encoder         : GStreamer matroskamux version 1.8.1.1
    creation_time   : 2017-10-16 14:13:24
  Duration: 00:05:50.61, start: 13.498000, bitrate: 50 kb/s
    Stream #2:0(eng): Audio: opus, 48000 Hz, stereo, fltp (default)
    Metadata:
      title           : Audio

above files are output of video conference call, want to merge all files together and show as side by side video.

start time of video and audio are different, want to sync the video and audio respectively and merge the video side by side.

Initially used the following command to merge two videos

ffmpeg -i first.mkv -i second.mkv -filter_complex "
[0:v]scale=320:240,pad=325:240,setsar=1[l];[1:v]scale=320:240,setsar=1[r];
[l][r]hstack" -c:v libx264 -preset ultrafast -crf 0 merged.mp4

After that use the following command to merge as suggested by @mulvya

ffmpeg -ss 00:00:09.692 -i first.mkv -i second.mkv -i first.mka -i second.mka -filter_complex "[0:v]scale=320:240,pad=325:240,setsar=1[l];[1:v]scale=320:240,setsar=1[r];[l][r]hstack=shortest=1[v];[3]adelay=9712|9712[3a];[2][3a]amerge[a]" -map '[v]' -map '[a]' -c:v libx264 -preset slower -crf 0 -c:a aac -ac 2 merged.mp4

for the -ss value taken the difference in video start time and adelay value taken the difference in audio start time

Sample test conference files

1. https://drive.google.com/open?id=0ByVMq5U43FGlbXpXR3JtSnFTaWM

2. https://drive.google.com/open?id=0ByVMq5U43FGlbENVRWlTWktQb3M

3. https://drive.google.com/open?id=0ByVMq5U43FGldndlZDNpNWxWY2M

4. https://drive.google.com/open?id=0ByVMq5U43FGlei1oRjNKeXRZbE0

now facing audio sync issues and second audio hearing low.

Expected result is first video and second video merged side by side and audio should sync with merged video.

Now I can able to get desired output using the below command

ffmpeg -i first.mkv -i second.mkv -i first.mka -i second.mka -filter_complex "[0]scale=320:240,pad=645:240,setsar=1[l];[1]scale=320:240,setpts=PTS-STARTPTS+9.723/TB,setsar=1[1v];[l][1v]overlay=x=325[v];[3]adelay=9712|9712[1a];[2]adelay=31|31[2a];[2a][1a]amerge=inputs=2[a]" -map '[v]' -map '[a]' -c:v libx264 -preset slower -crf 0 -c:a aac -ac 2 merged.mp4

but again facing following issues

1. Second Video not encoded properly stuck in middle and playing.
2. Audio Sync issues.
3. Conversion process is slow. how can be above work done using hstack?.

any suggestions or help?

Answer 1

Use

ffmpeg -i first.mkv -i second.mkv -i 1st.wav -i 2nd.wav -filter_complex "
[0:v]scale=320:240,pad=325:240,setsar=1[l];[1:v]setpts=PTS+6/TB,scale=320:240,setsar=1[r];
[l][r]hstack=shortest=1,drawbox=325:0:c=black:t='max':enable='lt(t,6)'[v];
[3]adelay=6000|6000[3a];[2][1a]amerge[a]" -map '[v]' -map '[a]' -c:v libx264 -preset ultrafast -crf 0 -c:a aac -ac 2 merged.mp4

---

UPDATE: For the current set of uploaded files, use the command below

ffmpeg -i first.mkv -i second.mkv -c:a libopus -i first.mka -c:a libopus -i second.mka
 -filter_complex
        "[0]fps=30,scale=320:240,pad=645:240,setsar=1[l];
         [1]fps=30,scale=320:240,setpts=PTS-STARTPTS+3.981/TB,setsar=1[1v];
         [l][1v]overlay=x=325[v];
         [2]atrim=0.025,asetpts=PTS-STARTPTS,aresample=async=1,pan=stereo|c0=c0|c1=c1[2a];
         [3]atrim=0.012,asetpts=PTS-STARTPTS,aresample=async=1,adelay=3981|3981,pan=stereo|c0=c0|c1=c1[3a];
         [2a][3a]amerge=inputs=2[a]"
 -map "[v]" -map "[a]" -c:v libx264 -preset fast -crf 10 -c:a aac -ac 2 merged.mp4

first.mkv start time is 07.930
first.mka start time is 07.905
second.mkv start time is 11.911
second.mka start time is 11.899

These audio files use codec features not implemented in ffmpeg's native Opus decoder, so external decoder is forced.

The first video file fps isn't detected so for both videos, I force the same framerate using fps filter.

The atrim filters are added for both audios with value equal to difference of start times between audio and corresponding video. The timestamps of the trimmed audio has to be reset us9ing asetpts. Any gaps in the recording have to be filled out using aresample. Only the late-starting a/v should have its audio adelay-ed, with value equal to difference of video start times. pan filters added to fix channel layout, required for amerge.

preset changed to fast. CRF increased to 10 (0 is a large waste of space).