Merge multiple audio and video files

Question

I am new to using ffmpeg and what I am trying to do is combine multiple video and audio files and have them being displayed in a mosaic.

What i currently have:

2 video files and 2 audio files, where 1 audio file belongs to 1 video file while the other audio file belongs to the other video file

input1.mkv
input1.mka
input2.mkv
input2.mka

What i'm trying to accomplish

1. Combine input1.mkv with input1.mka and input2.mkv with input2.mka
2. I want to display input1.mkv and input2.mkv in a horizontal mosaic (https://stackoverflow.com/questions/11552565/vertically-or-horizontally-stack-mosaic-several-videos-using-ffmpeg)
3. Input2.mkv should start 120 seconds after input1.mkv in output.mkv

Way of combining that works via multiple calls

ffmpeg -i input1.mkv -i input1.mka -map 0:0 -map 1:0 input1.mkv
ffmpeg -i input2.mkv -i input2.mka -map 0:0 -map 1:0 input2.mkv

Can the above be done in a single call?

What i currently have in terms of command line argument:

ffmpeg -i input1.mkv -i input1.mka -itsoffset 120 -i input2.mkv -i input2.mka -filter_complex hstack=inputs=2 -map 0:0 -map 1:0 -map 2:0 -map 3:0 -acodec copy -vcodec copy output.mkv

Error i'm getting:

Streamcopy requested for output stream 0:0, which is fed from a complex filtergraph. Filtering and streamcopy cannot be used together.

Am i going about it correctly in the task i'm trying to accomplish? Should i execute multiple ffmpegs call from C# to combine each input's audio and video file separately? Thank you for your help!

You can do this in one call, but the error is telling you that you cannot use `-acodec copy -vcodec copy` if you want to filter streams (which you do). So, get rid of these options first. You'll likely need to use `tpad` filter. Revise items 2 & 3 of your todo list as they don't make sense if you need further help. — kesh, Mar 22 '22 at 16:40
@kesh Hi, thank you for replying. I've Revised 2 & 3 so hopefully they are clearer! — Spartan 117, Mar 22 '22 at 16:48
how about audios? input2.mka also delayed by 120 seconds? How do you want to mix 2 audios? — kesh, Mar 22 '22 at 17:04
@kesh I was assuming that since i would be combining input1.mkv with input1.mka to output a new input1.mkv and input2.mkv with input2.mka which would output a new input2.mkv, when i go to combine the new mkv files to show in a mosaic, delaying input2.mkv by 120 seconds would also take care of the audio part no? If that's not the correct way, then what you are suggesting is correct. Newbie here so i don't understand what you mean by wanting to mix the 2 audios? I apologize if my wording is unclear because i'm just unfamiliar with ffmpeg. — Spartan 117, Mar 22 '22 at 17:11
no need to apologize. I know dealing with media files can be very tricky with so many variables involved. So, do you want one output file (output.mkv) which plays one stream of video along with a single audio track (not switching between 2 tracks, like 2 languages)? If so, you need to combine 2 audio tracks together somehow, and that's the mixing process. Do you want both audio to be heard at the same level? One louder than the other? Or do you only want to hear one? — kesh, Mar 22 '22 at 17:30
@kesh So what i'm trying to implement is 1 output file (output.mkv) which has input1.mkv and input2.mkv horizontally stacked (i think this is the correct terminology)? Each one of those has an audio file so what i would want is the audio file corresponding to each video file to start playing as soon as the video starts (in the case of input2, 120 secs after input 1). I want the audio heard at the same level for both tracks so i guess i want both tracks? How does the single audio tracks work vs switching between the 2 tracks? — Spartan 117, Mar 22 '22 at 17:40
Think of movie DVDs, they often come with multiple languages (e.g., English, Spanish, and French in North America) and they may even include additional track for director's commentary. To listen to another track, listener must switch manually. That's what you'd get if you keep your 2 audio tracks (or streams) separate. To play both audio at the same time, you'd use [`amix`](https://ffmpeg.org/ffmpeg-filters.html#amix) filter, in which you can set `weights` to emphasize one track over the other (e.g., foreground voice to be louder than the background music) by picking unequal weights. — kesh, Mar 22 '22 at 17:47
Ah ok, that's a great explanation. Then yes i guess i would like both input1.mka and input2.mka to be part of a single audio stream so that they start playing when the corresponding input video shows up. — Spartan 117, Mar 22 '22 at 17:54
I'd be happy to put together a filtergraph for you when I get home in a few hours, but meanwhile if you want to give it a stab, I'd use `tpad`, `hstack`, `apad`, and `amix` filters in `filter_complex` option. — kesh, Mar 22 '22 at 18:02
@kesh Thank you, will give it a shot. In the particular case i'm working on, i also have an input3.mka file which doesn't have a corresponding video file and needs to play about 180 seconds after the input1 file does so wanted to ask if that would be complicated to add in the filtergraph as well? — Spartan 117, Mar 22 '22 at 18:15
`amix` can take multiple audio streams, so it should be a simple extension of 2-audio case you have laid out — kesh, Mar 22 '22 at 18:18
@kesh Thank you so much for all your help. Will chat in a few hours :) — Spartan 117, Mar 22 '22 at 18:19
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/243202/discussion-between-spartan-117-and-kesh). — Spartan 117, Mar 23 '22 at 00:25

kesh · Accepted Answer · 2022-03-23T02:37:46.373

Try this:

ffmpeg -i input1.mkv -i input1.mka -i input2.mkv -i input2.mka -i input3.mka \
  -filter_complex \
    "[2:v]tpad=start_duration=120:color=black[vpad]; \
     [3:a]adelay=120000:all=1[a2pad]; \
     [4:a]adelay=180000:all=1[a3pad]; \
     [0:v][vpad]hstack=inputs=2[vout]; \
     [1:a][a2pad][a3pad]amix=inputs=3:weights=1|1|1[aout]" \
  -map [vout] -map [aout] \
  output.mkv

To delay the second set of A/V files, you use tpad for video and adelay for audio. Here you see how the 120s and 180s delays are coded. For tpad you can also set the color of the padded video frames.
[edit] adelay has to specify delay for all channels. Assuming audios are stereo, you need to add all=1 option.
amix I've already explained in the comments. I've added weights option just in case you decide to change the mixing balance.
the filtergraph outputs a video stream [vout] and a audio stream [aout]. You then assign them to the output file using -map option, one for each of them.
BTW, both [vout] and [aout] will last until the longest of 2 streams ends (shorter audio stream drops out while shorter video stream holds its last frame) If this is a problem, describe the situation and desired.
You cannot copy output streams of a filtergraph. So -vcodec copy and -acodec copy are no goes. I've removed them to use the default codecs for .mkv file, but if you need to customize it, add output options before output.mkv.

Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackoverflow.com/rooms/243260/discussion-on-answer-by-kesh-merge-multiple-audio-and-video-files). — Samuel Liew, Mar 24 '22 at 01:21

Merge multiple audio and video files

1 Answers1