1

I've already read FFmpeg - Overlay one video onto another video?, How to overlay 2 videos at different time over another video in single ffmpeg command?, FFmpeg - Multiple videos with 4 areas and different play times (and many similar questions tagged [ffmpeg] about setpts), and the following code is working, but I'm sure we can simplify it, and have a more elegant solution.

I'd like to mix multiple sources (image and sound) , with different starting points:

t (seconds)           0   1   2   3   4   5   6   7   8   9  10  11  12  13    
test.png              [-------------------------------]
a.mp3                         [-------]
without_sound.mp4                                 [-------------------]        (overlay at x,y=200,200)
b.mp3                                     [---]
with_sound.mp4                    [---------------------------------------]    (overlay at x,y=100,100)

This works:

ffmpeg -i test.png 
       -t 2 -i a.mp3 
       -t 5 -i without_sound.mp4 
       -t 1 -i b.mp3 
       -t 10 -i with_sound.mp4 
       -filter_complex "
            [0]setpts=PTS-STARTPTS[s0];
            [1]adelay=2000^|2000[s1];
            [2]setpts=PTS-STARTPTS+7/TB[s2];
            [3]adelay=5000^|5000[s3];
            [4]setpts=PTS-STARTPTS+3/TB[s4];
            [4:a]adelay=3000^|3000[t4];
            [s1][s3][t4]amix=inputs=3[outa];
            [s0][s4]overlay=100:100[o2];
            [o2][s2]overlay=200:200[outv]
       " -map [outa] -map [outv]
       out.mp4 -y

but:

  • is it normal that we have to use both setpts and adelay? I have tried without adelay and then the sound is not shifted. Said differently, is there a way to simplify:

    [4]setpts=PTS-STARTPTS+3/TB[s4];
[4:a]adelay=3000^|3000[t4];

    ?

  • is there a way to do it with setpts and asetpts only? When I replaced adelay=5000|5000 with asetpts=PTS-STARTPTS+5/TB and also for the other one, it didn't give the expected time-shifting (see below)

  • in similar questions/answers I often see overlay=...:enable='between(t,...,...)', here it seems it is not needed, why?

More generally, how would you simplify this "mix multiple audio and video" ffmpeg code?

More details about the second bullet point: if we replace adelay by asetpts,

-filter_complex "
            [0]setpts=PTS-STARTPTS[s0];
            [1]asetpts=PTS-STARTPTS+2/TB[s1];
            [2]setpts=PTS-STARTPTS+7/TB[s2];
            [3]asetpts=PTS-STARTPTS+5/TB[s3];
            [4]setpts=PTS-STARTPTS+3/TB[s4];
            [4:a]asetpts=PTS-STARTPTS+3/TB[t4];
            [s1][s3][t4]amix=inputs=3[outa];
            [s0][s4]overlay=100:100[o2];
            [o2][s2]overlay=200:200[outv]

it doesn't work: [3] should begin at 0'05", and [4:a] at 0'03" but they all begin at the same time than [1], i.e. at 0'02".

It seems that amix only takes the first asetpts in consideration, and discards the others; is it true?

Basj
  • 41,386
  • 99
  • 383
  • 673
  • "*is there a way to simplify*" Ran out of time to look at this today, but one lazy suggestion is to use `adelay=5s:all=1` so you can work in seconds and not have to know the number of channels (make sure your ffmpeg isn't too old or it won't work). – llogan Feb 18 '21 at 19:36

1 Answers1

2

is it normal that we have to use both setpts and adelay?

Yes, the former is for video streams; the latter, for audio. asetpts is not suitable for use with amix since the latter ignores starting time offsets. adelay fills in with silence from 0 to the desired offset.

I often see overlay=...:enable='between(t,...,...)', here it seems it is not needed, why?

Overlay syncs its main and overlay video frames by timestamps. enable is needed if one wishes to disable overlay when synced frames are available for both inputs.

Gyan
  • 85,394
  • 9
  • 169
  • 201