36

Based on this article it seems that it is possible to use FFMPEG to detect scene change in videos: http://www.luckydinosaur.com/u/ffmpeg-scene-change-detector

Now I have a video that displays a book text and when the text (word or sentence) is spoken it gets highlighted. Something like this audio book: https://youtu.be/lA7L6ZNVKjc

I need to know the timestamp when the text gets highlighted (hence scene change), this will allow me to add timestamp tags on my youtube video, so it becomes easier for listeners to navigate through the audiobook.

What is the magic command line that would do this?

Thank you very much!

Mozart
  • 735
  • 2
  • 6
  • 10
  • 1
    Anyone landing on this question/post [may find this useful](https://video.stackexchange.com/questions/28613/how-to-extract-each-video-scene-with-ffmpeg) – dtmland Jun 27 '20 at 18:19

4 Answers4

50

Combining the scene filter (for detecting scene changes) and the showinfo filter should achieve what you want:

ffmpeg -i input.flv  \
       -filter:v "select='gt(scene,0.4)',showinfo" \
       -f null \
       - 2> ffout

This command extracts all frames that differ from the previous frame by more than (gt) 0.4 (on a scale from 0 to 1). For these frames, information is printed out (showinfo) like this

[Parsed_showinfo_1 @ 0x2d85e60] n:   0 pts:2537204 pts_time:2.5372  pos:  2998114 fmt:rgb24 sar:1/1 s:1920x1200 i:P iskey:1 type:I checksum:5616582E plane_checksum:[5616582E]

Now you only have to extract the timestamp. I think you're interested in pts_time. You could do it like this:

grep showinfo ffout | grep pts_time:[0-9.]* -o | grep [0-9.]* -o > timestamps

This will give you the list of all timestamps:

2.5372
4.37799
6.65301
8.09344

For this approach to work, you must have a version of FFmpeg that implements the scene detection. Also, you have to select a suitable value for the threshold (0.4 in the first command). You can try to find an optimal threshold by extracting frames for different thresholds (and afterwards examine the frames manually) like this

ffmpeg -i input.flv \
       -filter:v "select='gt(scene,0.1)',showinfo" \
       -vsync 0 frames/%05d.jpg

Just for clarification: grep [0-9.]* does not exclude integers as claimed in another answer. It matches any character sequence consisting of digits and periods but it would also match non-numbers like '4.4.4'. However, ffmpeg shouldn't output such ill-formed timestamps.

Martin Delille
  • 11,360
  • 15
  • 65
  • 132
ckoehn
  • 624
  • 7
  • 7
  • the last grep above should be `grep '[0-9]*\.[0-9]*' -o` so that the final grep is `grep showinfo ffout | grep pts_time:[0-9.]* -o | grep '[0-9]*\.[0-9]*' -o > timestamps` the `.` needs to be escaped, otherwise it matches any character – po.studio Oct 19 '16 at 12:38
  • 5
    @keypulsations, `[.]` does not match any character `.` does. `[.]` is a bracket expression, which matches any single character between the brackets. See also [the grep manual on "Character Classes and Bracket Expressions"](https://www.gnu.org/software/grep/manual/html_node/Character-Classes-and-Bracket-Expressions.html#Character-Classes-and-Bracket-Expressions). While `grep [0-9.]*` would also match more than floats and integers (but in reality does not in this pipeline), your regexp might exclude integers as pointed out in the other answer. – ckoehn May 30 '17 at 14:59
  • Hi, I'm interested in scenes change detection. You wrote about a version of FFmpeg that implements the scene detection. Can you tell me which version of ffmpeg implements this feature? – Laura Jun 17 '18 at 12:13
  • @Laura, the latest and - at least - every version since July '16. I don't know when this was added. – ckoehn Jun 18 '18 at 11:14
  • @MartinDelille zhs tries to expand the * and complains that it cannot do that. Using quotes should fix that. – ckoehn Dec 19 '18 at 16:21
  • @ckoehn Thanks it works better. The only problem I have is with the last grep which is replacing `pts_time:80.24` by an empty line. – Martin Delille Dec 21 '18 at 10:20
  • @ckoehn I tried it and it works for "when the camera is moving" but it doesn't detect new scenes, like when I transition from a video to a new title ex: "CHAPTER 2"... it doesn't detecte those changes, and I'm looking for that. – Conrad C Jun 06 '19 at 19:26
  • @ConradC Did you try lower thresholds? If that does not work for you, you should consider asking a new question with your specific problem. – ckoehn Jun 18 '19 at 08:20
  • Don't forget to quote the regex, otherwise the regex might return zero results. – Danny Nov 28 '19 at 11:28
17

You can simply use the command:

ffmpeg -i inputvideo.mp4 -filter_complex "select='gt(scene,0.3)',metadata=print:file=time.txt" -vsync vfr img%03d.png

This will save just the relevant information in the time.txt file like below.

frame:0    pts:108859  pts_time:1.20954
lavfi.scene_score=0.436456
frame:1    pts:285285  pts_time:3.16983
lavfi.scene_score=0.444537
frame:2    pts:487987  pts_time:5.42208
lavfi.scene_score=0.494256
frame:3    pts:904654  pts_time:10.0517
lavfi.scene_score=0.462327
frame:4    pts:2533781 pts_time:28.1531
lavfi.scene_score=0.460413
frame:5    pts:2668916 pts_time:29.6546
lavfi.scene_score=0.432326

The frame is the serial number of the detected shot change from the starting. Also, choose your threshold value (here 0.3) appropriately for your use case to get correct outputs

jdhao
  • 24,001
  • 18
  • 134
  • 273
Legolas
  • 653
  • 1
  • 9
  • 17
  • This command doesn't work and creates the following error: "Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)" – bashan Aug 22 '20 at 20:41
  • Try changing the frame rate of the input file, seems a problem for the input mapping. – Legolas Aug 24 '20 at 07:15
  • 2
    it should be `ffmpeg -i inputvideo.mp4`, otherwise, it get error message that `output file #0 does not contain any stream`. – jdhao Dec 03 '21 at 08:08
8

I don't have the rep to post a comment on the above answer but I wanted to point out that the grep posted by both @ckoehn and @keypulsations will only grab timestamps which are floating point. To grab both floating point and integer timestamps use the following regex

grep showinfo ffout | grep pts_time:[0-9.]* -o | grep -E '[0-9]+(?:\.[0-9]*)?' -o > timestamps
Luke Harrison
  • 81
  • 1
  • 3
3

I was trying the @ckoehn answer and it worked, until it stopped working, the asterisk in the last grep was causing trouble. To avoid this I recommend using double quotes in the grep sentences like:

grep showinfo ffout | grep pts_time:[0-9.]* -o | grep "[0-9.]*" -o > timestamps
Ignacio Peletier
  • 2,056
  • 11
  • 24