0

This question has been asked several times on this forum, with the accepted answer using ffmpeg to assess the integrity of the file with these example commands:

# scan a single file
ffmpeg.exe -v error -i C:\to\path\file.avi -f null - >error.log 2>&1

# batch scan
find C:\to\path\ -name "*.mp4" -exec sh -c "ffmpeg -v error -i '{}' -map 0:1 -f null - 2>'{}.log'" \;

The Problem:

The above commands work without issue, taking anywhere between 2-20 mins to assess a single video file. But when running the above batch command on a large number of video files (1000+) (assuming an average of 5 minutes per file), the process could take over a week to finish.

The Objective:

Looking for a faster solution to verify integrity of my files. Either to modify the ffmpeg command, or to use as a different binary entirely. Anything is accepted as long as I can run the new command in the terminal/bash. Would like to get the processing time down from a few days, to a few hours.

References:

https://superuser.com/questions/100288/how-can-i-check-the-integrity-of-a-video-file-avi-mpeg-mp4

Quickly check the integrity of video files inside a directory with ffmpeg

How can I tell if a video file is corrupted? FFmpeg?

https://gist.github.com/ridvanaltun/8880ab207e5edc92a58608d466095dec

Update

I never did find a "quick" way of scanning the video files. I just accepted that for the sake of thoroughness it will take some time. However, I made a GUI Python program that may benefit others:

https://github.com/nhershy/CorruptVideoFileInspector

nhershy
  • 643
  • 1
  • 6
  • 20
  • I guess using `ffprobe` should be better for this – jvx8ss Mar 09 '23 at 19:28
  • 1
    Apparently, `ffprobe` only looks at metadata, and not the file itself: *Another option is ffprobe that comes in package with ffmpeg. It doesn't do any conversion but simply reads metadata info from file. Therefore it will detect errors in metadata but won't find any problems within file itself.* https://superuser.com/questions/100288/how-can-i-check-the-integrity-of-a-video-file-avi-mpeg-mp4 – nhershy Mar 09 '23 at 22:16

1 Answers1

0

If you can assume that files are valid at time of creation, then store the checksum (sum or cksum) as a separate parallel file for each of those videos, then you could have that checksum as a verification tool for any subsequent file move/copy/transfer.

For sum:

{checksum} {block count}
07664 39404

For cksum:

{CRC checksum} {byte count} {filename}
3609965344 40348936 testfile.mp4

If you are looking for a method to verify that a video file is valid at time of first creation, the only way is to use the ffmpeg method for that verification.

The term "valid" itself is also vague.

If you mean "not corrupt", the above checksum methods are what you need.

If you mean "playable", you could use any multimedia tool and attempt playing the video at command line (as background process) and if it fails immediately, then you know it is invalid (assuming the codec used is one that the tool recognizes).

If, after starting the the video playing in background, you have a loop to wait for 30 seconds before testing for, and discovering, that the process is still running, then you would deem the file playable/valid, kill the background video process and move on to the next file in the list for testing.

Basically, videos are both disk and CPU intensive data for which data integrity-maintenance schemes need to be established up front, before starting to collect those. Otherwise, you run into the post-collection nightmare with which you are now facing.

I came face to face with my own "Rubicon" when I started my own post-collection integrity checking.

As for ffprobe, for this context, that is more limited to its use as a preliminary scan, to assist in setup for a more comprehensive ffmpeg assessment.

Eric Marceau
  • 1,601
  • 1
  • 8
  • 11
  • You have a point about the ambiguity of "valid". I did in fact mean "playable". However, the process is not as black & white as just "playable" or "non-playable". All my video files are playable. But some may have buggy/fuzzy/glitchy sections within several timestamps of the video. For example, the first hour of the video will run smoothly, then every 5 minutes there will be a 10 second chunk of freezy-glitchiness, only to resume normally once that bad sector has passed. (If there is a more accurate term to describe this behavior, please let me know). So I have to thoroughly scan the file. – nhershy Mar 10 '23 at 04:31
  • It seems like after talking this out, `ffmpeg` is my best option here. I guess what question remains is, how can I speed up the current process/command. I.e. use the same command as shared above, but refactor the code to use multithreading or parallel processing? Is it possible? And if so, how can I do it? (rhetorical questions, but would be great if someone knows the answer to that). Thanks – nhershy Mar 10 '23 at 04:34
  • So, for the "**freezy-glitchiness**" issue, if the video was downloaded, then you need to compare the checksums at source/copy. If they don't match, the glitchiness exists. Otherwise, there shouldn't be any, unless the video creator did not ensure sufficient CPU/buffering at time of creation to prevent creating the video with that glitchiness at the outset. If videos created on site, then you need to review the video creation code to add some mechanism to give priority and reserve bandwidth to ensure there is no insufficiency of buffering for the process that would cause the glitchiness. – Eric Marceau Mar 11 '23 at 00:44
  • I don't know if it is possible, but possibly the glitchiness is associated to a lower than average bitrate during a range of frames. If that is a direct relationship, and some process exists to provide that bitrate during fixed time-slices, for the entire video, that might provide a post-creation automatic diagnostic tool. – Eric Marceau Mar 11 '23 at 00:49
  • I ended up making a GUI program as a wrapper for ffmpeg to simplify this process of batch scanning video files for myself and others – nhershy Mar 12 '23 at 22:07