0

I have a service that needs to transcode large amount of videos with different formats. The service spawns five threads, one for a single video and each thread runs ffmpeg with the following command:

ffmpeg -i %%FILEPATH%% -vf scale=X:Y -ab 128k -c:a aac -movflags faststart -strict -2 -ar 22050 -r 24 -c:v libx264 -crf 25 -y %%OUTPUT.MP4%%

where X and Y are the desired dimensions based on the orientation of the original file basically its either 640:trunc(ow*a/2)*2 for landscape or trunc(oh*a/2)*2:640 for portrait.

This is my ffmpeg info:

ffmpeg version 2.4.3-1ubuntu1~trusty6 Copyright (c) 2000-2014 the FFmpeg developers
  built on Nov 22 2014 17:07:19 with gcc 4.8 (Ubuntu 4.8.2-19ubuntu1)
  configuration: --prefix=/usr --extra-version='1ubuntu1~trusty6' --build-suffix=-ffmpeg --toolchain=hardened --extra-cflags= --extra-cxxflags= --libdir=/usr/lib/x86_64-linux-gnu --shlibdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --enable-shared --disable-stripping --enable-avresample --enable-avisynth --enable-fontconfig --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-opengl --enable-x11grab --enable-libxvid --enable-libx265 --enable-libdc1394 --enable-libiec61883 --enable-libzvbi --enable-libzmq --enable-frei0r --enable-libx264 --enable-libsoxr --enable-openal --enable-libopencv
  libavutil      54.  7.100 / 54.  7.100
  libavcodec     56.  1.100 / 56.  1.100
  libavformat    56.  4.101 / 56.  4.101
  libavdevice    56.  0.100 / 56.  0.100
  libavfilter     5.  1.100 /  5.  1.100
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  0.100 /  3.  0.100
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  0.100 / 53.  0.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

The service is written in Java and running on Ubuntu Server 14.04 and the machine is an 64-bit octa-core server.

This is the code block that executes ffmpeg:

try 
{
    ProcessBuilder procBuilder = null;


    String sArgs = String.format("ffmpeg -i %s -vf scale=%s:%s -ab 128k -c:a aac -movflags faststart -strict -2 -ar 22050 -r 24 -c:v libx264 -crf 25 -y %s",
                originalPath,
                outWidth,
                outHeight,
                targetPath
        );

    }

    String[] arrArgs = sArgs.split("\\s+");
    procBuilder = new ProcessBuilder(Arrays.asList(arrArgs));
    procBuilder.redirectErrorStream(true);
   procBuilder.redirectOutput();


    Process process = procBuilder.start();

    try (BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()))) 
    {
        String line = null;
        while ((line = reader.readLine()) != null) 
        {
            System.out.println(line);
        }
        errorCode = process.waitFor();
    }
}
catch(Throwable ex)
{
}

I am currently spawning five threads, and each thread runs a single instance of ffmpeg targeting a single video file. This works fine most of the time, but every once in a while threads start to hang. I noticed from top that ffmpeg hangs indefinitely on some files with one of the threads using 100% of the core CPU and no progress is made. It happened with different file types as I noticed this on mkv, avi, wmv and mp4 files.

I am not sure what is causing ffmpeg to hang, it does not happen right at the beginning of the transcoding process, ffmpeg starts converting the file fine but somewhere in the middle it gets stuck.

Now this is not a problem with the files as when I try the same command on the same file manually it runs fine. And it only seems to happen when there are multiple instances of ffmpeg running at the same time as I now changed my service to only run a single thread and it has been running for almost a month with no issues.

Is there an option that I need to use to allow multiple instances of ffmpeg to run at the same time? Is it something in the command line that I currently use that causes this?

Cœur
  • 37,241
  • 25
  • 195
  • 267
Zaid Amir
  • 4,727
  • 6
  • 52
  • 101
  • So you have 8 cores (real cores)? How do you measure hangs again? Can you monitor the disk activity for each target file? Note that ffmpeg write a lot to stderr so you need to redirect the error stream to the output stream otherwise you will miss a lot of information. – UmNyobe May 12 '16 at 09:42
  • @UmNyobe Yes I also tried redirecting stderr but ffmpeg does not write any output. The first time I noticed this is when my service stopped pulling files to process. I checked the output directory where I store my transcoded files and I noticed that no files are getting changed. Also from top I saw that I had 5 processes of ffmpeg running. each process has spawned about 20+ threads and only one of those was using 100% of the cpu. I left it running for about an hour and came back and it was still the same, output file sizes remained the same and the same exact threads was using 100% CPU – Zaid Amir May 12 '16 at 10:02
  • @UmNyobe just to clarify, when I said 'a single thread' I meant each ffmpeg process had a single thread running at 100% CPU – Zaid Amir May 12 '16 at 10:05
  • @UmNyobe I rechecked the code and it seems I missed the redirect calls. So I updated the question body. – Zaid Amir May 12 '16 at 10:12
  • Can you try killing this process with 100% usage to see if it progresses? Can you also increase the verbosity of ffmpeg (-loglevel debug)? – UmNyobe May 12 '16 at 11:05
  • @UmNyobe killing one process does not do a thing, the other processes stay in a hanging state. I will try to switch to a debug loglevel but it might take sometime before the issue happens again, as I said this issue happens infrequently and sometime it takes days to reproduce. – Zaid Amir May 12 '16 at 11:27
  • Can you use a different aac encoder? The aac encoder you are using was experimental till very recently (2016). As I see it, it is the most probable point of failure. You will need a compiled version of ffmpeg with better aac encoders. See https://trac.ffmpeg.org/wiki/Encode/AAC – UmNyobe May 12 '16 at 11:54
  • @UmNyobe will also try that, so if it fails on the newest version, what type of aac encoder do you recommend? – Zaid Amir May 12 '16 at 11:58
  • you need to check the error and the input stream - all those redirects confuse me but ffmpeg writes CONTINOUSLY output - to say that ffmpeg doesnt write anything seems you are missing – gpasch May 12 '16 at 17:09
  • @gpasch hence I used the word 'hangs' in the title – Zaid Amir May 12 '16 at 21:11

1 Answers1

1

libx264 uses by default a number of threads equal to 1.5 * logical cores (for frame-based multi-threading). Multiply this with five instances and you get a lot of threads, and this is just for encoding the video stream.

You can try reducing the number of threads used on each instance by setting the -x264opts threads=n option.

aergistal
  • 29,947
  • 5
  • 70
  • 92
  • And what would that achieve? – Zaid Amir May 15 '16 at 05:51
  • These threads are CPU intensive. The default settings may work for a single command but since you run multiple instances you're creating a high CPU demand. Reducing the individual number of threads might help you run multiple instances while avoiding that deadlock. Performance-wise you need to benchmark to see what happens, since too many threads will just wait on CPU anyway. – aergistal May 15 '16 at 08:05
  • Threads configuration based on no. of CPU-cores http://stackoverflow.com/questions/13834692/threads-configuration-based-on-no-of-cpu-cores – aergistal May 15 '16 at 08:07