Convert animated webp to mp4 faster

Question

After searching around, I saw some one had suggested this for a animated webp to webm. But that seemed cumbersome. So I made this to convert a animated webp to mp4 or webm, which I have live here. It takes some logic of converting a gif to video and applies it. The issue is, it takes a bit.

I was wondering if anyone had suggestions on how to improve the speed?

import os
import moviepy.video.io.ImageSequenceClip

def analyseImage(path):
    '''
    Pre-process pass over the image to determine the mode (full or additive).
    Necessary as assessing single frames isn't reliable. Need to know the mode
    before processing all frames.
    '''
    im = PIL.Image.open(path)
    results = {
        'size': im.size,
        'mode': 'full',
    }
    try:
        while True:
            if im.tile:
                tile = im.tile[0]
                update_region = tile[1]
                update_region_dimensions = update_region[2:]
                if update_region_dimensions != im.size:
                    results['mode'] = 'partial'
                    break
            im.seek(im.tell() + 1)
    except EOFError:
        pass
    return results


def processImage(path):
    '''
    Iterate the animated image extracting each frame.
    '''
    images = []
    mode = analyseImage(path)['mode']

    im = PIL.Image.open(path)

    i = 0
    p = im.getpalette()
    last_frame = im.convert('RGBA')

    try:
        while True:
            print("saving %s (%s) frame %d, %s %s" % (path, mode, i, im.size, im.tile))

            '''
            If the GIF uses local colour tables, each frame will have its own palette.
            If not, we need to apply the global palette to the new frame.
            '''
            if '.gif' in path:
                if not im.getpalette():
                    im.putpalette(p)

            new_frame = PIL.Image.new('RGBA', im.size)

            '''
            Is this file a "partial"-mode GIF where frames update a region of a different size to the entire image?
            If so, we need to construct the new frame by pasting it on top of the preceding frames.
            '''
            if mode == 'partial':
                new_frame.paste(last_frame)

            new_frame.paste(im, (0, 0), im.convert('RGBA'))
            nameoffile = path.split('/')[-1]
            output_folder = path.replace(nameoffile, '')

            name = '%s%s-%d.png' % (output_folder, ''.join(os.path.basename(path).split('.')[:-1]), i)
            print(name)
            new_frame.save(name, 'PNG')
            images.append(name)
            i += 1
            last_frame = new_frame
            im.seek(im.tell() + 1)
    except EOFError:
        pass
    return images



def webp_mp4(filename, outfile):
    images = processImage("%s" % filename)
    fps = 30
    clip = moviepy.video.io.ImageSequenceClip.ImageSequenceClip(images, fps=fps)
    clip.write_videofile(outfile)
    return [outfile]

webp_mp4(filename, outfile)

How it works currently, is it when you run webp_mp4(filename, outfile) it calls processImage which calls analyseImage. In the end all this works fine. Just want it faster.

Have you considered writing a backend in a system programming language (ie. C, C++, Go, Rust, D, ...), if performance is really an issue? It's probably easier than trying to squeeze all the performance you can out of a slow language. — jthulhu, Feb 18 '22 at 08:45

sudo · Accepted Answer · 2022-01-17T20:05:54.457

3

Video transcoding is usually an "embarrassingly parallel" task, and processImage is doing things in one big sequence. If processImage is the slow part, you can use multiprocessing.Pool and assign each worker (which can run on a separate CPU core) its own range of frames to process. PIL objects aren't pickle-able, so you'll have to write temp files, which it seems you're already doing.

I don't know much about PIL, so if there's a better way to use the lib instead, I'm not going to see it. Maybe saving each frame as PNG is slow; worth trying TIF or JPEG. (I'd try it myself, but my Python installation isn't set up on this laptop.)

edited Jan 17 '22 at 20:05

answered Jan 17 '22 at 20:00

sudo

5,604
5
40
78

Probably the best way to paralelize transcoding is to work in chunks between keyframes (the ones with the full image). The work between keyframes can be easily paralelized and probably will yield the biggest speed bump. – Iñigo González Jan 17 '22 at 20:43
@sudo I put a bounty on this for a code example if you wish – nadermx Feb 17 '22 at 05:43

score 2 · Answer 2 · answered Feb 17 '22 at 21:24

A few optimisation pointers from looking at the code:

You read the image from the disk in both analyseImage and processImage. This isn't a huge thing since it only happens once per image, but reading from disk is still a relatively slow operation so it's best not to do it more than necessary. Instead, you could just open in in processImage and pass the opened image to analyseImage.

Your code for handling partial frames does more work than it needs to:

new_frame = PIL.Image.new('RGBA', im.size)
if mode == 'partial':
    new_frame.paste(last_frame)
new_frame.paste(im, (0, 0), im.convert('RGBA'))

(one new image, two pastes, one conversion)

could become

im = im.convert("RGBA")
if mode == 'partial':
    new_frame = last_frame
    new_frame.paste(im, (0, 0), im)
else:
    new_frame = im

(no new images, one paste, one conversion)

I suspect the biggest slowdown is writing every frame to the disk, then reading them again. It would be better to keep them in memory (and ideally, only one in memory at once).

imageio, the library that moviepy uses internally, provides a way to do this. You just create a video writer using imageio.get_writer(path), and add frames to it sequentially with writer.append_data(frame_data).

I tried this myself but I ended up with mixed up colours, so I don't have working code to provide. As a hint though, you can convert a PIL image to the raw frame data that imageio expects with something like numpy.array(im).

score 0 · Answer 3 · answered Feb 17 '22 at 02:36

0

    threads
      Number of threads to use for ffmpeg. Can speed up the writing of
      the video on multicore computers.

* clip.write_videofile(outfile, threads=4)

You can set the number of threads to increase the write speed.

answered Feb 17 '22 at 02:36

lazy

744
3
13

The speed difference on this was neglible. I believe @sudo is right in which the bottle kneck is `processImage` not `clip`. But his answer doesn't offer code. – nadermx Feb 17 '22 at 05:42

score 0 · Answer 4 · answered Feb 18 '22 at 05:05

Keep everything in memory. ImageSequenceClip accepts NumPy arrays for the frames. Instead of saving the frames to files, convert each frame to a NumPy array and yield them out of processImage.

This should improve performance significantly. Once you have this improvement in place, you can see if lazy's suggestion of using multiple threads for encoding to mp4 offers further improvement.

Or you could append the frames to the writer (e.g., in a for loop), as Artemis suggests.

score 0 · Answer 5 · answered Mar 06 '22 at 08:43

Add this function to remove the images after being used for creating the MP4 file.

def processImage2(path):
'''
Iterate the animated image extracting each frame.
'''
images = []
mode = analyseImage(path)['mode']

im = Image.open(path)

i = 0
p = im.getpalette()
last_frame = im.convert('RGBA')

try:
    while True:
        print("saving %s (%s) frame %d, %s %s" % (path, mode, i, im.size, im.tile))

        '''
        If the GIF uses local colour tables, each frame will have its own palette.
        If not, we need to apply the global palette to the new frame.
        '''
        if '.gif' in path:
            if not im.getpalette():
                im.putpalette(p)

        new_frame = Image.new('RGBA', im.size)

        '''
        Is this file a "partial"-mode GIF where frames update a region of a different size to the entire image?
        If so, we need to construct the new frame by pasting it on top of the preceding frames.
        '''
        if mode == 'partial':
            new_frame.paste(last_frame)

        new_frame.paste(im, (0, 0), im.convert('RGBA'))
        nameoffile = path.split('/')[-1]
        output_folder = path.replace(nameoffile, '')

        name = '%s%s-%d.png' % (output_folder, ''.join(os.path.basename(path).split('.')[:-1]), i)
        print(name)
        os.remove(name)
        i += 1
        last_frame = new_frame
        im.seek(im.tell() + 1)
except EOFError:
    pass

Just edit this function by adding one line "processImage2("%s" % filename)" and you are good to go.

def webp_mp4(filename, outfile):
images = processImage("%s" % filename)
fps = 30
clip = moviepy.video.io.ImageSequenceClip.ImageSequenceClip(images, fps=fps)
clip.write_videofile(outfile)

processImage2("%s" % filename)
return [outfile]

This doesn't improve speed but saves space which will be a problem no matter if it is executed on the cloud or offline. So, I thought of sharing this here. As this was something that should be present in the script. — Ayush Singh, Mar 09 '22 at 07:05

Convert animated webp to mp4 faster

5 Answers5