Convert HTML5 Canvas Sequence to a Video File

Question

I'd like to convert an animation in HTML5 canvas to a video file that could be uploaded to YouTube. Is there any sort of screen capture API or something that could allow me to do this programatically?

Answer 1

Back to 2020

Solved it by using MediaRecorder API. It builds exactly to do that, among other things.

Here is a solution that recorded X ms of canvas video you can extend it with Buttons UI to start, pause, resume, stop, generate URL.

function record(canvas, time) {
    var recordedChunks = [];
    return new Promise(function (res, rej) {
        var stream = canvas.captureStream(25 /*fps*/);
        mediaRecorder = new MediaRecorder(stream, {
            mimeType: "video/webm; codecs=vp9"
        });
        
        //ondataavailable will fire in interval of `time || 4000 ms`
        mediaRecorder.start(time || 4000);

        mediaRecorder.ondataavailable = function (event) {
            recordedChunks.push(event.data);
             // after stop `dataavilable` event run one more time
            if (mediaRecorder.state === 'recording') {
                mediaRecorder.stop();
            }

        }

        mediaRecorder.onstop = function (event) {
            var blob = new Blob(recordedChunks, {type: "video/webm" });
            var url = URL.createObjectURL(blob);
            res(url);
        }
    })
}

How to use:

const recording = record(canvas, 10000)
// play it on another video element
var video$ = document.createElement('video')
document.body.appendChild(video$)
recording.then(url => video$.setAttribute('src', url) )

// download it
var link$ = document.createElement('a')
link$.setAttribute('download','recordingVideo') 
recording.then(url => {
 link$.setAttribute('href', url) 
 link$.click()
})

Answer 2

Firefox has an experimental feature (disabled by default) that is called HTMLCanvasElement.captureStream()

Essentially it captures the canvas element as a video stream which can then be sent to another computer using RTCPeerConnection() or perhaps you can use the YouTube Live Streaming API to stream directly.

See: https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/captureStream

Also: https://developers.google.com/youtube/v3/live/getting-started

Answer 3

There exist the whammy library which claims to produce webm videos from stills using JavaScript:
http://antimatter15.com/wp/2012/08/whammy-a-real-time-javascript-webm-encoder/

Note that there are limitations (as to be expected). This encoder bases itself on the webp image format which is currently only supported in Chrome (perhaps the new Opera too but I haven't checked). This means you can't encode in other browsers unless you find a way to encode the image you want to use as a webp image first (see this link for possible solution for that).

Beyond that there is no way to create a video file from images using JavaScript and canvas using native browser APIs.

Answer 4

FileSaver.js + ffmpeg on the command line

With FilSaver.js we can download each canvas frame as PNG: Save to Local File from Blob

Then we just convert the PNGs to any video format with ffmpeg from the command line: How to create a video from images with FFmpeg?

Chromium 75 asks if you want to allow it to save multiple images. Then once you say yes, it downloads the images automatically one by one under your download folder, named as 0.png, 1.png, etc.

It also worked in Firefox 68, but less well, because the browser opens a bunch of "Do you want to save this file" windows. They do have a "do the same for similar downloads" popup, but you have to be quick to select it and hit enter, or else a new popup comes along!

To stop it, you have to close the tab, or add a stop button and some JavaScript logic.

var canvas = document.getElementById("my-canvas");
var ctx = canvas.getContext("2d");
var pixel_size = 1;
var t = 0;

/* We need this to fix t because toBlob calls are asynchronous. */
function createBlobFunc(t) {
  return function(blob) {
    saveAs(blob, t.toString() + '.png');
  };
}

function draw() {
    console.log("draw");
    for (x = 0; x < canvas.width; x += pixel_size) {
        for (y = 0; y < canvas.height; y += pixel_size) {
            var b = ((1.0 + Math.sin(t * Math.PI / 16)) / 2.0);
            ctx.fillStyle =
                "rgba(" +
                (x / canvas.width) * 255 + "," +
                (y / canvas.height) * 255 + "," +
                b * 255 +
                ",255)"
            ;
            ctx.fillRect(x, y, pixel_size, pixel_size);
        }
    }
    canvas.toBlob(createBlobFunc(t));
    t++;
    window.requestAnimationFrame(draw);
}
window.requestAnimationFrame(draw);

<canvas id="my-canvas" width="512" height="512" style="border:1px solid black;"></canvas>
<script src="https://cdnjs.cloudflare.com/ajax/libs/FileSaver.js/1.3.8/FileSaver.min.js"></script>

GitHub upstream.

Here's an image to GIF output using this instead: https://askubuntu.com/questions/648244/how-do-i-create-an-animated-gif-from-still-images-preferably-with-the-command-l

Frames get skipped if the FPS is too high

This can be observed by reducing the size of the canvas in the above demo to speed things up. At 32x32, my Chromium 77 download in chunks of about 10 files and skips about 50 files in between...

Unfortunately, there is no way to wait for the downloads to finish... close window after file save in FileSaver.js

So the only solution I can see if you have high framerate is framerate limiting... Controlling fps with requestAnimationFrame? Here is a live demo: https://cirosantilli.com/#html-canvas

Maybe one day someone will answer:

• H.264 video encoder in javascript
• Running ffmpeg in browser - options?

and then we will be able to download the video directly!

Here is an OpenGL version if you decide that the browser is not for you :-) How to use GLUT/OpenGL to render to a file?

Tested in Ubuntu 19.04.

Answer 5

This should help, it allows you to drop some images that get converted into HTML5 CANVAS and then converted into webm video: http://techslides.com/demos/image-video/create.html

Answer 6

Pure javascript, no other 3rd-package.

If you have a video and want to take some frames, you can try as below

class Video2Canvas {
  /**
   * @description Create a canvas and save the frame of the video that you are giving.
   * @param {HTMLVideoElement} video
   * @param {Number} fps
   * @see https://developer.mozilla.org/en-US/docs/Web/Guide/Audio_and_video_manipulation#video_manipulation
   * */
  constructor(video, fps) {
    this.video = video
    this.fps = fps
    this.canvas = document.createElement("canvas");
    [this.canvas.width, this.canvas.height] = [video.width, video.height]
    document.querySelector("body").append(this.canvas)
    this.ctx =  this.canvas.getContext('2d')
    this.initEventListener()
  }

  initEventListener() {
    this.video.addEventListener("play", ()=>{
      const timeout = Math.round(1000/this.fps)
      const width = this.video.width
      const height = this.video.height
      const recordFunc = ()=> {
        if (this.video.paused || this.video.ended) {
          return
        }
        this.ctx.drawImage(this.video, 0, 0, width, height)
        const frame = this.ctx.getImageData(0, 0, width, height)
        // ... // you can make some modifications to change the frame. For example, create the grayscale frame: https://developer.mozilla.org/en-US/docs/Web/Guide/Audio_and_video_manipulation#video_manipulation

        //  Below is the options. That saves each frame as a link. If you wish, then you can click the link to download the picture.
        const range = document.createRange()
        const frag = range.createContextualFragment('<div><a></a></div>')
        const tmpCanvas = document.createElement('canvas')
        tmpCanvas.width = this.canvas.width
        tmpCanvas.height = this.canvas.height
        tmpCanvas.getContext('2d').putImageData(frame, 0, 0)
        const a = frag.querySelector('a')
        a.innerText = "my.png"
        a.download = "my.png"
        const quality = 1.0
        a.href = tmpCanvas.toDataURL("image/png", quality)
        a.append(tmpCanvas)
        document.querySelector('body').append(frag)
        setTimeout(recordFunc, timeout)
      }
      setTimeout(recordFunc, timeout)
    })
  }
}
const v2c = new Video2Canvas(document.querySelector("video"), 1)

<video id="my-video" controls="true" width="480" height="270" crossorigin="anonymous">
  <source src="http://jplayer.org/video/webm/Big_Buck_Bunny_Trailer.webm" type="video/webm">
</video>

If you want to edit the video (for example, take 5~8_sec+12~15_sec and then create a new one) you can try

class CanvasRecord {
  /**
   * @param {HTMLCanvasElement} canvas
   * @param {Number} fps
   * @param {string} mediaType: video/webm, video/mp4(not support yet) ...
   * */
  constructor(canvas, fps, mediaType) {
    this.canvas = canvas
    const stream = canvas.captureStream(25) // fps // https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/captureStream
    this.mediaRecorder = new MediaRecorder(stream, { // https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/MediaRecorder
      mimeType: mediaType
    })
    this.initControlBtn()

    this.chunks = []
    this.mediaRecorder.ondataavailable = (event) => {
      this.chunks.push(event.data)
    }
    this.mediaRecorder.onstop = (event) => {
      const blob = new Blob(this.chunks, {
        type: mediaType
      })
      const url = URL.createObjectURL(blob)

      //  Below is a test code for you to know you are successful. Also, you can download it if you wish.
      const video = document.createElement('video')
      video.src = url
      video.onend = (e) => {
        URL.revokeObjectURL(this.src);
      }
      document.querySelector("body").append(video)
      video.controls = true
    }
  }

  initControlBtn() {
    const range = document.createRange()
    const frag = range.createContextualFragment(`<div>
    <button id="btn-start">Start</button>
    <button id="btn-pause">Pause</button>
    <button id="btn-resume">Resume</button>
    <button id="btn-end">End</button>
    </div>
    `)
    const btnStart = frag.querySelector(`button[id="btn-start"]`)
    const btnPause = frag.querySelector(`button[id="btn-pause"]`)
    const btnResume  = frag.querySelector(`button[id="btn-resume"]`)
    const btnEnd   = frag.querySelector(`button[id="btn-end"]`)
    document.querySelector('body').append(frag)
    btnStart.onclick = (event) => {
      this.chunks = [] // clear
      this.mediaRecorder.start() // https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/start
      console.log(this.mediaRecorder.state) // https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/state

    }

    btnPause.onclick = (event) => { // https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/pause
      this.mediaRecorder.pause()
      console.log(this.mediaRecorder.state)
    }

    btnResume.onclick = (event) => {
      this.mediaRecorder.resume()
      console.log(this.mediaRecorder.state)
    }

    btnEnd.onclick = (event) => {
      this.mediaRecorder.requestData() // trigger ``ondataavailable``  // https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/requestData
      this.mediaRecorder.stop()
      console.log(this.mediaRecorder.state)
    }
  }
}


class Video2Canvas {
  /**
   * @description Create a canvas and save the frame of the video that you are giving.
   * @param {HTMLVideoElement} video
   * @param {Number} fps
   * @see https://developer.mozilla.org/en-US/docs/Web/Guide/Audio_and_video_manipulation#video_manipulation
   * */
  constructor(video, fps) {
    this.video = video
    this.fps = fps
    this.canvas = document.createElement("canvas");
    [this.canvas.width, this.canvas.height] = [video.width, video.height]
    document.querySelector("body").append(this.canvas)
    this.ctx =  this.canvas.getContext('2d')
    this.initEventListener()
  }

  initEventListener() {
    this.video.addEventListener("play", ()=>{
      const timeout = Math.round(1000/this.fps)
      const width = this.video.width
      const height = this.video.height
      const recordFunc = ()=> {
        if (this.video.paused || this.video.ended) {
          return
        }
        this.ctx.drawImage(this.video, 0, 0, width, height)
        /*
        const frame = this.ctx.getImageData(0, 0, width, height)
        // ... // you can make some modifications to change the frame. For example, create the grayscale frame: https://developer.mozilla.org/en-US/docs/Web/Guide/Audio_and_video_manipulation#video_manipulation

        //  Below is the options. That saves each frame as a link. If you wish, then you can click the link to download the picture.
        const range = document.createRange()
        const frag = range.createContextualFragment('<div><a></a></div>')
        const tmpCanvas = document.createElement('canvas')
        tmpCanvas.width = this.canvas.width
        tmpCanvas.height = this.canvas.height
        tmpCanvas.getContext('2d').putImageData(frame, 0, 0)
        const a = frag.querySelector('a')
        a.innerText = "my.png"
        a.download = "my.png"
        const quality = 1.0
        a.href = tmpCanvas.toDataURL("image/png", quality)
        a.append(tmpCanvas)
        document.querySelector('body').append(frag)
        */
        setTimeout(recordFunc, timeout)
      }
      setTimeout(recordFunc, timeout)
    })
  }
}

(()=>{
  const v2c = new Video2Canvas(document.querySelector("video"), 60)
  const canvasRecord = new CanvasRecord(v2c.canvas, 25, 'video/webm')

  v2c.video.addEventListener("play", (event)=>{
    if (canvasRecord.mediaRecorder.state === "inactive") {
      return
    }
    document.getElementById("btn-resume").click()
  })

  v2c.video.addEventListener("pause", (event)=>{
    if (canvasRecord.mediaRecorder.state === "inactive") {
      return
    }
    document.getElementById("btn-pause").click()
  })
})()

<video id="my-video" controls="true" width="480" height="270" crossorigin="anonymous">
  <source src="http://jplayer.org/video/webm/Big_Buck_Bunny_Trailer.webm" type="video/webm">
</video>