JS use video frame as canvas background

Question

I'm attempting to use a the first frame of a video as the background of a canvas, as you probably guessed from the title of the question. I can set a background, I can get a frame from a video, but I can't make the frame be the background. It looks like I need a url for the frame to set as the background in the CSS style. For that, I need a blob on which I can call createObjURL() (although I'm discovering now that that method may be deprecated/ing).

Here's some code:

    <!DOCTYPE html>
<html lang="en">
<canvas id="back" style="background-image: url('path/to/.png')";></canvas>
<input type="file" accept="video/*" />


<script>

  document.querySelector('input').addEventListener('change', extractFrame, false);

  var canvas = document.getElementById('back');
  var context = canvas.getContext('2d');
  var imageObj = new Image();
  var image;
  imageObj.onload = function() {
          var height = imageObj.height;
          var width = imageObj.width;
          canvas.height = height;
          canvas.width = width;
          context.drawImage(imageObj, 0, 0, back.width, back.height);
  };
  imageObj.src = "path/to/.png";



  var draw_canvas = document.getElementById('back');
  var draw_context = draw_canvas.getContext('2d');
  var corners = [];
  draw_canvas.addEventListener('mousedown',startDrawing,false);

  function startDrawing(e){
    corners.length=0;
    corners.push([e.layerX, e.layerY]);
    corners.push([e.layerX+1, e.layerY+1]);
    draw_canvas.addEventListener('mousemove', updateCorners,false);
    draw_canvas.addEventListener('mouseup',function(){
        draw_canvas.removeEventListener('mousemove', updateCorners,false);
    },false);
  }

  function updateCorners(e){
    corners[1] = [e.layerX, e.layerY]
    redraw();
  }

  function redraw(){
    draw_context.clearRect(0, 0, canvas.width, canvas.height);
    draw_context.beginPath()
    draw_context.lineWidth = "2";
    draw_context.strokeStyle = "red";
    draw_context.rect(corners[0][0], corners[0][1], corners[1][0] - corners[0][0], corners[1][1] - corners[0][1]);
    draw_context.stroke();
  }

  function extractFrame(){
    var video = document.createElement('video');
    var array = [];

    function initCanvas(){
        canvas.width = this.videoWidth;
        canvas.height = this.videoHeight;
    }

    function drawFrame(e) {
        this.pause();
        context.drawImage(this, 0, 0);
        var blob = canvas.toBlob(saveFrame, 'image/png');
        var dataUrl = URL.createObjectURL(array[0]);
        document.getElementById('back').style.background = "url('${dataUrl}')";
        URL.revokeObjectURL(dataUrl);
    }

    function saveFrame(blob) {
        console.log(array);
        array.push(blob);
        console.log(array);
    }

    video.muted = true;
    video.addEventListener('loadedmetadata', initCanvas, false);
    video.addEventListener('timeupdate', drawFrame, false);
    video.src = URL.createObjectURL(this.files[0]);
    video.play();
  }

</script>

I've been messing with the ordering of the code, the objects I'm passing, etc. all day now and I have no idea what's keeping this from working. If it helps, one of the errors I get most often is

Failed to execute 'createObjectURL' on 'URL': Overload resolution failed.

Which a different question said may be because the object in the array is undefined or None or however JS describes it, but the logged array says that it is populated with blobs.

Any suggestions are appreciated. Happy to provide more code or specs if I can. Thanks!

EDIT: I've added more code so that it can be reproduced (I hope). I'm not hosting it anywhere but locally, so I hope this works for y'all.

Hi there, do you have a repo where we could try your code? In the meantime, 2 things stood out to me. First, I would expect you assign `background` a value like `\`url('${dataUrl}')\`` rather than just `dataUrl`. Secondly, aren't you revoking the objectURL too soon? — Dan Macak, May 29 '21 at 18:42
I've made changes to meet your suggestions, but the issue comes with the line ```var dataUrl = URL.createObjectURL(array[0]);``` For whatever reason, HTML doesn't want to convert the blob object to a URL. — Elkelthen, May 29 '21 at 19:20
`Canvas.toBlob(fn)` will call `fn` asynchronously. So if you try to access the first item of the array synchronously after thos call, you wont have anything. Move all the lines after `toBlob` inside `saveFrame`. — Kaiido, May 30 '21 at 01:23

Elkelthen · Answer 1 · 2021-05-30T14:05:57.457

0

EDITED

    function drawFrame(e) {
        this.pause();
        context.drawImage(this, 0, 0);
        canvas.toBlob(saveFrame,'image/png');
    }

    function saveFrame(blob) {
        url = URL.createObjectURL(blob);
        document.getElementById('back').style.background = 'url(' + url + ')';
        URL.revokeObjectURL(dataUrl);
    }

Per @Kaiido's suggestion, I found the way to use a blob instead of a data url.

edited May 30 '21 at 14:05

answered May 30 '21 at 00:21

Elkelthen

17
8

No, don't use data: URL. A blob: URP os way better memory wise than a data: URL one. – Kaiido May 30 '21 at 01:21
1

Please note that handling asynchronous code correctly is only part of the solution if you would still use the object URL. Second part is **not** calling `revokeObjectURL` prematurely as in your code sample, since that releases the object url from memory and it can't be used anymore (eg. for background). – Dan Macak May 30 '21 at 07:59

JS use video frame as canvas background

1 Answers1