HTML5 Canvas hangs when changing the sprite sheet being animated (frame by frame video playing)

Question

I need to play videos that have a transparent background on mobile devices using HTML5. On the Windows Phone, I can capture the video tag's current frame and show that in the canvas. But that doesn't work on Android and iOS devices (I think for security reasons?)

My solution was to split up the .flv using FFMPEG and then stitch those frames together into large images, like sprite sheets.

The Problem is that the animation 'hangs' when I switch over to a new frame sheet. I've only checked this visually and through the console (by logging when I change the current sprite sheet row.) I've tested this by seeing how it hangs when I change the sprite sheet, and how it doesn't hang when I just loop the same sheet over and over.

I pre-load all of the images before hand:

var frameImages = [];

for(var i = 0; i < 35; i++)
{
  frameImages.push(new Image());
  frameImages[i].src = 'frame' + i + '.png';

  console.log(frameImages[i].src);

  frameImages[i].onload = function()
  {
    // Notify us that it's been loaded.
    console.log("Image loaded");
  }
}

And then play it like so:

processFrame = function()
{
    outputCanvas.width = outputCanvas.width;
    output.drawImage(frameImages[currentFrame], (curCol*153), (curRow*448), 153, 448, 0, 0, 153, 448);
    curCol += 1;

    if(curCol >= maxCol)
    {
      curCol = 0;
      curRow += 1;

      if(curRow >= maxRow)
      {
        curRow = 0;
        currentFrame++;
      }
    }
  }
}

var mozstart = window.mozAnimationStartTime;

step = function(timestamp) {

  var diff = (new Date().getTime() - start) - time;
  if(diff >= frameDelay)
  {
    processFrame();
    time += frameDelay;
  }
}

I've tried this in Chrome v 23.0.1271.97 m on a Win 7 machine and on a Nexus 7 with Chrome.

See it in action here:
http://savedbythecode.com/spokes/mozanimation.php - This is using mozAnimationStartTime
and http://savedbythecode.com/spokes/newplayer.php - This is using regular JS timers that are adjusted each step (from http://www.sitepoint.com/creating-accurate-timers-in-javascript/)

Any ideas? Was the problem clear enough?

Thanks, Kevin

Answer 1

Cool code. Why couldn't you just use a video tag (see option 6) ? Anyway I will just list issues I can think of and hope it helps you :

1. Change ++ to += 1 : for consistency with your previous code, and because channeling Doug Crockford's benevolent ghost may help to remove glitches from your javascript code.

2. You stitch together a bunch of images into a single 'sprite sheet'. The code then hangs when you want to go the next of these sheets. So, because your code does not work yet, as a test, remove all the loops cycling through the sub-images in each frame, and just directly output each frame in your frameImages array to canvas at each render to see if it still hangs. The result of this will let you know if your code can get to the next frame in frameImages or if it can't. This could lead you closer to the source of the bug.

3. Is currentFrame declared with var and in scope of processFrame ? If not, just put var currentFrame = 0; somewhere above where you define function processFrame(...

4. When you say "and then I play it like so" do you use a promise or equivalently call your animation start function after all the onloads have fired? A way to do this is : You can increment a counter once with each onload that fires and each time check if the counter is equal to the total number of loads needed, and only then call your animation start. My hunch is it is a caching issue as these seem to be the most variable between different browsers due to different internals.

5. Try clearing the canvas between drawImage calls and when you update currentFrame, since it may help to reset the internal state of the canvas when you change images.

6. You can do alpha and transparency with HTML5 video tags. Just use some svg transforms. These are easy to do as it says in this SO answer

Answer 2

Here's another, less orthodox solution - use the video as your canvas input and use getImageData to green screen each frame.

basically you've got your video tag and your canvas

<video id='source' style='display: none;'>
   <source>whatever.mp4</source>
   <source>whatever.ogg</source>
</video>
<canvas id='screen' width='videoWidth' height='videoHeight></canvas>

then you setup you draw function

var t = null;
document.getElementById('source').addEventListener('playing', function(){
   var vid = this;   
   t = setInterval(function(){
       if (vid.paused || vid.ended){
          clearInterval(t);
          return
       } else {
          drawFrame();
       }
   }, 20)
}, false)

and your draw frame function is

function drawFrame(){
   var ctx = document.getElementById('screen').getContext('2d'),
       vid = document.getElementById('source');

   ctx.drawImage(vid, 0, 0);
   var tolerance = 0.2,
       bgColor = [0, 255, 0], //or whatever 
       imageData = ctx.getImageData(0, 0, ctx.canvas.width, ctx.canvas.height),
       len = imageData.data.length;

   for (var i=0; i<len; i+=4){
      var r = imageData.data[i],
          g = imageData.data[i+1],
          b = imageData.data[i+2];

      //There's lots of different color comparing methods, just check google
      var dif = colorCompare([r, g, b], bgColor);
      if (dif < tolerance){
         imageData.data[i+3] = 255;
      }
   }

   ctx.putImageData(imageData, 0, 0);
}

You can play around with different parts of it to achieve the performance level you want. For the relatively small (dimension-wise) video you have performance shouldn't be an issue. You might have to use a buffer canvas to avoid screen flickering, but I doubt it. And you can adjust the tolerance variable to avoid any halo effect you might get.

Not saying it's the best solution, but it's definitely a workable solution, and one that would work if you wanted to change the video by just changing the input video with no extra work required.

Answer 3

Not sure if you've looked at this or not...but have you thought about utilizing the jquery queue? When I load it into my browser, i see that some frames don't show, or blanks out, then comes back, etc, etc. I've used queues before to do synchronous processing of say ajax calls that i wanted to "chain" together in a specific order....so that might be an area you could try to load each "pictured frame" in a synchronous way, ensuring each frames gets a load in the right order?

http://api.jquery.com/queue/