8

I am building online web application which renders a video on a canvas and then records the canvas using canvas.captureStream() and mediaRecorder. The problem is that when the user switches tab or minimizes the window the canvas freezes. My animation keeps running as I have used webWorkerSetInterval(Hacktimer.js). As per chrome they have not yet provided a solution https://bugs.chromium.org/p/chromium/issues/detail?id=639105.

Can anyone suggest a work-around? I tried new window which doesn't allow to minimize but was unsuccessful. The recording doesn't stop when switching window.(stops only when tab is switched or minimized)

Amritesh Anand
  • 893
  • 11
  • 22
  • Possible duplicate of [CanvasCaptureMediaStream / MediaRecorder Frame Synchronization](https://stackoverflow.com/questions/40687010/canvascapturemediastream-mediarecorder-frame-synchronization) – Kaiido May 24 '17 at 11:24
  • Ps to close-vote: I know the questions look different, but the core issue is almost the same (rAF limitations there), and the solution will probably do for you too. – Kaiido May 24 '17 at 11:32
  • rAF limitations have already been eliminated by using WebWorker. But I am trying your solution don't close it now. Also this does not concern hidden canvas but inactive tabs. – Amritesh Anand May 24 '17 at 11:44
  • I checked your solution but the recording stops when I switch tabs. – Amritesh Anand May 24 '17 at 11:48
  • on chrome? it doesn't for me.... – Kaiido May 24 '17 at 11:48
  • yes on chrome can you provide a jsfiddle of your solution? – Amritesh Anand May 24 '17 at 11:49
  • I'm on my phone right now can't you jist copy the snippet on jsfiddle? Also, the snippet didn't worked on chrome until they fixed the bug mentioned on the answer (I filled it) – Kaiido May 24 '17 at 11:52
  • untested https://jsfiddle.net/96rck7r0/ – Kaiido May 24 '17 at 11:57
  • your example looks good I will have to experiment a little before closing/ edititng the question as to why it is not working for me. I am using three.js videoTexture and maybe that is causing some issue. Thanks will update the issue. – Amritesh Anand May 24 '17 at 12:02
  • It doesn't work for webgl context. Only works for 2d context. I tested it and the video hangs at the moment tab was changed. – Amritesh Anand May 24 '17 at 14:55
  • its intentional to stop the WebGL event loop when its tab is not active – Scott Stensland Jun 27 '17 at 01:16

1 Answers1

6

NB:
Now that this question has been specifically edited to treat webgl contexts, it may not be completely a duplicate of this previous answer, which indeed doesn't work with webgl contexts ; but because of an chrome bug...

So this answer will show you how to workaround this bug, while waiting for a fix from chrome.


The linked answer made use of the WebAudio API's timing method to create an Timed loop, not tied to the screen refresh-rate nor the window / tab's visibility.

But as said in the header, this currently doesn't work with webgl contexts on chrome.

The easy workaround, is to use an offscreen 2d context as the stream source, and to draw our webgl canvas onto this 2d context :

function startRecording(webgl_renderer, render_func) {
  // create a clone of the webgl canvas
  var canvas = webgl_renderer.domElement.cloneNode();
  // init an 2D context
  var ctx = canvas.getContext('2d');
  function anim(){
    // render the webgl Animation
    render_func();
    // draw the wegbl canvas on our 2D one
    ctx.clearRect(0,0,canvas.width, canvas.height);
   ctx.drawImage(webgl_renderer.domElement, 0,0);
  }
 var fps = 60;
  // start our loop @60fps
  var stopAnim = audioTimerLoop(anim, 1000 / fps);
  // maximum stream rate set as 60 fps
  var cStream = canvas.captureStream(fps);

  let chunks = [];
  var recorder = new MediaRecorder(cStream);
  recorder.ondataavailable = e => chunks.push(e.data);
  recorder.onstop = e => {
    // we can stop our loop
    stopAnim();
    var url = URL.createObjectURL(new Blob(chunks));
    var v = document.createElement('video');
    v.src = url;
    v.controls = true;
    document.body.appendChild(v);
  }
  recorder.start();
  // stops the recorder in 20s, try to change tab during this time
  setTimeout(function() {
    recorder.stop();
  }, 20000);
  btn.parentNode.removeChild(btn);
}


/*
    An alternative timing loop, based on AudioContext's clock

    @arg callback : a callback function 
        with the audioContext's currentTime passed as unique argument
    @arg frequency : float in ms;
    @returns : a stop function

*/
function audioTimerLoop(callback, frequency) {

  var freq = frequency / 1000;      // AudioContext time parameters are in seconds
  var aCtx = new AudioContext();
  // Chrome needs our oscillator node to be attached to the destination
  // So we create a silent Gain Node
  var silence = aCtx.createGain();
  silence.gain.value = 0;
  silence.connect(aCtx.destination);

  onOSCend();

  var stopped = false;       // A flag to know when we'll stop the loop
  function onOSCend() {
    var osc = aCtx.createOscillator();
    osc.onended = onOSCend; // so we can loop
    osc.connect(silence);
    osc.start(0); // start it now
    osc.stop(aCtx.currentTime + freq); // stop it next frame
    callback(aCtx.currentTime); // one frame is done
    if (stopped) {  // user broke the loop
      osc.onended = function() {
        aCtx.close(); // clear the audioContext
        return;
      };
    }
  };
  // return a function to stop our loop
  return function() {
    stopped = true;
  };
}

/* global THREE */
/* Note that all rAF loop have been removed
   since they're now handled by our 'audioTimerLoop' */


(function() {

    'use strict';
    var WIDTH = 500, HEIGHT = 500;
    var scene = new THREE.Scene();
    var camera = new THREE.PerspectiveCamera(75, WIDTH / HEIGHT, 0.1, 1000);

    var renderer = new THREE.WebGLRenderer();
    renderer.setSize(WIDTH , HEIGHT);
    document.body.appendChild(renderer.domElement);

    var geometry = new THREE.CubeGeometry(5, 5, 5);
    var material = new THREE.MeshLambertMaterial({
        color: 0x00fff0
    });
    var cube = new THREE.Mesh(geometry, material);
    scene.add(cube);

    camera.position.z = 12;
    
    var pointLight = new THREE.PointLight(0xFFFFFF);

    pointLight.position.x = 10;
    pointLight.position.y = 50;
    pointLight.position.z = 130;

    scene.add(pointLight);

    var render = function() {        
        var delta = Math.random() * (0.06 - 0.02) + 0.02;

        cube.rotation.x += delta;
        cube.rotation.y += delta;
        cube.rotation.z -= delta;

        renderer.render(scene, camera);
    };
    render();
    console.clear();
    
    btn.onclick = function(){startRecording(renderer, render);};

}());
body {
    margin: 0;
    background: #000;
}
button{
  position: absolute;
  top: 0;
  }
<script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/85/three.min.js"></script>
<!-- Mobile devices need an user interaction to start the WebAudio API -->
<button id="btn">Start</button>
Scott Stensland
  • 26,870
  • 12
  • 93
  • 104
Kaiido
  • 123,334
  • 13
  • 219
  • 285
  • Aweome hack. But the drawImage is a bottleneck for my situtaiton as it takes around 80-90ms. I want to achieve atleast 8 fps of recording. But with this method a lot of frames are skipped even at a very slow frame rate. Can you suggest an optimization? Can we bind 2d canvas and gl canvas or speed up the process using context.readPixels? – Amritesh Anand May 25 '17 at 12:34
  • @AmriteshAnand, 80ms oO!? What's the size of your canvas ? With the same animation as in the snippet, but with an 2500*2500 canvas, the longest time I got for `clearRect + drawImage` was 10ms, with an average of 1.22ms. I didn't got 60 fps either, but the culprit was in webgl part, not in drawImage. And AFAIK, drawImage is the fastest way to draw an webgl canvas onto a 2d one. – Kaiido May 25 '17 at 14:20
  • Thanks a lot, I tried on mac and it gave me good performance. – Amritesh Anand May 26 '17 at 12:11