Is there any explanation why when tab is hidden, Canvas stops receive frames from OffscreenCanvas drawImage in worker thread?

Question

I have a video merger that gets different MediaStreamTrack from navigator.mediaDevices, Camera, and Display and sends them to a worker via MediaStreamTrackProcessor.reader which reads each stream frame by frame and draws into an OffscreenCanvas. So, the Canvas inside the main thread which is synced to that OffscreenCanvas would work just fine and renders each frame drawn into OffscreenCanvas, Until the tab gets blurred and the main thread goes hidden (background tab). In that case, the main canvas stops rendering any frame (with OffscreenCanvas working well on other side) which would be impossible to record its captured stream with MediaRecorder.

I already have the solution to get offscreen.transferToImageBitmap() and post it back to the main thread and then render it to the main canvas via canvasContext.transferFromImageBitmap(). And it is working all good. But it gets tricky on some gaming laptops and PCs with 2 or more GPUs installed and causes an fps drop from 60fps to something around 10fps. That would not be ideal for my application.

I have been searching for an answer for so many days now and it started to bother me why it's acting like that.

Any tips/answers leading to fix this issue and making that background tab effect go away would be much appreciated <3

Here is a sample extracted from my code, in charge of merging tracks in a worker thread:

const canvas = document.querySelector('#canvas');
const offscreen = canvas.transferControlToOffscreen();

const coWorker = () => {
    let offscreen;
    let ctx;
    let streams = new Map();
    let started = false;
    let framesCount = 0;
    let lastFramesCount = 0;
    let fpsReport = 0;

    setInterval(() => {
        console.log('worker_fps', fpsReport)
        fpsReport = framesCount - lastFramesCount;
        lastFramesCount = framesCount;
    }, 1000);

    const renderFrames = () => {
        streams.forEach(async (reader) => {
            const {value} = await reader.read();
            ctx.drawImage(value, 0,0, 1280, 720);
            value.close();
            framesCount += 1;
            renderFrames();
        });
    };

    self.onmessage = event => {
        const {type, id, readable} = event.data;
        switch (type) {
            case 'Canvas': {
                offscreen = event.data.offscreen;
                ctx = offscreen.getContext('2d');
                break;
            }
            case 'AddStream': {
                streams.set(id, readable.getReader());
                if (!started) {
                    started = true;
                    renderFrames();
                }
                break;
            }
            default:
                self.postMessage(`Error: ${event.data.msg}`, []);
        }
    };

}

const worker = new Worker(
    window.URL.createObjectURL(
        new Blob(["(" + coWorker.toString() + ")()"], {type: "text/javascript"})
    )
);

worker.postMessage({type: 'Canvas', offscreen}, [offscreen]);

// Starts here by getting screen-share video track

navigator.mediaDevices.getDisplayMedia({video: {frameRate: 60}}).then((stream) => {
    const videoTrack = stream.getVideoTracks()[0];
    if (videoTrack) {
        const processor = new MediaStreamTrackProcessor({track: videoTrack});
        worker.postMessage({type: 'AddStream', id: stream.id, readable: processor.readable,}, [processor.readable]);
    }
});

// recorder
const chunks = [];
const recorder = new MediaRecorder(canvas.captureStream(30), {mimeType: 'video/webm;'});
recorder.start();
const recorderInterval = setInterval(() => {
    recorder.requestData();
}, 1000)
recorder.ondataavailable = event => {
    const currentTarget = event.currentTarget;
    const blob = new Blob([event.data]);
    console.log('MediaRecorded:', currentTarget.state, event.data.size, currentTarget.mimeType, {
        data: event.data,
        blob,
    });
    if (event.data.size > 0) chunks.push(blob);
    if (chunks.length > 20 && recorder.state === 'recording') {
        recorder.stop();
        clearInterval(recorderInterval);
    }
};
recorder.onstop = () => {
    const data = new Blob(chunks, {type: this.type});
    const url = window.URL.createObjectURL(data);
    const a = document.createElement('a');
    a.style.display = 'none';
    a.href = url;
    a.download = 'test.webm';
    document.body.appendChild(a);
    a.click();
    setTimeout(() => {
        document.body.removeChild(a);
        window.URL.revokeObjectURL(url);
    }, 100);
};

When you run it and share your screen, until you stay on the tab it will work fine and record canvas stream. When you go to another tab or window, it will start recording 0 bytes.

Answer 1

The OffscreenCanvas API has been modeled so that it commits new frames to its placeholder <canvas> in requestAnimationFrame callbacks.
requestAnimationFrame has been modeled so that it runs only when new data can be painted to the monitor. When your tab is blurred, no new data can be painted to the monitor, so requestAnimationFrame callbacks won't fire. This means that your OffscreenCanvas's new drawings won't be committed to its placeholder <canvas>, and its captured MediaStream won't receive new data that your MediaRecorder will be able to record.

This is actually a design flaw. There used to be a .commit() method on the OffscreenCanvasRenderingContext2D interface, but it's been removed from implementations. Now implementations have to have some hooks that check when a <canvas> stream is being recorded and to not throttle requestAnimationFrame in that case. Firefox only recently did that for foreground requestAnimationFrame (they don't expose the Worker's one yet), and Chrome team is actively working on fixing it on their Worker's requestAnimationFrame.
So the best for you might unfortunately be to wait, though you may already have your case fixed in the latest Chrome Canary.