5

I've recently been trying to generating video in the browser, and have thus been playing with two approaches:

The whammy approach works well, but is only supported in Chrome, since it's the only browser that currently supports webp encoding (canvas.toDataURL("image/webp")). And so I'm using the captureStream approach as a backup for Firefox (and using libwebpjs for Safari).

So now on to my question: Is there a way to control the video quality of the canvas stream? And if not, has something like this been considered by the browsers / w3c?

Here's a screenshot of one of the frames of the video generated by whammy:

using whammy js library to combine webp frames into webm video

And here's the same frame generated by the MediaRecorder/canvas.captureStream approach:

using MediaRecorder and canvas.captureStream

My first thought is to artificially increase the resolution of the canvas that I'm streaming, but I don't want the output video to be bigger.

I've tried increasing the frame rate passed to the captureStream method (thinking that there may be some strange frame interpolation stuff happening), but this doesn't help. It actually degrades quality if I make it too high. My current theory is that the browser decides on the quality of the stream based on how much computational power it has access to. This makes sense, because if it's going to keep up with the frame rate that I've specified, then something has to give.

So the next thought is that I should slow down the rate at which I'm feeding the canvas with images, and then proportionally lower the FPS value that I pass into captureStream, but the problem with that is that even though I'd likely have solved the quality problem, I'd then end up with a video that runs slower than it's meant to.

Edit: Here's a rough sketch of the code that I'm using, in case it helps anyone in a similar situation.

  • Can you share some of your code? – Helder Sepulveda Sep 17 '18 at 19:25
  • 1
    These are compression artifacts, and I'm afraid there no such options no... So first, these are not due to captureStream. You can check it by passing your stream to a videoElement's srcObject directly and you won't notice as much artifacts. The main problem is that most common video codecs are not meant for displaying such graphics, think of it as what JPEG does. There is the videoBitsPerSecond option of the MediaRecorder but it will probably won't help here. (I tried and it didn't). On Chrome, I've got slightly less horrible results using vp8 codec than any other. – Kaiido Sep 18 '18 at 02:22
  • @HelderSepu I've tried to strip away the irrelevant parts of my code, so this may have a bug or two: https://gist.github.com/josephrocca/ec073b3a90f936bec87bbd8e3e4c3486 –  Sep 18 '18 at 03:39
  • Just a note about your gist: `if(MediaRecorder)` should be `if(window.MediaRecorder)` otherwise you'll throw a ReferenceError, and instead of a 1ms setTimeout loop, you'd be better run a requestAnimationFrame one (I don't think you can record at more than 60FPS anyway) – Kaiido Sep 18 '18 at 05:30
  • Ah, thanks for the `MeidiaRecorder` catch. The reason I went for `setTimeout` rather than `requestAnimationFrame` is because I figured that the latter is more prone to be throttled by the browser in case of lag - I'd prefer the user's browser to get laggy than to risk skipping frames. I also didn't set the timeout to `1/opts.fps` because I figured that I should ideally update the canvas much faster than the `captureStream` frame rate or else I'd risk a sort of [temporal aliasing](https://en.wikipedia.org/wiki/Stroboscopic_effect) effect. I might be mistaken there though. –  Sep 18 '18 at 06:25
  • setTimeout will get throttled the same way as rAF... See [this Q/A](https://stackoverflow.com/questions/40687010/canvascapturemediastream-mediarecorder-frame-synchronization/40691112#40691112) for an alternative timing method which shouldn't get throttled. But note that this throttling is only for when the window is blurred. If it's beeing throttled because the browser is busy, even your canvas drawings will get throttled anyway. – Kaiido Sep 19 '18 at 01:59
  • Okay, switched the gist to `requestAnimationFrame`. I did some tests and neither approach seems to be throttled in Chrome or Firefox when the tab/browser is unfocussed. Bit confused by that. –  Sep 19 '18 at 05:21

1 Answers1

5

These are compression artifacts, and there is not much you can do for now...

Video codecs are built mainly with the idea of showing real-life colors and shapes, a bit like JPEG with a really low quality. They will also do their best to keep as less information as they can between keyframes (some using motion detection algorithm) so that they need less data to be stored.

These codecs normally have some configurable settings that will allow us to improve the constant-quality of the encoding, but MediaRecorder's specs being codec agnostic, they didn't provide (yet) an option in the API for us web-devs to set any other option than a fixed bit-rate (which won't help us more in here).

There is this proposal, which asks for such a feature though.

Kaiido
  • 123,334
  • 13
  • 219
  • 285
  • Thanks for looking into this! Great to see that this stuff is being discussed by w3c people. –  Sep 18 '18 at 03:38
  • I'm curious as to why the fixed bit-rate stuff (i.e. the [`videoBitsPerSecond`](https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/MediaRecorder) option in the `MediaRecorder` constructor) wouldn't help improve the quality here? Isn't that what the bit-rate is meant to control? I must have some misunderstanding. –  Feb 25 '19 at 11:21
  • @Joe, first there is no CBR option, videoBitsPerSecond is only the target rate of whatever mode (I guess it's usually VBR). Then I guess that what masters the video quality in VP8 is actually an other option: https://www.webmproject.org/docs/encoder-parameters/ – Kaiido Feb 25 '19 at 15:54
  • 3
    Actually, increasing videoBitsPerSecond makes things look better – Max Aug 08 '19 at 12:00