Video Seek Scroll (60FPS)

Question

Trying to achieve an effect of seeking through a video when the page is scrolled. This has been achieved by exporting all frames of the video to JPEG images, pre-loading the images, and rendering them to a canvas. However, this approach uses a lot of bandwidth and takes a long time to load on slow networks.

Trying to achieve the same effect by rendering a video to a canvas does not play as smoothly as the image-based approach.

Here is a working demo with the video-based approach:

https://codesandbox.io/s/infallible-chaum-grvi0r?file=/index.html

Since HTMLMediaElement.fastSeek() doesn't have widespread browser coverage, how can one achieve a realtime playback rate of 30-60 FPS?

Here is the relevant code for the effect (see CSB link above for the full code):

const video = document.querySelector("video");
const canvas = document.querySelector("canvas");
const ctx = canvas.getContext("2d");

(function tick() {
  requestAnimationFrame(tick);

  const { scrollHeight, clientHeight, scrollTop } = document.body;
  const maxScroll = scrollHeight - clientHeight;
  const scrollProgress = scrollTop / maxScroll;

  canvas.width = document.body.clientWidth;
  canvas.height = document.body.clientHeight;

  // This is the line that causes the issue
  video.currentTime = video.duration * scrollProgress;

  ctx.clearRect(0, 0, canvas.width, canvas.height);
  ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
})();

since `scroll*` properties don't update for every pixel user scrolls (aka it can jump from 0 to 100 in a single step), your best bet would be capture mouse wheel event instead? — vanowm, Apr 03 '22 at 19:03
@vanowm the issue is not with the scroll tracking; you can see this for yourself by adding a `console.log` inside the`requestAnimationFrame` callback. The issue is with the video not seeking fast enough to be rendered smoothly at 60 FPS — Enijar, Apr 03 '22 at 19:06

score 3 · Answer 1 · answered Apr 09 '22 at 00:01

The problem is due to specifics of drawImage() as noted here, it should be called only after firing of seeked event. The code below solves the problem by using a sentinel value (seeked = false) which will block the execution of drawImage() until the next seeked event:

const video = document.querySelector("video");
const canvas = document.querySelector("canvas");
const ctx = canvas.getContext("2d");
let seeked;
(function tick() {
    requestAnimationFrame(tick);
    if (seeked) {
        seeked = false;
        const { scrollHeight, clientHeight, scrollTop } = document.body;
        const maxScroll = scrollHeight - clientHeight;
        const scrollProgress = scrollTop / Math.max(maxScroll, 1);

        canvas.width = document.body.clientWidth;
        canvas.height = document.body.clientHeight;

        // This is the line that causes the issue
        video.currentTime = video.duration * scrollProgress;

        ctx.clearRect(0, 0, canvas.width, canvas.height);
        ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
    }
})();
video.addEventListener('seeked', () => {
    seeked = true;
});
video.currentTime = .001;

Thank you, and especially thank you for the reference to the MDN doc notes. I've not seen that note before. Thanks again! — Enijar, Apr 10 '22 at 15:30
@Enijar What FPS do you get with this? Seek speed depends on many things. If you have control over the video files you can increase it: [Keyframe interval](/q/30979714): smaller → faster seekable; framerate and video resolution: smaller → faster; GPU accelerated video decoding: x264(mp4) faster than unaccelerated webm. — cachius, May 09 '22 at 12:19
@cachius thanks for the advice, this seeks even faster now: https://codesandbox.io/s/fast-seek-video-q9zzv5?file=/index.html — Enijar, May 10 '22 at 08:35

Video Seek Scroll (60FPS)

1 Answers1