0

I'd like to be able to stream a website (e.g. youtube) as texture data into a WebGL scene. I've considered solutions where I'd simply overlay an iframe of the website on top of a WebGL scene, but ideally, I'd like to be able to use the website's content and render it as a texture into a WebGL scene for interesting post processing effects.

I've seen solutions where we I can snapshot the site (e.g. Html2Canvas) and stream the image data, however, being able to capture and render at 30+ fps would probably be way too expensive.

Any thoughts?

Detuned
  • 3,652
  • 4
  • 27
  • 54
  • @gman OP seems to know how to take a snapshot of a page, but none of the dupes you linked to did point to a solution able to render at 30+ fps – Kaiido Nov 22 '17 at 23:07
  • All of the links point out there is no solution at 30+ fps. In fact there's really no solution period, especially since the poster mentioned youtube and question strongly implies wanting a client side solution. [There maybe be a solution in the future](https://www.khronos.org/registry/webgl/extensions/proposals/WEBGL_texture_source_iframe/) though it won't allow streaming youtube, only 100% same origin domain iframes. – gman Nov 23 '17 at 03:53
  • @gman [MediaCapture](https://w3c.github.io/mediacapture-screen-share/) API allows screen-share and window-share as MediaStream, this is a solution @30+fps. – Kaiido Nov 23 '17 at 05:57
  • The question is closed but you may be interested in this: https://github.com/dtinth/html5-animation-video-renderer – Juribiyan Dec 15 '21 at 16:53

1 Answers1

3

One way would be using the being-defined MediaCapture API.
This will allow you to create a MediaStream from a application window, or a browser.

So you could run a browser server-side, stream its tab, grab it in a video element and then use this video as a texture.

However, I am not sure if any headless browser did implement anything like that currently.

Firefox and chrome do have something similar to this API in their mediaDevices.getUserMedia API, with the {video: { mediaSource: "window" }} constraint, but chrome does allow this only for chrome:// scripts (extensions).

Here is a rough proof of concept that will only work on Firefox for now.

Kaiido
  • 123,334
  • 13
  • 219
  • 285
  • I upvoted your answer since it provides a new solution. Does it belong on one of the other questions or do you think this is really a different question? Seems to me it's the same question. I'm sure all the others would like a fast solution as well. Currently though, your answer as worded is a little strange. First paragraph says "use this client side solution" and second paragraph says "So use a server". – gman Nov 23 '17 at 08:54
  • @gman the API is a WebAPI (front-side), but intended to be p2p, so to be able to navigate through this streamed page in the same time as seeing it in the receiving one, I guess going through server-side offers a better UX (like here in my fiddle, the user has to select the window, and can't really navigate through it while watching the anim. – Kaiido Nov 23 '17 at 08:58
  • As for other questions, I don't think they do really ask for a 30fps solution. html2canvas is still the best way for still images. – Kaiido Nov 23 '17 at 08:59
  • The first link mentions video. The last link mentions wanting events and wanting to be able cover animations (css animations are implied which run at 60fps). As for your 2 paragraphs, I get it now. Sorry it seemed confusing to me. – gman Nov 23 '17 at 09:08