Android - draw YouTube video on SurfaceTexture

Question

I'm trying to draw a WebView on SurfaceTexture so I can render it in OpenGL.

So far, I successfully played youtube video in WebView the standard way:

  ...
  webView.getSettings().setJavaScriptEnabled(true);

  if (Build.VERSION.SDK_INT < 8) 
  {
     webView.getSettings().setPluginsEnabled(true);
  } 
  else 
  {
     webView.getSettings().setPluginState(WebSettings.PluginState.ON);
  }
  webView.setWebChromeClient(new WebChromeClient() { });
  webView.setWebViewClient(new WebViewClient());
  ...

also the hardware acceleration is turned on:

 <application
        android:hardwareAccelerated="true"
 ...

And I also successfully rendered the WebView in OpenGL (except playing the video) based on this tutorial: http://anuraagsridhar.wordpress.com/2013/03/13/rendering-an-android-webview-or-for-that-matter-any-android-view-directly-to-opengl/

Additional initialization of the WebView (so it is always 'portrait' and fits into the texture):

 Point p = new Point();
 activity.getWindow().getWindowManager().getDefaultDisplay().getSize(p);
 webView.setLayoutParams(new ViewGroup.LayoutParams(p.x, p.y));
 surfaceTexture.setDefaultBufferSize(p.x, p.y);

Overriden onDraw method of the WebView:

@Override
public void onDraw(Canvas c)
{
   try
   {
      Point p = new Point();
      activity.getWindow().getWindowManager().getDefaultDisplay().getSize(p);
      Canvas canvas = surfaceTexture.lockCanvas(new Rect(0,0,p.x,p.y));
      canvas.save();
      canvas.translate(-scrollL, -scrollT);
      super.onDraw(canvas);
      canvas.restore();
      surfaceTexture.unlockCanvasAndPost(canvas);
   }
   catch (Surface.OutOfResourcesException e)
   {
      throw new RuntimeException(e);
   }
}

When trying to play YouTube video on webview rendered using the above hack, the video rectangle is black (however the sound can be heard). I suspect that the hardware acceleration does not work on surface texture.

So the question is: Is there any way to play a youtube video on OpenGL texture?

You've got an interesting problem here. The WebView is going to use a SurfaceView to play the video, but you don't know where to cut a hole in the texture where that SurfaceView should be displayed. Unless your texture is full screen, the video won't be in the right place anyway. This may not be possible. — Dave, Oct 10 '13 at 11:57
The video (whole webpage for that matter) IS in the right place. I achieved this by just making SurfaceTexture's buffer size fit all of the pixels of the webview (which is in fullscreen). — Piotr Trzpil, Oct 11 '13 at 10:24
Actually, you have a black rectangle where the video is supposed to be, right? The video itself is playing on a separate window behind the view hierarchy's window. The SurfaceView's most important job is usually to cut a hole in the view hierarchy window so you can see the other window. Rendering things as you are, nothing can cut that hole. If you could see it, the video would be where the WebView originally put since has no concept of being rendered on a texture. — Dave, Oct 11 '13 at 10:59
Thanks, I misunderstood your first comment. You might want to convert it to answer. — Piotr Trzpil, Oct 13 '13 at 20:29
I have actually looked at some of the Android source again trying to come up with a way to do this. Unfortunately, I haven't come up with much, but I'll post an answer... — Dave, Oct 14 '13 at 03:51
There is a request for enhancement open for this particular feature. Up to now it's completely unnoticed, but if more people will star it, maybe it could get some attention: https://code.google.com/p/gdata-issues/issues/detail?can=2&start=0&num=100&q=label%3AAPI-YouTube%20surface&colspec=API%20ID%20Type%20Status%20Priority%20Stars%20Summary&groupby=&sort=&id=4690 — Rick77, Apr 03 '15 at 23:29
Any update on this issue? My take is that videos and canvases are promoted to a native view inside the webview and for some reason, they are not correctly renderered in the surface texture. I tried using a framelayout that contains the webview and do the overriding of the dispatchDraw in it but did not work either. — Iker Jamardo Zugaza, Mar 26 '17 at 00:46
I spent whole day trying to fix it, and finally found this question. At least now I know, that it is an issue with Android, not my code — Lev Leontev, Jun 23 '19 at 16:37

score 7 · Accepted Answer · answered Oct 14 '13 at 04:22

The problem is this: the video is being drawn to a separate window that is behind the view hierarchy's window and is not readily accessible to be drawn on your SurfaceTexture. To complicate the matter, the WebView will have positioned and sized the video window without regard for itself being drawn to a Canvas other than the one that would be displayed on the screen. Furthermore, the video is being rendered in native code, and you wouldn't be able to draw to a Canvas fast enough to keep up anyway (that's the whole idea behind giving the video a separate window).

Any satisfactory solution would almost necessarily require use of the private APIs to get access to the video window in native code. Depending on how you plan to use your SurfaceTexture, you might come up with a workaround that involves intercepting URLs in a WebViewClient, stripping out any video tags before rendering, and placing the video in a SurfaceView or TextureView under your control. You can then put it where it should be in relation to where you've drawn the SurfaceTexture, but that limits your usage of the SurfaceTexture to billboarding. That sort of hack would get very ugly very quickly, though.

Put simply, I don't see a clean way to do this. It's a nice idea, but it's just not workable with the public API.

It seems this is doable. I found an Android app called [FullDive](https://play.google.com/store/apps/details?id=in.fulldive.shell). It's based on Cardboard and successfully renders Youtube videos onto opengl texture. — user2060386, Apr 26 '15 at 06:17
I tried to find that Window (to copy it to Bitmap using PixelCopy API) using this question https://stackoverflow.com/q/19669984, but it didn't actually appear! And, as WebView developer states here https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/3wrULcul8lw, WebView uses hardware compositing (though idk what it is), which is impossible to copy from... — Lev Leontev, Jan 09 '20 at 14:11

score 1 · Answer 2 · answered Jun 19 '19 at 20:27

1

Try using lockHardwareCanvas, added in API level 23.

answered Jun 19 '19 at 20:27

sirbrialliance

3,612
1
25
15

Android - draw YouTube video on SurfaceTexture

2 Answers2

Linked