How to "apply fragment shader to GLSurfaceView/TextureView"?

Question

My goal is to have ExoPlayer display a color video as black and white.

According to this this Github issue I should be able to achieve this with Open GL:

ExoPlayer will render video to any Surface. You could use SurfaceTexture, at which point you'd have video rendered into an OpenGL ES texture. Once you have that you can do anything that OpenGL lets you do, including using a pixel shader to transform the video to black and white.

In an older, but related, discussion in the Android Developers Google group Romain Guy gives some details as to how this should be done:

Create an OpenGL context

Generate an OpenGL texture name

Create a SurfaceTexture with the texture name

Pass the SurfaceTexture to Camera

Listen for updates On SurfaceTexture update, draw the texture with OpenGL using the shader you want

Simple :)

And by playing around with Google's Cardboard example project I have determined that a fragment shader like the following one should be about right:

precision mediump float;
varying vec4 v_Color;

void main() {
    float grayscale = v_Color[0] * 0.3 + v_Color[1] * 0.59 + v_Color[2] * 0.11;
    gl_FragColor = vec4(grayscale, grayscale, grayscale, 0.1);
}

I am also managed to have ExoPlayer render into a TextureView instead of the usual SurfaceView:

mPlayer.sendMessage(
    videoRenderer,
    MediaCodecVideoTrackRenderer.MSG_SET_SURFACE,
    new Surface(mVideoTextureView.getSurfaceTexture())
);

Now, how do I wire everything together?

Where can I "apply" the shader to the TextureView? Is that even possible or should I use a GLSurfaceView instead? What would I need to do in the Renderer?

Have you seen this: https://github.com/google/ExoPlayer/issues/1220 and the link within it? — Morrison Chang, Feb 26 '16 at 16:19
Thanks, the linked blog post did indeed provide a working solution! Thanks a lot! — david.mihola, Mar 09 '16 at 12:54
Hey david, I am also trying to do the same thing but I'm stuck on this for the past 2 days. I took a look at the VidEffect's VideoSurfaceView code but when I integrate exoplayer the video is rendered black but with audio. If you still have the working solution, it would mean a lot if you can share that. — Udit Mukherjee, Feb 15 '17 at 12:26

score 1 · Accepted Answer · edited May 23 '17 at 12:03

1

I have applied fragment shaders to a video being played by an ExoPlayer before. The most effective way seems to be to play the video onto a GLSurfaceView, since they have good support for custom renderers.

A working example of applying OpenGL shaders to a GLSurfaceView is available from here. It even has a greyscale fragment shader already as part of its effects library. The existing code will require some minor refactoring since it uses a MediaPlayer rather than an ExoPlayer.

edited May 23 '17 at 12:03

Community

1
1

answered Mar 05 '16 at 22:35

M.S.

1,091
1
11
27

1

Thanks! I built my solution from the blog post that @Morrison Chang's comment led me to - it uses a `TextureView` and passes the `Surface` to the ExoPlayer. But maybe I'll still have switch to `GLSurfaceView` because I'll also need to support VisualOn and that player only accepts a `SurfaceView` (not a `Surface` directly). – david.mihola Mar 09 '16 at 12:57

How to "apply fragment shader to GLSurfaceView/TextureView"?

1 Answers1