3

I'm pretty new to opengl-es and currently have a problem rendering my video output on a screen in Unity.

I was developing a video player project with Unity. I bought the EasyMovieTexture plugin and replaced the video player module with another open-source video player (Ijkplayer) years ago which worked fine all the time.

Now I want to replace it with newer VLC using libvlcjni. I compiled and just replaced the old Ijkplayer but it didn't work as I expected. The screen just started flashing one color from every video frame but the video was going and the audio track was normally playing too.

Screenshot - A test scene with a screen, sorry there's mistake with texcoord but only color flashing

I'd like to provide some further information hope that I can find some help (Sorry if I have some mistakes or misunderstandings):

  1. As far as I know, these video player modules need a Surface (Or SurfaceTexture) in the Android layer and the video decoder will work as a data producer. The SurfaceTexture consumes data from the producer and converts it to the texture with the type of GL_TEXTURE_EXTERNAL_OES which can be directly consumed and displayed with components like TextureView. But this texture data cannot be consumed in the Unity layer unless I use a GLSL shader directly sampling my OES texture data, like this:
    // Unity shader with GLSL
    GLSLPROGRAM
        #pragma only_renderers gles3
        #include "UnityCG.glslinc"
    // ...
    // Ignoring vertex shader
    // ...
        in vec2 textureCoord;
        layout(binding = 0) uniform samplerExternalOES _MainTex;
        out vec4 fragColor;
        void main()
        {
            fragColor  = texture(_MainTex, textureCoord);
        }
    ENDGLSL
    
  2. My approach was to convert the texture to GL_TEXTURE_2D with the native library which came along with the EasyMovieTexture plugin. Here I cannot provide the source code of this .so library but I've decompiled it in IDAPro and I know it can work along with GLES and render the external texture data to another 2d texture using FrameBuffer Object and external shader program.
    • Here is a random example to explain the procedure, it is NOT the accurate code from the binary: FilterFBOTexture.java
    • Though I cannot edit the .so file, luckily it was reading two external files as the shader program:
    // vertex shader
    attribute highp   vec3  inVertex;
    attribute mediump vec3  inNormal;
    attribute mediump vec2  inTexCoord;
    uniform highp   mat4  MVPMatrix;
    uniform mediump vec2 TexCoordMove;
    varying mediump vec2   TexCoord;
    
    void main()
    {
        highp vec4 vPos = vec4(0,0,0,1);
    
        vPos.x = ( inTexCoord.x * 2.0 - 1.0 );
        vPos.y = ( inTexCoord.y * 2.0 - 1.0 );
    
        gl_Position = vPos;
    
        mediump vec4 vec4Temp  = vec4(inTexCoord.x - TexCoordMove.x,inTexCoord.y - TexCoordMove.y,0,1.0);
        vec4Temp  = MVPMatrix * vec4Temp;
        vec4Temp.xyz = vec4Temp.xyz / vec4Temp.w;
        TexCoord = vec4Temp.xy;
    }
    
    // fragment shader
    #extension GL_OES_EGL_image_external : require
    uniform samplerExternalOES sTexture;
    uniform lowp float AlphaValue;
    varying mediump vec2  TexCoord;
    void main()
    {
        lowp vec4 color = texture2D(sTexture, TexCoord) ;
        color = vec4(color.rgb, color.a * AlphaValue);
        gl_FragColor = color;
    }
    

I don't know whether I have to check the vertex shader or just dive into the source code of libvlcjni so that I can correctly render my video output. Any idea will be grateful, thanks.

Update 2022-11-4:

I turned around and began to use VLC for Android.

Big appreciation to @mfkl, I created an issue days ago on the VLC repo.

https://code.videolan.org/videolan/vlc-unity/-/issues/164

The problem still remains but at least I can work for something now.

Virtroid
  • 43
  • 7
  • Can you successfully render the converted GL_TEXTURE_2D in android? – shingo Oct 08 '22 at 10:43
  • @shingo I think the answer is no, sorry I'm also new to android too and at least I didn't develop any pure Android program before. I just tested with a TextureView component and it worked fine with VLC. But when I was setting a new SurfaceTexture object to the TextureView or GLSurfaceView, it did not even show the color and only the audio track playing. – Virtroid Oct 08 '22 at 10:53
  • I've tried converting texture to GL_TEXTURE_2D with only GLES30 API but didn't succeed. And It was difficult to debug my aar plugin in Unity program. – Virtroid Oct 08 '22 at 10:56
  • @shingo is it better to try and confirm whether I can render the converted GL_TEXTURE_2D in android? In fact, I've read many articles and tried for over 2 weeks now. Sadly I cannot stick to this question cause I have too much to learn about opengl-es, android or vlc and also I have some other tasks to deal with :P – Virtroid Oct 08 '22 at 11:16
  • This looks like a conversion issue. My opinion 1) Yes, try to test the converted texture in android first. 2) Start with a simplest shader. – shingo Oct 08 '22 at 11:21
  • @shingo thanks. I will try to figure that out first. – Virtroid Oct 09 '22 at 02:19
  • https://code.videolan.org/videolan/vlc-unity – mfkl Oct 11 '22 at 01:48
  • @mfkl yes I‘ve learned about the repositories provided by VideoLAN official. The earlier one was built for the standalone platform and recently there is also a solution for android. But I checked the package content and the plugin called "libVLCUnityPlugin" is a black box which means I may not have enough flexibility to fix or modify my video plugin cause the plugin is not open-source. – Virtroid Oct 11 '22 at 02:33
  • Also in the unity assets store, these plugin packages are sold for $100 – Virtroid Oct 11 '22 at 02:34
  • The plugin is fully opensource, there is no blackbox. – mfkl Oct 11 '22 at 08:46
  • @mfkl sorry and yes, I see the plugin source for video rendering. But I don't know the difference between the open-source repo and the asset in the unity store. Is technical support the only difference between these options? – Virtroid Oct 12 '22 at 03:17
  • 1
    please take a look on that question for anyone having the answer: https://stackoverflow.com/questions/35561796/allow-for-multiple-pass-renders-in-glsurfaceview-renderer-with-surfacetexture I have started a bounty there – Mohammad Elsayed Feb 08 '23 at 21:11

0 Answers0