I'm pretty new to opengl-es and currently have a problem rendering my video output on a screen in Unity.
I was developing a video player project with Unity. I bought the EasyMovieTexture plugin and replaced the video player module with another open-source video player (Ijkplayer) years ago which worked fine all the time.
Now I want to replace it with newer VLC using libvlcjni. I compiled and just replaced the old Ijkplayer but it didn't work as I expected. The screen just started flashing one color from every video frame but the video was going and the audio track was normally playing too.
Screenshot - A test scene with a screen, sorry there's mistake with texcoord but only color flashing
I'd like to provide some further information hope that I can find some help (Sorry if I have some mistakes or misunderstandings):
- As far as I know, these video player modules need a
Surface
(OrSurfaceTexture
) in the Android layer and the video decoder will work as a data producer. TheSurfaceTexture
consumes data from the producer and converts it to the texture with the type ofGL_TEXTURE_EXTERNAL_OES
which can be directly consumed and displayed with components likeTextureView
. But this texture data cannot be consumed in the Unity layer unless I use a GLSL shader directly sampling my OES texture data, like this:// Unity shader with GLSL GLSLPROGRAM #pragma only_renderers gles3 #include "UnityCG.glslinc" // ... // Ignoring vertex shader // ... in vec2 textureCoord; layout(binding = 0) uniform samplerExternalOES _MainTex; out vec4 fragColor; void main() { fragColor = texture(_MainTex, textureCoord); } ENDGLSL
- My approach was to convert the texture to
GL_TEXTURE_2D
with the native library which came along with the EasyMovieTexture plugin. Here I cannot provide the source code of this.so
library but I've decompiled it in IDAPro and I know it can work along with GLES and render the external texture data to another 2d texture using FrameBuffer Object and external shader program.- Here is a random example to explain the procedure, it is NOT the accurate code from the binary: FilterFBOTexture.java
- Though I cannot edit the
.so
file, luckily it was reading two external files as the shader program:
// vertex shader attribute highp vec3 inVertex; attribute mediump vec3 inNormal; attribute mediump vec2 inTexCoord; uniform highp mat4 MVPMatrix; uniform mediump vec2 TexCoordMove; varying mediump vec2 TexCoord; void main() { highp vec4 vPos = vec4(0,0,0,1); vPos.x = ( inTexCoord.x * 2.0 - 1.0 ); vPos.y = ( inTexCoord.y * 2.0 - 1.0 ); gl_Position = vPos; mediump vec4 vec4Temp = vec4(inTexCoord.x - TexCoordMove.x,inTexCoord.y - TexCoordMove.y,0,1.0); vec4Temp = MVPMatrix * vec4Temp; vec4Temp.xyz = vec4Temp.xyz / vec4Temp.w; TexCoord = vec4Temp.xy; }
// fragment shader #extension GL_OES_EGL_image_external : require uniform samplerExternalOES sTexture; uniform lowp float AlphaValue; varying mediump vec2 TexCoord; void main() { lowp vec4 color = texture2D(sTexture, TexCoord) ; color = vec4(color.rgb, color.a * AlphaValue); gl_FragColor = color; }
I don't know whether I have to check the vertex shader or just dive into the source code of libvlcjni so that I can correctly render my video output. Any idea will be grateful, thanks.
Update 2022-11-4:
I turned around and began to use VLC for Android.
Big appreciation to @mfkl, I created an issue days ago on the VLC repo.
https://code.videolan.org/videolan/vlc-unity/-/issues/164
The problem still remains but at least I can work for something now.