SurfaceTexture in Android plugin doesn't work in Unity

Question

I can't get the texture tied to a SurfaceTexture to display in Unity.

Update 4: Based on the pipeline in update 1 (surface->external texture via surface texture -> fbo -> texture 2d) I know the SurfaceTexture isn't properly converting its surface to a texture. I can get correctly drawn pictures from its surface via pixelcopy and I can confirm my FBO drawing to texture2d pipeline works with some test colors. So the question is, why can't the SurfaceTexture convert its surface to a texture?

I generate a Texture in Java and pass its pointer back to Unity:

public void initGLTexture()
{
    Log.d("Unity", "initGLTexture");
    int textures[] = new int[1];
    GLES20.glGenTextures(1, textures, 0);
    mTextureId = textures[0];

    GLES20.glBindTexture(GL_TEXTURE_EXTERNAL_OES, mTextureId);
    GLES20.glTexParameterf(GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR);
    GLES20.glTexParameterf(GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
    GLES20.glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    GLES20.glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
}

I create a SurfaceTexture from the id (in Java):

mSurfaceTexture = new SurfaceTexture(mTextureId);
mSurfaceTexture.setDefaultBufferSize(512, 512);

I use a third-party library, GeckoView, to render onto the Surface of the SurfaceTexture. I call the following method from Unity's OnRenderObject() to keep all GL rendering on the same thread:

mSurfaceTexture.updateTexImage();

I know the above code allows proper drawing onto the surface.

I call the following in Unity to load the texture:

_imageTexture2D = Texture2D.CreateExternalTexture(
        512,512,TextureFormat.RGBA32,false,true,(IntPtr) mTextureId);
_rawImage.texture = _imageTexture2D;

Why does the RawImage with the texture applied show only this sprite-looking thing, which should be a webpage?

Update 1: So I've been working on the hypothesis of: use Gecko to draw to the Surface, and use a SurfaceTexture to render this surface to a GL_TEXTURE_EXTERNAL_OES. Since I can't display this on Unity (not sure why) I am drawing this texture to a frame buffer and copying the pixels in the framebuffer to a GL_TEXTURE_2D. I am getting a web page in the texture_2d (in the emulator with an imageview and glReadPixels). However, when I import the work into Unity to test if the pipeline is okay thus far I just get a black screen. I CAN get images of the surface via the PixelCopy api.

Here is my FBO overview code - my rendering code comes from grafika's texture2D program:

    // bind display buffer
    GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, mFrameBufferId);
    GlUtil.checkGlError("glbindframebuffer");

    // unbind external texture to make sure it's fresh
    GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
    GlUtil.checkGlError("glunbindexternaltex");

   // bind source texture (done in drawFrame as well )
    GLES20.glBindTexture(GL_TEXTURE_EXTERNAL_OES, mOffscreenTextureId);
    GlUtil.checkGlError("glBindFramebuffer");

    // draw to frame buffer
    GLES20.glClearColor(0.0f, 0.0f, 0.0f, 1.0f);    // again, only really need to
    GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);     //  clear pixels outside rect

    mFullScreen.drawFrame(mOffscreenTextureId, mIdentityMatrix);

    // unbind source texture
    GLES20.glBindTexture(GLES20.GL_TEXTURE_2D,0);
    GlUtil.checkGlError("glBindTexture2d");

    GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
    GlUtil.checkGlError("glunbindexternaltex");

    // make sure we're still bound to fbo
    GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, mFrameBufferId);
    GlUtil.checkGlError("glBindTexture2d");

    // copy pixels from frame buffer to display texture
     GLES20.glCopyTexImage2D(GLES20.GL_TEXTURE_2D,0,GLES20.GL_RGBA,0,0,512,512,0);

    // read pixels from the display buffer to imageview for debugging
    BitmapDisplay.mBitmap = SavePixels(0,0,512,512);

    // unbind texture
    GLES20.glBindTexture(GLES20.GL_TEXTURE_2D,0);

Here's my player settings > other:

Update 2: Possible pipeline to try: call the draw function of the external texture to FBO (attached to Unity's texture_2d) in C++ via this interface.

Update 3: Calling the Java functions from native code that are responsible for drawing the texture from the SurfaceTexture to the FBO to Unity's texture via the GL.IssuePluginEvent produce a black texture as in the first update. It will show images in the emulator but not in Unity.

@Shaman do you have one of the pipelines' logs in mind? e.g. 1. try and display the external texture attached to the surface texture, 2. the external texture-> fbo -> teture_2d for display, 3. the same as 2 but with a `textureview` attached to the surface texture — pale bone, Jan 20 '19 at 19:07
Alright, can you show me Player settings -> Other settings? (Unity) — Hamid Yusifli, Jan 20 '19 at 19:17
https://docs.unity3d.com/Manual/NativePluginInterface.html probably has to be used. The Unity texture reference should be used in Android, to fill with data, instead to the SurfaceTexture. — Maistor Kokir, Jan 26 '19 at 00:01
Do you know how to use the gpu to get data off of a surface onto a unity texture2d? The only way I've seen is to use a surfacetexture, which only uses external textures — pale bone, Jan 26 '19 at 00:14
From what I see, yes SurfaceTexture can't be avoided, if you want to get the camera frames in Android, and yes the target is GL_TEXTURE_EXTERNAL_OES. — Maistor Kokir, Jan 26 '19 at 01:51

score 2 · Answer 1 · answered Jan 22 '19 at 15:33

2

I had to do a similar task a couple of months ago and found out that the correct pipeline is creating a texture in Unity, obtaining a native pointer in C and finally updating it in the Java layer.

Please take a look at this sample project, it should give you a different perspective.

https://github.com/robsondepaula/unity-android-native-camera

Regards

answered Jan 22 '19 at 15:33

Robson de Paula

39
8

I'll check this out and get back to you – pale bone Jan 22 '19 at 18:18
You're using an `ImageReader` to get frames, do you know the performance implications for this? I.e. impact on CPU and frame rate - I want to render videos, especially youtube vids which is why I'm using a texture and opengl calls. – pale bone Jan 22 '19 at 20:33
At the time, the camera feed was not available using Android NDK hence this design. To speed up things a little, some color transformations are done using Renderscript which offloads some calculations on the GPU. Since you are aiming for videos, and possible higher resolutions than the camera preview you should stay on your direction. Perhaps that source code can help you on getting the right texture pointer? – Robson de Paula Jan 23 '19 at 11:39
Trying to give you more clearer suggestion, sorry. Instead of creating the texture in Java and passing to Unity, try the other way around. Create the texture in Unity, get the pointer in C, and give it to Java (the provided git helps on this task). Let's see what happens keeping the code you already have but changing the texture creation part. Cheers – Robson de Paula Jan 23 '19 at 11:51
Gotcha, thanks for the suggestion I'll try it out today. – pale bone Jan 23 '19 at 14:06
Made sure this worked on the emulator but when I port it to unity I still get just a black texture. – pale bone Jan 24 '19 at 18:26
Furthermore, what is the value of getting the pointer to the texture in C? It's my understanding that it's simply an integer that any language can use to reference the texture from the GPU. – pale bone Jan 24 '19 at 20:07
Well this came after troubleshooting, the value got in Unity or Java didn't work when the value obtained in C did. If you can trim down your code to unit test this pipeline I might be able to take a look and see if I can help. – Robson de Paula Jan 25 '19 at 13:19
I appreciate your offer to help as well as the repo to look at, both of which are kind. However, I've A/B tested the rendering pipeline I have thus far calling native code that calls java code via `GL.IssuePluginEvent` as well as just calling the java code in Unity's `OnRenderObject()` loop. They both give the same results. I can confirm drawing the FBO to the texture2d as well as the library drawing on my surface works in both cases. In neither case does the `SurfaceTexture` successfully convert its surface to a texture. If your post's help becomes apparent later on I'll mark as the answer. – pale bone Jan 26 '19 at 04:34
No problem @palebone, glad to participate on this thread. Hope you get it working. Cheers! – Robson de Paula Jan 26 '19 at 21:20
@palebone I'm curious what you ended up with? I've got a demo working with video recordings and it would be interesting to compare with your approach. – Troy Oct 21 '19 at 11:56
1

@TroyLamerton check out OVROverlay: https://developer.oculus.com/documentation/unity/latest/concepts/unity-ovroverlay/?locale=en_US – pale bone Oct 21 '19 at 20:14
1

Haven't solved it yet. Drawing a Surface or SurfaceTexture into Unity is super useful so I've added a bounty to this question: https://stackoverflow.com/questions/35227222/rendering-surfacetexture-to-unity-texture2d – Troy Nov 27 '19 at 07:48

SurfaceTexture in Android plugin doesn't work in Unity

1 Answers1

Linked