I'm having a weird issue with my OpenGL/Skia Android Camera2 app.
My Camera renders frames into a SurfaceTexture, which is a GL_TEXTURE_EXTERNAL_OES texture in OpenGL.
I can then simply render this OpenGL texture to all outputs (1920x1080 Preview EGLSurface, 4000x2000 Video Recorder EGLSurface) using a simple pass-through shader.
Camera --> GL_TEXTURE_EXTERNAL_OES
GL_TEXTURE_EXTERNAL_OES --> PassThroughShader --> Preview Output EGLSurface
GL_TEXTURE_EXTERNAL_OES --> PassThroughShader --> Video Recorder Output EGLSurface
Now I want to introduce Skia into this, which allows me to render onto the Camera Frame before passing it along to my outputs (e.g. to draw a red box onto the Frame). Since I can't directly render onto the same GL_TEXTURE_EXTERNAL_OES again, I created a separate offscreen texture (GL_TEXTURE_2D), and a separate offscreen frame buffer (FBO1) and attached them.
Now, when I render to the FBO1, the offscreen texture GL_TEXTURE_2D gets updated, and I want to pass that GL_TEXTURE_2D to my outputs then:
Camera --> GL_TEXTURE_EXTERNAL_OES
GL_TEXTURE_EXTERNAL_OES --> Skia to FBO1 + drawing a red box --> GL_TEXTURE_2D
GL_TEXTURE_2D --> PassThroughShader --> Preview Output EGLSurface
GL_TEXTURE_2D --> PassThroughShader --> Video Recorder Output EGLSurface
But for some reason, this draws the first frame only, then gets stuck and renders weird glitching artefacts on the screen. See video demo here: https://github.com/mrousavy/react-native-vision-camera/assets/15199031/254a5455-b9cf-4c1e-9cb9-85f9b60d0cd5
The relevant files are:
- onFrame(..): Called once the Camera put a new Frame into
_inputTexture(GL_TEXTURE_EXTERNAL_OES).// Camera texture OpenGLTexture& cameraTexture = _inputTexture.value(); // Render to new texture using Skia auto newTexture = skia->renderFrame(_glContext, cameraTexture); // Reset the bindings glBindTexture(GL_TEXTURE_2D, newTexture.id); glBindFramebuffer(GL_FRAMEBUFFER, DEFAULT_FRAMEBUFFER); // Render to all outputs if (_previewOutput) { _previewOutput->renderTextureToSurface(newTexture, transformMatrix); } if (_recordingSessionOutput) { _recordingSessionOutput->renderTextureToSurface(newTexture, transformMatrix); } - SkiaRenderer::renderFrame(..): Called to render the Frame onto an offscreen texture (
GL_TEXTURE_2D) and add some additional Skia commands (draw a red rectangle).if (_skiaContext == nullptr) { _skiaContext = GrDirectContext::MakeGL(); } if (_offscreenSurface == nullptr) { GrBackendTexture skiaTex = _skiaContext->createBackendTexture(cameraTexture.width, cameraTexture.height, SkColorType::kN32_SkColorType, GrMipMapped::kNo, GrRenderable::kYes); GrGLTextureInfo info; skiaTex.getGLTextureInfo(&info); _offscreenSurfaceTextureId = info.fID; SkSurfaceProps props(0, kUnknown_SkPixelGeometry); _offscreenSurface = SkSurfaces::WrapBackendTexture(_skiaContext.get(), skiaTex, kBottomLeft_GrSurfaceOrigin, 0, SkColorType::kN32_SkColorType, nullptr, &props, // TODO: Delete texture! nullptr); } GrGLTextureInfo textureInfo { // OpenGL will automatically convert YUV -> RGB because it's an EXTERNAL texture .fTarget = GL_TEXTURE_EXTERNAL_OES, .fID = cameraTexture.id, .fFormat = GR_GL_RGBA8, .fProtected = skgpu::Protected::kNo, }; GrBackendTexture skiaTexture(cameraTexture.width, cameraTexture.height, GrMipMapped::kNo, textureInfo); sk_sp<SkImage> frame = SkImages::BorrowTextureFrom(_skiaContext.get(), skiaTexture, kBottomLeft_GrSurfaceOrigin, kN32_SkColorType, kOpaque_SkAlphaType, nullptr, nullptr); SkCanvas* canvas = _offscreenSurface->getCanvas(); canvas->clear(SkColors::kCyan); canvas->drawImage(frame, 0, 0); SkRect rect = SkRect::MakeXYWH(150, 250, random() * 200, random() * 400); SkPaint paint; paint.setColor(SkColors::kGreen); canvas->drawRect(rect, paint); _offscreenSurface->flushAndSubmit(); - OpenGLRenderer::renderTextureToSurface(..): Called with the newly rendered to offscreen texture (
GL_TEXTURE_2D) that contains the frame and the red rectangle. This will render the Frame to the output EGLSurface.if (_surface == EGL_NO_SURFACE) { _context->makeCurrent(); _surface = eglCreateWindowSurface(_context->display, _context->config, _outputSurface, nullptr); } // 1. Activate the OpenGL context for this surface _context->makeCurrent(_surface); // 2. Set the viewport for rendering glViewport(0, 0, _width, _height); glDisable(GL_BLEND); glClearColor(1.0f, 0.0f, 0.0f, 1.0f); // <-- red for debug glClear(GL_COLOR_BUFFER_BIT); // 3. Bind the input texture glBindTexture(GL_TEXTURE_2D, newTexture.id); // 4. Draw it using the pass-through shader which also applies transforms _passThroughShader.draw(newTexture, transformMatrix); // 5. Swap buffers to pass it to the window surface _context->flush(); - PassThroughShader::draw(..): Actually doing the pass-through rendering of the 2D texture that contains my Skia drawing. This is the shader that it uses. (
sampler2Dinstead ofsamplerExternalOES, right?)// 1. Set up Shader Program if (_programId == NO_SHADER || _shaderTarget != texture.target) { _programId = createProgram(texture.target); glUseProgram(_programId); _vertexParameters = { .aPosition = glGetAttribLocation(_programId, "aPosition"), .aTexCoord = glGetAttribLocation(_programId, "aTexCoord"), .uTransformMatrix = glGetUniformLocation(_programId, "uTransformMatrix"), }; _fragmentParameters = { .uTexture = glGetUniformLocation(_programId, "uTexture"), }; _shaderTarget = texture.target; } glUseProgram(_programId); // 2. Set up Vertices Buffer if (_vertexBuffer == NO_BUFFER) { glGenBuffers(1, &_vertexBuffer); } // TODO: I shouldn't be doing this each frame, but if I don't I just get a blackscreen. // Maybe Skia is overwriting those values? glBindBuffer(GL_ARRAY_BUFFER, _vertexBuffer); glBufferData(GL_ARRAY_BUFFER, sizeof(VERTICES), VERTICES, GL_STATIC_DRAW); // 3. Pass all uniforms/attributes for vertex shader glEnableVertexAttribArray(_vertexParameters.aPosition); glVertexAttribPointer(_vertexParameters.aPosition, 2, GL_FLOAT, GL_FALSE, sizeof(Vertex), reinterpret_cast<void*>(offsetof(Vertex, position))); glEnableVertexAttribArray(_vertexParameters.aTexCoord); glVertexAttribPointer(_vertexParameters.aTexCoord, 2, GL_FLOAT, GL_FALSE, sizeof(Vertex), reinterpret_cast<void*>(offsetof(Vertex, texCoord))); // TODO: Does this transformation matrix need to be applied before, in Skia already? glUniformMatrix4fv(_vertexParameters.uTransformMatrix, 1, GL_FALSE, transformMatrix); // 4. Pass texture to fragment shader glActiveTexture(GL_TEXTURE0); // TODO: Do I need to use GL_TEXTURE0 here? Does Skia overwrite this value? glBindTexture(texture.target, texture.id); glUniform1i(_fragmentParameters.uTexture, 0); // 5. Draw! glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
And the Shaders are here:
static constexpr Vertex VERTICES[] = {
{{-1.0f, -1.0f}, {0.0f, 0.0f}}, // bottom-left
{{1.0f, -1.0f}, {1.0f, 0.0f}}, // bottom-right
{{-1.0f, 1.0f}, {0.0f, 1.0f}}, // top-left
{{1.0f, 1.0f}, {1.0f, 1.0f}} // top-right
};
static constexpr char VERTEX_SHADER[] = R"(
attribute vec4 aPosition;
attribute vec2 aTexCoord;
uniform mat4 uTransformMatrix;
varying vec2 vTexCoord;
void main() {
gl_Position = aPosition;
vTexCoord = (uTransformMatrix * vec4(aTexCoord, 0.0, 1.0)).xy;
}
)";
// TODO: This is NOT samplerExternalOES because we draw the TEXTURE_2D that Skia already rendered.
// Is that correct?
static constexpr char FRAGMENT_SHADER[] = R"(
precision mediump float;
varying vec2 vTexCoord;
uniform sampler2D uTexture;
void main() {
gl_FragColor = texture2D(uTexture, vTexCoord);
}
)";
OpenGL setup is here, I basically have a 1x1 pbuffer for creating my textures, and then I switch between the output EGLSurfaces to render. When I render to the offscreen framebuffer (TEXTURE_2D), the 1x1 pbuffer is active.
I'm not sure if Skia is doing some texture/buffer binding that I should undo, if there is a memory issue, if the pass-through shader is wrong, or if the transformation matrix also needs to be applied to the Skia Shader.
If you want to take a look at this;
- Clone this PR/branch: https://github.com/mrousavy/react-native-vision-camera/pull/1731
- Run
yarn && cd example && yarn - Open
example/androidin Android Studio
Any help appreciated!
