How to have a live wallpaper that can scroll while showing a video?

Question

Background

I've been familiar with using Canvas to draw a Bitmap in a live wallpaper, and eventually I also tried out playing GIF animation in a live wallpaper (here) and a simple video playing (here)

The problem

I wanted to make it more efficient by using what other people have made, which uses OpenGL together with ExoPlayer, and still have it playing the video properly, including being able to scroll it horizontally.

Using a Canvas won't work here as it is used for simple drawing. Using the classes I saw for videos wouldn't work because they don't offer a good control of the movement&scaling, as well as showing other content (background color/image).

Sadly there isn't much that I've found that I can read, but I've found a very few solutions that could help me:

1. Muzei - used for images only (with some effects). Works well and can scroll horizontally well. However, it's a huge app and very hard to understand what's going on there. Seems it's using OpenGL3 alone.

2. alynx-live-wallpaper (my updated, Kotlin fork here) - can show a video and supports both OpenGL2 and OpenGL3. I think it plays the video in the middle, but something is wrong with its horizontal scrolling (here). The issue there is that you can't reach the edges of the video (on the left/right sides), even though you should be able to do so.

3. LibGdx - I barely found any resources of it being used on a live wallpaper, but I'm sure it can play both videos and images. I asked about it on reddit after trying it out, here.

What I've tried

I've decided to improve "alynx-live-wallpaper" solution and I've forked it (here), converting all to Kotlin and trying then to understand what's going on, to fix the issue that it has, and modernize it on the way.

To make it even easier to work on it, I've created yet another repository (based on my fork), and this time trying to minimize the code as much as possible, playing a video that's built into the app, just to focus on the issue, and then see what I should do. This repository can be found here.

I will focus on OpenGL3, even though the same seems to exist on OpenGL2.

At first I thought it might be a thread-related issue (because I can see some functions are called on the UI thread, and some on a background thread), but even though I think there is technically a thread-related issue here, it's not the real reason for what I'm seeing. Then I thought that the mistake was in setOffset, as it used maxXOffset for the new value of y-offset, instead of maxYOffset (reported here), but it still wasn't the reason.

For Canvas with a Bitmap, what I have created in the past and it can work fine even for horizontal scrolling, is something like that:

val canvasWidth = canvas.width
val canvasHeight = canvas.height
val bitmapWidth = bitmap.width.toFloat()
val bitmapHeight = bitmap.height.toFloat()
val scale = max(canvasWidth / bitmapWidth, canvasHeight / bitmapHeight)
val x = currentxOffset!! * (canvasWidth - bitmapWidth * scale)
val y = (canvasHeight - bitmapHeight * scale) / 2
canvas.save()
canvas.translate(x, y)
canvas.drawBitmap(bitmap, 0f, 0f, null)
canvas.restore()

On the project I've created for videos, it seems it works as such:

GLWallpaperService.kt It passes the callbacks to the OpenGL renderer class:

@UiThread
override fun onOffsetsChanged(xOffset: Float, yOffset: Float, xOffsetStep: Float, yOffsetStep: Float, xPixelOffset: Int, yPixelOffset: Int) {
    super.onOffsetsChanged(xOffset, yOffset, xOffsetStep, yOffsetStep, xPixelOffset, yPixelOffset)
    if (allowSlide && !isPreview) {
        renderer!!.setOffset(0.5f - xOffset, 0.5f - yOffset)
    }
}

@UiThread
override fun onSurfaceChanged(surfaceHolder: SurfaceHolder, format: Int, width: Int, height: Int) {
    super.onSurfaceChanged(surfaceHolder, format, width, height)
    renderer!!.setScreenSize(width, height)
}

In addition, it finds the video properties:

videoRotation = rotation!!.toInt()
videoWidth = width!!.toInt()
videoHeight = height!!.toInt()

GLES30WallpaperRenderer.kt This is responsible for the actual calculations and drawing. I will try to minimize code here as much as possible:

@UiThread
override fun setScreenSize(width: Int, height: Int) {
    if (screenWidth != width || screenHeight != height) {
        screenWidth = width
        screenHeight = height
        maxXOffset =
                (1.0f - screenWidth.toFloat() / screenHeight / (videoWidth.toFloat() / videoHeight)) / 2
        maxYOffset =
                (1.0f - screenHeight.toFloat() / screenWidth / (videoHeight.toFloat() / videoWidth)) / 2
        updateMatrix()
    }
}

override fun setOffset(xOffset: Float, yOffset: Float) {
    val newXOffset = xOffset.coerceAtLeast(-maxXOffset).coerceAtMost(maxXOffset)
    val newYOffset = yOffset.coerceAtLeast(-maxYOffset).coerceAtMost(maxYOffset)
    if (this.xOffset != newXOffset || this.yOffset != newYOffset) {
        this.xOffset = newXOffset
        this.yOffset = newYOffset
        updateMatrix()
    }
}

@UiThread
private fun updateMatrix() {
    for (i in 0..15) {
        mvp[i] = 0.0f
    }
    mvp[15] = 1.0f
    mvp[10] = mvp[15]
    mvp[5] = mvp[10]
    mvp[0] = mvp[5]
    val videoRatio = videoWidth.toFloat() / videoHeight
    val screenRatio = screenWidth.toFloat() / screenHeight
    if (videoRatio >= screenRatio) {
        Matrix.scaleM(mvp, 0, videoWidth.toFloat() / videoHeight / (screenWidth.toFloat() / screenHeight), 1f, 1f)
        if (videoRotation % 360 != 0) {
            Matrix.rotateM(mvp, 0, -videoRotation.toFloat(), 0f, 0f, 1f)
        }
        Matrix.translateM(mvp, 0, xOffset, 0f, 0f)
    } else {
        Matrix.scaleM(mvp, 0, 1f, videoHeight.toFloat() / videoWidth / (screenHeight.toFloat() / screenWidth), 1f)
        if (videoRotation % 360 != 0) {
            Matrix.rotateM(mvp, 0, -videoRotation.toFloat(), 0f, 0f, 1f)
        }
        Matrix.translateM(mvp, 0, 0f, yOffset, 0f)
    }
}

The questions

1. What is wrong here in the scrolling of the video?

2. This is move of a bonus: how can I also put a background color and/or image (which moves with the scrolling)? I remember I did something like that on OpenGL 1.x, but things probably changed since then...

Answer 1

Try make GLES30WallpaperRenderer.kt

class GLES30WallpaperRenderer : GLSurfaceView.Renderer {

    // ... Ur code ...

    private var bGroundTextureId: Int = -1
    private var bGroundShaderProgram: Int = -1

    // New variables to control the movement
    private var offsetX: Float = 0.0f
    private var offsetY: Float = 0.0f

    private fun setupBGround() {
        // Load b-ground image texture if you have one
        bGroundTextureId = TextureUtils.loadTexture(context, R.drawable.background_image)

        // Create and compile b-ground shader program
        val vertexShaderCode = getVertexShaderCodeWithOffset() // Vertex shader code with offset
        val fragmentShaderCode = ... // Fragment shader code for b-ground quad
        bGroundShaderProgram = ShaderUtils.createProgram(vertexShaderCode, fragmentShaderCode)
    }

    // Add a uniform variable to the vertex shader to control the offset
    private fun getVertexShaderCodeWithOffset(): String {
        return """
            // ... Your other vertex shader code ...
            
            // Uniform variable for offset
            uniform vec2 u_Offset;

            void main() {
                // ... Your other vertex shader code ...

                // Apply the offset to the vertex position
                gl_Position = vec4(a_Position.xy + u_Offset, 0.0, 1.0);
            }
        """.trimIndent()
    }

    // Function to update the offset values for movement
    fun updateOffset(offsetX: Float, offsetY: Float) {
        this.offsetX = offsetX
        this.offsetY = offsetY
    }

    private fun drawBGround() {
        // Bind the b-ground shader program
        GLES30.glUseProgram(bGroundShaderProgram)

        // Set the offset uniform in the shader
        val offsetLocation = GLES30.glGetUniformLocation(bGroundShaderProgram, "u_Offset")
        GLES30.glUniform2f(offsetLocation, offsetX, offsetY)

        // Bind the texture and draw the b-ground quad
        GLES30.glActiveTexture(GLES30.GL_TEXTURE0)
        GLES30.glBindTexture(GLES30.GL_TEXTURE_2D, bGroundTextureId)

        // Set texture coordinates, vertex coordinates, and other attributes for the b-ground quad
        // Draw the b-ground quad using GLES30.glDrawArrays() or GLES30.glDrawElements()
    }

    // ... Ur other code ...

}