How to input video (frames) into a GLSL shader

Question

I'm trying to do video processing using GLSL. I'm using OpenCV to open a video file up and take each frame as a single image an then I want to use each frame in a GLSL shader

What is the best/ideal/smart solution to using video with GLSL?

Reading From Video

VideoCapture cap("movie.MOV");
Mat image;
bool success = cap.read(image);
if(!success)
{
    printf("Could not grab a frame\n\7");
    exit(0);
}

Image to Texture

 GLuint tex;
 glGenTextures(1, tex);
 glActiveTexture(GL_TEXTURE0);
 glBindTexture(GL_TEXTURE_2D, tex);
 glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, image.cols, image.rows, 0,
                GL_BGR, GL_UNSIGNED_BYTE, image.data);

 glUniform1i(glGetUniformLocation(shaderProgram, "Texture"), 0);

What needs to be in my render while loop?

Do I need to recompile/reattach/relink my shader every time? Or Once my shader is created and compiled and I use glUseProgram(shaderProgram) can I keep sending it new textures?

The current loop I've been using to render a texture to the screen is as follows. How Could I adapt this to work with video? Where would I need to make my calls to update the texture being used in the shader?

while(!glfwWindowShouldClose(window))
{
    glClearColor(0.0f, 0.0f, 0.0f, 1.0f);
    glClear(GL_COLOR_BUFFER_BIT);

    glViewport(0,0,512,512);

    glBindFramebuffer(GL_READ_FRAMEBUFFER, frameBuffer);
    glReadBuffer(GL_COLOR_ATTACHMENT0);
    glDrawElements(GL_TRIANGLES, 6, GL_UNSIGNED_INT, 0);
    glBindFramebuffer(GL_DRAW_FRAMEBUFFER, 0);
    glBlitFramebuffer(0, 0, image.cols, image.rows, 0, 0, image.cols, image.rows, GL_COLOR_BUFFER_BIT, GL_LINEAR);


    glfwSwapBuffers(window);
    glfwPollEvents();
}

IplImages are a blast from the past. use cv::Mat and cv::VideoCapture — berak, Sep 16 '14 at 21:56
downvote revoked. please do the same to the capture. your post will stay *forever* on SO — berak, Sep 16 '14 at 22:02
@karlphillip I just started the webpage, its a work in progress, thanks for the input though :) — Maggick, Sep 17 '14 at 02:51

score 3 · Accepted Answer · answered Sep 17 '14 at 01:06

Let's clarify a few things that needs to happen before the loop:

Set the pixel storage mode with glPixelStorei();
Generate only one texture with glGenTextures(), because at every iteration of the loop its content will be replaced with new data;
Compile the shader with glCompileShader(), use glCreateShader() to create a shader object, invoke glCreateProgram()to create a program, call glAttachShader() to attach the shader object to the new program, and finally glLinkProgram() to make everything ready to go.

That said, every iteration of the loop must:

Clear the color and depth buffer;
Load the modelview matrix with the identity matrix;
Specify the location where the drawing is going to happen glTranslatef();
Retrieve a new frame from the video;
Enable the appropriate texture target, bind it and then transfer the frame to the GPU with glTexImage2D();
Invoke glUseProgram() to activate your GLSL shader;
Draw a 2D face using GL_QUADS or whatever;
Disable the program with glUseProgram(0);
Disable the texture target with glDisable(GL_TEXTURE_YOUR_TEXTURE_TARGET);

This is more or less what needs to be done.

By the way: here's my OpenCV/OpenGL/Qt application that retrieves frames from the camera and displays it in a window. No shaders, though.

Good luck!

Thank you for outlining the steps it is very easy to follow – Maggick Sep 17 '14 at 03:15 — Maggick, Sep 17 '14 at 03:15

score 2 · Answer 2 · answered Sep 16 '14 at 22:45

2

You don't need to use a framebuffer to send textures to the shader. Once you've got texture 0 selected as the active texture and 0 set as the value of the uniform sampler2D in your shader, every time you call glBindTexture(), it will set the sampler2D to whichever texture you've specified in the function parameter. So no, you don't need to relink or recompile your shader each time you want to change texture.

answered Sep 16 '14 at 22:45

Tristram

176
1
8

Oh the reason I had the frame buffer object in my example code. Is because I'm actually processing two streams of video together. and I need to get two frame out of my shader. So I was using `COLOR_ATTACHMENT0` and `COLOR_ATTACHMENT1` – Maggick Sep 16 '14 at 23:09
You could always bind two textures simultaneously, one using texture slot 0 and one using texture slot 1, and then have two `sampler2D`s in your shader. – Tristram Sep 17 '14 at 12:33

How to input video (frames) into a GLSL shader

Reading From Video

Image to Texture

What needs to be in my render while loop?

2 Answers2

Linked