Rotating Android Camera Preview by 90 degrees on GLSurfaceView using OpenGL 2.0 APIs

Question

Note I am new to both Android and OpenGL.

I using the code to display Android Camera Preview on the screen. I would like to rotate the Camera angle by 90 degrees using OpenGL 2.0 (to match my camera in portrait resolution since my App is in portrait mode) in the class ' MainRenderer' below. I have looked into Android camera rotate but it did not execute rotation.

Setting "android:rotation='90'" in layout xml rotates surface view but yields black surface - no camera preview.

Since I am using GLSurfaceView, I am now looking to actually rotate the camera preview in OpenGL 2.0 instead by performing Matrix rotation and aspect ratio adjustment. There were few posts like How to rotate a camera view in OpenGL 2.0 ES and http://www.programcreek.com/java-api-examples/index.php?class=android.opengl.Matrix&method=rotateM but I could not relate API with the code below due to my unfamiliarity with OpenGL (which I am hoping to get familiar soon).

I understand that maybe rotateM and translateM (guess using ModelViewProjection Matrix) may do what I intend to accomplish.

I am wondering vtmp and ttmp may need to be altered below?

Any help is appreciated.

The original code is from http://maninara.blogspot.com/2012/09/render-camera-preview-using-opengl-es.html

// in Manifest

< uses-feature android:glEsVersion="0x00020000" android:required="true"/>
< uses-feature android:name="android.hardware.camera"/>
< uses-permission android:name="android.permission.CAMERA"/>
< uses-permission android:name="android.permission.WAKE_LOCK"/>
... android:screenOrientation="landscape" ... // activity property

// Activity

public class MainActivity extends Activity {
  private MainView mView;
  private WakeLock mWL;

  @Override
  public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    // full screen & full brightness
    requestWindowFeature ( Window.FEATURE_NO_TITLE);
    getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN, WindowManager.LayoutParams.FLAG_FULLSCREEN);
    getWindow().setFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON, WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
    mWL = ((PowerManager)getSystemService ( Context.POWER_SERVICE )).newWakeLock(PowerManager.FULL_WAKE_LOCK, "WakeLock");
    mWL.acquire();
    mView = new MainView(this);
    setContentView ( mView );
  }

  @Override
  protected void onPause() {
    if ( mWL.isHeld() )
      mWL.release();
    mView.onPause();
    super.onPause();
  }

  @Override
  protected void onResume() {
    super.onResume();
    mView.onResume();
    if(!mWL.isHeld()) mWL.acquire();
  }
}

// View

class MainView extends GLSurfaceView {
  MainRenderer mRenderer;

  MainView ( Context context ) {
    super ( context );
    mRenderer = new MainRenderer(this);
    setEGLContextClientVersion ( 2 );
    setRenderer ( mRenderer );
    setRenderMode ( GLSurfaceView.RENDERMODE_WHEN_DIRTY );
  }

  public void surfaceCreated ( SurfaceHolder holder ) {
    super.surfaceCreated ( holder );
  }

  public void surfaceDestroyed ( SurfaceHolder holder ) {
    mRenderer.close();
    super.surfaceDestroyed ( holder );
  }

  public void surfaceChanged ( SurfaceHolder holder, int format, int w, int h ) {
    super.surfaceChanged ( holder, format, w, h );
  }
}

// Renderer

public class MainRenderer implements GLSurfaceView.Renderer, SurfaceTexture.OnFrameAvailableListener {
  private final String vss =
      "attribute vec2 vPosition;\n" +
      "attribute vec2 vTexCoord;\n" +
      "varying vec2 texCoord;\n" +
      "void main() {\n" +
      "  texCoord = vTexCoord;\n" +
      "  gl_Position = vec4 ( vPosition.x, vPosition.y, 0.0, 1.0 );\n" +
      "}";

  private final String fss =
      "#extension GL_OES_EGL_image_external : require\n" +
      "precision mediump float;\n" +
      "uniform samplerExternalOES sTexture;\n" +
      "varying vec2 texCoord;\n" +
      "void main() {\n" +
      "  gl_FragColor = texture2D(sTexture,texCoord);\n" +
      "}";

  private int[] hTex;
  private FloatBuffer pVertex;
  private FloatBuffer pTexCoord;
  private int hProgram;

  private Camera mCamera;
  private SurfaceTexture mSTexture;

  private boolean mUpdateST = false;

  private MainView mView;

  MainRenderer ( MainView view ) {
    mView = view;
    float[] vtmp = { 1.0f, -1.0f, -1.0f, -1.0f, 1.0f, 1.0f, -1.0f, 1.0f };
    float[] ttmp = { 1.0f, 1.0f, 0.0f, 1.0f, 1.0f, 0.0f, 0.0f, 0.0f };
    pVertex = ByteBuffer.allocateDirect(8*4).order(ByteOrder.nativeOrder()).asFloatBuffer();
    pVertex.put ( vtmp );
    pVertex.position(0);
    pTexCoord = ByteBuffer.allocateDirect(8*4).order(ByteOrder.nativeOrder()).asFloatBuffer();
    pTexCoord.put ( ttmp );
    pTexCoord.position(0);
  }

  public void close()
  {
    mUpdateST = false;
    mSTexture.release();
    mCamera.stopPreview();
    mCamera.release();
    mCamera = null;
    deleteTex();
  }

  public void onSurfaceCreated ( GL10 unused, EGLConfig config ) {
    //String extensions = GLES20.glGetString(GLES20.GL_EXTENSIONS);
    //Log.i("mr", "Gl extensions: " + extensions);
    //Assert.assertTrue(extensions.contains("OES_EGL_image_external"));

    initTex();
    mSTexture = new SurfaceTexture ( hTex[0] );
    mSTexture.setOnFrameAvailableListener(this);

    mCamera = Camera.open();
    try {
      mCamera.setPreviewTexture(mSTexture);
    } catch ( IOException ioe ) {
    }

    GLES20.glClearColor ( 1.0f, 1.0f, 0.0f, 1.0f );

    hProgram = loadShader ( vss, fss );
  }

  public void onDrawFrame ( GL10 unused ) {
    GLES20.glClear( GLES20.GL_COLOR_BUFFER_BIT );

    synchronized(this) {
      if ( mUpdateST ) {
    mSTexture.updateTexImage();
    mUpdateST = false;
      }
    }

    GLES20.glUseProgram(hProgram);

    int ph = GLES20.glGetAttribLocation(hProgram, "vPosition");
    int tch = GLES20.glGetAttribLocation ( hProgram, "vTexCoord" );
    int th = GLES20.glGetUniformLocation ( hProgram, "sTexture" );

    GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
    GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, hTex[0]);
    GLES20.glUniform1i(th, 0);

    GLES20.glVertexAttribPointer(ph, 2, GLES20.GL_FLOAT, false, 4*2, pVertex);
    GLES20.glVertexAttribPointer(tch, 2, GLES20.GL_FLOAT, false, 4*2, pTexCoord );
    GLES20.glEnableVertexAttribArray(ph);
    GLES20.glEnableVertexAttribArray(tch);

    GLES20.glDrawArrays(GLES20.GL_TRIANGLE_STRIP, 0, 4);
    GLES20.glFlush();
  }

  public void onSurfaceChanged ( GL10 unused, int width, int height ) {
    GLES20.glViewport( 0, 0, width, height );
    Camera.Parameters param = mCamera.getParameters();
    List psize = param.getSupportedPreviewSizes();
    if ( psize.size() > 0 ) {
      int i;
      for ( i = 0; i < psize.size(); i++ ) {
    if ( psize.get(i).width < width || psize.get(i).height < height )
      break;
      }
      if ( i > 0 )
    i--;
      param.setPreviewSize(psize.get(i).width, psize.get(i).height);
      //Log.i("mr","ssize: "+psize.get(i).width+", "+psize.get(i).height);
    }
    param.set("orientation", "landscape");
    mCamera.setParameters ( param );
    mCamera.startPreview();
  }

  private void initTex() {
    hTex = new int[1];
    GLES20.glGenTextures ( 1, hTex, 0 );
    GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, hTex[0]);
    GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_CLAMP_TO_EDGE);
    GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_CLAMP_TO_EDGE);
    GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_NEAREST);
    GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_NEAREST);
  }

  private void deleteTex() {
    GLES20.glDeleteTextures ( 1, hTex, 0 );
  }

  public synchronized void onFrameAvailable ( SurfaceTexture st ) {
    mUpdateST = true;
    mView.requestRender();
  }

  private static int loadShader ( String vss, String fss ) {
    int vshader = GLES20.glCreateShader(GLES20.GL_VERTEX_SHADER);
    GLES20.glShaderSource(vshader, vss);
    GLES20.glCompileShader(vshader);
    int[] compiled = new int[1];
    GLES20.glGetShaderiv(vshader, GLES20.GL_COMPILE_STATUS, compiled, 0);
    if (compiled[0] == 0) {
      Log.e("Shader", "Could not compile vshader");
      Log.v("Shader", "Could not compile vshader:"+GLES20.glGetShaderInfoLog(vshader));
      GLES20.glDeleteShader(vshader);
      vshader = 0;
    }

    int fshader = GLES20.glCreateShader(GLES20.GL_FRAGMENT_SHADER);
    GLES20.glShaderSource(fshader, fss);
    GLES20.glCompileShader(fshader);
    GLES20.glGetShaderiv(fshader, GLES20.GL_COMPILE_STATUS, compiled, 0);
    if (compiled[0] == 0) {
      Log.e("Shader", "Could not compile fshader");
      Log.v("Shader", "Could not compile fshader:"+GLES20.glGetShaderInfoLog(fshader));
      GLES20.glDeleteShader(fshader);
      fshader = 0;
    }

    int program = GLES20.glCreateProgram();
    GLES20.glAttachShader(program, vshader);
    GLES20.glAttachShader(program, fshader);
    GLES20.glLinkProgram(program);

    return program;
  }
}

If you really want to go the OpenGL ES route, see "texture from camera" in Grafika (https://github.com/google/grafika). It demonstrates arbitrary position, rotation, scale, and "zoom" of the Camera preview, with output to a SurfaceView. Also: you seem to be requesting the wake lock permission; be sure to read http://stackoverflow.com/questions/2131948/force-screen-on/2134602#2134602 . — fadden, Apr 18 '16 at 23:59

Volodymyr Lykhonis · Accepted Answer · 2016-04-18T23:49:42.303

6

I think there are few ways depends what suits you more:

Change vertex array values based on angle (probably the easiest one)
Camera.setDisplayOrientation
You could change your vertex little bit (it is little bit different from what you have so far, I used vec4 instead of vec2, some other changes will need to be done to account that in your code):

static final String vss =
    "uniform mat4 uMVPMatrix;\n" +
    "uniform mat4 uSTMatrix;\n" +
    "attribute vec4 vPosition;\n" +
    "attribute vec4 vTexCoord;\n" +
    "varying vec2 texCoord;\n" +
    "void main() {\n" +
    "    gl_Position = uMVPMatrix * vPosition;\n" +
    "    texCoord = (uSTMatrix * vTexCoord).xy;\n" +
    "}\n";

and then

float[] mvpMatrix = new float[16];
int uMVPMatrixHandle;

uMVPMatrixHandle = GLES20.glGetUniformLocation(program, "uMVPMatrix");
checkLocation(uMVPMatrixHandle, "uMVPMatrix");

when drawing specify any angle you would like to:

Matrix.setIdentityM(mvpMatrix, 0);
Matrix.rotateM(mvpMatrix, 0, angle, 0f, 0f, 1f);
GLES20.glUniformMatrix4fv(uMVPMatrixHandle, 1, false, mvpMatrix, 0);
EGL.checkGlError("glUniformMatrix4fv");

edited Apr 18 '16 at 23:49

answered Apr 18 '16 at 23:43

Volodymyr Lykhonis

2,936
2
17
13

Thanks. I added checkGLError and checkLocation functions from here http://stackoverflow.com/questions/35227222/rendering-surfacetexture-to-unity-texture2d and other pieces from above. However, when I ran the code with your suggestions, it seems to crash. Just to be clear, which parts of the code you have listed should go in 'onDraw' and which in 'onSurfaceCreated', 'onSurfaceChanged'? – enthusiasticgeek Apr 19 '16 at 04:01
4

Camera.setDisplayOrientation didn't work for me. I used the first option, I took paper and pencil and drew a simple diagram on coordinate system. Then I placed all coordinates logically and based on it I changed `vtmp` as follows, `float[] vtmp = { -1.0f, -1.0f, -1.0f, 1.0f, 1.0f, -1.0f, 1.0f, 1.0f };` to rotate by 90 degrees. And it worked fine for my application. – enthusiasticgeek Apr 19 '16 at 04:42
Great, yes, last approach is good when needs more control and extra steps. I think first one is easiest for your case, but make sure account that angle 90 degrees is a dynamic/variable that can be different per device you run (see http://developer.android.com/reference/android/hardware/Camera.html#setDisplayOrientation(int) for more info about angle handling), rather than that, you should be fine. – Volodymyr Lykhonis Apr 19 '16 at 15:27
1

Hi, I have checked the ist answer and it works correctly but there is one pblm, how to overcome the mirror effect ? – ranjith May 30 '16 at 17:29
If you have mirror display, `float[] vtmp = { 1.0f, -1.0f, 1.0f, 1.0f, -1.0f, -1.0f, -1.0f, 1.0f};` is worked for me. – Talha Ç May 31 '22 at 11:49

Rotating Android Camera Preview by 90 degrees on GLSurfaceView using OpenGL 2.0 APIs

1 Answers1

Linked