5

I am encoding NV21 frames coming from camera preview. For some reason schema which is working fine on other devices works incorrectly on Sony Xperia Z1 with Android 4.3. It sends back encoded frames with incorrect (low) quality.

MediaCodec's format is COLOR_FormatYUV420SemiPlanar which is NV12 (I convert NV21 to NV12 by swapping U and V components). Output buffers sent back to me by MediaCodec are with very low size which don't correspond to resolution (1280x720) and bitrate (1000000) I am using. First few frames are coming with a good quality, but then it's dropping significantly:

int encoderStatus = mMediaCodec.dequeueOutputBuffer(mBufferInfo, TIMEOUT_USEC);
// a few encoderStatus checks skipped
ByteBuffer encodedData = outputBuffers[encoderStatus];
Log.i(Constants.TAG, "Buffer size " + mBufferInfo.size);

Which gives me the following log on Xperia Z1:

Buffer size 26
Buffer size 52172
Buffer size 23650
Buffer size 14394
Buffer size 3591
Buffer size 1849
Buffer size 3908

...

Buffer size 1043
Buffer size 248
Buffer size 836
Buffer size 518
Buffer size 1112

Example log from Sony Xperia ZR on which this works properly:

Buffer size 21
Buffer size 51048
Buffer size 21063
Buffer size 24228
Buffer size 28040
...
Buffer size 44959
Buffer size 44972
Buffer size 44957
Buffer size 45004
Buffer size 44999
Buffer size 44957

Any advice would be appreciated.

fadden
  • 51,356
  • 5
  • 116
  • 166
Andrey Chernih
  • 3,513
  • 2
  • 27
  • 27
  • 1
    The Xperia Z1 encoder appears to have some special behaviors, e.g. http://stackoverflow.com/questions/20475332/ . For 1280x720 you probably want more than 1Mbps though -- at 30fps that's 4KB per frame, and the Xperia ZR is outputting 10x that. So it's possible the ZR is broken and the Z1 is doing what you told it to. Does increasing the bit rate improve matters? – fadden Dec 18 '13 at 15:41
  • Thanks, fadden, your comment and mstorsjo's answer have pointed me into the right direction. – Andrey Chernih Dec 19 '13 at 06:26

1 Answers1

5

Are you sure you pass timestamps in the right unit (microseconds) - and that it has set a sensible framerate?

  • Some encoders might ignore the timestamps and only set a fixed bitrate budget per frame, based on the framerate.

  • Others might try to calculate how many bits they're allowed to use per frame based on the timestamp.

If the timestamps are given e.g. in milliseconds instead, this could lead to the encoder reducing the size of encoded frames down towards zero.

New Alexandria
  • 6,951
  • 4
  • 57
  • 77
mstorsjo
  • 12,983
  • 2
  • 39
  • 62
  • That was the issue. Thanks a lot. I was passing System.currentTimeMillis() as 4th parameter to mMediaCodec.queueInputBuffer() while what I should have pass is System.nanoTime() / 1000. – Andrey Chernih Dec 19 '13 at 06:25
  • 1
    Related: `System.nanoTime()` is the time when the frame was processed by you, not the time the frame was generated, and the time base may "wobble" a bit depending on the vagaries of the task scheduler. You can use `SurfaceTexture#getTimestamp()` to get the frame generation time. – fadden Dec 19 '13 at 15:52
  • Thanks for the information, @fadden Unfortunately, I am not using `SurfaceTexture`. I am getting my frames in preview callback set by `Camera.setPreviewCallbackWithBuffer()`. – Andrey Chernih Dec 20 '13 at 09:20
  • 2
    Ah, I should've known that from the U/V swap remark. If you're okay limiting your app to API 18+, and you don't need to edit the frames in YUV space, you should consider switching to Surface -- it's much more efficient. Grafika happily encodes 60fps preview on my Nexus 7 (2012). (https://github.com/google/grafika) – fadden Dec 20 '13 at 15:45
  • 60fps sounds crazy. Yet I am still not sure if I should use `SurfaceTexture`. I am doing some real-time processing as frames are passed to my callback and this processing happens on CPU side. Hence I need camera frames available in RAM, but `glReadPixels` is pretty slow according to my tests... – Andrey Chernih Dec 20 '13 at 18:39
  • To clarify: this processing is done with `OpenCV` library and hardly can be reproduced on GPU. – Andrey Chernih Dec 20 '13 at 18:40
  • 1
    60fps does sound crazy. I re-checked and it's actually coming through at 30fps (at 1280x720). Grafika is using the camera defaults; haven't experimented to see what the actual maximum is. FWIW, Grafika includes a 1280x720 glReadPixels speed test; it takes 6ms per frame on some devices, 14ms on other devices, and 170ms on yet another. That last must be hitting some sort of slow path. So YMMV. – fadden Dec 20 '13 at 19:47