4

When I encode a video via Surface -> MediaCodec -> MediaMuxer, I get a very strange result when testing on the Samsung Galaxy S7. For other devices tested (emulator with Marshmallow and HTC Desire), the video comes out correctly, but on this device the video is garbled.

enter image description here

Using MediaCodec to save series of images as Video had a similar output of video, but I don't see how the solution could apply here because I am using a Surface as input and set the color format to COLOR_FormatSurface.

I also tried messing with the video resolution (settled on 1280 x 720) per MediaCodec Encoded video has green bar at bottom and chrominance screwed up, but that didn't solve the problem either. (c.f. Nexus 7 2013 mediacodec video encoder garbled output)

Does anyone have suggestions for what I might try to get the video formatted correctly?

Here is part of the log from the encoding:

D/ViewRootImpl: #1 mView = android.widget.LinearLayout{1dc79f2 V.E...... ......I. 0,0-0,0 #102039c android:id/toast_layout_root}
I/ACodec:  [] Now uninitialized
I/OMXClient: Using client-side OMX mux.
I/ACodec: [OMX.qcom.video.encoder.avc] Now Loaded
W/ACodec: [OMX.qcom.video.encoder.avc] storeMetaDataInBuffers (output) failed w/ err -1010
W/ACodec: do not know color format 0x7fa30c06 = 2141391878
W/ACodec: do not know color format 0x7fa30c04 = 2141391876
W/ACodec: do not know color format 0x7fa30c08 = 2141391880
W/ACodec: do not know color format 0x7fa30c07 = 2141391879
W/ACodec: do not know color format 0x7f000789 = 2130708361
D/ViewRootImpl: MSG_RESIZED_REPORT: ci=Rect(0, 0 - 0, 0) vi=Rect(0, 0 - 0, 0) or=1
I/ACodec: setupVideoEncoder succeeded
W/ACodec: do not know color format 0x7f000789 = 2130708361
I/ACodec: [OMX.qcom.video.encoder.avc] Now Loaded->Idle
I/ACodec: [OMX.qcom.video.encoder.avc] Now Idle->Executing
I/ACodec: [OMX.qcom.video.encoder.avc] Now Executing
I/MPEG4Writer: setStartTimestampUs: 0
I/MPEG4Writer: Earliest track starting time: 0

The 5th unrecognized color seems to be COLOR_FormatSurface... Is that a problem?

Other details:

  • MIME: video/avc
  • Resolution: 1280 x 720
  • Frame rate: 30
  • IFrame interval: 2
  • Bitrate: 8847360
Community
  • 1
  • 1
Grant
  • 442
  • 5
  • 19
  • In which case is the video garbled? When decoded and played back with MediaCodec on the same device, or when decoded and played back somewhere else? – mstorsjo Jan 25 '17 at 06:51
  • I only tried playing the video on the default video player on the device itself. – Grant Jan 25 '17 at 17:41
  • Ok, then it's probably still an encoder issue. The pattern looks a little like what a tiled format might look like, and the qualcomm decoder can output data in that pattern - a player that doesn't support it could end up with such an issue, but I guess the default player on the device should be fine. Then I don't have any other suggestions other than checking how other reference examples using surface input behave on the same device – mstorsjo Jan 25 '17 at 18:54

2 Answers2

4

Per Android Docs for MediaCodec.createInputSurface():

The Surface must be rendered with a hardware-accelerated API, such as OpenGL ES. lockCanvas(android.graphics.Rect) may fail or produce unexpected results.

I must have missed (or ignored) that in writing the code. Since I was using lockCanvas() to get a canvas upon which to draw my video frames, the code broke. I have put a quick fix on the problem by using lockHardwareCanvas() if API level >= 23 (since it is unavailable prior to that and since the code ran fine on API level 19).

Long term however (for me and anyone else who might stumble across this), I may have to get into more OpenGL stuff for a more permanent and stable solution. It's not worth going that route though unless I find an example of a device which will not work with my quick fix.

Grant
  • 442
  • 5
  • 19
  • So you can record videos using Canvas (without this scary GL stuff?) I did something similar with MediaRecorder (but it's very buggy): https://stackoverflow.com/questions/51332386/mediarecorder-and-videosource-surface-stop-failed-1007-a-serious-android-bug can you give me an example with MediaCodec? – user25 Jul 14 '18 at 08:29
  • I found that it's not good using `API level >= 23` cause, yes, `lockHardwareCanvas` works ok for my 6.0.1 (API 23) Device but `lockCanvas` gives similar result as on your screenshot for my 4.4.4 (API 19) Device (Samsung) so you can't be sure that it's going to work ok for devices with API < 23 - `lockCanvas`. So mb would better to support devices only from 23 API but KitKat and Lollipop are still very popular devices https://developer.android.com/about/dashboards/ – user25 Jul 14 '18 at 23:08
  • So I guess **OpenGL stuff** will catch us anyway – user25 Jul 14 '18 at 23:10
  • @user25 ```java configureFormat = ...; configureFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface); codec = MediaCodec.createEncoderByType(...); codec.configure(configureFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE); surface = codec.createInputSurface(); while(!done) { if(Build.VERSION.SDK_INT >= 23) { canv = surface.lockHardwareCanvas(); } else { canv = surface.lockCanvas(null); } // Draw to canv here mSurface.unlockCanvasAndPost(canv); } codec.signalEndOfInputStream(); ``` – Grant Aug 04 '18 at 18:54
  • it's not going to work on all devices (believe me https://issuetracker.google.com/issues/111433520#comment13) – user25 Aug 05 '18 at 08:56
  • I've found that lockHardwareCanvas isn't always reliable, In some devices the counterpart unlockCanvasAndPost doesn't always send each frame into the MediaCodec. I ended up going through the OpenGL route, and wasn't that painful. Grafika is a good starting point. – PerracoLabs Sep 12 '19 at 20:20
-2
If you are still looking for an example for rendering bitmaps to a InputSurface. 

I was able to get this to work.
Look at my answers here.
https://stackoverflow.com/a/49331192/7602598
https://stackoverflow.com/a/49331352/7602598
https://stackoverflow.com/a/49331295/7602598
David Knight
  • 43
  • 1
  • 5