1

There is some good documentation on this site called big flake about how to use media muxer and mediacodec to encode then decode video as mp4, or extract video then encode it again and more stuff.

But it doesnt seem that there is a way to encode audio with video at the same time, no documentation or code about this. It doesn't seem impossible.

Question

Do you know any stable way of doing it that will work on all devices greater than android 18?

Why no one implemented it, is it hard to implement?

1 Answers1

2

You have to create 2 Mediacodec instances, one for video and one for audio and then use MediaMuxer to mux the video with audio after encoding, you can take a look at ExtractDecodeEditEncodeMuxTest.java and at this project to capture camera/mic and save to mp4 file using Mediamuxer and Mediacodec

E.Abdel
  • 1,992
  • 1
  • 13
  • 24
  • this link shows you how to extract audio and video and then mux them, but the question is from where do i get the audio when I am encoding the video, plus is this a stable way? –  May 18 '18 at 07:26
  • Probabelly you will get your audio as RAW data from a source (Phone mic for example), and yes it is the most stable way to do it, see my edit (https://github.com/saki4510t/AudioVideoRecordingSample/blob/master/app/src/main/java/com/serenegiant/encoder/MediaAudioEncoder.java) – E.Abdel May 18 '18 at 07:33
  • Lets say that I encoded a video in a way such that I - frames are 1 sec away or 2 sec away from each other, does this affect the end video when audio is also muxed? –  May 18 '18 at 07:38
  • or audio and video formats are independent of each other? –  May 18 '18 at 07:39
  • audio and video format are independant because you use 2 mediacodec instances, but you have to sync the audio with the video returned the encoders (how many audio samples must play for each video frame) – E.Abdel May 18 '18 at 07:47
  • I wanted to ask you what is meant by (track) in terms of media muxer it confuses me, and does it mean that each audio and video are added to same track? –  May 18 '18 at 08:08
  • MP4 (and others format) is a container wich can contain different type of data (metadata, audio, video) called tracks, so a track can be a video data, audio data or metadata (ex : gyro infos) – E.Abdel May 18 '18 at 08:13
  • so when muxing video and audio I add them as 2 different tracks? –  May 18 '18 at 08:16
  • if yes then how do they exactly sync? –  May 18 '18 at 08:17
  • please take a look here https://stackoverflow.com/questions/20972049/how-to-provide-both-audio-data-and-video-data-to-mediamux – E.Abdel May 18 '18 at 08:26
  • I tried to record video and audio seperately at same time and then mixed them together is it safe or bad idea. –  May 18 '18 at 08:26
  • Yes, it is the safe idea, the question is how to sync them, please take a look at the link above – E.Abdel May 18 '18 at 08:30
  • No I meant record the video alone to a file and then the audio alone to a file, and then get both files and merge them together using media extractor .....is it bad or safe? –  May 18 '18 at 08:31
  • It is possible to do that with mediacodec, you can use extractor if you muxed audio and video files before (each one in a .mp4) and then mux them together or just mediamuxer if you saved the files without muxing before (e.g : video .h264/.264, audio .aac/3gp) – E.Abdel May 18 '18 at 08:37
  • ahh okay ...do you by any chance know anything about mediaprojection? –  May 18 '18 at 08:39
  • If you want to capture the device screen, you can follow this example https://github.com/googlesamples/android-ScreenCapture (without using mediacodec) or this one https://github.com/saki4510t/ScreenRecordingSample (using mediacodec) – E.Abdel May 18 '18 at 08:44
  • yeah exactly, I want to ask you about this issue, what is the recommended way using media recorder or mediacodec to use mediaprojection? what is your advice? –  May 18 '18 at 08:46
  • Depending on what you want to do, MediaRecorder is (easier) a high level Interface. But you cannot process frames because they aren't accessible for example, with MediaCodec you can do what you want with frames because you can manipulate the codec buffers, take a look here https://stackoverflow.com/questions/37248199/mediacodec-vs-mediaplayer-and-mediarecorder?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa and here http://www.hnwatcher.com/r/1170899/Video-recording-and-processing-in-Android – E.Abdel May 18 '18 at 08:56
  • Excuse me if I am asking too much, I am trying to learn from you as you are an expert in media stuff. You mean by processing frames that we can edit them or delete them or crop them when we use mediacodec but not when we use mediarecorder.....plus has anyone added audio while using media projection in mediacodec? –  May 18 '18 at 09:03
  • OH no I am not an expert, just in this moment I deal a lot with mediacodec API. So yes, with mediarecorder you cannot modify frames directly, with mediacodec you can. And yes, as I said before, with mediamuxer you can added audio track to a video track recorder by mediacodec through mediaprojection virtual display surface – E.Abdel May 18 '18 at 09:17