7

I'm trying to get video raw data by Preview, get audio raw data by AudioRecord. Then I will send them to MediaCodec(I will set two Codec instance). After that I will send the video data and audio data to MediaMux to get a mp4 file. I have two questions:

1) I've used MediaMux to process the video data before. For video data, the MediaMux process it by frame, but the video record is continuous. How could the MediaMux handle the video and audio in synchronization.

2) I found only of variable for inputbuffer in the function writeSampleData. A nice guy on StackOverflow also provided a demo of MediaMux. But this demo only has one variable for source data. How could this one variable stand both video data and audio data?

Thank you!

Michael
  • 57,169
  • 9
  • 80
  • 125
Brendon Tsai
  • 1,267
  • 1
  • 17
  • 31

2 Answers2

10

I have had some success passing synced audio and video to MediaMuxer. I calculated the number of audio samples that should play for each frame of video: based on the audio sample rate and the video frame rate. I then had a loop that wrote one video frame and one block of audio in each iteration. Use the presentation time to ensure they will be synced on playback. Use the track index from addTrack in calls to writeSampleData to allow writing video and audio to two separate tracks.

fadden
  • 51,356
  • 5
  • 116
  • 166
lighty
  • 147
  • 1
  • 4
  • Thank you! Could you tell me how do you handle the record of audio? Since audio record is constant, do you keep it recording or do you stop it between frames? I review the MediaMux API, but it seems like we use addTrack to add MediaFormat. What if we want to provide the audio source and video source to MediaMux? Besides, there is only one variable in writeSampleData for inpurbuffer. – Brendon Tsai Jan 10 '14 at 03:15
  • I am muxing video and audio that I recorded earlier, so my scenario is different to yours. I'm using `AudioRecord` in an `AsyncTask` and assume you do the same. Stopping audio recording between frames would produce stuttering audio, so is not ideal. You will need to manage video and audio recording threads carefully to get smooth recording - the writing/muxing task can take lower priority as they will not affect the user experience. Call `addTrack` once to add the video track and again to add the audio track. It returns index of the added track. Then call `writeSampleData` for each track. – lighty Jan 10 '14 at 16:34
  • Thanks! So I should use addTrack twice and get two indexs. Then I use writeSampleData twice, the first variable of the function is the indexI get, the second is the source of raw data. Is that right? – Brendon Tsai Jan 13 '14 at 09:09
  • Correct, and the third parameter to `writeSampleData` is `bufferInfo` which you can use to read or set the presentation time of the frame. – lighty Jan 13 '14 at 14:12
  • However you would normally set the presentation time in your calls to `queueInputBuffer`. – lighty Jan 13 '14 at 20:18
  • Thanks! I will check that – Brendon Tsai Jan 14 '14 at 05:50
  • @BrendonTsai how do you use the AddTrack it only gets int.How can I add Track in it?? – Robin Royal Nov 09 '14 at 08:20
  • I have same problem in this post Can you look this post https://stackoverflow.com/questions/53832469/h264-format-doesnt-audio-how-to-get-audio-in-h264 – Diego Dec 19 '18 at 07:13
  • 1
    Make sure you use @Synchronized (Kotlin) on a function that handles the actual writeSampleData call. Never call writeSampleData in two or more different locations. It is important that when the muxer is writing data with writeSampleData, you never call it again while it is writing. That may not seem obvious. It is easy to assume that writeSampleData can handle multiple tracks internally asynchronously, but it can't. – Johann May 16 '21 at 16:37
1

In aggregate to answer @robin-royal, in order to encode both video and audio y should call twice the Addtrack method of MediaMuxer, One track index (int) to each one. so then when you call WriteSampleData in the MediaMuxer, the first parameter specifies the track index, e. g. if audioTrackIndex=2 and videoTrackIndex=1, if you call WriteSampleData with the first parameter equals 1, you would be writing video. (Sorry I have no privileges to answer him in the comment) thanks

Gabriel Bursztyn
  • 656
  • 8
  • 24