13

I am trying to stream video from android camera through local unix socket and write file from stream to sdcard. Everything works fine, except file is not playable with any player. It's because Android not filling some gaps in the file because socket is not seekable. As I understand I need to make some modifications after video stream is over. I read several articles here, here and here, but none of them helped me. I am playing with hex editor to learn how to do it manually, so afterwards it will be trivial to do the same in the Android code.

Here is sample file that saved from stream: https://dl.dropbox.com/u/17510473/sample_not_playable.3gp

Can anyone fix to make it playable and tell how he done it?

EDIT: I erase header of the 3gp file and write new one as follows:

00 00 00 18 66 74 79 70 33 67 70 34 00 00 03 00 33 67 70 34 33 67 70 36 00 00 00 00

Then I find starting location of mdat and moov atoms with following command:

grep -aobE "ftyp|mdat|moov" sample_not_playable.3gp

And it gives me following output:

4:ftyp
28:mdat
1414676:moov

Then make 1414676 - 28 = 1,414,648 = 0x1595F8

Then I write 0x1595F8 as 25-28 bytes, just prior mdat atom. So my header now looks like this:

00 00 00 18 66 74 79 70 33 67 70 34 00 00 03 00 33 67 70 34 33 67 70 36 00 15 95 F8

And when I try to play it with mplayer I get some damaged video and audio output. Here's some part from mplayer output:

[amrwb @ 0x7f72ad652380]Frame too small (33 bytes). Truncated file?
[amrwb @ 0x7f72ad652380]Encountered a bad or corrupted frame
[amrwb @ 0x7f72ad652380]Encountered a bad or corrupted frame
[amrwb @ 0x7f72ad652380]Frame too small (33 bytes). Truncated file?
[amrwb @ 0x7f72ad652380]Encountered a bad or corrupted frame
[amrwb @ 0x7f72ad652380]Encountered a bad or corrupted frame
[amrwb @ 0x7f72ad652380]Encountered a bad or corrupted frame
A:  11.0 V:   1.4 A-V:  9.650 ct:  0.023   0/  0 10%  1%  1.6% 0 0                                                        
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f72adeafc40]stream 1, offset 0x15e62b: partial file
[h263 @ 0x7f72ad652380]Bad picture start code
[h263 @ 0x7f72ad652380]header damaged
Error while decoding frame!
[h263 @ 0x7f72ad652380]Bad picture start code
[h263 @ 0x7f72ad652380]header damaged
Error while decoding frame!
[h263 @ 0x7f72ad652380]Bad picture start code
[h263 @ 0x7f72ad652380]header damaged
Error while decoding frame!
A:  11.1 V:   1.5 A-V:  9.558 ct:  0.027   0/  0  9%  1%  1.4% 0 0                                                        
[h263 @ 0x7f72ad652380]Bad picture start code
[h263 @ 0x7f72ad652380]header damaged
Error while decoding frame!

What I am doing wrong?

Community
  • 1
  • 1
Alex Amiryan
  • 1,374
  • 1
  • 18
  • 30

2 Answers2

14

What you need to understand is that mp4 is not a live streamable format. hence there is no way you will be able to hack anyway around to make it live streamable. The header [moov atom] is written at the end. Android creates a in memory table of frame sizes and other parameters which it then writes at the begining of the file at the end of recording because of which it needs the seekability of the file handle. [which a socket is not]

If you are writing encrypted content to the disk and you want to do it so that nobody can play the file you don't have to encrypt the whole file. You just have to encrypt the header and the entire file is unplayable.

If you desperately need full file encoding because you are not convinced of my previous para then use something like ffmpeg to do the encoding. Modify it give you encrypted output itself and save it to hard disk again - removes the socket part again.

There is no way you can do live streaming of mp4 files in android. I have seen tons of people trying it in vain and if you understand video/formats there is not way you can do it. mp4 is not designed for live streaming. Unless you know your frame sizes in advance [which you don't] and the exact length of encoding [which you most probably don;t] you cannot pre-create the header.

ps. mp4 and 3gp are cousins so the same thing applies.

Many people confuse live streaming with http pd and pseudo streaming. Live streaming means I do not have the whole file, it is being created and also being streamed on the fly. http pd and pseudo streaming take place with files that are fully available.

EDIT:

If your aim is to encrypt the recorded file before storing in sdcard you need encoder support. This implies getting your own encoder in there. Take ffmpeg, cross compile it to android. Write a small JNI interface for your application. Get this up first without encryption. Once done, where ffmpeg fwrites the stream, add your encryption module. Same thing while decoding. Y3ng creation. This is the cleanest way to do it.

EDIT 2: See spydroid to get some of the features for you. There are similar ones out there.

EDIT 3: To improve the quality of the answer I am explaining an imperfect workaround also given by other answers:

One can still stream the AV by parsing the mp4 as it gets generated and sending the elementary streams seperately over a socket. The only issue you will face is that you will not get perfect AV sync as you do not know the exact AV sample timestamps. Only android knows that and it writes that in the mp4 header at the end. So no good for you. You have to be making an assumption on perfect sampling rate of video frames and your audio will have to be amr to assume 20ms packets. In other audio cases you will start seeing drift in long runs [especially where you start having scenes of high motion]. This is because each audio packet generated does not correspond to a fixed time duration [except for amr and other speech codecs]

av501
  • 6,645
  • 2
  • 23
  • 34
  • 1
    Thanks for nice explanation of my problem. The thing is I desperately need to encrypt whole file. I use sockets only for purpose I need to encrypt video on the fly (that requirement is desperate too). So, what you can suggest? I have following requirements: 1. Encrypt video on the fly as it being recorded on the android device(without writing any unencrypted data to disk), 2. Be able to play file on the device, thus decrypting on the fly and feeding decrypted stream to video player. 3. Be able to decrypt file and have fully playable video which can be played with any player on PC. – Alex Amiryan Aug 28 '12 at 22:35
  • 1
    Then like I suggested you need to put in your own encoder to do the encoding. Take ffmpeg and get that up on the device. Once you have that just change it a bit to encrypt the output before giving it out. Also use ffmpeg to decode the stream and therefore decrypt the input before decoding it. For point 3 you can write a very simple decryption module yourself which simply takes one file in and gives another file out. – av501 Aug 29 '12 at 03:22
  • 1
    Thanks. But I am not coding in NDK. Is there any implementation in Java of video encoding? Can you suggest to what format I can encode my video to? Can you provide some code snippets for this purpose? – Alex Amiryan Aug 29 '12 at 09:49
  • Ok. The problem is I already have working encryption module which is in Java not JNI. I need to use that. I read whole source code of spydroid. In fact I learned how to stream video through local sockets from there. Spydroid uses it's h263 packetizer to transform stream into RTP. Can you suggest to what format I need to encode my video to match 3 criterias listed above and can you provide some code snippets or documentation on video format that you will suggest. Thank you. – Alex Amiryan Aug 29 '12 at 10:08
  • I can assure you will hit a dead end without doing something in jni because underlying system is not designed to do what you want to do. The only way out is if on the server [in your java code] you parse the data out, understand what needs to go in the header and at the end, write the header youself. Which means you need to understand the mp4 format clearly and parse the data coming to you to recreate the header.The streamable formats are transport stream [ts] or mjpeg or webm. So if you have ability to encoding to one of those your existing scheme will work without much trouble. Not mp4/3gpp – av501 Aug 29 '12 at 14:48
  • You know I don't agree with you. When you say to Android to write video file it writes it in the same way and then upon completion makes some corrections in file. He is able to do that because he has seekable descriptor in his hands. But, when say to write to socket it does same recording process just it is unable to make final corrections, because socket is not seekable. So in fact I need to replicate final adjustments part of video recording and everything will be fine. – Alex Amiryan Aug 29 '12 at 20:11
  • Exactly what I am saying, you need to keep parsing the data coming to you because the header contains offsets to every frame and full file size. Android creates that in memory. In the end it writes it at the begining of the file. So those final adjustments are the mp4 header. So you need to parse the data that comes to you to ensure what goes in the mp4 header at the end write it. That is what you are calling "adjustments". – av501 Aug 29 '12 at 20:23
  • And my final question. Any ideas how to implement it? Maybe you can point to some documentation, because I am googleling 2nd week already and all I can find is some junk. Please help to find documentation and bounty is yours. – Alex Amiryan Aug 29 '12 at 20:36
  • Well, the bounty is secondary. Look at movenc.c in ffmpeg source code in libavformat folder. There is a function called mov_write_header function. mov/3gpp/mp4 are pretty much the same. The mp4 documentation is unwieldy. It is written as a big standard that is difficult to read. So forget that. Also look at mov.c which is the demuxer for mp4 format. That will explain how to parse the data. This is a bit hard you know. understanding a file format will take reasonable effort. – av501 Aug 29 '12 at 20:44
  • @av501: You are partly wrong. I saw sipdroid and couple of other projects which does live streaming. First of all, there are some protocols (like h263 which are parsable). Second, you can access video and audio directly and write streaming on your own (I did this). This approach has nothing to do with mp4, however it does the trick (streams video/audio) – Victor Ronin Sep 04 '12 at 14:48
  • @VictorRonin, did you stream the mp4 directly? My whole point is on mp4 not being streamable. sipdroid and other projects are encoding video and audio seperately via rtp by parsing the output from the encoding in android. They stream them as elementary streams. H.263 is a codec [not a protocol]. I have mentioned to Alex also that he can also parse and do the streaming. See the comments above. But mp4 format is not streamable by itself. So you cannot stream the output from android directly. You have to parse and send via rtp seperately. Let me know if I am wrong now! – av501 Sep 04 '12 at 14:53
  • I mean to say streaming video and audio seperately in my previous comment [3rd sentence]. Also how can you access audio and video seperately without parsing? I am guessing from your answer below you are also parsing it. So it is no longer mp4. It is elementary streams being sent. – av501 Sep 04 '12 at 15:10
  • One other problem with sending audio and video this way though is that you will not get perfect AV sync. This is because only the android system knows the exact timestamps for the audio and video samples. In your parsing you do not know that. So you will see some avsync issues in long running output when you do it that way - parse mp4 and stream elementary files. That is where I said the perfect way to do it is install ffmpeg and use that. – av501 Sep 04 '12 at 15:20
4

First of all, it's not very clear what you are trying to do.

Do you want just to save a video programmatically to a playable file? If so, you just need to use file descriptor instead of unix socket. This API of MediaRecord was designed to work with a file (not unix socket), precisely for the reason of random access. So, my first advise would be to use file descriptor and you will get correct file at the end of recording.

You can get examples here: How can I capture a video recording on Android?

In the case, if you are trying to write an application which streams video from device, you will need to parse the stream in a realime, divide it by frames and send frames separately. And the most complex part is parsing the stream (some video codecs, as example H263 can be parsed and other can't be, especially if the data is interleaved with the audio).

I believe one of these two projects implements such functionality: http://sipdroid.org/ http://code.google.com/p/imsdroid/

Community
  • 1
  • 1
Victor Ronin
  • 22,758
  • 18
  • 92
  • 184
  • The thing is I want to encrypt video on the fly and then write it to disk. That's why I need to use unix socket to be able to encrypt video in real time. The problem is this encrypted file is not playable after I decrypt it. So I just need to write streamed video to a file and make it playable. – Alex Amiryan Aug 23 '12 at 19:14
  • I hate when somebody asks me this question, but still I have to ask it "What is the core reason why do you need to encrypt the video in run time"? I mean do you actually need to encrypt it in runtime? Do you want to protect this file from access by other apps? I just think that it may be easier to work around encryption than fixing half written stream. – Victor Ronin Aug 23 '12 at 21:34
  • BTW. One more strange thing. It looks like mdat in your file is before moov. I was under impression that it's written at the very end of file, because it contains info on frames. – Victor Ronin Aug 23 '12 at 21:35
  • Encrypting video files actually is the main aim of the app :). So we need to do it. When you say to Android's MediaRecorder to write stream not into file, but in the socket it streams `mdat` atom and as soon as you press Stop, it gives you `moov` atom and final file looks like the one is attached in question. So any ideas on how to fix it make it playable? – Alex Amiryan Aug 24 '12 at 00:12
  • You are right. moov is written last (I forgot about this). I looked in the file and found following: – Victor Ronin Aug 24 '12 at 15:00
  • You are right. moov is written last (I forgot about this). I looked in the file with the hex editor and found following: "mdat" marker is located at offset 0x28 (which is 40 decimal) and "moov" marker is at offset 0x159620 (1414688). I have no idea, why grep showed wrong values. Based on this. Based on this your header should be 00 00 00 24 00 15 95 F8 (which is 141688-40) – Victor Ronin Aug 24 '12 at 15:10
  • Is it playable now? Can you give me the modified playable file please so I can make hex compare? – Alex Amiryan Aug 24 '12 at 16:04
  • I didn't try it yet. I will try on Monday and will send it to you, if it works. – Victor Ronin Aug 26 '12 at 16:37
  • Here is the link. The video quality is terrible (only at the end, it looks ok), but it works (doesn't show any error messages).https://www.dropbox.com/sh/vklu3qf8hvl5nta/DhTaf3Kvsr#f:sample_not_playable.3gp – Victor Ronin Aug 26 '12 at 18:47
  • I am getting same result with above instructions in my original question. I want to get fully normal playable video with sound. I think we need to change something in moov atom too. – Alex Amiryan Aug 26 '12 at 21:05
  • Did you try the file which I uploaded on dropbox? I tried it on Windows and Mac and player showed it without any errors? – Victor Ronin Aug 27 '12 at 02:26
  • BTW. What kind of player are you using? – Victor Ronin Aug 27 '12 at 02:27
  • No, it's not playing even a one frame correctly. I am using mplayer, totem, vlc on linux. – Alex Amiryan Aug 27 '12 at 14:15
  • That's strange. It plays fine on Windows in VLC (v.1.1) and in MacOS Chrome browser. It doesn't play in MacOS VLC (v.2.0). I am not sure what's going on. I always thought that media playback is binary thing (it works or it doesn't it). It's very strange that it behaves differently in different players. – Victor Ronin Aug 27 '12 at 16:53
  • No, it plays incorrectly and no audio. We are doing something wrong definitely. – Alex Amiryan Aug 28 '12 at 19:11
  • Also it's very poor documentation on 3GP. Can anyone find concrete specification of 3GPP format? – Alex Amiryan Aug 28 '12 at 19:11
  • Here is the specification: http://www.3gpp.org/ftp/Specs/archive/26_series/26.244/26244-b00.zip – Victor Ronin Aug 30 '12 at 15:31