6

I have an H264 stream (IIS - smooth streaming) that I would like to play with SilverLight. Apparently SilverLight can do it, but how?

Note:
the VC-1 stream can be played by the SilverLight, but H264 not.

Also, I can provide a stream and any additional information required. H264 encoder is the one in Media Foundation (MFT). Same goes for the VC-1 that works (although is it impossible to create equal chunks for smooth streaming because forcing key-frame insertion makes video jerky.

EDIT: MPEG2VIDEOINFO values for H264:

MPEG2VIDEOINFO values for H264

user1764961
  • 673
  • 7
  • 21
  • I can certainly help with this. But it is a large topic. The issue can be in the source media, the server or the client. What server are you using? IIS? Wowza? Where kind of media are you streaming? ismv? what does the ismc and ism look like? Where did the media come from? Did you encode it? forcing key-frame insertion will not make video jerky if done correctly. – szatmary Aug 07 '13 at 18:48
  • I'm using IIS. I have my own implementation of stream encoding using MFT VC-1 and H264. As I sad, the VC-1 works fine, but when I call IWMVideoForceKeyFrame::SetKeyFrame() just before IMFTransform::ProcessInput(), the key frame is inserted as expected, but the video is jerky. By "jerky" I mean that you can see how some previous frame has been inserted. I can run the stream and show you that. Regarding H264, I did everything as described here: http://msdn.microsoft.com/en-us/library/ff402310(v=vs.90).aspx. The encoding process is running, the IIS is receiving it, but clients cannot decode it. – user1764961 Aug 07 '13 at 20:27
  • @szatmary, if you need some data or you want to see that 'jerky' video, just let me know and I will arrange it. – user1764961 Aug 07 '13 at 20:35
  • I want to make sure I understand. You can not get AVC to work al all. VC-1 works, but is 'jerky' if you force keyframe intervals. This correct? – szatmary Aug 07 '13 at 21:29
  • VC-1 works, but if I force key frame insertion in order to create 50 frames long chunks before sending them to IIS, then codec inserts some previous frames in and the result is jerky video. Do you want to see that? AVC1 encoding works - no errors. Creating chunks and pushing to IIS is also fine. But, when you open that stream in the browser, there is only black screen.. nothing. Also, I can demonstrate it too. – user1764961 Aug 07 '13 at 21:36
  • Here is the jerky VC-1 stream: http://access9.streamsink.com/smooth/SmoothStreamingPlayer.html Key frame inserted on every 50th frame. If you play the stream slower, you will notice how previous frame is inserted. If I don't insert key frames manually on every 50th frame, then it is fine. Ignore the interlaced content. – user1764961 Aug 07 '13 at 21:44

2 Answers2

9

Just a guess. Based on your question 18009152. I am guessing you are encoding h.264 using the annexb bitstream format. According to comments, you can not tell the encoder to use AVCC format. Therefore, you must perform this conversion manually (Annex B WILL NOT work in an ISO container). You can do this by looking for start codes in your AVC stream. A start code is 3 or 4 bytes (0x000001, 0x00000001). You get the length of the NALU by locating the next start code, or the end of the stream. Strip the start code (throw it away) and in its place write the size of the NALU in a 32bit integer big endian. Then write this data to the container. Just to be clear, this is performed on the video frames that come out of the encoder. The extra data is a separate step that appears you have mostly figure out (except for the NALUSizeLength). Because we uses a 4 byte integer to write the NALU sizes, you MUST set NALUSizeLength to 4.

szatmary
  • 29,969
  • 8
  • 44
  • 57
  • Although I didn't try it, I think you will nail this down. The start code is 4 bytes long in my case. But, since I'm not writing this to any container (I'm streaming (PUSH) it to the IIS), does it mean that I should send SPS/PPS with NALU size before sending each packet with frames (e.g. 50 frames) to the IIS? Or should I send it with every frame? Or... – user1764961 Aug 12 '13 at 20:17
  • Be careful, Just because you see one 4 byte start code, doesn't mean they will all be 4. I'm not sure what you mean by frames in this case. But you will have to do this for every NALU (startcode). Remember NALU != frame. A frame may contain several NALUs. – szatmary Aug 12 '13 at 20:26
  • Yes, I noticed about 10-12 NALUs per frame. I added the size in front of each NALU exactly as you said, but unfortunately it didn't help. :-( – user1764961 Aug 12 '13 at 23:29
  • It works!!!!!! I forgot to put the NALU size in big endian. You earned every single one from the 500. I would buy you a 6 pack also.. saved my ass. – user1764961 Aug 12 '13 at 23:47
  • Awesome, Glad I could Help. Sorry I got testy in the comments. I know this stuff very well and get frustrated when I can not communicate my thoughts. like a little kid :) – szatmary Aug 13 '13 at 00:03
  • Same here. I lost my temper, because I was debugging working code for a month. Usually till 7am. But I needed a guy on this who really knows the stuff. So, you provided the missing part - NALU size for each NALU in each compressed frame. I'm getting drunk tomorrow, definitely :-) Thanks again!!! – user1764961 Aug 13 '13 at 00:09
-2

Silverlight 3 can play H264 files. Use MediaStreamSource for this.

Here is the interface description: http://msdn.microsoft.com/en-us/library/system.windows.media.mediastreamsource(v=vs.95).aspx

Also, this blog entry is related to H264 playing sing Silverlight 3: http://nonsenseinbasic.blogspot.ru/2011/05/silverlights-mediastreamsource-some.html It will help you with other issues that may arise.

Sergey K.
  • 24,894
  • 13
  • 106
  • 174
  • Regarding NAL units and 3-byte start codes, I don't see a way to accomplish that. According to MSDN, I have to send SPS and PPS concatenated together and separated by the 2-byte length field. The SSFMuxAddStream() understands that and when I call SSFMuxGetHeader(), that function will create SPS and PPS with 4 byte start codes. The 2-byte length fields are simply replaced by 4-byte start codes. – user1764961 Aug 07 '13 at 20:33
  • Have you checked this example? http://stackoverflow.com/questions/1966215/how-to-play-mp4-h-264-video-in-silverlight-3-or-4-from-a-url – Sergey K. Aug 07 '13 at 20:41
  • yes, I have. I downloaded some example from the net and later I wrote one on my own from scratch. I got the same result. – user1764961 Aug 07 '13 at 21:04
  • As szatmary suggested, perhaps there is a problem in the stream.. but it is being sliced to chunks and pushed to the IIS in the same way as VC-1. – user1764961 Aug 07 '13 at 21:11
  • There is a lot of it, which part would you like to see? – user1764961 Aug 07 '13 at 21:26
  • 2 byte length? Are you referring to the avcC box? There is more to it than a simple SPS+PPS – szatmary Aug 07 '13 at 21:31
  • @user1764961: where you access the file and create ``MediaStreamSource`` – Sergey K. Aug 07 '13 at 21:35
  • I'm referring to this: http://msdn.microsoft.com/en-us/library/ff402310(v=vs.90).aspx. Look the H.264 /AVC1 section. – user1764961 Aug 07 '13 at 21:37
  • "The codec private data in the case of H.264/AVC1 is assumed to be the portion of the MPEG2VIDEOINFO structure immediately following the VIDEOINFOHEADER2." - do you respect this guideline? – Sergey K. Aug 07 '13 at 21:40
  • Im sorry user1764961 I can't help any more until you post some code. There are WAY to many things it can be. – szatmary Aug 07 '13 at 21:46
  • I'm putting the codec private data (SPS/PPS) at the dwSequenceHeader's address in the MPEG2VIDEOINFO. – user1764961 Aug 07 '13 at 21:49
  • I cannot post 5000 lines of code. You have to say what exactly are you interested in. – user1764961 Aug 07 '13 at 21:49
  • What are you putting for all the other values of that structure? – szatmary Aug 07 '13 at 21:50
  • We need your help to answer your question. If you don't help us, we cant help you. How are we supposed to read that image. My last guess is to make sure your encoder is NOT encoding in annex B, and check the size length. It is probably 4, not 2. Good Luck. – szatmary Aug 07 '13 at 22:44
  • And you sure you encoder is not set to annex B? Can you post the section of code that proves this? Can you post the documentation or code thats says NALUSizeLength = 2? – szatmary Aug 07 '13 at 23:19
  • @szatmary, I think the NALU size length is irrelevant here. I'm pretty sure about that, because according to MSDN: MPEG2VIDEOINFO::dwFlags is the number of bytes used for the length field that appears before each NALU. The length field indicates the size of the following NALU in bytes. It should be set to the CodecPrivateData NALUSizeLength + 1. Valid values are 1, 2, and 4. No matter what size I use, SSFMuxAddStream() API will add the stream, and SSFMuxGetHeader() will return the correct SPS+PPS with 4-byte start code. Based on that I would say the NALU size length is always interpreted right – user1764961 Aug 08 '13 at 11:02
  • Yes, you are missing something. I have said it 3 times now. Post the part of the code where you configure the h.264 encoder to use Avcc stream format as apposed to annex B. code or GTFO. – szatmary Aug 08 '13 at 18:48
  • It is in the attached picture! All H264 parameters. In the right pane (memory view) are SPS/PPS values. Red rectangle is around it. What else can I do, read you the numbers?! Jesus... – user1764961 Aug 08 '13 at 20:51
  • It is not in the attached picture. You think that the NALUSizeLength is used to set the values in the extradata field. This is false. It is used when the decoder is reading the NALU sizes while processing the elementary stream. This value must match the values being produced by your encoder. You do not know enough about the elementary stream. I will ass again. Show me in the code where you set the bitstreram format to avcc (not annexb) and show me where in the encoder (NOT THE MUXER) that the NALUSizeLength is 2. – szatmary Aug 12 '13 at 03:22
  • I'm not able to set it, because I'm on Win 7 and minimum required client for setting the NALU size on the encoder is Win 8. But still, the documentation says that the encoder can be used on Win 7 too. And people really use it on Win 7. – user1764961 Aug 12 '13 at 15:16