19

I have a card that produces a H264 stream with a SPS (Sequence Parameter Set) and a PPS (Picture Parameter Set), in that order, directly before each I-Frame. I see that most H264 streams contain a PPS and SPS at the first I-Frame.

Is this recommended? Do decoders/muxers typically support multiple PPS and SRS?

szatmary
  • 29,969
  • 8
  • 44
  • 57
Paul Knopf
  • 9,568
  • 23
  • 77
  • 142

3 Answers3

38

H.264 comes in a variety of stream formats. One variation is called "Annex B".

(AUD)(SPS)(PPS)(I-Slice)(PPS)(P-Slice)(PPS)(P-Slice) ... (AUD)(SPS)(PPS)(I-Slice).

Typically you see SPS/PPS before each I frame and PPS before other slices.

Most decoders/muxers are happy with "Annex B" and the repetition of SPS/PPS.

Most decoders/muxers won't do anything meaningful if you change the format and SPS/PPS midstream.

Most decoders/muxers parse the first SPS/PPS as part of a setup process and ignore subsequent SPSs.

Some decoders/muxers prefer H.264 without the (AUD), start codes and SPS/PPS. Then you have to feed SPS/PPS out of band as part of setting up the decoders/muxers.

Markus Schumann
  • 7,636
  • 1
  • 21
  • 27
  • 2
    The MP4 format can be used with in-mdat SPS and PPS, but almost every file I've seen puts the SPS and PPS in the track description under stsd.avc1.avcC and then the mdat only contains sample data. Changing SPS and PPS in the middle of the stream is the simplest wacky thing that can happen. I'm pretty sure it's possible for a single stream to have various samples that are based on different SPS and PPS. Consider the fields slice_header.pic_parameter_set_id and picture_parameter_set.seq_parameter_set_id . I suspect most decoders don't support this as of 2013. – Mutant Bob Dec 19 '13 at 17:12
30

An IDR frame, or an I-slice can not be decoded without a SPS and PPS. In the case of a container like mp4, the SPS and PPS is stored away from the video data in the file header. Upon playback the mp4 is parsed, the SPS/PPS is used to configure the AVC decoder once, then video can be played back starting at any IDR/I-slice.

There is a second scenario, Live video. With live video, there is no file header, because there is no file. So when a TV tunes into a channel, where does it get the SPS/PPS? Because television is broadcast, meaning the television has no way to request the SPS/PPS, it is repeated in the stream.

So when you start encoding video, your encoder does not know what you intend to do with the video. Now if the extra SPS/PPS show up in an mp4, the decoder just ignores them, but if you are streaming to a TV, without them the stream would never play. So most of the default to repeating SPS/PPS just in case.

szatmary
  • 29,969
  • 8
  • 44
  • 57
1

I know about matroska(mkv) spec so here SPS and PPS are stored only once as codec private data section. So they are not repeating with every i frame or IDR frame.

If your h264 stream's each i frame/IDR frame has SPS/PPS then matroska muxer will store only 1 copy in codec private data.

So while storing usecase based container format suggest to use only one copy of SPS/PPS but broadcasting and streaming based container formats suggest send SPS/PPS before every iFrame/IDR frame or whenever any codec changes change in h264 stream at that time

Jeegar Patel
  • 26,264
  • 51
  • 149
  • 222
  • While mkv does store "private codec data" which is the SPS and PPS for H264, nothing stops you from writing these packets to the stream. Players like VLC and ffplay will change the codec settings when they see SPS/PPS in the stream. – David Weber Jun 11 '18 at 08:14
  • Agreed, Matroska Demuxer will treat those SPS/PPS + encoded data as encoded data only and give them to decoder and All h264 decoder would be able to decode them. But i dont see any benefit of doing this if its not a streaming usecase or codec is changing its setting – Jeegar Patel Jun 12 '18 at 10:18