I experience occasional stuttering of varying severity in my implementation of streaming video over network. I've tested my application on an Iphone 7 and an Ipad 9.7, both suffer from occasional stuttering but the Iphone 7 seems to stutter the most. I'd like to think that this is not purely a hardware issue, since I can stream video through youtube without any problems at all.
My first implementation of video streaming over network was just sending jpeg images from my pc to my Iphone 7, this suffered from the same problem. I checked if there were dropped packages by incrementing a number and adding it to every package I sent, there was stuttering but no dropped packages. I checked if the issue was that there were no pictures to render because the packages arrived late by delaying the render and keeping the received images in a buffer, still stutters.
My guess was that the stuttering happened because my event scheduler didn't always fire right on time (which it didn't do), I tested different event schedulers but I didn't manage to implement anything precise enough. I thought that if I could just pass a h264 encoded bitstream to some built-in objective-c class then it would manage the decoding and rendering of my image on time for me. So that's what I've tried doing.
I started of by following this guide on streaming h264 encoded video to IOS. I had to make a few changes to get it working, since my h264 bitstream contains several picture nal units per frame. I append each one of them to the CMBlockBuffer, instead of just creating a block buffer encapsulating the first picture nal unit like the linked guide does.
My sample buffer attachments also look like this instead of how they are shown in the guide
CFMutableDictionaryRef dict = (CFMutableDictionaryRef)CFArrayGetValueAtIndex(attachments, 0);
//CFDictionarySetValue(dict, kCMSampleAttachmentKey_DisplayImmediately, kCFBooleanTrue);
CFDictionarySetValue(dict, kCMSampleAttachmentKey_IsDependedOnByOthers, kCFBooleanTrue);
if (naluType == 1) {
// P-frame
CFDictionarySetValue(dict, kCMSampleAttachmentKey_NotSync, kCFBooleanTrue);
CFDictionarySetValue(dict, kCMSampleAttachmentKey_DependsOnOthers, kCFBooleanTrue);
} else {
// I-frame
CFDictionarySetValue(dict, kCMSampleAttachmentKey_NotSync, kCFBooleanFalse);
CFDictionarySetValue(dict, kCMSampleAttachmentKey_DependsOnOthers, kCFBooleanFalse);
}
These are the attachments that the Moonlight Streaming Service uses. I don't display the picture immediately because I set the presentation timestamp of each frame to 1/60th of a second more than the previous frame with CMSampleBufferSetOutputPresentationTimeStamp.
But I still suffer from stuttering. I've read about IOS hogging threads and that it can mess up drawing frames on time, but all the websites that stream video can do it on my device without stuttering. Surely I should be able to do the same for my app? I've also tried building my app with a release build but that did not help. I know that asking "How do I fix my video stream stuttering" is a pretty broad question but I hope that mentioning what I've attempted, how my implementation looks and the fact that web sites like youtube can stream without stuttering on my hardware should be enough information for someone to be able to point me to the right direction. I know I could try a web-based solution like WebRTC but if possible I'd like to fix the issue I have, instead of making something completely new.
Update 1
In my project I print the time between two picture packets arriving. What used to happen was that the stream would stutter at the same time a packet would arrive late even when delaying the stream playback. Reading something online made me think that fixing my mem leaks might solve my problem. I don't know if it's related, or if my previous tests were done incorrectly or if it all was one big coincidence. But after fixing my mem leaks, my project now stutters when packets arrive late, but the stutters are also delayed if my stream is. Now it might just be that I set PTS incorrectly.
Update 2
I can delay my playback properly with PTS and I can make it playback faster or slower depending on the time stamps I feed the sample buffer so I don't think I've made a mistake with setting PTS. I recorded my screen to show how it looks like. The example stores 600 frames worth of data in a container and then decodes all of them in one go to make sure that the stuttering is not because of packets arriving late. The video also prints out the time between the packet arriving and the previous packet arriving, if the time between them is longer than a bit over 1/60th of a second. The video and PTS relevant code of the example is below.
Video: https://youtu.be/Ym5rfHwg-eM
// init function
CMTimeBaseCreateWithMasterClock(CFAllocatorGetDefault(), CMClockGetHostTimeClock(), &_controlTimebase);
_displayLayer.controlTimebase = _controlTimebase;
CMTimebaseSetTime(_displayLayer.controlTimebase, CMTimeMake(0, 60));
CMTimebaseSetRate(_displayLayer.controlTimebase, 1.0);
// f = frame. Starts at 630 to delay playback with 630 frames.
f = 630;
......................
// received packet containing frame function
[frames addObject:data]; // data is an NSData of an encoded frame
// store 600 frames worth of data then queue all of them in one go to make sure that the stutter is not because packets arrive late
if ([frames count] == 600)
{
for (int i = 0; i < 600; i++)
{
uint8_t *bytes = (uint8_t*)[frames[i] bytes];
[self->cameraView.streamRenderer decodeFrame:bytes length:frames[i].length;
}
}
......................
// decode frame function
CMSampleBufferSetOutputPresentationTimeStamp(sampleBuffer, CMTimeMake(f, 60));
f++;
Update 3
Now I have also tried using CADisplayLink to get callbacks to when I should draw my image, where I then render the CVPixelBuffer with the Metal framework. I use the same example as the one described in update 2 and I still suffer from the same issues. I've made sure to print the time it takes for Metal to render my CVPixelBuffer and it take around 3 milliseconds per CVPixelBuffer, well below the refresh rate of the screen.
Update 4
I tried rendering the index of the current frame above the rendered pixel buffer. With this I can check whether the display is lagging or if the pixel buffers are bad. After stepping through the video frame by frame I could see that the rendered index was incrementing even when the stream was lagging. Now I'm thinking that it's either the decoding or encoding of the video stream that is the problem.
