IOSurfaces - Artefacts in video and unable to grab video surfaces

Question

This is a 2-part Question. I have the following code working which grabs the current display surface and creates a video out of the surfaces (everything happens in the background).

for(int i=0;i<100;i++){
        IOMobileFramebufferConnection connect;
        kern_return_t result;
        IOSurfaceRef screenSurface = NULL;

        io_service_t framebufferService = IOServiceGetMatchingService(kIOMasterPortDefault, IOServiceMatching("AppleH1CLCD"));
        if(!framebufferService)
            framebufferService = IOServiceGetMatchingService(kIOMasterPortDefault, IOServiceMatching("AppleM2CLCD"));
        if(!framebufferService)
            framebufferService = IOServiceGetMatchingService(kIOMasterPortDefault, IOServiceMatching("AppleCLCD"));

        result = IOMobileFramebufferOpen(framebufferService, mach_task_self(), 0, &connect);

        result = IOMobileFramebufferGetLayerDefaultSurface(connect, 0, &screenSurface);

        uint32_t aseed;
        IOSurfaceLock(screenSurface, kIOSurfaceLockReadOnly, &aseed);
        uint32_t width = IOSurfaceGetWidth(screenSurface);
        uint32_t height = IOSurfaceGetHeight(screenSurface);
        m_width = width;
        m_height = height;
        CFMutableDictionaryRef dict;
        int pitch = width*4, size = width*height*4;
        int bPE=4;
        char pixelFormat[4] = {'A','R','G','B'};
        dict = CFDictionaryCreateMutable(kCFAllocatorDefault, 0, &kCFTypeDictionaryKeyCallBacks, &kCFTypeDictionaryValueCallBacks);
        CFDictionarySetValue(dict, kIOSurfaceIsGlobal, kCFBooleanTrue);
        CFDictionarySetValue(dict, kIOSurfaceBytesPerRow, CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &pitch));
        CFDictionarySetValue(dict, kIOSurfaceBytesPerElement, CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &bPE));
        CFDictionarySetValue(dict, kIOSurfaceWidth, CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &width));
        CFDictionarySetValue(dict, kIOSurfaceHeight, CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &height));
        CFDictionarySetValue(dict, kIOSurfacePixelFormat, CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, pixelFormat));
        CFDictionarySetValue(dict, kIOSurfaceAllocSize, CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &size));

        IOSurfaceRef destSurf = IOSurfaceCreate(dict);

        IOSurfaceAcceleratorRef outAcc;
        IOSurfaceAcceleratorCreate(NULL, 0, &outAcc);

        IOSurfaceAcceleratorTransferSurface(outAcc, screenSurface, destSurf, dict, NULL);

        IOSurfaceUnlock(screenSurface, kIOSurfaceLockReadOnly, &aseed);
        CFRelease(outAcc);

        // MOST RELEVANT PART OF CODE

        CVPixelBufferCreateWithBytes(NULL, width, height, kCVPixelFormatType_32BGRA, IOSurfaceGetBaseAddress(destSurf), IOSurfaceGetBytesPerRow(destSurf), NULL, NULL, NULL, &sampleBuffer);

        CMTime frameTime = CMTimeMake(frameCount, (int32_t)5);

        [adaptor appendPixelBuffer:sampleBuffer withPresentationTime:frameTime];

        CFRelease(sampleBuffer);
        CFRelease(destSurf);
        frameCount++;
    }

P.S: The last 4-5 lines of code are the most relevant(if you need to filter).

1) The video that is produced has artefacts. I have worked on videos previously and have encountered such an issue before as well. I suppose there can be 2 reasons for this:
i. The PixelBuffer that is passed to the adaptor is getting modified or released before the processing (encoding + writing) is complete. This can be due to asynchronous calls. But I am not sure if this itself is the problem and how to resolve it.
ii. The timestamps that are passed are inaccurate (e.g. 2 frames having the same timestamp or a frame having a lower timestamp than the previous frame). I logged out the timestamp values and this doesn't seem to be the problem.

2) The code above is not able to grab surfaces when a video is played or when we play games. All I get is a blank screen in the output. This might be due to hardware accelerated decoding that happens in such cases.

Any inputs on either of the 2 parts of the questions will be really helpful. Also, if you have any good links to read on IOSurfaces in general, please do post them here.

I was trying to get screenshots in background, your code works great in most apps but in games it returns black screens as described in the second question. Have you got any ideas how to fix that? Also, RecordMyScreen works well in games, but it's not open source now so I can't figure out how it capture screens. — cloudycliff, Feb 09 '14 at 08:15
The above code isn't able to capture screens rendered using openGL. RecordMyScreen uses CARenderServerRenderDisplay() API. The code is at github - https://github.com/coolstar/RecordMyScreen. The file CSScreenRecorder.m contains the core code. I compared the performance for both the API's and found out that IOMobileFramebufferGetLayerDefaultSurface() is faster compared to CARenderServerRenderDisplay(). — Hrishikesh_Pardeshi, Feb 09 '14 at 15:23
RecordMyScreen's code at github is out of date now. I tried CARenderServerRenderDisplay() in iOS7, only returns black screen. — cloudycliff, Feb 10 '14 at 00:05
You are right. CARenderServerRenderDisplay() doesn't work on iOS7. Have you tried the above code on iOS 7? I was working on this project a year back, I haven't checked it out on iOS 7. — Hrishikesh_Pardeshi, Feb 10 '14 at 12:53
The code above works well on iOS7, but not with games...Games only give me black screen. I tried to reverse engineering the latest RecordMyScreen, seems that they still use CARenderServerRenderDisplay() to capture screenshot, have no idea how it works. — cloudycliff, Feb 10 '14 at 13:27
I worked more with RecordMyScreen's code, CARenderServerRenderDisplay() works on iOS7 as a springboard tweak. — cloudycliff, Feb 11 '14 at 00:24
I am also trying to capture the screen from a background app and use IOSurface/IOMobileFramebufferGetLayerDefaultSurface. I have code that works reasonably well in iOS <=8.x, but it appears to no longer work in iOS 9. In iOS 9 the app crashes at runtime complaining that the framework CoreSurface can't be loaded. Have you had any luck making this work in iOS 9? — Duncan C, Sep 02 '15 at 17:02

score 5 · Accepted Answer · answered Jan 04 '13 at 06:13

I did a bit of experimentation and concluded that the screen surface from which the content is copied is changing even before the transfer of contents is complete (call to IOSurfaceAcceleratorTransferSurface() ). I am using a lock (tried both asynchronous and read-only) but it is being overridden by the iOS. I changed the code between the lock/unlock part to the following minimal:

IOSurfaceLock(screenSurface, kIOSurfaceLockReadOnly, &aseed);
aseed1 = IOSurfaceGetSeed(screenSurface);
IOSurfaceAcceleratorTransferSurface(outAcc, screenSurface, destSurf, dict, NULL);
aseed2 = IOSurfaceGetSeed(screenSurface);
IOSurfaceUnlock(screenSurface, kIOSurfaceLockReadOnly, &aseed);

The GetSeed function tells if the contents of the surface have changed. And, I logged a count indicating the number of frames for which the seed changes. The count was non-zero. So, the following code resolved the problem:

if(aseed1 != aseed2){
//Release the created surface
continue; //Do not use this surface/frame since it has artefacts
}

This however does affect performance since many frames/surfaces are rejected due to artefacts. Any additions/corrections to this will be helpful.

I am having very similar problems trying to figure out why the output video is distorted and still couldnt find an answer. Is there a way i can access the remainder of the functions you used to get screen capture working? Like can you upload them to github? take care — aytunch, Apr 20 '15 at 15:24
Did you try the solution I mentioned i.e. comparing aseed1 and aseed2 and discarding if the seeds aren't the same? This will ensure that the image was copied correctly. — Hrishikesh_Pardeshi, Apr 28 '15 at 09:45

IOSurfaces - Artefacts in video and unable to grab video surfaces

1 Answers1

Linked