Access framebuffer for arbitrary MacOS application

Question

I am trying to capture the frame buffer for an arbitrary application on Mac and process the frame buffers, frame by frame. Is there a way to tap into an applications internal frame buffer? I also noticed there are some Graphics API functions like this that can capture the entire screen, but I worry that

A) I'm going to need to figure out where my window is and crop it out of the entire screen image which may take a non-zero amount of time

B) Since it's capturing the entire screen instead of just one window, I imagine it'll take a longer amount of time.

I'm hoping to be able to capture and process 20 frames per second, so speed is pretty important.

score 3 · Accepted Answer · edited May 23 '17 at 12:01

I have a couple of ideas...

Firstly, and most simply, you can start Quicktime (which is supplied with macOS) and go to File->New screen recording and record an arbitrary area of the screen and save it in a movie and analyse the frames later.

Secondly, you can use screencapture (/usr/sbin/screencapture) and either specify a rectangle to capture or a window id. The manual page is incorrect and rubbish, so use the following to see the actual options:

screencapture -h

Output

usage: screencapture [-icMPmwsWxSCUtoa] [files]
  -c         force screen capture to go to the clipboard
  -b         capture Touch Bar - non-interactive modes only
  -C         capture the cursor as well as the screen. only in non-interactive modes
  -d         display errors to the user graphically
  -i         capture screen interactively, by selection or window
               control key - causes screen shot to go to clipboard
               space key   - toggle between mouse selection and
                             window selection modes
               escape key  - cancels interactive screen shot
  -m         only capture the main monitor, undefined if -i is set
  -M         screen capture output will go to a new Mail message
  -o         in window capture mode, do not capture the shadow of the window
  -P         screen capture output will open in Preview
  -I         screen capture output will in a new Messages message
  -s         only allow mouse selection mode
  -S         in window capture mode, capture the screen not the window
  -t<format> image format to create, default is png (other options include pdf, jpg, tiff and other formats)
  -T<seconds> Take the picture after a delay of <seconds>, default is 5
  -w         only allow window selection mode
  -W         start interaction in window selection mode
  -x         do not play sounds
  -a         do not include windows attached to selected windows
  -r         do not add dpi meta data to image
  -l<windowid> capture this windowsid
  -R<x,y,w,h> capture screen rect
  -B<bundleid> screen capture output will open in app with bundleidBS
  files   where to save the screen capture, 1 file per screen

As you can see, the -l, -R options are very useful.

I wrote a little program to get a list of windowids in another answer, here.

The size of the window and the file format make a difference to the speed. I find JPEG is normally fastest, and PNG is normally slowest. I can get 20 frames a second with a reasonable size window using this:

time for i in {0..99}; do screencapture -l 56 -t jpg fred-$i.jpg; done

where I got the 56 from the windowlist program in my other, linked answer.

Quicktime won't be sufficient since I need to process in realtime, but screencapture looks perfect. I'll look into it, thank you so much! — Nat, Mar 26 '17 at 19:19

Access framebuffer for arbitrary MacOS application

1 Answers1