3

I am interested in quickly, at very low-latency, capturing the pixels of what a user might see on a screen. Assuming I can control the host, how might you get those pixels if they're stored in GPU memory? What would be the fastest way?

I've looked in DRM but haven't had much luck.

There's not too much information out there since most applications are centered around drawing things instead.

I am specifically interested in using NVIDIA GPUs.

Error_001_
  • 5
  • 3
Suhail Doshi
  • 716
  • 1
  • 10
  • 23
  • You probably need to be more specific about technologies you are using. For NVIDIA, see [Direct access to frame buffer?](https://devtalk.nvidia.com/default/topic/453751/cuda-programming-and-performance/direct-access-to-frame-buffer-/) and questions like [How to read a pixel off of the screen?](https://stackoverflow.com/q/4425668/608639) for OpenGL (and related questions like [How to render offscreen on OpenGL?](https://stackoverflow.com/q/12157646/608639), which discusses `glReadPixels`) – jww Aug 03 '19 at 00:07
  • I think the most sensible things to use are X11, NVIDIA GPUs. You'd need the desktop GUI to be stored in the GPU and then you'd have to work with the GPU to get frame. Those links you posted didn't seem to go anywhere. – Suhail Doshi Aug 03 '19 at 00:12
  • [assuming X11] You get the pixels with XGetImage(3) on the root window (it's plenty fast on any machine newer than 20 years, and using shared memory is a waste of time that only complicates things). If you want "quickly, low-latency" you get only the parts which were modified since the last time, not the whole screen. To determine which parts were modified, you use the [Damage](https://www.x.org/releases/X11R7.5/doc/damageproto/damageproto.txt) extension. –  Aug 03 '19 at 03:16
  • To be clear, I am looking for something that's < 5ms fast for frame capture. My worry with XGetImage is that it's not a zero-copy way of getting the image to then later h264 encode the frame since you'd have to copy it from host memory to GPU memory which will take time. – Suhail Doshi Aug 03 '19 at 06:05

1 Answers1

0

I'm not sure if this fullfills your needs, but, since you have Linux tagged, you may use the framebuffer to receive the pixels.

It is usually located at /dev/fb0

It is a regular linux file, hence, you can read and write to it as usual.

Okan Barut
  • 279
  • 1
  • 7
  • That's close but my understanding is that NVIDIA doesn't support the Linux Framebuffer I think? https://devtalk.nvidia.com/default/topic/984904/nvidia-devs-any-eta-on-fbdev-console-mode-setting-implementation-/ – Suhail Doshi Aug 05 '19 at 06:57
  • Well, according to your source, there's this line: **However, the kernel compiled-in efifb module supports high-resolution nvidia console on EFI systems**, so it depends on the kernel, I guess? – Okan Barut Aug 05 '19 at 16:14
  • Also, I work on a linux machine with propriety and open source NVIDIA drivers. It may be possible that on the open-source version you may get access to the GPU pixels. I think the driver was called "Nouveau" – Okan Barut Aug 05 '19 at 16:17