How can I access a graphics card's output directly?

Question

Do graphics cards typically write their output to some place in memory which I can access? Do I have to use the driver? If so, can I use OpenGL?

I want to know if it's possible to "capture" the output of a VM on Linux which has direct access to a GPU, and is running Windows. Ideally, I could access the output from memory directly, without touching the GPU, since this code would be able to run on the Linux host.

The other option is to maybe write a Windows driver which reads the output of the GPU and writes it to some place in memory. Then, on the Linux side, a program can read this memory. This seems somewhat impossible, since I'm not really sure how to get a process on the host to share memory with a process on the guest.

Is it possible to do option 1 and simply read the output from memory?

Answer 1

I do not code under Linux but in Windows (you are running it in emulator anyway) you can use WinAPI to directly access canvas of any window or even desktop from 3th party App. Some GPU overlays could be problematic to catch (especially DirectX based) but I had no problems with mine GL/GLSL for now.

If you got access to App source you can use glReadPixels for image extraction from GL directly (but that works only for current GL based rendering).

1. Using glReadPixels

   As mentioned this must be implemented directly in the targeted app so you need to have it source code or inject your code in the right place/time. I use for screenshoting this code:

   void OpenGLscreen::screenshot(Graphics::TBitmap *bmp)
       {
       if (bmp==NULL) return;
       int *dat=new int[xs*ys],x,y,a,*p;
       if (dat==NULL) return;
       bmp->HandleType=bmDIB;
       bmp->PixelFormat=pf32bit;
       if ((bmp->Width!=xs)||(bmp->Height!=ys)) bmp->SetSize(xs,ys);
       if ((bmp->Width==xs)&&(bmp->Height==ys))
           {
           glReadPixels(0,0,xs,ys,GL_BGRA,GL_UNSIGNED_BYTE,dat);
           glFinish();
           for (a=0,y=ys-1;y>=0;y--)
            for (p=(int*)bmp->ScanLine[y],x=0;x<xs;x++,a++)
             p[x]=dat[a];
           }
       delete[] dat;
       }
   

   where xs,ys is OpenGL window resolution, you can ignore the whole bmp stuff (it is VCL bitmap I use to store screenshot) and also can ignore the for it just copy the image from buffer to bitmap. So the important stuff is just this:

   int *dat=new int[xs*ys]; // each pixel is 32bit int
   glReadPixels(0,0,xs,ys,GL_BGRA,GL_UNSIGNED_BYTE,dat);
   glFinish();
   

   You need to execute this code after the rendering is done otherwise you will obtain unfinished or empty buffer. I use it after redraw/repaint events. As mentioned before this will obtain only the GL rendered stuff so if your App combines GDI+OpenGL it is better to use the next approach.

2. WinAPI approach

   To obtain Canvas image of any window I wrote this class:

   //---------------------------------------------------------------------------
   //--- screen capture ver: 1.00 ----------------------------------------------
   //---------------------------------------------------------------------------
   class scrcap
       {
   public:
       HWND hnd,hnda;
       TCanvas *scr;
       Graphics::TBitmap *bmp;
       int x0,y0,xs,ys;
       scrcap()
           {
           hnd=NULL;
           hnda=NULL;
           scr=new TCanvas();
           bmp=new Graphics::TBitmap;
           #ifdef _mmap_h
           mmap_new('scrc',scr,sizeof(TCanvas()        ));
           mmap_new('scrc',bmp,sizeof(Graphics::TBitmap));
           #endif
           if (bmp)
               {
               bmp->HandleType=bmDIB;
               bmp->PixelFormat=pf32bit;
               }
           x0=0; y0=0; xs=1; ys=1;
           hnd=GetDesktopWindow();
           }
       ~scrcap()
           {
           #ifdef _mmap_h
           mmap_del('scrc',scr);
           mmap_del('scrc',bmp);
           #endif
           if (scr) delete scr; scr=NULL;
           if (bmp) delete bmp; bmp=NULL;
           }
       void init(HWND _hnd=NULL)
           {
           RECT r;
           if (scr==NULL) return;
           if (bmp==NULL) return;
           bmp->SetSize(1,1);
           if (!IsWindow(_hnd)) _hnd=hnd;
           scr->Handle=GetDC(_hnd);
           hnda=_hnd;
           resize();
           }
       void resize()
           {
           if (!IsWindow(hnda)) return;
           RECT r;
   //      GetWindowRect(hnda,&r);
           GetClientRect(hnda,&r);
           x0=r.left; xs=r.right-x0;
           y0=r.top; ys=r.bottom-y0;
           bmp->SetSize(xs,ys);
           xs=bmp->Width;
           ys=bmp->Height;
           }
       void capture()
           {
           if (scr==NULL) return;
           if (bmp==NULL) return;
           bmp->Canvas->CopyRect(Rect(0,0,xs,ys),scr,TRect(x0,y0,x0+xs,y0+ys));
           }
       };
   //---------------------------------------------------------------------------
   //---------------------------------------------------------------------------
   //---------------------------------------------------------------------------
   

   Again it uses VCL so rewrite bitmap bmp and canvas scr to your programing environment style. Also igneore the _mmap_h chunks of code they are just for debugging/tracing of memory pointers related to some nasty compiler bug I was facing at that time I wrote this.

   The usage is simple:

   // globals
   scrcap cap;
   // init
   cap.init();
   
   // on screenshot
   cap.capture();
   // here use cap.bmp
   

   If you call cap.init() it will lock on the whole windows desktop. If you call cap.init(window_handle) it will lock on specific visual window/component instead. To obtain window handle from 3th app side see:

   • is ther a way an app can display a message without the use of messagebox API?

   Sorry it is on SE/RE instead of here on SE/SO but My answer here covering this topic was deleted. I use this for video capture ... all of the animated GIFs in my answers where created by this code. Another example could be seen on the bottom of this answer of mine:

   • Image to ASCII art conversion

   As you can see it works also for DirectX overlay with media player classic (even Windows PrintScreen function cant do it right). As I wrote I got no problems with this yet.

   Beware visual stuff WinAPI calls MUST BE CALLED FROM APP MAIN THREAD (WNDPROC) otherwise serious problems could occur leading to random unrelated WinAPI calls exceptions anywhere in the App.