Capturing Display / Monitor Images, Sending Keyboard Input on Linux

Question

I need to process images sent to my laptop's video display, and I need to send keyboard input to my Linux system, using a C++ or shell program.

My goal is to process images that are part of an FPS game, then taking action inside that game (hence the keyboard input) based on these images. Instead of trying to understand (if it's even possible) how to interface to game X or Y, using some API, I figured this is the quickest way to interface to any game, hijacking Linux input and output somehow.

Is there any way to do this without any kernel, or device driver hacking? I used recordmydesktop to record my desktop as a video before, I guess I could hack its code and try to reverse-engineer something from that. Any other ideas? I am on Ubuntu 11.

Related Question

Answer 1

You don't need to do anything as low level as the kernel or device drivers to do this.

You can use the XTest X11 extension to fake input events programatically, for example (from this posting, there's another example for keyboard).

#include <X11/extensions/XTest.h>  
#include <unistd.h>  

int main ()  
{  
  Display *dpy = NULL;  
  XEvent event;  

  dpy = XOpenDisplay (NULL);  

  /* Get the current pointer position */  
  XQueryPointer (dpy, RootWindow (dpy, 0),  
        &event.xbutton.root, &event.xbutton.window,  
        &event.xbutton.x_root, &event.xbutton.y_root,  
        &event.xbutton.x, &event.xbutton.y,  
        &event.xbutton.state);  

  /* Fake the pointer movement to new relative position */  
  XTestFakeMotionEvent (dpy, 0, event.xbutton.x + 100,  
        event.xbutton.y + 50, CurrentTime);  
  XSync(dpy, 0);  
  XCloseDisplay (dpy);  
  return 0;  
}   

To capture the images the simplest way is to use function interposition (via LD_PRELOAD) to "intercept" calls to glXSwapBuffers, which will be called once each frame is drawn. From there you can copy the contents of the framebuffer, using glReadPixels and do with it what you want.

E.g. untested outline for intercepting OpenGL frames:

// Function pointer to the *real* glXSwapBuffers
static void (*glx_fptr)(Display*, GLXDrawable) = NULL;

// Make sure init gets called when the shared object is loaded. GCC specific.
static void init(void)  __attribute__((constructor));

static void init(void) {
    dlerror();
    // find the real glXSwapBuffers
    glx_fptr = dlsym(RTLD_NEXT, "glXSwapBuffers");
    if (NULL == glx_fptr)
        fprintf(stderr, "[glvidcap] %s\n", dlerror());
}

void glXSwapBuffers(Display *dpy, GLXDrawable drawable) {
    unsigned int w = 0;
    unsigned int h = 0;
    static int x,y;
    static Window win;
    static unsigned int border,depth;
    // Find the window size. (You could skip this and make it all static if you
    // Trust the window not to change size
    XGetGeometry(dpy, drawable, &win, &x, &y, &w, &h, &border, &depth);

    // Assuming frame is some memory you want the frame dumped to:
    glReadPixels(0,0,w,h,GL_BGR,GL_UNSIGNED_BYTE, frame);

    // Call the real function:
    assert(glx_fptr);
    glx_fptr(dpy, drawable);
}

Then you want to compile this as a shared object and LD_PRELOAD that shared object before running whatever game you're looking at.

If it happens to be an SDL application instead you can intercept the calls to SDL_Flip or SDL_UpdateRect as appropriate.

Answer 2

Thanks to @awoodland's answer, I looked for related resources, and found this

http://bzr.sesse.net/glcapture/glcapture.c

I compile this code as

gcc -shared -fPIC -o glcapture.so glcapture.c -ldl

and load it for FPS game Urban Terror in a script as

LD_PRELOAD=`pwd`/glcapture.so [DIR]/UrbanTerror/ioUrbanTerror.i386

The moment this script is ran, the game is loaded. For some reason, the game graphics do not show, but I can improve that later. When I exit the game, I see gettimeofday messages printed on the console, which tells me the hook worked. I'll provide more details as I continue to work this code.

For sending key presses, I followed the link

http://bharathisubramanian.wordpress.com/2010/03/14/x11-fake-key-event-generation-using-xtest-ext/

After I installed the necessary package on Ubuntu with

sudo apt-get install libxtst-dev

Then fakeKey.c compiled and worked without problems.

Note: I started a project on Github for this, anyone is welcome to fork, help, etc.

Answer 3

I finally have a solution. I believe UrT loads OpenGL on its own so that things such as wallhacks, etc are not possible. Then the best remaining option is taking X screenshots. This worked quite fast, even from a scripting language like Python. The following code takes successive screenshots and displays them as an animation through OpenCV. You need to start UrT in minimized mode of course. The rest of the details are in my project.

import gtk.gdk
import PIL
from opencv.cv import *
from opencv.highgui import *
from opencv.adaptors import PIL2Ipl

w = gtk.gdk.get_default_root_window()
sz = w.get_size()
print "The size of the window is %d x %d" % sz

size_x = 600
size_y = 400
start_x = 0
start_y = 100
end_x = start_x+size_x
end_y = start_y+size_y
box = (start_x, start_y, start_x+size_x, start_y+size_y)

while True:
    pb = gtk.gdk.Pixbuf(gtk.gdk.COLORSPACE_RGB,False,8,sz[0],sz[1])
    pb = pb.get_from_drawable(w,w.get_colormap(),0,0,0,0,sz[0],sz[1])
    width,height = pb.get_width(),pb.get_height()
    im = PIL.Image.fromstring("RGB",(width,height),pb.get_pixels())
    im = im.crop(box)
    cv_img = PIL2Ipl(im)
    cvNamedWindow("fps")
    cvShowImage("fps", cv_img)
    cvWaitKey(30) 

Also, for sending keys to the game, the method above did not work, I had to use xdotool in order to send forward walk key to UrT,

xdotool search --name ioUrbanTerror  windowactivate keydown W