Capturing Display / Monitor Images, Sending Keyboard Input on Linux

I need to process images sent to my laptop's video display, and I need to send keyboard input to my Linux system, using a C++ or shell program.

My goal is to process images that are part of an FPS game, then taking action inside that game (hence the keyboard input) based on these images. Instead of trying to understand (if it's even possible) how to interface to game X or Y, using some API, I figured this is the quickest way to interface to any game, hijacking Linux input and output somehow.

Is there any way to do this without any kernel, or device driver hacking? I used recordmydesktop to record my desktop as a video before, I guess I could hack its code and try to reverse-engineer something from that. Any other ideas? I am on Ubuntu 11.

You don't need to do anything as low level as the kernel or device drivers to do this.

You can use the XTest X11 extension to fake input events programatically, for example (from this posting, there's another example for keyboard).

#include <X11/extensions/XTest.h>  
#include <unistd.h>  

int main ()  
{  
  Display *dpy = NULL;  
  XEvent event;  

  dpy = XOpenDisplay (NULL);  

  /* Get the current pointer position */  
  XQueryPointer (dpy, RootWindow (dpy, 0),  
        &event.xbutton.root, &event.xbutton.window,  
        &event.xbutton.x_root, &event.xbutton.y_root,  
        &event.xbutton.x, &event.xbutton.y,  
        &event.xbutton.state);  

  /* Fake the pointer movement to new relative position */  
  XTestFakeMotionEvent (dpy, 0, event.xbutton.x + 100,  
        event.xbutton.y + 50, CurrentTime);  
  XSync(dpy, 0);  
  XCloseDisplay (dpy);  
  return 0;  
}   

To capture the images the simplest way is to use function interposition (via LD_PRELOAD) to "intercept" calls to glXSwapBuffers, which will be called once each frame is drawn. From there you can copy the contents of the framebuffer, using glReadPixels and do with it what you want.

E.g. untested outline for intercepting OpenGL frames:

// Function pointer to the *real* glXSwapBuffers
static void (*glx_fptr)(Display*, GLXDrawable) = NULL;

// Make sure init gets called when the shared object is loaded. GCC specific.
static void init(void)  __attribute__((constructor));

static void init(void) {
    dlerror();
    // find the real glXSwapBuffers
    glx_fptr = dlsym(RTLD_NEXT, "glXSwapBuffers");
    if (NULL == glx_fptr)
        fprintf(stderr, "[glvidcap] %s\n", dlerror());
}

void glXSwapBuffers(Display *dpy, GLXDrawable drawable) {
    unsigned int w = 0;
    unsigned int h = 0;
    static int x,y;
    static Window win;
    static unsigned int border,depth;
    // Find the window size. (You could skip this and make it all static if you
    // Trust the window not to change size
    XGetGeometry(dpy, drawable, &win, &x, &y, &w, &h, &border, &depth);

    // Assuming frame is some memory you want the frame dumped to:
    glReadPixels(0,0,w,h,GL_BGR,GL_UNSIGNED_BYTE, frame);

    // Call the real function:
    assert(glx_fptr);
    glx_fptr(dpy, drawable);
}

Then you want to compile this as a shared object and LD_PRELOAD that shared object before running whatever game you're looking at.

If it happens to be an SDL application instead you can intercept the calls to SDL_Flip orSDL_UpdateRect as appropriate.

Posted in: Embedded System