Faster alternative to java.awt.Robot.createScreenCapture?

Question

I'm making a program that requires at least 24 screenshots per second to be captured. Currently with the code below I'm only getting 1 per every ~94 milliseconds, so about 10 per second.

I'd prefer not to use any 3rd party libraries because I'm trying to keep it as small as possible, but if I'd get significant performance increase I'd be willing to. I'm also trying to keep this platform independent, but again, if it would be a really significant performance increase I'd be willing to keep it limited to Windows.

edit: I've now tried it two different ways as well; using a snippet found on oracles website and the one pointed out in the comments below. All three took around the same time, 2.1-2.2 million nanoseconds which is pretty damn inefficient.

public abstract class Benchmark {

    private final int iterations;

    public Benchmark(int iterations) {
        this.iterations = iterations;
    }

    public abstract void logic();

    public void start() {
        long start = System.nanoTime();
        for (int iteration = 0; iteration < iterations; iteration++) {
            long iterationStart = System.nanoTime();
            logic();
            System.out.println("iteration: " + iteration + " took: " + (System.nanoTime() - iterationStart) + " nanoseconds.");
        }
        long total = (System.nanoTime() - start);
        System.out.println(iterations + " iterations took: " + total + " nanoseconds.  Average iteration was: " + (total / iterations));
    }
}

_

import java.awt.AWTException;
import java.awt.Rectangle;
import java.awt.Robot;
import java.awt.Toolkit;

public class RobotBenchmark extends Benchmark {

    private final Robot robot;
    private final Rectangle screen;

    public static void main(String[] args) {
        Benchmark benchmark;
        try {
            benchmark = new RobotBenchmark(24);
            benchmark.start();
        } catch (AWTException e) {
            e.printStackTrace();
        }
    }

    public RobotBenchmark(int iterations) throws AWTException {
        super(iterations);
        robot = new Robot();
        screen = new Rectangle(Toolkit.getDefaultToolkit().getScreenSize());
    }

    @Override
    public void logic() {
        robot.createScreenCapture(screen);
    }

}

_

import java.awt.AWTException;
import java.awt.GraphicsDevice;
import java.awt.HeadlessException;
import java.awt.Rectangle;

public class DirectRobotBenchmark extends Benchmark {

    private final GraphicsDevice device;
    private final Rectangle screenRectangle;
    private final DirectRobot robot;

    private int[] screen;

    public static void main(String[] args) {
        Benchmark benchmark;
        try {
            benchmark = new DirectRobotBenchmark(24);
            benchmark.start();
        } catch (HeadlessException | AWTException e) {
            e.printStackTrace();
        }
    }

    public DirectRobotBenchmark(int iterations) throws HeadlessException, AWTException {
        super(iterations);
        device = DirectRobot.getDefaultScreenDevice();
        screenRectangle = new Rectangle(1920, 1080);
        robot = new DirectRobot(device);
        screen = new int[screenRectangle.width * screenRectangle.height];
    }

    @Override
    public void logic() {
        screen = robot.getRGBPixels(screenRectangle);
    }
}

_

import java.awt.AWTException;
import java.awt.GraphicsEnvironment;
import java.awt.Rectangle;
import java.awt.Robot;
import java.awt.Toolkit;
import java.awt.peer.RobotPeer;

import sun.awt.SunToolkit;

@SuppressWarnings("restriction")
public class RobotPeerBenchmark extends Benchmark {

    private final SunToolkit toolkit;
    private final RobotPeer peer;
    private final Rectangle screenRectangle;

    private int[] screen;

    public static void main(String[] args) {
        try {
            Benchmark robotPeerBenchmark = new RobotPeerBenchmark(24);
            robotPeerBenchmark.start();
        } catch (AWTException e) {
            e.printStackTrace();
        }
    }

    public RobotPeerBenchmark(int iterations) throws AWTException {
        super(iterations);
        toolkit = (SunToolkit) Toolkit.getDefaultToolkit();
        peer = toolkit.createRobot(new Robot(), GraphicsEnvironment.getLocalGraphicsEnvironment().getDefaultScreenDevice());
        screenRectangle = new Rectangle(toolkit.getScreenSize());
        screen = new int[screenRectangle.width * screenRectangle.height];
    }

    @Override
    public void logic() {
        screen = peer.getRGBPixels(screenRectangle);
    }
}

Answer 1

The only way to do this will be through JNI or possibly JNA. I did some benchmarking and native screen capture API and it was able to sustain about 45 FPS vs the Robots 8 FPS. I might be starting on a JNI project to solve this issue in the near future. I will update this post with the project URL if that goes forward.

Answer 2

I've currently set up a working example of using VLCJ and then using the DirectMediaPlayer (https://github.com/caprica/vlcj/blob/master/src/test/java/uk/co/caprica/vlcj/test/direct/DirectTestPlayer.java) to get the BufferedImage.

The JFrame isn't necessary for it to work correctly.

I know this is an old question, but this is still a problem today, so I thought I'd share.

VLCJ are Java bindings for LibVLC.

Example code:

    private BufferedImage image;
    private MediaPlayerFactory factory;
    private DirectMediaPlayer mediaPlayer;

    public void start() {

            image = GraphicsEnvironment.getLocalGraphicsEnvironment().getDefaultScreenDevice().getDefaultConfiguration().createCompatibleImage(width, height);
            image.setAccelerationPriority(1.0f);

            String mrl = "screen://";
            String[] options = {
                    ":screen-fps=30",
                    ":live-caching=0",
                    ":screen-width=1920",
                    ":screen-height=1080",
                    ":screen-left=0",
                    ":screen-top=0"
            };
            factory = new MediaPlayerFactory();
            mediaPlayer = factory.newDirectMediaPlayer(new TestBufferFormatCallback(), new TestRenderCallback());
            mediaPlayer.playMedia(mrl, options);
    }

    // Callbacks are required.
    private final class TestRenderCallback extends RenderCallbackAdapter {

        public TestRenderCallback() {
            super(((DataBufferInt) image.getRaster().getDataBuffer()).getData());
        }

        @Override
        public void onDisplay(DirectMediaPlayer mediaPlayer, int[] data) {
            // The image data could be manipulated here...

            /* RGB to GRAYScale conversion example */
//            for(int i=0; i < data.length; i++){
//                int argb = data[i];
//                int b = (argb & 0xFF);
//                int g = ((argb >> 8 ) & 0xFF);
//                int r = ((argb >> 16 ) & 0xFF);
//                int grey = (r + g + b + g) >> 2 ; //performance optimized - not real grey!
//                data[i] = (grey << 16) + (grey << 8) + grey;
//            }
//            imagePane.repaint();
        }
    }

    private final class TestBufferFormatCallback implements BufferFormatCallback {

        @Override
        public BufferFormat getBufferFormat(int sourceWidth, int sourceHeight) {
            return new RV32BufferFormat(width, height);
        }

    }

Answer 3

Frame rates around 30FPS with pure Java can be achieved using mutiple java.awt.Robot in parallel. An example implementation can be found on GitHub