Screen sharing in the ImageView of JavaFX

Question

I am trying to build a JavaFX application, where I have a button named "Start" and an ImageView. With the robot class of JavaFX-12, I am trying to take a screenshot of the laptop screen when the button is clicked and show the images one by one during the runtime in the ImageView. My problem is that the JavaFX window does not respond and the program crashes (probably). Even putting the thread into sleep does not seem to work. I assume that it isn't working as I have not set any fps rule, but how can I do that? At the moment, I am creating writable images, converting them into a separate image with a number, saving them, and again reusing them. My goal is to create a screen sharing of the same laptop in the image view. I know that's difficult. I'm new to the JavaFx robot class (not he awt one). Any help is appreciated.

P.S.: The images are properly formed in the directory.

package sample;

import javafx.application.Application;
import javafx.embed.swing.SwingFXUtils;
import javafx.fxml.FXMLLoader;
import javafx.geometry.Rectangle2D;
import javafx.scene.Parent;
import javafx.scene.Scene;
import javafx.scene.control.Button;
import javafx.scene.image.Image;
import javafx.scene.image.ImageView;
import javafx.scene.image.WritableImage;
import javafx.scene.layout.VBox;
import javafx.scene.robot.Robot;
import javafx.stage.Screen;
import javafx.stage.Stage;

import javax.imageio.ImageIO;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;

public class Main extends Application {

    @Override
    public void start(Stage primaryStage) throws Exception{
        Parent root = FXMLLoader.load(getClass().getResource("sample.fxml"));
        primaryStage.setTitle("Hello World");

        ImageView iv = new ImageView();
        iv.setFitWidth(100);
        iv.setFitHeight(100);
        Button b = new Button("Start");
        VBox v = new VBox(10);
        v.getChildren().addAll(b,iv);
        b.setOnAction(event -> {
            Robot r = new Robot();
            WritableImage wi = new WritableImage(300,300);
            WritableImage i;
            Rectangle2D rect = Screen.getPrimary().getVisualBounds();
            while(true){
                i = r.getScreenCapture(wi,rect);
                try {
                    ImageIO.write(SwingFXUtils.fromFXImage(i,null),"png",new File("F:/Pic/pic" + x + ".png"));
                    iv.setImage(new Image(new FileInputStream("F:/Pic/pic" + x + ".png")));
                    //Thread.sleep(500);
                    //iv.setImage(null);
                } catch (Exception e) {
                    System.out.println(e);
                }
            }
        });
        primaryStage.setScene(new Scene(v, 500, 500));
        primaryStage.show();
    }


    public static void main(String[] args) {
        launch(args);
    }
}

Answer 1

JavaFX is, like most UI frameworks, single-threaded. You must never block (e.g. sleep) or otherwise monopolize (e.g. while (true)) the JavaFX Application Thread. That thread is responsible for everything related to the UI and if it's not free to do its job then the UI will become unresponsive. Note that a render pass cannot happen until the FX thread returns from whatever it's doing, so setting the image of an ImageView in a loop will have no visible effect until some time after the loop terminates.

Also, a full-throttle while loop will attempt to get a screen capture as fast as the CPU can execute said loop. That is likely to be much faster than the rate at which your UI refreshes and is thus a waste of resources. The rate of screen captures should not exceed the frame rate.

If you need to loop on the FX thread and/or be constrained by the (JavaFX's) frame rate then use the javafx.animation API. In your case, an AnimationTimer seems apt. Here's an example which continuously screenshots the primary screen:

import javafx.animation.AnimationTimer;
import javafx.application.Application;
import javafx.geometry.Rectangle2D;
import javafx.scene.Scene;
import javafx.scene.image.ImageView;
import javafx.scene.image.WritableImage;
import javafx.scene.layout.StackPane;
import javafx.scene.robot.Robot;
import javafx.stage.Screen;
import javafx.stage.Stage;

public class App extends Application {

  @Override
  public void start(Stage primaryStage) {
    ImageView view = new ImageView();
    view.setFitWidth(720);
    view.setFitHeight(480);

    // Keep a reference to the AnimationTimer instance if
    // you want to be able to start and stop it at will
    new AnimationTimer() {

      final Robot robot = new Robot();
      final Rectangle2D bounds = Screen.getPrimary().getBounds();

      @Override
      public void handle(long now) {
        WritableImage oldImage = (WritableImage) view.getImage();
        WritableImage newImage = robot.getScreenCapture(oldImage, bounds);
        if (oldImage != newImage) {
          view.setImage(newImage);
        }
      }
    }.start();

    primaryStage.setScene(new Scene(new StackPane(view)));
    primaryStage.show();
  }
}

Some notes:

• The AnimationTimer#handle(long) method is invoked once per frame. JavaFX tries to target 60 frames-per-second. If that's too fast (the above lags somewhat on my computer) then you can use the method's argument, which is the timestamp of the current frame in nanoseconds, to throttle the rate of screen captures. You could also look into using a Timeline or PauseTransition instead of an AnimationTimer. See JavaFX periodic background task for more information.
• The above gives a Droste Effect (I think that's the term?) since the screen capture is displayed on the screen which is being captured.

My example does not include saving each image to a file. You seem to already understand how to do that so you should be able to easily adapt the code. It'd probably be a good idea, however, to move the I/O to a background thread. Unfortunately, that will likely require using different WritableImage instances for each capture to avoid the image being mutated by the FX thread while the background thread reads it. It may also require some tuning or dropped images; I'm not sure how well the I/O will keep up with the influx of screen captures (i.e. standard producer-consumer problems).

---

As an aside, your question explains you're attempting to share the entire screen. If that's the case then continue using Robot. However, if you only need to share something from the JavaFX application itself then consider using Node#snapshot(SnapshotParameters,WritableImage) or Scene#snapshot(WritableImage) (assuming you can't just send model data instead of images).