JavaFX bug with fast setImage call

Question

I'm using Java 8 with JavaFX from Oracle, and I want to create a desktop remote app where the client continuously sends images as byte[] over sockets to the server. The server receives these images and displays them as an image in an ImageView. However, after a few seconds, the program starts crashing, and I get this exception: java.lang.NullPointerException at com.sun.prism.d3d.D3DTexture.getContext(D3DTexture.java:84) at com.sun.prism.d3d.D3DTexture.update(D3DTexture.java:207) at com.sun.prism.d3d.D3DTexture.update(D3DTexture.java:151)...

After a lot of googling, I found out that by enabling "-Dprism.verbose=true," I can get more information about the problem. I discovered that the console keeps outputting: Growing pool D3D Vram Pool target to 524,167,168 Growing pool D3D Vram Pool target to 524,167,168 Growing pool D3D Vram Pool target to 535,226,368...

always with a larger number. This output becomes even faster if, for example, I resize the window in which the ImageView is located, even if the ImageView itself doesn't change in size, which doesn't make sense to me.

After some testing, I added a "System.gc();" call after each "setImage();" method call of the ImageView, and suddenly it works without any issues. There is no more "growing pool" log, and the program no longer crashes with the aforementioned NullPointerException. However, continuously calling "System.gc();" is not a solution for me as it severely impacts the performance of the program, and it's generally recommended to avoid it. Besides, Java should handle this automatically.

Another thing that worked was setting "-Dprism.order=sw -Xmx1024m" in the VM options. However, this caused the entire UI to be relatively buggy, so it's not a solution for me either.

To trigger the garbage collector, I also tried calling "setImage(null);" every time, hoping that the garbage collector would "clean up." Unfortunately, this didn't help at all.

I have created a code example that indirectly reproduces the same issue.

Code-Example:

package example;

import javafx.application.Application;
import javafx.application.Platform;
import javafx.geometry.Pos;
import javafx.scene.Scene;
import javafx.scene.control.Button;
import javafx.scene.image.Image;
import javafx.scene.image.ImageView;
import javafx.scene.layout.VBox;
import javafx.stage.Stage;

import javax.imageio.ImageIO;
import java.awt.*;
import java.awt.image.BufferedImage;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;

public class Main extends Application {

    private ImageView imageView;
    private Button button;


    @Override
    public void start(Stage primaryStage) throws Exception {
        primaryStage.setTitle("ImageView Example");

        button = new Button("Start");
        button.setOnAction(e -> start());

        imageView = new ImageView();
        imageView.setFitWidth(720);
        imageView.setFitHeight(480);

        VBox root = new VBox();
        root.setAlignment(Pos.CENTER);
        root.getChildren().addAll(imageView, button);

        Scene scene = new Scene(root, 720, 510);

        primaryStage.setScene(scene);
        primaryStage.show();
    }

    private void start() {
        button.setDisable(true);
        new Thread(() -> {
            while (true) {
                Platform.runLater(() -> {
                    try {
                        imageView.setImage(new Image(new ByteArrayInputStream(robotScreenshot("jpg"))));
//System.gc(); <- would fix the "bug" but is not a option for me
                    } catch (Exception e) {
                        e.printStackTrace();
                        System.exit(-1);
                    }
                });

                try {
                    Thread.sleep(100);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                    System.exit(-1);
                }
            }
        }).start();
    }

    private static final GraphicsDevice graphicDevice = GraphicsEnvironment.getLocalGraphicsEnvironment().getScreenDevices()[0];

    public static byte[] robotScreenshot(String format) throws Exception {
        Rectangle rectangle = graphicDevice.getDefaultConfiguration().getBounds();
        BufferedImage bufferedImage = new Robot().createScreenCapture(rectangle);

        ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
        ImageIO.write(bufferedImage, format.toLowerCase(), byteArrayOutputStream);
        return byteArrayOutputStream.toByteArray();
    }

    public static void main(String[] args) {
        launch(args);
    }
}

I tried various approaches to resolve the issue with the growing D3D VRAM pool in JavaFX. Initially, I enabled the "-Dprism.verbose=true" option to gather more information about the problem. My expectation was to identify the cause of the crash and find a solution. However, I discovered that the console continuously displayed messages indicating the growth of the VRAM pool with increasingly larger numbers.

To address the issue, I experimented with using "System.gc();" after each "setImage();" method call in the ImageView. Surprisingly, this temporarily resolved the problem. However, I realized that constantly triggering the garbage collector negatively impacted the program's performance, which was not a viable long-term solution.

I also attempted setting the VM options to "-Dprism.order=sw -Xmx1024m," but this led to UI bugs, making it unsuitable as a resolution.

Additionally, I tried calling "setImage(null);" instead of "System.gc();" to trigger the garbage collector, but it had no effect on the issue.

In summary, despite my attempts to mitigate the growing VRAM pool problem, I have yet to find a satisfactory solution that maintains the program's performance while avoiding the NullPointerException and excessive memory usage.

Answer 1

It's not possible for me to test this, but the code you have is flawed because this line puts a task into the Platform JavaFXUI thread every 100ms, but you have no idea whether the Platform task finished before you queue up another task:

Platform.runLater(() -> { ... });
Thread.sleep(100);

If the action takes over 100ms you will start to get a lot of backed up tasks in the UI thread, and then JavaFX will not have a chance to repaint the image properly after each new setImage making the use of more calls to Platform.runLater pointless because you never see the newer image on screen before your code tries to replace it. The app will become jerky / unresponsive when this starts to occur, and you will get memory issues with a large backlog.

A simple way to fix this is to change the while loop to wait for each item in the runLater call, one way would be with CountDownLatch:

while (true) {
    CountDownLatch done = new CountDownLatch(1);
    Platform.runLater(() -> {
        try {
            imageView.setImage(new Image(new ByteArrayInputStream(robotScreenshot("jpg"))));
        } catch (Exception e) {
            e.printStackTrace();
            System.exit(-1);
        }
        done.countDown();
    });

    try {
        done.await();
        Thread.sleep(100);
    } catch (InterruptedException e) {
        e.printStackTrace();
        System.exit(-1);
    }
}

In addition, it is better to have as much processing outside of the JavaFX UI thread so try moving the screen grab out of the Platform.runLater call and diff byte[] with previous and only submit new task for setImage when the new image is changed.