JavaFX Mouse Move or Drag events "skip" pixels - can this be prevented?

Question

Below example demonstrates "gaps" in drawing due to mouse events arriving too slowly (I presume). I'd like to capture all "mouseovered" pixels, without drawing lines from the last pixel written to the current one. Doing this assumes the mouse moved in a straight line, and that may not be the case for very fast movements.

Do we have any workarounds to get the missing events?

import javafx.application.Application;
import javafx.scene.Scene;
import javafx.scene.canvas.Canvas;
import javafx.scene.canvas.GraphicsContext;
import javafx.scene.layout.AnchorPane;
import javafx.scene.paint.Color;
import javafx.stage.Stage;
import java.awt.*;

public class DrawTest extends Application {
    
    private static Point point;
    
    @Override
    public void start(Stage stage) {
        Canvas canvas = new Canvas(800, 600);
        AnchorPane anchorPane = new AnchorPane(canvas);
        GraphicsContext gc = canvas.getGraphicsContext2D();
        
        point = MouseInfo.getPointerInfo().getLocation();
        
        gc.setFill(Color.WHITE);
        gc.fillRect(0, 0, 800, 600);
        
        canvas.setOnMouseDragged((event) -> {
            gc.getPixelWriter().setColor((int) event.getX(), (int) event.getY(), Color.BLACK);
            System.out.println("X " + event.getX() + " Y " + event.getY());
        });
        
        stage.setScene(new Scene(anchorPane));
        stage.show();
    }
    
    public static void main(String[] args) {
        Application.launch(DrawTest.class, args);
    }
}

This is an example of some of the event coordinates that get rendered with large gaps:

X 189.0 Y 248.0
X 193.0 Y 248.0
X 199.0 Y 248.0
X 204.0 Y 247.0
X 211.0 Y 244.0
X 225.0 Y 240.0

I've tried using this flag but it didn't help (on Linux): -Djavafx.animation.fullspeed=true

I'd be willing to try limited Swing options or maybe even more "native" workarounds, but obviously would prefer a JFX resolution or at least an explanation why JFX doesn't do this. Would need a Linux compatible solution, Windows too I guess. :P

This question is very similar, but the solutions proposed are still of a "connect the dots" nature and not exactly suitable if you need to accurately record or react to the pixel-by-pixel mouse path, so please don't close this as a duplicate - it's not.

How to draw a continuous line with mouse on JavaFX canvas?

Edit:

Well... it turns out I guess JavaFX is vindicated after all. After adding the following bit in the middle of the example above, if you'll forgive the expected thread racing, it does look to me like JFX is matching AWT event for event with this crude polling method. Probably the best that can be done is to fill in the gaps somehow.

    Thread t = new Thread(() -> {
        while (true) {
            Point p = MouseInfo.getPointerInfo().getLocation();
            if (point.x != p.x || point.y != p.y) {
                point = p;
                System.out.println(point);
            }
            try {
                Thread.sleep(1);
            } catch (InterruptedException e) {
            }
        }
    });
    t.setDaemon(true);
    t.start();

Output:

java.awt.Point[x=2890,y=301]
X 2890.0 Y 301.0
java.awt.Point[x=2884,y=303]
X 2884.0 Y 303.0
java.awt.Point[x=2870,y=308]
X 2870.0 Y 308.0
X 2848.0 Y 314.0
java.awt.Point[x=2848,y=314]
java.awt.Point[x=2822,y=322]
X 2822.0 Y 322.0
X 2790.0 Y 330.0
java.awt.Point[x=2790,y=330]
java.awt.Point[x=2760,y=338]
X 2760.0 Y 338.0
X 2726.0 Y 344.0
java.awt.Point[x=2726,y=344]
X 2694.0 Y 350.0
java.awt.Point[x=2694,y=350]
X 2668.0 Y 356.0
java.awt.Point[x=2668,y=356]
java.awt.Point[x=2646,y=360]
X 2646.0 Y 360.0
java.awt.Point[x=2631,y=366]
X 2631.0 Y 366.0
X 2623.0 Y 367.0
java.awt.Point[x=2623,y=367]
X 2620.0 Y 369.0
java.awt.Point[x=2620,y=369]
java.awt.Point[x=2621,y=369]
X 2621.0 Y 369.0
X 2622.0 Y 368.0
java.awt.Point[x=2622,y=368]

I have definitely encountered the issue you are speaking of. My belief on this (based on nothing but my own intuition on computers) is that Swing/Java is sending these events as quickly as it gets them from the OS. So then the question becomes how do you get the OS to send more rapid events: I am not sure there is an answer to that. — ControlAltDel, Dec 03 '21 at 23:48
The short answer is, no, you can't "fix" this. This is how the OS interrupts the movement of the cursor across the screen — MadProgrammer, Dec 03 '21 at 23:51
You can with java SE draw Bezier curves through the points and such which _looks_ more realistic. In a way it also a data reduction; you not really want to store 1000 pixel positions. — Joop Eggen, Dec 03 '21 at 23:52
@Joop Eggen I'm not actually storing the coordinates, but I'd like to use these events to modify buffered image data as accurate as possible. If this is truly as accurate as it gets and I just need to interpolate between the data points, that's that, but the point of asking this is so I can know that I'm not being any less accurate than every other (non Java) application out there. — Manius, Dec 04 '21 at 00:13
One can set the mouse cursor's speed to get an idea of the possible data sampling of a mouse trajectory. In my experience, the larger the gap, the faster the mouse, the more a straight uncurved line. — Joop Eggen, Dec 04 '21 at 00:23
I guess I'll put together at least some kind of AWT comparison. If AWT can do no better, that might be convincing enough. I don't want to assume this is the best the OS can do without at least a little proof. — Manius, Dec 04 '21 at 00:29
@JoopEggen Obviously, you can draw Bézier curves with JavaFX, as well as with Swing/AWT, so there’s no need to suggest mixing toolkits here. — James_D, Dec 04 '21 at 01:33
I didn’t downvote (yet) but my guess is it’s because this kind of question really requires a [mre] to make it reasonably straightforward to answer. — James_D, Dec 04 '21 at 01:38
[*Line painting in painting program JavaFX*](https://stackoverflow.com/q/69348082/230513) examines tracking the _endpoints_ of a continuous line as it's being drawn, storing the result in a `List`. — trashgod, Dec 04 '21 at 01:48
I suppose this point is also worth making. The JVM is going to be limited as to how quickly it detects mouse events by how quickly the native system detects mouse events. So it doesn’t really matter if the JVM detect rate and the native detection rate are the same, or if not if there’s a way to bypass the difference: the bottom line is that the native system imposes some limit which can always be exceeded by how fast the user moves the mouse. So there will always be the possibility of “skipped pixels” and you need to code for that possibility. Joop’s Bezier suggestion is likely optimal. — James_D, Dec 04 '21 at 01:51
[This](https://math.hws.edu/javanotes/source/chapter6/SimplePaint.java) may be slightly better curve-wise than the answer I posted on the other post you linked. — SedJ601, Dec 04 '21 at 03:04
@James_D I thought I had a "minex" (coining that term right now) basically but I'll make it a little more copy-paste friendly and complete now. Also I don't really mind coding around the gaps if a good result can be reached. For me this is more about being certain that there isn't some bug or performance issue with JFX specifically. I can't remember where I've seen it, but I vaguely remember one reading a discussion about an alleged JavaFX bug causing "missing" mouse events. — Manius, Dec 04 '21 at 06:49
Found an old browser tab with the JFX ticket that made me suspicious, if anyone's curious. Slightly different issue. https://bugs.openjdk.java.net/browse/JDK-8087922 — Manius, Dec 04 '21 at 07:30

Manius · Answer 1 · 2021-12-04T19:16:39.757

Since I've asked if 1) this can be prevented (presumably not) and 2) for workarounds, I feel like I should at least supply the basis for the workaround to this that I'll probably be settling on.

Since a "connect the dots" method is the only way, but I don't want to draw directly to the Canvas, I can't use the GraphicsContext line drawing calls in the other linked answer, which appear to invoke external native code for drawing lines and such. So we'll have to implement our own line drawing algorithm (maybe there's some library out there to do this?)...

This early draft workaround involves drawing an integer color to a generic buffer in memory (this could be a WritableImage in JFX or just a one dimensional int[] buffer etc.) so that the written values can easily be read back later, rather than drawing directly to a Canvas.

/*
 * Bresenham's algorithm
 */
public static void drawLine(RasterLayer layer, int x0, int y0, int x1, int y1, int color) {
    if (Math.abs(y1 - y0) < Math.abs(x1 - x0)) {
        if (x0 > x1) {
            plotLineLow(layer, x1, y1, x0, y0, color);
        } else {
            plotLineLow(layer, x0, y0, x1, y1, color);
        }
    } else {
        if (y0 > y1) {
            plotLineHigh(layer, x1, y1, x0, y0, color);
        } else {
            plotLineHigh(layer, x0, y0, x1, y1, color);
        }
    }
}

private static void plotLineLow(RasterLayer layer, int x0, int y0, int x1, int y1, int color) {
    int dx = x1 - x0;
    int dy = y1 - y0;
    int yi = 1;
    if (dy < 0) {
        yi = -1;
        dy = -dy;
    }
    int D = (2 * dy) - dx;
    int y = y0;
    
    for (int x = x0; x < x1; x++) {
        layer.setPixel(x, y, color);
        
        if (D > 0) {
            y = y + yi;
            D = D + (2 * (dy - dx));
        } else {
            D = D + 2 * dy;
        }
    }
}

private static void plotLineHigh(RasterLayer layer, int x0, int y0, int x1, int y1, int color) {
    int dx = x1 - x0;
    int dy = y1 - y0;
    int xi = 1;
    if (dx < 0) {
        xi = -1;
        dx = -dx;
        y1++;
    }
    int D = (2 * dx) - dy;
    int x = x0;
    
    for (int y = y0; y < y1; y++) {
        layer.setPixel(x, y, color);
        
        if (D > 0) {
            x = x + xi;
            D = D + (2 * (dx - dy));
        } else {
            D = D + 2 * dx;
        }
    }
}

To make the drawn line thicker I suppose I'll just have to draw additional adjacent lines by offsetting the x or y coordinates for the additional lines, selected depending on the slope of the line. (Draw additional adjacent lines with offset y values for more horizontal lines, and offset x values for more vertical lines.) Testing so far this seems to work reasonably well. Obviously you could also get fancy with anti-aliased lines and such...

JavaFX Mouse Move or Drag events "skip" pixels - can this be prevented?

1 Answers1