2

I'm developing a program that will be able to draw the audio waveform using the streamed amplitude data from a microphone or Line-in. The way I thought to do this would be to draw each point from the sample data at a rate equal to the sample rate, going 1 step over in the x direction with each point drawn. Therefore I would need to update the JavaFx Application thread around 44100 times a second to draw each point. Before I start doing this I wanted to test my idea out by just drawing a straight line and only updating each point every half a second. I'm using the Timeline class to do this. My code looks like this: 

public class JavaFxPractice extends Application { 
  private int xValue = 50;

  @Override 
  public void start(Stage primaryStage) {      
    Pane pane = new Pane();

    EventHandler<ActionEvent> eventHandler = e -> {
      xValue++;
      Line point = new Line(xValue,50,xValue,50);
      pane.getChildren().add(point);
    };

    Timeline animation = new Timeline(new KeyFrame(Duration.millis((500)), eventHandler)); 
    animation.setCycleCount(500);
    animation.play();

    Scene scene = new Scene(pane, 600, 500);
    primaryStage.setTitle("Streaming Test");
    primaryStage.setScene(scene);
    primaryStage.show();  
  } 
}

However every time I do this my program becomes unresponsive and I have to force close it. I noticed that if I do the same thing but instead make Text blink on and off, it works perfectly fine. Is there a reason Lines aren't able to be drawn using the Timeline class? Does it put too much of a load on the thread? If so, what ways can I go around solving my idea. I just want to be able to draw a waveform in real time, updating at around 44,100 times a second.

Ryan Foster
  • 103
  • 9
  • 2
    I can't reproduce the freezing problem you describe when executing the provided example. Though keep in mind your code ends up adding 500 `Line` objects to the UI; it would be better to simply update the same `Line` to end at the new x-coordinate. As for updating the UI at around 44,100 times a second, if that entails posting a new runnable with `Platform.runLater` for each update then note that will flood the FX thread and can (probably will) lead to a frozen UI. On top of that, JavaFX typically runs at 60 frames per second and trying to update faster than that will have no visible effect. – Slaw Apr 09 '20 at 03:07
  • 2
    Your current code runs fine for me as well - but I'd reiterate the concerns expressed above for the actual application you are proposing. Simply adding that many nodes to a scene graph will likely cause performance problems. You probably want to consider a strategy such as updating something off-screen in your background thread, and using an `AnimationTimer` to grab the latest update each time the FX scene graph refreshes. Using a `Canvas` or `WritableImage` may be a better solution than add so many nodes, unless you can make this work by updating existing nodes. – James_D Apr 09 '20 at 03:18
  • 1
    Related recent question (with other useful links in the comments): https://stackoverflow.com/q/60668758/2189127 – James_D Apr 09 '20 at 03:21
  • @Slaw Yes that would make more sense, but you have to remember this is just practice. In reality I'm trying to draw a waveform that won't be a straight line. Hence why I want to draw points. But if javafx can only update at 60 frames per second. How do you supposed I draw out a waveform that needs to be refreshed at roughly 41000 times a second? – Ryan Foster Apr 09 '20 at 03:40
  • 3
    I understand this is just an example, but the concept of reusing existing nodes and using as few nodes as possible may be transferable to the real code. And regarding the update rate of your waveform, it does not _need_ to be updated 41,000 times a second. For one, no monitor in the world has a refresh rate anywhere near 41,000Hz, so even if the software/GPU was capable of/allowed that refresh rate the monitor wouldn't be able to show it. For another, no human can perceive 41,000fps. – Slaw Apr 09 '20 at 04:37
  • I've been reading what you guys have said and they have been very helpful. If I understand correctly, are you saying I should use a canvas in a background thread to draw my waveform which gets updated around 60 times a second. Each canvas would be for a a single frame and then use an AnimationTimer to grab each updated canvas? If that's the case this would mean that my canvas would also only be updated 60 times a second. I have around 44,100 samples that are coming per second. Is the canvas capable of drawing out 735 points (44,100/ 60) every frame or will that also lead to performance issues? – Ryan Foster Apr 09 '20 at 04:40
  • 1
    The idea behind using an `AnimationTimer`, as suggested by @James (check out the link he posted), is that it'd poll the latest data and update the UI. It'd do that as fast as the UI can handle it (e.g. 60fps). While the timer is doing that, your background thread can be updating the model state as fast as you want it to. But updates between polls (i.e. between frames) would be _dropped_. Basically, the `AnimationTimer` is sampling the incoming data at 60Hz. – Slaw Apr 09 '20 at 04:43
  • @Slaw Ahhh gotcha. So you are saying It's OK to skip some chunks of data in between frames since the user won't be able to visually see this. Also would you recommend using a canvas instead of nodes just as James_D noted? Last question sorry. :) – Ryan Foster Apr 09 '20 at 04:44
  • @Slaw Awesome, you are such a huge help. Would the waveform not appear to be jumping all around? For example, the audio is being streamed at 44100hz. Im polling it every 60hz. That means in order to display 44100 samples every second I would need to display 735 points in each frame, in the proper order. Remember It's audio data so the waveform needs to match the rate of the song so its not out of sync with the audio. But would this make the waveform look like its continuously flowing? Or would it appear to be jumping when it transitions to a different mapping of points for a different frame – Ryan Foster Apr 09 '20 at 05:19
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/211263/discussion-between-ryan-foster-and-slaw). – Ryan Foster Apr 09 '20 at 05:22

1 Answers1

0

I recommend AnimationTimer for any ongoing animation. It attempts to update as close to 60 fps as it can. (I just read some comments following the first one, recommending AnimationTimer. These folks are right. IDK why they didn't post that as an answer themselves.)

The question then becomes what to display. If I were attacking this problem, here is what I would try:

  1. make an array to hold positions, one bucket per pixel, for the size of the display.
  2. make a function that draws the data from such an array, that can be called by the animation timer
  3. make a concurrent-safe queue to hold these arrays (e.g., ConcurrentLinkedQueue)
  4. load the ConcurrentLinkedQueue with reads from your AudioInputLine
  5. poll from the ConcurrentLinkedQueue from the AnimationTimer

To get the timings to work out, you may need to use decimation (e.g., throwing out every 2nd or 2 out of 3 or more PCM data points), or linear interpolation if the needed decimation doesn't come out to an easy-to-use rational fraction. In other words, you are not tied to a 1:1 correspondence between the array (tied to pixels) and the PCM data points. The more decimation you use, the more high frequencies will be lost.

Phil Freihofner
  • 7,645
  • 1
  • 20
  • 41