Java swing - Processing simultaneous key presses with key bindings

Question

Info

Having read through several related questions, I think I have a bit of a unique situation here.

I am building a Java swing application to help drummers make simple shorthand song charts. There's a dialog where the user can "key in" a rhythm, which is to be recorded to a MIDI sequence and then processed into either tabulature or sheet music. This is intended to be used with short sections of a song.

Setup

The idea is when the bound JButtons fire their action while the sequence is being recorded, they'll generate a MidiMessage with timing information. I also want the buttons to visually indicate that they've been activated.

The bound keys are currently firing correctly using the key bindings I've implemented (except for simultaneous keypresses)...

Problem

It's important that simultaneous keypresses are registered as a single event--and the timing matters here.

So, for example, if the user pressed H (hi-hat) and S (snare) at the same time, it would register as a unison hit at the same place in the bar.

I have tried using a KeyListener implementation similar to this: https://stackoverflow.com/a/13529058/13113770 , but with that setup I ran into issues with focus, and though it could detect simultaneous key presses, it would also process them individually.

Could anyone shed some light on this for me?

  // code omitted

  public PunchesDialog(Frame owner, Song partOwner, Part relevantPart)
  {
    super(owner, ModalityType.APPLICATION_MODAL);

    this.partOwner = partOwner;
    this.relevantPart = relevantPart;

    // code omitted

    /*
     * Voices Panel
     */

    voices = new LinkedHashMap<>() {{
      put("crash",    new VoiceButton("CRASH (C)",         crashHitAction));
      put("ride",     new VoiceButton("RIDE (R)",          rideHitAction));
      put("hihat",    new VoiceButton("HI-HAT (H)",        hihatHitAction));
      put("racktom",  new VoiceButton("RACK TOM (T)",      racktomHitAction));
      put("snare",    new VoiceButton("SNARE (S)",         snareHitAction));
      put("floortom", new VoiceButton("FLOOR TOM (F)",     floortomHitAction));
      put("kickdrum", new VoiceButton("KICK DRUM (SPACE)", kickdrumHitAction));
    }};

    Action crashHitAction = new CrashHitAction();
    Action rideHitAction = new RideHitAction();
    Action hihatHitAction = new HihatHitAction();
    Action racktomHitAction = new RacktomHitAction();
    Action snareHitAction = new SnareHitAction();
    Action floortomHitAction = new FloortomHitAction();
    Action kickdrumHitAction = new KickdrumHitAction();

    KeyStroke key;
    InputMap inputMap = ((JPanel) getContentPane()).
      getInputMap(JComponent.WHEN_IN_FOCUSED_WINDOW);
    ActionMap actionMap = ((JPanel) getContentPane()).getActionMap();

    key = KeyStroke.getKeyStroke(KeyEvent.VK_C, 0);
    inputMap.put(key, "crashHit");
    actionMap.put("crashHit", crashHitAction);

    key = KeyStroke.getKeyStroke(KeyEvent.VK_R, 0);
    inputMap.put(key, "rideHit");
    actionMap.put("rideHit", rideHitAction);

    key = KeyStroke.getKeyStroke(KeyEvent.VK_H, 0);
    inputMap.put(key, "hihatHit");
    actionMap.put("hihatHit", hihatHitAction);

    key = KeyStroke.getKeyStroke(KeyEvent.VK_T, 0);
    inputMap.put(key, "racktomHit");
    actionMap.put("racktomHit", racktomHitAction);

    key = KeyStroke.getKeyStroke(KeyEvent.VK_S, 0);
    inputMap.put(key, "snareHit");
    actionMap.put("snareHit", snareHitAction);

    key = KeyStroke.getKeyStroke(KeyEvent.VK_F, 0);
    inputMap.put(key, "floortomHit");
    actionMap.put("floortomHit", floortomHitAction);

    key = KeyStroke.getKeyStroke(KeyEvent.VK_SPACE, 0);
    inputMap.put(key, "kickdrumHit");
    actionMap.put("kickdrumHit", kickdrumHitAction);

    final JPanel pnlVoices = new JPanel(new MigLayout(
          "Insets 0, gap 0, wrap 2", "[fill][fill]", "fill"));
    pnlVoices.add(voices.get("crash"),    "w 100%, h 100%, grow");
    pnlVoices.add(voices.get("ride"),     "w 100%");
    pnlVoices.add(voices.get("hihat"),    "w 100%");
    pnlVoices.add(voices.get("racktom"),  "w 100%");
    pnlVoices.add(voices.get("snare"),    "w 100%");
    pnlVoices.add(voices.get("floortom"), "w 100%");
    pnlVoices.add(voices.get("kickdrum"), "span");

    // code omitted

  }

  private class CrashHitAction extends AbstractAction
  {
    @Override
    public void actionPerformed(ActionEvent e) {
      // voices.get("crash").doClick(100);
      kfMgr.clearFocusOwner();

      logger.debug("hit crash");
    }
  }

  private class RideHitAction extends AbstractAction
  {
    @Override
    public void actionPerformed(ActionEvent e) {
      // voices.get("ride").doClick(100);
      kfMgr.clearFocusOwner();

      logger.debug("hit ride");
    }
  }

  private class HihatHitAction extends AbstractAction
  {
    @Override
    public void actionPerformed(ActionEvent e) {
      // voices.get("hihat").doClick(100);
      kfMgr.clearFocusOwner();

      logger.debug("hit hihat");
    }
  }

  private class RacktomHitAction extends AbstractAction
  {
    @Override
    public void actionPerformed(ActionEvent e) {
      // voices.get("racktom").doClick(100);
      kfMgr.clearFocusOwner();

      logger.debug("hit racktom");
    }
  }

  private class FloortomHitAction extends AbstractAction
  {
    @Override
    public void actionPerformed(ActionEvent e) {
      // voices.get("floortom").doClick(100);
      kfMgr.clearFocusOwner();

      logger.debug("hit floortom");
    }
  }

  private class SnareHitAction extends AbstractAction
  {
    @Override
    public void actionPerformed(ActionEvent e) {
      // voices.get("snare").doClick(100);
      kfMgr.clearFocusOwner();

      logger.debug("hit snare");
    }
  }

  private class KickdrumHitAction extends AbstractAction
  {
    @Override
    public void actionPerformed(ActionEvent e) {
      // voices.get("kickdrum").doClick(100);
      kfMgr.clearFocusOwner();

      logger.debug("hit kickdrum");
    }
  }

Screenshot of dialog here: https://i.stack.imgur.com/n4RzY.png

Answer 1

You need to decouple some of your concepts a little more. For example, the Action API allows you to make use of the same Action (same instance of multiple instance of the same Action) on buttons (all buttons) as well as the key bindings.

In this case, you want to find away in which the Action is decoupled from the possible trigger (ie, don't assume it's a button or a key binding if possible)

For me, when the key binding is triggered, I'd want to notify some kind of observer or manager that the action has taken place. A possible consideration would also be, what do when it's pressed as apposed to when it's released, is there a difference?

KeyStroke allows you to define both "pressed" and "released" triggers. I would then use some kind of monitor to manage the state, ie a series of booleans which are either true or false depending on the state of the action, this, however, doesn't scale well, so, instead, I'd consider using a enum and a Set instead.

The following example will only highlight the labels while the key for the associated action is been held down.

import java.awt.Color;
import java.awt.EventQueue;
import java.awt.event.ActionEvent;
import java.awt.event.KeyEvent;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
import java.util.TreeSet;
import javax.swing.AbstractAction;
import javax.swing.ActionMap;
import javax.swing.InputMap;
import javax.swing.JComponent;
import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JPanel;
import javax.swing.KeyStroke;
import javax.swing.border.CompoundBorder;
import javax.swing.border.EmptyBorder;
import javax.swing.border.LineBorder;

public class Main {
    public static void main(String[] args) {
        new Main();
    }

    public Main() {
        EventQueue.invokeLater(new Runnable() {
            @Override
            public void run() {
                JFrame frame = new JFrame();
                frame.add(new TestPane());
                frame.pack();
                frame.setLocationRelativeTo(null);
                frame.setVisible(true);
            }
        });
    }

    public class TestPane extends JPanel {

        enum UserAction {
            CRASH_HIT, RIDE_HIT, HI_HAT_HIT, RACK_TOM_HIT, SNARE_HIT, FLOOR_TOM_HIT, KICK_DRUM_HIT;
        }

        public interface Observer {
            public void didActivateAction(UserAction action);
            public void didDeactivateAction(UserAction action);
        }

        private Map<UserAction, JLabel> labels;
        private Set<UserAction> activeActions = new TreeSet<>();
        private final Set<UserAction> allActions = new TreeSet<>(Arrays.asList(UserAction.values()));

        public TestPane() {            
            labels = new HashMap<>();
            for (UserAction action : UserAction.values()) {
                JLabel label = new JLabel(action.name());
                label.setBorder(new CompoundBorder(new LineBorder(Color.DARK_GRAY), new EmptyBorder(8, 8, 8, 8)));
                label.setOpaque(true);
                add(label);

                labels.put(action, label);
            }

            InputMap inputMap = getInputMap(JComponent.WHEN_IN_FOCUSED_WINDOW);
            ActionMap actionMap = getActionMap();

            Observer observer = new Observer() {
                @Override
                public void didActivateAction(UserAction action) {
                    if (activeActions.contains(action)) {
                        // We don't want to deal with "repeated" key events
                        return;
                    }
                    activeActions.add(action);
                    // I could update the labels here, but this is a deliberate 
                    // example of how to decouple the action from the state
                    // so the actions can be dealt with in as a single unit
                    // of work, you can also take into consideratoin any
                    // relationships which different inputs might have as well
                    updateUIState();
                }

                @Override
                public void didDeactivateAction(UserAction action) {
                    activeActions.remove(action);
                    updateUIState();
                }
            };

            inputMap.put(KeyStroke.getKeyStroke(KeyEvent.VK_C, 0, false), "pressed-crashHit");
            inputMap.put(KeyStroke.getKeyStroke(KeyEvent.VK_C, 0, true), "released-crashHit");
            actionMap.put("pressed-crashHit", new InputAction(UserAction.CRASH_HIT, true, observer));
            actionMap.put("released-crashHit", new InputAction(UserAction.CRASH_HIT, false, observer));

            inputMap.put(KeyStroke.getKeyStroke(KeyEvent.VK_R, 0, false), "pressed-rideHit");
            inputMap.put(KeyStroke.getKeyStroke(KeyEvent.VK_R, 0, true), "released-rideHit");
            actionMap.put("pressed-rideHit", new InputAction(UserAction.RIDE_HIT, true, observer));
            actionMap.put("released-rideHit", new InputAction(UserAction.RIDE_HIT, false, observer));

            inputMap.put(KeyStroke.getKeyStroke(KeyEvent.VK_H, 0, false), "pressed-hihatHit");
            inputMap.put(KeyStroke.getKeyStroke(KeyEvent.VK_H, 0, true), "released-hihatHit");
            actionMap.put("pressed-hihatHit", new InputAction(UserAction.HI_HAT_HIT, true, observer));
            actionMap.put("released-hihatHit", new InputAction(UserAction.HI_HAT_HIT, false, observer));

            inputMap.put(KeyStroke.getKeyStroke(KeyEvent.VK_T, 0, false), "pressed-racktomHit");
            inputMap.put(KeyStroke.getKeyStroke(KeyEvent.VK_T, 0, true), "released-racktomHit");
            actionMap.put("pressed-racktomHit", new InputAction(UserAction.RACK_TOM_HIT, true, observer));
            actionMap.put("released-racktomHit", new InputAction(UserAction.RACK_TOM_HIT, false, observer));

            inputMap.put(KeyStroke.getKeyStroke(KeyEvent.VK_S, 0, false), "pressed-snareHit");
            inputMap.put(KeyStroke.getKeyStroke(KeyEvent.VK_S, 0, true), "released-snareHit");
            actionMap.put("pressed-snareHit", new InputAction(UserAction.SNARE_HIT, true, observer));
            actionMap.put("released-snareHit", new InputAction(UserAction.SNARE_HIT, false, observer));

            inputMap.put(KeyStroke.getKeyStroke(KeyEvent.VK_F, 0, false), "pressed-floortomHit");
            inputMap.put(KeyStroke.getKeyStroke(KeyEvent.VK_F, 0, true), "released-floortomHit");
            actionMap.put("pressed-floortomHit", new InputAction(UserAction.FLOOR_TOM_HIT, true, observer));
            actionMap.put("released-floortomHit", new InputAction(UserAction.FLOOR_TOM_HIT, false, observer));

            inputMap.put(KeyStroke.getKeyStroke(KeyEvent.VK_SPACE, 0, false), "pressed-kickdrumHit");
            inputMap.put(KeyStroke.getKeyStroke(KeyEvent.VK_SPACE, 0, true), "released-kickdrumHit");
            actionMap.put("pressed-kickdrumHit", new InputAction(UserAction.KICK_DRUM_HIT, true, observer));
            actionMap.put("released-kickdrumHit", new InputAction(UserAction.KICK_DRUM_HIT, false, observer));
        }

        protected void updateUIState() {
            Set<UserAction> inactiveActions = new TreeSet<>(allActions);
            inactiveActions.removeAll(activeActions);

            for (UserAction action : inactiveActions) {
                JLabel label = labels.get(action);
                label.setBackground(null);
                label.setForeground(Color.BLACK);
            }
            for (UserAction action : activeActions) {
                JLabel label = labels.get(action);
                label.setBackground(Color.BLUE);
                label.setForeground(Color.WHITE);
            }
        }

        // This could act as a base class, from which other, more dedicated
        // implementations could be built, which did focused jobs, for example
        // protected class ActivateCrashHit extends InputAction {
        //    public ActivateCrashHit(Observer observer) {
        //        super(UserAction.CRASH_HIT, true, observer);
        //    }
        //    // Override actionPerformed
        // }
        protected class InputAction extends AbstractAction {

            private UserAction action;
            private boolean activated;
            private Observer observer;

            public InputAction(UserAction action, boolean activated, Observer observer) {
                this.action = action;
                this.activated = activated;
                this.observer = observer;
            }

            @Override
            public void actionPerformed(ActionEvent e) {
                // This could perform other actions, but the intention of the
                // observer is provide an oppurunity for the interested party
                // to also make some kind of update, to allow the user to
                // see that that action occured
                if (activated) {
                    observer.didActivateAction(action);
                } else {
                    observer.didDeactivateAction(action);
                }
            }
        }
    }
}

You should also beware that that there is a hardware limitation on some keyboards which limit the number of simultaneous keys which can be pressed at any one time, although to be honest, I found it hard to press all the keys are once for this example any way

Answer 2

I personally would use the KeyListener interface to keep track of what the user has typed and then just register within a List the keys that have been pressed (without releasing) and then collect them once any key has been released.

Make sure though to add to the list only the keys that are not present yet because the keyPressed event is fired multiple times while the user is holding down a key.

I've also created a little sample to give you the idea.

public class MyClass extends JFrame implements KeyListener {

    private JTextArea textArea;
    private List<Character> listKeys;

    public MyClass() {
        setTitle("test");

        listKeys = new ArrayList<>();
        textArea = new JTextArea();
        textArea.addKeyListener(this);

        setLayout(new BorderLayout());
        add(textArea, BorderLayout.CENTER);

        setLocation(50, 50);
        setSize(500, 500);
        setVisible(true);
    }

    @Override
    public void keyTyped(KeyEvent e) {
    }

    @Override
    public void keyPressed(KeyEvent e) {
        if (!listKeys.contains(e.getKeyChar())) {
            listKeys.add(e.getKeyChar());
        }
    }

    @Override
    public void keyReleased(KeyEvent e) {
        if (listKeys.isEmpty()) {
            return;
        }

        if (listKeys.size() > 1) {
            System.out.print("The key combination ");
        } else {
            System.out.print("The key ");
        }
        for (Character c : listKeys) {
            System.out.print(c + " ");
        }
        System.out.println("has been entered");
        listKeys.clear();
    }

    public static void main(String[] args) {
        new MyClass();
    }
}