Play .mp3 and a sequence of notes (MIDI) simultaneously in Java

Question

I am currently delopping an application where the user can load a .mp3 file and enter a sequence of notes. The goal for the user is to match this sequence of notes with the song of the .mp3 file.

This requires the possibility to play the .mp3 file and the sequence of notes simultaneously. After some research I found out that either the Java Sound API or JFuge can do the job to produce a sequence of notes (MIDI). (The input given by the user). As stated here, JLayer can be used to play mp3 files in Java. (I could also transform the .mp3 to .wav and use another way to play the transformed .wav).

However, would it be possible to play this .mp3 and sequence of notes together without any problems, or should I first convert them to one single file?

The user should be able to play the .mp3 and his/her sequence of notes at any random timestamp simultaneously. Preferably without any delay so the user can easily adapt a note to match the pitch of the file. It seems that merging them together to one file, before playing them, would be too much overhead when the user is almost constantly changing a note and replaying to check if it matches the pitch.

Thanks in advance!

Answer 1

Java supports playback from multiple threads. All you need to do is run the .mp3 from one thread, and the midi-generated notes on another concurrently running thread.

There used to be a few Linux systems that could only handle output from one audio source at a time. I don't know if this is still an issue.

Another, much more elaborate possibility that would let you do live mixing and output to a single line would be to read the song file using AudioInputStream, convert the bytes to PCM on the fly (e.g., to floats ranging from -1 to 1) (or preload and store the audio as PCM), and then add this to PCM data coming from a do-it-yourself synth, and then convert this back to bytes and output via a SourceDataLine.

That is a lot of trouble and you probably don't want to go that route, but if you did, following is some info to help break down the various steps of one possible realization.

Loading .wav data and converting it into an internal PCM form can be seen in the open-source AudioCue (line 359 loadURL method). And here is an example (free download) of a real-time Java synth I made that runs via keystrokes. One of the voices is a simple organ, which outputs PCM audio data by just adding four sine waves at harmonic frequencies. Making other sounds is possible if you want to get into other forms of synthesis but gets more involved.

(IDK how to convert data coming from a MIDI-controlled synth, unless maybe a TargetDataLine can be identified, and data from it converted to PCM similar to the conversion used in reading from an AudioInputStream in the AudioCue source example.)

Given two PCM sources, the two can be mixed in real time using addition, converted to bytes and output via a single SourceDataLine (see line 1387 convertBufferToAudioBytes method). The SourceDataLine can be kept running indefinitely if you input zeros from the contributors when they are not playing. An SDL spends the vast majority of its time in a blocked state as audio data processing is much quicker than the rate it is consumed by the system, and so uses very little cpu.