22

I'm creating an app that generates live instrument sounds and I'm planning on using the new Midi API featured in Android Marshmallow (version 6.0). I've read the package overview document here http://developer.android.com/reference/android/media/midi/package-summary.html and I know how to generate Midi notes but i'm still unsure: how do I actually play these notes after I've generated their Midi data?

Do I need a synthesizer program to play Midi notes? If so, do I have to make my own or is one provided by Android or a 3rd party?

I am a novice with Midi so please be as descriptive as possible with your answer.

What i've tried so far: I've created a Midi manager object and opened an input port

MidiManager m = (MidiManager)context.getSystemService(Context.MIDI_SERVICE); 
MidiInputPort inputPort = device.openInputPort(index);

Then, i've sent a test noteOn midi message to the port

byte[] buffer = new byte[32];
int numBytes = 0;
int channel = 3; // MIDI channels 1-16 are encoded as 0-15.
buffer[numBytes++] = (byte)(0x90 + (channel - 1)); // note on
buffer[numBytes++] = (byte)60; // pitch is middle C
buffer[numBytes++] = (byte)127; // max velocity
int offset = 0;
// post is non-blocking
inputPort.send(buffer, offset, numBytes);

I've also set up a class to receive the midi note messages

class MyReceiver extends MidiReceiver {
    public void onSend(byte[] data, int offset,
            int count, long timestamp) throws IOException {
        // parse MIDI or whatever
    }
}
MidiOutputPort outputPort = device.openOutputPort(index);
outputPort.connect(new MyReceiver());

Now, here's where i'm most confused. The use case of my app is to be an all-in-one composition & playback tool for making music. In other words, my app needs to contain or use a virtual midi device (like an intent of another app's midi synthesizer). Unless someone already made such a synthesizer, I must create one myself within my app's lifecycle. How do I actually actually convert a received midi noteOn() into sound coming out of my speakers? I'm especially confused because there also has to be a way to programmatically decide what type of instrument the note sounds like it's coming from: is this also done in a synthesizer?

Midi support in Android Marshmallow is fairly new so I haven't been able to find any tutorials or sample synthesizer apps online. Any insight is appreciated.

Cody
  • 1,801
  • 3
  • 28
  • 53
  • Which device did you connect to? What are its type and name? – CL. Mar 24 '16 at 07:23
  • I haven't actually connected to a device yet, this is part of my question. The "device" I would be connecting to would be a virtual device: a program to accept midi notes as input and output sound. Is there such a program in existence or do I have to make my own? – Cody Mar 25 '16 at 00:05
  • And what is the variable `device` in your code? Anyway, read the [documentation](http://developer.android.com/reference/android/media/midi/package-summary.html#get_list_of_already_plugged_in_entities). – CL. Mar 25 '16 at 09:11
  • 1
    I've already read that documentation, your answer is not helpful at all. "device" is not a real object, I merely posted the relevant sample code from that document in my question to give context. In a real program, device would be an object that accepts midi instructions (like noteOn) and synthesizes them into real sounds. If you would've read the documentation you would've realized that there is no mention of a pre-packaged synthesizer to use to actually play noteOn() messages, which is what i'm asking about: does such a prebuilt Android synthesizer exist? – Cody Mar 25 '16 at 22:12
  • A comment is not an answer. Anyway, it appears nobody knows the answer yet. Please run the enumeration code on your Marshmallow device (I don't have one), and post the answer here. – CL. Mar 26 '16 at 08:36
  • Unfortunately, I don't either. I might be able to emulate one but if not I appear to be stuck. There are many posts about this issue but none since Marshmallows release. Apparently there is a native Android synth called Sonivox EAS/JETPlayer but working documentation is sparse at best. – Cody Mar 26 '16 at 15:42
  • Do you need it to play real-time? If your app generates a .mid file, and you need to play back that file (from the device), you don't need to jump through hoops of lava... .mid playback is supported. – CaptJak Apr 06 '16 at 15:22

3 Answers3

29

I haven't found any "official" way to control the internal synthesizer from Java code.

Probably the easiest option is to use the Android midi driver for the Sonivox synthesizer.

Get it as an AAR package (unzip the *.zip) and store the *.aar file somewhere in your workspace. The path doesn't really matter and it doesn't need to be inside your own app's folder structure but the "libs" folder inside your project could be a logical place.

With your Android project open in Android Studio:

File -> New -> New Module -> Import .JAR/.AAR Package -> Next -> Find and select the "MidiDriver-all-release.aar" and change the subproject name if you want. -> Finish

Wait for Gradle to do it's magic and then go to your "app" module's settings (your own app project's settings) to the "Dependencies" tab and add (with the green "+" sign) the MIDI Driver as a module dependency. Now you have access to the MIDI Driver:

import org.billthefarmer.mididriver.MidiDriver;
   ...
MidiDriver midiDriver = new MidiDriver();

Without having to worry anything about NDK and C++ you have these Java methods available:

// Not really necessary. Receives a callback when/if start() has succeeded.
midiDriver.setOnMidiStartListener(listener);
// Starts the driver.
midiDriver.start();
// Receives the driver's config info.
midiDriver.config();
// Stops the driver.
midiDriver.stop();
// Just calls write().
midiDriver.queueEvent(event);
// Sends a MIDI event to the synthesizer.
midiDriver.write(event);

A very basic "proof of concept" for playing and stopping a note could be something like:

package com.example.miditest;

import android.os.Bundle;
import android.support.v7.app.AppCompatActivity;
import android.util.Log;
import android.view.MotionEvent;
import android.view.View;
import android.widget.Button;

import org.billthefarmer.mididriver.MidiDriver;

public class MainActivity extends AppCompatActivity implements MidiDriver.OnMidiStartListener,
        View.OnTouchListener {

    private MidiDriver midiDriver;
    private byte[] event;
    private int[] config;
    private Button buttonPlayNote;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        buttonPlayNote = (Button)findViewById(R.id.buttonPlayNote);
        buttonPlayNote.setOnTouchListener(this);

        // Instantiate the driver.
        midiDriver = new MidiDriver();
        // Set the listener.
        midiDriver.setOnMidiStartListener(this);
    }

    @Override
    protected void onResume() {
        super.onResume();
        midiDriver.start();

        // Get the configuration.
        config = midiDriver.config();

        // Print out the details.
        Log.d(this.getClass().getName(), "maxVoices: " + config[0]);
        Log.d(this.getClass().getName(), "numChannels: " + config[1]);
        Log.d(this.getClass().getName(), "sampleRate: " + config[2]);
        Log.d(this.getClass().getName(), "mixBufferSize: " + config[3]);
    }

    @Override
    protected void onPause() {
        super.onPause();
        midiDriver.stop();
    }

    @Override
    public void onMidiStart() {
        Log.d(this.getClass().getName(), "onMidiStart()");
    }

    private void playNote() {

        // Construct a note ON message for the middle C at maximum velocity on channel 1:
        event = new byte[3];
        event[0] = (byte) (0x90 | 0x00);  // 0x90 = note On, 0x00 = channel 1
        event[1] = (byte) 0x3C;  // 0x3C = middle C
        event[2] = (byte) 0x7F;  // 0x7F = the maximum velocity (127)

        // Internally this just calls write() and can be considered obsoleted:
        //midiDriver.queueEvent(event);

        // Send the MIDI event to the synthesizer.
        midiDriver.write(event);

    }

    private void stopNote() {

        // Construct a note OFF message for the middle C at minimum velocity on channel 1:
        event = new byte[3];
        event[0] = (byte) (0x80 | 0x00);  // 0x80 = note Off, 0x00 = channel 1
        event[1] = (byte) 0x3C;  // 0x3C = middle C
        event[2] = (byte) 0x00;  // 0x00 = the minimum velocity (0)

        // Send the MIDI event to the synthesizer.
        midiDriver.write(event);

    }

    @Override
    public boolean onTouch(View v, MotionEvent event) {

        Log.d(this.getClass().getName(), "Motion event: " + event);

        if (v.getId() == R.id.buttonPlayNote) {
            if (event.getAction() == MotionEvent.ACTION_DOWN) {
                Log.d(this.getClass().getName(), "MotionEvent.ACTION_DOWN");
                playNote();
            }
            if (event.getAction() == MotionEvent.ACTION_UP) {
                Log.d(this.getClass().getName(), "MotionEvent.ACTION_UP");
                stopNote();
            }
        }

        return false;
    }
}

The layout file just has one button that plays the predefined note when held down and stops it when released:

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:paddingBottom="@dimen/activity_vertical_margin"
    android:paddingLeft="@dimen/activity_horizontal_margin"
    android:paddingRight="@dimen/activity_horizontal_margin"
    android:paddingTop="@dimen/activity_vertical_margin"
    tools:context="com.example.miditest.MainActivity"
    android:orientation="vertical">

    <Button
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="Play a note"
        android:id="@+id/buttonPlayNote" />
</LinearLayout>

It is actually this simple. The code above could well be a starting point for a touch piano app with 128 selectable instruments, very decent latency and a proper "note off" functionality which many apps lack.

As for choosing the instrument: You'll just need to send a MIDI "program change" message to the channel on which you intend to play to choose one of the 128 sounds in the General MIDI soundset. But that's related to the details of MIDI and not to the usage of the library.

Likewise you'll probably want to abstract away the low level details of MIDI so that you can easily play a specific note on a specific channel with a specific instrument at a specific velocity for a specific time and for that you might find some clues from all the open source Java and MIDI related applications and libraries made so far.

This approach doesn't require Android 6.0 by the way. And at the moment only 4.6 % of devices visiting the Play Store run Android 6.x so there wouldn't be much audience for your app.

Of course if you want to use the android.media.midi package you could then use the library to implement a android.media.midi.MidiReceiver to receive the MIDI events and play them on the internal synthesizer. Google already has some demo code that plays notes with square and saw waves. Just replace that with the internal synthesizer.

Some other options could be to check out what's the status with porting FluidSynth to Android. I guess there might be something available.

Edit: Other possibly interesting libraries:

Markus Kauppinen
  • 3,025
  • 4
  • 20
  • 30
  • 3
    I tried out creating a more complete quick and dirty "touch piano" with one full octave where you can also select the instrument via a MIDI program change message. (See the `selectInstrument()` method for reference) The Android Studio project is [here](https://bitbucket.org/MarkusKauppinen/midipianotest). Everything is Public Domain from my side and the library's documentation says: _"Licensed under GPL or Apache license - your choice."_ – Markus Kauppinen Apr 10 '16 at 12:29
  • Can you provide documentation for the available midi instrument soundfonts on Android? I see an integer is passed into selectInstrument() as a parameter but how do I know which integer maps to which instrument? If I wanted to add custom instruments (such as specific flute models) how would I go about doing so? – Cody Apr 21 '16 at 01:53
  • 2
    The internal synth is "General MIDI level 1" compatible. The GM1 specification defines 128 instrument sounds (part of them being sound F/X instead of actual instruments). You can find all kinds of MIDI details for example at [https://midi.org/specifications](https://midi.org/specifications). The instrument set is [here](https://midi.org/specifications/item/gm-level-1-sound-set). You can't add custom instruments unless the internal synth supports [downloadable sounds (DLS)](https://midi.org/specifications/category/dls-specifications). I'm not familiar with DLS myself. – Markus Kauppinen Apr 21 '16 at 06:13
  • 1
    @MarkusKauppinen This is working great. Thanks. But, can you tell me how do I store in file notes played and then play it again afterwards? – SwapnilD Sep 20 '16 at 07:52
  • 2
    @SwapnilD That's a separate and quite broad topic on its own. And I haven't done it myself. But you would have to understand MIDI files and MIDI events and store the MIDI events (program change, note on, note off etc.) produced by the user into a MIDI file. I believe the javax.sound.midi package would be useful in that. MIDI file playback on the other hand is already supported on Android, so that's the easy part. – Markus Kauppinen Sep 20 '16 at 09:48
  • 1
    @MarkusKauppinen i had a question regarding your code. How do i manage the tempo ? I mean i have 4 Whole Notes that i want to play i know everything except the tempo feat. However how do i set a tempo or how do i manage it ? – Giuseppe Oct 01 '17 at 19:54
  • 1
    @Giuseppe The code is just playing single notes and there's no tempo information involved in any way. If playing notes this way, just use any timer mechanism provided by Java/Android. Or you can consider constructing an actual sequence of MIDI events (a "MIDI file") and feeding that to the MediaPlayer. – Markus Kauppinen Oct 02 '17 at 16:38
  • 1
    In `build.gradle`, I had to change `implementation fileTree(dir: 'libs', include: ['*.jar'])` to `implementation fileTree(dir: 'libs', include: ['*.jar', '*.aar'])` to find the module – alexdriedger Mar 06 '18 at 22:13
6

Do I need a synthesizer program to play Midi notes? If so, do I have to make my own or is one provided by Android or a 3rd party?

No, fortunately you don't need to make your own synthesizer. Android already has one built in: the SONiVOX Embedded Audio Syntehesizer. Android states in the docs on SONiVOX JETCreator:

JET works in conjunction with SONiVOX's Embedded Audio Synthesizer (EAS) which is the MIDI playback device for Android.

It wasn't clear whether or not you want real-time playback, or if you want to create a composition first and play it later within the same app. You also state that you want to play midi notes, not files. But, just so you know, Midi playback is supported on android devices. So playing a .mid file should be done the same way you would play a .wav file using MediaPlayer.

To be honest, I haven't use the midi package, or done midi playback, but if you can create a .mid file and save it to disk, then you should be able to play it back using straight MediaPlayer.

Now, if you want to play straight midi notes, not files, then you can use this mididriver package. Using this package you should be able to write midi data to the Embedded Synthesizer:

/**
* Writes midi data to the Sonivox synthesizer. 
* The length of the array should be the exact length 
* of the message or messages. Returns true on success, 
* false on failure.
*/

boolean write(byte buffer[])  

If you want to step even lower than that, you could even play straight PCM using AudioTrack.

For additional info, here is a blog post (archive link) I found from someone who seemed to have similar troubles to yours. He states:

Personally I solved the dynamic midi generation issue as follows: programmatically generate a midi file, write it to the device storage, initiate a mediaplayer with the file and let it play. This is fast enough if you just need to play a dynamic midi sound. I doubt it’s useful for creating user controlled midi stuff like sequencers, but for other cases it’s great.

Hope I covered everything.

Morrison Chang
  • 11,691
  • 3
  • 41
  • 77
CaptJak
  • 3,592
  • 1
  • 29
  • 50
  • 1
    "Live instrument sounds" implies real-time playback. – CL. Apr 06 '16 at 18:37
  • The question actually is whether (and how) Marshmallow allows accessing the synthesizer through the native MIDI package, without having to muck around with the NDK. – CL. Apr 06 '16 at 18:41
  • @CL. I didn't get that from the question, thanks for clarifying. No, it looks like the only way is to muck around with the NDK. It looks like the package I linked (on GitHub) does the mucking for you though. From the blog post (also linked above): *"But it’s not exposed to us poor developers. I also don’t know if it’s possible to use the NDK to just use these C(++) API’s. I don’t have the C knowledge for that, and the NDK docs don’t mention them as supported/stable."* There *was* a [feature request](https://code.google.com/p/android/issues/detail?id=8201) – CaptJak Apr 06 '16 at 18:53
  • This describes the situation before Marshmallow. Did it change anything? – CL. Apr 06 '16 at 19:46
  • 1
    @CL. Due to the inability to find anything which states otherwise, the fact that midi streaming is not listed in the `android.media.midi` overview, that the only Docs I could find going over midi streaming was from the [SONiVOX EAS API](https://github.com/android/platform_external_sonivox/tree/master/docs), and that the feature request I listed above was closed in March 2015 (7 months before release), I would say that it didn't change anything. – CaptJak Apr 06 '16 at 20:10
  • However, @CL. I could be wrong, since I haven't checked *everywhere*. It's much easier to find something that exists than something which doesn't... :P – CaptJak Apr 06 '16 at 20:12
  • @CaptJak can you tell how you wrote a midi file in android? – Beniamin Ionut Dobre Jun 11 '20 at 12:37
1

To generate sound using Android MIDI API, you need a synthesizer app which accepts MIDI input. Unfortunately, this is the only such app I have found on Google Play: https://play.google.com/store/apps/details?id=com.mobileer.midisynthexample

I was able to play music by sending note on and note off messages to this app. But program change worked poorly. Unless I did something wrong in my code, it seems the app has only two instruments.

However, there are some guys working on other synthesizer apps, so I expect more apps will be available soon. This app looks promising, though I haven't tested it myself yet: https://github.com/pedrolcl/android/tree/master/NativeGMSynth

Elazar
  • 20,415
  • 4
  • 46
  • 67