36

I'm reviewing all kinds of Android sound API and I'd like to know which one I should use. My goal is to get low latency audio or, at least, deterministic behavior regarding delay of playback.

We've had a lot of problems and it seems that Android sound API is crap, so I'm exploring possibilities.

The problem we have is that there is significant delay between sound_out.write(sound_samples); and actual sound played from the speakers. Usually it is around 300 ms. The problem is that on all devices it's different; some don't have that problem, but most are crippled (however, CS call has zero latency). The biggest issue with this ridiculous delay is that on some devices this delay appears to be some random value (i.e. it's not always 300ms).

I'm reading about OpenSL ES and I'd like to know if anybody had experience with it, or it's the same shit but wrapped in different package?

I prefer to have native access, but I don't mind Java layer indirection as long as I can get deterministic behavior: either the delay has to be constant (for a given device), or I'd like to get access to current playback position instead of guessing it with a error range of ±300 ms...

EDIT:
1.5 years later I tried multiple android phones to see how I can get best possible latency for a real time voice communication. Using specialized tools I measured the delay of waveout path. Best results were over 100 ms, most phones were in 180ms range. Anybody have ideas?

Pavel P
  • 15,789
  • 11
  • 79
  • 128
  • 1
    If you're targeting Android 2.3 and up, you can use OpenSl ES. It is not a wrapper, and is faster, but don't expect anything below 40ms either. – Phonon Sep 01 '11 at 21:40
  • 2
    You might also check out http://music.columbia.edu/pipermail/andraudio/ - it's a mailing list for android audio. My sense is that many people share your pain. – James Moore Sep 14 '11 at 05:38
  • You may be interested to look into Android source code. There are some underlying clases like AudioPolicyManager and audio_policy which, as I understand, are used to work with audio on low level. They may be useful. – Victor Ronin Sep 06 '12 at 19:11
  • 2
    @JamesMoore: As of Jellybean the doc you quoted is no longer strictly accurate. Jellybean includes scheduler improvements that OpenSL ES can take advantage of but Java cannot. Please see my answer below. – Ian Ni-Lewis Sep 13 '12 at 17:54
  • @Phonon: 30ms is achievable on some Jellybean devices. It depends a lot on how you measure it, though. Audio output latency is only part of the issue. Touch latency is also important, and varies from device to device. – Ian Ni-Lewis Sep 13 '12 at 17:56
  • +1 for @IanNi-Lewis's comment - 4.1 makes a big difference with OpenSL. – James Moore Sep 15 '12 at 14:49

5 Answers5

27

SoundPool is the lowest-latency interface on most devices, because the pool is stored in the audio process. All of the other audio paths require inter-process communication. OpenSL is the best choice if SoundPool doesn't meet your needs.

Why OpenSL? AudioTrack and OpenSL have similar latencies, with one important difference: AudioTrack buffer callbacks are serviced in Dalvik, while OpenSL callbacks are serviced in native threads. The current implementation of Dalvik isn't capable of servicing callbacks at extremely low latencies, because there is no way to suspend garbage collection during audio callbacks. This means that the minimum size for AudioTrack buffers has to be larger than the minimum size for OpenSL buffers to sustain glitch-free playback.

On most Android releases this difference between AudioTrack and OpenSL made no difference at all. But with Jellybean, Android now has a low-latency audio path. The actual latency is still device dependent, but it can be considerably lower than previously. For instance, http://code.google.com/p/music-synthesizer-for-android/ uses 384-frame buffers on the Galaxy Nexus for a total output latency of under 30ms. This requires the audio thread to service buffers approximately once every 8ms, which was not feasible on previous Android releases. It is still not feasible in a Dalvik thread.

This explanation assumes two things: first, that you are requesting the smallest possible buffers from OpenSL and doing your processing in the buffer callback rather than with a buffer queue. Second, that your device supports the low-latency path. On most current devices you will not see much difference between AudioTrack and OpenSL ES. But on devices that support Jellybean+ and low-latency audio, OpenSL ES will give you the lowest-latency path.

Ian Ni-Lewis
  • 2,377
  • 20
  • 20
  • Ian, I actually performed tests and it looks that OpenSL performs worse. – Pavel P Feb 28 '13 at 21:24
  • Can you describe the tests you performed? There are a number of ways you can set up OpenSL ES so that its latency is higher than AudioTrack. See my answer to http://stackoverflow.com/questions/14842803/low-latency-audio-playback-on-android/16205370#16205370 for details on how to set up OpenSL to be low latency. – Ian Ni-Lewis Apr 25 '13 at 18:29
  • I used Android's OpenSL waveout code from WebRTC for playing audio. The idea of my test was simple: I generate two distinctive short burst of noise from loudspeaker and I pick it up on the microphone of the same device. Then I measure the time from when I wrote these bursts of noise to waveout and the time when I detected it on wavein. We need lowest possible waveout latency for real-time VoIP calls, we also need all kinds of controls available for AudioTrack, like switching earpiece/loudspeaker, switching automatic gain or acoustic echo cancellation; Is all of that unavailable with OpenSL? – Pavel P Apr 25 '13 at 20:07
  • The WebRTC code doesn't enable fast path low latency. (Why would it?) Watch http://www.youtube.com/watch?v=d3kfEeMZ65c. The important takeaway is that if you do not set the magic buffer size and sample rate – Ian Ni-Lewis May 31 '13 at 20:00
2

IIRC, OpenSL is passed through the same interface as AudioTrack, so at best it will match AudioTrack. (FWIW, I'm currently using OpenSL for "low latency" output)

The sad truth is there is no such thing as low latency audio on Android. There isn't even a proper way to flag and/or filter devices based on audio latency.

What interface you'll want to use to minimize latency is going to depend on what you are trying to do. If you want to have an audio stream you'll be looking at either OpenSL or AudioTrack.

If you want to trigger some static oneshots you may want to use SoundPool. For static oneshots SoundPool will have low latency as the samples are preloaded to the hardware. I think it's possible to preload oneshots using OpenSL as well, but I haven't tried.

Drakonite
  • 1,309
  • 9
  • 15
  • I tried some tests, and they confirm: at best OpenSL gets the same result as AudioTrack. – Pavel P Feb 28 '13 at 21:28
  • As above, this is true if OpenSL is not using the fast path. If you configure it so that it can use the fast path, your latency should be significantly better on most devices. – Ian Ni-Lewis May 31 '13 at 20:05
1

About the problem of a deterministic/constant latency, here you can find an interesting article:

APPROACHES FOR CONSTANT AUDIO LATENCY ON ANDROID

The core of their investigations is: Because the Audio HAL, which is one of the deeper levels of the audiopath and responsible for the timing of the audio-callback-events, is vendor-implemented the relative latencies can vary, especially in cheap hardware. So they suggest two approaches to reduce the variance of the latency. One is to take care of the callback-timing by inserting audio in fixed intervals, the other one is to filter the callback times to estimate the time at which a constant latency callback should have occured by appliyng a smoothing filter. With this two approaches the could significantly reduce the variance of the latency.

It should also be mentioned, that there is a new native Android-Audio-API, AAudio.

AAudio API

It's available/stable from Android Oreo 8.1 (API Level 27). There is also a wrapper, which dynamically chooses between OpenSL ES and AAudio and is much easier to code then OpenSL ES. It's still in developer preview.

Oboe Audio Library

Niklas Dada
  • 311
  • 4
  • 17
1

The lowest latency you can get is from SoundPool. There's a limit on how big of a sound you can play that way, but if you're under the limit (1Mb, IIRC) it's pretty low latency. Even that's probably not 40ms, though.

But it is faster than what you can get by streaming, at least in my experience.

Caveat: You may see occasional crashes in SoundPool on Samsung devices. I'm have a theory that it only happens when you access the SoundPool from multiple threads, but I haven't verified this.

EDIT: OpenSL ES apparently has extremely HIGH latency on Kindle Fire, while SoundPool is much better, but the reverse may be true on other platforms.

SomeCallMeTim
  • 4,503
  • 2
  • 28
  • 27
  • 4
    The OpenSL ES buffer size is chosen by the OEM. Kindle Fire apparently has a very large buffer, which makes sense--larger buffers mean fewer buffer servicing calls, which means longer battery life. And you don't need low latency if all you're doing is playing music and movies. – Ian Ni-Lewis Sep 06 '12 at 23:02
0

The best way to get low latency for native code on Android is to use Oboe.

https://github.com/google/oboe

Oboe wraps AAudio on newer devices. AAudio offers the lowest possible latency paths. If AAudio is not available then Oboe calls OpenSL ES. Oboe is much easier to use than OpenSL ES.

AAudio either calls through AudioTrack or through a new MMAP data path. AAudio makes it easier to get a FAST track because you can leave some parameters unspecified. AAudio will then choose the right parameters needed for a FAST track.

philburk
  • 691
  • 5
  • 13