42

I'm currently attempting to minimize audio latency for a simple application:

I have a video on a PC, and I'm transmitting the video's audio through RTP to a mobile client. With a very similar buffering algorithm, I can achieve 90ms of latency on iOS, but a dreadful ±180ms on Android.

I'm guessing the difference stems from the well-known latency issues on Android.

However, after reading around for a bit, I came upon this article, which states that:

  1. Low-latency audio is available since Android 4.1/4.2 in certain devices.

  2. Low-latency audio can be achieved using libpd, which is Pure Data library for Android.

I have 2 questions, directly related to those 2 statements:

  1. Where can I find more information on the new low-latency audio in Jellybean? This is all I can find but it's sorely lacking in specific information. Should the changes be transparent to me, or is there some new class/API calls I should be implementing for me to notice any changes in my application? I'm using the AudioTrack API, and I'm not even sure if it should reap benefits from this improvement or if I should be looking into some other mechanism for audio playback.

  2. Should I look into using libpd? It seems to me like it's the only chance I have of achieving lower latencies, but since I've always thought of PD as an audio synthesis utility, is it really suited for a project that just grabs frames from a network stream and plays them back? I'm not really doing any synthesizing. Am I following the wrong trail?

As an additional note, before someone mentions OpenSL ES, this article makes it quite clear that no improvements in latency should be expected from using it:

"As OpenSL ES is a native C API, non-Dalvik application threads which call OpenSL ES have no Dalvik-related overhead such as garbage collection pauses. However, there is no additional performance benefit to the use of OpenSL ES other than this. In particular, use of OpenSL ES does not result in lower audio latency, higher scheduling priority, etc. than what the platform generally provides."

eerock
  • 188
  • 7
Sergio Morales
  • 2,600
  • 6
  • 32
  • 40
  • 13
    I'm a member of the Android team and I work closely with the authors of the article you cite. The passage you quoted is no longer strictly true. When the article was written, the smallest buffers available to OpenSL were still quite large. Now that the buffer size has been reduced in Jellybean, latency has dropped to the point where "Dalvik-related overhead such as garbage collection pauses" is a very significant consideration. The only way to reliably take advantage of the smaller Jellybean buffers is to use OpenSL. – Ian Ni-Lewis Apr 25 '13 at 00:46

7 Answers7

69

For lowest latency on Android as of version 4.2.2, you should do the following, ordered from least to most obvious:

  1. Pick a device that supports FEATURE_AUDIO_PRO if possible, or FEATURE_AUDIO_LOW_LATENCY if not. ("Low latency" is 50ms one way; pro is <20ms round trip.)

  2. Use OpenSL. The Dalvik GC has a low amortized cost, but when it runs it takes more time than a low-latency audio thread can allow.

  3. Process audio in a buffer queue callback. The system runs buffer queue callbacks in a thread that has more favorable scheduling than normal user-mode threads.

  4. Make your buffer size a multiple of AudioManager.getProperty(PROPERTY_OUTPUT_FRAMES_PER_BUFFER). Otherwise your callback will occasionally get two calls per timeslice rather than one. Unless your CPU usage is really light, this will probably end up glitching. (On Android M, it is very important to use EXACTLY the system buffer size, due to a bug in the buffer handling code.)

  5. Use the sample rate provided by AudioManager.getProperty(PROPERTY_OUTPUT_SAMPLE_RATE). Otherwise your buffers take a detour through the system resampler.

  6. Never make a syscall or lock a synchronization object inside the buffer callback. If you must synchronize, use a lock-free structure. For best results, use a completely wait-free structure such as a single-reader single-writer ring buffer. Loads of developers get this wrong and end up with glitches that are unpredictable and hard to debug.

  7. Use vector instructions such as NEON, SSE, or whatever the equivalent instruction set is on your target processor.

  8. Test and measure your code. Track how long it takes to run--and remember that you need to know the worst-case performance, not the average, because the worst case is what causes the glitches. And be conservative. You already know that if it takes more time to process your audio than it does to play it, you'll never get low latency. But on Android this is even more important, because the CPU frequency fluctuates so much. You can use perhaps 60-70% of CPU for audio, but keep in mind that this will change as the device gets hotter or cooler, or as the wifi or LTE radios start and stop, and so on.

Low-latency audio is no longer a new feature for Android, but it still requires device-specific changes in the hardware, drivers, kernel, and framework to pull off. This means that there's a lot of variation in the latency you can expect from different devices, and given how many different price points Android phones sell at, there probably will always be differences. Look for FEATURE_AUDIO_PRO or FEATURE_AUDIO_LOW_LATENCY to identify devices that meet the latency criteria your app requires.

Ian Ni-Lewis
  • 2,377
  • 20
  • 20
  • Re point 5. If the source audio is recorded at say 44100 but PROPERTY_OUTPUT_SAMPLE_RATE is 48000 then the audio will play too quickly. What is the best way around this? – Ian1971 Sep 16 '13 at 16:47
  • Ian, I believe PROPERTY_OUTPUT_SAMPLE_RATE (POSR) refers to the native audio sampling rate of your device's sound processor. This means that it if you feed it audio sampled at 48kHz and the POSR of your device is 48kHz, then the system will not have to do any "extra work" and thus can be played without additional latency. – rmigneco Mar 16 '14 at 23:11
  • Giving it audio sampled at say 44.1kHz does't mean it will be played at a different speed, but you're requiring the OS/device to do an additional re-sampling step to process the audio at 48kHz, which adds latency. – rmigneco Mar 16 '14 at 23:12
  • The point is to initialize audio playback at device's frequency, and do possible resampling of audio data on application side. – Pointer Null Apr 25 '14 at 08:09
  • Re. #4 "Make your buffer size a multiple of AudioManager.getProperty(PROPERTY_OUTPUT_FRAMES_PER_BUFFER)..." My read of this is that the OpenSL buffer size needs to be 1x, 2x, 3x, etc. the PROPERTY_OUTPUT_FRAMES_PER_BUFFER. Is this still true? This property varies across devices but I've seen as high as 3840 - which results in high latency. Or instead, do you mean the OpenSL can be a factor of PROPERTY_OUTPUT_FRAMES_PER_BUFFER (i.e. 64)? – windup Nov 03 '14 at 03:48
  • Hmm, good question. I think theoretically it *could* be a factor, because OpenSL will call your callback repeatedly until it has enough data. But that would be less efficient than just making your buffer big enough, and it wouldn't affect latency at all. 3840 is pretty big, but remember that higher latency == lower power, so it's not surprising that some devices might make that tradeoff. Those devices just have lousy latency, and there's nothing to be done about it. – Ian Ni-Lewis Feb 03 '15 at 15:48
  • Thanks. Point 5 is really important and should really be #1 since everyone can use it no matter the project. It helped me vanquish the wowza orcs. – sbaar Nov 17 '15 at 00:57
  • 3
    "On Android M, it is very important to use EXACTLY the system buffer size, due to a bug in the buffer handling code" Is there any more information on this? I seem to have this issue. I guess people are forced to use low-latency audio in this case. – Matthew Mitchell Sep 16 '16 at 20:56
  • @PointerNull Thanks for your comment. Could you tell me if I understood you correctly? Say, I have a sound file recorded at 24000Hz and a device with 48000Hz. In this case, I have two choices. 1. Initialize at 24000 and let the System resampler do its job. 2. Initialize at 48000 and resample the sound source myself. – Jenix May 19 '19 at 04:11
  • But the latter is better latency-wise. Is this correct? – Jenix May 19 '19 at 04:12
  • 1
    Yes @Jenix, you are correct. But maybe if you try to achieve low latency resampling of the source should be done beforehand, and not on the fly. – Stéphane Jun 09 '19 at 07:29
6

From the link at your point 1:

"Low-latency audio

Android 4.2 improves support for low-latency audio playback, starting from the improvements made in Android 4.1 release for audio output latency using OpenSL ES, Soundpool and tone generator APIs. These improvements depend on hardware support — devices that offer these low-latency audio features can advertise their support to apps through a hardware feature constant."

Your citation in complete form:

"Performance

As OpenSL ES is a native C API, non-Dalvik application threads which call OpenSL ES have no Dalvik-related overhead such as garbage collection pauses. However, there is no additional performance benefit to the use of OpenSL ES other than this. In particular, use of OpenSL ES does not result in lower audio latency, higher scheduling priority, etc. than what the platform generally provides. On the other hand, as the Android platform and specific device implementations continue to evolve, an OpenSL ES application can expect to benefit from any future system performance improvements."

So, the api to comunicate with drivers and then hw is OpenSl (in the same fashion Opengl does with graphics). The earlier versions of Android have a bad design in drivers and/or hw, though. These problems were addressed and corrected with 4.1 and 4.2 versions, so if the hd have the power, you get low latency using OpenSL.

Again, from this note from the puredata library website, is evident that the library uses OpenSL itself to achieve low latency:

Low latency support for compliant devices The latest version of Pd for Android (as of 12/28/2012) supports low-latency audio for compliant Android devices. When updating your copy, make sure to pull the latest version of both pd-for-android and the libpd submodule from GitHub.

At the time of writing, Galaxy Nexus, Nexus 4, and Nexus 10 provide a low-latency track for audio output. In order to hit the low-latency track, an app must use OpenSL, and it must operate at the correct sample rate and buffer size. Those parameters are device dependent (Galaxy Nexus and Nexus 10 operate at 44100Hz, while Nexus 4 operates at 48000Hz; the buffer size is different for each device).

As is its wont, Pd for Android papers over all those complexities as much as possible, providing access to the new low-latency features when available while remaining backward compatible with earlier versions of Android. Under the hood, the audio components of Pd for Android will use OpenSL on Android 2.3 and later, while falling back on the old AudioTrack/AudioRecord API in Java on Android 2.2 and earlier.

AndrewBloom
  • 2,171
  • 20
  • 30
6

When using OpenSL ES you should fulfil the following requirements to get low latency output on Jellybean and later versions of Android:

  • The audio should be mono or stereo, linear PCM.

  • The audio sample rate should be the same same sample rate as the output's native rate (this might not actually be required on some devices, because the FastMixer is capable of resampling if the vendor configures it to do so. But in my tests I got very noticeable artifacts when upsampling from 44.1 to 48 kHz in the FastMixer).

  • Your BufferQueue should have at least 2 buffers. (This requirement has since been relaxed. See this commit by Glenn Kasten. I'm not sure in which Android version this first appeared, but a guess would be 4.4).

  • You can't use certain effects (e.g. Reverb, Bass Boost, Equalization, Virtualization, ...).

The SoundPool class will also attempt to make use of fast AudioTracks internally when possible (the same criteria as above apply, except for the BufferQueue part).

Michael
  • 57,169
  • 9
  • 80
  • 125
3

Those of you more interested in Android’s 10 Millisecond Problem ie low latency audio on Android. We at Superpowered created the Android Audio Path Latency Explainer. Please see here:

http://superpowered.com/androidaudiopathlatency/#axzz3fDHsEe56

Patrick Vlaskovits
  • 194
  • 9
  • 2
    This is a good explanation, but not 100% up to date. The AudioFlinger buffering was reduced in L, removing a total of two buffers from the path. Currently there's one effective buffer of latency between the application and the HAL (it looks like more, but some of the code is synchronous--more memcpys doesn't always mean more latency.) Also, not sure where the 6ms "bus latency" figure is coming from. That seems way too high. – Ian Ni-Lewis Apr 05 '16 at 13:22
  • Hi Ian, Hope you are well. We have a more recently updated version here: http://superpowered.com/android-marshmallow-latency – Patrick Vlaskovits Apr 07 '16 at 15:19
  • 2
    That looks approximately correct. The one thing that's missing is the Nexus 9 DSP--it's called AHUB in the Tegra K1 datasheet. I'd guess that FIFOs there are accounting for more post-software latency than the bus is. I think you might also be overlooking the fact that the HAL write call is blocking. The call doesn't return until the next interrupt fires, so the entire engine is scheduled based on the audio interrupt, not on a rate monotonic scheduler. There are some serious issues with this, many of which are unique to Linux-based OSes, but I don't think a "push" model is one of them. – Ian Ni-Lewis Apr 11 '16 at 13:43
  • Hey Ian, Well, the GIF at http://superpowered.com/images/Android-Marshmallow-Audio-Path-Latency-Superpowered-Audio.gif shows the round-trip latency using a loopback connector with which none of the Tegra DSP is active. The HAL write call may block on some HALs and may not block on others. Because of the current "sleep-based" AudioFlinger architecture and other factors, AudioFlinger and the user application side (AudioTrack and co.) add more than 1 buffer to the round-trip latency. – Patrick Vlaskovits Apr 12 '16 at 19:33
  • "No DSP is active" -- you probably meant that OpenSL effects are off. Those effects are not the only thing the DSP does--it also handles mixing, routing, speaker protection, etc. "Sleep-based" --I can see why you'd say that, because one of the code paths in the mixer loop does contain a sleep. What's not obvious from the code is how infrequently that code gets called, because almost all HALs do blocking writes. We don't know of any that don't. That said, we have some data that implies that a HAL with async writes + sleep actually gets better latency than one with blocking writes. – Ian Ni-Lewis May 13 '16 at 20:09
  • Hey Patrick, I do think I need to walk back my statement about "push" just a bit. Android's underlying architecture isn't really what I'd call a push model, but I guess it's debatable. What *is* a push model, however, is OpenSL ES. You can't tell it how big you want your buffers to be (only how many of them), and it can't tell you how much audio it needs (only that it's consumed some). It's possible to approximate a pull model by getting all of the numbers just right and making some careful assumptions, but it's by no means easy or robust. – Ian Ni-Lewis May 13 '16 at 20:12
  • Hi Ian, Head's up on our latest. The Superpowered Media Server for Android. We're going sub-10ms roundtrip here. More here: http://superpowered.com/Superpowered-Android-Media-Server#AndroidProblemSolved – Patrick Vlaskovits Jun 16 '16 at 10:22
2

Another database of audio latencies and buffer sizes used:

http://superpowered.com/latency/#table

Source code:

https://github.com/superpoweredSDK/SuperpoweredLatency

Christopher Fraser
  • 511
  • 5
  • 6
2

There is a new C++ Library Oboe which help with reducing Audio Latency. I have used it in my projects and it works good. It has this features which help in reducing audio latency:

  • Automatic latency tuning
  • Chooses the audio API (OpenSL ES on API 16+ or AAudio on API 27+)
Wrobel
  • 1,042
  • 12
  • 21
0

Application for measuring sampleRate and bufferSize: https://code.google.com/p/high-performance-audio/source/checkout and http://audiobuffersize.appspot.com/ DB of results

Mickey Tin
  • 3,408
  • 10
  • 42
  • 71