Android Audio Latency Problem Just Got Worse

Gabor Szanto

Follow @szantog

Oh no, Android 8.0, say it ain't so!

Being the bearer of bad news isn't pleasant -- but with Android 8.0, Android's 10 ms Problem is worse than ever. You read that right: Android 8.0 Oreo has worse latency than Android 7.0 Nougat.

If you're building interactive audio applications that will be embedded in an Android substrate: AR/VR, voice assistants, VOIP apps, conferencing apps, virtual instruments, DAWs, karaoke apps to name a few: the performance of those sorts of apps will be impacted negatively.

TLDR

Our testing shows that Android 8.0 has demonstrably and materially worse round-trip audio latency performance than previous Android versions, when audio input is not mono. We've shared our findings with Google's Android audio team and updated our latency test app. Data and methods below.

Update for Nov 2nd 2017

A Developer Advocate at Android Audio has responded to our claims, our response is here.

Table of Contents

• Technical Background
  Read this if you care about the boring technical details on how we discovered and fixed things.
• Android 8.0 Is Not There Yet
  The more interesting part begins here.
• The Path Forward Remains Unclear
  Business folks check this.

Technical Background

A few weeks ago a member of the Android audio team notified us that our Superpowered Audio Latency Measurement app was behaving strangely on Android 8. The variance between the latency measurement steps was too great, and the latency results were also higher than Google's own measurements.

There are two apps to measure round-trip audio latency on Android. Google's own "Rick'O'Rang app" (source and APK) and Superpowered's Latency Test (source and APK). Both are fully open-source. Their respective measurement methods are different, but the results have always been similar, in fact, virtually identical.

Google's Rick'O'Rang app requires a special hardware dongle known as the Loopback Connector to be connected into the headphone plug. The app emits a signal, which loops back via this connector, causing audio feedback (the "Larsen Effect"). The app calculates the round-trip audio latency from the periodicity of this feedback. It also measures "audio continuity" to detect glitches.

The Superpowered Latency Test app works without any special hardware. It can work using the built-in speaker and microphone, a USB sound card, the Loopback Connector or any other audio device. The app emits a signal on the audio output, then measures the elapsed time until it's received back into the audio input. This process is repeated 10 times to filter out dispersion and other factors.

It bears restating: the results of both apps have been historically similar, virtually identical. In this way, they were a natural check-and-balance on one another.

Until Android 8…

Putting some log messages into our app revealed the main anomaly on our Nexus 6P phone. As a part of Google's efforts to improve audio latency, the required number of input and output buffers on the app side was only 1 (one), between Android 4.4 and Android 7, using the OpenSL ES audio API. Input and output callbacks filling and consuming these buffers monotonically followed one another:

IN, OUT, IN, OUT, IN, OUT, …

But the Nexus 6P (with Android 8) behaved wildly differently:

IN, IN, IN, IN, OUT, OUT, OUT, OUT, IN, IN, IN, OUT, OUT, OUT, OUT, …

The number of consecutive inputs or outputs fluctuated between three and five. This behavior is unexpected and abnormal. In fact, so unexpected, that both the Google and the Superpowered latency apps were using only 1 buffer. But one buffer is not enough to cover five, therefore all measurements become erratic and inaccurate.

The fundamental impact of non-monotonic scheduling of input/output callbacks is that it increases round-trip audio latency since an app is unable to react to an input buffer with an output at the next callback, since the next callback may be an input again.

Before Our Fix of the Superpowered Latency Test App

Our app displayed a latency value fluctuation. Every Android 8 user could immediately see that something is wrong - yet Google's app displayed good, plausible, and moreover, low latency values.

Although Google's glitch measurement feature suggested that something was wrong, many users simply overlook this, as glitch detection is not as "exciting" as latency measurement (it's a separate feature). The number of glitches should be zero, but it's two or three in every case now (under Android 8, on the Nexus 6P).

After Our Fix of the Superpowered Latency Test App

We have modified the Superpowered Latency app to be able to handle the anomalous buffering, up to 128 buffers (which is extremely high). Our implementation is not Android 8 specific. Our goal is that it works on every Android version without any adjustment.

We've carefully tested it in previous Android versions on multiple devices to make sure that it still provides the same, trusted values on those.

During the past few years our latency app has collected the largest mobile audio latency database in the world with more than 320,000 measurements on almost 8,000 different device models/builds, so we've had plenty of data to benchmark against.

Google's Fix?

We've sent our findings to the author of Google's Rick'O'Rang app. We'll update this article when the fix for Google's app is ready.

What happened to the fast flag?

An important feature of the low latency audio efforts is the so-called "Android fast mixer". If an app chooses the fast mixer audio route (and the device supports it), then the audio latency is much, much better. The Nexus 6P happily did everything on the fast path until Android 8. But now we see this:

The fast mixer is denied for the audio input. The fast mixer is available for the output with AAudio only, but the frame count is too high, equivalent to the "non-fast" frame count.

It is worth noting, AAudio doesn't go through the "fast mixer" in exclusive mode (we're using that), but talks to the HAL directly.

Android 8.0 Is Not There Yet

So we fixed our latency measurement app and everybody was curious, how much better is Android 8.0 now? Uh-oh, it isn't better. It's worse…

It's clear that something has happened in the bowels of the Android audio stack.

The new AAudio API

Android 8 comes with the new API, AAudio, which is much better than the older OpenSL ES API. It's easier to understand, it's easier to work with and it provides tighter, shorter code as well.

The announcement of this API helped create hype around Android 8's better audio capabilities. It also comes with the promise of lower latency. Let's see how it performs currently:

Unfortunately, no difference in audio latency can be measured on the Pixel XL (between OpenSL ES and AAudio). On the Nexus 6P the situation is this:

On the Nexus 6P you get much better audio latency with Android 7, regardless of which API is used on Android 8.

UPDATE: use mono!

After informing Google about these measurements, they notified us about an Google-internal bug. If the input channel count is reduced to 1 (mono), the fast flag is approved for the audio input, and latency improves:

Important to repeat: this workaround works with mono input only. No improvement over Android 7 still.

Google is trying to convince us that mono is a good solution, but we think it's not. Mono input is not an option for several apps (such as multi-track recording or DVS DJ), but more importantly: really???

AAudio and Older Androids?

If you simply build with the AAudio library, your app will crash on older Android versions. There is no "official" way published on how to create apps which are compatible with older Android versions yet.

In the Superpowered Latency Test app we show a way to create apps compatible with older Android versions. Our method also saves you from having to maintain two distinct native libraries and code bases.

The solution in a nutshell: although CMakeLists.txt will create two distinct libraries (one with and one without AAudio), the source for both will be the same.

AAudio Buffer Size Warning

If you develop for AAudio, please keep in mind that the buffer size for each audio callback is not constant. On the Nexus 6P it typically looks like this sequence: 192, 128, 64, 192

This is not unknown for audio driver developers, but for regular app developers it may be shockingly unexpected.

Superpowered Recommendations and Takeaways for Android Audio Developers

The Technical Background section shows that there is a serious issue with OpenSL ES under Android 8. This is our recommendation for you:

If you wrote the "1 buffer" OpenSL ES thing for audio I/O, then you should update your app immediately, because it outputs garbled audio under Android 8. It receives garbled audio too.

If you used SuperpoweredAndroidAudioIO for audio I/O, then you are completely safe. It has a mechanism inside which is not affected by the issue. It just works.

Hold off with AAudio until we see how it works on non-Google devices with Android 8, and how Google addresses the latency issues on the Nexus 6P and Pixel devices.

As of October 2017, SuperpoweredAndroidAudioIO will not support AAudio, but we're following the situation closely, and if it improves, we'll make it happen.

SAPA is dead

We interrupt our broadcast for a moment. We've removed SAPA (Samsung Professional Audio SDK) support from the Superpowered Latency Test app, because even it's latest update doesn't support 64-bit.

Maintaining a separate codebase and library for 32-bit devices with SAPA support is not worth the effort anymore. Fact, that the effort was too high for most developers anyway.

SAPA was an interesting try by Samsung to solve Android's audio latency problem, but unfortunately it never really took off. We've tried to support it as long as we could. Vaya con Dios old friend!

The Path Forward Remains Unclear

Google has been trying to satisfy pro audio developers for a long time now. An announcement has been regularly made every year from 2011 onwards.

The Music Category in Google Play is much further behind the Music Category in the App Store. Yes, Google Play revenue is generally lower than the App Store, but the Music Category in Google Play is much much lower than expected. The technological problems of Android audio are prime contributors to this.

Most of us has the hope that things gradually progress forward. That some day it becomes economically worthwhile to create pro audio apps for Android, compared to iOS. Naively, we imagine it like an exponential curve in our head:

In other words, most entrepreneurs are thinking like:

The situation is getting better, the number of pro audio Android devices are accumulating to the point where it makes sense for me to do something in Android pro audio. I'm closely watching how it progresses.

But we get this instead:

Android 7.0.1 had a serious bug in CPU scheduling, making processing-heavy audio apps produce heavy audio dropouts in background mode. And now we get Android 8, which significantly increases audio latency on some devices. Bringing us to:

"The situation is getting better in general, but the major hiccups on the road are preventing the accumulation of pro audio Android devices to the point where it makes sense for me to do something in Android pro audio. Unfortunately, many Android devices are not getting updates past some point. The devices stopping at or updated to Android 7.0.1 or Android 8 have been lost. I'm closely watching how it progresses.

We'll continue helping Google with everything we can. We'll continue providing pro audio tools to make pro audio Android projects happen. Our sincere hope is that Android matches iOS audio performance some day soon, giving tools and opportunity to the billions using this platform.