What Developers Can and Cannot Do to Lower Android Audio Latency: The Android Audio Low Latency Primer to Android’s 10ms Problem.

Gabor Szanto

Follow @szantog

To best understand the origins (and fixes) of Android audio latency, it is best to segregate contributions to total round-trip Android audio latency into two parts:

1. Audio latency in Android user space. In userland, developers have many methods to lower latency. They are described below.
2. Audio latency stemming from Android system space. In system space, developers cannot affect latency, but OEMs and device manufacturers can. Options to lower audio latency originating in Android system space are also described below.

Latency is the lag or delay between between when an audio signal is sent, processed and received. Audio latency is considered professional-grade or ‘real-time’ when round-trip latency is 10 ms or less.

Audio latency in Android User Space

This includes the Android app itself, application frameworks, and 3rd party and native API and libraries such as OpenSL ES. Developers can should remember low latency best practices in Android app programming logic such as:

1. Do not do any audio processing in Java. Keep all audio processing in C++.
2. Use the JNI (Java Native Interface) for successful interoperability between native (C++) code and Java. (Prepare yourself, this is not necessarily fun).
3. Always make your code sample-rate and buffer size agnostic. Your app should work with any sample-rate or buffer size. Do not expect buffer sizes to be a power of 2, they can be any multiples of 8.
4. Do not depend on audio features offered by Android because Android audio feature implementation/function is erratic, unpredictable and may be device dependent.
5. The CPU governor reduces CPU frequency and may cause audio drop-outs. We once advised developers to use 'fake touch events' to keep the CPU governor from activating. However, this only works on Nexus devices. SuperpoweredCPU is a universal solution for Android CPU Frequency Scaling and is now included in Superpowered.
6. Keep in mind that there are multiple, different audio signal paths and each has its own specific latency.
7. Using Bluetooth audio will add minimum 100ms to latency.
8. Wifi will also add latency. Highly variable, depends on implementation and needs to be measured on site. Can range from 40 ms to several seconds.
9. For audio processing, do not use OpenSL ES ‘wrappers’ or OpenSL ES itself. Use self-contained android low latency audio API in static libraries like Superpowered Audio SDK.
10. Use OpenSL ES for audio I/O only.

How do I know any Android device’s round-trip audio latency?

Android audio latency is not device-specific or tied to specific hardware.

The round-trip latency on a HTC Nexus 9 LTE running Android 6.0.1 (MMB29R 2482564) is 14ms, while the same device running Android 5.1.1 (LMY47X 1849464) clocks in at 42 ms round-trip latency. The same device may exhibit different latencies depending on Android version and build as well as carrier specific modifications in the HAL (hardware abstraction layer), audio driver, and/or audio hardware settings.

To help developers determine device latency, Superpowered maintains a free, open-source iOS and Android audio latency test app as well as continuously updated public database of latencies and native buffer sizes for Android devices.

All things being equal, the lower the buffer size, the lower the latency. However, all things are rarely equal in Android.

The latency test app also records the presence of hardware feature flags:

• android.hardware.audio.low_latency indicates a continuous output latency of 45 ms or less.
• android.hardware.audio.pro indicates a continuous round-trip latency of 20 ms or less.

How does the Superpowered iOS and Android audio latency test app work?

Our latency test app measures the total round-trip latency of every component of the entire audio chain.

NOTE: the Superpowered audio latency test app for Android and iOS does not use Superpowered, and in fact, does not do any audio processing whatsoever. It measures the theoretical minimum round-trip latency of any device by measuring audio in and audio out. The latency data-table displays the best (lowest) measured value, if the dispersion is within 2x. We display a latency range if dispersion is outside of 2x.

For full transparency, the latency test app's source is open and available on GitHub. We use the same code on all Android and iOS devices, and it works as follows:

How one measurement step works:

1. Listen and measure the average loudness of the environment for 1 second.
2. Create a threshold value 24 decibels higher than the average loudness.
3. Begin playing a 1000 Hz sine wave and start counting the samples elapsed.
4. Stop counting and playing if the input's loudness is higher than the threshold, as the output wave is coming back (probably).
5. Divide the the elapsed samples with the sample rate to get the round-trip audio latency value in seconds.
6. We expect the threshold exceeded within 1 second. If it did not, then fail with error. Usually happens when the environment is too noisy (loud).

How the measurement process works:

1. Perform 10 measurement steps.
2. Repeat every step until it returns without an error.
3. Store the results in an array of 10 floats.
4. After each step, check the minimum and maximum values.
5. If the maximum is higher than the minimum's double, stop the measurement process with an error. It indicates an unknown error, perhaps an unwanted noise happened. Double jitter (dispersion) is too high, an audio system can not be so bad.

The latency test app does not distinguish between input and output latency. In theory, input latency and output latency are symmetrical, however devices may have an uneven split, due to different period sizes and period counts. Furthermore, many devices feature built-in mic processing to prevent echo, suppress noise and the Larsen effect (feedback).

Lastly, the latency test app only tests for one of the four main audio signal paths. The four main possible audio signal path options, that can and do vary in latency on the same device, are:

A casual user might think each audio path returns a similar result, however, this is not the case. Below we tested a few common devices to demonstrate the differences in latencies between audio paths.

What about Android USB audio and Android MIDI audio latency?

This article is for readers interested in Android USB audio and Android MIDI.

How should developers work with OpenSL ES?

OpenSL ES is the standard audio API for Android apps. Android mandates that all audio in and audio out pass through OpenSL ES. However, it does not mandate that apps use OpenSL ES to process audio in the app. This allows developers to use 3rd party C++ audio libraries such as Superpowered for low latency audio processing, which radically reduces latency as compared to audio features offered by OpenSL ES. OpenSL ES feature availability also varies wildly between the different devices.

If I use Superpowered to process audio and then send it back it out through OpenSL ES, is Superpowered a wrapper?

No. Wrappers are an abstraction layer above and around the audio features of an operating system and built on top of the existing audio environment. Wrappers attempt to make it easier to interact and program for audio APIs but have no impact on the fundamental technological capabilities such as latency, audio quality or power consumption.

For example, FooAudioWrapper might allow you to interface with OpenSL ES in less complex ways but cannot deliver better actual audio performance because it still processes audio in OpenSL ES.

With Superpowered, a true audio ‘engine’, every Superpowered feature is implemented via the Superpowered library. Therefore, you not only get better performance, but since it is cross-platform, you get the same features, processing speed and audio quality regardless you run it on iOS or Android.

Audio latency in Android System (Kernel) Space

[To be more precise, Android has a system space while Linux has a kernel. The audio driver is a Linux system component, while the HAL is an Android system space component.]

For the purposes of understanding audio latency reduction, Android system space is where all the system drivers reside. These cannot be modified directly by app developers.

Of most interest in the Android audio path and its contributions to latency in system space are (in no particular order) the ALSA audio driver, the HAL (hardware abstraction layer) and AudioFlinger inside the Android Media Server.

One can find a detailed breakdown of the Android audio path in Android 5.0 Lollipop as well as details on changes in audio latency and audio path in Android 6.0 Marshmallow.

What are the fundamental contributors to Android audio latency in system space?

There are many misconceptions as to the root cause of latency in Android system space. They are typically thought to be hardware related, Linux audio related or ALSA driver related.

Our research suggests that the root cause lies in how the ALSA and AudioFlinger communicate. Specifically, AudioFlinger ‘pushes’ audio buffers to the ALSA. ‘Pushing’ causes scheduling problems, which cause audio quality problems. All pro audio, low latency systems use the pull mechanism as does the Superpowered Media Server for Android.

If Android system space latency is non-trivial, what should developers do?

While developers cannot affect Android audio latency directly, they can lobby OEMs and device manufacturers to license the Superpowered Low Latency Media Server for Android. The Superpowered Media Server is injected into system space and radically improves audio latency performance.

How does Superpowered offer low latency audio performance?

The heart of Superpowered technology involves patent-pending innovation in algorithm selection and optimization for audio processing on low-power mobile devices. Superpowered technology use fewer CPU clock cycles which means not only is processing faster, but because it uses fewer clock cycles, it uses less power as well -- allowing a mobile device to run other non-audio related tasks concurrently.

It’s like taking a VW Bug, adding technology to it, which would make it accelerate like a Porsche and get the fuel efficiency of a Prius at the same time.

The physics of automobiles won’t actually allow you do that -- but the physics of digital signal processing do.

More questions you’d like to see answered here?

Send an email to androidlatency@superpowered.com.

Superpowered products and services include:

• Superpowered Audio SDK
• Superpowered HLS (HTTP Live Streaming) for Android and iOS
• Superpowered USB Audio and USB MIDI SDK for Android
• Superpowered Paid Support
• Superpowered Media Server for Android
• Superpowered 3D Audio and Spatializer for iOS, Android & Unity

Professional audio on Android just got Superpowered.