Rebooting Android's 10 Millisecond Problem: Audio Latency Improvements in Android 6.0 Marshmallow

Gabor Szanto

Follow @szantog

Superpowered Audio superpowers over 250 million app-installs across iOS and Android. Our audio latency data is the most up-to-date, open-source and freely available data.

Contents

• Android 6.0 on the verge of Pro Audio
• What's changed in Android 6.0 Marshmallow?
• Android's Four Main Audio Signal Paths
• The Major Shift: The Android Professional Audio Flag
• Android MIDI
• Why the Low Latency Flag is Unimportant
• Updates to Samsung Professional Audio SDK ("SAPA")
• Updated Superpowered Latency App

On the Verge of Pro Audio

[For practical tips on dealing with latency, Android developers please also see Android Audio Low Latency Primer.]

[UPDATED AS OF JUNE 2016: Please see Android's 10 ms Problem? SOLVED.]

Since we published "Android's 10 Millisecond Problem: The Android Audio Path Latency Explainer" in early 2015, Google has made tremendous strides in improving round-trip audio latency on Android OS. With the deployment of Android 6.0 Marshmallow, the Huawei Nexus 6P clocks in at a much improved 18ms round-trip audio latency and the HTC Nexus 9 at 15 ms.

As readers may recall, 10ms round-trip audio latency is the threshold that must be met by Android to be considered truly 'pro audio'.

One second of round-trip audio latency sounds like an eternity. 250 ms is still terrible. 100 ms is 10% of one second, and still sounds unacceptable. At 25ms, things begin to approach acceptable ranges. Unless you are Terence Fletcher, the nightmare-ish jazz instructor from the movie Whiplash, 10 ms should be perceived as instantaneous.

Today, the HTC Nexus 9 is not quite at 10ms, but at 14ms demonstrates incredible progress in enabling Android to become a true 'pro audio' platform.

Want to embed this image? Copy the following code:

What's changed in Android 6.0 Marshmallow?

Improvements in Android system-wide thread scheduling in the recent Android versions have allowed Google to finally decrease the buffer size settings in the audio drivers for the Nexus 9.

Using smaller buffer sizes means that the audio signal is composed of smaller, more frequent packets - which when compared to an audio signal made up of larger and less frequent packets - means lower latencies.

In our previous article, we discussed that the the Android Audio Hardware Abstraction Layer ("HAL") implementations, the component between the audio driver and the media server, are often poorly implemented across the the Android device landscape. Google has also ensured that the HAL has been implemented properly for the most recent Nexus devices.

However, the Android media server itself does not look like it has been significantly improved from Android Lollipop to Android Marshmallow. While it was already good in Lollipop, it appears as if Google is now hitting the hard limits of the media server’s "push" mechanism.

We have no doubt, it took heroic efforts by Google’s team in rate monotonic thread scheduling to make the push mechanism work with low buffer sizes.

Yet still, the OpenSL ES layer "pushes" audio down into the audio stack -- when it should be ‘pulled’ -- this is an unfortunate artifact from Android’s early days as the creators of Android did not imagine real-time pro audio use cases. A "pull-mechanism" media server would work much better for dropout-free, low latency audio, where the interruptions of the audio driver stack direct the flow. In other words, the Android media server is still reliant on push, meaning this is a show-stopper for uses-cases where audio must be flawless, such as live stage performances eg, a guitarist might want to use a real-time distortion effect.

Other major improvements include the Professional Audio Flag as well as MIDI support. Covered below.

What do I need to know?

To stay up-to-date with the latest Android audio latency improvements, you should understand the following points moving forward.

The four main possible audio signal path options are:

A casual user might think each audio path returns a similar result, however, this is not the case. Below we tested a few common devices to demonstrate the differences in latencies between audio paths.

Additional audio latency on the built-in microphone

Almost every Android device has additional processing for the built-in microphone for better audio quality, better separation between the speaker and background noise, and to reduce echo.

Our latency measurement data shows that in most cases, this processing is bypassed if audio input comes from a source other than the inbuilt mic.

That’s why latency measurements with a loopback connector (dongle) or USB audio show lower latencies. Google’s audio latency measurements are all taken using an audio loopback dongle.

We’ve improved our latency measurement app to report the headphone status, so we provide data for other, "more natural" cases as well.

Detecting if something is plugged into the headphone jack

Starting from Android 5.0 (Lollipop, level 21) there is now an API to detect if something is in the headphone jack. There was no reliable way to detect this in previous Android versions. However, there is still no way to get precise routing information, such as what exactly is plugged in (just a headphone? a mic?). Currently, an Android app can not reliably detect which signal path is active, unfortunately.

Built-in microphone processing can be turned off...

...on devices supporting that feature. We don’t know exactly which devices support this, but the Nexus 6P and 5X do.

Check for SL_ANDROID_RECORDING_PRESET_VOICE_RECOGNITION when you set up OpenSL ES, or use our SuperpoweredAndroidAudioIO class for an easy implementation.

iOS has this as well, AVAudioSessionModeMeasurement mode should be used. Our latency measurement app automatically disables built-in mic processing for the lowest audio latency, on both iOS and Android.

Note: The difference in latencies observed on the iPad Air 1 may be due to different audio hardware.

The Android Professional Audio Flag

This flag portends a major shift for the better. Starting with Android 6.0 (Marshmallow, level 23), devices may report "professional audio level of functionality and performance" with the FEATURE_AUDIO_PRO flag.

Developers should expect proper Android HAL implementations and good latencies if this flag is enabled. More precisely:

• 20 ms or less round-trip audio latency over the headphone jack path.
• Compatibility with USB Audio Class Compliant audio devices (sound cards).
• Great multi-channel HDMI output.
• Android MIDI compatibility.

We’ve improved our latency measurement app to report this as well. We’ll see how many devices will support this.

Android MIDI

Yes, Android 6.0 (Marshmallow, level 23) finally has a CoreMIDI rival. However, having this in the native layer would be even better, as routing MIDI through Java is suboptimal. Why?

Because some events will be received too late as Java may be busy with other stuff, such as other Android features or garbage collection. Having low latency and low jitter (!) is very important for a musician playing on a MIDI instrument.

As professional audio must be implemented in the native layer, professional Android MIDI would be better in native as well.

The Low Latency Flag

Since Android 2.3 (Gingerbread, level 9), devices may report the FEATURE_AUDIO_LOW_LATENCY flag. Since the recommendation here is for 45 ms output latency (of a 50 ms round-trip) which is much too great, our latency app doesn’t collect information about this flag.

Well, what about the Samsung Professional Audio SDK ("SAPA")?

We have reached out to Samsung, and they have informed us that they will be releasing Samsung Professional Audio SDK 3.0 with their new flagship phone sometime in 1H 2016. The details will be announced at SDC in April.

In addition, they have expanded SAPA available devices to include some mid-range devices and tablets. As of 12/20/2015, the SAPA-enabled devices are listed below.

Phones:

• Galaxy S5
• Galaxy S5 Neo
• Galaxy S5 Wideband LTE-A
• Galaxy S6 Active
• Galaxy S6 Edge
• Galaxy S6
• Galaxy S6 Plus
• Galaxy A9 (2016)
• Galaxy A7 (2016)
• Galaxy A5 (2016)

Note Line:

• Galaxy Note Edge
• Galaxy Note 4
• Galaxy Note 5
• Galaxy Note 10.1 (2014E) Wi-Fi
• Galaxy Note 10.1 (2014E)
• Galaxy Note 10.1 (2014E) LTE

Tablets:

• Galaxy Tab S2 (9.7)
• Galaxy Tab S2 (8.0)
• Galaxy Tab S2 Wi-Fi (8.0)
• Galaxy Tab S2 Wi-Fi (9.7)
• Galaxy View
• Galaxy View Wi-Fi
• Galaxy Tab S Wi-Fi (10.5)
• Galaxy Tab S (10.5)
• Galaxy Tab S Wi-Fi (8.4)
• Galaxy Tab S (8.4)

KitKat version of Galaxy S5 and Galaxy Tab S are not supported. Basically, the latencies with SAPA depends on SOC and the internal signal path. As a rule of thumb:

• Qualcomm: 26 ms
• SAMSUNG Exynos: 17-18 ms
• SAMSUNG Exynos + S/W Mixer: 27-28 ms

Latencies were measured using Google’s Audio Loopback Dongle, buffer size was 240 (low latency setting).

Some devices apply auto-normalization to the input signal from the built-in mic that causes 7 to 8ms additional latency.

Exynos's H/W mixer has insufficient number of ports for handling all the internal audio paths including professional audio, and the S/W Mixer with buffers (total 10 ms) remedies this.

Unlike Exynos, Qualcomm has its own DSP which has sufficient audio signal chains. However, the Qualcomm DSP adds 2 to 3ms latency at the minimal latency setting.

Superpowered Latency App update

As such, we have updated the Superpowered Latency App with the following features:

• New reporting features: Pro Audio flag, headphone jack, Android build ID.
• SAPA can disabled so that developers can test OpenSL ES latency on Samsung Galaxy devices.
• Data send to Superpowered server is opt-in.
• Testers can override buffer sizes manually to check whether the buffer size recommended by AudioManager is correct.

The results table has also updated to show a better representation with the new data.

Download the Superpowered Latency App or see the results for 2800+ different mobile devices here.

iOS 8
and above   Android 3.0
and above