Music Creation and Low Latency Audio on Android and iOS

Patrick Vlaskovits

Follow @Pv

With digital music creation and real time audio processing, the chief concern is almost always low latency audio. But exactly how low is low latency? In other words, what number constitutes acceptably low latency? Many assume this number to be zero, based predominantly on the false assumption that latency is an inconvenient byproduct of digital signal processing, something that does not exist with "real" instruments.

This, however, is not the case. Regardless of instrument choice, all musicians have to deal with varying degrees of audio latency.  In fact, latency is inherent in pretty much any system--—including the time taken for a sound wave to travel to the ear of the person who generated it.

For instance, the violin, which tends to be six inches from the player's ear, has roughly 0.5 ms of latency, whereas latency with harps, pianos, and drums can be as much as 3 ms thanks to the varying distances between the musician and the strings, hammers, and skins. The pipe organ represents an extreme case, in which the pipes may be up to 150 feet away from the player sitting at the keys, resulting in as much as 131 ms of latency for the lowest notes in the farthest rank.

Low latency audio in digital signal processing depends on a similarly wide array of factors—including operating system shortcomings and hardware considerations. Yet latency also varies quite a bit from system to system, with more powerful laptop and desktop resulting in much lower latency than mobile devices thanks to the aforementioned factors.

And though the skilled organist can work around the latency inherent to that instrument, that's a little harder to do with digital signal processing because it's isn't the same across device, operating system, and application. Furthermore, just because latency is a naturally occurring phenomenon, that doesn't mean musicians of all stripes don't strive for a tighter, more cohesive sound.

But the goal isn't zero latency, because there's no such thing. As a general rule for digital music creation, up to 25 ms constitutes an acceptably low level—a number that feels tight, cohesive, and natural to musician and listeners alike. But the best feel is achieved below 10 ms.

Low Latency Audio on Android and Issue 3434

Android latency issues on music apps are well-documented and are caused by Android’s audio stack simply not being capable of low latency audio output. Despite Google’s well communicated efforts (Google I/O 2012, 2013 and 2014), and improvements in Ice Cream Sandwich audio latency (Android 4.0) and Jelly Bean audio latency (Android 4.1), the so-called “fast mixer” in Android is still not able to reach the magic number of latency below 10 ms. The Google Code android sound latency issue has became a hotly debated piece of Android audio development history already.

Example Time Scale of System Latencies. pic.twitter.com/3o31b5J85b

— Anton (@PieCalculus) April 25, 2014

Google has yet to provide proper elevated scheduling for the audio callback. Although the fast mixer runs with the SCHED_FIFO policy, the priority should be in the 90s range, not 2 or 3. And it appears that the underlying HAL layer may need additional care too. In contrast, iOS has been able to do this since Day 1.

Furthermore, the “fast mixer” is available on only a few devices, it is missing in action even on high-end models devices such as the Samsung Galaxy family. Audio latency depends a lot on how the OEMs configure their particular Android flavor, and low latency audio doesn't appear to be a high priority for most of them. This is yet another Android fragmentation problem, and Google should really develop higher standards for the audio subsystem.

Samsung ported JACK2 as a workaround and made it available under the "Professional Audio SDK" moniker, yet only two Samsung devices support it. We hear that this is actually one audio enthusiast Samsung engineer's effort and as such, without strong organizational support, it doesn't perform like a serious audio solution.

In contrast, Apple's hardware-software symbiotic paradigm is required here to make provide a real low latency audio on Android solution with buffer sizes 128 or even less. For Android, this would require a SoC supplier, who can fine-tune the firmware and the audio drivers as well as providing a good Android low latency audio api instead of the OpenSL ES....

And then we'll have a low-latency audio in/out foundation to build upon.

Fortunately, Superpowered has everything on top of that.