How Android Mutes The Next Billions With Android USB Audio And Android MIDI

Gabor Szanto

Follow @szantog

In people’s pockets all around the world, there are more 3 billion super-computers that run on either ARM+iOS or ARM+Android. Android boasts 1.4 billion (30 day active) users, and Android’s share of the global smartphone OS market (share in unit shipments) is greater than 80%.

And on not a single one of those Android devices does either USB audio or MIDI -- two indisputable, keystone technologies for audio, music creation, production and performance -- ‘run’ without crippling problems.

On iOS, aside from orders of magnitude more music production apps than on Android, the burgeoning USB audio/MIDI hardware industry includes plug-and-play, class-compliant devices like USB sound cards, USB mixers, USB microphones, USB headphones, USB recording interfaces (guitar, voice), MIDI keyboards, MIDI button pads, MIDI controllers, MIDI-enabled instruments (such as MIDI wind instruments), synthesizers, effects, drumpads, mixing consoles, DAW workstations and DJ Controllers.

All of which hardware manufacturers would happily release to eager Android users, but can’t.

This means there are entire markets missing from Android. Markets with latent demand, waiting to explode. Markets that would materialize instantaneously on Android.

All because Android doesn’t handle USB audio and MIDI well.

Will the next billionbillions of Android users not get access to these technologies? Speaking of billions, will Google stop leaving billions on the table by not equipping Android devices with the core music and audio technologies that are part and parcel of iOS?

Will Android users remain muted?

Contents

• State of Android USB Digital Audio Support
• Android DVS (Digital Vinyl System) USB Audio Case Study
• iOS Core Audio vs. Android USB Audio vs. Superpowered USB Audio for Android
• Android USB MIDI
• iOS Core MIDI vs. Android USB MIDI vs. Superpowered USB MIDI for Android
• Superpowered USB Audio and MIDI SDK for Android

State of Android USB Digital Audio Support

USB Audio is an interface designed to allow audio recording and playback between analog peripherals and digital devices such as smartphones and computers. USB Audio hardware is often used to overcome the limitations of a digital device’s built-in audio, with professional audio features such as high audio quality, low latency, multiple channels and various analog audio connectivity options.

Native Android USB support was only introduced with Jelly Bean, Android 4.1 (API level 16) and USB Android audio performance is still severely limited, suffering from multiple issues such as lack of API, high audio latency, fixed bit depth and a limited number of sample rates as illustrated in the Android DVS (Digital Vinyl System) USB Audio Case Study below.

USB (Universal Serial Bus), is an industry standard for connection, communication, and power protocols between a host and its peripherals. For the purposes of this article focusing on mobile audio on Android, the host will be considered to be an Android device (smartphone, tablet or phablet) and peripherals are typically the USB hardware mentioned above.

Android has allowed Android devices to act as host since Honeycomb, Android 3.1 (API level 12). Most Android devices are manufactured with a micro-USB port, which itself does not allow for host operation. As such, an inexpensive ($5) USB On-The-Go (OTG) adapter is required for USB audio operation.

Common USB Android Audio Use Cases include:

• USB audio recording
• USB audio playback
• USB headset for telephony and VOIP

For each of these use-cases, Android automatically routes audio through the Android USB audio path. The Android USB audio path differs from the other audio paths we have described in Android Audio’s 10ms Problem and Rebooting the Android Audio’s 10ms Problem.

The default software USB Audio Stack on Android

1. USB audio device connected to the USB host controller
2. USB host controller driver
3. USB core
4. Linux USB audio driver (for class compliant devices)
5. ALSA kernel driver
6. Android HAL for USB
7. AudioFlinger (Android media server)
8. AudioTrack (OpenSL ES)
9. User Application

Android DVS (Digital Vinyl System) USB Audio Case Study

As a proxy for real-time musical instruments powered by USB audio, we wanted to create the first ever real-time Android DVS (Digital Vinyl System).

A DVS system consists of a DJ turntable playing a ‘control vinyl’ connected to a computer running DJ software via a USB sound card.

This is what DVS sounds and looks like on iOS with USB audio (using DJ Player App built with Superpowered Audio SDK).

The DJ ‘scratches’ the control vinyl back and forth on the turntable. Unlike a regular vinyl record that plays music, the control vinyl emits a single control tone. The control tone then directs the DJ software audio playback by direction (forward/backward), playback speed and position. What is exciting about DVS is that the physical (analog) scratching action makes for fun effects and a great performance with digital audio.

To bring DVS to Android, we connected a Behringer USB sound card to the Nexus 9 tablet with an OTG adapter. The Android audio stack recognized the sound card, but we ran into a number of problems soon after:

1. Android is incapable of telling the user application whether the audio is routed via the USB sound card or via the built-in microphone/speaker. But the DVS application depends on USB audio path and is useless with the built-in microphone/speaker audio path.
2. Because Android cannot tell us the current audio routing status, there is no programmatic method to tell the user to connect a USB sound card. (In comparison, the iOS USB API allows for this.)
3. Android only reports the native sample rate of the built-in audio device, but not the USB sound card available sample rates. Meaning that Android cannot determine programmatically which sample rate to use for low latency audio performance.
4. Without the correct sample rate, Android’s audio stack may refuse “Fast Mixer” mode, which is essential for DVS (~10 ms round-trip audio latency is the target for real “turntable-like” operation).
5. Android cannot report on what buffer size to set either. (Again, the iOS USB API allows for this.) Since USB audio is transmitted in 1 ms chunks and our sound card is able to operate at 48000 Hz, which matches the native sample rate of the Nexus 9, we manually tested multiples of 1 ms, such as 48, 96, 144 etc. samples.
6. The results were marred by audio glitches, despite we had confirmation of “Fast Mixer” mode.
7. Glitch-free operation was achieved with 480 samples. However, this made round-trip latency unacceptably high (+30 ms). Please note, this was on an untouched, vanilla Google Nexus 9 system with Android Marshmallow 6.0.

Cutting to the chase, the Superpowered USB Audio and MIDI SDK for Android solves all of the above problems. In the case of the Android DVS, it provides glitch-free low latency audio with the very same sound card and the Nexus 9. The round-trip latency is 11 ms with this setup.

Routing, sample rate and buffer size are properly reported and configurable by the Superpowered USB Audio API and all of these features will now be available to +1.1 billion Android devices.

What happens when you connect a USB audio device for playback and/or recording to an Android device?

The Android device may or may not have the appropriate audio driver to handle USB audio. If the device has a USB audio driver and Android’s media server (AudioFlinger) is configured to handle it, then audio can be managed by the Android media server.

It has several severe limitations though:

• It’s impossible to configure it via OpenSL ES for consistent, glitch free, low latency audio on most devices.
• Audio is limited to 2 channels (stereo). For example, if the USB audio device has multiple channels, such as a sound card designed for DJs, then only a single stereo output will work, which makes it effectively useless.
• Audio is limited to 16-bits. For example, if the sound card is designed for 24-bit audio recording for musicians, the user will not be able to fully utilize the sound card’s capability for high quality.
• Audio is limited to maximum 48000 Hz. For example, if the sound card is designed for 96000 Hz playback or recording, the user will not be able to take the advantage of higher audio quality.

Comparison between iOS Core Audio USB Audio, Android USB Audio and Superpowered USB Audio for Android

Beyond the above tabular representation of the facts, what really matters is how do we empower the creative output of Android users?

What will USB MIDI on Android mean to most of the world?

Android USB MIDI

MIDI is the industry standard set of communication protocols that allow digital musical instruments to communicate with one another and with other digital devices to transmit musical information (eg notation, pitch and velocity, volume, vibrato, audio panning, cues, and clock signals). MIDI is the equivalent of the musician's’ keyboard and mouse.

MIDI is used mostly for live performances, such as musicians playing on a keyboard (piano) or DJs on controllers. Low latency and low jitter (time variation) is paramount for pleasing live performances.

What happens if you connect a USB MIDI device to an Android device?

Currently, only 7.5% of all active Android devices will recognize the MIDI device and handle it appropriately (Android 6.0 and up), the rest -- 92.5% of all Android devices -- will simply ignore it.

Even if the Android device has Marshmallow installed, the MIDI package by Google is implemented in Java, which hampers real-time communication. The Android MIDI suffers from scheduling problems with the Android Runtime (the virtual machine running Java), and Java’s garbage collection. Even if garbage collection is significantly improved in Android, it still may stop all execution for a few milliseconds, delaying the next stroke on the piano for example.

Comparison between iOS Core MIDI, Android USB MIDI and Superpowered USB MIDI for Android

What would making 1.1 billion Android devices MIDI-compliant sound like?

It might sound like this:

Superpowered USB Audio and MIDI SDK for Android

The Superpowered USB Audio and MIDI SDK for Android will fully take over MIDI device handling with none of the limitations described above, providing low latency and low jitter (below 2 ms) access to MIDI devices for 1.1 billion Android devices, that is, 75.6% of all Android devices.

It’s the Android equivalent of iOS Core Audio and Core MIDI for USB devices.

• Fully built-in to the user application (your application).
• No special permissions (such as rooting) are required. There is no additional installable component. It doesn’t install anything.
• I/O is fully independent of the Android Runtime (and Dalvik), with no Java dependency.
• Entirely independent of the Android audio and MIDI stack. It takes over the attached USB audio and MIDI devices.

The Superpowered USB Audio Stack on Android

1. USB audio device connected to the USB host controller
2. USB host controller driver
3. USB core
4. Linux USB user space communication (devio.c)
5. Superpowered USB Audio library
6. User Application

The Superpowered USB Audio and MIDI SDK provides stable, professional, low latency, low jitter, multi-channel audio and MIDI functionality for more than 1 billion active Android devices, letting Android users fully experience and express the sonic power of that Android powered super-computer in their pocket unmuted.

How do I get access to Superpowered USB Audio and MIDI SDK for Android?

The Superpowered USB Audio and MIDI SDK is part of the Superpowered Audio SDK. Download it today.