Problems/Suggestions : Unity Forum, Discord, or mail to 5argon@exceed7.com

Native Audio

Lower audio latency via OS's native audio library.
Requirement : Unity 2017.1+ For iOS and Android.
(Technically should work with all versions since this is almost pure native, but I provides support for only 2017.1+)

Latest : v3.0.0 (01/11/2018)
Demo : APK | Changelog : Click here

So your Unity game outputs WAY slower audio than other apps even on the same device? Turns out, Unity adds as much as 79% of the total latency you hear. But not to worry because Native Audio will take care all of it. I have researched into the cause for so long and the solution is right here.

Unity adds audio latency

This plugin was developed when I noticed that both my Android AND iOS game has a considerably worse audio latency than other music apps such as Garage Band installed on the same device. I have confirmed by creating a basic Unity app which just play an audio on touch down vs. a basic Xcode iOS app/Android Studio app which also plays an audio on touch down. You can clone the project on GitHub to confirm this by yourself.

If you can't feel the difference, I encourage you to do a simple sound wave recording test. Keep your phone at the same distance from your computer's mic and tap the screen nail-down. The sound wave interval between the peak of nail's sound and the peak of response sound should immediately reveals the difference visually. Newer device might have a better latency, but the difference between Unity and native app should be relative for every devices.

This perceived result is because of 2 things : touch input latency AND audio latency. Native Audio's job is to greatly reduce the audio latency.

What can we skip by going directly to the native API?

Unity has an internal mixing system, backed by FMOD. You can go crazy with a method like AudioSource.PlayOneShot and the sound magically overlaps over itself,even though at native side Unity only asked Android for only 1 "Audio Track". How can that be? Because Unity spend time mixing them together to 1 bus. Plus you have all the wonders of Unity 5.0.0 introduced audio mixer system. Effects, sends, mixers, etc. all stacked onto that.

A great design for a game engine. But we certain subset of game developers absolutely do not want any of the "spend time" if possible. Unfortunately Unity does not give us a choice to bypass and just go native.

For some genre of apps and games that needs critical timing for audio (basically any response sound from input, rhythm game, etc.) this is not good. The idea to fix this is to directly call into the native methods and have it read the raw audio file we prepared without Unity's importer. Bypassing Unity's audio path entirely.

Also I suspected Unity have to collect all audio commands to summarize them together at the end of each frame. Selecting a number of loudest AudioSource to mix according to audio settings, mix with the mixer system introduced in Unity 5.0 which supports various effects and many channel/buses/sidechain, then send the mixed 1 audio stream to native.

While with Native Audio, after native calling there is no concept of "frame" anymore. It will just play audio right there in the middle of the frame! (The hardware might wait for something, but that is already outside of Unity's domain)

I have researched for long time into the fastest native way of each respective platform. For iOS it is to use OpenAL (Objective-C/Swift) and for Android it is to use OpenSL ES (NDK/C). For more information about other alternatives why they are not good enough, please go to implementation page.

But having to interface with multiple different set of libraries separately from Unity is a pain, so Native Audio is here to help...

Native Audio is a plugin that helps you easily loads and plays an audio using each platform's fastest native method, from the same common interface in Unity.

Android High-Performance Audio Ready

It improves latency for iOS greatly, but I guess many came here to fix the already-horrible-without-Unity Android latency.

I am proud to present that Native Audio is following all of Google's official best practices required to achieve High-Performance Audio in Android. Because Native Audio is not as versatile, I can additionally fix all the latency-related tradeoffs that Unity have to pay for their more versatile audio engine.

Of course, with publicly available thorough research and confirmations. This means it can perform even better than native Android app that was coded naively/lazily in regarding to audio. Even pure Android developers might not want to go out of Java to C/NDK.

How faster?

We will do a measurement by placing the device in fixed position, recording the peak of nail sound touching the screen to the response sound produced by the app. While this is not a standard latency measurement approach like loopback cable method and the number alone is not comparable between devices, but time difference of the same device truly show how much the latency decreased without doubt.

The pure Unity's audio was measured with "Best Latency" settings on Audio Settings, the fastest playing (smallest buffer) Unity can achieve without any plugins. At first I wasn't annoyed by Unity's latency that much, but after a clear contrasting with Native Audio (especially on Android) the latency reduction is insane. It is surprising how I was able to live with that bad latency when there's nothing to compare with.

Device (introduction year)Event System -> AudioSource Best LatencyEvent System -> Native Audio (Efficiency)Native Touch -> Native Audio (Efficiency of Native Touch)
Xiaomi Mi A2 (2018 / 8.1.0 Oreo)321.2 ms78.2 ms (-75.65%)74.2 ms (-5.12%)
Xperia Z5 (2015 / 7.1.1 Nougat)120.6 ms69 ms (-42.79%)57.8 ms (-16.23%)
Lenovo A1000 (2015 / 5.1 Lollipop)366.2 ms217.4 ms (-40.63%)209.2 ms (-3.77%)
OnePlus One (2014 / 9.0 Pie POSP ROM)102.6 ms59.4 ms (-42.11%)53.8 ms (-9.43%)
Samsung Galaxy Note 5 (2015 / 7.0 Nougat)260.4 ms85.8 ms (-67.05%)54.2 ms (-17.38%)
Samsung Galaxy Note 8 (2017 / 8.0.0 Oreo)263.6 ms65.6 ms (-75.11%)71.2 ms (-17.02%)
Samsung Galaxy S7 Edge (2016 / 8.0.0 Oreo)243.4 ms67.4 ms (-72.31%)60.4 ms (-10.39%)
Samsung Galaxy S9 Plus (2018 / 8.0.0 Oreo)98 ms56.6 ms (-42.24%)40.2 ms (-28.98%)
iPod Touch Gen 5 (2012 / 9.3.5)94.6 ms58.8 ms (-37.84%)48 ms (-18.37%)
iPad 3 (2012 / 9.3.5)115.6 ms67.4 ms (-41.7%)52.8 ms (-21.66%)

View benchmark details and analysis.

For perspective, 23ms is approximately an acceptable latency for a pianist (the time from your finger hitting the key to hammer hitting the strings inside and finally travels to your ear). Native Audio could potentially make multiple times of this difference.

The last column improves perceived latency further by an indirect mean : speeding up the touch input that produces the response sound. It has a very pronounced effect on iOS and quite random but possible to be better effect on Android, because of how touch event works on each platform.

Why do you need... ?

Certain kind of games or apps relies heavily on feedback sound. The keyword is not an "audio application" but "feedback sound". For example if you are making a music player, that is clearly an audio app but audio latency won't affect the experience at all because all the interaction you do is press play and listen. It's not like if the song starts a bit late then an entire song is ruined. The core experience is on the song itself not a timing.

But if a feedback sound lags? It is not concerning for non-gameplay elements like a UI button that sounds when you press it, but imagine a drumming game that you have to hit at the correct moment. If you hit perfectly and the game says so, the sound will come later. If you hit early the game punishes you, but the sound will be exact. It's this kind of problem.

Click to learn more about 3 classes of musical apps.

3 Classes Of Musical Apps

Sequencer

Application like digital audio workstation (DAW) on mobile phone or live performing musical apps like Looper, Launchpad falls into this category. The app is interactive, but the reference of what is the "correct" timing are all controllable. Imagine you start a drum loop. Each sound might have delay based on device, but all delays are equal, results in a perfect sequence albeit variable start time. When starting another loops, it is 100% possible for the software to compensate and match the beat that is currently playing. This class of application is immune to mobile audio latency.

Instrument

Apps like GarageBand (in live playing mode) is in this category. The sound have to respond when you touch the screen. A latency can impact the experience, but if you are rehearsing by yourself you might be able to ignore the latency since if you play perfectly, the output sound will all have equal latency and will be perfect with a bit of delay.

Music Games

There are many music games on mobile phone made by Unity like Cytus, Deemo, Dynamix, VOEZ, Lanota, Arcaea, etc. If there is a sound feedback on hitting the note, this is the hardest class of the latency problem. Unlike Sequencer class, even though the song is predictable and the game know all the notes at all points in the song you cannot predict if the sound will play or not since it depends on player's performance. (Unless the sound is played regardless of hit or miss or bad judgement, then this class can be reduced to Sequencer class.) It is harder than Instrument class, since now we have backing track playing as a reference and also a visual indicator. If you hit on time according to the visuals or music, you will get "Perfect" judgement but the sound will be off the backing track. When this happen, even though you get Perfect already you will automatically adapt to hit earlier to make that respond sound match with the song, in which case you will not get the Perfect judgement anymore. In the Instrument class, if you are live jamming with others this might happen too but if you adapt to hit early you can get accurate sound and not be punished by the judgement like in games.

What I am making is a music game. Even a little bit of latency will be very obvious. Since there is a beat in the song for reference, players will be able to tell right away that he/she is hearing 2 separate sound (the beat in the song and the response sound) even if the player scores a perfect.

No magic

Native Audio aims to abstract the difference between native audio libraries only programmatically. I believe you as a programmer needs full understanding of exactly what is happening on each platform after calling those abstraction. This is why I put in explanations in the code explaining things in a non-abstract way for every platform supported. It will also shows up in your Intellisense. You will proceed with confidence and feel just like you are making games with Android Studio/Xcode with this knowledge!

A purchase also gives you all the source code. That is : Unity's managed C# side, iOS's .mm and .h files, and since .aar android plugin file is not readable I have also bundled Android Studio project that produces .aar file used by Native Audio in a zip file.

Feature list, what doesn't work?

You are not wrong to think that this has to be less flexible than audio function in Unity. I have received many e-mails for some use case that Native Audio sadly does not support so this section is to stop you from buying the plugin if it does not fit your case...

iOSAndroid
Volume/GainYou can use gain between 0.0f to 1.0f on every play or even while playing. On iOS it is possible to go over 1.0 for louder volume, but will clip if over 0dB.OKOK
2D Panning (Balance)Fully hearing both channels on center, and completely silence one channel on full left or right. OKOK
ConcurrencyHow many audio you can play over each other "mixerless" at once? On Android if you decided to go over the default 3 that's possible with some risks (usually possible up to around 14~16, on some phone even up to 31), read the "implementation" page.163+
Get/Set Playback TimeGet the audio time similar to AudioSettings.dspTime or audioSource.time but ask from native side. (iOS, the same lock step as Unity's dspTime. Android, different lock step as Unity's dspTime) Setting a playback time is possible even while playing.OKOK
ResamplingCurrently please just use 44100Hz because iOS still cannot do it. But if your game is Android only then go nuts.XOK
Off-main thread compatibilityMost Unity API including audio API could not be used from other than main thread, but Native Audio does not care about Unity so it works. Combo nicely with Native Touch's Android since the touch returned not on the main thread and you can play audio right away if you want. Note that this does not mean Native Audio is thread safe.OKOK
Playing audio downloaded dynamicallyIf you manage to put that in the same folder as StreamingAssets destination then Native Audio can read it. Which means if you use Asset Bundle it will still not in the right place.OKOK
PrepareFurther minimize the latency by preemptively specifying what you will play next without actually playing.OKX
UnloadUnload to save space because .wav in memory can get big.OKOK
StopYou can prematurely stop all audio.OKOK
Audio still playing even when minimizedSince it skips Unity and Unity cannot cut off the sound any more, it is the default behavior of both Android and iOS. On iOS you will need to add some settings to your game to enable this, please see the How To Use section.OKOK
Pause/ResumePause/resume is track-based. Additionally you can ask the current playback time and store it to use later with play with start offset.OKOK
Play from an arbitrary pointYou can use start offset in seconds.OKOK
LoopingOn Android to add this feature it will disqualify us from using fast audio track so I won't add it. On iOS it is possible to add an arbitrary loop point (not just at the end) but it is not implemented yet. If you need looping on Android and need less latency than Unity, I recommend the SoundPool-based solution Android Native Audio by Christopher Creates, since SoundPool can loop natively, or do a "manual loop" as explained in the How To Use (advanced use) section.XX
Pitch Shifting/Set RateOn Android to add this feature it will disqualify us from using fast audio track so I won't add it. On iOS I have not yet investigate this.XX
3D Positional SoundOn Android OpenSL ES can just do gain and 2D balance of each channels, so you have to emulate position sound with just this 2 settings. On iOS, actually OpenAL is a full-blown 3D positional sound engine but emulated to be able to do 2D panning for Native Audio. To make everything accessible like what AudioSource & AudioListener's interaction of position, distance, facing, falloff curves, doppler, etc. it would greatly go out of scope what Native Audio is supposed to do. Native Audio uses OpenAL just because it is the fastest and not that it is 3D. (You would require something like "Unity OpenAL" kind of plugin...) XX
Audio EffectsHaving any effects (even Android's native effect) will disqualify us from using fast audio track so I won't add it. On iOS I haven't investigate.XX
Low Latency Wireless/Bluetooth HeadphonesFast audio track does not work with wireless headset and you will get a slower play. On iOS I haven't investigate.XX
Playing compressed audio/music/BGMPossible but requires some native decoding work which hopefully each platform can do on its own. For now you have to use .wav which means it is not feasible to play music as it will eat up too much of your game's space. Only SFX for now which does not take much space in .wavXX
Playing .wav with custom header dataLike predetermine loop point embeded in the wav file. Unfortunately the wav reader I wrote is really dumb. (actually just a file header remover)XX

Evidence backed by an extensive personal research

(06/04/2017) "Unity's Mobile Audio Latency Problem And Solutions"

While containing many false assumptions I have get my feet (fingers) wet with many native solutions and arrives that AudioTrack (Android) and AVAudioPlayer (iOS) are better than the rest. Watch the research video or read the first experiment's note here. The plugin is still in development at this point.

(31/10/2017) "Unity iOS/Android Native Audio, Native Touch, and latency"

This time the detailed write-up was hosted on GitHub with repro project, I have found that touch/input latency also plays a role in the perceived audio latency. By going native not just audio but also on touch, we can get up to 1 frame earlier (can be 2 on Android) of touch response and that translate to FREE -16ms audio latency as long as that audio plays on touch response. (That's a lot when you "feel" it as a part of interaction)

A 1-take video which is my final proof that the plugin will have any benefit. At this point the first version of Native Audio is out along with "iOS Native Touch" to solve the aforementioned input latency.

Though, in that video I still insisted that AVAudioPlayer is the best for iOS. That is wrong.

(12/04/2018) "Unity's Mobile Audio Latency Problem And Solutions - iOS Build & OpenAL"

On iOS OpenAL does not introduce frame rate drop when playing plus we get an even better latency. The plugin was updated to use OpenAL on iOS.

(28/06/2018) "The Android OpenSL ES migration"

The research took place entirely in a Discord channel I have recently started. A user searching for audio solution is not satisfied with other solution but at the same time using Native Audio introduce a strange frame skip on Android. I have no choice but to abandon AudioTrack and blindly go for even more native OpenSL ES and luckily it solves the problem, also luckily for me I found many more faster path than the old Java way. It leads to the entire Android article located in the implementation page. You can read the epic discovery I and that user made together there.

(17/08/2018) "Faster input is possible on Android"

I have discovered a way to get faster touch on Android and indirectly related when used with Native Audio you will get up to 10-33ms free "perceived" lantency reduction. Previously named "iOS Native Touch" it will be renamed to just "Native Touch" with Android support. You are free to use Native Audio in combination with that one if you want even more.

Back to the top