So your Unity game outputs WAY slower audio than other apps even on the same device? Turns out, Unity adds as much as 79% of the total latency you hear. But not to worry because Native Audio will take care all of it. I have researched into the cause for so long and the solution is right here.
This plugin was developed when I noticed that both my Android AND iOS game has a considerably worse audio latency than other music apps such as Garage Band installed on the same device. I have confirmed by creating a basic Unity app which just play an audio on touch down vs. a basic Xcode iOS app/Android Studio app which also plays an audio on touch down. You can clone the project on GitHub to confirm this by yourself.
If you can't feel the difference, I encourage you to do a simple sound wave recording test. Keep your phone at the same distance from your computer's mic and tap the screen nail-down. The sound wave interval between the peak of nail's sound and the peak of response sound should immediately reveals the difference visually. Newer device might have a better latency, but the difference between Unity and native app should be relative for every devices.
This perceived result is because of 2 things : touch input latency AND audio latency. Native Audio's job is to greatly reduce the audio latency.
Unity has an internal mixing system, backed by FMOD. You can go crazy with a method like
AudioSource.PlayOneShot and the sound magically overlaps over itself,even though at native side Unity only asked Android for only 1 "Audio Track". How can that be? Because Unity spend time mixing them together to 1 bus. Plus you have all the wonders of Unity 5.0.0 introduced audio mixer system. Effects, sends, mixers, etc. all stacked onto that.
A great design for a game engine. But we certain subset of game developers absolutely do not want any of the "spend time" if possible. Unfortunately Unity does not give us a choice to bypass and just go native.
For some genre of apps and games that needs critical timing for audio (basically any response sound from input, rhythm game, etc.) this is not good. The idea to fix this is to directly call into the native methods and have it read the raw audio file we prepared without Unity's importer. Bypassing Unity's audio path entirely.
Also I suspected Unity have to collect all audio commands to summarize them together at the end of each frame. Selecting a number of loudest
AudioSource to mix according to audio settings, mix with the mixer system introduced in Unity 5.0 which supports various effects and many channel/buses/sidechain, then send the mixed 1 audio stream to native.
While with Native Audio, after native calling there is no concept of "frame" anymore. It will just play audio right there in the middle of the frame! (The hardware might wait for something, but that is already outside of Unity's domain)
I have researched for long time into the fastest native way of each respective platform. For iOS it is to use
OpenAL (Objective-C/Swift) and for Android it is to use
OpenSL ES (NDK/C). For more information about other alternatives why they are not good enough, please go to implementation page.
But having to interface with multiple different set of libraries separately from Unity is a pain, so Native Audio is here to help...
It improves latency for iOS greatly, but I guess many came here to fix the already-horrible-without-Unity Android latency.
I am proud to present that Native Audio is following all of Google's official best practices required to achieve High-Performance Audio in Android. Because Native Audio is not as versatile, I can additionally fix all the latency-related tradeoffs that Unity have to pay for their more versatile audio engine.
OpenSL ESand not Java
AudioTrack. Plus, most latency-critical interfacing methods from Unity are by
externto C and not
AndroidJavaClassto Java. Feature set of
OpenSL ESthat would add latency has been deliberately removed.
Of course, with publicly available thorough research and confirmations. This means it can perform even better than native Android app that was coded naively/lazily in regarding to audio. Even pure Android developers might not want to go out of Java to C/NDK.
We will do a measurement by placing the device in fixed position, recording the peak of nail sound touching the screen to the response sound produced by the app. While this is not a standard latency measurement approach like loopback cable method and the number alone is not comparable between devices, but time difference of the same device truly show how much the latency decreased without doubt.
The pure Unity's audio was measured with "Best Latency" settings on Audio Settings, the fastest playing (smallest buffer) Unity can achieve without any plugins. At first I wasn't annoyed by Unity's latency that much, but after a clear contrasting with Native Audio (especially on Android) the latency reduction is insane. It is surprising how I was able to live with that bad latency when there's nothing to compare with.
|Device (introduction year)||Event System -> AudioSource Best Latency||Event System -> Native Audio (Efficiency)||Native Touch -> Native Audio (Efficiency of Native Touch)|
|Xiaomi Mi A2 (2018 / 8.1.0 Oreo)||321.2 ms||78.2 ms (-75.65%)||74.2 ms (-5.12%)|
|Xperia Z5 (2015 / 7.1.1 Nougat)||120.6 ms||69 ms (-42.79%)||57.8 ms (-16.23%)|
|Lenovo A1000 (2015 / 5.1 Lollipop)||366.2 ms||217.4 ms (-40.63%)||209.2 ms (-3.77%)|
|OnePlus One (2014 / 9.0 Pie POSP ROM)||102.6 ms||59.4 ms (-42.11%)||53.8 ms (-9.43%)|
|Samsung Galaxy Note 5 (2015 / 7.0 Nougat)||260.4 ms||85.8 ms (-67.05%)||54.2 ms (-17.38%)|
|Samsung Galaxy Note 8 (2017 / 8.0.0 Oreo)||263.6 ms||65.6 ms (-75.11%)||71.2 ms (-17.02%)|
|Samsung Galaxy S7 Edge (2016 / 8.0.0 Oreo)||243.4 ms||67.4 ms (-72.31%)||60.4 ms (-10.39%)|
|Samsung Galaxy S9 Plus (2018 / 8.0.0 Oreo)||98 ms||56.6 ms (-42.24%)||40.2 ms (-28.98%)|
|iPod Touch Gen 5 (2012 / 9.3.5)||94.6 ms||58.8 ms (-37.84%)||48 ms (-18.37%)|
|iPad 3 (2012 / 9.3.5)||115.6 ms||67.4 ms (-41.7%)||52.8 ms (-21.66%)|
For perspective, 23ms is approximately an acceptable latency for a pianist (the time from your finger hitting the key to hammer hitting the strings inside and finally travels to your ear). Native Audio could potentially make multiple times of this difference.
The last column improves perceived latency further by an indirect mean : speeding up the touch input that produces the response sound. It has a very pronounced effect on iOS and quite random but possible to be better effect on Android, because of how touch event works on each platform.
Certain kind of games or apps relies heavily on feedback sound. The keyword is not an "audio application" but "feedback sound". For example if you are making a music player, that is clearly an audio app but audio latency won't affect the experience at all because all the interaction you do is press play and listen. It's not like if the song starts a bit late then an entire song is ruined. The core experience is on the song itself not a timing.
But if a feedback sound lags? It is not concerning for non-gameplay elements like a UI button that sounds when you press it, but imagine a drumming game that you have to hit at the correct moment. If you hit perfectly and the game says so, the sound will come later. If you hit early the game punishes you, but the sound will be exact. It's this kind of problem.Click to learn more about 3 classes of musical apps.
Application like digital audio workstation (DAW) on mobile phone or live performing musical apps like Looper, Launchpad falls into this category. The app is interactive, but the reference of what is the "correct" timing are all controllable. Imagine you start a drum loop. Each sound might have delay based on device, but all delays are equal, results in a perfect sequence albeit variable start time. When starting another loops, it is 100% possible for the software to compensate and match the beat that is currently playing. This class of application is immune to mobile audio latency.
Apps like GarageBand (in live playing mode) is in this category. The sound have to respond when you touch the screen. A latency can impact the experience, but if you are rehearsing by yourself you might be able to ignore the latency since if you play perfectly, the output sound will all have equal latency and will be perfect with a bit of delay.
There are many music games on mobile phone made by Unity like Cytus, Deemo, Dynamix, VOEZ, Lanota, Arcaea, etc. If there is a sound feedback on hitting the note, this is the hardest class of the latency problem. Unlike Sequencer class, even though the song is predictable and the game know all the notes at all points in the song you cannot predict if the sound will play or not since it depends on player's performance. (Unless the sound is played regardless of hit or miss or bad judgement, then this class can be reduced to Sequencer class.) It is harder than Instrument class, since now we have backing track playing as a reference and also a visual indicator. If you hit on time according to the visuals or music, you will get "Perfect" judgement but the sound will be off the backing track. When this happen, even though you get Perfect already you will automatically adapt to hit earlier to make that respond sound match with the song, in which case you will not get the Perfect judgement anymore. In the Instrument class, if you are live jamming with others this might happen too but if you adapt to hit early you can get accurate sound and not be punished by the judgement like in games.
What I am making is a music game. Even a little bit of latency will be very obvious. Since there is a beat in the song for reference, players will be able to tell right away that he/she is hearing 2 separate sound (the beat in the song and the response sound) even if the player scores a perfect.
Native Audio aims to abstract the difference between native audio libraries only programmatically. I believe you as a programmer needs full understanding of exactly what is happening on each platform after calling those abstraction. This is why I put in explanations in the code explaining things in a non-abstract way for every platform supported. It will also shows up in your Intellisense. You will proceed with confidence and feel just like you are making games with Android Studio/Xcode with this knowledge!
A purchase also gives you all the source code. That is : Unity's managed C# side, iOS's
.h files, and since
.aar android plugin file is not readable I have also bundled Android Studio project that produces
.aar file used by Native Audio in a zip file.
You are not wrong to think that this has to be less flexible than audio function in Unity. I have received many e-mails for some use case that Native Audio sadly does not support so this section is to stop you from buying the plugin if it does not fit your case...
|Volume/Gain||You can use gain between 0.0f to 1.0f on every play or even while playing. On iOS it is possible to go over 1.0 for louder volume, but will clip if over 0dB.||OK||OK|
|2D Panning (Balance)||Fully hearing both channels on center, and completely silence one channel on full left or right.||OK||OK|
|Concurrency||How many audio you can play over each other "mixerless" at once? On Android if you decided to go over the default 3 that's possible with some risks (usually possible up to around 14~16, on some phone even up to 31), read the "implementation" page.||16||3+|
|Get/Set Playback Time||Get the audio time similar to ||OK||OK|
|Resampling||Currently please just use 44100Hz because iOS still cannot do it. But if your game is Android only then go nuts.||X||OK|
|Off-main thread compatibility||Most Unity API including audio API could not be used from other than main thread, but Native Audio does not care about Unity so it works. Combo nicely with Native Touch's Android since the touch returned not on the main thread and you can play audio right away if you want. Note that this does not mean Native Audio is thread safe.||OK||OK|
|Playing audio downloaded dynamically||If you manage to put that in the same folder as ||OK||OK|
|Prepare||Further minimize the latency by preemptively specifying what you will play next without actually playing.||OK||X|
|Unload||Unload to save space because .wav in memory can get big.||OK||OK|
|Stop||You can prematurely stop all audio.||OK||OK|
|Audio still playing even when minimized||Since it skips Unity and Unity cannot cut off the sound any more, it is the default behavior of both Android and iOS. On iOS you will need to add some settings to your game to enable this, please see the How To Use section.||OK||OK|
|Pause/Resume||Pause/resume is track-based. Additionally you can ask the current playback time and store it to use later with play with start offset.||OK||OK|
|Play from an arbitrary point||You can use start offset in seconds.||OK||OK|
|Looping||On Android to add this feature it will disqualify us from using fast audio track so I won't add it. On iOS it is possible to add an arbitrary loop point (not just at the end) but it is not implemented yet. If you need looping on Android and need less latency than Unity, I recommend the ||X||X|
|Pitch Shifting/Set Rate||On Android to add this feature it will disqualify us from using fast audio track so I won't add it. On iOS I have not yet investigate this.||X||X|
|3D Positional Sound||On Android ||X||X|
|Audio Effects||Having any effects (even Android's native effect) will disqualify us from using fast audio track so I won't add it. On iOS I haven't investigate.||X||X|
|Low Latency Wireless/Bluetooth Headphones||Fast audio track does not work with wireless headset and you will get a slower play. On iOS I haven't investigate.||X||X|
|Playing compressed audio/music/BGM||Possible but requires some native decoding work which hopefully each platform can do on its own. For now you have to use .wav which means it is not feasible to play music as it will eat up too much of your game's space. Only SFX for now which does not take much space in .wav||X||X|
|Playing .wav with custom header data||Like predetermine loop point embeded in the wav file. Unfortunately the wav reader I wrote is really dumb. (actually just a file header remover)||X||X|
While containing many false assumptions I have get my feet (fingers) wet with many native solutions and arrives that
AudioTrack (Android) and
AVAudioPlayer (iOS) are better than the rest. Watch the research video or read the first experiment's note here. The plugin is still in development at this point.
This time the detailed write-up was hosted on GitHub with repro project, I have found that touch/input latency also plays a role in the perceived audio latency. By going native not just audio but also on touch, we can get up to 1 frame earlier (can be 2 on Android) of touch response and that translate to FREE -16ms audio latency as long as that audio plays on touch response. (That's a lot when you "feel" it as a part of interaction)
A 1-take video which is my final proof that the plugin will have any benefit. At this point the first version of Native Audio is out along with "iOS Native Touch" to solve the aforementioned input latency.
Though, in that video I still insisted that
AVAudioPlayer is the best for iOS. That is wrong.
OpenAL does not introduce frame rate drop when playing plus we get an even better latency. The plugin was updated to use
OpenAL on iOS.
The research took place entirely in a Discord channel I have recently started. A user searching for audio solution is not satisfied with other solution but at the same time using Native Audio introduce a strange frame skip on Android. I have no choice but to abandon AudioTrack and blindly go for even more native OpenSL ES and luckily it solves the problem, also luckily for me I found many more faster path than the old Java way. It leads to the entire Android article located in the implementation page. You can read the epic discovery I and that user made together there.
I have discovered a way to get faster touch on Android and indirectly related when used with Native Audio you will get up to 10-33ms free "perceived" lantency reduction. Previously named "iOS Native Touch" it will be renamed to just "Native Touch" with Android support. You are free to use Native Audio in combination with that one if you want even more.