Like any artistic medium, games are emotional experiences filled with joy, sadness, frustration, and more. Special moments can bring tears, cause shouting, or even screams. But imagine during one of those emotional highs if the audio simply died, and the game continued onward in a deafening silence before eventually freezing. That kind of marred experience was commonplace under Dolphin's old way of handling HLE audio.

The users of today don't have to face those problems; modern High Level Emulation (HLE) audio is both fast and accurate, mostly matching the conventions and output of its high-accuracy counterpart, Low Level Emulation (LLE) audio. This change in behavior is thanks to the work by delroth and magumagu that corrected the main fundamental flaw that afflicted old HLE audio. Fixing this defect and cleaning up the audio brought a multitude of features and fixes to the emulator that helped bring us into this modern era of speedy accuracy.

While the end result is great, the journey was anything but smooth, filled with uncovered bugs, regressions, controversy and more! Thanks to the help of contributors degasus, booto, skidau, konpie and phire Dolphin has nearly perfect audio emulation.

The Beginning of Audio in Dolphin¶

As the first GameCube emulator to boot commercial games, there was a time when having any audio emulation was an achievement! The GameCube's Digital Signal Processor (DSP) was effectively a black box to the early developers of Dolphin. As such, they did what any normal person would do and poked it with data and analyzed what it put out. While not the easiest or most efficient way to work, ector (hrydgard), StapleButter and many others brought the DSP-HLE plugin into Dolphin before the 1.0 releases. The original DSP-HLE mostly just played music in games that streamed audio, but was capable of sometimes playing sounds and other music.

This HLE audio core continued to evolve along with the rest of the emulator. By the time Dolphin hit open source, games could play music and sounds, even if things didn't sound exactly right. With fresh life breathed into Dolphin from the move to open source, DSP-HLE continued to evolve, providing the source of primary audio for the emulator for year after year. But that evolution would slow to a crawl in its final years, even as glaring issues such as crashing, hangs, and audio stopping plagued the emulator. The truth was that the original HLE audio implementation handled things in a broken way that could not be reconciled.

A Fundamental Flaw¶

To understand this flaw, one must first understand how the GameCube/Wii's DSP operates. The audio processing code runs on the DSP, which is a secondary processor engineered to be fast at auxiliary tasks like audio mixing. Every 5ms, the CPU sends the DSP a list of data blocks about sounds to process. Each of these blocks contain information like “location of the sound data in memory”, “volume”, “looping or oneshot”, as well as a list of commands for the DSP to run. When the DSP is done running the commands, it loads the mixed sound samples into RAM and sends an interrupt to the CPU to signal that it is done, then the CPU outputs the audio and goes to the next 5ms block.

The old HLE code was unable to do this properly, thanks to an asynchronous processing method. The 5ms interrupts weren't guaranteed to happen after 5ms* of CPU time. Old HLE audio could play these samples with up to 100ms of variance. In a worst case scenario, twenty updates could happen instantly, followed by 100ms of total silence or messed up audio. And if the emulator wasn't running full speed, things could get even more confusing as the DSP would plow ahead as if everything was fine. These behaviors resulted in many of the numerous problems that plagued DSP-HLE for years.

*Wii games and a small minority of GameCube games use 3ms interrupts.

The left side shows how asynchronous audio handles DSP interrupts, while the right side shows how it would work on a typical game run on the GameCube.

AUX Processing¶

AUX processing is a feature of the DSP program that allows the DSP to apply audio effects on the sounds, such as echo, reverb, chorus, etc. AUX uses DSP commands to download and upload data to the CPU in order for the CPU to process the AUX effect while the DSP is working on something else.

This simply cannot be done with asynchronous audio processing. Not only is DSP<->CPU communication impossible (they don’t run in sync anymore), but all of the AUX code is designed to handle a fixed number of samples. This matches how many samples the DSP handles at a time (32×5, for 5ms at 32KHz). Without AUX processing almost all audio effects are lost, and games sound considerably inferior.

Games Requiring Tight Timing¶

Since the DSP always operates at set intervals, game developers used this to time when sounds should start and stop, especially when composing music from samples. Of course, 5ms accuracy is often not enough for music; so the DSP provides a feature called "sub-5ms updates" which can specify some slight changes on the sound data blocks for each millisecond. Failing to emulate this nifty feature can cause games to get exceedingly confused as the DSP doesn't make the changes it's expecting. Once the timings are off, anything can happen: audio garbling, instruments out of sync, popping/cracking, along with many, many more possible problems. It's not pretty.

Games Using the Sound Processing Status to Trigger Events¶

Double Dash is notorious for music that just quits.

One of the more infamous problems in Dolphin emulation history was the fact that The Legend of Zelda: The Wind Waker would hang a lot when collecting items. It was so common, cheat codes were added to the GameINI to work around the hang. Link picks up an item, the jingle plays, and then the game just sits there waiting, not frozen, not crashed, still emulating, just... waiting. What has happened in this case?

In a basic sense, some games wait for sounds to be completed before they will continue. When the sound processing is done asynchronously, the game may miss the moment when the sound has finished playing (because it went too fast compared to the emulated CPU speed) and just freeze. This tended to affect the Zelda µcode games more than anything. This separate µcode posed extra challenges for Dolphin due to tighter timings and tons of features unique to itself. In HLE audio, it's completely isolated from the handling of the AX µcode.

There were a lot of problems unique to these games because of this, such as issues with certain sounds and tons of music hangs. Mario Kart: Double Dash!! and Super Mario Galaxy 1/2 both suffered consistent music hangs, leading to the infamous Grand Star hang in the Super Mario Galaxy series.

The Solution¶

To work around the issues of DSP-HLE, nakeee, LordMark and others created the DSP-LLE Interpreter. It represented an extremely accurate way of handling audio by perfectly reproducing what the DSP does, including synchronous audio processing. HLE audio could only approximate what the DSP does with C++ code, while LLE audio translated the binary code into x86, producing far more accurate results.

DSP-LLE and DSP-HLE still stand beside each other to this day.

Unfortunately, using an interpreter for the DSP is painfully slow and grinds the entire emulator to a halt. skidau and kiesel created DSP-LLE Recompiler to combat this slowdown the same way Dolphin uses a JIT for the main processor of the GameCube/Wii. Everything in DSP-LLE recompiler is the same as the interpreter, except it produces those very same results much faster.

However, DSP-LLE required DSP ROM dumps from a GameCube or Wii. While the procedure was fairly involved on a GameCube, later on the ease of softmodding a Wii would make retrieving DSP dumps simple - if you had one. Because of the dumps requirement and the chore required to get DSP ROM dumps, DSP-LLE could not be set as a default, limiting user exposure. That mixed with its slower way of handling audio made it difficult for users to swallow.

New-AX-HLE Audio¶

A change in audio hit in 3.5-78, right after the release of Dolphin 3.5 late December 2012. Without having to worry about marring the release with a potentially buggy feature, delroth sprung one of the largest rewrites to Dolphin in years: New-AX-HLE. This not only rewrote and improved huge portions of the HLE audio system, but it also finally brought synchronous audio to DSP-HLE within the GameCube AX µcode.

The result was astounding; many games began outputting audio in DSP-HLE for the very first time, stability increased and the emulator was altogether faster because less titles needed DSP-LLE in order to run.

While this merge was sudden to Dolphin users, Delroth had already spent several months prior rewriting most of the AX HLE code. Of course, the first merge was only the beginning. His reverse engineering of the DSP resulted in him being able to write his own DSP-ROM that could be included with dolphin! For the first time, users could get through problematic games with LLE audio without needing to go through the trouble of dumping one from their console!

Then DSP-LLE became mostly unneeded; delroth's second AX-HLE merger hit in 3.5-1154, fixing up the problems from the first one while also adding support for Wii AX games. Hundreds of games saw huge improvements in stability, HLE audio became usable in popular features such as netplay, and Dolphin found itself in a much stronger position for the future.

So what was the reaction to this increase in both speed and accuracy? Surprisingly mixed.

The controversy behind Synchronous HLE Audio¶

Sometimes unexpected consequences completely alter how a fix is perceived. The developers expected New-AX-HLE to be one of the most heralded features added to Dolphin in years; it improved speed and accuracy equally. We thought users would celebrate these changes! However, several factors mixed together to make New-AX-HLE a controversial feature.

1. New-AX-HLE addition of synchronous audio brought over another behavior of DSP-LLE - when the game was not running full speed, audio wouldn't run full speed either.

2. New-AX-HLE made HLE audio behave akin to DSP-LLE, so it began to suffer a lot of the issues that went unaddressed in LLE audio due to its lack of use.

3. While not related to audio; Dolphin was going through a phase where it was getting slower due to hacks being removed and compatibility being raised. Games weren't running as fast as they were in older releases.

4. There was no official blog back then, so most users did not see the explanations as to why the behavior changed and the roadmap for the future.

By far the biggest change in behavior was for users who couldn't run games fullspeed. The asynchronous behavior of old HLE allowed them to mostly ignore the slowdown because the game would sound reasonable unless audio outright crashed. The addition of synchronous audio processing took away that ability and made it very obvious that the game was lagging. Complaints mounted against the new way of handling HLE audio, and a small minority of users demanded the old system back.

The developers didn't just change audio overnight, though. The past months prior to merge were filled with discussions about what synchronous audio would bring to the emulator, and the best ways to transition. All that could be said is that the changes in behavior were intended; if the host machine could not run the emulator at 100%, sound should not be fullspeed. In order to guarantee stability in games, the audio must be synchronous with the action.

As you might expect, the latent explanation wasn't very convincing to the people that just wanted to play their games and didn't care how the emulator worked. What was thought to be one of the biggest features added to Dolphin ended up being one of its most controversial. Threads were started, emails sent, and it became very loud and agitated. To ease the process, delroth created an article to explain the changes in more detail, but this was long after the initial outrage.

As with any outcry against a new feature, there were a lot of very valid complaints mixed in with the unreasonable ones. Long-time issues in Dolphin's audio interface made the jump from LLE to HLE thanks to the two systems working in similar manners. This included a massive 200ms latency on audio and a strange buffer bug that caused audio distortion after ninety minutes of play! By 4.0-879 those bugs and several others were squashed.

The Final Steps Toward Total Synchronous Audio¶

New-AX-HLE covered roughly 96% of the games supported by Dolphin, but there were two small subsets of that did not fall under the scope of the New-AX-HLE merges. One set of games was extremely small, but included some of the most popular games in Dolphin: Zelda µcode audio.

The Zelda µcode, named after the fact it was primarily used in Zelda and Mario games (but surprisingly not Skyward Sword or New Super Mario Bros. Wii), is very difficult to emulate properly. It has more stringent timings and can use a lot of extra DSP features that the AX counterpart does not. That couldn't scare away magumagu. Without warning, he implemented synchronous HLE audio for Zelda µcode games in 4.0-1300. This didn't handle all of the problems, instead it focused on fixing the critical problem of the asynchronous audio handling.

Then there was one more category of games that escaped the grasp of both rewrites: streaming audio, also known as Disc TracK (DTK) audio. DTK audio was mostly used for music in the background of levels, so there wasn't any real demand to make it synchronous except for the fact that it had a few issues with looping. Except, it wasn't always used for just background music. Some clever titles used streaming audio for voice acting and cutscenes, which absolutely demanded perfect timings.

After a lot of tweaking and testing, magumagu's DTK Rewrite hit Dolphin finally bringing synchronous audio to the last holdouts.

The difference it makes can be easily seen in the following cutscene from Star Fox Adventures.

The Cleanup Crew¶

While delroth and magumagu earn top honors for the transition to synchronous audio in the AX-HLE, Zelda µcode HLE and DTK Audio, without the help of the rest of the development team, a lot of nasty bugs would still be lurking in the code!

• degasus cleaned up a lot of the bugs in the audio interface that old HLE skipped. This included the aforementioned 90 minutes bug and latency issues.

• skidau actually made a big change to Dolphin even before AX-HLE by adding the OpenAL audio backend and implementing time-stretching in it. Another fun-fact is that he applied that time-stretch to DTK audio, meaning although it was still buggy, it was at least synchronous when using OpenAL! More recently, he has made many contributions that include fixing many audio errors, missing audio, and weird behaviors that have emerged due to better timings. A lot of these changes have affected both LLE and HLE audio, including instrument loop-points and ARAM DMA timings. He also used the audio mixing in Dolphin to hookup play emulated Wiimote audio through the default audio device (usually system speakers).

• Phire handled a very strange bug that cropped up after the DTK Audio Rewrite. It turns out a missed static variable caused resampling issues between the DSP mixer and DVD mixer, resulting in the audio static that was plaguing games. His relentless search for the bug is the reason it why it was fixed before it became a well-known problem.

• booto started an audio assault after being dragged in to fix other regressions in DTK audio. He used extensive hardware testing and reverse engineering to figure out how audio streaming worked and tuned up the emulator to get everything even remotely related to audio streaming working. The result is that not only were DTK audio looping bugs fixed, but games like Pac-Man Fever began playing the right audio tracks for the very first time. He also fixed games with tight timing like Pokemon Snap, Harvest Moon: Magical Melody and others with his audio assault.

• konpie has shown up to provide critical information and patches just when everyone else was out of ideas on several bugs. Two of the patches he provided fixed severe crackling in several games, and missing music in Tom and Jerry in War of the Whiskers.

Conclusion¶

While the benefits of Synchronous HLE took some time to take hold, Dolphin's audio emulation is in a far better place. With extremely high compatibility mixed with low requirement, DSP-HLE is now the default option for almost every title compatible with the emulator. This was something that was absolutely unthinkable just two years ago. There is still work to be done, but DSP-HLE has become a solid audio emulation option that will serve users well into the future.

Some information and examples used in this article were taken from delroth's original blog post, which covers the GameCube's audio technology in much more detail. It can be read here.