• Create BookmarkCreate Bookmark
  • Create Note or TagCreate Note or Tag
  • PrintPrint
Share this Page URL

Chapter 4. Design > Adaptive Audio

Adaptive Audio

Adaptive audio has been defined and redefined many times since game audio began; currently it is understood as any audio that is nonlinear or nonreactive in a game. For example, in a linear medium such as a cut scene, a movie, or a full-motion video clip, the audio is linear: It doesn't change no matter how many times the medium is played. The same is true when someone is playing a level of a game with a looped piece of music. If the player presses a button and something happens either through sound effects or music, the audio is reactive; however, if the player's actions cause a number of different transitions in the soundtrack, the audio can be classified as adaptive on the game design side, since the game itself is changing based on the player's actions. Taking it yet another step further, if the audio influences the player's choices, the audio can be classified as adaptive—but on the player side. When the previously mentioned transitions take place, the soundtrack or sound effects “adapt” to a player's gameplay choices and in turn match or enhance the experience.

What makes videogames different from movies is that in a game, a player influences the outcome, which can be different each time the game is played. This is the first thing every composer needs to know when creating music for a game. Without this knowledge, a composer assumes that the game experience will be linear and thus creates only one piece of music for any given situation, rather than several pieces or several variations on a piece. It's not just about getting your hands on a live orchestra; it's about creating an entirely new means of musical expression. So why is music more important than sound for adaptive audio?

For the most part, music is what drives adaptive audio, since music is more abstract than sound and can be less attached to the visuals. For example, a game's depiction of an explosion should have some sort of sound associated with it to indicate that there is an explosion going on. Music does not have this sort of requirement.

No bones about it, creating an adaptive audio soundtrack is hard. Few people have done it well. This is because most uninitiated game composers consider adaptive audio as daunting as any kind of programming. Even some hardened veterans refuse to delve into it because it is so far removed from writing music and creating sound effects. Even fewer have done it so well that they are known for it. I will present just a few of the souls worthy of mention who have contributed adaptive and dynamic soundtracks to games.


The examples I give in this chapter are almost exclusively PC games. This is not because console games haven't used adaptive soundtracks, but because I haven't experienced any of them. With apologies to the console crowd, I've heard whispers of such console games, but the people I talk to about adaptive audio are primarily PC developers. I encourage everyone in the audio community to add to your store of knowledge about adaptive soundtrack usage by going to www.adaptiveaudio.org (Figure 4.2).

Figure 4.2. Adaptive Audio Now is an initiative started by Guy Whitmore for professionals and amateurs alike to post tales of their adaptive-audio techniques. Check it out at www.adaptiveaudio.org.

LucasArts and iMUSE

LucasArts has made some of the best computer games ever, and part of the company's greatness comes from its advanced and groundbreaking approach to audio. You can find a fantastic history of LucasArts on the company's Web site, www.lucasarts.com/20th/history_1.htm, but how inconsiderate of me to tempt those nestled in bed with this book to reach over to their laptop to read it. To sum it up, the founding members of LucasArts had state-of-the-art games in mind when they began the company in 1983, but they took sound more seriously than most in the industry did. Indeed, they hired some very imaginative folks, among them Michael Land (LucasArts' audio director), Peter McConnell, and Clint Bajakian—the triumvirate behind the musical magic in LucasArts games.

In the late 1980s, Michael spearheaded an effort to create something called the Interactive Music Streaming Engine—hence the name iMUSE. Essentially this was an engine that handled first MIDI tracks and later digital audio tracks, and enabled the MIDI tracks to branch and loop. This was the first time this sort of endeavor had been pursued in earnest; more than $1 million was spent on its development over a period of ten years, and the company applied for a patent. It's also the only interactive-adaptive music system that has its own fan site: http://imuse.mixnmojo.com/what.shtml.

The iMUSE engine was designed specifically for the kind of games that LucasArts developed, which at the time were mostly graphic adventures. The company is still most renowned for its adventure games, the latest and greatest being Knights of the Old Republic. Back in the old days, a graphic adventure consisted of a character wandering through environments that took up a whole screen at a time. When the character wandered toward the edge of the screen, the scene cut to the next place they were going. The game's design could be mapped on a flowchart fairly easily, and the musicians were able to create music based on the chart. This is perhaps the ultimate example of a development process map (DPM) driving the design of the game as well as the music composition.

The first title to use iMUSE was Monkey Island 2: LeChuck's Revenge (Figure 4.3), released in 1991. LucasArts' audio team used the system to allow different tracks (either single MIDI instruments or sets of them, depending on the situation) to be introduced and then removed as the player moved from place to place within the same screen, not just from screen to screen. Thus sometimes the music shifted when the player moved to a different room in the same screen.

Figure 4.3. Monkey Island 2: LeChuck's Revenge was the first title to use the iMUSE sound engine.

Later, iMUSE was used with great success with streamed stereo audio files in The Dig, another LucasArts classic. Many people thought they were forward-thinking when they claimed that a soundtrack might one day have a sweeping orchestral score. Well, they had no idea: The Dig presented an incredible orchestral score by Michael Land, with plenty of dynamics and an almost exclusive use of Wagnerian strings. The LucasArts team licensed recordings of Wagner, then cut them up and used them in conjunction with synthesized samples of other instruments. The occasional solo live instrument was recorded in-house at LucasArts to complete the soundtrack. The clever use of cross-fading made a soundtrack composed in pieces sound seamless.

The original LucasArts music team is pursuing other interests now, but its legacy lives on in the eight or so major LucasArts titles that used iMUSE. The engine will take its place in videogame music history, as it had a hand in perhaps the greatest adaptive soundtracks of its time.

3D First-Person Adaptive Soundtracks

Nearly every game uses three-dimensional sound in some form or another nowadays, but with the remake of Castle Wolfenstein in 1990 a whole new game genre was born: 3D FP, or 3D first-person. People were now seeing the screen as though they were in the adventure looking through the eyes of the player character.

In 1992, the first major title to use an interactive soundtrack with a first-person perspective was the Looking Glass/Origin collaboration Ultima Underworld: The Stygian Abyss (Figure 4.4).

Figure 4.4. Ultima Underworld: The Stygian Abyss provided a truly enriching experience in 3D first-person, with an adaptive soundtrack to boot.

George Sanger (aka The Fatman) and Dave Govett created a number of pieces related to the player's actions, from wandering to fighting. The music shifted based on variables such as an enemy attacking the player (not just being within sight of the player, mind you).

Unfortunately, this title was produced during a transitional period in game instrumentation. Small sets of samples were built into sound cards (such sets came to be known as General MIDI, since the sets were supposed to be the same on all sound cards) and triggered by MIDI commands; they were expertly crafted so that if you concentrated you could hear a full, beautiful orchestral score. These days you don't have to concentrate to hear the richness of the music, so let's give credit where credit is due: to the pioneers.

In 1995, another 3D FP game with an adaptive soundtrack was released: Delphine Software's Fade to Black. In theory, this title should have taken adaptive audio a step further, as it was released three years later, but it didn't. Instead, its music switched in a split second from a tense, ambient General MIDI track to a pulse-pounding timpani, regardless of whether you could see an enemy nearby. If the enemy was behind you on a catwalk, the music tipped you off. This was a failure on the part of the adaptive track, as it unnecessarily influenced players with something completely outside the playing environment. This isn't to say the music itself was bad, but the way it was integrated screwed up the overall immersiveness of the game.

Another 3D FP game, Unreal by Epic Megagames (now Epic Games), used MOD files, so called because the first music files used the .mod extension in their filenames. These are not General MIDI files; rather, they use whatever samples the composer chooses, up to a certain size limit per file. For Unreal the MOD file size limit was 1 MB. Considering that the sound bank for General MIDI on sound cards at the time (the SoundBlaster AWE32 and AWE64, circa 1997) was only 1 MB, Unreal was able to have truly unique and high-quality sound. MOD files could also be manipulated more than MIDI files could: By simply typing a number, you could tweak vibrato for each time subdivision, whether it was an eighth note or a sixty-fourth note. It sounds programmer-centric, but MODs exploded in the PC music scene around 1993 and haven't stopped since

With the MOD file format, the composers at Straylight Productions were able to loop sections of music devoted to as many gameplay situations as were needed, from suspense to action to death to exploration. They could also create various fade (not cross-fade) settings so that music transitions would be smoother.

Still, even with the more advanced techniques used in Unreal and other games like it, the audio wasn't nearly as convincing as that of a film when it came to smooth transitions, good modulation, and emotional impact. The quality of MOD files still paled in comparison with a live score, and the audio transitions based on combat or exploration were getting old. It would take better techniques and another four years before orchestras and exponentially larger orchestral samples would allow game composers to create a more dramatic and powerful sonic landscape using a soundtrack.

The audio folks behind Deus Ex: Invisible War used a different technique. As the audio director of this project, I arranged for various areas in the game to change instrumentation or change the soundtrack completely, using cross-fades for the environment. This capability was always there in conjunction with newly available ambient sound effects. Ambient sound effects made an overwhelming impact in the title Thief in 1998 (Figure 4.5)—and in a subtler way much earlier, in 1989, with the Amiga title Dungeon Master, influencing players to make major gameplay decisions based on ambient sounds alone for the first time. In addition to the ambience and varied environmental music, triggers based on various situations could either switch the entire soundtrack or play a one-shot piece of music. It was very dynamic but not used in nearly enough places, so the soundtrack ended up being mostly ambient, while it could have been far more expressive. Yep, I blame myself.

Figure 4.5. Thief may not have been a looker graphically, but its alternative stealth gameplay and stunningly atmospheric sound put it in a class of its own.

Remember the ambient sound I mentioned earlier? And how Thief blew the doors off the competition with this kind of sound? Nowadays, people like Erik Kraber at Electronic Arts are jumping on the “hard work on sound effects makes a better game” bandwagon with painfully serious, top-notch sound design and mixing work. Let's explore a bit more of what's going on in the increasingly complex realm of game audio.

  • Creative Edge
  • Create BookmarkCreate Bookmark
  • Create Note or TagCreate Note or Tag
  • PrintPrint