The obligatory thesis word cloud about immersive audio

The word is out: I hereby present you my PhD thesis! After the final writeup in 2009, it took some time to get my viva arranged in August 2010. Sorry to keep you waiting…but now, it can be downloaded via this blog.

This thesis addresses the role of audio for immersion. Yep, the I-word…almost just as Impossible as it is Irresistible to designers. Yet, I’ve tried to come up with some useful theory for those who want to know more about the design of sound, music and speech in games and the conceptual decisions that are involved.

Let’s get things straight. This thesis does not:

• tell you which frequencies to amplify in order to make your music more immersive or explain you how to record immersive voice recordings (it’s not a cook book).
• provide you an answer to the question which psychological processes can be linked to immersion. There’s a lot of valuable literature available that investigates this subject.
• give you a final answer what to do when your game isn’t ‘immersive enough’. Game design is a discipline and you normally can’t fix game design flaws with audio!

Now what does it do? This work addresses the design decisions with respect to audio and immersion. I hope it offers audio designers, game designers and design researchers that are interested in the conceptualisation of game audio some illuminating insight into this matter.

Now go and create some Captivating Sound!

PhD award

ABSTRACT
This thesis explores the relation between game audio and (computer) game immersion. The main contribution in this research is a framework for the conceptual design of game audio in relation to immersion. A model for game audio is presented defining the communication by means of game audio during active gameplay. This model, named IEZA (Huiberts & van Tol, 2008), defines four conceptual domains of communication of game audio. IEZA was developed to enable a more detailed and consistent investigation into game audio and has been evaluated as a design resource in educational, academic and practical settings.

Immersion is considered to be one of the key factors making games worthy to play. The unity that seems to exist in the fact that many parties value this phenomenon is more apparent than real: there is a lot of dispute on the scope of immersion. Several classifications of immersion are compared for a better understanding of the nature of an immersive experience. The three‐dimensional SCI‐model by Ermi & Mäyrä (2005) is considered as valuable representation of the multi‐dimensional character of immersion.

Audio is an important constituent of most games and its role for immersion in games has hardly been investigated. In this thesis, audio is studied using the IEZA‐model and the SCI‐model, and several design issues are described. This yields a conceptual framework that describes various audio design issues that can be used to reflect upon conceptual decisions relevant for the design of audio in relation to immersion. This framework can be used for a conceptual analysis of design and is relevant for audio designers as well as game designers.

My PhD was a part-time PhD study at the University of Portsmouth in collaboration with the Utrecht School of the Arts.

Download the PDF here.

View the video files.

Huiberts, S., Captivating Sound: the Role of Audio for Immersion in Games. Doctoral Thesis. University of Portsmouth and Utrecht School of the Arts, Portsmouth, 2010.

A notification about your experience with this subject is warmly appreciated. Please send me a message using the contact form or use the comment form below.

I would like to express my sincere gratitude to all who supported the writing of this thesis: Jan IJzermans, Richard van Tol, Barbara Lotti, Tony Kalus, Adriaan Braat, Chris Creed, Han Onno ter Hoor, Micah Hrehovcsik, Rens Machielse, Paul Newland, Dan Pinchbeck, Dick Rijken, Hans Timmermans, Gerard van Wolferen, the other members of the Music Design Research Group, the gamers and moderators from the forums (Tweakers Forum, the 3Dgamers Forum, the Insidegamer Forum, the Gamers‐Forum, AudioGames Forum), the participants of PUGS, Espressofabriek and Hans and Riet Huiberts.

• Audacity: free audio editor
• 25 Free Digital Audio Editors You Should Know: list of free audio editors
• The Freesound Project: The Freesound Project aims to create a huge collaborative database of audio snippets, samples, recordings, bleeps, … mostly CC-by license
• OPsound – “free love, free music” (ahum)
• FreePlayMusic
• Free sound links
• Looperman: cool loops
• Loopasonic: free loops
• Rebirth (free classic “drum computer”)
• Radio Daddy (voice over community voor radio voices, place a request for leaders, one-liners, etc)
• Findsounds.com slightly out-dated database with sound files
• http://digifx.hku.nl/ [Only for HKU-students, access limited to KMT network in Hilversum]
• Sound design guide: how to make specific sounds ?
• FMOD MP3 licensing issues: All about using MP3 in games.
• …more to be added soon…

Game Audio Lab: an educational framework for research and design of realtime, nonlinear sound and music design. Photo © Sander Huiberts

The Music Design Group of the Utrecht School of the Arts under Jan IJzermans in the Netherlands has developed a lab for academic purposes which enables research and rapid prototyping of nonlinear audio. The system offers designers the ability to construct composite variables based on variables that are available in the original game system. In this paper, the architecture of the hardware and software as well as the motivation for this setup are discussed.

Nonlinear and adaptive systems for sound and music in games are gaining popularity due to their potential to enhance the game experience. The framework enables academics to map composite variables and adapt sound and music design in real-time during active game play. Composite variables were found to be more useful than mapping simple game parameters directly to the sound design. To give an example: mapping the state of an adaptive environment to the health level of the avatar often is too obvious for the player.

The paper also discusses the need for open game systems, which help designers to study sound design in realtime settings. Most game platforms are closed and protected by intellectual property. Interestingly, many academics mentioned this during the conference.

Unfortunately, I can’t publish the article here, so if you are interested it’s probably best to visit the AES website. The reference to the article is:

Huiberts, S. Van Tol, R and Went, K. (2009) GAME AUDIO LAB – AN ARCHITECTURAL FRAMEWORK FOR NONLINEAR AUDIO IN GAMES. In: Kelly, M. Proceedings AES 35th International Conference Audio for Games, 2009. Royal Academy of Engineering, London, UK.

Kees presenting at the AES. Photo: Sander Huiberts

Some video demos of applications (Half-Life 2 Source engine, OSC, Bridge software, FMOD and GStoolkit in Max/MSP) are listed below:

Credits for the game lab and these applications go to Arjen Schut, Mark Doeze, Maurice Alberts, Ramon Kerstens and Rogier Habraken. They truly did a wonderful job in this project.

Full demonstration Game Audio Lab Half Life 2: actual game play (select watch in high quality):

Simple demonstration: gunsound impact changes:

Simple demonstration: atmosphere dynamically changes with health status:

[For more information and the slides of this presentation, head over to adaptivemusicsystems.hku.nl]

Download PDF

Guitar player photo by puja

General websites:

• Audio Features at Gamasutra featured articles.
• WIKINDX by Mark Grimshaw: very comprehensive library with (academic) game audio resources.
• Gamesound.com articles on game audio.
• Game audio section at Filmsound.org articles.
• Music4games features: features on game music, primarily behind-the-scenes, interviews and post mortems with notorious game composers.
• Resources at AudioGames.net: Although the primary concern of this website is audio games, other sources on game audio are mentioned.
• Articles on filmsound.org
• Audio resources by Nick Koumentakis.
• Pretty Ugly Gamesound Study: good and bad examples of game audio.
• Links on The Game Audio Tutorial.
• Links on Sound-Music-Interactive-Games Blog.

Publications by academics and designers:

• The Game Audio Tutorial by Richard Stevens and Dave Raybould.
• Videogameaudio.com publications by Leonard Paul
• Gamessound.com by Karen Collins
• Sounddesign.org.uk articles by Rob Bridgett

Forums

Two topics on the gameaudio forum discussing new books:

• Recent books
• Which book to get

Some more resources can be found in this article by West Latta: Demistifying sound design.

Two active discussion lists are:

• sound_design@yahoogroups
• gameaudiopro@yahoogroups.com

And an extra pointer: Richard van Tol and I published an article about the IEZA framework on Gamasutra. This might be useful for designers to increase insight in the communication by means of game audio. Mentioning it here is not to promote ourselves, we found that this framework can help designers conceptualising and structuring their work. If the article valuable for you, the other frameworks that are listed in the first section of this article might. Visit: IEZA framework for game audio at Gamasutra.

IEZA is further discussed in my PhD-thesis on game audio and immersion. There you also find models and theory about game audio.

In the past years, I’ve designed quite some audio menus for audio games and supervised projects that used audio-only interaction for blind users. Below I share some of my experiences concerning these menus, and include the original recommendations of the thesis.

illustration by zkukkuiz

An updated summary of Audio Menus (2002).

An audio menu is a menu, which utilizes sound as primary feedback mechanic. The user interacts in various ways with the menu system, for example with keys or speech control. Audio menus are mostly found in applications where the user is not able to use the visual alternatives of menus. Also, when the menu is not in sight (controlling the menu of a ‘smarthome’-system throughout the building) or when a visual task is required (e.g. in a car), audio menus can be valuable. At present, audio menus are most frequently found in audio games [2] or games for the blind.

In general, these menus do not seem to be as pleasant and user-friendly as most visual menus are. Current menus found in phone systems often read out all the options, before letting the user make his choice. In this case, the user often feels passive due to this very passive way of interaction. In this case, the user is waiting continuously.
When developing an audio menu, one could choose a ‘more active’ interaction model, in which the user can browse with two keys and listen to each option in his own tempo. In that case, he can interrupt options and confirm his choice. With this model the waiting time is decreased because the menu-system does not have to tell the user which option corresponds to which key every time, which prevents listener-fatigue when used frequently. Also, the user might feel ‘in control’ instead of slaved to the application, while carrying out instructions. Yet, the instructions for a more active interaction model requires more explanation and some players have difficulties, because of their non-existent expectations.

One key issue to acknowledge is that the domain of audio fundamentally differs from the visual domain. The information in the auditory domain is presented in time instead of at the same time on screen. For the user, it can be rather difficult to keep overview. This also has implications for the number of options that can be represented in one menu layer: after 7 options, most users have forgotten what the first options were. A good source on this aspect comes from Buxton, Gaver & Bly (1991) [3]. The following information is abstracted from their article, which shows the strengths and weaknesses of using audio in the interface:

Other considerations to improve usability are:

• decrease the number of choices within one level of the menu, to ease navigation and increase overview;
• rearrange the options in a more logical order so the desired option is found faster, preferably adaptive to the presumed task of the user, following user profiles;
• using background sound to give the user an impression of the menu-layers and the structure of the menu [4]. In a prototype that was made for the menu, most users understood the function of an ambient sound layer that altered when a different layer of the menu was accessed. The main menu can be easily accentuated with sound;
• choosing the correct terminology in the explanation to inform the user about the menu-interaction. Visual references can be misleading when interacting in the auditory domain. To give examples: what is ‘back to top’ when using sound? What is the most right option?

One should also take into account that the user goes through a learning process. Few users have encountered an audio menu, and do not have experiences with audio menus on regular computers. The presence of a computer screen often has high impact on the expectations of the user. It can be desirable to replay the spoken explanation for users that do not interact with the menu and use and alternative – more complete – explanation when they do not seem to understand at all how to interact with the system.

A difficulty with audio menus is that experienced users as well as novice users have to be able to work with the menu. In graphical menus, this is not an issue, as the user knows the location of the button and does not have to wait for the options to appear.
A way of handling this is to assume that experienced users use higher interaction speeds: they know the options and are already acquainted with the controls. A designer can choose to wait for a short period before playing the explanation files, so experienced users do not have to hear “This is an audio menu. Use these keys to browse…”. Also, auditory icons can accomplish this ‘need for speed’, when they replace the function of the speech samples. In the audio game Drive (2002) [5], these techniques were applied to allow the player to navigate with high speed, while hearing only the short and subtle icons (so only the beginning of the files). The short silence between the tones and the speech fragment prevents clicks in the audio. Novice users just listen to the complete sound file and make their choice in their own time.

The short auditory icon at the beginning of the sound file allows fast navigation for experienced users.

A more frequent use of audio menu’s and a (perhaps spontaneously arised) standardization are likely to have a positive effect on the expectation and experience of users.

References

[1] The author would like to express his gratitude to Jan IJzermans, Richard van Tol,Hugo Verweij and Kees Went.

[2] AudioGames.net

[3] Buxton, W., Gaver, W. & Bly, S. (1991). The use of non-speech audio at the interface. Tutorial no. 8. In: CHI’91 Conference proceedings, Human Factors in Computing Systems, ‘Reaching through technology,’ New Orleans, ACM Press: Addison-Wesley.

[4] This technique was implemented in the main menu of the Audio Game Maker by The Bartiméus Accessibility Foundation.

[5] Huiberts, Van Tol, Verweij. Utrecht School of the Arts. Review and research report at AudioGames.net

Reference:

Huiberts, S. (2002). Audio menus. Unpublished thesis. (Utrecht School of the Arts, The Netherlands). Updated abstract, 2008, Retrieved 15 December, 2008, from
http://captivatingsound.com/audio-only-menus/