The Full-AAR project

Full-AAR is an artistic and tech development project exploring the narrative possibilities of Audio Augmented Reality (AAR or Audio AR). In AAR, reality is enhanced with virtual sounds embedded in the environment. In some applications, real-world sounds can also be removed, replaced, or enhanced like isolating speech from background noise.

The Full-AAR project focuses on headphone-based AAR utilising the six degrees of freedom (6DoF) principle, enabling the user to move around and turn head freely while the virtual sounds stay fixed in their three-dimensional positions. The project is run by WHS, a contemporary circus and visual theatre group based in Helsinki, Finland.

To learn more about AAR with 6DoF, check the page What is 6DoF AAR?.

To read about recent events related to the project, check the News.

This non-profit project aims at developing AAR as a storytelling medium with its own narrative language. The emphasis is on indoor experiences in complex multi-room environments such as historical buildings, home museums, heritage sites, etc.

Another focus area is group-oriented applications with multiple simultaneous users being able to communicate with each other and maintain situation awareness while being acoustically immersed in the narrative.

To our knowledge, no comprehensive technical platforms or artistic tools yet exist for the medium. A handful of proprietary technical solutions exist enabling 6DoF AAR applications, but these are not open for public. Hence, to be able to run our experiments, we have constructed our own prototype platform based on thorough tests on a number of available virtual audio solutions, indoor tracking systems, game engine workflows, system configurations, and narrative techniques. The prototype enables 6DoF AAR experiences for two or more simultaneous users in a multi-room indoor space.

Further, to maximise the quality of acoustic illusion while keeping the system convenient to setup and run, a custom virtual audio solution is being developed in collaboration with Aalto University Acoustics Lab.

Photo: Actor Emma Castrén recording dialogue in an anechoic chamber at Aalto Acoustics Lab

These pages aim at providing detailed information on the Full-AAR project with insights and recommendations on creating 6DoF AAR experiences. The content is still very much a work in progress and will be constantly updated with new stuff as the project progresses.

The Full-AAR project is financially supported by the European Union NextGenerationEU fund .

In the project, we are particularly interested in the following topics:

• Narrative possibilities of AAR, mainly utilising 6DoF
  • What kind of stories would benefit this medium?
  • Identifying and testing characteristic narrative techniques in practice
  • Interaction between users; Letting simultaneous users experience the same story with different narrative viewpoints and alternate audio content
  • Interplay between real and virtual
  • Plausibility of virtual characters when portrayed through different spatial and contextual modes (invisible/acousmatic, attached to an object, heard behind a visual obstacle, etc.)
  • Role of interaction and multimodality in supporting the narrative immersion despite any imperfections in the virtual acoustic rendering
• Virtual audio
  • Finding and developing the most optimal spatialisation and auralisation solution to enable plausible acoustic illusions through 6DoF dynamic binaural audio while keeping the computational demands convenient
• Positional tracking
  • Experimenting with various 6DoF indoor tracking methods suitable for temporary installation in complex multi-room venues
  • Using body tracking for kinaesthetic interaction
• Workflows and best practices
  • Finding useable and fluent workflows and methods for content-creation

Photo: Teodors Kerimovs measuring acoustic properties of the venue with spatial room impulse responses (SRIRs)

The project started in September 2021, and thanks to the current funding, will continue at least until summer 2025. During the initial two years, we have conducted tests on various technologies for indoor tracking and virtual audio, and built a prototype 6DoF AAR setup. Simultaneously, we have been brainstorming with countless narrative ideas and researching on narrative techniques characteristic to the medium.

During this first stage of the project, we have also produced a bio-fictional, interactive story titled Valtakunnan liitto (The Reign Union). The experience is located in WHS’ gallery space in Helsinki and is based on the businesses operating in the premises, namely a laundry Vaahto Oy. The story revolves around the connections between Vaahto and a subversive, underground Nazi group during wartime. Two simultaneous users navigating the space will share the same story, but do they hear the same narrative? Who to trust, and can you trust what you hear?

We are inviting visitors and different audiences to test and discuss the setup, narrative content, and level of immersion. The demo serves as a data source for several research topics about narrative AAR.

After the first demo, the project enters its second stage and will be collaborating with an external partner (yet to be confirmed as of writing this). While creating and producing an 6DoF AAR experience at their respective location – out of their countless intriguing stories –, the production process and experiences from both makers and users will be again collected for research purposes in order to develop the medium's best practices and narrative language further .

Matias Harju - Project lead, sound and narrative design, programming, research
Ville Walo - Producing, narrative and concept design, adminstration
Miranda Kastemaa - Lead programming
Teodors Kerimovs - Virtual audio development and research (2023)
Mikko Honkanen - Programming (2022)
Anne Jämsä - Background story research

For any questions on this wiki or the Full-AAR project, please drop a message to Matias:

All photos and diagrams on these pages by authors unless credited otherwise.