The Full-AAR project

Full-AAR is an artistic and tech development project by WHS, a contemporary circus and visual theatre group based in Helsinki, Finland. The project explores the narrative possibilities of audio augmented reality (AAR), a medium that enhances reality with virtual sounds embedded in the environment. The Full-AAR project focuses on AAR utilising the six degrees of freedom (6DoF) principle, enabling the user to move around and turn head freely while the virtual sounds stay fixed in their three-dimensional positions.

To learn more about AAR with 6DoF, check the page What is 6DoF AAR?.

To read about recent events related to the project, check the News.

This non-profit project aims at developing AAR as a storytelling medium with its own narrative language. The emphasis is on indoor experiences in complex multi-room environments such as historical buildings, home museums, heritage sites, etc.

Another focus area is group-oriented applications with multiple simultaneous users being able to communicate with each other and maintain situation awareness while being acoustically immersed in the narrative.

To our knowledge, no comprehensive technical platforms or artistic tools yet exist for the medium. A handful of proprietary technical solutions exist enabling 6DoF AAR applications, but these are not open for public. Hence, we have been testing a number of available virtual audio solutions, indoor tracking systems, game engine workflows, system configurations, and narrative techniques since the start of the project in autumn 2021. Based on that work, a prototype setup has been constructed enabling 6DoF AAR experiences for two or more simultaneous users in a multi-room indoor space.

Further, to maximise the quality of acoustic illusion while keeping the system convenient to setup and run, a custom virtual audio solution is being developed in collaboration with Aalto University Acoustics Lab.

Photo: Actor Emma Castrén recording dialogue in an anechoic chamber at Aalto Acoustics Lab

These pages aim at providing detailed information on the Full-AAR project with insights and recommendations on creating 6DoF AAR experiences. The content is still very much a work in progress and will be constantly updated with new stuff as the project progresses.

The Full-AAR project is financially supported by the European Union NextGenerationEU fund .

In the project, we are particularly interested in the following topics:

• Narrative possibilities of AAR, mainly utilising 6DoF
  • What kind of stories would benefit this medium?
  • Identifying and testing characteristic narrative techniques in practice
  • Interaction between users; Letting simultaneous users experience the same story with different narrative viewpoints and alternate audio content
  • Interplay between real and virtual
  • Plausibility of virtual characters when portrayed through different spatial and contextual modes (invisible/acousmatic, attached to an object, heard behind a visual obstacle, etc.)
  • Role of interaction and multimodality in supporting the narrative immersion despite any imperfections in the virtual acoustic rendering
• Virtual audio
  • Finding and developing the most optimal spatialisation and auralisation solution to enable plausible acoustic illusions through 6DoF dynamic binaural audio while keeping the computational demands convenient
• Positional tracking
  • Experimenting with various 6DoF indoor tracking methods suitable for temporary installation in complex multi-room venues
  • Using body tracking for kinaesthetic interaction
• Workflows and best practices
  • Finding useable and fluent workflows and methods for content-creation

Photo: Teodors measuring acoustic properties of the venue with spatial room impulse responses (SRIRs)

The project started in September 2021, and thanks to the current funding, will continue at least until summer 2025. During the initial two years, we have conducted tests on various technologies for indoor tracking and virtual audio, and built a prototype 6DoF AAR setup. Simultaneously, we have been brainstorming with countless narrative ideas and researching on narrative techniques characteristic to the medium.

During this first stage of the project, we have also produced an hour-long demo experience for two simultaneous users to be opened for public in early 2024. We are inviting visitors and different audiences to test and discuss the setup, narrative content, and level of immersion. The demo will serve as a data source for several research topics about narrative AAR.

After the first demo, the project enters its second stage and will be collaborating with an external partner (yet to be confirmed as of writing this). While creating and producing an 6DoF AAR experience at their respective location – out of their countless intriguing stories –, the production process and experiences from both makers and users will be again collected for research purposes in order to develop the medium's best practices and narrative language further.

Matias Harju - Project lead, sound and narrative design, programming, research
Ville Walo - Producing, narrative and concept design, adminstration
Miranda Kastemaa - Lead programming
Teodors Kerimovs - Virtual audio development and research
Mikko Honkanen - Programming (2022)
Anne Jämsä - Background story research

For any questions on this wiki or the Full-AAR project, please drop a message to Matias:

All photos and diagrams on these pages by authors unless credited otherwise.