This week, we launched Soundscape Explorer on 91热爆 Taster. It's an experimental web app for listening to, personalising, and making immersive soundscapes using . In this post, we'll introduce the app and how we went about making it, including our findings from a co-design study on the use of augmented reality (AR), binaural audio, and sound design.
What is Soundscape Explorer?
Soundscape Explorer is a web application that plays back mixes made using the . The mixer tool lets anyone add sounds and do some simple mixing (like setting volumes, start times, and loop points). The mix can then be shared as a link. Our Soundscape Explorer tool decodes that shared link to grab the sounds from the Sound Effects library and then positions them (randomly) in 3D space to be explored. If you use the app on a mobile device, moving the device can change the viewpoint. You can use the arrow keys to move around the scene on a computer.
You can use Soundscape Explorer with mixes you've made yourself, but we launched with some ready-made soundscapes to try out.
- Winter Outside is a poem written specially for this app by 91热爆 Research & Development's published poet, Philip Robinson. Philip's poem was read by Mary Ann Hobbs (91热爆 Radio 6 Music) and set to a wintry soundscape designed by Emma Young.
- A Visit from St. Nicholas is one of the best-known Christmas poems. It was read by Nick Bright (91热爆 Radio 1Xtra, 91热爆 Radio 5 Live) and set to narrative sound design by Jon Francombe.
- Tropical Beach, Spring Awakening, and Coastal Winds are short natural soundscapes designed by Sian Eleri (91热爆 Radio 1), Jack Reynolds (91热爆 R&D), and Emma Young (91热爆 R&D), respectively.
As well as 3D audio, the soundscapes have 360-degree background images (including one that was captured by Florian Schweiger with 91热爆 Research & Development's lightfield capture rig). A sphere represents each sound, and you can interact with the sounds to move them or change their volume.
We used the Sound Effects Mixer to find audio clips from the archive and set the timing for each effect. We then imported the mix into Soundscape Explorer to listen in 3D and position each sound source relative to the background image.
How did this work come about?
This work was inspired by our conversations with the team to explore how spatial audio and augmented reality (AR) could lead to different uses of the collection and the project, which investigates the wellbeing benefits of nature sounds.
We ran a short co-design study with our colleagues to better understand how we could use augmented reality audio with soundscapes. We hoped this would also lead to a user-led design process, ultimately producing a prototype that would interest our audiences. In the co-design study, we aimed to investigate how we could use natural soundscapes to enhance wellbeing, specifically how we could use augmented reality audio techniques to do this. We asked our participants about spatial audio, graphics, and interaction, how and if they could enhance soundscapes.
Responses from these interviews were analysed for common themes, with immersion, relaxation, and connection emerging as the key themes. Based on these themes, we chose a set of features that fulfilled the desires expressed by our participants. These features include binaural sound and visual augmented reality to enhance immersion; pre-curated soundscapes to deliver easily accessed relaxation; and the ability to share a link directly to soundscapes to improve participants' sense of connection with others. Additionally, the participants expressed a desire to customise and create their own soundscapes, leading us to add interaction with the soundscape to our set of desired features.
We created a prototype implementing these features and using visual augmented reality; a visual representation of the sounds in the mix was layered on top of the camera view of the space (so-called "magic window" AR). We decided to work in the browser and to prototype ideas from the workshop quickly we relied on existing open-source software: Google's library for dynamic spatial audio rendering and the toolkit for describing relatively simple virtual and augmented reality scenes. We could quickly create an explorable 3D scene visualising the sound effect positions on top of the real world camera view and render them to a binaural headphone signal. However, there were some stumbling blocks: the experimental WebXR API is not yet available on Apple devices, and enabling the camera view on our Android phones tended to interfere with the performance of the audio rendering.
In developing the app, we first prototyped with the augmented reality view (left), then added user interface elements (centre), and finally found the audio performance to be much better when using a static 360 image instead, especially with many sound sources (right).
So, we took the project in two directions. For a public trial on 91热爆 Taster, we wanted the app to work as well as possible with as many devices as possible. Given the problems described above, we decided to scale back our ambitions for the augmented reality view. Instead, we represented our sound sources on a 360-degree background image while still using the phone's sensors to rotate the view. But we didn't drop the idea of audio augmented reality altogether, and we continued to build a slightly scaled-back version for some focus group testing and evaluation. That's happening now, and we should have some findings in early 2022.
Find out more
We hope you enjoy using Soundscape Explorer鈥攅ither to escape into one of our authored soundscapes or to make your own. We'll use feedback from the 91热爆 Taster pilot to inform our thinking on immersive audio experiences and add findings from this release (and the earlier prototyping) to add to a growing body of research on audio augmented reality (for example, you can read about earlier trials with augmented reality audio on the blog).
Using the sound effects mixer and positioning sound sources in a virtual space is much like the work of professional sound designers, albeit using slightly different tools (for example, digital audio workstation plugins like our EAR Production Suite) and often working with composers and original recordings.
If you're inspired to take your mixes further, why not check out some of the sessions from our recent Sounds Amazing event, including a talk on the soundscapes of 91热爆 drama Vigil and insight from producer Benbrick into the detail of his work. If you'd like to hear how binaural sound is used in professional mixes, we have some examples in a .