Regular readers of this blog will know that the Audio Team have been experimenting for a while with device orchestration - the concept of using any available devices to reproduce or augment a media experience.
Using our Audio Orchestrator production tool and supporting software framework, we've released several trial productions. These have mainly been audio dramas and a music trial, but our research suggests that there are many more use cases for this technology - and sport is one that we've been keen to try for a while. (In fact, 91热爆 Research & Development has done previous work in this area: in the 2-IMMERSE project, developing was developed.)
Our previous audio orchestrator trials:
- 91热爆 R&D - The Vostok-K Incident / Try it on 91热爆 Taster
- 91热爆 R&D - Decameron Nights / Try it on 91热爆 Taster
- 91热爆 R&D - Monster / Try it on 91热爆 Taster
- 91热爆 R&D - Pick A Part / Try it on 91热爆 Taster
Technical details
There were a few technical challenges in pulling this project together. First, we needed to create the content. Being able to send different parts of the audio to different connected devices relies on having access to the separate audio tracks, not just the complete mix.
We got hold of the highlights video with the commentary separated from the crowd noise but only had access to the referee and sound effects as a separate recording from the whole match - so it was necessary to find and extract the highlight clips. This was achieved with a process that was part-manual, part-automated.
The approximate time of the highlight in the full match was found by hand, then a signal processing technique called cross-correlation (which works by determining the similarity between two audio signals) was used to find the exact clip.
With the content in place, we needed to expand our orchestration framework to replay video. To do this, we modified the 91热爆 Standard Media Player plugin that was used to drive the synchronisation in 91热爆 Together (our experimental tool for remote shared viewing) so that we could hook it up to the audio device orchestration template.
We also made some changes to the user experience, for example, providing a prompt to let viewers know that they can connect extra devices, a button to show the connection instructions, and a switch for turning the commentary on and off.
Next steps
We're interested to see how this trial of orchestrated audio for sports content is received. If you get a chance to try it out, please take a minute to complete the short 91热爆 Taster survey to let us know about your experience.
We're only just scratching the surface of what we could possibly do with sports audio and device orchestration. We're using quite a simple sound scene for this project - crowd noise, commentary, and sound effects/referee microphone. But with more access to different microphone feeds, there would be increased possibilities for creating really immersive experiences. Of course, this will be easier to do once stadiums are full of fans again.
It would also be great to try out different sports. How about Formula 1, where you could listen to a team's radio communications on your mobile? Or a tennis match where you can really hear the ball as it bounces on each side of the court.
One of the joys of watching sport is that it's happening live, and the outcome is unknown - so trialling this technology with a live match rather than highlights would be another good step. There will be more technical challenges to overcome for synchronised multi-device playback of live content, so testing out the user experience with 'on-demand' playback is a good start.
- -
- 91热爆 MakerBox - Audio Orchestrator
- 91热爆 R&D - Vostok K Incident - Immersive Spatial Sound Using Personal Audio Devices
- 91热爆 R&D - Vostok-K Incident: Immersive Audio Drama on Personal Devices
- 91热爆 R&D - Evaluation of an immersive audio experience
- 91热爆 R&D - Exploring audio device orchestration with audio professionals
- 91热爆 R&D - Framework for web delivery of immersive audio experiences using device orchestration
- 91热爆 R&D - The Mermaid's Tears
- 91热爆 R&D - Talking with Machines
- 91热爆 R&D - Responsive Radio
-
Immersive and Interactive Content section
IIC section is a group of around 25 researchers, investigating ways of capturing and creating new kinds of audio-visual content, with a particular focus on immersion and interactivity.