The virtual orchestra is a tour-friendly, audiovisual installation that my company, MOD Productions, created as a pilot project for the Australian Chamber Orchestra. The purpose of this pilot was to test the viability of producing a portable, interactive and educational promotional tool that was adaptable for different environments.
The Virtual Orchestra pilot: Screenshot from the Unity app using the warped fisheye mesh
These included a dome application giving users the immersive experience of standing amongst musicians while they perform, and a virtual reality/augmented reality mobile phone application.
One of the challenges with producing the Virtual Orchestra pilot was capturing audio and footage that was able to be used in these different formats. The solution was provided, both by the choices made when shooting the footage, and by the use of editing software.
We decided to film the players on greenscreen to provide complete flexibility in terms of manipulating the background and foreground elements in the Dome application. This enabled the option of placing the musicians within different scenes – from simple backgrounds like a blue sky with clouds, to complex, moving backgrounds like snow storms or water droplets. The Dome App also allowed us to screen the performers as silhouettes against these background.
We recorded two pieces of music, totaling six minutes. These were the Winter movement from Antonio Vivaldi’s Four Seasons and an arrangement of Niccolò Paganini’s Caprice No. 5. After testing these in the Dome environment, the ACO players involved suggested that repertoire for the Dome should keep all players busy, as in the Vivaldi piece, where the viewer is immersed in the surrounding sound as well as the surrounding visuals. Music like the Paganini piece, which featured solos by individual instrumentalists, was less effective in a Dome environment. It did however well suit the mobile application with limited visuals but the ability to separate musicians’ performances and concentrate on the actions of one musician.
The Virtual Orchestra screenshot with snow storm scene. Photographer: Michela Ledwidge
An important aspect of the pilot was to ensure that the software framework was easily configurable to suit different exhibition venues. We achieved this using video game software (Unity) to develop a Dome App with a number of presets that could be triggered from a mobile controller app (TouchOSC). Unity allowed video elements to be manipulated in real-time, and provided a convenient packaging mechanism for delivering versions of the Dome App across different platforms. It also made simplified the testing of parameters. The Dome app was built to meet de-facto standard MirrorDome specifications pioneered by Paul Bourke at University of Western Australia. Each presets built into the Unity app warped the imagery to test a different screen type – a flatscreen computer display, the vertical (half dome) iDome and a inflatable MirrorDome theatre system.
Theoretically, the use of middleware (pre-existing software) means that it will be straightforward to distribute software-only packages to exhibition venues that already show MirrorDome and flat screen video content. In practice, however, the virtual orchestra distribution process and packaging will evolve based on testing at initial sites to make installation more straightforward.
Production still from ACO shoot with full orchestra, March 2011. Photographer: Matthew Vaughan.
Our final pilot event gave viewers an opportunity to experience the ACO quartet in 3D. Upon entering the installation space (AVIE at iCinema, UNSW), the user was surrounded by images of the players, with the sound appearing to be coming from the image of each player’s location in the projected visuals. The Virtual Orchestra pilot was successful in terms of audience interest and technical achievement. Late March, we shot the full orchestra at Fox Studios with 13 stereo camera rigs. The next phase of the project – and the next challenge for MOD Productions – is the post production and creation of the installation.
Michela Ledwidge
Michela Ledwidge is an award winning artist, entrepreneur and internet pioneer (MOD Films, MOD Productions, Rack&Pin). Since setting up the first web site in NSW in 1993 she has worked on a wide range of projects around the globe including the BBC iPlayer service and the multi-platform children’s TV series dirtgirlworld.
This article is an edited version of Michela Ledwidge’s report ACO Virtual Orchestra Pilot. Read the full report here
Read More
http://modprods.com/virtual orchestra
Information on the MirrorDome
http://www.astronomy.swin.edu.au
http://paulbourke.net/miscellaneous/domemirror/
Watch More
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Australia.