Augmented Reality in Music
AR in music provides experiences in learning an instrument and reading sheet music, fun applications that visualize music for marketing, and mobile AR technology that even allows for audience participation during a live concert. We have come a long way from John Mayer's 2009 music video for Heartbreak Warfare. It is the very first music video to use Augmented Reality.
AR Piano – 3D Piano Concerts is a popular augmented reality app to watch a virtual pianist superimposed on your piano. There is a feature where the user can see a virtual set of hands playing and a piano roll above to help you place your fingers and for how long. However, the music the in the app is quite advanced, pieces by Beethoven and Chopin to name a few, which require many years of study to gain the technical skill.
A more approachable app for music education is the Augmented Songbook. Augmented Songbook was an app for tablets that was in development that would superimpose augmented contents, such as a keyboard, over the pages of songbook. The camera would detect the page by finding the corners, just like a mobile scanning application, then “read” the music by highlighting the note, and then the user would play the notes on the piano. This is a challenge because it is difficult to identify the sheet music, maintain the superimposed image while reading and animating the notes, so that the use can play the correct notes. There would be some problem
In a live musical performance, there is possibility for the audience to perform with the artist. Usually, an audience member will hold their mobile device up and view an experience through their screen – U2 did this during a recent tour, more can be read here. But, then there was Echobo. Echobo is the mobile music instrument designed for an audience to participate in a live performance. In 2012, 109 audience participated in a collaborative performance. There was a live clarinet player as well as a “master musician.” The while the clarinetist played, the master musician gave the audience members constraints as to which notes they had access to play. The constraints were chords, so the audience members could only play notes in that chord and of that particular key.
In regard to musical entertainment, AR in the popular music industry has been used primarily used as a marketing tool for artists. The Los Angeles based group, Haim, has created an AR album cover for their Women in Music III release in November 2020. It can be seen here. Sam Smith’s track “Diamonds,” has an AR experience through Spotify (access is here). This requires two devices, one lays flat, and the other can rotate around a 360° Sam Smith. These examples are excellent marketing tools. The user will often find themselves staying on the page of the artist for further discover of their music.
Beatsy is an AR music visualizer on your smartphone. The app uses a grid that detects a flat surface. Once the surface is detected, the microphone is activated and starts listening to ambient sounds. The amplitude of the sound is animated on the surface through the screen. There is further marketing opportunity here for the artist. There is a feature where an artist can put up a Beatsy sticker in town, and a user can use the app to listen to a song get information about the artist as well as an AR experience.