AI Pulling the Strings in Live Music Performance

With all the noise surrounding Generative-AI, it is easy to miss the other contributions Artificial Intelligence are making to the music industry. In the context of live performance, AI is currently augmenting performances, enhancing audience experiences, revolutionizing sound production, and the visual production. And, this is just the intro to what’s coming up

AI as a Musical Partner: Performance Enhancement

The integration of artificial intelligence into live music performances is moving beyond simple automation, with AI now acting as a dynamic and even responsive musical partner.

One notable innovation in this area is Yamaha’s AI Music Ensemble Technology. This system is designed to synchronize in real time with a performer, meticulously analyzing their playing. By comparing the notes being played to the sheet music, the AI discerns the current position in the score, the tempo, and the nuances of musical expression.

This intricate analysis allows the AI to predict how the performer will interpret upcoming sections, enabling it to provide accompaniment precisely aligned with the performer’s timing. This technology is versatile, applicable to a wide range of instruments, from pianos and solo instruments like violins or flutes to multi-person ensembles and even complex orchestral performances.

This system can even learn the unique stylistic tendencies of a specific performer, including their characteristic tempo changes and common errors. This technology was used in a recreation of the late maestro Sviatoslav Richter’s performance. Through AI ensemble technology, it was possible for the Berliner Philharmoniker’s Scharoun Ensemble to perform alongside a digital representation of the legendary pianist, bridging the boundaries of time and space

.Future, Center Stage

From the moment that hologram technology became more sophisticated and life-like we started to see live performances incorporating a hologram of passed artists. Arguably the first performance was in 2012 at Coachella Festival in California where the late rapper Tupac Shakur «appeared» on stage with Snoop Dogg and Dr.Dre.

Another application of AI in live music is the LyricJam Sonic system. This is designed to generate real-time music and lyrics based on an artist’s existing body of work or even live instrumental input. The artist can play an instrument, and the system will “listen” and respond in real time by generating a compatible musical accompaniment drawn from the artist’s own catalog. This interactive capability is blurring the traditional lines between human and artificial creativity and a collaborative potential for improvisation and on-the-fly composition.

Is Music even (a)live anymore?

It is nothing new that the concept of a live performance is somewhat being blurred by creative use of technology. Most of the famous live albums in rock history have been «doctored» before release, rerecording vocal and instrumental performances and even adding crowd noise as in the case of «Kiss Alive!» on which producer Eddie Kramer lifted the audience cheers from a Slade live album!

With AI technology like the systems mentioned becoming ubiquitous, the notion of a «live performance» might be overdue for redefinition. We’re already at the stage where most artists’ performance is a man-machine hybrid event. The use of auto-tune and «pre-taped» vocals and instruments have become the norm rather than the embarrassing exception, further undermining «the realness» of a live experience.

The praxis of «Real-Time» is also a key concept when questioning the relationship between man and (AI) technology. One of the major capacities of AI tech is to do «things» in real-time, however in this «hybrid event» of a live performance one needs to question both ideas «Real» and «Time». The concept of real-time refers to something «communicated, shown, presented, etc. at the same time as events actually happen.»

One context where this is challenged is YouTube where instrumentalist influencers edit both what is «real» and what is «time» of a performance creating an illusion out of what is in actuality an unreal performance (sped up and quantized etc.) and presented in «faux time» – a prerecorded track disguised as a live performance. And, it’s just a matter of time before AI generated «performances» will flood these channels.

The Audience is Listening…and Participating

There are several options for gig-discovery using AI, but AI is enabling new forms of interaction and engagement during live performances themselves. AI can be used to generate music and visuals that respond in real time to audience feedback, such as applause, gestures, or even detected emotions.

This creates a more dynamic and participatory concert experience, moving beyond the traditional model of passive spectating. For instance, AI could manipulate audio effects or generate visual displays based on the energy levels of the crowd, making the audience feel like active participants in shaping the performance.

AI algorithms can help artists adjust their setlist on the fly to better match the mood and energy of the crowd, sort of what DJs have always done! A great DJ generally goes into a venue with a plan (set-list), but is constantly modifying the sequence of tracks, and even the real-time mixing. Reacting to the collective emotional state of the audience, the vibes, can be a powerful tool to create the perfect concert experience.

However, letting AI systems «scan» your emotions, whether through wearables, face-scan etc. might not be something you’re comfortable with when going to a gig to let loose, or wind down…And, this is a classic AI dilemma, the attempt at enhancing human interaction via invasive measures.

Lights, Sound, Camera, AI!

In the domain of sound engineering, AI is used to optimize sound levels, balance different instruments and vocals, and even identify and troubleshoot audio problems as they pop up. This is particularly beneficial in large venues or complex musical arrangements where achieving optimal sound quality can be challenging. AI-assisted sound engineering can ensure that every note and instrument is audible to the audience, improving the overall listening experience.

World-renowned live sound engineer, Dave Natale (Rolling Stones etc.) claimed (in an interview with Rick Beato):

«6 million years of evolution is not gonna be fooled by 30 years of chip design»,

making the case for keeping the live sound engineering 100% analog. Still, Natale might agree that by automating certain aspects of sound adjustment and problem-solving, AI allows human sound engineers to focus on the more nuanced and creative aspects of the mix.

On the visual side of the experience, AI can generate abstract visuals that evolve in response to the harmonic progression of a song or control the intensity and color of stage lights to match the emotional arc of the music. The incorporation of AI-generated holograms by K-pop groups offers a vivid example of this approach, blurring the lines between the physical presence of performers and virtual elements.

The Future is Waiting in the Wings

I don’t think we are far from a situation where it is commonplace that AI generated «artists» go on tour with fully AI-generated «live» shows. The success of virtual artists like Hatsune Miku (a Vocaloid software voicebank developed by Crypton Future Media) who has performed sold-out concerts as an AI hologram, suggests a growing audience appetite for such experiences. Future advancements could lead to even more realistic and interactive virtual performers, opening up entirely new forms of entertainment.

Musicians could also leverage AI to create interactive narratives within their live performances. AI’s ability to process and respond to audience input in real time could enable interactive storytelling elements, and adding VR to the mix there are many creative paths to take to incorporate a narrative arch to a performance. What could a band like Rush, for instance, have done with the tools to present their sci-fi concept album 2112 in a live setting enhanced by AI and VR?

Coda

There is a lot of trepidation in the music industry, these days and for good reason. A scenario where 95% of all music consumed in any setting is AI generated is the wet dream of the tech-bros who are spearheading the investment boom in AI. No songwriters, performers, management etc. to dilute profits. The most dangerous argument promoted by proselytizers like Mark Zuckerberg and Sam Altman is the argument that AI should be able to freely train on historical data without any recognition of copyright etc. as not to lag behind competitors (read:China).

The accelerating adoption of AI in live music necessitates addressing complex economic and ethical considerations. Issues related to potential job displacement for live music professionals (e.g., sound engineers, lighting technicians), copyright protection for AI-generated content, ownership of AI-created works etc. Fair compensation of artists and other industry professionals will need clear legal and ethical frameworks to ensure a sustainable and equitable future for a music industry being «remixed» by Artificial Intelligence.