Interview: “AI pushes you out of your musical comfort zone”

Foto: Jens Thomas/CCB Magazin

Valerio Velardo is co-founder and CEO of Melodrive, a Berlin-based start-up that uses Artificial Intelligence (AI) to automatically generate soundtracks for video games in real time. Valerio combines two worlds: He is a classically trained musician, he has a PhD in music and AI, and he studied astrophysics at degree level. Valerio is one of our judges in the Beats & Bits competition.

Text: Michael Wingens


I would like to start with one or two personal questions: You are a classically trained musician – how have you ended up in the AI music business?

My personal background consists of two different paths. The main path has always been classical music: I studied and worked as a concert pianist, conductor and composer. At the same time, I studied physics and astrophysics at degree level. Eventually, I have combined my musical and scientific passions and decided to look deeper into AI by pursuing a PhD in AI music, which is a highly interdisciplinary field.

You are currently working in the AI music business – but do you still compose analogue music?

I stopped playing a while ago, but I still compose music from time to time. We compose music as part of our daily work at Melodrive and on the sidelines, I have finished a piano suite that I am going to release at some point.

Did AI have any role in your piano suite?

No, not at all. AI is fantastic, but sometimes you just want to have a little space for yourself. As a human being you don’t want to always think of music in terms of algorithms, but as a personal expression in a free, or at least in an apparently free, way.

Speaking about AI in more general terms, the most important question upfront: How does AI make music?

Obviously, generative music is a very broad field. You can generate music on many different levels. One level would be to take human-made music and use it as a basis for intelligent recombination. This, in a sense, is a form of generative music. Historically, generative music was not intelligent at all. You could use simple algorithms by mapping certain mathematical formula into music and get relatively satisfying outcomes. Using AI is a whole other level. The most fashionable and commonly used approach right now is machine learning: You have a neuronal network, train it with music and you get music that sounds more or less like the original. The problem with that approach is that the music will lack structure.
Music is extremely complex and multi-dimensional, so you can’t just solve that problem by training a neuronal network with a ten thousand or even a million songs. There are many aspects that current algorithms and forms of machine learning miss. Therefore, what becomes important is to inject music knowledge representation and music theory in your algorithms and combine them with machine learning. This will allow you to create a hybrid system that will hopefully write highly complex music. This hybrid system would be able to know what it is doing in a semantic way, similar to what we are doing as human beings all the time.
That is also our philosophy at Melodrive. We have deliberately stayed clear of pure machine learning approaches due to the complexity of music. You need to have a deeper understanding about music, to decide which algorithms to build and at which level to do so. If you want to create an algorithm that generates a full orchestral score, there are far too many decisions that you have to think about, such as instrumentation, structure, or melody. All of these factors cannot just be outsourced to a single algorithm. What you need is a complex system and several algorithms that will handle different aspects at different points in the overall structure.


“At the end of the day, everything that we build has the potential to be creative, regardless of the creator.”


If you create a complex structure based on a neuronal network, AI is able to compose new songs on its own. Would you say that machines can be creative?

That is a very difficult question. If you talk to the 7.7 billion people on earth, probably everyone has his or her own definition of creativity that is slightly different from anyone else. The problem with creativity is that humans love to think they are the only species that is creative. Intelligence is not a stronghold anymore. AI is more “intelligent” than us at specific tasks, like chess or even the board game go. Creativity, on the other hand, has always been seen like a god-like feature that has been bestowed upon humanity. That may not be the case after all. Creativity has different forms and can be replicated. But even if we do not have an agreed idea of what creativity is, a key point that you will find in most research is that in order for a piece of music to be creative, for example, it needs to fulfill at least two conditions: First, it has to be novel. Second, it has to have value for someone.
Take for example Beethoven’s Ninth Symphony. It used to be completely novel and it still has a lot of value for a multitude of people. In that sense, it is a creative piece of music. Still, if you think about it, creativity is a relative attribute. Value depends on what other people think, it depends on culture and society at a particular point in time. Hence, something that was not appreciated yesterday could be appreciated today. In order for you to be creative, there has to be a balance between creating something novel, while at the same time having your art tied to something that people will more or less understand. If you push the boundaries too much, people will reject it. At the end of the day, everything that we build has the potential to be creative, regardless of the creator. When you ask: “Is a machine creative?”, the question is whether it is capable of creating something new and also whether it is able to create something that is valuable for others. If you only focus on the second condition I would say that it can be fulfilled. Machines can create pieces of music that are valuable for some people. But with regard to novelty, I doubt that we have reached that point. You can argue that machines are able to create something new, but that output is not completely novel, it is non-transformational. Machine generation is more similar to exploring a certain conceptual space and finding new music, for example in the style of Bach or Pink Floyd, than to find a crazy unexpected result. We don’t have a machine that can come up with the equivalent of dodecaphony yet, unless you value randomness above all! That is what we are working on right now in the generative music field: transformational machines, which create something that has not been heard before and that has some value on its own.


“AI can challenge composers as a sparring partner.”


Considering the different music genres that are out there: How can AI support human composers?

Obviously, the less structured a style of music is, the easier it is for an AI to replicate. It is not a surprise that a lot of AI music is in some kind of ambient style. I would argue that as of now, (decent!) human composers are far better than artificial composers at creating any form of music. Still, AI has one main advantage against humans: It can create an endless stream of music. We use that advantage at Melodrive when generating music for video games in real time. That is a particularly interesting field for composers working with AI. Our vision is not to provide generic background music, but instead having AI and human composers working together at different levels.
One example could be an amateur playing around and having the thrill of influencing certain aspects of music with AI – similar to the amateur section in the Beats and Bits competition. Another example: AI can challenge composers as a sparring partner. Sometimes you can be stuck as a composer, similar to a writer’s block. This is when AI can help you come up with new ideas: You can train an AI with your own music, or you can take some AI-generated music as a useful input and repurpose it for your artistic or production needs. Moreover, each composer comes from a certain background and is influenced by certain types of music which he or she usually replicates. When you want to explore new musical spaces, AI can provide you with new ideas and push you out of your musical comfort zone.

You mentioned Melodrive’s work for video game soundtracks. Could you summarize the role of AI in this specific field of work?

For video game soundtracks we are working on interactive, non-linear music – a field that will be completely disrupted by AI in the near future. When you are playing a video game, the music should follow what you do. While a film composer already knows what is going to happen in two minutes or the next scene, that does not happen in a video game. The player is always free to decide how to continue. The music has to react to each decision, which means that you cannot provide a linear soundtrack for a non-linear piece of content. However, that comes with a huge burden for the composer. Potentially, you may need to have an infinite amount of music that is able to match the infinite possibilities that a player has when he goes through a experience. To solve this issue, AI can become an infinite composer inside video games, which is what we are working on at Melodrive. The composer still has almost full control of the music. What we envision is the composer providing high-level directions to the AI, which then uses them to implement the composer’s musical choices.


“I cannot see any risk for composers to be replaced by machines anytime soon.”


Do musicians and music composers criticize what you are doing? Are they worried or are they just happy to get another tool?

That last part is what we hope to convey when we are talking with musicians and music makers in general. The answer to your question depends on the kind of musician we are talking to. Video game composers work under the constant pressure of having to produce non-linear content with human capacity. When we started Melodrive three years ago, we did a lot of interviews with game composers and found that up to 90 percent of them were optimistic about the idea of AI getting into their field of work. They understand that AI is not going to replace their jobs but that it is just going to be another tool to work with.
If you talk to linear musicians instead, you face more skepticism. There is a lot of misunderstanding and skepticism because most people do not really understand what weird companies such as Melodrive are doing. Still, it is just a matter of perception because the quality that humans can provide is much higher than anything that can come from AI. I cannot see any risk for composers to be replaced by machines anytime soon. Obviously, it is another scenario if you talk about background music for advertisement. But then again, I think the important message to convey is that AI is just going to be another tool. I call it the AI music revolution. Specifically for generating music, AI is an incredible tool entering the musical landscape, which is going to be as revolutionary as the digital music revolution before. Intelligent agents can help us to be more creative. It is like a conversation between a music maker and the AI.

As a judge of our Beats & Bits competition, what kind of submissions do you expect from our participants?

When we started with Melodrive it took us more than a year and a half to build a working system. We invested a lot of time in AI and music research. I hope to find creative solutions that are out of the usual schemes that we normally use in generative music. I would expect different forms of collaborations between humans and machines. I am very curious about how people are going to handle this challenge in a short amount of time.

Do you have an advice for our participants before they are going to start their own work?

The competition is there to explore new things, to be open to paths that have not been walked by other people and to experiment with AI software. The history of music has always been a history of experimentation. It is an evolutionary path. I do not see exploration in AI music to be any different from that. As a judge I am going to be more interested in the processes than in the actual result. I love to see things that have not been explored yet. Obviously, in the last sixty years a lot has been done, but I am curious to see what people will come up with beyond what is normally done in the field.