AudioCraft – Meta’s Generative AI for Audio and Music

Facebook parent company Meta, released a new generative AI tool, AudioCraft, that creates sounds from text-based prompts. In a post, the company opens with a thought experiment. “Imagine a professional musician being able to explore new compositions without having to play a single note on an instrument.”

These are some pretty heavy words coming from the company as they claim that the new AI tool allows for the generation of high-quality, realistic audio, and music from text prompts. What’s more interesting, is that this is coming at a time of greater scrutiny toward generative AI content.

According to the post, AudioCraft consists of three models: MusicGen, AudioGen, and EnCodec. The first, MusicGen was trained with Meta’s own and licensed music and can generate music from text prompts.

AudioGen which was trained using public sound effects, can generate audio from text prompts as well. A thing of creating environmental sounds or other sound effects such as a dog’s bark, car horns, footsteps a specific type of floor, and more.

It seems that the difference between the two is music creation and sound effects focused on the latter. Then, there is Encodec. This model can create higher-quality music generation with fewer artifacts according to the company.

All three models are being open-sourced and the company is giving researchers and practitioners access so they can train their models with their datasets. The results could be quite interesting considering that the world of AI-generated music and sounds is still in its infancy.

Right now, the models aren’t available for anyone to just write in a prompt and hope a lovely sound is created. The models, according to Meta, still require more work due to their complex nature. But the team promises that the AudioCraft family of models is capable of producing high-quality audio.

The company is hoping that the AudioCraft family of models will be seen as new tools to help musicians and sound designers explore their craft. This hope also extends to inspiring their work. If proven successful, and popular, this should be the future that DJ David Guetta saw when he mentioned AI as being a critical part of the future of music.