EvoAudio: Recursive Self-Improvement for Audio Understanding

A closed loop in which the current model decides what it should practise next, a library of audio tools builds the waveforms that ask it, and construction rather than annotation fixes the answers.

Paper Code (coming soon) Listen to the demos
The Chinese University of Hong Kong, Shenzhen Tsinghua University Tencent Hunyuan

Abstract

Audio language models understand what is said far better than how it sounds. Closing this gap takes more than data. Detailed acoustic annotation is costly, labels from stronger models inherit their errors and limits, and fixed data cannot adapt as the learner improves. We therefore propose EvoAudio, a recursive self-improvement system for audio understanding. To our knowledge, it is the first to evolve the model, waveforms, questions, and difficulty in one closed loop. EvoAudio uses the current model's performance to set the focus and difficulty of the next training data. A library of audio tools then constructs questions whose answers follow from how the audio was made, providing verifiable supervision without new human annotation. Reinforcement learning updates the model, and validation decides whether it enters the next evolution round. Across 13 rounds, EvoAudio improves five models with different audio encoders and language backbones on MMSU, MMAU-Pro, and MMAR. It achieves the highest average for every backbone, raising overall performance by up to 6.3 points. The improvement unfolds over successive rounds, with each stronger model starting the next round.

Overview of the EvoAudio system: diagnose and plan, generate and calibrate, then learn and evolve.
Overview of the EvoAudio system. Each round profiles the current solver across 47 skills, an LLM proposer decides which skills to practise next and whether they should get easier or harder, the tool library renders the waveforms and composes the questions, acoustic verification confirms the rendered cue survived, a text-shortcut probe removes items that can be answered without listening, and GRPO trains a candidate that only becomes the next solver if held-out validation agrees.

Self-evolution across five backbones

Attempt 0 is the base model; every later attempt is the candidate that round produced, including the ones promotion rejected. All five backbones improve as evolution proceeds, and the gains accumulate over rounds rather than arriving all at once.

Accuracy against evolution attempt for five audio language models, on the three-benchmark average and on MMSU, MMAU-Pro and MMAR separately. Accuracy against evolution attempt for five audio language models, on the three-benchmark average and on MMSU, MMAU-Pro and MMAR separately.
Accuracy (%) against evolution attempt. The first panel is the unweighted mean over the three suites; the rest break it out per benchmark. MMSU gains the most, which follows the reach of the tool library: rate, pitch, loudness and event structure are what the tools control and what MMSU asks about, while the semantic and cultural reasoning of MMAR lies mostly beyond what they can build.
Three radar charts, one per backbone, comparing five training regimes across nine benchmark axes. Three radar charts, one per backbone, comparing five training regimes across nine benchmark axes.
The gain comes from the loop, not simply from more training. Each radar holds one backbone fixed and compares five regimes on the same nine benchmark axes. Every regime after Base receives an identical optimization budget and identical training settings, and they differ only in where their questions come from: Static-profile GRPO keeps the round schedule but reads its skill profile, quotas and difficulty once from the base model; Pooled GRPO makes a single pass over the union of all rounds' curricula; EvoAudio rebuilds the curriculum against the current solver every round. The EvoAudio (GRPO) polygon sits outside the others on almost every axis of all three backbones.

Demos

Every clip below was built by EvoAudio's tool library as training data, and every answer follows from how the audio was made rather than from a model's opinion about it. Pick a task family, then drag the difficulty slider to hear the same skill tighten.

Difficulty Mode

Citation

Citation details will be posted here once the paper is available.