Alibaba Tongyi Lab Releases PrismAudio Framework: More Accurate Synchronization of Environmental Sound Effects from Videos

Alibaba Tongyi Lab recently released the PrismAudio video-to-audio framework, focusing on environmental sound and effect synthesis—such as hoofbeats, wind and rain sounds—synchronized with video content, not character dubbing. It is the first environmental sound generation tool combining reinforcement learning with chain-of-thought, letting the model 'think' before generating audio via a decomposed chain-of-thought process.

PrismAudio introduces a four-dimensional scoring mechanism and uses the Fast-GRPO efficient training algorithm to reduce training time. Tests show it outperforms state-of-the-art methods on VGGSound and the self-built AudioCanvas dataset. With only 518 million parameters, it generates 9 seconds of audio in 0.63 seconds. This research is accepted by ICLR 2026, and code will be open-sourced soon.