Xiaomi open-sources ControlFoley for controllable video sound effects

Video sound effects generation is moving from simple scene matching to intent-based control. Xiaomi's large model application team today open-sourced ControlFoley, a unified framework for controllable video sound effects, addressing creators' difficulty in precisely controlling audio output.

The model supports three tasks: text-guided video-to-audio (TV2A), text-controlled video-to-audio (TC-V2A), and reference audio-controlled video-to-audio (AC-V2A). Its key innovations include joint visual encoding, a timbre-temporal decoupling strategy, and modality-robust training, ensuring stable performance across multimodal inputs while prioritizing user intent.

ControlFoley achieves open-source SOTA results on benchmarks like VGGSound-Test, outperforming existing methods in semantic alignment, temporal synchronization, and audio quality. Code, model weights, and online demos are now available, rivaling commercial systems like Kling-Foley.