ByteDance launches 3B open-source model Lance for image and video tasks

ByteDance has released an open-source multimodal model called Lance, with only 3B activated parameters. It can handle image understanding, video understanding, image generation, video generation, and cross-modal editing all in one system. Unlike common approaches that separate understanding and generation into different modules, Lance unifies them from the start, aiming to handle tasks like text-to-image and text-to-video with a single model.

The model uses a shared context with decoupled capabilities, employing a dual-stream expert architecture for understanding and generation. It introduces modality-aware rotary positional encoding to avoid confusion between different visual token types. Training was done in four stages, using about 1.5T tokens in pre-training, followed by continued training, supervised fine-tuning, and reinforcement learning, all on up to 128 GPUs.

Benchmark results show Lance scores 0.90 on GenEval for image generation, 85.11 on VBench for video generation, 7.30 on GEdit-Bench for image editing, and 62.0 on MVBench for video understanding. The model is released under Apache 2.0 license, with weights available on Hugging Face. It requires Python 3.10+, CUDA 12.4+, and at least 40GB VRAM.