Ant Group's Bailing Open-Sources Ling-3.0 Base Models with 6 Checkpoints

Ant Group's Bailing lab has open-sourced the Ling-3.0-tiny and Ling-3.0-flash models this month, and now released the Base versions. The release includes not only the final Base checkpoints but also weights from pre-training and mid-training stages, totaling 6 checkpoints. This helps researchers with tasks like continued pre-training, fine-tuning, preference optimization, RLHF, and distillation.

Ling-3.0-tiny-base has 7.9B total parameters with 1.3B active, using about 50% fewer parameters than the previous generation, yet it scores higher in most benchmarks, especially in coding. Ling-3.0-flash-base has 124B total parameters with 5.1B active, using a sparse activation design, offering a solid base for larger-scale training and real-world adaptation.

The team notes that Base models are not aligned for chat, so they shouldn't be used directly in user-facing services or safety-critical apps without proper post-training. Also, they replaced traditional learning rate decay with a method called WSM (Warmup-Stable and Merge) after mid-training, which makes the models better for continued pre-training and dynamic data expansion.