Baidu opens up ERNIE 5.5 Flash Base weights, a 280B sparse MoE model

Baidu's ERNIE team has released ERNIE-5.5-Flash-Base, opening up the weights at the end of pre-training. It is a sparse MoE model with 280B total parameters and 14B activated per token, pre-trained from scratch on 33.5T tokens using a Muon optimizer and FP8 mixed precision. The model trains with up to 128K context and supports 1M tokens for fine-tuning and inference. The training data focused on web pages, code, and long documents, with code data covering commits, issues, code reviews, and CI-verified fixes. These weights have not gone through mid or post training, so they can serve as a shared starting point for domain adaptation, instruction fine-tuning, and reinforcement learning research. Baidu tested the model on 30 benchmarks across seven skill areas, and it performed well overall. A post-trained ERNIE 5.5 Flash is coming soon.