ByteDance Discusses Training a Model with Over 5 Trillion Parameters, the Largest in China

ByteDance is reportedly discussing building a model with over 5 trillion parameters, which would be the largest in China, surpassing Alibaba's Qwen and Moonshot's K3. The plan is still early, led by the head of Seed Foundation in collaboration with the lead for LLM pre-training data.

Insiders say several Seed teams have been reflecting this year, partly driving this move. The founder reassured the team, saying it's okay to lag temporarily but aiming for top-tier intelligence globally.

He also stressed coding as a key focus and opposed distillation, calling it just copying existing abilities rather than true innovation.