AMD launches open-source Instella-MoE models with 2.8B active parameters

AMD released the Instella-MoE open-source model series on July 24. It has 16 billion total parameters but only 2.8 billion active ones. The models use 27 decoder layers and were trained on AMD's own Instinct MI300X and MI325X GPUs.

The series includes six versions, covering pre-training, long-context, and supervised fine-tuning. The Instella-MoE-16B-A3B-Think version, after reinforcement learning, beats Google's Gemma-4-E4B-it with fewer active parameters.

Training was done entirely on AMD's ROCm software stack. The FarSkip-Collective technique boosted pre-training speed by 12.7%. For inference, the SGLang framework cuts first-token response time by up to 39.2%, showing AMD's strength in AI infrastructure.