New AI benchmark for phones, tested on iPhone 17 Pro

Artificial Analysis and Liquid AI released a new benchmark for phone AI models, focusing on how they run on the iPhone 17 Pro. The test includes five tasks: function calling, instruction following, knowledge, tough questions, and math. Models must be under 8GB after 4-bit compression, like Gemma 4 E2B and LFM2-2.6B-Exp.

With a 16K token limit, Nanbeige4.2-3B and LFM2.5-2.6B scored highest. When limited to one minute, LFM2.5-8B-A1B came in first. Nanbeige4.2-3B, made by BOSS Zhipin's lab, uses a special design to boost depth without adding more parameters, with about 4 billion in total.