Huge Kimi K3 AI model runs without a GPU using old Intel Optane memory

Optane is a non-volatile memory technology co-developed by Intel and Micron, based on 3D XPoint. Its Persistent Memory 200 series looks like server RAM sticks, offering 128GB to 512GB per stick for 3rd Gen Intel Xeon Scalable processors, and it has been discontinued.

A Reddit user shared a home lab experiment, running Moonshot AI's Kimi K3 model with 2.8 trillion parameters without any graphics card. They used a 40-core 3rd Gen Intel Xeon Scalable processor with the Optane Persistent Memory 200 series in App Direct mode, and used llama.cpp for inference.

It is still just a test, and the token speed is slow. With 1-bit quantization, it started at about 0.85 tokens per second, rising to 1.50 tokens per second after turning off DAX so hot data could use the DRAM cache. A 3-bit test only hit 0.75 tokens per second, and making 50,000 tokens took about 18.5 hours.