GitHub Copilot will run AI models on your device this month

GitHub Copilot is Microsoft's AI coding assistant. It works inside tools like Visual Studio Code and the command line, suggesting code completions and refactors based on context. So far, it has relied on cloud-hosted models.

At its event in San Francisco, Microsoft announced that GitHub Copilot will support local AI model inference by the end of the month. Developers can switch between cloud and on-device models automatically or manually, with settings available in Copilot CLI, the Copilot app, and Visual Studio Code.

The companion model, MAI Code 1.1 Flash, uses a mixture-of-experts design with 137 billion total parameters and 6.8 billion active ones, optimized through quantization and speculative decoding. In tests on a Surface Laptop Ultra, decoding throughput ranged from 40 to 63 tokens per second for prompts between 2K and 256K tokens.

Local setups can point to a specific provider, model, or endpoint. Developers can call the model through Windows ML or connect an OpenAI-compatible endpoint, keeping coding help available on weak networks or in privacy-sensitive scenarios.