Jack Dorsey Recommends mesh-llm: Idle GPUs Can Form Decentralized Inference Networks

Frustrated by insufficient GPU memory to run large models? mesh-llm, recommended by Jack Dorsey (co-founder of Twitter and Block), might help. This tool connects idle GPUs into a peer-to-peer network for open-source large models, developed by Block's engineer and part of the Goose ecosystem under MIT license.

mesh-llm adapts automatically: Dense models split into layers for parallelism, MoE models by experts with zero inter-node traffic. Tests show GLM-4.7-Flash runs at 68 tok/s alone, dropping to 21 tok/s with 2 nodes over WiFi—but for users with 24GB GPUs wanting 142GB models, it makes 'running possible'.

Feasibility comes from different communication modes: inference transfers only activation values (low traffic, latency doesn't affect token speed), while training needs gradient sync (high bandwidth).