iroh's Mesh LLM pools GPUs across machines into one OpenAI-compatible endpoint

iroh released Mesh LLM on July 11, an 18 MB tool that pools GPUs and memory across multiple machines and exposes the combined compute as a single OpenAI-compatible API endpoint at localhost:9337/v1. The pitch is running large models on whatever hardware is on hand, without renting cloud GPUs, behind an endpoint that existing OpenAI-compatible apps can call unchanged.

iroh, legally n0, inc., is the company behind the peer-to-peer networking library of the same name. Mesh LLM routes each request one of three ways: to a local GPU, to a peer that already has the model loaded, or by splitting a large model across machines by layer range — a mode it calls Skippy — for models up to 235 billion parameters.

It ships with more than 40 supported models, from roughly 500-million-parameter models that fit on a laptop to 235-billion-parameter mixture-of-experts systems. Connections run over iroh’s authenticated QUIC transport using three application-layer protocols, with two relays in separate regions as a fallback when direct peer connections fail.

The distributed-inference approach is not new in research, but packaging it into a small client with a drop-in API is the selling point here. iroh’s post is the only public account of Mesh LLM so far, and neither the Skippy layer-split throughput nor its performance on real multi-machine workloads has been independently tested. The Mesh LLM launch post details the protocol design and the supported model list.