Mesh LLM: Distributed AI Computing Based on iroh

Mesh LLM is a distributed inference framework that combines the computing resources of individual devices (laptops, GPU rigs, mini PCs, servers, etc.) into one, providing them via an OpenAI-compatible API.

This system uses the iroh protocol to form direct QUIC connections between devices without a central server, and supports communication even in complex network environments through NAT traversal and hole punching.

The main operating modes are as follows:

• Local execution: Directly uses the GPU of the current device
• Routing: Forwards requests to another peer that already has the model loaded
• Split Mode: Applies the 'Skippy' technique, which splits a massive model that cannot run on a single device across multiple devices on a layer-by-layer basis

Mesh LLM supports a plugin architecture through MCP, HTTP, inference, and offers a catalog of more than 40 models, ranging from small models with 500 million parameters up to MoE models with 235B parameters.