WebLLM Releases Browser-Based LLM Inference Engine Powered by WebGPU

WebLLM is a high-performance engine that enables large language model (LLM) inference directly within web browsers without server support. It leverages WebGPU for hardware acceleration and was developed as a companion project to the MLC LLM project.

Key Features

• OpenAI API Compatible: Provides an interface identical to the OpenAI API, including streaming, JSON mode, and function calling (WIP), making integration with existing applications easy.
• Broad Model Support: Natively supports major open-source models such as Llama 3, Phi 3, Gemma, Mistral, and Qwen (Tongyi Qianwen).
• Structured JSON Generation: Supports structured generation in JSON mode with optimized performance via WebAssembly components.
• Plug and Play: Easily installable via NPM, Yarn, CDN, etc., and supports Chrome extensions and Web Workers/Service Workers to optimize UI performance.

Usage and Deployment

Developers can install the @mlc-ai/web-llm package or import the module via CDN to create an engine and load models using the CreateMLCEngine() factory function. This allows building AI assistants or chatbots that preserve privacy while benefiting from GPU acceleration.