LLMxRay Released: Open-Source Observability Tool for Local Ollama Models

LLMxRay is an open-source observability tool designed for local LLMs running on Ollama. It offers a web-based interface to monitor performance, debug prompts, and analyze model behavior in real-time. The tool acts as a transparent proxy, allowing developers to inspect traffic from various clients like VS Code Copilot, Open WebUI, and custom scripts.

Key Features

• Prompt X-ray: Visualizes prompt structure, showing which parts are cached, re-read, or dropped by the daemon. It helps identify cache misses caused by changing tokens like timestamps.
• Shared Box: Diagnoses performance bottlenecks by showing how different tools impact model availability and context usage.
• GPU Map: Displays video memory usage, showing loaded models, context costs, and free space.
• Live Monitoring: A second port (11435) proxies requests to Ollama, logging client, model, prompt size, cache hits, and speeds for all connected tools.
• Cache Lab: Measures the impact of prompt structure on cache reuse, demonstrating significant speed improvements (e.g., 3.5x faster prefill) by optimizing token placement.
• Surgical Benchmark: Evaluates models on curated suites (MMLU-Pro, ARC-Challenge, etc.) using real token logprobs to assess confidence versus accuracy.
• Protocol Observatory: Compares Ollama's native, OpenAI-compatible, and Anthropic-compatible endpoints side-by-side.

Installation and Compatibility

LLMxRay can be installed via npx llmxray or Docker. It is tested against Ollama 0.40.x and works with versions 0.20 and newer. The tool runs locally, storing data in the browser's IndexedDB, and includes an MCP server for agent integration.