#load-balancing

The latest AI and developer news about #load-balancing, with the original source and a short summary.

vLLM Router: Inference Request Routing Considering Sessions and Caches

vLLM Router provides routing functionality that distributes inference requests while considering sessions and caches.

·2026.11.12 20:00

Olla v0.1.0 released as lightweight open-source LLM proxy for homelabs and on-prem inference

Olla v0.1.0 adds Hugging Face TEI support, a reworked dashboard, and improved failover for managing local LLM inference fleets.

·1·2026.11.03 20:00

Bifrost - Ultra-fast Enterprise AI Gateway

Bifrost has unveiled an OpenAI-compatible Gateway that unifies 15+ providers and 1,000+ models.

·1·2026.05.11 10:00

The Current State of Model Serving Routing

Netflix routed model serving traffic by purpose using Switchboard.

·1·2026.05.02 06:00

An Almost-Free Method to Improve MoE LLM Training Performance: Global-Batch Load Balancing

Applying load balancing at the global-batch level instead of the micro-batch level significantly improves MoE model performance and expert specialization.

·1·2025.01.21 01:00

Global Repl Deployment, Part 3: Getting the Engines Running

Replit distributed its infrastructure worldwide to reduce latency and built a cluster-based, isolated architecture.

·2023.03.17 01:00

Worldwide Repls Part 2: Load Balancing for Fun (Not for Profit)

Replit built its own load balancer to solve geographic routing limitations and uneven load distribution problems.

·1·2022.09.24 03:00