Worldwide Repls Part 2: Load Balancing for Fun (Not for Profit)

Replit had previously been using Google Cloud Load Balancer (GCLB), but faced two key problems while scaling its global infrastructure.

The first is inconsistency in geographic routing. GCLB creates containers based on the user's request location, but it did not fully reflect the logic Replit intended for placing containers in the optimal location. This caused unnecessary network latency, such as requests from users in India being processed in the US.

The second is imbalance in load distribution. It was discovered that load would concentrate on specific machines, causing the service to slow down or time out, while other machines remained idle.

To address this, Replit began building its own load balancer on top of the Control Plane. At the prototype stage, they used Prometheus metrics to collect CPU usage and the number of running Repls, and validated a scalable load balancing model by applying a subsetting algorithm.