Replit Service Outage on May 18

On May 18 (Pacific Standard Time), an outage occurred in which all users were unable to access Replit for about 2 hours.

The root cause of the outage was a latent bug in the configuration push mechanism. When a handler was missing while adding a new configuration kind, the read-lock on a Golang RwMutex was leaked instead of being released.

At 11:45, during a routine build deployment, an attempted write-lock triggered a deadlock across all VMs. As a result, all VMs were reported as unhealthy, and the internal load balancer was unable to send traffic.

The recovery process faced the following technical challenges:

• Rollback delay: The rollback process did not proceed smoothly, and it took 40 minutes to deploy the last healthy build.
• Autoscaler malfunction: After the rollback reset system capacity, the Autoscaler detected low utilization and entered a vicious cycle of further shrinking instance size.

The system was fully recovered at 13:56, and the issue was resolved by deploying a build that fixed the deadlock bug.