A one-line Kubernetes fix that saved 600 hours a year

There was a problem where every restart of Atlantis, which plans and applies Terraform changes, would stall for 30 minutes. With about 100 restarts needed per month, every credential rotation and onboarding blocked the entire engineering workflow and even triggered on-call alerts.

The cause was Kubernetes' default behavior. Atlantis runs as a singleton StatefulSet and used repository state stored on a PersistentVolume (PV), but as the number of files on the PV grew to millions, the mount process at restart became abnormally slow. Inodes ran out and the PV had to be expanded, and it was during this process that the true nature of the problem surfaced.

Running kubectl rollout restart statefulset atlantis would immediately create a new Pod, but in reality it stayed stuck in the Init:0/1 state for a long time. Pod events alone weren't enough to find the cause, so digging deeper by checking the kubelet logs revealed a long gap right after the PV mount.

The key clue was the following log:

• Setting volume ownership ... and fsGroup set
• A warning that ownership changes can be slow when there are many files

In other words, when Kubernetes mounted the PV, it was recursively changing group ownership across the entire filesystem because of fsGroup. With so many files and directories, an operation close to chgrp -R was happening every single time, and as a result restart time stretched to 30 minutes.

The fix was simple. fsGroupChangePolicy, supported since Kubernetes 1.20, was added to the Pod's securityContext, and the default Always was changed to OnRootMismatch. If the permissions on the root directory already match, the entire PV isn't scanned again, avoiding unnecessary recursive permission changes.

• Before: restart took about 30 minutes
• After: restart takes about 30 seconds
• Savings: over 50 hours per month, roughly 600 hours per year

The default is reasonable for small volumes, but as data grows it silently becomes a bottleneck. For workloads using large PVs, it's worth checking fsGroup and fsGroupChangePolicy and confirming whether OnRootMismatch is safe. This case shows that a single question about why the system behaves the way it does can save a huge amount of time — more than any complex configuration.