Data Loss: A Sad Story's Happy Ending

Earlier this year, it was discovered that some users' data was being lost or that changes were not being saved. Initially, the team responded by fixing individual bugs, but this actually accelerated data loss, revealing a problem with load-bearing bugs (bugs that, when fixed, cause other problems).

To resolve the issue, the following structural approach was introduced:

• Strengthened logging and monitoring at points suspected of data loss
• Applied logic to salvage whatever portions were possible instead of deleting everything when corrupted data was found
• Analyzed the output of the filesystem snapshot tool to prevent corrupted data from being permanently stored
• Built a Dataflow pipeline to perform bulk recovery of existing, old repl data

The root cause was an interaction between the way Golang's exec.Cmd handles stdin/stdout and a filesystem snapshot tool that failed to properly check for failures of certain syscalls (write(2), splice(2)) under high system load or low disk space. This caused silent truncation of data without any errors.

All mitigations have now been completed, and monitoring and testing have been strengthened to improve the system so that similar data loss does not recur in the future.