Healthchecks.io started using self-hosted object storage

Healthchecks.io stores the first 100kB of the ping request body, putting small bodies into PostgreSQL and offloading large bodies to S3-compatible object storage.

Previously it used AWS S3, OVHcloud, and UpCloud in that order, and while cost and operating conditions were fine, over time it repeatedly ran into performance degradation, server errors, timeouts, and especially delays with DeleteObjects.

The operating scale is as follows.

• 14 million objects, 119GB
• Object size: 100 bytes ~ 100,000 bytes, average 8KB
• Average 30 uploads/sec, peak 150 uploads/sec
• Continuous upload/delete churn

As self-hosted alternatives, Minio, SeaweedFS, and Garage were considered, but for a one-person operation, taking on cluster automation, upgrade procedures, failed-node replacement, and monitoring made the operational complexity too heavy a burden.

The final choice was Versity S3 Gateway. This tool makes a local filesystem behave like an S3 server; there's no separate metadata DB, and file create/read/delete directly correspond to S3's Put/Get/Delete. Upgrades were as simple as swapping the binary and restarting systemd.

The current setup is as follows.

• The S3 API runs on a dedicated server
• Application servers access it through a Wireguard tunnel
• Storage is 2 NVMe drives in RAID 1 with a Btrfs filesystem
• Changes are synced to a backup server via rsync every 2 hours
• The backup server encrypts a full daily backup and stores it offsite
• Full backups are kept for the last 30 days

The biggest tradeoff of this approach is durability. If both the object storage server and the two disks fail simultaneously, up to 2 hours worth of ping bodies could be lost. Still, this was judged to have less impact than an outage of PostgreSQL, the primary data store.

After the switch, S3 latency dropped, and the backlog of pending ping body queues also decreased. It's only been a few weeks so far, so long-term stability still needs to be watched, but there have been no availability issues so far.

In conclusion:

• Cost increased compared to managed object storage
• In exchange, performance and controllability improved
• While keeping the simplicity a one-person team can handle, they found a better operational tradeoff
• They remain open to migrating again if a system with a better tradeoff comes along