Digging into the Postgres source to build a WAL receiver from scratch

Curious about the internals of PostgreSQL's pg_receivewal, I dug into the source code and ended up implementing a WAL receiver directly in Go.

After building PostgreSQL from source and attaching in debug mode, I inserted extensive tracing to check the replication loop, disconnect recovery, the completion timing of .partial files, timeline switches, and fsync timing one by one.

I also compared existing tools.

• pgBackRest: a very solid engineering product, strong in large-scale environments
• Barman: a simple, reliable orchestrator built on top of pg_receivewal and pg_basebackup

The initial goal was to reproduce the behavior of pg_receivewal as closely as possible, and within a few weeks I had a stable prototype. But just receiving WAL wasn't the end of it.

Once compression, encryption, S3/SFTP upload, old-file cleanup, and monitoring were added, I needed an internal management structure that didn't rely on external bash scripts.

• Controlling backups so they don't overlap
• Preventing conflicts between archiving and cleanup
• Safe shutdown and state management
• Managing job lifecycles with a job queue, worker pool, supervisor, and goroutine coordination

From there, the tool grew into a small system coordinating multiple background processes. The core wasn't WAL reception itself, but building a backup system that consistently controls retry, cleanup, transfer, retention, and shutdown.

I also added base backup support. I acknowledged the single-thread limitation of the replication protocol, but judged it practical enough as a several-hour overnight job for a DB of roughly 200 GiB.

On the other hand, I excluded incremental / differential backup. Considering the restore stage as well, it would require complexity on the level of pg_combinebackup, which I judged to be realistically beyond scope.