How GitHub Uses eBPF to Improve Deployment Safety

To reduce circular dependency problems that arise from the structure where all of GitHub's own code lives on github.com, GitHub controlled the network behavior of deployment scripts using eBPF.

The biggest problem wasn't simply when GitHub itself became inaccessible, but cases where deployment scripts relied on internal services or external binaries, creating direct dependencies, hidden dependencies, and transient dependencies. In particular, since stateful hosts also handle customer traffic, blocking all of github.com wasn't an option.

To solve this, GitHub attached eBPF programs to a cGroup that isolates specific processes, creating a way to selectively monitor and block egress traffic from deployment scripts. The PoC consisted of Go code using cilium/ebpf, tracking packets via BPF_PROG_TYPE_CGROUP_SKB.

DNS blocking was handled with an extra layer of sophistication. Using BPF_PROG_TYPE_CGROUP_SOCK_ADDR to intercept connect4 calls, DNS requests were redirected to a userspace DNS proxy at localhost:53, which then determined whether to allow the request based on a blocklist and worked together with eBPF Maps to perform the blocking.

On top of this, GitHub also tracked which command a blocked request originated from.

• Recording the DNS transaction ID and PID in an eBPF Map
• Checking the actual command executed via /proc/{PID}/cmdline
• Delivering the cause to teams via logs in the format WARN DNS BLOCKED ...

This approach made it possible to keep an audit list of domains contacted during deployment and immediately notify teams of dependencies causing problems. At the same time, by using cGroups to also apply CPU and memory limits, it prevents deployment scripts from negatively impacting workloads.

This circular dependency detection system has been in operation following a 6-month rollout, and it automatically detects and alerts teams whenever new dependencies arise or existing tools add external dependencies. As a result, GitHub can deploy more reliably, and recovery time during outages has also decreased.