Karrot SRE builds action-runner-job-exporter to monitor GitHub Actions ARC runners
·2026.11.17 18:49
Key point
The tool uses three components to map jobs to pods, track queue wait times, and identify controller throughput issues in self-hosted ARC environments.
Details
- Karrot SRE developed action-runner-job-exporter to visualize which GitHub Actions jobs are waiting for runners and which runner pods are executing specific jobs, addressing gaps in ARC's built-in metrics.
- The exporter consists of three binaries: Collector maps job names to runner pods using EphemeralRunner status, Webhook tracks queued and in-progress jobs via GitHub webhooks, and StateStore persists state to S3 to survive restarts.
- Production incidents revealed state management challenges, including stale snapshots causing 72 completed jobs to appear queued and webhook delivery failures leading to incorrect counts, resolved by separating retention periods and implementing reconciliation logic.
- The tool identified an ARC controller bottleneck where increasing runnerMaxConcurrentReconciles from 2 to 5 reduced the controller work queue from 669 to 233 and waiting jobs from 380 to 168 within 10 minutes.
- Metrics also enable right-sizing runners by comparing actual CPU and memory usage against configured requests, helping teams optimize resource allocation and reduce scheduling delays.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.