How Agents Self-Heal in Production

After deploying GTM Agent, we built a self-healing deployment pipeline so that if something breaks, the system detects and fixes it before a human steps in. The core flow is: watch the logs after deployment, determine whether it's a real regression, and only when necessary, call Open SWE to automatically open a fix PR.

The automated checks right after deployment split into two paths.

• Docker build failure: the build log and the git diff of the immediately preceding commit are passed straight to Open SWE for a fix.
• Server-side regression: error logs from the 7 days before deployment are collected as a baseline, and errors from the 60 minutes after deployment are normalized the same way and compared.

Error comparison isn't just a raw count difference—it's gated by a Poisson test. After stripping UUIDs, timestamps, and long numeric strings and truncating to 200 characters, identical errors are grouped under the same signature and compared against the expected per-hour rate from the baseline; if p < 0.05, it's flagged as a regression candidate. New errors not present in the baseline only trigger a warning if they recur repeatedly within the monitoring window.

Since numbers alone can't distinguish the cause, a triage agent steps in next. This agent takes the diff and the errors, classifies the changed files into runtime, prompt/config, test, docs, CI, etc., and only when it's a runtime change does it have to explain the causal relationship between specific code lines and the error. The output is a structured verdict containing the decision, confidence, reasoning, and the error signatures attributed to the change.

Only issues filtered this way get passed on to Open SWE, so the agent receives a narrowed investigation prompt instead of a full error dump. This structure is especially useful for catching silent failures that don't visibly crash, config mismatches, and cascading regressions that only surface in the next deployment.

Going forward, we're considering applying a wider lookback window to trace bugs planted in earlier versions, embedding error messages into vector space for clustering, and using a smaller model to classify errors before passing the results to Open SWE. We also present an alternative approach, like Ramp, of pre-generating monitors at PR merge time.

Finally, we note that right now it's always fix-forward—leaving the broken deployment as is and fixing it via PR—but in real operations, it would be better to decide whether to rollback immediately or push forward with a patch based on severity, error rate, and triage confidence. Ultimately, the goal is to automate the loop of deployment, monitoring, triage, and fixing to minimize human intervention.