How to Build an Autonomous SRE Agent for Kubernetes

LangChain built an autonomous SRE agent to automatically classify failures in Kubernetes clusters and reduce response times. The agent checks cluster status, diagnoses issues, and suggests fixes when necessary, but is designed so that actual changes always require human approval.

Kubernetes emits a vast amount of signals, including Pod status, restart counts, HPA status, node conditions, warning events, and deployment readiness, but does not synthesize them. The agent performs the following tasks:

• Proactive monitoring: A scheduler periodically collects cluster status using the Kubernetes Python client and makes a single call to Claude Haiku to deliver a health report by severity level to Slack.
• On-demand investigation: When an issue arises, specialized sub-agents such as pod-inspector, scaling-analyzer, performance-analyzer, log-analyzer, security-auditor, and reliability-auditor investigate in parallel and then generate a prioritized report.
• Safe change management: The agent can read the entire cluster but cannot make changes on its own. Write operations such as scaling deployments, restarting rollouts, and patching HPAs are routed to the change-executor sub-agent, and every tool call goes through HITL (human-in-the-loop) approval.

The implementation uses Deep Agents running on LangGraph, leveraging planning, sub-agents, and HITL interrupts as core features. Instead of one giant prompt, it composes specialized agents with narrow roles to improve parallelism and context efficiency, assigning Claude Sonnet to the orchestrator for reasoning tasks and Claude Haiku to read-only tasks and periodic checks.

For periodic checks, it does not run the full orchestrator; instead, it collects status with standard Python code and calls Haiku only once. This results in a 95–99% cost reduction per check while maintaining failure detection performance, with complex investigations handled on-demand. Read operations are automated, while write operations require human approval via Slack, and the same principle is enforced through cluster RBAC.