Configuring Self-Hosted Grafana Webhooks to Trigger Automatic Investigations with AWS DevOps Agent

• Uses the Webhook Contact Point in self-hosted Grafana 12.0 or later to send alerts to the DevOps Agent
• Supports direct Webhook transmission compared to AMG and allows flexible configuration of Service Account Token expiration dates
• Automates incident creation and RCA report generation through Webhook secret authentication and Grafana Viewer token lookup
• In multi-account environments, configures a parallel setup using OAM for metrics/log sharing and IAM AssumeRole for direct investigation

Configuration and Authentication

• Authenticates by entering the Webhook secret as a Bearer token in the Webhook Contact Point
• The DevOps Agent uses a Service Account Token with the Viewer role for Grafana API queries
• Accesses Grafana inside the VPC via HTTPS (TLS 1.2 or higher) endpoints and Private Connection

Multi-Account Integration

• Allows cross-account CloudWatch metrics and log queries through OAM Sink/Link configuration
• Creates an IAM role in the secondary account and connects it as a Secondary source in the Agent Space to perform direct investigations
• A 403 error occurs if the IAM principal in Account A lacks the iam:PassRole permission

Verification and Testing

• Verifies integration in three stages: direct curl calls, Contact Point tests, and E2E Alert Rule tests
• Sets 'Continue matching subsequent sibling nodes' to ON in the Notification Policy to maintain existing alerts