Agents as a Stepping Stone for Repetitive Tasks

For tasks like security vulnerability patching, which are repetitive but full of exceptions, it's more effective to start by prototyping with an agent and later move to a code-centric workflow.

In one case, Dependabot webhooks were fed into an internal agent framework, aiming to filter out only high-priority issues and send them to Slack. After upgrading from GPT 4.1 to GPT 5.4 high reasoning, the agent got quite good at using Github MCP to find the right owner based on clues like the Codeowners file and recent commits.

However, the constraint of only sending critical issues was never reliably maintained. Even with repeated CRITICAL: you must... instructions, high and occasionally even medium issues slipped through, and an approximation at the level of "send a notification to a human" wasn't good enough.

The solution was a structure where code handles flow control, not the agent.

• After receiving the webhook, a script extracts the severity and action
• Low priority or non-actionable items are filtered out first
• Code packages issues into bundles by repository
• Each bundle is passed to an agent to identify the responsible team or person
• A second agent formats the result into a Slack message

With this change, the operation became 100% reliable, while still keeping the internal ownership skill and Github MCP. This made it possible to roll out more aggressively.

The same pattern was later applied to weekly follow-up notifications, and the next step is to automate the vulnerability fix itself. The goal is to progressively reduce the points where humans need to intervene, eventually reaching a structure where humans only review changes, leading into automatic deployment.

There are three core patterns:

• First, prototype with an agent to identify where the task is hard
• Then move control logic into code, increasing the deterministic portion
• Finally, leave only the parts requiring ambiguous judgment, such as identifying code owners, to the agent, where it's strong

This approach is faster, cheaper, and easier to maintain. The switching cost is also low—with just recent execution logs, prompts, and brief instructions, you can build an alternative implementation with Codex or Claude in minutes.