Datadog Adopts Codex for System-Level Code Review

As Datadog operates a platform that monitors complex distributed systems, understanding how code changes affect the entire system is critically important. Existing static analysis or rule-based tools could only catch surface-level issues, with limits in grasping the complex interactions across the whole system.

To address this, Datadog's AI DevX team integrated OpenAI's coding agent, Codex, into its development workflow. Codex goes beyond a simple linter, providing system-level reasoning that can infer how code changes affect other modules or services.

Datadog verified Codex's performance through an 'incident replay harness' that reproduces past incident cases. Test results showed that Codex found risk factors that engineers had missed in about 22% of the incident cases reviewed, a performance that surpassed other previously evaluated tools.

Codex identifies the following key risks:

• Identifying interactions with modules not included in the diff
• Checking for missing test coverage in areas of high coupling between services
• Flagging API contract changes that could cause downstream risk

Currently, over 1,000 engineers use Codex on a regular basis. Codex plays a role in helping engineers focus more on system design, beyond simple error detection.