Human-Augmenting Agent Workflow for Causal Inference

As data analysis is increasingly delegated to software agents, the importance of Oversight to verify the validity of results is growing. In particular, in the field of Observational Causal Inference (OCI), which requires specialized expertise, there is the challenge that it is difficult to judge whether an agent has properly handled bias.

To address this, Netflix designed an agent workflow called oci-agent. This system automates repetitive and error-prone tasks (covariate balance checks, sensitivity analysis, etc.), pursuing a 'Human-Augmenting' approach that helps human experts focus on higher-order tasks such as framing questions or testing hypotheses.

This workflow adheres to the following Design Diagnostics principles.

• Covariate balance: Keep the standardized mean difference between the treatment and control groups below 0.2 after weighting
• Overlap: Restrict the propensity score to between 0.1 and 0.9
• Placebo outcome: Verify that the effect on pre-treatment variables is close to zero
• Sensitivity to hidden confounders analysis

Netflix tested this agent on the ACIC (Atlantic Causal Inference Conference) dataset, and it systematically outperformed the one-shot approach, demonstrating competitive results even when compared against manually tuned benchmarks.