How to Build an AI Software Factory: Agents That Open, Review, and Merge PRs

Based on Addy Osmani's concept of 'harnessing loops at scale', this article presents a 5-stage gate architecture for a software factory where AI agents generate, review, and merge PRs. The core principles are queue-based processing, ephemeral isolated environments, and rigorous verification before review.

5-Stage Gate Architecture

1. Intake: Adopts Anthropic's brain/hands/durable log structure. Shopify allows only public Slack channels, while Stripe prevents duplicate PRs via a Firecrawl index. Analysis of Microsoft dotnet/runtime shows a success rate of 76-80% for clear change tasks, but vulnerability in architectural design.
2. Isolation: Choose between Git worktree, Container, or Cloud sandbox as needed. Ramp achieved faster speeds and infinite concurrency compared to local environments through cloud sandboxes.
3. Tools: MCP-based tool integration is key. Microsoft improved agent success rates from 41.4% to 54.5% using AGENTS.md and data-driven preparation.
4. Verification: The differentiator of the factory. Faire demonstrated that model self-reported confidence is unsuitable as a quality metric (3% variance in acceptance rate), while Uber applies visual verification and precision-first reviews.
5. Merge Gate: Uses tiered routing where mechanical changes are auto-merged and complex changes undergo human review. Stripe mandates human review after two CI runs.

Measured Data and Industry Trends

• Productivity: Stripe merges over 1,300 agent PRs per week, and Shopify processed approximately 1/8 of all PRs via agents over 30 days. Airbnb used LLMs to migrate 3,500 test files in 6 weeks.
• Quality and Risk: According to the METR benchmark, approximately 50% of PRs passing SWE-bench are impossible to actually merge. GitHub analysis shows that after AI adoption, duplicate code blocks increased 8-fold and refactoring rates plummeted.
• ROI: According to the DORA 2026 report, the first-year return on AI adoption is approximately 39%; efficiency for simple tasks improves by 35-40%, but remains below 10% for legacy code.

Conclusion and Limitations

Verified workloads are limited to large-scale mechanical changes (migrations, maintenance) and are unsuitable for greenfield design. Key limitations cited include review capacity bottlenecks, parallel branch conflicts, and unclear cost efficiency. Open-source projects (such as tldraw, curl) are trending toward suspending or blocking external reviews due to a surge in AI vulnerability reports.