AI BriefingMeta's Catching JiTTests System Identifies 74 Serious Privacy Bugs in H1 2026
·2026.10.30 01:00
Key point
The system achieved a 60% true-positive rate and engaged 589 engineers through a human-on-the-loop workflow.
Details
- Meta deployed Catching JiTTests into production infrastructure, moving beyond research prototypes to catch bugs before code lands.
- In the first half of 2026, the system identified 74 serious (SEV) privacy bugs at the diff stage, reducing overall privacy SEV occurrence by 20%.
- 589 engineers (64% of outreach) engaged with the system, reporting a 60% true-positive rate for actionable findings.
- The approach uses LLMs to generate executable tests that demonstrate concrete behavioral differences, distinct from standard LLM code reviews.
- Two tandem workflows drive test generation: an "intent-aware" workflow targeting likely failure points and a "dodgy diff" workflow treating every change as a potential mutant.
- False positives are filtered using an "LLM-as-Judge" ensemble and a deterministic rule-based assessor called RubFake.
- A human-on-the-loop model reduces engineer load by having humans set criteria and review aggregate outcomes rather than supervising every individual test.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.