GitHub Replaces LLM-Based Secret Scanning with Fine-Tuned ModernBERT Classifier

GitHub has deployed a new secret scanning pipeline that replaces its previous LLM-based system with a fine-tuned ModernBERT classifier, achieving significant gains in speed, cost, and accuracy. The new approach uses a two-stage process: regex filters generate candidate secrets with high recall, and the compact classifier evaluates each candidate's context to distinguish real credentials from lookalikes. This shift allows GitHub to extend generic secret detection to push protection, providing feedback before code is committed rather than only after.

Performance and Cost Improvements

The transition to a specialized encoder model has delivered measurable improvements in production:

• Serving Cost: Reduced by more than 98% compared to remote LLM calls.
• Execution Time: Median post-push scan execution time dropped by 86%.
• User Acceptance: Repository owner-weighted acceptance increased by 11.5%, indicating fewer false positives and higher trust in alerts.

Offline evaluations showed the new pipeline achieved 13% higher precision and 19% higher recall than the previous LLM pipeline on the same public traffic. The classifier is exported to ONNX format with 8-bit quantization to ensure fast inference on GitHub's infrastructure.

How the Pipeline Works

The system addresses the challenge of "hard negatives"—strings that match credential patterns but are not usable secrets, such as test fixtures, documentation placeholders, or checksums.

1. Candidate Generation: Regex patterns scan content for potential secrets, prioritizing speed and broad coverage.
2. Context-Aware Classification: The fine-tuned ModernBERT model analyzes the candidate value alongside its file path, variable names, and surrounding code. It learns to recognize contextual signals (e.g., words like "example" or "dummy") that indicate a false positive.

Training and Future Outlook

The model was trained on a combination of product-derived examples from GitHub repositories, high-recall regex candidates, and synthetic examples targeting difficult cases. Training emphasized hard negatives to help the model distinguish between syntactically similar but semantically different strings.

GitHub notes that model size does not equate to usefulness; a smaller, fine-tuned classifier is often superior for narrow tasks with measurable decision boundaries. Future work focuses on building a continuous improvement flywheel using production feedback and exploring architectures that combine detection and extraction in a single step.