Performance Limitations of Open-Source AI Detectors

An evaluation of six major open-source AI text detection models using the same protocol showed that four out of six models failed to achieve the 0.5% false positive rate (FPR) threshold.

Key Findings

• Below Threshold: The MAGE model misclassified 26% of general human web text with scores above 0.9999, making it impossible to achieve a 0.5% FPR. OpenAI's existing RoBERTa detector performed worse than a coin flip against the latest generative models, with an AUC of 0.31.
• Vulnerable to Humanized Text: The ability to detect AI text that has been polished or modified to look human (humanizer-paraphrased) was extremely low. Even the best-performing model recorded a 42% detection rate (recall), while the second-best model achieved only 4%.
• False Positives on Non-Native Text: All models exhibited systematic errors, classifying non-native English text, such as TOEFL essays, as AI-generated at rates significantly higher than the baseline.

Model Performance (FPR 0.5% Threshold)

| Model | ROC-AUC | Original AI Detection Rate | Humanized AI Detection Rate | Latest Model Detection Rate | | :--- | :--- | :--- | :--- | :--- | | tropa-mini | 0.968 | 93.2% | 41.6% | 33.6% | | desklib/ai-text-detector-v1.01 | 0.875 | 83.9% | 4.0% | 1.8% | | SuperAnnotate/ai-detector | 0.824 | 0.5% | 1.4% | 0.6% | | Hello-SimpleAI/chatgpt-detector-roberta | 0.571 | 0.8% | 0.4% | 0.2% | | yaful/MAGE | 0.507 | - | - | - | | roberta-large-openai-detector | 0.313 | 0.0% | 0.1% | 0.0% |

• Note: MAGE cannot achieve a 0.5% FPR at any threshold.

This evaluation was conducted using public datasets (NBER, TOEFL, FineWeb, etc.) and text generated by the latest frontier models (GPT-5.x, Claude Opus 5, Gemini 3.x). The full dataset and methodology are available via Hugging Face model cards for reproducibility.