Study Finds 39.3% of AI Search Links Are Unused in DeepSeek and Qwen Responses

A new empirical study analyzing 160,860 cleaned citations across four major Chinese AI assistants—DeepSeek, Qwen, Doubao, and Yuanbao—reveals a significant disconnect between displayed sources and actual content generation. The audit found that 39.3% of the links shown to users were never cited in a single sentence of the final response.

Key Findings on Source Usage

The study, titled "What Do Chinese-Language Generative Search Engines Cite and Surface?" (arXiv:2607.15771), tested 614 questions three times each on both web and mobile versions of the assistants. Key metrics include:

• Unused Footnotes: 39.3% of listed links were ignored by the model when writing the answer.
• Dropped Brand Names: Of 632,256 brand mentions found within cited pages, only 8.3% appeared in the AI's final output.
• Ghost Mentions: 13.02% of names recommended in the answers did not appear in the cited sources at all, suggesting reliance on pre-trained memory over live search results. This rate rose to 27.5% for vague questions.
• Repetition vs. Authority: Traditional domain authority scores ranked 10th out of 11 factors influencing inclusion. Instead, repetition across separate websites was far more critical; a name appearing in 11 or more links had a 59.9% chance of inclusion, compared to just 2.1% for names in a single link.

Platform Discrepancies

The study highlighted significant inconsistencies between platforms and interfaces:

• Web vs. Mobile: On Qwen, 64.2% of identical questions shared zero exact URLs between the web browser and mobile app. DeepSeek showed the highest consistency with a 40.8% link overlap, followed by Doubao (31.0%) and Yuanbao (9.7%).
• Pre-trained Memory Override: Broad prompts frequently triggered answers based on pre-trained knowledge rather than retrieved web data, leading to higher rates of unmatched names.

Limitations

The authors noted that their scraper only analyzed main body text, excluding pop-ups, footers, and code, which may account for a portion of the 13.02% unmatched rate. Additionally, semantic scoring was performed using an LLM rubric rather than human annotators.