AI Briefing
KOSign in

AI scraper bots overwhelm university libraries, blocking user access

·2026.11.14 23:51

Key point

A GLAM-E Lab survey found 39 of 43 cultural collection maintainers saw traffic rise, with 27 blaming AI bots that ignore robots.txt.

Details

  • AI scraper bots are overwhelming university library systems, causing lockouts for students and staff; UNC Chapel Hill recorded over 11,000 searches for Finnish music in a single day from thousands of IP addresses.
  • A 2025 GLAM-E Lab survey of 43 online cultural collection maintainers found that 39 reported increased traffic over the previous year, and 27 attributed the rise directly to AI training data bots.
  • Bots ignore robots.txt, request thousands of documents per second, and download content in every available format, sometimes causing sites to go down completely.
  • Anthropic, OpenAI, Google, and Microsoft support a standardized robots.txt opt-out signal, but compliance is voluntary and does not cover smaller AI startups or third-party scrapers.
data-collectionai-scrapingrobots-txtuniversity-librariesinfrastructure-impact
Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.

Related

Hacker NewsAI scraping bots disrupt university library access as major firms support opt-out standardTLDR AIHUMXN offers free home repairs in three cities to train AI robots with real-world dataCloudflare BlogCloudflare introduces continuous trust scoring for Verified botsGeekNewsAnthropic Engineer Clarifies Claude Code Prompt Suggestions Are Not Used for RLHF TrainingTLDR AIPantheon Team Builds 1M-Task UMI Dataset for $10/Hour Using Custom Hardware and Argus PipelineHacker NewsOpenAI Tracks User Behavior on External Websites via Cookies Linked to ChatGPT Accounts