Beyond Bots and Humans
Key point
The core issue on the web is not distinguishing bots from humans, but discerning the intent and behavior behind a request.
Details
On the web, people interact online through gateways that pass through keyboards, screens, browsers, and devices. But recent human detection can no longer be explained as a simple human/bot distinction. Users of news-summarizing browsers, tech enthusiasts running automated ticket booking at dawn, visually impaired users with screen readers turned on, and even enterprise traffic behind a zero trust proxy all leave the same surface-level signals.
Website operators still have to handle data protection, resource management, control over content distribution, and abuse prevention. But what's needed is not 'human or not,' but intent and behavior. The real questions that matter are things like: Is this attack traffic? Is the load from a known crawler proportional to the traffic it returns? Did we expect a user to connect from a new country? Is advertising being manipulated?
The web's traditional balance was built around the browser. As the user agent between server and user, the browser rendered pages and mediated conflicts between publishers and users within rules like accessibility standards and extensions. But AI agents bypass this rendering step and pull raw data directly, making it hard for publishers to tell whether it's a personal report or large-scale collection for model training. As a result, the web's implicit agreements and revenue structures are being shaken.
This situation is also evident in the client-server model. The server only sees the request, and after responding it cannot know whether the content is being consumed in a single person's browser, or automatically collected, indexed, and repurposed by a program. For a server to apply trust, prioritization, blocking, and rate limits, it needs more clues—and those same clues can also become fingerprints used for tracking.
Current signals for handling requests are divided into three layers.
- Passive client signals: signals essential to the request itself, like IP address and TLS session
- Active client signals: signals the client voluntarily provides, like User-Agent and credentials
- Server signals: signals the server observes, like edge server location and time of receipt
The problem is that all of this information is incomplete. If 10x more requests come from a single IP, you might suspect something's off, but it could also be a VPN or shared infrastructure. So typically, traffic is handled by estimating its capability and intent, but the risk of misjudgment and privacy violations remains.
The piece frames this as the rate limit trilemma. You cannot fully have decentralized, anonymous, and accountable all at once. The web is fundamentally decentralized and anonymous by default, but weak in accountability. Conversely, increasing accountability requires things like account registration or disclosure of activity, and achieving both anonymity and accountability together would require strong governance like Web PKI—but no such system exists on the client side.
Ultimately, the distinction that matters is not bot vs. human, but what is being attempted. For identifiable high-volume traffic like search crawlers, cloud platforms, and enterprise infrastructure, disclosing identity and having stable access can be mutually beneficial. On the other hand, cases like attacks, scraping, ad fraud, and fake account creation require trustworthy identification and strong controls, and future web protection needs to shift toward this purpose-based classification.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.