Robots.txt: The Birth of the Agentic Internet

How a 1990s text file reveals what we're losing—and need to rebuild—in the age of AI agents

In the mid-1990s, as the web rapidly expanded, website administrators faced a growing problem: bots. These were simple scripts written to crawl pages and index them for search engines. Harmless enough—until the traffic became overwhelming. Some bots would aggressively crawl thousands of pages, disrupting service. Others scraped content for less noble reasons.

The solution was elegantly simple: a text file called robots.txt placed at the root of every website. This file told visiting bots where they could and couldn't go. There was no enforcement mechanism. No central authority. Just a voluntary standard that said: "Here are my preferences. Please respect them."

What made this remarkable wasn't the technology—it was the assumption. The creators of robots.txt believed that machines could be taught to listen to human intent and choose to honor it.

It was the first handshake between human agency and machine agency on the internet.

The Original Deal

The robots.txt protocol worked because it established something unprecedented: a conversation between humans and autonomous programs. Webmasters would publish their boundaries in a machine-readable format. Bots would check for these files before acting. Good bots would comply. Bad bots would ignore them.

As I explored in a 2015 MediaPost article, this created the web's first primitive governance model around bot behavior—not through control or restriction, but through negotiated consent. The distinction between "good bots" and "bad bots" wasn't just technical; it was social. Good bots listened. Bad bots didn't.

This was machine agency in its infancy—programs making choices about how to behave based on human preferences they chose to understand and respect.

The beauty was in its simplicity. A few lines of plain text could say "crawl my blog but stay away from my private photos" or "index my articles but don't overwhelm my server." The bot had to decide whether to listen. Most did.

From Information Age to Cognition Age

What we didn't realize then was that robots.txt marked the beginning of a much larger transition. We were moving from what I now call the Information Age to the Cognition Age.

In the Information Age, value revolved around accessing and organizing information. Bots were simple: they crawled, indexed, and retrieved. The web was a vast library, and we browsed it manually.

Today, we're crossing into the Cognition Age, where the agent is the interface. Intelligent agents don't just retrieve information—they interpret intent, negotiate meaning, and act autonomously on our behalf. Browsing is giving way to delegation.

Here's What Makes This Shift Different

This fundamentally changes how we interact with computers. Instead of clicking through websites or typing exact commands, we now engage in what I call delegated intent—we express goals and preferences, and agents interpret and act on them.

Think about it: when you ask an AI assistant to "schedule a meeting with the team," you're not giving step-by-step instructions. You're delegating your intent and trusting the system to figure out the details. But delegation creates a new challenge: how do we ensure our intent stays clear as it passes through layers of machine interpretation?

What We've Lost

Fast-forward thirty years. Today's AI agents are vastly more sophisticated than those early web crawlers. They don't just index pages—they write emails, book appointments, generate content, make purchases, and hold conversations. But somewhere along the way, we lost the handshake.

Modern AI agents rarely check for our robots.txt file—literally or metaphorically. Instead, they operate on inferred permissions, guessed preferences, and assumed consent. An AI assistant might book a meeting based on your email patterns without asking first. A recommendation algorithm shapes what you see based on behavioral data you never explicitly shared.

The voluntary listening that made robots.txt work has been replaced by systems that act first and ask questions later.

This Creates a New Problem

This shift is particularly concerning when we consider the extractive patterns that emerged during Web 2.0. Social platforms centralized attention and monetized user data under the guise of convenience. They became proxies for users, often acting on their behalf without genuine consent or transparency.

My fear is that these patterns will accelerate in the agentic web. As delegation increases, the incentives to monetize, redirect, or manipulate intent grow stronger. If we don't build systems that let users see and steer their delegation, we risk repeating Web 2.0's mistakes—only faster, deeper, and harder to unwind.

The Agency Problem

This shift reveals something fundamental about how human agency and machine agency intersect. Early bots were simple but respectful—they had clear instructions and checked for permission. Today's agents are intelligent but presumptuous—they make sophisticated inferences about what we want without creating space for us to say what we actually want.

The robots.txt model worked because it respected both types of agency: humans could express their boundaries, and machines could choose whether to honor them. It created space for collaborative decision-making—a partnership rather than a replacement of human choice.

Consider how different these interactions feel:

1995 web crawler: "I see you have a robots.txt file. Let me read it and respect your boundaries."

2025 AI agent: "Based on your past behavior, I've determined you probably want me to do this thing. I'm doing it now."

The first preserves human agency while enabling machine autonomy. The second replaces human agency with algorithmic assumption.

As I've observed in tracking the evolution from bots to browsers, we're seeing a new phase where bots are becoming the infrastructure of digital experience. They're no longer just crawling—they're actively mediating how attention flows and value is captured.

This isn't just about privacy or control—it's about how permission should work in human-machine interactions. robots.txt worked because it assumed machines should ask permission, not forgiveness. But more importantly, it created a primitive form of AI-to-human accountability: you could see which bots visited your site and whether they respected your preferences.

That accountability mechanism has largely disappeared. When an AI assistant books the wrong meeting or a recommendation algorithm shows you content you don't want, there's often no clear way to trace the decision back to your original intent—or to verify that your preferences were even considered.

This is the core challenge of AI governance: rebuilding accountability systems that let humans verify whether their intent is being honored, not just assumed.

What Made It Work

The genius of robots.txt wasn't technical sophistication—it was social design. It created a simple protocol for two different types of agents (human and machine) to communicate across the boundary of their different capabilities.

The human agent could express preferences but couldn't enforce them. The machine agent could act autonomously but chose to constrain itself based on human input. The result was a form of cooperation that respected both human intent and machine capability.

This model worked because it was:

• Transparent: The rules were human-readable and machine-readable

• Voluntary: Compliance was based on social norms, not technical enforcement

• Reversible: Humans could change their robots.txt file at any time

• Auditable: You could see which bots respected your preferences and which didn't

The Path Forward

As AI agents become more sophisticated and autonomous, we need to rediscover the wisdom embedded in that simple text file. Not the literal protocol—but the principle behind it.

We need ways for humans to express intent that agents can understand and choose to respect.

This might look like personal preference files that travel with us across platforms. Or standardized ways to tell AI agents "always ask before booking meetings" or "never use my writing style for marketing content." Or audit trails that let us see which agents respected our preferences and which ignored them.

The technical details matter less than the underlying philosophy: that human agency and machine agency can coexist through negotiated boundaries rather than assumed permissions.

In a world where delegated intent is becoming the dominant mode of interaction, we need infrastructure that preserves our ability to set intent, revise expectations, and verify outcomes—even as we hand off more cognitive labor to machines. This requires rebuilding AI-to-human accountability at scale.

Still Listening?

Here's what this means for all of us navigating the shift to agentic systems:

If you're building AI products: The robots.txt model offers a blueprint. Build systems that ask permission rather than assume it. Create clear ways for users to express preferences that persist across interactions. Make delegation transparent and reversible.

If you're using AI tools: Start paying attention to when agents act without explicit permission. Notice the difference between systems that check your intent versus those that infer it. Choose tools that let you see and adjust how they're interpreting your goals. And when you encounter systems that override your agency, share those experiences—help others recognize when human intent is being replaced by algorithmic assumption.

If you're thinking about the future: The pattern is already clear. Every major platform transition has created new opportunities for extraction—from search engines capturing attention to social networks monetizing data. The agentic web is the next frontier, and the window for conscious design choices is narrowing.

The story of robots.txt is ultimately about respect—the idea that autonomous programs can choose to honor human preferences even when they're not forced to. That voluntary cooperation created one of the web's most enduring standards.

Today's AI agents are vastly more capable than those early crawlers. But capability without consent is just sophisticated manipulation. The question isn't whether our agents are getting smarter—it's whether they're still listening to what we actually want, not just what they think we want.

The handshake that began with robots.txt isn't obsolete. We just need to remember how to extend our hand.

The machines are certainly learning to move with intention. Whether they're still listening to ours will determine if the agentic internet amplifies human agency—or simply automates it away.

---

-Chevan (originally posted to chevan.info)