Google-Agent Just Changed The Rules For Bot Traffic And Robots.txt

On March 20, 2026, Google quietly added a new name to its list of web fetchers. Not Googlebot. Not a training crawler. The new entry is called Google-Agent, and it shows up when an AI assistant on Google infrastructure visits a page on behalf of a real person. The rules for Google-Agent are not the rules you have used for the last two decades, as Search Engine Journal reported on May 17.

If your team blocks bots with robots.txt, monitors crawler logs for fraud, or uses user-agent strings to decide who sees what, Google-Agent changes your job. Google-Agent does not follow the same rules as Googlebot, and it signs some requests with a new cryptographic protocol. That is the first time a major search company has put a real identity on agent traffic. Here is what changed and the five-step plan your team should follow this month.

What Google-Agent Is And Why It Is Not Googlebot

Google-Agent is the user-agent string for AI systems that run on Google infrastructure and browse websites on behalf of a person. Google’s Project Mariner, the experimental AI browsing tool, was the first product to use it. When a user asks a Mariner-powered assistant to compare two products, fill out a form, or pull a quote from a competitor page, Google-Agent is the thing that actually loads the page.

Google-Agent is a very different job from Googlebot. Googlebot crawls the open web on a schedule, builds the search index, and respects your robots.txt file. Google-Agent only shows up when a person asks an AI assistant to do something. The Google-Agent fetch is tied to a single user request, not a crawl plan.

Google classifies Google-Agent as a “user-triggered fetcher.” That category also includes Google Read Aloud, NotebookLM, and Feedfetcher. All of them share one rule. They generally ignore robots.txt because a person started the request. The logic is simple. If you type a URL into Chrome, the browser fetches the page no matter what robots.txt says. The agent is the user’s proxy, so it works the same way. You can read the full classification in the new Google web bot auth documentation.

OpenAI and Anthropic made a different call. ChatGPT-User and Claude-User are also user-triggered fetchers, but both companies told website owners those agents will respect robots.txt directives. If you block ChatGPT-User in robots.txt, ChatGPT will not fetch the page when a user asks. Google picked the opposite path. That gap matters for any team that built access policy on the old assumption that robots.txt was a universal door for every bot.

Why Robots.txt No Longer Stops Agent Traffic

For most of the last 20 years, robots.txt has been the front door for bot access. You added a disallow rule, and most well-behaved crawlers stayed out. That worked because the web was crawler-led. Today the web is becoming agent-led, and the agent model breaks the old assumption.

Google-Agent treats your page like a human visitor with a browser would. Google-Agent will load the page even if robots.txt says no. If you want to block or rate-limit Google-Agent traffic, you now need server-side controls. That means access rules at the firewall, in your CDN, or in your web application layer. The same tools you would use to stop a scraper or to fence off a logged-in area.

A page that is closed to Googlebot can still be wide open to Google-Agent. Our team covered how to think about crawlability and access control in an earlier post. The principles hold, but the access layer for agent traffic now lives outside robots.txt.

If your site relies on robots.txt for anything more serious than a polite request, treat that as a gap. Map your robots.txt rules to server-side equivalents. Then decide which agents you want to allow, which you want to challenge, and which you want to block.

Web Bot Auth And The Cryptographic Identity Layer

The bigger story is buried in one line of Google’s documentation. Google-Agent is testing a protocol called Web Bot Auth using the identity https://agent.bot.goog. Web Bot Auth is an IETF draft standard. It works like a digital passport. Each agent holds a private key, publishes its public key in a well-known directory, and cryptographically signs the HTTP requests it sends. Your server can verify the signature and know, with high confidence, that the Google-Agent visitor is who it claims to be.

This is a sharp change from how bot identity has worked for decades. A user-agent string can be set to any value. An IP address can be shared by many services. A cryptographic signature is much harder to fake. Major bot detection vendors, CDNs, and WAFs already support Web Bot Auth, including Akamai, Cloudflare, and Amazon’s AgentCore Browser. Google joining gives the protocol the critical mass it needed.

Google’s own guidance says the rollout is still experimental. Not every Google-Agent request is signed yet. Site owners are told to keep using IP ranges, reverse DNS, and user-agent strings in parallel. That mirrors the advice we shared on ad fraud and bot traffic protection. Belt and suspenders is the right approach while the standard hardens.

The story matters because the web is heading into a real identity problem. Agent traffic is going to grow fast. Sites need a way to separate legitimate AI agents from scrapers that pretend to be agents. Cryptographic signatures scale better than IP allowlists.

The Three-Tier Visitor Model And What Each Tier Wants

Google-Agent locks in a three-tier visitor model that every digital team should be planning for now.

The first tier is human visitors. They use a browser, click links, fill out forms, and convert into leads or customers. Your conversion tracking, your landing pages, and your CRO work are all built around them.

The second tier is crawlers. Googlebot, GPTBot, Google-Extended, and the rest. Their job is to index pages for search and for training. They follow robots.txt when they are well behaved, and they are the audience for most traditional SEO work. AI search visibility still depends in part on how well crawlers can read and store your content.

The third tier is agents. Google-Agent, ChatGPT-User, Claude-User, and others. They show up only when a real person asks them to do something. They want to complete a specific task, not index your library. A Google-Agent visit might read a product page once to pull a price, fill out a contact form, or extract a comparison table.

Each tier has different rules and different goals. Mixing them in one access policy is the fastest way to either block real customers or let too much non-human traffic through. The Google generative AI features guide we shared last week is a good companion read. Google Analytics 4 also now has an AI Assistant default channel group. The new GA4 AI assistant channel separates traffic from ChatGPT, Gemini, and other chatbot referrers, and Google-Agent visits that send a human back to the site will start showing up there.

What This Means For Schema, AI Overviews, And SEO Reporting

The Google-Agent story arrives in the same month as two other shifts. Google dropped FAQ rich results from search. A new Ahrefs study on 1,885 pages showed that adding schema markup did not lift AI citations in any meaningful way. We unpacked both in earlier posts on FAQ rich results going away and on schema markup and AI citations.

Google-Agent fits the same theme. The old SEO playbook was built around getting crawlers to read structured signals and rank you in the SERP. The new playbook has to plan for Google-Agent and other agents that visit your page once on behalf of a user, take what they need, and leave. That is closer to a customer support interaction than a crawl.

Reporting will need to change with it. The AI search terms report Google rolled out last week is one early signal. Expect more reports that split human visits, crawler visits, and Google-Agent visits into separate views. Pages that win in the agent tier load fast, present clean key facts above the fold, and avoid heavy JavaScript.

A Five-Step Plan For The Next 30 Days With Google-Agent

Here is a clean 30-day plan your team can run starting this week.

Step one. Audit your logs for Google-Agent. Search the last 90 days of access logs for the string “compatible; Google-Agent”. Volume will likely be low today, but the baseline matters. Track the trend weekly. Also pull traffic for ChatGPT-User, Claude-User, and Perplexity-User so you have the full agent tier in one view.

Step two. Map your robots.txt rules to server-side equivalents. List every disallow rule you rely on. Decide which ones need a real access control at the firewall, the CDN, or the application layer. Robots.txt should stay in place as a polite signal, but it is not a security boundary for agent traffic.

Step three. Talk to your CDN or WAF provider about Web Bot Auth. Akamai, Cloudflare, and Amazon already support the protocol. Ask whether your current plan can verify the Signature-Agent header and the public key directory at https://agent.bot.goog for Google-Agent traffic. If yes, turn on verified-agent reporting in your dashboard. If no, get a timeline.

Step four. Update your access policy for the three-tier model. Write a short policy that says how you treat humans, crawlers, and agents. Pick the agents you want to allow. Decide which content is open, which is gated behind a form, and which is closed to non-humans. Share the policy with content, legal, and security in one document.

Step five. Brief your reporting team. Set up a monthly view that pulls Google-Agent visits from the server logs and pairs them with the new GA4 AI Assistant channel data. Track which pages get the most agent visits. Those pages are the ones an AI is choosing to read on behalf of users, and they deserve the same care your top organic pages get today.

Want a partner to help your team work through Google-Agent in the next month? Book a free consultation and we will walk through your robots.txt rules, your CDN setup, and your log data together. Let’s Grow!