Web Bot Auth Explained: How Websites Can Verify AI Agent Requests

Traditional bot verification relies heavily on user-agent strings, IP ranges, and reverse DNS. These methods remain useful, but they can be difficult to maintain and may be vulnerable to impersonation. Web Bot Auth is an experimental approach that allows automated agents to cryptographically sign HTTP requests.

Google has begun testing Web Bot Auth with some AI-agent traffic. The goal is to give website operators stronger evidence that a request genuinely came from the claimed agent provider.

Why User-Agent Verification Is Not Enough

An attacker can send:

User-Agent: Google-Agent

without actually being Google. IP allowlists help, but cloud infrastructure, changing networks, proxies, and large address ranges make maintenance difficult.

How Web Bot Auth Works

At a high level:

1. The agent provider publishes public verification keys.
2. The agent signs selected HTTP request components.
3. The website receives signature-related headers.
4. The website retrieves or uses cached public keys.
5. The signature is verified according to HTTP Message Signatures.
6. The request is associated with a verified agent identity.

Benefits of Signed Bot Requests

• Reduced user-agent spoofing
• Less dependence on fixed IP ranges
• Clearer AI-agent analytics
• More precise access-control policies
• Better separation of legitimate and malicious automation
• Stronger audit trails

Current Limitations

Web Bot Auth is still experimental. Not every request is signed, not every crawler supports it, and implementations may change as standards work continues.

Websites should continue using layered verification:

• User-agent matching
• Published IP ranges
• Reverse DNS checks
• CDN verified-bot services
• Rate limiting
• Request signatures

What Website Operators Need

A custom implementation may need to:

• Read Signature-Agent, Signature, and Signature-Input headers
• Fetch the agent provider's public-key directory
• Cache keys according to Cache-Control
• Remove expired or revoked keys
• Verify signature timestamps and covered components
• Handle unsigned fallback traffic safely

DNS and Key-Directory Availability

Signature verification depends on reaching the provider's key directory. DNS failure, DNSSEC problems, certificate errors, or network filtering can prevent key retrieval.

Operators should monitor:

• DNS resolution time
• HTTPS certificate validity
• Key-directory response status
• Cache expiration
• Signature verification failures

Using Web Bot Auth with a CDN or WAF

The easiest implementation may be through a CDN, bot-management platform, or WAF that validates signatures before the request reaches the origin server.

Ask the provider whether it supports:

• Web Bot Auth verification
• Verified AI-agent categories
• Rules based on verified identity
• Logs containing signature results
• Fallback behavior for unsigned requests

Frequently Asked Questions

Does Web Bot Auth replace robots.txt?

No. Web Bot Auth verifies identity; robots.txt communicates crawl preferences.

Does it replace DNS verification?

No. DNS remains part of reaching key directories and web services, and traditional verification may still be needed.

Should unsigned requests be blocked?

Not automatically during the experimental phase because some legitimate requests may remain unsigned.

Conclusion

Web Bot Auth is an important step toward trusted AI-agent traffic. It can reduce spoofing and enable more accurate policies, but current deployments should combine signed-request verification with existing bot, DNS, and network controls.