How to Verify AI Crawlers with Reverse DNS, IP Ranges and Signed Requests
AI crawler names can be spoofed. A malicious bot can claim to be GPTBot, Google-Agent, or another recognized crawler by copying its user-agent string. Reliable verification requires multiple signals.
Method 1: Published IP Ranges
Some providers publish JSON files containing IP ranges used by their crawlers. A website can compare the source IP against the current list.
Advantages:
- Simple to automate
- Works before application processing
- Supported by many firewalls and CDNs
Limitations:
- Lists can change
- Overly broad ranges increase exposure
- Cached lists can become stale
- Not every provider publishes stable ranges
Method 2: Reverse DNS Verification
Reverse DNS checks the PTR record for the source IP. A secure workflow should use forward-confirmed reverse DNS:
- Look up the PTR record for the source IP.
- Confirm the hostname belongs to the provider's documented domain.
- Resolve that hostname forward.
- Confirm the original IP appears in the forward result.
This reduces the risk of trusting a PTR name controlled by an attacker.
Method 3: Verified-Bot Services
CDNs and bot-management platforms maintain their own verification systems. They may combine network ownership, behavior, signatures, and provider relationships.
This can simplify rules such as:
- Allow verified AI search bots
- Challenge unverified automation
- Rate-limit agent traffic
- Block training crawlers
Method 4: Cryptographic Request Signatures
Experimental protocols such as Web Bot Auth allow agents to sign requests. Signature verification can establish stronger identity than a user-agent string.
Signed requests are particularly useful when the agent uses changing cloud IP addresses.
Why You Need More Than One Method
Each verification method can fail:
- IP lists may be stale
- Reverse DNS may be unavailable
- A CDN may misclassify traffic
- Not every legitimate request is signed
- Proxy infrastructure may hide the source
A layered policy can use stronger verification for sensitive or expensive endpoints.
Suggested Verification Levels
Public Content
Allow verified search crawlers and use moderate rate limits.
High-Cost Dynamic Pages
Require verified-bot status or signatures and apply strict request limits.
Account and Administrative Pages
Use authentication and authorization. Do not rely on crawler identity.
DNS Tools for Verification
Use a PTR lookup for reverse DNS and an A or AAAA lookup for forward confirmation. DNSCheckers.com can help compare DNS results from different locations.
Common Verification Mistakes
- Trusting the user-agent string alone
- Checking PTR without forward confirmation
- Hardcoding IPs permanently
- Allowlisting an entire cloud provider
- Ignoring IPv6
- Skipping rate limits for verified bots
- Failing open when verification systems break
Frequently Asked Questions
Is reverse DNS proof of identity?
It is a useful signal, especially with forward confirmation, but cryptographic verification is stronger.
Can a verified bot still create high server load?
Yes. Apply reasonable crawl-rate and resource controls even to legitimate bots.
Should I block all unverifiable AI agents?
The answer depends on your content, business model, and risk tolerance.
Conclusion
AI crawler verification should combine identity, network, behavior, and cryptographic evidence. Use published IP ranges, forward-confirmed reverse DNS, verified-bot services, signed requests, and sensible rate limits.