- Treat search crawlers, AI training bots and transactional AI agents as three separate audiences with separate rules.
- Keep every page you want ranked fully crawlable; gate only actions like booking, checkout and account access.
- Verify bots by IP and reverse DNS, not user-agent strings, which are trivially spoofed.
- Review server logs monthly, because agent traffic and naming conventions keep changing.
A large airline requiring permission for AI agents while leaving search indexing open is a signal worth reading carefully. It shows a pattern many businesses are moving toward: welcome discovery, control action. Visibility in search and AI answers drives demand, but software that logs in, searches fares, or completes purchases on a customer's behalf creates risks that a ranking crawler never did. Here is how to apply the same thinking to your own site.
Three kinds of automated visitors, three different decisions
Lumping every bot together is the most common mistake. In practice you are dealing with distinct groups:
- Search crawlers that index pages so people can find you. Blocking them costs traffic.
- AI retrieval and training bots that fetch content to ground answers or build models. Whether to allow them is a business judgment about attribution, citations and licensing.
- Transactional AI agents that act for a user: comparing prices, filling forms, booking, or scraping live inventory. These touch your systems, your terms and your liability.
The airline-style approach keeps the first group wide open, makes a deliberate choice on the second, and requires explicit permission for the third.
Why gating agents makes commercial sense
Agents behave unlike humans. They can hammer search endpoints, hold inventory, trigger fraud checks, bypass upsell flows, and present your prices stripped of context or fees. If an agent books the wrong thing, the customer blames you. Requiring permission, through approved partners, API access or authenticated tokens, lets you set rate limits, keep brand and pricing integrity, and define who is accountable when something goes wrong.
Meanwhile, indexing stays open because organic search remains a major acquisition channel, and being citable in AI-generated answers depends on your content being readable by retrieval systems.
How to implement the split
1. Use robots.txt for intent, not enforcement
Robots.txt is a polite request. Reputable crawlers honor it; bad actors ignore it. Write separate user-agent blocks for search engines and for each AI bot you have decided about, and keep the file readable and commented so teammates understand the reasoning.
2. Enforce with real controls
For anything sensitive, rely on server-side measures:
- Authentication and session requirements for booking, pricing engines and account areas.
- Rate limiting and bot-management rules at your CDN or firewall.
- Dedicated APIs or partner programs for approved agents, with keys and usage terms.
- Updated terms of service stating what automated access is allowed.
3. Verify, don't trust user-agent strings
Anyone can claim to be a major search bot. Confirm identity through published IP ranges or reverse DNS lookups. Otherwise you may accidentally block real crawlers or wave through impostors.
Protect your SEO while you tighten access
Gating done carelessly can damage rankings. Avoid these traps:
- Do not put informational pages, category pages or fare and product landing pages behind logins or JavaScript walls that crawlers cannot render.
- Do not serve search bots a different experience from users; that edges toward cloaking.
- Do not block CSS, JavaScript or image paths that search engines need to render pages.
- Test changes in a staging environment, then watch indexing and coverage reports after release.
A good rule: if a page exists to earn a click from a person, it should be crawlable. If a page exists to complete a transaction, it can be controlled.
Make your content agent-friendly where it helps you
Gating actions does not mean hiding facts. Clear structured data, accurate product and organization markup, current policies, and well-organized FAQs help both search engines and AI systems represent you correctly. Consider publishing a clean, public source of truth for things like hours, service areas, return rules or baggage-style policies, so agents quote you accurately instead of guessing from third-party sites.
Monitor and adjust
Agent behavior changes quickly, and new identifiers appear often. Build a simple routine:
- Review server logs monthly for new user agents and unusual request patterns.
- Track referral traffic from AI assistants and note which pages they cite.
- Revisit your policy per bot group each quarter as partnership options and business goals shift.
- Document decisions so legal, product and marketing stay aligned.
Actionable checklist
- List every automated visitor type hitting your site.
- Decide allow, limit or require permission for each.
- Align robots.txt, firewall rules and terms of service.
- Confirm search crawlers can reach and render all rankable pages.
- Offer an approved route, such as an API, for agents you want to work with.
Conclusion
The lesson is not to fear AI agents or to block everything. It is to be intentional: stay discoverable, stay in control of transactions. If you want help auditing crawl rules, log data and AI visibility, Growth Asteroid can map your bot traffic and build an access policy that protects both rankings and revenue.
Frequently asked questions
Will blocking AI agents hurt my Google rankings?
Not if you only restrict agent actions and AI-specific bots while leaving search crawlers able to fetch and render your pages. Problems arise only when search bots are blocked or served different content.
Is robots.txt enough to stop unwanted AI agents?
No. It is a voluntary signal. Use authentication, rate limiting, bot management and clear terms of service for anything sensitive.
How do I tell a real search crawler from a fake one?
Verify requests against the search engine's published IP ranges or perform reverse DNS checks. User-agent strings alone are easy to spoof.