Audit: Nearly 90% of Sites Block AI Crawlers
5 min read
A new study released on October 12, 2026 reveals that the vast majority of websites are inadvertently blocking themselves from AI-powered search engines and chatbots. Stradiji, an SEO and generative engine optimization consultancy, released its GEO Score Card framework along with an audit of more than 100 websites in seven countries, finding that nearly nine in ten sites had closed at least one of three critical conditions that determine whether AI engines can reach, recognize, and believe a business. The findings highlight a growing disconnect as businesses struggle to adapt to the rise of AI-powered search through platforms like ChatGPT, Gemini, and Perplexity.
The Three Barriers to AI Visibility
The framework groups 70 checks into three stages called Reach, Recognize, and Believe, where Reach measures whether AI engines can access a site, Recognize measures whether they identify the brand as a distinct entity, and Believe measures whether independent sources support what is said about it.
The first and most fundamental barrier proved especially widespread. About three in four of the audited sites give AI crawlers no explicit access in robots.txt, and some block GPTBot, ClaudeBot, or PerplexityBot outright. This technical configuration prevents AI systems from even reading site content, effectively rendering businesses invisible to users searching through AI platforms.
The sites were audited through September 2026 as part of client and prospect engagements, with each site crawled in full and its robots.txt, llms.txt, ai.txt, and AGENTS.md files reviewed, plus live tests run in ChatGPT, Perplexity, Gemini, and Google’s AI features. The sites are based in Turkey, the United States, the United Kingdom, Germany, Estonia, Romania, and South Korea and cover sectors including health, e-commerce, B2B software, real estate, media, law, manufacturing, and travel.
The Shift from Traditional SEO to AI Search
The study arrives as businesses face mounting pressure to optimize for what industry experts call generative engine optimization, or GEO. Unlike traditional search engine optimization, which focuses on ranking in lists of blue links, GEO requires ensuring that AI systems can access, understand, and confidently cite a brand when generating answers to user queries.
This shift reflects broader changes in how people search for information online. Multiple industries are adapting to the reality that users increasingly turn to AI chatbots for research before visiting websites directly. Companies that remain invisible to these systems risk losing discovery opportunities as AI-powered search continues to gain market share.
Growing Industry Response
The visibility challenge has sparked a wave of new tools and services designed to help businesses track and improve their presence in AI-generated answers. The emergence of specialized auditing frameworks, tracking platforms, and optimization consultancies signals recognition that AI visibility requires distinct strategies from traditional web search.
Several major technology companies are also responding to the changing landscape. Reports indicate that some organizations are building in-house capabilities to manage their visibility across AI platforms, treating it as a strategic priority rather than an extension of existing marketing functions.
Technical and Strategic Gaps
Beyond basic crawler access, the Stradiji audit identified multiple layers where websites fall short of AI visibility requirements. The framework examines whether sites properly declare entity information that helps AI systems distinguish one business from another with similar names, and whether external authoritative sources validate the claims businesses make about themselves.
These requirements differ substantially from traditional SEO best practices. While conventional search optimization focuses on keywords and backlinks, AI visibility depends on structured data, clear entity signals, and verifiable information that AI models can confidently include in generated responses.
Implications for Business Discovery
The audit findings suggest most businesses face a gap between how AI systems know them for branded queries versus how they appear in competitive recommendation scenarios. A company might be correctly described when users ask about it by name, but fail to surface when potential customers pose broader questions about solutions, services, or product categories.
This distinction matters because discovery queries represent new business opportunities, while branded searches typically come from users already familiar with a company. Businesses that appear only in the latter scenario miss the chance to reach buyers earlier in their research process.
Key Facts
- Stradiji audited more than 100 websites in seven countries, finding nearly nine in ten had closed at least one of three conditions needed for AI visibility
- About three in four audited sites give AI crawlers no explicit access in robots.txt, with some blocking GPTBot, ClaudeBot, or PerplexityBot entirely
- The GEO Score Card framework groups 70 checks into three stages: Reach, Recognize, and Believe
- Each site was tested live in ChatGPT, Perplexity, Gemini, and Google’s AI features
- Audited sites spanned sectors including health, e-commerce, B2B software, real estate, media, law, manufacturing, and travel
Sources
- Stradiji AI Visibility Study and Audit Finds Most Websites Close Doors to LLMs
- Stradiji
- 11 of the Best-Funded GEO Startups Racing to Win AI Search
- Zeta Brings GEO In-House as AI Search Reshapes Brand Visibility
