Contact Book a call

Home / Tools / AI Crawler Checker

Free tool · AI search

Can ChatGPT and AI search read your site?

Enter your website. In seconds you see which AI and search bots can crawl it, what is blocking you, and the exact robots.txt lines to fix it.

  • GPTBot
  • OAI-SearchBot
  • ClaudeBot
  • PerplexityBot
  • Google-Extended
  • +6 more

Built by the AWISEE AI search team. No sign-up, no credit card.

AI crawler checkerFree

We email you the report and may follow up with AI search tips. Privacy policy.

Robots.txt not public, or site blocks checks?
ExampleThis is a sample report. Enter your website above to see yours.

Results

Your AI crawl report.

AI search bots–

can find and link your pages in ChatGPT, Claude and Perplexity search.

All bots–

of 11 search and AI crawlers can crawl your homepage.

llms.txt–

An optional file some AI tools read. Google doesn't need it.

BotUsed forStatus
What this means

What to do next

    Copy and paste

    Suggested robots.txt lines

    

    

    Rules follow Google's robots.txt spec: a bot uses its own group if one exists, otherwise *; the longest matching rule wins, and Allow wins a tie.
    

    


    

    How it works
    

    Three steps, under a minute.
    

    1. 1
      Enter your site
      Type your domain and email. Nothing to install.
    2. 2
      We read your rules
      We fetch your robots.txt and llms.txt and test 11 search and AI bots against them.
    3. 3
      Get the fix
      See who is blocked, why it matters, and copy the robots.txt lines to change.
    

    

    

    Know the difference
    

    Not all AI bots do the same job.
    

    

    01 · Search bots
    Keep these open
    OAI-SearchBot, Claude-SearchBot and PerplexityBot find pages to show and link in AI search answers. Blocking them can cost you visibility.
    

    02 · Training bots
    Your call
    GPTBot, ClaudeBot and Google-Extended collect content for model training. Blocking them does not remove you from search.
    

    03 · User fetchers
    Hard to block
    ChatGPT-User and Perplexity-User visit a page because a person asked. They may not follow robots.txt.
    

    

    

    

				
    
				
    
		
    
    
    
    			
    Questions
    
					
    AI crawler FAQs.
    
		
    
    			
    
    
    
    
    
    			
    Should I block AI crawlers?
    
		
    


    

    
    			
    Block training bots like GPTBot, ClaudeBot or Google-Extended if you don't want your content used to train models. Keep search bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot open if you want to be found and cited in AI answers.
    
		

    


    


    

					
    
    
    
    
    
    			
    Does blocking Google-Extended affect Google Search?
    
		
    


    

    
    			
    No. Google-Extended only controls use in Gemini apps and Vertex AI. Google Search and AI Overviews use Googlebot, so blocking Google-Extended does not change your rankings.
    
		

    


    


    

					
    
    
    
    
    
    			
    Do I need an llms.txt file for AI search?
    
		
    


    

    
    			
    No. llms.txt is an optional file some AI tools read. Google has said it does not use it. It does no harm, but crawl access and content quality matter far more.
    
		

    


    


    

					
    
    
    
    
    
    			
    How does this tool decide what is allowed?
    
		
    


    

    
    			
    It follows Google's robots.txt rules. A bot uses its own User-agent group if one exists, otherwise the * group. The longest matching rule wins, and Allow wins a tie.
    
		

    


    


    

					
    
    
    
    
    
    			
    What if the tool can't reach my site?
    
		
    


    

    
    			
    Some sites block automated checks, or sit behind a login or firewall. Paste your robots.txt into the box and we check it right in your browser.
    
		

    


    


    

					
    
    
    
    
    
    			
    What do you do with my email?
    
		
    


    

    
    			
    We save your email, website and the result so we can send the report and follow up with AI search tips. We never sell it. You can ask us to delete it any time via our privacy policy.