Free tool · includes AI crawlers
Robots.txt Generator
Make a correct robots.txt for your site, choose which AI bots may read it, and download it.
Want to show up in AI answers?
Get your business cited by ChatGPT, Perplexity and Google AI.
Letting AI crawlers in is step one. My AI search optimisation makes your site the source AI tools quote when people ask about your services.
Common robots.txt mistakes
- Blocking CSS or JavaScript folders: Google needs them to see your pages properly.
- Leaving "Disallow: /" from a staging site on the live site, which hides it from Google completely.
- Using robots.txt to hide private pages: the file is public, so it tells everyone where they are. Use passwords instead.
- Forgetting the sitemap line, an easy way to help search engines find every page.
Frequently asked questions
What does robots.txt do?
It tells search engines and other crawlers which parts of your site they may visit. It must be at the top level of your domain, like example.com/robots.txt.
Does Disallow remove a page from Google?
No. It stops crawling, but a blocked page can still appear in results if other sites link to it. To keep a page out of Google, let it be crawled and add a noindex meta tag instead.
Should I block AI crawlers?
It is your choice. Blocking training bots such as GPTBot and Google-Extended stops your content being used to train models. Keeping search bots such as OAI-SearchBot and PerplexityBot lets AI search tools find and cite your site, which can send you visitors.
How do I add it to my site?
Upload robots.txt to the main folder of your website. In WordPress, SEO plugins like Yoast and Rank Math let you edit it under their tools settings. Then test it with the robots.txt checker.
More website tools
Page Speed Checker
Test load speed and Core Web Vitals on mobile and desktop, with fixes ranked by impact.
SEO Checker & Page Audit
35+ SEO checks for any page: meta tags, content, keyword, technical SEO and links, plus competitor comparison.
AI Search Visibility Checker
Check if ChatGPT, Claude, Perplexity and Google AI can find, understand and recommend your site.
Mobile-Friendly Test
See your site on a phone and check viewport, tap targets and mobile speed.
Website Screenshot
Capture any website on mobile and desktop, and download a device mockup image.
DNS & WHOIS Lookup
Domain age, registrar and expiry, every DNS record, and email security (SPF, DMARC).
CMS & Theme Detector
See what any site is built with: WordPress theme and plugins, Shopify theme, hosting and more.
SSL Certificate Checker
Check SSL expiry, trust, certificate chain, TLS version and HTTPS redirects.
Broken Link Checker
Find dead links, 404 errors, redirects and broken images on any page.
Meta Tag & Social Preview
Preview how a page looks in Google, Facebook, LinkedIn, X and WhatsApp, and fix its tags.
Robots.txt & Sitemap Checker
Check robots.txt for mistakes, see if Google and AI crawlers can access your site, and validate sitemaps.
HTTP Header & Redirect Checker
Trace redirect chains, check www and https versions, and grade your security headers.
Website Down Checker
Is it down for everyone or just you? Check status, response time and DNS.
Website Speed Comparison
Compare your site's speed with a competitor's: scores, Core Web Vitals and real-visitor data.
XML Sitemap Generator
Crawl your site and create sitemap.xml for Google, plus a list of broken links.
llms.txt Generator
Create an llms.txt file from your site's main pages to help AI tools understand it.
Who Hosts This Site?
Find any website's hosting company, server IP, CDN, DNS and email provider.
Cookie & Tracker Scanner
See which cookies and trackers a site loads, and whether it has a GDPR consent banner.
Accessibility Checker
Test any page for WCAG accessibility problems: contrast, labels, alt text and more.
Website Generator
Build a one-page business website from your details, preview it and download it free.
Website Cost Calculator
Estimate what your website will cost to build and run, then get an exact quote.