Robots.txt Checker
Confirm search and AI crawlers can reach the pages that matter, and see exactly which bots your robots.txt allows or blocks.
This is one slice of the full scan
A full Stackra scan adds a Growth Readiness Score, expert AI reviews, and a prioritized action plan across your whole site.
Understanding robots.txt
Robots.txt is a plain text file at the root of your site that tells search and AI crawlers which pages to visit and which to skip. It does not hide content from human visitors, but a single bad rule can silently block your entire site from every search engine at once.
How do I check if my robots.txt is blocking Google?
Load yoursite.com/robots.txt in a browser. If you see "Disallow: /" under "User-agent: *", every crawler is blocked from your entire site. This checker reads that file for you and flags any rule that prevents your homepage from being reached. A blocked homepage will not rank, regardless of how good the content is.
What should a healthy robots.txt look like?
For most small business sites, simple is correct: allow all crawlers, point them to your sitemap with a "Sitemap:" directive, and only block paths that genuinely should not be indexed (admin panels, cart pages, duplicate parameter URLs). The most common mistake is an overly aggressive Disallow rule added during a site rebuild that never gets removed before launch.
What are AI crawlers, and why does blocking them matter?
AI companies run their own crawlers to build the datasets that power ChatGPT, Perplexity, Gemini, and similar tools. If those crawlers are blocked in your robots.txt, your business does not appear in their answers when someone asks for a recommendation in your space. Most sites that block AI crawlers do it accidentally, by copying a template written before AI search existed. This checker shows you exactly which ones can and cannot reach you.
Does blocking a page in robots.txt remove it from Google?
Not immediately, and not completely. A blocked page can still appear in search if Google discovers it through links from other sites. It just cannot read the content, so it shows a blank or outdated snippet. To actually remove a page from search results, you need a noindex meta tag, not just a robots.txt rule.
My site has no robots.txt. Is that a problem?
Not a critical one. Without a file, crawlers treat everything as allowed by default. The main downside is you cannot point them to your sitemap, which slows page discovery for new content. Most CMS platforms generate one automatically, so if yours is missing, check whether something in your configuration is suppressing it.
Related guides and checklists
Keep going with hands-on guides and a checklist you can work through on your own site.