Check whether robots.txt blocks your URL
Paste the domain and the path you want to test. The tool fetches the site's robots.txt, identifies the rules that apply to Googlebot, and tells you whether the URL would be blocked or allowed, and which rule was applied.
note: no data is stored. The analysis runs in real time and results are not saved.
what robots.txt is and what it is used for
robots.txt is a text file hosted at the root of a domain, always available at yoursite.com/robots.txt. It tells crawlers which paths on the site they may or may not access. According to Google's official documentation , the file is mainly used to manage crawl traffic and avoid overloading the server.
The most common directives are:
User-agentSpecifies which crawler the rules apply to. The value * represents all crawlers. Specific values such as Googlebot apply only to that bot.
DisallowSpecifies the paths the crawler should not access. A Disallow: /admin/ rule blocks everything inside that directory.
AllowCreates exceptions inside a blocked section. If an entire directory is blocked with Disallow, an Allow rule can open up a specific path inside it.
SitemapPoints to the location of the site's sitemap. It isn't a crawling rule, just a content discovery hint.
when to use block or allow rules
Disallow makes sense for paths with no indexing value: admin panels, internal search results, pages generated by filters and parameters that duplicate existing content, and cart or checkout areas.
One important limitation: blocking a URL in robots.txt does not guarantee it will drop out of Google. According to Google's documentation , a URL blocked from crawling can still be indexed if external links point to it. Google can know the URL exists without accessing its content. In that case, it may appear in results without a description.
To remove a page from the index, the correct approach is to allow crawling and use the noindex meta tag. If crawling is blocked, Googlebot cannot read the noindex instruction and the page may stay indexed.
frequently asked questions
No. A URL blocked in robots.txt can keep appearing in search results if external links point to it. Google can index the URL without being able to crawl its content, showing the result without a description. To remove a URL from the index, you need to allow crawling and use the noindex meta tag, or use the removals tool in Google Search Console for urgent removals.
Need help with your site? Reach out to me on LinkedIn. I've worked in SEO for almost 10 years and I can help.
talk on LinkedIn