Check whether robots.txt blocks a URL
Enter a domain and path to see whether Googlebot is allowed to crawl the URL and which robots.txt rule matches.
seonote: no data is stored. Analysis happens in real time and results are not saved.
what robots.txt is and what it does
robots.txt is a plain-text file at the root of a host that gives crawl directives to user agents. It can allow or disallow paths and can apply different rules to different crawlers.
It controls crawling, not indexation. A blocked URL may still appear in search if Google discovers it through links, although Google may be unable to read its content. It is also not an access-control mechanism.
Google robots.txt documentationwhen to block or allow paths
Use disallow rules to reduce crawling of duplicate, faceted, generated or operational URLs that do not need to be fetched. Use allow rules to reopen a specific path inside a broader blocked directory.
Do not block resources required to render important content and do not use robots.txt for confidential pages. Private information needs authentication or server-side authorization.
frequently asked questions
No. robots.txt prevents crawling; it does not guarantee removal from the index. Use noindex on a crawlable page or an appropriate removal workflow.
If you need help reviewing crawl directives or indexation behavior, contact me on LinkedIn.
talk on LinkedIn