Description from extension meta
This extension will report if a page is disallowed in robots.txt
Image from store
Description from store
See if a URL is allowed or disallowed in robots.txt for a specific user-agent token. If the URL is disallowed, it will not be crawled by the crawler with the matching user-agent token.