This extension will report if a page is disallowed in robots.txt
See if a URL is allowed or disallowed in robots.txt for a specific user-agent token. If the URL is disallowed, it will not be crawled by the crawler with the matching user-agent token.