This extension will report if a page is disallowed in robots.txt
See if a URL is allowed or disallowed in robots.txt for a specific user-agent token. If the URL is disallowed, it will not be crawled by the crawler with the matching user-agent token.
Statistics
Installs
3
history
Category
Rating
0.0 (0 votes)
Last update / version
2020-06-05 / 1.1
Listing languages
en