Robots Exclusion Checker
Checks robots.txt, meta robots, x-robots-tag with URL alerts. Canonical warnings, HTTP header info. An SEO extension, robots tester.
Robots Exclusion Checker is designed to visually indicate whether any robots exclusions are preventing your page from being crawled or indexed by Search Engines.
## The extension reports on 5 elements:
1. Robots.txt
2. Meta Robots tag
3. X-robots-tag
4. Rel=Canonical
5. UGC, Sponsored and Nofollow attribute values
- Robots.txt
If a URL you are visiting is being affected by an "Allow” or “Disallow” within robots.txt, the extension will show you the specific rule within the extension, making it easy to copy or visit the live robots.txt. You will also be shown the full robots.txt with the specific rule highlighted (if applicable). Cool eh!
- Meta Robots Tag
Any Robots Meta tags that direct robots to “index", “noindex", “follow" or “nofollow" will flag the appropriate Red, Amber or Green icons. Directives that won’t affect Search Engine indexation, such as “nosnippet” or “noodp” will be shown but won’t be factored into the alerts. The extension makes it easy to view all directives, along with showing you any HTML meta robots tags in full that appear in the source code.
- X-robots-tag
Spotting any robots directives in the HTTP header has been a bit of a pain in the past but no longer with this extension. Any specific exclusions will be made very visible, as well as the full HTTP Header - with the specific exclusions highlighted too!
- Canonical Tags
Although the canonical tag doesn’t directly impact indexation, it can still impact how your URLs behave within SERPS (Search Engine Results Pages). If the page you are viewing is Allowed to bots but a Canonical mismatch has been detected (the current URL is different to the Canonical URL) then the extension will flag an Amber icon. Canonical information is collected on every page from within the HTML <head> and HTTP header response.
- UGC, Sponsored and Nofollow
A new addition to the extension gives you the option to highlight any visible links that use a "nofollow", "ugc" or "sponsored" rel attribute value. You can control which links are highlighted and set your preferred colour for each. I’d you’d prefer this is disabled, you can switch off entirely.
## User-agents
Within settings, you can choose one of the following user-agents to simulate what each Search Engine has access to:
1. Googlebot
2. Googlebot news
3. Bing
4. Yahoo
## Benefits
This tool will be useful for anyone working in Search Engine Optimisation (SEO) or digital marketing, as it gives a clear visual indication if the page is being blocked by robots.txt (many existing extensions don’t flag this). Crawl or indexation issues have a direct bearing on how well your website performs in organic results, so this extension should be part of your SEO developer toolkit for Google Chrome. An alternative to some of the common robots.txt testers available online.
This extension is useful for:
- Faceted navigation review and optimisation (useful to see the robot control behind complex / stacked facets)
- Detecting crawl or indexation issues
- General SEO review and auditing within your browser
## Avoid the need for multiple SEO Extensions
Within the realm of robots and indexation, there is no better extension available. In fact, by installing Robots Exclusion Checker you will avoid having to run multiple extensions within Chrome that will slow down its functionality.
Similar plugins include:
NoFollow
https://chrome.google.com/webstore/detail/nofollow/dfogidghaigoomjdeacndafapdijmiid
Seerobots
https://chrome.google.com/webstore/detail/seerobots/hnljoiodjfgpnddiekagpbblnjedcnfp
NoIndex,NoFollow Meta Tag Checker
https://chrome.google.com/webstore/detail/noindexnofollow-meta-tag/aijcgkcgldkomeddnlpbhdelcpfamklm
CHANGELOG:
1.0.2: Fixed a bug preventing meta robots from updating after a URL update.
1.0.3: Various bug fixes, including better handling of URLs with encoded characters. Robots.txt expansion feature to allow the viewing of extra-long rules. Now JavaScript history.pushState() compatible.
1.0.4: Various upgrades. Canonical tag detection added (HTML and HTTP Header) with Amber icon alerts. Robots.txt is now shown in full, with the appropriate rule highlighted. X-robots-tag now highlighted within full HTTP header information. Various UX improvements, such as "Copy to Clipboard” and “View Source” links. Social share icons added.
1.0.5: Forces a background HTTP header call when the extension detects a URL change but no new HTTP header info - mainly for sites heavily dependant on JavaScript.
1.0.6: Fixed an issue with the hash part of the URL when doing a canonical check.
1.0.7: Forces a background body response call in addition to HTTP headers, to ensure a non-cached view of the URL for JavaScript heavy sites.
1.0.8: Fixed an error that occurred when multiple references to the same user-agent were detected within robots.txt file.
1.0.9: Fixed an issue with the canonical mismatch alert.
1.1.0: Various UI updates, including a JavaScript alert when the extension detects a URL change with no new HTTP request.
1.1.1: Added additional logic Meta robots user-agent rule conflicts.
1.1.2: Added a German language UI.
1.1.3: Added UGC, Sponsored and Nofollow link highlighting.
1.1.4: Switched off nofollow link highlighting by default on new installs and fixed a bug related to HTTP header canonical mismatches.
1.1.5: Bug fixes to improve robots.txt parser.
1.1.6: Extension now flags 404 errors in Red.
1.1.7: Not sending cookies when making a background request to fetch a page that was navigated to with pushstate.
1.1.8: Improvements to the handling of relative vs absolute canonical URLs and unencoded URL messaging.
1.2.0.11: Updating to Google's new manifest V3 and fixing small bugs.
1.2.0.12: Added a Spanish language version and made improvements to existing translations. Linking to new website https://www.checkrobots.com
1.2.0.13: Fixed pushState navigation data extraction, resolved inconsistent icon display, and added security protections to prevent logout issues with enterprise websites.
Found a bug or want to make a suggestion? Please email extensions @ samgipson.com
Latest reviews
Save this extension from extinction, Please
this extension adds an annoying red dotted border box around certain links, when working on websites, this becomes an issue as i just spent the last 3 hours going through my 100+ extensions one at a time to narrow it down. I understand this extension helps with checking robots.txt exclusion, but there was nothing telling me that this was being caused by your extension, you need some type of info next to it, so people know your extension is doing this. Otherwise this extension is great *** Update *** went through the settings and see there is an option called nofollow thats preselected that says highlights in red. I disabled that and the dotted border line is gone. You should really have something either on a highlight or a popup box saying that the dotted border line is being caused by your extention, even did an inspect and couldnt figure out what extension was causing it, looked it up google, and google said it was an accessibility issue. I have rerated this plugin as 5 stars now
NICE!!!!!!
This works, but it is super sticky. I was tuning robots.txt expecting changes to be reflected immediately. Tried clearing caches/cookies and even restarting the webserver. Eventually, I figured out I had to select View Live robots.txt and then disable/enable the extension to get the changes loaded.
The Robots Exclusion Checker Chrome extension is a must-have tool for webmasters and SEO professionals who want to efficiently manage their website's visibility in search engine results. With its user-friendly interface and comprehensive features, this extension simplifies the process of checking and validating robots.txt files, ensuring your website is effectively crawled by search engine robots. https://www.tranktechnologies.com/usa/app-development-company-new-york
واقعا عالیه من برای سایتم لینک میزنم قبلا متوجه نبود که سایت ایندکسه یا نه ولی الان راحت متوجه میشم. خیلی ممنون از سازندش https://www.xn--mgbatejc1d2cmr8bab45pla.net/
A great SEO addon which is also one of a few that can inform you of the index status of the page without the need of opening the plugin. It turns red from green automatically so you know what's going on.
A great SEO addon which is also one of a few that can inform you of the index status of the page without the need of opening the plugin. It turns red from green automatically so you know what's going on.
A must have extension for every SEO! I advize every client of mine to install this as well. I just love how you instantly know for any page if it's indexable or not.
Very helpful extension! Very useful and works perfectly – 100% recommended. Just a question: looking at the chrome task manager it uses a lot of memory footprint compared to other processes and extensions (specially when working with multiple tabs). @Sam can you take a look at it or is it just me? Thanks in advance!
Very helpful extension! Very useful and works perfectly – 100% recommended. Just a question: looking at the chrome task manager it uses a lot of memory footprint compared to other processes and extensions (specially when working with multiple tabs). @Sam can you take a look at it or is it just me? Thanks in advance!
Good Extension, unfortunately NOINDEX Link Highlighting is missing.
Good Extension, unfortunately NOINDEX Link Highlighting is missing.
Working now again :)
Working now again :)
This is the best extension. I haven't found any worthy counterparts for it. Thanks to the developer for this
This is the best extension. I haven't found any worthy counterparts for it. Thanks to the developer for this
Would recommend!
Would recommend!
A great and useful extension that I use with my marketing team. The little extra, a reactive person named Sam, thanks for this great tool that we use several times a day.
A great and useful extension that I use with my marketing team. The little extra, a reactive person named Sam, thanks for this great tool that we use several times a day.
Fantastic extension! It saves me so much time and is really easy to understand and use. SEOs and DEVs, it's your loss if you don't install it! :D
Fantastic extension! It saves me so much time and is really easy to understand and use. SEOs and DEVs, it's your loss if you don't install it! :D
SEO for 10 years, I was mad when the extension SeeRobots was withdrawn but this one is a great successor !
SEO for 10 years, I was mad when the extension SeeRobots was withdrawn but this one is a great successor !
Possibly my favourite SEO extension...the brilliance is in the simplicity of the design. Key information, presented clearly. It's not trying to add loads of superfluous SEO "insights" or solve every issue. Would have reviewed it sooner as I have been recommending it for years :-)
Possibly my favourite SEO extension...the brilliance is in the simplicity of the design. Key information, presented clearly. It's not trying to add loads of superfluous SEO "insights" or solve every issue. Would have reviewed it sooner as I have been recommending it for years :-)
Buenísima, me la acaban de descubrir, la he instalado y probado, y me da que ha llegado para quedarse. 👌
I never write reviews on extensions, but I use this bad boy every day and it's excellent. Well done Sam :)
I never write reviews on extensions, but I use this bad boy every day and it's excellent. Well done Sam :)
Top notch plugin, Simple but very effective. Use it daily. Best robots plugin on chrome store
Top notch plugin, Simple but very effective. Use it daily. Best robots plugin on chrome store
One of my favourite SEO extensions, I use this daily. Changing its colour so you can see problems at a glance is incredibly time-saving.
One of my favourite SEO extensions, I use this daily. Changing its colour so you can see problems at a glance is incredibly time-saving.
A must have Chrome extension for your SEO toolkit. 4 key checks across: - Robots.txt - Meta robots tag - X-robots header - Canonical tag Provide quick insights relating to the index-ability of a given URL without having to manually review the source yourself. Color coded notifications in the extension as well as the ability to click and copy any data retrieved make this extension a prerequisite.
A must have Chrome extension for your SEO toolkit. 4 key checks across: - Robots.txt - Meta robots tag - X-robots header - Canonical tag Provide quick insights relating to the index-ability of a given URL without having to manually review the source yourself. Color coded notifications in the extension as well as the ability to click and copy any data retrieved make this extension a prerequisite.
Superb extension!
Superb extension!
Amazing plugin that comes handy on a daily basis. Recently I noticed an issue with the plugin, reached out to the OP, Sam, and he had it resolved quickly. Thanks for the great, free extension!
Super useful and better than my previous extension 'see robots' as has more options which are more extensive when checking if a page is actually no-indexed.
Super useful and better than my previous extension 'see robots' as has more options which are more extensive when checking if a page is actually no-indexed.
Really useful extension, works well, saves lots of time and very easy to use. Definitely recommend!
Amazing extension! Saves me so much time and I love the new canonical feature. Highly recommended for SEO's and devs to use!
Amazing extension! Saves me so much time and I love the new canonical feature. Highly recommended for SEO's and devs to use!
Have needed something handy like this for a long time. A real time saver! Highly recommended to all SEO's out there.
Have needed something handy like this for a long time. A real time saver! Highly recommended to all SEO's out there.
Easy and worthwhile. Saves me a ton of time.
Easy and worthwhile. Saves me a ton of time.
Great Extension! It is very useful and easy to use :)
Great Extension! It is very useful and easy to use :)
