Website crawl errors can prevent your pages from appearing in search results. The Crawl Errors report in the Search Console provides details about these errors, by showing a list of all site URLs that Google is unable to crawl, or that are returning HTTP error codes.
The Crawl Errors report has two sections: Site Errors and URL Errors. Here’s an overview of each of them, and how to fix the errors that you might come across.
Site Errors
This section of the report shows data from the last 90 days. In most cases, you’ll find a green check mark “status” next to each error type, meaning that your site is functioning well. If your status is not a green check, you can click on the box to expand it and view the historical crawling data from the last three months. This would only happen if your website is down, overloaded, or configured improperly. If this is the case, Google will send you a message to give you the heads up.
Site Errors are split into three different types:
DNS Errors
A DNS error means that a bot crawling your site cannot communicate with the DNS server — either because the server is down, or perhaps there is an issue with the routing to your domain. Whatever the case, a DNS error might indicate speed issues, which won’t affect the bot’s ability to access your site, but will affect your users and their ability to use your site.
Server Errors
A server error means that the bot cannot access your URL for some reason. Perhaps the request timed out or maybe the site was busy; either way, as a result, the bot had to quit the original request and cannot access your site.
Robot Failure
A Robot Failure is an error in retrieving your site’s robots.txt file. Approximately once a day, before a bot crawls your site, it retrieves your robots.txt file in order to see which pages it should not crawl. If your robots.txt file is unavailable, Google will stop the crawl so that it doesn’t accidentally crawl URLs that should not be crawled.
URL Errors
URL Errors are ranked by Google and sorted to show the most important and urgent issues at the top of the report, to a maximum of 1,000. They are also split between Desktop, Smartphone and Feature Phone errors and you can mark them fixed as you go through each one (which is what I find satisfying!). However, if you do mark a URL as “fixed” and it actually remains unfixed, the URL will reappear the next time Google crawls your site until you actually sort out the issue.
Common URL errors include:
Server Error: just like under the DNS errors, a server error for your URLs means the same: that a Googlebot cannot access your URL, the request was timed out, or your site was busy, and the bot was forced to abandon the request.
404: this error is one that most people see on an almost-daily basis. A 404 error occurs when a user tries to access a webpage that doesn’t (or no longer) exists. The causes of this error could be from broken links, incorrect URLs, or a redirected page that is no longer valid.
Access Denied: these errors occur when a Googlebot cannot access your site. Perhaps a URL requires a login in order to view it, or your server requires proxy authentication, or your hosting provider is blocking Google from accessing your site.
Not Followed: these errors list URLs that Google cannot completely follow. Perhaps your redirects are not redirecting to the right page, or because some features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash can make it difficult for search engines to crawl your site.
URLs Blocked for Smartphone: this “Blocked” error appears on the Smartphone tab. If you get the “Blocked” error for a URL on your site, that means that the URL is blocked for Google’s smartphone Googlebot in your site’s robots.txt file.
Flash Content: most mobile devices cannot process Flash content, so those Smartphone URLs will return this error. You may need to upgrade to a more responsive web design to avoid this error.
Summary
Some of these errors are definitely more technical than others; it’s our recommendation that you make use of Google’s support documents to help you learn which are important to act on, and which might be safe to ignore (at least for now).
In the meantime, you should also check out some of our blog posts on the topic of Google Search Console (formerly Google Webmaster Tools) and website errors:
Who knows…maybe you’ll enjoy fixing these errors as much as I do 🙂 And while you’re making these updates to your site, you might want to make other improvements as well. For a list of 25 website “must-haves” download a copy of our free eBook:
