SEO / Web Marketing Articles:

Search Engine Indexing Errors

Recently, I worked with a client who had issues getting indexed by Google. Funny thing is Google really wanted to index them but they had server side issues which prevented indexing. Here is some of the things I learned that helped diagnois the problem and eventually resolve it.

First step is to set up Google webmaster tools for the site with the problem and run their report. You'll get a quick understanding of the problem.

In the Google webmaster tools you'll see the following items under "Crawl Errors": HTTP, In Sitemaps, Not Followed, Not Found, Restricted by Robots.txt, Timed Out, Unreachable

Restricted by Robots.txt - are pages that you have prevented from being crawled through your site's robots.txt file. There are a lot of misunderstandings about this file. Bascially, it is a text file that is on your server that tells the search engines what they are not allowed to crawl. When a crawler visits your site it is the first thing they check. If there is no robots.txt file, they will treat the site as if it is free to crawl. Robots.txt are best used to block crawling of secure files or potential duplicate content. To see how Google sees your site map do the following: click the "site configuration" link then "crawler access".

Timed Out - if you see errors, it means that sever could have been down during the time of the crawl.

Network unreachable - Here is word straight from Google: "This can happen when Googlebot encounters a time-out or other network related issue when requesting a file from your site, and thus is forced to abandon the request. This can be caused by one or more of the following:

A DNS configuration issue."

For dealing with network unreachable errors that can be time consuming see the article I created to help.

Are you experiencing Google indexing issues. Please feel free to contact me at 905-417-9470 or by email at allanp73@gmail.com