This article was originally written in April 2004…it is cool to read because gives some history and insight to the way the web is now.
The Current State of the Search Engines
Yesterday, I received a piece of Spam that told me they could submit my site to 100,000 search engines for only $99/year. I was amazed that people still offer this type of service. Maybe I have been an insider for too long, but doesn’t everyone know that you don’t need to submit your site to the search engines. Also, don’t they realize that there aren’t 100,000 search engines? The fact there really only two search engines: Yahoo and Google. Recently, the lines were drawn and war was declared between the two. Yahoo spent last year building its arsenal buying major web properties Overture, AlltheWeb/Fast, AltaVista, and Inktomi. Yahoo basically bought the ability to do everything that Google once provided for them. Yahoo is now using a variation of Inktomi index and crawler (called Yahoo serp) to power its search results (serps), having dropped Google at the end of February. Yahoo is still working to integrate its search technologies, but has announced that on April 15th it will officially launch SiteMatch. SiteMatch replaces the paid inclusion programs offered by AltaVista and Inktomi. The new system will charge $49 per URL plus $.15 per click. Sites listed through SiteMatch will be given no preference in the natural serps over sites that were crawled for free, but will be indexed more often then non-paid for inclusion sites. Yahoo now offers a free submission service, for those for do not wish to pay. One problem for those using Inktomi’s paid for inclusion service is their sites will be dropped from Yahoo’s serps after April 15th. Also, Yahoo serps will soon replace the serps at AltaVista and AlltheWeb.
On a side note for those who might have said there are really three major search engines: Yahoo, Google and MSN. Realize Inktomi also powers MSN, who is still a year out from releasing its new search technology codename “Longhorn”. Also, MSN dropped Looksmart so its results are almost the same as Yahoo’s where Overture sponsored listings are on top then soon followed by the natural ranked sites.
How does Google respond to Yahoo’s challenge? Google made several important acquisitions as well, purchasing Applied Semantics, Pyra Labs (Blogs), and the patent for the Hilltop Algorithm. Google’s method was to defeat Yahoo by having not only a larger index but to have more relevant search results. At the fall of last year, Google went from updating its index once a month to a constant update in order to provide the freshest results. In November (Black November for many) it launched its Hilltop Algo called the Florida update. Hilltop runs a series of checks where it determines whether a site is an authority ranks it accordingly. Sites, which are not deemed an authority, are dropped. To be classified as an authority ones site must have a considerable number of links in and at the same time many links out to relevant sites. As a result many commercial sites which lack as two attributes which filtered by the system. Many business owners (myself included) complained that Google was purposely filtering commercial sites in order to generate more Adwords revenue. To further fuel this accusation, was the fact that the Hilltop algorithm was only applied to popularly searched terms, which was dominated by commercial keywords. In January, Google increased its usage of Hilltop targeting secondary as well as primary key phrases. This update called Austin further upset commercial site owners who found their sites lost from the very terms they targeted. Searches for “real estate” for example would return weather, university, and major directories. It seems that Hilltop is here to stay, whether it better or not. Yahoo even pointed out Google’s blunder by referring to the fact that doing a search for a particular city hotel would get hotels in that city, where on Google the same search leads to only directories in the results. Google’s spokespeople defended their new algorithm and told those who wish to target commercial terms they should spend money on Adwords. Strangely Google has alienated the very people who made it a success. However, Google has increased its revenue and stands ready for its IPO.
How the search engine wars will play out is anyone’s guess. But the question many commercial site owners are asking is: Are free search listings and free rankings going to be a casualty? For now don’t pay the $99/year; both Yahoo and Google will crawl your site for free if it has links from other sites.
Any questions about the actual current state of the search engines, please feel free to contact me at 905-417-9470 or by email at allanp73@gmail.com
