| Crawler-based search engines automatically visit web pages to compile their listings. This means that, unlike directories, you are likely to have several if not many pages listed with them. This also means that by taking care in how you build your pages, you might rank well in crawler-produced results.
Optimizing pages for crawlers is covered more in the Optimizing For Crawlers section of Search Engine Watch. If you have time, it is recommended that you read the pages in this section, especially the Search Engine Placement Tips page. However, by simply following the submission tips below, you can at least get your pages listed with crawlers, where they might naturally rank well for certain terms.
Submitting To Google
One of the most important crawler-based search engines is Google, because many people search at it, plus it "powers" the main results of several other services (see the Search Engine Results Chart for what "main" results are).
The absolute best way to get listed with Google is to build links to your web site. Indeed, this is the best way to get listed for free with all the major crawlers listed on this page. Crawlers follow links, so if you have good links pointing at your web site, the crawlers are more likely to find and include your pages.
Here's the good news: if you submitted your site to the major directories and got listed with one of them, then Google and other crawlers will almost certainly pick up the URL that was listed. This means you may not need to do additional work to get listed with crawlers.
Nevertheless, you may find it helpful to do some link building beyond the directories. Basic tips on building good links are covered on the Search Engine Placement Tips page, while the Link Analysis & Link Building page available to Search Engine Watch members provides in-depth advice on building relevant links to your web site. Consider reading what's covered on one or both of these pages.
The Submitting & Encouraging Crawlers page available to Search Engine Watch members also has advice on how your site architecture can be improved to naturally allow more of your pages to be added by crawlers.
Aside from link building, Google provides an Add URL page that lets you submit a URL directly to its crawler. There's no guarantee that Google will actually include a URL submitted to it this way, however. Despite this, it makes sense to submit your home page and perhaps one or two other URLs from "inside" your web site via the Add URL page.
You really don't need to submit more than this. The only reason for submitting some of your inside pages is in case there is a problem reaching your home page. This gives Google an alternate route into your site. From whatever page it visits, it will look for links to other pages that you have and perhaps include those. This is true for other crawlers, as well.
If you have a brand new web site, it will probably take about a month before Google lists your web pages. Because of this, you might consider making use of its paid placement program, which is covered in the next part of this guide.
Finally, Search Engine Watch members have access to a detailed How Google Works page that guides you even more through the process of submitting to the crawler and ranking well within its results. To learn more about becoming a member to access this information, visit the membership information page.
Submitting To Yahoo
Yahoo is an important crawler-based search engine because many people use the Yahoo site and it provides main results of several other services (see the Search Engine Results Chart for more about this).
As covered with Google, building links is the best way to get listed for free. Yahoo also offers a free URL submission form that you'll find listed on this page. Submit according to the same instructions as for Google, above.
What if you aren't picked up for free? Yahoo has paid inclusion programs that guarantee to add the pages you submit quickly. The downside to these programs is that you'll be charged every time someone clicks on your listing. If you run out of money, potentially, your listing may be dropped. However, there's still a chance that even if you run out of money, you might continue to be listed for free.
Confused? You're to be forgiven, if so. The programs are so complex that beginners are not recommended to bother with them. Instead, there's a good chance that many pages in your site will just naturally get listed for free.
By the way, Yahoo's crawler incorporates technology from three different crawlers that it purchased in 2002 and 2003: Inktomi, AltaVista and FAST's AllTheWeb. Any references you hear about those crawlers are now superceded by the single Yahoo crawler that operates.
Submitting To Teoma
Teoma is an important crawler-based search engine because it powers the main of the results that appear at the popular Ask Jeeves web site (see the Search Engine Results Chart). In fact, Ask Jeeves owns Teoma.
Teoma has no free Add URL page. This doesn't mean that you can't get listed, however. Teoma crawls the web, so if you have links pointing at your web site, you may get included naturally
Basic tips on building good links are covered on the Search Engine Placement Tips page, while the Link Analysis & Link Building page available to Search Engine Watch members provides in-depth advice on building relevant links to your web site. Consider reading what's covered on one or both of these pages, to help yourself with Teoma.
Submitting To MSN
MSN Search is an important crawler-based search engine used by many people. It is currently powered by Yahoo's crawler-based results. However, by the end of 2004, a new crawler-based search engine being developed by Microsoft should be taking over.
That new Microsoft search engine is already available to the public in a limited fashion. More about it, and how it relates to the current MSN service, can be found in this article from Search Engine Watch: MSN Search Gets New Look; Microsoft Gets New Search Engine.
|  |
