Search Bots, Crawlers, and Spiders

If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?

Not quite sure what I am talking about? Here is a few examples of various bots searching my website:

207.68.146.40 (msnbot.msn.com)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
This is the MSN Search bot.

207.68.146.40 (lj2070.inktomisearch.com)
Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)
This is Yahoos Search Bot.

66.249.65.147 (crawl-66-249-65-147.googlebot.com)
Mediapartners-Google/2.1
This is Googles bot, that searches your webpages for AdSense.

What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites.

What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered.

Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard.

How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do.

How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website.

What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days.

Ken Dennis
http://KenDennis-RSS.homeip.net/

Business Consulting Web Development Web Site Design Graphic Design Ecommerce
Marketing Web Hosting Domain Name Registration Clients Web Maintenance Voice Overs Talent
Contact Us Company Info Home Page
>> Link Exchange Directory - Ad Your Site Now! <<

INFORMATION AND HELPFUL ARTICLES
Web Design Web Hosting Web Development Search Engine Optimization
Web Design Samples Graphic Design Samples Glossary Pantone Chart RGB HTML

Tampa Web Design and Development Tampa Bay Web PHP Development and Web Hosting
Graphic Design Tampa Bay Small Business Consultant

Search Engine Watch

Search Engine Optimization Inc. Announces New Addition to Senior ... SYS-CON Media, NJ - 1 hour ago As CDO for Search Engine Optimization Inc. (SEO Inc.), Ed will develop new strategies for future sales growth and company expansion as well as spearhead the ... Bruce Clay, Inc. Releases New Free SEO Tools MSNBC Search Engine Optimization(On Page) PWO Seo infotech is providing free search engine optimization ... PR-Inside.com (Pressemitteilung) Promotion World (press release) - dBusinessNews Cleveland (press release) all 69 news articles

Baytech Releases Long-Awaited Integrated SEO / PPC Marketing Solution Business Wire (press release), CA - 2 hours ago Search engine optimization, pay-per-click advertising (PPC) and email marketing services will be offered as one powerful and integrated marketing solution. ...

Mark Your Calendars: Search Engine Strategies (SES) Returns to New ... MarketWatch (press release) - Jan 7, 2009 Incisive Media's Search Engine Strategies is a leading global conference and training series focused on search engine optimization and search engine ...

Best Syndication

A-1 Internet Design - Keeping Pace with the Global Internet marketing Best Syndication, CA - 18 hours ago Their package of affordable search engine marketing services include effective search engine optimization, and in their competitors circle they are heralded ...

Derek Edmond Becomes Partner in Massachusetts-Based Search Engine ... Business Wire (press release), CA - Jan 6, 2009 KoMarketing Associates, LLC is pleased to announce that Derek Edmond, currently the Director of Search Engine Optimization & Search Marketing Strategies at ...

IndiaBook Indian SEO Company Launches Search Marketing Outsourcing ... PowerHomeBiz.com (press release) - 22 hours ago India Book SEO company India offers all type of Internet search engine optimization services including competitor analysis, keyword research,on page ...

Adweek

Search Engine Enlightenment Traffick - Jan 6, 2009 Although pretty heavy-handed in some cases, this also goes for Google's "optimization suggestions." While Google is pushing the "room for improvement" ... Yahoo Modifies Search Advertisers Terms And Conditions eBrandz Yahoo Angers Advertisers After Terms & Conditions Update Notification Search Engine Land all 60 news articles

Search Engine Marketing Agency, Formic Media, Launches PR Web (press release), WA - Jan 5, 2009 Formic's initial SEM service offerings include web analytics optimization, keyword research & recommendations, website search engine optimization (SEO) ... Oregon Marketing Firm Focuses On Local SEM Services Mediapost.com all 4 news articles

BizAtomic Launches Version 5.1 of its SEO Optimized E3 Ecommerce ... Emediawire (press release), WA - 23 hours ago "We feel that every eBusiness needs to have a basic understanding of the Search Engine Optimization, even if they are not performing their own Search Engine ...

OptimizeMyROI, a Leader in Website Marketing Continues to ... PR.com (press release), NY - Jan 7, 2009 SportsBanners4you.com , said: “In a matter of only a few weeks, Heath Nolan and his Search Engine Optimization Consultant team at OptimizeMyROI produced a ...

© MediaRocket.com All rights reserved.
 
Business Consulting Web Development Web Site Design Graphic Design Ecommerce
Marketing Web Hosting Domain Name Registration Clients Web Maintenance

voice overs work, onhold music, on hold message, on hold messages, message for voice mail, voicemail greetings, voice mail greetings, voice mail greeting, voicemail messages, messages for voicemail, messages for voicemails, voice services, radio voices, radio voice, voice over recordings, voice over auditions, voice acting, voice over artists, voice over artist, voiceover artists, voiceover artist, voiceover auditions, voiceover audition, voiceovers auditions, voice over agents


Tampa web site design voice talent and voice overs, small business consulting web design graphic design business consulting services tampa web design web design tampa business consulting st. petersburg web design florida web design sarasota web design jacksonville web design orlando web design miami web design business consultant  radio station web design radio web design clearwater web design tallahassee web design small business consultant real estate web design west palm beach web design ecommerce web design business web design hosting web design professional web design flash web design flash graphics web design web design company web page designers color chart Web Design -- Business Consulting Services Web Development Web Site Design Graphic Design Ecommerce Marketing Web Hosting Clients Web Maintenance Contact Us Company  Home Page Employment  Privacy Glossary, Voice Talent, Voice Overs voice over talent voiceover artist voice overs