All the search engines such as Google, Yahoo, MSN, etc., require a way of indexing all the billions of websites on the net and the pages contained within these sites. Thousands of new sites/blogs/personal pages are also added to the www daily so there really is no way for a human to keep up. The solution that the search engines, and many directories choose is in the form of automatic indexing called bots, indexers or spiders.
Essentially, a bot is programmed to look for certain criteria, then it goes out and scans the www reporting back what it finds to be indexed automatically.
Here's a good site that gives should provide all the info you'll ever need to understanding how a bot, spider or indexer works:
http://www.searchtools.com/robots/