Yahoo started as a heirarchical web catalog ... they didn't crawl the web. Other services did crawl the web. Among them was Inktomi. Google crawled the web really well, and began to assign relevance based on the strength and number of backlinks. In between those two developments, Yahoo decided they needed a crawler and so they bought Inktomi. For some years that stood them in relatively good stead, alongside other search engines of the day. Ultimately, Google prevailed. Yahoo! still has some good stuff, and they have untapped ways of making money that Microsoft grasps better than they do. That's why Microsoft would love to become Microhoo! and give Google a run for their money.
Basically Slurp crawls the web page to index sites to become a part of Yahoo!'s Web Results, and other Inktomi powered search engine sites. There used to be a WHOLE lot more spiders which showed up in web site visitor logs. Mainly, if it's not IE/Netscape/Firefox/Safari/Opera, chances are it's a spider of some sort (except for WAP browsers).
