Site uses cookies to provide basic functionality.

OK
"were creating crawlers that would serve as search tools for the Web. These included the WWW Wanderer built by Matthew Gray at MIT, WebCrawler by Brian Pinkerton at the University of Washington, AltaVista by Louis Monier at the Digital Equipment Corporation, Lycos by Michael Mauldin at Carnegie Mellon University, OpenText by a team from Canada's University of Waterloo, and Excite by six friends from Stanford. All of them used link-hopping robots, or bots, that could dart around the Web like a binge drinker on a pub crawl, scarfing up URLs and information about each site. This would then be tagged, indexed, and placed in a database that could be accessed by a query server. Filo and Yang did not build their own web crawler; instead they decided to license one to add to their home page. Yahoo! continued to emphasize the importance of its directory, which was compiled by humans. When a user typed in a phrase, the Yahoo! computers would see if it related to an entry in the directory, and if so that handcrafted list of sites would pop up. If not, the query would be handed off to the Web-crawling search engine. The Yahoo! team believed, mistakenly, that most users would navigate the Web by exploring rather than seeking something specific. "The shift from exploration and discovery to the intent-based search of today was inconceivable," recalled Srinija Srinivasan, Yahoo!'s first editor in chief, who oversaw a newsroom of more than sixty young editors and directory compilers.114 This reliance on the human factor meant that Yahoo! would be much better than its rivals over the years (and even to the present) in choosing news stories, although not in providing search tools."