This article is pretty interesting, it’s about a project called Grub.
“It will be the first comprehensive index (of the Net),” said Kord Campbell, the programmer behind the Grub software. “We can conceivably crawl every Web page, every day.”
By contrast, today’s fastest search engines, such as Google or Inktomi, crawl about 150 million pages a day. Google indexes about a third of the Web, and refreshes its index every 30 days, according to LookSmart.
Google does make some good points, however:
Peter Norvig, director of search quality at Google, said while the Grub project is topical and interesting; improving Web searches isn’t a problem of widening an index, but narrowing it.
“It isn’t a problem of computing resources but deciding what parts of the Web should be updated more frequently than others,” he said.
