dear readers: I would like to do a basic web search and download a
list of all web pages matching the results. I don't care very much
about ordering, because my own perl programs will then wget the
resulting web pages and see if they meet other needs of mine. (I do
need to sift through 1000's of result pages, though.)
of course, I could use one of the many publicly accessible spider
program, and crawl the web myself, but this seems like a waste of
bandwidth. are there public repositories that avoid the need for me to
crawl? google.com used to have an API, but apparently just dropped it.
moreover, I don't need much google or pagerank sophistication---I need
the old altavista-like comprehensiveness more than cleverness.
any pointers would be appreciated.
sincerely,
/iaw
