Newsgroups : Alt : alt.internet.search-engines : 2006 Aug : What is this thing spidering my site?

www.cryer.info
Managed Newsgroup Archive

What is this thing spidering my site?

Subject:What is this thing spidering my site?
Posted by:"Roy Schestowitz" (newsgrou..@schestowitz.com)
Date:Wed, 02 Aug 2006 05:46:47 +0100

I would generally hate to turn away legit crawlers, but a particaulr address
( sls-gc8p11.dca2.superb.net ) has been harvesting one of my sites for 4
days, taking up about 300 MB of pages per day. What can it be? Using HTTP on
that address gives the default Apache installtion page. Should I just block
all such mysterious requests? Any idea what it might be?

Glossary

Replies:

www.cryer.info
Managed Newsgroup Archive