Okay - I have been getting some extremely unusual crawler activity on
my website, which is actually pulling its bandwidth usage into overload
(and is going to start costing me money).
I do have a robots.txt file, and I thought I would check its syntax.
Fair enough. Now this is where it got wierd. When I go to any part of
robotstxt.org, it crashes my browser, before loading much of the site
at all.
Fine, I am using firefox, try it in IE. It also crashes the browser.
Okay - try it on a different computer, with IE - crashes the browser.
This is mighty odd behaviour.
Try it a different subnet - I get the same. Try it on a different
network - ouch still got a crash.
The only thing that showed a difference was my PDA (A Pocket PC 2003
device) - which simply said "the page you are looking for cannot be
found".
Is something really fishy - like a defaced site being doctored to send
out malformed headers and crashing browsers? If I link it with strange
crawler behaviour, it makes me wander if there is some general attack
going on, that nobody has clocked yet. I may just be overreacting or
missing something, but I have used that site before, and I have *never*
seen this before. Not the same site, regardless of deep linking,
regardless of browser, and regardless of network/machine.
Any ideas here?
Orion
