Hi All
Just wondered if you could clarify a few things for me.
a) Do the main search engines still look at the robots.txt file for indexing
purposes?
b) I once set up a robots.txt to disallow certain folders, which I wanted to
keep hidden from the rest of the world and therefore didn't want to be
spidered, and half a dozen alt.www.webmaster regulars found out about my
hidden folders straightaway because everybody looks for
www.mydomain.co.uk/robots.txt.
Somebody kindly advised me that I should put a blank robots.txt file in the
root of my hidden folders so that the search engines ignore them and
would-be funny hackers don't see any disallow list, but I feel that I may be
missing out in the search engines by not supplying them with a robots.txt
file with pages in it.
Could somebody put my mind at rest.
Many thanks.
Rgds
Robbie
