On Tuesday 29 June 2004 12:16, Paul Williams wrote:
> Al Kossow wrote:
> > As soon as bitsavers came on line again, google crawlers started
> > downloading EVERYTHING from multiple IP adrs.
>
> Put this in your robots.txt:
>
> User-agent: Googlebot
> Disallow: /*.pdf$
Grr. Don't do this. I really hate it when people disallow google to
index content. It always makes it harder to find stuff. The only time
I'd consider doing it is if the "webserver" is on a dialup connection
or something that won't stay at the same IP address.
Pat
--
Purdue University ITAP/RCS --- http://www.itap.purdue.edu/rcs/
The Computer Refuge --- http://computer-refuge.org
Received on Tue Jun 29 2004 - 12:58:39 BST