Al Kossow wrote:
>
> As soon as bitsavers came on line again, google crawlers started downloading
> EVERYTHING from multiple IP adrs.
Put this in your robots.txt:
User-agent: Googlebot
Disallow: /*.pdf$
> FTP for mirroring isn't up yet, I've told Jay that I'll email when it works.
> Haven't heard any more from Patrick about the mirror he was starting.
Mirrors are clearly like buses. I'm one of the "dozens of people"[1] who
has a private mirror, for maintaining Manx and OCRing, but that is now
going online at VT100.net as I've moved to a dedicated server. Bandwidth
is, as Rolls-Royce would say, "adequate". However, I'm populating the
mirror from home at only 100 MiB an hour, so it could take weeks to get
up to speed!
[1] Al, 2004-05-07.
--
Paul
http://vt100.net/manx/ - a catalogue of online computer manuals
Received on Tue Jun 29 2004 - 12:16:59 BST