Any problem with Document Archive in Bitsavers.org ?

From: Paul Williams <paul_at_frixxon.co.uk>
Date: Tue Jun 29 12:16:59 2004

Al Kossow wrote:
>
> As soon as bitsavers came on line again, google crawlers started downloading
> EVERYTHING from multiple IP adrs.

Put this in your robots.txt:

User-agent: Googlebot
Disallow: /*.pdf$

> FTP for mirroring isn't up yet, I've told Jay that I'll email when it works.
> Haven't heard any more from Patrick about the mirror he was starting.

Mirrors are clearly like buses. I'm one of the "dozens of people"[1] who
has a private mirror, for maintaining Manx and OCRing, but that is now
going online at VT100.net as I've moved to a dedicated server. Bandwidth
is, as Rolls-Royce would say, "adequate". However, I'm populating the
mirror from home at only 100 MiB an hour, so it could take weeks to get
up to speed!

[1] Al, 2004-05-07.

-- 
Paul
http://vt100.net/manx/ - a catalogue of online computer manuals
Received on Tue Jun 29 2004 - 12:16:59 BST

This archive was generated by hypermail 2.3.0 : Fri Oct 10 2014 - 23:37:01 BST