Googlers

From: Tothwolf <tothwolf_at_concentric.net>
Date: Thu Oct 9 22:20:43 2003

On Thu, 9 Oct 2003, jim wrote:
> Fred Cisin wrote:
>
> > I do NOT believe that they are spammers trolling for addresses, nor
> > any other nefarious purpose. They seem to be just hapless folk
> > without a clue.
>
> are the archive sites set to "no robots" so that they are not indexed on
> the search engines?
>
> If not, then anyone typing in PDP 11 on google will see all the PDP 11
> posts in the archive.
>
> Unless Jay has converted to an ASP or CGI script that won't serve up the
> contents to robots.

I think that allowing Google and other search engines to crawl the
messages is actually a *good* idea. There are a number of mailing lists
that I've dealt with that denied search engines, and only had their own
search function. Those search functions never did work very well.

If not for Google, I'd have not ever found the list. I'm sure I'm not the
only one.

-Toth
Received on Thu Oct 09 2003 - 22:20:43 BST