Sellam Ismail wrote:
>
> On Fri, 24 May 2002, Megan wrote:
> >
> > "Per the request of the site owner, http://www.dec.com
> > is no longer available in the Wayback Machine.
>
> But I wonder if they do in fact have the pages archived? My guess
> is that they do, but they just can't make them available.
This "per the request of the site owner" is nothing more than an
instruction placed in the website's robots.txt file, which can request
that either specific or all crawlers do not retrieve pages from the
site. When archive.org's crawler sees this instruction, it definitely
does not crawl the site, so they don't have a large stash of pages that
are not available for viewing.
- Paul
Received on Sat May 25 2002 - 06:33:12 BST