--- "Dwight K. Elvey" <dwightk.elvey_at_amd.com> wrote:
> The closest I could find was this below. I'm not sure if it
> pertains to auto bidding? I don't know what a robot exclusion header
> is??
It is a convention amongst robot/spider writers to pick up and
respect the contents of a "robots.txt" file. This file contains
optional instructions for respecful software to not index or cache
listed web pages and/or directories. I have one on my Antarctic
web site to keep them from indexing things like my web stats
directories (who needs to cache months-old webified access reports,
after all).
I learned about robots.txt when I started seeing a lot of non-existent page
"errors" in the Wusage logs. Did a bit of digging and discovered
that it's a good idea to have even an empty robots.txt file on a
web page that is likely to see a crawler go by. I think as much as
30% of the traffic on my site is automated.
-ethan
__________________________________________________
Do You Yahoo!?
Yahoo! - Official partner of 2002 FIFA World Cup
http://fifaworldcup.yahoo.com
Received on Mon Jun 03 2002 - 19:32:36 BST