[UPHPU] Default robots.txt file

[Richard K Miller]

On Aug 7, 2007, at 3:34 PM, Scott Hill wrote:

> OK, I have a minute so I was wondering what is robots.txt and what  
> is it's purpose and/or uses?
>

The robots.txt file communicates to complying search engines which  
pages should not be indexed, and which search engines should not  
crawl a site.

For example:

http://www.apple.com/robots.txt

http://discussions.apple.com/robots.txt

http://php.net/robots.txt

For more info:

http://www.robotstxt.org/