[UPHPU] Default robots.txt file

[Velda Christensen]

Jon Jensen wrote:
> It's a way for web sites to tell bots/spiders/crawlers how to behave 
> (e.g. what they can and cannot view), though it is of course entirely 
> up to the bot to comply.
>
> http://www.robotstxt.org/wc/norobots.html 
We find that there are a ton of bots that do the exact opposite of what 
you tell them to.  I'd use .htaccess to deny all traffic in directories 
and files that no one needs to access via http, to password protect 
admin / private directories, and to just deny bots in directories you 
simply don't want indexed.

Robots.txt files are great though for telling google which pictures it 
can and can't use :-)

-V