[UPHPU] Default robots.txt file

[Richard K Miller]

On Aug 7, 2007, at 3:27 PM, Jon Jensen wrote:

>>> ###mod_rewrite
>>> RewriteEngine On
>>> RewriteCond %{REQUEST_FILENAME} !-f
>>> # Don't really need to verify robots.txt is a directory
>>> RewriteRule /robots.txt http://masterhost.domain.com/robots.txt [L]
>>
>> If I use "http://" in the substitution string, it issues an HTTP  
>> redirect, which I don't want. (I want each robots.txt to be served  
>> from its own domain.)
>>
>>> ###Or even easier with mod_alias
>>> Alias /robots.txt /www/public/robots.txt
>>
>> The Alias command works, with one exception: The file at /www/ 
>> public/robots.txt is always served, even if a VirtualHost has its  
>> own robots.txt file. Is there a way to do the equivalent of  
>> RewriteCond %{REQUEST_FILENAME} !-f with an Alias command?
>
> You probably want a combination of both, since mod_rewriting to  
> another hostname always does a redirect ... Apache isn't going to  
> fetch that external URL for the browser, instead it tells the  
> browser to go there. Try this:
>
> Alias /robots-default.txt /www/public/robots.txt
>
> RewriteEngine On
> RewriteCond %{REQUEST_FILENAME} !-f
> RewriteRule /robots.txt /robots-default.txt

Ingenious idea to use both mod_alias and mod_rewrite. I added the  
[PT] flag ("pass through") and it worked:

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule /robots.txt /robots-default.txt [PT]

Some of my VirtualHosts that have their own RewriteRules aren't  
working, but many of them are.