[UPHPU] throttling bots and spiders

phpninja phpninja at gmail.com
Fri Feb 22 12:37:43 MST 2008


One thing just off the top of my head would be a list similar to a
c:\windows\system32\drivers\etc\hosts file but for your website site. You
could write an array, or db table of malicious HTTP_USER_AGENT and if they
match just give them an IP/site ban. A quick google search gave me this
list: http://www.pgts.com.au/pgtsj/pgtsj0208d.html but I am sure there are
probably more.

Another thing could be the mod_throttle module (i think it only supports the
1.* version of apache though)
http://gunnm.org/~soda/work/oldstuff/vhffs2/vhffs-modthrottle/doc/ . You can
set policies with this, such as concurrent connections.

Regards,
-phpninja


On 2/22/08, Wade Preston Shearer <lists at wadeshearer.com> wrote:
>
> Are there any effective methods for throttling bots and spiders from
> spidering your site? If your sessions are database-based, how do you
> keep them from bringing your server to it's knees?
>
> _______________________________________________
>
> UPHPU mailing list
> UPHPU at uphpu.org
> http://uphpu.org/mailman/listinfo/uphpu
> IRC: #uphpu on irc.freenode.net
>
>


More information about the UPHPU mailing list