[support] Getting Control of Spiders...

[Shai Gluskin]

Support folk,

I'm getting Page not Found errors in my Drupal log every hour, on multiple
sites on my server, for the files I deleted from the root directory:


   - install.php
   - CHANGELOG.txt
   - INSTALL.txt
   - INSTALL.mysql.txt
   - INSTALL.pgsql.txt
   - LICENSE.txt
   - MAINTAINERS.txt
   - UPGRADE.txt
   - COPYRIGHT.txt


The robots.txt tells the bots to ignore those pages... but???

Should I just ignore it? Any good wisdom someone has on advising on how to
handle all those bots that are pinging my server all the time?

Thanks,

Shai
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.drupal.org/pipermail/support/attachments/20100618/744ce0d0/attachment.html