.htaccess.
robots.txt is optional, and not all bots respect it, or even check for it. I'd also wager that it's not bots but scripts trying to find many of those files and execute exploits. install.php? Where would a search engine bot find a link to that, or CHANGELOG.txt?
Alternatively, you could just put all the file back in place ;)
-Mike __________________ Michael Prasuhn 503.512.0822 office mike@mikeyp.net http://mikeyp.net
On Jun 18, 2010, at 2:53 PM, Shai Gluskin wrote:
Support folk,
I'm getting Page not Found errors in my Drupal log every hour, on multiple sites on my server, for the files I deleted from the root directory:
• install.php • CHANGELOG.txt • INSTALL.txt • INSTALL.mysql.txt • INSTALL.pgsql.txt • LICENSE.txt • MAINTAINERS.txt • UPGRADE.txt • COPYRIGHT.txt
The robots.txt tells the bots to ignore those pages... but???
Should I just ignore it? Any good wisdom someone has on advising on how to handle all those bots that are pinging my server all the time?
Thanks,
Shai
[ Drupal support list | http://lists.drupal.org/ ]