[support] Getting Control of Spiders...

[Michael Prasuhn]

.htaccess.

robots.txt is optional, and not all bots respect it, or even check for it. I'd also wager that it's not bots but scripts trying to find many of those files and execute exploits. install.php? Where would a search engine bot find a link to that, or CHANGELOG.txt?

Alternatively, you could just put all the file back in place ;)

-Mike
__________________
Michael Prasuhn
503.512.0822 office
mike at mikeyp.net
http://mikeyp.net

On Jun 18, 2010, at 2:53 PM, Shai Gluskin wrote:

> Support folk,
> 
> I'm getting Page not Found errors in my Drupal log every hour, on multiple sites on my server, for the files I deleted from the root directory:
> 
> 	• install.php 
> 	• CHANGELOG.txt 
> 	• INSTALL.txt 
> 	• INSTALL.mysql.txt 
> 	• INSTALL.pgsql.txt 
> 	• LICENSE.txt 
> 	• MAINTAINERS.txt 
> 	• UPGRADE.txt 
> 	• COPYRIGHT.txt
> 
> The robots.txt tells the bots to ignore those pages... but???
> 
> Should I just ignore it? Any good wisdom someone has on advising on how to handle all those bots that are pinging my server all the time?
> 
> Thanks,
> 
> Shai
> -- 
> [ Drupal support list | http://lists.drupal.org/ ]