I'm getting Page not Found errors in my Drupal log every hour, on multiple sites on my server, for the files I deleted from the root directory:
- install.php
- CHANGELOG.txt
- INSTALL.txt
- INSTALL.mysql.txt
- INSTALL.pgsql.txt
- LICENSE.txt
- MAINTAINERS.txt
- UPGRADE.txt
- COPYRIGHT.txt
The robots.txt tells the bots to ignore those pages... but???
Should I just ignore it? Any good wisdom someone has on advising on how to handle all those bots that are pinging my server all the time?
Thanks,
Shai