[support] Problematic Spiders, Suggestions for Managing

[Lists]

It appears that the MSN spider practically lives at my web site.

My watchdog logs are filling up with requests to non-existent pages, pages
that are member-only (access denied errors) and the like.

The culprit is at a narrow range of Microsoft IP's.

    207.46.98.129 - 207.46.98.134


Unrelated, I have some kind of bot that keeps trying to hit a (non-existent)
page at:  node/14

This is a varying address, but always in the same IP block (not the same
IP).  The attempt is made (to this and other places) about 6 to 8 times per
day, usually with 10 or so attempts at various "admin/*" pages along the
way.

What tools are others using to limit this bandwidth-sucking and log-clogging
activity?


Cheers,
--
Gary


P.S.  ...for those with a sense of humor...


Bad Behavior seemed reasonable, but the saga of my trying it out today goes,
so far, something like this:

    I installed this on my localhost server and on a live server, and I get
the same error in both environments.

    This is the same error reported here:

        http://www.angier.co.uk/drupal/downloads

    I got my copy from the 4.6.x link here:

        http://www.angier.co.uk/drupal/downloads

    I followed the installation instructions here:

        http://www.angier.co.uk/drupal/instructions/drupal-badbehavior-122

    Which lead me to get another copy from here:

        http://www.ioerror.us/software/bad-behavior/bad-behavior-download/

    Which produces the same error.  Okay, it's me.

    I now have a 'bad-behavior' v1.2.4 download, whose ReadMe.txt file says:

        This README.txt file applies to WordPress installations.
        For other types of installations, please see the documentation at

<http://www.ioerror.us/software/bad-behavior/installing-and-using-bad-behavi
or/>

        Which then tells me to go here:

<http://www.ioerror.us/software/bad-behavior/installing-and-using-bad-behavi
or/on-drupal/>

        Which says:

        "The installation process for Bad Behavior for Drupal is a little
different from other software on which Bad Behavior runs. You will need to
download Bad Behavior for Drupal and be sure to follow the installation
directions given there."

        With a download (but not really) link to:

            http://www.angier.co.uk/david/2005-08-19-drupal-bad-behaviour


        Which I follow, and which then tells me:

        "So, this is a fairly complete implementation, ready to roll.
Download here."


        With a (not yet, really) download link to:

            http://www.angier.co.uk/drupal/downloads


        And which also goes on to tell me:


        "To install, unpack Bad Behavior from here into a sub-directory of
the modules directory and then copy badbehaviour.module into that folder."


        With 'here' being linked to:

            http://www.ioerror.us/2005/08/16/bad-behavior-12/


So, split over two sites, 3 blogs and 4 interfaces, the author of this
software still hasn't informed me how to install and evaluate the software.

Whew.

I'm still trying to crack the code that is the 'bad behavior' module.

And it will probably turn out to be simple, or "me", but how can a software
allow such spiraling uncertainty from a user?  A simple paragraph from the
author, and a single source of information, would probably tell me what I
need to do.  That part, at least, can not be considered "user error".

I shall prevail!

Roses are #ff0000
Violets are #0000ff
All my base
are belong to you.

--
Mwa ha ha!