[support] Eliminating bot-noise

[Tibor Liktor]

On Thu, 30 Aug 2007 08:11:19 -0400
Earnie Boyd <earnie at users.sourceforge.net> wrote:

> Quoting Tibor Liktor <liktor at gmail.com>:
> 
> > Hi,
> >
> >
> > I've got a watchdog problem.
> >
> > The watchdog log is essential for me to discover bugs and errors, and 
> > monitor the site's
> > performance, and blahblablah - you know that.
> >
> > But watchdog became quite useless for me, because it is full of 404 
> > errors triggered by Google and
> > other bots.
> >
> > Now nearly 90% of the log is crap. It is impossible to dig out any 
> > useful info from that. (Not
> > speaking about the additional server load and database size issues.)
> >
> > Is there any solution to filter out the tons of messages caused by 
> > searchbots?
> >
> > Do you face with similar issues? How do you handle those?
> >
> 
> There is a filter by message type list box that can help but that is 
> too simple and I must not be understanding you.  I do know that the 
> "page not found" errors tend to be ridiculous and the drupal engine 
> itself causes a several; i.e. the referrer is the site on which the 
> watchdog log is being reviewed.  It would be nice if I could filter to 
> "all messages except page not found messages"; is that what you mean?  
> Can't do it out of the box but that doesn't mean you can't program for 
> it.
> 
> Earnie -- http://for-my-kids.com/
> -- http://give-me-an-offer.com/
> 

Hi,


no, I need the "normal" 404 messages, since they provide me useful infos about broken links on the
site.

My problem is the Googlebot tries links thousand years dead, etc. and generates a massive amount of
unnecessary entries in my watchdog.

I only would like to filter for domainnames, IP-s, etc.


Best,
Tibor