[support] Eliminating bot-noise

[Tibor Liktor]

Hi Mike,

I was afraid that I get a similar answer, but I also don't see other solution.

Thx all!

Best,
Tibor

On Thu, 30 Aug 2007 08:15:15 -0500
Michael MacKenna <mpmackenna at gmail.com> wrote:

> This may seem like a lot of work for solving your issue, but depending on how desperate you are
> it should work.  You could right a python script that digs through the log files and skips all
> the bogus lines that match a certain pattern and writes the lines that don't fit that pattern to
> a new log file.  The end result would be a new log file that contains only the information you
> need.
> 
> Mike
> 
> 
> 
> Tibor Liktor wrote:On Thu, 30 Aug 2007 08:11:19 -0400
> Earnie Boyd <earnie at users.sourceforge.net> wrote:
> 
>   Quoting Tibor Liktor <liktor at gmail.com>:
> 
>     Hi,
> 
> 
> I've got a watchdog problem.
> 
> The watchdog log is essential for me to discover bugs and errors, and 
> monitor the site's
> performance, and blahblablah - you know that.
> 
> But watchdog became quite useless for me, because it is full of 404 
> errors triggered by Google and
> other bots.
> 
> Now nearly 90% of the log is crap. It is impossible to dig out any 
> useful info from that. (Not
> speaking about the additional server load and database size issues.)
> 
> Is there any solution to filter out the tons of messages caused by 
> searchbots?
> 
> Do you face with similar issues? How do you handle those?
> 
>       There is a filter by message type list box that can help but that is 
> too simple and I must not be understanding you.  I do know that the 
> "page not found" errors tend to be ridiculous and the drupal engine 
> itself causes a several; i.e. the referrer is the site on which the 
> watchdog log is being reviewed.  It would be nice if I could filter to 
> "all messages except page not found messages"; is that what you mean?  
> Can't do it out of the box but that doesn't mean you can't program for 
> it.
> 
> Earnie -- http://for-my-kids.com/
> -- http://give-me-an-offer.com/
> 
>     
> Hi,
> 
> 
> no, I need the "normal" 404 messages, since they provide me useful infos about broken links on the
> site.
> 
> My problem is the Googlebot tries links thousand years dead, etc. and generates a massive amount
> of unnecessary entries in my watchdog.
> 
> I only would like to filter for domainnames, IP-s, etc.
> 
> 
> Best,
> Tibor
>