Eliminating bot-noise
Hi, I've got a watchdog problem. The watchdog log is essential for me to discover bugs and errors, and monitor the site's performance, and blahblablah - you know that. But watchdog became quite useless for me, because it is full of 404 errors triggered by Google and other bots. Now nearly 90% of the log is crap. It is impossible to dig out any useful info from that. (Not speaking about the additional server load and database size issues.) Is there any solution to filter out the tons of messages caused by searchbots? Do you face with similar issues? How do you handle those? Best, Tibor ps.: Afais, it is might be caused by two reasons: 1. I use GoogleAds 2. the site had been dramatically changed, and bot searches for the 'old' content fixme.
Quoting Tibor Liktor <liktor@gmail.com>:
Hi,
I've got a watchdog problem.
The watchdog log is essential for me to discover bugs and errors, and monitor the site's performance, and blahblablah - you know that.
But watchdog became quite useless for me, because it is full of 404 errors triggered by Google and other bots.
Now nearly 90% of the log is crap. It is impossible to dig out any useful info from that. (Not speaking about the additional server load and database size issues.)
Is there any solution to filter out the tons of messages caused by searchbots?
Do you face with similar issues? How do you handle those?
There is a filter by message type list box that can help but that is too simple and I must not be understanding you. I do know that the "page not found" errors tend to be ridiculous and the drupal engine itself causes a several; i.e. the referrer is the site on which the watchdog log is being reviewed. It would be nice if I could filter to "all messages except page not found messages"; is that what you mean? Can't do it out of the box but that doesn't mean you can't program for it. Earnie -- http://for-my-kids.com/ -- http://give-me-an-offer.com/
On Thu, 30 Aug 2007 08:11:19 -0400 Earnie Boyd <earnie@users.sourceforge.net> wrote:
Quoting Tibor Liktor <liktor@gmail.com>:
Hi,
I've got a watchdog problem.
The watchdog log is essential for me to discover bugs and errors, and monitor the site's performance, and blahblablah - you know that.
But watchdog became quite useless for me, because it is full of 404 errors triggered by Google and other bots.
Now nearly 90% of the log is crap. It is impossible to dig out any useful info from that. (Not speaking about the additional server load and database size issues.)
Is there any solution to filter out the tons of messages caused by searchbots?
Do you face with similar issues? How do you handle those?
There is a filter by message type list box that can help but that is too simple and I must not be understanding you. I do know that the "page not found" errors tend to be ridiculous and the drupal engine itself causes a several; i.e. the referrer is the site on which the watchdog log is being reviewed. It would be nice if I could filter to "all messages except page not found messages"; is that what you mean? Can't do it out of the box but that doesn't mean you can't program for it.
Earnie -- http://for-my-kids.com/ -- http://give-me-an-offer.com/
Hi, no, I need the "normal" 404 messages, since they provide me useful infos about broken links on the site. My problem is the Googlebot tries links thousand years dead, etc. and generates a massive amount of unnecessary entries in my watchdog. I only would like to filter for domainnames, IP-s, etc. Best, Tibor
Hi Mike, I was afraid that I get a similar answer, but I also don't see other solution. Thx all! Best, Tibor On Thu, 30 Aug 2007 08:15:15 -0500 Michael MacKenna <mpmackenna@gmail.com> wrote:
This may seem like a lot of work for solving your issue, but depending on how desperate you are it should work. You could right a python script that digs through the log files and skips all the bogus lines that match a certain pattern and writes the lines that don't fit that pattern to a new log file. The end result would be a new log file that contains only the information you need.
Mike
Tibor Liktor wrote:On Thu, 30 Aug 2007 08:11:19 -0400 Earnie Boyd <earnie@users.sourceforge.net> wrote:
Quoting Tibor Liktor <liktor@gmail.com>:
Hi,
I've got a watchdog problem.
The watchdog log is essential for me to discover bugs and errors, and monitor the site's performance, and blahblablah - you know that.
But watchdog became quite useless for me, because it is full of 404 errors triggered by Google and other bots.
Now nearly 90% of the log is crap. It is impossible to dig out any useful info from that. (Not speaking about the additional server load and database size issues.)
Is there any solution to filter out the tons of messages caused by searchbots?
Do you face with similar issues? How do you handle those?
There is a filter by message type list box that can help but that is too simple and I must not be understanding you. I do know that the "page not found" errors tend to be ridiculous and the drupal engine itself causes a several; i.e. the referrer is the site on which the watchdog log is being reviewed. It would be nice if I could filter to "all messages except page not found messages"; is that what you mean? Can't do it out of the box but that doesn't mean you can't program for it.
Earnie -- http://for-my-kids.com/ -- http://give-me-an-offer.com/
Hi,
no, I need the "normal" 404 messages, since they provide me useful infos about broken links on the site.
My problem is the Googlebot tries links thousand years dead, etc. and generates a massive amount of unnecessary entries in my watchdog.
I only would like to filter for domainnames, IP-s, etc.
Best, Tibor
On Thu, Aug 30, 2007 08:15:15 AM -0500, Michael MacKenna (mpmackenna@gmail.com) wrote:
This may seem like a lot of work for solving your issue, but depending on how desperate you are it should work. You could right a python script that digs through the log files and skips all the bogus lines that match a certain pattern
There is probably no need to start python or other interpreted languages. Something like this: grep -v FIRST_IRRELEVANT_PATTERN log_file | \ grep -v SECOND_IRRELEVANT_PATTERN | \ grep -v THIRD_IRRELEVANT_PATTERN | \ etc etc,
purged_log_file
would probably be lighter on the server, I think. HTH, Marco -- Help *everybody* love Free Standards and Free Software http://digifreedom.net/
On Thu, 30 Aug 2007 20:27:34 +0200 "M. Fioretti" <mfioretti@nexaima.net> wrote:
There is probably no need to start python or other interpreted languages. Something like this:
grep -v FIRST_IRRELEVANT_PATTERN log_file | \ grep -v SECOND_IRRELEVANT_PATTERN | \ grep -v THIRD_IRRELEVANT_PATTERN | \
N forks. grep -vE '(FIRST_IRRELEVANT_PATTERN)|(SECOND_IRRELEVANT_PATTERN)...' oh btw this doesn't make it the right solution, it should just make yours a bit faster. -- Ivan Sergio Borgonovo http://www.webthatworks.it
Do you have the GSiteMap / XML Sitemap module installed? If you don't, I'd recommend it. Not only does it help with SEO immensely, but it may help convince Google that gee, those other files don't exist anymore, stop trying to find them. I don't know for sure if Google will take the hint, but generally that module does help with SEO anyway and reduces server load (because Google knows where all of your important pages are) so I'd recommend it regardless. On Thursday 30 August 2007, Tibor Liktor wrote:
Hi,
I've got a watchdog problem.
The watchdog log is essential for me to discover bugs and errors, and monitor the site's performance, and blahblablah - you know that.
But watchdog became quite useless for me, because it is full of 404 errors triggered by Google and other bots.
Now nearly 90% of the log is crap. It is impossible to dig out any useful info from that. (Not speaking about the additional server load and database size issues.)
Is there any solution to filter out the tons of messages caused by searchbots?
Do you face with similar issues? How do you handle those?
Best, Tibor
ps.:
Afais, it is might be caused by two reasons:
1. I use GoogleAds 2. the site had been dramatically changed, and bot searches for the 'old' content
fixme.
-- Larry Garfield AIM: LOLG42 larry@garfieldtech.com ICQ: 6817012 "If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it." -- Thomas Jefferson
participants (6)
-
Earnie Boyd -
Ivan Sergio Borgonovo -
Larry Garfield -
M. Fioretti -
Michael MacKenna -
Tibor Liktor