Would it make sense to change the "Discard watchdog/access log entries older than" settings to use the number of SQL table rows, rather than the number of days? The problem? Again, at drupal.org, I had these set to "3 days" which I felt was short enough for the tables not to grow unwieldy. It wasn't until this week that I found out -- through extensive profiling -- that 3 days was too long. The point? Unless you are a Drupal developer, you won't profile your site. This means, most users have no clue about how much data is kept track of, or why their site becomes slower and slower. The question? How can we make it so that this doesn't happen? To me, having an option "Number of records to keep" rather than "Discard records older than" makes sense, because (i) it gives me control over the maximum number of table rows (ii) without having to worry about traffic bursts or a growing number of visitors (the cost is more likely to be constant over time). It gives me a clue about the performance/resource ramifications of certain features. Having a similar setting for the sessions- and history-table would make sense too. Would this degrade or improve usability? Suggestions? -- Dries Buytaert :: http://www.buytaert.net/
On Wed, 11 May 2005, Dries Buytaert wrote:
Would it make sense to change the "Discard watchdog/access log entries older than" settings to use the number of SQL table rows, rather than the number of days?
The problem? Again, at drupal.org, I had these set to "3 days" which I felt was short enough for the tables not to grow unwieldy. It wasn't until this week that I found out -- through extensive profiling -- that 3 days was too long.
The point? Unless you are a Drupal developer, you won't profile your site. This means, most users have no clue about how much data is kept track of, or why their site becomes slower and slower.
The question? How can we make it so that this doesn't happen? To me, having an option "Number of records to keep" rather than "Discard records older than" makes sense, because (i) it gives me control over the maximum number of table rows (ii) without having to worry about traffic bursts or a growing number of visitors (the cost is more likely to be constant over time). It gives me a clue about the performance/resource ramifications of certain features. Having a similar setting for the sessions- and history-table would make sense too.
Would this degrade or improve usability? Suggestions?
I am not too fond of the idea. Admins don't want statistics over number of sql data rows but over time. What I'd rather see is more than one watchdog / accesslog table. There would be a small "current" table. When it has reached maybe 500 entries, those get moved to a big backup table and the current table gets pruned. This could be done by cron. This would require locking, I think. Which advantages does that have? - We work with a small table on normal page views. Inserts are fast. - We still have all the data we want on admin pages. Cheers, Gerhard
On 10 May 2005, at 3:41 PM, Gerhard Killesreiter wrote:
On Wed, 11 May 2005, Dries Buytaert wrote:
Would it make sense to change the "Discard watchdog/access log entries older than" settings to use the number of SQL table rows, rather than the number of days?
I am not too fond of the idea. Admins don't want statistics over number of sql data rows but over time.
What I'd rather see is more than one watchdog / accesslog table.
There would be a small "current" table. When it has reached maybe 500 entries, those get moved to a big backup table and the current table gets pruned. This could be done by cron. This would require locking, I think.
Which advantages does that have?
- We work with a small table on normal page views. Inserts are fast. - We still have all the data we want on admin pages.
Cheers, Gerhard
We could use use 'Merge tables' in MySQL (and have the same functionality with views in PostgreSQL)... keeping logs is exactly the kind of scenario these were designed for. The 'con' is that this is pretty database specific... Djun
In a previous incarnation of my Drupal web site, I needed access logs for certain range of time in order to generate graphs and charts. So my argument is there should be an option to use a time-based limit. In what other situations are the log tables used? Nic On May 10, 2005, at 5:06 PM, Dries Buytaert wrote:
Would it make sense to change the "Discard watchdog/access log entries older than" settings to use the number of SQL table rows, rather than the number of days?
The problem? Again, at drupal.org, I had these set to "3 days" which I felt was short enough for the tables not to grow unwieldy. It wasn't until this week that I found out -- through extensive profiling -- that 3 days was too long.
The point? Unless you are a Drupal developer, you won't profile your site. This means, most users have no clue about how much data is kept track of, or why their site becomes slower and slower.
The question? How can we make it so that this doesn't happen? To me, having an option "Number of records to keep" rather than "Discard records older than" makes sense, because (i) it gives me control over the maximum number of table rows (ii) without having to worry about traffic bursts or a growing number of visitors (the cost is more likely to be constant over time). It gives me a clue about the performance/resource ramifications of certain features. Having a similar setting for the sessions- and history-table would make sense too.
Would this degrade or improve usability? Suggestions?
-- Dries Buytaert :: http://www.buytaert.net/
In a previous incarnation of my Drupal web site, I needed access logs for certain range of time in order to generate graphs and charts. So my argument is there should be an option to use a time-based limit.
Plus, if you look at watchdog logs daily or something, you would not like to loose the possibilty to have an overview of the stuff happening on your site, if it had a traffic peak (which would cause rows to get removed inbetween the timeframe you are able to look at the logs). Goba
i think your proposal is reasonable. would we benefit from a well placed throttle here? if the site is under heavy load, stop trying to insert history records. -moshe On May 10, 2005, at 6:06 PM, Dries Buytaert wrote:
Would it make sense to change the "Discard watchdog/access log entries older than" settings to use the number of SQL table rows, rather than the number of days?
The problem? Again, at drupal.org, I had these set to "3 days" which I felt was short enough for the tables not to grow unwieldy. It wasn't until this week that I found out -- through extensive profiling -- that 3 days was too long.
The point? Unless you are a Drupal developer, you won't profile your site. This means, most users have no clue about how much data is kept track of, or why their site becomes slower and slower.
The question? How can we make it so that this doesn't happen? To me, having an option "Number of records to keep" rather than "Discard records older than" makes sense, because (i) it gives me control over the maximum number of table rows (ii) without having to worry about traffic bursts or a growing number of visitors (the cost is more likely to be constant over time). It gives me a clue about the performance/resource ramifications of certain features. Having a similar setting for the sessions- and history-table would make sense too.
Would this degrade or improve usability? Suggestions?
-- Dries Buytaert :: http://www.buytaert.net/
What I'd rather see is more than one watchdog / accesslog table.
There would be a small "current" table. When it has reached maybe 500 entries, those get moved to a big backup table and the current table gets pruned. This could be done by cron. This would require locking, I think.
Which advantages does that have?
- We work with a small table on normal page views. Inserts are fast. - We still have all the data we want on admin pages.
I would rather not do this. This is table partitioning, which complicates later access to the data. Some databases support table partitioning because they cannot deal with large data under certain workloads (e.g. Decision Support Systems), and it is a nightmare writing queries in a partitioned data environment.
In a previous incarnation of my Drupal web site, I needed access logs for certain range of time in order to generate graphs and charts. So my argument is there should be an option to use a time-based limit.
I agree that whatever we do, we should keep a time-based options anyway.
On 10-May-05, at 5:47 PM, K B wrote:
What I'd rather see is more than one watchdog / accesslog table.
There would be a small "current" table. When it has reached maybe 500 entries, those get moved to a big backup table and the current table gets pruned. This could be done by cron. This would require locking, I think.
Which advantages does that have?
- We work with a small table on normal page views. Inserts are fast. - We still have all the data we want on admin pages.
Can we/should we dump to disk, like "regular" log files? The Drupal statistics really aren't very useful right now, other than popular files. Either that or I'm too dense to understand the tables vs. log file parsers like (e.g.) AWStats. -- Boris Mann http://www.bmannconsulting.com
On Tue, 10 May 2005, Boris Mann wrote:
On 10-May-05, at 5:47 PM, K B wrote:
What I'd rather see is more than one watchdog / accesslog table.
There would be a small "current" table. When it has reached maybe 500 entries, those get moved to a big backup table and the current table gets pruned. This could be done by cron. This would require locking, I think.
Which advantages does that have?
- We work with a small table on normal page views. Inserts are fast. - We still have all the data we want on admin pages.
Can we/should we dump to disk, like "regular" log files?
Sure we can, but why should we, ie what would we gain?
The Drupal statistics really aren't very useful right now, other than popular files. Either that or I'm too dense to understand the tables vs. log file parsers like (e.g.) AWStats.
The Drupal stats are very usefull if you are interested in popular pages or what individual users are doing. The last thing is something awstats and Co. cannot track (unless Apache would be logging session info). I use this feature if a user is having some difficulties on the site. Cheers, Gerhard
I suggest we enhance useability by emulating two *nix tools: swatch and logrotate. Logrotate would use a separate table, or even database, to store he rotted log entries. For normal users logrotate will be set to 0, meaning the functionality is not used. Advanced admins can set this to any amount/size or time. swatch will use reg.expressions or other settings to pick out only certain log entries. These can be 1) saved for a longer time in the watchdog table and/or 2) emailed to the administrator. That way the logs can be kept small, while ensuring that admins still see or get crucial errors. Ber Op woensdag 11 mei 2005 00:06, schreef Dries Buytaert:
Would it make sense to change the "Discard watchdog/access log entries older than" settings to use the number of SQL table rows, rather than the number of days?
The problem? Again, at drupal.org, I had these set to "3 days" which I felt was short enough for the tables not to grow unwieldy. It wasn't until this week that I found out -- through extensive profiling -- that 3 days was too long.
The point? Unless you are a Drupal developer, you won't profile your site. This means, most users have no clue about how much data is kept track of, or why their site becomes slower and slower.
The question? How can we make it so that this doesn't happen? To me, having an option "Number of records to keep" rather than "Discard records older than" makes sense, because (i) it gives me control over the maximum number of table rows (ii) without having to worry about traffic bursts or a growing number of visitors (the cost is more likely to be constant over time). It gives me a clue about the performance/resource ramifications of certain features. Having a similar setting for the sessions- and history-table would make sense too.
Would this degrade or improve usability? Suggestions?
-- Dries Buytaert :: http://www.buytaert.net/ Regards, Bèr -- [ Bèr Kessels | Drupal services www.webschuur.com ]
Logrotate would use a separate table, or even database, to store he rotted log entries. For normal users logrotate will be set to 0, meaning the functionality is not used. Advanced admins can set this to any amount/size or time.
Dont force a separater database, because many people on shared hosting have only one db. I agree the logs are not very useful compared to raw logs or tools that use raw logs (e.g. Webalizer, AWStats, ...etc.) However, there is some usefulness in them nonetheless. Some people dont even have raw log capability. So leave the functionality as it is for the sake of those.
participants (9)
-
Boris Mann -
Bèr Kessels -
Dries Buytaert -
Gabor Hojtsy -
Gerhard Killesreiter -
K B -
Moshe Weitzman -
Nicholas Ivy -
puregin