[development] 2k qps
remorse at partners.org
Wed Oct 11 20:08:52 UTC 2006
At one point I ran into an issue where my server kept hanging. The
load average didn't go up quite so quickly as yours, but it would
climb up. The problem turned out to be that the version of MySQL I
was using had an issue where queries that involved inner joins with
views (especially when those views were made of some 20+ inner joins
themselves) would never return. Do you have any odd modules
enabled? Any you wrote yourself? I solved it my upgrading MySQL
(well, my first solution was to change the database structure to
realize a view every hour using cron, but when I upgraded and the bug
went away, I went back to the view).
On Oct 11, 2006, at 9:02 AM, Tamir Khason wrote:
> We have sysstat installed (SAR tool)
> And what we see nothing in it besides some high load average until
> everything is stucked. It doesn't succeed to log the real situation
> when it comes to it, since server doesn't have any resources...
> No iowait noticed at all. Everything is between php/mysql. Server has
> good dsics, raid1 and iowait is getting to 10% maximum in bad
> times..also I run iostat from time to time to see what goes on. I also
> think change io scheduling but imo the current problem has nothing
> with io. Dmesg also doesn't point to anything like that.
> We are trying to dump the apache/mysql status once there is a problem
> (through mytop/server-status and other things) the problem is that
> it's too late in most cases and nothing "cooperates" with us.
>> ---------- Forwarded message ----------
>> From: Khalid B <kb at 2bits.com>
>> Date: Wed, 11 Oct 2006 07:05:58 -0400
>> Subject: Re: [development] 2k qps
>> To: development at drupal.org
>> I have another client site that shoots up to 3,000 QPS, but I don't
>> the statistics of page access for it. This one has 4 Xeons 3.2GHZ
>> and 4GB.
>> Write a simple script to run vmstat every 15 seconds or so, log the
>> output to a file. Keep it running until the "hang" happens, and then
>> see what it tells you about the system. Are you out of CPU? Do
>> you have too many blocked processes or runable processes?
>> Do you have a lot of wait on i/o? Are you swapping to the point of
>> Only after you do this analysis you should decide whether to
>> upgrade the RAM or just tune things.
>> Regarding decreasing the number of Apache processes, yes, that
>> means you can only serve so many users, but you prevent the chaos
>> caused by swapping. Think of it as a popular restaurant. If they let
>> in more people than they have tables/seats, would the customer be
>> happy? Would the staff cope with it?
More information about the development