Performance Problem - support

5 Jan 2012


      Posted by toolsmythe http://drupal.org/user/1746988  on January 4, 2012 at
8:18pm
I'm dealing with an odd performance issue on a production Drupal 7.7 site.
I have a machine that is configured thus:
4gig
Quad proc
900GB disk
OS: Debian 6
For apache we are using version
ii apache2 2.2.16-6+squeeze1 Apache HTTP Server metapackage
ii apache2-mpm-prefork 2.2.16-6+squeeze1 Apache HTTP Server - traditional
non-threaded model
For PHP we are running version
ii php5 5.3.3-7+squeeze3 server-side, HTML-embedded scripting language
(metapackage)
ii libapache2-mod-php5 5.3.3-7+squeeze3 server-side, HTML-embedded scripting
language (Apache 2 module)
ii mysql-client-5.1 5.1.49-3 MySQL database client binaries
Database is on a seperate server.
The Problem
We are getting a pretty consistant 5+ second delay before any page on the
site renders. It renders quickly once it starts to render, but there is a
delay of 5 or more seconds before that happens.
Oddly enough, we have a copy of the site running under cPanel that has no
delay whatsoever. That box is configured thus:
Apache 2.2
PHP 5.2.17
Mod SuPHP
CentOS release 5.7 (Final)
Not sure if this is worthy of note, but the production server has Suhosin
(PHP hardening) and the copy does not.
We made a second copy on the prod server and it is experiencing the same
delay.
All the database instances are practically empty.
What we've tried so far
We've tried optimizing the tables in MySQL.
We've tried applying various optimizations to Apache and MySQL recommended
on Drupal pages (for example: http://drupal.org/node/215516 and
http://drupal.org/node/2601).
No joy.
We did an ab benchmark on both servers. as follows:
ab -n 500 -c 50
The test server showed this:
Benchmarking .com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Finished 500 requests
Server Software: Apache
Server Hostname: .com
Server Port: 80
Document Path: /clients///
Document Length: 18543 bytes
Concurrency Level: 50
Time taken for tests: 50.229747 seconds
Complete requests: 500
Failed requests: 0
Write errors: 0
Total transferred: 9663634 bytes
HTML transferred: 9327129 bytes
Requests per second: 9.95 [#/sec] (mean)
Time per request: 5022.975 [ms] (mean)
Time per request: 100.459 [ms] (mean, across all concurrent requests)
Transfer rate: 187.88 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.0 0 9
Processing: 703 4901 5312.1 3312 38328
Waiting: 657 4653 4788.4 3134 33105
Total: 703 4902 5312.4 3312 38330
Percentage of the requests served within a certain time (ms)
50% 3312
66% 4721
75% 5718
80% 6374
90% 10416
95% 14444
98% 24669
99% 33351
100% 38330 (longest request)
Meanwhile, here's the same test for our prod server:
Server Software: Apache/2.2.16
Server Hostname: .com
Server Port: 80
Document Path: /clients//
Document Length: 18760 bytes
Concurrency Level: 50
Time taken for tests: 314.671 seconds
Complete requests: 500
Failed requests: 0
Write errors: 0
Total transferred: 9757000 bytes
HTML transferred: 9380000 bytes
Requests per second: 1.59 [#/sec] (mean)
Time per request: 31467.146 [ms] (mean)
Time per request: 629.343 [ms] (mean, across all concurrent requests)
Transfer rate: 30.28 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.3 0 1
Processing: 11813 29570 12525.7 27341 126615
Waiting: 11745 29509 12525.6 27281 126535
Total: 11813 29570 12525.7 27341 126615
Percentage of the requests served within a certain time (ms)
50% 27341
66% 31105
75% 33440
80% 35335
90% 40918
95% 47849
98% 66717
99% 84633
100% 126615 (longest request)
We tried turnning on Long Running Queries in MySQL; it logged nothing.
top showed nothing of interest.
We're stumped over here (though to be quite honest, stumping us is no great
feat <grin>, at least when it comes to Drupal). Any thoughts would be much
appreciated.