Posted by toolsmythe <http://drupal.org/user/1746988> on January 4, 2012 at 8:18pm I'm dealing with an odd performance issue on a production Drupal 7.7 site. I have a machine that is configured thus: 4gig Quad proc 900GB disk OS: Debian 6 For apache we are using version ii apache2 2.2.16-6+squeeze1 Apache HTTP Server metapackage ii apache2-mpm-prefork 2.2.16-6+squeeze1 Apache HTTP Server - traditional non-threaded model For PHP we are running version ii php5 5.3.3-7+squeeze3 server-side, HTML-embedded scripting language (metapackage) ii libapache2-mod-php5 5.3.3-7+squeeze3 server-side, HTML-embedded scripting language (Apache 2 module) ii mysql-client-5.1 5.1.49-3 MySQL database client binaries Database is on a seperate server. The Problem We are getting a pretty consistant 5+ second delay before any page on the site renders. It renders quickly once it starts to render, but there is a delay of 5 or more seconds before that happens. Oddly enough, we have a copy of the site running under cPanel that has no delay whatsoever. That box is configured thus: Apache 2.2 PHP 5.2.17 Mod SuPHP CentOS release 5.7 (Final) Not sure if this is worthy of note, but the production server has Suhosin (PHP hardening) and the copy does not. We made a second copy on the prod server and it is experiencing the same delay. All the database instances are practically empty. What we've tried so far We've tried optimizing the tables in MySQL. We've tried applying various optimizations to Apache and MySQL recommended on Drupal pages (for example: http://drupal.org/node/215516 and http://drupal.org/node/2601). No joy. We did an ab benchmark on both servers. as follows: ab -n 500 -c 50 The test server showed this: Benchmarking .com (be patient) Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Finished 500 requests Server Software: Apache Server Hostname: .com Server Port: 80 Document Path: /clients/// Document Length: 18543 bytes Concurrency Level: 50 Time taken for tests: 50.229747 seconds Complete requests: 500 Failed requests: 0 Write errors: 0 Total transferred: 9663634 bytes HTML transferred: 9327129 bytes Requests per second: 9.95 [#/sec] (mean) Time per request: 5022.975 [ms] (mean) Time per request: 100.459 [ms] (mean, across all concurrent requests) Transfer rate: 187.88 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 1.0 0 9 Processing: 703 4901 5312.1 3312 38328 Waiting: 657 4653 4788.4 3134 33105 Total: 703 4902 5312.4 3312 38330 Percentage of the requests served within a certain time (ms) 50% 3312 66% 4721 75% 5718 80% 6374 90% 10416 95% 14444 98% 24669 99% 33351 100% 38330 (longest request) Meanwhile, here's the same test for our prod server: Server Software: Apache/2.2.16 Server Hostname: .com Server Port: 80 Document Path: /clients// Document Length: 18760 bytes Concurrency Level: 50 Time taken for tests: 314.671 seconds Complete requests: 500 Failed requests: 0 Write errors: 0 Total transferred: 9757000 bytes HTML transferred: 9380000 bytes Requests per second: 1.59 [#/sec] (mean) Time per request: 31467.146 [ms] (mean) Time per request: 629.343 [ms] (mean, across all concurrent requests) Transfer rate: 30.28 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.3 0 1 Processing: 11813 29570 12525.7 27341 126615 Waiting: 11745 29509 12525.6 27281 126535 Total: 11813 29570 12525.7 27341 126615 Percentage of the requests served within a certain time (ms) 50% 27341 66% 31105 75% 33440 80% 35335 90% 40918 95% 47849 98% 66717 99% 84633 100% 126615 (longest request) We tried turnning on Long Running Queries in MySQL; it logged nothing. top showed nothing of interest. We're stumped over here (though to be quite honest, stumping us is no great feat <grin>, at least when it comes to Drupal). Any thoughts would be much appreciated.
Since you are using apache on both the servers, can you list the apache modules enabled on both these servers? (a command like apache2 -l can be helpful) Also, is there any kind of web logging software (like AWStats) installed and enabled on your prod server? On Thu, Jan 5, 2012 at 10:49 PM, John Pagakis <drupal@pagakis.com> wrote:
Posted by toolsmythe <http://drupal.org/user/1746988> on *January 4, 2012 at 8:18pm*****
I'm dealing with an odd performance issue on a production Drupal 7.7 site. ****
I have a machine that is configured thus:****
4gig Quad proc 900GB disk****
OS: Debian 6 ****
For apache we are using version ii apache2 2.2.16-6+squeeze1 Apache HTTP Server metapackage ii apache2-mpm-prefork 2.2.16-6+squeeze1 Apache HTTP Server - traditional non-threaded model****
For PHP we are running version ii php5 5.3.3-7+squeeze3 server-side, HTML-embedded scripting language (metapackage) ii libapache2-mod-php5 5.3.3-7+squeeze3 server-side, HTML-embedded scripting language (Apache 2 module)****
ii mysql-client-5.1 5.1.49-3 MySQL database client binaries****
Database is on a seperate server.****
*The Problem* We are getting a pretty consistant 5+ second delay before any page on the site renders. It renders quickly once it starts to render, but there is a delay of 5 or more seconds before that happens.****
Oddly enough, we have a copy of the site running under cPanel that has no delay whatsoever. That box is configured thus:****
Apache 2.2 PHP 5.2.17 Mod SuPHP CentOS release 5.7 (Final)****
Not sure if this is worthy of note, but the production server has Suhosin (PHP hardening) and the copy does not.****
We made a second copy on the prod server and it is experiencing the same delay.****
All the database instances are practically empty.****
*What we've tried so far* We've tried optimizing the tables in MySQL.****
We've tried applying various optimizations to Apache and MySQL recommended on Drupal pages (for example: http://drupal.org/node/215516 and http://drupal.org/node/2601).****
No joy.****
We did an ab benchmark on both servers. as follows:****
ab -n 500 -c 50 ****
The test server showed this:****
Benchmarking .com (be patient) Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Finished 500 requests****
Server Software: Apache Server Hostname: .com Server Port: 80****
Document Path: /clients/// Document Length: 18543 bytes****
Concurrency Level: 50 Time taken for tests: 50.229747 seconds Complete requests: 500 Failed requests: 0 Write errors: 0 Total transferred: 9663634 bytes HTML transferred: 9327129 bytes Requests per second: 9.95 [#/sec] (mean) Time per request: 5022.975 [ms] (mean) Time per request: 100.459 [ms] (mean, across all concurrent requests) Transfer rate: 187.88 [Kbytes/sec] received****
Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 1.0 0 9 Processing: 703 4901 5312.1 3312 38328 Waiting: 657 4653 4788.4 3134 33105 Total: 703 4902 5312.4 3312 38330****
Percentage of the requests served within a certain time (ms) 50% 3312 66% 4721 75% 5718 80% 6374 90% 10416 95% 14444 98% 24669 99% 33351 100% 38330 (longest request)****
Meanwhile, here's the same test for our prod server:****
Server Software: Apache/2.2.16 Server Hostname: .com Server Port: 80 Document Path: /clients// Document Length: 18760 bytes Concurrency Level: 50 Time taken for tests: 314.671 seconds Complete requests: 500 Failed requests: 0 Write errors: 0 Total transferred: 9757000 bytes HTML transferred: 9380000 bytes Requests per second: 1.59 [#/sec] (mean) Time per request: 31467.146 [ms] (mean) Time per request: 629.343 [ms] (mean, across all concurrent requests) Transfer rate: 30.28 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.3 0 1 Processing: 11813 29570 12525.7 27341 126615 Waiting: 11745 29509 12525.6 27281 126535 Total: 11813 29570 12525.7 27341 126615 Percentage of the requests served within a certain time (ms) 50% 27341 66% 31105 75% 33440 80% 35335 90% 40918 95% 47849 98% 66717 99% 84633 100% 126615 (longest request)****
We tried turnning on Long Running Queries in MySQL; it logged nothing.****
top showed nothing of interest.****
We're stumped over here (though to be quite honest, stumping us is no great feat <grin>, at least when it comes to Drupal). Any thoughts would be much appreciated.****
** **
** **
-- [ Drupal support list | http://lists.drupal.org/ ]
-- Cheers, Mukesh Agarwal ________________________________ Innoraft Solutions <http://www.innoraft.com> || +91 8017220799
What is the speed of the processors (both machines)? Have you checked with Devel's long-running queries? Nancy Injustice anywhere is a threat to justice everywhere. -- Dr. Martin L. King, Jr.
________________________________ From: John Pagakis
I'm dealing with an odd performance issue on a production Drupal 7.7 site. I have a machine that is configured thus: 4gig Quad proc 900GB disk OS: Debian 6 Database is on a seperate server. The Problem We are getting a pretty consistant 5+ second delay before any page on the site renders. It renders quickly once it starts to render, but there is a delay of 5 or more seconds before that happens.
Try running and analysing a tcpdump on the mysql port. Wireshark could provide some answers, or you may even get a visual clue just running tcpdump in a terminal. What happens if you run the db locally? On Thu, 2012-01-05 at 10:19 -0700, John Pagakis wrote:
Posted by toolsmythe on January 4, 2012 at 8:18pm
I'm dealing with an odd performance issue on a production Drupal 7.7 site.
I have a machine that is configured thus:
4gig Quad proc 900GB disk
OS: Debian 6
For apache we are using version ii apache2 2.2.16-6+squeeze1 Apache HTTP Server metapackage ii apache2-mpm-prefork 2.2.16-6+squeeze1 Apache HTTP Server - traditional non-threaded model
For PHP we are running version ii php5 5.3.3-7+squeeze3 server-side, HTML-embedded scripting language (metapackage) ii libapache2-mod-php5 5.3.3-7+squeeze3 server-side, HTML-embedded scripting language (Apache 2 module)
ii mysql-client-5.1 5.1.49-3 MySQL database client binaries
Database is on a seperate server.
The Problem We are getting a pretty consistant 5+ second delay before any page on the site renders. It renders quickly once it starts to render, but there is a delay of 5 or more seconds before that happens.
participants (5)
-
Jeff Brown -
John Pagakis -
Ms. Nancy Wichmann -
Mukesh Agarwal -
Ted