<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span lang=EN>Posted by <a href="http://drupal.org/user/1746988" title="View user profile.">toolsmythe</a> on <em><span style='font-family:"Calibri","sans-serif"'>January 4, 2012 at 8:18pm</span></em><o:p></o:p></span></p><p><span lang=EN>I'm dealing with an odd performance issue on a production Drupal 7.7 site.<o:p></o:p></span></p><p><span lang=EN>I have a machine that is configured thus:<o:p></o:p></span></p><p><span lang=EN>4gig<br>Quad proc<br>900GB disk<o:p></o:p></span></p><p><span lang=EN>OS: Debian 6 <o:p></o:p></span></p><p><span lang=EN>For apache we are using version<br>ii apache2 2.2.16-6+squeeze1 Apache HTTP Server metapackage<br>ii apache2-mpm-prefork 2.2.16-6+squeeze1 Apache HTTP Server - traditional non-threaded model<o:p></o:p></span></p><p><span lang=EN>For PHP we are running version<br>ii php5 5.3.3-7+squeeze3 server-side, HTML-embedded scripting language (metapackage)<br>ii libapache2-mod-php5 5.3.3-7+squeeze3 server-side, HTML-embedded scripting language (Apache 2 module)<o:p></o:p></span></p><p><span lang=EN>ii mysql-client-5.1 5.1.49-3 MySQL database client binaries<o:p></o:p></span></p><p><span lang=EN>Database is on a seperate server.<o:p></o:p></span></p><p><b><u><span lang=EN>The Problem</span></u></b><span lang=EN><br>We are getting a pretty consistant 5+ second delay before any page on the site renders. It renders quickly once it starts to render, but there is a delay of 5 or more seconds before that happens.<o:p></o:p></span></p><p><span lang=EN>Oddly enough, we have a copy of the site running under cPanel that has no delay whatsoever. That box is configured thus:<o:p></o:p></span></p><p><span lang=EN>Apache 2.2<br>PHP 5.2.17<br>Mod SuPHP<br>CentOS release 5.7 (Final)<o:p></o:p></span></p><p><span lang=EN>Not sure if this is worthy of note, but the production server has Suhosin (PHP hardening) and the copy does not.<o:p></o:p></span></p><p><span lang=EN>We made a second copy on the prod server and it is experiencing the same delay.<o:p></o:p></span></p><p><span lang=EN>All the database instances are practically empty.<o:p></o:p></span></p><p><b><u><span lang=EN>What we've tried so far</span></u></b><span lang=EN><br>We've tried optimizing the tables in MySQL.<o:p></o:p></span></p><p><span lang=EN>We've tried applying various optimizations to Apache and MySQL recommended on Drupal pages (for example: <a href="http://drupal.org/node/215516" title="http://drupal.org/node/215516">http://drupal.org/node/215516</a> and <a href="http://drupal.org/node/2601" title="http://drupal.org/node/2601">http://drupal.org/node/2601</a>).<o:p></o:p></span></p><p><span lang=EN>No joy.<o:p></o:p></span></p><p><span lang=EN>We did an ab benchmark on both servers. as follows:<o:p></o:p></span></p><p><span lang=EN>ab -n 500 -c 50 <o:p></o:p></span></p><p><span lang=EN>The test server showed this:<o:p></o:p></span></p><p><span lang=EN>Benchmarking .com (be patient)<br>Completed 100 requests<br>Completed 200 requests<br>Completed 300 requests<br>Completed 400 requests<br>Finished 500 requests<o:p></o:p></span></p><p><span lang=EN>Server Software: Apache<br>Server Hostname: .com<br>Server Port: 80<o:p></o:p></span></p><p><span lang=EN>Document Path: /clients///<br>Document Length: 18543 bytes<o:p></o:p></span></p><p><span lang=EN>Concurrency Level: 50<br>Time taken for tests: 50.229747 seconds<br>Complete requests: 500<br>Failed requests: 0<br>Write errors: 0<br>Total transferred: 9663634 bytes<br>HTML transferred: 9327129 bytes<br>Requests per second: 9.95 [#/sec] (mean)<br>Time per request: 5022.975 [ms] (mean)<br>Time per request: 100.459 [ms] (mean, across all concurrent requests)<br>Transfer rate: 187.88 [Kbytes/sec] received<o:p></o:p></span></p><p><span lang=EN>Connection Times (ms)<br>min mean[+/-sd] median max<br>Connect: 0 0 1.0 0 9<br>Processing: 703 4901 5312.1 3312 38328<br>Waiting: 657 4653 4788.4 3134 33105<br>Total: 703 4902 5312.4 3312 38330<o:p></o:p></span></p><p><span lang=EN>Percentage of the requests served within a certain time (ms)<br>50% 3312<br>66% 4721<br>75% 5718<br>80% 6374<br>90% 10416<br>95% 14444<br>98% 24669<br>99% 33351<br>100% 38330 (longest request)<o:p></o:p></span></p><p><span lang=EN>Meanwhile, here's the same test for our prod server:<o:p></o:p></span></p><p><span lang=EN>Server Software: Apache/2.2.16<br>Server Hostname: .com<br>Server Port: 80<br>Document Path: /clients//<br>Document Length: 18760 bytes<br>Concurrency Level: 50<br>Time taken for tests: 314.671 seconds<br>Complete requests: 500<br>Failed requests: 0<br>Write errors: 0<br>Total transferred: 9757000 bytes<br>HTML transferred: 9380000 bytes<br>Requests per second: 1.59 [#/sec] (mean)<br>Time per request: 31467.146 [ms] (mean)<br>Time per request: 629.343 [ms] (mean, across all concurrent requests)<br>Transfer rate: 30.28 [Kbytes/sec] received<br>Connection Times (ms)<br>min mean[+/-sd] median max<br>Connect: 0 0 0.3 0 1<br>Processing: 11813 29570 12525.7 27341 126615<br>Waiting: 11745 29509 12525.6 27281 126535<br>Total: 11813 29570 12525.7 27341 126615<br>Percentage of the requests served within a certain time (ms)<br>50% 27341<br>66% 31105<br>75% 33440<br>80% 35335<br>90% 40918<br>95% 47849<br>98% 66717<br>99% 84633<br>100% 126615 (longest request)<o:p></o:p></span></p><p><span lang=EN>We tried turnning on Long Running Queries in MySQL; it logged nothing.<o:p></o:p></span></p><p><span lang=EN>top showed nothing of interest.<o:p></o:p></span></p><p><span lang=EN>We're stumped over here (though to be quite honest, stumping us is no great feat <grin>, at least when it comes to Drupal). Any thoughts would be much appreciated.<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>