Re: [support] Improving performance
Hi Jamie, Phil, thanks for the info Jamie, much appreciated. As to sharing Phil, happy to do so. A little bit about my setup. I am retired and run a Centos 6.5, Apache, MySQL webserver in my lounge room. The server is home built - Gigabyte mobo, Intel processor, Kingston RAM, etc. I have a second similar server as a backup and development machine. I get 1.5Mb upload speed which is adequate to support half a dozen low traffic Drupal based sites. When I say low traffic, we are looking at 30-50 hits a day for each site The sites tend to be more personal interest sites rather than commercial ie. personal Blogs etc. The hits tend to be clustered based on chronology and demographic. Typically might get 20-30 hits over a couple of hours on a site and then nothing for 20 hours. During the quiet period had problems with Drupal cache's timing out, so first hit was then incredibly slow. I installed memcache via Drupal.org and memcache status using their instructions. This seemed to help, but still getting very slow first responses. I then did following: [root@webserver ~] yum install php-pear php-devel httpd-devel pcre-devel gcc make [root@webserver ~] pecl install apc [root@webserver ~] service httpd restart This being suggested by a google search. The pecl install seems to have included an initial setup as many questions were asked. As I have massive amounts of RAM available, I increased all resources in /etc/php.d/apc.ini apc.shm_size = 128M apc.num_files = 4096 apc.user-entries = 4096 apc.ttl - 21600 apc.user_ttl - 21600 apc.gc_ttl = 14400 apc.max_file_size = 5M. Then did a [root@webserver ~] service httpd restart. There is a really neat tool to see what's happening. Copy apc.php from docs directory to site root, and the run example.com/apc.php. Now I need to wait a few days to see what happens. Alison PS. Latest site just built is at http://www.trans-aspie.com At 01:57 AM 04-01-14, you wrote:
Hi Alison, Please share what you find. I have been intending to look into this myself. I'm concerned about the initial page load time on my site. It seems pretty slow. Regards, Phil
From: Jamie Holly <hovercrafter@earthlink.net> To: support@drupal.org, Date: 01/03/2014 09:31 AM Subject: Re: [support] Improving performance Sent by: support-bounces@drupal.org
The high performance group on drupal.org is really good:
https://groups.drupal.org/high-performance
The most common recipe is APC for opcode caching, memcache for object caching and Drupal's database to cache the form table.
APC also offers object/user caching, but you do have to spend sometime getting it tuned right so the memory does fragment. In one high performance site I manage I use the same recipe as above, but I use APC to handle page caching.
Of course all this varies greatly by site. Some sites are mostly anonymous users, meaning page caching is the key, while others have a heavy logged-in user base, so object caching becomes the big factor.
Also depending on how the site is built, there's other cache modules out there that really help, like views cache, authcache and panels cache.
Jamie Holly http://hollyit.net
On 1/3/2014 5:08 AM, Alison wrote:
Hi Guys and Girls,
can anyone recommend a good general discussion site or place where I can read about Drupal caching. Currently running Memcache and APC - the PHP accelerator, but not sure whether there would be anything to gain by implementing Drupals caching as well. Wouldn't mind finding some way of measuring the effects of the various performance enhancements. Are there any downsides to Memcache and APC?
Hugs
Alison
-- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ]
Your problem isn't really caching, but regeneration. If you have items on the page taking a few seconds to generate, from queries that aren't optimized, huge processing, etc., then whenever that cache is invalidated, either through expiration, content change or a cache clear call, then you're going to get hit with the same problem. Put another way, the hangup isn't getting the data from cache, but rather Drupal generating the data to put into the cache. The best thing to do is run a development setup and enable the devel module and enable query output in there. Then clear your cache and look on your slow pages to see what queries are holding things up. Another very common issue is negligence in tuning the actual server. What happens here is that you are throwing performance enhancements like caching into the mix and that is nothing more than a band-aid covering a much larger problem. A key place to start is always with the database. Installing MySQL is very easy, but the configuration it ships with is in no way meant for production (simply because configuration depends upon usage and usage is different between applications and even between Drupal sites). You need to monitor some key MySQL metrics to make sure they are within appropriate ranges. A good tool for the is tuning-primer: https://launchpad.net/mysql-tuning-primer Run that after the server has been running a day or two and it will check existing status variables and make recommendations on what to change. Jamie Holly http://hollyit.net On 1/3/2014 8:21 PM, Alison wrote:
Hi Jamie, Phil,
thanks for the info Jamie, much appreciated. As to sharing Phil, happy to do so. A little bit about my setup.
I am retired and run a Centos 6.5, Apache, MySQL webserver in my lounge room. The server is home built - Gigabyte mobo, Intel processor, Kingston RAM, etc. I have a second similar server as a backup and development machine. I get 1.5Mb upload speed which is adequate to support half a dozen low traffic Drupal based sites. When I say low traffic, we are looking at 30-50 hits a day for each site
The sites tend to be more personal interest sites rather than commercial ie. personal Blogs etc. The hits tend to be clustered based on chronology and demographic. Typically might get 20-30 hits over a couple of hours on a site and then nothing for 20 hours. During the quiet period had problems with Drupal cache's timing out, so first hit was then incredibly slow.
I installed memcache via Drupal.org and memcache status using their instructions. This seemed to help, but still getting very slow first responses. I then did following:
[root@webserver ~] yum install php-pear php-devel httpd-devel pcre-devel gcc make [root@webserver ~] pecl install apc [root@webserver ~] service httpd restart
This being suggested by a google search. The pecl install seems to have included an initial setup as many questions were asked.
As I have massive amounts of RAM available, I increased all resources in /etc/php.d/apc.ini apc.shm_size = 128M apc.num_files = 4096 apc.user-entries = 4096 apc.ttl - 21600 apc.user_ttl - 21600 apc.gc_ttl = 14400 apc.max_file_size = 5M. Then did a [root@webserver ~] service httpd restart.
There is a really neat tool to see what's happening. Copy apc.php from docs directory to site root, and the run example.com/apc.php.
Now I need to wait a few days to see what happens.
Alison PS. Latest site just built is at http://www.trans-aspie.com
At 01:57 AM 04-01-14, you wrote:
Hi Alison, Please share what you find. I have been intending to look into this myself. I'm concerned about the initial page load time on my site. It seems pretty slow. Regards, Phil
From: Jamie Holly <hovercrafter@earthlink.net> To: support@drupal.org, Date: 01/03/2014 09:31 AM Subject: Re: [support] Improving performance Sent by: support-bounces@drupal.org
The high performance group on drupal.org is really good:
https://groups.drupal.org/high-performance
The most common recipe is APC for opcode caching, memcache for object caching and Drupal's database to cache the form table.
APC also offers object/user caching, but you do have to spend sometime getting it tuned right so the memory does fragment. In one high performance site I manage I use the same recipe as above, but I use APC to handle page caching.
Of course all this varies greatly by site. Some sites are mostly anonymous users, meaning page caching is the key, while others have a heavy logged-in user base, so object caching becomes the big factor.
Also depending on how the site is built, there's other cache modules out there that really help, like views cache, authcache and panels cache.
Jamie Holly http://hollyit.net
On 1/3/2014 5:08 AM, Alison wrote:
Hi Guys and Girls,
can anyone recommend a good general discussion site or place where I can read about Drupal caching. Currently running Memcache and APC - the PHP accelerator, but not sure whether there would be anything to gain by implementing Drupals caching as well. Wouldn't mind finding some way of measuring the effects of the various performance enhancements. Are there any downsides to Memcache and APC?
Hugs
Alison
-- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ]
Wow that's almost scary. With the following settings my server leaves burnt rubber on the road. Tweaked the my.cnf on my devel server to test, and then my production server per following from https://drupal.org/node/85768 [mysqld] max_connections = 150 max_user_connections = 150 key_buffer = 96M myisam_sort_buffer_size = 128M join_buffer_size = 8M read_buffer_size = 8M sort_buffer_size = 12M table_cache = 1024 thread_cache_size = 286 table_definition_cache = 1000 interactive_timeout = 25 wait_timeout = 1800 connect_timeout = 10 max_allowed_packet = 4M max_connect_errors = 2000 query_cache_limit = 4M query_cache_size = 64M query_cache_type = 1 tmp_table_size = 64M innodb-flush-log-at-trx-commit = 2 Don't profess to understand most of them, although did a stint as an IDMS/IMS DBA thirty years ago. Above probably need some more tweaking as I go along BUT...... Thanks for tips Alison At 02:09 PM 04-01-14, you wrote:
Your problem isn't really caching, but regeneration. If you have items on the page taking a few seconds to generate, from queries that aren't optimized, huge processing, etc., then whenever that cache is invalidated, either through expiration, content change or a cache clear call, then you're going to get hit with the same problem. Put another way, the hangup isn't getting the data from cache, but rather Drupal generating the data to put into the cache.
The best thing to do is run a development setup and enable the devel module and enable query output in there. Then clear your cache and look on your slow pages to see what queries are holding things up.
Another very common issue is negligence in tuning the actual server. What happens here is that you are throwing performance enhancements like caching into the mix and that is nothing more than a band-aid covering a much larger problem. A key place to start is always with the database. Installing MySQL is very easy, but the configuration it ships with is in no way meant for production (simply because configuration depends upon usage and usage is different between applications and even between Drupal sites). You need to monitor some key MySQL metrics to make sure they are within appropriate ranges. A good tool for the is tuning-primer:
https://launchpad.net/mysql-tuning-primer
Run that after the server has been running a day or two and it will check existing status variables and make recommendations on what to change.
Jamie Holly http://hollyit.net
On 1/3/2014 8:21 PM, Alison wrote:
Hi Jamie, Phil,
thanks for the info Jamie, much appreciated. As to sharing Phil, happy to do so. A little bit about my setup.
I am retired and run a Centos 6.5, Apache, MySQL webserver in my lounge room. The server is home built - Gigabyte mobo, Intel processor, Kingston RAM, etc. I have a second similar server as a backup and development machine. I get 1.5Mb upload speed which is adequate to support half a dozen low traffic Drupal based sites. When I say low traffic, we are looking at 30-50 hits a day for each site
The sites tend to be more personal interest sites rather than commercial ie. personal Blogs etc. The hits tend to be clustered based on chronology and demographic. Typically might get 20-30 hits over a couple of hours on a site and then nothing for 20 hours. During the quiet period had problems with Drupal cache's timing out, so first hit was then incredibly slow.
I installed memcache via Drupal.org and memcache status using their instructions. This seemed to help, but still getting very slow first responses. I then did following:
[root@webserver ~] yum install php-pear php-devel httpd-devel pcre-devel gcc make [root@webserver ~] pecl install apc [root@webserver ~] service httpd restart
This being suggested by a google search. The pecl install seems to have included an initial setup as many questions were asked.
As I have massive amounts of RAM available, I increased all resources in /etc/php.d/apc.ini apc.shm_size = 128M apc.num_files = 4096 apc.user-entries = 4096 apc.ttl - 21600 apc.user_ttl - 21600 apc.gc_ttl = 14400 apc.max_file_size = 5M. Then did a [root@webserver ~] service httpd restart.
There is a really neat tool to see what's happening. Copy apc.php from docs directory to site root, and the run example.com/apc.php.
Now I need to wait a few days to see what happens.
Alison PS. Latest site just built is at http://www.trans-aspie.com
At 01:57 AM 04-01-14, you wrote:
Hi Alison, Please share what you find. I have been intending to look into this myself. I'm concerned about the initial page load time on my site. It seems pretty slow. Regards, Phil
From: Jamie Holly <hovercrafter@earthlink.net> To: support@drupal.org, Date: 01/03/2014 09:31 AM Subject: Re: [support] Improving performance Sent by: support-bounces@drupal.org
The high performance group on drupal.org is really good:
https://groups.drupal.org/high-performance
The most common recipe is APC for opcode caching, memcache for object caching and Drupal's database to cache the form table.
APC also offers object/user caching, but you do have to spend sometime getting it tuned right so the memory does fragment. In one high performance site I manage I use the same recipe as above, but I use APC to handle page caching.
Of course all this varies greatly by site. Some sites are mostly anonymous users, meaning page caching is the key, while others have a heavy logged-in user base, so object caching becomes the big factor.
Also depending on how the site is built, there's other cache modules out there that really help, like views cache, authcache and panels cache.
Jamie Holly http://hollyit.net
On 1/3/2014 5:08 AM, Alison wrote:
Hi Guys and Girls,
can anyone recommend a good general discussion site or place where I can read about Drupal caching. Currently running Memcache and APC - the PHP accelerator, but not sure whether there would be anything to gain by implementing Drupals caching as well. Wouldn't mind finding some way of measuring the effects of the various performance enhancements. Are there any downsides to Memcache and APC?
Hugs
Alison
-- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ]
A better way to get a configuration to start off with is using this tool: https://tools.percona.com/ You got to register an account there, but it's free. If you're running Drupal 7, then you need to also account for tuning InnoDB. Again, there is no "one size fits all" scenario here. One common example is innodb_file_per_table. If you're running a lot of small sites that aren't updated that much, then you can get by with out it, but if you got a couple large sites or a bunch of inserts and deletes going on, then it's best to use that option. The thing with that option is that it doesn't take affect by just changing it. You actually have to either dump all the databases and reimport them, or issue an alter on each table, setting the engine to InnoDB again. Jamie Holly http://hollyit.net On 1/3/2014 11:25 PM, Alison wrote:
Wow that's almost scary. With the following settings my server leaves burnt rubber on the road.
Tweaked the my.cnf on my devel server to test, and then my production server per following from https://drupal.org/node/85768
[mysqld] max_connections = 150 max_user_connections = 150 key_buffer = 96M myisam_sort_buffer_size = 128M join_buffer_size = 8M read_buffer_size = 8M sort_buffer_size = 12M table_cache = 1024 thread_cache_size = 286 table_definition_cache = 1000 interactive_timeout = 25 wait_timeout = 1800 connect_timeout = 10 max_allowed_packet = 4M max_connect_errors = 2000 query_cache_limit = 4M query_cache_size = 64M query_cache_type = 1 tmp_table_size = 64M innodb-flush-log-at-trx-commit = 2
Don't profess to understand most of them, although did a stint as an IDMS/IMS DBA thirty years ago. Above probably need some more tweaking as I go along BUT......
Thanks for tips
Alison
At 02:09 PM 04-01-14, you wrote:
Your problem isn't really caching, but regeneration. If you have items on the page taking a few seconds to generate, from queries that aren't optimized, huge processing, etc., then whenever that cache is invalidated, either through expiration, content change or a cache clear call, then you're going to get hit with the same problem. Put another way, the hangup isn't getting the data from cache, but rather Drupal generating the data to put into the cache.
The best thing to do is run a development setup and enable the devel module and enable query output in there. Then clear your cache and look on your slow pages to see what queries are holding things up.
Another very common issue is negligence in tuning the actual server. What happens here is that you are throwing performance enhancements like caching into the mix and that is nothing more than a band-aid covering a much larger problem. A key place to start is always with the database. Installing MySQL is very easy, but the configuration it ships with is in no way meant for production (simply because configuration depends upon usage and usage is different between applications and even between Drupal sites). You need to monitor some key MySQL metrics to make sure they are within appropriate ranges. A good tool for the is tuning-primer:
https://launchpad.net/mysql-tuning-primer
Run that after the server has been running a day or two and it will check existing status variables and make recommendations on what to change.
Jamie Holly http://hollyit.net
On 1/3/2014 8:21 PM, Alison wrote:
Hi Jamie, Phil,
thanks for the info Jamie, much appreciated. As to sharing Phil, happy to do so. A little bit about my setup.
I am retired and run a Centos 6.5, Apache, MySQL webserver in my lounge room. The server is home built - Gigabyte mobo, Intel processor, Kingston RAM, etc. I have a second similar server as a backup and development machine. I get 1.5Mb upload speed which is adequate to support half a dozen low traffic Drupal based sites. When I say low traffic, we are looking at 30-50 hits a day for each site
The sites tend to be more personal interest sites rather than commercial ie. personal Blogs etc. The hits tend to be clustered based on chronology and demographic. Typically might get 20-30 hits over a couple of hours on a site and then nothing for 20 hours. During the quiet period had problems with Drupal cache's timing out, so first hit was then incredibly slow.
I installed memcache via Drupal.org and memcache status using their instructions. This seemed to help, but still getting very slow first responses. I then did following:
[root@webserver ~] yum install php-pear php-devel httpd-devel pcre-devel gcc make [root@webserver ~] pecl install apc [root@webserver ~] service httpd restart
This being suggested by a google search. The pecl install seems to have included an initial setup as many questions were asked.
As I have massive amounts of RAM available, I increased all resources in /etc/php.d/apc.ini apc.shm_size = 128M apc.num_files = 4096 apc.user-entries = 4096 apc.ttl - 21600 apc.user_ttl - 21600 apc.gc_ttl = 14400 apc.max_file_size = 5M. Then did a [root@webserver ~] service httpd restart.
There is a really neat tool to see what's happening. Copy apc.php from docs directory to site root, and the run example.com/apc.php.
Now I need to wait a few days to see what happens.
Alison PS. Latest site just built is at http://www.trans-aspie.com
At 01:57 AM 04-01-14, you wrote:
Hi Alison, Please share what you find. I have been intending to look into this myself. I'm concerned about the initial page load time on my site. It seems pretty slow. Regards, Phil
From: Jamie Holly <hovercrafter@earthlink.net> To: support@drupal.org, Date: 01/03/2014 09:31 AM Subject: Re: [support] Improving performance Sent by: support-bounces@drupal.org
The high performance group on drupal.org is really good:
https://groups.drupal.org/high-performance
The most common recipe is APC for opcode caching, memcache for object caching and Drupal's database to cache the form table.
APC also offers object/user caching, but you do have to spend sometime getting it tuned right so the memory does fragment. In one high performance site I manage I use the same recipe as above, but I use APC to handle page caching.
Of course all this varies greatly by site. Some sites are mostly anonymous users, meaning page caching is the key, while others have a heavy logged-in user base, so object caching becomes the big factor.
Also depending on how the site is built, there's other cache modules out there that really help, like views cache, authcache and panels cache.
Jamie Holly http://hollyit.net
On 1/3/2014 5:08 AM, Alison wrote:
Hi Guys and Girls,
can anyone recommend a good general discussion site or place where I can read about Drupal caching. Currently running Memcache and APC - the PHP accelerator, but not sure whether there would be anything to gain by implementing Drupals caching as well. Wouldn't mind finding some way of measuring the effects of the various performance enhancements. Are there any downsides to Memcache and APC?
Hugs
Alison
-- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ]
Dear Alison, Have you considered using varnish? Best wishes, Shérab.
Think the problem is now resolved to my satisfaction. Thanks. I have previously used Memcache successfully, and now with APC opcode caching and a bit of MySQL tuning, all looks good. Alison At 07:50 PM 04-01-14, you wrote:
Dear Alison, Have you considered using varnish? Best wishes, Shérab. -- [ Drupal support list | http://lists.drupal.org/ ]
I learned a whole lot from the excellent book High Performance Drupal that just came out this fall. An excellent resource. Gregg Marshall -----Original Message----- From: support-bounces@drupal.org [mailto:support-bounces@drupal.org] On Behalf Of Alison Sent: Saturday, January 04, 2014 2:42 AM To: support@drupal.org Subject: Re: [support] Improving performance Think the problem is now resolved to my satisfaction. Thanks. I have previously used Memcache successfully, and now with APC opcode caching and a bit of MySQL tuning, all looks good. Alison At 07:50 PM 04-01-14, you wrote:
Dear Alison, Have you considered using varnish? Best wishes, Shérab. -- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ]
Wow, Thanks Alison and all, That's a lot of info. Not sure how much translates over to windows. I will take a look. Phil From: Alison <penguin@alisoncc.com> To: support@drupal.org, Date: 01/03/2014 09:58 PM Subject: Re: [support] Improving performance Sent by: support-bounces@drupal.org Hi Jamie, Phil, thanks for the info Jamie, much appreciated. As to sharing Phil, happy to do so. A little bit about my setup. I am retired and run a Centos 6.5, Apache, MySQL webserver in my lounge room. The server is home built - Gigabyte mobo, Intel processor, Kingston RAM, etc. I have a second similar server as a backup and development machine. I get 1.5Mb upload speed which is adequate to support half a dozen low traffic Drupal based sites. When I say low traffic, we are looking at 30-50 hits a day for each site The sites tend to be more personal interest sites rather than commercial ie. personal Blogs etc. The hits tend to be clustered based on chronology and demographic. Typically might get 20-30 hits over a couple of hours on a site and then nothing for 20 hours. During the quiet period had problems with Drupal cache's timing out, so first hit was then incredibly slow. I installed memcache via Drupal.org and memcache status using their instructions. This seemed to help, but still getting very slow first responses. I then did following: [root@webserver ~] yum install php-pear php-devel httpd-devel pcre-devel gcc make [root@webserver ~] pecl install apc [root@webserver ~] service httpd restart This being suggested by a google search. The pecl install seems to have included an initial setup as many questions were asked. As I have massive amounts of RAM available, I increased all resources in /etc/php.d/apc.ini apc.shm_size = 128M apc.num_files = 4096 apc.user-entries = 4096 apc.ttl - 21600 apc.user_ttl - 21600 apc.gc_ttl = 14400 apc.max_file_size = 5M. Then did a [root@webserver ~] service httpd restart. There is a really neat tool to see what's happening. Copy apc.php from docs directory to site root, and the run example.com/apc.php. Now I need to wait a few days to see what happens. Alison PS. Latest site just built is at http://www.trans-aspie.com At 01:57 AM 04-01-14, you wrote:
Hi Alison, Please share what you find. I have been intending to look into this myself. I'm concerned about the initial page load time on my site. It seems pretty slow. Regards, Phil
From: Jamie Holly <hovercrafter@earthlink.net> To: support@drupal.org, Date: 01/03/2014 09:31 AM Subject: Re: [support] Improving performance Sent by: support-bounces@drupal.org
The high performance group on drupal.org is really good:
https://groups.drupal.org/high-performance
The most common recipe is APC for opcode caching, memcache for object caching and Drupal's database to cache the form table.
APC also offers object/user caching, but you do have to spend sometime getting it tuned right so the memory does fragment. In one high performance site I manage I use the same recipe as above, but I use APC to handle page caching.
Of course all this varies greatly by site. Some sites are mostly anonymous users, meaning page caching is the key, while others have a heavy logged-in user base, so object caching becomes the big factor.
Also depending on how the site is built, there's other cache modules out there that really help, like views cache, authcache and panels cache.
Jamie Holly http://hollyit.net
On 1/3/2014 5:08 AM, Alison wrote:
Hi Guys and Girls,
can anyone recommend a good general discussion site or place where I can read about Drupal caching. Currently running Memcache and APC - the PHP accelerator, but not sure whether there would be anything to gain by implementing Drupals caching as well. Wouldn't mind finding some way of measuring the effects of the various performance enhancements. Are there any downsides to Memcache and APC?
Hugs
Alison
-- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ]
participants (5)
-
Alison -
Gregg Marshall -
Jamie Holly -
Philip_Wetzel＠nhd.uscourts.gov -
Shérab