Thank you for this in depth discussion it is much appreciated, and yes, I do appreciate the ease with which Drupal "sets up" a full blown site. I also very much appreciate the highly professional way people respond, no flame wars, no ridicule, just patient insightful information. Thank you.
My observations to date: -- Our server runs Centos 6 and php 5.3 -- Drupal is a great system but can suffer from less than optimal module coding as explained below. -- Some sites have 60 and more modules and work flawlessly. -- Our site works flawlessly for the most part. -- We have 41 modules soon to be about 45ish when I start the newsletter system so number of modules, in our case, is not a problem because most are not used in normal operation or called infrequently but are still required. -- We have a very minimal site with very low demand from users, minimal demand on modules, minimal calls on database (until I get the newsletter system working) and few display calls, little of the below discussion relates to our requirements. -- Add Content Type is still a problem, it seems to unexpectedly fail. -- Based on a previous response, adding a Content Type, seems to have much broader issues with far greater overhead than I first understood. -- Apart from altering php memory_limit, which works sometimes but has wider implications for the whole site, there is no definitive answer to this Add Content Type problem. --There seems to be no pressing need to have greater than 128M allocated to PHP. Our ISP support team advise against increasing above 128M.
Conclusion: It seems that Add Content Type puts outrageously heavy demand on a system. Why it has 17 includes unrelated to creating a content type, Calls 18 scripts, all but node/content_types, completely unrelated and unnecessary at this point, a huge jQuery that has nothing at all to do with anything, irrelevant menu lists that are switched off, not in use, or not in the least related to this display, I am not sure.
Reading about Ruby on Rails and separating out the business logic from data and views. Looking at the underlying code for Drupal Add Content Type it seems there is a need to separate out much of the business logic. But I do not know if this can be done with PHP.
As below, optimising Drupal means stripping things away, but when faced with this, stripping can't help much. Roger
<snip> Every module is coded in a different way. While some can be benign, others might be less so. Some modules create additional load because they force pages to be re-rendered, or for permissions to be re-checked and so forth. This is a community-project, which means lots of free great code, but also not everything is coded perfectly at all times, since everyone is contributing in an open source fashion. Each module that is activated also creates some additional function calls, depending on the functions found, those functions will either run once and be cached, or be perpetually accessed every time their hook_ is called. Is adding one more hook_ a problem? Not inherently, but clearly the more you add, the more calls need to be made. If you want a system that can handle hundreds of modules, each with potentially tens of functions, each of which is doing a calculation then surely that increases your server's stress? There is nothing magic about any CMS, it is, as you say, just a bunch of PHP code. An optimal system needs careful consideration. If you want a site like facebook, where you are dealing with tremendous stress considerations, you generally just build the site up from scratch. No CMS, you put all the parts together, with the absolute zero-line that anything non-essential is scrapped. If you have the budget to build your site from the ground-up, you just don't bother with any CMS and build your own (or use an extremely lightweight framework as your base point). Drupal gives you a lot out of the box, not many CMS's can crunch out full blown applications with such flexibility and extensibility, but flexibility comes with a price. Optimizing Drupal sometimes involves stripping things away, researching how things were done by others, and improving the code that others have made. Your question and concern is extremely general, so it is hard to give you anything concrete in return. Can you run a complex application on Drupal? Absolutely Can you run a site that has an extremely fast response time? Most definitely Can Drupal be used on a website that has high traffic, sometimes with burst traffic intervals? For sure can I get a high performant site, by installing any module I find on the net, keeping it on, and just expect that clicking "setup" is enough to achieve high performance, and high speed? Doubtful? You need to be good at web development to be good at performance and optimization. It is a very complex subject, and many solutions, depending on need, can bring you beyond drupal, to things like memcache / apache / mysql optimization. Drupal gives you a giant flexible application building API/environment. You then need to take care of it to make sure it retains high performance. Hope this helps! -- Info Razor Sent with Sparrow (http://www.sparrowmailapp.com/?sig) </snip>
Hi Roger,
I still wonder why don't you do this:
1. Downloads and install from source apache & php 5.3. There are penty of tutorials how to do this. Compile only php modules you need.(!)
2. Install in /usr/local/ (It goes there by default)
3. Modify firewall to allow port 8080
4. Edit httpd.conf of the newly compiled Apache so you have correct DocumentRoot and make it listen 8080.
5. Start the new Apache on 8080
6. Check how site behaves on port 80 and port 8080.
A) If it is better on 8080 shut down primary Apache, and change teh port of the new one on 80.
B) If it is the same, check database - run check and repair mysql tables - might be broken indexes or something.
Peter
On 07/04/2013 03:08, Roger wrote:
Thank you for this in depth discussion it is much appreciated, and yes, I do appreciate the ease with which Drupal "sets up" a full blown site. I also very much appreciate the highly professional way people respond, no flame wars, no ridicule, just patient insightful information. Thank you.
My observations to date: -- Our server runs Centos 6 and php 5.3 -- Drupal is a great system but can suffer from less than optimal module coding as explained below. -- Some sites have 60 and more modules and work flawlessly. -- Our site works flawlessly for the most part. -- We have 41 modules soon to be about 45ish when I start the newsletter system so number of modules, in our case, is not a problem because most are not used in normal operation or called infrequently but are still required. -- We have a very minimal site with very low demand from users, minimal demand on modules, minimal calls on database (until I get the newsletter system working) and few display calls, little of the below discussion relates to our requirements. -- Add Content Type is still a problem, it seems to unexpectedly fail. -- Based on a previous response, adding a Content Type, seems to have much broader issues with far greater overhead than I first understood. -- Apart from altering php memory_limit, which works sometimes but has wider implications for the whole site, there is no definitive answer to this Add Content Type problem. --There seems to be no pressing need to have greater than 128M allocated to PHP. Our ISP support team advise against increasing above 128M.
Conclusion: It seems that Add Content Type puts outrageously heavy demand on a system. Why it has 17 includes unrelated to creating a content type, Calls 18 scripts, all but node/content_types, completely unrelated and unnecessary at this point, a huge jQuery that has nothing at all to do with anything, irrelevant menu lists that are switched off, not in use, or not in the least related to this display, I am not sure.
Reading about Ruby on Rails and separating out the business logic from data and views. Looking at the underlying code for Drupal Add Content Type it seems there is a need to separate out much of the business logic. But I do not know if this can be done with PHP.
As below, optimising Drupal means stripping things away, but when faced with this, stripping can't help much. Roger
<snip> Every module is coded in a different way. While some can be benign, others might be less so. Some modules create additional load because they force pages to be re-rendered, or for permissions to be re-checked and so forth. This is a community-project, which means lots of free great code, but also not everything is coded perfectly at all times, since everyone is contributing in an open source fashion. Each module that is activated also creates some additional function calls, depending on the functions found, those functions will either run once and be cached, or be perpetually accessed every time their hook_ is called. Is adding one more hook_ a problem? Not inherently, but clearly the more you add, the more calls need to be made. If you want a system that can handle hundreds of modules, each with potentially tens of functions, each of which is doing a calculation then surely that increases your server's stress? There is nothing magic about any CMS, it is, as you say, just a bunch of PHP code. An optimal system needs careful consideration. If you want a site like facebook, where you are dealing with tremendous stress considerations, you generally just build the site up from scratch. No CMS, you put all the parts together, with the absolute zero-line that anything non-essential is scrapped. If you have the budget to build your site from the ground-up, you just don't bother with any CMS and build your own (or use an extremely lightweight framework as your base point). Drupal gives you a lot out of the box, not many CMS's can crunch out full blown applications with such flexibility and extensibility, but flexibility comes with a price. Optimizing Drupal sometimes involves stripping things away, researching how things were done by others, and improving the code that others have made. Your question and concern is extremely general, so it is hard to give you anything concrete in return. Can you run a complex application on Drupal? Absolutely Can you run a site that has an extremely fast response time? Most definitely Can Drupal be used on a website that has high traffic, sometimes with burst traffic intervals? For sure can I get a high performant site, by installing any module I find on the net, keeping it on, and just expect that clicking "setup" is enough to achieve high performance, and high speed? Doubtful? You need to be good at web development to be good at performance and optimization. It is a very complex subject, and many solutions, depending on need, can bring you beyond drupal, to things like memcache / apache / mysql optimization. Drupal gives you a giant flexible application building API/environment. You then need to take care of it to make sure it retains high performance. Hope this helps! -- Info Razor Sent with Sparrow (http://www.sparrowmailapp.com/?sig) </snip>
On 07/04/13 13:23, Zyumbilev, Peter wrote:
Hi Roger,
I still wonder why don't you do this:
- Downloads and install from source apache & php 5.3. There are penty
of tutorials how to do this. Compile only php modules you need.(!)
Install in/usr/local/ (It goes there by default)
Modify firewall to allow port 8080
Edit httpd.conf of the newly compiled Apache so you have correct
DocumentRoot and make it listen 8080.
Start the new Apache on 8080
Check how site behaves on port 80 and port 8080.
A) If it is better on 8080 shut down primary Apache, and change teh port of the new one on 80.
B) If it is the same, check database - run check and repair mysql tables
- might be broken indexes or something.
Peter
Because it's a waste of time. If he's running on RHEL or CentOS (he said he'd test on CentOS) he's using the same software that carries the certification demanded by all the largest companies in the world and on which red Hat bets its business.
If he's still testing on Fedora, then testing on CentOS is essential. Fedora is quite likely to be broken, but probably better than Apache's apache and PHP's PHP because they have undergone additional testing and hammering by Good Folk from Drupal, Joomla, Wordpress and any other significant open source project using PHP.
I think it has to be something not certified such as Drupal or Roger.
I cannot agree. Centos is not certified at all :-) So we cannot speak about certification at all here.
Compilation and setup can be done in 1-2 hours - I think he wasted more time than that.
Going the compilation way will probably help Roger go around his mistake e.g wrong httpd or php config.
Peter
On 07/04/2013 12:07, John Summerfield wrote:
On 07/04/13 13:23, Zyumbilev, Peter wrote:
Hi Roger,
I still wonder why don't you do this:
- Downloads and install from source apache & php 5.3. There are penty
of tutorials how to do this. Compile only php modules you need.(!)
Install in/usr/local/ (It goes there by default)
Modify firewall to allow port 8080
Edit httpd.conf of the newly compiled Apache so you have correct
DocumentRoot and make it listen 8080.
Start the new Apache on 8080
Check how site behaves on port 80 and port 8080.
A) If it is better on 8080 shut down primary Apache, and change teh port of the new one on 80.
B) If it is the same, check database - run check and repair mysql tables
- might be broken indexes or something.
Peter
Because it's a waste of time. If he's running on RHEL or CentOS (he said he'd test on CentOS) he's using the same software that carries the certification demanded by all the largest companies in the world and on which red Hat bets its business.
If he's still testing on Fedora, then testing on CentOS is essential. Fedora is quite likely to be broken, but probably better than Apache's apache and PHP's PHP because they have undergone additional testing and hammering by Good Folk from Drupal, Joomla, Wordpress and any other significant open source project using PHP.
I think it has to be something not certified such as Drupal or Roger.
On 07/04/13 18:00, Zyumbilev, Peter wrote:
I cannot agree. Centos is not certified at all:-) So we cannot speak about certification at all here.
It's build from RHEL source and RHEL is.
It would be more useful to rebuild Apache and PHP from Red Hat's source which has undergone more testing and fixing since it was downloaded from the upstream providers, That is exactly what CentOS has already done.
You can safely increase memory limit to much higher value.
The price of a block of RAM is very small. Increase php memory limit to 512Mb and see how it goes. That might the simplest solution to try :-)
Peter
On 07/04/2013 03:08, Roger wrote:
Thank you for this in depth discussion it is much appreciated, and yes, I do appreciate the ease with which Drupal "sets up" a full blown site. I also very much appreciate the highly professional way people respond, no flame wars, no ridicule, just patient insightful information. Thank you.
My observations to date: -- Our server runs Centos 6 and php 5.3 -- Drupal is a great system but can suffer from less than optimal module coding as explained below. -- Some sites have 60 and more modules and work flawlessly. -- Our site works flawlessly for the most part. -- We have 41 modules soon to be about 45ish when I start the newsletter system so number of modules, in our case, is not a problem because most are not used in normal operation or called infrequently but are still required. -- We have a very minimal site with very low demand from users, minimal demand on modules, minimal calls on database (until I get the newsletter system working) and few display calls, little of the below discussion relates to our requirements. -- Add Content Type is still a problem, it seems to unexpectedly fail. -- Based on a previous response, adding a Content Type, seems to have much broader issues with far greater overhead than I first understood. -- Apart from altering php memory_limit, which works sometimes but has wider implications for the whole site, there is no definitive answer to this Add Content Type problem. --There seems to be no pressing need to have greater than 128M allocated to PHP. Our ISP support team advise against increasing above 128M.
Conclusion: It seems that Add Content Type puts outrageously heavy demand on a system. Why it has 17 includes unrelated to creating a content type, Calls 18 scripts, all but node/content_types, completely unrelated and unnecessary at this point, a huge jQuery that has nothing at all to do with anything, irrelevant menu lists that are switched off, not in use, or not in the least related to this display, I am not sure.
Reading about Ruby on Rails and separating out the business logic from data and views. Looking at the underlying code for Drupal Add Content Type it seems there is a need to separate out much of the business logic. But I do not know if this can be done with PHP.
As below, optimising Drupal means stripping things away, but when faced with this, stripping can't help much. Roger
<snip> Every module is coded in a different way. While some can be benign, others might be less so. Some modules create additional load because they force pages to be re-rendered, or for permissions to be re-checked and so forth. This is a community-project, which means lots of free great code, but also not everything is coded perfectly at all times, since everyone is contributing in an open source fashion. Each module that is activated also creates some additional function calls, depending on the functions found, those functions will either run once and be cached, or be perpetually accessed every time their hook_ is called. Is adding one more hook_ a problem? Not inherently, but clearly the more you add, the more calls need to be made. If you want a system that can handle hundreds of modules, each with potentially tens of functions, each of which is doing a calculation then surely that increases your server's stress? There is nothing magic about any CMS, it is, as you say, just a bunch of PHP code. An optimal system needs careful consideration. If you want a site like facebook, where you are dealing with tremendous stress considerations, you generally just build the site up from scratch. No CMS, you put all the parts together, with the absolute zero-line that anything non-essential is scrapped. If you have the budget to build your site from the ground-up, you just don't bother with any CMS and build your own (or use an extremely lightweight framework as your base point). Drupal gives you a lot out of the box, not many CMS's can crunch out full blown applications with such flexibility and extensibility, but flexibility comes with a price. Optimizing Drupal sometimes involves stripping things away, researching how things were done by others, and improving the code that others have made. Your question and concern is extremely general, so it is hard to give you anything concrete in return. Can you run a complex application on Drupal? Absolutely Can you run a site that has an extremely fast response time? Most definitely Can Drupal be used on a website that has high traffic, sometimes with burst traffic intervals? For sure can I get a high performant site, by installing any module I find on the net, keeping it on, and just expect that clicking "setup" is enough to achieve high performance, and high speed? Doubtful? You need to be good at web development to be good at performance and optimization. It is a very complex subject, and many solutions, depending on need, can bring you beyond drupal, to things like memcache / apache / mysql optimization. Drupal gives you a giant flexible application building API/environment. You then need to take care of it to make sure it retains high performance. Hope this helps! -- Info Razor Sent with Sparrow (http://www.sparrowmailapp.com/?sig) </snip>
On 07/04/13 13:29, Zyumbilev, Peter wrote:
You can safely increase memory limit to much higher value.
The price of a block of RAM is very small. Increase php memory limit to 512Mb and see how it goes. That might the simplest solution to try:-)
At home, yes. On a plan, maybe not.
And, I think he said he's done that anyway.
You can safely increase memory limit to much higher value.
The price of a block of RAM is very small. Increase php memory limit to 512Mb and see how it goes. That might the simplest solution to try:-)
Funny thing is my machines do not exhibit lack of memory. I run <System Monitor> most of the time and do "top" checks in a terminal frequently. I mentioned in a previous email that I had tested this on my machine and went as far as 2048M but after 512M the problem worsened by degrees, so throwing more memory isn't the solution. Roger
On 07/04/13 18:27, Roger wrote:
Funny thing is my machines do not exhibit lack of memory. I run <System Monitor> most of the time and do "top" checks in a terminal frequently. I mentioned in a previous email that I had tested this on my machine and went as far as 2048M but after 512M the problem worsened by degrees, so throwing more memory isn't the solution. Roger
Exactly what is running on the shared hosting system? Maybe you can replicate that?
Roger,
I would not expect any increase in total memory to affect the problem at all, since you're still capping php at 128M per page process. The only place where overall memory comes into play is when you have lots of people hitting pages that consume lots of memory.
I'd recommend getting back to trying to figure out why you are getting the errors that are bothering you.
I'm running views, panels etc on sites with 96M memory setting and create content types without trouble all the time, so I think you might want to find out what kind of problem you have.
I'd download the devel module if you haven't already. In the devel settings page you can enable "display memory usage". Next you can start disabling modules and switching themes to see if you can identify which module or theme is causing you the most trouble. Try visiting different pages to find out more.
For example, when I enabled the panels module, and views module, the memory consumption went up from a peak of 2.5 to a peak of 6.5 on the modules page, but when visiting the front page of my site, it was only at 3.0.
Remember that the PHP Memory limit is about MAX consumption and not typical consumption, so increasing its value does not necessarily mean that you are opening up your site for problems at load. In older versions of drupal we used to have to worry about visiting the modules page, which would load every module (regardless of whether installed), but the memory consumption for all the other pages wasn't affected. The memory limit is just a break so that you don't have a runaway process consuming all memory so that you can't even debug it.
It seems to me that you may have a module or theme with a memory leak, and that your memory consumption is not that of a typical drupal install. I'd be working hard to verify this.
That is why many continue to ask for your module list. They're trying to figure out if you've got any non-standard modules that might have a leak or performance problem.
Hopefully that helps, (I fear we've gotten a bit off topic as we've started debating the age old question of when it's cheaper just to throw hardware resources at the problem or spend manpower resources trying to optimize).
-----Original Message----- From: support-bounces@drupal.org [mailto:support-bounces@drupal.org] On Behalf Of Roger Sent: Sunday, April 07, 2013 3:27 AM To: Zyumbilev, Peter Cc: support@drupal.org Subject: Re: [support] Fed up with Drupal 7 ... Desperately.....modules.etc
You can safely increase memory limit to much higher value.
The price of a block of RAM is very small. Increase php memory limit to 512Mb and see how it goes. That might the simplest solution to try:-)
Funny thing is my machines do not exhibit lack of memory. I run <System Monitor> most of the time and do "top" checks in a terminal frequently. I mentioned in a previous email that I had tested this on my machine and went as far as 2048M but after 512M the problem worsened by degrees, so throwing more memory isn't the solution. Roger
On 07/04/13 08:08, Roger wrote:
-- Add Content Type is still a problem, it seems to unexpectedly fail.
Just for the hell of it, I just created a new content type at my test site, testserver.example.com which is my CentOS6 test system.
I got a couple of messages from a highlighter module, but nothing to do with the content type. I added an extra field to me - again just for the fun of it, I created an email address field (the email module).
Created a new "new type," all good.
Why don't you post a list of all the modules you are using?
-
John Summerfield -
Metzler, David -
Roger -
Zyumbilev, Peter