Re: [development] How many modules is too many?
I have nearly 100 projects, many with multiple modules, and am not done adding, yet. It does really eat up memory like crazy but I haven't heard a solution to it. The fact is that some sites, especially social networking ones, use a lot of modules. Most of them are fairly small but needed. Would combining the smaller modules into one big one help any? Or would that still use the same amount of memory? Michelle On 11/21/2007 12:29:46 PM, Khalid Baheyeldin (kb@2bits.com) wrote:
(Changing the subject)
I regularly see 80+ to 110+, and that causes Apache to eat 100MB per process, which is really not good ...
This is the open buffet binge syndrome detailed here:
http://2bits.com/articles/server-indigestion-the-drupal-contributed- modules-open-buffet-binge-syndrome.html
On Nov 21, 2007 12:50 PM, Jim Li <jimmydami@gmail.com> wrote:
Cool, thanks!
It'd be interesting to have a poll on how many modules people use on a
social network site. I heard someone uses 160 modules at his dev site, it may go down a bit later, but it probably will still in the 100+ ragnge :)
On Nov 20, 2007 6:31 PM, Earl Miles <merlin@logrus.com> wrote:
On the other hand, having 150+ modules load *is* going to eat a whole lot of memory; so actually activating all these modules is a really bad idea.
And yes, the modules page under 4.7 is going to choke like [insert really bad sports team metaphor here].
Sean Robertson wrote:
As I understand it, Drupal 5+ no longer does that. That's what the
On Nov 21, 2007 12:00 PM, Michelle Cox <mcox@charter.net> wrote:
I have nearly 100 projects, many with multiple modules, and am not done adding, yet. It does really eat up memory like crazy but I haven't heard a solution to it. The fact is that some sites, especially social networking ones, use a lot of modules. Most of them are fairly small but needed. Would combining the smaller modules into one big one help any? Or would that still use the same amount of memory?
I have the same question. Does the number of modules enabled significantly affect memory usage? I tend to create tons of tiny modules, many of which do something simple - like provide a single hook. If the overhead from this approach is significant, then I need to change my ways! Ryan
Michelle
On 11/21/2007 12:29:46 PM, Khalid Baheyeldin (kb@2bits.com) wrote:
(Changing the subject)
I regularly see 80+ to 110+, and that causes Apache to eat 100MB per process, which is really not good ...
This is the open buffet binge syndrome detailed here:
http://2bits.com/articles/server-indigestion-the-drupal-contributed- modules-open-buffet-binge-syndrome.html
On Nov 21, 2007 12:50 PM, Jim Li <jimmydami@gmail.com> wrote:
Cool, thanks!
It'd be interesting to have a poll on how many modules people use on a
social network site. I heard someone uses 160 modules at his dev site, it may go down a bit later, but it probably will still in the 100+ ragnge :)
On Nov 20, 2007 6:31 PM, Earl Miles <merlin@logrus.com> wrote:
On the other hand, having 150+ modules load *is* going to eat a whole lot of memory; so actually activating all these modules is a really bad idea.
And yes, the modules page under 4.7 is going to choke like [insert really bad sports team metaphor here].
Sean Robertson wrote:
As I understand it, Drupal 5+ no longer does that. That's what the
If the code volume (# of lines, kilobytes of file size) is the same, then the memory imprint will not really be affected. But speed will suffer since each file that has to be opened slows things down. Drupal 6 will better optimize this issue by allowing module developers to better include only the needed files through the menu system, thus reducing overall memory usage without adding much overhead for opening more files. I am sure chx would give a better summary. -----Original Message----- From: development-bounces@drupal.org [mailto:development-bounces@drupal.org] On Behalf Of Ryan Courtnage ? Sent: Wednesday, November 21, 2007 11:41 AM To: development@drupal.org Subject: Re: [development] How many modules is too many? On Nov 21, 2007 12:00 PM, Michelle Cox <mcox@charter.net> wrote:
I have nearly 100 projects, many with multiple modules, and am not done adding, yet. It does really eat up memory like crazy but I haven't heard a solution to it. The fact is that some sites, especially social networking ones, use a lot of modules. Most of them are fairly small but needed. Would combining the smaller modules into one big one help any? Or would that still use the same amount of memory?
I have the same question. Does the number of modules enabled significantly affect memory usage? I tend to create tons of tiny modules, many of which do something simple - like provide a single hook. If the overhead from this approach is significant, then I need to change my ways! Ryan
Michelle
On 11/21/2007 12:29:46 PM, Khalid Baheyeldin (kb@2bits.com) wrote:
(Changing the subject)
I regularly see 80+ to 110+, and that causes Apache to eat 100MB per process, which is really not good ...
This is the open buffet binge syndrome detailed here:
http://2bits.com/articles/server-indigestion-the-drupal-contributed- modules-open-buffet-binge-syndrome.html
On Nov 21, 2007 12:50 PM, Jim Li <jimmydami@gmail.com> wrote:
Cool, thanks!
It'd be interesting to have a poll on how many modules people use on
a
social network site. I heard someone uses 160 modules at his dev site, it may go down a bit later, but it probably will still in the 100+ ragnge :)
On Nov 20, 2007 6:31 PM, Earl Miles <merlin@logrus.com> wrote:
On the other hand, having 150+ modules load *is* going to eat a whole lot of memory; so actually activating all these modules is a really bad idea.
And yes, the modules page under 4.7 is going to choke like [insert really bad sports team metaphor here].
Sean Robertson wrote:
As I understand it, Drupal 5+ no longer does that. That's what the
Greg Holsclaw wrote:
Drupal 6 will better optimize this issue by allowing module developers to better include only the needed files through the menu system, thus reducing overall memory usage without adding much overhead for opening more files. I am sure chx would give a better summary.
I was just thinking, what might be nice, then, is for there to be some standard way of componentizing a module, and bringing in bits of it based on the URL. I take it from the mention of menu that that's what D6 does? If you introduce a precompiler to minimize the file hits in such a system as Khalid mentions, would that nullify the effect you're after on the memory? Do precompilers typically work within the allocated memory of the apache processes and hence bloat them just the same? Cheers, J-P -- J-P Stacey +44 (0)1608 811870 http://torchbox.com
On Nov 22, 2007 4:41 AM, J-P Stacey <jp.stacey@torchbox.com> wrote:
Greg Holsclaw wrote:
Drupal 6 will better optimize this issue by allowing module developers to better include only the needed files through the menu system, thus reducing overall memory usage without adding much overhead for opening more files. I am sure chx would give a better summary.
I was just thinking, what might be nice, then, is for there to be some standard way of componentizing a module, and bringing in bits of it based on the URL. I take it from the mention of menu that that's what D6 does?
Module splitting has happened as part of D6, mostly along the lines of admin stuff and regular stuff. However, it can be further tuned to be more granular, but it has to be on a module by module basis. No general rule can work for all possible permutations.
If you introduce a precompiler to minimize the file hits in such a system as Khalid mentions, would that nullify the effect you're after on the memory? Do precompilers typically work within the allocated memory of the apache processes and hence bloat them just the same?
Effectively, if you have a shared memory based op-code cache (e.g. APC), then yes, the splitting becomes moot. Remember that not all people can have an op-code cache (e.g. shared hosts), so there is benefit to those from splitting the modules. Op-code caches do save memory, since the code is loaded ONCE, parsed, tokenized and stored as tokens. So, the memory footprint becomes less per Apache process.
Cheers, J-P -- J-P Stacey +44 (0)1608 811870 http://torchbox.com
-- Khalid M. Baheyeldin 2bits.com http://2bits.com Drupal optimization, development, customization and consulting.
On Thursday 22 November 2007, Khalid Baheyeldin wrote:
On Nov 22, 2007 4:41 AM, J-P Stacey <jp.stacey@torchbox.com> wrote:
Greg Holsclaw wrote:
Drupal 6 will better optimize this issue by allowing module developers to better include only the needed files through the menu system, thus reducing overall memory usage without adding much overhead for opening more files. I am sure chx would give a better summary.
I was just thinking, what might be nice, then, is for there to be some standard way of componentizing a module, and bringing in bits of it based on the URL. I take it from the mention of menu that that's what D6 does?
Module splitting has happened as part of D6, mostly along the lines of admin stuff and regular stuff. However, it can be further tuned to be more granular, but it has to be on a module by module basis. No general rule can work for all possible permutations.
What Drupal 6 does is allow hook_menu() and hook_theme() to specify an extra file to be loaded into memory before the page handler or theme function(s) are called. The new theme system has also moved many to most theme functions off into template files, which has the double benefit of making them easier to theme as well as not being loaded unless actually required. As of beta 3, all core modules have been split up with the exception of locale, which is already split in a somewhat bizarre way by putting all of its page handlers into locale.inc. I'm not certain it's worth refactoring that when in beta 3 stage, when there wouldn't be any significant difference in the amount of code that's loaded. (Gabor, I defer to you on that one.) I need to do some benchmarks to see just how much time we save this way, but based on earlier partial benchmarks I suspect it will be considerable. Of course, it also is dependent on contrib authors taking advantage of this new functionality. If you maintain a contrib module, *please* make sure to read this page: http://drupal.org/node/146172 I am already working on a new system to take that process a step further in Drupal 7, but I don't want to get into details on the dev list when Drupal 6 isn't even out yet. If you want to hear about it, go test patches. :-) -- Larry Garfield AIM: LOLG42 larry@garfieldtech.com ICQ: 6817012 "If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it." -- Thomas Jefferson
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Larry Garfield schrieb:
to theme as well as not being loaded unless actually required. As of beta 3, all core modules have been split up with the exception of locale, which is already split in a somewhat bizarre way by putting all of its page handlers into locale.inc.
*ggg* As you mention this, I have to remember how much I had to beg Dries for being allowed to split it up in that bizarre way way back for D 4.5 or so. tempora mutantur Cheers, Gerhard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHRdAcfg6TFvELooQRAhN/AKCM7LHmP5oh9OhG/ydpzEk0q/u7GwCguleZ GbZIXKbGjrALGk2nO0nYA8s= =wDhS -----END PGP SIGNATURE-----
Throttling should shut down excess modules when necessary.
On Nov 22, 2007 7:45 PM, Larry Garfield <larry@garfieldtech.com> wrote:
... As of beta 3, all core modules have been split up with the exception of locale, which is already split in a somewhat bizarre way by putting all of its page handlers into locale.inc. I'm not certain it's worth refactoring that when in beta 3 stage, when there wouldn't be any significant difference in the amount of code that's loaded. (Gabor, I defer to you on that one.)
I have commented on the issue with some immediate concerns, but none of the tools (my issues, issue search, etc) is good for people like me who comment on an enourmous amount of patches, so a direct patch URL would be nice :) Also, it needs some reroll, as we fixed quite some stuff around there, since it was submitted. I still think to complete the split, locale stuff would be great to move properly to modules/locale and only the really required include code left in includes, which that patch is up to do. Gabor
On Thursday 22 November 2007, Gábor Hojtsy wrote:
On Nov 22, 2007 7:45 PM, Larry Garfield <larry@garfieldtech.com> wrote:
... As of beta 3, all core modules have been split up with the exception of locale, which is already split in a somewhat bizarre way by putting all of its page handlers into locale.inc. I'm not certain it's worth refactoring that when in beta 3 stage, when there wouldn't be any significant difference in the amount of code that's loaded. (Gabor, I defer to you on that one.)
I have commented on the issue with some immediate concerns, but none of the tools (my issues, issue search, etc) is good for people like me who comment on an enourmous amount of patches, so a direct patch URL would be nice :) Also, it needs some reroll, as we fixed quite some stuff around there, since it was submitted. I still think to complete the split, locale stuff would be great to move properly to modules/locale and only the really required include code left in includes, which that patch is up to do.
Gabor
I was asking more in general terms rather than about the specific patch. For locale, it would really just be code tidying, not a performance boost, so I wasn't sure if it was worth it in beta 3 state. If you say yes, though, I'll see about redoing it. (The last version of the patch sucked for various reasons.) -- Larry Garfield AIM: LOLG42 larry@garfieldtech.com ICQ: 6817012 "If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it." -- Thomas Jefferson
Hello all, I just wrote a blog post that makes reference to a post that popped up at the SUPPORT list that I believe is timely to your discussion here. More on Drupal and Usability http://lizasabater.com/more_on_drupal_and_usability Interesting enough, here's another post at the SUPPORT list that ends with words that echo the usability issues I feel are involved with the module bloat discussion :
BTW, it would be so easy to be able to move body and title (and taxonomy) in the CCK configuration screen (they have fixed weight)
http://lists.drupal.org/pipermail/support/2007-November/006752.html Cheers, / liza On 22.Nov.2007, at 01:22 PM, Khalid Baheyeldin wrote:
On Nov 22, 2007 4:41 AM, J-P Stacey <jp.stacey@torchbox.com> wrote: Greg Holsclaw wrote:
Drupal 6 will better optimize this issue by allowing module developers to better include only the needed files through the menu system, thus reducing overall memory usage without adding much overhead for opening more files. I am sure chx would give a better summary.
I was just thinking, what might be nice, then, is for there to be some standard way of componentizing a module, and bringing in bits of it based on the URL. I take it from the mention of menu that that's what D6 does?
Module splitting has happened as part of D6, mostly along the lines of admin stuff and regular stuff. However, it can be further tuned to be more granular, but it has to be on a module by module basis. No general rule can work for all possible permutations.
If you introduce a precompiler to minimize the file hits in such a system as Khalid mentions, would that nullify the effect you're after on the memory? Do precompilers typically work within the allocated memory of the apache processes and hence bloat them just the same?
Effectively, if you have a shared memory based op-code cache (e.g. APC), then yes, the splitting becomes moot. Remember that not all people can have an op-code cache (e.g. shared hosts), so there is benefit to those from splitting the modules.
Op-code caches do save memory, since the code is loaded ONCE, parsed, tokenized and stored as tokens. So, the memory footprint becomes less per Apache process.
Cheers, J-P -- J-P Stacey +44 (0)1608 811870 http://torchbox.com
-- Khalid M. Baheyeldin 2bits.com http://2bits.com Drupal optimization, development, customization and consulting.
On Nov 21, 2007 2:40 PM, Ryan Courtnage ☠ <ryan@courtnage.ca> wrote:
On Nov 21, 2007 12:00 PM, Michelle Cox <mcox@charter.net> wrote:
I have nearly 100 projects, many with multiple modules, and am not done adding, yet. It does really eat up memory like crazy but I haven't heard a solution to it. The fact is that some sites, especially social networking ones, use a lot of modules. Most of them are fairly small but needed. Would combining the smaller modules into one big one help any? Or would that still use the same amount of memory?
I have the same question. Does the number of modules enabled significantly affect memory usage?
Not only memory, but loading time. The php include_once() that is used to load modules is an expensive operatin. Both can be remedied (to a large degree) by using an op-code cache that caches in memory (such as what APC does by default, others require certain config options). There is an article on 2bits coming up soon on details. Stay tuned.
I tend to create tons of tiny modules, many of which do something simple - like provide a single hook. If the overhead from this approach is significant, then I need to change my ways!
Ryan
Michelle
On 11/21/2007 12:29:46 PM, Khalid Baheyeldin (kb@2bits.com) wrote:
(Changing the subject)
I regularly see 80+ to 110+, and that causes Apache to eat 100MB per process, which is really not good ...
This is the open buffet binge syndrome detailed here:
http://2bits.com/articles/server-indigestion-the-drupal-contributed- modules-open-buffet-binge-syndrome.html
On Nov 21, 2007 12:50 PM, Jim Li <jimmydami@gmail.com> wrote:
Cool, thanks!
It'd be interesting to have a poll on how many modules people use on a
social network site. I heard someone uses 160 modules at his dev
site,
it may go down a bit later, but it probably will still in the 100+ ragnge :)
On Nov 20, 2007 6:31 PM, Earl Miles <merlin@logrus.com> wrote:
On the other hand, having 150+ modules load *is* going to eat a whole lot of memory; so actually activating all these modules is a really bad idea.
And yes, the modules page under 4.7 is going to choke like [insert really bad sports team metaphor here].
Sean Robertson wrote:
As I understand it, Drupal 5+ no longer does that. That's what the
-- Khalid M. Baheyeldin 2bits.com http://2bits.com Drupal optimization, development, customization and consulting.
participants (10)
-
blogdiva＠culturekitchen.com -
Darren Oh -
Gerhard Killesreiter -
Greg Holsclaw -
Gábor Hojtsy -
J-P Stacey -
Khalid Baheyeldin -
Larry Garfield -
Michelle Cox -
Ryan Courtnage ☠