[drupal-devel] menu memory usage
Hi! Probably most of the devels have not noticed http://drupal.org/node/34755 "when I enable all modules in core, the menu array eats over 800K of RAM". As we now have menu-otf which will be badly abused and (almost) all nodes will be put in the menu hierarchy instead of stuff like About Us, FAQ etc, we can look ahead for situations where someone locks himself out of his drupal cold after a node submission because the menu tree will grow too large. I know this can not be solved for 4.7, it's too late, but let's discuss it. Regards NK
On Sun, 23 Oct 2005, Karoly Negyesi wrote:
Probably most of the devels have not noticed http://drupal.org/node/34755 "when I enable all modules in core, the menu array eats over 800K of RAM". As we now have menu-otf which will
I have noticed it of course.
be badly abused and (almost) all nodes will be put in the menu hierarchy instead of stuff like About Us, FAQ etc, we can look ahead for situations where someone locks himself out of his drupal cold after a node submission because the menu tree will grow too large. I know this can not be solved for 4.7, it's too late, but let's discuss it.
I'll set my memory allowance a bit higher and restart apache when that happens. ;p I actually doubt that this will happen. Having all nodes in a menu does not make too much sense as the menu will get either very long or very deep. Both things are not desirable. Cheers, Gerhard
Op zondag 23 oktober 2005 23:31, schreef Karoly Negyesi:
Hi!
Probably most of the devels have not noticed http://drupal.org/node/34755 "when I enable all modules in core, the menu array eats over 800K of RAM". As we now have menu-otf which will be badly abused and (almost) all nodes will be put in the menu hierarchy instead of stuff like About Us, FAQ etc, we can look ahead for situations where someone locks himself out of his drupal cold after a node submission because the menu tree will grow too large. I know this can not be solved for 4.7, it's too late, but let's discuss it.
Regards
NK
I love to edit my menus and do so for all my sites. However, I hardly evern tweak my admin menus. Would it be a good idea to cache that part. I mean: not load it in the tree, but cached/loaded in a *separate* block? * Navigation ** site navigation ** stuff * Administration ** Other stuff ** etc: The last block can be pulled from a cache and loaded only when needed (ie 1 - 20% of all page loads, or so). if administration pages are loaded more then that, you should get a hobby (or get some ads online), because you are editing more then that your site is read ;) Bèr -- PGP ber@webschuur.com http://www.webschuur.com/sites/webschuur.com/files/ber_webschuur.asc PGP berkessels@gmx.net http://www.webschuur.com/sites/webschuur.com/files/ber_gmx.asc
Op maandag 24 oktober 2005 00:07, schreef Álvaro Ortiz:
Another step in the separation of front and back.
No. I never proposed this with that in mind. Never! Just that it will speed up things. :) Bèr -- PGP ber@webschuur.com http://www.webschuur.com/sites/webschuur.com/files/ber_webschuur.asc PGP berkessels@gmx.net http://www.webschuur.com/sites/webschuur.com/files/ber_gmx.asc
Hello, recently, I have been experimenting with nested sets. Nested sets provide a rather simple technology to store huge (menu) trees in a database without having to loop through them using recursion. As recursion eats more time as the tree grows large and if there are more levels, it is not adviced to use this technology in relational databases. The nested set (or nested tree) works by adding two meta values to the record, one left value and one right value. Let's take the following menu structure: root [1/24] -- create content [2/7] -- page [3/4] -- story [5/6] -- my account [8/9] -- administer [10/23 -- access control [11/12] -- settings [13/20] -- content types [14/15] -- posts [16/17] -- users [18/19] -- users [21/22] For an extensive example, see http://www.phpriot.com/d/articles/php/application-design/nested-trees-1/inde.... The left and right values are in the square brackets, the first is the left one and the second the right one. The advantage is, that it is easy to obtain all necessary leafs with one single query. I don't like the current taxonomy system. I just want a tree where I can attach nodes to and let drupal build the complete navigation structure. Another disadvantage of the taxonomy system is also that building real breadcrumbs is very costly (due to recursion). Yesterday, I started a module (named ouline) that uses a nested tree instead of taxonomy to classify content. At the moment it is already capable of building a breadcrumb with one single query: SELECT node.nid as nid, node.title as title FROM outline AS selector CROSS JOIN outline AS outline ON (selector.lft BETWEEN outline.lft AND outline.rgt AND selector.tid = outline.tid) CROSS JOIN node AS node ON (outline.nid = node.nid AND node.status > 0 AND node.moderate = 0) WHERE selector.nid = <node id> AND outline.nid != selector.nid ORDER BY outline.lft If the outline table would be integrated within the node table, you even need to JOIN one table less. Another advantage is that the nested set does not mess around with "weight" propertys. There is only one single allowed order (that can be changed very easly though, for example using drag and drop with AJAX and regular buttons as fallback). Konstantin Käfer Karoly Negyesi schrieb:
Hi!
Probably most of the devels have not noticed http://drupal.org/node/34755 "when I enable all modules in core, the menu array eats over 800K of RAM". As we now have menu-otf which will be badly abused and (almost) all nodes will be put in the menu hierarchy instead of stuff like About Us, FAQ etc, we can look ahead for situations where someone locks himself out of his drupal cold after a node submission because the menu tree will grow too large. I know this can not be solved for 4.7, it's too late, but let's discuss it.
Regards
NK
you just described the general relation system :) read more on: http://drupal.org/node/34168 http://www.webschuur.com/relations_system http://drupal.org/node/28480 and have a look at the tables at http://drupal.org/node/28480#comment-42850 (and the commetn above it for some descriptions) Ber Op maandag 24 oktober 2005 12:30, schreef Konstantin Käfer:
Hello,
recently, I have been experimenting with nested sets. Nested sets provide a rather simple technology to store huge (menu) trees in a database without having to loop through them using recursion. As recursion eats more time as the tree grows large and if there are more levels, it is not adviced to use this technology in relational databases.
The nested set (or nested tree) works by adding two meta values to the record, one left value and one right value. Let's take the following menu structure:
root [1/24] -- create content [2/7] -- page [3/4] -- story [5/6] -- my account [8/9] -- administer [10/23 -- access control [11/12] -- settings [13/20] -- content types [14/15] -- posts [16/17] -- users [18/19] -- users [21/22]
For an extensive example, see http://www.phpriot.com/d/articles/php/application-design/nested-trees-1/ind ex.html.
The left and right values are in the square brackets, the first is the left one and the second the right one. The advantage is, that it is easy to obtain all necessary leafs with one single query.
I don't like the current taxonomy system. I just want a tree where I can attach nodes to and let drupal build the complete navigation structure. Another disadvantage of the taxonomy system is also that building real breadcrumbs is very costly (due to recursion). Yesterday, I started a module (named ouline) that uses a nested tree instead of taxonomy to classify content. At the moment it is already capable of building a breadcrumb with one single query:
SELECT node.nid as nid, node.title as title FROM outline AS selector CROSS JOIN outline AS outline ON (selector.lft BETWEEN outline.lft AND outline.rgt AND selector.tid = outline.tid) CROSS JOIN node AS node ON (outline.nid = node.nid AND node.status > 0 AND node.moderate = 0) WHERE selector.nid = <node id> AND outline.nid != selector.nid ORDER BY outline.lft
If the outline table would be integrated within the node table, you even need to JOIN one table less. Another advantage is that the nested set does not mess around with "weight" propertys. There is only one single allowed order (that can be changed very easly though, for example using drag and drop with AJAX and regular buttons as fallback).
Konstantin Käfer
Karoly Negyesi schrieb:
Hi!
Probably most of the devels have not noticed http://drupal.org/node/34755 "when I enable all modules in core, the menu array eats over 800K of RAM". As we now have menu-otf which will be badly abused and (almost) all nodes will be put in the menu hierarchy instead of stuff like About Us, FAQ etc, we can look ahead for situations where someone locks himself out of his drupal cold after a node submission because the menu tree will grow too large. I know this can not be solved for 4.7, it's too late, but let's discuss it.
Regards
NK Bèr -- PGP ber@webschuur.com http://www.webschuur.com/sites/webschuur.com/files/ber_webschuur.asc PGP berkessels@gmx.net http://www.webschuur.com/sites/webschuur.com/files/ber_gmx.asc
Konstantin describes a node hierarchy system powered by nested sets. Thats far more powerful than Ber's relations as described in those links. Nested sets allow you to express complex hierarchies with excellent performance. Thats not true with flat relations table. Many of us in Amsterdam discussed nested sets as a promising solution to our widespread neet for hierarchy. I encourage Konstantin and others to pursue this solution. Book.module is also excellent test case IMO. -moshe Bèr Kessels wrote:
you just described the general relation system :)
read more on: http://drupal.org/node/34168 http://www.webschuur.com/relations_system http://drupal.org/node/28480
and have a look at the tables at http://drupal.org/node/28480#comment-42850 (and the commetn above it for some descriptions)
Ber
Op maandag 24 oktober 2005 12:30, schreef Konstantin Käfer:
Hello,
recently, I have been experimenting with nested sets. Nested sets provide a rather simple technology to store huge (menu) trees in a database without having to loop through them using recursion. As recursion eats more time as the tree grows large and if there are more levels, it is not adviced to use this technology in relational databases.
The nested set (or nested tree) works by adding two meta values to the record, one left value and one right value. Let's take the following menu structure:
root [1/24] -- create content [2/7] -- page [3/4] -- story [5/6] -- my account [8/9] -- administer [10/23 -- access control [11/12] -- settings [13/20] -- content types [14/15] -- posts [16/17] -- users [18/19] -- users [21/22]
For an extensive example, see http://www.phpriot.com/d/articles/php/application-design/nested-trees-1/ind ex.html.
The left and right values are in the square brackets, the first is the left one and the second the right one. The advantage is, that it is easy to obtain all necessary leafs with one single query.
I don't like the current taxonomy system. I just want a tree where I can attach nodes to and let drupal build the complete navigation structure. Another disadvantage of the taxonomy system is also that building real breadcrumbs is very costly (due to recursion). Yesterday, I started a module (named ouline) that uses a nested tree instead of taxonomy to classify content. At the moment it is already capable of building a breadcrumb with one single query:
SELECT node.nid as nid, node.title as title FROM outline AS selector CROSS JOIN outline AS outline ON (selector.lft BETWEEN outline.lft AND outline.rgt AND selector.tid = outline.tid) CROSS JOIN node AS node ON (outline.nid = node.nid AND node.status > 0 AND node.moderate = 0) WHERE selector.nid = <node id> AND outline.nid != selector.nid ORDER BY outline.lft
If the outline table would be integrated within the node table, you even need to JOIN one table less. Another advantage is that the nested set does not mess around with "weight" propertys. There is only one single allowed order (that can be changed very easly though, for example using drag and drop with AJAX and regular buttons as fallback).
Konstantin Käfer
Karoly Negyesi schrieb:
Hi!
Probably most of the devels have not noticed http://drupal.org/node/34755 "when I enable all modules in core, the menu array eats over 800K of RAM". As we now have menu-otf which will be badly abused and (almost) all nodes will be put in the menu hierarchy instead of stuff like About Us, FAQ etc, we can look ahead for situations where someone locks himself out of his drupal cold after a node submission because the menu tree will grow too large. I know this can not be solved for 4.7, it's too late, but let's discuss it.
Regards
NK Bèr
On Mon, 24 Oct 2005 14:47:33 +0200, Moshe Weitzman <weitzman@tejasa.com> wrote:
Konstantin describes a node hierarchy system powered by nested sets. Thats far more powerful than Ber's relations as described in those links. Nested sets allow you to express complex hierarchies with excellent performance. Thats not true with flat relations table.
Many of us in Amsterdam discussed nested sets as a promising solution to our widespread neet for hierarchy. I encourage Konstantin and others to pursue this solution. Book.module is also excellent test case IMO.
Let me point out that Allie's hier module wants to use nested sets. See http://donorge.org/d_items/view/331 and http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/vauxia/hier/READM... Regards NK
Konstantin describes a node hierarchy system powered by nested sets. Thats far more powerful than Ber's relations as described in those links. Nested sets allow you to express complex hierarchies with excellent performance. Thats not true with flat relations table.
Many of us in Amsterdam discussed nested sets as a promising solution to our widespread neet for hierarchy. I encourage Konstantin and others to pursue this solution. Book.module is also excellent test case IMO.
I'm compiling a writeup on a variation on the nested set algorithm, which should help with frequently changed nested sets - to avoid frequent re-indexing. I think Aliie, has a prototype of a hirarchy module in her sandbox as well (vauxia?) I think it would merit thinking through an approach, where we use "specialised" indexes for different ordering types like linear, tree, graph, etc... Cheers, Vlado
On Oct 24, 2005, at 8:03 AM, vlado wrote:
Many of us in Amsterdam discussed nested sets as a promising solution to our widespread neet for hierarchy. I encourage Konstantin and others to pursue this solution. Book.module is also excellent test case IMO.
I think Aliie, has a prototype of a hirarchy module in her sandbox as well
Thanks for noticing! I've been trying to explain what Moshe and Konstantin describe succinctly - that relating items (not just nodes) is a universal problem that deserves a more comprehensive solution than what can be handled by an adjacency model. I've seen the adjacency model in a heavily-related system, and it wasn't good. The other concept in hier is that it's not all about nodes. I'm proposing a system that includes users, nodes, terms, even the website itself. And I'm proposing that we investigate permissions handling there too. I hope to be part of this conversation as it progresses, and I would love to spend more time on the hier module. As is often the case, I seem find myself with time OR resources, but never both.
I'm compiling a writeup on a variation on the nested set algorithm, which should help with frequently changed nested sets - to avoid frequent re-indexing.
I agree that thrashing on a heavy-update system would be the biggest problem with a nested set algorithm. My first modification was to introduce a depth field to be able to move about the tree more easily. More important are the inserts and deletes. There are proposals out there which "leave room", so the tree is only re-hashed every n modifications, which I hope will be useful.
I think it would merit thinking through an approach, where we use "specialised" indexes for different ordering types like linear, tree, graph, etc...
I hope we can try a few of these out and benchmark them against realistic data. Unless there are clear delineations for what works when, I'd lean to a "one size fits all" approach. Of course, I'm saying this without the benefit of context. I was really, really, REALLY bummed about not being able to make Amsterdam. Hopefully, financial constraints won't stand in my way next time there's a DrupalCON. I hope you all had a great time! Allie Micka pajunas interactive, inc. http://www.pajunas.com scalable web hosting and open source strategies
Not at all :)
I've been trying to explain what Moshe and Konstantin describe succinctly - that relating items (not just nodes) is a universal problem that deserves a more comprehensive solution than what can be handled by an adjacency model. I've seen the adjacency model in a heavily-related system, and it wasn't good. Well, this reminds me of the Medaimatic's presentation about AnyMeta - a CMS the guys developed and maintain. They were refering to everything as a "thing". Apart from the fact that that forced us to drink a lot of beer in short time, we realised that their "everything is a thing" eaquals the "everything is a node" theory circulating the Drupal community. I think that merits consideration. Apart from purely philosophical arguments - this concept has the benefit of simplifying things like relationships, classification, permissions, etc...
Relations between nodes (in the above context) can be explicit: * books for example Or implicit: * via taxo terms * search results * linking between pages Relations are defined within certain domains: * for example by different vocabularies * child-parent * part of Observations like that make me believe that relation should be considered a more general term - reflecting only fact that things (cheers) are somehow related to other things, and there maybe an order imposed in that retaionship. The exact structures behind the relationships should be indicated by relationship types - trees, graphs, whatever. This will allow using uniformly both explicit and implicit ones - treating similarly relations within a book, and realtions implied by the occurrence of different taxo-terms. As an example for the latter have a look at http://mediamatic.net Each different page has a central thing and related things. The guys calculate those reations it using a 'distance' metric based on the tags and maybe additional info. It has some really cool concepts behind it. Click on keywords related to a thing to see how the relation changes.
I hope to be part of this conversation as it progresses, and I would love to spend more time on the hier module. As is often the case, I seem find myself with time OR resources, but never both. Why do I have the same feeling.
I agree that thrashing on a heavy-update system would be the biggest problem with a nested set algorithm. My first modification was to introduce a depth field to be able to move about the tree more easily. Just drop the sequential numbering of indexes and use the whole integer or float number space, while preserving the ordering in the tree. This will require a far rarer re-indexing after inserts or updates.
I hope we can try a few of these out and benchmark them against realistic data. Unless there are clear delineations for what works when, I'd lean to a "one size fits all" approach. Of course, I'm saying this without the benefit of context. me too
Cheers, Vlado
On 24-Oct-05, at 3:30 AM, Konstantin Käfer wrote:
I don't like the current taxonomy system. I just want a tree where I can attach nodes to and let drupal build the complete navigation structure. Another disadvantage of the taxonomy system is also that building real breadcrumbs is very costly (due to recursion). Yesterday, I started a module (named ouline) that uses a nested tree instead of taxonomy to classify content. At the moment it is already capable of building a breadcrumb with one single query:
Outline sounds good. As noted elsewhere, looking at/working on Allie's hier.module is probably a good thing (chx has done some work, too, I think). I wanted to note that I believe that using the current taxonomy for outlines (taxonomy menu, etc.) is a bad practice (i.e. the reverse of a best practice). If it were me, I would recommend using a book outline...except we don't have book-outline-on-the-fly. Both breadcrumbs and menus get generated consistently and correctly when using a book outline. Cheers, -- Boris Mann http://www.bmannconsulting.com
Karoly Negyesi wrote:
Hi!
Probably most of the devels have not noticed http://drupal.org/node/34755 "when I enable all modules in core, the menu array eats over 800K of RAM". As we now have menu-otf which will be badly abused and (almost) all nodes will be put in the menu hierarchy instead of stuff like About Us, FAQ etc, we can look ahead for situations where someone locks himself out of his drupal cold after a node submission because the menu tree will grow too large. I know this can not be solved for 4.7, it's too late, but let's discuss it.
Yes, the menu system is memory intensive. I noticed it using the Zend Studio profiler. We are carrying a large array of data around from page to page, even though most of that array was not used on any given page. I think this is a good area to look at for improved performance. -- chrisxj
participants (10)
-
Allie Micka -
Boris Mann -
Bèr Kessels -
Chris Johnson -
Gerhard Killesreiter -
Karoly Negyesi -
Konstantin Käfer -
Moshe Weitzman -
vlado -
Álvaro Ortiz