- Development - lists.drupal.org

[drupal-devel] [feature] Improve functionality of block generation for book module
by andremolnar 20 Jan '05

20 Jan '05

Project: Drupal Version: cvs Component: book.module Category: feature requests Priority: normal Assigned to: nysus Reported by: nysus Updated by: andremolnar Status: patch Attachment: http://drupal.org/files/issues/book_module_3.patch (6.66 KB) Here is an updated patch. Same comments as followup 51 - except that the known issue has been resolved. This changes book module so that any action taken on a book, including adding new books or book pages will create a menu in the menu system for that book - and thus create blocks for those menus that can be administered in the usual ways. This is my first attempt at a major patch to core - comments are welcome I will be happy to update documentation once revisions to the code have been taken care of. andre andremolnar Previous comments: ------------------------------------------------------------------------ December 9, 2004 - 06:12 : nysus Attachment: http://drupal.org/files/issues/book_19.patch (4.85 KB) Attached is a patch for the book module that does the following: 1) Allows book blocks to appear on any page at any time, not just when a node from the book is being viewed. 2) Allows multiple book blocks to appear on the same page. This functionality is achieved by the automatic creation of individual blocks for each book when the book is created. Simply enable the book's block to enjoy the benefits of 1 & 2 above. If the blocks are not enabled, the blocks will appear only when a node from that block is being viewed (the same way it works now). ------------------------------------------------------------------------ December 9, 2004 - 12:35 : andremolnar Attachment: http://drupal.org/files/issues/book_19_1.patch (4.84 KB) +1 This is great. A good many people have asked for something like this, and I think its a nice solution. But in the end this isn't up to me. One minor error in the patch... + $result = db_query('SELECT n.title, b.block, b.nid FROM {book} b INNER JOIN {node} n on n.nid = b.nid WHERE b.parent = 0'); b.block is not a valid field in this query. this updated patch removes reference to it. andre ------------------------------------------------------------------------ December 9, 2004 - 13:54 : nysus Glad you like it and thanks for fixing that up. It was left over from an older version of my patch. ------------------------------------------------------------------------ December 9, 2004 - 13:54 : andremolnar Steve, If I apply this patch, and I attempt to configure one of the newly created blocks. I noticed that for some reason the block.module is returning "true" at line 249 (of the block module)- and is creating a form for module-specific configuration. But all that shows up on the screen is the word "array". Can you trace this back to see why - and maybe update the patch. I can continue to test your changes - anything to help this patch make it into core. andre ------------------------------------------------------------------------ December 9, 2004 - 15:29 : nysus Hmmm...probably because I tested my patch on my version of Drupal, version 4.5.1. I'm having no problems. Are you using a cvs version of Drupal to test. If so, I'll set up cvs on my site and track this down. ------------------------------------------------------------------------ December 9, 2004 - 18:53 : drumm See http://drupal.org/node/12347 for information on how the block system has been updated. When I saw that the elseif ($op == 'view') was taken out I knew immedaitely that there was something weird about this patch. ------------------------------------------------------------------------ December 9, 2004 - 18:58 : nysus OK, thanks for the tip. Sorry for the confusion. Still kind of new to making open source contributions and it's easy for me to overlook some obvious stuff like this. I'll fix this up when I get a chance. ------------------------------------------------------------------------ December 9, 2004 - 19:06 : Dries Please do, because this being a new feature, it has no chance getting committed to the DRUPAL-4-5 branch. The DRUPAL-4-5 branch is in bugfix-mode. New features go into CVS HEAD. ------------------------------------------------------------------------ December 9, 2004 - 22:32 : Dries There is quite a bit of duplicated code in the patch. Maybe it can be simplified (using a function)? Either way, it is a little weird. I haven't tried the path, and I'm not sure I understood the description. It's somewhat vague. Is the book module exporting multiple blocks that are nearly identical, yet have different display behavior? If so, why not add a simple block configuration setting to the original book block? ------------------------------------------------------------------------ December 10, 2004 - 02:51 : nysus Dries, Yes, there is some code that can be factored out and some general cleaning up that can be done. It was a little tricky to write so I left it kind of ragged around the edges until I'm sure there are no bugs. It has worked very well on 4.5.1 but I obviously need to update it to work with cvs. I'll be on that soon. As far as what it does: 1) For every new book that a user creates, a new block is associated with it. So if you create "Book A", "Book B", "Book C", you will get three new blocks visible on the block administration page called "Book A", "Book B", "Book C". The original "Book Navigation" block is still there, too. The functionality of the "Book Navigation" block is not affected at all by this patch. 2) If Book A's block is enabled, the block containing its menu will appear not only when a node within Book A is viewed, but at all times (unless the user suppresses it on certain pages with the "path" feature). When any node is that is NOT in Book A is viewed, Book A's menu still appears but it is fully collapsed. When a node that DOES belong to Book A is viewed, Book A's Book menu expands accordingly. 3) The user can also enable Book B & C's block, and have their menus appear in a block at all times as well. 4) If none of the book's blocks are enabled by the user, the module will behave just as it did without the patch. That is, when the "Book Navigation" block is enabled, the only time any book menu will appear is when a book node is being viewed. 5) It's important to note (and this was the tricky part to write) that if both the "Book Navigation" block is enabled and "Book A" is enabled, they will play nice with each other and not do nasty things like create the same book menu twice. ------------------------------------------------------------------------ December 10, 2004 - 05:51 : nysus Attachment: http://drupal.org/files/issues/book_20.patch (5.37 KB) Andre, Attached is a new patch that will resolve the problem of the block-specific stuff showing on the block configuration form. Let me know if you spot any other bugs. If it looks good to you, I'll go to work making the code look leaner and prettier. ------------------------------------------------------------------------ December 10, 2004 - 07:26 : nysus Attachment: http://drupal.org/files/issues/book_21.patch (4 KB) Dries, Andre: Here is a new and improved streamlined version of the patch. Have a look if you get a chance. ------------------------------------------------------------------------ December 10, 2004 - 08:11 : nysus Attachment: http://drupal.org/files/issues/book_22.patch (4.15 KB) Found a bug in the last version that would cause the block to jump to a different location. I think this should do it. Everything appear to work well (famous last words). ------------------------------------------------------------------------ December 10, 2004 - 14:49 : andremolnar Steve: bugs appear to be gone, and I didn't run across any other errors. This is functioning exactly as described. everyone: I personally would encourage support for this functionality. Book is a powerful navigation building tool in a site, not only with its ability to move next and back through a hierarchy of pages - but also its ability to build the appropriate navigation block without further user intervention (unlike the admin features in the menu module or taxonomy). The most frequently cited complaint about the book module is its inflexibility when it comes to when and where the block shows up. I also frequently hear requests for the ability to show multiple book blocks at the same time. Up until now the best alternatives suggested required users to do a hack (e.g. build a custom block that calls such and such a hook). Most abandon their request at that point because its over their heads. With this patch all those requests are covered and more. Now all books can automatically have their own block and admins can easily decide when and where each of those individual blocks show up (left right, up down) and coupled with the new configuration features of the block module - its very easy for admins to decide on which individual pages a block will show up. I would be interested what other have to say about this feature. My only reservation (which is minor compared to the benefits of the functionality offered) is that there is no way to turn this functionality off. i.e. The default behaviour is to build individual blocks for each book. If there could be a way to toggle this feature on and off somewhere - it would be perfect. Still, AS IS - this is a major improvement and offers great flexibility to admins and site creators. andre ------------------------------------------------------------------------ December 10, 2004 - 15:04 : Anonymous Andre, Thanks for the feedback on the usefulness of this module. Glad I could pitch in and help. I agree about the inability to turn the feature on/off and I was thinking about that myself. I think it could easily be accomplished by creating a checkbox in the "book navigation" block individual configuration's settings. Call it "Enable individual book blocks." When enabled, the individual book blocks will appear on the block administration menu. One question: Where would the state of this checkbox get saved? Has Drupal moved away from serializing data in the data base? ------------------------------------------------------------------------ December 10, 2004 - 18:36 : Dries I'm afraid that 'Enable individual book blocks.' is not descriptive/clear at all. Are you suggesting a setting to toggle between 'show block on all pages' and 'show block on book pages only'? ------------------------------------------------------------------------ December 10, 2004 - 19:45 : nysus Dries, No. Andre and I suggest a setting within the "Book Navigation" cofiguration page, that would toggle whether or not individual books appear on the list of all blocks on the block administration page. Hence the name 'Enable individual book navigation blocks.' The help text for this checkbox might read something like: "By default, a book's navigation block is visible only when a page from that book is being viewed. Check this box if you want more control over where and when an book's navigation block is visible. You will then be able to control the book's navigation block location and visibility settings on the "admin/block" page." Hope this makes it pretty clear. ------------------------------------------------------------------------ December 10, 2004 - 20:08 : Dries I understand what you are trying to do, but not how you are trying to do it, or how the setting is supposed to work. I guess I'll have to try it when a new patch lands. ------------------------------------------------------------------------ December 11, 2004 - 09:52 : nysus Attachment: http://drupal.org/files/issues/book_23.patch (5.02 KB) Alright, fellas, I'm proud to unveil my crowning achievement in the open source development world (no big deal to most of you guys but pretty good for a hack like me). Thanks for all the input so far. It's been helpful. I've streamlined the heck out of it per Dries suggestion and I've created an option to turn this functionality on an off per Andre's suggestion. Does this look good to you guys? Anything else I have to fix or improve? Thanks. ------------------------------------------------------------------------ December 11, 2004 - 13:02 : Dries I tried the patch. It works great but to me, the 'Give books their own block' settings seems to be redundant. Why not export the current book navigation block, along with an additional block for each book? Looks a lot simpler to me. I think I spotted a bug: orphaned book pages (or possibly book pages that are unpublished) appear to be getting book navigation blocks. ------------------------------------------------------------------------ December 11, 2004 - 17:34 : nysus I'll see if I can fix the bug. Might be tricky. But I don't understand your recommendation to "export the current book navigation block, along with an additional block for each book". Can you expand on this thought? ------------------------------------------------------------------------ December 11, 2004 - 18:06 : nysus Dries, I am unable to duplicate the bug. I have three orphaned pages. I also tried unpublishing some pages. But as far as I can tell, the patch works as expected. ------------------------------------------------------------------------ December 11, 2004 - 19:00 : Dries If you can't reproduce the problem, chances are my node/book table is somewhat fubar. As for the configuration option. I suggest removing it and to always make these new blocks available on the /admin/block configuration screen. ------------------------------------------------------------------------ December 11, 2004 - 19:52 : moshe weitzman I am hoping that we maintain the option to keep the behavior where the appropriate book block only shows up when its book page viewed. this is a nice, tidy arrangement. ------------------------------------------------------------------------ December 11, 2004 - 20:03 : nysus Yes, if you don't enable any of the individual book blocks, the a book's block (i.e. navigation menu) will only appear when a page in a book is viewed. In other words, you have the option to have the book block act like this patch isn't even installed. ------------------------------------------------------------------------ December 11, 2004 - 20:54 : Dries I guess I'll have to try the patch again, because I don't understand why it works like this -- or at least, why it can't be made simpler. ------------------------------------------------------------------------ December 11, 2004 - 21:13 : nysus Can you be more specific? Why it works like what? ------------------------------------------------------------------------ December 11, 2004 - 21:24 : nysus Attachment: http://drupal.org/files/issues/book_24.patch (4.1 KB) This patch reverses a change made in the last patch which required a user to enable individual book block before they could enable any individual book blocks. ------------------------------------------------------------------------ December 11, 2004 - 21:41 : andremolnar As I mentioned earlier I am *fine* with either version of this patch as long as it makes it to core. But, as I said earlier, I clearly think the preference for admins would be to have the option to enable or disable this functionality. BTW - if this does make it in, I would be happy to create a Handbook page that describes the new features - something like "how to build robust site navigation using the book module". (on or after December 17th). andre ------------------------------------------------------------------------ December 12, 2004 - 12:11 : Anonymous I am not at all happy with these features. They are incosistant, confusing and should use exising methods and UIs. May main concern is the incosistancy: it is confusing, will require extra attention with each core code change, adds extra logic to the core, and is not re-useable. so here are my questions: 1) why do we not use the menu system and apis to build and adminster the trees? Saves code, does not add extra UIs, and gives users more power. 2) why do we need that extra showing logic? a book block should not get exeptional if clauses, it should use the existing path setting methods on block admin. extra logic is confusing for administrators (hey, i set the path so that the book-block should show up here, but it does not, why?) we should really not provide extra logic in the block hook, but should rather use default settings in block admin (the book could fill the bookblock sql cells with custom paths, for example) 3) we should avoid extra UIs. We already have far too much, and far too much different ones. Please rather improve the block admin, than add new separate interfaces. 4) why do book blocks need al these expeptions in the first place? If they are so exeptional, we could consider not using blocks, but something else, like in-line book layout (pages with the index etc) 5) why did you not choose for a general, standard, block gerneation API? that way modules, such as taxonomy, image gallery, weblinks, article, etc can reuse it and introduce block gernation. All that sayd, i like the idea of this functionality, but i fear for a great useability downfall if we start introducing all sorts of exeptions for all sorts of modules. Because now chances are very big that taxonomy, image gallery etc will need to introduce other UIs, other code, new methods and new documentation, if they too want some sort of better block handling. so a -1 from me. ------------------------------------------------------------------------ December 12, 2004 - 12:12 : Bèr Kessels ^^--- That was me (bèr kessels) forgot to log in..... ------------------------------------------------------------------------ December 12, 2004 - 12:34 : nysus I'm not going to pretend I can argue if my patch does or does not fit well into Drupal's larger architecture. My motivation for writing it is that I had an immediate need to create an easy way to make it easy for users to create menus. I'll let others decide whether or not the patch has merit from the big picture perspective. But if it doesn't, why not just use it for its immediate benefits and then throw it away when something better comes along? ------------------------------------------------------------------------ December 12, 2004 - 13:01 : Dries My summary is this: +1 on the functionality, -1 on the implementation. The code itself is good, but the usability/integration is not. ------------------------------------------------------------------------ December 12, 2004 - 13:13 : nysus When you say "usability" is that from the user's perspective or the code maintainers? I'm guessing it's the latter but I'm unsure. What about the idea of using the patch until a more permanent solution comes along? Yes, it's much better to live in a home with indoor plumbing but why not use the outhouse while you wait for a toilet to get installed? Or are there other considerations I'm overlooking that would make this a bad idea? ------------------------------------------------------------------------ December 12, 2004 - 13:20 : Goba nysus it is just generally against the Drupal philosophy to add improperly implemented functionality until something better comes along. There even used to be ocassions in Drupal releases, when some functionality was removed (not fixed) for a release, because its implementation was not adequate. ------------------------------------------------------------------------ December 12, 2004 - 13:55 : Dries Usability for the user. The extra setting on the block configuration page is both confusing and awkward. I don't understand why things must be configured/enabled that way (see my and Ber's previous comments on this issue). ------------------------------------------------------------------------ December 12, 2004 - 14:03 : nysus Well, just for the record, I reversed that functionality per your suggestion and uploaded the patch. The indiviudal book blocks now appear automatically. ------------------------------------------------------------------------ December 13, 2004 - 13:56 : Bèr Kessels Hi, I am sure you can make not only simpler, but better usefull for admins and users. All you need to do is use the menus for this. i.e. make a menu entry for each book page. For each book make a menu on level 0, without a parent. that way they become a seoprate menu, each with an own block. it saves code, makes things more consistent, and most important, uses drupal functionality where it should. ------------------------------------------------------------------------ December 13, 2004 - 14:50 : nysus Ber, Are you suggesting that for every single book page that a menu item be created? That's really not practical. That was my main motivation for writing this patch: to make it easy to put links, not necessarily related to one another, into a block. Any more pages than 10 and the sheer tedium of the job would prevent anyone from ever doing that. The menu.module is great, but adding new menu items is far from quick and painless. I just added about 10 to my menu for different taxonomies on my site and it wasn't fun. Plus, if you do as you suggest, there is also the problem of the book showing up twice. It will be generated by the menu and then it will be generated again by the book module which is programmed to design a block. You'd have to put some logic in the book.module _block hook to try to anticipate if a user has enabled a book in the menu. That wouldn't be pretty code. I'm all for putting automatic generation of book navigation blocks as part of the menu module. It does make more sense to have it there. But it forces me to ask the question: "Then why do we currently have code in the book module that generates a menu? Shouldn't that belong in the menu.module, too?" ------------------------------------------------------------------------ December 13, 2004 - 22:31 : killes(a)www.drop.org I think what Ber is trying to say is that you can write a contrib module that monitors the changes to the book table and creates menu items automatically. nodeapi is your friend. I would also prefer this solution. ------------------------------------------------------------------------ December 15, 2004 - 16:39 : Anonymous "Then why do we currently have code in the book module that generates a menu? Shouldn't that belong in the menu.module, too?" Because it's old code. It would be nice if book.module generated this block using its _menu hook, so that the admin would have a few options in terms of configuration. Plus, if you do as you suggest, there is also the problem of the book showing up twice. It will be generated by the menu and then it will be generated again by the book module which is programmed to design a block. No. The old code which manually builds a block in book.module would be removed. Book blocks would only be generated by the menu. This would also have the added benefit of allow the administrator to easily place a book in an appropriate spot in the menu tree, while still allowing the possibility of displaying it in a separate block. Because of menu caching, I don't expect a large performance hit for creating the menu items. That was my main motivation for writing this patch: to make it easy to put links, not necessarily related to one another, into a block. It sounds like instead of (mis)using book.module, your time would be better spent in a usability improvement to menu.module so it's easier to do this. ------------------------------------------------------------------------ December 16, 2004 - 06:55 : nysus Thanks for the feedback and input. I appreciate it. However, I would also appreciate if you took more care to avoid the condescending tone in your post: It sounds like instead of (mis)using book.module, your time would be better spent in a usability improvement to menu.module so it's easier to do this. It's really quite unnecessary and off-putting. Though it won't stop me from contributing to Drupal in the future, I'm sure others would be really turned off by such a patronizing comment and it could dissuade them. I'd like the Drupal community to be a welcoming and friendly place that will inspire people to contribute, not discourage them. Thanks. ------------------------------------------------------------------------ December 16, 2004 - 10:07 : Bèr Kessels nyesus, Please do not start /that/ discussion here. :) Drupal community is known fo being direct, maybe because of the big number of western-Europeans attending, maybe because of other reasons. No-one commented that you are wasting your time. But the commentor was telling you somthing likethis: "If you would follow the previous sugeestions, your added feature would be much better appreciated, and will probably work much better for you too". He/she was by no means telling you to stop your silly coding, or anything in that line. He/She only wanted to show you the obvious and better direction. We often deal with issues that add some feature, and a complete new UI, because the author does not like, or cannot use the existing UIs and features. This is nogt good, because if that same author would have spend his/her time on improving that existing functionality or UI (improving is not neccesarily the same as extending!!) that code and time would benefit all much better. Thats what the commentor tried to say. And so is it here: If you dislike the way the books handle the blocks, and if you do not want to use the menu, because you do not like its UI, then do not add another UI and more features, but rather merge these, and improve the parts (in the menu) you dislike. ------------------------------------------------------------------------ December 16, 2004 - 10:35 : Dries We can worry about the menu integration later. Let's focus on the new option's usability/interaction design first. ------------------------------------------------------------------------ December 16, 2004 - 14:51 : nysus I understand what the commenter was saying and like I said, I appreciate and understand it. I'm not upset and I'm not looking for an argument, I was just being direct as well. :) As part of the Drupal community (albeit a minor player), all I'm saying is that I would like to see folks not have a tin ear to the humanity in all of us. It will help make Drupal an even stronger community and attract even more talented developers. Human interaction is part of the development process. Whether we like it or not, we must cope and deal with it. ------------------------------------------------------------------------ December 18, 2004 - 09:54 : Dries I thought some more about this and am starting to believe that integration with the menu system should take priority. Here are common cool scenario's: I want to create a separate navigation block for the 'Drupal handbook'. I want to add a menu item called 'handbook' to the user block. That is, I want the book navigation to be part of the existing user navigation block. I want to add a menu item called 'handbook' to the top-level navigation menu. How would that work from a user's point of view? What do I have to click and where to configure this? ------------------------------------------------------------------------ December 21, 2004 - 18:47 : andremolnar I was actually thinking about the same thing last night (must have been something in the arctic air). 2. I want to add a menu item called 'handbook' to the user block. That is, I want the book navigation to be part of the existing user navigation block. 3. I want to add a menu item called 'handbook' to the top-level navigation menu. This is already possible (to a certain extent) with the current Book.module and Menu.module - A 'Books' menu item is created in the user navigation by having the Book.module enabled. Menu.module allows you to enable/disable this menu item. Menu.module also allows you to re-name this menu item. But, this only helps if you only intend to have 1 book (or else the name 'Handbook' is misleading if the user finds more than 1 book listed). - so this isn't good enough (or is it). 1. I want to create a separate navigation block for the 'Drupal handbook'. This is what I was trying to figure out. Not just this, but a different way to do what Nysus was attempting. i.e. create a menu block for each and every book that is created. There is a way to write code that would (re)build a 'custom menu' on every add/edit/update to a book page - or book outline update. Custom menu's automatically have a block created for them. But, this I think would be a misleading use of the 'custom' menu - as the menu would not be custom if they are a part of core. So, I would think that two new constants could be added to the menu.inc file - MENU_BOOK_MENU - and MENU_BOOK_ITEM - each would behave as custom menu's, but would be reserved for books. These menu types should NOT show up in the Menu.module administration - because the administration of the book_menu items would be done by administering the book itself (adding an item, removing an item, moving an item up/down in the hierarchy, assigning parents etc.). However, the blocks that the book_menus would create would show up in the block administration (so users could enable/disable each block - and decide where on the site they show up). The existing book block logic would be removed. So the logic would be: If a creates a book in the book administration page - the Book.module automatically creates a new MENU_BOOK_MENU Any time a user adds to or updates or delets a book page - the entire book menu is deleted and recreated based on the hierarchy defined by the book itself. The blocks for each book would show up in block administration. Any thoughts - I know that this doesn't exactly address points 1 and 2 - but it could act as a first step. Is this approach a bad idea? It would add special conditions for the proposed book menu's - but books would be a special case. Even if people don't like this solution, maybe it will give someone a better idea. I'd love to hear them. andre ------------------------------------------------------------------------ December 21, 2004 - 22:55 : Boris Mann +1 to this andre I had promised to put down my thoughts on this matter, as it relates to 1) primary and secondary links and how they are managed and 2) auto-generation of primary/secondary navigation based on book outlines So, for 1), we currently have functional-but-not-very-usable plain textfields to manage primary and secondary links. I would like to see menu.module used to control all navigation links, whether it is the navigation block or primary/secondary links. What is needed: a) default system menus labelled "primary" and "secondary" which would store; this is where modules could insert navigation b) support for full URLs (e.g. http://myexternaldomain.com) instead of just path 2) if we got 1 working, and andre does his book menus, this could get taken care of automatically. Basically, for brochureware/business/etc. sites that have static content, you could have a root book as one of the primary navigation links, and then the secondary links are generated automatically. ------------------------------------------------------------------------ December 22, 2004 - 21:02 : Dries I agree with Andre that the book module's integration with the menu system needs to be worked on. I support any effort that makes it easier to structure pages and that makes it easy to link pages/nodes from within a menu. However, I'm opposed to putting book module specific names in the menu system (eg. MENU_BOOK_MENU and MENU_BOOK_ITEM). I can imagine a handful of modules that want to maintain a menu tree (or part thereof) so I'm in favor of generic names. I'd have to read up on the menu system code, but last time I checked there was a MENU_MODIFIABLE_BY_ADMIN flag. You could choose not to set this flag for the menu items generated by the book module. Maybe it's already possible to implement to implement Andre's suggestion without having to modify/extend the menu system. Are you exploring this path? ------------------------------------------------------------------------ December 24, 2004 - 10:01 : andremolnar I've had a few spare hours to work on this. I've started to cobble together a solution - but in doing so I discovered a bug in the menu module (for which I will submit a seperate issue - if one doesn't already exist). The principal I suggested works. I added some code to the book module that creates custom menu's (MENU_CUSTOM_MENU with MENU_CUSTOM_ITEMs) for each Book that exists in a site. This is just as a proof of concept - I chose this menu type to start with because they automatically have a block created for them in admin/block (which is ultimatly the functionality we want). I ran a test and the menus are created as expected - the blocks are also created. But when I tested the menu blocks by enabling them I ran into a problem. It seems as though the menu system does not expand/explode sub menu items if the node type is book. I'm not sure why this is, and I haven't traced the source code yet - I thought I would ask if anyone has intimate knowledge of the menu system if they can point me in the right direction. No patch attached because until that problem is fixed this proposed change won't fly :( andre ------------------------------------------------------------------------ December 24, 2004 - 11:11 : Dries Just a wild guess: maybe it doesn't work because the book module's URI scheme is not hierarchical? ------------------------------------------------------------------------ January 7, 2005 - 10:45 : andremolnar Attachment: http://drupal.org/files/issues/book_module_1.patch (5.1 KB) I finally took some time out to do some work on this. As mentioned in a post to dev the problem I was having with menus not expanding/contracting properly was some crud in my database. Once that was cleared up my changes worked fine. I am attaching a patch for comments and for the brave willing to give it a test drive. History: This thread and http://drupal.org/node/15198 and http://drupal.org/node/15153 node/15198 has a patch that is required for this patch to work. What this does: Pretty straight forward. Any time a user adds/updates/deletes book or book pages (including via outline) - a function is called to create a new menu for each book. Any existing book menus are wiped out and then the new book menus are inserted - and the system menu is rebuilt to reflect the changes. The menus created consist of type MENU_MODULE_MENU and MENU_MODULE_ITEM. These menus show up in the menu/admin page so that admins can be aware of them, but the menu types are not editable via menu/admin (all changes are handled by the book module). Since the menus are in the menu table, the menu_block() hook handles the creation of the blocks for each of them. The blocks can then be administered in the usual ways via the block/admin interface. KNOWN ISSUE: (suggested solutions welcome) Since the menu table is updated on every change to books - the blocks associated with the menus are also recreated with default settings (i.e. disabled, and with no path or throttle settings) requiring a user to take an extra step and re-configure the book blocks for viewing on their site. I think this is unacceptable. For the casual user of the book module that only has a single book that doesn't change often, this would not be a big deal. But, if anyone wanted to make use of book module to handle dynamic site navigation this would create more work than it saves. Looking forward: If the block generation issue can be solved in a tidy way, this patch could allow users to use book to handle all their site navigation generation needs. Also, this patch could allow for the removal of a large chunk of code in the book module dedicated to building its own block via the block system. I left it there for now because I suppose there may be those out there that want to have book.module work the way book.module always worked (only show the book block when viewing a book page). Even so, since each book would have its own block, an admin could specify when and where the block shows via admin/block. I would appreciate feedback. If nothing else I hope this gives someone some new ideas. I'll continue to work on the block regeneration issue. andre ------------------------------------------------------------------------ January 7, 2005 - 10:47 : andremolnar Attachment: http://drupal.org/files/issues/book_module_2.patch (4.98 KB) sorry - here's a patch with prefered line breaks. andre ------------------------------------------------------------------------ January 7, 2005 - 16:47 : nysus Andre, I've been meaning to give this a throrough look when I get a chance. Hopefully this weekend. ---Steve -- View: http://drupal.org/node/14120 Edit: http://drupal.org/project/comments/add/14120

1 0

[drupal-devel] [task] Put revisions in their own table
by killes 20 Jan '05

20 Jan '05

Project: Drupal Version: cvs Component: node system Category: tasks Priority: normal Assigned to: killes(a)www.drop.org Reported by: killes(a)www.drop.org Updated by: killes(a)www.drop.org Status: patch - database upgrades for the core modules with an own table - contrib modules need an upgrade too. - do we need nid and vid in both the node and the node_revisions table? - the amount of sql queries means a good stress testing for large databases. These issues are still open, btw. Especially the first one needs to be tackled. killes(a)www.drop.org Previous comments: ------------------------------------------------------------------------ May 5, 2004 - 17:25 : killes(a)www.drop.org Currently all node revisions are stored in a serialized field in node.table and retrieved for _each_ page view although they are rarely needed. However, we have agreed that serializing data is bad and that we should try to keep the memory foot print pf Drupal small. Therefore I propose to create a separate revisions table which would be in principle identical to the node table, only that it could have several old copies of the same node. Extra data added by other modules could be added in a serialized field unless we find a better solution. ------------------------------------------------------------------------ May 5, 2004 - 18:06 : jhriggs I too think the serialized approach is less than desirable, but here's an alternative. This would likely take some considerable rework in core and contrib, but the following is how we handle similar types of situations in our databases at work. It is more elegant that a separate table, and avoids the (almost exact) duplication of a table. Instead of separate tables, keep all revisions of nodes in the node table as follows: * add field: active (0/1 or Y/N) * add field: revision * every revision of a node is stored in the node table; however, only one revision can be active at any given time * nid can no longer be unique -- primary/unique key becomes (nid, active) * any time a node is loaded, updated (without revision), etc., the active version is used. Thoughts? ------------------------------------------------------------------------ May 5, 2004 - 18:57 : killes(a)www.drop.org I am not opposed to your scheme, but I want to stress the following: * Duplicating a table's structure is not bad (IMHO) as long as the content is different. * having two tables will allow us to have a rather small node table. This is (maybe) a performance gain. ------------------------------------------------------------------------ May 5, 2004 - 19:37 : jhriggs I don't necessarily think that duplicating a table's structure is _bad_. It just seems to be wasteful and a pain to maintain. (Every change to the node table is made twice...easy to do, but also easy to miss perhaps.) As for performance, as long as nid and the active indicator are indexed, there shouldn't be any performance loss. Also, archiving an old version when making a new revision will be much simpler: just change the active indicator rather than copying an entire node to another table (and ensuring everything gets copied...again a potential maintainance issue). To be honest, I would just like to see the serialized data go away, regarless of what approach is taken. ------------------------------------------------------------------------ July 30, 2004 - 20:49 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_07-30-2004.p… (10.47 KB) I'm interested in using Drupal for a large scale wiki-type project. In order to do this, I need revisions to be in their own table. Attached is a patch to do just that. Most of the changes are pretty self explanatory. Spreading out node data across two tables meant that I had to add database functions to do locking/transactions. Without this, race conditions in which the database becomes corrupted are possible. ------------------------------------------------------------------------ July 30, 2004 - 20:54 : Nick Nassar Oh yeah... The patch is a diff against Drupal CVS ------------------------------------------------------------------------ July 31, 2004 - 01:00 : Anonymous Gerhard speaking. Nick, thanks a lot for your nice patch! It saves me a great deal of labour. I looked through it and immediately liked it. You not only put the old revisions into a new table but also the current one. Do you have an estimate how much more expensive the additional join is? Besides a few minor coding style issues I found a major one: Just a few hours before you uploaded your patch JonBob's node access patch hit core. That means your patch won't apply anymore as all the queries you change have been changed. Can I bug you to update your patch? ------------------------------------------------------------------------ July 31, 2004 - 02:11 : Anonymous Also I think that your upgrade path loses existing revisions. ------------------------------------------------------------------------ July 31, 2004 - 03:39 : drumm I think this is the proper way to do things. No columns are duplicated, there is no serialized data, and only the fields that are logically revised are stored. Nothing jumped out at me as a way to have my node modules be able to keep a table of revisions of additional fields. I'm guessing this could be done within the confines of _insert and _update. Assuming the upgrade path works and modules can extend it I give it a +1. ------------------------------------------------------------------------ July 31, 2004 - 15:40 : Nick Nassar It figures that just as I finish a big patch, another patch comes along and breaks it. Oh well, it should be a pretty easy to fix. I'll work on it. Fixing the upgrade path to keep revisions should be fairly painless. I found another issue that needs to be fixed before this patch gets merged. There format of a node needs to be stored for each revision. Otherwise, for modules that store a format for the nodes, such as page and book, if you write one revision in PHP and the next in HTML, the PHP revision will be displayed as HTML. This is part of a larger issue of how node modules should store revisions of additional fields. I think each module that wants to do this should create another table with (nid, revid) as the primary key. Just as when they want to add fields to a node they create another table with nid as the primary key. As far as performance goes, for sites that make heavy use of revisions, an extra join on primary keys is going to be a lot faster than grabbing all of the revisions from that database everytime. We would need to run benchmarks to determine is the overall difference in speed is for an average site is a gain or a loss. I'm guessing it's very minor either way. ------------------------------------------------------------------------ August 23, 2004 - 14:55 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_08-23-2004.p… (10.92 KB) Here's an updated patch against CVS that puts revisions in their own table, provides an upgrade path, and fixes the format related bugs in the last patch. Hopefully, this can make it into CVS as soon as the freeze is over. ------------------------------------------------------------------------ August 23, 2004 - 15:10 : moshe weitzman Interesting patch ... drumm's question is still outstanding. how do modules store revisions of their fields? Are they expected to manage this on their own? Thats not how it works today. As an aside, i am seeing profile_ fields in my node.revisions column. One could argue that those need not be saved. They pertain to the node author, not to the node itself. ------------------------------------------------------------------------ August 23, 2004 - 17:14 : Nick Nassar Having modules be responsible for storing revisions of their own fields is a side-effect of storing revision data in tables. There's really no way around it. However, revisions generally don't make sense for node types that don't have PHP/HTML content, such as polls. I think it's going to be a pretty rare scenario for a new node type to want another field to change per-revision, so it's a pretty good trade-off. Storing fields that shouldn't be part of revisions, such as the profile_ fields, is a side-effect of storing revisions as serialized objects. Applying this patch will free up that wasted space. :) ------------------------------------------------------------------------ August 23, 2004 - 18:20 : Anonymous There should be a hook that let's the module choose whether it supports history. This way a module author can prevent the user from doing something that may break his module or just cause undefined behavior. If the module doesn't support history then don't let the user/admin choose to add history to nodes of that type. Craig ------------------------------------------------------------------------ August 23, 2004 - 20:23 : Nick Nassar I agree, there should be an API change to make specifying support for revisions easier. In the interests of keeping patches small and keeping to one change per patch, I think the API change should be a separate issue. A sort of ad-hoc API to decide whether or not a module supports revisions by default already exists. Instead of having a hook, modules set the default value of the "Create new revision" field in the edit form. The admin can change this option in admin/node/configure/defaults. This patch doesn't change that. Revisions are broken for node types that have their own database structure, like polls, even when storing them as serialized objects. This patch doesn't change that, either. ------------------------------------------------------------------------ October 26, 2004 - 03:35 : moshe weitzman I'm guessing that someone is going to have to demonstrate that this patch performs as well as current drupal before it gets comitted. i think this patch is a few benchmarks from being comitted. ------------------------------------------------------------------------ October 27, 2004 - 02:04 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004.p… (11 KB) I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ October 27, 2004 - 02:05 : Nick Nassar I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ October 27, 2004 - 02:05 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004.p… (11 KB) I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ November 15, 2004 - 06:05 : elias1884 please overthink the revision system default workflow as well. don't look at the revision system as an isolated system but as a part of the whole workflow system! if you combine revisions with the moderation queue the most logic default workflow would be like that: auth user creates node (revision #0) admin approves the node (status = 1, moderation = 0) => node publicly available auth user finds typo and changes node (revision #1, status = 0, moderation = 1) ------- what happens at that point at the moment is, that the node is not accessible anymore at all until the new revision is approved by admin. of course the new revision should not go online until reviewed and approved, this is absolutely correct, but there is no reason to not take the old revision offline, since it was already approved and should therefore be online until the new revision is approved. it is not practical if a node disappears only because the author corrected a typo. ------- admin approves the node (status = 1, moderation = 0) eventhough I first thought a plain boolean active field would not be capable of providing that functionality if finally came to the conclusion, that it can. The only thing to do is to not set that bit, when a new revision is created, but when it is approved (in case moderation is activated under default workflow). Every revision should have its own moderation, status and active field and on approval they are set like this (status=1, moderation=0, active=Y). When you wanna rollback to an old revision, you can chose between all revisions that already have the moderation bit set back to 0 again and the published to 1. There should be an extra permission for rollback! another concern that I have about the default workflow is, that users can't see the content, they have just created, when moderation is enabled. Eventhough, there is a big fat "submission accepted" presented after submissions, unexperienced users tend to question the information those stupid tincans give them, if they can't find their content afterwards. Many users are really lazy bastards and they don't even read the status messages. The best feedback about whether his story was submitted successfully or not of course is, if he can find the story somewhere on the site, maybe with a status message on top of it, mentioning, that the content is currently not publicly available since it has not been approved yet. there should be a my content section under my account, like somebody is trying to do with the workspace module I guess. so my suggestion is to make (status=0, moderation=1) still available for the creator under a my content section somewhere! ------------------------------------------------------------------------ November 24, 2004 - 05:21 : Nick Nassar I agree. The current workflow for moderation queues and revisions needs to change, but this patch isn't the place for it. The patch is already too big, and it only does the backend stuff. Instead of adding more to this patch and making it take even longer to get into core, would you mind creating a new issue for your UI suggestions, so the those changes can be added as a separate patch? Thanks, Nick ------------------------------------------------------------------------ December 11, 2004 - 13:26 : Dries This patch is _much_ needed so I'd love to see someone revive it. In order for this patch to be accepted, the following needs to be done: Update this patch to CVS HEAD. Rename revid to vid. Rename node_rev to node_revisions. Rename node_rev.changed to node_revisions.timestamp. Rename $rnode to $revision. Fix the coding style to match Drupal's: proper spacing, single quotes where possible, proper variable names. Benchmark this patch with a large database with enough revisions. I'd be happy to benchmark this on my local copy of the drupal.org database. The book.log field should probably move to the node_revisions table. This can be done in a separate patch. Investigate whether transactions are well-supported. ------------------------------------------------------------------------ December 13, 2004 - 01:25 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004-r… (11.02 KB) I've worked a bit on the patch (coding style issues as mentioned by Dries). One thing I noticed is that the patch uses REPLACE. IIRC this needs to be chagned to "UPDATE, if fail INSERT" for pgsql compatibility. Nick, are you still interested in working on that patch? I'd like to know how it works on your site and work on getting it into core. ------------------------------------------------------------------------ December 13, 2004 - 13:33 : Dries Gerhard: your patch does not apply. ------------------------------------------------------------------------ December 13, 2004 - 18:10 : killes(a)www.drop.org Yes, I know, that was the same version as I mailed to you earlier. ------------------------------------------------------------------------ December 13, 2004 - 22:02 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions.patch (52.96 KB) Ok, upüdated the patch to cvs. ------------------------------------------------------------------------ December 14, 2004 - 09:58 : Dries Some more comments: db_begin_transaction() and db_end_transaction() do not belong in database.inc, but in database.mysql.inc and database.pgsql.inc respectively. The node module calls node_revisionsision_list() which is not defined. (Fxed that on my local copy.) Do db_begin_transaction() and db_end_transaction() deprecate Jeremy's table locking patch? The upgrade path assigns the wrong user ID to each revision. The upgrade path assigns the wrong date to each revision (that or a node's revision page shows the wrong usernames/dates). The coding style needs a bit of work, but we can worry about that later. ------------------------------------------------------------------------ December 14, 2004 - 18:34 : Nick Nassar If you need any help getting those things fixed, just let me know. ------------------------------------------------------------------------ December 14, 2004 - 18:50 : Nick Nassar How this relates to Jeremy's node locking patch: There was lots of discussion, and node locking was decided against because from an end user point of view you never want a node to be locked. He's now advocating for a much simpler patch that warns users if their changes will overwrite someone elses. That patch still has a race condition, which might be fixed using db_begin_transaction(). http://drupal.org/node/6025 ------------------------------------------------------------------------ December 14, 2004 - 23:26 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_0.patch (55.96 KB) Here is an updated patch that tries to address Dries concerns. ------------------------------------------------------------------------ December 15, 2004 - 09:32 : Dries Attachment: http://drupal.org/files/issues/revisions-bug.png (76.06 KB) It didn't fix the aforementioned bugs. See attached screenshot. ------------------------------------------------------------------------ January 6, 2005 - 21:15 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_1.patch (51.77 KB) Ok, here is a new version. Dries and myself worked hart at it, so please have a look. what is still missing - database upgrades for the core modules with an own table - contrib modules need an upgrade too. - do we need nid and vid in both the node and the node_revisions table? - the amount of sql queries means a good stress testing for large databases. ------------------------------------------------------------------------ January 19, 2005 - 22:43 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_2.patch (49.49 KB) Here is an updated patch. We discussed to keep the current title in node module and also in the revisiosn table. This is content duplication but will save many joins as many queries only need the title of a node. Discussion is welcome. ------------------------------------------------------------------------ January 20, 2005 - 00:33 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_3.patch (29.93 KB) I've implemented the aforementioned solution. This makes the patch much smaller. The patch now also removes taxonomy_node_has_term() which wasn't used anywhere. I'd really apprciate if some people could test drive the patch. It will be another huge improvement for 4.6. ------------------------------------------------------------------------ January 20, 2005 - 01:05 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_4.patch (30 KB) Another revision. Steven didn't like my literal $node->vid in queries. -- View: http://drupal.org/node/7582 Edit: http://drupal.org/project/comments/add/7582

1 0

[drupal-devel] [task] Put revisions in their own table
by killes 20 Jan '05

20 Jan '05

Project: Drupal Version: cvs Component: node system Category: tasks Priority: normal Assigned to: killes(a)www.drop.org Reported by: killes(a)www.drop.org Updated by: killes(a)www.drop.org Status: patch Attachment: http://drupal.org/files/issues/revisions_4.patch (30 KB) Another revision. Steven didn't like my literal $node->vid in queries. killes(a)www.drop.org Previous comments: ------------------------------------------------------------------------ May 5, 2004 - 17:25 : killes(a)www.drop.org Currently all node revisions are stored in a serialized field in node.table and retrieved for _each_ page view although they are rarely needed. However, we have agreed that serializing data is bad and that we should try to keep the memory foot print pf Drupal small. Therefore I propose to create a separate revisions table which would be in principle identical to the node table, only that it could have several old copies of the same node. Extra data added by other modules could be added in a serialized field unless we find a better solution. ------------------------------------------------------------------------ May 5, 2004 - 18:06 : jhriggs I too think the serialized approach is less than desirable, but here's an alternative. This would likely take some considerable rework in core and contrib, but the following is how we handle similar types of situations in our databases at work. It is more elegant that a separate table, and avoids the (almost exact) duplication of a table. Instead of separate tables, keep all revisions of nodes in the node table as follows: * add field: active (0/1 or Y/N) * add field: revision * every revision of a node is stored in the node table; however, only one revision can be active at any given time * nid can no longer be unique -- primary/unique key becomes (nid, active) * any time a node is loaded, updated (without revision), etc., the active version is used. Thoughts? ------------------------------------------------------------------------ May 5, 2004 - 18:57 : killes(a)www.drop.org I am not opposed to your scheme, but I want to stress the following: * Duplicating a table's structure is not bad (IMHO) as long as the content is different. * having two tables will allow us to have a rather small node table. This is (maybe) a performance gain. ------------------------------------------------------------------------ May 5, 2004 - 19:37 : jhriggs I don't necessarily think that duplicating a table's structure is _bad_. It just seems to be wasteful and a pain to maintain. (Every change to the node table is made twice...easy to do, but also easy to miss perhaps.) As for performance, as long as nid and the active indicator are indexed, there shouldn't be any performance loss. Also, archiving an old version when making a new revision will be much simpler: just change the active indicator rather than copying an entire node to another table (and ensuring everything gets copied...again a potential maintainance issue). To be honest, I would just like to see the serialized data go away, regarless of what approach is taken. ------------------------------------------------------------------------ July 30, 2004 - 20:49 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_07-30-2004.p… (10.47 KB) I'm interested in using Drupal for a large scale wiki-type project. In order to do this, I need revisions to be in their own table. Attached is a patch to do just that. Most of the changes are pretty self explanatory. Spreading out node data across two tables meant that I had to add database functions to do locking/transactions. Without this, race conditions in which the database becomes corrupted are possible. ------------------------------------------------------------------------ July 30, 2004 - 20:54 : Nick Nassar Oh yeah... The patch is a diff against Drupal CVS ------------------------------------------------------------------------ July 31, 2004 - 01:00 : Anonymous Gerhard speaking. Nick, thanks a lot for your nice patch! It saves me a great deal of labour. I looked through it and immediately liked it. You not only put the old revisions into a new table but also the current one. Do you have an estimate how much more expensive the additional join is? Besides a few minor coding style issues I found a major one: Just a few hours before you uploaded your patch JonBob's node access patch hit core. That means your patch won't apply anymore as all the queries you change have been changed. Can I bug you to update your patch? ------------------------------------------------------------------------ July 31, 2004 - 02:11 : Anonymous Also I think that your upgrade path loses existing revisions. ------------------------------------------------------------------------ July 31, 2004 - 03:39 : drumm I think this is the proper way to do things. No columns are duplicated, there is no serialized data, and only the fields that are logically revised are stored. Nothing jumped out at me as a way to have my node modules be able to keep a table of revisions of additional fields. I'm guessing this could be done within the confines of _insert and _update. Assuming the upgrade path works and modules can extend it I give it a +1. ------------------------------------------------------------------------ July 31, 2004 - 15:40 : Nick Nassar It figures that just as I finish a big patch, another patch comes along and breaks it. Oh well, it should be a pretty easy to fix. I'll work on it. Fixing the upgrade path to keep revisions should be fairly painless. I found another issue that needs to be fixed before this patch gets merged. There format of a node needs to be stored for each revision. Otherwise, for modules that store a format for the nodes, such as page and book, if you write one revision in PHP and the next in HTML, the PHP revision will be displayed as HTML. This is part of a larger issue of how node modules should store revisions of additional fields. I think each module that wants to do this should create another table with (nid, revid) as the primary key. Just as when they want to add fields to a node they create another table with nid as the primary key. As far as performance goes, for sites that make heavy use of revisions, an extra join on primary keys is going to be a lot faster than grabbing all of the revisions from that database everytime. We would need to run benchmarks to determine is the overall difference in speed is for an average site is a gain or a loss. I'm guessing it's very minor either way. ------------------------------------------------------------------------ August 23, 2004 - 14:55 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_08-23-2004.p… (10.92 KB) Here's an updated patch against CVS that puts revisions in their own table, provides an upgrade path, and fixes the format related bugs in the last patch. Hopefully, this can make it into CVS as soon as the freeze is over. ------------------------------------------------------------------------ August 23, 2004 - 15:10 : moshe weitzman Interesting patch ... drumm's question is still outstanding. how do modules store revisions of their fields? Are they expected to manage this on their own? Thats not how it works today. As an aside, i am seeing profile_ fields in my node.revisions column. One could argue that those need not be saved. They pertain to the node author, not to the node itself. ------------------------------------------------------------------------ August 23, 2004 - 17:14 : Nick Nassar Having modules be responsible for storing revisions of their own fields is a side-effect of storing revision data in tables. There's really no way around it. However, revisions generally don't make sense for node types that don't have PHP/HTML content, such as polls. I think it's going to be a pretty rare scenario for a new node type to want another field to change per-revision, so it's a pretty good trade-off. Storing fields that shouldn't be part of revisions, such as the profile_ fields, is a side-effect of storing revisions as serialized objects. Applying this patch will free up that wasted space. :) ------------------------------------------------------------------------ August 23, 2004 - 18:20 : Anonymous There should be a hook that let's the module choose whether it supports history. This way a module author can prevent the user from doing something that may break his module or just cause undefined behavior. If the module doesn't support history then don't let the user/admin choose to add history to nodes of that type. Craig ------------------------------------------------------------------------ August 23, 2004 - 20:23 : Nick Nassar I agree, there should be an API change to make specifying support for revisions easier. In the interests of keeping patches small and keeping to one change per patch, I think the API change should be a separate issue. A sort of ad-hoc API to decide whether or not a module supports revisions by default already exists. Instead of having a hook, modules set the default value of the "Create new revision" field in the edit form. The admin can change this option in admin/node/configure/defaults. This patch doesn't change that. Revisions are broken for node types that have their own database structure, like polls, even when storing them as serialized objects. This patch doesn't change that, either. ------------------------------------------------------------------------ October 26, 2004 - 03:35 : moshe weitzman I'm guessing that someone is going to have to demonstrate that this patch performs as well as current drupal before it gets comitted. i think this patch is a few benchmarks from being comitted. ------------------------------------------------------------------------ October 27, 2004 - 02:04 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004.p… (11 KB) I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ October 27, 2004 - 02:05 : Nick Nassar I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ October 27, 2004 - 02:05 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004.p… (11 KB) I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ November 15, 2004 - 06:05 : elias1884 please overthink the revision system default workflow as well. don't look at the revision system as an isolated system but as a part of the whole workflow system! if you combine revisions with the moderation queue the most logic default workflow would be like that: auth user creates node (revision #0) admin approves the node (status = 1, moderation = 0) => node publicly available auth user finds typo and changes node (revision #1, status = 0, moderation = 1) ------- what happens at that point at the moment is, that the node is not accessible anymore at all until the new revision is approved by admin. of course the new revision should not go online until reviewed and approved, this is absolutely correct, but there is no reason to not take the old revision offline, since it was already approved and should therefore be online until the new revision is approved. it is not practical if a node disappears only because the author corrected a typo. ------- admin approves the node (status = 1, moderation = 0) eventhough I first thought a plain boolean active field would not be capable of providing that functionality if finally came to the conclusion, that it can. The only thing to do is to not set that bit, when a new revision is created, but when it is approved (in case moderation is activated under default workflow). Every revision should have its own moderation, status and active field and on approval they are set like this (status=1, moderation=0, active=Y). When you wanna rollback to an old revision, you can chose between all revisions that already have the moderation bit set back to 0 again and the published to 1. There should be an extra permission for rollback! another concern that I have about the default workflow is, that users can't see the content, they have just created, when moderation is enabled. Eventhough, there is a big fat "submission accepted" presented after submissions, unexperienced users tend to question the information those stupid tincans give them, if they can't find their content afterwards. Many users are really lazy bastards and they don't even read the status messages. The best feedback about whether his story was submitted successfully or not of course is, if he can find the story somewhere on the site, maybe with a status message on top of it, mentioning, that the content is currently not publicly available since it has not been approved yet. there should be a my content section under my account, like somebody is trying to do with the workspace module I guess. so my suggestion is to make (status=0, moderation=1) still available for the creator under a my content section somewhere! ------------------------------------------------------------------------ November 24, 2004 - 05:21 : Nick Nassar I agree. The current workflow for moderation queues and revisions needs to change, but this patch isn't the place for it. The patch is already too big, and it only does the backend stuff. Instead of adding more to this patch and making it take even longer to get into core, would you mind creating a new issue for your UI suggestions, so the those changes can be added as a separate patch? Thanks, Nick ------------------------------------------------------------------------ December 11, 2004 - 13:26 : Dries This patch is _much_ needed so I'd love to see someone revive it. In order for this patch to be accepted, the following needs to be done: Update this patch to CVS HEAD. Rename revid to vid. Rename node_rev to node_revisions. Rename node_rev.changed to node_revisions.timestamp. Rename $rnode to $revision. Fix the coding style to match Drupal's: proper spacing, single quotes where possible, proper variable names. Benchmark this patch with a large database with enough revisions. I'd be happy to benchmark this on my local copy of the drupal.org database. The book.log field should probably move to the node_revisions table. This can be done in a separate patch. Investigate whether transactions are well-supported. ------------------------------------------------------------------------ December 13, 2004 - 01:25 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004-r… (11.02 KB) I've worked a bit on the patch (coding style issues as mentioned by Dries). One thing I noticed is that the patch uses REPLACE. IIRC this needs to be chagned to "UPDATE, if fail INSERT" for pgsql compatibility. Nick, are you still interested in working on that patch? I'd like to know how it works on your site and work on getting it into core. ------------------------------------------------------------------------ December 13, 2004 - 13:33 : Dries Gerhard: your patch does not apply. ------------------------------------------------------------------------ December 13, 2004 - 18:10 : killes(a)www.drop.org Yes, I know, that was the same version as I mailed to you earlier. ------------------------------------------------------------------------ December 13, 2004 - 22:02 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions.patch (52.96 KB) Ok, upüdated the patch to cvs. ------------------------------------------------------------------------ December 14, 2004 - 09:58 : Dries Some more comments: db_begin_transaction() and db_end_transaction() do not belong in database.inc, but in database.mysql.inc and database.pgsql.inc respectively. The node module calls node_revisionsision_list() which is not defined. (Fxed that on my local copy.) Do db_begin_transaction() and db_end_transaction() deprecate Jeremy's table locking patch? The upgrade path assigns the wrong user ID to each revision. The upgrade path assigns the wrong date to each revision (that or a node's revision page shows the wrong usernames/dates). The coding style needs a bit of work, but we can worry about that later. ------------------------------------------------------------------------ December 14, 2004 - 18:34 : Nick Nassar If you need any help getting those things fixed, just let me know. ------------------------------------------------------------------------ December 14, 2004 - 18:50 : Nick Nassar How this relates to Jeremy's node locking patch: There was lots of discussion, and node locking was decided against because from an end user point of view you never want a node to be locked. He's now advocating for a much simpler patch that warns users if their changes will overwrite someone elses. That patch still has a race condition, which might be fixed using db_begin_transaction(). http://drupal.org/node/6025 ------------------------------------------------------------------------ December 14, 2004 - 23:26 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_0.patch (55.96 KB) Here is an updated patch that tries to address Dries concerns. ------------------------------------------------------------------------ December 15, 2004 - 09:32 : Dries Attachment: http://drupal.org/files/issues/revisions-bug.png (76.06 KB) It didn't fix the aforementioned bugs. See attached screenshot. ------------------------------------------------------------------------ January 6, 2005 - 21:15 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_1.patch (51.77 KB) Ok, here is a new version. Dries and myself worked hart at it, so please have a look. what is still missing - database upgrades for the core modules with an own table - contrib modules need an upgrade too. - do we need nid and vid in both the node and the node_revisions table? - the amount of sql queries means a good stress testing for large databases. ------------------------------------------------------------------------ January 19, 2005 - 22:43 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_2.patch (49.49 KB) Here is an updated patch. We discussed to keep the current title in node module and also in the revisiosn table. This is content duplication but will save many joins as many queries only need the title of a node. Discussion is welcome. ------------------------------------------------------------------------ January 20, 2005 - 00:33 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_3.patch (29.93 KB) I've implemented the aforementioned solution. This makes the patch much smaller. The patch now also removes taxonomy_node_has_term() which wasn't used anywhere. I'd really apprciate if some people could test drive the patch. It will be another huge improvement for 4.6. -- View: http://drupal.org/node/7582 Edit: http://drupal.org/project/comments/add/7582

1 0

[drupal-devel] [task] Put revisions in their own table
by killes 20 Jan '05

20 Jan '05

Project: Drupal Version: cvs Component: node system Category: tasks Priority: normal Assigned to: killes(a)www.drop.org Reported by: killes(a)www.drop.org Updated by: killes(a)www.drop.org Status: patch Attachment: http://drupal.org/files/issues/revisions_3.patch (29.93 KB) I've implemented the aforementioned solution. This makes the patch much smaller. The patch now also removes taxonomy_node_has_term() which wasn't used anywhere. I'd really apprciate if some people could test drive the patch. It will be another huge improvement for 4.6. killes(a)www.drop.org Previous comments: ------------------------------------------------------------------------ May 5, 2004 - 17:25 : killes(a)www.drop.org Currently all node revisions are stored in a serialized field in node.table and retrieved for _each_ page view although they are rarely needed. However, we have agreed that serializing data is bad and that we should try to keep the memory foot print pf Drupal small. Therefore I propose to create a separate revisions table which would be in principle identical to the node table, only that it could have several old copies of the same node. Extra data added by other modules could be added in a serialized field unless we find a better solution. ------------------------------------------------------------------------ May 5, 2004 - 18:06 : jhriggs I too think the serialized approach is less than desirable, but here's an alternative. This would likely take some considerable rework in core and contrib, but the following is how we handle similar types of situations in our databases at work. It is more elegant that a separate table, and avoids the (almost exact) duplication of a table. Instead of separate tables, keep all revisions of nodes in the node table as follows: * add field: active (0/1 or Y/N) * add field: revision * every revision of a node is stored in the node table; however, only one revision can be active at any given time * nid can no longer be unique -- primary/unique key becomes (nid, active) * any time a node is loaded, updated (without revision), etc., the active version is used. Thoughts? ------------------------------------------------------------------------ May 5, 2004 - 18:57 : killes(a)www.drop.org I am not opposed to your scheme, but I want to stress the following: * Duplicating a table's structure is not bad (IMHO) as long as the content is different. * having two tables will allow us to have a rather small node table. This is (maybe) a performance gain. ------------------------------------------------------------------------ May 5, 2004 - 19:37 : jhriggs I don't necessarily think that duplicating a table's structure is _bad_. It just seems to be wasteful and a pain to maintain. (Every change to the node table is made twice...easy to do, but also easy to miss perhaps.) As for performance, as long as nid and the active indicator are indexed, there shouldn't be any performance loss. Also, archiving an old version when making a new revision will be much simpler: just change the active indicator rather than copying an entire node to another table (and ensuring everything gets copied...again a potential maintainance issue). To be honest, I would just like to see the serialized data go away, regarless of what approach is taken. ------------------------------------------------------------------------ July 30, 2004 - 20:49 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_07-30-2004.p… (10.47 KB) I'm interested in using Drupal for a large scale wiki-type project. In order to do this, I need revisions to be in their own table. Attached is a patch to do just that. Most of the changes are pretty self explanatory. Spreading out node data across two tables meant that I had to add database functions to do locking/transactions. Without this, race conditions in which the database becomes corrupted are possible. ------------------------------------------------------------------------ July 30, 2004 - 20:54 : Nick Nassar Oh yeah... The patch is a diff against Drupal CVS ------------------------------------------------------------------------ July 31, 2004 - 01:00 : Anonymous Gerhard speaking. Nick, thanks a lot for your nice patch! It saves me a great deal of labour. I looked through it and immediately liked it. You not only put the old revisions into a new table but also the current one. Do you have an estimate how much more expensive the additional join is? Besides a few minor coding style issues I found a major one: Just a few hours before you uploaded your patch JonBob's node access patch hit core. That means your patch won't apply anymore as all the queries you change have been changed. Can I bug you to update your patch? ------------------------------------------------------------------------ July 31, 2004 - 02:11 : Anonymous Also I think that your upgrade path loses existing revisions. ------------------------------------------------------------------------ July 31, 2004 - 03:39 : drumm I think this is the proper way to do things. No columns are duplicated, there is no serialized data, and only the fields that are logically revised are stored. Nothing jumped out at me as a way to have my node modules be able to keep a table of revisions of additional fields. I'm guessing this could be done within the confines of _insert and _update. Assuming the upgrade path works and modules can extend it I give it a +1. ------------------------------------------------------------------------ July 31, 2004 - 15:40 : Nick Nassar It figures that just as I finish a big patch, another patch comes along and breaks it. Oh well, it should be a pretty easy to fix. I'll work on it. Fixing the upgrade path to keep revisions should be fairly painless. I found another issue that needs to be fixed before this patch gets merged. There format of a node needs to be stored for each revision. Otherwise, for modules that store a format for the nodes, such as page and book, if you write one revision in PHP and the next in HTML, the PHP revision will be displayed as HTML. This is part of a larger issue of how node modules should store revisions of additional fields. I think each module that wants to do this should create another table with (nid, revid) as the primary key. Just as when they want to add fields to a node they create another table with nid as the primary key. As far as performance goes, for sites that make heavy use of revisions, an extra join on primary keys is going to be a lot faster than grabbing all of the revisions from that database everytime. We would need to run benchmarks to determine is the overall difference in speed is for an average site is a gain or a loss. I'm guessing it's very minor either way. ------------------------------------------------------------------------ August 23, 2004 - 14:55 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_08-23-2004.p… (10.92 KB) Here's an updated patch against CVS that puts revisions in their own table, provides an upgrade path, and fixes the format related bugs in the last patch. Hopefully, this can make it into CVS as soon as the freeze is over. ------------------------------------------------------------------------ August 23, 2004 - 15:10 : moshe weitzman Interesting patch ... drumm's question is still outstanding. how do modules store revisions of their fields? Are they expected to manage this on their own? Thats not how it works today. As an aside, i am seeing profile_ fields in my node.revisions column. One could argue that those need not be saved. They pertain to the node author, not to the node itself. ------------------------------------------------------------------------ August 23, 2004 - 17:14 : Nick Nassar Having modules be responsible for storing revisions of their own fields is a side-effect of storing revision data in tables. There's really no way around it. However, revisions generally don't make sense for node types that don't have PHP/HTML content, such as polls. I think it's going to be a pretty rare scenario for a new node type to want another field to change per-revision, so it's a pretty good trade-off. Storing fields that shouldn't be part of revisions, such as the profile_ fields, is a side-effect of storing revisions as serialized objects. Applying this patch will free up that wasted space. :) ------------------------------------------------------------------------ August 23, 2004 - 18:20 : Anonymous There should be a hook that let's the module choose whether it supports history. This way a module author can prevent the user from doing something that may break his module or just cause undefined behavior. If the module doesn't support history then don't let the user/admin choose to add history to nodes of that type. Craig ------------------------------------------------------------------------ August 23, 2004 - 20:23 : Nick Nassar I agree, there should be an API change to make specifying support for revisions easier. In the interests of keeping patches small and keeping to one change per patch, I think the API change should be a separate issue. A sort of ad-hoc API to decide whether or not a module supports revisions by default already exists. Instead of having a hook, modules set the default value of the "Create new revision" field in the edit form. The admin can change this option in admin/node/configure/defaults. This patch doesn't change that. Revisions are broken for node types that have their own database structure, like polls, even when storing them as serialized objects. This patch doesn't change that, either. ------------------------------------------------------------------------ October 26, 2004 - 03:35 : moshe weitzman I'm guessing that someone is going to have to demonstrate that this patch performs as well as current drupal before it gets comitted. i think this patch is a few benchmarks from being comitted. ------------------------------------------------------------------------ October 27, 2004 - 02:04 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004.p… (11 KB) I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ October 27, 2004 - 02:05 : Nick Nassar I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ October 27, 2004 - 02:05 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004.p… (11 KB) I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ November 15, 2004 - 06:05 : elias1884 please overthink the revision system default workflow as well. don't look at the revision system as an isolated system but as a part of the whole workflow system! if you combine revisions with the moderation queue the most logic default workflow would be like that: auth user creates node (revision #0) admin approves the node (status = 1, moderation = 0) => node publicly available auth user finds typo and changes node (revision #1, status = 0, moderation = 1) ------- what happens at that point at the moment is, that the node is not accessible anymore at all until the new revision is approved by admin. of course the new revision should not go online until reviewed and approved, this is absolutely correct, but there is no reason to not take the old revision offline, since it was already approved and should therefore be online until the new revision is approved. it is not practical if a node disappears only because the author corrected a typo. ------- admin approves the node (status = 1, moderation = 0) eventhough I first thought a plain boolean active field would not be capable of providing that functionality if finally came to the conclusion, that it can. The only thing to do is to not set that bit, when a new revision is created, but when it is approved (in case moderation is activated under default workflow). Every revision should have its own moderation, status and active field and on approval they are set like this (status=1, moderation=0, active=Y). When you wanna rollback to an old revision, you can chose between all revisions that already have the moderation bit set back to 0 again and the published to 1. There should be an extra permission for rollback! another concern that I have about the default workflow is, that users can't see the content, they have just created, when moderation is enabled. Eventhough, there is a big fat "submission accepted" presented after submissions, unexperienced users tend to question the information those stupid tincans give them, if they can't find their content afterwards. Many users are really lazy bastards and they don't even read the status messages. The best feedback about whether his story was submitted successfully or not of course is, if he can find the story somewhere on the site, maybe with a status message on top of it, mentioning, that the content is currently not publicly available since it has not been approved yet. there should be a my content section under my account, like somebody is trying to do with the workspace module I guess. so my suggestion is to make (status=0, moderation=1) still available for the creator under a my content section somewhere! ------------------------------------------------------------------------ November 24, 2004 - 05:21 : Nick Nassar I agree. The current workflow for moderation queues and revisions needs to change, but this patch isn't the place for it. The patch is already too big, and it only does the backend stuff. Instead of adding more to this patch and making it take even longer to get into core, would you mind creating a new issue for your UI suggestions, so the those changes can be added as a separate patch? Thanks, Nick ------------------------------------------------------------------------ December 11, 2004 - 13:26 : Dries This patch is _much_ needed so I'd love to see someone revive it. In order for this patch to be accepted, the following needs to be done: Update this patch to CVS HEAD. Rename revid to vid. Rename node_rev to node_revisions. Rename node_rev.changed to node_revisions.timestamp. Rename $rnode to $revision. Fix the coding style to match Drupal's: proper spacing, single quotes where possible, proper variable names. Benchmark this patch with a large database with enough revisions. I'd be happy to benchmark this on my local copy of the drupal.org database. The book.log field should probably move to the node_revisions table. This can be done in a separate patch. Investigate whether transactions are well-supported. ------------------------------------------------------------------------ December 13, 2004 - 01:25 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004-r… (11.02 KB) I've worked a bit on the patch (coding style issues as mentioned by Dries). One thing I noticed is that the patch uses REPLACE. IIRC this needs to be chagned to "UPDATE, if fail INSERT" for pgsql compatibility. Nick, are you still interested in working on that patch? I'd like to know how it works on your site and work on getting it into core. ------------------------------------------------------------------------ December 13, 2004 - 13:33 : Dries Gerhard: your patch does not apply. ------------------------------------------------------------------------ December 13, 2004 - 18:10 : killes(a)www.drop.org Yes, I know, that was the same version as I mailed to you earlier. ------------------------------------------------------------------------ December 13, 2004 - 22:02 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions.patch (52.96 KB) Ok, upüdated the patch to cvs. ------------------------------------------------------------------------ December 14, 2004 - 09:58 : Dries Some more comments: db_begin_transaction() and db_end_transaction() do not belong in database.inc, but in database.mysql.inc and database.pgsql.inc respectively. The node module calls node_revisionsision_list() which is not defined. (Fxed that on my local copy.) Do db_begin_transaction() and db_end_transaction() deprecate Jeremy's table locking patch? The upgrade path assigns the wrong user ID to each revision. The upgrade path assigns the wrong date to each revision (that or a node's revision page shows the wrong usernames/dates). The coding style needs a bit of work, but we can worry about that later. ------------------------------------------------------------------------ December 14, 2004 - 18:34 : Nick Nassar If you need any help getting those things fixed, just let me know. ------------------------------------------------------------------------ December 14, 2004 - 18:50 : Nick Nassar How this relates to Jeremy's node locking patch: There was lots of discussion, and node locking was decided against because from an end user point of view you never want a node to be locked. He's now advocating for a much simpler patch that warns users if their changes will overwrite someone elses. That patch still has a race condition, which might be fixed using db_begin_transaction(). http://drupal.org/node/6025 ------------------------------------------------------------------------ December 14, 2004 - 23:26 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_0.patch (55.96 KB) Here is an updated patch that tries to address Dries concerns. ------------------------------------------------------------------------ December 15, 2004 - 09:32 : Dries Attachment: http://drupal.org/files/issues/revisions-bug.png (76.06 KB) It didn't fix the aforementioned bugs. See attached screenshot. ------------------------------------------------------------------------ January 6, 2005 - 21:15 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_1.patch (51.77 KB) Ok, here is a new version. Dries and myself worked hart at it, so please have a look. what is still missing - database upgrades for the core modules with an own table - contrib modules need an upgrade too. - do we need nid and vid in both the node and the node_revisions table? - the amount of sql queries means a good stress testing for large databases. ------------------------------------------------------------------------ January 19, 2005 - 22:43 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_2.patch (49.49 KB) Here is an updated patch. We discussed to keep the current title in node module and also in the revisiosn table. This is content duplication but will save many joins as many queries only need the title of a node. Discussion is welcome. -- View: http://drupal.org/node/7582 Edit: http://drupal.org/project/comments/add/7582

1 0

Re: [drupal-devel] [bug] form_weight should use dleta:15 in book outline admin
by Boris Mann 19 Jan '05

19 Jan '05

On Wed, 19 Jan 2005 16:17:33 +0100, Bèr Kessels <drupal-devel(a)drupal.org> wrote: > This is a useability improvement *with code* ;) > > In book-pages node forms we use a delta of fifteen (-15 to +15) but in > the book-outline admin we dont dewfine this, resulting in a default > delta of ten. This should be consistent and thus both 15. > Bèr Kessels Actually, the delta should be infinite, weight should only be used internally, and we should have +/-, up/down arrows :P (And a javascript, drag-and-drop version from Chris Messina) But thanks, Ber. -- Boris Mann http://www.bryght.com

4 3

[drupal-devel] [task] Put revisions in their own table
by killes 19 Jan '05

19 Jan '05

Project: Drupal Version: cvs Component: node system Category: tasks Priority: normal Assigned to: killes(a)www.drop.org Reported by: killes(a)www.drop.org Updated by: killes(a)www.drop.org Status: patch Attachment: http://drupal.org/files/issues/revisions_2.patch (49.49 KB) Here is an updated patch. We discussed to keep the current title in node module and also in the revisiosn table. This is content duplication but will save many joins as many queries only need the title of a node. Discussion is welcome. killes(a)www.drop.org Previous comments: ------------------------------------------------------------------------ May 5, 2004 - 17:25 : killes(a)www.drop.org Currently all node revisions are stored in a serialized field in node.table and retrieved for _each_ page view although they are rarely needed. However, we have agreed that serializing data is bad and that we should try to keep the memory foot print pf Drupal small. Therefore I propose to create a separate revisions table which would be in principle identical to the node table, only that it could have several old copies of the same node. Extra data added by other modules could be added in a serialized field unless we find a better solution. ------------------------------------------------------------------------ May 5, 2004 - 18:06 : jhriggs I too think the serialized approach is less than desirable, but here's an alternative. This would likely take some considerable rework in core and contrib, but the following is how we handle similar types of situations in our databases at work. It is more elegant that a separate table, and avoids the (almost exact) duplication of a table. Instead of separate tables, keep all revisions of nodes in the node table as follows: * add field: active (0/1 or Y/N) * add field: revision * every revision of a node is stored in the node table; however, only one revision can be active at any given time * nid can no longer be unique -- primary/unique key becomes (nid, active) * any time a node is loaded, updated (without revision), etc., the active version is used. Thoughts? ------------------------------------------------------------------------ May 5, 2004 - 18:57 : killes(a)www.drop.org I am not opposed to your scheme, but I want to stress the following: * Duplicating a table's structure is not bad (IMHO) as long as the content is different. * having two tables will allow us to have a rather small node table. This is (maybe) a performance gain. ------------------------------------------------------------------------ May 5, 2004 - 19:37 : jhriggs I don't necessarily think that duplicating a table's structure is _bad_. It just seems to be wasteful and a pain to maintain. (Every change to the node table is made twice...easy to do, but also easy to miss perhaps.) As for performance, as long as nid and the active indicator are indexed, there shouldn't be any performance loss. Also, archiving an old version when making a new revision will be much simpler: just change the active indicator rather than copying an entire node to another table (and ensuring everything gets copied...again a potential maintainance issue). To be honest, I would just like to see the serialized data go away, regarless of what approach is taken. ------------------------------------------------------------------------ July 30, 2004 - 20:49 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_07-30-2004.p… (10.47 KB) I'm interested in using Drupal for a large scale wiki-type project. In order to do this, I need revisions to be in their own table. Attached is a patch to do just that. Most of the changes are pretty self explanatory. Spreading out node data across two tables meant that I had to add database functions to do locking/transactions. Without this, race conditions in which the database becomes corrupted are possible. ------------------------------------------------------------------------ July 30, 2004 - 20:54 : Nick Nassar Oh yeah... The patch is a diff against Drupal CVS ------------------------------------------------------------------------ July 31, 2004 - 01:00 : Anonymous Gerhard speaking. Nick, thanks a lot for your nice patch! It saves me a great deal of labour. I looked through it and immediately liked it. You not only put the old revisions into a new table but also the current one. Do you have an estimate how much more expensive the additional join is? Besides a few minor coding style issues I found a major one: Just a few hours before you uploaded your patch JonBob's node access patch hit core. That means your patch won't apply anymore as all the queries you change have been changed. Can I bug you to update your patch? ------------------------------------------------------------------------ July 31, 2004 - 02:11 : Anonymous Also I think that your upgrade path loses existing revisions. ------------------------------------------------------------------------ July 31, 2004 - 03:39 : drumm I think this is the proper way to do things. No columns are duplicated, there is no serialized data, and only the fields that are logically revised are stored. Nothing jumped out at me as a way to have my node modules be able to keep a table of revisions of additional fields. I'm guessing this could be done within the confines of _insert and _update. Assuming the upgrade path works and modules can extend it I give it a +1. ------------------------------------------------------------------------ July 31, 2004 - 15:40 : Nick Nassar It figures that just as I finish a big patch, another patch comes along and breaks it. Oh well, it should be a pretty easy to fix. I'll work on it. Fixing the upgrade path to keep revisions should be fairly painless. I found another issue that needs to be fixed before this patch gets merged. There format of a node needs to be stored for each revision. Otherwise, for modules that store a format for the nodes, such as page and book, if you write one revision in PHP and the next in HTML, the PHP revision will be displayed as HTML. This is part of a larger issue of how node modules should store revisions of additional fields. I think each module that wants to do this should create another table with (nid, revid) as the primary key. Just as when they want to add fields to a node they create another table with nid as the primary key. As far as performance goes, for sites that make heavy use of revisions, an extra join on primary keys is going to be a lot faster than grabbing all of the revisions from that database everytime. We would need to run benchmarks to determine is the overall difference in speed is for an average site is a gain or a loss. I'm guessing it's very minor either way. ------------------------------------------------------------------------ August 23, 2004 - 14:55 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_08-23-2004.p… (10.92 KB) Here's an updated patch against CVS that puts revisions in their own table, provides an upgrade path, and fixes the format related bugs in the last patch. Hopefully, this can make it into CVS as soon as the freeze is over. ------------------------------------------------------------------------ August 23, 2004 - 15:10 : moshe weitzman Interesting patch ... drumm's question is still outstanding. how do modules store revisions of their fields? Are they expected to manage this on their own? Thats not how it works today. As an aside, i am seeing profile_ fields in my node.revisions column. One could argue that those need not be saved. They pertain to the node author, not to the node itself. ------------------------------------------------------------------------ August 23, 2004 - 17:14 : Nick Nassar Having modules be responsible for storing revisions of their own fields is a side-effect of storing revision data in tables. There's really no way around it. However, revisions generally don't make sense for node types that don't have PHP/HTML content, such as polls. I think it's going to be a pretty rare scenario for a new node type to want another field to change per-revision, so it's a pretty good trade-off. Storing fields that shouldn't be part of revisions, such as the profile_ fields, is a side-effect of storing revisions as serialized objects. Applying this patch will free up that wasted space. :) ------------------------------------------------------------------------ August 23, 2004 - 18:20 : Anonymous There should be a hook that let's the module choose whether it supports history. This way a module author can prevent the user from doing something that may break his module or just cause undefined behavior. If the module doesn't support history then don't let the user/admin choose to add history to nodes of that type. Craig ------------------------------------------------------------------------ August 23, 2004 - 20:23 : Nick Nassar I agree, there should be an API change to make specifying support for revisions easier. In the interests of keeping patches small and keeping to one change per patch, I think the API change should be a separate issue. A sort of ad-hoc API to decide whether or not a module supports revisions by default already exists. Instead of having a hook, modules set the default value of the "Create new revision" field in the edit form. The admin can change this option in admin/node/configure/defaults. This patch doesn't change that. Revisions are broken for node types that have their own database structure, like polls, even when storing them as serialized objects. This patch doesn't change that, either. ------------------------------------------------------------------------ October 26, 2004 - 03:35 : moshe weitzman I'm guessing that someone is going to have to demonstrate that this patch performs as well as current drupal before it gets comitted. i think this patch is a few benchmarks from being comitted. ------------------------------------------------------------------------ October 27, 2004 - 02:04 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004.p… (11 KB) I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ October 27, 2004 - 02:05 : Nick Nassar I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ October 27, 2004 - 02:05 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004.p… (11 KB) I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ November 15, 2004 - 06:05 : elias1884 please overthink the revision system default workflow as well. don't look at the revision system as an isolated system but as a part of the whole workflow system! if you combine revisions with the moderation queue the most logic default workflow would be like that: auth user creates node (revision #0) admin approves the node (status = 1, moderation = 0) => node publicly available auth user finds typo and changes node (revision #1, status = 0, moderation = 1) ------- what happens at that point at the moment is, that the node is not accessible anymore at all until the new revision is approved by admin. of course the new revision should not go online until reviewed and approved, this is absolutely correct, but there is no reason to not take the old revision offline, since it was already approved and should therefore be online until the new revision is approved. it is not practical if a node disappears only because the author corrected a typo. ------- admin approves the node (status = 1, moderation = 0) eventhough I first thought a plain boolean active field would not be capable of providing that functionality if finally came to the conclusion, that it can. The only thing to do is to not set that bit, when a new revision is created, but when it is approved (in case moderation is activated under default workflow). Every revision should have its own moderation, status and active field and on approval they are set like this (status=1, moderation=0, active=Y). When you wanna rollback to an old revision, you can chose between all revisions that already have the moderation bit set back to 0 again and the published to 1. There should be an extra permission for rollback! another concern that I have about the default workflow is, that users can't see the content, they have just created, when moderation is enabled. Eventhough, there is a big fat "submission accepted" presented after submissions, unexperienced users tend to question the information those stupid tincans give them, if they can't find their content afterwards. Many users are really lazy bastards and they don't even read the status messages. The best feedback about whether his story was submitted successfully or not of course is, if he can find the story somewhere on the site, maybe with a status message on top of it, mentioning, that the content is currently not publicly available since it has not been approved yet. there should be a my content section under my account, like somebody is trying to do with the workspace module I guess. so my suggestion is to make (status=0, moderation=1) still available for the creator under a my content section somewhere! ------------------------------------------------------------------------ November 24, 2004 - 05:21 : Nick Nassar I agree. The current workflow for moderation queues and revisions needs to change, but this patch isn't the place for it. The patch is already too big, and it only does the backend stuff. Instead of adding more to this patch and making it take even longer to get into core, would you mind creating a new issue for your UI suggestions, so the those changes can be added as a separate patch? Thanks, Nick ------------------------------------------------------------------------ December 11, 2004 - 13:26 : Dries This patch is _much_ needed so I'd love to see someone revive it. In order for this patch to be accepted, the following needs to be done: Update this patch to CVS HEAD. Rename revid to vid. Rename node_rev to node_revisions. Rename node_rev.changed to node_revisions.timestamp. Rename $rnode to $revision. Fix the coding style to match Drupal's: proper spacing, single quotes where possible, proper variable names. Benchmark this patch with a large database with enough revisions. I'd be happy to benchmark this on my local copy of the drupal.org database. The book.log field should probably move to the node_revisions table. This can be done in a separate patch. Investigate whether transactions are well-supported. ------------------------------------------------------------------------ December 13, 2004 - 01:25 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004-r… (11.02 KB) I've worked a bit on the patch (coding style issues as mentioned by Dries). One thing I noticed is that the patch uses REPLACE. IIRC this needs to be chagned to "UPDATE, if fail INSERT" for pgsql compatibility. Nick, are you still interested in working on that patch? I'd like to know how it works on your site and work on getting it into core. ------------------------------------------------------------------------ December 13, 2004 - 13:33 : Dries Gerhard: your patch does not apply. ------------------------------------------------------------------------ December 13, 2004 - 18:10 : killes(a)www.drop.org Yes, I know, that was the same version as I mailed to you earlier. ------------------------------------------------------------------------ December 13, 2004 - 22:02 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions.patch (52.96 KB) Ok, upüdated the patch to cvs. ------------------------------------------------------------------------ December 14, 2004 - 09:58 : Dries Some more comments: db_begin_transaction() and db_end_transaction() do not belong in database.inc, but in database.mysql.inc and database.pgsql.inc respectively. The node module calls node_revisionsision_list() which is not defined. (Fxed that on my local copy.) Do db_begin_transaction() and db_end_transaction() deprecate Jeremy's table locking patch? The upgrade path assigns the wrong user ID to each revision. The upgrade path assigns the wrong date to each revision (that or a node's revision page shows the wrong usernames/dates). The coding style needs a bit of work, but we can worry about that later. ------------------------------------------------------------------------ December 14, 2004 - 18:34 : Nick Nassar If you need any help getting those things fixed, just let me know. ------------------------------------------------------------------------ December 14, 2004 - 18:50 : Nick Nassar How this relates to Jeremy's node locking patch: There was lots of discussion, and node locking was decided against because from an end user point of view you never want a node to be locked. He's now advocating for a much simpler patch that warns users if their changes will overwrite someone elses. That patch still has a race condition, which might be fixed using db_begin_transaction(). http://drupal.org/node/6025 ------------------------------------------------------------------------ December 14, 2004 - 23:26 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_0.patch (55.96 KB) Here is an updated patch that tries to address Dries concerns. ------------------------------------------------------------------------ December 15, 2004 - 09:32 : Dries Attachment: http://drupal.org/files/issues/revisions-bug.png (76.06 KB) It didn't fix the aforementioned bugs. See attached screenshot. ------------------------------------------------------------------------ January 6, 2005 - 21:15 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_1.patch (51.77 KB) Ok, here is a new version. Dries and myself worked hart at it, so please have a look. what is still missing - database upgrades for the core modules with an own table - contrib modules need an upgrade too. - do we need nid and vid in both the node and the node_revisions table? - the amount of sql queries means a good stress testing for large databases. -- View: http://drupal.org/node/7582 Edit: http://drupal.org/project/comments/add/7582

1 0

[drupal-devel] RE: drupal-devel Digest, Vol 25, Issue 99
by Michael Jervis 19 Jan '05

19 Jan '05

Tim: >You can find my current work in my sandbox. Hmm, well, you may be known enough in drupal for others to find your sand box, but I'm drawing a blank here. No taltman, tim or altman or other obviously named sandbox sub-dir. >If at all possible, we should use the same framework for importation >scripts, so it's easy to make a new importer plug-in. If you have a framework, I'd like to take a look, so if you could point me the right way. Cheers, Mike

1 0

Re: [drupal-devel] Introduction and Questions
by Steven Wittens 19 Jan '05

19 Jan '05

>I've read the contributors guide, which states I should "and then notify the >contributions manager to request a review of your work before posting". I >can't see anything that tells me how to go about this, so I am assuming the >method is to post to the list? Should I check into contributions before >review to have it somewhere to be reviewed, or post it in some other form >somewhere? Or does "before posting" mean before filling in the form to list >under modules? > >I have attached the module in a RAR, I hope this won't cause too much >offence, it is less than 1kb. > Yeah the "contributions manager" thing should probably be changed to something clearer. You can commit anything you think is reasonable into the contrib repository (and of course within the rules), but it won't show up on Drupal.org until you create a project and have it approved by one of the people with the mojo to do this. Your module seems okay, granted it's also very short ;). I have two comments: - You search for the string "livejournal" in the server name, which would include more than the real livejournal.com. Using a regexp anchored to the end is probably a better idea (e.g. '/livejournal\.com$/i'). Granted, this issue will probably never pop up, but it's still nice to be exact. - There are a couple of things that don't fit the Drupal code style (a couple of spaces and a brace). This doesn't matter for the contributions repository, as they are not required there, but if you plan on submitting core patches later, it might be a good idea to start with good habits. If you want a review next time, it's probably best to just commit it to the repository and ask on this list. The people here are almost all developers, so keep that in mind. If you want to reach general Drupal users, try the forums instead. As far as attachments go, if they are small (like yours) then it's acceptable, though I would suggest using .tar.gz or .zip instead of .rar as they are more widespread. Steven Wittens

2 2

[drupal-devel] [bug] Inconsistent behavior of URL aliases in user-specified paths
by JonBob 19 Jan '05

19 Jan '05

Project: Drupal Version: cvs Component: base system Category: bug reports Priority: normal Assigned to: JonBob Reported by: JonBob Updated by: JonBob -Status: active +Status: patch Attachment: http://drupal.org/files/issues/menu_17.patch (781 bytes) Here's a patch to bring the menu system in line with #5, which seems to be the user expectation. Any advice on where to document the appropriate behavior for modules? Perhaps the PHPdoc for drupal_get_path_alias()? JonBob Previous comments: ------------------------------------------------------------------------ July 23, 2004 - 08:14 : JonBob Currently Drupal is rather schizophrenic when it comes to handling user-specified paths. We don't require the user to type in paths often, which is good, but there are a few situations where this is unavoidable. Examples are menu administration, settings for 403/404 pages, the front page selection, and block path regular expressions. In these locations, there is no clear policy for whether the user is expected to enter an aliased or real Drupal path, and we certainly don't tell the user in the help text. We should decide on a policy, document it in the code and/or user help, and enforce it in the modules. The possible policies I can imagine are: 1) Require the user to enter a real Drupal path. 2) Require the user to enter an aliased path. 3) Allow the user to enter either, but always store a real Drupal path. 4) Allow the user to enter either, but always store an aliased path. 5) Allow the user to enter either, store what the user entered, and convert at read time. Now 1 and 3 have the advantage that if the alias for a page changes, links to that page in the various settings pages do not break. 2 and 4, on the other hand, allow the administrator to affect what page a setting refers to by simply changing a path alias, which could be construed as good behavior. 3 and 4 have the downside of possibly presenting the admin with different information than she entered when she goes back to edit it later. 5 is the most permissive, but is the hardest policy to enforce because the value could be read in many different places in the code. Also, with 5 the behavior is like 1 and 3 if the user enters a real path or 2 and 4 if the user enters an alias. This is good because it allows the administrator to have the behavior she wants, but bad because it is inconsistent. What policy should we go with? I'm leaning toward either 3 or 5. Once we decide, I will: - Add documentation about the decision in the code - Change existing usage to match the decision - Update user documentation on the administration pages (probably only necessary with approach 1 or 2) ------------------------------------------------------------------------ July 23, 2004 - 10:11 : JonBob Another note: policies 3 and 4 will not work with block paths, as we cannot reliably convert the regex to the appropriate form. With these, we must perform any necessary conversion on the path that is being matched against the regex instead of on the regex itself. ------------------------------------------------------------------------ July 27, 2004 - 11:15 : JonBob The issue Xtemplate mission does not work with an alias [1] is really a part of this one. [1] http://drupal.org/node/view/7685 -- View: http://drupal.org/node/9477 Edit: http://drupal.org/project/comments/add/9477

1 0

[drupal-devel] [feature] Implement google's rel="nofollow"
by walkah 19 Jan '05

19 Jan '05

Project: Drupal Version: 4.5.0 Component: filter.module Category: feature requests Priority: normal Assigned to: walkah Reported by: walkah Updated by: walkah Status: patch Attachment: http://drupal.org/files/issues/filter-no-follow.patch (1.74 KB) google just announced a new attribute rel="nofollow" in an attempt to combat comment spam. the attached patch adds the option to add rel="nofollow" as part of HTML Filter for more info check: http://www.google.com/googleblog/2005/01/preventing-comment-spam.html walkah -- View: http://drupal.org/node/15847 Edit: http://drupal.org/project/comments/add/15847

5 6