Development

Download

development@drupal.org

9354 discussions

[drupal-devel] [bug] Put revisions in their own table
by Bèr Kessels 25 Jul '05

25 Jul '05

Issue status update for http://drupal.org/node/7582 Post a follow up: http://drupal.org/project/comments/add/7582 Project: Drupal Version: cvs Component: node system Category: bug reports Priority: normal Assigned to: killes(a)www.drop.org Reported by: killes(a)www.drop.org Updated by: Bèr Kessels Status: patch I was not referring to not testing it. If it is just the upgrade path that proves to be cumbersome, then why can we not fix it afterwards, I.e. when everyone has had a good chance to look at it. Testing such a huge patch requires a lot of work. something no-one just does in his spare hours. We had discussions before to deal in a slightly different way with big changes; to commit them quicker and to leave the ironing out of any left overs for the community. I hooked into that discussion here. For two reasons. One is that Gerhard has spent numerous hours on maintaining this patch. The other one is that the community can be of help with ironing out issues in such a large change, much better than that Gerhard can do on his own. And yes, dataloss is very bad, but no-one should loose data, when he/she followed the instructions (backing up)... Bèr Kessels Previous comments: ------------------------------------------------------------------------ Wed, 05 May 2004 16:25:27 +0000 : killes(a)www.drop.org Currently all node revisions are stored in a serialized field in node.table and retrieved for _each_ page view although they are rarely needed. However, we have agreed that serializing data is bad and that we should try to keep the memory foot print pf Drupal small. Therefore I propose to create a separate revisions table which would be in principle identical to the node table, only that it could have several old copies of the same node. Extra data added by other modules could be added in a serialized field unless we find a better solution. ------------------------------------------------------------------------ Wed, 05 May 2004 17:06:35 +0000 : jhriggs I too think the serialized approach is less than desirable, but here's an alternative. This would likely take some considerable rework in core and contrib, but the following is how we handle similar types of situations in our databases at work. It is more elegant that a separate table, and avoids the (almost exact) duplication of a table. Instead of separate tables, keep all revisions of nodes in the node table as follows: * add field: active (0/1 or Y/N) * add field: revision * every revision of a node is stored in the node table; however, only one revision can be active at any given time * nid can no longer be unique -- primary/unique key becomes (nid, active) * any time a node is loaded, updated (without revision), etc., the active version is used. Thoughts? ------------------------------------------------------------------------ Wed, 05 May 2004 17:57:48 +0000 : killes(a)www.drop.org I am not opposed to your scheme, but I want to stress the following: * Duplicating a table's structure is not bad (IMHO) as long as the content is different. * having two tables will allow us to have a rather small node table. This is (maybe) a performance gain. ------------------------------------------------------------------------ Wed, 05 May 2004 18:37:29 +0000 : jhriggs I don't necessarily think that duplicating a table's structure is _bad_. It just seems to be wasteful and a pain to maintain. (Every change to the node table is made twice...easy to do, but also easy to miss perhaps.) As for performance, as long as nid and the active indicator are indexed, there shouldn't be any performance loss. Also, archiving an old version when making a new revision will be much simpler: just change the active indicator rather than copying an entire node to another table (and ensuring everything gets copied...again a potential maintainance issue). To be honest, I would just like to see the serialized data go away, regarless of what approach is taken. ------------------------------------------------------------------------ Fri, 30 Jul 2004 19:49:33 +0000 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_07-30-2004.p… (10.47 KB) I'm interested in using Drupal for a large scale wiki-type project. In order to do this, I need revisions to be in their own table. Attached is a patch to do just that. Most of the changes are pretty self explanatory. Spreading out node data across two tables meant that I had to add database functions to do locking/transactions. Without this, race conditions in which the database becomes corrupted are possible. ------------------------------------------------------------------------ Fri, 30 Jul 2004 19:54:36 +0000 : Nick Nassar Oh yeah... The patch is a diff against Drupal CVS ------------------------------------------------------------------------ Sat, 31 Jul 2004 00:00:08 +0000 : Anonymous Gerhard speaking. Nick, thanks a lot for your nice patch! It saves me a great deal of labour. I looked through it and immediately liked it. You not only put the old revisions into a new table but also the current one. Do you have an estimate how much more expensive the additional join is? Besides a few minor coding style issues I found a major one: Just a few hours before you uploaded your patch JonBob's node access patch hit core. That means your patch won't apply anymore as all the queries you change have been changed. Can I bug you to update your patch? ------------------------------------------------------------------------ Sat, 31 Jul 2004 01:11:59 +0000 : Anonymous Also I think that your upgrade path loses existing revisions. ------------------------------------------------------------------------ Sat, 31 Jul 2004 02:39:12 +0000 : drumm I think this is the proper way to do things. No columns are duplicated, there is no serialized data, and only the fields that are logically revised are stored. Nothing jumped out at me as a way to have my node modules be able to keep a table of revisions of additional fields. I'm guessing this could be done within the confines of _insert and _update. Assuming the upgrade path works and modules can extend it I give it a +1. ------------------------------------------------------------------------ Sat, 31 Jul 2004 14:40:15 +0000 : Nick Nassar It figures that just as I finish a big patch, another patch comes along and breaks it. Oh well, it should be a pretty easy to fix. I'll work on it. Fixing the upgrade path to keep revisions should be fairly painless. I found another issue that needs to be fixed before this patch gets merged. There format of a node needs to be stored for each revision. Otherwise, for modules that store a format for the nodes, such as page and book, if you write one revision in PHP and the next in HTML, the PHP revision will be displayed as HTML. This is part of a larger issue of how node modules should store revisions of additional fields. I think each module that wants to do this should create another table with (nid, revid) as the primary key. Just as when they want to add fields to a node they create another table with nid as the primary key. As far as performance goes, for sites that make heavy use of revisions, an extra join on primary keys is going to be a lot faster than grabbing all of the revisions from that database everytime. We would need to run benchmarks to determine is the overall difference in speed is for an average site is a gain or a loss. I'm guessing it's very minor either way. ------------------------------------------------------------------------ Mon, 23 Aug 2004 13:55:49 +0000 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_08-23-2004.p… (10.92 KB) Here's an updated patch against CVS that puts revisions in their own table, provides an upgrade path, and fixes the format related bugs in the last patch. Hopefully, this can make it into CVS as soon as the freeze is over. ------------------------------------------------------------------------ Mon, 23 Aug 2004 14:10:39 +0000 : moshe weitzman Interesting patch ... drumm's question is still outstanding. how do modules store revisions of their fields? Are they expected to manage this on their own? Thats not how it works today. As an aside, i am seeing profile_ fields in my node.revisions column. One could argue that those need not be saved. They pertain to the node author, not to the node itself. ------------------------------------------------------------------------ Mon, 23 Aug 2004 16:14:39 +0000 : Nick Nassar Having modules be responsible for storing revisions of their own fields is a side-effect of storing revision data in tables. There's really no way around it. However, revisions generally don't make sense for node types that don't have PHP/HTML content, such as polls. I think it's going to be a pretty rare scenario for a new node type to want another field to change per-revision, so it's a pretty good trade-off. Storing fields that shouldn't be part of revisions, such as the profile_ fields, is a side-effect of storing revisions as serialized objects. Applying this patch will free up that wasted space. :) ------------------------------------------------------------------------ Mon, 23 Aug 2004 17:20:57 +0000 : Anonymous There should be a hook that let's the module choose whether it supports history. This way a module author can prevent the user from doing something that may break his module or just cause undefined behavior. If the module doesn't support history then don't let the user/admin choose to add history to nodes of that type. Craig ------------------------------------------------------------------------ Mon, 23 Aug 2004 19:23:29 +0000 : Nick Nassar I agree, there should be an API change to make specifying support for revisions easier. In the interests of keeping patches small and keeping to one change per patch, I think the API change should be a separate issue. A sort of ad-hoc API to decide whether or not a module supports revisions by default already exists. Instead of having a hook, modules set the default value of the "Create new revision" field in the edit form. The admin can change this option in admin/node/configure/defaults. This patch doesn't change that. Revisions are broken for node types that have their own database structure, like polls, even when storing them as serialized objects. This patch doesn't change that, either. ------------------------------------------------------------------------ Tue, 26 Oct 2004 02:35:06 +0000 : moshe weitzman I'm guessing that someone is going to have to demonstrate that this patch performs as well as current drupal before it gets comitted. i think this patch is a few benchmarks from being comitted. ------------------------------------------------------------------------ Wed, 27 Oct 2004 01:04:09 +0000 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004.p… (11 KB) I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ Wed, 27 Oct 2004 01:05:16 +0000 : Nick Nassar I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ Wed, 27 Oct 2004 01:05:26 +0000 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004.p… (11 KB) I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ Mon, 15 Nov 2004 05:05:30 +0000 : elias1884 please overthink the revision system default workflow as well. don't look at the revision system as an isolated system but as a part of the whole workflow system! if you combine revisions with the moderation queue the most logic default workflow would be like that: auth user creates node (revision #0) admin approves the node (status = 1, moderation = 0) => node publicly available auth user finds typo and changes node (revision #1, status = 0, moderation = 1) ------- what happens at that point at the moment is, that the node is not accessible anymore at all until the new revision is approved by admin. of course the new revision should not go online until reviewed and approved, this is absolutely correct, but there is no reason to not take the old revision offline, since it was already approved and should therefore be online until the new revision is approved. it is not practical if a node disappears only because the author corrected a typo. ------- admin approves the node (status = 1, moderation = 0) eventhough I first thought a plain boolean active field would not be capable of providing that functionality if finally came to the conclusion, that it can. The only thing to do is to not set that bit, when a new revision is created, but when it is approved (in case moderation is activated under default workflow). Every revision should have its own moderation, status and active field and on approval they are set like this (status=1, moderation=0, active=Y). When you wanna rollback to an old revision, you can chose between all revisions that already have the moderation bit set back to 0 again and the published to 1. There should be an extra permission for rollback! another concern that I have about the default workflow is, that users can't see the content, they have just created, when moderation is enabled. Eventhough, there is a big fat "submission accepted" presented after submissions, unexperienced users tend to question the information those stupid tincans give them, if they can't find their content afterwards. Many users are really lazy bastards and they don't even read the status messages. The best feedback about whether his story was submitted successfully or not of course is, if he can find the story somewhere on the site, maybe with a status message on top of it, mentioning, that the content is currently not publicly available since it has not been approved yet. there should be a my content section under my account, like somebody is trying to do with the workspace module I guess. so my suggestion is to make (status=0, moderation=1) still available for the creator under a my content section somewhere! ------------------------------------------------------------------------ Wed, 24 Nov 2004 04:21:18 +0000 : Nick Nassar I agree. The current workflow for moderation queues and revisions needs to change, but this patch isn't the place for it. The patch is already too big, and it only does the backend stuff. Instead of adding more to this patch and making it take even longer to get into core, would you mind creating a new issue for your UI suggestions, so the those changes can be added as a separate patch? Thanks, Nick ------------------------------------------------------------------------ Sat, 11 Dec 2004 12:26:02 +0000 : Dries This patch is _much_ needed so I'd love to see someone revive it. In order for this patch to be accepted, the following needs to be done: * Update this patch to CVS HEAD. * Rename revid to vid. * Rename node_rev to node_revisions. * Rename node_rev.changed to node_revisions.timestamp. * Rename $rnode to $revision. * Fix the coding style to match Drupal's: proper spacing, single quotes where possible, proper variable names. * Benchmark this patch with a large database with enough revisions. I'd be happy to benchmark this on my local copy of the drupal.org database. * The book.log field should probably move to the node_revisions table. This can be done in a separate patch. * Investigate whether transactions are well-supported. ------------------------------------------------------------------------ Mon, 13 Dec 2004 00:25:40 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004-r… (11.02 KB) I've worked a bit on the patch (coding style issues as mentioned by Dries). One thing I noticed is that the patch uses REPLACE. IIRC this needs to be chagned to "UPDATE, if fail INSERT" for pgsql compatibility. Nick, are you still interested in working on that patch? I'd like to know how it works on your site and work on getting it into core. ------------------------------------------------------------------------ Mon, 13 Dec 2004 12:33:08 +0000 : Dries Gerhard: your patch does not apply. ------------------------------------------------------------------------ Mon, 13 Dec 2004 17:10:12 +0000 : killes(a)www.drop.org Yes, I know, that was the same version as I mailed to you earlier. ------------------------------------------------------------------------ Mon, 13 Dec 2004 21:02:06 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions.patch (52.96 KB) Ok, upüdated the patch to cvs. ------------------------------------------------------------------------ Tue, 14 Dec 2004 08:58:36 +0000 : Dries Some more comments: * db_begin_transaction() and db_end_transaction() do not belong in database.inc, but in database.mysql.inc and database.pgsql.inc respectively. * The node module calls node_revisionsision_list() which is not defined. (Fxed that on my local copy.) * Do db_begin_transaction() and db_end_transaction() deprecate Jeremy's table locking patch? * The upgrade path assigns the wrong user ID to each revision. * The upgrade path assigns the wrong date to each revision (that or a node's revision page shows the wrong usernames/dates). * The coding style needs a bit of work, but we can worry about that later. ------------------------------------------------------------------------ Tue, 14 Dec 2004 17:34:44 +0000 : Nick Nassar If you need any help getting those things fixed, just let me know. ------------------------------------------------------------------------ Tue, 14 Dec 2004 17:50:30 +0000 : Nick Nassar How this relates to Jeremy's node locking patch: There was lots of discussion, and node locking was decided against because from an end user point of view you never want a node to be locked. He's now advocating for a much simpler patch that warns users if their changes will overwrite someone elses. That patch still has a race condition, which might be fixed using db_begin_transaction(). http://drupal.org/node/6025 ------------------------------------------------------------------------ Tue, 14 Dec 2004 22:26:19 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_0.patch (55.96 KB) Here is an updated patch that tries to address Dries concerns. ------------------------------------------------------------------------ Wed, 15 Dec 2004 08:32:50 +0000 : Dries Attachment: http://drupal.org/files/issues/revisions-bug.png (76.06 KB) It didn't fix the aforementioned bugs. See attached screenshot. ------------------------------------------------------------------------ Thu, 06 Jan 2005 20:15:01 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_1.patch (51.77 KB) Ok, here is a new version. Dries and myself worked hart at it, so please have a look. what is still missing - database upgrades for the core modules with an own table - contrib modules need an upgrade too. - do we need nid and vid in both the node and the node_revisions table? - the amount of sql queries means a good stress testing for large databases. ------------------------------------------------------------------------ Wed, 19 Jan 2005 21:43:49 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_2.patch (49.49 KB) Here is an updated patch. We discussed to keep the current title in node module and also in the revisiosn table. This is content duplication but will save many joins as many queries only need the title of a node. Discussion is welcome. ------------------------------------------------------------------------ Wed, 19 Jan 2005 23:33:32 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_3.patch (29.93 KB) I've implemented the aforementioned solution. This makes the patch much smaller. The patch now also removes taxonomy_node_has_term() which wasn't used anywhere. I'd really apprciate if some people could test drive the patch. It will be another huge improvement for 4.6. ------------------------------------------------------------------------ Thu, 20 Jan 2005 00:05:54 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_4.patch (30 KB) Another revision. Steven didn't like my literal $node->vid in queries. ------------------------------------------------------------------------ Thu, 20 Jan 2005 01:10:50 +0000 : killes(a)www.drop.org - database upgrades for the core modules with an own table - contrib modules need an upgrade too. - do we need nid and vid in both the node and the node_revisions table? - the amount of sql queries means a good stress testing for large databases. These issues are still open, btw. Especially the first one needs to be tackled. ------------------------------------------------------------------------ Tue, 25 Jan 2005 20:11:59 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_5.patch (51.13 KB) Here is a patch that has the database tables updated for forum, book, and page module. ------------------------------------------------------------------------ Sat, 29 Jan 2005 22:55:59 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_6.patch (49.18 KB) Yet another update to keep it working with head. The patch now also removes the table definitons for the page table. ------------------------------------------------------------------------ Sat, 29 Jan 2005 22:57:40 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_7.patch (55.69 KB) Sorry, that was the old version, this is the right one. ------------------------------------------------------------------------ Mon, 31 Jan 2005 19:55:03 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_8.patch (55.71 KB) Updated once more. ------------------------------------------------------------------------ Mon, 31 Jan 2005 20:52:08 +0000 : Dries Anyone to help review/test this? ------------------------------------------------------------------------ Mon, 31 Jan 2005 21:22:36 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_9.patch (49.29 KB) Updated again, the update functions occurred twice. Thanks Bart. ------------------------------------------------------------------------ Wed, 02 Feb 2005 00:27:05 +0000 : killes(a)www.drop.org Don't know if the db I am using is corrupted or what. I still do have some didficulties. The latest patch is attached. ------------------------------------------------------------------------ Wed, 02 Feb 2005 00:27:49 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_10.patch (49.67 KB) I am probably slowly going mad ... ------------------------------------------------------------------------ Wed, 02 Feb 2005 01:54:58 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_11.patch (48.95 KB) The update issue still needs investigating. This patch is updated for cvs. ------------------------------------------------------------------------ Wed, 02 Feb 2005 20:20:51 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_12.patch (49.83 KB) Ok, here is a new version. I've solved my troubles with book.module. There are still some issues with forum module. Possibly due to inconsistent database. ------------------------------------------------------------------------ Wed, 02 Feb 2005 21:31:05 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_13.patch (49.83 KB) Turns out the drupal.org database had indeed some quirks. Please run this query in your oldest db and tell me the result: select nid,type from node where type like '%/%'; If you get a non-zero result we might need to add another security update. The patch could use still more testing, though. ------------------------------------------------------------------------ Thu, 03 Feb 2005 01:16:54 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_14.patch (49 KB) Ok, we are getting somewhere. At a first glance the update is working. There is a problem remaining: the revisions tab will be shown whether the node has revisions or not. Not sure we can/need to fix this. People with a drupal.org account can log in at http://killes.drupaldevs.org/revision/ and poke around. Your permissions will be the same as on drupal.org. Feel free to vreak everything but don't forget to file complaints here. (Note: this is only a pruned version of the drupal.org database with all project nodes and nodes with nids > 7000 dropped). ------------------------------------------------------------------------ Thu, 03 Feb 2005 04:19:14 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_15.patch (52.39 KB) There was some error in node_save and also the patches to the database.inc files got lost... ------------------------------------------------------------------------ Thu, 03 Feb 2005 07:07:27 +0000 : robertDouglass Submitting book pages doesn't work on your test site. It puts the entire content of the preview inside the body textarea. I wrote a sentence in the body and the log, and pressing preview put several lines of HTML containing both sentences in the body textarea on the preview page, plus the book page wouldn't submit. -R ------------------------------------------------------------------------ Thu, 03 Feb 2005 07:50:59 +0000 : Junyor 0 results here. I started using Drupal with version 4.4, though. ------------------------------------------------------------------------ Thu, 03 Feb 2005 23:56:18 +0000 : killes(a)www.drop.org @Junyor: Thanks, that's a good sign. Maybe somebody else has an older db to try. @robertDouglass: The first effect you describe is due to drupaldevs running on PHP 5. I am unsure why the second thing does not work. In node_save() the node object has a nid although there is none in the form. Very strange. I've enabled display of db queries on the testsite. ------------------------------------------------------------------------ Fri, 04 Feb 2005 19:17:55 +0000 : dmjossel No results here on the query: select nid,type from node where type like '%/%'; On a database that was put in place prior to Drupal 4 and is now running on 4.5.2. ------------------------------------------------------------------------ Fri, 04 Feb 2005 20:44:23 +0000 : killes(a)www.drop.org @dmjossel: thanks. @all. The strange problem I reported was apparently php 5 related. After applying Steven's php 5 patch it went away. One error is remaining: If I create a new forum topic it will be shown as part of the book on preview. Hmm, that was due to a db that got corrupted during testing so that is fixed too. Please poke around at the test site and look if you find more errors. ------------------------------------------------------------------------ Sat, 05 Feb 2005 07:16:22 +0000 : Steven By the way, I just remembered that Drupal.org has some blogs lingering on in the database even though blog.module is not enabled. Perhaps this is causing troubles? ------------------------------------------------------------------------ Sat, 05 Feb 2005 11:22:59 +0000 : Anonymous I can't see why it would. Drupal.org will need extra updates for images and project nodes because those have their own tables. GK. ------------------------------------------------------------------------ Sun, 06 Feb 2005 12:49:55 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_16.patch (52.49 KB) Updated to apply to cvs again. ------------------------------------------------------------------------ Tue, 22 Feb 2005 20:15:40 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_17.patch (49.64 KB) Updated again. All we need is a patch to upload module and an upgrade path for it. ------------------------------------------------------------------------ Fri, 04 Mar 2005 04:22:58 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_18.patch (52.31 KB) Updated once more. Moved log field from book to node_revisions table as discussed in Antwerp. upload module still missing. We need to decide under which circumstances the log field should be displayed. Should that be added to the workflow? Should it depend on the revisions setting? ------------------------------------------------------------------------ Sat, 05 Mar 2005 19:27:03 +0000 : Anonymous Attachment: http://drupal.org/files/issues/revisions_20.patch (75.52 KB) Ok, here it is: Yet another revision of this grrrrreat patch. Changes from previous versions: - supports versioning for uploaded files. A problem is that if you delete a file, it will be gone for all revisions. - the log field is now in the node_revsions table, but each module has to decide whether to show it or not. I've implemented it for the page and the book type odes. Also, the field can be edited when adding non-book nodes to the book. The log is displayed on the revisions page and if a node is moderated. - the revisions are moved to an old_revisions table to a) get the node table smaller and b) still leave the mavailable for contrib modules that want to retreive old version data. The patch has been applied to killes.drupaldevs.org/revision where it can be tested by anybody (especially people who have "site admin" rights on drupal.org) The database is from drupal.org and you should b able to log in with your pass or simply mail yourself a new one. Gerhard ------------------------------------------------------------------------ Sat, 05 Mar 2005 19:51:56 +0000 : Anonymous Attachment: http://drupal.org/files/issues/revisions_21.patch (59.42 KB) BTW, I marked this a bug because atm the revisions field can grow quite big. Neil has reported problems from some users who were not able to load some nodes due to to many large revisions. Also, som unrelated stuff crept into the patch. New version attached. ------------------------------------------------------------------------ Tue, 08 Mar 2005 05:56:01 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_22.patch (60.29 KB) Ok, I think I got it. Changes to last version: - uploads are no properly versioned. Missing are still pgsql checks and updates. ------------------------------------------------------------------------ Thu, 10 Mar 2005 16:58:41 +0000 : Anonymous Was able to get http://drupal.org/files/issues/revisions_21.patch to work with drupal-cvs.tar.gz (10 March 2005) by: - includes/database.mysql.inc: Commenting out duplicates for functions function db_begin_transaction and function db_commit_transaction - modules: node.module: Removing "'title' => $node->title," from $node_table_values variable declaration and removing "'title' => "'%s'"," from "$node_table_types" variable declaration. Happy to submit a patch if requested. I'll watch this thread. ------------------------------------------------------------------------ Fri, 11 Mar 2005 23:59:45 +0000 : killes(a)www.drop.org The duplicate function has been removed in rev 22 of this patch. Why do you think the changes in node_save are needed? Titles are saved in both tables for performance reasons. ------------------------------------------------------------------------ Sun, 13 Mar 2005 16:12:21 +0000 : jlerner Hi - I posted comment #62. The changes to node_save appear to be needed because recent patches (both 21 and 22) remove the field 'title' from table 'node'. So without the changes to node_save, node.module is broken and generates errors. Joshua ------------------------------------------------------------------------ Sun, 13 Mar 2005 16:29:42 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_23.patch (61.17 KB) Thanks, Joshua, for catching this. node:title is there to stay. ------------------------------------------------------------------------ Wed, 13 Apr 2005 16:29:23 +0000 : moshe weitzman since HEAD is open again, perhaps it is a good time to revisit this patch. once this is committed, lets address - http://drupal.org/node/11071 "node_validate does not respect group editing" ------------------------------------------------------------------------ Mon, 18 Apr 2005 15:43:42 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_24.patch (60.39 KB) Updated. ------------------------------------------------------------------------ Mon, 18 Apr 2005 16:16:32 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_24_0.patch (60.39 KB) Updated. ------------------------------------------------------------------------ Tue, 19 Apr 2005 05:19:42 +0000 : Dries I'll commit this patch later this week! If you haven't checked this patch already, I urge you to test/check it out because it will have significant impact on existing code and modules! ------------------------------------------------------------------------ Tue, 19 Apr 2005 05:21:33 +0000 : Dries Also, what do people think about the n.title being duplicated? ------------------------------------------------------------------------ Tue, 19 Apr 2005 05:26:58 +0000 : chx I won't lose any sleep because of duplicated titles... ------------------------------------------------------------------------ Tue, 19 Apr 2005 18:35:58 +0000 : killes(a)www.drop.org Let me explain why I have chosen to duplicate the title (and also the uid): If you look at all the queries in Drupal, you will find that most of them only need the title and th uid of a node. That is, by duplicating it, we save expensive joins on the node_revisions table. Due to this fact, this patch is actually a performance improvement. A note about updating contrib module: Strictly speaking they wouldn't need to be updated. They only need to if their authors decide that their info should be available for revisioning. The upgrade path for forum.module in my update.inc patch (+ the forum patch) should show you what needs to be done. I will write a note for the update page once the patch hits core. ------------------------------------------------------------------------ Sun, 24 Apr 2005 21:21:19 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_25.patch (60.38 KB) Updated to cvs. Dries: Based on some remarks in #drupal this is the last update I am going to do. Apply it or won't fix it. ------------------------------------------------------------------------ Sat, 30 Apr 2005 03:42:39 +0000 : Jeremy(a)kerneltrap.org Attachment: http://drupal.org/files/issues/revisions_25.patch.patch (528 bytes) That's a big patch. I've only started looking through it. I noticed one little typo, affecting updates. A patch to your last patch is attached. I'm running with the revision patch on my dev server now happily. I like the concept. What happens if you click 'stop' on your browser in the middle of a MySQL "transaction"? I assume that kills the connection to MySQL, and the lock is freed? But this then leaves changes only partially applied? What exactly does locking/unlocking the tables buy us in MySQL? I don't see anywhere that we detect if an apply fails part way through, and thus roll back...? What am I missing? ------------------------------------------------------------------------ Sat, 30 Apr 2005 07:11:28 +0000 : Dries Jeremy: many of us are worried about the performance ramifications this patch introduces. Early experiments showed a small performance improvement (while a performance regression might be expected). More performance reports from large sites like kerneltrap.org will certainly help this patch. Mind to do a quick performance comparision and to report back with some numbers? Thanks. ------------------------------------------------------------------------ Sat, 30 Apr 2005 12:38:02 +0000 : Jeremy(a)kerneltrap.org Dries: I'm not running HEAD on kerneltrap, so this really isn't a possibility. Furthermore, until I understand why we're locking tables, I don't like it. The idea of revisions in their own tables is great. The idea of locking tables to get (without any obvious benefit) there really worries me. ------------------------------------------------------------------------ Sat, 30 Apr 2005 14:16:01 +0000 : killes(a)www.drop.org @Jeremy: Thanks for looking at the patch! Also for catching the typo. :) Did you try to upgrade your database? If yes, how did it go? One of Dries' concerns is the complexity of the upgrade. How many nodes and revisions did the db have? About database locking: This part of the patch was created by Nick and I simply continued to use it. Maybe the code should rather be: if(db_begin_transaction(array('{node}', '{node_revisions}', '{watchdog}', '{sessions}', '{files}'))) { db_query($node_query, $node_table_values); db_query($revisions_query, $revisions_table_values); db_commit_transaction(); ... } The idea is probably to avoid two updates at the same time. I don't think the locking helps if you abort the script at an inconvenient time. Rollbacks aren't implemented in all mysql versions. We could omit the db locking if deemed inappropriate. Maybe Nick can explain his ideas behind this. @Dries: I wonder who the "many of us" are. They certainly haven't spoken to me. Moshe had some reservations about the upgrade path and project module, but the time that project module abused revisions to store issue updates was long ago and his reservations were resolved. Nobody else (besides you of course and now Jeremy) has voiced reservations in a way that was audible to me. If you grep through the patch you will notice that there are only four queries which have a join on the node_revisions table. Two of them are in node_load and in the other cases the join replaced a join on the node table. The two queries in node_load are the only ones that have both a join on the node and the revisions query. Thus, loading of individual nodes might become somewhat slower. All other queries will be faster since the node table is now much smaller. Also, node loading does not have to be slower, it depends on your node table. If you had a lot of revisions and thus a large table, then the new scheme will make your queries actually faster since we do not load the revisions on each and every node load anymore. If you didn't have many revisions your node_load migth be somewhat slower. WRT to the update script Karoly pointed out that we could use multiple insert queries instead one query per revision. This would probably make the update somewhat faster. I am willing to work on this iff you declare that you will commit the patch afterwards. I'd need to know if this will work on pgsql and on all supported mysql versions before I work on it. About locking: Database locking is dog slow, at least on mysql. I was using locks in an earlier version of the upgrade script but had to remove it for (serious!) performance reasons. ------------------------------------------------------------------------ Mon, 09 May 2005 15:07:34 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_26.patch (46.45 KB) Ok, another update, cause I need it myself. I've left out the transaction stuff for now. It is in principle unrelated to this patch and should be discussed elsewhere. This also makes the patch smaller and easier to review (hint, hint). ------------------------------------------------------------------------ Mon, 09 May 2005 20:32:09 +0000 : killes(a)www.drop.org The patch contained the update functions twice. ------------------------------------------------------------------------ Mon, 09 May 2005 20:32:26 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_27.patch (39.05 KB) The patch contained the update functions twice. ------------------------------------------------------------------------ Mon, 09 May 2005 21:23:06 +0000 : Dries Got one error during the upgrade path: ALTER TABLE {book} ADD PRIMARY KEY vid (vid) FAILED ------------------------------------------------------------------------ Mon, 09 May 2005 21:26:19 +0000 : killes(a)www.drop.org This had happend to me as well, when I tested this patch. The reason is that for some reason the vid is not unique. Most likely there are some entries with vid = 0 in there. Can you check which node types those have? it always was an error in the test database. See: http://drupal.org/node/7582#comment-20678 ------------------------------------------------------------------------ Mon, 09 May 2005 21:27:06 +0000 : Dries Actually, I got 2850 errors during the upgrade. Some of these: sprintf() [function.sprintf]: Too few arguments in drupal-cvs/includes/database.inc on line 154. Some of these: Query was empty query: in drupal-cvs/includes/database.mysql.inc on line 66. And this: Unknown table 'n' in field list query: SELECT n.nid, n.vid FROM node INNER JOIN files f ON n.nid = f.nid in drupal-cvs/includes/database.mysql.inc on line 66. :-) ------------------------------------------------------------------------ Mon, 09 May 2005 21:29:19 +0000 : Dries Or this: user error: Unknown column 'log' in 'field list' query: SELECT parent, weight, log FROM book WHERE nid = 1 in drupal-cvs/includes/database.mysql.inc on line 66 ------------------------------------------------------------------------ Mon, 09 May 2005 21:52:12 +0000 : Dries The time required to generate my main page went from 902 ms (before upgrade) to 2139 ms (after upgrade). The time required to generate a forum listing (?q=forum/x) went from 1872 ms (before upgrade) to 2874 ms (after upgrade). Maybe this is because my database is not consistent as the result of the upgrade errors (yet I don't see any errors on the pages I benchmarked). ------------------------------------------------------------------------ Tue, 10 May 2005 00:24:38 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_28.patch (53.47 KB) Ok, let me get to this from the bottom to the top: - my test runs indicated a different development wrt timing. If I had gotten your results, I had stopped working on this long ago. So your results are wrong for some reason. - user error: Unknown column 'log' in 'field list' Wasn't my day, the book patch got lost. It is contained now. First -R the old patch, then apply this one. - Unknown table 'n' in field list query: Walkah found this, but I forgot to fix it. Fixed now. - I've no idea where the other queries come from. I am hoping that either your test db is borken or they are follow ups from the other ones. If you let me have your test db, I'll try some debugging. Thanks for wasting your time, too. ------------------------------------------------------------------------ Tue, 10 May 2005 05:07:31 +0000 : Dries I double-checked and the numbers don't seem to lie. I'll test some more after work on another machine to make sure it is not platform-specific. ------------------------------------------------------------------------ Wed, 11 May 2005 03:32:47 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_29.patch (54.83 KB) Ok, here I am again. What I did: 1) Ask Dries to let me have drupal.org database 2) get 400MB of SQL inserts... 3) take 23 minutes to import said data 4) Remove all image and project nodes (don't want to install their modules), 11765 nodes left 5) back up data 6) take tests on non-cached /node page (as anonymous user). ab results: -c 1 -n 25: Requests per second: 1.29 [#/sec] (mean) Connection Times (ms) min mean[+/-sd] median max Total: 663 775 179.7 689 1264 7) Do the same for the tracker page: Requests per second: 0.83 [#/sec] (mean) Total: 1182 1199 7.4 1199 1217 8) Apply my patch (rev. 28). 9) run db update and hold breath 10) update times out... 11) play back backup from 5) 12) wait 13) getting annoyed and removing cache, watchdog, and accesslog before playing in backup. 14) wait again. Understand why Dries doesn't try this patch often. Maybe a smaller DB would do for testing? 15) wait more. get really annoyed. 16) Set time limit to 18000 in update.php 17) try again 18) fails again before the second update is completed. 19) curse. 20) delete search stuff from db. Ooops, sooo much smaller... 21) import again, below 2 minutes... 22) rewrite to use extended insert. Found a bug. 23) still does not complete. Mysql logging to the rescue! 24) tid = 0? Not good. 25) Well, the update works fine till node 10834. 5595 nodes done, 6136 to go. 26) Writing shell based update script. Discovery: 24MB aren't enough. Hopefully 64 are. Nope. extended inserts for revisions are apparently not the brightest idea: Huge memory consumption. Hmm, no, all updates got through. Selecting the revisions to put them into old_revisions table screwed it. Learned about CREATE TABLE old_revisions SELECT syntax. Yay! finally. 24 MB are just not enough the update.php script seems to still break. 27) Benchmarks! /node Requests per second: 1.54 [#/sec] (mean) Connection Times (ms) min mean[+/-sd] median max Total: 632 649 40.5 636 791 /tracker Requests per second: 0.86 [#/sec] (mean) Total: 1119 1165 65.8 1160 1461 Ok: So we get an improvement for many node_loads, but none for simple selects from node. More tests can be done. 28) roll new patch Ain't Drupal fun? ------------------------------------------------------------------------ Wed, 18 May 2005 13:38:05 +0000 : Dries I did another round of tests on _another_ machine and it is not looking good: Before upgrade After upgrade ?q= (main page) 218 ms/request 340 ms/request ?q=forum (forum overview) 754 ms/request 1520 ms/request ?q=about (book page) 375 ms/request 5400 ms/request The upgrade process itself gave me a number of 'query was empty' and 'sprintf(): too few arguments' reports. Everything seems to work fine though. Looking at the ?q=about page, I see that the following query is executed twice _and_ that each time, it take more than 2 seconds to complete: SELECT n.nid, n.title, b.parent, b.weight FROM node n INNER JOIN book b ON n.vid = b.vid WHERE n.status = 1 AND n.moderate = 0 ORDER BY b.weight, n.title; --8 SHOW INDEX FROM book; +-------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | +-------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+ | book | 1 | book_parent | 1 | parent | A | 92 | NULL | NULL | | BTREE | | | book | 1 | nid | 1 | nid | A | 369 | NULL | NULL | | BTREE | | +-------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+ 2 rows in set (0.00 sec) The book module does not appear to have a primary key? Sounds like a bad idea so I added one: mysql> ALTER TABLE book ADD PRIMARY KEY nid (nid); Query OK, 369 rows affected (0.02 sec) Records: 369 Duplicates: 0 Warnings: 0 Next, I wanted to make the vid column a unique key in all node tables: mysql> ALTER TABLE node ADD UNIQUE vid (vid); Query OK, 20392 rows affected (0.81 sec) Records: 20392 Duplicates: 0 Warnings: 0 mysql> ALTER TABLE book ADD UNIQUE vid (vid); ERROR 1062: Duplicate entry '0' for key 2 mysql> ALTER TABLE forum ADD UNIQUE vid (vid); Query OK, 10806 rows affected (0.10 sec) Records: 10806 Duplicates: 0 Warnings: 0 As you can see, it fails for the book table which makes me believe there is some inconsistent data ... I set out to fix that: mysql> SELECT nid, COUNT(nid) AS vids FROM book GROUP BY vid HAVING vids > 1; +-----+------+ | nid | vids | +-----+------+ | 871 | 2 | +-----+------+ 1 row in set (0.00 sec) mysql> SELECT title FROM node WHERE nid = 871; Empty set (0.00 sec) mysql> DELETE FROM book WHERE nid = 871; Query OK, 1 row affected (0.00 sec) mysql> ALTER TABLE book ADD UNIQUE vid (vid); Query OK, 368 rows affected (0.01 sec) Records: 368 Duplicates: 0 Warnings: 0 Looks like everything is well now. Ran some new benchmarks: Before upgrade After upgrade With indices ?q= (main page) 218 ms/request 340 ms/request 336 ms/request ?q=forum (forum overview) 754 ms/request 1520 ms/request 1531 ms/request ?q=about (book page) 375 ms/request 5400 ms/request 475 ms/request Unfortunately, we're still slower than the original code. ------------------------------------------------------------------------ Wed, 18 May 2005 21:53:31 +0000 : killes(a)www.drop.org Dries, thanks for testing it again. I do think the broken queries you observer have something to do with the bad performance after the update. Please log the queries any I will have a look at them. I've never seen any such queries. My update script also tries to create the appropriate indices, but it will of course fail if the database contains cruft. the indices for the forum are probably missing, too. I am still convinced that the patch is core worthy. ------------------------------------------------------------------------ Thu, 19 May 2005 04:36:09 +0000 : Dries It wouldn't hurt if more people would benchmark this patch. The patch's current performance worries me. Did you check your watchdog messages after upgrading the drupal.org database? Depending on your settings, errors might only be shown in the watchdog. I'll look into the remaining glitches as time permits. Thanks for your persistence in keeping this patch up-to-date. :) ------------------------------------------------------------------------ Thu, 19 May 2005 11:59:22 +0000 : killes(a)www.drop.org Dries: Can you please let me have your updated database? I want to have a look at it and try my own benchmarks with it. And yes, if I did learn something on this project is how to be persistant. ;) ------------------------------------------------------------------------ Fri, 24 Jun 2005 16:25:34 +0000 : killes(a)www.drop.org Here is an idea that occurred to me: The problem with the upgrade process is that keeping the existing revisions requires a lot of work to do. This generates a huge amount of sql queries for a large database and also requires a huge amount of memory. My suggestions is to let update.php only handle the basic upgrade, ie without old revisions. An additional module could be created that would implement a cron based approach to upgrading old revisions one node at a time. it could expose a hook to let contrib modules do their own upgrades. Dries, what do you think? (I am writing "Dries" because he seems to be the only one who is interested in getting this into core...) ------------------------------------------------------------------------ Fri, 24 Jun 2005 22:25:11 +0000 : Junyor Killes: I'm also interested in seeing this hit core. What about adding something to legacy.module to do it? ------------------------------------------------------------------------ Sun, 26 Jun 2005 21:14:54 +0000 : chx This is a sensible approach. Maybe this is the _only_ sensible approach. I have a little problem though: while the conversion is running somehow both revision handlers should be available. ------------------------------------------------------------------------ Sun, 26 Jun 2005 22:16:21 +0000 : killes(a)www.drop.org hehe, one only has to whine and bug enough and one gets some feedback. ;) @junyor: legacy.module would be a good place. my current idea is to auto-enable it in update.php and then disable it again in legacy_cron after all nodes are updated.. ;) @chx: When somebody wants to look at revisions of a node that node could be auto-updated. The only problem are contrib modules: they's need to have some hook in order to update their own data. When somebody looks at the revisions of a node than cannot be updated because the contrib module in question has no such hook, we can optionally let the user discard old revisions I guess. Dries, what do you think? ------------------------------------------------------------------------ Mon, 25 Jul 2005 16:48:51 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_30.patch (53.13 KB) Sooooo; I've updated this patch once again. Dries didn't like my idea of legacy updates so we will have an option to discard old revisions in case the update should prove difficult. Always keep a backup. ------------------------------------------------------------------------ Mon, 25 Jul 2005 17:31:12 +0000 : Bèr Kessels can we please accept and commit this patch? We can iron out any issues later. This patch is just far too big and complex to be perfect-at-once. ------------------------------------------------------------------------ Mon, 25 Jul 2005 19:22:50 +0000 : Dries @Ber: no, we can't commit this patch blindly. Data loss is a much bigger problem than a syntax error or other code glitch. We can break Drupal but we can't break their data. I've spent quite a bit of time testing the previous version of this patch and noticed significant performance degradation. Tell me, Ber, why can't you test this patch first?

1 0

[drupal-devel] [bug] Put revisions in their own table
by Dries 25 Jul '05

25 Jul '05

Issue status update for http://drupal.org/node/7582 Post a follow up: http://drupal.org/project/comments/add/7582 Project: Drupal Version: cvs Component: node system Category: bug reports Priority: normal Assigned to: killes(a)www.drop.org Reported by: killes(a)www.drop.org Updated by: Dries Status: patch @Ber: no, we can't commit this patch blindly. Data loss is a much bigger problem than a syntax error or other code glitch. We can break Drupal but we can't break their data. I've spent quite a bit of time testing the previous version of this patch and noticed significant performance degradation. Tell me, Ber, why can't you test this patch first? Dries Previous comments: ------------------------------------------------------------------------ Wed, 05 May 2004 16:25:27 +0000 : killes(a)www.drop.org Currently all node revisions are stored in a serialized field in node.table and retrieved for _each_ page view although they are rarely needed. However, we have agreed that serializing data is bad and that we should try to keep the memory foot print pf Drupal small. Therefore I propose to create a separate revisions table which would be in principle identical to the node table, only that it could have several old copies of the same node. Extra data added by other modules could be added in a serialized field unless we find a better solution. ------------------------------------------------------------------------ Wed, 05 May 2004 17:06:35 +0000 : jhriggs I too think the serialized approach is less than desirable, but here's an alternative. This would likely take some considerable rework in core and contrib, but the following is how we handle similar types of situations in our databases at work. It is more elegant that a separate table, and avoids the (almost exact) duplication of a table. Instead of separate tables, keep all revisions of nodes in the node table as follows: * add field: active (0/1 or Y/N) * add field: revision * every revision of a node is stored in the node table; however, only one revision can be active at any given time * nid can no longer be unique -- primary/unique key becomes (nid, active) * any time a node is loaded, updated (without revision), etc., the active version is used. Thoughts? ------------------------------------------------------------------------ Wed, 05 May 2004 17:57:48 +0000 : killes(a)www.drop.org I am not opposed to your scheme, but I want to stress the following: * Duplicating a table's structure is not bad (IMHO) as long as the content is different. * having two tables will allow us to have a rather small node table. This is (maybe) a performance gain. ------------------------------------------------------------------------ Wed, 05 May 2004 18:37:29 +0000 : jhriggs I don't necessarily think that duplicating a table's structure is _bad_. It just seems to be wasteful and a pain to maintain. (Every change to the node table is made twice...easy to do, but also easy to miss perhaps.) As for performance, as long as nid and the active indicator are indexed, there shouldn't be any performance loss. Also, archiving an old version when making a new revision will be much simpler: just change the active indicator rather than copying an entire node to another table (and ensuring everything gets copied...again a potential maintainance issue). To be honest, I would just like to see the serialized data go away, regarless of what approach is taken. ------------------------------------------------------------------------ Fri, 30 Jul 2004 19:49:33 +0000 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_07-30-2004.p… (10.47 KB) I'm interested in using Drupal for a large scale wiki-type project. In order to do this, I need revisions to be in their own table. Attached is a patch to do just that. Most of the changes are pretty self explanatory. Spreading out node data across two tables meant that I had to add database functions to do locking/transactions. Without this, race conditions in which the database becomes corrupted are possible. ------------------------------------------------------------------------ Fri, 30 Jul 2004 19:54:36 +0000 : Nick Nassar Oh yeah... The patch is a diff against Drupal CVS ------------------------------------------------------------------------ Sat, 31 Jul 2004 00:00:08 +0000 : Anonymous Gerhard speaking. Nick, thanks a lot for your nice patch! It saves me a great deal of labour. I looked through it and immediately liked it. You not only put the old revisions into a new table but also the current one. Do you have an estimate how much more expensive the additional join is? Besides a few minor coding style issues I found a major one: Just a few hours before you uploaded your patch JonBob's node access patch hit core. That means your patch won't apply anymore as all the queries you change have been changed. Can I bug you to update your patch? ------------------------------------------------------------------------ Sat, 31 Jul 2004 01:11:59 +0000 : Anonymous Also I think that your upgrade path loses existing revisions. ------------------------------------------------------------------------ Sat, 31 Jul 2004 02:39:12 +0000 : drumm I think this is the proper way to do things. No columns are duplicated, there is no serialized data, and only the fields that are logically revised are stored. Nothing jumped out at me as a way to have my node modules be able to keep a table of revisions of additional fields. I'm guessing this could be done within the confines of _insert and _update. Assuming the upgrade path works and modules can extend it I give it a +1. ------------------------------------------------------------------------ Sat, 31 Jul 2004 14:40:15 +0000 : Nick Nassar It figures that just as I finish a big patch, another patch comes along and breaks it. Oh well, it should be a pretty easy to fix. I'll work on it. Fixing the upgrade path to keep revisions should be fairly painless. I found another issue that needs to be fixed before this patch gets merged. There format of a node needs to be stored for each revision. Otherwise, for modules that store a format for the nodes, such as page and book, if you write one revision in PHP and the next in HTML, the PHP revision will be displayed as HTML. This is part of a larger issue of how node modules should store revisions of additional fields. I think each module that wants to do this should create another table with (nid, revid) as the primary key. Just as when they want to add fields to a node they create another table with nid as the primary key. As far as performance goes, for sites that make heavy use of revisions, an extra join on primary keys is going to be a lot faster than grabbing all of the revisions from that database everytime. We would need to run benchmarks to determine is the overall difference in speed is for an average site is a gain or a loss. I'm guessing it's very minor either way. ------------------------------------------------------------------------ Mon, 23 Aug 2004 13:55:49 +0000 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_08-23-2004.p… (10.92 KB) Here's an updated patch against CVS that puts revisions in their own table, provides an upgrade path, and fixes the format related bugs in the last patch. Hopefully, this can make it into CVS as soon as the freeze is over. ------------------------------------------------------------------------ Mon, 23 Aug 2004 14:10:39 +0000 : moshe weitzman Interesting patch ... drumm's question is still outstanding. how do modules store revisions of their fields? Are they expected to manage this on their own? Thats not how it works today. As an aside, i am seeing profile_ fields in my node.revisions column. One could argue that those need not be saved. They pertain to the node author, not to the node itself. ------------------------------------------------------------------------ Mon, 23 Aug 2004 16:14:39 +0000 : Nick Nassar Having modules be responsible for storing revisions of their own fields is a side-effect of storing revision data in tables. There's really no way around it. However, revisions generally don't make sense for node types that don't have PHP/HTML content, such as polls. I think it's going to be a pretty rare scenario for a new node type to want another field to change per-revision, so it's a pretty good trade-off. Storing fields that shouldn't be part of revisions, such as the profile_ fields, is a side-effect of storing revisions as serialized objects. Applying this patch will free up that wasted space. :) ------------------------------------------------------------------------ Mon, 23 Aug 2004 17:20:57 +0000 : Anonymous There should be a hook that let's the module choose whether it supports history. This way a module author can prevent the user from doing something that may break his module or just cause undefined behavior. If the module doesn't support history then don't let the user/admin choose to add history to nodes of that type. Craig ------------------------------------------------------------------------ Mon, 23 Aug 2004 19:23:29 +0000 : Nick Nassar I agree, there should be an API change to make specifying support for revisions easier. In the interests of keeping patches small and keeping to one change per patch, I think the API change should be a separate issue. A sort of ad-hoc API to decide whether or not a module supports revisions by default already exists. Instead of having a hook, modules set the default value of the "Create new revision" field in the edit form. The admin can change this option in admin/node/configure/defaults. This patch doesn't change that. Revisions are broken for node types that have their own database structure, like polls, even when storing them as serialized objects. This patch doesn't change that, either. ------------------------------------------------------------------------ Tue, 26 Oct 2004 02:35:06 +0000 : moshe weitzman I'm guessing that someone is going to have to demonstrate that this patch performs as well as current drupal before it gets comitted. i think this patch is a few benchmarks from being comitted. ------------------------------------------------------------------------ Wed, 27 Oct 2004 01:04:09 +0000 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004.p… (11 KB) I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ Wed, 27 Oct 2004 01:05:16 +0000 : Nick Nassar I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ Wed, 27 Oct 2004 01:05:26 +0000 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004.p… (11 KB) I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ Mon, 15 Nov 2004 05:05:30 +0000 : elias1884 please overthink the revision system default workflow as well. don't look at the revision system as an isolated system but as a part of the whole workflow system! if you combine revisions with the moderation queue the most logic default workflow would be like that: auth user creates node (revision #0) admin approves the node (status = 1, moderation = 0) => node publicly available auth user finds typo and changes node (revision #1, status = 0, moderation = 1) ------- what happens at that point at the moment is, that the node is not accessible anymore at all until the new revision is approved by admin. of course the new revision should not go online until reviewed and approved, this is absolutely correct, but there is no reason to not take the old revision offline, since it was already approved and should therefore be online until the new revision is approved. it is not practical if a node disappears only because the author corrected a typo. ------- admin approves the node (status = 1, moderation = 0) eventhough I first thought a plain boolean active field would not be capable of providing that functionality if finally came to the conclusion, that it can. The only thing to do is to not set that bit, when a new revision is created, but when it is approved (in case moderation is activated under default workflow). Every revision should have its own moderation, status and active field and on approval they are set like this (status=1, moderation=0, active=Y). When you wanna rollback to an old revision, you can chose between all revisions that already have the moderation bit set back to 0 again and the published to 1. There should be an extra permission for rollback! another concern that I have about the default workflow is, that users can't see the content, they have just created, when moderation is enabled. Eventhough, there is a big fat "submission accepted" presented after submissions, unexperienced users tend to question the information those stupid tincans give them, if they can't find their content afterwards. Many users are really lazy bastards and they don't even read the status messages. The best feedback about whether his story was submitted successfully or not of course is, if he can find the story somewhere on the site, maybe with a status message on top of it, mentioning, that the content is currently not publicly available since it has not been approved yet. there should be a my content section under my account, like somebody is trying to do with the workspace module I guess. so my suggestion is to make (status=0, moderation=1) still available for the creator under a my content section somewhere! ------------------------------------------------------------------------ Wed, 24 Nov 2004 04:21:18 +0000 : Nick Nassar I agree. The current workflow for moderation queues and revisions needs to change, but this patch isn't the place for it. The patch is already too big, and it only does the backend stuff. Instead of adding more to this patch and making it take even longer to get into core, would you mind creating a new issue for your UI suggestions, so the those changes can be added as a separate patch? Thanks, Nick ------------------------------------------------------------------------ Sat, 11 Dec 2004 12:26:02 +0000 : Dries This patch is _much_ needed so I'd love to see someone revive it. In order for this patch to be accepted, the following needs to be done: * Update this patch to CVS HEAD. * Rename revid to vid. * Rename node_rev to node_revisions. * Rename node_rev.changed to node_revisions.timestamp. * Rename $rnode to $revision. * Fix the coding style to match Drupal's: proper spacing, single quotes where possible, proper variable names. * Benchmark this patch with a large database with enough revisions. I'd be happy to benchmark this on my local copy of the drupal.org database. * The book.log field should probably move to the node_revisions table. This can be done in a separate patch. * Investigate whether transactions are well-supported. ------------------------------------------------------------------------ Mon, 13 Dec 2004 00:25:40 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004-r… (11.02 KB) I've worked a bit on the patch (coding style issues as mentioned by Dries). One thing I noticed is that the patch uses REPLACE. IIRC this needs to be chagned to "UPDATE, if fail INSERT" for pgsql compatibility. Nick, are you still interested in working on that patch? I'd like to know how it works on your site and work on getting it into core. ------------------------------------------------------------------------ Mon, 13 Dec 2004 12:33:08 +0000 : Dries Gerhard: your patch does not apply. ------------------------------------------------------------------------ Mon, 13 Dec 2004 17:10:12 +0000 : killes(a)www.drop.org Yes, I know, that was the same version as I mailed to you earlier. ------------------------------------------------------------------------ Mon, 13 Dec 2004 21:02:06 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions.patch (52.96 KB) Ok, upüdated the patch to cvs. ------------------------------------------------------------------------ Tue, 14 Dec 2004 08:58:36 +0000 : Dries Some more comments: * db_begin_transaction() and db_end_transaction() do not belong in database.inc, but in database.mysql.inc and database.pgsql.inc respectively. * The node module calls node_revisionsision_list() which is not defined. (Fxed that on my local copy.) * Do db_begin_transaction() and db_end_transaction() deprecate Jeremy's table locking patch? * The upgrade path assigns the wrong user ID to each revision. * The upgrade path assigns the wrong date to each revision (that or a node's revision page shows the wrong usernames/dates). * The coding style needs a bit of work, but we can worry about that later. ------------------------------------------------------------------------ Tue, 14 Dec 2004 17:34:44 +0000 : Nick Nassar If you need any help getting those things fixed, just let me know. ------------------------------------------------------------------------ Tue, 14 Dec 2004 17:50:30 +0000 : Nick Nassar How this relates to Jeremy's node locking patch: There was lots of discussion, and node locking was decided against because from an end user point of view you never want a node to be locked. He's now advocating for a much simpler patch that warns users if their changes will overwrite someone elses. That patch still has a race condition, which might be fixed using db_begin_transaction(). http://drupal.org/node/6025 ------------------------------------------------------------------------ Tue, 14 Dec 2004 22:26:19 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_0.patch (55.96 KB) Here is an updated patch that tries to address Dries concerns. ------------------------------------------------------------------------ Wed, 15 Dec 2004 08:32:50 +0000 : Dries Attachment: http://drupal.org/files/issues/revisions-bug.png (76.06 KB) It didn't fix the aforementioned bugs. See attached screenshot. ------------------------------------------------------------------------ Thu, 06 Jan 2005 20:15:01 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_1.patch (51.77 KB) Ok, here is a new version. Dries and myself worked hart at it, so please have a look. what is still missing - database upgrades for the core modules with an own table - contrib modules need an upgrade too. - do we need nid and vid in both the node and the node_revisions table? - the amount of sql queries means a good stress testing for large databases. ------------------------------------------------------------------------ Wed, 19 Jan 2005 21:43:49 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_2.patch (49.49 KB) Here is an updated patch. We discussed to keep the current title in node module and also in the revisiosn table. This is content duplication but will save many joins as many queries only need the title of a node. Discussion is welcome. ------------------------------------------------------------------------ Wed, 19 Jan 2005 23:33:32 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_3.patch (29.93 KB) I've implemented the aforementioned solution. This makes the patch much smaller. The patch now also removes taxonomy_node_has_term() which wasn't used anywhere. I'd really apprciate if some people could test drive the patch. It will be another huge improvement for 4.6. ------------------------------------------------------------------------ Thu, 20 Jan 2005 00:05:54 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_4.patch (30 KB) Another revision. Steven didn't like my literal $node->vid in queries. ------------------------------------------------------------------------ Thu, 20 Jan 2005 01:10:50 +0000 : killes(a)www.drop.org - database upgrades for the core modules with an own table - contrib modules need an upgrade too. - do we need nid and vid in both the node and the node_revisions table? - the amount of sql queries means a good stress testing for large databases. These issues are still open, btw. Especially the first one needs to be tackled. ------------------------------------------------------------------------ Tue, 25 Jan 2005 20:11:59 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_5.patch (51.13 KB) Here is a patch that has the database tables updated for forum, book, and page module. ------------------------------------------------------------------------ Sat, 29 Jan 2005 22:55:59 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_6.patch (49.18 KB) Yet another update to keep it working with head. The patch now also removes the table definitons for the page table. ------------------------------------------------------------------------ Sat, 29 Jan 2005 22:57:40 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_7.patch (55.69 KB) Sorry, that was the old version, this is the right one. ------------------------------------------------------------------------ Mon, 31 Jan 2005 19:55:03 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_8.patch (55.71 KB) Updated once more. ------------------------------------------------------------------------ Mon, 31 Jan 2005 20:52:08 +0000 : Dries Anyone to help review/test this? ------------------------------------------------------------------------ Mon, 31 Jan 2005 21:22:36 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_9.patch (49.29 KB) Updated again, the update functions occurred twice. Thanks Bart. ------------------------------------------------------------------------ Wed, 02 Feb 2005 00:27:05 +0000 : killes(a)www.drop.org Don't know if the db I am using is corrupted or what. I still do have some didficulties. The latest patch is attached. ------------------------------------------------------------------------ Wed, 02 Feb 2005 00:27:49 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_10.patch (49.67 KB) I am probably slowly going mad ... ------------------------------------------------------------------------ Wed, 02 Feb 2005 01:54:58 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_11.patch (48.95 KB) The update issue still needs investigating. This patch is updated for cvs. ------------------------------------------------------------------------ Wed, 02 Feb 2005 20:20:51 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_12.patch (49.83 KB) Ok, here is a new version. I've solved my troubles with book.module. There are still some issues with forum module. Possibly due to inconsistent database. ------------------------------------------------------------------------ Wed, 02 Feb 2005 21:31:05 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_13.patch (49.83 KB) Turns out the drupal.org database had indeed some quirks. Please run this query in your oldest db and tell me the result: select nid,type from node where type like '%/%'; If you get a non-zero result we might need to add another security update. The patch could use still more testing, though. ------------------------------------------------------------------------ Thu, 03 Feb 2005 01:16:54 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_14.patch (49 KB) Ok, we are getting somewhere. At a first glance the update is working. There is a problem remaining: the revisions tab will be shown whether the node has revisions or not. Not sure we can/need to fix this. People with a drupal.org account can log in at http://killes.drupaldevs.org/revision/ and poke around. Your permissions will be the same as on drupal.org. Feel free to vreak everything but don't forget to file complaints here. (Note: this is only a pruned version of the drupal.org database with all project nodes and nodes with nids > 7000 dropped). ------------------------------------------------------------------------ Thu, 03 Feb 2005 04:19:14 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_15.patch (52.39 KB) There was some error in node_save and also the patches to the database.inc files got lost... ------------------------------------------------------------------------ Thu, 03 Feb 2005 07:07:27 +0000 : robertDouglass Submitting book pages doesn't work on your test site. It puts the entire content of the preview inside the body textarea. I wrote a sentence in the body and the log, and pressing preview put several lines of HTML containing both sentences in the body textarea on the preview page, plus the book page wouldn't submit. -R ------------------------------------------------------------------------ Thu, 03 Feb 2005 07:50:59 +0000 : Junyor 0 results here. I started using Drupal with version 4.4, though. ------------------------------------------------------------------------ Thu, 03 Feb 2005 23:56:18 +0000 : killes(a)www.drop.org @Junyor: Thanks, that's a good sign. Maybe somebody else has an older db to try. @robertDouglass: The first effect you describe is due to drupaldevs running on PHP 5. I am unsure why the second thing does not work. In node_save() the node object has a nid although there is none in the form. Very strange. I've enabled display of db queries on the testsite. ------------------------------------------------------------------------ Fri, 04 Feb 2005 19:17:55 +0000 : dmjossel No results here on the query: select nid,type from node where type like '%/%'; On a database that was put in place prior to Drupal 4 and is now running on 4.5.2. ------------------------------------------------------------------------ Fri, 04 Feb 2005 20:44:23 +0000 : killes(a)www.drop.org @dmjossel: thanks. @all. The strange problem I reported was apparently php 5 related. After applying Steven's php 5 patch it went away. One error is remaining: If I create a new forum topic it will be shown as part of the book on preview. Hmm, that was due to a db that got corrupted during testing so that is fixed too. Please poke around at the test site and look if you find more errors. ------------------------------------------------------------------------ Sat, 05 Feb 2005 07:16:22 +0000 : Steven By the way, I just remembered that Drupal.org has some blogs lingering on in the database even though blog.module is not enabled. Perhaps this is causing troubles? ------------------------------------------------------------------------ Sat, 05 Feb 2005 11:22:59 +0000 : Anonymous I can't see why it would. Drupal.org will need extra updates for images and project nodes because those have their own tables. GK. ------------------------------------------------------------------------ Sun, 06 Feb 2005 12:49:55 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_16.patch (52.49 KB) Updated to apply to cvs again. ------------------------------------------------------------------------ Tue, 22 Feb 2005 20:15:40 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_17.patch (49.64 KB) Updated again. All we need is a patch to upload module and an upgrade path for it. ------------------------------------------------------------------------ Fri, 04 Mar 2005 04:22:58 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_18.patch (52.31 KB) Updated once more. Moved log field from book to node_revisions table as discussed in Antwerp. upload module still missing. We need to decide under which circumstances the log field should be displayed. Should that be added to the workflow? Should it depend on the revisions setting? ------------------------------------------------------------------------ Sat, 05 Mar 2005 19:27:03 +0000 : Anonymous Attachment: http://drupal.org/files/issues/revisions_20.patch (75.52 KB) Ok, here it is: Yet another revision of this grrrrreat patch. Changes from previous versions: - supports versioning for uploaded files. A problem is that if you delete a file, it will be gone for all revisions. - the log field is now in the node_revsions table, but each module has to decide whether to show it or not. I've implemented it for the page and the book type odes. Also, the field can be edited when adding non-book nodes to the book. The log is displayed on the revisions page and if a node is moderated. - the revisions are moved to an old_revisions table to a) get the node table smaller and b) still leave the mavailable for contrib modules that want to retreive old version data. The patch has been applied to killes.drupaldevs.org/revision where it can be tested by anybody (especially people who have "site admin" rights on drupal.org) The database is from drupal.org and you should b able to log in with your pass or simply mail yourself a new one. Gerhard ------------------------------------------------------------------------ Sat, 05 Mar 2005 19:51:56 +0000 : Anonymous Attachment: http://drupal.org/files/issues/revisions_21.patch (59.42 KB) BTW, I marked this a bug because atm the revisions field can grow quite big. Neil has reported problems from some users who were not able to load some nodes due to to many large revisions. Also, som unrelated stuff crept into the patch. New version attached. ------------------------------------------------------------------------ Tue, 08 Mar 2005 05:56:01 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_22.patch (60.29 KB) Ok, I think I got it. Changes to last version: - uploads are no properly versioned. Missing are still pgsql checks and updates. ------------------------------------------------------------------------ Thu, 10 Mar 2005 16:58:41 +0000 : Anonymous Was able to get http://drupal.org/files/issues/revisions_21.patch to work with drupal-cvs.tar.gz (10 March 2005) by: - includes/database.mysql.inc: Commenting out duplicates for functions function db_begin_transaction and function db_commit_transaction - modules: node.module: Removing "'title' => $node->title," from $node_table_values variable declaration and removing "'title' => "'%s'"," from "$node_table_types" variable declaration. Happy to submit a patch if requested. I'll watch this thread. ------------------------------------------------------------------------ Fri, 11 Mar 2005 23:59:45 +0000 : killes(a)www.drop.org The duplicate function has been removed in rev 22 of this patch. Why do you think the changes in node_save are needed? Titles are saved in both tables for performance reasons. ------------------------------------------------------------------------ Sun, 13 Mar 2005 16:12:21 +0000 : jlerner Hi - I posted comment #62. The changes to node_save appear to be needed because recent patches (both 21 and 22) remove the field 'title' from table 'node'. So without the changes to node_save, node.module is broken and generates errors. Joshua ------------------------------------------------------------------------ Sun, 13 Mar 2005 16:29:42 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_23.patch (61.17 KB) Thanks, Joshua, for catching this. node:title is there to stay. ------------------------------------------------------------------------ Wed, 13 Apr 2005 16:29:23 +0000 : moshe weitzman since HEAD is open again, perhaps it is a good time to revisit this patch. once this is committed, lets address - http://drupal.org/node/11071 "node_validate does not respect group editing" ------------------------------------------------------------------------ Mon, 18 Apr 2005 15:43:42 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_24.patch (60.39 KB) Updated. ------------------------------------------------------------------------ Mon, 18 Apr 2005 16:16:32 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_24_0.patch (60.39 KB) Updated. ------------------------------------------------------------------------ Tue, 19 Apr 2005 05:19:42 +0000 : Dries I'll commit this patch later this week! If you haven't checked this patch already, I urge you to test/check it out because it will have significant impact on existing code and modules! ------------------------------------------------------------------------ Tue, 19 Apr 2005 05:21:33 +0000 : Dries Also, what do people think about the n.title being duplicated? ------------------------------------------------------------------------ Tue, 19 Apr 2005 05:26:58 +0000 : chx I won't lose any sleep because of duplicated titles... ------------------------------------------------------------------------ Tue, 19 Apr 2005 18:35:58 +0000 : killes(a)www.drop.org Let me explain why I have chosen to duplicate the title (and also the uid): If you look at all the queries in Drupal, you will find that most of them only need the title and th uid of a node. That is, by duplicating it, we save expensive joins on the node_revisions table. Due to this fact, this patch is actually a performance improvement. A note about updating contrib module: Strictly speaking they wouldn't need to be updated. They only need to if their authors decide that their info should be available for revisioning. The upgrade path for forum.module in my update.inc patch (+ the forum patch) should show you what needs to be done. I will write a note for the update page once the patch hits core. ------------------------------------------------------------------------ Sun, 24 Apr 2005 21:21:19 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_25.patch (60.38 KB) Updated to cvs. Dries: Based on some remarks in #drupal this is the last update I am going to do. Apply it or won't fix it. ------------------------------------------------------------------------ Sat, 30 Apr 2005 03:42:39 +0000 : Jeremy(a)kerneltrap.org Attachment: http://drupal.org/files/issues/revisions_25.patch.patch (528 bytes) That's a big patch. I've only started looking through it. I noticed one little typo, affecting updates. A patch to your last patch is attached. I'm running with the revision patch on my dev server now happily. I like the concept. What happens if you click 'stop' on your browser in the middle of a MySQL "transaction"? I assume that kills the connection to MySQL, and the lock is freed? But this then leaves changes only partially applied? What exactly does locking/unlocking the tables buy us in MySQL? I don't see anywhere that we detect if an apply fails part way through, and thus roll back...? What am I missing? ------------------------------------------------------------------------ Sat, 30 Apr 2005 07:11:28 +0000 : Dries Jeremy: many of us are worried about the performance ramifications this patch introduces. Early experiments showed a small performance improvement (while a performance regression might be expected). More performance reports from large sites like kerneltrap.org will certainly help this patch. Mind to do a quick performance comparision and to report back with some numbers? Thanks. ------------------------------------------------------------------------ Sat, 30 Apr 2005 12:38:02 +0000 : Jeremy(a)kerneltrap.org Dries: I'm not running HEAD on kerneltrap, so this really isn't a possibility. Furthermore, until I understand why we're locking tables, I don't like it. The idea of revisions in their own tables is great. The idea of locking tables to get (without any obvious benefit) there really worries me. ------------------------------------------------------------------------ Sat, 30 Apr 2005 14:16:01 +0000 : killes(a)www.drop.org @Jeremy: Thanks for looking at the patch! Also for catching the typo. :) Did you try to upgrade your database? If yes, how did it go? One of Dries' concerns is the complexity of the upgrade. How many nodes and revisions did the db have? About database locking: This part of the patch was created by Nick and I simply continued to use it. Maybe the code should rather be: if(db_begin_transaction(array('{node}', '{node_revisions}', '{watchdog}', '{sessions}', '{files}'))) { db_query($node_query, $node_table_values); db_query($revisions_query, $revisions_table_values); db_commit_transaction(); ... } The idea is probably to avoid two updates at the same time. I don't think the locking helps if you abort the script at an inconvenient time. Rollbacks aren't implemented in all mysql versions. We could omit the db locking if deemed inappropriate. Maybe Nick can explain his ideas behind this. @Dries: I wonder who the "many of us" are. They certainly haven't spoken to me. Moshe had some reservations about the upgrade path and project module, but the time that project module abused revisions to store issue updates was long ago and his reservations were resolved. Nobody else (besides you of course and now Jeremy) has voiced reservations in a way that was audible to me. If you grep through the patch you will notice that there are only four queries which have a join on the node_revisions table. Two of them are in node_load and in the other cases the join replaced a join on the node table. The two queries in node_load are the only ones that have both a join on the node and the revisions query. Thus, loading of individual nodes might become somewhat slower. All other queries will be faster since the node table is now much smaller. Also, node loading does not have to be slower, it depends on your node table. If you had a lot of revisions and thus a large table, then the new scheme will make your queries actually faster since we do not load the revisions on each and every node load anymore. If you didn't have many revisions your node_load migth be somewhat slower. WRT to the update script Karoly pointed out that we could use multiple insert queries instead one query per revision. This would probably make the update somewhat faster. I am willing to work on this iff you declare that you will commit the patch afterwards. I'd need to know if this will work on pgsql and on all supported mysql versions before I work on it. About locking: Database locking is dog slow, at least on mysql. I was using locks in an earlier version of the upgrade script but had to remove it for (serious!) performance reasons. ------------------------------------------------------------------------ Mon, 09 May 2005 15:07:34 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_26.patch (46.45 KB) Ok, another update, cause I need it myself. I've left out the transaction stuff for now. It is in principle unrelated to this patch and should be discussed elsewhere. This also makes the patch smaller and easier to review (hint, hint). ------------------------------------------------------------------------ Mon, 09 May 2005 20:32:09 +0000 : killes(a)www.drop.org The patch contained the update functions twice. ------------------------------------------------------------------------ Mon, 09 May 2005 20:32:26 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_27.patch (39.05 KB) The patch contained the update functions twice. ------------------------------------------------------------------------ Mon, 09 May 2005 21:23:06 +0000 : Dries Got one error during the upgrade path: ALTER TABLE {book} ADD PRIMARY KEY vid (vid) FAILED ------------------------------------------------------------------------ Mon, 09 May 2005 21:26:19 +0000 : killes(a)www.drop.org This had happend to me as well, when I tested this patch. The reason is that for some reason the vid is not unique. Most likely there are some entries with vid = 0 in there. Can you check which node types those have? it always was an error in the test database. See: http://drupal.org/node/7582#comment-20678 ------------------------------------------------------------------------ Mon, 09 May 2005 21:27:06 +0000 : Dries Actually, I got 2850 errors during the upgrade. Some of these: sprintf() [function.sprintf]: Too few arguments in drupal-cvs/includes/database.inc on line 154. Some of these: Query was empty query: in drupal-cvs/includes/database.mysql.inc on line 66. And this: Unknown table 'n' in field list query: SELECT n.nid, n.vid FROM node INNER JOIN files f ON n.nid = f.nid in drupal-cvs/includes/database.mysql.inc on line 66. :-) ------------------------------------------------------------------------ Mon, 09 May 2005 21:29:19 +0000 : Dries Or this: user error: Unknown column 'log' in 'field list' query: SELECT parent, weight, log FROM book WHERE nid = 1 in drupal-cvs/includes/database.mysql.inc on line 66 ------------------------------------------------------------------------ Mon, 09 May 2005 21:52:12 +0000 : Dries The time required to generate my main page went from 902 ms (before upgrade) to 2139 ms (after upgrade). The time required to generate a forum listing (?q=forum/x) went from 1872 ms (before upgrade) to 2874 ms (after upgrade). Maybe this is because my database is not consistent as the result of the upgrade errors (yet I don't see any errors on the pages I benchmarked). ------------------------------------------------------------------------ Tue, 10 May 2005 00:24:38 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_28.patch (53.47 KB) Ok, let me get to this from the bottom to the top: - my test runs indicated a different development wrt timing. If I had gotten your results, I had stopped working on this long ago. So your results are wrong for some reason. - user error: Unknown column 'log' in 'field list' Wasn't my day, the book patch got lost. It is contained now. First -R the old patch, then apply this one. - Unknown table 'n' in field list query: Walkah found this, but I forgot to fix it. Fixed now. - I've no idea where the other queries come from. I am hoping that either your test db is borken or they are follow ups from the other ones. If you let me have your test db, I'll try some debugging. Thanks for wasting your time, too. ------------------------------------------------------------------------ Tue, 10 May 2005 05:07:31 +0000 : Dries I double-checked and the numbers don't seem to lie. I'll test some more after work on another machine to make sure it is not platform-specific. ------------------------------------------------------------------------ Wed, 11 May 2005 03:32:47 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_29.patch (54.83 KB) Ok, here I am again. What I did: 1) Ask Dries to let me have drupal.org database 2) get 400MB of SQL inserts... 3) take 23 minutes to import said data 4) Remove all image and project nodes (don't want to install their modules), 11765 nodes left 5) back up data 6) take tests on non-cached /node page (as anonymous user). ab results: -c 1 -n 25: Requests per second: 1.29 [#/sec] (mean) Connection Times (ms) min mean[+/-sd] median max Total: 663 775 179.7 689 1264 7) Do the same for the tracker page: Requests per second: 0.83 [#/sec] (mean) Total: 1182 1199 7.4 1199 1217 8) Apply my patch (rev. 28). 9) run db update and hold breath 10) update times out... 11) play back backup from 5) 12) wait 13) getting annoyed and removing cache, watchdog, and accesslog before playing in backup. 14) wait again. Understand why Dries doesn't try this patch often. Maybe a smaller DB would do for testing? 15) wait more. get really annoyed. 16) Set time limit to 18000 in update.php 17) try again 18) fails again before the second update is completed. 19) curse. 20) delete search stuff from db. Ooops, sooo much smaller... 21) import again, below 2 minutes... 22) rewrite to use extended insert. Found a bug. 23) still does not complete. Mysql logging to the rescue! 24) tid = 0? Not good. 25) Well, the update works fine till node 10834. 5595 nodes done, 6136 to go. 26) Writing shell based update script. Discovery: 24MB aren't enough. Hopefully 64 are. Nope. extended inserts for revisions are apparently not the brightest idea: Huge memory consumption. Hmm, no, all updates got through. Selecting the revisions to put them into old_revisions table screwed it. Learned about CREATE TABLE old_revisions SELECT syntax. Yay! finally. 24 MB are just not enough the update.php script seems to still break. 27) Benchmarks! /node Requests per second: 1.54 [#/sec] (mean) Connection Times (ms) min mean[+/-sd] median max Total: 632 649 40.5 636 791 /tracker Requests per second: 0.86 [#/sec] (mean) Total: 1119 1165 65.8 1160 1461 Ok: So we get an improvement for many node_loads, but none for simple selects from node. More tests can be done. 28) roll new patch Ain't Drupal fun? ------------------------------------------------------------------------ Wed, 18 May 2005 13:38:05 +0000 : Dries I did another round of tests on _another_ machine and it is not looking good: Before upgrade After upgrade ?q= (main page) 218 ms/request 340 ms/request ?q=forum (forum overview) 754 ms/request 1520 ms/request ?q=about (book page) 375 ms/request 5400 ms/request The upgrade process itself gave me a number of 'query was empty' and 'sprintf(): too few arguments' reports. Everything seems to work fine though. Looking at the ?q=about page, I see that the following query is executed twice _and_ that each time, it take more than 2 seconds to complete: SELECT n.nid, n.title, b.parent, b.weight FROM node n INNER JOIN book b ON n.vid = b.vid WHERE n.status = 1 AND n.moderate = 0 ORDER BY b.weight, n.title; --8 SHOW INDEX FROM book; +-------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | +-------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+ | book | 1 | book_parent | 1 | parent | A | 92 | NULL | NULL | | BTREE | | | book | 1 | nid | 1 | nid | A | 369 | NULL | NULL | | BTREE | | +-------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+ 2 rows in set (0.00 sec) The book module does not appear to have a primary key? Sounds like a bad idea so I added one: mysql> ALTER TABLE book ADD PRIMARY KEY nid (nid); Query OK, 369 rows affected (0.02 sec) Records: 369 Duplicates: 0 Warnings: 0 Next, I wanted to make the vid column a unique key in all node tables: mysql> ALTER TABLE node ADD UNIQUE vid (vid); Query OK, 20392 rows affected (0.81 sec) Records: 20392 Duplicates: 0 Warnings: 0 mysql> ALTER TABLE book ADD UNIQUE vid (vid); ERROR 1062: Duplicate entry '0' for key 2 mysql> ALTER TABLE forum ADD UNIQUE vid (vid); Query OK, 10806 rows affected (0.10 sec) Records: 10806 Duplicates: 0 Warnings: 0 As you can see, it fails for the book table which makes me believe there is some inconsistent data ... I set out to fix that: mysql> SELECT nid, COUNT(nid) AS vids FROM book GROUP BY vid HAVING vids > 1; +-----+------+ | nid | vids | +-----+------+ | 871 | 2 | +-----+------+ 1 row in set (0.00 sec) mysql> SELECT title FROM node WHERE nid = 871; Empty set (0.00 sec) mysql> DELETE FROM book WHERE nid = 871; Query OK, 1 row affected (0.00 sec) mysql> ALTER TABLE book ADD UNIQUE vid (vid); Query OK, 368 rows affected (0.01 sec) Records: 368 Duplicates: 0 Warnings: 0 Looks like everything is well now. Ran some new benchmarks: Before upgrade After upgrade With indices ?q= (main page) 218 ms/request 340 ms/request 336 ms/request ?q=forum (forum overview) 754 ms/request 1520 ms/request 1531 ms/request ?q=about (book page) 375 ms/request 5400 ms/request 475 ms/request Unfortunately, we're still slower than the original code. ------------------------------------------------------------------------ Wed, 18 May 2005 21:53:31 +0000 : killes(a)www.drop.org Dries, thanks for testing it again. I do think the broken queries you observer have something to do with the bad performance after the update. Please log the queries any I will have a look at them. I've never seen any such queries. My update script also tries to create the appropriate indices, but it will of course fail if the database contains cruft. the indices for the forum are probably missing, too. I am still convinced that the patch is core worthy. ------------------------------------------------------------------------ Thu, 19 May 2005 04:36:09 +0000 : Dries It wouldn't hurt if more people would benchmark this patch. The patch's current performance worries me. Did you check your watchdog messages after upgrading the drupal.org database? Depending on your settings, errors might only be shown in the watchdog. I'll look into the remaining glitches as time permits. Thanks for your persistence in keeping this patch up-to-date. :) ------------------------------------------------------------------------ Thu, 19 May 2005 11:59:22 +0000 : killes(a)www.drop.org Dries: Can you please let me have your updated database? I want to have a look at it and try my own benchmarks with it. And yes, if I did learn something on this project is how to be persistant. ;) ------------------------------------------------------------------------ Fri, 24 Jun 2005 16:25:34 +0000 : killes(a)www.drop.org Here is an idea that occurred to me: The problem with the upgrade process is that keeping the existing revisions requires a lot of work to do. This generates a huge amount of sql queries for a large database and also requires a huge amount of memory. My suggestions is to let update.php only handle the basic upgrade, ie without old revisions. An additional module could be created that would implement a cron based approach to upgrading old revisions one node at a time. it could expose a hook to let contrib modules do their own upgrades. Dries, what do you think? (I am writing "Dries" because he seems to be the only one who is interested in getting this into core...) ------------------------------------------------------------------------ Fri, 24 Jun 2005 22:25:11 +0000 : Junyor Killes: I'm also interested in seeing this hit core. What about adding something to legacy.module to do it? ------------------------------------------------------------------------ Sun, 26 Jun 2005 21:14:54 +0000 : chx This is a sensible approach. Maybe this is the _only_ sensible approach. I have a little problem though: while the conversion is running somehow both revision handlers should be available. ------------------------------------------------------------------------ Sun, 26 Jun 2005 22:16:21 +0000 : killes(a)www.drop.org hehe, one only has to whine and bug enough and one gets some feedback. ;) @junyor: legacy.module would be a good place. my current idea is to auto-enable it in update.php and then disable it again in legacy_cron after all nodes are updated.. ;) @chx: When somebody wants to look at revisions of a node that node could be auto-updated. The only problem are contrib modules: they's need to have some hook in order to update their own data. When somebody looks at the revisions of a node than cannot be updated because the contrib module in question has no such hook, we can optionally let the user discard old revisions I guess. Dries, what do you think? ------------------------------------------------------------------------ Mon, 25 Jul 2005 16:48:51 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_30.patch (53.13 KB) Sooooo; I've updated this patch once again. Dries didn't like my idea of legacy updates so we will have an option to discard old revisions in case the update should prove difficult. Always keep a backup. ------------------------------------------------------------------------ Mon, 25 Jul 2005 17:31:12 +0000 : Bèr Kessels can we please accept and commit this patch? We can iron out any issues later. This patch is just far too big and complex to be perfect-at-once.

1 0

[drupal-devel] [bug] Put revisions in their own table
by Bèr Kessels 25 Jul '05

25 Jul '05

Issue status update for http://drupal.org/node/7582 Post a follow up: http://drupal.org/project/comments/add/7582 Project: Drupal Version: cvs Component: node system Category: bug reports Priority: normal Assigned to: killes(a)www.drop.org Reported by: killes(a)www.drop.org Updated by: Bèr Kessels Status: patch can we please accept and commit this patch? We can iron out any issues later. This patch is just far too big and complex to be perfect-at-once. Bèr Kessels Previous comments: ------------------------------------------------------------------------ Wed, 05 May 2004 16:25:27 +0000 : killes(a)www.drop.org Currently all node revisions are stored in a serialized field in node.table and retrieved for _each_ page view although they are rarely needed. However, we have agreed that serializing data is bad and that we should try to keep the memory foot print pf Drupal small. Therefore I propose to create a separate revisions table which would be in principle identical to the node table, only that it could have several old copies of the same node. Extra data added by other modules could be added in a serialized field unless we find a better solution. ------------------------------------------------------------------------ Wed, 05 May 2004 17:06:35 +0000 : jhriggs I too think the serialized approach is less than desirable, but here's an alternative. This would likely take some considerable rework in core and contrib, but the following is how we handle similar types of situations in our databases at work. It is more elegant that a separate table, and avoids the (almost exact) duplication of a table. Instead of separate tables, keep all revisions of nodes in the node table as follows: * add field: active (0/1 or Y/N) * add field: revision * every revision of a node is stored in the node table; however, only one revision can be active at any given time * nid can no longer be unique -- primary/unique key becomes (nid, active) * any time a node is loaded, updated (without revision), etc., the active version is used. Thoughts? ------------------------------------------------------------------------ Wed, 05 May 2004 17:57:48 +0000 : killes(a)www.drop.org I am not opposed to your scheme, but I want to stress the following: * Duplicating a table's structure is not bad (IMHO) as long as the content is different. * having two tables will allow us to have a rather small node table. This is (maybe) a performance gain. ------------------------------------------------------------------------ Wed, 05 May 2004 18:37:29 +0000 : jhriggs I don't necessarily think that duplicating a table's structure is _bad_. It just seems to be wasteful and a pain to maintain. (Every change to the node table is made twice...easy to do, but also easy to miss perhaps.) As for performance, as long as nid and the active indicator are indexed, there shouldn't be any performance loss. Also, archiving an old version when making a new revision will be much simpler: just change the active indicator rather than copying an entire node to another table (and ensuring everything gets copied...again a potential maintainance issue). To be honest, I would just like to see the serialized data go away, regarless of what approach is taken. ------------------------------------------------------------------------ Fri, 30 Jul 2004 19:49:33 +0000 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_07-30-2004.p… (10.47 KB) I'm interested in using Drupal for a large scale wiki-type project. In order to do this, I need revisions to be in their own table. Attached is a patch to do just that. Most of the changes are pretty self explanatory. Spreading out node data across two tables meant that I had to add database functions to do locking/transactions. Without this, race conditions in which the database becomes corrupted are possible. ------------------------------------------------------------------------ Fri, 30 Jul 2004 19:54:36 +0000 : Nick Nassar Oh yeah... The patch is a diff against Drupal CVS ------------------------------------------------------------------------ Sat, 31 Jul 2004 00:00:08 +0000 : Anonymous Gerhard speaking. Nick, thanks a lot for your nice patch! It saves me a great deal of labour. I looked through it and immediately liked it. You not only put the old revisions into a new table but also the current one. Do you have an estimate how much more expensive the additional join is? Besides a few minor coding style issues I found a major one: Just a few hours before you uploaded your patch JonBob's node access patch hit core. That means your patch won't apply anymore as all the queries you change have been changed. Can I bug you to update your patch? ------------------------------------------------------------------------ Sat, 31 Jul 2004 01:11:59 +0000 : Anonymous Also I think that your upgrade path loses existing revisions. ------------------------------------------------------------------------ Sat, 31 Jul 2004 02:39:12 +0000 : drumm I think this is the proper way to do things. No columns are duplicated, there is no serialized data, and only the fields that are logically revised are stored. Nothing jumped out at me as a way to have my node modules be able to keep a table of revisions of additional fields. I'm guessing this could be done within the confines of _insert and _update. Assuming the upgrade path works and modules can extend it I give it a +1. ------------------------------------------------------------------------ Sat, 31 Jul 2004 14:40:15 +0000 : Nick Nassar It figures that just as I finish a big patch, another patch comes along and breaks it. Oh well, it should be a pretty easy to fix. I'll work on it. Fixing the upgrade path to keep revisions should be fairly painless. I found another issue that needs to be fixed before this patch gets merged. There format of a node needs to be stored for each revision. Otherwise, for modules that store a format for the nodes, such as page and book, if you write one revision in PHP and the next in HTML, the PHP revision will be displayed as HTML. This is part of a larger issue of how node modules should store revisions of additional fields. I think each module that wants to do this should create another table with (nid, revid) as the primary key. Just as when they want to add fields to a node they create another table with nid as the primary key. As far as performance goes, for sites that make heavy use of revisions, an extra join on primary keys is going to be a lot faster than grabbing all of the revisions from that database everytime. We would need to run benchmarks to determine is the overall difference in speed is for an average site is a gain or a loss. I'm guessing it's very minor either way. ------------------------------------------------------------------------ Mon, 23 Aug 2004 13:55:49 +0000 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_08-23-2004.p… (10.92 KB) Here's an updated patch against CVS that puts revisions in their own table, provides an upgrade path, and fixes the format related bugs in the last patch. Hopefully, this can make it into CVS as soon as the freeze is over. ------------------------------------------------------------------------ Mon, 23 Aug 2004 14:10:39 +0000 : moshe weitzman Interesting patch ... drumm's question is still outstanding. how do modules store revisions of their fields? Are they expected to manage this on their own? Thats not how it works today. As an aside, i am seeing profile_ fields in my node.revisions column. One could argue that those need not be saved. They pertain to the node author, not to the node itself. ------------------------------------------------------------------------ Mon, 23 Aug 2004 16:14:39 +0000 : Nick Nassar Having modules be responsible for storing revisions of their own fields is a side-effect of storing revision data in tables. There's really no way around it. However, revisions generally don't make sense for node types that don't have PHP/HTML content, such as polls. I think it's going to be a pretty rare scenario for a new node type to want another field to change per-revision, so it's a pretty good trade-off. Storing fields that shouldn't be part of revisions, such as the profile_ fields, is a side-effect of storing revisions as serialized objects. Applying this patch will free up that wasted space. :) ------------------------------------------------------------------------ Mon, 23 Aug 2004 17:20:57 +0000 : Anonymous There should be a hook that let's the module choose whether it supports history. This way a module author can prevent the user from doing something that may break his module or just cause undefined behavior. If the module doesn't support history then don't let the user/admin choose to add history to nodes of that type. Craig ------------------------------------------------------------------------ Mon, 23 Aug 2004 19:23:29 +0000 : Nick Nassar I agree, there should be an API change to make specifying support for revisions easier. In the interests of keeping patches small and keeping to one change per patch, I think the API change should be a separate issue. A sort of ad-hoc API to decide whether or not a module supports revisions by default already exists. Instead of having a hook, modules set the default value of the "Create new revision" field in the edit form. The admin can change this option in admin/node/configure/defaults. This patch doesn't change that. Revisions are broken for node types that have their own database structure, like polls, even when storing them as serialized objects. This patch doesn't change that, either. ------------------------------------------------------------------------ Tue, 26 Oct 2004 02:35:06 +0000 : moshe weitzman I'm guessing that someone is going to have to demonstrate that this patch performs as well as current drupal before it gets comitted. i think this patch is a few benchmarks from being comitted. ------------------------------------------------------------------------ Wed, 27 Oct 2004 01:04:09 +0000 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004.p… (11 KB) I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ Wed, 27 Oct 2004 01:05:16 +0000 : Nick Nassar I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ Wed, 27 Oct 2004 01:05:26 +0000 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004.p… (11 KB) I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ Mon, 15 Nov 2004 05:05:30 +0000 : elias1884 please overthink the revision system default workflow as well. don't look at the revision system as an isolated system but as a part of the whole workflow system! if you combine revisions with the moderation queue the most logic default workflow would be like that: auth user creates node (revision #0) admin approves the node (status = 1, moderation = 0) => node publicly available auth user finds typo and changes node (revision #1, status = 0, moderation = 1) ------- what happens at that point at the moment is, that the node is not accessible anymore at all until the new revision is approved by admin. of course the new revision should not go online until reviewed and approved, this is absolutely correct, but there is no reason to not take the old revision offline, since it was already approved and should therefore be online until the new revision is approved. it is not practical if a node disappears only because the author corrected a typo. ------- admin approves the node (status = 1, moderation = 0) eventhough I first thought a plain boolean active field would not be capable of providing that functionality if finally came to the conclusion, that it can. The only thing to do is to not set that bit, when a new revision is created, but when it is approved (in case moderation is activated under default workflow). Every revision should have its own moderation, status and active field and on approval they are set like this (status=1, moderation=0, active=Y). When you wanna rollback to an old revision, you can chose between all revisions that already have the moderation bit set back to 0 again and the published to 1. There should be an extra permission for rollback! another concern that I have about the default workflow is, that users can't see the content, they have just created, when moderation is enabled. Eventhough, there is a big fat "submission accepted" presented after submissions, unexperienced users tend to question the information those stupid tincans give them, if they can't find their content afterwards. Many users are really lazy bastards and they don't even read the status messages. The best feedback about whether his story was submitted successfully or not of course is, if he can find the story somewhere on the site, maybe with a status message on top of it, mentioning, that the content is currently not publicly available since it has not been approved yet. there should be a my content section under my account, like somebody is trying to do with the workspace module I guess. so my suggestion is to make (status=0, moderation=1) still available for the creator under a my content section somewhere! ------------------------------------------------------------------------ Wed, 24 Nov 2004 04:21:18 +0000 : Nick Nassar I agree. The current workflow for moderation queues and revisions needs to change, but this patch isn't the place for it. The patch is already too big, and it only does the backend stuff. Instead of adding more to this patch and making it take even longer to get into core, would you mind creating a new issue for your UI suggestions, so the those changes can be added as a separate patch? Thanks, Nick ------------------------------------------------------------------------ Sat, 11 Dec 2004 12:26:02 +0000 : Dries This patch is _much_ needed so I'd love to see someone revive it. In order for this patch to be accepted, the following needs to be done: * Update this patch to CVS HEAD. * Rename revid to vid. * Rename node_rev to node_revisions. * Rename node_rev.changed to node_revisions.timestamp. * Rename $rnode to $revision. * Fix the coding style to match Drupal's: proper spacing, single quotes where possible, proper variable names. * Benchmark this patch with a large database with enough revisions. I'd be happy to benchmark this on my local copy of the drupal.org database. * The book.log field should probably move to the node_revisions table. This can be done in a separate patch. * Investigate whether transactions are well-supported. ------------------------------------------------------------------------ Mon, 13 Dec 2004 00:25:40 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004-r… (11.02 KB) I've worked a bit on the patch (coding style issues as mentioned by Dries). One thing I noticed is that the patch uses REPLACE. IIRC this needs to be chagned to "UPDATE, if fail INSERT" for pgsql compatibility. Nick, are you still interested in working on that patch? I'd like to know how it works on your site and work on getting it into core. ------------------------------------------------------------------------ Mon, 13 Dec 2004 12:33:08 +0000 : Dries Gerhard: your patch does not apply. ------------------------------------------------------------------------ Mon, 13 Dec 2004 17:10:12 +0000 : killes(a)www.drop.org Yes, I know, that was the same version as I mailed to you earlier. ------------------------------------------------------------------------ Mon, 13 Dec 2004 21:02:06 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions.patch (52.96 KB) Ok, upüdated the patch to cvs. ------------------------------------------------------------------------ Tue, 14 Dec 2004 08:58:36 +0000 : Dries Some more comments: * db_begin_transaction() and db_end_transaction() do not belong in database.inc, but in database.mysql.inc and database.pgsql.inc respectively. * The node module calls node_revisionsision_list() which is not defined. (Fxed that on my local copy.) * Do db_begin_transaction() and db_end_transaction() deprecate Jeremy's table locking patch? * The upgrade path assigns the wrong user ID to each revision. * The upgrade path assigns the wrong date to each revision (that or a node's revision page shows the wrong usernames/dates). * The coding style needs a bit of work, but we can worry about that later. ------------------------------------------------------------------------ Tue, 14 Dec 2004 17:34:44 +0000 : Nick Nassar If you need any help getting those things fixed, just let me know. ------------------------------------------------------------------------ Tue, 14 Dec 2004 17:50:30 +0000 : Nick Nassar How this relates to Jeremy's node locking patch: There was lots of discussion, and node locking was decided against because from an end user point of view you never want a node to be locked. He's now advocating for a much simpler patch that warns users if their changes will overwrite someone elses. That patch still has a race condition, which might be fixed using db_begin_transaction(). http://drupal.org/node/6025 ------------------------------------------------------------------------ Tue, 14 Dec 2004 22:26:19 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_0.patch (55.96 KB) Here is an updated patch that tries to address Dries concerns. ------------------------------------------------------------------------ Wed, 15 Dec 2004 08:32:50 +0000 : Dries Attachment: http://drupal.org/files/issues/revisions-bug.png (76.06 KB) It didn't fix the aforementioned bugs. See attached screenshot. ------------------------------------------------------------------------ Thu, 06 Jan 2005 20:15:01 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_1.patch (51.77 KB) Ok, here is a new version. Dries and myself worked hart at it, so please have a look. what is still missing - database upgrades for the core modules with an own table - contrib modules need an upgrade too. - do we need nid and vid in both the node and the node_revisions table? - the amount of sql queries means a good stress testing for large databases. ------------------------------------------------------------------------ Wed, 19 Jan 2005 21:43:49 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_2.patch (49.49 KB) Here is an updated patch. We discussed to keep the current title in node module and also in the revisiosn table. This is content duplication but will save many joins as many queries only need the title of a node. Discussion is welcome. ------------------------------------------------------------------------ Wed, 19 Jan 2005 23:33:32 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_3.patch (29.93 KB) I've implemented the aforementioned solution. This makes the patch much smaller. The patch now also removes taxonomy_node_has_term() which wasn't used anywhere. I'd really apprciate if some people could test drive the patch. It will be another huge improvement for 4.6. ------------------------------------------------------------------------ Thu, 20 Jan 2005 00:05:54 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_4.patch (30 KB) Another revision. Steven didn't like my literal $node->vid in queries. ------------------------------------------------------------------------ Thu, 20 Jan 2005 01:10:50 +0000 : killes(a)www.drop.org - database upgrades for the core modules with an own table - contrib modules need an upgrade too. - do we need nid and vid in both the node and the node_revisions table? - the amount of sql queries means a good stress testing for large databases. These issues are still open, btw. Especially the first one needs to be tackled. ------------------------------------------------------------------------ Tue, 25 Jan 2005 20:11:59 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_5.patch (51.13 KB) Here is a patch that has the database tables updated for forum, book, and page module. ------------------------------------------------------------------------ Sat, 29 Jan 2005 22:55:59 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_6.patch (49.18 KB) Yet another update to keep it working with head. The patch now also removes the table definitons for the page table. ------------------------------------------------------------------------ Sat, 29 Jan 2005 22:57:40 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_7.patch (55.69 KB) Sorry, that was the old version, this is the right one. ------------------------------------------------------------------------ Mon, 31 Jan 2005 19:55:03 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_8.patch (55.71 KB) Updated once more. ------------------------------------------------------------------------ Mon, 31 Jan 2005 20:52:08 +0000 : Dries Anyone to help review/test this? ------------------------------------------------------------------------ Mon, 31 Jan 2005 21:22:36 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_9.patch (49.29 KB) Updated again, the update functions occurred twice. Thanks Bart. ------------------------------------------------------------------------ Wed, 02 Feb 2005 00:27:05 +0000 : killes(a)www.drop.org Don't know if the db I am using is corrupted or what. I still do have some didficulties. The latest patch is attached. ------------------------------------------------------------------------ Wed, 02 Feb 2005 00:27:49 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_10.patch (49.67 KB) I am probably slowly going mad ... ------------------------------------------------------------------------ Wed, 02 Feb 2005 01:54:58 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_11.patch (48.95 KB) The update issue still needs investigating. This patch is updated for cvs. ------------------------------------------------------------------------ Wed, 02 Feb 2005 20:20:51 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_12.patch (49.83 KB) Ok, here is a new version. I've solved my troubles with book.module. There are still some issues with forum module. Possibly due to inconsistent database. ------------------------------------------------------------------------ Wed, 02 Feb 2005 21:31:05 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_13.patch (49.83 KB) Turns out the drupal.org database had indeed some quirks. Please run this query in your oldest db and tell me the result: select nid,type from node where type like '%/%'; If you get a non-zero result we might need to add another security update. The patch could use still more testing, though. ------------------------------------------------------------------------ Thu, 03 Feb 2005 01:16:54 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_14.patch (49 KB) Ok, we are getting somewhere. At a first glance the update is working. There is a problem remaining: the revisions tab will be shown whether the node has revisions or not. Not sure we can/need to fix this. People with a drupal.org account can log in at http://killes.drupaldevs.org/revision/ and poke around. Your permissions will be the same as on drupal.org. Feel free to vreak everything but don't forget to file complaints here. (Note: this is only a pruned version of the drupal.org database with all project nodes and nodes with nids > 7000 dropped). ------------------------------------------------------------------------ Thu, 03 Feb 2005 04:19:14 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_15.patch (52.39 KB) There was some error in node_save and also the patches to the database.inc files got lost... ------------------------------------------------------------------------ Thu, 03 Feb 2005 07:07:27 +0000 : robertDouglass Submitting book pages doesn't work on your test site. It puts the entire content of the preview inside the body textarea. I wrote a sentence in the body and the log, and pressing preview put several lines of HTML containing both sentences in the body textarea on the preview page, plus the book page wouldn't submit. -R ------------------------------------------------------------------------ Thu, 03 Feb 2005 07:50:59 +0000 : Junyor 0 results here. I started using Drupal with version 4.4, though. ------------------------------------------------------------------------ Thu, 03 Feb 2005 23:56:18 +0000 : killes(a)www.drop.org @Junyor: Thanks, that's a good sign. Maybe somebody else has an older db to try. @robertDouglass: The first effect you describe is due to drupaldevs running on PHP 5. I am unsure why the second thing does not work. In node_save() the node object has a nid although there is none in the form. Very strange. I've enabled display of db queries on the testsite. ------------------------------------------------------------------------ Fri, 04 Feb 2005 19:17:55 +0000 : dmjossel No results here on the query: select nid,type from node where type like '%/%'; On a database that was put in place prior to Drupal 4 and is now running on 4.5.2. ------------------------------------------------------------------------ Fri, 04 Feb 2005 20:44:23 +0000 : killes(a)www.drop.org @dmjossel: thanks. @all. The strange problem I reported was apparently php 5 related. After applying Steven's php 5 patch it went away. One error is remaining: If I create a new forum topic it will be shown as part of the book on preview. Hmm, that was due to a db that got corrupted during testing so that is fixed too. Please poke around at the test site and look if you find more errors. ------------------------------------------------------------------------ Sat, 05 Feb 2005 07:16:22 +0000 : Steven By the way, I just remembered that Drupal.org has some blogs lingering on in the database even though blog.module is not enabled. Perhaps this is causing troubles? ------------------------------------------------------------------------ Sat, 05 Feb 2005 11:22:59 +0000 : Anonymous I can't see why it would. Drupal.org will need extra updates for images and project nodes because those have their own tables. GK. ------------------------------------------------------------------------ Sun, 06 Feb 2005 12:49:55 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_16.patch (52.49 KB) Updated to apply to cvs again. ------------------------------------------------------------------------ Tue, 22 Feb 2005 20:15:40 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_17.patch (49.64 KB) Updated again. All we need is a patch to upload module and an upgrade path for it. ------------------------------------------------------------------------ Fri, 04 Mar 2005 04:22:58 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_18.patch (52.31 KB) Updated once more. Moved log field from book to node_revisions table as discussed in Antwerp. upload module still missing. We need to decide under which circumstances the log field should be displayed. Should that be added to the workflow? Should it depend on the revisions setting? ------------------------------------------------------------------------ Sat, 05 Mar 2005 19:27:03 +0000 : Anonymous Attachment: http://drupal.org/files/issues/revisions_20.patch (75.52 KB) Ok, here it is: Yet another revision of this grrrrreat patch. Changes from previous versions: - supports versioning for uploaded files. A problem is that if you delete a file, it will be gone for all revisions. - the log field is now in the node_revsions table, but each module has to decide whether to show it or not. I've implemented it for the page and the book type odes. Also, the field can be edited when adding non-book nodes to the book. The log is displayed on the revisions page and if a node is moderated. - the revisions are moved to an old_revisions table to a) get the node table smaller and b) still leave the mavailable for contrib modules that want to retreive old version data. The patch has been applied to killes.drupaldevs.org/revision where it can be tested by anybody (especially people who have "site admin" rights on drupal.org) The database is from drupal.org and you should b able to log in with your pass or simply mail yourself a new one. Gerhard ------------------------------------------------------------------------ Sat, 05 Mar 2005 19:51:56 +0000 : Anonymous Attachment: http://drupal.org/files/issues/revisions_21.patch (59.42 KB) BTW, I marked this a bug because atm the revisions field can grow quite big. Neil has reported problems from some users who were not able to load some nodes due to to many large revisions. Also, som unrelated stuff crept into the patch. New version attached. ------------------------------------------------------------------------ Tue, 08 Mar 2005 05:56:01 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_22.patch (60.29 KB) Ok, I think I got it. Changes to last version: - uploads are no properly versioned. Missing are still pgsql checks and updates. ------------------------------------------------------------------------ Thu, 10 Mar 2005 16:58:41 +0000 : Anonymous Was able to get http://drupal.org/files/issues/revisions_21.patch to work with drupal-cvs.tar.gz (10 March 2005) by: - includes/database.mysql.inc: Commenting out duplicates for functions function db_begin_transaction and function db_commit_transaction - modules: node.module: Removing "'title' => $node->title," from $node_table_values variable declaration and removing "'title' => "'%s'"," from "$node_table_types" variable declaration. Happy to submit a patch if requested. I'll watch this thread. ------------------------------------------------------------------------ Fri, 11 Mar 2005 23:59:45 +0000 : killes(a)www.drop.org The duplicate function has been removed in rev 22 of this patch. Why do you think the changes in node_save are needed? Titles are saved in both tables for performance reasons. ------------------------------------------------------------------------ Sun, 13 Mar 2005 16:12:21 +0000 : jlerner Hi - I posted comment #62. The changes to node_save appear to be needed because recent patches (both 21 and 22) remove the field 'title' from table 'node'. So without the changes to node_save, node.module is broken and generates errors. Joshua ------------------------------------------------------------------------ Sun, 13 Mar 2005 16:29:42 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_23.patch (61.17 KB) Thanks, Joshua, for catching this. node:title is there to stay. ------------------------------------------------------------------------ Wed, 13 Apr 2005 16:29:23 +0000 : moshe weitzman since HEAD is open again, perhaps it is a good time to revisit this patch. once this is committed, lets address - http://drupal.org/node/11071 "node_validate does not respect group editing" ------------------------------------------------------------------------ Mon, 18 Apr 2005 15:43:42 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_24.patch (60.39 KB) Updated. ------------------------------------------------------------------------ Mon, 18 Apr 2005 16:16:32 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_24_0.patch (60.39 KB) Updated. ------------------------------------------------------------------------ Tue, 19 Apr 2005 05:19:42 +0000 : Dries I'll commit this patch later this week! If you haven't checked this patch already, I urge you to test/check it out because it will have significant impact on existing code and modules! ------------------------------------------------------------------------ Tue, 19 Apr 2005 05:21:33 +0000 : Dries Also, what do people think about the n.title being duplicated? ------------------------------------------------------------------------ Tue, 19 Apr 2005 05:26:58 +0000 : chx I won't lose any sleep because of duplicated titles... ------------------------------------------------------------------------ Tue, 19 Apr 2005 18:35:58 +0000 : killes(a)www.drop.org Let me explain why I have chosen to duplicate the title (and also the uid): If you look at all the queries in Drupal, you will find that most of them only need the title and th uid of a node. That is, by duplicating it, we save expensive joins on the node_revisions table. Due to this fact, this patch is actually a performance improvement. A note about updating contrib module: Strictly speaking they wouldn't need to be updated. They only need to if their authors decide that their info should be available for revisioning. The upgrade path for forum.module in my update.inc patch (+ the forum patch) should show you what needs to be done. I will write a note for the update page once the patch hits core. ------------------------------------------------------------------------ Sun, 24 Apr 2005 21:21:19 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_25.patch (60.38 KB) Updated to cvs. Dries: Based on some remarks in #drupal this is the last update I am going to do. Apply it or won't fix it. ------------------------------------------------------------------------ Sat, 30 Apr 2005 03:42:39 +0000 : Jeremy(a)kerneltrap.org Attachment: http://drupal.org/files/issues/revisions_25.patch.patch (528 bytes) That's a big patch. I've only started looking through it. I noticed one little typo, affecting updates. A patch to your last patch is attached. I'm running with the revision patch on my dev server now happily. I like the concept. What happens if you click 'stop' on your browser in the middle of a MySQL "transaction"? I assume that kills the connection to MySQL, and the lock is freed? But this then leaves changes only partially applied? What exactly does locking/unlocking the tables buy us in MySQL? I don't see anywhere that we detect if an apply fails part way through, and thus roll back...? What am I missing? ------------------------------------------------------------------------ Sat, 30 Apr 2005 07:11:28 +0000 : Dries Jeremy: many of us are worried about the performance ramifications this patch introduces. Early experiments showed a small performance improvement (while a performance regression might be expected). More performance reports from large sites like kerneltrap.org will certainly help this patch. Mind to do a quick performance comparision and to report back with some numbers? Thanks. ------------------------------------------------------------------------ Sat, 30 Apr 2005 12:38:02 +0000 : Jeremy(a)kerneltrap.org Dries: I'm not running HEAD on kerneltrap, so this really isn't a possibility. Furthermore, until I understand why we're locking tables, I don't like it. The idea of revisions in their own tables is great. The idea of locking tables to get (without any obvious benefit) there really worries me. ------------------------------------------------------------------------ Sat, 30 Apr 2005 14:16:01 +0000 : killes(a)www.drop.org @Jeremy: Thanks for looking at the patch! Also for catching the typo. :) Did you try to upgrade your database? If yes, how did it go? One of Dries' concerns is the complexity of the upgrade. How many nodes and revisions did the db have? About database locking: This part of the patch was created by Nick and I simply continued to use it. Maybe the code should rather be: if(db_begin_transaction(array('{node}', '{node_revisions}', '{watchdog}', '{sessions}', '{files}'))) { db_query($node_query, $node_table_values); db_query($revisions_query, $revisions_table_values); db_commit_transaction(); ... } The idea is probably to avoid two updates at the same time. I don't think the locking helps if you abort the script at an inconvenient time. Rollbacks aren't implemented in all mysql versions. We could omit the db locking if deemed inappropriate. Maybe Nick can explain his ideas behind this. @Dries: I wonder who the "many of us" are. They certainly haven't spoken to me. Moshe had some reservations about the upgrade path and project module, but the time that project module abused revisions to store issue updates was long ago and his reservations were resolved. Nobody else (besides you of course and now Jeremy) has voiced reservations in a way that was audible to me. If you grep through the patch you will notice that there are only four queries which have a join on the node_revisions table. Two of them are in node_load and in the other cases the join replaced a join on the node table. The two queries in node_load are the only ones that have both a join on the node and the revisions query. Thus, loading of individual nodes might become somewhat slower. All other queries will be faster since the node table is now much smaller. Also, node loading does not have to be slower, it depends on your node table. If you had a lot of revisions and thus a large table, then the new scheme will make your queries actually faster since we do not load the revisions on each and every node load anymore. If you didn't have many revisions your node_load migth be somewhat slower. WRT to the update script Karoly pointed out that we could use multiple insert queries instead one query per revision. This would probably make the update somewhat faster. I am willing to work on this iff you declare that you will commit the patch afterwards. I'd need to know if this will work on pgsql and on all supported mysql versions before I work on it. About locking: Database locking is dog slow, at least on mysql. I was using locks in an earlier version of the upgrade script but had to remove it for (serious!) performance reasons. ------------------------------------------------------------------------ Mon, 09 May 2005 15:07:34 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_26.patch (46.45 KB) Ok, another update, cause I need it myself. I've left out the transaction stuff for now. It is in principle unrelated to this patch and should be discussed elsewhere. This also makes the patch smaller and easier to review (hint, hint). ------------------------------------------------------------------------ Mon, 09 May 2005 20:32:09 +0000 : killes(a)www.drop.org The patch contained the update functions twice. ------------------------------------------------------------------------ Mon, 09 May 2005 20:32:26 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_27.patch (39.05 KB) The patch contained the update functions twice. ------------------------------------------------------------------------ Mon, 09 May 2005 21:23:06 +0000 : Dries Got one error during the upgrade path: ALTER TABLE {book} ADD PRIMARY KEY vid (vid) FAILED ------------------------------------------------------------------------ Mon, 09 May 2005 21:26:19 +0000 : killes(a)www.drop.org This had happend to me as well, when I tested this patch. The reason is that for some reason the vid is not unique. Most likely there are some entries with vid = 0 in there. Can you check which node types those have? it always was an error in the test database. See: http://drupal.org/node/7582#comment-20678 ------------------------------------------------------------------------ Mon, 09 May 2005 21:27:06 +0000 : Dries Actually, I got 2850 errors during the upgrade. Some of these: sprintf() [function.sprintf]: Too few arguments in drupal-cvs/includes/database.inc on line 154. Some of these: Query was empty query: in drupal-cvs/includes/database.mysql.inc on line 66. And this: Unknown table 'n' in field list query: SELECT n.nid, n.vid FROM node INNER JOIN files f ON n.nid = f.nid in drupal-cvs/includes/database.mysql.inc on line 66. :-) ------------------------------------------------------------------------ Mon, 09 May 2005 21:29:19 +0000 : Dries Or this: user error: Unknown column 'log' in 'field list' query: SELECT parent, weight, log FROM book WHERE nid = 1 in drupal-cvs/includes/database.mysql.inc on line 66 ------------------------------------------------------------------------ Mon, 09 May 2005 21:52:12 +0000 : Dries The time required to generate my main page went from 902 ms (before upgrade) to 2139 ms (after upgrade). The time required to generate a forum listing (?q=forum/x) went from 1872 ms (before upgrade) to 2874 ms (after upgrade). Maybe this is because my database is not consistent as the result of the upgrade errors (yet I don't see any errors on the pages I benchmarked). ------------------------------------------------------------------------ Tue, 10 May 2005 00:24:38 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_28.patch (53.47 KB) Ok, let me get to this from the bottom to the top: - my test runs indicated a different development wrt timing. If I had gotten your results, I had stopped working on this long ago. So your results are wrong for some reason. - user error: Unknown column 'log' in 'field list' Wasn't my day, the book patch got lost. It is contained now. First -R the old patch, then apply this one. - Unknown table 'n' in field list query: Walkah found this, but I forgot to fix it. Fixed now. - I've no idea where the other queries come from. I am hoping that either your test db is borken or they are follow ups from the other ones. If you let me have your test db, I'll try some debugging. Thanks for wasting your time, too. ------------------------------------------------------------------------ Tue, 10 May 2005 05:07:31 +0000 : Dries I double-checked and the numbers don't seem to lie. I'll test some more after work on another machine to make sure it is not platform-specific. ------------------------------------------------------------------------ Wed, 11 May 2005 03:32:47 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_29.patch (54.83 KB) Ok, here I am again. What I did: 1) Ask Dries to let me have drupal.org database 2) get 400MB of SQL inserts... 3) take 23 minutes to import said data 4) Remove all image and project nodes (don't want to install their modules), 11765 nodes left 5) back up data 6) take tests on non-cached /node page (as anonymous user). ab results: -c 1 -n 25: Requests per second: 1.29 [#/sec] (mean) Connection Times (ms) min mean[+/-sd] median max Total: 663 775 179.7 689 1264 7) Do the same for the tracker page: Requests per second: 0.83 [#/sec] (mean) Total: 1182 1199 7.4 1199 1217 8) Apply my patch (rev. 28). 9) run db update and hold breath 10) update times out... 11) play back backup from 5) 12) wait 13) getting annoyed and removing cache, watchdog, and accesslog before playing in backup. 14) wait again. Understand why Dries doesn't try this patch often. Maybe a smaller DB would do for testing? 15) wait more. get really annoyed. 16) Set time limit to 18000 in update.php 17) try again 18) fails again before the second update is completed. 19) curse. 20) delete search stuff from db. Ooops, sooo much smaller... 21) import again, below 2 minutes... 22) rewrite to use extended insert. Found a bug. 23) still does not complete. Mysql logging to the rescue! 24) tid = 0? Not good. 25) Well, the update works fine till node 10834. 5595 nodes done, 6136 to go. 26) Writing shell based update script. Discovery: 24MB aren't enough. Hopefully 64 are. Nope. extended inserts for revisions are apparently not the brightest idea: Huge memory consumption. Hmm, no, all updates got through. Selecting the revisions to put them into old_revisions table screwed it. Learned about CREATE TABLE old_revisions SELECT syntax. Yay! finally. 24 MB are just not enough the update.php script seems to still break. 27) Benchmarks! /node Requests per second: 1.54 [#/sec] (mean) Connection Times (ms) min mean[+/-sd] median max Total: 632 649 40.5 636 791 /tracker Requests per second: 0.86 [#/sec] (mean) Total: 1119 1165 65.8 1160 1461 Ok: So we get an improvement for many node_loads, but none for simple selects from node. More tests can be done. 28) roll new patch Ain't Drupal fun? ------------------------------------------------------------------------ Wed, 18 May 2005 13:38:05 +0000 : Dries I did another round of tests on _another_ machine and it is not looking good: Before upgrade After upgrade ?q= (main page) 218 ms/request 340 ms/request ?q=forum (forum overview) 754 ms/request 1520 ms/request ?q=about (book page) 375 ms/request 5400 ms/request The upgrade process itself gave me a number of 'query was empty' and 'sprintf(): too few arguments' reports. Everything seems to work fine though. Looking at the ?q=about page, I see that the following query is executed twice _and_ that each time, it take more than 2 seconds to complete: SELECT n.nid, n.title, b.parent, b.weight FROM node n INNER JOIN book b ON n.vid = b.vid WHERE n.status = 1 AND n.moderate = 0 ORDER BY b.weight, n.title; --8 SHOW INDEX FROM book; +-------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | +-------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+ | book | 1 | book_parent | 1 | parent | A | 92 | NULL | NULL | | BTREE | | | book | 1 | nid | 1 | nid | A | 369 | NULL | NULL | | BTREE | | +-------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+ 2 rows in set (0.00 sec) The book module does not appear to have a primary key? Sounds like a bad idea so I added one: mysql> ALTER TABLE book ADD PRIMARY KEY nid (nid); Query OK, 369 rows affected (0.02 sec) Records: 369 Duplicates: 0 Warnings: 0 Next, I wanted to make the vid column a unique key in all node tables: mysql> ALTER TABLE node ADD UNIQUE vid (vid); Query OK, 20392 rows affected (0.81 sec) Records: 20392 Duplicates: 0 Warnings: 0 mysql> ALTER TABLE book ADD UNIQUE vid (vid); ERROR 1062: Duplicate entry '0' for key 2 mysql> ALTER TABLE forum ADD UNIQUE vid (vid); Query OK, 10806 rows affected (0.10 sec) Records: 10806 Duplicates: 0 Warnings: 0 As you can see, it fails for the book table which makes me believe there is some inconsistent data ... I set out to fix that: mysql> SELECT nid, COUNT(nid) AS vids FROM book GROUP BY vid HAVING vids > 1; +-----+------+ | nid | vids | +-----+------+ | 871 | 2 | +-----+------+ 1 row in set (0.00 sec) mysql> SELECT title FROM node WHERE nid = 871; Empty set (0.00 sec) mysql> DELETE FROM book WHERE nid = 871; Query OK, 1 row affected (0.00 sec) mysql> ALTER TABLE book ADD UNIQUE vid (vid); Query OK, 368 rows affected (0.01 sec) Records: 368 Duplicates: 0 Warnings: 0 Looks like everything is well now. Ran some new benchmarks: Before upgrade After upgrade With indices ?q= (main page) 218 ms/request 340 ms/request 336 ms/request ?q=forum (forum overview) 754 ms/request 1520 ms/request 1531 ms/request ?q=about (book page) 375 ms/request 5400 ms/request 475 ms/request Unfortunately, we're still slower than the original code. ------------------------------------------------------------------------ Wed, 18 May 2005 21:53:31 +0000 : killes(a)www.drop.org Dries, thanks for testing it again. I do think the broken queries you observer have something to do with the bad performance after the update. Please log the queries any I will have a look at them. I've never seen any such queries. My update script also tries to create the appropriate indices, but it will of course fail if the database contains cruft. the indices for the forum are probably missing, too. I am still convinced that the patch is core worthy. ------------------------------------------------------------------------ Thu, 19 May 2005 04:36:09 +0000 : Dries It wouldn't hurt if more people would benchmark this patch. The patch's current performance worries me. Did you check your watchdog messages after upgrading the drupal.org database? Depending on your settings, errors might only be shown in the watchdog. I'll look into the remaining glitches as time permits. Thanks for your persistence in keeping this patch up-to-date. :) ------------------------------------------------------------------------ Thu, 19 May 2005 11:59:22 +0000 : killes(a)www.drop.org Dries: Can you please let me have your updated database? I want to have a look at it and try my own benchmarks with it. And yes, if I did learn something on this project is how to be persistant. ;) ------------------------------------------------------------------------ Fri, 24 Jun 2005 16:25:34 +0000 : killes(a)www.drop.org Here is an idea that occurred to me: The problem with the upgrade process is that keeping the existing revisions requires a lot of work to do. This generates a huge amount of sql queries for a large database and also requires a huge amount of memory. My suggestions is to let update.php only handle the basic upgrade, ie without old revisions. An additional module could be created that would implement a cron based approach to upgrading old revisions one node at a time. it could expose a hook to let contrib modules do their own upgrades. Dries, what do you think? (I am writing "Dries" because he seems to be the only one who is interested in getting this into core...) ------------------------------------------------------------------------ Fri, 24 Jun 2005 22:25:11 +0000 : Junyor Killes: I'm also interested in seeing this hit core. What about adding something to legacy.module to do it? ------------------------------------------------------------------------ Sun, 26 Jun 2005 21:14:54 +0000 : chx This is a sensible approach. Maybe this is the _only_ sensible approach. I have a little problem though: while the conversion is running somehow both revision handlers should be available. ------------------------------------------------------------------------ Sun, 26 Jun 2005 22:16:21 +0000 : killes(a)www.drop.org hehe, one only has to whine and bug enough and one gets some feedback. ;) @junyor: legacy.module would be a good place. my current idea is to auto-enable it in update.php and then disable it again in legacy_cron after all nodes are updated.. ;) @chx: When somebody wants to look at revisions of a node that node could be auto-updated. The only problem are contrib modules: they's need to have some hook in order to update their own data. When somebody looks at the revisions of a node than cannot be updated because the contrib module in question has no such hook, we can optionally let the user discard old revisions I guess. Dries, what do you think? ------------------------------------------------------------------------ Mon, 25 Jul 2005 16:48:51 +0000 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_30.patch (53.13 KB) Sooooo; I've updated this patch once again. Dries didn't like my idea of legacy updates so we will have an option to discard old revisions in case the update should prove difficult. Always keep a backup.

1 0

[drupal-devel] [feature] Patch to add the guid tag to the rss feed
by Steven 25 Jul '05

25 Jul '05

Issue status update for http://drupal.org/node/26223 Post a follow up: http://drupal.org/project/comments/add/26223 Project: Drupal Version: cvs Component: node system Category: feature requests Priority: normal Assigned to: Anonymous Reported by: frjo Updated by: Steven Status: patch Possible remedy: generate GUIDs by hashing the base url with the node id (optionally the word "node" too if we use this technique elsewhere). Steven Previous comments: ------------------------------------------------------------------------ Sat, 02 Jul 2005 07:29:30 +0000 : frjo Attachment: http://drupal.org/files/issues/common_rss_guid.patch (537 bytes) This patch add the guid (globally unique identifier) tag to the rss feed. This is in line with the RSS 2.0 Specification, se below. It's a very simpel, one line patch, to the format_rss_item function in common.inc. I want this personally so my news reader (NetNewsWire) can distinguish between new and updates posts. >From http://blogs.law.harvard.edu/tech/rss "A frequently asked question about s is how do they compare to s. Aren't they the same thing? Yes, in some content systems, and no in others. In some systems, is a permalink to a weblog item. However, in other systems, each is a synopsis of a longer article, points to the article, and is the permalink to the weblog entry. In all cases, it's recommended that you provide the guid, and if possible make it a permalink. This enables aggregators to not repeat items, even if there have been editing changes. " ------------------------------------------------------------------------ Tue, 19 Jul 2005 06:09:38 +0000 : frjo Attachment: http://drupal.org/files/issues/common_rss_guid_02.patch (524 bytes) I made a patch for the cvs version of Drupal also. I believe this is a smal but very nice improvement of Drupals RSS support. ------------------------------------------------------------------------ Tue, 19 Jul 2005 18:32:06 +0000 : Dries guid has to be permanent. I don't think we can guarantuee that at the moment. People can set URL aliases and stuff, and these changes would change the guid.

1 0

[drupal-devel] [feature] Allow annoymous users to log in on access denied
by Steven 25 Jul '05

25 Jul '05

Issue status update for http://drupal.org/node/24050 Post a follow up: http://drupal.org/project/comments/add/24050 Project: Drupal Version: cvs Component: user system Category: feature requests Priority: normal Assigned to: gordon Reported by: gordon Updated by: Steven Status: patch You say you can't use a custom 403 page because it doesn't know where it came from... so, why not set a destination parameter instead? I think all that needs to be done is set $_REQUEST['destination'] = $_GET['q'] before calling menu_set_active_item() in drupal_access_denied(). Probably the same for drupal_not_found(). The destination is then picked up by the login-block which adds destination=drupal_get_destination() in the form's action. Steven Previous comments: ------------------------------------------------------------------------ Wed, 01 Jun 2005 05:51:25 +0000 : gordon Attachment: http://drupal.org/files/issues/denied_login.diff (1.41 KB) If the user is anonymous, then the user will be presented with the login page 'user/login' instead of a the 403 default page. If the user then logins in and has access to the page they will then get redirected to the page they were trying to access. If the user is already logged in then they will get the defined 403 page. With the standard system you cannot just set user/login as the 403 page as you will not be redirected to the page that you were originally requesting, but instead to the user page. ------------------------------------------------------------------------ Sun, 05 Jun 2005 00:22:06 +0000 : gordon Attachment: http://drupal.org/files/issues/denied_login2.patch (3.43 KB) here is a newer version of the patch which adds an option to admin/settings page so this functionality can be turned off and on. ------------------------------------------------------------------------ Sun, 05 Jun 2005 01:38:39 +0000 : moshe weitzman I have been wanting this since i first saw drupal (i.e. a long time). +1 ------------------------------------------------------------------------ Sun, 05 Jun 2005 10:29:05 +0000 : Dries Because the 403 page is configurable, this is already possible. This patch only makes it more convenient. I tempted to say: "Won't commit, but let's extend the form description a bit so it is clear this is possible.". That or we add a link to the login page (without the extra setting), cfr. to what the comment module does. ------------------------------------------------------------------------ Sun, 05 Jun 2005 12:06:40 +0000 : moshe weitzman dries - upon executing the 403 handler, the original $_GET['q'] is lost. That means that we cannot redirect to the destination page after login and thats one of the main benefits of this patch. ------------------------------------------------------------------------ Sun, 05 Jun 2005 12:40:57 +0000 : gordon Yes you can use the user/login link, but once you login you get the user page and not the page you were triing to get to. Also if you are logged in you will still get the user/login page when really you should get the custom access denied page. ------------------------------------------------------------------------ Sun, 05 Jun 2005 20:57:20 +0000 : slower What version of Drupal is this for? I'm using 4.5. ------------------------------------------------------------------------ Sun, 05 Jun 2005 23:47:19 +0000 : gordon It is for cvs (4.7) I would have to rewrite it if I was to port it back to 4.5 ------------------------------------------------------------------------ Mon, 13 Jun 2005 23:49:02 +0000 : gordon Dries, What is the verdict on this patch. I have found no way of prompting anonymous users to login and continue seemlessly without this patch. Please correct me if I am wrong, as this is some functionality that I require. ------------------------------------------------------------------------ Tue, 14 Jun 2005 01:00:12 +0000 : neofactor Glad to see it... I had always just added the following custom pages: http://neofactor.com/error403 (with a login link) http://neofactor.com/error404 (Adding the auto search) My 403's are only for non-members and all members have the same access. I use PHP to control groups differently for lots of different clients, so this works out well form me. The auto redirect function is nice.... keeping the user's initial request in place. hey.... When is Drupal going to add images to pages at core? Visual appeal.... It would be great if menu items and other areas started to do this... then people could just swap them out. Just a wish list. ;) ------------------------------------------------------------------------ Tue, 14 Jun 2005 03:02:01 +0000 : gordon Just adding the pages does work, but does not give the effect that I need. You can always use the login box on the sidebar, which will do the same thing. You said that the 403's aren't used to logged in users as they have the same privledge, But not everyone has the same privledge on your site (unless everyone has the same rights as user 1 ;-)) So what if a user who is already logged in goes to /admin you do not want to tell them that they need to login to get to the admin page, you want to send them to a big go away page. ------------------------------------------------------------------------ Tue, 19 Jul 2005 11:34:19 +0000 : jjeff Before I saw this thread, I posted another take on this bug at http://drupal.org/node/26659. It explains the problem a little more in-depth. Here's the thing: What makes Drupal sites look like Drupal sites is that they've got a login/password box on every page until the user is logged in. For Drupal to become a mature platform, it's going to need to move away from dependence on this crutch. Drupal should provide the infrastructure to allow the option of a simple "Login/Register" link. However, if a user gets an access denied message, they should be prompted to log in. As it is, the best that you can do after login is direct them back to the node that defines the "Access Denied" message. This is more than just confusing for the user. It's downright wrong! Dries, please reconsider this patch and/or some variation that solves this problem. It is essential for all current versions of Drupal. Thanks, Jeff ------------------------------------------------------------------------ Tue, 19 Jul 2005 11:52:21 +0000 : Kobus In concept, a definitive +1 from me. Especially the mention that the login box does not even display unless the user has to log in. I have started to remove the block completely and put it in a menu somewhere else so that it is not so obtrusive. For a commercial site, the exaggerated login page makes it look "as if you have something to hide". Not good. Regards,

1 0

[drupal-devel] [feature] add inline functionaliy to uploads
by Bèr Kessels 25 Jul '05

25 Jul '05

Issue status update for http://drupal.org/node/26288 Post a follow up: http://drupal.org/project/comments/add/26288 Project: Drupal Version: cvs Component: upload.module Category: feature requests Priority: normal Assigned to: Anonymous Reported by: Bèr Kessels Updated by: Bèr Kessels Status: patch I will fix the broken Xhttml. I chose "image" because it is exactly the same as the .image class used by image.module. But I will see to it, to add more functionality. I agree that it is not up to the themer for some good defaults, thus I will add a line of CSS to drupal.css (*shiver* ;) ) upload, IMO does not communicate well enough what the Image is doing. Not sure yet how I ill fix this, but I will see to it to come up with a better CSS/theme solution. Thanks! Bèr Kessels Previous comments: ------------------------------------------------------------------------ Sun, 03 Jul 2005 18:03:35 +0000 : Bèr Kessels Attachment: http://drupal.org/files/issues/upload_inline.patch (19.05 KB) One of the most often asked features is proper inlnie handling of files. Look at the amount of solutions, the popularity of image_assist, and the amount of peolpe dowloading image.module! That alone should be enough proof that Drupal lacks proper inline image support. This patch adds that to core. In fact, it does little more then appending a link of img tag to the body or the teaser. Off course that is configurable per file. Next to the [] list checkbox, this patch adds an [] inline checkbox. Simplicity is the foundation of this patch. I want no stles for inline editing, no fancy html wrappers, no tokens, just $node->body or teaser appended with a small html string. Another small themable funtion is introduced, (hey, you cannot expect me to develop something without adding more power for themers, now, can you? ;) ), that allows people to theme the string that is appended to the body or the teaser. Oh, and also note hat the biggest part of this patch is some cleaning I had to do in order to be able to develop properly. I dont like Ifs inside cases in foreaches inside swiches. in other words: nodeapi now calls functions instead of executing code directly. ------------------------------------------------------------------------ Sun, 03 Jul 2005 18:19:25 +0000 : Bèr Kessels Attachment: http://drupal.org/files/issues/inline.patch.screenshot.png (26.68 KB) here is how the form now looks ------------------------------------------------------------------------ Sun, 03 Jul 2005 18:19:58 +0000 : Bèr Kessels Attachment: http://drupal.org/files/issues/inline.patch.screenshot3.png (30.53 KB) and this is an example of inlined images and a .doc file. ------------------------------------------------------------------------ Sun, 03 Jul 2005 20:44:00 +0000 : sepeck changing to patch per request from berkes ------------------------------------------------------------------------ Tue, 05 Jul 2005 07:46:13 +0000 : Kobus This gets a +1 from me in principle, however, the [inline:xx] tag with inline.module gives you greater freedom as to where the inline image must be displayed. If you can add this functionality (I haven't checked the code, I don't know if it is in there) it would be a great addition for Core. This same strategy can be used for inline blocks, I am sure. Regards, Kobus ------------------------------------------------------------------------ Tue, 05 Jul 2005 08:57:50 +0000 : Bèr Kessels there are a couple of reasons why i did not include the [inline] tags. * I aimed for extreme simplicity: a checkbox shows an image inline: its up to the theme where it appears (if one does not like it before/above the body and teaser.). Simplicity was the main goal. * We don't have any tokens in core. And we should not have them. * Tokens are a very bad substitute for a good interface. They give less power then plain HTLM. Are much worst documented then HTML, but in the mean time, they are still as hard to learn as HTML. (Yes, I know people _think_ they are easier, but there _is_ not difference between [ ] and , only that its a different ascii char. So, no. I don't allow any placement of the image. I leave that that for dedicated modules, or the themer to decide. ------------------------------------------------------------------------ Tue, 05 Jul 2005 09:34:06 +0000 : Kobus So you will provide an API to do this? For example, with perhaps minor modifications, the inline.module will be able to display these files? In that case I am happy. If not, then I can't give my support to this patch (as if that matters...). A themer can't do this task as inline images (and blocks for that matter) is too dynamic for theming; it can be placed anywhere in a node where the user pleases. This means for me that there should be a module for this, such as inline module that allows you to define [inline:xx] tags. If your module emits an array of uploaded files (such as upload.module), inline.module can be, with minor changes, adapted to show these files inline. To show you what I mean with the content is too dynamic for a themer to perform this task, look at this screenshot related to inline blocks (inline images can follow the same pattern): http://drupal.org/files/issues/regions---possibility-3.png ------------------------------------------------------------------------ Tue, 05 Jul 2005 11:26:15 +0000 : Bèr Kessels Kobus, We are dealing with different issues here: You want a method to place images, files or so anywhere in the post. That is fine, but certainly NOT addressed in the patch! I made patch that *only* adds marked files to the body. its really nothing more then <?php $node->body = $filestring . $node->body ?> No APIS, no, dynamic tokens, no filters, nothing. However, what I meant with themers, is that there now is a theme_upload_inline available, so you can theme the abovementioned $filestring. On top of that $node->files[FILEID]->inline is TRUE if a file is flagged for inline. So in node.tpl.php, or wherever you want to theme a node, you can print nice images inline, when that flag is set. And about the comment that a themer cant do this: Simply not true. On most sites images are always placed in the same places. REally, even the sites see which use img_assist or inline, use them to place the images on the exact same places in every node. People /think/ they want the power to place images anywhere, but they hardly ever use that power. Just look at all the big news/publishing sites out there (BBC, CNN, BoigBoing, r even freshmeat) images are all placed acc. to the theme. They are not placed in random places by authors. So if you are one of the few that still want that power, there aer mots of power tools like inline module or img_assist. We should offer a good default, one that is simple. ------------------------------------------------------------------------ Tue, 05 Jul 2005 11:41:34 +0000 : Gerhard Killesreiter I fully agree that tags are evil. Kobus: You can always look for tags in $node->body in your theme's node function. ------------------------------------------------------------------------ Tue, 05 Jul 2005 11:50:52 +0000 : stefan nagtegaal Ber, the theme-function in your code looks like: <?php +/** + * Theme function for rendering of inline images + * @param $file a file object. + * @param $image a Boolean, indicating whether an img tag (TRUE) or an anchor tag (FALSE) should be used. + * @ingroup themable + */ +function theme_upload_inline($file, $image) { + if ($image) { + return '<img src="'. check_url(($file->fid ? file_create_url($file->filepath) : url(file_create_filename($file->filename, file_create_path())))) .'" alt="'. check_plain($file->filename) .'" class="upload inline">'; + } + else { + return ''. check_plain($file->filename) .' [1]'; + } + +} ?> After looking at your code it's not clear to me when $image is TRUE or FALSE, can you elaborate on me please? [1] http://drupal.org/'. check_url(($file->fid ? file_create_url($file->filepath) : url(file_create_filename($file->filename, file_create_path())))) .'\" class=\"upload inline ------------------------------------------------------------------------ Tue, 05 Jul 2005 12:06:39 +0000 : Bèr Kessels The function calling the theme function should decide whether its an image. IMO that is far too hardcore code for a themer ;) <?php $image = ereg('^(image/)', $file->filemime); ?> inside _upload_inline() does the trick. I did find one issue, though, with svgs, which are image/svg so maybe we should limit this to really only inline jpg, png, and gif, by extension? But I am no fan of determining files by extension, and IMO getimgsize is too heavy; ------------------------------------------------------------------------ Tue, 05 Jul 2005 12:13:52 +0000 : stefan nagtegaal Can't you make use of PHP's <?php image_type_to_mime_type(); ?> or <?php image_type_to_extension(); ?> ------------------------------------------------------------------------ Tue, 05 Jul 2005 12:45:43 +0000 : Kobus Well, if not in the patch, I need inline functionality one way or another. inline.module works perfectly with the current upload.module to fulfill my needs, and if there is a way where someone can extend your proposed upload.module, that would get my +1, otherwise I will just remain neutral about this issue. inline.module makes use of $node->files and displays that image. If your patch still uses $node->files, then this is a non-issue. In other words, if I can, with inline.module, or with minor modifications to inline.module still show files inline, I am +1 for this. As for the themability question... I still disagree. You say "Simply not true. On most sites images are always placed in the same places." That is ONLY true because they have NO alternatives. Means the content is in the same places, because they have no other places to put them. I have made tens of static sites where the content does not resemble a fixed pattern or structure. For me, free-flow layout is my expression of my creativity. This is why I still even BOTHER with static sites; when Drupal don't do what I want it to do. If Drupal could do inline display of content properly, I'd never even build another static site. It is a great mistake people make (including myself), and that is they design the site before ANY content is available, and this happens a lot in static sites. But in dynamic sites, you have far less control over this, unless you have a VERY clean structure with limited information that can be added, and only a few people posting content. If you can have your content before you start the design, you will have a better fit. I can't see how you can anticipate every possible posting posted on your site if you have a diversity of users. Especially not if you're using a site such as an designer's showcase site where your creativity is shown by your web sites. Possible, but not easy. If I claim that I have "unique, fresh designs" I can certainly NOT give them a "box-like" site with "left and right side bars only". That's why I need inline content, this includes images AND boxes. Gerhard: Your concern about tags is answered as well above, if I am not mistaken. The patch need not provide the tags, but should provide for someone to develop a module that provide the filters or tags. To summarize: I don't need your patch to do the inline images. I just need your patch to be able to allow someone else to develop a module that can display inline files. Kobus ------------------------------------------------------------------------ Mon, 18 Jul 2005 09:56:38 +0000 : Bèr Kessels okay, so for all clarity, let me sumarise: * this patch introduces a chackbox "inline". If checked, the file will show up inline. IT will be appended to the teaser and the body. * this patch does *not* allow one to place images of files in a userdefined place in the body. * this patch does *not* do any resizing nor any thumbnailing. * This patch is meant to be simple, clear and transparant. No tokens, no javascript, no nothing. ------------------------------------------------------------------------ Mon, 18 Jul 2005 11:21:54 +0000 : Kobus Then I can't see how this patch could replace the existing upload.module, which allows you to use inline.module to place images inline. I therefore withdraw my +1 and go -1 on this (not that it matters, I believe). I think you didn't properly read my previous message. I said I don't need your patch to do all this, but I can't live without the functionality currently provided by the upload.module and inline.module combination. I agree that your patch don't need to have that functionality, but if your patch takes away this functionality, in other words, I can't (by hook or by crook) manage what I need to do, my vote is -1. Regards, Kobus ------------------------------------------------------------------------ Mon, 18 Jul 2005 19:58:34 +0000 : Bèr Kessels Kobus, And all the others looking fofr advanced inline systems: please do not -1 this. It will hep inline module a lot too. allthough not yet in this patch, but can you imagine a perission system for inline module, that comes for free? Or a core system, that makes inline module ten times smaller? it is this path! so, please, if it is not /exactly/ what you are looking for, at least look at the advantages for Joe Average, and even for the possibilties for your favorite inline-module. even imag_assist will gain an anomous usability impcact when usiong this module, for it can then finally use the upload module in full scale. Ber ------------------------------------------------------------------------ Sat, 23 Jul 2005 18:59:10 +0000 : Bèr Kessels Sohodjo jim: " I hope it matters. A big -1 on enforcing over-simplification at the expense of important and existing functionality. If automatic, uncontrollable placement of images becomes the standard, it will mean fewer not more image-rich Drupal sites as site owners/developers get complaints about "Why can't I control my images!" I _strongly_ encourage Ber to provide a "have it both ways" solution. Why not have an admin configuration setting for "allow inline tags" with default off. Change it to on and you get an additional upload checkbox for 'inline at tag' to accommodate the current functionality of the inline module while simultaneously allowing for default placement. " NOOO. people; please; look at the PATCH. Again: and hopefully last time: Simplicity. This patch does NOT replace ANYTHING inline module or img_assist wants to do. This patch hands better data to such modules. It hands a variable over to these modules, $node->file[FID]->inline, which tells these sorts of modules if people wnat that file to appear inline. AND it CAN (by default will) render a file in the most simple way inline. So really, if you want to -1 this patch, fine. But do not -1 it, because it does too little IYO. It still allows MUCH more than what you can do no, without that patch! And any solution, for core, that allows advanced handling of inline images will either require an enourmous amount of work, or it will simply no get in. So, unless you come up with a good core-worthy patch, this is still a big leap forward from what we have now. ------------------------------------------------------------------------ Sat, 23 Jul 2005 19:01:51 +0000 : Bèr Kessels :$ this is an attempt to fix the borken HML in the previous follow up. ------------------------------------------------------------------------ Sun, 24 Jul 2005 04:07:34 +0000 : TDobes Please don't combine unrelated changes together in the same patch. Moving stuff in/out of the _nodeapi hook has absolutely nothing to do with the patch and makes it much harder to read through the patch and see what it's actually doing. I'd appreciate it if you could separate the changes made here which are really relevant and move the other changes into another issue. Also: You change form_group_collapsible(t('File attachments') to form_group_collapsible(t('File Attachments'). A quick glance at some other core code seems to indicate that we've standardized on capitalizing only the first word in multi-word group titles. As for the functionality itself, I don't think this belongs in upload.module. It would be useful to have the functionality in core, but as a separate module which can be switched on and off. (upload_inline.module or something like that) If a site were set up using an inlining system from contrib and the functionality you're proposing could not be switched off, it would be extremely confusing to users that there are two different ways of inlining images. A minor comment: I'm not so sure whether the results of this patch, when inlining non-image attachments, will be the one users expect. A non-web-savvy person may expect his/her Word document (for instance) to appear inline with the node when they click "inline" -- just like their images do. I'd suggest that we limit inlining to images only. I'm not as adamant about this as the other issues, so I could be convinced otherwise if others disagree. ------------------------------------------------------------------------ Sun, 24 Jul 2005 07:58:19 +0000 : Bèr Kessels Thank you for your thoughts Tdobbes. As for the "moving things in and out of nodeapi" This was necessary, because otherwise the patch would have made the code even more unreadable. I agree with you on this point, but I simply had to clean it up a little, in order to be able to work in it. As it stands now, it is simply not possible to make this a separate module. We cannot hook into the files table. In a next stage, I might work out some hook for that table. But first things first. Can Dries or Steven please comment on his. I want to know If i should bother spending this enormous amount of time on commenting and debating this issue. If the chances are small this is accepted, Ill just drop it This ridiculous long thread reminds me again why I try to avoid making patches for core. ------------------------------------------------------------------------ Mon, 25 Jul 2005 05:49:37 +0000 : Steven Overall I like the concept, but I think it at least needs some default styling to make the image appear okay, like floating it. Right now it appears inline between whatever was there... very ugly. Sure you can say "leave it up to the themer" but if the goal is to provide a simple default, we can't expect people to dive into theming if they are going to use this feature instead of img_assist or whatever. Also, I wonder about the classes "upload" and "image". Is there a reason to use two classes? Isn't "image" a bit too generic? Oh and there is a missing XHTML / for the img tag.

1 0

[drupal-devel] [bug] Tablesort and $_REQUEST array handling
by Steven 25 Jul '05

25 Jul '05

Issue status update for http://drupal.org/node/5371 Post a follow up: http://drupal.org/project/comments/add/5371 Project: Drupal Version: cvs Component: base system Category: bug reports Priority: normal Assigned to: mathias Reported by: mathias Updated by: Steven Status: patch By the way, what exactly is the point of this check: + if (!strstr($query_string, $key_as_uri)) { + $query_string .= '&'. $key_as_uri. '='. urlencode($value); + } It seems to be there to avoid duplicates, but there is no similar check below. If it is required, it needs a comment to explain it. Steven Previous comments: ------------------------------------------------------------------------ Fri, 23 Jan 2004 16:07:37 +0000 : mathias Attachment: http://drupal.org/files/issues/tablesort_and_REQUEST_arrays.patch (652 bytes) Tablesort tries its best to rebuild the URI from which it was called. The problem is when $_REQUEST holds arrays such as $edit. Tablesort will process this as: example.com?edit=Array This can be problematic since many module conditions rely on whether or not the $edit array is set. My patch disallows tablesort to append "array $_REQUEST variables" to it's new query string, since it serves no purpose. ------------------------------------------------------------------------ Sat, 31 Jan 2004 11:00:20 +0000 : Dries I'd like to hear Moshe's take on this as I'm not sure this is the proper fix. I'd think that the tablesort code knows exactly what fields it should add to the URL. ------------------------------------------------------------------------ Sat, 31 Jan 2004 13:24:36 +0000 : moshe weitzman I don't think this is the proper fix either. For example, - browse to the Drupal.org issue search page [1]. - press Search button - On the results page, look at the links in the table header and the pager. The query state gets passed along by tablesort. This patch would break this behavior. Changing Status to 'Active', since this patch shouldn't be applied without modification. [1] http://drupal.org/project/drupal/issues/search ------------------------------------------------------------------------ Sat, 31 Jan 2004 17:01:56 +0000 : mathias I'm pretty sure the project system is hacking the request object just like I am before sending it to tablesort: <?php if ($_SERVER['REQUEST_METHOD'] == 'POST' && is_array($_POST['edit'])) { foreach ($_POST['edit'] as $key => $value) { if (!empty($value) && in_array($key, $fields)) { $query->$key = $value; $_POST[$key] = is_array($value) ? implode(',', $value) : $value; } } unset($_POST['edit'], $_POST['op']); } ?> Basically, we're both moving the edit array into separate name / value pairs and then destroying it before handing it off to tablesort. I don't understand what the benefit is of passing an array into a GET query string since the key/value pairs aren't retained. We could set an option in tablesort to ask it to decompose the array into it's key/value pairs for us and pass them back in the query string. For example an edit array such as: <?php $edit['a'] => 1 $edit['b'] => 2 would have a query string of: &edit_a=1&edit_b=2 instead of the currently meaningless: &edit=Array ?> ------------------------------------------------------------------------ Sat, 31 Jan 2004 17:12:57 +0000 : Goba You mean: <?php $edit['a'] => 1 $edit['b'] => 2 would have a query string of: &edit[a]=1&edit[b]=2 instead of the currently meaningless: &edit=Array ?> This reuses the wonderful array handling provided by PHP (as long as nested arrays are handled properly). ------------------------------------------------------------------------ Sun, 01 Feb 2004 15:02:25 +0000 : Kjartan Indeed project module hacks the request data. I don't really like this as a general fix as it is a hack, but for now it is the best way. I dont think the original patch will break this as it just ignors arrays(). Project module loops through the array and converts them to normal array string elements; $_POST['edit']['status'] = value becomes $_POST['status'] = value, then I think I unset the edit array to clean up the urls. ------------------------------------------------------------------------ Thu, 05 Feb 2004 13:42:16 +0000 : moshe weitzman After feedback from Matias and Kjartan, I now consider this patch to be desireable ------------------------------------------------------------------------ Thu, 12 Aug 2004 15:10:30 +0000 : killes(a)www.drop.org Patch doesn't apply anymore. ------------------------------------------------------------------------ Fri, 08 Oct 2004 04:14:30 +0000 : mathias Attachment: http://drupal.org/files/issues/tablesort_and_REQUEST_arrays_0.patch (2.55 KB) This patch resolves the hacks project module and the ecommerce package use to rebuild the $_REQUEST arrays when implementing paged resultsets and/or tablesorting. I know some other developers have ran into this bug as well usually after form submissions. Tablesort will now fully handle multi-dimensional $_REQUEST arrays and convert them to a valid querystring and preserve user-submitted data. ------------------------------------------------------------------------ Fri, 08 Oct 2004 04:34:12 +0000 : moshe weitzman very nicely done ... bonus points for converting pager to use the new array2uri() function ... tablesort's poor handling of arrays frustrated me in a recent client project. this will simplify almost all cases where a form post leads to a sortable "results" table, such as in project.module. ------------------------------------------------------------------------ Fri, 15 Oct 2004 14:01:42 +0000 : mathias Attachment: http://drupal.org/files/issues/tablesort_and_REQUEST_arrays_1.patch (2.99 KB) Moved the new array2uri function to live with its array2* counterparts in common.inc and cleaned up the function documentation a wee bit. ------------------------------------------------------------------------ Tue, 21 Dec 2004 16:43:33 +0000 : moshe weitzman this is a nice code consolidation IMO ------------------------------------------------------------------------ Wed, 22 Dec 2004 10:50:33 +0000 : Dries This patch adds more code than it removes. That aside, it would be nice if the function's PHPdoc comments were extended. Looking at the patch, it is easy to see how/when this function is being used, but reading the PHPdoc comments it is not. I'd add a simple example and some context information. Also, it is unclear what $current_key is used for without inspecting the function's code. For readability, maybe rename $array to $attributes? If that makes sense, maybe rename 'array2uri' to 'attributes2uri'? Which makes me wonder: how does this stack with the existing drupal_attributes() function? Do we really need the recursive array handling? ------------------------------------------------------------------------ Sun, 13 Mar 2005 20:25:06 +0000 : killes(a)www.drop.org Doesn't apply anymore. ------------------------------------------------------------------------ Tue, 12 Jul 2005 22:49:07 +0000 : mathias Attachment: http://drupal.org/files/issues/tablesort_and_REQUEST_arrays_2.patch (3.19 KB) Updating this patch since I've been bitten by this issue again. The new array2uri() function really shines for tablesorting and/or paging search results after a $_POST request. The only coding change involved for developers is to use $_REQUEST['edit'] and $_REQUEST['op'] instead of $_POST['edit'] and $_POST['op'] within their function callback where this functionality is desired. As an added benefit these pages could also be bookmarked. Dries, I've updated the documentation to hopefully makes things a little easier to understand. I don't think renaming the $array variable to $attributes or calling the function attributes2uri instead of array2uri would make the function and its purpose more transparent. It's not used for HTML attributes like the drupal_attributes function, but rather for form submitted data (e.g., the $edit array) that needs to be tablesorted/paged. Project module and the ecommerce package still currently hack the $_POST vars to circumvent this problem. Not sure if other modules are also doing this. ------------------------------------------------------------------------ Fri, 15 Jul 2005 16:25:46 +0000 : Steven If we're going to have an array2uri() function, we might as well add urlencode() calls in there... this would centralize a lot of existing urlencode() calls, I think. ------------------------------------------------------------------------ Mon, 18 Jul 2005 13:51:56 +0000 : Dries Not going to commit this yet. 1. Let's wait for Mathias' feedback on Steven's comment. 2. I suggest using query or query_string, not querystring. 3. It is not clear why this can't be part of drupal_attributes($array)? If there is an important difference, it would be nice to make that clear in the PHPdoc comments. ------------------------------------------------------------------------ Fri, 22 Jul 2005 03:18:19 +0000 : mathias Attachment: http://drupal.org/files/issues/tablesort_and_REQUEST_arrays_3.patch (3.23 KB) Agreed with Steven that this is a logical place to urlencode values. The patch has been updated accordingly. There was some IRC discussion as to whether this function should be called drupal_uri, drupal2uri or just plain ol' array2uri. I think since this code doesn't make any Drupal specific function calls, array2uri is the best name. Which brings up a whole new issue for a whole new thread: Perhaps drupal_attributes should be renamed array2attributes? ------------------------------------------------------------------------ Mon, 25 Jul 2005 08:50:00 +0000 : Steven Dries: drupal_attributes is used for outputting attributes of an HTML tag. This patch is about putting data into the query (GET) part of an URL. They are completely different. mathias: couldn't we use this function in drupal_get_destination() as well ?

1 0

[drupal-devel] [bug] Tablesort and $_REQUEST array handling
by Steven 25 Jul '05

25 Jul '05

Issue status update for http://drupal.org/node/5371 Post a follow up: http://drupal.org/project/comments/add/5371 Project: Drupal Version: cvs Component: base system Category: bug reports Priority: normal Assigned to: mathias Reported by: mathias Updated by: Steven Status: patch Dries: drupal_attributes is used for outputting attributes of an HTML tag. This patch is about putting data into the query (GET) part of an URL. They are completely different. mathias: couldn't we use this function in drupal_get_destination() as well ? Steven Previous comments: ------------------------------------------------------------------------ Fri, 23 Jan 2004 16:07:37 +0000 : mathias Attachment: http://drupal.org/files/issues/tablesort_and_REQUEST_arrays.patch (652 bytes) Tablesort tries its best to rebuild the URI from which it was called. The problem is when $_REQUEST holds arrays such as $edit. Tablesort will process this as: example.com?edit=Array This can be problematic since many module conditions rely on whether or not the $edit array is set. My patch disallows tablesort to append "array $_REQUEST variables" to it's new query string, since it serves no purpose. ------------------------------------------------------------------------ Sat, 31 Jan 2004 11:00:20 +0000 : Dries I'd like to hear Moshe's take on this as I'm not sure this is the proper fix. I'd think that the tablesort code knows exactly what fields it should add to the URL. ------------------------------------------------------------------------ Sat, 31 Jan 2004 13:24:36 +0000 : moshe weitzman I don't think this is the proper fix either. For example, - browse to the Drupal.org issue search page [1]. - press Search button - On the results page, look at the links in the table header and the pager. The query state gets passed along by tablesort. This patch would break this behavior. Changing Status to 'Active', since this patch shouldn't be applied without modification. [1] http://drupal.org/project/drupal/issues/search ------------------------------------------------------------------------ Sat, 31 Jan 2004 17:01:56 +0000 : mathias I'm pretty sure the project system is hacking the request object just like I am before sending it to tablesort: <?php if ($_SERVER['REQUEST_METHOD'] == 'POST' && is_array($_POST['edit'])) { foreach ($_POST['edit'] as $key => $value) { if (!empty($value) && in_array($key, $fields)) { $query->$key = $value; $_POST[$key] = is_array($value) ? implode(',', $value) : $value; } } unset($_POST['edit'], $_POST['op']); } ?> Basically, we're both moving the edit array into separate name / value pairs and then destroying it before handing it off to tablesort. I don't understand what the benefit is of passing an array into a GET query string since the key/value pairs aren't retained. We could set an option in tablesort to ask it to decompose the array into it's key/value pairs for us and pass them back in the query string. For example an edit array such as: <?php $edit['a'] => 1 $edit['b'] => 2 would have a query string of: &edit_a=1&edit_b=2 instead of the currently meaningless: &edit=Array ?> ------------------------------------------------------------------------ Sat, 31 Jan 2004 17:12:57 +0000 : Goba You mean: <?php $edit['a'] => 1 $edit['b'] => 2 would have a query string of: &edit[a]=1&edit[b]=2 instead of the currently meaningless: &edit=Array ?> This reuses the wonderful array handling provided by PHP (as long as nested arrays are handled properly). ------------------------------------------------------------------------ Sun, 01 Feb 2004 15:02:25 +0000 : Kjartan Indeed project module hacks the request data. I don't really like this as a general fix as it is a hack, but for now it is the best way. I dont think the original patch will break this as it just ignors arrays(). Project module loops through the array and converts them to normal array string elements; $_POST['edit']['status'] = value becomes $_POST['status'] = value, then I think I unset the edit array to clean up the urls. ------------------------------------------------------------------------ Thu, 05 Feb 2004 13:42:16 +0000 : moshe weitzman After feedback from Matias and Kjartan, I now consider this patch to be desireable ------------------------------------------------------------------------ Thu, 12 Aug 2004 15:10:30 +0000 : killes(a)www.drop.org Patch doesn't apply anymore. ------------------------------------------------------------------------ Fri, 08 Oct 2004 04:14:30 +0000 : mathias Attachment: http://drupal.org/files/issues/tablesort_and_REQUEST_arrays_0.patch (2.55 KB) This patch resolves the hacks project module and the ecommerce package use to rebuild the $_REQUEST arrays when implementing paged resultsets and/or tablesorting. I know some other developers have ran into this bug as well usually after form submissions. Tablesort will now fully handle multi-dimensional $_REQUEST arrays and convert them to a valid querystring and preserve user-submitted data. ------------------------------------------------------------------------ Fri, 08 Oct 2004 04:34:12 +0000 : moshe weitzman very nicely done ... bonus points for converting pager to use the new array2uri() function ... tablesort's poor handling of arrays frustrated me in a recent client project. this will simplify almost all cases where a form post leads to a sortable "results" table, such as in project.module. ------------------------------------------------------------------------ Fri, 15 Oct 2004 14:01:42 +0000 : mathias Attachment: http://drupal.org/files/issues/tablesort_and_REQUEST_arrays_1.patch (2.99 KB) Moved the new array2uri function to live with its array2* counterparts in common.inc and cleaned up the function documentation a wee bit. ------------------------------------------------------------------------ Tue, 21 Dec 2004 16:43:33 +0000 : moshe weitzman this is a nice code consolidation IMO ------------------------------------------------------------------------ Wed, 22 Dec 2004 10:50:33 +0000 : Dries This patch adds more code than it removes. That aside, it would be nice if the function's PHPdoc comments were extended. Looking at the patch, it is easy to see how/when this function is being used, but reading the PHPdoc comments it is not. I'd add a simple example and some context information. Also, it is unclear what $current_key is used for without inspecting the function's code. For readability, maybe rename $array to $attributes? If that makes sense, maybe rename 'array2uri' to 'attributes2uri'? Which makes me wonder: how does this stack with the existing drupal_attributes() function? Do we really need the recursive array handling? ------------------------------------------------------------------------ Sun, 13 Mar 2005 20:25:06 +0000 : killes(a)www.drop.org Doesn't apply anymore. ------------------------------------------------------------------------ Tue, 12 Jul 2005 22:49:07 +0000 : mathias Attachment: http://drupal.org/files/issues/tablesort_and_REQUEST_arrays_2.patch (3.19 KB) Updating this patch since I've been bitten by this issue again. The new array2uri() function really shines for tablesorting and/or paging search results after a $_POST request. The only coding change involved for developers is to use $_REQUEST['edit'] and $_REQUEST['op'] instead of $_POST['edit'] and $_POST['op'] within their function callback where this functionality is desired. As an added benefit these pages could also be bookmarked. Dries, I've updated the documentation to hopefully makes things a little easier to understand. I don't think renaming the $array variable to $attributes or calling the function attributes2uri instead of array2uri would make the function and its purpose more transparent. It's not used for HTML attributes like the drupal_attributes function, but rather for form submitted data (e.g., the $edit array) that needs to be tablesorted/paged. Project module and the ecommerce package still currently hack the $_POST vars to circumvent this problem. Not sure if other modules are also doing this. ------------------------------------------------------------------------ Fri, 15 Jul 2005 16:25:46 +0000 : Steven If we're going to have an array2uri() function, we might as well add urlencode() calls in there... this would centralize a lot of existing urlencode() calls, I think. ------------------------------------------------------------------------ Mon, 18 Jul 2005 13:51:56 +0000 : Dries Not going to commit this yet. 1. Let's wait for Mathias' feedback on Steven's comment. 2. I suggest using query or query_string, not querystring. 3. It is not clear why this can't be part of drupal_attributes($array)? If there is an important difference, it would be nice to make that clear in the PHPdoc comments. ------------------------------------------------------------------------ Fri, 22 Jul 2005 03:18:19 +0000 : mathias Attachment: http://drupal.org/files/issues/tablesort_and_REQUEST_arrays_3.patch (3.23 KB) Agreed with Steven that this is a logical place to urlencode values. The patch has been updated accordingly. There was some IRC discussion as to whether this function should be called drupal_uri, drupal2uri or just plain ol' array2uri. I think since this code doesn't make any Drupal specific function calls, array2uri is the best name. Which brings up a whole new issue for a whole new thread: Perhaps drupal_attributes should be renamed array2attributes?

1 0

[drupal-devel] [bug] Some URLs get ?PHPSESSID added to them
by kbahey 25 Jul '05

25 Jul '05

Issue status update for http://drupal.org/node/4109 Post a follow up: http://drupal.org/project/comments/add/4109 Project: Drupal Version: cvs Component: base system Category: bug reports Priority: normal Assigned to: Eric Scouten Reported by: Eric Scouten Updated by: kbahey Status: patch Here is some references: issue 21170 [1], and a forum discussion [2]. [1] http://drupal.org/node/21170 [2] http://drupal.org/node/17947 kbahey Previous comments: ------------------------------------------------------------------------ Thu, 13 Nov 2003 00:22:39 +0000 : Eric Scouten Some URLs get ?PHPSESSID=(whatever) added to them. I find this distracting and unnecessary for anonymous users. Submitting patch #147 for consideration to remove this issue. ------------------------------------------------------------------------ Fri, 05 Dec 2003 14:39:25 +0000 : Kjartan This is a local PHP configuration issue. Don't see it as something Drupal should mess with, the less PHP settings Drupal messes with the better. ------------------------------------------------------------------------ Fri, 05 Dec 2003 15:11:33 +0000 : killes(a)www.drop.org Could you elaborate which config setting one has to tweak? ------------------------------------------------------------------------ Fri, 05 Dec 2003 15:18:11 +0000 : Anonymous php_flag session.use_trans_sid off php_flag session.use_only_cookies on The first will disable the automatic addition of the session ID to internal URLs, the second will remove support for passing session IDs in URLs. See http://php.net/session [3] for more info. BTW the above settings are for Apache, in PHP.ini you should not add the php_flag part, and you should use = inbetween the name and value... [3] http://php.net/session ------------------------------------------------------------------------ Mon, 07 Jun 2004 22:42:06 +0000 : shane It seems that Drupal (via the .htaccess file) already "messes with" a bunch of PHP settings. For example, my Drupal install "came with" these messed with settings: php_value register_globals 0 php_value track_vars 1 php_value short_open_tag 1 php_value magic_quotes_gpc 0 php_value magic_quotes_runtime 0 php_value magic_quotes_sybase 0 php_value arg_separator.output "&" php_value session.cache_expire 200000 php_value session.gc_maxlifetime 200000 php_value session.cookie_lifetime 2000000 php_value session.auto_start 0 php_value session.save_handler user php_value session.cache_limiter none php_value allow_call_time_pass_reference On I'm not sure that adding two more "php_value" flags to the stock .htaccess file is going to arguably mess with more than already is. Personally I find it extremely un-professional and very un-clean to have funky PHPSESSID url strings tacked onto a majority of my URLs. I've worked extremely hard to implement clean URLs (above and beyond the already great "clean url" feature of Drupal - I don't like them being turned into muck. I'd vote that these two flags be added to the .htaccess file that is distributed with Drupal. However, I'm uncertain of the overall true impact of making these changes, so barring an technical reason - I would absolutely request this as a feature. I've implemented these in many of my production sites already (just today) - we'll see what impact they have. ------------------------------------------------------------------------ Tue, 08 Jun 2004 02:56:00 +0000 : marky It's a performance thing. You need to remember that while not everyone has edit access to php.ini, those of us that do control our own servers see no need for .htaccess directives that set php options. >From the Apache docs [4]: "Apache will look in every directory for .htaccess files. Thus, permitting .htaccess files causes a performance hit, whether or not you actually even use them! Also, the .htaccess file is loaded every time a document is requested." Ok, I'm lucky in that I control my own server, so I can effectively remove the .htaccess file and move its config options to php.ini and the vhost section of httpd.conf, thus speeding up the responsiveness of my server. But if you don't have that option there's nothing wrong with adding those directives to your directory .htaccess file. That's what it's for - it allows directory specific config options for those that don't have access to their server config. As Kjartan said, "the less PHP settings Drupal messes with the better". I've got to agree - as far as I'm concerned it's already big enough. The "php_value short_open_tag 1", "php_value session.auto_start 0" and "php_value session.cache_limiter none" for example serve little purpose, as they mirror the installation default settings (apart from the last, where the available options are: nocache, private or public). If you don't have the choice, use .htaccess, and add whatever directives you see fit. But please remember that not everyone uses it. Anyway, just my 2 beads. [4] http://httpd.apache.org/docs-2.0/howto/htaccess.html#when ------------------------------------------------------------------------ Tue, 08 Jun 2004 04:38:32 +0000 : cel4145 Why not add them, but commented out with an explanation? When I encountered this problem, I had to spend a little time on drupal.org finding the solution. However, I am familiar with the .htaccess file from doing installs. If it had been included there commented out, I would have probably remembered it. After all, this is much the same way that httpd.conf functions with it's many, optional, commented out configuration options. Just makes it easier when a non-standard configuration option has to be used. Besides, I've also seen many Drupal sites with this problem. I suspect that some of those sites would have implemented these flags if they had been an option in the .htaccess file. And it looks bad. People that visit the sites that don't know any better will assume it's Drupal, not the lack of proper PHP settings. ------------------------------------------------------------------------ Tue, 08 Jun 2004 10:39:27 +0000 : killes(a)www.drop.org It wouldn't hurt if Drupal had a better documented and possibly extended .htaccess file. Those who whish not to use it should really not care... session.save_handler = user should probably included as well. ------------------------------------------------------------------------ Tue, 08 Jun 2004 12:17:06 +0000 : robertDouglass If you use these settings: php_flag session.use_trans_sid off php_flag session.use_only_cookies on won't it make it so that people with cookies disabled cannot have session state? And if the PHPSESSID is in the URL, isn't that an indication that the browser being used refused to take a cookie? I'd really like clarification on this, if anyone knows the answers. ------------------------------------------------------------------------ Tue, 27 Jul 2004 13:25:46 +0000 : JonBob The original issue is definitely by design, as these URLs need the session ID to preserve the session when running under the given PHP settings. A better-documented .htaccess is a different issue entirely. ------------------------------------------------------------------------ Thu, 19 Aug 2004 18:04:13 +0000 : gtoddv It hasn't been mentioned in this thread or anywhere else that I can find, but this session ID issue breaks validation. I don't know is this will cause accessibility issues too. I have tried the .htaccess fix and that doesn't change anything. ------------------------------------------------------------------------ Sat, 06 Nov 2004 07:17:21 +0000 : wazdog The non-validating problem is another problem with your server setup. PHP can be setup to used encoded &'s or not. Your server isn't if the url isn't validating. You'd have to talk to your server admin to fix that. ------------------------------------------------------------------------ Thu, 30 Dec 2004 20:32:35 +0000 : m_freeman2004 This is what you have to include in the .htaccess file: # Fix for ?PHPSESSID in clean URLs php_value session.use_trans_sid 0 php_value session.use_only_cookies 1 # End of fix ------------------------------------------------------------------------ Wed, 01 Jun 2005 19:07:53 +0000 : dr05 The code in .htaccess file: # Fix for ?PHPSESSID in clean URLs php_value session.use_trans_sid 0 php_value session.use_only_cookies 1 # End of fix don't work! Error: "Internal Server Error!" ------------------------------------------------------------------------ Thu, 14 Jul 2005 15:25:22 +0000 : jasonwhat Anyone else looking into this? Can one get rid of the id added onto the url but still keep the sessions going? ------------------------------------------------------------------------ Mon, 25 Jul 2005 06:16:22 +0000 : Eric Scouten dr05, what version of Apache and what OS are you using? I've been using this same patch with no problems on Mac OS X, Linux, and FreeBSD-based servers using both Apache 1.3.x and 2.0.x. Attached patch will apply these changes to .htaccess. (IMHO, this is useful for the majority of users; those who are smart enough to understand the performance issues can comment out these lines.) ------------------------------------------------------------------------ Mon, 25 Jul 2005 06:57:18 +0000 : clydefrog You forgot the patch, Eric. ------------------------------------------------------------------------ Mon, 25 Jul 2005 08:02:26 +0000 : Eric Scouten Attachment: http://drupal.org/files/issues/htaccess-no-session-ids.patch (383 bytes) D'oh! ------------------------------------------------------------------------ Mon, 25 Jul 2005 08:45:48 +0000 : kbahey There are cases where no parameter will fix this issue. On shared hosting systems where you do not have access to php.ini, and the hosting company has setup things in a way that they cannot be overriden. I found this out the hard way, after lobbying for the settings to be included in the init_set() statements in the settings.php prior to 4.6 being released. They are simply ignored on some hosts. On other hosts, you need to have a php.ini of your own, if suphp or SuExec is being used. In some cases, the only way to make things work is to compile your own PHP as a CGI (detailed writeup) [5]. This can have disadvantages too, such some performance impact, as well as excessive CPU utilization (hosting company may not like it). [5] http://baheyeldin.com/drupal/how-to-get-rid-of-phpsessid-in-drupal-and-othe…

1 0

[drupal-devel] [bug] Some URLs get ?PHPSESSID added to them
by kbahey 25 Jul '05

25 Jul '05

Issue status update for http://drupal.org/node/4109 Post a follow up: http://drupal.org/project/comments/add/4109 Project: Drupal Version: cvs Component: base system Category: bug reports Priority: normal Assigned to: Eric Scouten Reported by: Eric Scouten Updated by: kbahey Status: patch There are cases where no parameter will fix this issue. On shared hosting systems where you do not have access to php.ini, and the hosting company has setup things in a way that they cannot be overriden. I found this out the hard way, after lobbying for the settings to be included in the init_set() statements in the settings.php prior to 4.6 being released. They are simply ignored on some hosts. On other hosts, you need to have a php.ini of your own, if suphp or SuExec is being used. In some cases, the only way to make things work is to compile your own PHP as a CGI (detailed writeup) [1]. This can have disadvantages too, such some performance impact, as well as excessive CPU utilization (hosting company may not like it). [1] http://baheyeldin.com/drupal/how-to-get-rid-of-phpsessid-in-drupal-and-othe… kbahey Previous comments: ------------------------------------------------------------------------ Thu, 13 Nov 2003 00:22:39 +0000 : Eric Scouten Some URLs get ?PHPSESSID=(whatever) added to them. I find this distracting and unnecessary for anonymous users. Submitting patch #147 for consideration to remove this issue. ------------------------------------------------------------------------ Fri, 05 Dec 2003 14:39:25 +0000 : Kjartan This is a local PHP configuration issue. Don't see it as something Drupal should mess with, the less PHP settings Drupal messes with the better. ------------------------------------------------------------------------ Fri, 05 Dec 2003 15:11:33 +0000 : killes(a)www.drop.org Could you elaborate which config setting one has to tweak? ------------------------------------------------------------------------ Fri, 05 Dec 2003 15:18:11 +0000 : Anonymous php_flag session.use_trans_sid off php_flag session.use_only_cookies on The first will disable the automatic addition of the session ID to internal URLs, the second will remove support for passing session IDs in URLs. See http://php.net/session [2] for more info. BTW the above settings are for Apache, in PHP.ini you should not add the php_flag part, and you should use = inbetween the name and value... [2] http://php.net/session ------------------------------------------------------------------------ Mon, 07 Jun 2004 22:42:06 +0000 : shane It seems that Drupal (via the .htaccess file) already "messes with" a bunch of PHP settings. For example, my Drupal install "came with" these messed with settings: php_value register_globals 0 php_value track_vars 1 php_value short_open_tag 1 php_value magic_quotes_gpc 0 php_value magic_quotes_runtime 0 php_value magic_quotes_sybase 0 php_value arg_separator.output "&" php_value session.cache_expire 200000 php_value session.gc_maxlifetime 200000 php_value session.cookie_lifetime 2000000 php_value session.auto_start 0 php_value session.save_handler user php_value session.cache_limiter none php_value allow_call_time_pass_reference On I'm not sure that adding two more "php_value" flags to the stock .htaccess file is going to arguably mess with more than already is. Personally I find it extremely un-professional and very un-clean to have funky PHPSESSID url strings tacked onto a majority of my URLs. I've worked extremely hard to implement clean URLs (above and beyond the already great "clean url" feature of Drupal - I don't like them being turned into muck. I'd vote that these two flags be added to the .htaccess file that is distributed with Drupal. However, I'm uncertain of the overall true impact of making these changes, so barring an technical reason - I would absolutely request this as a feature. I've implemented these in many of my production sites already (just today) - we'll see what impact they have. ------------------------------------------------------------------------ Tue, 08 Jun 2004 02:56:00 +0000 : marky It's a performance thing. You need to remember that while not everyone has edit access to php.ini, those of us that do control our own servers see no need for .htaccess directives that set php options. >From the Apache docs [3]: "Apache will look in every directory for .htaccess files. Thus, permitting .htaccess files causes a performance hit, whether or not you actually even use them! Also, the .htaccess file is loaded every time a document is requested." Ok, I'm lucky in that I control my own server, so I can effectively remove the .htaccess file and move its config options to php.ini and the vhost section of httpd.conf, thus speeding up the responsiveness of my server. But if you don't have that option there's nothing wrong with adding those directives to your directory .htaccess file. That's what it's for - it allows directory specific config options for those that don't have access to their server config. As Kjartan said, "the less PHP settings Drupal messes with the better". I've got to agree - as far as I'm concerned it's already big enough. The "php_value short_open_tag 1", "php_value session.auto_start 0" and "php_value session.cache_limiter none" for example serve little purpose, as they mirror the installation default settings (apart from the last, where the available options are: nocache, private or public). If you don't have the choice, use .htaccess, and add whatever directives you see fit. But please remember that not everyone uses it. Anyway, just my 2 beads. [3] http://httpd.apache.org/docs-2.0/howto/htaccess.html#when ------------------------------------------------------------------------ Tue, 08 Jun 2004 04:38:32 +0000 : cel4145 Why not add them, but commented out with an explanation? When I encountered this problem, I had to spend a little time on drupal.org finding the solution. However, I am familiar with the .htaccess file from doing installs. If it had been included there commented out, I would have probably remembered it. After all, this is much the same way that httpd.conf functions with it's many, optional, commented out configuration options. Just makes it easier when a non-standard configuration option has to be used. Besides, I've also seen many Drupal sites with this problem. I suspect that some of those sites would have implemented these flags if they had been an option in the .htaccess file. And it looks bad. People that visit the sites that don't know any better will assume it's Drupal, not the lack of proper PHP settings. ------------------------------------------------------------------------ Tue, 08 Jun 2004 10:39:27 +0000 : killes(a)www.drop.org It wouldn't hurt if Drupal had a better documented and possibly extended .htaccess file. Those who whish not to use it should really not care... session.save_handler = user should probably included as well. ------------------------------------------------------------------------ Tue, 08 Jun 2004 12:17:06 +0000 : robertDouglass If you use these settings: php_flag session.use_trans_sid off php_flag session.use_only_cookies on won't it make it so that people with cookies disabled cannot have session state? And if the PHPSESSID is in the URL, isn't that an indication that the browser being used refused to take a cookie? I'd really like clarification on this, if anyone knows the answers. ------------------------------------------------------------------------ Tue, 27 Jul 2004 13:25:46 +0000 : JonBob The original issue is definitely by design, as these URLs need the session ID to preserve the session when running under the given PHP settings. A better-documented .htaccess is a different issue entirely. ------------------------------------------------------------------------ Thu, 19 Aug 2004 18:04:13 +0000 : gtoddv It hasn't been mentioned in this thread or anywhere else that I can find, but this session ID issue breaks validation. I don't know is this will cause accessibility issues too. I have tried the .htaccess fix and that doesn't change anything. ------------------------------------------------------------------------ Sat, 06 Nov 2004 07:17:21 +0000 : wazdog The non-validating problem is another problem with your server setup. PHP can be setup to used encoded &'s or not. Your server isn't if the url isn't validating. You'd have to talk to your server admin to fix that. ------------------------------------------------------------------------ Thu, 30 Dec 2004 20:32:35 +0000 : m_freeman2004 This is what you have to include in the .htaccess file: # Fix for ?PHPSESSID in clean URLs php_value session.use_trans_sid 0 php_value session.use_only_cookies 1 # End of fix ------------------------------------------------------------------------ Wed, 01 Jun 2005 19:07:53 +0000 : dr05 The code in .htaccess file: # Fix for ?PHPSESSID in clean URLs php_value session.use_trans_sid 0 php_value session.use_only_cookies 1 # End of fix don't work! Error: "Internal Server Error!" ------------------------------------------------------------------------ Thu, 14 Jul 2005 15:25:22 +0000 : jasonwhat Anyone else looking into this? Can one get rid of the id added onto the url but still keep the sessions going? ------------------------------------------------------------------------ Mon, 25 Jul 2005 06:16:22 +0000 : Eric Scouten dr05, what version of Apache and what OS are you using? I've been using this same patch with no problems on Mac OS X, Linux, and FreeBSD-based servers using both Apache 1.3.x and 2.0.x. Attached patch will apply these changes to .htaccess. (IMHO, this is useful for the majority of users; those who are smart enough to understand the performance issues can comment out these lines.) ------------------------------------------------------------------------ Mon, 25 Jul 2005 06:57:18 +0000 : clydefrog You forgot the patch, Eric. ------------------------------------------------------------------------ Mon, 25 Jul 2005 08:02:26 +0000 : Eric Scouten Attachment: http://drupal.org/files/issues/htaccess-no-session-ids.patch (383 bytes) D'oh!

1 0

Jump to page: