March 2005 - Development - lists.drupal.org

[drupal-devel] [feature] move check_query() into the database interface files
by Dries 10 Mar '05

10 Mar '05

Issue status update for http://drupal.org/node/13180 Project: Drupal Version: cvs Component: database system Category: feature requests Priority: normal Assigned to: chx Reported by: chx Updated by: Dries Status: patch Can we read up on that somewhere? Dries Previous comments: ------------------------------------------------------------------------ November 19, 2004 - 21:35 : chx Attachment: http://drupal.org/files/issues/check_output_patch.txt (1.54 KB) The check_query documentation says 'Prepare user input for use in a database query, preventing SQL injection attacks.' But the code itself is a simple addslashes. This is simply not correct -- for mysql, yes, but the world is not just mysql. There are _escape_string functions for that: mysql_escape_string, pg_escape_string, sqlite_escape_string... Read the manual for pg_escape_string: "Use of this function is recommended instead of addslashes()." Or read sqlite_escape_string: "addslashes() should NOT be used to quote your strings for SQLite queries; it will lead to strange results when retrieving your data." Ooooops. So I propose moving check_query() declaration from bootstrap.inc to database.*.inc. I've already written a letter to the devel list about this, but drumm said "code is better than talk" so here is a patch. At this moment, this does absolutely nothing, only opens the possibility to use the aforementioned escape_string functions. But database.sqlite.inc simply will not work without this change. ------------------------------------------------------------------------ November 19, 2004 - 21:42 : drumm Looks straightfoward enough. Is there any reason to not put in pg_escape_string() now? ------------------------------------------------------------------------ November 19, 2004 - 21:47 : chx Actually, yes. Read on the mentioned manual page: "Note: This function requires PostgreSQL 7.2 or later." I have no idea how the check it. There is a pg_version() but I will leave this postgres business to someone who has a working postgres install to code and check this thing. ------------------------------------------------------------------------ November 19, 2004 - 22:58 : Chris Johnson PostgreSQL 7.2 was released February 4, 2002. Major production releases 73. and 7.4, along with many point releases, have been issued since then. Current production is 7.4.6 and 8.0 is in the 4th revision of beta. It seems pretty safe to assume that any Drupal installation using PostgreSQL would have version 7.2 or later. Document if a user encounters errors with PostgreSQL that they need to check the version is 7.2 or later. ------------------------------------------------------------------------ November 20, 2004 - 15:42 : Dries +1 I suggest creating a db_escape_string() function, and removing check_query() all together. ------------------------------------------------------------------------ November 20, 2004 - 15:55 : Goba +1 on Dries ------------------------------------------------------------------------ November 20, 2004 - 19:32 : chx Attachment: http://drupal.org/files/issues/check_output_patch_0.txt (1.6 KB) db_escape_string sounds find to me, it's really a better name, thanks Dries. But removing check_query? I'd suggest deprecating it and removing it later. I am afraid It would break a lot of custom modules. Here is a slightly modified patch with pg_escape_string included and documented. ------------------------------------------------------------------------ November 20, 2004 - 19:43 : killes(a)www.drop.org We were never afraid of breakign things for the good of the project. Please don't change this policy. ------------------------------------------------------------------------ November 20, 2004 - 20:26 : chx Attachment: http://drupal.org/files/issues/db_escape_string.txt (20.87 KB) Let's do it. ------------------------------------------------------------------------ November 20, 2004 - 20:29 : chx Attachment: http://drupal.org/files/issues/db_escape_string_0.txt (20.88 KB) Something funny happened to the PostgreSQL db_escape_string comment and I have missed it, sorry. Reposting the patch. ------------------------------------------------------------------------ November 21, 2004 - 10:29 : Dries I committed the patch. Thanks chx. Keep it coming. I've also updated the upgrade notes at http://drupal.org/node/12347. I'll update some contributed modules unless someone objects. ------------------------------------------------------------------------ November 21, 2004 - 13:27 : chx Attachment: http://drupal.org/files/issues/mysql_escape_patch.txt (572 bytes) Keep it coming? Well, let's see the MySQL case... It's far from being trivial, and I expect some controversy over it. ------------------------------------------------------------------------ March 10, 2005 - 01:36 : chx Attachment: http://drupal.org/files/issues/mysql_escape_patch_0.txt (573 bytes) OK, here is mysql version a bit corrected, this should get into 4.6 -- addslashes is not a correct way any more.

1 0

[drupal-devel] [bug] Put revisions in their own table
by Anonymous 10 Mar '05

10 Mar '05

Issue status update for http://drupal.org/node/7582 Project: Drupal -Version: cvs +Version: 4.5.2 Component: node system Category: bug reports Priority: normal Assigned to: killes(a)www.drop.org Reported by: killes(a)www.drop.org Updated by: Anonymous Status: patch Was able to get http://drupal.org/files/issues/revisions_21.patch to work with drupal-cvs.tar.gz (10 March 2005) by: - includes/database.mysql.inc: Commenting out duplicates for functions function db_begin_transaction and function db_commit_transaction - modules: node.module: Removing "'title' => $node->title," from $node_table_values variable declaration and removing "'title' => "'%s'"," from "$node_table_types" variable declaration. Happy to submit a patch if requested. I'll watch this thread. Anonymous Previous comments: ------------------------------------------------------------------------ May 5, 2004 - 16:25 : killes(a)www.drop.org Currently all node revisions are stored in a serialized field in node.table and retrieved for _each_ page view although they are rarely needed. However, we have agreed that serializing data is bad and that we should try to keep the memory foot print pf Drupal small. Therefore I propose to create a separate revisions table which would be in principle identical to the node table, only that it could have several old copies of the same node. Extra data added by other modules could be added in a serialized field unless we find a better solution. ------------------------------------------------------------------------ May 5, 2004 - 17:06 : jhriggs I too think the serialized approach is less than desirable, but here's an alternative. This would likely take some considerable rework in core and contrib, but the following is how we handle similar types of situations in our databases at work. It is more elegant that a separate table, and avoids the (almost exact) duplication of a table. Instead of separate tables, keep all revisions of nodes in the node table as follows: * add field: active (0/1 or Y/N) * add field: revision * every revision of a node is stored in the node table; however, only one revision can be active at any given time * nid can no longer be unique -- primary/unique key becomes (nid, active) * any time a node is loaded, updated (without revision), etc., the active version is used. Thoughts? ------------------------------------------------------------------------ May 5, 2004 - 17:57 : killes(a)www.drop.org I am not opposed to your scheme, but I want to stress the following: * Duplicating a table's structure is not bad (IMHO) as long as the content is different. * having two tables will allow us to have a rather small node table. This is (maybe) a performance gain. ------------------------------------------------------------------------ May 5, 2004 - 18:37 : jhriggs I don't necessarily think that duplicating a table's structure is _bad_. It just seems to be wasteful and a pain to maintain. (Every change to the node table is made twice...easy to do, but also easy to miss perhaps.) As for performance, as long as nid and the active indicator are indexed, there shouldn't be any performance loss. Also, archiving an old version when making a new revision will be much simpler: just change the active indicator rather than copying an entire node to another table (and ensuring everything gets copied...again a potential maintainance issue). To be honest, I would just like to see the serialized data go away, regarless of what approach is taken. ------------------------------------------------------------------------ July 30, 2004 - 19:49 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_07-30-2004.p… (10.47 KB) I'm interested in using Drupal for a large scale wiki-type project. In order to do this, I need revisions to be in their own table. Attached is a patch to do just that. Most of the changes are pretty self explanatory. Spreading out node data across two tables meant that I had to add database functions to do locking/transactions. Without this, race conditions in which the database becomes corrupted are possible. ------------------------------------------------------------------------ July 30, 2004 - 19:54 : Nick Nassar Oh yeah... The patch is a diff against Drupal CVS ------------------------------------------------------------------------ July 31, 2004 - 00:00 : Anonymous Gerhard speaking. Nick, thanks a lot for your nice patch! It saves me a great deal of labour. I looked through it and immediately liked it. You not only put the old revisions into a new table but also the current one. Do you have an estimate how much more expensive the additional join is? Besides a few minor coding style issues I found a major one: Just a few hours before you uploaded your patch JonBob's node access patch hit core. That means your patch won't apply anymore as all the queries you change have been changed. Can I bug you to update your patch? ------------------------------------------------------------------------ July 31, 2004 - 01:11 : Anonymous Also I think that your upgrade path loses existing revisions. ------------------------------------------------------------------------ July 31, 2004 - 02:39 : drumm I think this is the proper way to do things. No columns are duplicated, there is no serialized data, and only the fields that are logically revised are stored. Nothing jumped out at me as a way to have my node modules be able to keep a table of revisions of additional fields. I'm guessing this could be done within the confines of _insert and _update. Assuming the upgrade path works and modules can extend it I give it a +1. ------------------------------------------------------------------------ July 31, 2004 - 14:40 : Nick Nassar It figures that just as I finish a big patch, another patch comes along and breaks it. Oh well, it should be a pretty easy to fix. I'll work on it. Fixing the upgrade path to keep revisions should be fairly painless. I found another issue that needs to be fixed before this patch gets merged. There format of a node needs to be stored for each revision. Otherwise, for modules that store a format for the nodes, such as page and book, if you write one revision in PHP and the next in HTML, the PHP revision will be displayed as HTML. This is part of a larger issue of how node modules should store revisions of additional fields. I think each module that wants to do this should create another table with (nid, revid) as the primary key. Just as when they want to add fields to a node they create another table with nid as the primary key. As far as performance goes, for sites that make heavy use of revisions, an extra join on primary keys is going to be a lot faster than grabbing all of the revisions from that database everytime. We would need to run benchmarks to determine is the overall difference in speed is for an average site is a gain or a loss. I'm guessing it's very minor either way. ------------------------------------------------------------------------ August 23, 2004 - 13:55 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_08-23-2004.p… (10.92 KB) Here's an updated patch against CVS that puts revisions in their own table, provides an upgrade path, and fixes the format related bugs in the last patch. Hopefully, this can make it into CVS as soon as the freeze is over. ------------------------------------------------------------------------ August 23, 2004 - 14:10 : moshe weitzman Interesting patch ... drumm's question is still outstanding. how do modules store revisions of their fields? Are they expected to manage this on their own? Thats not how it works today. As an aside, i am seeing profile_ fields in my node.revisions column. One could argue that those need not be saved. They pertain to the node author, not to the node itself. ------------------------------------------------------------------------ August 23, 2004 - 16:14 : Nick Nassar Having modules be responsible for storing revisions of their own fields is a side-effect of storing revision data in tables. There's really no way around it. However, revisions generally don't make sense for node types that don't have PHP/HTML content, such as polls. I think it's going to be a pretty rare scenario for a new node type to want another field to change per-revision, so it's a pretty good trade-off. Storing fields that shouldn't be part of revisions, such as the profile_ fields, is a side-effect of storing revisions as serialized objects. Applying this patch will free up that wasted space. :) ------------------------------------------------------------------------ August 23, 2004 - 17:20 : Anonymous There should be a hook that let's the module choose whether it supports history. This way a module author can prevent the user from doing something that may break his module or just cause undefined behavior. If the module doesn't support history then don't let the user/admin choose to add history to nodes of that type. Craig ------------------------------------------------------------------------ August 23, 2004 - 19:23 : Nick Nassar I agree, there should be an API change to make specifying support for revisions easier. In the interests of keeping patches small and keeping to one change per patch, I think the API change should be a separate issue. A sort of ad-hoc API to decide whether or not a module supports revisions by default already exists. Instead of having a hook, modules set the default value of the "Create new revision" field in the edit form. The admin can change this option in admin/node/configure/defaults. This patch doesn't change that. Revisions are broken for node types that have their own database structure, like polls, even when storing them as serialized objects. This patch doesn't change that, either. ------------------------------------------------------------------------ October 26, 2004 - 02:35 : moshe weitzman I'm guessing that someone is going to have to demonstrate that this patch performs as well as current drupal before it gets comitted. i think this patch is a few benchmarks from being comitted. ------------------------------------------------------------------------ October 27, 2004 - 01:04 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004.p… (11 KB) I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ October 27, 2004 - 01:05 : Nick Nassar I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ October 27, 2004 - 01:05 : Nick Nassar Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004.p… (11 KB) I ran some really unscientific benchmarks, and it looks like this patch has a negligible affect on performance. I used apache bench and the database from theregular.org, which doesn't contain any revisions (worst case scenario for this patch) and contains several hundred nodes. Both the patched and unpatched versions hovered between 2.36 and 2.38 requests per second. The command I used was: ab -n50 -C 'PHPSESSID=b01a9f92880ef215b0ed6f1314a5eba2' http://192.168.0.100/ An updated patch that should apply to CVS is attached. ------------------------------------------------------------------------ November 15, 2004 - 05:05 : elias1884 please overthink the revision system default workflow as well. don't look at the revision system as an isolated system but as a part of the whole workflow system! if you combine revisions with the moderation queue the most logic default workflow would be like that: auth user creates node (revision #0) admin approves the node (status = 1, moderation = 0) => node publicly available auth user finds typo and changes node (revision #1, status = 0, moderation = 1) ------- what happens at that point at the moment is, that the node is not accessible anymore at all until the new revision is approved by admin. of course the new revision should not go online until reviewed and approved, this is absolutely correct, but there is no reason to not take the old revision offline, since it was already approved and should therefore be online until the new revision is approved. it is not practical if a node disappears only because the author corrected a typo. ------- admin approves the node (status = 1, moderation = 0) eventhough I first thought a plain boolean active field would not be capable of providing that functionality if finally came to the conclusion, that it can. The only thing to do is to not set that bit, when a new revision is created, but when it is approved (in case moderation is activated under default workflow). Every revision should have its own moderation, status and active field and on approval they are set like this (status=1, moderation=0, active=Y). When you wanna rollback to an old revision, you can chose between all revisions that already have the moderation bit set back to 0 again and the published to 1. There should be an extra permission for rollback! another concern that I have about the default workflow is, that users can't see the content, they have just created, when moderation is enabled. Eventhough, there is a big fat "submission accepted" presented after submissions, unexperienced users tend to question the information those stupid tincans give them, if they can't find their content afterwards. Many users are really lazy bastards and they don't even read the status messages. The best feedback about whether his story was submitted successfully or not of course is, if he can find the story somewhere on the site, maybe with a status message on top of it, mentioning, that the content is currently not publicly available since it has not been approved yet. there should be a my content section under my account, like somebody is trying to do with the workspace module I guess. so my suggestion is to make (status=0, moderation=1) still available for the creator under a my content section somewhere! ------------------------------------------------------------------------ November 24, 2004 - 04:21 : Nick Nassar I agree. The current workflow for moderation queues and revisions needs to change, but this patch isn't the place for it. The patch is already too big, and it only does the backend stuff. Instead of adding more to this patch and making it take even longer to get into core, would you mind creating a new issue for your UI suggestions, so the those changes can be added as a separate patch? Thanks, Nick ------------------------------------------------------------------------ December 11, 2004 - 12:26 : Dries This patch is _much_ needed so I'd love to see someone revive it. In order for this patch to be accepted, the following needs to be done: Update this patch to CVS HEAD. Rename revid to vid. Rename node_rev to node_revisions. Rename node_rev.changed to node_revisions.timestamp. Rename $rnode to $revision. Fix the coding style to match Drupal's: proper spacing, single quotes where possible, proper variable names. Benchmark this patch with a large database with enough revisions. I'd be happy to benchmark this on my local copy of the drupal.org database. The book.log field should probably move to the node_revisions table. This can be done in a separate patch. Investigate whether transactions are well-supported. ------------------------------------------------------------------------ December 13, 2004 - 00:25 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/Drupal-Improved_Revision_Schema_10-26-2004-r… (11.02 KB) I've worked a bit on the patch (coding style issues as mentioned by Dries). One thing I noticed is that the patch uses REPLACE. IIRC this needs to be chagned to "UPDATE, if fail INSERT" for pgsql compatibility. Nick, are you still interested in working on that patch? I'd like to know how it works on your site and work on getting it into core. ------------------------------------------------------------------------ December 13, 2004 - 12:33 : Dries Gerhard: your patch does not apply. ------------------------------------------------------------------------ December 13, 2004 - 17:10 : killes(a)www.drop.org Yes, I know, that was the same version as I mailed to you earlier. ------------------------------------------------------------------------ December 13, 2004 - 21:02 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions.patch (52.96 KB) Ok, upüdated the patch to cvs. ------------------------------------------------------------------------ December 14, 2004 - 08:58 : Dries Some more comments: db_begin_transaction() and db_end_transaction() do not belong in database.inc, but in database.mysql.inc and database.pgsql.inc respectively. The node module calls node_revisionsision_list() which is not defined. (Fxed that on my local copy.) Do db_begin_transaction() and db_end_transaction() deprecate Jeremy's table locking patch? The upgrade path assigns the wrong user ID to each revision. The upgrade path assigns the wrong date to each revision (that or a node's revision page shows the wrong usernames/dates). The coding style needs a bit of work, but we can worry about that later. ------------------------------------------------------------------------ December 14, 2004 - 17:34 : Nick Nassar If you need any help getting those things fixed, just let me know. ------------------------------------------------------------------------ December 14, 2004 - 17:50 : Nick Nassar How this relates to Jeremy's node locking patch: There was lots of discussion, and node locking was decided against because from an end user point of view you never want a node to be locked. He's now advocating for a much simpler patch that warns users if their changes will overwrite someone elses. That patch still has a race condition, which might be fixed using db_begin_transaction(). http://drupal.org/node/6025 ------------------------------------------------------------------------ December 14, 2004 - 22:26 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_0.patch (55.96 KB) Here is an updated patch that tries to address Dries concerns. ------------------------------------------------------------------------ December 15, 2004 - 08:32 : Dries Attachment: http://drupal.org/files/issues/revisions-bug.png (76.06 KB) It didn't fix the aforementioned bugs. See attached screenshot. ------------------------------------------------------------------------ January 6, 2005 - 20:15 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_1.patch (51.77 KB) Ok, here is a new version. Dries and myself worked hart at it, so please have a look. what is still missing - database upgrades for the core modules with an own table - contrib modules need an upgrade too. - do we need nid and vid in both the node and the node_revisions table? - the amount of sql queries means a good stress testing for large databases. ------------------------------------------------------------------------ January 19, 2005 - 21:43 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_2.patch (49.49 KB) Here is an updated patch. We discussed to keep the current title in node module and also in the revisiosn table. This is content duplication but will save many joins as many queries only need the title of a node. Discussion is welcome. ------------------------------------------------------------------------ January 19, 2005 - 23:33 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_3.patch (29.93 KB) I've implemented the aforementioned solution. This makes the patch much smaller. The patch now also removes taxonomy_node_has_term() which wasn't used anywhere. I'd really apprciate if some people could test drive the patch. It will be another huge improvement for 4.6. ------------------------------------------------------------------------ January 20, 2005 - 00:05 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_4.patch (30 KB) Another revision. Steven didn't like my literal $node->vid in queries. ------------------------------------------------------------------------ January 20, 2005 - 01:10 : killes(a)www.drop.org - database upgrades for the core modules with an own table - contrib modules need an upgrade too. - do we need nid and vid in both the node and the node_revisions table? - the amount of sql queries means a good stress testing for large databases. These issues are still open, btw. Especially the first one needs to be tackled. ------------------------------------------------------------------------ January 25, 2005 - 20:11 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_5.patch (51.13 KB) Here is a patch that has the database tables updated for forum, book, and page module. ------------------------------------------------------------------------ January 29, 2005 - 22:55 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_6.patch (49.18 KB) Yet another update to keep it working with head. The patch now also removes the table definitons for the page table. ------------------------------------------------------------------------ January 29, 2005 - 22:57 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_7.patch (55.69 KB) Sorry, that was the old version, this is the right one. ------------------------------------------------------------------------ January 31, 2005 - 19:55 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_8.patch (55.71 KB) Updated once more. ------------------------------------------------------------------------ January 31, 2005 - 20:52 : Dries Anyone to help review/test this? ------------------------------------------------------------------------ January 31, 2005 - 21:22 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_9.patch (49.29 KB) Updated again, the update functions occurred twice. Thanks Bart. ------------------------------------------------------------------------ February 2, 2005 - 00:27 : killes(a)www.drop.org Don't know if the db I am using is corrupted or what. I still do have some didficulties. The latest patch is attached. ------------------------------------------------------------------------ February 2, 2005 - 00:27 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_10.patch (49.67 KB) I am probably slowly going mad ... ------------------------------------------------------------------------ February 2, 2005 - 01:54 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_11.patch (48.95 KB) The update issue still needs investigating. This patch is updated for cvs. ------------------------------------------------------------------------ February 2, 2005 - 20:20 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_12.patch (49.83 KB) Ok, here is a new version. I've solved my troubles with book.module. There are still some issues with forum module. Possibly due to inconsistent database. ------------------------------------------------------------------------ February 2, 2005 - 21:31 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_13.patch (49.83 KB) Turns out the drupal.org database had indeed some quirks. Please run this query in your oldest db and tell me the result: select nid,type from node where type like '%/%'; If you get a non-zero result we might need to add another security update. The patch could use still more testing, though. ------------------------------------------------------------------------ February 3, 2005 - 01:16 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_14.patch (49 KB) Ok, we are getting somewhere. At a first glance the update is working. There is a problem remaining: the revisions tab will be shown whether the node has revisions or not. Not sure we can/need to fix this. People with a drupal.org account can log in at http://killes.drupaldevs.org/revision/ and poke around. Your permissions will be the same as on drupal.org. Feel free to vreak everything but don't forget to file complaints here. (Note: this is only a pruned version of the drupal.org database with all project nodes and nodes with nids > 7000 dropped). ------------------------------------------------------------------------ February 3, 2005 - 04:19 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_15.patch (52.39 KB) There was some error in node_save and also the patches to the database.inc files got lost... ------------------------------------------------------------------------ February 3, 2005 - 07:07 : robertDouglass Submitting book pages doesn't work on your test site. It puts the entire content of the preview inside the body textarea. I wrote a sentence in the body and the log, and pressing preview put several lines of HTML containing both sentences in the body textarea on the preview page, plus the book page wouldn't submit. -R ------------------------------------------------------------------------ February 3, 2005 - 07:50 : Junyor 0 results here. I started using Drupal with version 4.4, though. ------------------------------------------------------------------------ February 3, 2005 - 23:56 : killes(a)www.drop.org @Junyor: Thanks, that's a good sign. Maybe somebody else has an older db to try. @robertDouglass: The first effect you describe is due to drupaldevs running on PHP 5. I am unsure why the second thing does not work. In node_save() the node object has a nid although there is none in the form. Very strange. I've enabled display of db queries on the testsite. ------------------------------------------------------------------------ February 4, 2005 - 19:17 : dmjossel No results here on the query: select nid,type from node where type like '%/%'; On a database that was put in place prior to Drupal 4 and is now running on 4.5.2. ------------------------------------------------------------------------ February 4, 2005 - 20:44 : killes(a)www.drop.org @dmjossel: thanks. @all. The strange problem I reported was apparently php 5 related. After applying Steven's php 5 patch it went away. One error is remaining: If I create a new forum topic it will be shown as part of the book on preview. Hmm, that was due to a db that got corrupted during testing so that is fixed too. Please poke around at the test site and look if you find more errors. ------------------------------------------------------------------------ February 5, 2005 - 07:16 : Steven By the way, I just remembered that Drupal.org has some blogs lingering on in the database even though blog.module is not enabled. Perhaps this is causing troubles? ------------------------------------------------------------------------ February 5, 2005 - 11:22 : Anonymous I can't see why it would. Drupal.org will need extra updates for images and project nodes because those have their own tables. GK. ------------------------------------------------------------------------ February 6, 2005 - 12:49 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_16.patch (52.49 KB) Updated to apply to cvs again. ------------------------------------------------------------------------ February 22, 2005 - 20:15 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_17.patch (49.64 KB) Updated again. All we need is a patch to upload module and an upgrade path for it. ------------------------------------------------------------------------ March 4, 2005 - 04:22 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_18.patch (52.31 KB) Updated once more. Moved log field from book to node_revisions table as discussed in Antwerp. upload module still missing. We need to decide under which circumstances the log field should be displayed. Should that be added to the workflow? Should it depend on the revisions setting? ------------------------------------------------------------------------ March 5, 2005 - 19:27 : Anonymous Attachment: http://drupal.org/files/issues/revisions_20.patch (75.52 KB) Ok, here it is: Yet another revision of this grrrrreat patch. Changes from previous versions: - supports versioning for uploaded files. A problem is that if you delete a file, it will be gone for all revisions. - the log field is now in the node_revsions table, but each module has to decide whether to show it or not. I've implemented it for the page and the book type odes. Also, the field can be edited when adding non-book nodes to the book. The log is displayed on the revisions page and if a node is moderated. - the revisions are moved to an old_revisions table to a) get the node table smaller and b) still leave the mavailable for contrib modules that want to retreive old version data. The patch has been applied to killes.drupaldevs.org/revision where it can be tested by anybody (especially people who have "site admin" rights on drupal.org) The database is from drupal.org and you should b able to log in with your pass or simply mail yourself a new one. Gerhard ------------------------------------------------------------------------ March 5, 2005 - 19:51 : Anonymous Attachment: http://drupal.org/files/issues/revisions_21.patch (59.42 KB) BTW, I marked this a bug because atm the revisions field can grow quite big. Neil has reported problems from some users who were not able to load some nodes due to to many large revisions. Also, som unrelated stuff crept into the patch. New version attached. ------------------------------------------------------------------------ March 8, 2005 - 05:56 : killes(a)www.drop.org Attachment: http://drupal.org/files/issues/revisions_22.patch (60.29 KB) Ok, I think I got it. Changes to last version: - uploads are no properly versioned. Missing are still pgsql checks and updates.

1 0

[drupal-devel] [bug] Old comments are displayed as new
by Goba 10 Mar '05

10 Mar '05

Issue status update for http://drupal.org/node/18692 Project: Drupal Version: cvs Component: comment.module Category: bug reports Priority: critical Assigned to: Goba Reported by: Goba Updated by: Goba Status: patch Attachment: http://drupal.org/files/issues/Drupal-fix-new-comment-counts.patch (751 bytes) Old comments (comments for nodes having no entry in the history table anymore) are counted as new comments, since the timestamp retrieved from the history table is used to check for new comments (which is 0, if an entry was not found). This patch sets the timestamp used to check for new comments to the NODE_NEW_LIMIT value at least (which is when the history table was/should have been pruned). Goba

1 0

[drupal-devel] [feature] Containers for taxonomy terms
by Axel 10 Mar '05

10 Mar '05

Issue status update for http://drupal.org/node/18647 Project: Drupal Version: 4.5.2 Component: taxonomy.module Category: feature requests Priority: minor Assigned to: Axel Reported by: Axel Updated by: Axel Status: patch Attachment: http://drupal.org/files/issues/taxonomy-containers.patch (4.48 KB) Patch allow to use taxonomy terms like containers - they visible in menus, but not allow to select it. This is especially usefull for forum module but not only for it. Tested with 4.5.2. Additionally patch requres changing db scheme (not included to patch): ALTER TABLE `term_data` ADD `container` TINYINT(1) AFTER `description`; I understand what patch will not be applied to official 4.5, just placed here for users who interested with such functionality. See some screenshots and explanations here (in English): http://drupal.ru/node/694 Axel

1 1

[drupal-devel] [feature] Description on e-mail field for anonymus comments
by frjo 10 Mar '05

10 Mar '05

Issue status update for http://drupal.org/node/18676 Project: Drupal Version: cvs Component: comment.module Category: feature requests Priority: normal Assigned to: Anonymous Reported by: frjo Updated by: frjo Status: patch Attachment: http://drupal.org/files/issues/comment.module01.patch (1.16 KB) When anonymus users are allowed to add comments there are a required e-mail field. This field are, as far as i can see, not shown publicly. A user asked me about this and after confirming that it is not shown publicly I added a description that inform the users of this. I have included a patch that add the description "The content of this field is kept private and will not be shown publicly". This is the same description that is used by the profile module. frjo

2 1

Re: [drupal-devel] A Folksonomy module
by Boris Mann 10 Mar '05

10 Mar '05

> This 1000 tag thing. I've found vocabularies with a tree of 100 tags a > bit awkward to use. The Admin vocab overview gets very long and the few > places where you have all tags for a vocab in a combo box will also get > awkward to use. This sounds like something that UI can overcome. I agree with the Admin vocab overview length issue, but if you use system vocabs, the folksonomy could have it's own admin display for this. Tag-per-line or comma-separated tags would be the preferred input, would they not? so I don't see the terms every necessarily being displayed in a combo box. I think that aiming to be able to implement folksonomies with our existing taxonomy system is a good thing to strive for, even if it means code changes to taxo.module to encompass this use. Now all we need to do is have aggregator create feed_item nodes, add a little publish/subscribe/workflow magic to delete/update them, and we can replace the strange "categories" used there with taxonomy as well... (let me dream, OK?) Hmm. And if profiles were nodes, you could "tag" users by tagging their associated profile(s). -- Boris Mann http://www.bryght.com

4 5

[drupal-devel] [bug] Prevent browser page caching of dynamic content
by Dries 10 Mar '05

10 Mar '05

Issue status update for http://drupal.org/node/5900 Project: Drupal Version: cvs Component: user system Category: bug reports Priority: critical Assigned to: mathias Reported by: christianlong(a)christianlong.com Updated by: Dries Status: patch -1 for the $_GET['q'] addition. While it might be the most problematic page, it is also the most popular page. The $_GET['q'] scenario merely hides the fact that the headers/caching are not working like they should. After a POST operation (eg. log in), Drupal should never send a cached page. This is checked for in page_get_cache() and page_get_cache(). Maybe those checks are bogus. Dries Previous comments: ------------------------------------------------------------------------ February 16, 2004 - 22:44 : christianlong(a)christianlong.com Attachment: http://drupal.org/files/issues/f_040214_01.txt (6.33 KB) This bug has been discussed before, but not resolved. http://drupal.org/node/view/1740 http://drupal.org/node/view/5686 One suggestion was to change the browser settings. That means I have to tell all my users to change their browser cache settings. Not a great way to run a site. Instead, the login should just work with default browser settings. Currently, login does not work, as described below. I log in to my drupal site, feingold.christianlong.com, and the logged-in home page appears, with my user name. OK. The address bar shows http://feingold.christianlong.com/node?PHPSESSID=867c4e2d57d290a3f59e138524… In the site menu, the "Home" link links to http://feingold.christianlong.com/ Here's the problem: when I click on the "Home" link, the browser takes me to the un-logged-in version of http://feingold.christianlong.com/, which was cached when I was not logged in. When I hit refresh, I do get the logged-in version of the page. So, to restate Start at home page of site, not logged in Log in - this works, and brings up a logged-in version of the page Click "Home" - this is where the problem is. I get the cached version of the home page (from when I was not logged in) Refresh browser - now I see the correct, logged-in version of the home page. It looks like I am logging in OK, but that Drupal is not telling my browser that there is a new version of the home page that it needs to check for Maybe the original (non-logged-in) version of the page is not marked for no-cache, and so when I click on the "Home" link, I get the cached (non-logged-in) version Browsers: MSIE 6 (happens a lot) and Firebird (happens sometimes). Also happens with christianlong.com Attached, find an annotated record of the HTTP header traffic. Thanks Christian ------------------------------------------------------------------------ February 16, 2004 - 22:48 : christianlong(a)christianlong.com Forum discussion is here http://drupal.org/node/view/5669 ------------------------------------------------------------------------ March 4, 2004 - 19:49 : moshe weitzman I am seing this same behavior at http://www.nshp.org [1]. To see for yourself, login using the box on the home page while using IE (Firefox doesn't have a problem). user: mwtest8 pass: testpass Here is the php config of the server [2]. Note that IE is set to "Check for newer pages: automatically" in Options -> Temporary Internet Files -> Settings Thoughts on how to resolve this are welcome. [1] http://www.nshp.org/phpinfo.php [2] http://www.nshp.org/phpinfo.php ------------------------------------------------------------------------ June 16, 2004 - 04:38 : duztin I had this problem, fix the date on your pc, mine was a month ahead and my cookies were expiring too soon. ------------------------------------------------------------------------ July 27, 2004 - 22:18 : killes(a)www.drop.org Setting to cvs. Is the fix proposed by duztin a real fix? if not: what else could we do? ------------------------------------------------------------------------ October 16, 2004 - 23:45 : mathias Attachment: http://drupal.org/files/issues/browser_caching.patch (644 bytes) I'm moving this to critical because in some cases this bug causes the hidden $edit['destination'] value of the login form to be set to 'logout', and to the user it appears they can't login since they are immediately logged out again. Very frustrating. The other side effect of this browser caching bug are, as stated above, the authenticated user will receive stale pages and if you have the login block enabled it will look as though they're were unexplicably logged out. It all depends on your browser and its settings, but to attempt to reproduce: 1. Login to your site. 2. Next, Click the homepage link. If you are served a stale copy of the page, you've hit the bug. This seems to happen more with IE and Safari than Firefox. A potential patch is to have Drupal issue the following header: header('Cache-Control: no-store, no-cache, must-revalidate'); It works, but I don't know the implications this has on other Drupal components such as RSS conditional GETS and gzip page serving. ------------------------------------------------------------------------ October 21, 2004 - 20:30 : mathias Attachment: http://drupal.org/files/issues/browser_caching_0.patch (593 bytes) Thanks to Ethereal [3] and LiveHTTPHeaders [4], I was able to trace this problem to the Cache-Control header being sent by Mac OS X server. On this OS, mod_expires [5] is enabled by default for Apache which sets the Cache-Control time to 60 seconds for dynamically rendered pages. The implications of this action are that once you login, re-visiting any page on the site will result in stale, locally-cached versions if you viewed that page within 60 seconds of logging in. Since you weren't logged in on those pages, the system will have appeared to logged you out. It will keep doing this unless you wait 60 seconds to login. Thus, users perceive this as a failure to successfully log in. This new patch doesn't tweak bootstrap.inc. Instead it uses .htaccess to test if mod_expires is enabled and resets the caching time to 1 second for dynamically rendered content. The benefit of this approach is that it doesn't interfere with other types of caching that may be in place for images, pdf files, etc. # Overload mod_expires variables. <IfModule mod_expires.c> # Reduce the time dynamically generated HTML pages are cache-able. ExpiresDefault A1 </IfModule> Down the road it may also be wise to consider sending our own caching headers to maintain control of our caching environment. I reviewed the following pieces of software, all of which intentionally disable caching by the client browser or proxy caches. I just grabbed the ones I thought were most popular. Plone (Cache-Control parameter is configurable) Wordpress (except for RSS feeds) eZpublish Mambo phpBB And if you're so inclined, here's the relevent code snippets [6] for each piece of software. [3] http://www.ethereal.com/ [4] http://livehttpheaders.mozdev.org/ [5] http://httpd.apache.org/docs/mod/mod_expires.html [6] http://asitis.org/tmp/cms_cache_review.txt ------------------------------------------------------------------------ October 21, 2004 - 20:39 : moshe weitzman I'v seen this bug on drupal.org, so this impacts more than OSX server ... Looks like a nice clean patch to me. ------------------------------------------------------------------------ October 26, 2004 - 18:21 : jvandyk +1 from me. This reduces confusion among end users while retaining caching ability for other mime types. ------------------------------------------------------------------------ October 27, 2004 - 10:05 : wazdog Does this end up having any effect on the RSS conditional GETS? If not, then +1 from me. (The WordPress code does check for RSS, so I'm thinking this may set them to expire too?) ------------------------------------------------------------------------ October 28, 2004 - 03:58 : jvandyk Attachment: http://drupal.org/files/issues/htaccess_0.patch (792 bytes) Regarding the concern about RSS caching: this patch is identical to the previous one except instead of targetting the default we target the text/html MIME type. <IfModule mod_expires.c> # Reduce the time dynamically generated HTML pages are cache-able. ExpiresByType text/html A1 </IfModule> This means that RSS feeds, which are MIME type text/xml, are not affected. ------------------------------------------------------------------------ February 11, 2005 - 03:47 : moshe weitzman i think this is a small, worthwhile patch. this bug plagues drupal.org as well. ------------------------------------------------------------------------ February 11, 2005 - 15:45 : Junyor @jvandyk: Is it just me or is there some junk before the meat of that patch? Better cache header handling would be a welcome addition here[1], but I'm not sure if the suggested patch will solve the problem efficiently. Isn't this change saying that all pages must always be redownloaded if mod_expires is enabled? Won't that cause unnecessary bandwidth and performance overhead? [1] I'm constantly seeing stale pages throughout all authenticated pages with Opera. ------------------------------------------------------------------------ February 11, 2005 - 16:42 : moshe weitzman "Isn't this change saying that all pages must always be redownloaded if mod_expires is enabled?" Correct. And that is desired behavior. A dynamic application like ours has no choice but to redownload every page (excluding RSS feeds). ------------------------------------------------------------------------ February 19, 2005 - 20:52 : killes(a)www.drop.org This patch only affects users of mod_expires and harms nobody else. +1 ------------------------------------------------------------------------ February 21, 2005 - 06:01 : Steven Applied to CVS. Do we still need our own Cache Control mechanism? I'm included to let the server handle this. ------------------------------------------------------------------------ March 9, 2005 - 08:27 : mathias Attachment: http://drupal.org/files/issues/control-browser-caching.patch (713 bytes) Browsers are sometimes pulling files from their cache and serving stale pages when instead they should be asking the server for a new copy. This is because Drupal doesn't issue its own Cache-Control headers like most other CMS's [7]. Instead those details are currently left up to each server, which users on shared hosting aren't authorized to configure. This bug rears it's ugly head when site admins use Drupal's caching mechanism. If a user clicks on the homepage after logging in, they're very likely to see a stale unauthenticated view. Or if they log out and returns to the homepage, it'll appear that they're still logged in. But perhaps the most confusing of these caching issues is when you click the login button and are once again presented with the same login form. These cached views are occurring because the browser still thinks those pages are valid in it's cache. In otherwards, it's sending if-Modified-Since headers and receiving a valid HTTP 304 response. Now this doesn't happen 100% of the time in all cases. It very much depends on the browser (usually Firefox or Safari but never Konqueror) and how the server is configured (Mac OS X server can be quite troublesome for example while a default FreeBSD install of apache usually sends the proper headers). The most elegant solution I've come up with is to issue our own Cache-Control headers after a request has been cached by Drupal's caching mechanism. The header workflow then becomes: // User requests a page that isn't in the Drupal cache table HTTP 200 (server issues it's own caching headers) // User requests the same page (now in the cache table) HTTP 200 (Drupal issues it's own caching headers explicitly stating not to cache the page) // User requests the same page (now in the cache table) HTTP 304 (server issues it's own caching headers) If I'm understanding how things work, this lets the browser use the cached page only as long as we're sending 304 responses. When the user logs in, the browser cache is invalidated since a different set of headers are emitted, causing the stale copies to expire. The end result is that browsers are still allowed to do caching of requests (including XML feeds) but only hold on to those copies until Drupal says otherwise. [7] http://asitis.org/tmp/cms_cache_review.txt ------------------------------------------------------------------------ March 9, 2005 - 08:37 : chx +1 ------------------------------------------------------------------------ March 9, 2005 - 10:25 : Bèr Kessels +1 from me. Nice clean patch. Works in FF and konq, cannot test in IE, which has very aggressive caching, AFAIK. ------------------------------------------------------------------------ March 9, 2005 - 20:43 : mathias Setting this to active at the moment since Safari users are still experiencing caching issues, albeit less in frequency. ------------------------------------------------------------------------ March 10, 2005 - 06:12 : mathias Attachment: http://drupal.org/files/issues/control-browser-caching_0.patch (1.14 KB) I think this is the elegant catch-all case. The most problematic page with stale caching is the frontpage. Not variable_get('site_frontpage'), but when $_GET['q'] is NULL. The solution that works is to make sure this request is never cached by the browser (Drupal can still cache it of course). So in summary. This patch should resolve all browser caching problems while still gracefully emitting HTTP 304 headers in all cases but the / request.

1 0

[drupal-devel] [bug] Prevent browser page caching of dynamic content
by mathias 10 Mar '05

10 Mar '05

Issue status update for http://drupal.org/node/5900 Project: Drupal Version: cvs Component: user system Category: bug reports Priority: critical Assigned to: mathias Reported by: christianlong(a)christianlong.com Updated by: mathias -Status: active +Status: patch Attachment: http://drupal.org/files/issues/control-browser-caching_0.patch (1.14 KB) I think this is the elegant catch-all case. The most problematic page with stale caching is the frontpage. Not variable_get('site_frontpage'), but when $_GET['q'] is NULL. The solution that works is to make sure this request is never cached by the browser (Drupal can still cache it of course). So in summary. This patch should resolve all browser caching problems while still gracefully emitting HTTP 304 headers in all cases but the / request. mathias Previous comments: ------------------------------------------------------------------------ February 16, 2004 - 14:44 : christianlong(a)christianlong.com Attachment: http://drupal.org/files/issues/f_040214_01.txt (6.33 KB) This bug has been discussed before, but not resolved. http://drupal.org/node/view/1740 http://drupal.org/node/view/5686 One suggestion was to change the browser settings. That means I have to tell all my users to change their browser cache settings. Not a great way to run a site. Instead, the login should just work with default browser settings. Currently, login does not work, as described below. I log in to my drupal site, feingold.christianlong.com, and the logged-in home page appears, with my user name. OK. The address bar shows http://feingold.christianlong.com/node?PHPSESSID=867c4e2d57d290a3f59e138524… In the site menu, the "Home" link links to http://feingold.christianlong.com/ Here's the problem: when I click on the "Home" link, the browser takes me to the un-logged-in version of http://feingold.christianlong.com/, which was cached when I was not logged in. When I hit refresh, I do get the logged-in version of the page. So, to restate Start at home page of site, not logged in Log in - this works, and brings up a logged-in version of the page Click "Home" - this is where the problem is. I get the cached version of the home page (from when I was not logged in) Refresh browser - now I see the correct, logged-in version of the home page. It looks like I am logging in OK, but that Drupal is not telling my browser that there is a new version of the home page that it needs to check for Maybe the original (non-logged-in) version of the page is not marked for no-cache, and so when I click on the "Home" link, I get the cached (non-logged-in) version Browsers: MSIE 6 (happens a lot) and Firebird (happens sometimes). Also happens with christianlong.com Attached, find an annotated record of the HTTP header traffic. Thanks Christian ------------------------------------------------------------------------ February 16, 2004 - 14:48 : christianlong(a)christianlong.com Forum discussion is here http://drupal.org/node/view/5669 ------------------------------------------------------------------------ March 4, 2004 - 11:49 : moshe weitzman I am seing this same behavior at http://www.nshp.org [1]. To see for yourself, login using the box on the home page while using IE (Firefox doesn't have a problem). user: mwtest8 pass: testpass Here is the php config of the server [2]. Note that IE is set to "Check for newer pages: automatically" in Options -> Temporary Internet Files -> Settings Thoughts on how to resolve this are welcome. [1] http://www.nshp.org/phpinfo.php [2] http://www.nshp.org/phpinfo.php ------------------------------------------------------------------------ June 15, 2004 - 20:38 : duztin I had this problem, fix the date on your pc, mine was a month ahead and my cookies were expiring too soon. ------------------------------------------------------------------------ July 27, 2004 - 14:18 : killes(a)www.drop.org Setting to cvs. Is the fix proposed by duztin a real fix? if not: what else could we do? ------------------------------------------------------------------------ October 16, 2004 - 15:45 : mathias Attachment: http://drupal.org/files/issues/browser_caching.patch (644 bytes) I'm moving this to critical because in some cases this bug causes the hidden $edit['destination'] value of the login form to be set to 'logout', and to the user it appears they can't login since they are immediately logged out again. Very frustrating. The other side effect of this browser caching bug are, as stated above, the authenticated user will receive stale pages and if you have the login block enabled it will look as though they're were unexplicably logged out. It all depends on your browser and its settings, but to attempt to reproduce: 1. Login to your site. 2. Next, Click the homepage link. If you are served a stale copy of the page, you've hit the bug. This seems to happen more with IE and Safari than Firefox. A potential patch is to have Drupal issue the following header: header('Cache-Control: no-store, no-cache, must-revalidate'); It works, but I don't know the implications this has on other Drupal components such as RSS conditional GETS and gzip page serving. ------------------------------------------------------------------------ October 21, 2004 - 12:30 : mathias Attachment: http://drupal.org/files/issues/browser_caching_0.patch (593 bytes) Thanks to Ethereal [3] and LiveHTTPHeaders [4], I was able to trace this problem to the Cache-Control header being sent by Mac OS X server. On this OS, mod_expires [5] is enabled by default for Apache which sets the Cache-Control time to 60 seconds for dynamically rendered pages. The implications of this action are that once you login, re-visiting any page on the site will result in stale, locally-cached versions if you viewed that page within 60 seconds of logging in. Since you weren't logged in on those pages, the system will have appeared to logged you out. It will keep doing this unless you wait 60 seconds to login. Thus, users perceive this as a failure to successfully log in. This new patch doesn't tweak bootstrap.inc. Instead it uses .htaccess to test if mod_expires is enabled and resets the caching time to 1 second for dynamically rendered content. The benefit of this approach is that it doesn't interfere with other types of caching that may be in place for images, pdf files, etc. # Overload mod_expires variables. <IfModule mod_expires.c> # Reduce the time dynamically generated HTML pages are cache-able. ExpiresDefault A1 </IfModule> Down the road it may also be wise to consider sending our own caching headers to maintain control of our caching environment. I reviewed the following pieces of software, all of which intentionally disable caching by the client browser or proxy caches. I just grabbed the ones I thought were most popular. Plone (Cache-Control parameter is configurable) Wordpress (except for RSS feeds) eZpublish Mambo phpBB And if you're so inclined, here's the relevent code snippets [6] for each piece of software. [3] http://www.ethereal.com/ [4] http://livehttpheaders.mozdev.org/ [5] http://httpd.apache.org/docs/mod/mod_expires.html [6] http://asitis.org/tmp/cms_cache_review.txt ------------------------------------------------------------------------ October 21, 2004 - 12:39 : moshe weitzman I'v seen this bug on drupal.org, so this impacts more than OSX server ... Looks like a nice clean patch to me. ------------------------------------------------------------------------ October 26, 2004 - 10:21 : jvandyk +1 from me. This reduces confusion among end users while retaining caching ability for other mime types. ------------------------------------------------------------------------ October 27, 2004 - 02:05 : wazdog Does this end up having any effect on the RSS conditional GETS? If not, then +1 from me. (The WordPress code does check for RSS, so I'm thinking this may set them to expire too?) ------------------------------------------------------------------------ October 27, 2004 - 19:58 : jvandyk Attachment: http://drupal.org/files/issues/htaccess_0.patch (792 bytes) Regarding the concern about RSS caching: this patch is identical to the previous one except instead of targetting the default we target the text/html MIME type. <IfModule mod_expires.c> # Reduce the time dynamically generated HTML pages are cache-able. ExpiresByType text/html A1 </IfModule> This means that RSS feeds, which are MIME type text/xml, are not affected. ------------------------------------------------------------------------ February 10, 2005 - 19:47 : moshe weitzman i think this is a small, worthwhile patch. this bug plagues drupal.org as well. ------------------------------------------------------------------------ February 11, 2005 - 07:45 : Junyor @jvandyk: Is it just me or is there some junk before the meat of that patch? Better cache header handling would be a welcome addition here[1], but I'm not sure if the suggested patch will solve the problem efficiently. Isn't this change saying that all pages must always be redownloaded if mod_expires is enabled? Won't that cause unnecessary bandwidth and performance overhead? [1] I'm constantly seeing stale pages throughout all authenticated pages with Opera. ------------------------------------------------------------------------ February 11, 2005 - 08:42 : moshe weitzman "Isn't this change saying that all pages must always be redownloaded if mod_expires is enabled?" Correct. And that is desired behavior. A dynamic application like ours has no choice but to redownload every page (excluding RSS feeds). ------------------------------------------------------------------------ February 19, 2005 - 12:52 : killes(a)www.drop.org This patch only affects users of mod_expires and harms nobody else. +1 ------------------------------------------------------------------------ February 20, 2005 - 22:01 : Steven Applied to CVS. Do we still need our own Cache Control mechanism? I'm included to let the server handle this. ------------------------------------------------------------------------ March 9, 2005 - 00:27 : mathias Attachment: http://drupal.org/files/issues/control-browser-caching.patch (713 bytes) Browsers are sometimes pulling files from their cache and serving stale pages when instead they should be asking the server for a new copy. This is because Drupal doesn't issue its own Cache-Control headers like most other CMS's [7]. Instead those details are currently left up to each server, which users on shared hosting aren't authorized to configure. This bug rears it's ugly head when site admins use Drupal's caching mechanism. If a user clicks on the homepage after logging in, they're very likely to see a stale unauthenticated view. Or if they log out and returns to the homepage, it'll appear that they're still logged in. But perhaps the most confusing of these caching issues is when you click the login button and are once again presented with the same login form. These cached views are occurring because the browser still thinks those pages are valid in it's cache. In otherwards, it's sending if-Modified-Since headers and receiving a valid HTTP 304 response. Now this doesn't happen 100% of the time in all cases. It very much depends on the browser (usually Firefox or Safari but never Konqueror) and how the server is configured (Mac OS X server can be quite troublesome for example while a default FreeBSD install of apache usually sends the proper headers). The most elegant solution I've come up with is to issue our own Cache-Control headers after a request has been cached by Drupal's caching mechanism. The header workflow then becomes: // User requests a page that isn't in the Drupal cache table HTTP 200 (server issues it's own caching headers) // User requests the same page (now in the cache table) HTTP 200 (Drupal issues it's own caching headers explicitly stating not to cache the page) // User requests the same page (now in the cache table) HTTP 304 (server issues it's own caching headers) If I'm understanding how things work, this lets the browser use the cached page only as long as we're sending 304 responses. When the user logs in, the browser cache is invalidated since a different set of headers are emitted, causing the stale copies to expire. The end result is that browsers are still allowed to do caching of requests (including XML feeds) but only hold on to those copies until Drupal says otherwise. [7] http://asitis.org/tmp/cms_cache_review.txt ------------------------------------------------------------------------ March 9, 2005 - 00:37 : chx +1 ------------------------------------------------------------------------ March 9, 2005 - 02:25 : Bèr Kessels +1 from me. Nice clean patch. Works in FF and konq, cannot test in IE, which has very aggressive caching, AFAIK. ------------------------------------------------------------------------ March 9, 2005 - 12:43 : mathias Setting this to active at the moment since Safari users are still experiencing caching issues, albeit less in frequency.

1 0

[drupal-devel] Allow overriding of links returned by modules
by K B 10 Mar '05

10 Mar '05

I dont think that this feature got its share of discussion. http://drupal.org/node/18260 This issue comes up in various forms for various modules and various themes.

1 0

[drupal-devel] caching issues
by Jeremy Andrews 10 Mar '05

10 Mar '05

Hello, I ran into a problem with the way Drupal caches data today. A spam bot started crawling my site for the past 36 hours or so, posting dozens of comments every minute (frequently several a second). Combined with normal traffic, my site was serving 3-400 pages every 60 seconds. Because the spam comments were being posted at such a high speed, the cache was being flushed too quickly to do any good. I may as well have disabled the cache. The site became sluggish. (It has handled that large a load before, just not with comments being posted so quickly) I am planning to patch my site to modify how the cache is flushed, and perhaps to work again on file-based caching. No matter how optimized a CMS, there will come a time when the limitations of the hardware prevent a site from updating in real-time. Large websites (ie Slashdot) have to rebuild their caches every n minutes, instead of every time a new comment is posted. Like it or not, this is significantly more efficient. I would like to target the effort for core inclusion into 4.7. Thus, I would like to brainstorm now and work out an acceptable design. If nobody is interested and this has no chance of getting into 4.7, I'll do it on my own anyway, but I'd much prefer to get something into core so I don't have to redo it with every release. Here are some proposals. I personally would like to see one or more of these available _in addition_ to the current method. ie, most sites would leave the cache as we're all used to. Busier sites would enable one of these alternative caching mechanisms (in order of coding complexity): 0) Current Drupal caching. What is in the cache is always valid. If new content is posted, the cache is potentially invalid so it is flushed and everything is rebuilt. (Some stuff sticks around, but that stuff will be unmodified by my proposals) 1) Time-based caching. Simply flush the cache every n minutes. When new content is posted, a message such as "your comment will become visible in n minutes" would need to be displayed. (This would have saved me most recently with the spambot problem I had today...) 2) Fuzzy time-based caching. Patches against 4.2 exist in CVS [1] to see what I'm referring to (patches apply in order: 1, then 2, then 3...). It's similar to idea #1, but slightly more complicated. The cache becomes "dirty" every n minutes. When a "dirty" cache page is requested, it may or may not be rebuilt by the requester (a call to random makes the determination). If after n+x minutes the cache entry still hasn't been rebuilt, it is flushed (forcing a rebuild). (The idea is to soften the affect of flushing the cache. In example 1, there will be a CPU spike every n minutes. In example 2, the CPU load is distributed randomly.) In other words, there's a soft timeout, and a hard timeout. After the soft timeout, the cache entry may be rebuilt. After the hard timeout, the cache entry has to be rebuilt. 3) File-based caching. Patches against 4.0 exist in CVS [2] to see what I'm referring to. The simplest mechanism would be like #1 above, but with files stored in the filesystem instead of in the database. When I utilized this in 4.0, the performance boost was phenomenal. Additionally, the site could continue to serve pages with the database stopped. 4) Fuzzy file-based caching. This is actually how I implemented file-based caching against 4.0 long ago. If you got this far and understood examples 1, 2, and 3, then no further explanation is needed here. Thoughts? Feedback? Suggestions? Dries? I'll work up patches. But if someone has better design ideas, now is a good time to suggest it. Thanks, -Jeremy [1] http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/jeremy/4.2.0/cac… [2] http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/jeremy/4.0.0/fil…

8 12