January 2005 - Development - lists.drupal.org

[drupal-devel] [bug] non-intuitive revisions
by mcd 14 Jan '05

14 Jan '05

Project: Drupal Version: 4.5.0 Component: node system Category: bug reports Priority: critical Assigned to: wazdog Reported by: wazdog Updated by: mcd Status: patch I tried this patch against 4.5.1 and it worked, but it took me some time to figure out that it was working because it's still not doing the intuitive thing. The user still can't see the revisions tab unless they have node administration privileges (which makes it unlike the outline tab). They can't see the checkbox to decide whether their edit is a new revision, so every edit they do becomes a revision from which they derive no benefit. I would let them see the revision checkbox and the revisions tab, and also roll back (non-destructively as suggested in http://drupal.org/node/12479) I don't think there is an intuitive way to handle the five node administration Options checkboxes as a group while appearing to enable them separately in the default workflow. I've had similar issues with the defaults, like a page I made sticky coming unstuck every time the owner edited it (with the default non-sticky in the workflow). Now I see why that was happening, but it's still highly counterintuitive. mcd Previous comments: ------------------------------------------------------------------------ November 1, 2004 - 03:33 : wazdog If you enable revisions for a node type via content > configure > default workflow, and the user who is editing a node of that type doesn't have the administer nodes permission, then a new revision is not created. I see this a huge bug. I am trying to have all edits to a node type saved as revisions (in this case a flexinode, but I also tested with blog types). I want to recreate the book pages, but with more fields (hence, flexinode). But if one of my authenticated users creates a node, and then later edits it, a revision is not made. However, if an admin or moderator (who both have the administer nodes permission) edits the node, a new revision is created. I don't want my regular users to have administer nodes permission, that gives them too much power, but their revisions need to be kept (just like I give them all publish permission via default workflow). If this is supposed to be how it works, what is the point of the default workflow setting? (and why do the other default workflow settings behave differently? i.e. they all apply no matter the users permissions) ------------------------------------------------------------------------ November 2, 2004 - 19:48 : wazdog Attachment: http://drupal.org/files/issues/node-rev.patch (721 bytes) Okay, here's a short little patch. It seems like node_create_revision() was being called before $node->revision was set. Since $node->revision is what tells Drupal whether or not to make a new revision, this was causing nodes to not keep revisions. I moved node_create_revision() to after this check. I'm not sure exactly why only regular users without 'administer nodes' permission were affected, but that seems the case. Now, if a particular node type has 'revision' checked in content > default workflow, a new revision is always created, no matter who is doing the editing... I'm setting this to critical because I think the current behavior goes against logic, and what admins may expect. And if they don't notice it's not working like they expect (i.e. revisions aren't being kept always), then content is lost... The patch is against 4.5.0, but always applies to current cvs (with a slight offset error). -- View: http://drupal.org/node/12401 Edit: http://drupal.org/project/comments/add/12401

1 0

[drupal-devel] Should there be a revisions permission?
by Robert Douglass 14 Jan '05

14 Jan '05

Hi all, I was wondering what thoughts there are on why revisions require the 'administer nodes' permission and don't have their own? There are many things that 'administer nodes' allows you to do (access node admin pages, different node validation, access to the admin form elements in node creation etc.) which are significantly different in nature and scope than revisions of nodes. It seems to me that there are cases where normal users should be able to make revisions without having access to the other stuff. Thoughts? cheers, Robert -- visit me at www.robshouse.net!

2 1

[drupal-devel] Inter-module function calls
by Sheldon Rampton 14 Jan '05

14 Jan '05

Can someone tell me what the best way is for me to invoke functions from an existing module when writing a new module? For example, "taxonomy_get_vocabularies" from the taxonomy module works fine if I call it directly from within a new module I'm writing, but is it kosher to do this? Should I be using module_invoke instead? -- -------------------------------- | Sheldon Rampton | Research director, Center for Media & Democracy (www.prwatch.org) | Author of books including: | Friends In Deed: The Story of US-Nicaragua Sister Cities | Toxic Sludge Is Good For You | Mad Cow USA | Trust Us, We're Experts | Weapons of Mass Deception | Banana Republicans -------------------------------- | Subscribe to our free weekly list serve by visiting: | http://www.prwatch.org/cmd/subscribe_sotd.html | | Donate now to support independent, public interest reporting: | https://secure.groundspring.org/dn/index.php?id=1118 --------------------------------

2 1

[drupal-devel] re: that node design mock-ups
by Charlie Lowe 14 Jan '05

14 Jan '05

Ber wrote, ". . . . This all dictates we take this project structurally, start bottom up: What can nodeapi do, where can it do that, what can appear, what not etc. The only thing that is required, ATM is the title, all other elements might (and thus eventually will) appear or disappear." I agree. Note that if a modular approach is used, once implemented, it will allow theme designers the ability to begin experimenting with various UI layouts. So FactoryJoe's particular input design might be the default in Drupal core for the next release. Theme designers will of course tweak it, change it around given enough felxibility, and the UI will continue to evolve. In subsequent Drupal releases, the best practices can be implemented as the default layout for Drupal core. But none of the theme designers need feel locked into it; they can always work to improve it on their own. IMHO, this is a good emergent model for developing the UI which will also make Drupal more attractive to designers. BTW: I also posted another mockup to the discussion thread in reply to yours. Charlie

2 1

[drupal-devel] [task] This is a Title
by Anonymous 14 Jan '05

14 Jan '05

Project: User experience Version: <none> Component: accessability Category: tasks Priority: critical Assigned to: Anonymous Reported by: Anonymous Updated by: Anonymous Status: patch this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, this it a description, Anonymous -- View: http://drupal.org/node/15606 Edit: http://drupal.org/project/comments/add/15606

1 0

[drupal-devel] [bug] Ignore unpublished comments when determining last_post
by killes 14 Jan '05

14 Jan '05

Project: Drupal Version: 4.5.0 Component: tracker.module Category: bug reports Priority: normal Assigned to: Morbus Iff Reported by: Morbus Iff Updated by: killes(a)www.drop.org Status: patch I think it makes sense to only select published comments. I think this bugfix should go in both 4.5 and cvs. +1 killes(a)www.drop.org Previous comments: ------------------------------------------------------------------------ January 12, 2005 - 17:02 : Morbus Iff The automated unpublishing is great for stopping spam from being displayed, but it doesn't stop spam from corrupting another feature: my "recent posts" (tracker.module). I'll regularly get bursts of 200 auto-unpublished spams which make the "recent posts" page useless - an unpublished spam still affects the modification date of a node, which makes "recent posts" display craploads of updates, even though there really isn't any. Thus, the feature request would be: - treat an unpublished spam as a deleted spam - when a spam is automatically unpublished, revert the node modification date to its previous value. I'm not entirely sure how easily possible this is, as you'd have to make sure the date is properly set when someone says "nah, this unpublished spam was actual ham", and then republishes it. ------------------------------------------------------------------------ January 12, 2005 - 20:41 : Jeremy(a)kerneltrap.org This is a known and annoying bug. As it affects me too, I assure you I'll get to fixing this eventually... Any ideas on how to cleanly implement this are welcome. ------------------------------------------------------------------------ January 13, 2005 - 22:27 : Morbus Iff Would it seem to you that this behavior should be part of core? Ignoring spam.module entirely, if a node is updated at 11:00, and then updated with a comment at 12:00, the 12:00 is saved in there as the node's modification time. But if a comment is unpublished, shouldn't the node also be reverted back to its previous state (being 11:00)? How is the deletion of data (permanently hiding it) different from merely temporarily hiding it (ie., unpublishing it)? Perhaps I should move request to Drupal. ------------------------------------------------------------------------ January 13, 2005 - 22:32 : Jeremy(a)kerneltrap.org Feel free to move it. If I come up with a clean solution (at least as far as spam is concerned), I'll update the issue wherever it is. ------------------------------------------------------------------------ January 13, 2005 - 22:39 : Morbus Iff Was talking about this with killes (who "frankly [doesn't] care too much about comments"), and I'm now wondering if I'm going nutty with all this talk about node and comment timestamps. We should investigate to see if tracker.module looks at /unpublished/ comment timestamps instead of just published ones. If it doesn't make a distinction, then it would theoretically be a quickie patch to a SQL statement to only check against published comment timestamps, not unpublished. ------------------------------------------------------------------------ January 13, 2005 - 22:42 : Morbus Iff With a quick look, it looks like tracker_page() only checks node.status, and not comment.status. We should be able to put a patch in for comment.status = 1 (unpublished being 2), and that should solve our problem, right? ------------------------------------------------------------------------ January 13, 2005 - 22:55 : Morbus Iff Apologies... "AND c.status = 0". Modifying the two queries in tracker_page() made things, seemingly, work as I desire. Can you confirm? ------------------------------------------------------------------------ January 13, 2005 - 23:02 : Jeremy(a)kerneltrap.org It didn't appear to help with regards to the spam module. I had a recent spam flood, and with or without the 'AND c.status = 0' I saw the same posts, including those with recently unpublished spam comments. ------------------------------------------------------------------------ January 13, 2005 - 23:08 : Morbus Iff Odd - I had one unpublished piece of spam that I tested against (for 4.5.0, not CVS), and modifying tracker.module seemed to work as I intended. So, the following bit of SQL still shows you erroneous nodes with unpublished comments? SELECT DISTINCT(n.nid), n.title, n.type, n.changed, n.uid, u.name, MAX(GREATEST(n.changed, c.timestamp)) AS last_post FROM node n LEFT JOIN comments c ON n.nid = c.nid INNER JOIN users u ON n.uid = u.uid WHERE n.status = 1 AND c.status = 0 AND '1' GROUP BY n.nid, n.title, n.type, n.changed, n.uid, u.name ORDER BY last_post DESC LIMIT 10; The above is what is actually run against my database with the proposed tracker.module change. (This code is currently active on gamegrene.com, so we can test live if need be. Submit a comment as an anonymous user with the words "poker" or "texas hold'em", and it should be a) unpublished automatically via your module, and b) not shown in the tracker. ------------------------------------------------------------------------ January 14, 2005 - 13:52 : Morbus Iff For what it's worth, I just woke up to another flood of 150 spams, and no corruption of my recent posts. ------------------------------------------------------------------------ January 14, 2005 - 14:05 : Jeremy(a)kerneltrap.org Then it sounds like a patch is in order... For the tracker module. ------------------------------------------------------------------------ January 14, 2005 - 14:59 : Morbus Iff Have you been able to confirm this though? ------------------------------------------------------------------------ January 14, 2005 - 15:46 : Morbus Iff Attachment: http://drupal.org/files/issues/tracker_0.patch (2.59 KB) Patch attached for 4.5.0. Haven't tried applying it to CVS, but it needs to go there too. -- View: http://drupal.org/node/15500 Edit: http://drupal.org/project/comments/add/15500

1 0

[drupal-devel] [bug] Ignore unpublished comments when determining last_post
by Morbus Iff 14 Jan '05

14 Jan '05

Project: Drupal Version: 4.5.0 Component: tracker.module Category: bug reports Priority: normal Assigned to: Morbus Iff Reported by: Morbus Iff Updated by: Morbus Iff Status: patch Attachment: http://drupal.org/files/issues/tracker_0.patch (2.59 KB) Patch attached for 4.5.0. Haven't tried applying it to CVS, but it needs to go there too. Morbus Iff Previous comments: ------------------------------------------------------------------------ January 12, 2005 - 11:02 : Morbus Iff The automated unpublishing is great for stopping spam from being displayed, but it doesn't stop spam from corrupting another feature: my "recent posts" (tracker.module). I'll regularly get bursts of 200 auto-unpublished spams which make the "recent posts" page useless - an unpublished spam still affects the modification date of a node, which makes "recent posts" display craploads of updates, even though there really isn't any. Thus, the feature request would be: - treat an unpublished spam as a deleted spam - when a spam is automatically unpublished, revert the node modification date to its previous value. I'm not entirely sure how easily possible this is, as you'd have to make sure the date is properly set when someone says "nah, this unpublished spam was actual ham", and then republishes it. ------------------------------------------------------------------------ January 12, 2005 - 14:41 : Jeremy(a)kerneltrap.org This is a known and annoying bug. As it affects me too, I assure you I'll get to fixing this eventually... Any ideas on how to cleanly implement this are welcome. ------------------------------------------------------------------------ January 13, 2005 - 16:27 : Morbus Iff Would it seem to you that this behavior should be part of core? Ignoring spam.module entirely, if a node is updated at 11:00, and then updated with a comment at 12:00, the 12:00 is saved in there as the node's modification time. But if a comment is unpublished, shouldn't the node also be reverted back to its previous state (being 11:00)? How is the deletion of data (permanently hiding it) different from merely temporarily hiding it (ie., unpublishing it)? Perhaps I should move request to Drupal. ------------------------------------------------------------------------ January 13, 2005 - 16:32 : Jeremy(a)kerneltrap.org Feel free to move it. If I come up with a clean solution (at least as far as spam is concerned), I'll update the issue wherever it is. ------------------------------------------------------------------------ January 13, 2005 - 16:39 : Morbus Iff Was talking about this with killes (who "frankly [doesn't] care too much about comments"), and I'm now wondering if I'm going nutty with all this talk about node and comment timestamps. We should investigate to see if tracker.module looks at /unpublished/ comment timestamps instead of just published ones. If it doesn't make a distinction, then it would theoretically be a quickie patch to a SQL statement to only check against published comment timestamps, not unpublished. ------------------------------------------------------------------------ January 13, 2005 - 16:42 : Morbus Iff With a quick look, it looks like tracker_page() only checks node.status, and not comment.status. We should be able to put a patch in for comment.status = 1 (unpublished being 2), and that should solve our problem, right? ------------------------------------------------------------------------ January 13, 2005 - 16:55 : Morbus Iff Apologies... "AND c.status = 0". Modifying the two queries in tracker_page() made things, seemingly, work as I desire. Can you confirm? ------------------------------------------------------------------------ January 13, 2005 - 17:02 : Jeremy(a)kerneltrap.org It didn't appear to help with regards to the spam module. I had a recent spam flood, and with or without the 'AND c.status = 0' I saw the same posts, including those with recently unpublished spam comments. ------------------------------------------------------------------------ January 13, 2005 - 17:08 : Morbus Iff Odd - I had one unpublished piece of spam that I tested against (for 4.5.0, not CVS), and modifying tracker.module seemed to work as I intended. So, the following bit of SQL still shows you erroneous nodes with unpublished comments? SELECT DISTINCT(n.nid), n.title, n.type, n.changed, n.uid, u.name, MAX(GREATEST(n.changed, c.timestamp)) AS last_post FROM node n LEFT JOIN comments c ON n.nid = c.nid INNER JOIN users u ON n.uid = u.uid WHERE n.status = 1 AND c.status = 0 AND '1' GROUP BY n.nid, n.title, n.type, n.changed, n.uid, u.name ORDER BY last_post DESC LIMIT 10; The above is what is actually run against my database with the proposed tracker.module change. (This code is currently active on gamegrene.com, so we can test live if need be. Submit a comment as an anonymous user with the words "poker" or "texas hold'em", and it should be a) unpublished automatically via your module, and b) not shown in the tracker. ------------------------------------------------------------------------ January 14, 2005 - 07:52 : Morbus Iff For what it's worth, I just woke up to another flood of 150 spams, and no corruption of my recent posts. ------------------------------------------------------------------------ January 14, 2005 - 08:05 : Jeremy(a)kerneltrap.org Then it sounds like a patch is in order... For the tracker module. ------------------------------------------------------------------------ January 14, 2005 - 08:59 : Morbus Iff Have you been able to confirm this though? -- View: http://drupal.org/node/15500 Edit: http://drupal.org/project/comments/add/15500

1 0

[drupal-devel] cvs log messages
by no-reply＠zind.net 14 Jan '05

14 Jan '05

File: /sandbox/dikini/binder/binder.sql 1.2 /sandbox/dikini/binder/state_machines_and_metadata.html 1.2 Date: January 14, 2005 - 12:06 User: dikini initiall commit ---------------------------------------------------------------------- File: /sandbox/dikini/binder/binder.module 1.2 Date: January 14, 2005 - 12:05 User: dikini initial commit ---------------------------------------------------------------------- File: /modules/sidecontent/README.txt 1.3 /modules/sidecontent/sidecontent.module 1.2 Date: January 14, 2005 - 11:27 User: MegaGrunt Permisions bug fix Side Content field made narrower ---------------------------------------------------------------------- File: /modules/mail_archive/CHANGELOG.txt 1.4 /modules/mail_archive/mail_archive.module 1.5 /modules/mail_archive/README.txt 1.5 Date: January 14, 2005 - 06:36 User: jeremy - mail_archive.module: o when determining if attachment is text, use 'encoding', not 'type' o formatting cleanup o fixed message filtering to use selected subscription filter format o enhancing parsing of 'message_id' and 'in_reply_to' to handle wider range of possible formats (also fixed related threading issues) o fixed rethreading of "corrupted" threads (ie, if top parent is deleted) A demo of this module can now be found at: http://kerneltrap.org/mailarchive ---------------------------------------------------------------------- File: /modules/action/action.module 1.5 /modules/action/actions.inc 1.2 Date: January 14, 2005 - 03:35 User: jvandyk parity with 4.5 ---------------------------------------------------------------------- File: /modules/search.module 1.111 Date: January 14, 2005 - 03:24 User: unconed Fixing search tabs going back to "content" when searching on other tabs. ---------------------------------------------------------------------- File: /modules/project/CHANGELOG.txt 1.1.2.1 DRUPAL-4-5 /modules/project/comment.inc 1.39.2.1 DRUPAL-4-5 /modules/project/INSTALL.txt 1.1.2.1 DRUPAL-4-5 /modules/project/issue.inc 1.84.2.1 DRUPAL-4-5 /modules/project/mail.inc 1.36.2.1 DRUPAL-4-5 /modules/project/MAINTAINERS.txt 1.1.2.1 DRUPAL-4-5 /modules/project/project.css 1.22.2.1 DRUPAL-4-5 /modules/project/project.inc 1.44.2.1 DRUPAL-4-5 /modules/project/project.module 1.175.2.1 DRUPAL-4-5 /modules/project/project.mysql 1.14.2.1 DRUPAL-4-5 /modules/project/README.txt 1.1.2.1 DRUPAL-4-5 /modules/project/release.inc 1.42.2.1 DRUPAL-4-5 /modules/project/update-project.php 1.1.8.1 DRUPAL-4-5 Date: January 13, 2005 - 23:20 User: unconed Tagging a slightly downgraded Project.module for 4.5 ---------------------------------------------------------------------- File: /update.php 1.142.2.3 DRUPAL-4-5 Date: January 13, 2005 - 19:25 User: dries - Patch #15337 by Gerhard: fixed bug in upgrade path. The 'update_start' variable got corrupted. ---------------------------------------------------------------------- File: /update.php 1.144 Date: January 13, 2005 - 19:24 User: dries - Patch #15337 by Gerhard: fixed bug in upgrade path. The 'update_start' variable got corrupted. ---------------------------------------------------------------------- File: /modules/ldap_integration/ldap_integration.module 1.2.2.2 DRUPAL-4-5 /modules/ldap_integration/README.txt 1.2.2.1 DRUPAL-4-5 Date: January 13, 2005 - 18:43 User: pablobm - ldap_integration.module: minor changes - README.txt: added a missing (important) detail concerning installation ---------------------------------------------------------------------- File: /modules/bookmarks/bookmarks.module 1.31.2.2 DRUPAL-4-5 Date: January 13, 2005 - 17:55 User: goba use tabs instead of menu items, to be consistent with the rest of Drupal + port some 4.6 changes which will work with 4.5 too ---------------------------------------------------------------------- File: /modules/bookmarks/bookmarks.module 1.35 Date: January 13, 2005 - 17:52 User: goba use tabs instead of menu items, to be consistent with the rest of Drupal ---------------------------------------------------------------------- File: /modules/search.module 1.110 Date: January 13, 2005 - 17:34 User: unconed - Some search doxygen tweaks ---------------------------------------------------------------------- File: /modules/flexinode/CHANGELOG 1.24 /modules/flexinode/field_textarea.inc 1.10 /modules/flexinode/flexinode.module 1.42 Date: January 13, 2005 - 16:32 User: JonBob - Fixed a DB prefixing bug. - Fixed error in table view pager. - Labels for empty text areas do not show. - Corrected overly-permissive node access rights. - Corrected duplicated listings when using node access modules. ---------------------------------------------------------------------- File: /modules/flexinode/CHANGELOG 1.23.2.1 DRUPAL-4-5 /modules/flexinode/field_textarea.inc 1.8.2.2 DRUPAL-4-5 /modules/flexinode/flexinode.module 1.36.2.2 DRUPAL-4-5 Date: January 13, 2005 - 16:31 User: JonBob - Fixed a DB prefixing bug. - Fixed error in table view pager. - Labels for empty text areas do not show. - Corrected overly-permissive node access rights. - Corrected duplicated listings when using node access modules. ---------------------------------------------------------------------- File: /INSTALL.txt 1.9 Date: January 13, 2005 - 16:15 User: unconed Making the installation instructions a bit clearer for newbies and it explicitly explains that the commands are for the command line. ---------------------------------------------------------------------- File: /modules/taxonomy_access/taxonomy_access.module 1.21 Date: January 13, 2005 - 15:38 User: pyromanfo Made SQL compatible with Mysql 3 ---------------------------------------------------------------------- File: /translations/fr/taxonomy-module.po 1.5 Date: January 13, 2005 - 15:25 User: alkahan Update by Nicolas Thauvin ---------------------------------------------------------------------- File: /modules/feedback/feedback.module 1.25 Date: January 13, 2005 - 15:13 User: kbahey 1. For performance reasons, the MX lookup is now off by default. Those who need it have to enable it themselves. 2. Newly settable Page header and response strings (successful send, or failed send). 3. Access perms description is now more descriptive. ---------------------------------------------------------------------- File: /modules/og/og.module 1.1.2.22 DRUPAL-4-5 /modules/og/readme.txt 1.1.2.8 DRUPAL-4-5 Date: January 13, 2005 - 14:50 User: weitzman fix a missing t() assure that only subscribers can post into a group. some readme improvements ---------------------------------------------------------------------- File: /modules/og/og.module 1.26 /modules/og/readme.txt 1.9 Date: January 13, 2005 - 14:50 User: weitzman fix a missing t() assure that only subscribers can post into a group. some readme improvements ----------------------------------------------------------------------

1 0

[drupal-devel] [Fwd: [droppies] Digital Communities 05 / Prix Ars Electronica]
by Dries Buytaert 14 Jan '05

14 Jan '05

-------- Original Message -------- Subject: [droppies] Digital Communities 05 / Prix Ars Electronica Date: Thu, 13 Jan 2005 13:20:12 +0100 From: AEC Communities <communities(a)prixars.aec.at> Dear Drupal Team, The category "Digital Communities", which met with great interest and participation in 2004, will be awarded again by Prix Ars Electronica in 2005. As you had been participating last year, we would kindly like to invite you and your team to submit your project "http://drupal.org/" - or a new project - in the category "Digital Communities" of this year's Prix Ars Electronica. The Digital Communities category of Prix Ars Electronica encompasses digital community projects as well as developments in the field of social software, mobile communications or wireless networks. For a detailed description of the category and about Prix Ars Electronica in general, please see our website: <http://www.aec.at/en/prix/communities/communities.asp> Prizes One Golden Nica with 10,000 Euro and two Awards of Distinction with 5,000 Euro each will be awarded and up to 12 Honorary Mentions made by the jury. The registration starts January 10, 2005. The deadline for submissions is March 11, 2005. Please use <http://www.aec.at/en/prix/registration/index.asp> for your submission and to obtain further details. If you need any further information or help, please do not hesitate to contact us. Looking forward to your participation! Sincerely yours, > Ingrid Fischer-Schreiber > Prix Ars Electronica 2005 > > AEC Ars Electronica Center Linz > Museumsgesellschaft mbH > Hauptstraße 2 > A-4040 Linz > > Tel. ++43.664.8126230 > Fax ++43.732.7272-674 > communities(a)prixars.aec.at > http://prixars.aec.at -- Dries Buytaert :: http://www.buytaert.net/ _______________________________________________ droppies mailing list droppies(a)drupal.org http://lists.drupal.org/listinfo/droppies

1 0

[drupal-devel] [feature] Centralize node queries somehow
by chx 14 Jan '05

14 Jan '05

Project: Drupal Version: cvs Component: node system Category: feature requests Priority: normal Assigned to: chx Reported by: chx Updated by: chx Status: patch Attachment: http://drupal.org/files/issues/node_rewrite_sql_8.patch (43.97 KB) PHPdoc is upgraded, I hope now it is up to date. chx Previous comments: ------------------------------------------------------------------------ December 23, 2004 - 22:25 : chx Attachment: http://drupal.org/files/issues/node_builder.patch (5.56 KB) Although I have posted this to the devel list, as Steve pointed out, putting patches to sandbox is not the best way to do things. So I open this thread. I have measured the time for build a node query in this is about 0.22-0.24ms (on an Athlon 933 MHz machine). ------------------------------------------------------------------------ December 23, 2004 - 23:33 : Dries I'm not likely to commit this. It's not conform with Drupal's coding conventions, but more importantly, a node query builder doesn't solve any real problems. It just adds a different way of doing things without offering a significant advantage. ------------------------------------------------------------------------ December 23, 2004 - 23:51 : chx Attachment: http://drupal.org/files/issues/node_builder_0.patch (5.67 KB) At this moment there are direct queries into the node table everywhere. After node_access_*_sql calls were invented, a lot of queries needed to be changed, and I think we will find more to be inserted. If someone does sg. else with the queries, permission, language etc. he needs to patch 60+ quieries in core alone. This is not a good thing. That's why I am proposing a central node query builder. As for code standards, code-style.pl node.module returns 17 errors, none of them comes from my patch. However, I should admit, there were a few spaces missing from the database.inc patch. So I resubmit. ------------------------------------------------------------------------ December 23, 2004 - 23:56 : chx Attachment: http://drupal.org/files/issues/node_builder_1.patch (5.67 KB) OK, I was wrong, there is a space missing in one of the rewritten queries. But I doubt this matters too much, as this is only a proposal and I think there will be a lot of revisions before it gets to the core. ------------------------------------------------------------------------ December 24, 2004 - 00:26 : drumm The blank lines between the comments and function starts will confuse the documentation parser on drupaldocs.org. I'm not sure if I support this quite yet. Reducing the number of queries and whatnot is great, but this patch doesn't do that quite yet. I'd only want this if the new system is more readable and easy to write. I'm not sure if this is true. As for code-style.pl, I don't think this is actually used much and it doesn't catch everything. Such as, most operators (=, =>, +, etc) like a space on either side of themselves. I'd support removing code-style.pl and throwing it in contributions somewhere. (If this needs further discussion or action, please file another issue, lets keep this focused on this patch). I'm going to wait for a few more revisions of this before I weigh in a +1 or -1. ------------------------------------------------------------------------ December 24, 2004 - 00:58 : Jose A Reyero I can really think of a number of advantages of this, and though not really for committing it yet, think is very good to see some discussion on the issue and some concrete implementations. When implementing i18n I run into this problem, of having to patch lots of queries -actually too many- in about all the modules doing any kind of node listing. This far, I'm stuck at this point. Then I realized it was just the same for node permissions, and will be the same again for whatever new functionality you want to add in the future which affects node listings. I think actually that direct access to 'node' table should be avoided when possible out of the node.module. And the only way to achieve this is to have some kind of query builder. And also think that chx's approach is very good. it provides some upper layer at the 'node' level, which can use some object level semantics while relying on a db layer query builder, which could be used later for other objects -users, taxonomy terms...- Just think of all the work that this could have saved when implementing permissions system, which I think is ugly and dirty anyway because of all that node_join, node_where... In the long run, adding such complex logic as sql code, patching and re-patching queries is bad, very bad, making any new change one higher level of complexity and driving us away from a OO model, which I think we should tend to. It's not using PHP objects. It is thinking of nodes as 'objects' and coding according to it. Oh, yes, sorry, this is growing too long to write it here :-). But we could talk also about db portability... ------------------------------------------------------------------------ December 24, 2004 - 01:14 : moshe weitzman I should add that the Organic Groups module could benefit from this node listing SQL builder. I have awkwardly worked around the problem for now but I don't like it. I don't know if this patch is good or not, but I do lean toward the functionality that it offers. Remember that any developer is free not to use the query builder and issue direct SQL as needed. If the query builder is making life hard, simply don't use, just like today. ------------------------------------------------------------------------ December 24, 2004 - 10:35 : Bèr Kessels I +1 for this functionality (i have no time to comment on coding style etc). It will not only improve maintainability, but will allow a far easier implementation of the i18n. i18n currently has a lot of patches, simpley to add logic to all those node sql queries referred to. ------------------------------------------------------------------------ December 24, 2004 - 13:20 : chx Attachment: http://drupal.org/files/issues/node_builder_2.patch (1.63 KB) OK, maybe the former approach was too complex. How about this? This requires a lot less change to the queries and no change to database.inc and ten times faster. As usual, a sample query is rewritten and an example of the proposed hook is provided. ------------------------------------------------------------------------ December 25, 2004 - 00:09 : chx Final thoughts for this day. If you like the last version, there is a possibility to automate the whole thing by adding the following three lines to the beginning of db_prefix_tables: Of course, there is very small chance of a loop here, so this would require more thought, but I think at the end this would be a good thing. ------------------------------------------------------------------------ December 27, 2004 - 12:55 : Dries I like the second approach better as it keep things readable and gives me a bit more control over the order in which things are written. Looking for more feedback from the others. Did you try updating more queries to make sure it works as intended in all cases? ------------------------------------------------------------------------ December 27, 2004 - 19:48 : chx Attachment: http://drupal.org/files/issues/node_builder_3.patch (1.23 KB) Yes, I tried with many different queries. And I have found one bug -- forgot the underscore in the regexp following FORM so I have rewritten the whole regexp. It is now a lot simpler :) And node_query is now reentrant, so if a function in the process of hook_sql calls node_query, it won't fall into an infinite loop. This situation did not occur but it's better to forestall such things. ------------------------------------------------------------------------ December 28, 2004 - 05:48 : moshe weitzman This patch is a step forward. I thought of another feature which becomes easier with the proposed hook_sql(). think of filtering like freshmeat.net. At freshmeat, you can say that you only want to see projects where OS=Windows and License=GPL (for example). A Drupal equivalent is 'only show me nodes related to Democrats and Poverty'. This sort if global node filtering is very hard in Drupal today. With this hook, it becomes easy. ------------------------------------------------------------------------ December 28, 2004 - 12:37 : Dries It's starting to look good (and you're getting more support)! Have you read the following threads: http://lists.drupal.org/archives/drupal-devel/2004-11/msg00198.html and http://lists.drupal.org/archives/drupal-devel/2004-11/msg00230.html? It's an open issue related to your work. If you start joining tables, you might have to add DISTINCT()s. When you are not joining tables, it would be nice if we'd not pay the cost of a DISTINCT(). It would be good if you could update the node queries in core to take advantage of it. It's not necessary while prototyping though. It would be good if the il8n team could take a closer look at this patch by trying to use it. It would be good if you could add some PHPDoc. ------------------------------------------------------------------------ January 4, 2005 - 07:55 : chx Here is a sum of changes in this new revision: Following Dries' advice, I read those threads and now DISTINCT(nid) is used only if there is something joined. Whether the access_grants system is used or not, is determined at the module configuration screen, so for efficiency reasons, I have patched system.module. I the very first version of my patch (you know, with arrays) I passed on the whole query so hook_sql implementations could easily determine whether they are interested in this query or not. In a very cool brainstorming with Jose, we found out, that passing the whole query is not a good thing, hints are enough. So when I will patch the forum queries, I will define a NODE_SQL_FORUM hint, and a filter which does something cool with forums (I have heard something about private forums not implemented -- I maybe wrong here) may do so. We now have a separate helper function to call hook_sql which is cool, 'cos with it the last node_access_*_sql calls which were lurking in node_search could be elinimated. ------------------------------------------------------------------------ January 4, 2005 - 07:56 : chx Attachment: http://drupal.org/files/issues/central_node_query.patch (6.46 KB) Funny thing. Forgot to attach the patch. ------------------------------------------------------------------------ January 6, 2005 - 00:46 : chx Another day, another revision. As this patch now has nothing to do with queries, but it is about rewriting node SQL statements, I have renamed it node_rewrite_sql, the helper _node_rewrite_sql and there is hook_node_rewrite_sql. hook_node_rewrite_sql expected return value has changed: it can return an array('join'=> ..., 'where'=>..., 'distinct'=> ) so it may determine whether DISTINCT is needed or not. Neither index is mandatory. That hook_node_rewrite_sql impementations will decide whether they are asking for DISTINCT or not, is -- I think -- a good opportunity to optimize for speed. For eg. I doubt every node access mechanism requires DISTINCT. One last note: hook return value can be also can be a string with a WHERE statement for the sake of simplicity. node_rewrite_sql now works with multiline queries. ------------------------------------------------------------------------ January 6, 2005 - 00:46 : chx Attachment: http://drupal.org/files/issues/central_node_query_0.patch (7.43 KB) I can't believe I once again did not attach the patch ------------------------------------------------------------------------ January 6, 2005 - 17:22 : chx Attachment: http://drupal.org/files/issues/central_node_query_1.patch (6.65 KB) Moshe, thanks for your patience and wisdom. Extract is thrown out, list is used. node_node_rewrite_sql looks better now. access_grants_present is determined in run time, once per page. ------------------------------------------------------------------------ January 6, 2005 - 21:35 : Dries The hint-stuff looks a bit hairy to me. It would be nice if this could be avoided but I see what it is necessary. The semaphore stuff is vague too. I don't see how you'd be able to get into an endless loop. It might actually be useful to turn node_access_grants_present() into a more generic module.inc function. Maybe something along the lines of module_implements($hook), which would return a list of modules implementing the specified $hook. I suggest you update more core modules. If you do, I'll measure the performance gain against a copy of drupal.org's database. ------------------------------------------------------------------------ January 6, 2005 - 22:47 : chx Attachment: http://drupal.org/files/issues/node_rewrite_sql.patch (15.72 KB) The hint stuff is something is pretty useful: hook implementations can fine tune themselves when to fire and when not. Good for features and performance, too. module_implements is done. Semaphore -- we will have tons of node_rewrite_sql calls in the core. Now someone writes a hook_node_rewrite_sql implementation which calls a function which in turn calls another one which calls node_rewrite_sql . I would rather not be the one who debugs such a scenario. Thanks for the offer! forum.module is now patched. Others will follow tomorrow. ------------------------------------------------------------------------ January 6, 2005 - 23:21 : Jose A Reyero I think this looks good enough, so I'm starting to test and use this for i18n. If this get into the node module, I don't think I will need any node.module patch anymore... A pair of performance hints: - As this node_rewrite_sql is going to be called several times for each page, we can cache the list of modules implementing the hook in a static variable. - We can even cache the whole result for each 'hint' value, so the whole hook will only be called once or twice for each page. Some suggested 'hint' values to be defined: NODE_REWRITE_SQL_LIST -> This one for simple generic list of nodes for the main page, could be default. NODE_REWRITE_SQL_ADMIN_LIST -> List of nodes for administration pages NODE_REWRITE_SQL_RELATED_LIST -> List of objects related to nodes, but no nodes themselves -like comments-, node table may have to be joined... NODE_REWRITE_SQL_BLOCK_LIST -> Listing for a block NODE_REWRITE_SQL_NO_LANGUAGE -> No need to add language conditions. This could hapen when you need a listing for other languages, or mixed languages, for translators or for administrators.. I would need something like this last one for i18n, just to have the permissions working and some flexibility with language conditions. And also the one you already have in place ...._SEARCH. All this will allow for options like search, admin, list current/all languages... ------------------------------------------------------------------------ January 7, 2005 - 17:30 : Jose A Reyero Attachment: http://drupal.org/files/issues/central_node_query_node_module.patch (7.07 KB) I found some problems when using the patch. The where conditions are not merged well if there are more than once, and also conditions need some parenthesis around. So I replaced the $where and $join strings in _node_rewrite_sql by arrays which are imploded at the end. Also added some 'hint' definitions. Patches for i18n module are coming next. This works like charm for node listings, comment listings, searches, etc... Btw, I also think 'semaphores' are not really needed. ------------------------------------------------------------------------ January 7, 2005 - 19:04 : Dries I don't see how the hint system can be portable: are contributed modules supposed to define their own hits? ------------------------------------------------------------------------ January 7, 2005 - 22:54 : chx The hints -- I was in a hurry yesterday evening, so I have forgot to define a NODE_REWRITE_SQL_FORUM in the forum.module patch and also forgot to use the hints. So I think core node types will have their hint defined. I have a vague idea about how contrib modules could define their own hint type: we shall have a NODE_REWRITE_SQL_HINT_MAX, and every hint define should like this -- beware this is not working! This is just an idea. Well, if defines does not work, maybe a global node_rewrite_sql_hints array should be introduced...? ------------------------------------------------------------------------ January 8, 2005 - 09:54 : chx Attachment: http://drupal.org/files/issues/node_rewrite_sql_0.patch (16.69 KB) Here is a list of changes: Complete rethought of hint system. Uses array of strings instead of defines. There are a number of examples. the new comment.module patch shows a good usage example for the third parameter of node_rewrite_sql which is at the moment called node_alias. It's really not a good name, it should be "the alias of the table which holds the nid field for this query". BTW. there should be an index on comment.nid. Incorporated Jose's "replaced the $where and $join strings in _node_rewrite_sql by arrays which are imploded at the end". I was a fool to lose these, the first version (which used array) used a proper implode for WHEREs. ------------------------------------------------------------------------ January 8, 2005 - 11:27 : Dries I have two fundamental problems with it this patch: The loop detection mechanism has to go. The hints are difficult to understand and I'm not convinced they are good/needed. Does the patch contain a good example of a node_rewrite_sql function/hook that uses these hints? I can't find one. I won't like these hints until I'm convinced they are needed and that they represent the best possible solution. Convince me. Could you extend the documentation about the hint-system? Currently it reads "An array of hint strings about the query, passed on to hook_sql handlers.". Clearly, that doesn't help me at all. The documentation should answer the following questions: (1) what are the hints for, (2) when should I use them (or not use them) and (3) what does hints look like (what is their format/syntax)? Note that I can't compare performance unless more queries have been updated. Thanks chx. ------------------------------------------------------------------------ January 8, 2005 - 11:30 : Dries Why does the hints in the comment module include 'related_list'. This is confusing. ------------------------------------------------------------------------ January 8, 2005 - 12:52 : Jose A Reyero Well, it seems we have different views about the 'hints'. At this point, the node_rewrite has two main uses: permissions and language conditions (i18n). I don't know whether they could have any use for permission system, but for i18n, some extra information about the rewritten queries will be needed: - Most of the queries will just select fields in the node table. Here, only a where condition is needed, but language condition may be added or not depending on the kind of query: _LIST would be a list for the main page, so only in this case, decissions can be made depending on the path. For some pages, you may want a list of nodes for all languages and for some others, only current language. _ADMIN_LIST is a list of nodes for administration. An option will be added in i18n for 'administering all languages together' or 'administer each language separately', or even a language field could be added in the search form. So language conditions will be merged diferently _BLOCK_LIST is a list of nodes for blocks. I'm thinking here of a global option like 'show mixed languages in blocks'. Anyway, this is a list for a block and you cannot have any logic depending on path here. _SEARCH, would be only for searching, and in this case you may decide to search 'all languages' or only the current one. This can be a global option or some extra field in search form. When doing text searches, searching all languages together will make a lot of sense, but also restricting search by language. About _RELATED_LIST, this is the case of comments, is a list of objects which are not nodes themselves, but related objects in a different table. Here, the node table itself may have to be joined as it is not yet in the query. When dealing with the translation system, the language conditions will be added or not by the module requesting the query itself, so no further language conditions will be needed. Queries will have to be run anyway through 'node_rewrite_sql' to have permissions in place. That's what the _NO_LANGUAGE hint is about. So this is my proposed use for the 'hint' thing, though maybe the hints themselves could be better defined or elaborated, but this is the main idea. I mean no specific hints for modules but generic ones for 'kind of listing' you are requesting. Maybe for the permission system, it could be also some use, as you could decide you want to show a list of objects to anyone, but deny access to the objects themselves, something like 'subscribe or register if you want to read the article'. So permissions here will apply differently depending on what you are doing. We can think also about more creative use of permissions like 'you can read latest articles' but old ones will be available only for selected people, or the opposite 'only old content available for everybody'. Or even 'you can read articles but not comments'. This kind of permissions would need also some 'hints'. ------------------------------------------------------------------------ January 8, 2005 - 13:17 : chx Attachment: http://drupal.org/files/issues/node_rewrite_sql_1.patch (26.89 KB) More queries rewritten. Loop detection gone. Will gather my thoughts on hint system later, but I do hope this amount of queries will be enough for a performance measurement. Now only blogapi, blog, book calls node_access_*_sql, all others are done. ------------------------------------------------------------------------ January 8, 2005 - 13:20 : chx Attachment: http://drupal.org/files/issues/node_rewrite_sql_2.patch (26.88 KB) Forgot to save taxonomy.module before rolling the patch, so a DISTINCT remained :( ------------------------------------------------------------------------ January 8, 2005 - 13:38 : chx I see Jose has written a lot about hints, thanks! "Well, it seems we have different views about the 'hints'." -- "we" probably means Dries vs. "Jose and I" 'cos I agree with Jose, but I see some more uses, namely what I have said before on making a filter on forums. If you want to that, you need to know which queries are coming from the forum.module. What shall we do? Pass on the whole query, and let the filter proccess the query to decide whether now it is the forum block it hunts for? I doubt. Let's give it a hint it's a forum and a block query. Some kind of extensionable hint mechanism is necessary, 'cos what about a contrib module which filters another contrib node, say flexinode? ------------------------------------------------------------------------ January 9, 2005 - 15:05 : Dries Could we drop the hint system if we'd choose not to support the il8n module for now? ------------------------------------------------------------------------ January 9, 2005 - 16:54 : Jose A Reyero :-( Ok, let's try to think positively. Could we at least pass the query along so i18n -or whatever module tries to implement this hook- could support itself? ------------------------------------------------------------------------ January 9, 2005 - 18:32 : chx Well, passing along the query would mean no problems, but I wonder whether a better documented hint system would be better? I had no problems with letting the loop detection system go it was just a safety belt for some distant future possibility. But the hint system is -- in my point -- is needed. So, here are the valid hint strings: node, forum, comment etc.name of the module from the query originates, like forum, comment, flexinode etc. genericthis would be be for generic listings, like node_default_page, forum_get_forums etc. relatednode table is not joined so if you want to use other fields from node table than nid you shall join node table yourself. However, if you have your table which is like JOIN {mytable} ON mt.nid=$nid_alias.nid that's fine. adminsomething meant only for administrators. blocknodes in a block searchsomething to do with a search ------------------------------------------------------------------------ January 9, 2005 - 18:36 : chx Attachment: http://drupal.org/files/issues/node_rewrite_sql_3.patch (28.31 KB) Killes said I did not had -F^f in the last version. ------------------------------------------------------------------------ January 9, 2005 - 22:37 : chx Attachment: http://drupal.org/files/issues/node_rewrite_sql_4.patch (28.77 KB) Automatic version as discussed and args for Killes. ------------------------------------------------------------------------ January 9, 2005 - 22:55 : chx Attachment: http://drupal.org/files/issues/node_rewrite_sql_5.patch (28.31 KB) Hm. It seems we have agreed on no automatization. However, arbitrary arguments are kept for Killes. ------------------------------------------------------------------------ January 10, 2005 - 21:02 : Dries The last version of this patch improved the performance of my test site's main page by 257%. That is, my machine can serve 2.57 times as much pages in the same time interval. Out of curiousity, I investigated this some more and it turns out this can be tracked down to the useless DISTINCT's which this patch removes. Unless someone objects to this patch or the approach taken, I'm going to commit this shortly. Please review this patch carefully as it marks an important change. ------------------------------------------------------------------------ January 10, 2005 - 22:39 : chx Attachment: http://drupal.org/files/issues/node_rewrite_sql_6.patch (44.18 KB) More DISTINCTs cleared, now there are no more node_access_*_sql calls in the core modules. This may be a final version. Or we can find a few more to rewrite. I've searched for {node} selects, dropped those which had nid = %d as I thought those do not need rewrite (I may be wrong) and dropped also those that are rewritten, dropped those that are indexing or cron. Here are the remaining queries, which may need rewriting: ------------------------------------------------------------------------ January 10, 2005 - 23:02 : Chris Johnson This patch seems like both a Good Thing and a Bad Thing to me. The idea of centralizing node queries is a good one, I think. It seems to be the original motivation for this patch. It allows better SQL optmization and hence performance. It puts all the code in one place for better future maintenance and enhancement. On the other hand, constructing complex SQL statements from programmatic logic is full of peril. The task quickly becomes very large and complex. Manipulating the SQL language elements via code means it is harder to see when one has made a syntactic mistake. Most such errors will be fatal. As the SQL construction code becomes more complex with development, the number of paths through it become huge and impossible to exhaustively test. Some day, someone will pass parameters that cause fatal SQL syntax to be generated in a "production" system. It also obscures SQL syntax differences between supported databases. It's more likely to introduce usage that works with MySQL but fails with Postgres. I've long been in favor of unifying and centralizing all of the database access in Drupal, to the greatest extent which does not adversely affect performance. So in this way, I think this patch is a big step in that general direction. But I've also been involved in maintaining a search engine which used this technique to create the SQL statements on the fly. It became extremely complex over time as one, then two, then three, etc. new possibilities were added (e.g. new usage, new search criteria). The more complex it became, the more regression errors we had and the harder to debug it became. It got to the point where making one small addition would result in 3 new bugs. It worked most of the time, and when it worked, it was fast. But it failed fatally in a significant number of ways. I'm not sure whether to vote +3 or -4 on this one. It might be good; it might be terrible. ------------------------------------------------------------------------ January 10, 2005 - 23:19 : chx Attachment: http://drupal.org/files/issues/node_rewrite_sql_7.patch (44.18 KB) Chris, I understand your concerns, but this method -- unlike the original -- tries to keep things as simple as possible. You can only JOIN tables and add WHERE conditions. We are not creating complex SQL on the fly, I tried that originally, just read comment #1 for the warm welcome by Dries: "I'm not likely to commit this". So do not worry, while I may be a little hotheaded, Dries keeps order with an iron hand. And the actual rewrite mechanism is also very simple. Ps. Two typos corrected in this version. ------------------------------------------------------------------------ January 13, 2005 - 23:20 : Dries I'm going to test and benchmark the latest version of this patch over the weekend. If all goes well, I'll commit this to HEAD as no concrete alternatives have been proposed. (I can't think of a better approach and believe that this patch does a good job balancing different values/requirements.) It would be much appreciated if those who maintain node-level permission modules could take a look at this patch to evaluate its usefulness/performance. If you maintain a module that queries the node table you might want to look at this as well -- after all, you'll have to update your module if this hits core. (Note that the PHPdoc code in the patch is not up-to-date.) -- View: http://drupal.org/node/14731 Edit: http://drupal.org/project/comments/add/14731

1 0