June 2008 - Development - lists.drupal.org

Drupal could be 10x times faster with correct indexing
by Jean-Michel Pouré 15 Jul '08

15 Jul '08

Dear friends, Today my Drupal 6.2 server was flooded and I discovered that there was a real lack of indexing in Drupal. Here is the issue with solutions: http://drupal.org/node/276742 When using MySQL, you don't really know how SQL queries are treated, except execution times that may be monitored (?). On the converse, PostgreSQL offers full logging of queries and access to query plans, showing how the database executes data. In the above issue, I could speed-up a simple query by 10 times. You may not notice this issue when using small Drupal sites (because the whole database executes in shared memory) or when using Drupal with caching. But ... on large installations like mine, you cannot cache everything without a huge amount of memory. And the result is SLOW. Using PostgreSQL, normal queries should run in less than one millisecond, usually 0,2 millisecond. Long queries with difficult JOINS should run in less that 5 milliseconds. This way, the database server is always faster than anay kind of PHP cache. And you can use large sites on small sites. There is an extensive query debugging needed on Drupal: 1) Turn on extensive logging in PostgreSQL, including query execution times. 2) Whenever a query executes in more than 1 millisecond, run psql or pgAdmin (graphical client) to run EXPLAIN ANALYSE. 3) Find sequential scans and fix them. Most of times, JOINS and clauses and ORDER BY should be on indexes. 4) Add proper indexing 5) Run EXPLAIN ANALYSE again. Times usualy are divided by 10. 6) Add indexes everywhere. After this hard work, a normal Drupal 6.2 installation should be able to answer thousands of queries. Do not hesitate to answer my issue : http://drupal.org/node/276742 The issue offers a pratical example. Again, this kind of problem will only show on large sites (like Drupal.org) not a small installation. But I consider this as a bug, a very important issue which should be fixed ASAP. Kind regards, Jean-Michel Pouré

8 9

programmatic creation of node
by Novák Áron 11 Jul '08

11 Jul '08

Hi! While the development of FeedAPI, i faced with a problem. FeedAPI has to create nodes programmatically, i believed that this is a really easy thing to do, feedapi now works like this: $node->data1 = "foo"; $node->data2 = "bar"; node_object_prepare($node); // this is about the default values node_save($node); But I got a report that this is not a good way to do: http://drupal.org/node/196273 Summary: "node_object_prepare() and node_prepare() functions are meant to simulate the demonstration of a node" And some users, who use FeedAPI + 3rd party modules together, really experience bugs around node creation / handling: http://drupal.org/node/195105 (summary: the core forum module uses form_alter to pass taxonomy-like data. And this data is lost now.) Can you suggest me a perfect way to handle this problem? mustafau (http://drupal.org/user/207559) suggested to use drupal_execute($form_id, $form_values), but in this case, i had another problem: drupal_execute has no useful return value and $node structure remains unaltered, so i had to do a node_load after this, which is quite expensive. Thanks, Aron Novak

11 20

Re: [development] Drupal could be 10x times faster with correct indexing
by David Timothy Strauss 08 Jul '08

08 Jul '08

When it comes to database capabilities, MySQL is quite a bit more than a glorified card catalog. When it comes to database architecture, Drupal developers are quite a bit more than rejected Microsoft Access programmers. ----- "Jean-Michel Pouré" <jm(a)poure.com> wrote: > Here is the issue with solutions: > http://drupal.org/node/276742 >From the issue, it seems like your indexes are missing or misconfigured. > When using MySQL, you don't really know how SQL queries are treated, > except execution times that may be monitored (?). Wrong. Use EXPLAIN. > This way, the database server is always faster than any kind of PHP cache. Wrong. Databases have higher latency for even simple queries compared to PHP static caching and memcache. > There is an extensive query debugging needed on Drupal: > [...] > 6) Add indexes everywhere. Wrong. Indexes add multiple forms of overhead. They should not be added gratuitously. Indexes should be added when they enhance performance more than they cost it or if they enhance data integrity (UNIQUE/PRIMARY keys). > [...] I consider this as a bug, [...] Wrong. Bugs are issues of correctness. Performance improvements are not issues of correctness.

7 11

breadcrumb override problem
by Augustin (Beginner) 02 Jul '08

02 Jul '08

Hello, I'm upgrading and cleaning directory.module for Drupal 6. I am running on a breadcrumb problem. I think the Drupal core implementation of breadcrumbs is broken. First of all, there can only be one set of breadcrumbs. Imagine a blog post outlined in some book, with a few taxonomy terms. We potentially have three sets of breadcrumbs: the one set by each module involved (blog, book, taxononomy, though core taxonomy.module doesn't set the breadcrumbs on a node view.) To decide which breadcrumb will prevail, timing is everything: it will be the one set by the last call of drupal_set_breadcrumb() BEFORE the first call of drupal_get_breadcrumb(). I was trying to override the taxonomy.module's breadcrumbs on a taxonomy/term/nnn page, but: - if I call drupal_set_breadcrumb() from hook_help(), it's too late: the theming engine has already called drupal_get_breadcrumb() and my breadcrumbs are ignored. - if I call drupal_set_breadcrumb() from hook_init(), it's too early: taxonomy.module will override mine just before drupal_get_breadcrumb() gets called! So, now, I am a bit lost and don't know where I can best call drupal_set_breadcrumb() so that it comes after taxonomy's call, and before the first drupal_get_breadcrumb() call... Searching d.o, I've found many other issues in other crontrib modules which seem to deal with more or less the same problem, but I've not found a solution yet. So: 1) how should I solve my particular problem (the timing of drupal_set_breadcrumb() in my module to override/enhance taxonomy's breadcrumbs). 2) Do you agree that Drupal's core breadcrumbs handling could be improved, to make it possible to have several sets of breadcrumbs (why not? A good themer could accommodate them easily), or at least to make it easier to override / complement each other's breadcrumbs. If there's some kind of agreement on the second point, I'll open an issue for D7 with a summary of the discussion here. Blessings, Augustin. --

4 3

code review for the "pivots" project
by Daniel Xiaodan Zhou 02 Jul '08

02 Jul '08

Hi all, We are working on the "pivots" project as an attempt to give module recommendations. The idea is to display on a module page the related forum conversations and related modules that are referenced in the same conversations. We think it could provide some useful information for users to decide whether to download a module or how to use it. The "pivots" system has been enabled for D.O. site maintainers and CVS account holders. More details can be found here: http://drupal.org/node/265450 We are about to deliver it to all interested D.O. users. And we hope to get some code reviews from the community to improve code quality. The code is published to CVS: contributions/modules/pivots4do Or, for a quick browse, you might go to the following links. PHP: http://danithaca.pastebin.com/m798c291d http://danithaca.pastebin.com/m2ea9911a Java Indexer: http://danithaca.pastebin.com/m1b147b25 http://danithaca.pastebin.com/m53f9621b http://danithaca.pastebin.com/m592c46ec For a short description of the architecture and security concerns, please refer to: http://drupal.org/node/265450#comment-884264 Your comments would be much appreciated. Thanks! Cheers, Daniel

2 2

Re: [development] upload module or maybe url() bug
by David Timothy Strauss 01 Jul '08

01 Jul '08

I'd try an extra step to encode "+" characters in your URLs. I would support adding such a feature to Drupal's url() if it makes the encoding less ambiguous. ----- "Matt Connolly" <matt(a)cabinetuk.com> wrote: > The + in the file name is being translated into a space and the file > > download does not work because the file is not found. > > When I change the code, as I have, by including a url_encode function > > on the filename, it all works perfectly. > > My question really is: should this be specifically for the file > download - like how I have patched my drupal installation, or is it > better served in the url() function if this would also solve other > urls in the site (page names, etc) > > -Matt > > On 30/06/2008, at 9:16 PM, David Timothy Strauss wrote: > > > See this issue: > > http://drupal.org/node/191116 > > > > Translating spaces to "+" is not RFC-compliant. For maximum > > compatibility, you should encode both spaces and plus signs using % > > > notation. > > > > ----- "Matt Connolly" <matt(a)cabinetuk.com> wrote: > > > >> I have a question about url encoding > >> > >> Spaces in URLS are normally translated to "%20" and spaces in > queries > >> > >> are normally translated into "+", although the "%20" is correctly > >> decoded. > >> > >> I'm discovering that private file uploads are not working when > there > >> > >> is a "+" in the file name. This is being translated into a " > >> " (space) > >> > >> which is quite normal for queries. > >> > >> With public downloads, (ie accessibly directly via > >> http://mydrupalsite/sites/files/fred+wilma.jpg > >> " the file works. So apache doesn't translate the + in the file's > > >> path > >> > >> name into a space, and the file is downloaded. > >> > >> With private downloads, the file name is placed into a query, > thanks > >> > >> to mod_rewrite (or, becomes > >> http://mydrupalsite/q?=system/files/fred+wilma.jpg > >> " . You can see where the problem is. > >> > >> > >> I can easily fix this by altering the "file_create_url" function in > > >> "/ > >> > >> includes/file.inc". I know I've been warned to not change drupal's > >> core modules and code, but ..... I think I need to. > >> > >> However, I wonder if the problem would be better handled in the > >> "url()" function? > >> > >> I notice that if you create page url's "bananas+pears" and "apples > > >> and > >> > >> oranges" the urls are all encoded with %20. So it seems you can't > use > >> > >> the "+" in a custom URL. Butt least here what you type and what > you > >> get are consistent. > >> > >> > >> Any thoughts? > >> -Matt

2 2

Re: [development] upload module or maybe url() bug
by David Timothy Strauss 30 Jun '08

30 Jun '08

See this issue: http://drupal.org/node/191116 Translating spaces to "+" is not RFC-compliant. For maximum compatibility, you should encode both spaces and plus signs using % notation. ----- "Matt Connolly" <matt(a)cabinetuk.com> wrote: > I have a question about url encoding > > Spaces in URLS are normally translated to "%20" and spaces in queries > > are normally translated into "+", although the "%20" is correctly > decoded. > > I'm discovering that private file uploads are not working when there > > is a "+" in the file name. This is being translated into a " " (space) > > which is quite normal for queries. > > With public downloads, (ie accessibly directly via > http://mydrupalsite/sites/files/fred+wilma.jpg > " the file works. So apache doesn't translate the + in the file's path > > name into a space, and the file is downloaded. > > With private downloads, the file name is placed into a query, thanks > > to mod_rewrite (or, becomes > http://mydrupalsite/q?=system/files/fred+wilma.jpg > " . You can see where the problem is. > > > I can easily fix this by altering the "file_create_url" function in "/ > > includes/file.inc". I know I've been warned to not change drupal's > core modules and code, but ..... I think I need to. > > However, I wonder if the problem would be better handled in the > "url()" function? > > I notice that if you create page url's "bananas+pears" and "apples and > > oranges" the urls are all encoded with %20. So it seems you can't use > > the "+" in a custom URL. Butt least here what you type and what you > get are consistent. > > > Any thoughts? > -Matt

2 1

Code freeze?
by Caleb Gilbert 30 Jun '08

30 Jun '08

Indeed this does go to a lot of people. Please try not to feel compelled to try and censor things that someone else might feel compelled to share -- especially if they are particularly exited and upbeat about something the community is doing. Some people might call this sharing, "feedback", and see value in receiving it. Regards, - Caleb This list goes to a lot of people. Please try to make your posts more substantive than "I agree!" ----- "Caleb Gilbert" wrote: > Reading this thread has me very excited - what a great decision to extend the code freeze of Drupal 7 for a while! In particular, I enjoyed/share chx's comment that 'Drupal 5/6 is quite good enough to hang-a-reputation-on for some time to come'. Indeed they are. :-) > > - Caleb

1 0

Drupal Function Frequency Report?
by Marjorie Roswell 30 Jun '08

30 Jun '08

I'm on a drupal learning curve. It occurs to me that I'd love to see a report of all the functions used in Drupal by frequency. (Then, I'd tackle the learning curve by starting with what's used most, and moving down the list!) I imagine starting with something like the api pull from the Module Builder Module (http://drupal.org/project/module_builder) and then somehow scanning the CVS for all the functions, and calculating a frequency. Seriously, would love to basically tackle this learning curve in a logical way, based on how often something appears in code, ideally in both core and contrib, combned. I recognize that little-used code functions may still be quite important, but this approach would certainly serve as a good guide. I don't have the skillset to write this report. Is there anyone for whom producing this function frequency report would be a relative cinch? Margie

9 10

upload module or maybe url() bug
by Matt Connolly 30 Jun '08

30 Jun '08

I have a question about url encoding Spaces in URLS are normally translated to "%20" and spaces in queries are normally translated into "+", although the "%20" is correctly decoded. I'm discovering that private file uploads are not working when there is a "+" in the file name. This is being translated into a " " (space) which is quite normal for queries. With public downloads, (ie accessibly directly via http://mydrupalsite/sites/files/fred+wilma.jpg " the file works. So apache doesn't translate the + in the file's path name into a space, and the file is downloaded. With private downloads, the file name is placed into a query, thanks to mod_rewrite (or, becomes http://mydrupalsite/q?=system/files/fred+wilma.jpg " . You can see where the problem is. I can easily fix this by altering the "file_create_url" function in "/ includes/file.inc". I know I've been warned to not change drupal's core modules and code, but ..... I think I need to. However, I wonder if the problem would be better handled in the "url()" function? I notice that if you create page url's "bananas+pears" and "apples and oranges" the urls are all encoded with %20. So it seems you can't use the "+" in a custom URL. Butt least here what you type and what you get are consistent. Any thoughts? -Matt

2 1