[drupal-devel] Project management contribs: Bundles: grouped modules.
Hi, Starting with the thesis: "The way contributions are managed now is not suitable anymore. Esp. modules are growing out of hand, not only the amount of modules, but mostly the way modules conflic,t overlap and even completely duplicate one-another is a problem." We can only fix that with some effort and rearranging the way we handle contribs. I do not beleive we can solve this with code, or extra features on drupal.org. Allthough that might /help/, the real solution lies in management of this project. Note that I am /not/ (yet) addressing the part where users cannot find approporate modules for downloading: frankly I don't (yet) care about these users: they should just look better. Once we manage the contribs better, we can look into that. If it is still neccesary then. I see a couple of solutions, but I will only deal with one, here: bundles: definition: A bundle is a group of modules that have features in one area. A bundle is not a subcalssification of single modules, but a single entity. A bundle has only one project entry on drupal.org and it lives in one directory (+subdirs) in CVS. It is therefore downloadable as a complete tarball only. Modules in a bundle should be: * aware of oneanother. If A.module is anabled, B.module can hook into that, or use additional features, in addition to the features A.module already delivers. * reusing APIs. No duplication of code, re-use of optional .inc files. * never conflicting. enabling module A should never mean module B starts spitting errors. Module B should know that it must "shut off" certain parts, when module A is enabled. * Reusing databse tables. Currently most modules have private tables, tables that even often confilct with others. Weblinks, for example, should be stored in a single (set of) table(s), where all weblinks-realted modules (in the weblink bundle) will store and get their info. It makes no sense if relatedlinks.module stores weblinks in tableA, weblinks.module in tableB, blogmarks in tableC, flexinodes url.inc in tableD etcetc. The e-commerce modules are a good example of this. For 4.6 we hope to have weblinks into a bundle too, this would Project management-wise a bundle would imply collaboration of a few developers. These developers must communicate amoungs eachother, they should use a separate directory in the sandbox CVS area as well as devide issues and features amoungst eachother. Pro: modules within same range of features can be managed in a central place. Pro: modules within same range of features can be found in one central place PRo: modules within same range of features will work together. Together they make a single and *complete* set of functions. separate managed modules (what we have now) often have a lot of simil;ar features. Take for example amazon_serach.module and amazon_items.module both offer some way to add amazon ads to your site, both allow some amazon content to be published on your site, but features differ slightly. Also, they use some moduified 3rd party .incs, which are not shared, buyboth modules. Ideally we would split them up and have: * one module that handles all ad-serving from amazon. * one module that handles content serving. so thogether they make a complete system, instead of having to conflicting modules. Con: Contributions of custommade modules is impossible/hard. If a module was developed for yoursite.com only, it will most likely not fit within one of the bundles. Question of course is, if we should host these modules at all. Con: Second-level discussions need to take place. So some small mailinglists, or even simply a list of CC-ed mailadresses are needed for each bundle. I beleive this is no real problem as yet, in future some module on Drupal.org could take care of this. Con: It is hard to manage. esp because of the open nature of the contribs. WE cannot dissallow people to contribute their custommade.module ATM, and we will not be able to implement such a thing soon. Thus the bundle systems would require common sense and good agreemenents between contributors. Con: a user will have to download a complete bundle (lets assumne weblink bundle) even if she only wants to have relatedlinks.module (which is part of the weblink bundle). So, please tell me what you think of this idea, and if it is worth discussion on drupalCON, or even on the lists only. Regards, Bèr -- [ Bèr Kessels | Drupal services www.webschuur.com ]
Hi Bèr, I like your intentions to improve the way things work with modules, I also like the idea of well-integrated 'bundles', but I also think that the ideas need more development/discussion before we will have an appropriate solution to the problem (out-of-hand contrib modules). A couple of points: - I appreciate the number and diversity of modules (even ones that are quite similar). - Being able to share 'special case' modules is a valuable dynamic of the Drupal community (even if they are not all compatible). What I think would be good would be the creation of two classes of modules: 1. 'Official' modules - these would be first class modules that provide popular functionality, be well maintained, use standard ways of doing things, and not conflict with any other 'official' module. 2. 'Contrib' modules - various and eclectic modules that do provide some useful functionality, work well with the core and also all of the 'official' modules, but are not required to work together with all of the other 'contrib' modules. This would combine the best of both worlds - *Diversity*, and *Quality* I think the core should stay 'lean and mean', then site admins (or 'distribution' providers) can customise the functionality of their site by selecting from trusted 'official' modules. Those who have particular special needs, can choose to develop their own modules, or make use of the shared 'Contrib' modules. As I said before, I think bundles could be a good idea (if managed well), but I personally don't think it will solve the problem. Other ideas to think about: - module rating system - module popularity system - dependency/conflict management (package management style) Cheers, Ross (Disclaimer: ideas offered solely for the purpose of discussion) Bèr Kessels wrote:
Hi,
Starting with the thesis: "The way contributions are managed now is not suitable anymore. Esp. modules are growing out of hand, not only the amount of modules, but mostly the way modules conflic,t overlap and even completely duplicate one-another is a problem."
We can only fix that with some effort and rearranging the way we handle contribs. I do not beleive we can solve this with code, or extra features on drupal.org. Allthough that might /help/, the real solution lies in management of this project. Note that I am /not/ (yet) addressing the part where users cannot find approporate modules for downloading: frankly I don't (yet) care about these users: they should just look better. Once we manage the contribs better, we can look into that. If it is still neccesary then.
I see a couple of solutions, but I will only deal with one, here: bundles:
definition: A bundle is a group of modules that have features in one area. A bundle is not a subcalssification of single modules, but a single entity. A bundle has only one project entry on drupal.org and it lives in one directory (+subdirs) in CVS. It is therefore downloadable as a complete tarball only. Modules in a bundle should be: * aware of oneanother. If A.module is anabled, B.module can hook into that, or use additional features, in addition to the features A.module already delivers. * reusing APIs. No duplication of code, re-use of optional .inc files. * never conflicting. enabling module A should never mean module B starts spitting errors. Module B should know that it must "shut off" certain parts, when module A is enabled. * Reusing databse tables. Currently most modules have private tables, tables that even often confilct with others. Weblinks, for example, should be stored in a single (set of) table(s), where all weblinks-realted modules (in the weblink bundle) will store and get their info. It makes no sense if relatedlinks.module stores weblinks in tableA, weblinks.module in tableB, blogmarks in tableC, flexinodes url.inc in tableD etcetc.
The e-commerce modules are a good example of this. For 4.6 we hope to have weblinks into a bundle too, this would
Project management-wise a bundle would imply collaboration of a few developers. These developers must communicate amoungs eachother, they should use a separate directory in the sandbox CVS area as well as devide issues and features amoungst eachother.
Pro: modules within same range of features can be managed in a central place. Pro: modules within same range of features can be found in one central place PRo: modules within same range of features will work together. Together they make a single and *complete* set of functions. separate managed modules (what we have now) often have a lot of simil;ar features. Take for example amazon_serach.module and amazon_items.module both offer some way to add amazon ads to your site, both allow some amazon content to be published on your site, but features differ slightly. Also, they use some moduified 3rd party .incs, which are not shared, buyboth modules. Ideally we would split them up and have: * one module that handles all ad-serving from amazon. * one module that handles content serving. so thogether they make a complete system, instead of having to conflicting modules.
Con: Contributions of custommade modules is impossible/hard. If a module was developed for yoursite.com only, it will most likely not fit within one of the bundles. Question of course is, if we should host these modules at all. Con: Second-level discussions need to take place. So some small mailinglists, or even simply a list of CC-ed mailadresses are needed for each bundle. I beleive this is no real problem as yet, in future some module on Drupal.org could take care of this. Con: It is hard to manage. esp because of the open nature of the contribs. WE cannot dissallow people to contribute their custommade.module ATM, and we will not be able to implement such a thing soon. Thus the bundle systems would require common sense and good agreemenents between contributors. Con: a user will have to download a complete bundle (lets assumne weblink bundle) even if she only wants to have relatedlinks.module (which is part of the weblink bundle).
So, please tell me what you think of this idea, and if it is worth discussion on drupalCON, or even on the lists only.
Regards, Bèr
Op vrijdag 18 februari 2005 15:56, schreef Ross Kendall: > Hi Bèr, > > I like your intentions to improve the way things work with modules, I > also like the idea of well-integrated 'bundles', but I also think that > the ideas need more development/discussion before we will have an > appropriate solution to the problem (out-of-hand contrib modules). > > A couple of points: > - I appreciate the number and diversity of modules (even ones that are > quite similar). That diversity is nice. But ATM that diversiti leads to a large diversity of not-finshed and not-perfect modules. Why cant we have one good set, instead of four unfinished ones? Answer is easy: because after scratching your itch, you are done, and the module is "dumped" in contribs. But other will have other itches. I am suggesting we stop scratching, and invent a "jar of crême" (i.e implement a management system) that will solve the itches too. Because scratching might help for that moment against the itch, that itch itself wont be solved. So: I dislike the model of diversity a lot. I know it is a common OSS problem. But without clear structures and management a lot of work is duplicated, even more work/time is wasted etc. > - Being able to share 'special case' modules is a valuable dynamic of > the Drupal community (even if they are not all compatible). Yes true, but does that need to be on drupal.org? maybe, but in a sandbox or separate area, it should not be part of official downloads Will special cases be maintained? most prolly No ,because the itch is scratched. How many special cases actually ever make it into maintained modules? Harddly any, nearly all the spacial cases are old, broken and must be removed after a new release. Its those nen-special case modules that survive. > What I think would be good would be the creation of two classes of modules: > 1. 'Official' modules > 2. 'Contrib' modules Can be combined perfectly. > This would combine the best of both worlds - *Diversity*, and *Quality* > > I think the core should stay 'lean and mean', then site admins (or > 'distribution' providers) can customise the functionality of their site > by selecting from trusted 'official' modules. Those who have particular > special needs, can choose to develop their own modules, or make use of > the shared 'Contrib' modules. We should keep core out of this. This regards only contribs. Lets not aim too high. We will deal with core later, if ever. > As I said before, I think bundles could be a good idea (if managed > well), but I personally don't think it will solve the problem. > > Other ideas to think about: > - module rating system -1. This will really not solve: * maintainability. * duplication of effort, APIs, code and features. * amoutn of modules contributed. It will *only* show what modules are more popular, /if/ it will show us anything at all. How often do you base your choise of a module on popularity. I never did, and never will. Based on quality, yes. > - module popularity system See above: -1, since It does not solve the issues we are discussing here. > - dependency/conflict management (package management style) Yes, but that is aiming very high. I do not beleive this will get in soon, if ever at all. Regards, Bèr -- [ Bèr Kessels | Drupal services www.webschuur.com ]
Hi Bèr, I think you slightly misunderstood my suggestions. Basically my ideas was just the same as Károly. 1. well controlled set of modules, listed for download with Drupal. 2. 'wild' modules, that are listed elsewhere. I just feel that if you try to control /everything/ to much, it stifles creativity, and people loose interest. So the answer, control a set, and let the rest free. Like I said, the best of /both/ worlds. It is easy to criticise all the half-finished modules, however this is where the ideas get worked out, and they end up being the basis for what /is/ kept and made 'standard'. more comments in message below... Bèr Kessels wrote:
That diversity is nice. But ATM that diversiti leads to a large diversity of not-finshed and not-perfect modules. Why cant we have one good set, instead of four unfinished ones? Answer is easy: because after scratching your itch, you are done, and the module is "dumped" in contribs. But other will have other itches. I am suggesting we stop scratching, and invent a "jar of crême" (i.e implement a management system) that will solve the itches too. Because scratching might help for that moment against the itch, that itch itself wont be solved.
So: I dislike the model of diversity a lot. I know it is a common OSS problem. But without clear structures and management a lot of work is duplicated, even more work/time is wasted etc.
I think is possible to have /both/ the diversity /and/ 'one good set', why not?
- Being able to share 'special case' modules is a valuable dynamic of the Drupal community (even if they are not all compatible).
Yes true, but does that need to be on drupal.org? maybe, but in a sandbox or separate area, it should not be part of official downloads
What I meant by my suggestion /is/ to put them in a different place. So I agree.
Will special cases be maintained? most prolly No ,because the itch is scratched. How many special cases actually ever make it into maintained modules? Harddly any, nearly all the spacial cases are old, broken and must be removed after a new release. Its those nen-special case modules that survive.
This is normal, not a problem. They do help start the process of development towards a more general solution. Even if they do not last forever, they do solve an immediate problem for some people. When you have a deadline to develop a website, there is no time to develop such elegant general solutions. Sometimes a 'quick fix' is the right way.
What I think would be good would be the creation of two classes of modules: 1. 'Official' modules 2. 'Contrib' modules
Can be combined perfectly.
Not sure what you mean? I was trying to say that I think it's a good idea for them to be separate.
This would combine the best of both worlds - *Diversity*, and *Quality*
I think the core should stay 'lean and mean', then site admins (or 'distribution' providers) can customise the functionality of their site by selecting from trusted 'official' modules. Those who have particular special needs, can choose to develop their own modules, or make use of the shared 'Contrib' modules.
We should keep core out of this. This regards only contribs. Lets not aim too high. We will deal with core later, if ever.
I was not suggesting anything should be different with the core. I like the way the core is.
As I said before, I think bundles could be a good idea (if managed well), but I personally don't think it will solve the problem.
Other ideas to think about: - module rating system
-1. This will really not solve: * maintainability. * duplication of effort, APIs, code and features. * amoutn of modules contributed. It will *only* show what modules are more popular, /if/ it will show us anything at all. How often do you base your choise of a module on popularity. I never did, and never will. Based on quality, yes.
Just for the non-official contrib modules, not for the high quality integrated modules. You can't tell me that you do not use the ratings on sourceforge.net to help you find software.
- module popularity system
See above: -1, since It does not solve the issues we are discussing here.
As above, I think it's a good idea. But as you say, it's a separate thing to worry about.
- dependency/conflict management (package management style)
Yes, but that is aiming very high. I do not beleive this will get in soon, if ever at all.
I just gave the last 3 ideas as future ideas, to help with thinking differently about the approach.
Regards, Bèr
Regards, Ross
Okay let's not loose focus. A summary: We agree on the poitn that we need: * a closed contribs section (official) * an open contribs section (sanbox) We disagree (or so it seems) on how to mamange the closed contribs. * I say we should bundle them by functionality, * You are saying we need some rating for them. * WE all say we need to pass modules trough a team of contrib maintainers before they are allowed in the closed sesction. * We should not forget that we actually have such a closed & open system, but too little people actually use it: All specific and unfinshed code *must* be in /sandbox/yourname/ Only releasable modules should be in contribs. Only finshed modules should be taggted for a Drupal release. PPl. seem to disagree a lot about when it the last two points are applicable. So: We do not disagree, But I am saying that we should go a bit further than only closed contribs: Thats where I am talking abou "combining". I meant to "combine" the bundles and close-open contribs ideas. Bèr
Okay let's not loose focus. A summary: We agree on the poitn that we need: * a closed contribs section (official) * an open contribs section (sanbox)
We disagree (or so it seems) on how to mamange the closed contribs. * I say we should bundle them by functionality, * You are saying we need some rating for them.
* WE all say we need to pass modules trough a team of contrib maintainers before they are allowed in the closed sesction.
* We should not forget that we actually have such a closed & open system, but too little people actually use it: All specific and unfinshed code *must* be in /sandbox/yourname/ Only releasable modules should be in contribs. Only finshed modules should be taggted for a Drupal release. PPl. seem to disagree a lot about when it the last two points are applicable.
Well, sometime last year there was a policy change of pushing development stage modules into /modules, and not into the sandboxes, since a lot of interesting stuff was buried in sandboxes back then. Since CVS allows people to tag their modules, and modules should only have a stable (non-HEAD) tag if they are useable, this policy change seemed to be a good idea. However this did not solve the functionality duplication problem you mention with the bundles...
So: We do not disagree, But I am saying that we should go a bit further than only closed contribs: Thats where I am talking abou "combining". I meant to "combine" the bundles and close-open contribs ideas.
What about cross-dependant modules? Like a cssvault type site, which links images to outside sites. It would need the image bundle (more image management then what core offers), and the link management (like broken link checking, etc). Where would such a module live? Goba
Getting closer to an understanding :) Bèr Kessels wrote:
Okay let's not loose focus. A summary: We agree on the poitn that we need: * a closed contribs section (official) * an open contribs section (sanbox)
Yes
We disagree (or so it seems) on how to mamange the closed contribs. * I say we should bundle them by functionality, * You are saying we need some rating for them.
Actually, no. I *like* your idea of bundles for closed contribs. The ratings I was thinking for the other modules (because there is many, and not always good quality). If there is a selection of good quality modules that all work together, there is no need for rating system, just good descriptions of what they do, and being well organised.
* WE all say we need to pass modules trough a team of contrib maintainers before they are allowed in the closed sesction.
Yes, good idea
* We should not forget that we actually have such a closed & open system, but too little people actually use it: All specific and unfinshed code *must* be in /sandbox/yourname/
One thing, sandbox is not very visible, maybe there could be another place that contributors can publish working modules, even if they are not 'approved'.
Only releasable modules should be in contribs. Only finshed modules should be taggted for a Drupal release. PPl. seem to disagree a lot about when it the last two points are applicable.
I think more control at the top level is needed, how it should work - I'm not so sure what's best - maybe other people will give some good ideas.
So: We do not disagree, But I am saying that we should go a bit further than only closed contribs: Thats where I am talking abou "combining". I meant to "combine" the bundles and close-open contribs ideas.
Yes, could be good. Apart from amazon and image, what else should be grouped, and how?
Bèr
Ross.
OK, just chiming in on this. I don't know if my perspective is unique among contrib developers, but my time spent on drupal modules is sporadic. I try, whenever possible, to develop useful modules as part of whatever projects I'm working on. Sometimes, they're really only useful to me, so I never even put them into CVS. I have half a dozen modules that I've reused personally, but couldn't ever see anyone else finding useful.
Okay let's not loose focus. A summary: We agree on the poitn that we need: * a closed contribs section (official) * an open contribs section (sanbox) I like this idea. I, too, like both quality and diversity and would want to see everything possible if I felt like looking for it.
We disagree (or so it seems) on how to mamange the closed contribs. * I say we should bundle them by functionality, * You are saying we need some rating for them. Perhaps just an overall status assessment for each module. If a module's been out for 1 week, then I wouldn't consider it stable, even if there are no current issues with it. I think it would be easy enough to provide a few metrics, like "age", "# of critical bugs", "ease of installation", "ease of operation" or something like that. As an example, I couldn't imagine a moblog module that didn't involve tons of outside configuration and dependent modules.
* WE all say we need to pass modules trough a team of contrib maintainers before they are allowed in the closed sesction. If there were such a team of contrib mantainers, people like myself might feel more comfortable about contributing modules. Whereas I might have enough time to properly develop a module, there's a good chance that I won't be around to fix bugs for a while. A team of contrib maintainers would keep decent modules from falling into deprecacy.
* We should not forget that we actually have such a closed & open system, but too little people actually use it: All specific and unfinshed code *must* be in /sandbox/yourname/ Only releasable modules should be in contribs. Only finshed modules should be taggted for a Drupal release. PPl. seem to disagree a lot about when it the last two points are applicable. I agree that is is misleading to have unfinished modules sitting beside working ones. I think if the sandbox were configured such that directories could be browsed and documented (and downloaded as tgz files) similar to the way the current releases are, people would be more apt to keep their unfinished code in there.
So: We do not disagree, But I am saying that we should go a bit further than only closed contribs: Thats where I am talking abou "combining". I meant to "combine" the bundles and close-open contribs ideas. I'm a big fan of bundles, but only for the purposes of easing installation and module selection. I would want the bundles to exist in addition to all of the other modules/sandboxes, not in place of them. One of the main reasons for the sandbox, IMO, is to put code that's not complete or requires patching of other code to operate properly. This code is no less valuable, though. Mark
Op vrijdag 18 februari 2005 22:46, schreef Mark Howell:
If there were such a team of contrib mantainers, people like myself might feel more comfortable about contributing modules. Whereas I might have enough time to properly develop a module, there's a good chance that I won't be around to fix bugs for a while. A team of contrib maintainers would keep decent modules from falling into deprecacy.
I do not think we currently have the resources to do this. A team maintaining contribs woiuld need to consist of at least 30 Developers @ 4 hours a week. At least. No, I think we should not talk about contrib *maintainers*, but more as a contrib door-guards. Those deciding what goes in, what not. Those keeping the heads together on what is happening, but mostly those that know who is doing what in these "official" contribs. Now, this might sound awfully "official", even burocratic, but its not. Think of these people as a team, ~15 developers with write CVS access. the way bugs, patches and featurews et al all are handled should be the exact same way as we do it with core: queues etc. Only that instead of having only three people with write access, we eill have five-fold, simply because a) there is more code in that repos. b) contrib "maintainers" will/might have less time to spend on these issues. c) have more toe filter out: from what I can see now, there are much more issues for contribs then for core. Regards, Bèr -- [ Bèr Kessels | Drupal services www.webschuur.com ]
No, I think we should not talk about contrib *maintainers*, but more as a contrib door-guards. Those deciding what goes in, what not. Those keeping the heads together on what is happening, but mostly those that know who is doing what in these "official" contribs.
it is quite possible that we don't have enough qualified developers with enough time on their hands to perform code checking and functionality checking for this. If you look at the patch queue, it is quite common for *noone* to review a given patch, and we just let dries do it. i would think that Contrib code would be even less worthy of review. I'm not opposed to the 'door guard' arrangement, i am just skeptical that people will do what they promised. the beauty of a rating system is that the whole community decides merit, and we don't need dedicated roles. it is true that ratings doesn't address the full range of issues that we want to solve here though.
On Fri, 18 Feb 2005, Moshe Weitzman wrote:
No, I think we should not talk about contrib *maintainers*, but more as a contrib door-guards. Those deciding what goes in, what not. Those keeping the heads together on what is happening, but mostly those that know who is doing what in these "official" contribs.
it is quite possible that we don't have enough qualified developers with enough time on their hands to perform code checking and functionality checking for this. If you look at the patch queue, it is quite common for *noone* to review a given patch, and we just let dries do it. i would think that Contrib code would be even less worthy of review. I'm not opposed to the 'door guard' arrangement, i am just skeptical that people will do what they promised.
I share your concern. We should, however, at least check sql queries for insertion vulnerabilities. Having Drupal modules which cause major havoc is not acceptable. I would also not split up the cvs repositories. Moving projects around from sandbox to contrib (and maybe back) will break the cvs history. I prefer to handle this through the website by de-moderating project modules.
the beauty of a rating system is that the whole community decides merit, and we don't need dedicated roles. it is true that ratings doesn't address the full range of issues that we want to solve here though.
I like this approach too. We should try it and see what happens. Cheers, Gerhard
On Sat, 19 Feb 2005, Gerhard Killesreiter wrote:
On Fri, 18 Feb 2005, Moshe Weitzman wrote:
it is quite possible that we don't have enough qualified developers with enough time on their hands to perform code checking and functionality checking for this. If you look at the patch queue, it is quite common for *noone* to review a given patch, and we just let dries do it. i would think that Contrib code would be even less worthy of review. I'm not opposed to the 'door guard' arrangement, i am just skeptical that people will do what they promised.
I share your concern. We should, however, at least check sql queries for insertion vulnerabilities. Having Drupal modules which cause major havoc is not acceptable.
We should also enforce use of our coding standards for contrib CVS. It is much easier to look at all db queries if you can just grep for them like this: egrep -r "SELECT|INSERT|UPDATE" */*.module */*.inc|less I currently get 1727 matches which might not be all of them. $result = db_query('SELECT nid, uid, created, body is part of the output from stats_plot/stats_plot_user_activity.inc which makes judging the query difficult. Cheers, Gerhard
Op zaterdag 19 februari 2005 05:54, schreef Moshe Weitzman:
No, I think we should not talk about contrib *maintainers*, but more as a contrib door-guards. Those deciding what goes in, what not. Those keeping the heads together on what is happening, but mostly those that know who is doing what in these "official" contribs.
it is quite possible that we don't have enough qualified developers with enough time on their hands to perform code checking and functionality checking for this. If you look at the patch queue, it is quite common for *noone* to review a given patch, and we just let dries do it. i would think that Contrib code would be even less worthy of review. I'm not opposed to the 'door guard' arrangement, i am just skeptical that people will do what they promised.
I share your concern. Thats why I aimed high: 15 -20 developers on this. Note that I already addressed the issue of developers being buisy.
the beauty of a rating system is that the whole community decides merit, and we don't need dedicated roles. it is true that ratings doesn't address the full range of issues that we want to solve here though.
Again: No-one is against a rating system here, it seems. But I and Károly were bot addresssing the same point, which cannot be solved wit rating. I suggest we either split focus here, or we make this into a huge-will-never-happen-issue: We have a security issue wiith CVS commits We have a problem with developers duplicating fatr too many code and making incompatible, somteimes even plain crappy modules. We have a problem with people not being able to find modules. Again: I cant be #rsed about the third one, in fact I dont beleive that is the case at all: If you cannot even take an hour to find the correct modules for your site: then you are prolly not wrth them ;). No, serious: This /is/ another issue, that should not be integrated into this project, for then we will really never get to a conclusion. The same goes for Dries's answer about firefox's extention site. That is fine, the idea is really great, and the amount of extensions/modules are very similar. But really: we should not start including that discussion here. Regards, Bèr -- [ Bèr Kessels | Drupal services www.webschuur.com ]
We have a problem with developers duplicating fatr too many code and making incompatible, somteimes even plain crappy modules.
Ber, you haven't convinced me that this is a problem that Dries' proposal wouldn't solve. Plain crappy modules would be easy to ignore because they wouldn't have high ratings and wouldn't bubble to the top of lists. The presence fo the code doesn't hurt anyone. In fact, as mentioned several times, it can have positive side-effects that are hard to predict. I don't think that you can ever solve the 'problem' of developers wanting to roll their own module. Sure there is duplicated effort, but organizing CVS differently or making fences and rules around the contrib repository won't change the basic urge that people have to start from scratch. -R
Bèr Kessels wrote:
We have a security issue wiith CVS commits We have a problem with developers duplicating fatr too many code and making incompatible, somteimes even plain crappy modules.
What do you think of the idea of having another CVS repository for high-quality integrated modules and 'bundles', with similar check-in rules to drupal core. * drupal * drupal-modules (maintained, secure, integrated, quality, non-core modules go here) * contributions (anyone using stuff here does at there own risk, some crap, some gold)
A few notes: Noone is expected to continously review even the "closed" CVS repository commits (if one such is created).. I agree that this would be a huge task. I thought that we restrict who may have write access here and also review the project starts. Another issue is the _current_ state. First, those modules that neither work with 4.5 nor with the forthcoming 4.6 surely does not worth a second look. Second, we check correct SQL usage and coding style. If this is OK, and there are two or more modules with colliding functionality we contact the authors asking them to merge. This raises the question that some nice functionality does not get moved to the closed contribs 'cos of some lacking whitespace. Well, we can send a note to the maintainer that his (I was thinking on a PC "her" here, but how many female drupal developers we have? never mind.) module failed the code style check. This is urgent. Please note that Drupal is spreading fast, and next weekend we are going to FOSDEM with IMHO the very intent to make it spread even more. Imagine inviting guests to your house where the actual house is hip, nice and clean but the back yard is ... you get the picture. Of course, we won't be able to clean up that back yard in a week and, after all, we are telling guests that the back yard is not maintained :) Regards Karoly Negyesi
Negyesi Karoly wrote:
Another issue is the _current_ state. First, those modules that neither work with 4.5 nor with the forthcoming 4.6 surely does not worth a second look.
This may not be as true in all senses as one might think. Allie Micka and I just updated the ezmlm module from 4.2 to 4.5 and HEAD, and will be maintaining and enhancing it. There may be other other modules which are not 4.5 current, but which someone may want to update because they already contain most of the code they need for a module they want. I agree that non-4.5 modules would not be worth looking at if creating a "reviewed" or otherwise endorsed repository. This is not really an argument against Karoly's posting. Sometimes old, non-working code is good for something, so I would say it's worth not just deleting things too quickly. -- Chris Johnson
I agree that non-4.5 modules would not be worth looking at if creating a "reviewed" or otherwise endorsed repository. This is not really an argument against Karoly's posting. Sometimes old, non-working code is good for something, so I would say it's worth not just deleting things too quickly.
There will be *NO* deletes. That's for sure. My proposal is that there should be a more closed contrib and something like to current. I said that pre-4.5 codes shall not get into the closed one. NK
On Sat, 19 Feb 2005, Bèr Kessels wrote:
Op zaterdag 19 februari 2005 05:54, schreef Moshe Weitzman:
skeptical that people will do what they promised.
I share your concern. Thats why I aimed high: 15 -20 developers on this. Note that I already addressed the issue of developers being buisy.
From which tree do you want to shake 15-20 developers? I guess it depends a bit on who you accept as being a developer, but still.
Cheers, Gerhard
On Sat, 19 Feb 2005 11:33:38 +0100, Bèr Kessels <berdrupal@tiscali.be> wrote:
We have a security issue wiith CVS commits We have a problem with developers duplicating fatr too many code and making incompatible, somteimes even plain crappy modules. We have a problem with people not being able to find modules.
Like some I don't think that having people submit redundant or even buggy modules is a problem. I think it is harmful to the project to limit 'playing around with the code'. However, thing that I think we all agree is a problem is that it is difficult to easily determine which modules are quality tested and which are experimental. I think a rating system, along with some other optional measures, could solve this problem. Furthermore, an automated method is more likely to succeed than a manual and bureaucratic one if it is implemented thoughtfully. In order for a rating system to be meaningful the ratings must be trustworthy as has been suggested. There are different ways to ensure that the ratings are trustworthy. It has been suggested to give each reviewer "reputation points" but I'm not convinced that it is possible to do this fairly. Rather, I like the idea of making module reviews moderatable. Also, reviewers should be able to comment on the module in addition to giving it a rating. Another optional that could be used, since we have the advantage of bug tracking for each module, is to use the number of bugs as an additional metric. This is another way of automating the rating system and create a competitive environment for module developers. If they want their module to show up at the top or have a good rating then they'll pay attention to bugs and reviewer comments. In conclusion, -1 for suggestions that promote bureaucratic process and user intervention and +1 for tools that automate. As far as bundles, I see this as a nice option for some groups of modules but not if it means losing the individual modules. Building modules which "require" other modules seems to go against the principle of Drupal (any module can be added but is not required) and should probably be avoided when possible. For this reason, having things like image handling in the core is a great bonus.
Moshe Weitzman wrote:
it is quite possible that we don't have enough qualified developers with enough time on their hands to perform code checking and functionality checking for this. If you look at the patch queue, it is quite common for *noone* to review a given patch, and we just let dries do it. i would think that Contrib code would be even less worthy of review. I'm not opposed to the 'door guard' arrangement, i am just skeptical that people will do what they promised.
Right on. I'm a True Believer of automation and self-organization (if possible). If we'd have a 'code check' script -- even an extremely basic one -- I could install it on drupal.org and run the script once or twice a day on _each_ project in the contributions repository and link the report from the project pages. For example, on the project page of the image module (http://drupal.org/project/image), there would be a link called 'View code report' that points to the image module's code report, generated by the script. Of course, it is nothing like a real review and it might generate false positives but it could easily catch dozens of issues and encourage good coding style without manual intervention and within a few hours. After all, it is easy enough to detect tabs, '} else' or the absence of '{}' in SQL queries. -- Dries Buytaert :: http://www.buytaert.net/
Dries Buytaert wrote:
Right on. I'm a True Believer of automation and self-organization (if possible).
If we'd have a 'code check' script -- even an extremely basic one -- I could install it on drupal.org and run the script once or twice a day on _each_ project in the contributions repository and link the report from the project pages.
For example, on the project page of the image module (http://drupal.org/project/image), there would be a link called 'View code report' that points to the image module's code report, generated by the script.
Of course, it is nothing like a real review and it might generate false positives but it could easily catch dozens of issues and encourage good coding style without manual intervention and within a few hours. After all, it is easy enough to detect tabs, '} else' or the absence of '{}' in SQL queries.
I slapped together the foundations of a code checker scripts and installed it on drupal.org. It took me 50 minutes to write and install. It catched hundreds of (small) issues already, and will continue to catch many more in future to it most certainly pays off. The code: --------- The code lives at: http://cvs.drupal.org/viewcvs/drupal/contributions/tricks/code-checker/ Currently, it checks for '} else', SQL queries that lack '{}' (prefix), and CamelCase function names. Drupal.org: ----------- At the bottom of the project pages, there is a new link called 'View code report'. For example, take a look at http://drupal.org/project/image. The image module's report looks like: http://drupal.org/files/projects/image.status. Note that the report is generated against the HEAD version of each project and is updated a few times a day. What is next? ------------- Of course, we'll want to integrate it better and make it visually more appealing. Also, we'll want to add many more tests. I won't be able to work on it until after DrupalCon but you're all invited to add more tests. I'd be happy to install newer versions on drupal.org as soon they hit CVS. So, who's first to add a new test? -- Dries Buytaert :: http://www.buytaert.net/
Op zaterdag 19 februari 2005 13:15, schreef Dries Buytaert:
Of course, we'll want to integrate it better and make it visually more appealing.
I spent 20 mins on a mockup to spice up the project pages. See it here: http://drupal.org/node/17573 Regards, Bèr -- [ Bèr Kessels | Drupal services www.webschuur.com ]
Nice work. Likely we could adapt some approaches from, e.g., the PEAR PHP_Beautifier package, http://pear.php.net/package/PHP_Beautifier. For actually fixing up non-compliant code, contributors could use PHP_Beautifier directly, if a noble volunteer would write a new "filter" for Drupal coding styles, modelled on, e.g., the existing PEAR filter (see the file Beatutifier/Filter/PEAR.filter.php in the distribution). Nedjo ----- Original Message ----- From: "Dries Buytaert" <dries@buytaert.net> To: <drupal-devel@drupal.org> Sent: Saturday, February 19, 2005 4:15 AM Subject: Re: [drupal-devel] Checking coding style of contributed modules
Dries Buytaert wrote:
Right on. I'm a True Believer of automation and self-organization (if possible).
If we'd have a 'code check' script -- even an extremely basic one -- I could install it on drupal.org and run the script once or twice a day on _each_ project in the contributions repository and link the report from the project pages.
For example, on the project page of the image module (http://drupal.org/project/image), there would be a link called 'View code report' that points to the image module's code report, generated by the script.
Of course, it is nothing like a real review and it might generate false positives but it could easily catch dozens of issues and encourage good coding style without manual intervention and within a few hours. After all, it is easy enough to detect tabs, '} else' or the absence of '{}' in SQL queries.
I slapped together the foundations of a code checker scripts and installed it on drupal.org. It took me 50 minutes to write and install. It catched hundreds of (small) issues already, and will continue to catch many more in future to it most certainly pays off.
The code: ---------
The code lives at: http://cvs.drupal.org/viewcvs/drupal/contributions/tricks/code-checker/
Currently, it checks for '} else', SQL queries that lack '{}' (prefix), and CamelCase function names.
Drupal.org: -----------
At the bottom of the project pages, there is a new link called 'View code report'. For example, take a look at http://drupal.org/project/image. The image module's report looks like: http://drupal.org/files/projects/image.status.
Note that the report is generated against the HEAD version of each project and is updated a few times a day.
What is next? -------------
Of course, we'll want to integrate it better and make it visually more appealing. Also, we'll want to add many more tests.
I won't be able to work on it until after DrupalCon but you're all invited to add more tests. I'd be happy to install newer versions on drupal.org as soon they hit CVS.
So, who's first to add a new test?
-- Dries Buytaert :: http://www.buytaert.net/
Nedjo Rogers said:
Nice work. Likely we could adapt some approaches from, e.g., the PEAR PHP_Beautifier package, http://pear.php.net/package/PHP_Beautifier. For actually fixing up non-compliant code, contributors could use PHP_Beautifier directly, if a noble volunteer would write a new "filter" for Drupal coding styles, modelled on, e.g., the existing PEAR filter (see the file Beatutifier/Filter/PEAR.filter.php in the distribution).
It would probably save a bit of work if all patches were run through the same kind of filter as Dries wrote. It could check for the same code style problems and also check if the patch was made using the correct diff parameters. -- Tim Altman
On Sat, 19 Feb 2005, Dries Buytaert wrote:
I slapped together the foundations of a code checker scripts and installed it on drupal.org. It took me 50 minutes to write and install. It catched hundreds of (small) issues already, and will continue to catch many more in future to it most certainly pays off.
talk == silver code == gold. :)
What is next? -------------
Of course, we'll want to integrate it better and make it visually more appealing. Also, we'll want to add many more tests.
I won't be able to work on it until after DrupalCon but you're all invited to add more tests. I'd be happy to install newer versions on drupal.org as soon they hit CVS.
So, who's first to add a new test?
I am. I added a test to check the integrity of database queries. It won't catch sql errors, but it will barf at variables in the middle of a query which could potentially be harmfull. Of course something like db_query('INSERT INTO {users} ('. implode(', ', $fields) .') VALUES ('. implode(', ', $s) .')', $values); _is_ secure, but we should try to avoid such stuff nevertheless. Because of the ambiguity the error message is quite soft: - The INSERT query at line 164 is potentially insecure as it does not use our database query wrapper properly. Cheers, Gerhard
Op zaterdag 19 februari 2005 22:53, schreef Gerhard Killesreiter:
talk == silver code == gold. :)
This statement is IMO the mayor cause for all these usability issues in OSS. But thats out of scope and offtopic. Just to say that I think OSS has hsome serious problems that will not be addressed because of silly statements like this. Still: good work. Regards, Bèr -- [ Bèr Kessels | Drupal services www.webschuur.com ]
Gerhard Killesreiter wrote:
On Sat, 19 Feb 2005, Dries Buytaert wrote:
I slapped together the foundations of a code checker scripts and installed it on drupal.org. It took me 50 minutes to write and install. It catched hundreds of (small) issues already, and will continue to catch many more in future to it most certainly pays off.
talk == silver code == gold. :)
I installed the updated version on drupal.org! Thanks. -- Dries Buytaert :: http://www.buytaert.net/
On Sun, 20 Feb 2005, Dries Buytaert wrote:
Gerhard Killesreiter wrote:
On Sat, 19 Feb 2005, Dries Buytaert wrote:
I slapped together the foundations of a code checker scripts and installed it on drupal.org. It took me 50 minutes to write and install. It catched hundreds of (small) issues already, and will continue to catch many more in future to it most certainly pays off.
talk == silver code == gold. :)
I installed the updated version on drupal.org! Thanks.
Thanks. :) Now we will get complaints about db queries that look like these: db_query("INSERT INTO {node} (". implode(", ", $k) .") VALUES(". implode(", ", $s) .")", $v); db_query("UPDATE {node} SET ". implode(', ', $q) ." WHERE nid = '$node->nid'", $v); The $node->nid in the last one is easy to fix, but the implodes for SET or VALUES aren't unless we simply assume that all fields need to be updated/inserted. This would be possible in this case (and require some additional changes for the update case) but not in other cases where similar constructs are used. The question is: can we extend our db wrapper to support a syntax like db_query("INSERT INTO {node} %a", $kv); Where $kv would contain $k as keys and $v as values. db_query("UPDATE {node} SET %a WHERE nid = %d", $kv, $node->nid); Note that %a is not supported in sprintf and we'd need to roll our own. I recall that Steven had some idea into this direction. I also recall that I wasn't too thrilled at the time he proposed it. ;-) But I realize know that it is just carrying our db abstraction layer a step further. Cheers, Gerhard
On Sun, 20 Feb 2005, Gerhard Killesreiter wrote:
Now we will get complaints about db queries that look like these:
db_query("INSERT INTO {node} (". implode(", ", $k) .") VALUES(". implode(", ", $s) .")", $v);
db_query("UPDATE {node} SET ". implode(', ', $q) ." WHERE nid = '$node->nid'", $v);
The $node->nid in the last one is easy to fix, but the implodes for SET
Actually, we cannot do the simple change this to db_query("UPDATE {node} SET ". implode(', ', $q) ." WHERE nid = %d", $v, $node->nid); because if we have an array as first argument, the rest of the arguments is discarded.
or VALUES aren't unless we simply assume that all fields need to be updated/inserted. This would be possible in this case (and require some additional changes for the update case) but not in other cases where similar constructs are used.
The question is: can we extend our db wrapper to support a syntax like
I've some ideas here, but while implementing them, I ran across the following code in node_save: foreach ($node as $key => $value) { if (in_array((string) $key, $fields)) { $k[] = db_escape_string($key); $v[] = $value; $s[] = "'%s'"; } } That is, we treat each field as a string although there are numeric fields in $fields (changed, uid, etc.). Why isn't that a problem? I'd at least expect postgreSQL to choke at this. I'd appreciate some feedback. Cheers, Gerhard P.S.: Dries, there is some dead code in node_save: $keysfmt = implode(', ', $s); // We need to quote the placeholders for the values. $valsfmt = "'". implode("', '", $s) ."'"; No patch, CVS was down.
On Sun, 20 Feb 2005, Gerhard Killesreiter wrote:
On Sun, 20 Feb 2005, Gerhard Killesreiter wrote:
Now we will get complaints about db queries that look like these:
db_query("INSERT INTO {node} (". implode(", ", $k) .") VALUES(". implode(", ", $s) .")", $v);
db_query("UPDATE {node} SET ". implode(', ', $q) ." WHERE nid = '$node->nid'", $v);
The $node->nid in the last one is easy to fix, but the implodes for SET
Actually, we cannot do the simple change this to
db_query("UPDATE {node} SET ". implode(', ', $q) ." WHERE nid = %d", $v, $node->nid);
because if we have an array as first argument, the rest of the arguments is discarded.
$args = array_merge(array($query), $args[1], array_slice($args, 2)); instead of $args = array_merge(array($query), $args[1]); in db_query would fix this.
or VALUES aren't unless we simply assume that all fields need to be updated/inserted. This would be possible in this case (and require some additional changes for the update case) but not in other cases where similar constructs are used.
The question is: can we extend our db wrapper to support a syntax like
I've some ideas here, but while implementing them, I ran across the following code in node_save:
[...]
That is, we treat each field as a string although there are numeric fields in $fields (changed, uid, etc.). Why isn't that a problem? I'd at least expect postgreSQL to choke at this. I'd appreciate some feedback.
If this is not a problem, the function in the attached scriopt could be used to update db_query(). Cheers, Gerhard
I noticed that the PayPal tip jar module still has a bug http://drupal.org/node/12969. Before I go digging into this one I would like to know if the maintainer is working on this or is it an abandoned module. Carl McDade
Carl McDade wrote:
I noticed that the PayPal tip jar module still has a bug http://drupal.org/node/12969. Before I go digging into this one I would like to know if the maintainer is working on this or is it an abandoned module.
Last time I checked, the module was abandoned. Maybe check with njivy (http://drupal.org/user/6194), who used to maintain the module. -- Dries Buytaert :: http://www.buytaert.net/
Op vrijdag 18 februari 2005 17:52, schreef Ross Kendall:
Just for the non-official contrib modules, not for the high quality integrated modules. You can't tell me that you do not use the ratings on sourceforge.net to help you find software. Continuing on the rating discussion:
Yes I do use that rating. But SF lists thousands of projects. Drupal only less than hundred. And we should better make less projects, that are of higher projects, thaen to have hundred vapourwares. Case: My client has a deadline, requires some "easy"mentod to add images to a node, and hires me to do so. I have three options: 1) start yet another image-node project, commit the code after finsishing. 2) take one of the exsiting projects and modify it to my needs, and send diff files to the maintainer. 3) contact the maintainer and work out an idea where we will both (and others too) benefit from. IMO, case 1 and 2 might be contributed, but *only in my sandbox*. Only case 3 should be contributed back. Yes, I beleive SF needs rating because of the overall poor quality of the projects in there. Its a common OSS problem, but we do not need to have that on Drupal, since we are small and aimed at one niche. We can prove that even OSS projets can work in a more efficient way, and that OSS can be organised too. After all: anarchy is nice to start off with, but some structure is far better for all parties, and has a much higher quality ratem, ass well as far higher efficiency. (at least that is my opninion) Regards, Bèr -- [ Bèr Kessels | Drupal services www.webschuur.com ]
Op vrijdag 18 februari 2005 17:52, schreef Ross Kendall:
Just for the non-official contrib modules, not for the high quality integrated modules. You can't tell me that you do not use the ratings on sourceforge.net to help you find software. Continuing on the rating discussion:
Yes I do use that rating. But SF lists thousands of projects. Drupal only less than hundred. And we should better make less projects, that are of higher projects, thaen to have hundred vapourwares.
In my experience the number of projects plays a much smaller role than how rigorous the process is to include the project on the listing service (in our case the drupal web site). 1) For example, with a couple clicks of a mouse on sourceforge you can have a project space created and let it sit there for a year before you submit anything. Even when you do you can set the status to production/stable at any time without any community input. 2) On the other extreme is Perl's CPAN where to include a module in the listing you must propose your idea and are heavly scrutinized by the more senior module maintaners. The process to get a module on CPAN is almost scary if you are doing it for the first time. Yet CPAN has many thousands of modules, and most of them are of a very high caliber of quality because of this process. There is duplication, but not too much, and the duplication tends to spur on innovation (like windows vs linux). So so coders will just have to live with submitting patches. This second model is the one that I would like to see drupal take. Most drupal users are scared of the word CVS, and the idea of writing PHP is just unheard of. So, with that we already have 75% of the people canceled out from submitting half-baked modules. The next step is to heavly scrutinize anything that gets in, but once its in the developers need to be absolutely trusted to do a good job and clean up after themselves. If they don't they get a quick slap and all eyes are on them for a while. Aran
Case: My client has a deadline, requires some "easy"mentod to add images to a node, and hires me to do so. I have three options: 1) start yet another image-node project, commit the code after finsishing. 2) take one of the exsiting projects and modify it to my needs, and send diff files to the maintainer. 3) contact the maintainer and work out an idea where we will both (and others too) benefit from.
IMO, case 1 and 2 might be contributed, but *only in my sandbox*. Only case 3 should be contributed back. Yes, I beleive SF needs rating because of the overall poor quality of the projects in there. Its a common OSS problem, but we do not need to have that on Drupal, since we are small and aimed at one niche. We can prove that even OSS projets can work in a more efficient way, and that OSS can be organised too. After all: anarchy is nice to start off with, but some structure is far better for all parties, and has a much higher quality ratem, ass well as far higher efficiency. (at least that is my opninion)
Regards, Bèr -- [ Bèr Kessels | Drupal services www.webschuur.com ]
Firefox's extension site looks like a good example to me: https://addons.update.mozilla.org/extensions/?application=firefox They can sort extensions (read: projects) by rating, popularity, date and have an "Editor's pick". In addition, they categorize all extensions. Furthermore, they don't overwhelm the user with version information. At least not until they actually want to download/install the extension. (First you find the module, then you worry about the details.) Frankly, I think it is as simple as making (part of) the project module behave more like Firefox's extension site. Of course, that doens't fully address the original issue -- improving CVS and project management -- but it would help nonetheless. The rating, popularity and editor's pick, make the "fitest projects" bubble to the top. -- Dries Buytaert :: http://www.buytaert.net/
Firefox's extension site looks like a good example to me:
https://addons.update.mozilla.org/extensions/?application=firefox
They can sort extensions (read: projects) by rating, popularity, date and have an "Editor's pick". In addition, they categorize all extensions.
Furthermore, they don't overwhelm the user with version information. At least not until they actually want to download/install the extension. (First you find the module, then you worry about the details.)
Frankly, I think it is as simple as making (part of) the project module behave more like Firefox's extension site. Of course, that doens't fully address the original issue -- improving CVS and project management -- but it would help nonetheless. The rating, popularity and editor's pick, make the "fitest projects" bubble to the top.
Well, excellent example. Categorization of contrib modules is long overdue, and it is needed even if bundles are introduced. Goba
participants (15)
-
Aran Clary Deltac -
Bèr Kessels -
Carl McDade -
Chris Cook -
Chris Johnson -
Dries Buytaert -
Gabor Hojtsy -
Gerhard Killesreiter -
Mark Howell -
Moshe Weitzman -
Nedjo Rogers -
Negyesi Karoly -
Robert Douglass -
Ross Kendall -
Tim Altman