Hi, Thanks for your answers !

In general we don't maintain backwards compatibility of code between major releases (although neither do we break it just for fun) - every release has detailed upgrade instructions for module maintainers as http://drupal.org/update/modules

So the implication here is that if a patch potentially causes backward compatibility problems with existing modules, then it should be moved to the next major release, while issues that are clearly fixing bugs can remain in the current release. Right, well I'll get back to that first patch then :) Thanks, Anselm -- (to send me personal email, remove the _list from this address)

On Wed, Nov 5, 2008 at 6:35 AM, Nathaniel Catchpole <catch56@googlemail.com> wrote:

Almost. Issues fixing bugs that are still bugs in the next major release should always be fixed in that release first, then backported to Drupal 6 and/or Drupal 5.

That way, we don't have bugs fixed in Drupal 6 which then crop up again in Drupal 7, and there's also much more active development there, which generally means more people available to review and refine proposed fixes for any given issue.

This has always been the assumption -- that there is more development in the newest version than in older versions. But it has always just been an assumption without proof -- and even I feel it was probably true most of the time, or in the past. If one only measures core development, than of course it's true, simply because past core releases are essentially frozen except security fixes. But right now, I would bet far more effort is being spent on Drupal 6 development than on Drupal 7 development. And it's part of this topic's problem. Issues and patches are piling up in the Drupal 6 issue queues, but the push is to look at Drupal 7 development. For example, I'm spending 100% of my effort to build Drupal 6 websites. I find a bunch of bugs in D6. I write issues and post patches. My motivation to check for the same problem in D7 and then develop a D7 patch, is going to be considerably less than my motivation for D6. I might not even be able to do that, if the D7 code is not sufficiently ready or stable. If I'm already waiting for patches to be applied to D6 modules, I'm not going to be interested in waiting even longer to have them applied to D7 and then get backported to D6. I need the fix yesterday, not next year. Really it's all about every member of the community having a different agenda, and everyone is negotiating with the community to get as much support for their own agenda as possible. Some people have more influence than others or more power than others in these negotiations (the Drupal community is much like the rest of life in this regard, after all). The question is whether the majority should continue to be facilitate the agenda of the minority, or if the majority should stand up, notice that it is the majority, and push more strongly for what they want.

On Wed, Nov 5, 2008 at 2:59 PM, Chris Johnson

This has always been the assumption -- that there is more development in the newest version than in older versions. But it has always just been an assumption without proof -- and even I feel it was probably true most of the time, or in the past.

If one only measures core development, than of course it's true, simply because past core releases are essentially frozen except security fixes.

Exactly, and we're talking about core patches. Most active contrib development is happening in Drupal 6, I generally don't see many patches posted for Drupal 4.7 versions of contrib modules, or even Drupal 5 depending on which module it is.

But right now, I would bet far more effort is being spent on Drupal 6 development than on Drupal 7 development. And it's part of this topic's problem.

Issues and patches are piling up in the Drupal 6 issue queues, but the push is to look at Drupal 7 development.

For example, I'm spending 100% of my effort to build Drupal 6 websites. I find a bunch of bugs in D6. I write issues and post patches. My motivation to check for the same problem in D7 and then develop a D7 patch, is going to be considerably less than my motivation for D6.

Well you could post the patch against D7 without verifying, someone watching that queue will take a look (if they have time), and if it doesn't apply to D7, move it back to D6, or re-roll for Drupal 7. This happens regularly, stating that the patch was rolled against Drupal 6 is all you need to do here.

I might not even be able to do that, if the D7 code is not sufficiently ready or stable.

D7 is exceptionally stable due to many hundreds of automated tests. Getting tests written to prevent bugs reappearing is one of the many reasons that doing active development there is useful. Since the tests verify that the bug is fixed (assuming they're written properly), then in general you're more likely to get a decent fix, and one that doesn't cause regressions elsewhere. And supplying patches with tests gets them committed much, much faster.

If I'm already waiting for patches to be applied to D6 modules, I'm not going to be interested in waiting even longer to have them applied to D7 and then get backported to D6. I need the fix yesterday, not next year.

Patches can get applied to Drupal 7 in as little as a few hours, depending on the patch, the availability of reviewers and core maintainers. An example from this week: http://drupal.org/node/329646 opened on third november. Fixed in both Drupal 6 and Drupal 7 on 4th November http://drupal.org/cvs?commit=151247 http://drupal.org/cvs?commit=151248 In which way is this slower than ignoring advice, posting a patch to the Drupal 6 queue, then trying to cajole a small number of over-stretched volunteers to change their workflow?

Really it's all about every member of the community having a different agenda, and everyone is negotiating with the community to get as much support for their own agenda as possible. Some people have more influence than others or more power than others in these negotiations (the Drupal community is much like the rest of life in this regard, after all).

The question is whether the majority should continue to be facilitate the agenda of the minority, or if the majority should stand up, notice that it is the majority, and push more strongly for what they want.

Well, the smallest minority is the core maintainers. There's two for Drupal 7, two for Drupal 6 (obviously Dries gets counted twice, so that's actually three people for both branches). Since only Dries can commit to both branches (like the example above), something has the best chance of being fixed in both branches if it's submitted to the Drupal 7 queue first of all. Also, webchick has a lot more time allocated to core development than Gabor, since Drupal 6 is in maintenance. If a patch goes through the review and RTBC cycle in Drupal 7, then more times than not it's a quick look over and commit for Drupal 6 - expecting all the back and forth of a core commit to happen in a maintenance release is asking too much IMO - Drupal 6 is likely to have a 2-3 year release cycle, after all. The other minority you're talking about is those actively working on core development on a regular (i.e. daily) basis - i.e. the people who are actually reviewing and RTBC-ing those patches. As far as I know , the majority of that minority prefers the current workflow too. This is volunteer project, and the ony real power as such lies with those with commit access - people can make their voices heard by complaining on the development list, or by getting stuck in, and change gets made on the strength of arguments. I've not actually seen any arguments put forward for changing the current workflow, except that some people 'don't have time'. Nat

Persuading the DRUPAL-5 and DRUPAL-6 branch maintainers to change their policy of not fixing bugs in their branch until they've been fixed in the development branch is what's important. On Nov 5, 2008, at 11:15 AM, Nathaniel Catchpole wrote:

One more thing ;)

If you want to see more active development on outstanding Drupal 6 issues, feedback on adding the Drupal 6 patch queue links back to the contributors block would go a long way - see here http://drupal.org/node/221510

Nat

On Wed, Nov 5, 2008 at 10:01 AM, Nathaniel Catchpole

Exactly, and we're talking about core patches. Most active contrib development is happening in Drupal 6, I generally don't see many patches posted for Drupal 4.7 versions of contrib modules, or even Drupal 5 depending on which module it is.

We were? (talking about core patches?) I don't see that in this thread. We were talking about all patches, as far as I could tell. The arguments are indeed very different for core and contrib. I was positing primarily about contrib, since that's where most of the need is. Still, there is a problem with core issue queues languishing, too. I can't remember the last time I posted a patch to core; it's become too hard for me to contribute meaningful new core code and get attention to it.

Quoting Chris Johnson <cxjohnson@gmail.com>:

On Wed, Nov 5, 2008 at 10:01 AM, Nathaniel Catchpole

...

Exactly, and we're talking about core patches. Most active contrib development is happening in Drupal 6, I generally don't see many patches posted for Drupal 4.7 versions of contrib modules, or even Drupal 5 depending on which module it is.

We were? (talking about core patches?)

I don't see that in this thread. We were talking about all patches, as far as I could tell.

Until today, I've been talking about core patches.

The arguments are indeed very different for core and contrib.

Indeed they are and the mechanism for contrib isn't the same nor as strong a force as core either.

I was positing primarily about contrib, since that's where most of the need is. Still, there is a problem with core issue queues languishing, too.

There are patches for core that are months, years old. Yes, they languish because no one tests and marks RTBC. So, naturally there are contrib patches that are also old. But those are more maintainer time related. However if those interested in a contrib patch reviewed and tested it then maybe the maintainer would be more willing to commit it.

I can't remember the last time I posted a patch to core; it's become too hard for me to contribute meaningful new core code and get attention to it.

Is that due to lack of reviewers? It seems people think there is a vast set of developers for core, that just isn't true. There are a few that know the ins and outs of core by heart but you only need a hand full of fingers to count those. Find a bug submit a patch, hope for a reviewer, don't be offended by any action of any reviewer, press forward and patch. Earnie http://r-feed.com Make a Drupal difference and review core patches.

Yes I can read the code but the question above can not be answered by reading the code - much more helpful would be a comment what the _aim_ of some lines of code is.

And maybe the group working on and with that particular module has a list of decisions (not a mailinglist) and why they where made - so one could go back to the original decision (should be referenced in the code) and look up what the original intention was.

Do you agree that these information could help a lot in similar situaltions?

This is actually a feature built into CVS (and most other version control systems) called CVS Annotate that does exactly that: http://www.lullabot.com/articles/cvs_annotate_or_what_the_heck_were_they_thi... For every line of code, you can discover who made the change, when they made it, and why. Assuming the maintainer is following standard commit message patterns, you can also reference the original issue that has all the background information on discussions on the code that were had, the development evolution of the feature over time, and why the decision was ultimately made to commit it. It's a pretty awesome resource because it's automatically updated with every commit, without the need for any manual intervention or extra overhead. -Angie

Quoting Thomas Zahreddin <tz@it-arts.org>:

My request is for that higher level: For Core the policy and a handbook is avialable; but you can find modules in contrib that even don't have a README - sure you can go to the sourcecode and look what they do - but this is not efficient.

I'm confused, are you discussing policy for core or for contrib? I don't think anyone enforces core policy on contrib much. Encourages it, yes, but doesn't enforce it. Each contrib maintainer chooses their own rules. Earnie -- http://for-my-kids.com/ -- http://give-me-an-offer.com/

Funny that I was having just this discussion with my Dev team at work today. How shall we document how/why something works. I think most of the folks agreed that closer to the code is better. So comments is the best answer but it's often difficult to document how disparate modules connect in code comments. Another resource that we have that's not used as much as it could be is the API module and it's document generation techniques. It already documents much of the bigger picture with core modules. Correct me if I'm wrong, but cannot developers maintain "extra" technical documentation in a separate document along with their modules in CVS? That might be a good thing to version control, but keep a static document, that feeds api documentation. I know this is possible with other modules such as Doxygen, but am not sure that it works in API module. (but I bet it could). This is the solution we landed on in my dev shop. That way rational/design info could(should?) be supplied with patches. In separate documentation and it would help those who do patch review understand how/why something is supposed to work before it gets committed. You see that's the one problem with using CVS annotate. The commiter provides that message, but the patch submitter is the one who needs to supply the documentation :). And remember our goal here is to make it easier for the review/commit phase, so adding doc work to the commiter is counter productive. So I like using CVS for why something changed, but not how something is supposed to work. That needs to be in code comments or lean text base documentation that goes with the code. But perhaps we near exhausting all that can be said on this issue. On Nov 6, 2008, at 8:21 PM, Angela Byron wrote:

On 11/6/08 6:19 AM, Thomas Zahreddin wrote:

...

Hi Angie,

thank you for answering the first third of my mail. I think we are not on the same page:

in the last third I wrote explicit, that I know all the tools and places where to find informations - but i also mention that it is impossible to follow all relevant information.

To lower the workload for us all we need a summery on a much higher level than CVS-messages (btw how many do we count a day - my estimation is two a minute).

I answered the first third, because it seemed to me that the last two thirds were based on the presumption that the information you seek is not readily available. But CVS annotate seems to do exactly what you request and it requires no manual effort on the part of our already over-burdened maintainers to keep up to date, since it's updated automatically with every commit.

In what specific ways does CVS annotate not get you what you're asking for? That would probably help the justification for your proposal be a bit more clear.

-Angie

drumm and I talked a fair bit about this in Szeged, and I have a todo list gathering dust on my desk for writing up some of the specifics pertaining to this, but... api.module has several areas that need working on - the large-scale priority list was a) rewrite the parser using php's tokenizer (we have also talked about the possibility of a Reflection-based parser, which has some problems but could be stupidly easy to extend), then b) port to d6. Neil/somebody, please do correct me if this is no longer accurate. And imo, those are top priority because api.module's lack of support for classes is debilitating for D7. However, there are some really interesting possibilities for including 'static' documentation, as David has suggested below. Doxygen does make it possible, although it's quite awkward. The Panels API documentation is done with doxygen, for example, and uses some such static documentation: http://doxy.samboyer.org/panels2/panels_api_plugins.html I wrote that reference with the intention of it being reminiscent of http://api.drupal.org/api/file/developer/topics/forms_api_reference.html, except that it's generated directly from documentation files that are checked in to Panels CVS here: http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/panels/docs/sam... then generated by doxygen using the doxyfile here: http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/panels/panels2.... (the ALIASES section is the substantive, and horrifying, part) I wouldn't wish the pain of that approach on anyone else, especially because I think there are a lot of better ways to do it. In particular, we might create our own @tags extending the standard set recognized by either phpdoc or doxygen. The value of an @invokes tag will be immediately obvious to most devs, but the potential for something like, say, @glossary tags for building up a network of terms for autolinking within api docs is really interesting. There are other possible avenues for expansion too, of course, but drumm's been rightfully hesitant to throw new @tags in willy-nilly. More importantly, though, ALL such discussion is moot until the basic rewrites and porting is taken care of. cheers Sam On Thursday 06 November 2008 23:28:13 David Metzler wrote:

Funny that I was having just this discussion with my Dev team at work today. How shall we document how/why something works. I think most of the folks agreed that closer to the code is better. So comments is the best answer but it's often difficult to document how disparate modules connect in code comments.

Another resource that we have that's not used as much as it could be is the API module and it's document generation techniques. It already documents much of the bigger picture with core modules. Correct me if I'm wrong, but cannot developers maintain "extra" technical documentation in a separate document along with their modules in CVS? That might be a good thing to version control, but keep a static document, that feeds api documentation. I know this is possible with other modules such as Doxygen, but am not sure that it works in API module. (but I bet it could). This is the solution we landed on in my dev shop.

That way rational/design info could(should?) be supplied with patches. In separate documentation and it would help those who do patch review understand how/why something is supposed to work before it gets committed.

You see that's the one problem with using CVS annotate. The commiter provides that message, but the patch submitter is the one who needs to supply the documentation :). And remember our goal here is to make it easier for the review/commit phase, so adding doc work to the commiter is counter productive.

So I like using CVS for why something changed, but not how something is supposed to work. That needs to be in code comments or lean text base documentation that goes with the code.

But perhaps we near exhausting all that can be said on this issue.

On Nov 6, 2008, at 8:21 PM, Angela Byron wrote:

...

On 11/6/08 6:19 AM, Thomas Zahreddin wrote:

...

Hi Angie,

thank you for answering the first third of my mail. I think we are not on the same page:

in the last third I wrote explicit, that I know all the tools and places where to find informations - but i also mention that it is impossible to follow all relevant information.

To lower the workload for us all we need a summery on a much higher level than CVS-messages (btw how many do we count a day - my estimation is two a minute).

I answered the first third, because it seemed to me that the last two thirds were based on the presumption that the information you seek is not readily available. But CVS annotate seems to do exactly what you request and it requires no manual effort on the part of our already over-burdened maintainers to keep up to date, since it's updated automatically with every commit.

In what specific ways does CVS annotate not get you what you're asking for? That would probably help the justification for your proposal be a bit more clear.

-Angie

http://www.google.de/search?q=commit+messages+site%3Adrupal.org

-----Original Message----- From: development-bounces@drupal.org [mailto:development-bounces@drupal.org] On Behalf Of Aaron Winborn Sent: Thursday, November 06, 2008 3:36 PM To: development@drupal.org Subject: Re: [development] Reviewing patches and making decisions -> Sociocracy could be a way to go!

What's the 'standard commit message pattern'? I thought I read once it was something like '#nid/username: comment', but can't find that anywhere. For readability, I just copy the comments from my changelogs, which usually follows something like 'Month, Year\n-----------\n * #nid: comment (username).' (Leaving out the M/Y obviously, since that's already present in the CVS logs). But I'd like to do this properly in the future.

Thanks, Aaron

Angela Byron wrote:

...

This is actually a feature built into CVS (and most other version control systems) called CVS Annotate that does exactly that:

http://www.lullabot.com/articles/cvs_annotate_or_what_the_heck_were_th

...

ey_thinking

For every line of code, you can discover who made the change, when they made it, and why. Assuming the maintainer is following standard commit message patterns, you can also reference the original issue that has all the background information on discussions on the code that were had, the development evolution of the feature over time, and why the decision was ultimately made to commit it.

It's a pretty awesome resource because it's automatically updated with every commit, without the need for any manual intervention or extra overhead.

-Angie

-- Aaron Winborn

Advomatic, LLC http://advomatic.com/

Drupal Multimedia Available Now! http://www.packtpub.com/create-multimedia-website-with-drupal/book

My blog: http://aaronwinborn.com/

On Wed, Nov 5, 2008 at 4:01 PM, Thomas Zahreddin <tz@it-arts.org> wrote:

Hallo,

thank you for bringing this topic up.

Since a long time I'm unsatisfied with this process.

(@ Dries: this is the e-mail i announced to you in Szeged)

...

Hello,

Having followed this thread for a while, I decided to do my bit, and start reviewing patches. I didn't get very far ; I stopped at the first patch.

Not because of the code - I could understand what the original code did, and what the patched code did - but because there are two decisions to be made :

1. In that case - and I guess this is not uncomon - the old behaviour might be a bug, but it might have been by design. I have no way of telling.

(caution: Ironical!) Oh, I can't count how often i heard: "the code tells it all!" or "read my code first" or "only code counts, not the words about it" ..

Yes I can read the code but the question above can not be answered by reading the code - much more helpful would be a comment what the _aim_ of some lines of code is.

And maybe the group working on and with that particular module has a list of decisions (not a mailinglist) and why they where made - so one could go back to the original decision (should be referenced in the code) and look up what the original intention was.

Do you agree that these information could help a lot in similar situaltions?

This is what proper commenting is for. Compare the two leading lines below. // iterate over items and trim away any whitespace. // trim items, whitespace coming in as a part of user input was breaking sorting later. see #889844 foreach($items as $i => $j) { $items[$i] = trim($j); } again something you can't do anything about besides trying to grok the code and comment it or ask the original developer.

2. Changing the old behaviour to the new one could break old modules that

...

implicitely relied on the old behaviour. Again, this is probably not uncomon. How do I weight the importance of the patch vs stability of old modules ?

IMHO persons, interested in a module (like programmers and users of this or a compatible module), shall have the possiblity to be heard with their requirements. It is up to the maintainer (or moderater) to listen to all and their arguments what is urgent and /or important and to come up with suggestions to which most (if not all) people can agree. The final decision is up to the maintainer (keeping in mind the long term goals, all persons interested in the module and the available resources) in cooperation with the person willing to contribute (e.g. code).

If the group of persons working on a module keeps a logbook about their decisions, then everybody is able to follow the decisions regarding a modul.

A short description of this method of dynamic selfgovernance can be found on http://en.wikipedia.org/wiki/Sociocracy

or as free pdf

http://www.governancealive.com/Links/The_Creative_Forces_of_Self_Organizatio...

A longer description is in Buck, John and Sharon Villines (2007). We the People: Consenting to a Deeper Democracy, A Guide to Sociocratic Principles and Methods. Sociocracy.info Press. ISBN 978-0-9792827-0-6.

Maintainers are already pressed for time... at least in contrib, I seriously doubt you will convince maintainers to keep such a list separate from inline comments.

So what do I do next ? I don't have a sufficient overview of that part of

...

Drupal to make such decisions myself. Should I mark it as reviewed, and add as a note what the potential implications are ? Or is there another process to follow in such situations ?

I agree absoluty: We do not have approriate structures in the drupal community to work efficently, make decissions traceable, keeping the teamspirit up and stay open for new developers to join.

I'm aware of: there are ca. one million notes on drupal.org, some IRC- channels, conferences, groups.drupal.org, mailinglists like this one, a CVS- Repository ... - but you can imagin alone by this list how hard it is to follow all relevant information / decissions.

So I suggest to read a little about Sociocracy and start discussing our decision making process.

IMHO: too much overhead , too few concrete suggestions, something that will get talked about a lot and nothing will come of it. google + api.drupal.org + inline comments + test cases should provide sufficient information to make most of your decisions. Nothing can replace a solid understanding of the code you're reviewing. .darrel.