<karoly@negyesi.net> wrote:

...

As Drupal 5 and 6 are very good, rock solid releases, people are satisfied and very busy there, much less is the need, the drive and the availability for core development. This means less patches are written. Also core got bigger. Very, very few people touch actions/triggers, fields, filter system, openid or for that matter, menu just to name a few. Once a patch is written , it needs a review.

I completely agree. However, how do you suggest we make this better?

Dmitri

I completely agree as well. Core is way ahead of contrib and far ahead of actual implementations. Only a minority of contrib modules has really adopted all new features of D6 thus far. There is an obvious gap between possible implementations and actual implementations of core features in modules. This also explains the fact that only a few contrib developers have the energy to work on improvements to Drupal core that would actually improve our APIs for contrib. The answer to solve this is simple, but somehow disliked by some: Let it mature. i.e. defer the code freeze for D7. I know that I'm not the only one who would prefer that (but capitulated in disagreement). sun

I don't think you are making much sense here. You claim that module developers haven't found time to add the new D6 features in their code and at the same time want to have a D7 which is more mature (and presumably thus contains even more features).

In the end your proposal would only serve to disconnect core and contrib even more.

Cheers, Ger»release early, release often«hard

In case this really was ambigious: Let it [D6] mature. => i.e. defer the code freeze for D7. Obviously, I mean the exact opposite of what you understood. Contrib needs time to catch up with core. sun

On Mon, Apr 20, 2009 at 7:06 AM, Bryan Ollendyke <bto108@psu.edu> wrote:

lol yeah, I believe the intent of the original author was to hold off on D7 until D6 matured more from a crontrib standpoint.  I'll +1 that idea.

I had concerns and I shared that with the community. I had no solutions. I stil do not have. I am not sure postponing Drupal 7 even further is the solution. Because then, as Gerhard says, what will you do when Drupal 7 gets out, send core development on vacation for two years?

Karoly Negyesi wrote:

what will you do when Drupal 7 gets out, send core development on vacation for two years? Hmm, I wonder what would happen if we all realized that every member of DO is a "core developer?" There may be a few people who are more or less directly responsible for consolidating proposed changes, but it is the community at large that is responsible for making those proposals - and patches. Even though we sometimes refer to those privileged (burdened) few as "Drupal-gods," they are not perfect; they cannot predict every way someone will use the code, nor can they test everything all together the way we put them together. Those people have real lives too (I think), and have to do the same thing we do: fit in time for Drupal amidst all our other demands. I don't always agree with everything they do or say - I dare say chx is one of them who knows that already. But Drupal is a great system, and it will stay that way only if we all think of ourselves as core developers.

No virus found in this outgoing message. Checked by AVG - http://www.avg.com Version: 8.0.176 / Virus Database: 270.12.0/2068 - Release Date: 4/19/2009 8:04 PM

Nedjo has given us the first response to chx's origninal problem statement that addresses the real intent of the concern, and proposes solutions. Of the three solutions Nedjo proposed, I favor c) giving targeted commit rights to people who have historically been the maintainers for certain subsystems. We would need to develop guidelines around where their turf ends and they are no longer free to commit without escalating to either the branch maintainer or a permanent core maintainer (Dries), but this is a feasible solution that could speed the development of certain subsystems (actions, triggers, openID, i18n). The other solution which chx and I have discussed, and which is a long time goal of mine, would be to reduce the size of core. We carry around a lot of modules that don't absolutely need to be part of core: - aggregator - blog - blogapi - forum - help - poll - profile - search - statistics - taxonomy - tracker I believe that all of these modules could have brilliant lives outside of core. A slimmer core means more focused development for the core team. It would also make us focus our attention on the APIs that support these very important modules. For example, the search API is in great need of being decoupled from the node module. If our goal were to remove search from core we'd need to really make sure the API were in place for letting core be searched. My argument squarely reflects opinion that "core" Drupal should be an application framework that doesn't offer any specialized functionality. Forums, polls, profiles, blogs and even taxonomy are specialized web application functions that don't need the 10,000 layers of red tape that core has. I firmly believe that taxonomy, for example, is hampered by its presence in core. There has been amazing work done on turning taxonomy into something far beyond what we have today, but the very presence of taxonomy IN CORE means that nobody uses it. Same with search. When I first wrote the ApacheSolr module it utilized as much of the search "API" as possible. Now we're moving to make it completely independent of core search because there is no way core can keep up with our development pace. So, in my opinion, if you want to take Drupal to the moon, you do two things: 1) give more people the keys to the car, and 2) shed a bunch of the pieces that we currently call "core". Robert Douglass The RobsHouse.net Newsletter: http://robshouse.net/newsletter/robshousenet-newsletter Follow me on Twitter: http://twitter.com/robertDouglass On Apr 20, 2009, at 5:35 PM, Nedjo Rogers wrote:

Karoly Negyesi wrote:

...

On Mon, Apr 20, 2009 at 7:06 AM, Bryan Ollendyke <bto108@psu.edu> wrote:

I had concerns and I shared that with the community. I had no solutions. I stil do not have. I am not sure postponing Drupal 7 even further is the solution. I agree that bottlenecks in the reviewing and applying of patches is a huge problem majorly holding up Drupal development. I suspect we need some significant structural changes to address it.

Partly I think we need to look at changes in the drivers of Drupal development.

There was a time when it was mainly hobbyists, who weren't particularly tied to external timelines. No longer. The biggest amount of focused development time now comes from medium to large companies needing specific functionality within set timelines. The typical development cycle for a large project (e.g., a major site or multisite installation) is a matter of months. Development can't wait the many months or even years before a new version is ready and can't deal with the uncertainty of fluxuating timelines. So the significant work they sponsor is almost never for core, which is years ahead of what can be used. It's instead for the current usable release version.

Meanwhile, with fewer resources, core goes its different direction, choosing its own solutions to many of the same problems. And so the contrib-core gap increases.

Even when a corporate-sponsored project is specifically designed to contribute to core, the results can be mixed at best. Take the internationalization work CivicActions did, sponsored by Sony Music, see http://drupal.org/node/383954. We worked just as hard or harder on numerous core fixes and enhancements as we did on work in contrib. But at the end of the six or seven months, when pretty much every significant patch had been accepted and applied in dozens of contrib modules (Views, CCK, Flag, Voting API, Internationalization, Date, etc.), practically none of our core patches had got in.

I conclude:

1. More than tweaks and further exhortations to just do more reviewing, we probably need to consider some major changes. 2. Any changes should increase the ease and incentive of contributing major improvements to core as part of the regular development work.

Here are some concrete ideas:

1. Expand our team of core committers.

Since 4.7 we have had a single maintainer per major version in addition to the permanent core committers (now reduced to one). We could consider changing this. Possible changes:

a. Each core committer either becomes a permanent core committer or is maintained for two or three versions instead of one. E.g., while Angie is the primary maintainer of 7.x, some or all of Gerhard, Neil and Gabor retain the ability to accept and apply patches for HEAD.

b. Two or three core committers are appointed per major release instead of just one.

c. The maintainers listed at http://cvs.drupal.org/viewvc.py/drupal/drupal/MAINTAINERS.txt?view=co are given permanent commit access to be used only in the areas they are responsible for.

2. Shorten our release cycle. Instead of trying to fit many major API changes and improvements into each major release, we instead limit ourselves to a focused few changes and a much shorter timeline. This quickens the pace of change, reducing the contrib-core gap.

3. Rather than always limiting new functionality to the new version, we accept specific, break-nothing backports in core of new functionality and APIs. E.g., we have two branches of each major version, one a feature-frozen branch (what we have now) and the other one (yes, somewhat less stable) that accepts selected new APIs and features that have been added already to HEAD. This allows contrib modules to use the new APIs in the current version.

I know that all of these potential changes - and others that I hope others will come up with - would have potential pitfalls and disasters. But I think these are the sorts of discussions we should be having. We've recognized a long-standing issue. We need some creative thinking, drawing on our strengths, to consider afresh how to keep core development accessible, rewarding, and effective.

Nedjo

On Apr 20, 2009, at 6:13 PM, Gábor Hojtsy wrote:

On Mon, Apr 20, 2009 at 6:03 PM, Robert Douglass <rob@robshouse.net> wrote:

...

So, in my opinion, if you want to take Drupal to the moon, you do two things: 1) give more people the keys to the car, and 2) shed a bunch of the pieces that we currently call "core".

Would that solve our other problem: that many contrib modules got updates later, so it turns out after a stable release that certain APIs are not complete or suitable for important contrib stuff? If we move more of the modules which use these APIs outside of core, we go to pure API design without validation of those APIs with real life modules, no? While I agree some modules would be best to go, maybe moving all is not the solution. Poll was joked to be one of the best API validation modules (for drag and drop, AHAH, etc). While that was a joke to some degree, it also has roots in reality.

Gábor

I agree that your concerns are valid, but I think that solving those concerns would be easier than solving the larger core concerns raised by chx if we continue to follow the exact strategy that we have now. API validation is vitally important, and should be driven by contrib. Contrib should be instructing core "hey, I need this in the API so that I can do what I want". Contrib module maintainers currently submit patches to core to tend to the important APIs that affect their modules. Witness Earl Mile's recent patch that has major ramifications on future Views development. http://drupal.org/node/322344 Contrib modules are real modules. They do a fine job of finding the limitations and strengths of core's APIs. The feedback loop from contrib to core is there, and it is strong. In one direction. Most of that feedback currently sits idle in the core issue queue, however, and contrib module authors often choose to just work around core instead of enhancing it. Poll, despite being "core's regression test", is the best example. The only time anyone pays any attention to poll module development is when other people scream "get poll out of core!" loudly enough :P These days, if we need regression tests, we should put them in the Simpletest framework. Another solution would be to split core into two parts. "Framework", and "CMS". The framework would be the things like user, node, block, system, actions, fields, etc., and the CMS would be things like poll, blog, forum and who knows what. The smart thing would be to make "Framework" in this case as small as possible, and push as much as we can into "Framework". Then "Framework" could adopt a nice, long release cycle and focus on big API changes, whereas "CMS" could focus on shorter release cycles upon any given "Framework" release. "CMS" would get a new set of "core" committers.

...

Another solution would be to split core into two parts. "Framework", and "CMS". The framework would be the things like user, node, block, system, actions, fields, etc., and the CMS would be things like poll, blog, forum and who knows what. The smart thing would be to make "Framework" in this case as small as possible, and push as much as we can into "Framework". Then "Framework" could adopt a nice, long release cycle and focus on big API changes, whereas "CMS" could focus on shorter release cycles upon any given "Framework" release.

I like this idea but it sort of goes into the whole "golden" module list that seems to come up ever now and again.

andrew

I think the problem is less the number of modules we have in core (mainly CMS-ish ones), and much more that the ones we have in core are only there for arbitrary / historical reasons. A patch to add forum, blog, help, poll, tracker, taxonomy to core now would be almost immediately won't fixed or stay at CNW for a very long time requiring years of work to get right - either because there are better contrib alternatives, or because the code is crappy, or both. However taking them out requires an upgrade path and a responsible contrib maintainer, not to mention consensus, and getting all three of those at the same time is hard too. On the other hand, getting eternally useful modules like token, CCK, Views etc. into core is a very long and drawn out process too - for good reason - if only in part because we know that once they're in core it's very hard to take them out, and any wrong design implications have a much longer lifespan and are harder to fix (hence the main query in tracker module which was bringing Drupal.org down regularly on Drupal 5 still hasn't been fixed in core for three releases). Packaging and better install profiles might help, getting those major APIs into core will hopefully help us push the old-style modules out of core eventually as well - i.e. replace tracker.module with a couple of default views. I don't think this is the root of the various recurring frustrations with core development though at all - either those of us very active in the core issue queue or those who avoid it like the plague. In my opinion it's a combination of factors, most very hard to fix - hence threads like this and discussions in #drupal on a regular basis. Nat

On Mon, Apr 20, 2009 at 10:06 AM, Peter Wolanin <pwolanin@gmail.com> wrote:

I would strongly agree with chx and others that in some way we need additional committers.

You can't agree with me because I have not said so. Check the thread again :) I have not suggested anything and will not (for some time at least). I deliberately avoid taking sides and rather play devil's advocate. Do not presume I agree or disagree with any idea presented in here. The rest of this mail will be reflections on many of the things were raised in here. It really must be noted that the Drupal 6 cycle was unusual because major contribs decided they do not want to maintain old crap and did a complete refactoring and started that only once D6 was out the door. This, hopefully, won't repeat so in the future contribs will be ready somewhat sooner. This does not help the contrib-core gap, of course. One of the reasons I left contrib is that my perfectionist half could not stand the crappy code my programmer half produced. I can't write pretty code without peer review. If you think you can, my best advice is to think again. Really, peer review is almost mandatory. Simply allowing people to commit patches will not help our code quality. Really, check most contrib vs core. I started with a post which showed how a shortage of patch writers, reviewers and committers create a self strenghening cycle. Making a half-assed fix at committers is not enough to fix the whole cycle. Slimming down core is an idea but as was noted we do we need them to validate. Also, if you slim down core and thus speed up development, how again will that help the contrib-core gap? Now, Drupal shops. Sure, you can't sell to a client directly core development but what about this, say, you would ask for 100/hr on a project which takes 400 hrs (less hree man-months, not that much). If you ask 110/hr instead then that client sponsored 40 hrs of core work, too. On the other hand if you can get away with 110/hr then why not just bill that and go to the beach?

Notwithstanding the fact that modules shed from core tend to die, I think some of the modules you mention could indeed be made contrib, because their are "application"-type (like aggregator, forum, or poll), whereas one (taxonomy) is what really defines Drupal to many people outside the coder group, and others (search, statistics, ....profile(!) ) are part of anyone's basic expectations in a CMS. ----- Original Message ----- From: "Robert Douglass" <rob@robshouse.net> To: <development@drupal.org> Sent: Monday, April 20, 2009 6:03 PM Subject: Re: [development] Very concerned over Drupal's core development [...] The other solution which chx and I have discussed, and which is a long time goal of mine, would be to reduce the size of core. We carry around a lot of modules that don't absolutely need to be part of core: - aggregator - blog - blogapi - forum - help - poll - profile - search - statistics - taxonomy - tracker I believe that all of these modules could have brilliant lives outside of core. A slimmer core means more focused development for the core team. It would also make us focus our attention on the APIs that support these very important modules. For example, the search API is in great need of being decoupled from the node module. If our goal were to remove search from core we'd need to really make sure the API were in place for letting core be searched. [...]

A few idea (I'm flinging IT at the wall to see what sticks): 1) Building on what Angie wrote about teams. We could put some time into tools to help people build teams. g.d.o helps with the tools. We could see what's missing and bring the tools there to help form the teams. Maybe come up with common solutions to making successful teams and sharing them. Maybe a guide to team building. It may seem basic but a guide can help people who don't know how. 2) The complex systems (like filters, menus, etc.) can be scary to people. Even really smart and capable developers. We could create training and teaching to explain how these work and why these work the way they do. The caching systems interaction with filters has caused more than a few questions, arguments, and violent beatings on drupal code. Tools to lower the barrier to entry to understand the more complex systems can invite people to learn them and learn how to code better raising the bar overall. 3) dww started to work on a distribution system for drupal some time ago. A package with an install profile, drupal core, modules, themes, etc. Imagine drupal.org creating distributions. There could be one for blogging, a video site, a knowledge management system, etc. We could be less worried about what core puts out for people to use and let it be a tool that things are built upon. The concern I've had is more along the lines of what chx noted. Go back and read it. It's not the committing of patches. It's the people with expertise in different areas working on and reviewing patches. For all the hundreds of people working on core there are areas we need more people qualified to work on the tough stuff, more people reviewing patches, and more people crafting them to be core worthy. More committers doesn't help with this part of the process (though that may eventually become a bottleneck). Additionally, there is the time aspect. Core development isn't a high priority for many people. How can we change that? If it's a priority people will find to be involved. Matt On Apr 20, 2009, at 1:25 PM, Angela Byron wrote:

On 20-Apr-09, at 11:35 AM, Nedjo Rogers wrote:

...

<snip> c. The maintainers listed at http://cvs.drupal.org/viewvc.py/drupal/drupal/MAINTAINERS.txt?view=co are given permanent commit access to be used only in the areas they are responsible for.

Let's examine the consequences of this, since it's always a popular option when it comes up. I'm going to choose the menu system as a random example, but you can easily substitute 'menu system' as $subsystem and 'chx and pwolanin' as $subsystem_maintainers in the following narrative.

The menu system is a bit tweaky and its innards not easily understood by the "common developer" in the Drupal community. As a result, patches that improve the menu system tend to languish in the queue in "code needs review" for weeks or months. When a review does happen, it's normally on more minor things like coding style or similar; not about the fundamental architecture of the menu system. This is extremely frustrating to chx and pwolanin, the menu system maintainers, since they want to keep going with their crazy development ideas and are blocked.

So, to resolve the issue of menu system patches languishing in the queue, chx and pwolanin are granted commit access to the menu system. Suddenly, all of our problems with languishing patches magically disappear! After a quick discussion in #drupal-dev, patch after patch that improves the menu system is committed. The menu system has never evolved so quickly in such a short period of time! And better still, the committers for the field API, the database system, actions/triggers, etc. are all empowered to do so as well, so Drupal evolves at lightning speed! WIN!

Of course, without any kind of peer oversight, while these patches that get committed make perfect sense to chx and pwolanin, they completely mystify the "common developer." When Drupal 7 is released, it becomes painfully obvious that we've developed extremely well-engineered subsystems without ever exposing a "real" human developer to having to work with them. And since these patches only needed to make sense to two subsystem maintainers with intimate knowledge of the underlying subsystem already, documentation is extremely sparse. It's now way harder than ever before to jump in and get involved in core development.

Worse, because all of these subsystems were rapidly evolving within their own sphere, 20, 30, perhaps 50 patches committed per day, it's become impossible for anyone to oversee all of them; at best you have people inspecting one-off patches here and there. That means things like coding standards compliance, performance benchmarking, algorithm optimization, and the collective DX (not to mention UX) experience of Drupal 7 plummets. Drupal 7 is usable by chx, pwolanin, DamZ, catch, and about 20 other people, and everyone else switches to Django. ;)

We have this kind of decentralized development model, where one or two people are solely responsible for code with basically zero peer review. It's called contrib. And it's notoriously filled with sub- standard, shoddily documented code that needs to be closely inspected by individual site maintainers before being deployed on any serious production sites.

The entire strength of Drupal core *is* that it is peer reviewed (sometimes mercilessly so :)), it has oversight from others who are *not* deeply immersed in APIs who can give sanity checks, and has people who are bringing a very *diverse* range of experience levels, specializations, and use cases. You remove that, you remove what I would argue is the very heart and soul of Drupal.

"But webchick!" I hear you say. "Obviously, we wouldn't let these subsystem maintainers work /completely/ bereft of peer review. We'd surely have someone else check over their work." And I say, "Great! Then *why aren't these mythical someones doing this right now*?" and *then* you get at what I believe is the /actual/ root of the problem. ;)

*We need to grow the base of people who have understanding in these subsystems.*

People who consider themselves subsystem maintainers should be actively working to build teams around their subsystem of choice. They should be documenting how their underlying APIs work so that someone without familiarity could jump in and get oriented quickly. They should be recruiting from contrib authors who are doing interesting/innovative work in their areas. They should be maintaining a "patch spotlight" page under http://drupal.org/community-initiatives/drupal-core so that people who want to jump in and don't know where to start have a direction. They should be guiding and mentoring "younger" contributors on how to do patch reviews, and actively advocating for others to do so as well.

If subsystem maintainers do this, the collective IQ of the Drupal development community goes up, we grow the larger pool of contributors, and no patch gets left as code needs review. Then you can come back to me when the RTBC queue is 6 pages long (as opposed to cleared down to zero about once every couple of months as it has been) and say "webchick, we need subsystem maintainers to have commit rights, because these patches have all been thoroughly peer reviewed and are languishing at RTBC." and I will probably back you up 110%.

-Angie

From my point of view, this discussion has some real value, but targets other areas that could use improvement.

Braindump: It's not about the speed of patches being committed. It's about core patches lacking review, and core evolving in ways and areas that leave contrib behind. It's about contrib development not being able to catch up with core. Core development is driven by some core developers. Only a few of them being contrib maintainers. But the real applications live in contrib. Contrib developers need to catch up with new core APIs and features before they can think about and work on improving core. But next to catching up with that, they also have to maintain, solve issues, and develop new stuff for their projects/applications. By doing so, they identify areas in core that need improvement. At the time they realize what needs improvement, new core APIs are released, so they start from scratch. Contrib should influence and drive core development. Currently, I experience the opposite. I have to decide whether I want to improve my XYZ module or whether I want to work on core without taking into account what XYZ module would really need in X months when new core is released. If I do both, neither core nor my module is mature. Gotta go. sun

On 20-Apr-09, at 2:04 PM, Earl Miles wrote:

Angela Byron wrote:

...

We have this kind of decentralized development model, where one or two people are solely responsible for code with basically zero peer review. It's called contrib. And it's notoriously filled with sub- standard, shoddily documented code that needs to be closely inspected by individual site maintainers before being deployed on any serious production sites.

I stop following your argument here. The only way this argument holds up is that if *all* of contrib is substandard crap. It is not. There is a level of contrib that is above that, and there is a reason those particular pieces of contrib are above that. Chew on that.

And this level contrib is the kind that has enough users so that peer reviews occur naturally, or at the very least peer QA after the fact. As we've noted, peer review and peer QA is *not* happening with these core subsystems, which makes it a lot closer to the "darker, scarier" contrib. So I'm not sure what there is to chew on. :) -Angie

On 20-Apr-09, at 2:20 PM, Earl Miles wrote:

Angela Byron wrote:

...

On 20-Apr-09, at 2:04 PM, Earl Miles wrote:

...

Angela Byron wrote:

...

We have this kind of decentralized development model, where one or two people are solely responsible for code with basically zero peer review. It's called contrib. And it's notoriously filled with sub-standard, shoddily documented code that needs to be closely inspected by individual site maintainers before being deployed on any serious production sites.

I stop following your argument here. The only way this argument holds up is that if *all* of contrib is substandard crap. It is not. There is a level of contrib that is above that, and there is a reason those particular pieces of contrib are above that. Chew on that. And this level contrib is the kind that has enough users so that peer reviews occur naturally, or at the very least peer QA after the fact. As we've noted, peer review and peer QA is *not* happening with these core subsystems, which makes it a lot closer to the "darker, scarier" contrib. So I'm not sure what there is to chew on. :)

Perhaps you attribute more peer review and QA to some of these modules than is truly there. I don't think there is very much. There is peer review pretty much after the fact; that's the review that said "it's ok to use this module."

But the responsibility of the committer is what made that happen in the first place. I personally believe that anyone who is given the right to commit directly to Drupal core is going to think several times longer before committing something than he or she would to contrib. Obviously this varies based on personality, but the suggestion that adding more committers to core will turn core into contrib does not work for me.

I guess I view it as the following simple pseudo-query, assuming an active maintainer: $relative_project_quality = db_result(db_query('SELECT (number of active participants in issues, filtering out '+1 subscribe') FROM {the_issue_queue} WHERE project = %d', $project->nid)). On a project such as Views, with a very high number, $relative_project_quality will be very high. On a project such as any of mine ;) with a very low number, code quality will be much lower. This is not because I'm a horrible coder or that I'm not conscientious about my work or that I don't care about quality. It's because there's no body of diverse people chiming in to say "actually that will break my use case and here's why," or "I actually find that a bit hard to grasp; what if we did it this way instead?" There's no one looking over my shoulder for obvious brain-os that I miss in the height of my caffeine-powered coding rush. And furthermore, even if I *wanted* to bounce ideas off of more experienced developers or peers (which I do!), I quite simply can't; they don't have expertise in this space. So I do what most contributed module authors do, which is muddle through and try my best. Now, if you do the above query with project = 3060, you will get an extremely healthy project. And by having only a small number of committers, through whom *all* changes are run, this number holds true. However, If we move to a subsystem maintainer committing model, we need to change that query for the Drupal project so that it's WHERE project_component = %d. And if you do /that/, you do not get a rosy Views-esque picture of things like the menu system. You get a very worst-case scenario picture of webchick-maintained contributed modules. So my argument is quite simply that without building this number much higher for each of the underlying subsystems, I don't see what choice core has *but* to turn into the dark, scary kind of contrib by adding more committers to the mix. -Angie

On Mon, Apr 20, 2009 at 2:48 PM, Angela Byron <drupal-devel@webchick.net> wrote:

So my argument is quite simply that without building this number much higher for each of the underlying subsystems, I don't see what choice core has *but* to turn into the dark, scary kind of contrib by adding more committers to the mix.

-Angie

I have to take issue with this in as much as any added committers would almost certainly not (as you do not) commit their own patches. If we were to move towards the notion of subsystem committers I would imagine that they would have to be in close communication with you and Dires about the overall direction and API design, get one or more serious reviews in addition to their own review, and probably would have much less time for writing patches as opposed to reviewing and committing them. In other words, they would behave w.r.t. a subset of the code in much as we observe you (and Gabor, and Neil) behaving. -Peter

Gerhard Killesreiter wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Earl Miles schrieb:

...

Angela Byron wrote:

...

We have this kind of decentralized development model, where one or two people are solely responsible for code with basically zero peer review. It's called contrib. And it's notoriously filled with sub-standard, shoddily documented code that needs to be closely inspected by individual site maintainers before being deployed on any serious production sites. I stop following your argument here. The only way this argument holds up is that if *all* of contrib is substandard crap. It is not. There is a level of contrib that is above that, and there is a reason those particular pieces of contrib are above that. Chew on that.

Because they follow core's development model?

They don't, actually, or at least, mine don't. I don't post patches and have half a dozen people review them before I commit code. If someone posts a patch, chances are there's only 1 or 2 reviews on it, and I have to do the real review prior to deciding whether or not to commit it. Sometimes some patch will be both simple enough and important enough for more than a couple people to review it, but that is the exception, not the rule. Right now, it's because I try to hold myself to a high standard. And I admit that I can be sloppy and commit too fast due to the workload, but ultimately even with that I do a better job than a lot of contrib, simply because I try to see the bigger picture prior to committing. The reason I think more committers in targeted areas will work is because I think it will create more activity in those areas. The people who have the right to commit will have a more vested interest, and therefore you will get more of their time, effort and energy, plus any that they can draw to them. webchick commented earlier that one of the issues was providing an incentive to do reviews. And that's certainly a key issue, right there. There isn't much incentive to do reviews. There's the good of the project, but there is no pride of ownership in a review. The pride of ownership drives a lot of the initial development, and it drives the committers, but the reviewers? I cannot remember a time in the 4 years I've been with this project that we haven't had a longstanding complaint about lack of good patch reviews. That is one of the first complaints I heard then, and it is one of the complaints I hear now. It has not changed, and asking people to do more reviews has already proven to be ineffective in combatting the issue. Give your reviewers some ownership and you'll find a lot more people interested in reviewing.

On Monday 20 April 2009 14:08:42 Karoly Negyesi wrote:

...

The people who have the right to commit will have a more vested interest

There isn't much incentive to do reviews. There's the good of the project, but there is no pride of ownership in a review. The pride of ownership drives a lot of the initial development, and it drives the committers, but the reviewers?

I am giving up my neutral position (already...) and I must say: I agree.

I *am* sloppy but if I am given the keys to Drupal core? I will second-guess every decision of mine for sure.

Hear hear for that. Nothing like the knowledge that if you commit a screw-up, it'll be noted by core devs within hours (if not minutes), which any self- respecting dev would find at least a smidge humiliating. When reading this little part of the thread, I got this mental image of Angie and Dries each driving enormous vans along a precarious cliffside road, trying to listen to a cacophony of conflicting navigational advice (ranging in quality from "local expert guide" to "pin the tail on the donkey") mixed in with a healthy dose of griping and "ARE WE THERE YET?!?" Not intended to paint us as whiny or unhelpful. My only point: personally, I got a lot better at navigating after I started driving.

I think catch hits a lot of good points about how our reward and trust system could be improved. I'm all for better profile pages, giving credit where credit is due, and providing better visibility into who is working on what. It might be useful to provide a "Reviewer rating" or "Reviewer score" -- something which could be an incentive for more people to review patches. Maybe this can be accomplished through some project module extensions? I'm not sure what the best approach would be, but it is worthy of a discussion. On Mon, Apr 20, 2009 at 10:30 PM, Nathaniel Catchpole <catch56@googlemail.com> wrote:

On Mon, Apr 20, 2009 at 8:38 PM, Sam Boyer

...

Hear hear for that. Nothing like the knowledge that if you commit a screw-up, it'll be noted by core devs within hours (if not minutes), which any self- respecting dev would find at least a smidge humiliating.

This is interesting, because I find it really comforting that we manage to pick up screw-ups both before and after commit, especially mine, maybe I'm not self-respecting enough. I don't really like the experience of coming across an issue which just got committed and have to mark it code needs work (sometimes with rollback patch ready if it's really, really bad) - but when people do it to me I don't get offended or overly humiliated - I'm just glad it never made it into a final release.

Actually I think a big part of the problem is this perception that core is hard and contrib is easy, personally I find the contrib workflow really hard.

A lot of people's first attempt to contribute to Drupal is often when they apply for a CVS account to contribute back a module. A lot of companies view that as their primary way to contribute back resources as well. What we end up with is 4,000 projects of very varying quality, and the projects which /everyone/ uses - Views, CCK etc. are crying out for people to help in their issue queues - not just writing and reviewing patches at a high level, but answering support requests, closing out duplicates, stuff like that.

One issue with this is the way our user profiles work in terms of highlighting contributions. I'd been contributing to core for more than a year before I got a CVS account, and that was when one of my core patches removed a slice of user.module on the condition it was maintained in contrib. So for a long time I never had any projects listed on my profile page, nor any entries in 'track code'. I've still only got commit access on a handful of projects, and don't actively maintain any of them - a situation I'm quite happy with. However if I posted a couple of crappy modules and committed loads of tiny patches to them, then to the casual observer that'd make the user profile look a lot beefier.

The only way that code contributions to core get measured is when Greg Knaddison parses commit logs and puts everything into a spreadsheet once or twice a year - our commit model also completely breaks sites like ohloh. Personally I'd really like better statistics on how many people are reviewing/submitting patches, getting them committed etc. since it'd at least help to spot clear trends when having discussions like these. And I think it'd be a bit of incentive to review and post patches more if there was some way of having that data recorded more firmly than just in commit messages themselves.

There was talk of having 'issue queue maintainers' without actual commit access for contrib projects as well - for people who answer lots of support requests and write documentation etc. Similar issue and it goes two ways. Identifying the people who you need to speak to about a particular area is hard unless you know who they are. Identifying what any particular individual is doing is hard unless you trawl through their tracker. There ought to be better ways to deal with this than filing patches to add people to maintainers.txt and having to write big long lists of names in commit messages (then what if you forgot someone or spell their name wrong).

Nat

-- Dries Buytaert :: http://buytaert.net/

On Monday 20 April 2009 23:19:03 Dries Buytaert wrote:

In fact, a lot of big projects, like the Linux kernel, Gnome, KDE, Ubuntu, Fedora and Wordpress all switched to 6 month release cycles. Mark Shuttleworth posted some good insights about that in a recent blog post: http://www.markshuttleworth.com/archives/288. I'm not suggesting that we should switch to 6 month release cycles, I'm merely bringing it up to get feedback on the idea. I don't think any of those projects have 6 month release cycles that break backwards compatibility, which I think major drupal releases do...

Marijn

On Monday 20 April 2009, Marijn Kruisselbrink wrote:

On Monday 20 April 2009 23:19:03 Dries Buytaert wrote:

...

In fact, a lot of big projects, like the Linux kernel, Gnome, KDE, Ubuntu, Fedora and Wordpress all switched to 6 month release cycles. Mark Shuttleworth posted some good insights about that in a recent blog post: http://www.markshuttleworth.com/archives/288. I'm not suggesting that we should switch to 6 month release cycles, I'm merely bringing it up to get feedback on the idea.

I don't think any of those projects have 6 month release cycles that break backwards compatibility, which I think major drupal releases do...

I think that's just another argument in favor of Robert's suggested move of core modules to contrib. The Form/Fields/File/whatever APIs deserve longer release cycles so that contrib has at least some chance to stick with feature work for a single core version only. However, for the modules built on top of the APIs (Forum, Aggregator, etc.) nobody really cares about API compatibility, those are the ones that would actually benefit from significantly shorter release cycles. While I'm not yet convinced that stripping down Drupal to a bare framework is the right approach, I do think that the two aspects of core should get different treatment. -- Jakob

Steven Peck wrote:

To my knowledge, every module removed from Drupal core has died. I use several of the modules on Roberts list (blog, aggregator, blogapi) and if they were removed, then I would be out of luck as one of the few remaining 'hobbyist' users.

And Stefan, I too miss the old front page where it wasn't all site implementations and Drupalcons but things people found interesting, but that's just what happened.

I don't believe this would be true if we were still packaging them together, but treating them differently. Think of the Drupal Framework as the Linux Kernel, and then the Drupal distribution as Linux. The framework includes only the APIs -- node, user, base object stuff. The foundation. Drupal then provides applications built upon that foundation. It starts with just one: Something resembling Drupal as we know it today. We split core, and we devote effort to the most important projects. This might allow us to better focus efforts, anyway. One of the issues I see is that core itself is too big. Neither of the two committers really understand everything they've been committing, lately. (If you doubt me...field API =)

Or we can have one Drupal Legacy Module that we can suport the modules of previous drupal release. Pros: * Lots of previous module will work. * The "contrib developer" have more time to decide to upgrade the module. (instead 1 years will have 2 years or more) * More users will update their Drupal version. * I think with that, instead contrib modules became more not up to date, will became more up to date. because with this we can use the modules with the new version of drupal, so the process of migrating modules will became more faster.. Why? because some developers wait until all others modules (that they use) became migrated, before start.. Cons: * Modules that use this module will run more slow * Some Modules will not run and others have some break's. -- ------------------------------------- Marco Sousa

See the joomla way: Legacy Mode: *http://tinyurl.com/cxr45d* -- ------------------------------------- Marco Sousa

On Tue, Apr 21, 2009 at 2:02 PM, Gerhard Killesreiter < gerhard@killesreiter.de> wrote:

Feel free to provide such a module in contrib and package it as part of an install profile. Or just use Joomla.

Gehard, this comment is useful? Please stop that. I just want to give more alternatives (used by one CMS), to we have all variables to choose the better one.. I'm not inflame nothing... -- ------------------------------------- Marco Sousa

Or we can have one Drupal Legacy Module that we can suport the modules of previous drupal release.

No we can't. The API changes in Drupal 7 include things like changing the way modules are defined in .info files to make them compatible with the registry, and changing the syntax of every single database query. It's impossible to make a module which would allow you to run a Drupal 6 module on Drupal 7 with no changes, and that's a very, very good thing. It's already been said once in this thread, but the reason it took so long for contrib to catch up to Drupal 6 was a massive refactoring of CCK, Views, wysiwyg API and other modules all the same time - and the fact that contrib has a very complex web of dependencies (baz module extends foo module which extends bar module, so can't do an update until both bar and foo are upgraded). A Views 2 which was backwards compatible with Views 1 wouldn't be half as much fun - and if you just want to use the same D5 modules you were using already without any of the advantages, why upgrade at all? We don't make API changes for fun, we make them because we have to - either to correct previous mistakes or open up new possibilities. If an API doesn't need to change then it usually doesn't - for example hook_menu() is pretty much exactly the same between 6 and 7. All the effort talking about backwards compatibility should instead be focused on http://drupal.org/project/deadwood and http://drupal.org/project/coder - and also on the Field API and Views related patches to core which will be necessary for field, views, and views-dependent modules to upgrade to D7. Coder already has an initial port to Drupal 7 and does a great deal of the work required to update a module. The better it gets, the less work module maintainers (or upgrade patch contributors) need to do to get a basic port of their module going. Combined with unstable releases, it's quite possible to start upgrading modules to Drupal 7 now - in fact there's 7 pages of modules which have some kind of port in the works, months before code freeze - something we've never, ever had before. http://drupal.org/project/modules?filters=drupal_core:103&solrsort=sort_titl... Nat

2009/4/21 Jakob Petsovits <jpetso@gmx.at>:

...

Or we can have one Drupal Legacy Module that we can suport the modules of previous drupal release.

This is proposed by someone for every Drupal version, it has been tried several times, and it has never worked for reasons that have been discussed at length in previous threads on this list. Please search the archives to read up those arguments.

Last night, just before I went to sleep, I was thinking if it would be good to have CHANGES.txt accompanied by DIDNOTCHANGE.txt to aid porting of contrib modules. In some cases people might even be able to have xyz.module and xyz.d6.inc as well as xyz.d7.inc. xyz.module would contain everthing mentioned in DIDNOTCHANGE.txt. But maybe this is silly... Eric

Quoting Jakob Petsovits <jpetso@gmx.at>:

On Tuesday 21 April 2009, Marco Sousa wrote:

...

Or we can have one Drupal Legacy Module that we can suport the modules of previous drupal release.

This is proposed by someone for every Drupal version, it has been tried several times, and it has never worked for reasons that have been discussed at length in previous threads on this list. Please search the archives to read up those arguments.

If we really go with module-supported "automatical" compatibility, the forward-facing way (deadwood) is much more promising than the legacy one.

I started to suggest the legacy issue yesterday but stopped because I realize that supporting legacy becomes an unending task that just postpones forever the upgrade to the newer API. I agree the deadwood module is a great benefit. One desire I would have with it is to report in foo.deadwood things of caution instead of inserting comments into foo.module. Yea, yea, I know a feature request in the deadwood issue queue. -- Earnie -- http://r-feed.com/ -- http://for-my-kids.com/ -- http://www.4offer.biz/ -- http://give-me-an-offer.com/

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Steven Peck schrieb:

To my knowledge, every module removed from Drupal core has died. I use several of the modules on Roberts list (blog, aggregator, blogapi) and if they were removed, then I would be out of luck as one of the few remaining 'hobbyist' users.

There are more hobbyist users than you think. They just are not on this list. I don't think that removing current core modules to the contrib repo is a good idea. It would screw up the upgrade path for thousands of existing sites. Things like the blog module simply do not need much development, they are feature complete. Adding bells and whistles would only make them harder to use.

And Stefan, I too miss the old front page where it wasn't all site implementations and Drupalcons but things people found interesting, but that's just what happened.

Yeah, I've read enough site implementations for the rest of my life I think. Cheers, Gerhard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkntcgUACgkQfg6TFvELooRw5QCdFu/6omXbEZnJbEi5vVblgPpR m9sAoMaV49l1DrFNfDZqaGkWPsccTXfI =YVlr -----END PGP SIGNATURE-----

On Tue, Apr 21, 2009 at 12:31 AM, Stefan Nagtegaal <development@standoutdesign.nl> wrote:

Op 21 apr 2009, om 08:20 heeft Steven Peck het volgende geschreven:

...

<snip></snip>

And Stefan, I too miss the old front page where it wasn't all site implementations and Drupalcons but things people found interesting, but that's just what happened.

-sepeck

Happy to hear that Sepeck, sometimes I wonder if it is just me who is experiencing this. Drupal is not for users anymore, drupal is for making money and profit. One example is the "Support the Drupal redesign! Donate Now"-button, which is currently cycling over the frontpage. Why can't we - as a community - build the new drupal.org design? It gives us a good reality check why drupal is (too) hard for true designers.

Now now, the redesign is complex and time consuming. People are busy, so outside perspective is useful.

That said, you could wonder that if some module is moved from core to contrib and dies silently, if it wasn't in core too long already. If nobody feels the urge to keep a certain module up to date, then you could argue about its usefullnes.

I still use them. I know a number of people I support do as well. If it's been working fine then it hasn't needed much attention, but we are still have a large non-professional, minimal coding group of people that use Drupal so I would rather see a gradual enhancement/generalization of core to absorb these remaining specialized modules.

I would rather see modules like views(1|2|3).module in core, which are very well designed and maintained and deserves to be in core, because of it's quality and filling the gap which there currently is in core. Building views/page-layouts is something drupal needs. Although, I do not think that the current implementation is right, the functionality surely is.

Different discussion. But on par with how CCK migrated into core as it and the community matured around it.

Getting rid of modules like the help.module is not that smart imo. Dries started a monster project to make drupal more accessible and easier to use by attracting Mark Boulton and Leisa, which I think is a good thing. Although, I do not doubt that they could make things easier to use, but I'm pretty convinced they could not make things as easy that the help texts are becoming redundant.

I liked the old built in help system, but we've gone 3 different ways since then.

Kind regards,

Stefan

Steven

Two thoughts occurred to me on the community. First, we are no longer what the community was 3 or 5 years ago. The front page of drupal has changed with a changing focus and need. Instead of a few thousand users there are hundreds of thousands of users. The front page of d.o has a different place in the ecosystem than it used to. But, we still need a place for some of those things that used to get posted. Is g.d.o the place for that? Second, we seem to have a bit of an identity crisis. Is drupal a framework or a CMS? If it's both should they be cleanly separated? How does this and should this work? Moving the CMS like features into contrib, as some have suggested, seems to be centered around this. If we have a drupal framework and a drupal cms do we have maintainers for each piece? How does this interaction work? Ah, growing pains. Now we just have to figure out how to bribe the kids to clean their rooms (i.e. review patches and clean the queue). On Apr 21, 2009, at 3:31 AM, Stefan Nagtegaal wrote:

Op 21 apr 2009, om 08:20 heeft Steven Peck het volgende geschreven:

...

<snip></snip>

And Stefan, I too miss the old front page where it wasn't all site implementations and Drupalcons but things people found interesting, but that's just what happened.

-sepeck

Happy to hear that Sepeck, sometimes I wonder if it is just me who is experiencing this. Drupal is not for users anymore, drupal is for making money and profit. One example is the "Support the Drupal redesign! Donate Now"- button, which is currently cycling over the frontpage. Why can't we - as a community - build the new drupal.org design? It gives us a good reality check why drupal is (too) hard for true designers.

That said, you could wonder that if some module is moved from core to contrib and dies silently, if it wasn't in core too long already. If nobody feels the urge to keep a certain module up to date, then you could argue about its usefullnes.

I would rather see modules like views(1|2|3).module in core, which are very well designed and maintained and deserves to be in core, because of it's quality and filling the gap which there currently is in core. Building views/page-layouts is something drupal needs. Although, I do not think that the current implementation is right, the functionality surely is.

Getting rid of modules like the help.module is not that smart imo. Dries started a monster project to make drupal more accessible and easier to use by attracting Mark Boulton and Leisa, which I think is a good thing. Although, I do not doubt that they could make things easier to use, but I'm pretty convinced they could not make things as easy that the help texts are becoming redundant.

Kind regards,

Stefan

Quoting Matt Farina <matt@mattfarina.com>:

Is drupal a framework or a CMS? If it's both should they be cleanly separated?

Drupal is both a framework and a CMS. The framework itself is geared toward managing dynamic content. The parts of Drupal that make it a traditional CMS must remain with the core of Drupal because that gives Drupal better definition.

How does this and should this work? Moving the CMS like features into contrib, as some have suggested, seems to be centered around this. If we have a drupal framework and a drupal cms do we have maintainers for each piece? How does this interaction work?

Since the framework is geared toward a CMS how can we separate all of those pieces. There may be some modules dwelling in core that may be moved but certainly not those that pertain to a minimal CMS. -- Earnie -- http://r-feed.com/ -- http://for-my-kids.com/ -- http://www.4offer.biz/ -- http://give-me-an-offer.com/

On Mon, Apr 20, 2009 at 10:19 PM, Dries Buytaert wrote:

Something else which is worth to discuss.

In private conversations, various people have expressed concerns that our release cycles are too long, and that, as a result, they are not interested in working on core.  If they have a problem, and they fix it in core, they have to wait 1-2 years before they can actual use their own improvement in a production environment.  In a lot of cases, people don't bother with it because they can't wait that long.

In other words, moving to shorter release cycles could help.  Not only does it increase the incentive for people to work on core, it could also provide additional focus.

With testing, dbtng and Field API I think it would have been impossible to get them all in during the six month release cycle, and hard to do any of them properly in that time either (unless they were committed the first week) - actually writing tests and converting modules has taken a long time, and that's been a good thing as all three APIs are maturing during the process. However, having made all those big changes in one release, it'd be really interesting to consider a shorter cycle for Drupal 8. Some things which are looking less likely for Drupal 7 (views in core, a non-crappy comment and forum module) would be great to have in a shorter timespan than another two year release cycle. Nat

In fact, a lot of big projects, like the Linux kernel, Gnome, KDE, Ubuntu, Fedora and Wordpress all switched to 6 month release cycles. Mark Shuttleworth posted some good insights about that in a recent blog post: http://www.markshuttleworth.com/archives/288.  I'm not suggesting that we should switch to 6 month release cycles, I'm merely bringing it up to get feedback on the idea.

On Mon, Apr 20, 2009 at 11:03 PM, Dries Buytaert <dries.buytaert@gmail.com> wrote:

...

I think catch hits a lot of good points about how our reward and trust system could be improved.  I'm all for better profile pages, giving credit where credit is due, and providing better visibility into who is working on what.  It might be useful to provide a "Reviewer rating" or "Reviewer score" -- something which could be an incentive for more people to review patches.  Maybe this can be accomplished through some project module extensions?  I'm not sure what the best approach would be, but it is worthy of a discussion.

-- Dries Buytaert :: http://buytaert.net/

Quoting Gerhard Killesreiter <gerhard@killesreiter.de>:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Dries Buytaert schrieb:

...

blog post: http://www.markshuttleworth.com/archives/288. I'm not suggesting that we should switch to 6 month release cycles, I'm merely bringing it up to get feedback on the idea.

I'd be very happy if we could at least break the trend towards longer and longer release cycles.

I can imagine that this could happen with fewer features added in one cycle. Things like dbTNG that requires a whole rewrite of queries will take a longer cycle and should be the only feature change for that release. Other changes like fields in core could be released earlier. This though requires a different paradigm in version control. -- Earnie -- http://r-feed.com/ -- http://for-my-kids.com/ -- http://www.4offer.biz/ -- http://give-me-an-offer.com/

Quoting David Metzler <metzlerd@metzlerd.com>:

I think think the right answer about drupal stagnating was really about having Views, Panels, and WYSIWG api all going through major refactoring on D6, all lagging significantly the D6 release. I know it's why I'm not using my own D6 ported modules in any production sites yet.

And to avoid this the module developer needs to agree to port the existing functionality of a module from one version to the next while the new features of the module are being developed. The deadwood and coder modules can help with that and should be one of the first modules upgraded out of the door. New functionality in a module is great but once a Drupal version is out and people are knocking on the modules door for "when is Dx going to be supported" carrying over the old functionality to the new Dx should have a priority over a more powerful, better looking model. D6 still lags because modules are improving their API and developers aren't willing to put out a module that is carries the sins of the older D5 version. A worthy goal but so is getting a module that is working just like the previous. This is really a sore sticking point making using Drupal sour in some eyes. I agree that Drupal has a nice feature rich framework for building many different applications (modules). I agree that the CMS work is what has drawn many to Drupal. However, a focus on CMS often drowns the other usefulness of the framework and you have a situation similar to HORDE/IMP. Does anyone think framework when they hear HORDE? Maybe as a second thought but usually you think an email client. Unfortunately when people here Drupal they think CMS and not framework. Most users of the CMS don't care that the framework of Drupal or the framework of their favorite module is now more feature rich at first. They care that their sites will work exactly the same as it does before upgrading. Then they might look at the newer features. There are more users of Drupal than there are hackers providing code, peer reviews and module functionality. Yes, these are important but not to the non-technical user of the product. People are tired of waiting on module maintainers and probably will go elsewhere or worse create their own version of the module. This is why it is important for contrib modules to be readily available within a month of a new Dx release. -- Earnie -- http://r-feed.com/ -- http://for-my-kids.com/ -- http://www.4offer.biz/ -- http://give-me-an-offer.com/

On Monday 20 April 2009 4:03:08 pm Dries Buytaert wrote:

I think catch hits a lot of good points about how our reward and trust system could be improved. I'm all for better profile pages, giving credit where credit is due, and providing better visibility into who is working on what. It might be useful to provide a "Reviewer rating" or "Reviewer score" -- something which could be an incentive for more people to review patches. Maybe this can be accomplished through some project module extensions? I'm not sure what the best approach would be, but it is worthy of a discussion.

Dear god this is a monster thread... :-) I'm going to jump in here for lack of anywhere better to do so. As others have noted, the problem isn't the speed of commits, it's the speed of reviews and the ability of sufficient people to grok them to review them usefully. I would add a related problem: The speed of decision making. Some have suggested giving subsystem maintainers commit access. I am not sure that's a good idea, given Drupal's current architecture. It's still too intertwined, with poor separation between systems. Hooks are well and good but they don't separate subsystems from each other. (I freely admit that one of my ulterior motives with the handlers project is to encourage further separation between components, making it easier to grok them separately, test them separately, and develop them separately, even in contrib. I hope to get back to that soon, as I've been heavily distracted by personal matters of late.) For subsystem committers to work, we would need to have a way for there to be layered levels of commit. That is, as others have mentioned before, the Linux model. Linus almost never reviews or even sees individual patches. He gets periodic rollups from subsystem maintainers, reviews them in general but not at a line-by-line level, and adds in a great deal of trust that said subsystem maintainer is doing his job and that what he's getting has already been vetted by someone who understands, say, the power management system far better than he does. It's the power management subsystem maintainer's job to go line by line, not Linus'. We could adopt that sort of model without giving subsystem maintainers core commit access, if and only if 1) We switched to a distributed VCS, because CVS really can't handle that. Really, I see greater autonomy for submaintainers and a DVCS for core as fundamentally linked; doing one without the other is pointless. 2) We need to trust that subsystem maintainers have and are implementing long-term planning that makes sense, and then let them sweat the line-by-line details. As webchick noted earlier, and as I discussed with her in IRC, we need more "maintainer teams" like we have for the DB system. The DB system is sort of an oddball case, though, since it breaks into subcomponents so easily (each driver is its own subcomponent). Of course, if we want to get more people to be domain experts, whether formally recognized or not, there needs to be an incentive. "Pleeeeeese???" is our current incentive model, and I don't think it works all that well. :-) Informal karma only goes so far. What is the incentive for people to become and remain domain experts? Now that I have finally managed to get query builders into core so that I never have to deal with the frickin' INSERT query syntax again, what's the incentive for me to remain active and on top of the DB system? (Not that I'm planning to wander off and ignore it, mind you; I'm just using that as an example.) I first became the DB maintainer right before Szeged. In Szeged, I asked Dries and Angie what they wanted me to do, exactly. "OK, so what did I just sign up for?" Their answer was "just keep doing what you're doing". That didn't really make much sense to me, as what I'd been doing was developing a several hundred KB patch over the course of a year and a half and not sleeping for two weeks in order to get it committed and I really didn't want to keep doing things that way. :-) I've not gotten a particularly different answer since. Sometimes I see "I'd like to get Crell's thoughts on this issue" type comments, and sometimes I get emails with an issue that a core committer wants me to have a look at, but usually I know about them already because I have the DB issue queue bookmarked. I don't mean that to sound like a whiny complaint. The point is that if I, as a current submaintainer / domain expert, don't really know what the benefit is for being one besides feeding my own ego, then what incentive is there for anyone else to become a domain expert on FAPI, or to give forum.module the extensive love it desperately needs? Which brings me to something else that Dries said earlier about the micro vs. macro level. Looking at the macro level and planning things out to ensure consistency and so forth is very important. It also has a name, which is even dirtier to mention around here than "backward compatibility" but I shall do so anyway: A roadmap. Roadmaps help ensure that small tasks fit together into a cohesive whole, and that everyone knows what that whole is. They also give people a clear task list of things that need to be done and WHY. That makes life easier for developers and for reviewers, because they can see why we're making this change to a given piece of code; it's a small change that is part of a larger effort, and the larger effort makes sense. OK, call them community initiatives or whatever; that's still what they are. We're seeing more of them appear haphazardly in the issue queue with multi-part issues that reference each other. That's a very good thing. Drupal is, really, past the size at which "scratch an itch somewhere" can be the primary organizational mechanism, especially if we actually want clean and consistent APIs. OK, so we don't necessarily need one huge roadmap for all of core. Given our development model I doubt that would work anyway. But, if we tone down the anti-planning rhetoric and instead allow for smaller, subsystem roadmaps or particular campaign roadmaps (like Fields-ifying core), that gives people more context in which to review, as well as more documentation on why something is being done, which again makes it easier to review. Then, allow submaintainers to direct roadmaps. If someone wants to become the big fish in the little pond for blog.module and do amazing things with it, and it's someone that has already shown that they are reliable in general, OK. Let's see what your plan (roadmap) is. Discuss and approve it in general. Let them work on it and also be the go-to person for anyone wanting to work on it to make sure that that particular piece of core is consistent and clean and robust. That's their responsibility... and also their authority. When something comes down to a subjective opinion question, the submaintainer's opinion needs to carry more than usual weight. (Objective measures like performance, sure, benchmarks win. I'm referring to judgment calls.) Then the subsystem maintainers, together with the maintainers (Dries and Angie for D7), coordinate to ensure consistency and cleanliness and elegance across core, so that the API decisions for node.module and the API decisions for comment.module and the decisions for FAPI all mirror each other where appropriate, which in turn leads to more consistent and therefore more learnable APIs, which lowers the barrier to entry for future reviewers, etc. Positive feedback loops are a good thing. -- Larry Garfield larry@garfieldtech.com

On Monday 20 April 2009, Nathaniel Catchpole wrote:

The only way that code contributions to core get measured is when Greg Knaddison parses commit logs and puts everything into a spreadsheet once or twice a year - our commit model also completely breaks sites like ohloh.

It's not the commit model that breaks Ohloh but the fact that we're using a distributed version control workflow with a centralized one (CVS) as tool. That's a mismatch, because most projects using centralized VCSs don't work that way. In fact, iirc then Linus Torvalds dropped CVS in favor of Bitkeeper (later Git) because the former did not work with his development model. Now they're working as a bunch of subsystem teams with each subsystem maintainer keeping track of his version of the Linux kernel, while the final word is still spoken by Linus who has a lot of say in terms of direction and the "large picture" (or should I say "macro-problems"). Sound familiar? Of course, even current DVCSs treat reviewed-by and approved-by as part of the textual message instead of core concept. As long as it's standardized though, Ohloh should be able to deal with it :)

On Monday 20 April 2009 15:30:03 Nathaniel Catchpole wrote:

One issue with this is the way our user profiles work in terms of highlighting contributions. I'd been contributing to core for more than a year before I got a CVS account, and that was when one of my core patches removed a slice of user.module on the condition it was maintained in contrib. So for a long time I never had any projects listed on my profile page, nor any entries in 'track code'. I've still only got commit access on a handful of projects, and don't actively maintain any of them - a situation I'm quite happy with. However if I posted a couple of crappy modules and committed loads of tiny patches to them, then to the casual observer that'd make the user profile look a lot beefier.

The only way that code contributions to core get measured is when Greg Knaddison parses commit logs and puts everything into a spreadsheet once or twice a year - our commit model also completely breaks sites like ohloh. Personally I'd really like better statistics on how many people are reviewing/submitting patches, getting them committed etc. since it'd at least help to spot clear trends when having discussions like these. And I think it'd be a bit of incentive to review and post patches more if there was some way of having that data recorded more firmly than just in commit messages themselves.

Heh. So I had a project that I started on last fall which would do exactly this kind of thing - collect richer statistics on drupal participation - and I'm gonna have to finish if I want to get my degree. Which reminds me that I've been meaning to ask if someone couldn't point me towards any colossal discussion(s) we've had about this kind of thing of thing in the past, so that I can digest what's already been thrown around. cheers Sam

On 20-Apr-09, at 2:44 PM, Earl Miles wrote:

The reason I think more committers in targeted areas will work is because I think it will create more activity in those areas. The people who have the right to commit will have a more vested interest, and therefore you will get more of their time, effort and energy, plus any that they can draw to them. webchick commented earlier that one of the issues was providing an incentive to do reviews. And that's certainly a key issue, right there. There isn't much incentive to do reviews. There's the good of the project, but there is no pride of ownership in a review. The pride of ownership drives a lot of the initial development, and it drives the committers, but the reviewers?

I cannot remember a time in the 4 years I've been with this project that we haven't had a longstanding complaint about lack of good patch reviews. That is one of the first complaints I heard then, and it is one of the complaints I hear now. It has not changed, and asking people to do more reviews has already proven to be ineffective in combatting the issue.

Give your reviewers some ownership and you'll find a lot more people interested in reviewing.

Now this "how to get more reviewers" is a line of discussion I'm *very* interested in having since, regardless if we have subsystem committers or not, we need to resolve it. Here are some ideas around that: 1. A community-led mentorship/journeyman program for people who want to get involved in core development and don't know where to start. Basically, taking what we do at code sprints at Drupalcons twice a year that always lead to a big increase in new core contributors, and making an e-version of it, or something that could be done in person at camps and local meetups. 2. Developing some sort of system that can formalize a bit more "patch review swaps." The biggest incentive people currently have for reviewing a patch is that they want one in core and need to get other reviewers to look at it in order to do so. Making it easier to identify people who are willing to engage in this would help everyone out. 3. Providing tools on g.d.o, d.o, or elsewhere to make it easier for core subsystem maintainers to nurture their pocket of Drupal core. Not really sure what those would be yet. Perhaps one aspect could be liaising with the documentation team to help get their subsystem documentation cleaned up and some clear step 1, step 2, step 3 stuff. 4. Monthly contests for "review queue smashing," like other projects have "bug fix days" or similar. Winners get their name in lights on the d.o front page, or we could pony up for some cash incentives or similar. 5. Calling specific attention out to people who did thorough reviews in commit messages, either by including them in the list of contributors or something like "#xxxx by dev1, dev1, and reviewed by rev1: wah wah wah." 6. Add more people to MAINTAINERS.txt, and have it reflect the current committer "trust" model. If that document can be trusted, this makes it easier for reviewers to know who to talk to if they're interested in helping out with X system. We could also add subsystem maintainer names to the pages on http://drupal.org/community-initiatives/drupal-core perhaps. Curious about others people might have as well. -Angie

On Apr 20, 2009, at 3:39 PM, Angela Byron wrote:

On 20-Apr-09, at 2:44 PM, Earl Miles wrote:

...

The reason I think more committers in targeted areas will work is because I think it will create more activity in those areas. The people who have the right to commit will have a more vested interest, and therefore you will get more of their time, effort and energy, plus any that they can draw to them. webchick commented earlier that one of the issues was providing an incentive to do reviews. And that's certainly a key issue, right there. There isn't much incentive to do reviews. There's the good of the project, but there is no pride of ownership in a review. The pride of ownership drives a lot of the initial development, and it drives the committers, but the reviewers?

I cannot remember a time in the 4 years I've been with this project that we haven't had a longstanding complaint about lack of good patch reviews. That is one of the first complaints I heard then, and it is one of the complaints I hear now. It has not changed, and asking people to do more reviews has already proven to be ineffective in combatting the issue.

Give your reviewers some ownership and you'll find a lot more people interested in reviewing.

Now this "how to get more reviewers" is a line of discussion I'm *very* interested in having since, regardless if we have subsystem committers or not, we need to resolve it.

A huge disincentive for getting involved in core is that it doesn't yield immediately deployable results. This keeps shops and individuals from putting more labor towards core work. Robert Douglass suggested on this thread to remove modules that don't absolutely need to be in core (aggregator, blog, blogapi, forum). I absolutely concur. This would not just make core easier to overlook and remove a huge tax from the drupal core infrastructure, but it would most importantly allow us to release and deploy these modules quicker. Like mentioned already by others on this thread, these modules could create a new tier of Drupal modules on d.o. that would be packaged with installer profiles and maintained for core compatibility for every release of the current Drupal release branch and Drupal HEAD. In my mind, such a third tier of modules would fill the gap between core and contrib: it would create a peer reviewed space between Drupal core and Drupal contrib while not suffering from long release cycles. In turn we would get more reviewers for the new tier of modules but I also for Drupal core as there is going to be more activity on modules that will be available in the current release branch *and* in HEAD. A third tier of Drupal modules does not solve all issues of course, but I think it makes our problem *a lot* smaller.

Here are some ideas around that:

1. A community-led mentorship/journeyman program for people who want to get involved in core development and don't know where to start. Basically, taking what we do at code sprints at Drupalcons twice a year that always lead to a big increase in new core contributors, and making an e-version of it, or something that could be done in person at camps and local meetups.

2. Developing some sort of system that can formalize a bit more "patch review swaps." The biggest incentive people currently have for reviewing a patch is that they want one in core and need to get other reviewers to look at it in order to do so. Making it easier to identify people who are willing to engage in this would help everyone out.

3. Providing tools on g.d.o, d.o, or elsewhere to make it easier for core subsystem maintainers to nurture their pocket of Drupal core. Not really sure what those would be yet. Perhaps one aspect could be liaising with the documentation team to help get their subsystem documentation cleaned up and some clear step 1, step 2, step 3 stuff.

4. Monthly contests for "review queue smashing," like other projects have "bug fix days" or similar. Winners get their name in lights on the d.o front page, or we could pony up for some cash incentives or similar.

5. Calling specific attention out to people who did thorough reviews in commit messages, either by including them in the list of contributors or something like "#xxxx by dev1, dev1, and reviewed by rev1: wah wah wah."

6. Add more people to MAINTAINERS.txt, and have it reflect the current committer "trust" model. If that document can be trusted, this makes it easier for reviewers to know who to talk to if they're interested in helping out with X system. We could also add subsystem maintainer names to the pages on http://drupal.org/community-initiatives/drupal-core perhaps.

Curious about others people might have as well.

-Angie

Alex Barth http://www.developmentseed.org/blog tel (202) 250-3633

Quoting Bryan Ollendyke <bto108@psu.edu>:

lol yeah, I believe the intent of the original author was to hold off on D7 until D6 matured more from a crontrib standpoint. I'll +1 that idea. D5 was and still is a very mature platform that you can develop and build off of, seemingly without "need" to upgrade if you have a site up and running stablly; this isn't always the case with 6 with some of the more major projects just getting to full releases in the last 6 months.

Especially when you have issues with core modules causing issues with cron actually executing. See http://drupal.org/node/436028 for what I'm referring to. -- Earnie -- http://r-feed.com/ -- http://for-my-kids.com/ -- http://www.4offer.biz/ -- http://give-me-an-offer.com/

Eric Schaefer wrote:

Why am I not participating in reviews of core patches? Naked fear. How could I possibly understand whats going on in core? Fire up your debugger (i use eclipse + pdt + zend debugger) and step through the major operations of the bootstrap process to discover the big picture. Then walk through the loading of a node that is decorated with cck fields and taxonomy terms. Finish up your tour with the editing of a form (node, admin) to get a good look at fapi and the form processing involved in form.inc and you have a VERY strong foundation from which to build. Most people involved in core don't understand 100% of it. Many specialize in a particular area. This is why we have reviews and the testbot. One reason for this is the fact that core is a moving target. This is a conscious and purposeful decision on the part of the project it will not change.

If I am not understanding the USE of those APIs how can I judge patches to the code behind them? api.drupal.org lists each function and in the declaration is detailed everywhere that function is also used in core. This is a good start but your IDE can help you alot more if you use one especially if you put a breakpoint in your function and start debugging to see how various functions arrive at the location and what the variables are set to at that time.

Please take the time and try it just once. Install D7, apply a couple patches, review their changes and you'll be surprised how quickly you fall in place. We want your help. I normally find that future contributors are simply scared away by the challenge certainly not actually unable to rise to it. Pick a patch that seems interesting to you. If you ever need help find me in IRC and I'll always be happy to lend a hand. -mf

I agree with Earl here. Several of my patches, including quite big popular patches that people have bought me beers for, have only got in because I'm a stubborn git - either re-rolling them 50 times due to patch conflicts or writing reams of arguments to get them committed. Not to mention nagging people time after time on irc to get reviews and/or advice. Plenty of other people give up (either on the individual patch or core as a whole), and then those patches either sit in the queue for years or sometimes get revived by another developer, and then they can still sit in the queue for another year or two. This doesn't mean that /all/ lingering patches are lingering for the wrong reasons, but there's a lot of overhead getting patches in other than just writing the code. Having said that, none of this overhead exists for reviewing patches - and it's slow review turnover which leads to a lot of the lingering, and being someone who reviews a lot of patches is one of the easiest ways to get your own patches reviewed faster. On Mon, Apr 20, 2009 at 7:07 PM, Earl Miles wrote:

Dries Buytaert wrote:

...

As core matures, we might see more patches getting stale.  I think the reason is (at least) two-fold:

 1) More patches compete for attention, but fewer of these patches are truly important.  In this case, patches getting stale is not necessarily a bad thing because history has proven that important patches will eventually bubble to the top for the right reasons.

If the 'right reasons' are because the developers got frustrated with the process and quit pushing their patch, whereas other developers are extremely stubborn and managed to push their patch through anyway, then sure. But just because some developer is bullheaded enough to wade through the crap doesn't mean that's the right reasons.

Plenty of other people give up (either on the individual patch or core as a whole),

Note: this is very very and a very sad important fact.

Karoly Negyesi wrote:

...

Plenty of other people give up (either on the individual patch or core as a whole),

Note: this is very very and a very sad important fact.

next fact: there won't be more good core reviewers joke1: your old feature patch is still not in core. Don't worry "There are at least two modules for that!" joke2: create another account on d.o and RTBC your own patch, then at least the core committer will give you a quick review on the changes. joke3: if you need a core patch review, then you should create a blog post about it first. Károly: sorry, I will not review patches created for the menu module, as I still do not know it enough, and no time to learn it. I have a huge list what should I work on, as many Drupal developers have, but core development is not on the list any more.. I don't know anyone other than Peter who would make the time and would give you a good review. I trust you, so your changes should be in the module. My real advice, if it is possible then create a contrib module which is better than the one in core. I will switch without hesitation.. My reviews will be submitted as bug reports. If something does not work you will know it sooner or later.. Advanced menu would be a good name..

Gerhard Killesreiter wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Earl Miles schrieb:

...

Dries Buytaert wrote:

...

As core matures, we might see more patches getting stale. I think the reason is (at least) two-fold:

1) More patches compete for attention, but fewer of these patches are truly important. In this case, patches getting stale is not necessarily a bad thing because history has proven that important patches will eventually bubble to the top for the right reasons. If the 'right reasons' are because the developers got frustrated with the process and quit pushing their patch, whereas other developers are extremely stubborn and managed to push their patch through anyway, then sure. But just because some developer is bullheaded enough to wade through the crap doesn't mean that's the right reasons.

Right or wrong: this is not new. You always had to be persistent (or stubborn) in order to push through larger changes.

I don't disagree with this. I disagree with the interpretation: Calling 'being stubborn' the right reasons is not an indication that the right patches float to the top automatically. It is a little bit like argument trickle-down economics: Giving money to the rich enriches the poor.

On Mon, Apr 20, 2009 at 1:07 PM, Earl Miles <merlin@logrus.com> wrote:

Dries Buytaert wrote:

...

As core matures, we might see more patches getting stale.  I think the reason is (at least) two-fold:

 1) More patches compete for attention, but fewer of these patches are truly important.  In this case, patches getting stale is not necessarily a bad thing because history has proven that important patches will eventually bubble to the top for the right reasons.

If the 'right reasons' are because the developers got frustrated with the process and quit pushing their patch, whereas other developers are extremely stubborn and managed to push their patch through anyway, then sure. But just because some developer is bullheaded enough to wade through the crap doesn't mean that's the right reasons.

That, and time. Core committers, as the DCDC comment noted, are those with more time to devote to the effort. It's tough to find the time to contribute to core because the entry point is high, but additionally because it's painful to spend lots of time on a core patch, only to see it languish. The same amount of effort can create and maintain several contrib modules.

If code is gold, then documentation is platinum. -----Original Message----- From: development-bounces@drupal.org [mailto:development-bounces@drupal.org] On Behalf Of Earl Miles Sent: Monday, April 20, 2009 2:15 PM To: development@drupal.org Subject: Re: [development] Very concerned over Drupal's core development Dries Buytaert wrote:

weigh in and participate in the review process. It is also important that you convince other people as why your patch should bubble up -- this is something a lot of people can get better at.

This statement right here is what's wrong with Drupal core development. At this time, you must be better at debate, arguing and general Drupal politics than you are at actual code design and implementation. At one point 'talk is silver, code is gold' was the motto. But this has changed. Now talk is gold and code is defined by those who talk the best.

I agree with Earl and Darrel, many patches are derailed with endless debates. The question is whether that is good or bad, because the context is missing. I can't speak much about the past, but is it possible that core's development process changed drastically? We introduced an awesome testing framework that ought to increase the quality of code that is committed to core. However, we are still introducing new core bugs (take "FAIL." [1] as an example), despite being reviewed by many eyeballs and having tests. I think this is caused by the steadily increasing complexity of Drupal core - and - by a lack of reviewers that *really* understand core, use-cases in contrib, many sub-systems (not one), and all implications of a core patch. This is the real issue (I believe) Karoly wanted to raise originally. I am not sure whether sub-system maintenance teams will lead to an improvement of this situation. Completely contrary to this proposal, I think our issue is that we are lacking the opposite of domain experts: Drupal core contributors/maintainers/reviewers who understand every single bit of core and potential implications of a changed bit. For the sake of clarity, I have revised Dries' "Drupal learning curve" graph, because I think the important part - the part we are really talking about - is missing: http://www.unleashedmind.com/files/drupal-learning-curve.png Adding new people above the "Core contributor" threshold is what we are mostly doing currently. However, it is my impression that there are very, very few people above the "Trustworthy for core maintainers" treshold. Even less, if you only count people who are known to be "trustworthy" for more than one sub-system. It is possible that you counted only one person (or less?) when climbing up the further tresholds. Maybe that is the issue we are facing? Patches languish in the queue for ages, because some sub-system maintainers subsequently chime in, each one needs own treatment, other contributors chime in, trying to understand the issue, the patch is re-rolled all over again, a branch maintainer shows up who questions the entire approach, and finally the patch may or may not be committed -- still lacking review by someone who understands all changes and implications of the patch. Given all other proposals in this thread, my guess is that adding more maintainers to core might really be a step in the right direction: Handing out responsibility (that comes with a fair amount of power) to core contributors who are already known to be trustworthy (in more than one area) will most likely let them advance in other areas, climbing up the "treshold-ladder" - not only, because people will start to ask about reviews/commits for arbitrary patches. Equally, this action could be done now. [1] http://drupal.org/node/446742 Thanks, sun (who hopes to have reached macro-level with these thoughts)