[development] Very concerned over Drupal's core development

[Larry Garfield]

On Monday 20 April 2009 4:03:08 pm Dries Buytaert wrote:
> I think catch hits a lot of good points about how our reward and trust
> system could be improved.  I'm all for better profile pages, giving
> credit where credit is due, and providing better visibility into who
> is working on what.  It might be useful to provide a "Reviewer rating"
> or "Reviewer score" -- something which could be an incentive for more
> people to review patches.  Maybe this can be accomplished through some
> project module extensions?  I'm not sure what the best approach would
> be, but it is worthy of a discussion.

Dear god this is a monster thread... :-)  I'm going to jump in here for lack 
of anywhere better to do so.

As others have noted, the problem isn't the speed of commits, it's the speed 
of reviews and the ability of sufficient people to grok them to review them 
usefully.  I would add a related problem: The speed of decision making.  

Some have suggested giving subsystem maintainers commit access.  I am not sure 
that's a good idea, given Drupal's current architecture.  It's still too 
intertwined, with poor separation between systems.  Hooks are well and good 
but they don't separate subsystems from each other.  (I freely admit that one 
of my ulterior motives with the handlers project is to encourage further 
separation between components, making it easier to grok them separately, test 
them separately, and develop them separately, even in contrib.  I hope to get 
back to that soon, as I've been heavily distracted by personal matters of 
late.)

For subsystem committers to work, we would need to have a way for there to be 
layered levels of commit.  That is, as others have mentioned before, the Linux 
model.  Linus almost never reviews or even sees individual patches.  He gets 
periodic rollups from subsystem maintainers, reviews them in general but not 
at a line-by-line level, and adds in a great deal of trust that said subsystem 
maintainer is doing his job and that what he's getting has already been vetted 
by someone who understands, say, the power management system far better than 
he does.  It's the power management subsystem maintainer's job to go line by 
line, not Linus'.

We could adopt that sort of model without giving subsystem maintainers core 
commit access, if and only if 1) We switched to a distributed VCS, because CVS 
really can't handle that.  Really, I see greater autonomy for submaintainers 
and a DVCS for core as fundamentally linked; doing one without the other is 
pointless.  2) We need to trust that subsystem maintainers have and are 
implementing long-term planning that makes sense, and then let them sweat the 
line-by-line details.

As webchick noted earlier, and as I discussed with her in IRC, we need more 
"maintainer teams" like we have for the DB system.  The DB system is sort of 
an oddball case, though, since it breaks into subcomponents so easily (each 
driver is its own subcomponent).

Of course, if we want to get more people to be domain experts, whether 
formally recognized or not, there needs to be an incentive.  "Pleeeeeese???" 
is our current incentive model, and I don't think it works all that well. :-)  
Informal karma only goes so far.  What is the incentive for people to become 
and remain domain experts?  Now that I have finally managed to get query 
builders into core so that I never have to deal with the frickin' INSERT query 
syntax again, what's the incentive for me to remain active and on top of the 
DB system?  (Not that I'm planning to wander off and ignore it, mind you; I'm 
just using that as an example.)

I first became the DB maintainer right before Szeged.  In Szeged, I asked Dries 
and Angie what they wanted me to do, exactly.  "OK, so what did I just sign up 
for?"  Their answer was "just keep doing what you're doing".  That didn't 
really make much sense to me, as what I'd been doing was developing a several 
hundred KB patch over the course of a year and a half and not sleeping for two 
weeks in order to get it committed and I really didn't want to keep doing 
things that way. :-)  I've not gotten a particularly different answer since.  
Sometimes I see "I'd like to get Crell's thoughts on this issue" type 
comments, and sometimes I get emails with an issue that a core committer wants 
me to have a look at, but usually I know about them already because I have the 
DB issue queue bookmarked.

I don't mean that to sound like a whiny complaint.  The point is that if I, as 
a current submaintainer / domain expert, don't really know what the benefit is 
for being one besides feeding my own ego, then what incentive is there for 
anyone else to become a domain expert on FAPI, or to give forum.module the 
extensive love it desperately needs?

Which brings me to something else that Dries said earlier about the micro vs. 
macro level.  Looking at the macro level and planning things out to ensure 
consistency and so forth is very important.  It also has a name, which is even 
dirtier to mention around here than "backward compatibility" but I shall do so 
anyway: A roadmap.  

Roadmaps help ensure that small tasks fit together into a cohesive whole, and 
that everyone knows what that whole is.  They also give people a clear task 
list of things that need to be done and WHY.  That makes life easier for 
developers and for reviewers, because they can see why we're making this 
change to a given piece of code; it's a small change that is part of a larger 
effort, and the larger effort makes sense.  OK, call them community initiatives 
or whatever; that's still what they are.  We're seeing more of them appear 
haphazardly in the issue queue with multi-part issues that reference each 
other.  That's a very good thing.  Drupal is, really, past the size at which 
"scratch an itch somewhere" can be the primary organizational mechanism, 
especially if we actually want clean and consistent APIs.  

OK, so we don't necessarily need one huge roadmap for all of core.  Given our 
development model I doubt that would work anyway.  But, if we tone down the 
anti-planning rhetoric and instead allow for smaller, subsystem roadmaps or 
particular campaign roadmaps (like Fields-ifying core), that gives people more 
context in which to review, as well as more documentation on why something is 
being done, which again makes it easier to review.  Then, allow submaintainers 
to direct roadmaps.  

If someone wants to become the big fish in the little pond for blog.module and 
do amazing things with it, and it's someone that has already shown that they 
are reliable in general, OK.  Let's see what your plan (roadmap) is.  Discuss 
and approve it in general.  Let them work on it and also be the go-to person 
for anyone wanting to work on it to make sure that that particular piece of 
core is consistent and clean and robust.  That's their responsibility... and 
also their authority.  When something comes down to a subjective opinion 
question, the submaintainer's opinion needs to carry more than usual weight.  
(Objective measures like performance, sure, benchmarks win.  I'm referring to 
judgment calls.)  Then the subsystem maintainers, together with the 
maintainers (Dries and Angie for D7), coordinate to ensure consistency and 
cleanliness and elegance across core, so that the API decisions for 
node.module and the API decisions for comment.module and the decisions for 
FAPI all mirror each other where appropriate, which in turn leads to more 
consistent and therefore more learnable APIs, which lowers the barrier to 
entry for future reviewers, etc.

Positive feedback loops are a good thing.

-- 
Larry Garfield
larry at garfieldtech.com