[development] Code freeze?

[Angela Byron]

Dries Buytaert wrote:

 > I do think we're doing great work on testing.  I've been committing
 > testing related patches on a (almost) daily basis.  That's pretty sweet.

Like catch and others, I've been dumping basically all the time I have 
available to work on D7 on testing stuff, in the hopes that it will help 
extend the code freeze out to when I will have much MORE time to dump 
into D7 in the fall. ;)

 > That said, we aren't able to measure our test coverage yet.  In other
 > words, it's really hard to tell how well we actually do.  Any updates on
 > that?

I think that our current test coverage by conventional tools is going to 
be close to 0%. It's also clear that the community has not in fact 
"embraced testing." Instead, a hardcore group of around 15-20 people 
(the "testing brigade" ;)) have, and are driving this effort forward on 
behalf of the rest of the development team.

I also think that both of these situations are okay right now. Because 
what the "testing brigade" has been focusing on is:

1. Improvement of testing tools. For example, that Batch API patch that 
was just committed I think is the *key* turning point that will make 
testing something that can be done by normal humans rather than just the 
"testing brigade."

2. Fixing of existing tests. Back when I wrote 
http://webchick.net/itch-of-the-week/fix-testing-crisis, we were in 
pretty sorry shape in this regard. I remember growing the critical bug 
queue by at least 20 in one night documenting all the tests that didn't 
pass. Now, we're down to 3 failing tests as of this morning.

3. Improving coverage of tests that run through the end-user experience 
via the browser. This helps save our reviewers from getting carpal 
tunnel clicking on forms to ensure that the basic system is running 
properly while they're testing a patch.

4. Developing testing guidelines, best practices. We're not totally 
there yet, but a fairly large amount of time has been spent on things 
like documentation, clean-up of tests so that they all follow basically 
the same conventions, etc. This type of work is important to getting new 
developers on the testing train.

However, all of this "clean-up" work has come at the expense of 
dramatically increasing our testing coverage. But I actually think that 
it would've been way too premature to dump a bunch of time into 
increasing our test coverage while the above 4 points were outstanding. 
My goal is for the "Awesome Testing Party" at Drupalcon Szeged (if it 
gets accepted) to help a lot in this regard.

-Angie