[development] The State of Testing (was Re: Code freeze?)

[Angela Byron]

Larry Garfield wrote:
> webchick lamented the lack of "embracing testing" and that we have only a "testing brigade".  I'm actually not surprised at that.  That's known as a "QA Team" in many circles, and is a very good thing.  A semi-dedicated QA team that beats people over the head is a more efficient use of resources than trying to get everyone to be everything for every patch.  My hats off to the "testing brigade" for their ongoing work, and may they continue to thwap the rest of us when appropriate.  (Incidentally, is it my imagination or does the Testing Brigade consist mostly of former GHOP students and mentors?)  Their work is made even harder by the time needed to basically rewrite our own testing framework, since we're apparently not sticking with any existing testing framework (which seems inline with Drupal's usual policy...)

A couple things:

1. I wasn't lamenting anything. I was simply stating the reality. The 
reality is that the community hasn't remotely "embraced testing," and a 
huge part of that is because the testing brigade has had its hands more 
than full hammering out larger over-arching issues with the testing 
framework and existing tests, which I proceeded to document in the rest 
of my reply.

2. I'm fine with this reality, and at the present point in time it would 
be silly to expect anything else. If we had tried to get 800+ developers 
to "embrace testing" back in March, it would've been a complete CF. I'm 
saying that NOW, we're in a much better position to have a larger body 
of developers working on testing, thanks to the hard work of the testing 
brigade over the past several months.

3. We /are/ going to need 800+ people fluent in writing tests by the 
time the D7 release is upon us. 15-20 people simply cannot provide 100% 
(or even 10%) test coverage on their own, nor should we want that to 
happen. The testing brigade should be focusing on doing "test reviews," 
mentoring developers on how to write good tests and answering their 
questions, checking for thorny areas which need more test coverage, fix 
issues in the testing framework that are beyond regular developers' 
expertise, etc.

4. Yes, there's a strong GHOP tie-in. I think that's because most of the 
people who helped out with and participated in GHOP are the same people 
who generally help out with and participate in any community initiative. 
And the same people who didn't... well... ;)

5. The testing framework re-write I was originally very opposed to. But 
the fact is that SimpleTest had all manner of problems running with our 
code, and going in there and monkeying with the code to fix it was a 
huge time-sink every single time. The code that's there now is at least 
understandable by more than 3 people, it's fully documented, and it also 
takes far less time to run the full test suite. In general, I'd say it's 
a huge win.

6. I agree about testable APIs being another huge component of this. The 
solution of course is to grow the size of the testing brigade, and the 
size of the body of people who know how to write tests. If this area was 
more well-covered, developers who'd normally be putting efforts into API 
re-writes could resume doing so. As it is currently, however, everyone 
in the testing brigade has more than enough to do on that single area of 
focus.

-Angie