[development] The State of Testing (was Re: Code freeze?)
drupal-devel at webchick.net
Thu Jun 26 19:25:25 UTC 2008
Larry Garfield wrote:
> webchick lamented the lack of "embracing testing" and that we have only a "testing brigade". I'm actually not surprised at that. That's known as a "QA Team" in many circles, and is a very good thing. A semi-dedicated QA team that beats people over the head is a more efficient use of resources than trying to get everyone to be everything for every patch. My hats off to the "testing brigade" for their ongoing work, and may they continue to thwap the rest of us when appropriate. (Incidentally, is it my imagination or does the Testing Brigade consist mostly of former GHOP students and mentors?) Their work is made even harder by the time needed to basically rewrite our own testing framework, since we're apparently not sticking with any existing testing framework (which seems inline with Drupal's usual policy...)
A couple things:
1. I wasn't lamenting anything. I was simply stating the reality. The
reality is that the community hasn't remotely "embraced testing," and a
huge part of that is because the testing brigade has had its hands more
than full hammering out larger over-arching issues with the testing
framework and existing tests, which I proceeded to document in the rest
of my reply.
2. I'm fine with this reality, and at the present point in time it would
be silly to expect anything else. If we had tried to get 800+ developers
to "embrace testing" back in March, it would've been a complete CF. I'm
saying that NOW, we're in a much better position to have a larger body
of developers working on testing, thanks to the hard work of the testing
brigade over the past several months.
3. We /are/ going to need 800+ people fluent in writing tests by the
time the D7 release is upon us. 15-20 people simply cannot provide 100%
(or even 10%) test coverage on their own, nor should we want that to
happen. The testing brigade should be focusing on doing "test reviews,"
mentoring developers on how to write good tests and answering their
questions, checking for thorny areas which need more test coverage, fix
issues in the testing framework that are beyond regular developers'
4. Yes, there's a strong GHOP tie-in. I think that's because most of the
people who helped out with and participated in GHOP are the same people
who generally help out with and participate in any community initiative.
And the same people who didn't... well... ;)
5. The testing framework re-write I was originally very opposed to. But
the fact is that SimpleTest had all manner of problems running with our
code, and going in there and monkeying with the code to fix it was a
huge time-sink every single time. The code that's there now is at least
understandable by more than 3 people, it's fully documented, and it also
takes far less time to run the full test suite. In general, I'd say it's
a huge win.
6. I agree about testable APIs being another huge component of this. The
solution of course is to grow the size of the testing brigade, and the
size of the body of people who know how to write tests. If this area was
more well-covered, developers who'd normally be putting efforts into API
re-writes could resume doing so. As it is currently, however, everyone
in the testing brigade has more than enough to do on that single area of
More information about the development