I know I do not yet have much credibility in the Drupal community, so I apologize in advance for making sweeping statements. I think the Drupal process could really benefit from a culture of obsessive automated testing. I have been programming for over 15 years, and one of the most valuable mottoes I have heard is "If a feature is not covered by an automated test, it does not exist". Practically, if something like Simpletest was in core, then the requirement for submitting a patch could be increased to also submitting a test that exposes the original bug & shows that it has been fixed. Skilled users could submit tests that expose the bug as part of the original issue submission. And submitting original tests could be a great way for new users to contribute. As the coverage of the submitted tests grows, developers will be able to have more and more confidence that the patch they are submitting does not cause regressions. Programming on complex systems is just so much more fun, when you know you are not breaking someone else's hard work. A comprehensive test framework can become a safety net, supporting Drupal's rapid evolution. Right now it seems like Drupal succeeds because it has such a robust community, which is willing to act as a test system, but that approach runs the risk of exhausting everyone involved. I know everyone has way too much on there plates right now to really consider culture change, but I wanted to try and get this idea out there. If there is one gift I can give to the Drupal community, out of half a lifetime of experience with many programming languages and cultures, this is it. Thanks for listening. -- Tao Starbow Web Architect, CITRIS
On Mar 6, 2006, at 9:33 AM, Tao Starbow wrote:
Skilled users could submit tests that expose the bug as part of the original issue submission. And submitting original tests could be a great way for new users to contribute.
My first full-time programming job was maintaining unit and regression tests for 1M+ C++ line code base with over 20 different types of software services for medical records. If something slipped through people could get the wrong diagnosis or the wrong medication. I agree, unit test are valuable. However, maintaining unit tests could be as difficult as building core itself initially. Maintaining unit tests attracts a different kind of personality type than people who are interested in adding new features. We need a way to appeal to the motivations of testers. I think coming up with a small number of tests maintained by a few people is the way to approach this. We just had a fun chat in #drupal about what the most basic tests for Drupal core should be. 1) Test checkboxes -apparently they were broken some 41 times in head 2) Changing Filter input to PHP type for security 3) Page edit 4) Bootstrap checking 5) Update.php checking I think a grand scheme to write hundreds of tests or force core contributors to write unit tests with their patches will fail. Worse it could delay emerging efforts for a small suite of maintained tests. If anyone is interested in maintaining at least one test over the 4.7 release cycle let's get together and see if we can provide a useful maintained suite that can be run against head to identify issues and provide core developers a toolset to improve the quality of their patches. Cheers, Kieran
-- Tao Starbow Web Architect, CITRIS
medication. I agree, unit test are valuable. However, maintaining unit tests could be as difficult as building core itself initially.
I agree with Kieran here. Tests become part of the project, adding to its overall size. When there is a crunch for time or money, the tests get neglegted, and fall into disuse. This makes them totally useless, since things change over time, and the old tests are a) not enough to certify all new functions of the project, and b) old function behaviors have changed, and the tests no longer pass. So, only do automated tests if you are willing to put the time and effort to maintain them for the lifetime of the project.
Khalid B wrote:
So, only do automated tests if you are willing to put the time and effort to maintain them for the lifetime of the project.
I summarized my thoughts on this here: http://drupal.org/node/52709 (I didn't want this discussion limited to just the development list where things often disappear in a tide; yet the forums often get missed).
Wow, Merlin. That's a exciting proposal. It's nice to know people are thinking about this issue on to that level. Earl Miles wrote:
Khalid B wrote:
So, only do automated tests if you are willing to put the time and effort to maintain them for the lifetime of the project.
I summarized my thoughts on this here: http://drupal.org/node/52709
(I didn't want this discussion limited to just the development list where things often disappear in a tide; yet the forums often get missed).
On Mar 6, 2006, at 2:30 PM, Earl Miles wrote:
I summarized my thoughts on this here: http://drupal.org/node/52709
(I didn't want this discussion limited to just the development list where things often disappear in a tide; yet the forums often get missed).
Thanks for that, Earl. It matches very well with my recollection of our conversation over that lovely pineapple fried rice.
On 07 Mar 2006, at 1:16 AM, Jonathan Chaffer wrote:
On Mar 6, 2006, at 2:30 PM, Earl Miles wrote:
I summarized my thoughts on this here: http://drupal.org/node/52709
(I didn't want this discussion limited to just the development list where things often disappear in a tide; yet the forums often get missed).
Thanks for that, Earl. It matches very well with my recollection of our conversation over that lovely pineapple fried rice.
One of the features of the next version of the forms api (the actions api) would be that each action has verifiable input, output and can be programmatically called. This would make it fairly simple to write unit tests for all forms (ie: node/add/book , with these paramaters, needs to create this node object). -- Adrian Rossouw Drupal developer and Bryght Guy http://drupal.org | http://bryght.com
Op maandag 6 maart 2006 19:57, schreef Kieran Lal:
If anyone is interested in maintaining at least one test over the 4.7 release cycle let's get together and see if we can provide a useful maintained suite that can be run against head to identify issues and provide core developers a toolset to improve the quality of their patches.
Again, here is Ber-with-his-never-ending-RoR stories :) Ignore this mail if you have heard enough of sympal (developers distro) and these Ror comaprisons. :) For Sympal (the Drupal distro for devels) I have scheduled, but not yet finalised, the following environments, simply by copying the RoR functionality: * production (the live version) * dev (hacking up stuff, core is even pulled out of the multisite during this stage) * test (yea, testing. But no Real clue yet) * live -> production (Live is the one active in apache, its there to allow switching of environment w/o touching apache) The test part is very and highly fuzzy right now. My ideal is that it will contain a testing environment. Where unit-test live. A place where we can also release an early betha for clients. A place where we can fix our work on development without the risk of 'i just saw "w000t first IF ent0r3d" appear on my site, sir, what is that about' etc. The problem is, that I never used testing environments before I ran into RoR. That one has it built in, and has it so easy to use, that even I used it :). The bigger problem in that, I have got nothing to show then a garbled an restructured core, in some BZR now.. So it aint very appealing to put a lot of effort into unit-test-for-that-distro out oif there is no distro yet. So, in practice: Have you got interest in a developers distro shipping with a default (automated) Drupal test environment: please let me know, I am a virgin in that area ;) , and would love to get some clue on how it can be used in Drupal development in a structured way, in the developers distro, if possible. Ber -- [ Bèr Kessels | Drupal services www.webschuur.com ]
We just had a fun chat in #drupal about what the most basic tests for Drupal core should be.
Note to self - log in to #drupal if I am going to propose anything on the list
I think a grand scheme to write hundreds of tests or force core contributors to write unit tests with their patches will fail. Worse it could delay emerging efforts for a small suite of maintained tests.
I don't think it needs to be a grand scheme. More like encouragement. I doubt most module-level developers even know about simpletest's existence. And a set of small tests is such a powerful way to communicate: either what a patch does, or the right way to use an API...it's really seductive once you start doing it.
participants (7)
-
Adrian Rossouw -
Bèr Kessels -
Earl Miles -
Jonathan Chaffer -
Khalid B -
Kieran Lal -
Tao Starbow