[development] Test automation

[Thomas Ilsche]

> Ok, let's completely abandon the idea of automatically unit testing
> patches.  Instead let's have a block on http://drupal.org/project/ 
> issues that has a file upload.   The file upload connects to a server
> that is available to unit test the patch.

> It spits back a test result number with the time the test was run and
> the results of the test.

> http://druapaluntitest.org/node/#####

> Instead of spending all this effort automating some thing simple we  
> focus on making it a valuable service to the developer community.   
> The person who posts the patch can just provide a link to the unit  
> test result.

> Keep It Simple S******

> Kieran

On the first view your approach has one advantage:
Developers can post patches before they "release" them to the issue
tracker.
Well but this is actually even more easy to do by just locally
installing the simpletest framework. So you can test your
modifications even before generating the patch!

While the narrowed approach would still be almost just as useful I
worry it will not be used that much.
It simply is more work to:
- post patch to test server
- wait for result (automated testing does take significant time)
- check result
- post patch to the issue tracker including a link

in contrast to:
- post patch (that you think is good) to issue tracker
- check result once its out (just while checking for other human
  replies)

I am not suspecting the devel community to be lazy but I know from myself
that I would only do this with the bigger patches. Testing generally
is a boring thing, thats why we want to automate it (which in contrast
is way more interesting).
This may lead to some smaller patches going through untested and thus
possibly breaking the system by making a fresh HEAD fail the tests,
this is a very bad situation for future testing.
That is why I believe the forced test process is way more valuable.

The amount of work needed for the reduced solution is less, no doubt.

- Thomas