[development] Test automation

[Rok Žlender]

Hello

I am working on unit test automation for this year Summer of Code.
After discussing possible solutions with some people on IRC I feel
that this mailing list is the right place to get some response on the
subject.
Here is the general idea how unit test automation will look like in the end.

Testing server - server which prepares the environment for tests and runs them
Project server - server with project module installed and where all
the issues and patches are gathered

1. Testing server queries project server for all untested patches.
This will require a xml-rpc handler in project module which would
generate a list of untested patches or patches marked "patch (needs
automated test)" and return it as a response.

2. Testing server tests every patch on the list. Testing is done in a
virtual server.

3. When a patch is tested testing server notifies project server with
result. Again this would require another xml-rpc handler inside
project module that would save the result into DB. Because there would
be too much information to display as a follow up all the detailed
information about a certain test should be saved on testing server.
This is an example how testing result could be displayed in follow up:
Automated test result for file: [file_name] PASS| FAIL.
[link_to_detailed_information].
Testing server could also gather some statistics regarding the tests
for example number of passes/fails etc.

Please feel free to comment and give ideas.

Cheers
Rok Žlender