[development] Test automation

[Rok Žlender]

> p.s. unrelated to all of the above, but another potential snag you'll
> run into: not everyone generates patches the same way.  some people
> do the appropriate, recommended thing, and generate the patch from
> the root of the drupal source tree.  others make them local to a
> specific directory.
You are absolutely right. I was browsing through submitted patches and
I noticed this problem. But every patch has the name of the file that
is it for written somewhere. If we know that the patch is for drupal
core or some specific module this narrows down the directories in
which to search for the appropriate file. I think this could be done.

> p.p.s. all that got me thinking of another thing... a given patch
> might apply just fine today, and pass all tests.  however, tomorrow,
> dries commits a change to HEAD which changes some API and now a) the
> patch no longer applies and/or b) now starts failing tests.
I agree with you that this is a big problem. Bacause if sth changes in
API it usually requires  tests to be rewritten. But this also means if
changes to API are tested together with tests for most important
modules they can FAIL the tests and module maintainers can get some
feedback what they need to change.

> Ok, let's completely abandon the idea of automatically unit testing
> patches.  Instead let's have a block on http://drupal.org/project/
> issues that has a file upload.   The file upload connects to a server
> that is available to unit test the patch.
This is a good idea and it bypasses a lot of problems. This way I
could focus on getting the tests running. One more thing that should
be in this block is for which module is this patch for because there
is a lot of projects for drupal and not knowing which one the patch is
for .. . well I don't see how we can test it.


> Of course this begs the question... where are all the extra
> unit tests going to come from :)
This is a big problem. There is a very small amount of tests written
so far only some basic functionality. Module maintainers should became
aware that tests are their tool and that they should use it.

> Whatever the unit-test site is (e.g. drupalunittest.org), we isolate the CVS working copies
> that the tests will be tried on, preferably on their own server, or perhaps in a VPS setup
> (Virtual Private Server - e.g. Xen, User Mode Linux). This server should have severe
> firewall limitations, e.g. almost all outbound ports blocked, only able to communicate with > the server on which the unit-test site is hosted, etc.
This is exactly what we had in mind. A virtual server, every patch
gets a fresh cvs check out and everything gets wiped clean every night
or so.

> User logs in at the unit-test site.
You are right that this posses a security threat. Unrevised code gets
executed automatically but on the other hand if we create all these
barriers it wouldn't be automatic testing any more and it looses its
effect. In my opinion it would be better to tighten the security on
this testing server and test everything that comes in.

Rok