p.s. unrelated to all of the above, but another potential snag you'll run into: not everyone generates patches the same way. some people do the appropriate, recommended thing, and generate the patch from the root of the drupal source tree. others make them local to a specific directory. You are absolutely right. I was browsing through submitted patches and I noticed this problem. But every patch has the name of the file that is it for written somewhere. If we know that the patch is for drupal core or some specific module this narrows down the directories in which to search for the appropriate file. I think this could be done.
p.p.s. all that got me thinking of another thing... a given patch might apply just fine today, and pass all tests. however, tomorrow, dries commits a change to HEAD which changes some API and now a) the patch no longer applies and/or b) now starts failing tests. I agree with you that this is a big problem. Bacause if sth changes in API it usually requires tests to be rewritten. But this also means if changes to API are tested together with tests for most important modules they can FAIL the tests and module maintainers can get some feedback what they need to change.
Ok, let's completely abandon the idea of automatically unit testing patches. Instead let's have a block on http://drupal.org/project/ issues that has a file upload. The file upload connects to a server that is available to unit test the patch. This is a good idea and it bypasses a lot of problems. This way I could focus on getting the tests running. One more thing that should be in this block is for which module is this patch for because there is a lot of projects for drupal and not knowing which one the patch is for .. . well I don't see how we can test it.
Of course this begs the question... where are all the extra unit tests going to come from :) This is a big problem. There is a very small amount of tests written so far only some basic functionality. Module maintainers should became aware that tests are their tool and that they should use it.
Whatever the unit-test site is (e.g. drupalunittest.org), we isolate the CVS working copies that the tests will be tried on, preferably on their own server, or perhaps in a VPS setup (Virtual Private Server - e.g. Xen, User Mode Linux). This server should have severe firewall limitations, e.g. almost all outbound ports blocked, only able to communicate with > the server on which the unit-test site is hosted, etc. This is exactly what we had in mind. A virtual server, every patch gets a fresh cvs check out and everything gets wiped clean every night or so.
User logs in at the unit-test site. You are right that this posses a security threat. Unrevised code gets executed automatically but on the other hand if we create all these barriers it wouldn't be automatic testing any more and it looses its effect. In my opinion it would be better to tighten the security on this testing server and test everything that comes in.
Rok