[development] Test automation

[Jeremy Epstein]

On 6/16/06, Khalid B <kb at 2bits.com> wrote:
> The potential for abuse here is very high. Someone may craft
> a patch that executes arbitrary PHP code that can be effectively
> anything.
>
> Having a virtual server to do this will help with some scenarios,
> but  others (e.g. sending spam, crawling, phishing) can still be
> useful to the cracker ...

Very good point. I'm thinking that a tiered approval system would help
to avoid security exploits. Possible workflow for unit-testing a
patch:

0. Whatever the unit-test site is (e.g. drupalunittest.org), we
isolate the CVS working copies that the tests will be tried on,
preferably on their own server, or perhaps in a VPS setup (Virtual
Private Server - e.g. Xen, User Mode Linux). This server should have
severe firewall limitations, e.g. almost all outbound ports blocked,
only able to communicate with the server on which the unit-test site
is hosted, etc.
1. User logs in at the unit-test site.
2. User submits URL of a patch (or submits patch file itself) for unit testing.
3. System checks if this user is 'trusted'. A trusted user has to be
explicitly given trusted status, which will generally happen only once
they are known as a core contributor.
4. If the user is 'trusted', test their patch immediately, and
generate test results for them.
5. If not, put their patch in an approval queue. Site admins will
review the patch, and will only approve it if they see no security
exploits in it. User gets notified when the patch gets approved /
rejected.
6. The unit-test site posts a link to the test results (and the
outcome of the tests, e.g. 'FAILED'), as a follow-up in the relevant
drupal.org issue. Also posts a 'patch foo was rejected due to security
vulnerabilities' message as a follow-up, if applicable.

This will make it a little bit tedious for untrusted users, but
really, this is about the least we should be doing, if we're planning
to execute the code of potentially anyone.

Cheers,
Jaza.