Hello I am working on unit test automation for this year Summer of Code. After discussing possible solutions with some people on IRC I feel that this mailing list is the right place to get some response on the subject. Here is the general idea how unit test automation will look like in the end. Testing server - server which prepares the environment for tests and runs them Project server - server with project module installed and where all the issues and patches are gathered 1. Testing server queries project server for all untested patches. This will require a xml-rpc handler in project module which would generate a list of untested patches or patches marked "patch (needs automated test)" and return it as a response. 2. Testing server tests every patch on the list. Testing is done in a virtual server. 3. When a patch is tested testing server notifies project server with result. Again this would require another xml-rpc handler inside project module that would save the result into DB. Because there would be too much information to display as a follow up all the detailed information about a certain test should be saved on testing server. This is an example how testing result could be displayed in follow up: Automated test result for file: [file_name] PASS| FAIL. [link_to_detailed_information]. Testing server could also gather some statistics regarding the tests for example number of passes/fails etc. Please feel free to comment and give ideas. Cheers Rok Žlender
Testing server - server which prepares the environment for tests and runs them Project server - server with project module installed and where all the issues and patches are gathered
this all makes sense to me. let me know if i can be of some help. i wrote some xml-rpc handlers recently.
overall, i'm excited about this work, and think it'll be very cool (both for drupal.org, and for other sites i'm working on to manage software development)... On Jun 15, 2006, at 4:52 PM, Rok Žlender wrote:
1. Testing server queries project server for all untested patches. This will require a xml-rpc handler in project module which would generate a list of untested patches or patches marked "patch (needs automated test)" and return it as a response.
given how issues and patches are currently handled, this is a very ambiguous query to attempt to fulfill: a) patches are just files attached to issue comments. there's no enforcement of a given naming convention, etc. it's therefore hard to separate patches worth testing from screenshots and other random attachments of all sorts. b) it's entire issues that are set to a given status, not individual comments. a given issue might have 10+ comments with different patches attached to them, some of which have been reviewed and already vetoed, etc. c) the *last* patch in a given issue isn't always the one people care about. sometimes a patch from early in the thread ends up being the best way to solve something, even if other patches are attached in later comments that attempt other approaches. because of all of this, even marking an issue "patch (needs auto- tests)" doesn't really help much. that'll make it easier to find all the issues that need testing, but not the individual patches. similarly, all this makes the reporting hard, too. when the testing server comes back to post a followup saying "patch foo.txt results: FAILED", that might be comment #45 in a long issue, but foo.txt might be from early in the the thread. it'd be nice to be able to group them more directly. instead of a new follow-up, *maybe* we want to edit the existing comment and add this additional info (though that won't show up as an update in the tracker, etc). i dunno, it's a sticky problem. i haven't thought about all of this long enough to have a proposal for how it should all work. but, i wanted to at least share my concerns so others who have a chance can keep this stuff in mind when thinking about this. some brainstorming... 1) project should use threaded comments, not it's own chronological followups? it'd help to group replies to specific comments more nicely, but it also might be harder to follow. 2) comments should have their own status, in addition to the entire issue? might be handy to deal with existing ambiguities in issues with lots of patches (e.g. comments like "#4 is broken, but #5 is RTBC"). probably too complicated and weird to be practical, but maybe worth considering. 3) we should reconsider how we handle file attachments to issues. instead of associating each patch with a specific comment, we associate all of them with the issue itself (so there's just a growing table of filenames at the top of the issue), and then individual comments refer to filenames when they want to discuss stuff. (there could be an "operations" column in the table, with a "reply" link that brings you to an add follow-up form with some pointer/link to that particular filename automatically filled in for you or something). this table could have metadata associated with each file, so we could clearly mark files as patches, screenshots, etc. maybe each file in this table needs its own status (another field in the metadata), instead of each comment in the issue. the callback from the test server could just add more meta data to specific files, change the status of files, etc. food for thought... thanks, -derek (dww) p.s. unrelated to all of the above, but another potential snag you'll run into: not everyone generates patches the same way. some people do the appropriate, recommended thing, and generate the patch from the root of the drupal source tree. others make them local to a specific directory. now that we're about to move core modules from modules/foo.module to modules/foo/foo.module, patches that just touch a given module would probably be better off from the local directory, so they can apply cleanly to both 4.7 and HEAD (bug fixes, of course). so, your test server is going to have to be very smart about analyzing patch files, trying different things to get them to apply to a given copy of the drupal source. some will just be faulty, bad patches, or patches that no longer apply (which should be reported with a different status than "FAILED TESTS". we probably want (at least) "FAILED TO APPLY", "FAILED TESTS", and "PASSED" as the possible results of a given patch. p.p.s. all that got me thinking of another thing... a given patch might apply just fine today, and pass all tests. however, tomorrow, dries commits a change to HEAD which changes some API and now a) the patch no longer applies and/or b) now starts failing tests. clearly, we need some mechanism/UI for re-testing patches, reporting the history of test attempts for a given patch through time, etc.
On Jun 15, 2006, at 7:38 PM, Derek Wright wrote:
overall, i'm excited about this work, and think it'll be very cool (both for drupal.org, and for other sites i'm working on to manage software development)...
food for thought...
Ok, let's completely abandon the idea of automatically unit testing patches. Instead let's have a block on http://drupal.org/project/ issues that has a file upload. The file upload connects to a server that is available to unit test the patch. It spits back a test result number with the time the test was run and the results of the test. http://druapaluntitest.org/node/##### Instead of spending all this effort automating some thing simple we focus on making it a valuable service to the developer community. The person who posts the patch can just provide a link to the unit test result. Keep It Simple S****** Kieran
On Jun 15, 2006, at 7:56 PM, Kieran Lal wrote:
Ok, let's completely abandon the idea of automatically unit testing patches. Instead let's have a block on http://drupal.org/project/ issues that has a file upload. ... Instead of spending all this effort automating some thing simple we focus on making it a valuable service to the developer community. The person who posts the patch can just provide a link to the unit test result.
Keep It Simple S******
fair enough. ;) to avoid storing duplicate copies of the patches (and to ensure we're testing the same patch we're commenting on and reviewing in the issue queue), we might want to just input a file URL in this testing block, that points to the copy of the patch already uploaded and attached to a given issue. so, you post your comment, attach your patch, and when you want it tested, you just drag the attachment URL from your comment into the test block (which should be visible on all issue pages, not just the issue query pages). however, many of the ambiguities and issues i raised in my previous message are true whether or not you're considering automated patch testing. i agree, it's probably not worth Rok's effort to solve all of them before getting *something* working for automated patch testing, and the upload block (or equivalent) seems like a fine place to start. certainly, he could get the block working first, and some day down the road when the other issues are solved in a more appealing way, we can worry about more direct automation between uploading a patch to an issue and having it tested. -derek (dww)
At 7:56 PM -0700 15/6/06, Kieran Lal wrote:
Ok, let's completely abandon the idea of automatically unit testing patches.
This is a useful fallback position, but the initial proposal to have patches automatically unit tested really rocks. If there's some way to get all uploaded patches automatically tested, I'd love to see that! Of course this begs the question... where are all the extra unit tests going to come from :) ...R.
Ok, let's completely abandon the idea of automatically unit testing patches. Instead let's have a block on http://drupal.org/project/ issues that has a file upload. The file upload connects to a server that is available to unit test the patch.
It spits back a test result number with the time the test was run and the results of the test.
Instead of spending all this effort automating some thing simple we focus on making it a valuable service to the developer community. The person who posts the patch can just provide a link to the unit test result.
Keep It Simple S******
Kieran
On the first view your approach has one advantage: Developers can post patches before they "release" them to the issue tracker. Well but this is actually even more easy to do by just locally installing the simpletest framework. So you can test your modifications even before generating the patch! While the narrowed approach would still be almost just as useful I worry it will not be used that much. It simply is more work to: - post patch to test server - wait for result (automated testing does take significant time) - check result - post patch to the issue tracker including a link in contrast to: - post patch (that you think is good) to issue tracker - check result once its out (just while checking for other human replies) I am not suspecting the devel community to be lazy but I know from myself that I would only do this with the bigger patches. Testing generally is a boring thing, thats why we want to automate it (which in contrast is way more interesting). This may lead to some smaller patches going through untested and thus possibly breaking the system by making a fresh HEAD fail the tests, this is a very bad situation for future testing. That is why I believe the forced test process is way more valuable. The amount of work needed for the reduced solution is less, no doubt. - Thomas
Derek Wright wrote:
overall, i'm excited about this work, and think it'll be very cool (both for drupal.org, and for other sites i'm working on to manage software development)...
On Jun 15, 2006, at 4:52 PM, Rok Žlender wrote:
1. Testing server queries project server for all untested patches. This will require a xml-rpc handler in project module which would generate a list of untested patches or patches marked "patch (needs automated test)" and return it as a response.
given how issues and patches are currently handled, this is a very ambiguous query to attempt to fulfill:
a) patches are just files attached to issue comments. there's no enforcement of a given naming convention, etc. it's therefore hard to separate patches worth testing from screenshots and other random attachments of all sorts.
b) it's entire issues that are set to a given status, not individual comments. a given issue might have 10+ comments with different patches attached to them, some of which have been reviewed and already vetoed, etc.
c) the *last* patch in a given issue isn't always the one people care about. sometimes a patch from early in the thread ends up being the best way to solve something, even if other patches are attached in later comments that attempt other approaches.
because of all of this, even marking an issue "patch (needs auto-tests)" doesn't really help much. that'll make it easier to find all the issues that need testing, but not the individual patches.
similarly, all this makes the reporting hard, too. when the testing server comes back to post a followup saying "patch foo.txt results: FAILED", that might be comment #45 in a long issue, but foo.txt might be from early in the the thread. it'd be nice to be able to group them more directly. instead of a new follow-up, *maybe* we want to edit the existing comment and add this additional info (though that won't show up as an update in the tracker, etc). i dunno, it's a sticky problem.
i haven't thought about all of this long enough to have a proposal for how it should all work. but, i wanted to at least share my concerns so others who have a chance can keep this stuff in mind when thinking about this.
some brainstorming...
1) project should use threaded comments, not it's own chronological followups? it'd help to group replies to specific comments more nicely, but it also might be harder to follow.
Casetracker has threaded comments. There is ongoing disscussion on whether this is a good idea. Cheers, Gerhard
On Jun 15, 2006, at 9:53 PM, Gerhard Killesreiter wrote:
Casetracker has threaded comments. There is ongoing disscussion on whether this is a good idea.
right. DaveNotik also submitted http://drupal.org/node/56292 since he wants to be able to automatically flatten comments for casetracker nodes, but allow threaded comments for others. ;) so, it's clear this isn't necessarily a good idea. as i said, i was just brainstorming. -derek (dww)
p.s. unrelated to all of the above, but another potential snag you'll run into: not everyone generates patches the same way. some people do the appropriate, recommended thing, and generate the patch from the root of the drupal source tree. others make them local to a specific directory. You are absolutely right. I was browsing through submitted patches and I noticed this problem. But every patch has the name of the file that is it for written somewhere. If we know that the patch is for drupal core or some specific module this narrows down the directories in which to search for the appropriate file. I think this could be done.
p.p.s. all that got me thinking of another thing... a given patch might apply just fine today, and pass all tests. however, tomorrow, dries commits a change to HEAD which changes some API and now a) the patch no longer applies and/or b) now starts failing tests. I agree with you that this is a big problem. Bacause if sth changes in API it usually requires tests to be rewritten. But this also means if changes to API are tested together with tests for most important modules they can FAIL the tests and module maintainers can get some feedback what they need to change.
Ok, let's completely abandon the idea of automatically unit testing patches. Instead let's have a block on http://drupal.org/project/ issues that has a file upload. The file upload connects to a server that is available to unit test the patch. This is a good idea and it bypasses a lot of problems. This way I could focus on getting the tests running. One more thing that should be in this block is for which module is this patch for because there is a lot of projects for drupal and not knowing which one the patch is for .. . well I don't see how we can test it.
Of course this begs the question... where are all the extra unit tests going to come from :) This is a big problem. There is a very small amount of tests written so far only some basic functionality. Module maintainers should became aware that tests are their tool and that they should use it.
Whatever the unit-test site is (e.g. drupalunittest.org), we isolate the CVS working copies that the tests will be tried on, preferably on their own server, or perhaps in a VPS setup (Virtual Private Server - e.g. Xen, User Mode Linux). This server should have severe firewall limitations, e.g. almost all outbound ports blocked, only able to communicate with > the server on which the unit-test site is hosted, etc. This is exactly what we had in mind. A virtual server, every patch gets a fresh cvs check out and everything gets wiped clean every night or so.
User logs in at the unit-test site. You are right that this posses a security threat. Unrevised code gets executed automatically but on the other hand if we create all these barriers it wouldn't be automatic testing any more and it looses its effect. In my opinion it would be better to tighten the security on this testing server and test everything that comes in.
Rok
Rok Žlender wrote:
Hello
I am working on unit test automation for this year Summer of Code. If I'm understanding you correctly, then strictly speaking this is not unit testing. Unit testing is done in isolation, so that you wouldn't need a web server nor the rest of Drupal, just the module in question. Checkout the PhpUnit project at http://phpunit.sourceforge.net/ for a PHP unit testing framework.
This doesn't mean I think your proposal is bad, it's just that QA is often about details, and accurately describing the type of testing is one of the more important details. It also doesn't mean I'm taking a position on using PhpUnit or doing true unit testing. I always encourage unit testing for traditional development languages, but I haven't done any in PHP, so I don't know what the pitfalls may be, if any. Gary
I am working on unit test automation for this year Summer of Code. If I'm understanding you correctly, then strictly speaking this is not unit testing.
You are right, strictly speaking we are doing a hybrid of web testing from a browsers view enhanced with direct code access (could be called unit testing) due to performance and some internal integrity checks that cannot be accieved by webtesting. I hope the naming does not confuse to much, its hard to be percise with language sometimes. - Thomas
I have created a group on groups.drupal.org named SOC: Unit test automation http://groups.drupal.org/soc-unit-test-automation. There you can find status report on some detailed information regarding automation. Cheers Rok
participants (8)
-
Derek Wright -
Gary Feldman -
Gerhard Killesreiter -
Kieran Lal -
Moshe Weitzman -
Richard Archer -
Rok Žlender -
Thomas Ilsche