[development] Test automation
gerhard at killesreiter.de
Fri Jun 16 04:53:34 UTC 2006
Derek Wright wrote:
> overall, i'm excited about this work, and think it'll be very cool
> (both for drupal.org, and for other sites i'm working on to manage
> software development)...
> On Jun 15, 2006, at 4:52 PM, Rok Žlender wrote:
>> 1. Testing server queries project server for all untested patches.
>> This will require a xml-rpc handler in project module which would
>> generate a list of untested patches or patches marked "patch (needs
>> automated test)" and return it as a response.
> given how issues and patches are currently handled, this is a very
> ambiguous query to attempt to fulfill:
> a) patches are just files attached to issue comments. there's no
> enforcement of a given naming convention, etc. it's therefore hard to
> separate patches worth testing from screenshots and other random
> attachments of all sorts.
> b) it's entire issues that are set to a given status, not individual
> comments. a given issue might have 10+ comments with different
> patches attached to them, some of which have been reviewed and already
> vetoed, etc.
> c) the *last* patch in a given issue isn't always the one people care
> about. sometimes a patch from early in the thread ends up being the
> best way to solve something, even if other patches are attached in
> later comments that attempt other approaches.
> because of all of this, even marking an issue "patch (needs
> auto-tests)" doesn't really help much. that'll make it easier to find
> all the issues that need testing, but not the individual patches.
> similarly, all this makes the reporting hard, too. when the testing
> server comes back to post a followup saying "patch foo.txt results:
> FAILED", that might be comment #45 in a long issue, but foo.txt might
> be from early in the the thread. it'd be nice to be able to group
> them more directly. instead of a new follow-up, *maybe* we want to
> edit the existing comment and add this additional info (though that
> won't show up as an update in the tracker, etc). i dunno, it's a
> sticky problem.
> i haven't thought about all of this long enough to have a proposal for
> how it should all work. but, i wanted to at least share my concerns
> so others who have a chance can keep this stuff in mind when thinking
> about this.
> some brainstorming...
> 1) project should use threaded comments, not it's own chronological
> followups? it'd help to group replies to specific comments more
> nicely, but it also might be harder to follow.
Casetracker has threaded comments. There is ongoing disscussion on
whether this is a good idea.
More information about the development