[development] Test automation

[Gerhard Killesreiter]

Derek Wright wrote:
> overall, i'm excited about this work, and think it'll be very cool 
> (both for drupal.org, and for other sites i'm working on to manage 
> software development)...
>
> On Jun 15, 2006, at 4:52 PM, Rok Žlender wrote:
>
>> 1. Testing server queries project server for all untested patches.
>> This will require a xml-rpc handler in project module which would
>> generate a list of untested patches or patches marked "patch (needs
>> automated test)" and return it as a response.
>
> given how issues and patches are currently handled, this is a very 
> ambiguous query to attempt to fulfill:
>
> a) patches are just files attached to issue comments.  there's no 
> enforcement of a given naming convention, etc.  it's therefore hard to 
> separate patches worth testing from screenshots and other random 
> attachments of all sorts.
>
> b) it's entire issues that are set to a given status, not individual 
> comments.  a given issue might have 10+ comments with different 
> patches attached to them, some of which have been reviewed and already 
> vetoed, etc.
>
> c) the *last* patch in a given issue isn't always the one people care 
> about.  sometimes a patch from early in the thread ends up being the 
> best way to solve something, even if other patches are attached in 
> later comments that attempt other approaches.
>
> because of all of this, even marking an issue "patch (needs 
> auto-tests)" doesn't really help much.  that'll make it easier to find 
> all the issues that need testing, but not the individual patches.
>
> similarly, all this makes the reporting hard, too.  when the testing 
> server comes back to post a followup saying "patch foo.txt results: 
> FAILED", that might be comment #45 in a long issue, but foo.txt might 
> be from early in the the thread.  it'd be nice to be able to group 
> them more directly.  instead of a new follow-up, *maybe* we want to 
> edit the existing comment and add this additional info (though that 
> won't show up as an update in the tracker, etc).  i dunno, it's a 
> sticky problem.
>
> i haven't thought about all of this long enough to have a proposal for 
> how it should all work.  but, i wanted to at least share my concerns 
> so others who have a chance can keep this stuff in mind when thinking 
> about this.
>
> some brainstorming...
>
> 1) project should use threaded comments, not it's own chronological 
> followups?  it'd help to group replies to specific comments more 
> nicely, but it also might be harder to follow.

Casetracker has threaded comments. There is ongoing disscussion on 
whether this is a good idea.


Cheers,
    Gerhard