The Drupal SimpleTest module has progressed far beyond the original SimpleTest we started with - we now have a more powerful, although Drupal-specific testing framework which still retains the somewhat misleading name "SimpleTest" (there's an issue somewhere to rename SimpleTest to testing). We currently do have both functional and unit tests working under the current framework - although javascript tests currently remain impossible, although there are several issues addressing that point (feel free to help!). I think what you're saying confirms that while 'far-beyond the original simpletest' - the current testing framework in Drupal still has this in common with the standard version of Simpletest -- it's primarily focused on unit/functional testing - not acceptance testing, which is a whole different focus. ( http://en.wikipedia.org/wiki/Acceptance_testing ) To help make what I'm talking about more understandable, here's a list of what selenium (for instance) offers that simpletest (and the Drupal implementation) does not: a) a top-rated browser plugin which offers an interface for scripting, exporting, and running tests specifically aimed at the browser-based aspect of a web application (again *not* unit/functional testing) (screenshots and explanation http://www.spreadfirefox.com/node/980 ) b) an additional tool, Selenium RC, which includes functionlity to automatically launch a variety of browsers and step through the tests in each browser - as an end-user would do c) testing of ajax/js ui elements But basically, our Drupal-specific extension of SimpleTest meets our needs for functional testing and unit testing so well, I don't think there's really any point to looking into other harder-to-install automated testing tools - they'll never be used by a sizable percentage of the Drupal community, whereas the SimpleTest module for Drupal requires very little in order to function properly. The current framework covers the vast majority of our testing needs; the tests are lagging far behind the testing framework here. We don't really need to rethink our framework or consider other frameworks until we've reached the limit of what the current framework can do. JavaScript tests would certainly be nice, but should we really worry ourselves with them when much important core php code remains entirely untested? Again, acceptance testing/selenium does not just = adding-javascript support to unit tests - it equals a whole other kind of testing - one I believe that a very sizable percentage of the Drupal community is interested in I'm sure - testing from the end-user/client perspective. So this would not be a matter of rethinking the existing framework - it's would be adding on to it and/or getting a way to more easily get selenium working with Drupal so that Drupal developers can include accetance test within their workflow. For anyone who doesn't have the bandwidth or the interest to work on finding solutions to better acceptance testing within Drupal I can understand - but I just wanted to point out the difference between what is in Drupal now and what I'm talking about. They are different and it's not just about what's in Drupal 'missing javascript support'. - Caleb
(Reposted because mailserve stripped blockquotes making things unreadable - apologies)
The Drupal SimpleTest module has progressed far beyond the original SimpleTest we started with - we now have a more powerful, although Drupal-specific testing framework which still retains the somewhat misleading name "SimpleTest" (there's an issue somewhere to rename SimpleTest to testing). We currently do have both functional and unit tests working under the current framework - although javascript tests currently remain impossible, although there are several issues addressing that point (feel free to help!).
I think what you're saying confirms that while 'far-beyond the original simpletest' - the current testing framework in Drupal still has this in common with the standard version of Simpletest -- it's primarily focused on unit/functional testing - not acceptance testing, which is a whole different focus. ( http://en.wikipedia.org/wiki/Acceptance_testing ) To help make what I'm talking about more understandable, here's a list of what selenium (for instance) offers that simpletest (and the Drupal implementation) does not: a) a top-rated browser plugin which offers an interface for scripting, exporting, and running tests specifically aimed at the browser-based aspect of a web application (again *not* unit/functional testing) (screenshots and explanation http://www.spreadfirefox.com/node/980 ) b) an additional tool, Selenium RC, which includes functionlity to automatically launch a variety of browsers and step through the tests in each browser - as an end-user would do c) testing of ajax/js ui elements
The Drupal SimpleTest module has progressed far beyond the original SimpleTest we started with - we now have a more powerful, although Drupal-specific testing framework which still retains the somewhat misleading name "SimpleTest" (there's an issue somewhere to rename SimpleTest to testing). We currently do have both functional and unit tests working under the current framework - although javascript tests currently remain impossible, although there are several issues addressing that point (feel free to help!).
Again, acceptance testing/selenium does not just = adding-javascript support to unit tests - it equals a whole other kind of testing - one I believe that a very sizable percentage of the Drupal community is interested in I'm sure - testing from the end-user/client perspective. So this would not be a matter of rethinking the existing framework - it's would be adding on to it and/or getting a way to more easily get selenium working with Drupal so that Drupal developers can include accetance test within their workflow. For anyone who doesn't have the bandwidth or the interest to work on finding solutions to better acceptance testing within Drupal I can understand - but I just wanted to point out the difference between what is in Drupal now and what I'm talking about. They are different and it's not just about what's in Drupal 'missing javascript support'. - Caleb
I suggested an automated javascript testing framework for gsoc, was thinking watir rather than selenium based on a few discussions about both, but it didn't go anywhere:http://groups.drupal.org/node/9511, While I'd love to see testing with Selenium or Watir or some other browser remote control that can handle stuff that simpletest or straight up javascript unit testing can't, like cwgordon said, it's a matter of effort and returns. The testing framework we have in core has the advantage that anyone can download and run tests without needing to install other software, tests can be written in PHP, and we're getting close to the tipping point where our coverage is high enough to start finding regressions before they're committed. On the other hand, and unfortunately, at the moment we don't have that many jQuery intensive patches - yes, it's really annoying trying to get cross-browser tests when a new jQuery is released, but that's once every few months - so in terms of saving click around time SimpleTest is giving higher returns at the moment. While it might appear to be a lack of interest at the moment, it's more a case of 1. a lack of resources (the 'testing brigade' is still pretty small) 2. a need to get the testing framework we actually have up to a point where it begins to actually save some work - it'll only do this when we're closer to 100% test coverage and tests are being run on patches automatically via testing.drupal.org That said - it's very clear that the core testing framework can't cover everything, so extending it would be the natural next step - and given the amount of time and work it's taken to get to where we are now, starting early surely wouldn't hurt. Nat
participants (2)
-
Caleb Gilbert -
Nathaniel Catchpole