One of my specialties (or at least something I get roped into doing fairly often ;)) is testing things with a fine-toothed comb. I have no formal background in testing methodologies or anything like that at all, but in my experience proper testing involves both: a) Figuring out all the various bits of functionality each sub-set of Drupal offers and making sure they all work. b) Intentionally throwing invalid stuff at the system or using it in unexpected ways to make sure it recovers. Over last weekend, chx and I were tag-teaming the forthcoming D6 issue queue; I was going through all aspects of core in detail identifying critical issues, chx was rolling patches for them, and I would then review/RTBC the patches he came up with. About halfway through, I decided it might be useful to document all the stuff that I was testing, and came up with this document: http://groups.drupal.org/node/5974 It's pretty rough at this point, and I was hoping to get much further with it before I posted it to the list, but I figured better early than never. ;) It's a wiki page, and I hope that others will contribute to extending the list to cover all of core. As new bugs come up, we can add tests that cover them to the list. Among the things we could do with this: - Use it purely as a means of cataloguing all of the bugs that have broken core in one way or another. Before a release, we "spot check" a few of these trouble areas and make sure they're good to go. - Before any release, we split up the various sections among individuals, so that each person can take a component (node system, user system, etc.) and make sure all of the tests there pass. This way we have no more of this "one person trying to test all of core by herself" business again. ;) - We can also use this as a check-list for our library of SimpleTest automated tests (or Selenium or whatever). While it takes awhile to write all of these, once they were written "in theory" no human would ever need to test those aspects again. If anyone can suggest improvements or wants to help fill out this list, by all means do! This is one of those tasks that literally anyone can help with, regardless of your Drupal experience, and it's a great way to get into that "critical eye" mindset that can help make you a good tester. Let's kick those critical bugs to the curb! :) -Angie
On Sep 7, 2007, at 10:06 AM, Angela Byron wrote:
b) Intentionally throwing invalid stuff at the system or using it in unexpected ways to make sure it recovers.
At UW-Madison's CS department (my day job) one of the groups has done a lot of work with "fuzz testing" [1]. They write code to automatically generate random input to any kinds of programs you wish to test. It's a great way to catch bugs in your error handling. Such tests would be fantastic for all aspects of the Drupal UI (FAPI, menu/path system, settings pages, etc), and should be relatively easy to automate. Any volunteers? Thanks, -Derek (dww) [1] http://pages.cs.wisc.edu/~bart/fuzz/fuzz.html
On 9/7/07, Angela Byron <drupal-devel@webchick.net> wrote:
Yay!
- Use it purely as a means of cataloguing all of the bugs that have broken core in one way or another. Before a release, we "spot check" a few of these trouble areas and make sure they're good to go.
- Before any release, we split up the various sections among individuals, so that each person can take a component (node system, user system, etc.) and make sure all of the tests there pass. This way we have no more of this "one person trying to test all of core by herself" business again. ;)
- We can also use this as a check-list for our library of SimpleTest automated tests (or Selenium or whatever). While it takes awhile to write all of these, once they were written "in theory" no human would ever need to test those aspects again.
Even more yay! Maybe you want to come and see this session: http://barcelona2007.drupalcon.org/node/621 "Creating a Drupal Specific QA Team" One of the things I'd like to do is create a system somewhat like Litmus - http://litmus.mozilla.org/ Litmus allows community testers to spend a very little amount of time testing the software, recording their results, and then aggregating them to make sure they are useful. It's pretty neat. Another fun tool would be the magic automated simpletester/Selenium machine we keep getting closer to implementing. This is another topic I'd love to cover in that QA brainstorming session in Barcelona. Anyway, just saying thanks for this work and (not so subtly) advertising my sessions here ;) Greg PS The other sessions I'm doing are: "Token Module: How I Learned To Use the Token API and Stop Re-Implementing Dynamic String Replacement" - http://barcelona2007.drupalcon.org/node/416 AND (Dis)Organizing a Drupal Local User Group: lessons learned - http://barcelona2007.drupalcon.org/node/433 -- Greg Knaddison Denver, CO | http://knaddison.com World Spanish Tour | http://wanderlusting.org/user/greg
(...)
- We can also use this as a check-list for our library of SimpleTest automated tests (or Selenium or whatever). While it takes awhile to write all of these, once they were written "in theory" no human would ever need to test those aspects again.
Jon Udell's talks on his blog about a tool that could be very interesting : http://blog.jonudell.net/2007/09/06/the-social-scripting-continuum/ It's called Coscripter (http://services.alphaworks.ibm.com/coscripter/browse/about) and it's a way to describe in human readable text a series of actions you perform on a website. It can then played back inside the browser (it's a firefox extension). "CoScripter is a system for recording, automating, and sharing processes performed in a web browser such as printing photos online, requesting a vacation hold for postal mail, or checking bank account information. Instructions for processes are recorded and stored in easy-to-read text here on the CoScripter web site, so anyone can make use of them. If you are having trouble with a web-based process, check to see if someone has written a CoScript for it!" This could be used to automate Drupal testing and see the results live in the browser. Very low barrier of entry it seems. I don't know what is the licence of this firefox extension trough... Philippe
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Karoly Negyesi schrieb:
It can then played back inside the browser (it's a firefox extension).
There is another , called Selenium and oh mine. I was impressed at first but working with Selenium is not pleasant. Simpletest rocks.
I think there's even a third one, TestGen4Web, and possibly more. While simpletest is great, it is much harder to use. It would be very convenient if we could settle for one of the browser based thingies and tell people to mail their tests. Everybody can use a browser but not everybody can write simpletest tests. So I see these as useful complements, not replacements. Cheers, Gerhard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFG4xbufg6TFvELooQRAv4dAJ4/VkkPPgcdqVczer8S5jFlq8CTnACePmYf qKqKUbuiK1mZo7YX4TIDGm0= =l6JL -----END PGP SIGNATURE-----
I think there's even a third one, TestGen4Web, and possibly more.
While simpletest is great, it is much harder to use. It would be very convenient if we could settle for one of the browser based thingies and tell people to mail their tests. Everybody can use a browser but not everybody can write simpletest tests. So I see these as useful complements, not replacements.
I just tried CoScripter, and it's very easy. Here is a sample script to test Drupal 6 book feature: http://services.alphaworks.ibm.com/coscripter/browse/script/1369 Notice that what you read on this page *is* the actual test, nothing else involved. You can run it on your own local Drupal install. The sentences containing "you" mean that human intervention is required. There is a problem currently with this test, it fails to select the right option in the book drop down menu. Still a neat idea imho. Philippe
That is so simple and elegant I could kiss the screen. we do need to get to a stage where there is no user intervention in our scripts and thus can be run unattended. On 9/14/07, Philippe Jadin <philippe.jadin@gmail.com> wrote:
I just tried CoScripter, and it's very easy. Here is a sample script to test Drupal 6 book feature: http://services.alphaworks.ibm.com/coscripter/browse/script/1369
Notice that what you read on this page *is* the actual test, nothing else involved. You can run it on your own local Drupal install. The sentences containing "you" mean that human intervention is required.
I have been really impressed with CoScripter too. The really nice thing is you can copy & paste it's output into a drupal issue, and then if other people have CoScripter they can copy & paste it into the sidebar and auto-run it, but even if they don't have CoScripter, they can still just follow the instructions. Moshe Weitzman wrote:
That is so simple and elegant I could kiss the screen.
we do need to get to a stage where there is no user intervention in our scripts and thus can be run unattended.
On 9/14/07, Philippe Jadin <philippe.jadin@gmail.com> wrote:
I just tried CoScripter, and it's very easy. Here is a sample script to test Drupal 6 book feature: http://services.alphaworks.ibm.com/coscripter/browse/script/1369
Notice that what you read on this page *is* the actual test, nothing else involved. You can run it on your own local Drupal install. The sentences containing "you" mean that human intervention is required.
thats a nice use case. unfortunately, CoScripter doesn't have assertions so it is useless as an automated unit test tool. AFAICT On 9/14/07, Tao Starbow <starbow@citris-uc.org> wrote:
I have been really impressed with CoScripter too. The really nice thing is you can copy & paste it's output into a drupal issue, and then if other people have CoScripter they can copy & paste it into the sidebar and auto-run it, but even if they don't have CoScripter, they can still just follow the instructions.
Moshe Weitzman wrote:
That is so simple and elegant I could kiss the screen.
we do need to get to a stage where there is no user intervention in our scripts and thus can be run unattended.
On 9/14/07, Philippe Jadin <philippe.jadin@gmail.com> wrote:
I just tried CoScripter, and it's very easy. Here is a sample script to test Drupal 6 book feature: http://services.alphaworks.ibm.com/coscripter/browse/script/1369
Notice that what you read on this page *is* the actual test, nothing else involved. You can run it on your own local Drupal install. The sentences containing "you" mean that human intervention is required.
participants (8)
-
Angela Byron -
Derek Wright -
Gerhard Killesreiter -
Greg Knaddison - GVS -
Karoly Negyesi -
Moshe Weitzman -
Philippe Jadin -
Tao Starbow