[WG: Sakai QA] Fwd: Automated testing -- Tools discussion

Corey McGarrahan corey at rsmart.com
Tue May 4 09:17:43 PDT 2010


Automated testing for Sakai with Selenium IDE, Selenium RC and TestNG

All of the existing Selenium scripts have been updated for 2.7.0-b08 -
https://source.sakaiproject.org/contrib/qa/trunk/functional/2.7/Selenium and
have been run against the latest build on
QA1-NL<http://qa1-nl.sakaiproject.org/>
.
The scripts are also updated in the Selenium RC/TestNG automated test suite
here -
https://source.sakaiproject.org/contrib/qa/trunk/functional/test_suite/

Instructions for running the Selenium scripts in the browser can be found
here  <http://confluence.sakaiproject.org/display/QA/Selenium>, while the
instructions for running the Selenium RC/TestNG automated test suite can be
found here<http://confluence.sakaiproject.org/display/QA/Selenium+Remote+Control+and+TestNG>
.

If you have any questions about setup or how to edit the scripts so they run
locally please don't hesitate to ask me. Some basic guidelines:

   1. Make sure to checkout the
TestingData<https://source.sakaiproject.org/contrib/qa/trunk/functional/2.7/Resources>
folder.
   All resources referenced in the scripts are found in there.
   2. All paths to uploads will need to be changed to point to the
   TestingData folder as they are not being hosted on a public server at this
   time.
   3. All scripts are written with the assumption that the site to be tested
   is already open in the browser.
      1. This does not apply to Selenium RC/TestNG
   4. All scripts were written against QA1-NL. Any deviations from that
   build will need to be accounted for; ie. admin pw

Corey McGarrahan
rSmart


On Wed, Apr 7, 2010 at 8:42 AM, Eli Cochran <eli at media.berkeley.edu> wrote:

> FYI
>
> Begin forwarded message:
>
> *From: *"Michelle D'Souza" <michelle.dsouza at utoronto.ca>
> *Date: *April 7, 2010 6:55:46 AM PDT
> *To: *Fluid Work <fluid-work at fluidproject.org>, Decapod Internal <
> decapod-internal at googlegroups.com>
> *Subject: **Fwd: [Work] Automated testing -- Tools discussion*
>
> An interesting summary of the research Kasper has done into automated
> testing tools.
>
> Michelle
>
>
>
> Begin forwarded message:
>
> *From: *Kasper Galschiot Markus <kasper2 at markus.dk>
> *Date: *April 7, 2010 7:39:53 AM GMT-04:00
> *To: *CollectionSpace Work list <work at lists.collectionspace.org>,
> obara.justin at gmail.com
> *Subject: **[Work] Automated testing -- Tools discussion*
>
>  I have been browsing around in the quite extensive amount of free web
> test automation tools around. Everyone seems to have their own preferences,
> and there are pros and cons for each tool. Below I'll try summarizing my
> observations (not experience, but what I have read). In the very bottom, I
> posted my personal opinion of choice.
>
> I'm sending this mail to the work list, because any experience, rumors,
> comments, suggestions, etc. would be highly appreciated. Also comments on
> the importance of: speed of tests, reality of tests, ease of programming,
> etc, as well as suggestions for other tools I should look into would be
> appreciated.
>
> ----
>
> *Selenium*: Probably the best known tool.
>
> Pros:
>
>    - It automates/scripts real browsers, which is an advantage as you get
>    close to reality.
>    - Well established and documented
>    - GUI record tool for firefox, making it easy to write test (for all
>    browsers)
>    - Support multiple programming languages (for writing the tests)
>    - Has been praised for being the easiest to use/get into
>    - Supports ant
>    - Has grid functionality to distribute the tests (to a certain extend)
>
> Cons:
>
>    - Events are simulated, which is sometimes buggy (based on rather old
>    articles, and justins (fluid) experiences X months ago, might have changed).
>    - Needs actual running windows+mac machines to run against
>    - Slow compared to emulated browser approach (can be somewhat remedied
>    by running tests in parallel)
>    - Disregards JS errors (unless they affect the asserts)!!!!
>    - Potential problem with waiting for pages to load (think this has
>    changed)
>
> *Canoo Webtest:* Also widely known tool.
>
> Pros:
>
>    - Do not need specific OS, as browsers are emulated (uses HtmlUnit)
>    - Record tool for writing easy tests
>    - Well established/documented
>    - Supports ant
>    - Easy to integrate/setup continuous testing
>    - Very good reports
>    - Fast
>
>
> Cons:
>
>    - Does not use real browsers, but emulates them using HtmlUnit. This
>    means some errors might potentially be not be caught.
>    - A bit steeper learning curve than Selenium
>    - Javascript support not as good as in "normal" browser
>    - Doesn't accept (too) badly formed html (Dont think this will be a
>    problem)
>
>
> *Doh.Robot: *A smaller project, brought to my attention by Justin.
>
> Pros:
>
>    - Does not simulate mouse/keys, but makes actual system calls using
>    java's robot
>    - As close as can be to real users (except for embedded scripts)
>
> Cons:
>
>    - Not too well documented/established
>    - Unsure if updated or discontinued
>    - requires code to be inserted on the page
>
> *
> Sahi:*
>
> Pros:
>
>    - GUI record for writing tests
>
> Cons:
>
>    - Small community
>    - Underdeveloped
>    - Confusing Interface
>
>
> *Watir:*
>
> Pros:
>
>    - Multi browser (& OS) support
>    - Has a rich API
>    - Has a ‘Simple’ class (for non-tech users)
>    - Watij & Watin (Java & .NET)
>
> Cons:
>
>
>    - Have to learn Ruby (unless you choose Watij or Watin)
>    - Every browser requires a different library
>
>
> *Selenium 2:*
>
> Pros:
>
>    - Has both standard selenium and HtmlUnit available for testing
>    - Circumvent browser restrictions by using whichever mechanism is most
>    appropriate to control the browser. For Firefox, this means that WebDriver
>    is implemented as an extension. For IE, WebDriver makes use of IE's
>    Automation controls, etc.
>    - WebDriver can make use of facilities offered by the OS (allows to get
>    closer to a real user, similar to robot i guess?!)
>
> Cons:
>
>    - Alpha state
>    - Sparse documentation, and probably somewhat buggy
>    - Steeper learning curve due to amount of features and lack of docs
>
> ---
>
> From what I can tell, Selenium and Canoo Webtest seems to be the most
> appealing choices, despite their shortcommings. It seems that Webtest
> receives less criticism than Selenium, but it is hard to tell whether this
> is due to Selenium being used/compared to more often. Generally, both people
> using Selenium and Webtest are satisfied. The others; Doh Robot, Sahi and
> Watir all have their advantages (except for Sahi :( ), but for various
> reasons didn't appeal to me (too little documentation, steep learning curve,
> lousier version of Webtest/Selenium). There is also the option of Selenium
> 2, but the problem with this is that it is still in alpha, and the
> documentation is rather sparse. Also, it might be kind of buggy.
>
> In many ways I am split between Selenium and Webtest, but I guess I am
> leaning towards Selenium. There are two main reasons for this:
>
>    1.
>
>    Using real browsers is a lot closer to reality, which I weigh high
>    2. (some of) the shortcommings of Selenium will be fixed in Selenium 2.
>    I take it that our tests written for Selenium will be compatible with
>    Selenium 2. When S2 will be at a decent stage it is easy to make the switch.
>
> The major problem with Selenium is the need of access to windows/mac
> machines (windows is doable, using Vms, I hate mac :).
> Another major problem is the speed. This can hopefully be remedied using
> Selenium Grid.
>
> As a final note, I guess I should mention that I might be a bit biased
> towards Selenium, since I knew that first, have read more of the
> documentation, etc.
> _______________________________________________
> Work mailing list
> Work at lists.collectionspace.org
>
> http://lists.collectionspace.org/mailman/listinfo/work_lists.collectionspace.org
>
>
>  ------------------------------------------------------
> Michelle D'Souza
> Software Developer, Fluid Project
> Adaptive Technology Resource Centre
>
> _______________________________________________________
> fluid-work mailing list - fluid-work at fluidproject.org
> To unsubscribe, change settings or access archives,
> see http://fluidproject.org/mailman/listinfo/fluid-work
>
>
>  . . . . . . . . . . .  .  .   .    .      .         .              .
>                 .
>
> Eli Cochran
> user interaction developer
> ETS, UC Berkeley
>
>
>
> _______________________________________________
> sakai-qa mailing list
> sakai-qa at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-qa
>
> TO UNSUBSCRIBE: send email to sakai-qa-unsubscribe at collab.sakaiproject.orgwith a subject of "unsubscribe"
>



-- 
Corey Jack McGarrahan
rSmart
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-qa/attachments/20100504/81c6551b/attachment-0001.html 


More information about the sakai-qa mailing list