[WG: Sakai QA] Fwd: Automated testing -- Tools discussion

[Jean-Francois Leveque]

I hear you.

I don't think Marist tests for GB2 are needed from a 2.7.0 release point 
of view.

Where are we regarding Marists tests for code in the 2.7.0 release that 
are not duplicates of rSmart tests?

Are the rSmart or Marists documented from a 
http://confluence.sakaiproject.org/display/QA/Test+Conditions+and+Scripts 
point of view?

https://source.sakaiproject.org/contrib/qa/trunk/functional/test_suite/readme.txt 
and 
http://confluence.sakaiproject.org/display/QA/Selenium+Remote+Control+and+TestNG
document the current state of 
https://source.sakaiproject.org/contrib/qa/trunk/functional/test_suite.

Looks like this documentation needs an update. cle-suite-nightly.xml in 
readme.txt looks like a specific rSmart name and cannot be found in 
subversion.

sakai-suite-regsuite-26.xml looks like a 2.6 suite.

sakai-suite-setup-27.xml looks like a 2.7 suite which includes 2.6 
teardown first.

sakai-suite-setup.xml looks like another 2.6 suite.

sakai-suite-single.xml looks like a specific rSmart teadrown.

sakai-suite-teardown.xml looks like the 2.6 teardown included in 
sakai-suite-setup-27.xml.

Which suite should be used against 2.6, 2.7 and trunk and how should 
they be used (is sakai-suite-teardown.xml needed before or after or 
should the suite be updated to include it)?

--
Jean-Francois Leveque
UPMC

Corey McGarrahan a écrit :
> The suite contains scripts from rSmart, but they are written for OOTB 
> Sakai. I know Marist tests for GB2 are going to be added soon. A lot of 
> the Marist tests are duplicates of what is already checked in. Anyone 
> who is writing scripts that are designed to run against OOTB Sakai 
> should be commiting those tests to the QA/trunk//functional/ area. What 
> I am worried about is having multiple copies of the same tests and I'll 
> be looking out for that.
> 
> For now I have been running the suite locally and have not been uploaded 
> the results anywhere as of yet, outside of reporting any bugs that I run 
> into. The results will be stored in the /test_suite/target/reports 
> directory as index.html once the Selenium RC/TestNG automated build is 
> running. This will address your third question as well. I've had 
> conversations with Alan about this and I know he has done some work on 
> this already.
> 
> The suite as it is now is rather thin. All of the existing test were 
> written as Acceptance Tests (positive path) and are going to need to be 
> updated as new fixes and functionality are added. This test suite can 
> (and should) become the automated regression testing suite for new 
> builds. My goal/hope/idea is that these scripts will be the foundation 
> for the regression suite and will be maintained and updated by the QA 
> community. 
> 
> I will be updating these existing scripts as we move through the betas 
> and into the RC's as things will obviously be changing. Anyone who is 
> interested in gettin involved can find information in the links of the 
> original message. 
> 
> 
> 
> On Wed, May 5, 2010 at 5:46 AM, Jean-Francois Leveque 
> <jean-francois.leveque at upmc.fr <mailto:jean-francois.leveque at upmc.fr>> 
> wrote:
> 
>     Does this suite contain scripts from Marist or scripts from rSmart only?
> 
>     Are the results of the run against 2.7.0-b08 stored somewhere?
> 
>     Could this suite be adapted and automatically run on the automated
>     builds of trunk and the 2.7.x, 2.6.x and 2.5.x branches?
> 
>     Jean-Francois Leveque
>     UPMC
> 
>     Corey McGarrahan a écrit :
> 
>         Automated testing for Sakai with Selenium IDE, Selenium RC and
>         TestNG
> 
>         All of the existing Selenium scripts have been updated for
>         2.7.0-b08 -
>         https://source.sakaiproject.org/contrib/qa/trunk/functional/2.7/Selenium
>         and have been run against the latest build on QA1-NL
>         <http://qa1-nl.sakaiproject.org/>.
> 
>         The scripts are also updated in the Selenium RC/TestNG automated
>         test suite here -
>         https://source.sakaiproject.org/contrib/qa/trunk/functional/test_suite/
> 
>         Instructions for running the Selenium scripts in the browser can
>         be found here
>          <http://confluence.sakaiproject.org/display/QA/Selenium>, while
>         the instructions for running the Selenium RC/TestNG automated
>         test suite can be found here
>         <http://confluence.sakaiproject.org/display/QA/Selenium+Remote+Control+and+TestNG>.
> 
> 
>         If you have any questions about setup or how to edit the scripts
>         so they run locally please don't hesitate to ask me. Some basic
>         guidelines:
> 
>           1. Make sure to checkout the TestingData
>            
>          <https://source.sakaiproject.org/contrib/qa/trunk/functional/2.7/Resources>
>         folder.
> 
>              All resources referenced in the scripts are found in there.
>           2. All paths to uploads will need to be changed to point to the
> 
>              TestingData folder as they are not being hosted on a public
>         server
>              at this time.    3. All scripts are written with the
>         assumption that the site to be
> 
>              tested is already open in the browser.
>                 1. This does not apply to Selenium RC/TestNG
>           4. All scripts were written against QA1-NL. Any deviations
>         from that
> 
>              build will need to be accounted for; ie. admin pw
> 
>         Corey McGarrahan
>         rSmart
> 
> 
> 
> 
> -- 
> Corey Jack McGarrahan
> rSmart