[WG: Sakai QA] Fwd: Automated testing -- Tools discussion

[Corey McGarrahan]

The Selenium tests are based on manual test procedures rSmart had written a
little over 2yrs ago? They were based off of the scripts found in Sakai
Confluence<http://confluence.sakaiproject.org/display/QA/Test+Conditions+and+Scripts>.
We rewrote the scripts in hopes of employing non Sakai people to help with
testing as well as for automation. The scripts in Sakai are not very useful
for people without intimate knowledge of Sakai and leave too many variables
open for automation. You can see the procedures we based the automation on
here <http://confluence.sakaiproject.org/display/QA/New+Test+Plans>.

Per your comments, I've updated the readme.txt to better reflect whats in
the test_suite. I've also updated the suite names. Some tests can be used
across 2.6 and 2.7 due to no functionality change. As of now, the html
scripts are in far better shape as well as there being many more available.
I have set up some time to finish exporting the rest of the 2.7 tests into
the TestNG format before the end of day Friday.

As for what to use when, it depends on the build and the frequency of the
build. The setup suites both now have a Teardown sequence before they setup
new users and courses.
The suites each have a url default that will need to be edited depending on
where you are trying to run them.
2.6 - http://nightly2.sakaiproject.org:8087/portal
2.7 - http://qa1-nl.sakaiproject.org/portal


On Thu, May 6, 2010 at 1:59 AM, Jean-Francois Leveque <
jean-francois.leveque at upmc.fr> wrote:

> I hear you.
>
> I don't think Marist tests for GB2 are needed from a 2.7.0 release point of
> view.
>
> Where are we regarding Marists tests for code in the 2.7.0 release that are
> not duplicates of rSmart tests?
>
> Are the rSmart or Marists documented from a
> http://confluence.sakaiproject.org/display/QA/Test+Conditions+and+Scriptspoint of view?
>
>
> https://source.sakaiproject.org/contrib/qa/trunk/functional/test_suite/readme.txtand
> http://confluence.sakaiproject.org/display/QA/Selenium+Remote+Control+and+TestNG
> document the current state of
> https://source.sakaiproject.org/contrib/qa/trunk/functional/test_suite.
>
> Looks like this documentation needs an update. cle-suite-nightly.xml in
> readme.txt looks like a specific rSmart name and cannot be found in
> subversion.
>
> sakai-suite-regsuite-26.xml looks like a 2.6 suite.
>
> sakai-suite-setup-27.xml looks like a 2.7 suite which includes 2.6 teardown
> first.
>
> sakai-suite-setup.xml looks like another 2.6 suite.
>
> sakai-suite-single.xml looks like a specific rSmart teadrown.
>
> sakai-suite-teardown.xml looks like the 2.6 teardown included in
> sakai-suite-setup-27.xml.
>
> Which suite should be used against 2.6, 2.7 and trunk and how should they
> be used (is sakai-suite-teardown.xml needed before or after or should the
> suite be updated to include it)?
>
> --
>
> Jean-Francois Leveque
> UPMC
>
> Corey McGarrahan a écrit :
>
>> The suite contains scripts from rSmart, but they are written for OOTB
>> Sakai. I know Marist tests for GB2 are going to be added soon. A lot of the
>> Marist tests are duplicates of what is already checked in. Anyone who is
>> writing scripts that are designed to run against OOTB Sakai should be
>> commiting those tests to the QA/trunk//functional/ area. What I am worried
>> about is having multiple copies of the same tests and I'll be looking out
>> for that.
>>
>>
>> For now I have been running the suite locally and have not been uploaded
>> the results anywhere as of yet, outside of reporting any bugs that I run
>> into. The results will be stored in the /test_suite/target/reports directory
>> as index.html once the Selenium RC/TestNG automated build is running. This
>> will address your third question as well. I've had conversations with Alan
>> about this and I know he has done some work on this already.
>>
>> The suite as it is now is rather thin. All of the existing test were
>> written as Acceptance Tests (positive path) and are going to need to be
>> updated as new fixes and functionality are added. This test suite can (and
>> should) become the automated regression testing suite for new builds. My
>> goal/hope/idea is that these scripts will be the foundation for the
>> regression suite and will be maintained and updated by the QA community.
>> I will be updating these existing scripts as we move through the betas and
>> into the RC's as things will obviously be changing. Anyone who is interested
>> in gettin involved can find information in the links of the original
>> message.
>>
>>
>> On Wed, May 5, 2010 at 5:46 AM, Jean-Francois Leveque <
>> jean-francois.leveque at upmc.fr <mailto:jean-francois.leveque at upmc.fr>>
>> wrote:
>>
>>    Does this suite contain scripts from Marist or scripts from rSmart
>> only?
>>
>>    Are the results of the run against 2.7.0-b08 stored somewhere?
>>
>>    Could this suite be adapted and automatically run on the automated
>>    builds of trunk and the 2.7.x, 2.6.x and 2.5.x branches?
>>
>>    Jean-Francois Leveque
>>    UPMC
>>
>>    Corey McGarrahan a écrit :
>>
>>        Automated testing for Sakai with Selenium IDE, Selenium RC and
>>        TestNG
>>
>>        All of the existing Selenium scripts have been updated for
>>        2.7.0-b08 -
>>
>> https://source.sakaiproject.org/contrib/qa/trunk/functional/2.7/Selenium
>>        and have been run against the latest build on QA1-NL
>>        <http://qa1-nl.sakaiproject.org/>.
>>
>>        The scripts are also updated in the Selenium RC/TestNG automated
>>        test suite here -
>>
>> https://source.sakaiproject.org/contrib/qa/trunk/functional/test_suite/
>>
>>        Instructions for running the Selenium scripts in the browser can
>>        be found here
>>         <http://confluence.sakaiproject.org/display/QA/Selenium>, while
>>        the instructions for running the Selenium RC/TestNG automated
>>        test suite can be found here
>>        <
>> http://confluence.sakaiproject.org/display/QA/Selenium+Remote+Control+and+TestNG
>> >.
>>
>>
>>        If you have any questions about setup or how to edit the scripts
>>        so they run locally please don't hesitate to ask me. Some basic
>>        guidelines:
>>
>>          1. Make sure to checkout the TestingData
>>                    <
>> https://source.sakaiproject.org/contrib/qa/trunk/functional/2.7/Resources
>> >
>>        folder.
>>
>>             All resources referenced in the scripts are found in there.
>>          2. All paths to uploads will need to be changed to point to the
>>
>>             TestingData folder as they are not being hosted on a public
>>        server
>>             at this time.    3. All scripts are written with the
>>        assumption that the site to be
>>
>>             tested is already open in the browser.
>>                1. This does not apply to Selenium RC/TestNG
>>          4. All scripts were written against QA1-NL. Any deviations
>>        from that
>>
>>             build will need to be accounted for; ie. admin pw
>>
>>        Corey McGarrahan
>>        rSmart
>>
>>
>>
>>
>> --
>> Corey Jack McGarrahan
>> rSmart
>>
>


-- 
Corey Jack McGarrahan
rSmart
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-qa/attachments/20100506/25066f04/attachment-0001.html