[Contrib: Evaluation System] 1.3.x next (and final?) RC

Beth Kirschner bkirschn at umich.edu
Thu Apr 28 07:41:03 PDT 2011


We have at least two volunteers for testing (thanks Rick & Bryan!) -- so now all we need are some structured test plans. What you put together is a great start, and I've just made some changes to add some UM ideas and also make sure all test cases are numbered for easy identification. 
	https://confluence.sakaiproject.org/display/EVALSYS/Test+Plans

   How about we do this:
	1) Ask all institutions to review and update the test plans -- adding or updating test cases to meet local institutional use cases.
	2) Ask any and all volunteers to start testing using both the structured test plan and any ad-hoc testing they can think of. I've created a Test Results page at https://confluence.sakaiproject.org/display/EVALSYS/Evaluation+Test+Results+1.3.0-RC3 with a suggested format.

   Here's some suggested milestones:
	1) May 6, 2011 -- finish review and update of test plans
	2) May 13, 2011 -- finish testing and update test results page
	3) May 16-20, 2011 -- we can have a t-con and decide if the release is ready to go or not

What we end up with might not approach complete coverage, but they'd be something we can improve on over time. They might also not be as detailed as the OSP test cases you reference, but given the resources we have for testing, I think they would be reasonably good. While the OSP test cases you reference (which I was involved in) provided detailed guidance, they also quickly became out of date, so I don't think we should feel negligent if our test cases do not approach that level of detail ;-)

Thoughts?
- Beth

On Apr 27, 2011, at 3:46 PM, Nicola Monat-Jacobs wrote:

> Beth -
> 
> I think structured QA sounds great - what did you have in mind? My experience from general Sakai release QA is that the testing is based on pre-written test plans - and we're struggling to put together plans that represent the breadth of all evalsys has to offer. Thoughts?
> 
> - Nicola
> 
> On Apr 26, 2011, at  11:18 AM, Beth Kirschner wrote:
> 
>> Hi Nicola,
>> 
>>  If we do decide to tag and release evaluations v1.3, I think the outstanding questions are:
>> 	1) Has there been any structured QA on this release? It sounds like the answer is no, but a rough test plan is in place.
>> 	2) Are there any outstanding 'blocker' issues? It sounds like there is one, but the priority is always subject to debate.
>> 	3) Has this release been tested in production anywhere? It sounds like the answer is yes.
>> 
>>  From UM's point of view, I think it would be useful to have some structured QA on this release, so that we can define a 'baseline' to better identify what, if any, issues would be introduced in a future v1.4 evaluations release.  Rick Moyer has volunteered to do some QA testing -- is there anyone else?
>> 
>>  I still think a phone conference would be a good idea, but I'd prefer if we had some structured QA first -- what do others think?
>> 
>> - Beth
>> 
>> On Apr 25, 2011, at 9:22 PM, Nicola Monat-Jacobs wrote:
>> 
>>> Beth, Rick, everyone -
>>> 
>>> Apologies on not responding sooner. We do have the start of a QA Test Plan here (really more of an outline at this point):
>>> 
>>> https://confluence.sakaiproject.org/display/EVALSYS/Test+Plans
>>> 
>>> If we feel like we still need more eyes on this, I can post to the dev list and encourage people to see if they can find any issues. 
>>> 
>>> Since Lovemore didn't respond to the query about the one blocker associated with 1.3.0 (EVALSYS-1000), I'm going to write him directly about it. 
>>> 
>>> I can say that almost all our clients are now running the 1.3.x branch and have been for a while and we've encountered no major issues with basic functionality. (If you use very specific functionality in a very specific way YMMV, of course.)
>>> 
>>> If you're curious to see the outstanding issues for 1.3.x, take a look here:
>>> 
>>> http://jira.sakaiproject.org/secure/IssueNavigator.jspa?reset=true&pid=10220&resolution=-1&version=11906&version=11919&sorter/field=issuekey&sorter/order=DESC&tempMax=1000
>>> 
>>> I'm happy to tag 1.3.0 tonight if we feel this has gone on long enough and we just need to move on. At this point, 1.3.0 will be far, far more reliable that 1.2.1, so it's an improvement no matter how you slice it.
>>> 
>>> Thanks,
>>> Nicola
>>> 
>>> 
>>> On Apr 19, 2011, at  6:35 AM, Beth Kirschner wrote:
>>> 
>>>> Thanks Rick -- I've added some more comments below, but I wonder:
>>>> 
>>>> 1) Is anyone planning on deploying 1.3 into production (or even a modified 1.3 evaluations)?
>>>> 2) I think a phone conference would be helpful, but would it be best to do before or after you (and others?!?) are able to do some testing?
>>>> 
>>>> - Beth
>>>> 
>>>> On Apr 15, 2011, at 3:17 PM, Rick Moyer wrote:
>>>> 
>>>>> Beth,
>>>>> 
>>>>> 	I can do some simple things such as create scales, items, templates, hierarchy.
>>>>> 
>>>> That would be good -- do we know who would want to be deploying 1.3 into production?
>>>> 
>>>>> 	I don't see any course data, forgot to try via adhoc hoc.
>>>>> 
>>>> There are some courses already defined on the server (e.g. 'CIS 101 A Spring 2011'), and you can always create new courses (demo course sites are defined). The "Evaluations Systems" tool will be automatically added to new users who login, but existing users will have to manually add it to their My Workspace (Worksite Setup -> Edit -> Edit Tools -> Select "Evaluation System" and Continue.
>>>> 
>>>> OSP Evaluations are also defined on this server, so be sure not to accidentally select that.
>>>> 
>>>>> 	Any chance of creating a test course?
>>>>> 
>>>> See above. 
>>>> 
>>>>> Rick.
>>>>> 
>>>>> 
>>>>> -----Original Message-----
>>>>> From: evaluation-bounces at collab.sakaiproject.org [mailto:evaluation-bounces at collab.sakaiproject.org] On Behalf Of Beth Kirschner
>>>>> Sent: Friday, April 15, 2011 8:35 AM
>>>>> To: evaluation at collab.sakaiproject.org
>>>>> Subject: Re: [Contrib: Evaluation System] 1.3.x next (and final?) RC
>>>>> 
>>>>> The QA5 server has been running the 1.3.0-RC3 tag of evaluations for over a month now -- has anyone had a chance to test? If not, should we have a conference call to coordinate the testing effort? Otherwise, we should probably be discussing any significant issues and then picking a release date.
>>>>> 
>>>>> Thanks,
>>>>> - Beth
>>>>> 
>>>>> On Mar 4, 2011, at 5:27 PM, Beth Kirschner wrote:
>>>>> 
>>>>>> Hi everyone -- the QA5 server has been updated with the latest evaluations tag -- please feel free to take it for a test drive.
>>>>>> 	http://qa5-us.sakaiproject.org/portal
>>>>>> 
>>>>>> - Beth
>>>>>> 
>>>>>> On Mar 4, 2011, at 8:26 AM, Beth Kirschner wrote:
>>>>>> 
>>>>>>> I'll send a note out when QA5 has been updated with the latest tag.
>>>>>>> 
>>>>>>> - Beth
>>>>>>> 
>>>>>>> On Mar 2, 2011, at 9:10 PM, Nicola Monat-Jacobs wrote:
>>>>>>> 
>>>>>>>> Beth and Everyone -
>>>>>>>> 
>>>>>>>> I have now merged all 1.3.x Status = merge JIRAs that made sense to merge in. There are currently two outstanding:
>>>>>>>> 
>>>>>>>> EVALSYS-1026 - waiting for more information from Rick, but I believe this should not be merged into 1.3.x and should have it's Fix Status changed to 1.4.x.
>>>>>>>> 
>>>>>>>> EVALSYS-870 - Folks suspect this may already be resolved. If testing proves it's not, I can merge in.
>>>>>>>> 
>>>>>>>> We still have about 75 issues that are tagged as affecting 1.3.x that remain unresolved and without patches, but the only one that is a blocker is:
>>>>>>>> 
>>>>>>>> https://jira.sakaiproject.org/browse/EVALSYS-1000
>>>>>>>> 
>>>>>>>> Lovemore, do you have more information on this?
>>>>>>>> 
>>>>>>>> I've made a new RC:
>>>>>>>> 
>>>>>>>> https://source.sakaiproject.org/contrib//evaluation/tags/1.3.0-RC3/
>>>>>>>> 
>>>>>>>> Perhaps this will be the final one if we don't find any major issues.
>>>>>>>> 
>>>>>>>> Beth, can you get it up on that UMich test server for others to test? I think it was qa5-us.sakaiproject.org right?
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Nicola
>>>>>>>> 
>>>>>>>> On Mar 2, 2011, at  10:51 AM, Beth Kirschner wrote:
>>>>>>>> 
>>>>>>>>> Hi Nicola and all,
>>>>>>>>> 
>>>>>>>>> It looks like there are 11 evaluation issues with a "1.3.x Status" of merge, and of these is resolved (EVALSYS-1026) but not merged to 1.3.x. Does anyone have a good feel for the current status of 1.3.x? Is JIRA up-to-date? Should EVALSYS-1026 be merged and the remaining 10 issues deferred until the 1.4.x release?
>>>>>>>>> 
>>>>>>>>> Thoughts?
>>>>>>>>> - Beth
>>>>>>>>> 
>>>>>>>>> On Feb 18, 2011, at 8:30 PM, Nicola Monat-Jacobs wrote:
>>>>>>>>> 
>>>>>>>>>> Thanks for your summary Adam, it's very helpful
>>>>>>>>>> 
>>>>>>>>>>> EVALSYS-1061 
>>>>>>>>>>> 
>>>>>>>>>>> Display login screen when a login is needed to see results.
>>>>>>>>>>> 
>>>>>>>>>>> * Nicola mentions this but it is still unresolved in Jira & no there is comment about being applied to 1.3
>>>>>>>>>> 
>>>>>>>>>> I've resolved it. Also, if you tab over to 'Subversion Commits' you should see my commits to trunk and 1.3.x - sometimes they take a little while show up and/or I think you have to be logged in to see this.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> EVALSYS-1072
>>>>>>>>>>> 
>>>>>>>>>>> PDF Report shows response rate of entire evaluation, rather than selected groups
>>>>>>>>>>> 
>>>>>>>>>>> * Nicola mentions this but it is still unresolved in Jira; it appears to have been committed to trunk but not 1.3
>>>>>>>>>> 
>>>>>>>>>> You should now see my 1.3.x commit under 'Subversion Commits' and I've resolved the ticket, as Will clarified that Rick had already committed the patch to trunk.
>>>>>>>>>> 
>>>>>>>>>>> Patches that have been applied but which aren't yet set to resolved
>>>>>>>>>> 
>>>>>>>>>> A number of patches have comments from Rick about committing them - he clarified today that he only committed to trunk. So, I'll be committing them to 1.3.x and then setting 1.3.x status = resolved and the JIRA to resolved once I've done so.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> _______________________________________________
>>>>>>>>>> evaluation mailing list
>>>>>>>>>> evaluation at collab.sakaiproject.org
>>>>>>>>>> http://collab.sakaiproject.org/mailman/listinfo/evaluation
>>>>>>>>>> 
>>>>>>>>>> TO UNSUBSCRIBE: send email to evaluation-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> evaluation mailing list
>>>>>> evaluation at collab.sakaiproject.org
>>>>>> http://collab.sakaiproject.org/mailman/listinfo/evaluation
>>>>>> 
>>>>>> TO UNSUBSCRIBE: send email to evaluation-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>>>>>> 
>>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> evaluation mailing list
>>>>> evaluation at collab.sakaiproject.org
>>>>> http://collab.sakaiproject.org/mailman/listinfo/evaluation
>>>>> 
>>>>> TO UNSUBSCRIBE: send email to evaluation-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>>>>> 
>>>>> 
>>>> 
>>>> _______________________________________________
>>>> evaluation mailing list
>>>> evaluation at collab.sakaiproject.org
>>>> http://collab.sakaiproject.org/mailman/listinfo/evaluation
>>>> 
>>>> TO UNSUBSCRIBE: send email to evaluation-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>>> 
>>> 
>>> 
>> 
> 
> 
> 



More information about the evaluation mailing list