[Building Sakai] Sakai OAE Usability Testing Package 1

Tue Sep 13 03:29:35 PDT 2011

Hi Allison,

Thanks for all the feedback!
I've finally found some time to add a note about 'clearing the changes
that participants made' between sessions. You can now find this extra
information in '2. Detailed Manual' under III.5, and I've also added it in
the intro on '13. Expected Outcomes'.  It's just a small note to make
people aware of it so nothing detailed of what specific changes have to be
cleared as this also depends on what participants do during the test (this
can vary a lot).

As these changes have now been made, I've also uploaded the newest version
of the ZIP on the confluence page, so institutions that plan to do the
testing in the future, should use this ZIP as well.

About changing the permissions on the test servers: don't worry about
making these notes because I could as well change these permissions to the
test servers instead.
As soon as Nico sets up another server, I'll make the changes to these
permissions.

Many thanks,
Anne-Sophie

> Hi folks,
>
> At UC Berkeley we've completed 4 usability tests and have 6 more scheduled
> by the end of next Tuesday. We decided not to modify the protocol by
> changing it to a project site or adding additional tasks.
>
> We have learned some things about the test environment --including a
> change that is needed to the permissions in the environment before
> starting the test, and some cleanup of the environment that is necessary
> between tests because the same "Elementary Calculus" course is being used
> by each tester.
>
> Is anyone else doing usability testing? If so I'll write up what we've
> learned about these things in detail.
>
> Cheers,
> Allison
>
> On Aug 22, 2011, at 6:05 PM, Allison Bloodworth wrote:
>
>> Hi Anne-Sophie,
>>
>> Thanks much for your responses and patience with us as Rachel, Daphne &
>> I came up with this feedback after taking a closer look at the test
>> together. :) Comments below...
>>
>> On Aug 22, 2011, at 12:30 PM, Anne-Sophie De Baets wrote:
>>
>>> Hi Allison,
>>>
>>> Thanks a lot for your feedback. Some comments inline:
>>>
>>>> Rachel & I are going over the protocol in detail now and it would be
>>>> very
>>>> helpful to actually run through it on a test server. However, we don't
>>>> want to start Berkeley's countdown clock on having our server for 3
>>>> weeks
>>>> quite yet--is there a server that is set up now that we could use to
>>>> run
>>>> through the test?
>>>
>>> That makes sense. You can experiment with the 3akai.sakaiproject.org
>>> server, using the same accounts and data as described in the manual,
>>> which have been created on there as well. However, the server is not
>>> running the latest code, so bugs might still exist. We'll reset the
>>> server to its initial state from time to time, meaning that you will
>>> lose all of the changes you make to the test accounts and data.
>>
>> Great, thanks! We took a quick look here and the server is a bit slow
>> but this should help us get an idea before we get our own server.
>>
>>>
>>>> In our current context (as we have no courses in Sakai OAE) it would
>>>> actually make more sense to test this on a group, e.g. a "Calculus
>>>> Research Project group." This would likely also allow us to obtain
>>>> information about how users in groups function. Do you think it would
>>>> be
>>>> problematic to make that change?
>>>
>>> In general, it would be good if we could keep the testing scenario
>>> as close to the original test as possible, just to make it easier
>>> to analyse results coming from different institutions. However, in
>>> this case, I think the change you'd like to make should be fine. You'll
>>> still be testing the same functionality and interface elements, as
>>> you're basically just changing the name and type.
>>>
>>> It's probably worth keeping in mind that the server will already have
>>> the Maths course pre-created for you. This means that you'll have
>>> to create the Mathematics Research Project group yourself and modify
>>> the content of the messages present in the system. It is important
>>> to keep the content of the group and messages as close as possible
>>> to the originals.
>>
>> We understand that there will be extra work involved if we make this
>> change, and will will weigh this when deciding whether or not to do it.
>> We definitely agree that the goals should be the same.
>>
>>>
>>>> Additionally, there are a few other things we'd like to add to the
>>>> test to
>>>> investigate the usability of other things that are important to us at
>>>> Berkeley (e.g. categories, profile). We are hoping information from
>>>> some
>>>> additional tasks we add would still be helpful to the managed project?
>>>
>>> I think it's great that you're keen to test other things as well, but
>>> for this first package I would encourage institutions to run the test
>>> as closely as possible to the original without changing or expanding
>>> the scope and goals. We're still trying to figure out how we can
>>> best analyse and summarise results coming from different institutions,
>>> so for now it would be more helpful if all participating institutions
>>> conducted the same focused test.
>>
>> We agree that analysis will be difficult. We had talked with Nico about
>> the new OAE UX group (of which we hope you will be a part! :)) working
>> on this. Are you envisioning the Sakai OAE UX group analyzing the
>> results in some way together, or that you would take the lead on
>> analysis and the rest of us would assist you in some fashion? Perhaps we
>> can talk sometime soon about how we can help. We have been meeting
>> (irregularly so far) Tuesdays at 9am PDT if that happens to work for
>> you, or we'd be happy to set up a time anytime this week.
>>
>>>
>>> Another reason why it might not be the best time to test categories and
>>> profiles is that both of these are currently undergoing design changes.
>>> It feels more useful to create a usability package that tests these
>>> changes, rather than the current solution which will disappear soon.
>>
>> Great, that's exactly the sort of feedback we were looking for. Not
>> testing these things makes sense if the changes aren't out by the time
>> we do our tests. Any idea when these changes will be made?
>>
>>>
>>> However, it's really valuable to find out what you would like to test
>>> as well. The goal is to send out small focused usability test packages
>>> at regular times, and that information could help us in deciding what
>>> to test next. Tomorrow, I will create a Confluence page where everyone
>>> can list what they would like to test in the next usability package.
>>> I'll
>>> send out a note once that page is up.
>>
>> It would be great to discuss strategy for future usability testing
>> sessions together--and we think it's a fantastic idea to create a
>> confluence page to discuss what to test next!  Since finding &
>> scheduling users takes us quite a bit of time (as we endeavor to get a
>> realistic sample as opposed to doing 'friends & family (or colleague)'
>> testing), it would be easier for us to have fewer testing sessions which
>> cover more ground. I'd be interested to hear about how other
>> institutions who plan to participate in the testing sessions feel.
>>
>> Cheers,
>> Allison
>>
>>>
>>> Best wishes,
>>> Anne-Sophie
>>>
>>
>> Allison Bloodworth
>> Senior User Experience Designer
>> Educational Technology Services
>> University of California, Berkeley
>> abloodworth at berkeley.edu
>> 510-289-6627
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> sakai-ui-dev mailing list
>> sakai-ui-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-ui-dev
>
> Allison Bloodworth
> Senior User Experience Designer
> Educational Technology Services
> University of California, Berkeley
> abloodworth at berkeley.edu
> 510-289-6627
>
>
>
>
>
>
>
>