[Building Sakai] Sakai OAE Usability Testing Package 1

[Allison Bloodworth]

Hi folks, 

At UC Berkeley we've completed 4 usability tests and have 6 more scheduled by the end of next Tuesday. We decided not to modify the protocol by changing it to a project site or adding additional tasks.

We have learned some things about the test environment --including a change that is needed to the permissions in the environment before starting the test, and some cleanup of the environment that is necessary between tests because the same "Elementary Calculus" course is being used by each tester.

Is anyone else doing usability testing? If so I'll write up what we've learned about these things in detail.

Cheers,
Allison

On Aug 22, 2011, at 6:05 PM, Allison Bloodworth wrote:

> Hi Anne-Sophie, 
> 
> Thanks much for your responses and patience with us as Rachel, Daphne & I came up with this feedback after taking a closer look at the test together. :) Comments below...
> 
> On Aug 22, 2011, at 12:30 PM, Anne-Sophie De Baets wrote:
> 
>> Hi Allison,
>> 
>> Thanks a lot for your feedback. Some comments inline:
>> 
>>> Rachel & I are going over the protocol in detail now and it would be very
>>> helpful to actually run through it on a test server. However, we don't
>>> want to start Berkeley's countdown clock on having our server for 3 weeks
>>> quite yet--is there a server that is set up now that we could use to run
>>> through the test?
>> 
>> That makes sense. You can experiment with the 3akai.sakaiproject.org
>> server, using the same accounts and data as described in the manual,
>> which have been created on there as well. However, the server is not
>> running the latest code, so bugs might still exist. We'll reset the
>> server to its initial state from time to time, meaning that you will
>> lose all of the changes you make to the test accounts and data.
> 
> Great, thanks! We took a quick look here and the server is a bit slow but this should help us get an idea before we get our own server.
> 
>> 
>>> In our current context (as we have no courses in Sakai OAE) it would
>>> actually make more sense to test this on a group, e.g. a "Calculus
>>> Research Project group." This would likely also allow us to obtain
>>> information about how users in groups function. Do you think it would be
>>> problematic to make that change?
>> 
>> In general, it would be good if we could keep the testing scenario
>> as close to the original test as possible, just to make it easier
>> to analyse results coming from different institutions. However, in
>> this case, I think the change you'd like to make should be fine. You'll
>> still be testing the same functionality and interface elements, as
>> you're basically just changing the name and type.
>> 
>> It's probably worth keeping in mind that the server will already have
>> the Maths course pre-created for you. This means that you'll have
>> to create the Mathematics Research Project group yourself and modify
>> the content of the messages present in the system. It is important
>> to keep the content of the group and messages as close as possible
>> to the originals.
> 
> We understand that there will be extra work involved if we make this change, and will will weigh this when deciding whether or not to do it. We definitely agree that the goals should be the same.
> 
>> 
>>> Additionally, there are a few other things we'd like to add to the test to
>>> investigate the usability of other things that are important to us at
>>> Berkeley (e.g. categories, profile). We are hoping information from some
>>> additional tasks we add would still be helpful to the managed project?
>> 
>> I think it's great that you're keen to test other things as well, but
>> for this first package I would encourage institutions to run the test
>> as closely as possible to the original without changing or expanding
>> the scope and goals. We're still trying to figure out how we can
>> best analyse and summarise results coming from different institutions,
>> so for now it would be more helpful if all participating institutions
>> conducted the same focused test.
> 
> We agree that analysis will be difficult. We had talked with Nico about the new OAE UX group (of which we hope you will be a part! :)) working on this. Are you envisioning the Sakai OAE UX group analyzing the results in some way together, or that you would take the lead on analysis and the rest of us would assist you in some fashion? Perhaps we can talk sometime soon about how we can help. We have been meeting (irregularly so far) Tuesdays at 9am PDT if that happens to work for you, or we'd be happy to set up a time anytime this week. 
> 
>> 
>> Another reason why it might not be the best time to test categories and
>> profiles is that both of these are currently undergoing design changes.
>> It feels more useful to create a usability package that tests these
>> changes, rather than the current solution which will disappear soon.
> 
> Great, that's exactly the sort of feedback we were looking for. Not testing these things makes sense if the changes aren't out by the time we do our tests. Any idea when these changes will be made? 
> 
>> 
>> However, it's really valuable to find out what you would like to test
>> as well. The goal is to send out small focused usability test packages
>> at regular times, and that information could help us in deciding what
>> to test next. Tomorrow, I will create a Confluence page where everyone
>> can list what they would like to test in the next usability package. I'll
>> send out a note once that page is up.
> 
> It would be great to discuss strategy for future usability testing sessions together--and we think it's a fantastic idea to create a confluence page to discuss what to test next!  Since finding & scheduling users takes us quite a bit of time (as we endeavor to get a realistic sample as opposed to doing 'friends & family (or colleague)' testing), it would be easier for us to have fewer testing sessions which cover more ground. I'd be interested to hear about how other institutions who plan to participate in the testing sessions feel. 
> 
> Cheers,
> Allison
> 
>> 
>> Best wishes,
>> Anne-Sophie
>> 
> 
> Allison Bloodworth
> Senior User Experience Designer
> Educational Technology Services
> University of California, Berkeley
> abloodworth at berkeley.edu
> 510-289-6627
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> sakai-ui-dev mailing list
> sakai-ui-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-ui-dev

Allison Bloodworth
Senior User Experience Designer
Educational Technology Services
University of California, Berkeley
abloodworth at berkeley.edu
510-289-6627