[Building Sakai] [DG: User Experience] [oae-urg] Sakai OAE Usability Testing Package 1

Tue Sep 13 03:40:51 PDT 2011

Hi Daphne,

I'm not really that familiar with Confluence to be able to say in what way
we could make things private (for example for the raw notes and videos).
I agree that it's probably okay to put the profiles on Confluence without
special privacy settings, as the profiles don't mention full names and
because no voice or face will be recognised through that.
However, I don't think it's beneficial for the tests if people already
look at these profiles before they've carried out their own tests, as they
might get prejudices before starting their own tests.
So yes I don't mind putting them on confluence, but only if people looking
at it, are people who don't intend to carry out tests themselves, or if
they've already finished their own tests.

Anne-Sophie

> Hi Keli,
>
> We'll be sharing our results back to Anne-Sophie in the form of the
> profiles she's asked along with our raw notes and video.  For privacy
> reasons, the raw notes and video should be kept private within the
> project team (do we have a way to post these things and restrict
> access?) but my assumption is Anne-Sophie will make all the profiles
> public.  We should have those available next week.
>
> -Daphne
>
> On Sep 9, 2011, at 10:16 AM, Keli Sato Amann wrote:
>
>> We have not, but we'd love to hear about your results.
>>
>> ----- Original Message -----
>> From: "Allison Bloodworth" <abloodworth at berkeley.edu>
>> To: "Sakai UI Development" <sakai-ui-dev at collab.sakaiproject.org>
>> Cc: "Anne-Sophie De Baets" <asd38 at caret.cam.ac.uk>, "Sakai OAE User
>> Reference Group" <oae-urg at collab.sakaiproject.org>, "Sakai UX"
>> <sakai-ux at collab.sakaiproject.org
>> >, "Sakai Developers" <sakai-dev at collab.sakaiproject.org>
>> Sent: Friday, September 9, 2011 9:27:54 AM
>> Subject: Re: [oae-urg] [Building Sakai] Sakai OAE Usability Testing
>> Package 1
>>
>> Hi folks,
>>
>> At UC Berkeley we've completed 4 usability tests and have 6 more
>> scheduled by the end of next Tuesday. We decided not to modify the
>> protocol by changing it to a project site or adding additional tasks.
>>
>> We have learned some things about the test environment --including a
>> change that is needed to the permissions in the environment before
>> starting the test, and some cleanup of the environment that is
>> necessary between tests because the same "Elementary Calculus"
>> course is being used by each tester.
>>
>> Is anyone else doing usability testing? If so I'll write up what
>> we've learned about these things in detail.
>>
>> Cheers,
>> Allison
>>
>> On Aug 22, 2011, at 6:05 PM, Allison Bloodworth wrote:
>>
>>> Hi Anne-Sophie,
>>>
>>> Thanks much for your responses and patience with us as Rachel,
>>> Daphne & I came up with this feedback after taking a closer look at
>>> the test together. :) Comments below...
>>>
>>> On Aug 22, 2011, at 12:30 PM, Anne-Sophie De Baets wrote:
>>>
>>>> Hi Allison,
>>>>
>>>> Thanks a lot for your feedback. Some comments inline:
>>>>
>>>>> Rachel & I are going over the protocol in detail now and it would
>>>>> be very
>>>>> helpful to actually run through it on a test server. However, we
>>>>> don't
>>>>> want to start Berkeley's countdown clock on having our server for
>>>>> 3 weeks
>>>>> quite yet--is there a server that is set up now that we could use
>>>>> to run
>>>>> through the test?
>>>>
>>>> That makes sense. You can experiment with the 3akai.sakaiproject.org
>>>> server, using the same accounts and data as described in the manual,
>>>> which have been created on there as well. However, the server is not
>>>> running the latest code, so bugs might still exist. We'll reset the
>>>> server to its initial state from time to time, meaning that you will
>>>> lose all of the changes you make to the test accounts and data.
>>>
>>> Great, thanks! We took a quick look here and the server is a bit
>>> slow but this should help us get an idea before we get our own
>>> server.
>>>
>>>>
>>>>> In our current context (as we have no courses in Sakai OAE) it
>>>>> would
>>>>> actually make more sense to test this on a group, e.g. a "Calculus
>>>>> Research Project group." This would likely also allow us to obtain
>>>>> information about how users in groups function. Do you think it
>>>>> would be
>>>>> problematic to make that change?
>>>>
>>>> In general, it would be good if we could keep the testing scenario
>>>> as close to the original test as possible, just to make it easier
>>>> to analyse results coming from different institutions. However, in
>>>> this case, I think the change you'd like to make should be fine.
>>>> You'll
>>>> still be testing the same functionality and interface elements, as
>>>> you're basically just changing the name and type.
>>>>
>>>> It's probably worth keeping in mind that the server will already
>>>> have
>>>> the Maths course pre-created for you. This means that you'll have
>>>> to create the Mathematics Research Project group yourself and modify
>>>> the content of the messages present in the system. It is important
>>>> to keep the content of the group and messages as close as possible
>>>> to the originals.
>>>
>>> We understand that there will be extra work involved if we make
>>> this change, and will will weigh this when deciding whether or not
>>> to do it. We definitely agree that the goals should be the same.
>>>
>>>>
>>>>> Additionally, there are a few other things we'd like to add to
>>>>> the test to
>>>>> investigate the usability of other things that are important to
>>>>> us at
>>>>> Berkeley (e.g. categories, profile). We are hoping information
>>>>> from some
>>>>> additional tasks we add would still be helpful to the managed
>>>>> project?
>>>>
>>>> I think it's great that you're keen to test other things as well,
>>>> but
>>>> for this first package I would encourage institutions to run the
>>>> test
>>>> as closely as possible to the original without changing or expanding
>>>> the scope and goals. We're still trying to figure out how we can
>>>> best analyse and summarise results coming from different
>>>> institutions,
>>>> so for now it would be more helpful if all participating
>>>> institutions
>>>> conducted the same focused test.
>>>
>>> We agree that analysis will be difficult. We had talked with Nico
>>> about the new OAE UX group (of which we hope you will be a
>>> part! :)) working on this. Are you envisioning the Sakai OAE UX
>>> group analyzing the results in some way together, or that you would
>>> take the lead on analysis and the rest of us would assist you in
>>> some fashion? Perhaps we can talk sometime soon about how we can
>>> help. We have been meeting (irregularly so far) Tuesdays at 9am PDT
>>> if that happens to work for you, or we'd be happy to set up a time
>>> anytime this week.
>>>
>>>>
>>>> Another reason why it might not be the best time to test
>>>> categories and
>>>> profiles is that both of these are currently undergoing design
>>>> changes.
>>>> It feels more useful to create a usability package that tests these
>>>> changes, rather than the current solution which will disappear soon.
>>>
>>> Great, that's exactly the sort of feedback we were looking for. Not
>>> testing these things makes sense if the changes aren't out by the
>>> time we do our tests. Any idea when these changes will be made?
>>>
>>>>
>>>> However, it's really valuable to find out what you would like to
>>>> test
>>>> as well. The goal is to send out small focused usability test
>>>> packages
>>>> at regular times, and that information could help us in deciding
>>>> what
>>>> to test next. Tomorrow, I will create a Confluence page where
>>>> everyone
>>>> can list what they would like to test in the next usability
>>>> package. I'll
>>>> send out a note once that page is up.
>>>
>>> It would be great to discuss strategy for future usability testing
>>> sessions together--and we think it's a fantastic idea to create a
>>> confluence page to discuss what to test next!  Since finding &
>>> scheduling users takes us quite a bit of time (as we endeavor to
>>> get a realistic sample as opposed to doing 'friends & family (or
>>> colleague)' testing), it would be easier for us to have fewer
>>> testing sessions which cover more ground. I'd be interested to hear
>>> about how other institutions who plan to participate in the testing
>>> sessions feel.
>>>
>>> Cheers,
>>> Allison
>>>
>>>>
>>>> Best wishes,
>>>> Anne-Sophie
>>>>
>>>
>>> Allison Bloodworth
>>> Senior User Experience Designer
>>> Educational Technology Services
>>> University of California, Berkeley
>>> abloodworth at berkeley.edu
>>> 510-289-6627
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> sakai-ui-dev mailing list
>>> sakai-ui-dev at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-ui-dev
>>
>> Allison Bloodworth
>> Senior User Experience Designer
>> Educational Technology Services
>> University of California, Berkeley
>> abloodworth at berkeley.edu
>> 510-289-6627
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> oae-urg mailing list
>> oae-urg at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/oae-urg
>> _______________________________________________
>> sakai-ux mailing list
>> sakai-ux at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-ux
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-ux-unsubscribe at collab.sakaiproject.org
>>  with a subject of "unsubscribe"
>
> Daphne Ogle-Glenn
> Senior Interaction Designer
> University of California, Berkeley
> Educational Technology Services
> daphne at media.berkeley.edu
> cell (925)348-4372
>
>
>
>
>
> _______________________________________________
> sakai-ux mailing list
> sakai-ux at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-ux
>
> TO UNSUBSCRIBE: send email to sakai-ux-unsubscribe at collab.sakaiproject.org
> with a subject of "unsubscribe"
>