[WG: Sakai QA] [Building Sakai] [DG: User Experience] QA and user testing

Fri Jul 31 10:56:12 PDT 2009

Both Clay's and Michael's points seem right on target to me.

Can we set an expectation of when a decision to add these Forums to  
the 2.7 code base will be made?  The end of August seems like a  
reasonable time frame to me.  It provides several weeks for reading  
documentation and code and for getting responses to questions without  
letting the process go on indefinitely. (The user testing is very  
important but since it is expected only to effect the decision if a  
blocker is unexpectedly found it shouldn't be a prerequisite to making  
a decision.) If we don't get the consideration pipeline running soon  
we'll end up faced with a lot of decisions that need to be made  
quickly when the freeze date approaches.

- Dave

David Haines
CTools Developer
Digital Media Commons
University of Michigan
dlhaines at umich.edu

On Jul 31, 2009, at 1:02 PM, Michael Korcuska wrote:

> There's a further point I want to emphasize here. We want increased
> attention to design in the development process. We'll have to work out
> the details together as we move forward but I think we eventually want
> to arrive at a place where designs are available for community review
> while there is still time to do something with that feedback. And we
> want project teams to build this thinking into the project plan. There
> are lots of variations in how this might work, from waterfall to
> agile, and this isn't the space for that discussion. Again, lots of
> detail to work out, but I think the destination is relatively clear.
>
> Of course we can't jump to that destination overnight, especially in
> the context of Sakai 2.x. There's a difference between "meets our
> ideal standards" and "is a meaningful improvement over what we have
> today". We'll likely put more emphasis on the latter in Sakai 2 and
> more on the former in Sakai 3.
>
> All of this takes practice, though, and in part I see Clay's desire to
> do this with Forums as part exercising the processes we're trying to
> establish to see what works in practice. So its a test case, of sorts,
> for a small part of the overall process.
>
> Michael
>
> On Jul 31, 2009, at 09:39, Clay Fenlason wrote:
>
>> I think you're hoping for a small answer, but I think I need to
>> attempt a big one anyway, though I hope it still manages to be
>> incisive on the point of your concern.
>>
>>> Is the usability review a part of the decision about accepting the
>>> changes into the release,
>>> or is it to collect information that could improve documentation
>>> and inform future changes?
>>
>> The second, especially since we haven't done this before as a
>> community.  It's too late in the game to think about using this sort
>> of thing as gateway criteria for 2.7, not to mention that user  
>> testing
>> should be part of the design process at the outset. Again I think the
>> best way to think of it is as analogous to other QA work. We're
>> looking for rough edges that might be easily sanded down, not a new
>> bar that needs to be vaulted. And yes, we're trying also to share
>> expertise about how to do this sort of work well. By extension, we're
>> also trying to work out more ways for the expertise in our UX
>> community to have a practical impact on the evolution of the product,
>> and as it does so perhaps even make our UX community a more engaging
>> place to be. My interest in this area is more longer term than  
>> focused
>> on 2.7 in particular.
>>
>> That said, if user testing reveals 'blocker' issues (where that means
>> essentially what it has always meant - a consensus that the severity
>> is such that it shouldn't be released in that form), then I think we
>> would have to take that seriously, and deal with it as we do other
>> blockers.  That seems a highly unlikely outcome to me in the case of
>> the Forums interface work we've mentioned, since much of the work has
>> emerged already from conversations between schools using Forums in
>> production, and Michigan's even been running this particular branch.
>> User testing may reveal UX issues of lesser severity, and these could
>> form documentation for what a next round of improvements should
>> tackle. That's my picture of how this could work, at any rate.
>>
>> On another note, it's been disappointing over the years to see how
>> quickly usability and accessibility issues are demoted to "minor" in
>> JIRA reporting.  By the same token I've been underwhelmed in the past
>> with some of our handwavy release claims of "UI enhancements."  We've
>> done relatively little to substantiate such claims or prepare the
>> training and support staff at partner institutions for how they'll
>> need to adapt, even if it is a clear improvement. So the  
>> documentation
>> in my view should not only be about best practices for future
>> development, it should also result in more substantive release
>> documentation for deploying institutions trying to plan whether or
>> when to upgrade.
>>
>> HTH
>>
>> ~Clay
>>
>> On Fri, Jul 31, 2009 at 10:06 AM, David Haines<dlhaines at umich.edu>
>> wrote:
>>> It would be good to be completely clear here about the purpose of
>>> this
>>> review in the context of the 2.7 release: Is the usability review a
>>> part of
>>> the decision about accepting the changes into the release, or is it
>>> to
>>> collect information that could improve documentation and inform
>>> future
>>> changes?
>>> I believe it is the second, but I'd like to be sure I'm not just
>>> jumping to
>>> that conclusion.
>>> - Dave
>>>
>>> David Haines
>>> CTools Developer
>>> Digital Media Commons
>>> University of Michigan
>>> dlhaines at umich.edu
>>>
>>>
>>> On Jul 28, 2009, at 12:52 PM, Clay Fenlason wrote:
>>>
>>> Yes, to be clear, I wasn't imagining changing the design, and even
>>> smoothing changes would I think have to be very minor - I think all
>>> would agree that trying to redesign during QA would be bad practice
>>> for a variety of reasons.  What's more, "comments on the changes"  
>>> was
>>> not what I would hope to get from the UX group.  I would be looking
>>> for systematic user testing with an analysis of results appropriate
>>> to
>>> expertise in the discipline.  It's not going to just be designers
>>> weighing in with their opinions, or I will have considered it a
>>> failure.
>>>
>>> The main aim is to bring some heft to a claim in the release notes
>>> that we have Forums UI improvements. I'm hoping that our product
>>> releases have clear rationales as a general rule - and demonstrated
>>> results - for the changes they introduce.  It seems only proper for
>>> our QA process to weigh a piece of work on relevant metrics of
>>> success, and for a UX improvement something will be missed if we  
>>> only
>>> look at technical failures, while at the same time I think it could
>>> also represent an opportunity for our UX community to ground itself
>>> in
>>> best practices in this area, as well as educate the rest of the
>>> community on how to do this work well (i.e. so that there is
>>> documentation that can be used by development teams for their
>>> projects
>>> in future).
>>>
>>> ~Clay
>>>
>>> On Tue, Jul 28, 2009 at 11:59 AM, David Haines<dlhaines at umich.edu>
>>> wrote:
>>>
>>> We're fully open to comments on the changes, but Michigan is  
>>> unlikely
>>>
>>> to find resources to be able to modify the current scope of the
>>>
>>> modifications beyond smoothing rough edges.  User testing data would
>>>
>>> certainly inform any future set of changes.
>>>
>>> Is there a timeline for getting feedback? It would be good to move
>>> the
>>>
>>> process along quickly (but not hastily).
>>>
>>> - Dave
>>>
>>> David Haines
>>>
>>> CTools Developer
>>>
>>> Digital Media Commons
>>>
>>> University of Michigan
>>>
>>> dlhaines at umich.edu
>>>
>>>
>>>
>>>
>>> On Jul 27, 2009, at 1:40 PM, Clay Fenlason wrote:
>>>
>>> I was reviewing the excellent documentation for a set of Forums UI
>>>
>>> changes (see http://confluence.sakaiproject.org/x/EoLgAw), and it
>>>
>>> reminded me of an old idea. We tend to test for technical failure
>>>
>>> foremost, but the main claim of the Forums UI work is that it
>>> improves
>>>
>>> usability. It would seem to me sound QA practice to test this, using
>>>
>>> community expertise in good user testing practice.  Could this be an
>>>
>>> area where the UX group and QA WG could join forces? Would anyone
>>> like
>>>
>>> to take a look at proposed 2.7 changes that have a clear UI impact
>>> and
>>>
>>> conduct reviews or testing along these lines?
>>>
>>> There's something that may feel a little misplaced in vetting a
>>> design
>>>
>>> when development has been essentially completed, I know, but I think
>>>
>>> it would help the community to have strong confirmation of the
>>>
>>> imagined benefits, and if there are still small rough edges to  
>>> smooth
>>>
>>> over it may not be too late to address, say, CSS and HTML during QA.
>>>
>>> ~Clay
>>>
>>> _______________________________________________
>>>
>>> sakai-ux mailing list
>>>
>>> sakai-ux at collab.sakaiproject.org
>>>
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-ux
>>>
>>> TO UNSUBSCRIBE: send email to sakai-ux-unsubscribe at collab.sakaiproject.org
>>>
>>> with a subject of "unsubscribe"
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> sakai-qa mailing list
>>>
>>> sakai-qa at collab.sakaiproject.org
>>>
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-qa
>>>
>>> TO UNSUBSCRIBE: send email to sakai-qa-unsubscribe at collab.sakaiproject.org
>>> with a subject of "unsubscribe"
>>>
>>> _______________________________________________
>>> sakai-dev mailing list
>>> sakai-dev at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>
>>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org
>>> with a subject of "unsubscribe"
>>>
>>>
>>>
>>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org
>> with a subject of "unsubscribe"
>
> -- 
> Michael Korcuska
> Executive Director, Sakai Foundation
> mkorcuska at sakaifoundation.org
> phone: +1 510-931-6559
> mobile (US): +1 510-599-2586
> skype: mkorcuska
>
>
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org 
>  with a subject of "unsubscribe"
>
>