[Building Sakai] [WG: Sakai QA] [DG: User Experience] QA and user testing

Fri Jul 31 16:24:12 PDT 2009

I've posted some proposed procedural details in that other thread that
I think answer this and other questions.  It's the end of the day on a
Friday, though, so the note was dashed off hastily.  I'm sure someone
will let me know where it's confusing or if there are important
details left out. Pretty please.

~Clay

On Fri, Jul 31, 2009 at 1:56 PM, David Haines<dlhaines at umich.edu> wrote:
> Both Clay's and Michael's points seem right on target to me.
>
> Can we set an expectation of when a decision to add these Forums to the 2.7
> code base will be made?  The end of August seems like a reasonable time
> frame to me.  It provides several weeks for reading documentation and code
> and for getting responses to questions without letting the process go on
> indefinitely. (The user testing is very important but since it is expected
> only to effect the decision if a blocker is unexpectedly found it shouldn't
> be a prerequisite to making a decision.) If we don't get the consideration
> pipeline running soon we'll end up faced with a lot of decisions that need
> to be made quickly when the freeze date approaches.
>
> - Dave
>
> David Haines
> CTools Developer
> Digital Media Commons
> University of Michigan
> dlhaines at umich.edu
>
>
>
>
> On Jul 31, 2009, at 1:02 PM, Michael Korcuska wrote:
>
>> There's a further point I want to emphasize here. We want increased
>> attention to design in the development process. We'll have to work out
>> the details together as we move forward but I think we eventually want
>> to arrive at a place where designs are available for community review
>> while there is still time to do something with that feedback. And we
>> want project teams to build this thinking into the project plan. There
>> are lots of variations in how this might work, from waterfall to
>> agile, and this isn't the space for that discussion. Again, lots of
>> detail to work out, but I think the destination is relatively clear.
>>
>> Of course we can't jump to that destination overnight, especially in
>> the context of Sakai 2.x. There's a difference between "meets our
>> ideal standards" and "is a meaningful improvement over what we have
>> today". We'll likely put more emphasis on the latter in Sakai 2 and
>> more on the former in Sakai 3.
>>
>> All of this takes practice, though, and in part I see Clay's desire to
>> do this with Forums as part exercising the processes we're trying to
>> establish to see what works in practice. So its a test case, of sorts,
>> for a small part of the overall process.
>>
>> Michael
>>
>> On Jul 31, 2009, at 09:39, Clay Fenlason wrote:
>>
>>> I think you're hoping for a small answer, but I think I need to
>>> attempt a big one anyway, though I hope it still manages to be
>>> incisive on the point of your concern.
>>>
>>>> Is the usability review a part of the decision about accepting the
>>>> changes into the release,
>>>> or is it to collect information that could improve documentation
>>>> and inform future changes?
>>>
>>> The second, especially since we haven't done this before as a
>>> community.  It's too late in the game to think about using this sort
>>> of thing as gateway criteria for 2.7, not to mention that user testing
>>> should be part of the design process at the outset. Again I think the
>>> best way to think of it is as analogous to other QA work. We're
>>> looking for rough edges that might be easily sanded down, not a new
>>> bar that needs to be vaulted. And yes, we're trying also to share
>>> expertise about how to do this sort of work well. By extension, we're
>>> also trying to work out more ways for the expertise in our UX
>>> community to have a practical impact on the evolution of the product,
>>> and as it does so perhaps even make our UX community a more engaging
>>> place to be. My interest in this area is more longer term than focused
>>> on 2.7 in particular.
>>>
>>> That said, if user testing reveals 'blocker' issues (where that means
>>> essentially what it has always meant - a consensus that the severity
>>> is such that it shouldn't be released in that form), then I think we
>>> would have to take that seriously, and deal with it as we do other
>>> blockers.  That seems a highly unlikely outcome to me in the case of
>>> the Forums interface work we've mentioned, since much of the work has
>>> emerged already from conversations between schools using Forums in
>>> production, and Michigan's even been running this particular branch.
>>> User testing may reveal UX issues of lesser severity, and these could
>>> form documentation for what a next round of improvements should
>>> tackle. That's my picture of how this could work, at any rate.
>>>
>>> On another note, it's been disappointing over the years to see how
>>> quickly usability and accessibility issues are demoted to "minor" in
>>> JIRA reporting.  By the same token I've been underwhelmed in the past
>>> with some of our handwavy release claims of "UI enhancements."  We've
>>> done relatively little to substantiate such claims or prepare the
>>> training and support staff at partner institutions for how they'll
>>> need to adapt, even if it is a clear improvement. So the documentation
>>> in my view should not only be about best practices for future
>>> development, it should also result in more substantive release
>>> documentation for deploying institutions trying to plan whether or
>>> when to upgrade.
>>>
>>> HTH
>>>
>>> ~Clay
>>>
>>> On Fri, Jul 31, 2009 at 10:06 AM, David Haines<dlhaines at umich.edu>
>>> wrote:
>>>>
>>>> It would be good to be completely clear here about the purpose of
>>>> this
>>>> review in the context of the 2.7 release: Is the usability review a
>>>> part of
>>>> the decision about accepting the changes into the release, or is it
>>>> to
>>>> collect information that could improve documentation and inform
>>>> future
>>>> changes?
>>>> I believe it is the second, but I'd like to be sure I'm not just
>>>> jumping to
>>>> that conclusion.
>>>> - Dave
>>>>
>>>> David Haines
>>>> CTools Developer
>>>> Digital Media Commons
>>>> University of Michigan
>>>> dlhaines at umich.edu
>>>>
>>>>
>>>> On Jul 28, 2009, at 12:52 PM, Clay Fenlason wrote:
>>>>
>>>> Yes, to be clear, I wasn't imagining changing the design, and even
>>>> smoothing changes would I think have to be very minor - I think all
>>>> would agree that trying to redesign during QA would be bad practice
>>>> for a variety of reasons.  What's more, "comments on the changes" was
>>>> not what I would hope to get from the UX group.  I would be looking
>>>> for systematic user testing with an analysis of results appropriate
>>>> to
>>>> expertise in the discipline.  It's not going to just be designers
>>>> weighing in with their opinions, or I will have considered it a
>>>> failure.
>>>>
>>>> The main aim is to bring some heft to a claim in the release notes
>>>> that we have Forums UI improvements. I'm hoping that our product
>>>> releases have clear rationales as a general rule - and demonstrated
>>>> results - for the changes they introduce.  It seems only proper for
>>>> our QA process to weigh a piece of work on relevant metrics of
>>>> success, and for a UX improvement something will be missed if we only
>>>> look at technical failures, while at the same time I think it could
>>>> also represent an opportunity for our UX community to ground itself
>>>> in
>>>> best practices in this area, as well as educate the rest of the
>>>> community on how to do this work well (i.e. so that there is
>>>> documentation that can be used by development teams for their
>>>> projects
>>>> in future).
>>>>
>>>> ~Clay
>>>>
>>>> On Tue, Jul 28, 2009 at 11:59 AM, David Haines<dlhaines at umich.edu>
>>>> wrote:
>>>>
>>>> We're fully open to comments on the changes, but Michigan is unlikely
>>>>
>>>> to find resources to be able to modify the current scope of the
>>>>
>>>> modifications beyond smoothing rough edges.  User testing data would
>>>>
>>>> certainly inform any future set of changes.
>>>>
>>>> Is there a timeline for getting feedback? It would be good to move
>>>> the
>>>>
>>>> process along quickly (but not hastily).
>>>>
>>>> - Dave
>>>>
>>>> David Haines
>>>>
>>>> CTools Developer
>>>>
>>>> Digital Media Commons
>>>>
>>>> University of Michigan
>>>>
>>>> dlhaines at umich.edu
>>>>
>>>>
>>>>
>>>>
>>>> On Jul 27, 2009, at 1:40 PM, Clay Fenlason wrote:
>>>>
>>>> I was reviewing the excellent documentation for a set of Forums UI
>>>>
>>>> changes (see http://confluence.sakaiproject.org/x/EoLgAw), and it
>>>>
>>>> reminded me of an old idea. We tend to test for technical failure
>>>>
>>>> foremost, but the main claim of the Forums UI work is that it
>>>> improves
>>>>
>>>> usability. It would seem to me sound QA practice to test this, using
>>>>
>>>> community expertise in good user testing practice.  Could this be an
>>>>
>>>> area where the UX group and QA WG could join forces? Would anyone
>>>> like
>>>>
>>>> to take a look at proposed 2.7 changes that have a clear UI impact
>>>> and
>>>>
>>>> conduct reviews or testing along these lines?
>>>>
>>>> There's something that may feel a little misplaced in vetting a
>>>> design
>>>>
>>>> when development has been essentially completed, I know, but I think
>>>>
>>>> it would help the community to have strong confirmation of the
>>>>
>>>> imagined benefits, and if there are still small rough edges to smooth
>>>>
>>>> over it may not be too late to address, say, CSS and HTML during QA.
>>>>
>>>> ~Clay
>>>>
>>>> _______________________________________________
>>>>
>>>> sakai-ux mailing list
>>>>
>>>> sakai-ux at collab.sakaiproject.org
>>>>
>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-ux
>>>>
>>>> TO UNSUBSCRIBE: send email to
>>>> sakai-ux-unsubscribe at collab.sakaiproject.org
>>>>
>>>> with a subject of "unsubscribe"
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>>
>>>> sakai-qa mailing list
>>>>
>>>> sakai-qa at collab.sakaiproject.org
>>>>
>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-qa
>>>>
>>>> TO UNSUBSCRIBE: send email to
>>>> sakai-qa-unsubscribe at collab.sakaiproject.org
>>>> with a subject of "unsubscribe"
>>>>
>>>> _______________________________________________
>>>> sakai-dev mailing list
>>>> sakai-dev at collab.sakaiproject.org
>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>
>>>> TO UNSUBSCRIBE: send email to
>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>>> with a subject of "unsubscribe"
>>>>
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> sakai-dev mailing list
>>> sakai-dev at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>
>>> TO UNSUBSCRIBE: send email to
>>> sakai-dev-unsubscribe at collab.sakaiproject.org
>>> with a subject of "unsubscribe"
>>
>> --
>> Michael Korcuska
>> Executive Director, Sakai Foundation
>> mkorcuska at sakaifoundation.org
>> phone: +1 510-931-6559
>> mobile (US): +1 510-599-2586
>> skype: mkorcuska
>>
>>
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>>
>>
>
>