[WG: Sakai QA] [Building Sakai] [DG: User Experience] QA and user testing

[Clay Fenlason]

I think you're hoping for a small answer, but I think I need to
attempt a big one anyway, though I hope it still manages to be
incisive on the point of your concern.

> Is the usability review a part of the decision about accepting the changes into the release,
> or is it to collect information that could improve documentation and inform future changes?

The second, especially since we haven't done this before as a
community.  It's too late in the game to think about using this sort
of thing as gateway criteria for 2.7, not to mention that user testing
should be part of the design process at the outset. Again I think the
best way to think of it is as analogous to other QA work. We're
looking for rough edges that might be easily sanded down, not a new
bar that needs to be vaulted. And yes, we're trying also to share
expertise about how to do this sort of work well. By extension, we're
also trying to work out more ways for the expertise in our UX
community to have a practical impact on the evolution of the product,
and as it does so perhaps even make our UX community a more engaging
place to be. My interest in this area is more longer term than focused
on 2.7 in particular.

That said, if user testing reveals 'blocker' issues (where that means
essentially what it has always meant - a consensus that the severity
is such that it shouldn't be released in that form), then I think we
would have to take that seriously, and deal with it as we do other
blockers.  That seems a highly unlikely outcome to me in the case of
the Forums interface work we've mentioned, since much of the work has
emerged already from conversations between schools using Forums in
production, and Michigan's even been running this particular branch.
User testing may reveal UX issues of lesser severity, and these could
form documentation for what a next round of improvements should
tackle. That's my picture of how this could work, at any rate.

On another note, it's been disappointing over the years to see how
quickly usability and accessibility issues are demoted to "minor" in
JIRA reporting.  By the same token I've been underwhelmed in the past
with some of our handwavy release claims of "UI enhancements."  We've
done relatively little to substantiate such claims or prepare the
training and support staff at partner institutions for how they'll
need to adapt, even if it is a clear improvement. So the documentation
in my view should not only be about best practices for future
development, it should also result in more substantive release
documentation for deploying institutions trying to plan whether or
when to upgrade.

HTH

~Clay

On Fri, Jul 31, 2009 at 10:06 AM, David Haines<dlhaines at umich.edu> wrote:
> It would be good to be completely clear here about the purpose of this
> review in the context of the 2.7 release: Is the usability review a part of
> the decision about accepting the changes into the release, or is it to
> collect information that could improve documentation and inform future
> changes?
> I believe it is the second, but I'd like to be sure I'm not just jumping to
> that conclusion.
> - Dave
>
> David Haines
> CTools Developer
> Digital Media Commons
> University of Michigan
> dlhaines at umich.edu
>
>
> On Jul 28, 2009, at 12:52 PM, Clay Fenlason wrote:
>
> Yes, to be clear, I wasn't imagining changing the design, and even
> smoothing changes would I think have to be very minor - I think all
> would agree that trying to redesign during QA would be bad practice
> for a variety of reasons.  What's more, "comments on the changes" was
> not what I would hope to get from the UX group.  I would be looking
> for systematic user testing with an analysis of results appropriate to
> expertise in the discipline.  It's not going to just be designers
> weighing in with their opinions, or I will have considered it a
> failure.
>
> The main aim is to bring some heft to a claim in the release notes
> that we have Forums UI improvements. I'm hoping that our product
> releases have clear rationales as a general rule - and demonstrated
> results - for the changes they introduce.  It seems only proper for
> our QA process to weigh a piece of work on relevant metrics of
> success, and for a UX improvement something will be missed if we only
> look at technical failures, while at the same time I think it could
> also represent an opportunity for our UX community to ground itself in
> best practices in this area, as well as educate the rest of the
> community on how to do this work well (i.e. so that there is
> documentation that can be used by development teams for their projects
> in future).
>
> ~Clay
>
> On Tue, Jul 28, 2009 at 11:59 AM, David Haines<dlhaines at umich.edu> wrote:
>
> We're fully open to comments on the changes, but Michigan is unlikely
>
> to find resources to be able to modify the current scope of the
>
> modifications beyond smoothing rough edges.  User testing data would
>
> certainly inform any future set of changes.
>
> Is there a timeline for getting feedback? It would be good to move the
>
> process along quickly (but not hastily).
>
> - Dave
>
> David Haines
>
> CTools Developer
>
> Digital Media Commons
>
> University of Michigan
>
> dlhaines at umich.edu
>
>
>
>
> On Jul 27, 2009, at 1:40 PM, Clay Fenlason wrote:
>
> I was reviewing the excellent documentation for a set of Forums UI
>
> changes (see http://confluence.sakaiproject.org/x/EoLgAw), and it
>
> reminded me of an old idea. We tend to test for technical failure
>
> foremost, but the main claim of the Forums UI work is that it improves
>
> usability. It would seem to me sound QA practice to test this, using
>
> community expertise in good user testing practice.  Could this be an
>
> area where the UX group and QA WG could join forces? Would anyone like
>
> to take a look at proposed 2.7 changes that have a clear UI impact and
>
> conduct reviews or testing along these lines?
>
> There's something that may feel a little misplaced in vetting a design
>
> when development has been essentially completed, I know, but I think
>
> it would help the community to have strong confirmation of the
>
> imagined benefits, and if there are still small rough edges to smooth
>
> over it may not be too late to address, say, CSS and HTML during QA.
>
> ~Clay
>
> _______________________________________________
>
> sakai-ux mailing list
>
> sakai-ux at collab.sakaiproject.org
>
> http://collab.sakaiproject.org/mailman/listinfo/sakai-ux
>
> TO UNSUBSCRIBE: send email to sakai-ux-unsubscribe at collab.sakaiproject.org
>
>  with a subject of "unsubscribe"
>
>
>
> _______________________________________________
>
> sakai-qa mailing list
>
> sakai-qa at collab.sakaiproject.org
>
> http://collab.sakaiproject.org/mailman/listinfo/sakai-qa
>
> TO UNSUBSCRIBE: send email to sakai-qa-unsubscribe at collab.sakaiproject.org
> with a subject of "unsubscribe"
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org
> with a subject of "unsubscribe"
>
>
>
>