[DG: Teaching & Learning] Fwd: UIEtips: 3 Questions You Shouldn't Ask During User Research

Wed Aug 18 15:49:05 PDT 2010

Thanks, Daphne. Interesting points, especially the one about asking for
future behavior.

I'm not sure that works in our domain, or at least that we need a different
rule of thumb than asking about past behavior. I'm one of those who thinks
that past behavior may be a big part of the problem. In my experience with
that sort of question (not extensive, but we've done a few rounds in the HCI
lab) I get descriptions of ingenious workarounds that seem like they might
be as unhelpful to the user research as speculating about a rational future.
For now, I tend to try to steer them toward describing what problem it is
they're trying to solve as directly as they can, on the assumption that none
of the tools they have available to them really speaks to that problem.

~Clay

On Wed, Aug 18, 2010 at 6:23 PM, Daphne Ogle <daphne at media.berkeley.edu>wrote:

> UIE talks about 3 common pitfalls in the kinds of questions we ask when
> doing user research.  I know I've fallen into these traps as it seems
> intuitive to ask users what they might do or how they would design something
> or putting words in their mouths by deducing from what we watched them do.
> Sometimes the hardest part of good research is not doing what might feel
> natural :).
>
> Enjoy!
>
> -Daphne
>
>
> Begin forwarded message:
>
> *From: *"Jared M. Spool" <jared.m.spool at uie.com>
> *Date: *August 18, 2010 1:17:46 PM PDT
> *To: *daphne at media.berkeley.edu
> *Subject: **UIEtips: 3 Questions You Shouldn't Ask During User Research*
> *Reply-To: *jared.m.spool at uie.com
>
> [image: Header Image]UIEtips: 3 Questions You Shouldn't Ask During User
> ResearchAugust 18, 2010
> [image: Horizontal Rule Image]Three Questions You Shouldn't Ask During
> User Research[image: Jared Spool]Jared Spool, User Interface Engineering
>
> The participant was struggling. While he was a high-volume customer who had
> bought tons from this site in the past, today he wasn't getting along with
> the checkout process. Confusion happened in both directions: the shopper
> didn't understand what the site was trying to tell him and the site
> definitely didn't understand what he wanted.
>
> The team spent several painful minutes watching as this poor customer made
> slow progress. His determination made the purchase happen, but it wasn't
> smooth.
>
> "It's usually not this bad," the customer shared. "I've had problems
> before, but this feels especially difficult today." It was just a perfect
> storm of things-not-going-well.
>
> Sitting next to me was a team member who was becoming increasingly anxious.
> As the e-commerce site's VP of Customer Service, he could feel this
> customer's pain. This wasn't the experience he wanted their customers to
> have, especially a loyal, regular customer like this one.
>
> At the appropriate moment, the session's moderator turned to the observers
> and asked, "Does anyone have any questions?"
>
> "Yes, I do," exploded the VP. "Do you think... I mean, if you were at home
> and you went through all this and had all this trouble... and there wasn't a
> room full of people watching you... do you think you would've called the 800
> number that was on every screen? Would you have called our support
> department?"
>
> The VP wanted to believe that, when customers have a traumatic experience
> like this, they don't just let it go—they call his people for help. After
> all, he's made sure they're trained to handle exactly this type of problem
> and shower the customer with great attention and results. The guy has to
> want to call.
>
> The customer thought about it carefully for a few moments. You could almost
> see into his mind as he recreated the situation in the comfort of his own
> home. Finally, he said, in a very calm voice, "I think so. Yes. I would've
> called the 800 number."
>
> The VP sat back in his chair, happy with the thought his team would be
> there for the rescue.
>
> Then I asked, "Have you ever called the support number when you've tried to
> purchase things in the past?"
>
> "No, I haven't," the customer answered immediately.
>
> That's the puzzle: The customer said he would call support, yet also said
> he had never done so in the past, even though he's had similar problems.
> Which should we believe?
> 1) Don't Do: Asking About the Future
>
> Unknowingly, the VP broke one of the rules of good participant questions:
> Don't ask about the future.
>
> When asked about a future scenario, many people will try to answer
> honestly. They'll answer about what they'd like as their ideal behavior.
> They want to think they'll behave in the most logical, effective manner. In
> short, they want to think they'll do the right thing.
>
> Yet, in the thick of the situation, people aren't always so logical. They
> don't do what's ideal.
>
> We want to predict future behaviors, so we can make designs that service
> them. Yet just asking participants may not get the actual answer.
>
> I've found the best way to predict future behavior is to look at past
> behavior. Asking a participant about what they've done in the past is a
> better way to get those answer.
>
> Instead of asking, "do you think you might need to store messages in
> different folders?", I've found it better to ask, "when you're done with
> messages, what do you do with them?" I focus on what they've done in the
> past, looking for that behavior to tell me what makes sense.
>
> Asking about the future is just one type of question I've learned we
> shouldn't ask our research participants.
> 2) Don't Do: Asking How They'd Design a Feature
>
> "If we gave you a way to annotate your files, how would that work?" "What
> would be the right way to organize the menu options?" "What other fields
> should we include on this form?" "How would you want to see this data
> displayed?"
>
> I've heard these questions many times. (I've even asked them on occasion.)
> After all, the designers want to create an effective design for the user.
> The best way is to just ask the participant what that design should be,
> right? Wrong.
>
> Users aren't designers. (If they were, they wouldn't need us.) They don't
> know how to deal with constraints. They haven't really thought the problem
> through.
>
> Any answer they give us is unlikely to be a great design. It'll be the
> first thing they think of and, unfortunately, they'll take a long time to
> get there.
>
> I saw this happen just the other day. A smart team was talking with a
> long-time customer of their product. "How would you..." the team leader
> asked.
>
> The customer's posture changed completely. Whereas before they'd been
> leaning towards the team members with solid eye contact for every
> interaction, they now relaxed in their chair and looked into space. "Hmmm. I
> guess I'd..." as they tried to imagine their brilliant idea.
>
> Except, it wasn't a brilliant idea. It was simplistic and couldn't handle
> the myriad of edge conditions that were patently obvious.
>
> The team could see the original problem and asking how to design it wasn't
> adding anything. They knew what to do.
> 3) Don't Do: Asking by Providing Their Reason
>
> "Is the reason you don't click this button because it's really hard to
> see?"
>
> This is a tough one for many observers. We're watching the participant and
> they aren't doing what we know to be the optimal method. Our brain races
> with possible reasons. Need to find out. Must ask.
>
> "Why didn't you click this button?" is only a hair better. It doesn't
> telegraph the reasons we're hypothesizing, but it implies the person has
> done something wrong. They'll tell you a reason, but it may be more because
> you've pointed it out, not because it's what they were actually thinking.
>
> Be creative about exploring the participant's understanding of the
> interface. For example, when I'm curious about why they didn't click a
> particular button, I ask them to talk about what each button does. That way,
> they won't associate the specific button with the action they just did (and
> I learn their overall understanding of the controls).
> One Bad Question Won't End The World
>
> Asking these questions isn't evil. We're not violating some
> Geneva-Convention-type of international agreement. We won't be hauled away
> by the User Research Police.
>
> However, there is a price to asking the wrong questions. When conducting
> user research, the most valuable moments are limited times that the team
> spends with each participant. It's important to make every second count.
>
> Asking one of these questions wastes those precious moments. They take time
> without returning value. The participant tries (and, usually tries really
> hard), but that trying doesn't pay off and eats down the clock.
>
> Learning to focus on the right questions can help you get the most out of
> each session.
> • • •
>
> Jared's article<http://r20.rs6.net/tn.jsp?et=1103623327778&s=4315&e=001XVL6_rbfznTK3hMUCmv7NbBEriMAR1QeaqiixQNzY3MxZhDdn5o_O2JUKk0Ng1n-T98cXgU7L0GoqZ4NJwYdMNOy5IbD4LoUxM_E9r66qfYKCjlRIc0WzGMD9mGuNCrXhFC3kI210aG_BWg72IGCkQRyCOK-N6mo>
>  is also available online.
> • • •
>
> Once you've learned what you need from your research, you'll want to put it
> in a form that helps you speed through your design process. That's exactly
> what scenarios help you do and, coincidentally, why we've asked Kim Goodwin
> to come to UI15. She'll teach us how to make scenarios work. You can find
> out about Kim and the other great UI15 experts at UICONF.com<http://r20.rs6.net/tn.jsp?et=1103623327778&s=4315&e=001XVL6_rbfznRbUK3F4AXyKmLiAHQwbn6OwkQBUqgwGoNycuQuzJaNW1SGhAGgt1-GzyBOcwJu07qoaWecSfE7t4mJucMDSfSIt52GmQC6jdQ=>
> .
> • • •
>
> Have you compiled your own questions that you shouldn't ask? Share your
> list at the UIE Brain Sparks blog<http://r20.rs6.net/tn.jsp?et=1103623327778&s=4315&e=001XVL6_rbfznRYIyhqMVBM5ujwl5rA4rpq5DRc7WNiQ27AKq24UC4g90oovsi1BSGuf3HMOcmd1a7ddFcUiy8knmK5kWs2wYKGOp6fO48HCD2DX-NQcRyBPyVpKmfLuAJFGj3OGsJE0xBAGuqzJPFnscyVCcA1dHp7dN1nSr5icvxxKXzhHoInSEFl1BFvawO8>
> .
> [image: Horizontal Rule Image]UI15: Where Good Designers Go to Become
> Great Designers
>
> Response to the User Interface 15 Conference<http://r20.rs6.net/tn.jsp?et=1103623327778&s=4315&e=001XVL6_rbfznRbUK3F4AXyKmLiAHQwbn6OwkQBUqgwGoNycuQuzJaNW1SGhAGgt1-GzyBOcwJu07qoaWecSfE7t4mJucMDSfSIt52GmQC6jdQ=>
>  has been amazing. During the sneak preview alone, we've had over 100
> people save their spot and take advantage of the lowest rate for all 3 days.
>
> If you've been thinking about attending UI15 in Boston on November 8-10,
> now is the time to register. This is the lowest rate you'll find, and you'll
> be guaranteed to get the workshops of your choice. Get the details at
> UICONF.com<http://r20.rs6.net/tn.jsp?et=1103623327778&s=4315&e=001XVL6_rbfznRbUK3F4AXyKmLiAHQwbn6OwkQBUqgwGoNycuQuzJaNW1SGhAGgt1-GzyBOcwJu07qoaWecSfE7t4mJucMDSfSIt52GmQC6jdQ=>
> .
> Why UI15 is Different
>
> The User Interface Conference is different from many UX conferences. Here's
> why:
>
>    1. You attend 2 full-day workshops. No skimming a topic for 45 minutes
>    or 60 minutes of "everything you need to know" presentations. We're talking
>    about deep dive, intense workshops on specific topics like scenarios,
>    content strategy, and visual thinking.
>    2. You don't just sit in these workshops—you work. The workshops are
>    made-up of lectures, hands-on exercises, and group discussions. If you're
>    expecting to sit back, relax, and listen to someone talk for 6 plus hours,
>    this conference may not be what you're looking for.
>    3. We know it's difficult to choose just two speakers to hear so we put
>    together a day between the workshops where you can sample other presenters.
>    Each presenter highlights 90 minutes of their material so you can get a
>    taste of what they have to offer.
>    4. There is no other conference that offers full-day workshops on
>    building personas, visual thinking, content strategy, visual design, new
>    techniques to communicate ideas, web form design, and designing scenarios,
>    components and libraries.
>
> Register now<http://r20.rs6.net/tn.jsp?et=1103623327778&s=4315&e=001XVL6_rbfznQDXavt2KFQd0jO1KvPHTtKQ4vYnW3y-b8hAKh6_6aAL5y5WWXKAETT7PODaVOzBBw_gTWIhI-_tre0cuWrwpzFWLtcMdI1YI2s7gVT7foLO40uEp2yju6ySksW_AD1Oqq-wseqk1OamoWiTHlJTB1b>
>  and pay the lowest rate of $1295. After August 26, the price goes up
> $400.
> [image: Horizontal Rule Image]Web App Design Advice from 11 Masters All in
> One Place
>
> Missed the Web App Masters Tour? No problem, we'll bring it to you.
>
> Here's your chance to hear 11 of the Tour Masters present on complex
> navigation, integrating social components, using design patterns, plus a
> range of topics that designers continually struggle with.
>
> Hear first hand about the design process that Facebook, 37signals, and
> Marriott incorporate. Share with your team methods to keep users engaged,
> steps needed to ensure a team's success, and the importance of creating an
> experience vision.
>
> When you purchase the Web App Masters Tour proceedings<http://r20.rs6.net/tn.jsp?et=1103623327778&s=4315&e=001XVL6_rbfznR_tuqrlUdlyolhP3XKyMSq6WSkfqJ0Xa0KbAV-lq9_zJ3Eib29WHC7SKO1NVT47nAs05lGKULDfe-xkqOczqbE5FX5QcqQ1QA=>
>  you'll get:
>
>    - In-depth presentation slides for 12 sessions from our world-renowned
>    speakers
>    - Audio recordings of all the presentations plus the Q&A from the
>    audience
>    - Podcasts of interviews with many of the Masters
>    - Discounts towards the books from our Masters
>
> Order Your Proceedings by August 31 and Save
>
> Save $50 when you order your proceedings by August 31. You'll get immediate
> access to the proceedings - no waiting. Down load the files and share them
> with everyone on your team.
>
> Learn more at UIETour.com<http://r20.rs6.net/tn.jsp?et=1103623327778&s=4315&e=001XVL6_rbfznR_tuqrlUdlyolhP3XKyMSq6WSkfqJ0Xa0KbAV-lq9_zJ3Eib29WHC7SKO1NVT47nAs05lGKULDfe-xkqOczqbE5FX5QcqQ1QA=>
> [image: Horizontal Rule Image]
>
> Get regular updates about UIE when you follow us on Twitter<http://r20.rs6.net/tn.jsp?et=1103623327778&s=4315&e=001XVL6_rbfznQsE6gecaYbAw7YKw0IAJEOUlKCBbD3PPGttoJEuOxroT59rG2aLGWMf9qxjid-Vvo6tyS8r3_KbuEJZCnGiotqatDnLauFWxqWGuBvxlS-hQ==>
>  or become a fan on Facebook<http://r20.rs6.net/tn.jsp?et=1103623327778&s=4315&e=001XVL6_rbfznQHs6WjiCNSjNl6zwJV4TpYVEB-FS8tqp1-a8KNEoMprSwO3IixdjJZofnMhHDQ8yYIqJYSdbDzx2iHOPxpkLNQtT0E8f_SEhsuZA7T5La1Qt0-DQaBMNwMW33kBhS6JdXX8esh1DBWaFtOTGIbbpoUvYz_ehnr1EfK1yo7tMSxMw==>
> .
> [image: Horizontal Rule Image]
>
> Do you have feedback or comments on our article? Send us your thoughts on
> our blog<http://r20.rs6.net/tn.jsp?et=1103623327778&s=4315&e=001XVL6_rbfznRYIyhqMVBM5ujwl5rA4rpq5DRc7WNiQ27AKq24UC4g90oovsi1BSGuf3HMOcmd1a7ddFcUiy8knmK5kWs2wYKGOp6fO48HCD2DX-NQcRyBPyVpKmfLuAJFGj3OGsJE0xBAGuqzJPFnscyVCcA1dHp7dN1nSr5icvxxKXzhHoInSEFl1BFvawO8>
> .
>
> *Forward email<http://ui.constantcontact.com/sa/fwtf.jsp?m=1102639163843&ea=daphne%40media.berkeley.edu&a=1103623327778>
> *
>
> [image: Safe Unsubscribe]<http://visitor.constantcontact.com/d.jsp?p=un&v=001pRmNkNNhb3_jCFmrxhX3x2Tl4k0LIn-Rqv8FuviJmG8ZCdwTPEEo5hVv69sD2Tmc>
> This email was sent to daphne at media.berkeley.edu by jared.m.spool at uie.com.
> Update Profile/Email Address<http://visitor.constantcontact.com/d.jsp?p=oo&v=001pRmNkNNhb3_jCFmrxhX3x2Tl4k0LIn-Rqv8FuviJmG8ZCdwTPEEo5hVv69sD2Tmc>
>  | Instant removal with SafeUnsubscribe<http://visitor.constantcontact.com/d.jsp?p=un&v=001pRmNkNNhb3_jCFmrxhX3x2Tl4k0LIn-Rqv8FuviJmG8ZCdwTPEEo5hVv69sD2Tmc>™
> | Privacy Policy<http://ui.constantcontact.com/roving/CCPrivacyPolicy.jsp>
> .
> Email Marketing <http://www.constantcontact.com/index.jsp?cc=custom01> by
> <http://www.constantcontact.com/index.jsp?cc=custom01>
> User Interface Engineering | 510 Turnpike Street | Suite 102 | North
> Andover | MA | 01845
>
>
> Daphne Ogle
> Senior Interaction Designer
> University of California, Berkeley
> Educational Technology Services
> daphne at media.berkeley.edu
> cell (925)348-4372
>
>
>
>
>
> _______________________________________________
> pedagogy mailing list
> pedagogy at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/pedagogy
>
> TO UNSUBSCRIBE: send email to pedagogy-unsubscribe at collab.sakaiproject.orgwith a subject of "unsubscribe"
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/pedagogy/attachments/20100818/239507d1/attachment-0001.html 