[DG: Teaching & Learning] Fwd: UIEtips: 3 Questions You Shouldn't Ask During User Research

[Daphne Ogle]

Great points!  Some thoughts below.

-Daphne

On Aug 18, 2010, at 3:49 PM, Clay Fenlason wrote:

> Thanks, Daphne. Interesting points, especially the one about asking  
> for future behavior.
>
> I'm not sure that works in our domain, or at least that we need a  
> different rule of thumb than asking about past behavior. I'm one of  
> those who thinks that past behavior may be a big part of the  
> problem. In my experience with that sort of question (not extensive,  
> but we've done a few rounds in the HCI lab) I get descriptions of  
> ingenious workarounds that seem like they might be as unhelpful to  
> the user research as speculating about a rational future. For now, I  
> tend to try to steer them toward describing what problem it is  
> they're trying to solve as directly as they can, on the assumption  
> that none of the tools they have available to them really speaks to  
> that problem.

This isn't a challenge particular to our environment.  Since much  
software doesn't work well for users it's a common issue.  Your onto  
something here about getting to the root of the problem.  Although,  
often it's hard for participants to describe the problem.  You know  
how that goes when you're in the middle of it and it can take quite a  
bit of reflection to get to the root of it all.  That's where the 3  
whys can help (keep asking why to get them deeper and deeper into the  
problem).

The other thing I'd say about asking about past behavior is that the  
goal of it isn't so much to try to duplicate what they've been doing  
but see what they've been doing and try to understand why.  What is  
the actual goal they are trying to reach?  Or as you get to above,  
what is the problem they are trying to solve.  Understanding the  
problem, their goals, their environment, constraints, etc. will then  
help us figure out better ways to enable reaching their goals -- or at  
least that's the ideal.  :)  And again, asking lots of questions.  If  
you're doing a contextual inquiry or user test where you want to watch  
users complete a task, having them go through it naturally once so you  
can see that natural uninterrupted flow is good and then asking them  
to go back and perhaps ask questions about what they did or why they  
did it.   Or like Jarod describes below, ask what they think something  
was supposed to do to get at what they were trying to do.   There are  
common methods, and tips and tricks to help do research well but I  
think it's for sure more an art than a science (unless you're doing  
big summative user testing).



>
> ~Clay
>
> On Wed, Aug 18, 2010 at 6:23 PM, Daphne Ogle <daphne at media.berkeley.edu 
> > wrote:
> UIE talks about 3 common pitfalls in the kinds of questions we ask  
> when doing user research.  I know I've fallen into these traps as it  
> seems intuitive to ask users what they might do or how they would  
> design something or putting words in their mouths by deducing from  
> what we watched them do.   Sometimes the hardest part of good  
> research is not doing what might feel natural :).
>
> Enjoy!
>
> -Daphne
>
>
> Begin forwarded message:
>
>> From: "Jared M. Spool" <jared.m.spool at uie.com>
>> Date: August 18, 2010 1:17:46 PM PDT
>> To: daphne at media.berkeley.edu
>> Subject: UIEtips: 3 Questions You Shouldn't Ask During User Research
>> Reply-To: jared.m.spool at uie.com
>>
>>
>> 	 UIEtips: 3 Questions You Shouldn't Ask During User Research
>> August 18, 2010
>>
>>
>>
>>
>> Three Questions You Shouldn't Ask During User Research
>>
>> 	
>> Jared Spool, User Interface Engineering
>>
>> The participant was struggling. While he was a high-volume customer  
>> who had bought tons from this site in the past, today he wasn't  
>> getting along with the checkout process. Confusion happened in both  
>> directions: the shopper didn't understand what the site was trying  
>> to tell him and the site definitely didn't understand what he wanted.
>>
>> The team spent several painful minutes watching as this poor  
>> customer made slow progress. His determination made the purchase  
>> happen, but it wasn't smooth.
>>
>> "It's usually not this bad," the customer shared. "I've had  
>> problems before, but this feels especially difficult today." It was  
>> just a perfect storm of things-not-going-well.
>>
>> Sitting next to me was a team member who was becoming increasingly  
>> anxious. As the e-commerce site's VP of Customer Service, he could  
>> feel this customer's pain. This wasn't the experience he wanted  
>> their customers to have, especially a loyal, regular customer like  
>> this one.
>>
>> At the appropriate moment, the session's moderator turned to the  
>> observers and asked, "Does anyone have any questions?"
>>
>> "Yes, I do," exploded the VP. "Do you think... I mean, if you were  
>> at home and you went through all this and had all this trouble...  
>> and there wasn't a room full of people watching you... do you think  
>> you would've called the 800 number that was on every screen? Would  
>> you have called our support department?"
>>
>> The VP wanted to believe that, when customers have a traumatic  
>> experience like this, they don't just let it go—they call his  
>> people for help. After all, he's made sure they're trained to  
>> handle exactly this type of problem and shower the customer with  
>> great attention and results. The guy has to want to call.
>>
>> The customer thought about it carefully for a few moments. You  
>> could almost see into his mind as he recreated the situation in the  
>> comfort of his own home. Finally, he said, in a very calm voice, "I  
>> think so. Yes. I would've called the 800 number."
>>
>> The VP sat back in his chair, happy with the thought his team would  
>> be there for the rescue.
>>
>> Then I asked, "Have you ever called the support number when you've  
>> tried to purchase things in the past?"
>>
>> "No, I haven't," the customer answered immediately.
>>
>> That's the puzzle: The customer said he would call support, yet  
>> also said he had never done so in the past, even though he's had  
>> similar problems. Which should we believe?
>>
>> 1) Don't Do: Asking About the Future
>>
>> Unknowingly, the VP broke one of the rules of good participant  
>> questions: Don't ask about the future.
>>
>> When asked about a future scenario, many people will try to answer  
>> honestly. They'll answer about what they'd like as their ideal  
>> behavior. They want to think they'll behave in the most logical,  
>> effective manner. In short, they want to think they'll do the right  
>> thing.
>>
>> Yet, in the thick of the situation, people aren't always so  
>> logical. They don't do what's ideal.
>>
>> We want to predict future behaviors, so we can make designs that  
>> service them. Yet just asking participants may not get the actual  
>> answer.
>>
>> I've found the best way to predict future behavior is to look at  
>> past behavior. Asking a participant about what they've done in the  
>> past is a better way to get those answer.
>>
>> Instead of asking, "do you think you might need to store messages  
>> in different folders?", I've found it better to ask, "when you're  
>> done with messages, what do you do with them?" I focus on what  
>> they've done in the past, looking for that behavior to tell me what  
>> makes sense.
>>
>> Asking about the future is just one type of question I've learned  
>> we shouldn't ask our research participants.
>>
>> 2) Don't Do: Asking How They'd Design a Feature
>>
>> "If we gave you a way to annotate your files, how would that work?"  
>> "What would be the right way to organize the menu options?" "What  
>> other fields should we include on this form?" "How would you want  
>> to see this data displayed?"
>>
>> I've heard these questions many times. (I've even asked them on  
>> occasion.) After all, the designers want to create an effective  
>> design for the user. The best way is to just ask the participant  
>> what that design should be, right? Wrong.
>>
>> Users aren't designers. (If they were, they wouldn't need us.) They  
>> don't know how to deal with constraints. They haven't really  
>> thought the problem through.
>>
>> Any answer they give us is unlikely to be a great design. It'll be  
>> the first thing they think of and, unfortunately, they'll take a  
>> long time to get there.
>>
>> I saw this happen just the other day. A smart team was talking with  
>> a long-time customer of their product. "How would you..." the team  
>> leader asked.
>>
>> The customer's posture changed completely. Whereas before they'd  
>> been leaning towards the team members with solid eye contact for  
>> every interaction, they now relaxed in their chair and looked into  
>> space. "Hmmm. I guess I'd..." as they tried to imagine their  
>> brilliant idea.
>>
>> Except, it wasn't a brilliant idea. It was simplistic and couldn't  
>> handle the myriad of edge conditions that were patently obvious.
>>
>> The team could see the original problem and asking how to design it  
>> wasn't adding anything. They knew what to do.
>>
>> 3) Don't Do: Asking by Providing Their Reason
>>
>> "Is the reason you don't click this button because it's really hard  
>> to see?"
>>
>> This is a tough one for many observers. We're watching the  
>> participant and they aren't doing what we know to be the optimal  
>> method. Our brain races with possible reasons. Need to find out.  
>> Must ask.
>>
>> "Why didn't you click this button?" is only a hair better. It  
>> doesn't telegraph the reasons we're hypothesizing, but it implies  
>> the person has done something wrong. They'll tell you a reason, but  
>> it may be more because you've pointed it out, not because it's what  
>> they were actually thinking.
>>
>> Be creative about exploring the participant's understanding of the  
>> interface. For example, when I'm curious about why they didn't  
>> click a particular button, I ask them to talk about what each  
>> button does. That way, they won't associate the specific button  
>> with the action they just did (and I learn their overall  
>> understanding of the controls).
>>
>> One Bad Question Won't End The World
>>
>> Asking these questions isn't evil. We're not violating some Geneva- 
>> Convention-type of international agreement. We won't be hauled away  
>> by the User Research Police.
>>
>> However, there is a price to asking the wrong questions. When  
>> conducting user research, the most valuable moments are limited  
>> times that the team spends with each participant. It's important to  
>> make every second count.
>>
>> Asking one of these questions wastes those precious moments. They  
>> take time without returning value. The participant tries (and,  
>> usually tries really hard), but that trying doesn't pay off and  
>> eats down the clock.
>>
>> Learning to focus on the right questions can help you get the most  
>> out of each session.
>>
>> • • •
>>
>> Jared's article is also available online.
>>
>> • • •
>>
>> Once you've learned what you need from your research, you'll want  
>> to put it in a form that helps you speed through your design  
>> process. That's exactly what scenarios help you do and,  
>> coincidentally, why we've asked Kim Goodwin to come to UI15. She'll  
>> teach us how to make scenarios work. You can find out about Kim and  
>> the other great UI15 experts at UICONF.com.
>>
>> • • •
>>
>> Have you compiled your own questions that you shouldn't ask? Share  
>> your list at the UIE Brain Sparks blog.
>>
>>
>> UI15: Where Good Designers Go to Become Great Designers
>>
>> Response to the User Interface 15 Conference has been amazing.  
>> During the sneak preview alone, we've had over 100 people save  
>> their spot and take advantage of the lowest rate for all 3 days.
>>
>> If you've been thinking about attending UI15 in Boston on November  
>> 8-10, now is the time to register. This is the lowest rate you'll  
>> find, and you'll be guaranteed to get the workshops of your choice.  
>> Get the details at UICONF.com.
>>
>> Why UI15 is Different
>>
>> The User Interface Conference is different from many UX  
>> conferences. Here's why:
>>
>> You attend 2 full-day workshops. No skimming a topic for 45 minutes  
>> or 60 minutes of "everything you need to know" presentations. We're  
>> talking about deep dive, intense workshops on specific topics like  
>> scenarios, content strategy, and visual thinking.
>>
>> You don't just sit in these workshops—you work. The workshops are  
>> made-up of lectures, hands-on exercises, and group discussions. If  
>> you're expecting to sit back, relax, and listen to someone talk for  
>> 6 plus hours, this conference may not be what you're looking for.
>>
>> We know it's difficult to choose just two speakers to hear so we  
>> put together a day between the workshops where you can sample other  
>> presenters. Each presenter highlights 90 minutes of their material  
>> so you can get a taste of what they have to offer.
>>
>> There is no other conference that offers full-day workshops on  
>> building personas, visual thinking, content strategy, visual  
>> design, new techniques to communicate ideas, web form design, and  
>> designing scenarios, components and libraries.
>>
>> Register now and pay the lowest rate of $1295. After August 26, the  
>> price goes up $400.
>>
>>
>> Web App Design Advice from 11 Masters All in One Place
>>
>> Missed the Web App Masters Tour? No problem, we'll bring it to you.
>>
>> Here's your chance to hear 11 of the Tour Masters present on  
>> complex navigation, integrating social components, using design  
>> patterns, plus a range of topics that designers continually  
>> struggle with.
>>
>> Hear first hand about the design process that Facebook, 37signals,  
>> and Marriott incorporate. Share with your team methods to keep  
>> users engaged, steps needed to ensure a team's success, and the  
>> importance of creating an experience vision.
>>
>> When you purchase the Web App Masters Tour proceedings you'll get:
>>
>> In-depth presentation slides for 12 sessions from our world- 
>> renowned speakers
>> Audio recordings of all the presentations plus the Q&A from the  
>> audience
>> Podcasts of interviews with many of the Masters
>> Discounts towards the books from our Masters
>> Order Your Proceedings by August 31 and Save
>>
>> Save $50 when you order your proceedings by August 31. You'll get  
>> immediate access to the proceedings - no waiting. Down load the  
>> files and share them with everyone on your team.
>>
>> Learn more at UIETour.com
>>
>>
>> Get regular updates about UIE when you follow us on Twitter or  
>> become a fan on Facebook.
>>
>>
>> Do you have feedback or comments on our article? Send us your  
>> thoughts on our blog.
>>
>>
>> Forward email
>>
>>
>> This email was sent to daphne at media.berkeley.edu by jared.m.spool at uie.com 
>> .
>> Update Profile/Email Address | Instant removal with  
>> SafeUnsubscribe™ | Privacy Policy.
>> Email Marketing by
>>
>>
>> User Interface Engineering | 510 Turnpike Street | Suite 102 |  
>> North Andover | MA | 01845
>>
>>
>
> Daphne Ogle
> Senior Interaction Designer
> University of California, Berkeley
> Educational Technology Services
> daphne at media.berkeley.edu
> cell (925)348-4372
>
>
>
>
>
> _______________________________________________
> pedagogy mailing list
> pedagogy at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/pedagogy
>
> TO UNSUBSCRIBE: send email to pedagogy-unsubscribe at collab.sakaiproject.org 
>  with a subject of "unsubscribe"
>

Daphne Ogle
Senior Interaction Designer
University of California, Berkeley
Educational Technology Services
daphne at media.berkeley.edu
cell (925)348-4372




-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/pedagogy/attachments/20100818/51d92783/attachment-0001.html