User Research & Data Analysis Starter Kit
Making Decisions From User Feedback Without Going Insane
Watch my recent lecture that goes with this blog:
As an XR Product Designer, I’ve been responsible for not just developing the concepts but also scoping out the development pipeline, doing user tests and distilling feedback into actionable development milestones.
This post will show you some basic concepts of managing data I’ve gained during user testing in the real (messy) world.
Before we get into it, remember to keep the high level perspective of what prototypes are for. You are trying to get to user value, not just go after your vision. Being a designer is part artist part scientist.
In order to get useful feedback, it’s helpful to have prototypes that demonstrate utility to your test users immediately. Don’t ask them to fill in the blanks of your vision, instead show them working user experiences that give them value right away while helping you validate what parts of your vision are working or not.
The prototypes are made, now time to show them to users!
But oh no, everyone is saying something different and giving me conflicting or even unclear feedback. What do I do? Use Data Coding.
Showing the same thing to multiple people will result in conflicting feedback. Don’t panic! Keep calm and ‘Code’ your data.
It does not take hard-core data science to turn qualitative data (subjective or non-structured transcripts) into quantitive data (structured, searchable, standardized).
Qualitative coding is a process of systematically categorizing excerpts in your qualitative data in order to find themes and patterns.
Take unstructured or semi-structured data such as transcripts from in-depth interviews or focus groups and structure it into themes and patterns for analysis.
Benefits of qualitative coding
Increase validity: Provides organization and structure to data so that you can examine it in a systematic way to increase the validity of your analysis.
Decrease bias: Enables awareness of potential biases in the way data is analyzed.
Accurately represent participants: allows you to evaluate if your analysis represents your participant base, and helps you avoid over representing one person or group of people.
Enable transparency: enables other researchers to methodically and systematically review your analysis.
There are two main approaches to deal with unstructured data: Inductive and Deductive coding.
Inductive is also known as ‘from the ground up’.
Derive your codes from the data. You don’t start with preconceived notions of what the codes should be, but allow the narrative or theory to emerge from the raw data itself. This is great for exploratory research or times when you want to come up with a new theories, ideas or concepts.
Deductive is also known as ‘from the top down’.
Start by developing a codebook with your initial set of codes.
This set could be based on your research questions or an existing research framework or theory. You then read through the data and assign excerpts to codes.
At the end of your analysis, your codes should still closely resemble the codebook that you started off with. This is good when you have a pre-determined structure for how you need your final findings to be.
For example, program evaluation studies may utilize a deductive coding approach.
In practice, research studies often combine both deductive and inductive approaches to coding. For example, you could deductively start with a set of codes, but then inductively come up with new codes and iterate on the codes as you sift through your data.
Your raw feedback can transform into proper metrics and give visibility to stakeholders on the leadership team and the development team, all while keeping you sane!
Some tips for using data codes aka tags.
Design 101 - User Testing & Feedback
Simple method: basic doc, use codes for different moments.
I’ve used some ones like
HMW - How Might We (User asks something very high level that is outside the scope of the experiment and/or product team’s roadmap, or even imagination)
UX - Comments about the user experience, positive and negative
UI - Specific UI issues or confusions, suggestions, etc
Bug - Not expected behavior or issue (fast triage to dev team)
Quote - specific user quotes that are important to keep and forward (good and bad)
Post Experimental Review.
Organize by your content by code, triage feedback.
Synthesize themes, translate valid requests into Product Requirements Documentation (PRD)
There are often constraints and/or requirements that must be taken into account in the design and development of any given technology solution.
A PRD specifies the exact details of what will be implemented.
Basic PRDs can contain and track:
Requirements
Research / Vision
How Might We statements
Related Feedback / Prior versions
Exact Scope
With some structure in place you are now free to create innovative prototypes and test with different users knowing that you can integrate the experiments into your workflow. Happy designing!
Check out a real world example of how this process led to a new paradigm for viewing content in XR.
