User research is to design and product development as statistical significance is to data.
You can’t be confident in figures if you haven’t carried out significance tests. And you can’t be confident in a design or product change if you haven’t carried out user research.
Yet businesses that baulk at treating data as gospel without statistical significance tests will make product or design decisions without a jot of user research.
I’ve worked for organisations like this, perhaps you have too.
What is user research?
User research is many things, but in practical terms it’s the tangible outcome of making your users, audience or customers the heart of what you do.
It’s one thing for a company to tell us that customers are their “number one priority”.
A company shows it by having user researchers who learn about their users: who they are, what they do, what they want, what they like and dislike, what influences them.
User researchers uncover these findings through interviews, observation, usability studies, surveys. Then they interpret and gather insight through multiple rounds of research.
Insight is the output: you find out who exactly your users are, and what their pain points and needs are. Insight is shared with the wider team (and the team should be joining in on research sessions too).
These findings sit within a goal. This can be a project, business or organisational goal, and how the product or service will best serve its users.
What is statistical significance?
Let’s say you’ve carried out A/B tests on two web pages and design B led to a 10% increase in goal completions.
Does this necessarily mean that design B is ‘better’? You could have got lucky with a hoard of spendthrift shoppers logging in together, or unlucky when the internet failed during design A’s slot.
The point of statistical significance is to be able to say that the results are likely to be true, a “low chance of an effect that actually is a false alarm”. That if you repeated this you had a good chance of getting similar results.
Newspapers and other everyday presentation of statistics typically omit statistical significance for simplicity. This is understandable, but statistics used in research and business must include it if they want to understand their data. And if the business does not run these tests, why not?
Statistical significance then helps give you the confidence — not certainty — that your findings are true. That your results weren’t due to a lucky (or unlucky) sample or events.
How user research ≣ statistical significance
User research gives you confidence. Confidence that what you’re doing has an effect due to changes your team made and not due to chance. Confidence that audience tastes are changing or a competitor has emerged.
Confidence that when the CEO says that they don’t like something that you can push back because the users say otherwise. That you have research, not just opinions.
User research gives you confidence but never certainty. That’s why research is an ongoing activity, much like how significance tests are carried out on each new result.
A danger of statistical significance is that it can give the appearance of scientific certainty when none is there. For example, produce Google Analytics data with statistical significance and it’ll appear the more ‘scientific’ result.
Yet the analytics is the outcome of people’s behaviour, and — unlike interviews — it’s hard to follow up and probe why a user did what they did with analytics.
Finally, both statistical significance and user research need to state the practical significance. Both can say that there is an effect but both need to say what the practical outcome is. For example, whether the problem is a mere annoyance or one that prevents users completing their task.
User research > statistics?
Both statistical significance and user research give you confidence in your results. But good user research includes the user impact by default.
A key part of user research is that the whole team should join in, and so will expand their own knowledge. How often does the team join the web analyst and contribute to their research?
User research can probe and build understanding across a team in a way that statistics by itself finds hard to achieve.
And any company that wants to be serious about its development needs user research as much as it needs statistical tests on its data.