Categories
General

Civil servants are user research participants too

Carrying out user research across the public sector is not the same as carrying it out with members of the public. That at least has been my experience of carrying out half a dozen different civil service-focused Discoveries.

As the Government Digital Service here in the UK likes to point out, civil servants are users too. But it’s a broad sector and my research projects have included central, devolved and local government, agencies, the police, and the NHS. Here are my experiences in light of the new guidance for services for civil servants.

Planning user research for civil servants

The first thing I do for all projects is meet the team and host a research question workshop. Where this differed from other workshops is how we thought about users.

We quickly decided that ‘civil servant’ was too broad a term for users and looked at the Civil Service profiles and Departmental IT profiles.

We found when reviewing civil servant personas from previous research that some are often just their job titles. So we did two things.

First, we adapted the job titles into roles to reflect that users across different teams may have the same fundamental duties and needs but different titles. This allowed us to see patterns and groups.

Second we wrote our potential users and their stories not just ‘As a…’ But ‘As a…’ ‘+ Who’ (eg” as an assistant who is in charge of a team’s room bookings”).

This helped us to really narrow down who our users were. It also helped us resolve one debate we had about who were end users of a service and who were our chief users for one Discovery, which to our surprise were not the same.

We also had a fairly clear idea about the end users but for the Discovery we determined it was more important to know who would implement and make decisions about the proposed service and their needs.

Cross-government help

What really saved time was posting what I was working on to the cross-government user research Slack channel and mailing list.

While my team had contacts, other user researchers put me in touch with their teams when relevant. In some cases they even had previous user research I could look at — with room bookings, for example, I had 3 different previous projects I could study and borrow form.

As you may be aware, government has a lot of meetings and forums and groups. Going along and inviting myself to relevant meetings helped in multiple ways: I got research from the meetings; I got contacts; and I got people to spread the word about what I was doing.

The tricky bits of civil service research

User research is mostly for getting information from users, but on my projects the civil servants I spoke to expected more from interviews, particularly if a team member was present.

In some interviews it did get bogged down when team members wanted to defend or tell why the problem the user mentioned was, and that’s not the aim of interviews. The decision has to be taken on the value of having a team member be there to take part in research and how to control the research session.

Confidentiality was also a concern. It’s hard to be truly frank as a user if the person who designed the system you’re criticising is in the same room.

One strategy was to allow for time at the end for Q&As between the team and users, and to shut it down if it went too off-topic.

This was even trickier in workshops, yet one reason we could get so many participants to attend was that our team of experts would be there. It was a question of balancing your desire to get information while rewarding the fact that professionals were giving up their time and so expected something in return.

Participants were also keen to know the next steps. Asking product managers to vow to blog at the end of the Discovery, Alpha or Beta meant I could tell users that there’d be a digest of learnings, and I invited many to the final Show and Tell.

What I learnt

We don’t share enough with other user researchers. And a lot of user researchers across government have worked on similar projects with similar problems and needs.

Contacting others can be easier due to Slack, public blogs, meet ups and so on but it requires more chasing and more channels to monitor. Combine this with projects using the same users and there can be a case of research fatigues for participants.

Some blockers were technical and unique to the civil service. GDS doesn’t have .gsi in its email (a ‘government secure initiative’ that’s going anyway in favour of better cyber security behaviour), and lacked a landline. For some not up to speed with the latest policies this was a red flag and was told I “couldn’t be trusted” with a response.

With so many departments and agencies (despite decentralisation) along with local authorities it can be tempting to stay in London and its area to meet users.

Yet bursting the London bubble and travelling the country was essential.

Hangouts, Appear.In and other remote tools are great but opportunities to observe other working areas were essential to get a proper view of work. They were often keen to meet someone who was willing to come see them so it was a positive session for all.

Overall it’s been an enjoyable experience. Civil servants are not just users, they’re people too. Shocking, I know.

Researching in government is rewarding as you have experts in their fields and they love talking about their work. Even those who are unhappy usually end their interviews with “sorry for the rant” despite having given you reams of information.

And you hope that by the end of the project your team will have had insights and findings that will help a range of talented people across the country do a better job and so help the public.

Note: this was originally written for the GDS blog but due to team changes got lost to the aether.

Categories
General

User research is statistical significance

User research is to design and product development as statistical significance is to data.

You can’t be confident in figures if you haven’t carried out significance tests. And you can’t be confident in a design or product change if you haven’t carried out user research.

Yet businesses that baulk at treating data as gospel without statistical significance tests will make product or design decisions without a jot of user research.

I’ve worked for organisations like this, perhaps you have too.

What is user research?

User research is many things, but in practical terms it’s the tangible outcome of making your users, audience or customers the heart of what you do.

It’s one thing for a company to tell us that customers are their “number one priority”.

A company shows it by having user researchers who learn about their users: who they are, what they do, what they want, what they like and dislike, what influences them.

User researchers uncover these findings through interviews, observation, usability studies, surveys. Then they interpret and gather insight through multiple rounds of research.

Credit

Insight is the output: you find out who exactly your users are, and what their pain points and needs are. Insight is shared with the wider team (and the team should be joining in on research sessions too).

These findings sit within a goal. This can be a project, business or organisational goal, and how the product or service will best serve its users.

This is a very quick overview of it and there are many places to find out more about user research, how it improves service design and why you should do it.

What is statistical significance?

Let’s say you’ve carried out A/B tests on two web pages and design B led to a 10% increase in goal completions.

Does this necessarily mean that design B is ‘better’? You could have got lucky with a hoard of spendthrift shoppers logging in together, or unlucky when the internet failed during design A’s slot.

Credit

The point of statistical significance is to be able to say that the results are likely to be true, a “low chance of an effect that actually is a false alarm”. That if you repeated this you had a good chance of getting similar results.

Newspapers and other everyday presentation of statistics typically omit statistical significance for simplicity. This is understandable, but statistics used in research and business must include it if they want to understand their data. And if the business does not run these tests, why not?

Statistical significance then helps give you the confidence — not certainty — that your findings are true. That your results weren’t due to a lucky (or unlucky) sample or events.

How user research statistical significance

User research gives you confidence. Confidence that what you’re doing has an effect due to changes your team made and not due to chance. Confidence that audience tastes are changing or a competitor has emerged.

Confidence that when the CEO says that they don’t like something that you can push back because the users say otherwise. That you have research, not just opinions.

Credit

User research gives you confidence but never certainty. That’s why research is an ongoing activity, much like how significance tests are carried out on each new result.

Caveats

A danger of statistical significance is that it can give the appearance of scientific certainty when none is there. For example, produce Google Analytics data with statistical significance and it’ll appear the more ‘scientific’ result.

Yet the analytics is the outcome of people’s behaviour, and — unlike interviews — it’s hard to follow up and probe why a user did what they did with analytics.

Finally, both statistical significance and user research need to state the practical significance. Both can say that there is an effect but both need to say what the practical outcome is. For example, whether the problem is a mere annoyance or one that prevents users completing their task.

User research > statistics?

Both statistical significance and user research give you confidence in your results. But good user research includes the user impact by default.

A key part of user research is that the whole team should join in, and so will expand their own knowledge. How often does the team join the web analyst and contribute to their research?

User research can probe and build understanding across a team in a way that statistics by itself finds hard to achieve.

And any company that wants to be serious about its development needs user research as much as it needs statistical tests on its data.

Categories
About Writing

The user researcher and the screenwriter

Screenwriting is getting a story onto paper which is then made into a film. User research is understanding how users behave when trying to complete a task or service, typically online. 

Putting it like that there may not seem to be too much similarity between the two, but explore further and I believe that they share the same goal – of documenting the human condition and producing a ‘truth’ within parameters.

Are there differences? Of course, but the similarities are that I find interesting.

User research
Time to interview – Ethan via Flickr

A history of two crafts

Screenwriting has been a craft for over a century, user research, in its current form, has only been embraced by governments on a widescale over the past few years. Being so new and flexible it does give me some room to manoeuvre, but in general the similarities are:

  • a quest for a truth, in defined parameters
  • a following of principles over rules
  • the aim of recording how people actually speak over how we think they should
  • show over tell

But what of market research? Well it’s similar but different to user research. Market research is about finding out users but is a more analytical approach, focusing on breadth often at the expense of depth. Government user research isn’t concerned so much about segmentation, weightings and the like (though they are not ignored). It’s about reaching the goal.

Screenwritng is similar — there are no rules, or if there are, there are too many exceptions. All that matters is writing a story that works.

A quest for truth

In very general terms, films aren’t necessarily about a truth – Superman has not saved the planet, the Inglourious Basterds didn’t kill the Third Reich’s ringleaders, Withnail never existed let alone acted.

But within their own universe, that created for the film, they must stay true to the rules created if they are to succeed. Superman can do almost anything, but even he must stay true to his rules — he will still ‘do good’ whatever is thrown at him.

The Inglourious Basterds burned Hitler and his cronies because to director Quentin Tarantino, that’s what worked in his story that included a glorified ‘kill the enemy film’, albeit from the German’s perspective.

Withnail may not have existed, but the relationships, tensions and ambitions Withnail & I explored are true enough in our world because it is set in the same rules as our universe.

And so on… So what am I getting at? You define the goals, you set parameters, and you stick to them if you want to succeed.

Individual approach

‘”There are no rules to follow, Donald, and
anybody who says there are, is just –”
“Not rules, principles.”‘ — The Kaufman twins, Adaptation.

Many crafts have principles rather than f rules. But user research is still fairly new in government and to a large extent it is still down to the individual or small team carrying out the user research to get to the goal. As such it is still down to the individual who does it.

This is reflected in that very few user researchers I’ve worked with have specialised in this for their careers. Instead they’ve come from a variety of backgrounds, and for myself it’s been content, journalism and anthropology.

It’s down to the individual.

Show, not tell

It’s rare for a film to succeed where all the characters do is tell you how good they are. In fact the audience largely forgives what we’re told about them if we see them doing wonderful things (ask Indiana Jones just hold old Marion Ravenwood was when he seduced her).

User research is about showing what is found — at show and tells, in videos of interviews, of producing quotes and examples. Don’t just tell us what is found, show it, and be consistent.

Getting to the heart of people

Ultimately screenwriting and user research and have one key goal – to show us that this is what life is really like. It is to produce something that is recognisable.

User research is like that. Taking something and passing it on to the next stage of the process. Looking for recognition that yes, this is what reality is and what we need to produce. You write down what people say, not what you think they say, and arrange it to make sense.

You also look for plot holes and inconsistencies and how to get rid of them, whether that’s more user research or revising the screenwriting.

A team sport

Finally it’s about others. User researchers don’t work alone, you’re encouraged to show, not tell, key parts of your process. You do not work around, your work forms the foundation of all that follows.

No script, no film; no user research, no project.

Ultimately it’s about getting a truth. The truth here but within the goal of getting a truth. Not the truth, but whatever will meet the goal of truth. And one that at the end, whether it is the final show-and-tell or handover, or a film, will leave the audience satisfied that they saw something true to what was set out.