“Success with Style” part 1: analysing great (and not so great) books

Computers can predict how successful a book will be — or so ran the headlines a few years ago following university researchers’ publication of Success with style.

A bold claim and if true and it could benefit many people; the writer looking for feedback, the literary agent or publishing house reader swamped among the piles of manuscripts, politicians looking for the next great speech. But can artificial intelligence or machine learning really predict the success of books?

Machine learning and writing

The researchers who came up with the Researchers at Stony Brook University in New York — Vikas Ashok, Song Feng and Yejin Choi — reduced stories and poems into their linguistic components and published the results. They claimed that success or failure is related to how types of words made up a text.

Their paper,
Success with style: using writing style to predict the success of novels, got a lot of attention at the time and has been cited multiple times since. Yet despite that happening in 2014, no publisher or agency has announced they’re replacing their readers with machines.

This may be in part because the authors didn’t detail their methods, and because the success rate was not 100%. Yet I had other issues with the paper. And now I have completed my statistics studies I can return to address them.

Investigating success

Things I wanted to investigate from the original paper:

  • the detailed methodology and how they got their results and why they chose to manipulate data in the way they did
  • the definition of success – not mentioned in the paper is that the downloads used isn’t the total downloads of all time, it was downloads over the previous 30 days so could be skewed. Is this the right measure?
  • the difference in proportions of success and failure was tiny, with no proportion being more than 1% – is this statistically significant?
  • the readability score – although not set out in their aims, their readability score is not divided by genre (like other scores). Why?

Success with style: the original process

The original researchers looked at 5 measures in a text (largely looking at how language was distributed and the sentiment), and we’re only go to focus on two, Part of speech (PoS) tag distribution and readability as these were fleshed out the most in the paper.

A PoS tagger tags words based on its position and context, categorises then reports how these are found in the text and there are several tools to do this along with different tags to use. The Penn tagger is used in this paper although it wasn’t their highest scoring predictor of success (they claim it only had a 2/3 success rate) this is still better than what we’d expect by chance (50% as it was a simple success/failure test).

Yet the question that wasn’t asked let alone answered “are these results statistically significant?”. That is, are these results we should sit up and pay attention to, or are these just down to chance that there happened to be a difference in the selected books. Throughout I’ll use p>0.05 as my measure of significance.

Success with style methodology

The original paper doesn’t detail the methods used, nor the hypothesis. As we’re going to perform statistical tests we’re going to we’re going to create a hypothesis.

If you haven’t come across a hypothesis before, the idea is that you start an experiment with a null hypothesis (H0), usually that the current situation is correct. You then offer an alternative hypothesis (HA) that represents your research question.

You test as if H0 is true. If the test results don’t provide convincing evidence for the alternative hypothesis, stick with H0. If they do then reject H0 in favour of HA (note this isn’t the same as saying that the test proves that the alternative hypothesis is true).

I interpreted the original PoS hypothesis as:

H0: There's no difference in the distribution of the proportion of parts of speech (PoS) tags in successful and unsuccessful books, regardless of the book's genre.
HA: There is a difference in the distribution of the proportion of PoS tags in successful and unsuccessful books, and the pattern will depend on a book's genre.

For readability (I’ll be using the Flesch-Kincaid grade measure, where the lower the score the more readable the work. The original researchers only used FOG and Flesch, other measures, but this is the measure I’ve used elsewhere and so:

H0: There's no difference in the readability measure of successful and unsuccessful books, regardless of the book's genre.
HA: There is a difference in the readability measure of successful and unsuccessful books, and the pattern will depend on a book's genre.

Recreating the Success with style method

The original team used the Stanford tagger. I used the Perl Tagger as I already have it setup but, like the Stanford tagger, it uses the Penn Treebank to assign PoS tags to English text. It’ll also be of interest to see if a different PoS program creates any difference.

I’m going to:

  1. Get the source books and the metadata used in the original study.
  2. Run the books through the PoS tagger and readability analyser.
  3. Recreate their output data and compare it with the original results.
  4. Carry out statistical tests for the significance of the results.

I also use LIWC 2015, an alternative language tagger that I’ve used for past projects and will repeat.

Finally, as the original results didn’t account for the 30 days success/failure measure I’m going to reuse the same books to see if they have different downloads and if the accuracy is repeated.

Recreating the original results

Recreating the original results wasn’t easy. The image below is from the original paper, but how they got this is missing.

Chart showing the distribution of parts of speech tags
From the original paper: Differences in POS tag distribution between more successful and less successful books across different genres. Negative (positive) value indicates higher percentage in less (more) successful class. Ashok et al 2014

The output data offers a range of ways of interpreting it, but this is how it was created (all data links are at the end if you want the raw and manipulated data):

  1. Split the data into success/failure and sum all the tag data (eg all the CC tags, all the CD tags and so on).
  2. Work out the proportion of all tags the individual tags represent (eg for CC you get CC/(CC+CD+NN+…) in success/failure, so for the Adventure genre CC is 140,393/3,386,774 = 4.1%).
  3. For each tag subtract the value of the failure from the success proportion to give a net value (eg 4.14% – 4.223% = 0.08% difference) as shown in my table.

Here’s my intiial output (next time I’ll detail how I got it):

Not precisely the same but very close, and I’ve used similar colours and the same (rather odd) arrangement of tags for easy comparison.

I labelled the y-axis label in the charts to draw attention to the tiny scale of difference. The scale is a maximum of 1% difference, with most differences within ±0.5%.

This is minuscule and begs the question of whether this is statistically significant.

Next time

Part 1 is about the original experiment and going about recreating it. Part 2 will be on testing it and stating the difference in findings and statistical significance.


This research was sponsored by Richardson Online Ltd to highlight how computers, analysis and content can come together.

About Writing

The 5Ws and whodunnits: a Chinatown character exercise

A whodunnit is a genre of film where, as the name suggests, we want to find out who did it. But knowing who is not just enough, we also want to know the why, where, when, what and how.

In journalism these points are generally known as the 5Ws (even though ‘how’ makes it 5W1H) and are seen as the fundamental information a news story needs to convey. This information should appear in a story as early as possible.

5Ws and whodunnits

The whos, whats, wheres, whens, whys and hows

“I keep six honest serving-men, (They taught me all I knew);
Their names are What and Why and When, And How and Where and Who.” – Rudyard Kipling

Naturally in a whodunnit we don’t want to know the answers too early or it spoils the fun, with the honourable exception of Colombo which made a point of showing all this at the start. Even then it became a mystery of how the rumpled detective would uncover the information.

What makes the television show Colombo with its one big crime to solve different to a mystery film such as Chinatown is that while both involve an investigation, in movies the general principle is that there are multiple mysteries. Act I typically has a smaller, more pedestrian mystery that leads into the bigger one, with Act III sometimes having its own mystery resolved, often one that stemmed from the Act I mystery but was not directly investigated at first.

As an exercise I did some work on looking at the mysteries within the classic 1974 film Chinatown written by Robert Towne and looked at the 5Ws for each of them. Spoilers follow, naturally.

Mysteries in Chinatown

In each of these not only have I labelled each mystery and the initial ‘answer’ to each of the 6Ws, I’ve iterated as to why the answer is what it is. Most end with a character motivation, or that it’s part of the setting.

The mysteries are listed in no particular order, and the act marks are where it’s first raised but not necessarily where it’s solved.

Mystery 1/Act I mystery – Is Mulwray having an affair? (The false mystery)

Is Mulwray (Evelyn’s husband) having an affair (False mystery)?

Answer/Why 1

Why 2

Why 3

Why 4


Mulwray, head of the Department of Water and Power

He’s a rich guy

Gittes, the private investigator, assumes pretty girls are mistressess

[It’s in his character to assume powerful men have affairs, he’s seen it before]


Attractive young girl

She’s pretty

Gittes assumes pretty girls are mistressess

[It’s in his character]


In a house Mulwray owns in Echo Park, LA

Away from the Mulwrays’ posh home

Gittes has seen this before

[It’s in his character]


During work hours

So his wife doesn’t know

Gittes has seen this before



Who cares

Jake doesn’t care about the whys in these case

 He’s in it for the money, he’s a professional  


He visits the love nest

Away from the wife

Gittes knew where to look as he has seen this before


Mystery 2/Act II – Who is stealing LA’s water?

Who is stealing the water?

Answer/Why 1

Why 2

Why 3

Why 4

Why 5


Noah Cross, a rich industrialist

He owns land that needs water

It’s desert and worthless without water

He bought it on the cheap knowing he could get water

[It’s his character to get what he wants regardless of the ethics]


Water is being diverted during a drought

Noah is a powerful man and can do this

He is extremely wealthy

[It’s his character]



To land Noah Cross, a rich industrialist, has bought

He wants it watered

To increase its value

It’s desert and worthless without water

LA is a desert [It’s the setting]


At night

So no one will see

As it’s theft

Water is precious in desert LA

Noah is rich but not so rich he can do whatever he wants [Conflict with used to getting what he wants]


To store water in land to make it more valuable

The land is dry

Noah wants to do it secretly

So that only he will benefit

He’s a greedy man [It’s his character]


Diverting water through channels

They were built there

[It’s the setting]


Mystery 3/Act IIb – Who set Gittes up?

Who set Gittes up?

Answer/Why 1

Why 2

Why 3

Why 4

Why 5


Noah Cross

Mulwray was blocking his plans to build a dam

The water would go to desert land and not benefit the citizens of LA

Mulwray believes water belongs to the people and can’t be corrupted

[It’s in his character]


Hired an actress to hire Jack Gittes to investigate ‘her’ husband

Gittes would believe her as the real Mrs Mulwray (Evelyn) wouldn’t do it

Mrs Mulwray loved her husband

[It’s in her character to love this father figure]



At Gittes office

It’s a city built in the desert

Water is a precious commodity here

[It’s the setting]



While Mulwray was seeing Evelyn and was head of the Department of Water and Power

Cross knew what it would look like

Mulwray could not reveal the truth of who the girl is, she is tainted

She is a product of incest

He won’t hurt her [It’s in his character to be good]


Cross wanted to blackmail Mulwray

So the dam will get built

The worthless land he bought will be worth millions

He wants a legacy

[It’s in his character to desire his name living on after him]


By having Gittes take photos of Mulwray with a girl and a ‘love nest’

Gittes is a well known PI so his evidence is credible

He will do what it takes to get the evidence

[It’s in his character]


Mystery 4/Act I and Act II – Who is Mulwray’s mistress?

Who is Mulwray’s mistress?

Answer/Why 1

Why 2

Why 3

Why 4


Katherine Cross

His wife’s daughter/sister

Incest in the past when Noah’s wife died

Noah can do what he want [It’s in his character]


Mulwray’s step-daughter (and sister-in-law)

He wants to protect and look after her

He’s a good guy

[It’s in his character to be good]


In a house Mulwray owns in Echo Park, LA

He wants to keep her away from Evelyn

He doesn’t feel Evelyn should have contact

Because the daughter is a product of incest


Since he married Evelyn

She’s her daughter

He feels responsible

[It’s in his character to do the right thing]


He wants to raise her well and keep Evelyn out of it

He wants Katherine to have a normal life

He doesn’t want her to know her past

It’s disturbing to know you’re a product of incest and he wants to protect her from it [It’s in his character]


He visits when he can and takes her places

He wants her to have a normal enough life

He puts value on people living well

[It’s in his character]

#### Mystery 5/Act II – Who killed Mulwray?

Who killed Mulwray?

Answer/Why 1

Why 2

Why 3

Why 4

Why 5

Why 6


Noah Cross (Evelyn’s father)

They were meeting to discuss the future

Mulwray has power to stop Cross’ plans

Mulwray is head of DWP

Mulwray sold his and Cross’ private water company to LA

Mulwray believes that water belongs to the people

[It’s in his character – he believes in ‘the people’]


Drowned Mulwray

Heat of the moment

He became angry at Mulwray’s refusal

Without water the land is worthless so his investment would be a waste

Noah gets furious if he doesn’t get his way

[It’s in his character to react angrily to refusals to his will]


In his own pond of saltwater

They were meeting at Mulwray’s house

Cross wanted somewhere private to discuss his plans

His plans are dodgy and involve defrauding LA voters

Cross is prepared to make dodgy deals to get his way

[It’s in his character]


After Mulwray said he’d reveal his plans to buy land on the cheap and divert water there

There’s a referendum soon on whether to build the dam

Mulwray publicly opposes it as head of the DWP

As head of the DWP and builder of another dam his voice carries weight

[It’s in his character to do the right thing]



To allow a damn to be built

It will provide water to a dry valley

Cross has been buying land on the cheap in the dry valley

He wants money and a legacy

[It’s in his character]



Cross pushed him in

Heat of the moment, Cross did not plan to kill Mulwary

Cross has a short temper

[It’s in his character]


Analysis of the analysis

When I started this exercise I didn’t plan to end each iteration on an answer. But with each mystery I broke down it seemed to naturally flow from the character or setting.

What is also satisfying is the number of iterations Chinatown offered for each mystery and its consistent character motivations. There wasn’t a straightforward answer to any of them, each took multiple iterations, and this fits in with master of mystery Raymond Chandler’s ‘Ten Commandments for the Detective Novel’; that we are honest with the reader (or viewer) and have given them the information through the 6Ws needed to make the inevitable once revealled.

Partly this may be because I am over analysing and making it seem more complicated than it is. But I’ve sat on this analysis for a while and having returned there is something satisfying with this approach.

The false mystery found

As stated earlier, Act I generally has the false mystery, one that segues into the larger mystery. It is also not what the film is known for.

Compare this with the Chinatown-inspired Who Framed Roger Rabbit. Private investigator Eddie Valiant’s Act I ‘puzzle’ is similar in that it’s notionally about him finding out about an affair but in reality this is staged setup for something bigger.

Neither protagonist is asked to investigate the film’s larger mystery — who killed an important man and the land-grab conspiracy behind it — that emerges in Act II but chooses to do so. Along the way they also stumble inadvertently into revealing answers to deeply personal puzzles they didn’t even want to know — that Evelyn is mother to her own sister, that Roger Rabbit‘s Judge Doom is a Toon.

Applying the 6Ws to characters and plot

I’m not pretending that this tool can be used to plot mysteries. But it can be used to sense check what you have written: first that you answer all the 5Ws that a sharp viewer will want to know (apart from JJ Abrams and his incomprehensible passion for Mystery Boxes); second that your answers have some depth beyond “it just is”.

As mentioned, there is a danger of over analysis – if you’re smart enough you can spin out anything. But with honest evaluation it may help as a tool to look for depth of mystery and consistency of character across the story.

It may be that this applies beyond mysteries, thrillers, whodunnits and the like but it seems an obvious start. In theory any protagonist and antagonist’s motives can be analysed this way too, and may be a way to check that a villain’s goals really are beyond ‘because he is evil’.

That’s for another analysis.

Research Writing

An Agile writers’ room: a better way of writing part 2

Last time we looked at the problem around writing and how too few individuals can write well enough consistently to reach the top. But together they may stand a better chance, and Agile methodology would be the way to do this.

That’s quite an assumption, but Agile (in all its forms, more on that later) is geared to testing and adaptation so the best thing is to plan how that would work and try it out in reality.

Agile writing room

Writing for publication is Waterfall but should it be Agile?

Agile is about working as a team to produce something together. Very idealistic, but doesn’t Waterfall and its related methodologies do the same?

The main difference is that Agile is not about working to produce one big, final, perfect result. Instead Agile is about breaking it down into small units, delivering the minimum needed in short sprints, testing, refining and adapting.

Agile v waterfall
Waterfall compared with Agile (via Agilenutshell)

This doesn’t mean Waterfall is bad, it suits big things where you can’t test, or update or move things. Things such as building projects… and writing? Certainly when I’ve written professionally or creatively it’s been comparable to this – set deadline, some editing and peer feedback then submit your best and forget about it once done.

This makes sense at first – if you’re aiming for a deadline you must produce your best and it must be complete and on time. Yet content teams are switching away from this in the non-creative sector due to the benefit of breaking things down into bits. And you can also break the team roles down into bits and split it between members.

The Agile writing team

As the roles are split you’ll need people who can do all these things working together, feeding back and being aware of what others are doing. A mantra of Agile is that the unit of delivery is the team. The best Agile teams may not have the best at their individual skills, the best developer, but it will have the best at working together to deliver what they need to.

You can be brilliant at your role but if you can’t work with others and adapt to help with them then you can’t write in an Agile team.

So writers are all you need in a writing team, right? Yes, of course you can’t have a writing team without writers, but you need more.

Here’s a table looking at the skills you’d need in an Agile writing team and how it’d map to a writers’ room. The roles aren’t all that different in many cases, it’d be how they work together that is. This is a big reduction, writing and Agile teams vary etc, I’ve taken liberties in both the writers room and Agile team for illustration.

Role Agile Writing teams
Deals with the vision and the bigger picture. Works with stakeholders. Decides on priorities and making decisions. Keeps the team informed of priorities. They work with the backlog and decide making deacons in a timely manner. Provide information in timely manner. Product owner (aka on-site customer or active stakeholder) Executive Producer Showrunner (depends on the team)
Create the right environment. They remove blockers and work with the product owner to make the vision happen. Doer of the visionary pairing. Delivery manager/scrum master Problem solver, project management, but not technical planning and scheduling as that is left to the team Works to hire the team Has a range of skills to do things properly Very practical person Co-producers Showrunner Writers assistant can help with some of the lower level tasks
Creator Content designer, developer Writers (story editors, staff writers etc)
Researches what the user needs, identifies the users User researcher Writers assistant (if asked by writer
Testing and stretch exercises Team develops this themselves Team develops this themselves
Specialists with knowledge brought on for key parts Technical or domain experts with specialist technical knowledge Consulting producer
Testers Independent test team, user researchers External editor Readers
Anyone who is a direct user, indirect user, manager or users, senior managers, staff member. “Gold owner” who funds the project. Representatives of the customer. Stakeholders (funder/commissioner) Executive producers, studio

Differences are many though. In Agile because it’s the team that’s responsible for delivery they are also collectively responsible for accepting work, allocating it and are responsible for producing it.

So while the show runner has editorial job, they are less of the tyrant of imaginings, but in return for this loss of control it should allow for a gain in innovation.

An example of how it works

Agile has already transformed other creative ways of working. I’ve mentioned government a lot but other areas have changed too, such as marketing:

“[Before Agile we didn’t have] a clear focus of our tasks and communicating them as a team […] Now, before the start of each quarter we’d meet and decide what our team priorities would be, then each team member would be assigned to the priorities and off we’d go. We’d meet two mornings a week to discuss the progress of our priorities, our KPIs, and our blockers.”

Which Agile do I mean?

Agile experts reading this probably long ago asked this question even though I said I’d look at the general principles. The main 3 forms of Agile are as the Harvard Business Review states:

  • scrum, which emphasises creative and adaptive teamwork in solving complex problems
  • lean development, which focuses on the continual elimination of waste
  • kanban, which concentrates on reducing lead times and the amount of work in process

My straw poll of Agile experts is that kanban would be a good way to start, as it’s about reduce the amount of work. But the beauty of Agile is that it can be adapted as needed.

Team writing in Agile is not for everyone for various reason.For instance, everyone needs to own a ticket. This responsibility is not for everyone. Consistency will be tricky. That is one for Agile to answer through the doing – there may not be a market, people may be afraid of ‘idea theft’ (not that that is really an issue). It may be less agile and more plodding.

Final thought: Agile writers, over complicating things?

It’s a fair question – is this overly complicated? My only defence is the William Goldman view of Hollywood – if, as he says, “no one knows anything” then who’s to say they know it won’t work?

Hollywood and TV (which this would be about writing scripts for) would be receptive to anything as long as it gets results. More and more places, including Amazon Studios, accept unsolicited scripts and only care if they tell a good story.

What they want is writers who can meet a specification on time, make changes as requested (and not be too difficult about pushing back) and do it on time.

From my time at BBC the thing that came up again and again when people asked “how does that person keep getting hired” was that while they may at worst be accused of mediocre scripts, they were never bad, they met the brief and most important of all, they were on time.

That’s not too high a bar to hit.

Next steps

Theory is one thing but it’s nothing without putting into action.

That’s what the plan is. It’ll be hard to get going – would this be voluntary or would I hire people; I have a breakdown of resources but will that work in practice?

So many questions, but the only way to answer them is not to speculate but to try.

Be prepared, be prepared to fail, but most importantly be prepared to learn to and to develop from that. Success in terms of the project is that it even works and we complete an initial script. Surely we can do that?

About Writing Original research

A better way of writing? Part 1: current problems

There is a problem with creative writing, and it’s an old one. It takes a lot of talent and energy to write a novel or screenplay, yet only a very few individuals have that combination of great story, great writing and a little luck to see it published or produced to widespread acclaim.

There are many reasons why so few people make a career as a writer. It takes time and commitment to a gripping idea and then the skill to write it in an engaging way. Even writing a great book or script is no guarantee a publisher, agent or studio will pick it up, due to market forces, personality clashes, bad luck or events.

Most writers write alone, they review alone, and beyond a small group of friends and family, and most fail alone. But what if they could have had help for others of equal talent to make their good story a great one?

via Drew Coffman

Writing quality and quantity

Very few people like to criticise others (although a few people do make a living from this kind of writing). Other than, usually effervescent, friends and family, the unpublished writer will get their criticism from writing groups.

I’ve been to multiple writing groups and the standard has been pretty good but not great (my own work included). That’s not to say there aren’t many clever people, with very good work, but it’s not been with that quality that is needed to make it to sales.

Writing groups can help with feedback to improve quality, and a good writing group will offer more feedback than “I really liked this line” and zero in on problems with the dialogue, pace, story and writing. Unfortunately, while the group is generally good at spotting the problem it’s bad at offering the right solution.

Writers may not take on board much feedback, dismissing it as the comments of amateurs. To act on feedback you generally need to hear it from others you regard as your peers or superiors.

Writers’ rooms

I recently went on a training session where we split into teams to complete tasks. Our team ‘won’ in that we completed the most tasks, but the secret was we should have worked with the other teams to complete all tasks because in the exercise we shared a ‘boss’. In writing our boss is the reader and as writers many of us are working solitary to complete a pretty good story rather than coming together for a completely satisfying story.

There are exceptions. In the US writers’ rooms, groups of writers working on a screenplay, are common for TV shows such as The Simpsons, Narcos and other top programmes. A writers’ room can lead to consistency over a series, draws together ideas, improves standards and enables a script to make it to read through and production fairly quickly.

So why aren’t they more common beyond US TV shows, why not in film or for novels, and why are they rare in the UK? Doctors on Radio 4 and Doctor Who are pretty much the only British writers’ rooms (the BBC Writers Room is the corporation’s submission site).

One reason is that UK TV series are shorter, typically 6 episodes, so one or two people can write it. US shows can be a run of 20 episodes in a season. Yet there are other places in the UK where writing is produced by a team to tight deadlines.

Just not in creative writing.

Content teams and writers’ rooms

I’ve written and led content teams to produce content for GOV.UK. Unlike a writing team, the content team’s output isn’t creative but the process to get there is: taking bureaucratic, legalistic documents and translating them into a language an audience not just understands but needs to understand (such as they need to get a passport or pay a fine) requires a lot of creative thinking.

So what makes a ‘content team’ different to a “writers’ room”? The secret sauce here is that a content team is multidisciplinary and Agile.

Using Agile delivery – breaking down projects into tasks assigned to individuals and agreed by a team as a whole to be delivered in a set time – and writing to a clear style (both for English and approach to work) focuses content designers.

Why content designers rather than writers? One reason is that writers were seen as rather servile, black boxes where someone sent a document to have its spelling and worst sentences corrected. Content designers do that too but have more power to shape the content; its structure, language (particularly that it meets the style guide), and can even reject the proposal.

Good writers do this too, but content designers probably work in a system more akin to that for software delivery rather than creative writing – in my experience in the media and publishing writing was not delivered this Agile way.

But is this process suitable for creative writing and can it help this push to greatness, or is the problem too great to solve with just one tool?

The writing problem – and a solution?

The problem with writing then is that writers aren’t being pushed enough. Some are of the calibre to push themselves, but for the majority the discipline and effort needed for the final push from good into great is too much.

Next time: how Agile methods can be used to achieve this push.

News Scientific Research

Scrivener: the best tool for organising user research

User research involves a lot of, well, research; a lot of notes, documents, videos, pictures, post its and more. And they all need organising.

There’s no one solution for the problem of what to do with all this, but after a bit of experimentation I find that using Scrivener has been the best for me for keeping things organised.

Scrivener is often seen as a writing tool, but it’s more than a word processor. Yes, it is a writing tool – from word processing to screenplays – but it is also an organiser. Most important it’s very simple to use, and has more advance features for those who want them.

Scrivener being used for user research
Scrivener lets you display folders and multiple documents at once

Renaming research in Scrivener

I’ve been using Scrivener for years, and coming from an anthropological and journalist background to user research I focus research that’s written up – observations, interviews, transcripts. But I also add photos, plan card sorts, organise thoughts with the card index display, and add spreadsheets, PDFs and presentations. Even if I don’t read the presentations directly in there, being able to search all relevant work in one search helps.

In Scrivener I like how easy it is to organise and rename documents, or duplicate them. Compared with doing this in Finder or Explorer, it is much less of a faff. Likewise documents open immediately rather than take a few seconds in Word or Google Drive (and often aren’t the one I want anyway).

While I still use Google Drive and Dropbox and to organise files, particularly video, due to the amount of research that is pure words, either as transcripts, proposals, documents or insights, I find that Scrivener is the best way to keep it all together.


I love tables. I like maths, I like spreadsheets. Really.

I like to organise interview questions in tables and use a Dewey-esque numbering system to help reorganise them. So question 101 is the first, but perhaps it needs to come later, so I reorganise it as 103 and sort.

Likewise when reviewing a transcript I like to have each question in its own cell with thoughts and insights in the cell next to it.

Scrivener could be friendlier with tables – don’t create one at the end of a page or you’ll never get out, and I always have to customise it. But once I created a good, blank table I could copy and paste that.

Sort code Quote Observation
101 I’m not really sure that it’s appropriate User not keen on this
102 Do I really have to give you a dummy quote? Prefers to be in control of speech
250 At this time, a friend shall lose his friend’s hammer and the young shall not know where lieth the things possessed by their fathers Likes Brian?

Good things about using Scrivener for user research

What’s great:

  • Easy to move documents around and organise into folders and rename them
  • Split view makes reviewing transcripts and images easy
  • Colour and icon coding makes it easy to find key files
  • Compiling documents means you can make it consistent output, or just select the ones you need to put into a single PDF or Word report, or output as multiple documents so you don’t have to worry about formatting until the end
  • Coding for things such as image captions means that you don’t have problems with Word getting confused about auto-numbers
  • Text file syncing – if out in the field you can create text notes and sync them automatically into the project 
  • Great search tool for searching titles or entire files
  • Corkboard views to organise thoughts, observations, insights etc
  • Good way to have a list of priorities and hierarchies
  • Importing documents automatically works pretty well, just drag and drop the Word docs to where you want them and it’ll convert them into a continuous webpage rather than multi page report

What’s not so great:

  • No dictation tool
  • Not always the best way to view documents and tables
  • No Android version, although there is one for iOS, although it’s rare that you need the entire project on ⁃ your phone
  • Adding weblink – it already fills in the https:// part but every time you copy and paste from Chrome it has that part, so you get ‘broken’ links as it’s https://https:// if you forget to remove that part
  • Can be fiddly with bullets

User research tools to support Scrivener

OneNote, which isn’t free, is good for:

  • Transcripts – jump to the audio where your notes are as it tracks your writing with recording (although only 15min recording on Android for some unknown reason). It can convert speech to text, though I find that’s a bit less reliable.
  • Optical character recognition – it’s not 100% accurate but it’s good enough for recognising text from images and these will be show in search
  • Syncs across devices

I also use Trello to track research questions, answers and insights.

Overall Scrivener with its files synced through the cloud (Dropbox, OneDrive etc) has been great for keeping track of research. Scrivener isn’t free, but I feel I got my $45 worth of use long ago, and it’s less than what Microsoft charges for Office 365 (which includes OneNote).

Scrivener hasn’t sponsored or otherwise provided incentives for me to write this (nor has Microsoft, though I’d feel weird if they did), I just want to spread the word for a useful tool.


Daily Mail v The Guardian: equally angry?

This week two British media giants, the Daily Mail and the Guardian, got into an inter-title fight about who encourages hate and negativity.

The Press Gazette best sums up the story, which started when the Guardian implied that the Mail and Sun are to blame for the recent attack on a mosque.

The Guardian published a cartoon of a white van outside Finsbury Park mosque, where one person was killed, with ‘Read the Sun and the Daily Mail’ on the vehicle. The Mail took this as implying that it incited the attacker to kill Muslims and fumed, replying with the editorial “Fake news, the fascist Left and the REAL purveyors of hatred”.

In short, both sides accuse the other of peddling noxious opinions, and in particular the Daily Mail effectively says that the Guardian can get off its high horse as its views are just as noxious. Are they?

The Mail has a point

Yes, the Daily Mail has a point. While the Guardian may not typically have immigrants, saboteurs or judges as targets of its wrath, it does similarly emotive language in descriptions of its enemies (usually tories).

What it comes down to is the Mail says that the Guardian’s views may be left politically, but they are just as negative as the Mail claims the Guardian thinks it is.

This chart shows the average proportion of ‘anger’ words in the body copy and headlines for 12,000 Mail and Guardian opinion pieces spanning the past couple of decades. They’re not so different in terms of the average about of anger and negative words they use in body copy and headlines, and use more on average than other British newspapers.

Negative newspapers?

In 2013 I analysed 60,000 opinion columns from 6 British newspapers — the Daily Express, Mail, Independent, Mirror, Guardian and Telegraph — for a range of measures. This included sentiment, and emotional proportions within text, using the LIWC 2007.

I was looking at a range of things, including the question of whether the internet had changed the way newspapers wrote — would they become more emotional to target their niches. I chose opinion columns for I took it that an opinion column — editorials, those written by regular as well as guest columnists and commentators — was the most suitable way to see what a paper really thinks as opposed to reporting a news event.

I split the headlines and body copy out as headlines are often written separately to the body, and can also give an idea of what phrasing the paper thinks will draw readers’ attention.

At the time I vowed to publish each week. I didn’t in the end, in part as I saw no market and in part I was looking around if someone was interested in publishing, and while I got some interest, it was a case of “what does this lead to”? This is what it leads to.

Whenever there are two colours, blue is the body copy, red is the headline. Y-axis is the proportion of content meeting that definition. Or just hover over the images for the legend to appear.

Average negative emotion in headlines and body for all newspapers

The following charts make it clearer, but there is a definitive difference between newspapers and their negativity, and a similarity between the Mail and Guardian.

Average anger in headlines and body for all newspapers

The Guardian has angrier content, on average, than the Mail – 0.884 v 0.839.

Most negative content

The Daily Mail is the most negative, but the Guardian isn’t far behind.

Angriest headlines and body (split out)

The Daily Mail has the angriest headlines, but not the angriest content — that’s the Guardian



Positive message

The Mirror is overall the most positive, although the Guardian is slightly more positive in its message than the Mail.

Negative emotions in headlines and body over time

Before 2006 I have less data, which may explain the variation (and is why the other charts are based on data from 2008 onwards), but while headlines change in tone, the body copy has largely been consistent. Zoom in to 1 or 2-year views and there’s no large change over the months, not even at Christmas.

Change in negativity over time for all papers

ALl newspapers have largely been consistent over the years. I had been expecting them to become more emotional as they strive to distinguish themselves on the internet.

Mail change in negativity over time

Love it or hate it, the Mail has largely stuck to its tone over the years, perhaps a little more negative of late.

Guardian change in negativity over time

As with the Mail, the Guardian has been roughly consistent in its tone.

Word count over time

This is the only chart that shows a real change over time. Many style guides for online suggest keeping the body length short (something I ought to be better at) and you can see that as the internet becomes more important for revenue around 2005 the length shortens.

Why creep up again? Honest answer, I don’t know, but it could be a suspicion that people are so quick to move onto another article that it doesn’t matter whether it was long or not — if the reader likes it, they’ll stick to the end, regardless of the length (within reason). Or it could be my data set.

The Daily Mail v Mail Online

Part of the beef the Daily Mail has is that it accuses the Guardian of confusing MailOnline with the Daily Mail and I use ‘the Mail’ in general terms partly due to reasons in this article. As such I can’t guarantee the data solely contains Daily Mail rather than MailOnline articles (they are apparently separately companies though both owned by DMGT), though if I reviewed it I probably could.

End thoughts

I should carry out significance tests, but for a quick and dirty evaluation (if 60,000 articles can be seen as that) it serves a point — that the Mail isn’t as wrong as many would like to think.

As this former journalist says, the Daily Mail isn’t all bad and this wasn’t published to bash it. In fact it was the Guardian accusing others of being so hateful that spurred me onto this data research back in the day.

What can both papers learn? I’ve not seen their sales, link shares and page views or other closed data as that would be the best way to see if there was a correlation between tone and readership. But they can both learn that while the topics of their wrath, their readership, their font, their style, all differ, there are more similarities than some would be comfortable with.

Contact me if you want the data of nearly 60,000 articles, including 5,200 from the Mail and 7,200 Guardian, or go to Google Drive, buy you must attribute if you use it.

About Writing

The user researcher and the screenwriter

Screenwriting is getting a story onto paper which is then made into a film. User research is understanding how users behave when trying to complete a task or service, typically online. 

Putting it like that there may not seem to be too much similarity between the two, but explore further and I believe that they share the same goal – of documenting the human condition and producing a ‘truth’ within parameters.

Are there differences? Of course, but the similarities are that I find interesting.

User research
Time to interview – Ethan via Flickr

A history of two crafts

Screenwriting has been a craft for over a century, user research, in its current form, has only been embraced by governments on a widescale over the past few years. Being so new and flexible it does give me some room to manoeuvre, but in general the similarities are:

  • a quest for a truth, in defined parameters
  • a following of principles over rules
  • the aim of recording how people actually speak over how we think they should
  • show over tell

But what of market research? Well it’s similar but different to user research. Market research is about finding out users but is a more analytical approach, focusing on breadth often at the expense of depth. Government user research isn’t concerned so much about segmentation, weightings and the like (though they are not ignored). It’s about reaching the goal.

Screenwritng is similar — there are no rules, or if there are, there are too many exceptions. All that matters is writing a story that works.

A quest for truth

In very general terms, films aren’t necessarily about a truth – Superman has not saved the planet, the Inglourious Basterds didn’t kill the Third Reich’s ringleaders, Withnail never existed let alone acted.

But within their own universe, that created for the film, they must stay true to the rules created if they are to succeed. Superman can do almost anything, but even he must stay true to his rules — he will still ‘do good’ whatever is thrown at him.

The Inglourious Basterds burned Hitler and his cronies because to director Quentin Tarantino, that’s what worked in his story that included a glorified ‘kill the enemy film’, albeit from the German’s perspective.

Withnail may not have existed, but the relationships, tensions and ambitions Withnail & I explored are true enough in our world because it is set in the same rules as our universe.

And so on… So what am I getting at? You define the goals, you set parameters, and you stick to them if you want to succeed.

Individual approach

‘”There are no rules to follow, Donald, and
anybody who says there are, is just –”
“Not rules, principles.”‘ — The Kaufman twins, Adaptation.

Many crafts have principles rather than f rules. But user research is still fairly new in government and to a large extent it is still down to the individual or small team carrying out the user research to get to the goal. As such it is still down to the individual who does it.

This is reflected in that very few user researchers I’ve worked with have specialised in this for their careers. Instead they’ve come from a variety of backgrounds, and for myself it’s been content, journalism and anthropology.

It’s down to the individual.

Show, not tell

It’s rare for a film to succeed where all the characters do is tell you how good they are. In fact the audience largely forgives what we’re told about them if we see them doing wonderful things (ask Indiana Jones just hold old Marion Ravenwood was when he seduced her).

User research is about showing what is found — at show and tells, in videos of interviews, of producing quotes and examples. Don’t just tell us what is found, show it, and be consistent.

Getting to the heart of people

Ultimately screenwriting and user research and have one key goal – to show us that this is what life is really like. It is to produce something that is recognisable.

User research is like that. Taking something and passing it on to the next stage of the process. Looking for recognition that yes, this is what reality is and what we need to produce. You write down what people say, not what you think they say, and arrange it to make sense.

You also look for plot holes and inconsistencies and how to get rid of them, whether that’s more user research or revising the screenwriting.

A team sport

Finally it’s about others. User researchers don’t work alone, you’re encouraged to show, not tell, key parts of your process. You do not work around, your work forms the foundation of all that follows.

No script, no film; no user research, no project.

Ultimately it’s about getting a truth. The truth here but within the goal of getting a truth. Not the truth, but whatever will meet the goal of truth. And one that at the end, whether it is the final show-and-tell or handover, or a film, will leave the audience satisfied that they saw something true to what was set out.


Bureaucrats for Brexit: the forthcoming multi-million pound gravy train

Last week just over half of us voted to leave the EU. The Leave campaign promised us massive savings, £350m a week no less (well actually, less, they admitted the morning of the result), but did not speak of the costs.

Not costs of the nosediving stock market, the torpedoed national credit rating, the plummeting investment or sinking trade figures. I’m talking about the cost of government producing laws and guiding the public how to follow them.

EU papers being crossed out

Getting legislation to laymen

The government, once it legislates, does not then just say “well we’ve passed a law, you people should read it and know what to do”. The various departments (HMRC, the Home Office etc) must produce guidance on how those affected need to follow the law and carry out its requirements. And that’s where I and others like me come in.

I’m a freelancer who turns laws into guidance the public understands – but there aren’t enough of people like me or civil servants to update the content in light of Brexit. This work will have to be done in stages:

  • review all laws to see which will need to be updated
  • review all guidance, categorising it as:
    • not needing an update
    • update without a change in the law
    • update with a change in the law
  • debate and update these laws in parliament
  • update the guidance that needed a change in the law

That’s a lot of work, but how much are we looking at?

Review all EU-related laws

According to, there are 12,272 laws related to “European”. This may not capture all laws and some may be superseded, some may not be directly related to the EU, but let’s assume that this is the right figure.

These laws must be reviewed within the 2 years notice period we give the EU telling it that we’re out (formally know as Article 50), so as to be ready for exit day. We haven’t submitted Article 50 as of time of writing, and the civil service can’t start the work till this is submitted.

So that’s 12,272 laws that will need to be examined. In 2 years.

House of Commons debating
Busy day at the office – UK Parliament via Flickr

This is just for existing laws of course, and ignores any amendments, and I’ve not even considered all the new laws we’ll need to create just to leave. But in theory this review shouldn’t cost us any more as this would be included in the MPs’ salaries, barring expenses for many late nights.

Department of Brexit?

MPs don’t draft laws alone, they work with civil servants. So if MPs have 12,300 laws to review, it’s the civil service that will do the work of examining and setting out the initial proposals to ministers to set to the House. Then the civil service will need to update the guidance to inform the public.

Perhaps a ‘Department of Brexit’ will be created to do this, or else the departments will create their own Brexit teams.

Yet the civil service is already running at high capacity. Even if some work can be ditched because it’s reviewing or enacting EU-related legislation that will no longer be needed, there simply isn’t enough staff to do this new, urgent mountain of work.

Update the guidance

There are around 12,000 EU-related publications on GOV.UK, the site where pan-UK (eg passports) and England-only guidance to the law is published. Scotland, Wales and Northern Ireland each have their own sites. Each page takes time to review and write, based on experience let’s say each page requires 2.5 working days.

In some cases it’s a simple 2 minute read through and no change will be needed. Other guidance, like farm grants, can take several weeks and involve several civil servants, the same ones doing all the reviewing for parliament. So 2.5 days seem fair.

To outsiders this may seem bureaucratic but the law is complex, often badly written, and subject to interpretation that requires a lot of input. Content is written, subbed, approved by other civil servants and amended as needed. The teams I work with go as quickly as possible but there are limits.

How long will it take?

Each pages requires half a person-week of work, or 20 pages per week for a team of 10. Again, this is reasonable, again this sounds crazy to an outsider. So let’s bring in a team of 20, that will increase output to 40 pages per week.

Great, that means that team would take 300 weeks, or 6 years, just for the English law. Scotland, Wales and Northern Ireland don’t have as much as they don’t need information on passports (yet…). So instead of quadrupling let’s call it a round 1,000 weeks, or 20 years for the team to review and update all guides.

So to get this done in two years, a tenth of the time, we’d need 10 times the people, 400. Instantly ready to go, interviewed, vetted and knowing how to write in style. In addition to existing civil servants. With offices and equipment.

How much will it cost?

We’ll take the average content designer salary as £40,000 per year (excluding pensions and benefits), which is higher than the current advert but accounts for senior roles. That works out at £16m a year, more like £20m with IT equipment, office space etc, or £40m for 2 years for the team (and this assumes all stay, there’s no hiring problems etc).

In the context of the crashing economy, and of course £350m a week Leave ‘claimed’ we’d save, this is not much. But the civil service has to find £3.5bn in cuts by 2019-20 and HMRC, which will have to update its guidance on trade with the EU, is already set to lose 20% of its staff, for example.

Hiring the civil servants would also mean that this £20m a year would be ongoing and keep rising, and if their workload does decrease when we are out of the EU it will be hard to get rid of them.

So the government would likely look to contractors, where the cost would be at least double, but can be released once the job’s done. Let’s say £80m over 2 years. Just to review and revise existing laws.

What else will taxpayers have to pay for?

We have a figure of £80m just for de-Euroising guidance. But the Department of Brexit would also have the budget for at least the following over 2 years:

  • trade negotiators – we’ll need to make trade deals with up to 50 countries, we have no negotiators as the EU did this. We have 2 years to exit, and Leave’s much-vaunted EU-Canada deal took 7 years to complete
  • special commissions – for the Irish border, Gibraltar and other areas that arise
  • referendum campaigns – for potential Scottish and North Irish referendums
  • reform unloved EU laws – VAT on tampons, reviewing farm grants for the ‘butter mountains’. If we’re to leave it’s only fair MPs examine these much-mocked rules
  • document updates – passports, driving licences and the like, and I doubt that a simple switch to “European Economic Area” in the wording or whatever we go with will suffice or be cheap
  • visa system updates – new and updated visa system to cope with EU migrants, this is already creaky and can involve processing a 41 page form and its supporting documents for each person
  • EU divorce negotiations – the big questions plus things like pensions for MEPs and eurocrats

Total price for that? If we’re involving lawyers we could well add another zero to the estimate for the updated guidance, shall we say £800m a year?


This £800m is a rough figure I’ve extrapolated. What’s terrible is not that I’ve made some very broad assumptions in my back-of-the-envelope calculations, but that this is an envelope more than what the Leave campaign told us.

In short taxpayers can expect a hefty bill they weren’t expecting. These are only initial costs. I suspect that as more people do the work the Leave campaign should have done and give detailed costs of untangling ourselves from Brussels the more bills we’ll find. But as many vote Leavers are saying already, freedom isn’t free, and they’ll be happy with it.

This poll suggests why this is the case. While those who voted remain find the economy (and so the costs of Brexit) the most important thing, for leave voters it was taking control of laws and immigration, and money’s no object there. So perhaps I shouldn’t be surprised no costing was done because it would only strengthen the remain arguments among its supporters but would do nothing for its own Leave voters.

In short then, half the country wanted us out, but all of us will have to pay. Start saving, taxpayers.

[poll id=”3″]

If you’re interested in the legal implications I recommend reading this blog on constitutional law.

Scientific Research Writing

Scraping, screenplays and sexism

In the past couple of days there have been two big data posts that analyses sex and screenplays.

Polygraph’s Hannah Anderson and Matt Daniels scraped and analysed 2,000 screenplays and their dialogue to get data on the division of dialogue according to sex, age and other factors.

The Economist looked at data from USC Annenberg on nudity and ‘sexualised attire’ (aka revealing outfits and the like) in film, along with lead and speaking roles by sex.


Getting screenplay data

Both reports focused on presenting the data and key thoughts rather than delving too deep into interpretation. Analysing Hollywood is a complex business – like William Goldman said “nobody knows anything” when it comes to predicting success, let alone Hollywood and sexism.

The main thing of interest for me is the methods of analysing screenplays. Matt has a long and detailed method with links to script sources, along with the code on Github and a list of where he got the data from.

Potential uses

Both studies used data to explore issues around gender and films, but there is further potential with the data. For example:

  • emotion and sentiment – not a fan due to the drawbacks but possible to trace emotion in scripts, looking at such things as whether beginning, middle or ends are more or less emotional and is there a pattern
  • the split of action and dialogue in a script – do successful scripts have a divide (aka an avoidance of walls of text)
  • are women more confident or not – an extension of their sexism report, but it could be a question of whether female characters tend to ask more characters (or use emotional language)
  • writing level – what is the typical readability for the dialogue of heroes and villains, along with scripts in general and how does this vary by genre (would The Imitation Game or A Beautiful Mind be more difficult to read, let alone film, than Die Hard?)
  • is good writing important in a successful script – as with the study of readability, does having too many adverbs and other things that Hemingway hates hinder scripts
  • statistical significance – as Matt acknowledges, there are no statistical tests in their report, what tests could be done

Why we need this data

Maybe nothing comes out, but there is no harm in trying and while I never expect any rules to come out (Goldman is already laughing) but perhaps some very broad principles could emerge from the data. Even a finding of nothing can be something to report. The only pity is that due to grey areas of scraping we’d have to start from scratch rather than use the script data the teams have already used.

But it will be worth it and we can get away from what the Polygraph article calls “all rhetoric and no data, which gets us nowhere in terms of having an informed discussion.”

In the meantime if you want to search the data you can either check out the links or use the Polygraph tool here.

CW News

Analysing content beyond Google Analytics

Last night I gave my talk at Content, Seriously: Real strategies for real content. It was a pleasure to address a room full of content professionals, and a relief to be able to answer their questions.

If you couldn’t make it, or wanted the links, here’s a copy.

[slideshare id=59687482&doc=analysingcontentbeyondgoogleanalyticspdf-160317164624]