Sentiment analysis – why should you do it?

Days before he bolted for the Soviet Union, MI5 man Kim Philby held a press conference affirming his loyalty to Britain. And he smiled as he did so.

True the smiles were fleeting, but as Malcolm Gladwell covers in Blink, they were enough. Beyond his words his true feelings were revealed. His words were denial, but his subtle smirks revealed him as a traitor

Whereas that analysis of Philby’s 1955 interview required replays and slow motion, judging the sentiment and tone behind the written word is something that is easy thanks to modern technology. But why would you want to do it in the first place?

This series of posts will look at the why, how and when of word and sentiment analysis, how it can improve writing and critiques, and explain some recent trends in writing.

Sentiment analysis and other tools

Sentiment analysis, word count and other tools give you a standardised way of analysing a block of text and they add layers to your understanding and critique of any text – not just novels and other traditional subjects of analysis, but blogs, news stories, emails and more.

Its pedigree is in psychology, from Freud and his slips and psychoanalysis, Rorschach and his inkblots, to other psychoanalytical methods. All these systems are used in conjunction with other means to analyse a person and to get beyond the words used in interviews to find out who they really are.

The cartography of text

But let’s get away leap psychology to cartography. Consider traditional writing analysis as a map, for where key features such as parks, lakes and buildings catch the eye, so too in the prime analysis of text do certain phrases and writing style stand out.

However, maps often have other layers of information overlaying these key features, most notably height contour lines. I consider sentiment analysis like contour lines on a map – they add another information layer to your understanding.

Of course we don’t always need height contours for stand-out features; we know that Ben Nevis is tall and steep without them, and contours without the map to overlay seldom are of any use.

Standardised sentiment analysis

But contour lines are standardised throughout maps and can add clarity to the landscape, and allow quantitative comparisons, particularly over large areas.

For instance you can say the the Netherlands are not just low and flat but can say that on average about 1m above sea level while the central Swiss plateau is about 588m. This is much more specific than saying that the Netherlands are flat and Switzerland hilly.

Is this of use to everyone? No, I wouldn’t use this in planning a stag do. But it is if you are looking at factors for planning a hike, or for understanding flood risk, or for building a road, then it is useful.

So sentiment analysis, if done in a standard way, may not lead to the unmasking of traitors (though it can find liars), it can lead to standard comparisons that are useful in particular situations and add to our understanding of the written word.

Next time we will look four of the best reasons for doing so.