One of the greatest SciFi movies of all times, in my view, was Stanley Kubrick’s 2001–A Space Odyssey . A key element in the story was HAL [ at left], a computer who got its feelings hurt, then caused havoc and human misery.
I spent three hours the other day, on the phone, clearing up a mess with a large utility company. I mentioned it in a tweet saying it was the kind of experience that made me miss chatting with my buddies at two other carrier companies.
Anyone with common sense would know that I was being sarcastic. But of course, to have common sense, you would need to be a living, breathing human being.
Computers are different than you or me. They can aggregate, compile and report numbers at far greater speed than you or me. And that might explain why I received thank you messages from two of the three companies a short while after posting my clearly sarcastic comment.
So what happened? Who did what? How did the thank you notes get generated?
I can only guess.
What I think happened is that these humongous companies are using sentiment analysis software. These are programs that scan the internet looking for company and product names, then scanning words near those names. So if I said “AT&T” for example and the software then saw that I started my tweet with the words, “What fun,” and then saw the word “buddies,” the machines then assumed my sentiments about these three companies were favorable.
The computer “bot” then generated a thank you tweet. It all happened very fast and very inexpensively because no humans were involved. Whoever managed the program nbever saw me, but I became a line on a spreadsheet, so that she/he could report how the software had helped the company scale, engage and respond.
Except the response was lame and inappropriate. But before I stand on a bull pulpit talking about how useless sentiment analysis software is, it is more complicated than that.
Social media has hyper escalated the number of times people and companies are being mentioned everyday in social media. People are also expecting to be heard with complaints and praises better than they used to be.
Five years ago, Scoble and I started our book, Naked Conversations with the observation, “We live in a time when people don’t trust big companies.” We went on to argue that this was caused, in part, because companies don’t 
listen, nor do they respond when we try to be heard by them.
Looking back, I see much evidence that companies are realizing this problem and trying to do something about it–but they are finding it to be really, really hard. Yet the demand–perhaps the requirement for it–gives modern companies little choice to listen and respond to social media conversations.
Sentiment analysis is about three years old. It is one of several social analytic software subcategories all working on the daunting challenges of measuring emotions, relevance, influence and other fluid nouns. Measurement guru KD Paine tells me that sentiment analysis is right “just about 50% of the time.”
This is a half full/empty glass situation. You may say 50% of the time, these programs provide crap. The other half of the time, you get some accurate results. Now, all you have to do is figure out which time is which.
In short, they have come a long, long way. Yet they have just as far to go.
I don’t think that sentiment analysis will reach 100% in my lifetime or in the lifetime of my yet unborn great-grandchildren. Computers are less than embryonic in demonstrating common sense.
For that, you still need people. Today’s solution in my view is to use the best of such tools in measuring public sentiment. But before you fire off a thank you tweet–slow down enough to have a human review the results.
Otherwise you have a 50% chance of looking really, really stupid in a public venue.
{ 1 trackback }
{ 13 comments }
Take a look at Twitter sentiment analysis tool http://smm.streamcrab.com, its written in python and uses Naive Bayes classifier with semi-supervised machine learning
Like Seth Grimes said, many automated sentiment analysis programs can exceed 50% reliability between human coders and the tool’s output (I haven’t tried the SAS offering, but I’ve seen success with some social media monitoring tools, such as Crimson Hexagon, as well as some of the older enterprise media measurement systems such as Attensity360 or Cymfony).
The largest hurdle with automating sentiment analysis is that the systems require a lot of human training and patience. Reliable sentiment analysis takes a long time to set up (think weeks or months) and it’s often not worth the time/financial commitment for one-off research projects or for brands that receive only modest amounts media coverage.
If you have the time, budget, and plan on analyzing large amounts of coverage over a long period of time, however, automated sentiment can be a practical alternative to human coders.
All these confusions actually can happen due to uncontrolled levels of audience. Leveraging the right audience will not require any bots/tools help even if you have great number of followers. In my point of view, when you try to say something and you have your audience as your community, they will definitely hear what you say. The rest is all useless!
I remember looking up Toyota on SocialMention.com during the height of the sudden acceleration recall crisis. The sentiment was largely positive. As I dug into the comments, it became clear that the sentiment analysis engine couldn’t distinguish sarcasm. Sentiment analysis is a fine first step but still requires human oversight.
Right on the money, Debra. Sentiment analysis is not about traditional ROI anymore. Those who try to measure it this way are playing a new game with an old rulebook.
Many tools can do a lot better than 50% accuracy, particularly with training. Katie Paine, who you quote, says she herself, in her case regarding SAS Social Media Analytics.
In any case, folks who are interested in a deeper dive should check out the Sentiment Analysis Symposium, http://sentimentsymposium.com .
Seth
Great example of the drawback of sentiment analysis. IMHO, I think sentiment analysis at companies is driven by the need to show ROI. In their case, increasingly positive sentiment as a ROI of social media involvement. Based on my assumption that this is a driving factor (and why so many social media monitoring companies are integrating it into their software), I would argue that in most cases it is the wrong ROI to measure. The right one would be the increased engagement and real interactions that lead to feeling more loyal to the company, or organization.
All these confusions actually can happen due to uncontrolled levels of audience. Leveraging the right audience will not require any bots/tools help even if you have great number of followers. In my point of view, when you try to say something and you have your audience as your community, they will definitely hear what you say. The rest is all useless!
Even if you account for sarcasm, the web is full of local folksonomies, slang and double entendres or puns. PEOPLE have a hard enough time communicating and understanding one another, let alone the software knowing what we meant.
I’m a little more optimistic about the likelihood of working sentiment analysis coming about – but I see it as being 10 – 15 years down the road with the research labs. Definitely not ’95% perfect now on our freemium web tool’.
All that said – I’m thinking that your thank you notes were not so much a product of sentiment analysis and more likely a case of your tweet being taken out of context. Too many monitoring teams are focusing on just the keyword hit and not taking the time to take a step back and take in the overall gist of the conversation. They don’t have the context of who you are. They don’t have the context of what you were talking about.
They aren’t being social but doing a very poor pantomime of being social.
Rob Clark
http://disclz.me/RobClark
Completely agree, and I would add that even some humans could not sentiment accurately! Generally it is hard to figure out sentiment when the message is written/typed. How many misunderstandings have arisen from jokey emails or texts?
This article is very accurate, and timely, as corporations start factoring social media marketing into their budgets.
Note: I would correct this omission though.
You said: We went on to argue that this was caused in part because companies listen, nor do they respond when we try to be heard by them.
I think you meant: We went on to argue that this was caused in part because companies DON’T listen, nor do they respond when we try to be heard by them.
I caught that because I’m not a bot. =]
[machine un-readable sarcasm alert] wow Shel. That was the lamest post ever (and by “lame” I mean awesome and dead on – but I don’t want the bots to know that. Too much sucking up spoils my street cred.)
I remember looking up Toyota on SocialMention.com during the height of the sudden acceleration recall crisis. The sentiment was largely positive. As I dug into the comments, it became clear that the sentiment analysis engine couldn’t distinguish sarcasm.
Sentiment analysis is a fine first step but still requires human oversight.
Comments on this entry are closed.