I’m a bit of a data packrat. My hard drive is littered with piles of spreadsheets, CSV files, MySQL databases, and more, which comes in handy more often than you’d think. When Klout announced a major change to their algorithm on October 26, 2011, I knew I had to take a look and see how scores had changed – but I had to do it in a statistically valid way. I strive to avoid producing “studies” and “social media science” that would be labeled cringeworthy by folks like Tom Webster.
Luckily, I had a pool of old Klout data with original Twitter IDs from July laying around, so I was able to do a longitudinal study of Klout scores for the same set of IDs over time. Let’s see what changed.
Data disclosure: this pool of approximately 5,000 Twitter IDs was originally randomly chosen from my Twitter followers. My audience tends to skew towards marketing professionals, so bear that in mind – this audience is not representative of all Twitter users.
Here’s the basic line chart for old Klout scores:
Here’s the basic line chart for new Klout scores:
Take note that scores declined nearly linearly once you were past the short head in the old model. In the new model, there’s a change in inflection right around 35 or so, and then again around 15. Also take note that in Old Klout, scores could be as low as 1; in New Klout, scores bottom out at 10.
The change in the floor score impacts the normal distribution of scores pretty significantly. Here’s Old Klout as a normal distribution:
You can see the pile of low level 1 scores at the very left. Now the same for New Klout:
The pile of level 1s are now piled up with the level 10s on the left side. For data quality purposes, this makes it VERY hard to distinguish between what’s a crap account (the old level 1s, which was a good indicator of bots) and brand new people to Twitter (usually the old level 10s). This is very unfortunate in itself.
Second, it almost looks like Klout tried to balance active, influential folks in around 45 on the new chart. To show you the best illustration of this, let’s filter out all scores below 11 on both data sets so that you can see people with at least some activity and/or influence.
Two things leap out: If you were above 45 in Old Klout, it looks like you might have gotten a downgrade. Second, look at the low end – a lot more people moved from the second quartile to the left side in the algorithm change.
So with all of these changes, is there a “good” Klout score in the new model for my dataset? In the old model that was activity based, anything above 15 was probably not too bad – active users of Twitter. In the new model, 15 is one of the break points, but right around 35 is where you see scores really pick up for this sample set. If I were looking for “influencers” in the new scoring model, I might want to start looking at scores of 35 and up.
GREAT BIG HUGE WARNING: Remember that this is a biased, non-representative sample. I am most assuredly NOT saying that you should run out and update all your social media marketing Powerpoint slides with a shiny new “35 or bust” bullet point. What I am saying is that Klout now appears to have two tiers in their data – lower influence in the 11-15 range and higher influence in the 35-50 range.
Does that mean you’re a social media failure if you have a Klout score below 35? No. It could mean you’re not going to get access to as many of the perks in their perks program, but that’s about it for consequences of a score under 35 as far as I can tell. Beyond that, keep doing everything that is a generally accepted best practice on Twitter: share interesting stuff, have real conversations, be human, etc.
Do Klout scores matter? In the old model, they were based on activity and could be gamed fairly easily. I don’t have enough data for the new model yet (working on that) to see what aspects of social media practice correlate less or more strongly with the score, so there’s no way to tell if their algorithm is an improvement or not for the purposes of judging who is influential. That means for now, they’re not any less or more accurate than they were before the update, so put as little or as much faith in them as you did before until we have better data.
For those folks who are data junkies, you are welcome to download the anonymized CSV files for these two datasets here:
Download Old Klout csv.
Download New Klout csv.
I’d love to hear about your conclusions in the comments.
You might also enjoy:
- Retiring Old Email Marketing Strategies
- What Is The Difference Between Analysis and Insight?
- Marketing Data Science: Introduction to Data Blending
- It's Okay to Not Be Okay Right Now
- Best Practices for Public Speaking Pages
Want to read more like this from Christopher Penn? Get updates here:
Get your copy of AI For Marketers
Wow, fascinating stuff Christopher especially the jump at 35 and baseline at 10. I haven’t paid much attention to my Klout score as a significant metric, but yesterday my score nearly doubled. The jump seems to connect in the way they computed my “amplification” score. I’m guessing it drops again soon to be more realistic or maybe they’ll award me for visiting their site so much in order to figure all of this stuff out.
There is a clear problem, though, when my score is higher than yours when you have so much more “real” clout.
I’m not sure you checked this (or that you could), but since Klout allows you to connect other social networks to determine the score, that could also influence the analysis – some users only having a score based on that single twitter account, while others based on multiple networks.
Well said George. It is really good analysis.
Excellent work there. Would love to hear how you see how our algorithm (PeerIndex) works out in your analysis. I am surprised that K decided to remove the signal for bots – but that must be a business choice for some reason. If you want to discuss our efforts, simply drop me a line.
The baseline for new Klout scores definitely is not 10. Actually, the majority of my Facebook friends have scores that are below 10, the lowest ones being 5.
However, the lowest scores among people I follow on Twitter are 10, and some of those people are ones I know IRL and who rarely, if ever, actually tweet.
Therefore, it seems possible that Klout has different baselines for different social networking sites.
agree to you. So I don’t put too much attention on my Klout Score as long as I’m updated with my Social media accounts.
Thanks for sharing your review Chris.
If you use more than one site, they tend to base your score mostly on your twitter activity, and the reason is probably the fact that on twitter, you are reliant on followers only, you cannot send friend requests, which makes life tougher, whereas on Facebook it is much more about friend requests only, at least it is for most of them that have less than 5000 Facebook friends.
Actually, even-though my score plummeted a precipitous 25% I’m actually glad they changed the algorithm. My Klout score was increasing steadily, however, my category changed from Specialist to Broadcaster before the change, and now I’m back as a Specialist, which is where I like to be.
I do agree that it is possible to game the system, and I may have been a bit guilty of that, as a result of joining Empire Avenue. Have you done any studies on that, or is it too new, OR worse, just another version of a social media game?
Very good study, well done. This throws some light on what has been a very hard situation to understand, so seeing these trends is very useful, thanks.
Love seeing the data Chris. I’ve decided to opt-out completely from Klout. Measuring influence online goes beyond one’s activity and shared activity within social media. Beyond that, the automatic creation of accounts and putting the onus on the user to opt-out, and then blanketing themselves in a weak excuse like “well Twitter data is public and you should protect your account” is a cop out. We shouldn’t have to protect our accounts or opt-out of a system that scores us.
They aren’t Google and they aren’t the credit reporting agencies.
What they are is a company using a healthy dose of psychology and game mechanics to sell word of mouth marketing to companies on the cheap.