So What? Do Instagram Hashtags Improve Engagement?

Hello everyone, welcome to So What the Marketing Analytics and Insights live show from Trust to Insights. Today Christopher Penn and I are here. Uh Katie Robert is off. Her husband works on the front lines. We appreciate his service.

And she's taking advantage of a little pre-holiday time to go off and just have some fun and have a little bit of a vacation. But yeah, Chris said we've got a lot of stuff up today. All right. Reminder, folks, if you're watching on Facebook, you need to authorize at StreamYard.com slash Facebook so we can see your comments. Otherwise, we get nothing at all.

So yeah, the uh a while back, our friends over at Agora Pulse uh put up his blog post. Uh and it's a really uh it's it's a well thought out post on do Instagram hashtags really lead to more engagement. This is about three years old now, so it's a little it's a little on the uh uh older side. Um I like stuff like this. I like stuff like this because I like to see people trying, you know, science.

Um the challenge that I think we run into sometimes is we don't necessarily have either the right data or the right methodology to really do uh the science in depth. So if you read through this article, it's a fairly long one here. Um they come up, they do uh some testing, they look at four Instagram accounts, they post some stuff with hashtags, some stuff with not. They conclude that um Instagram posts with hashtags increase likes by 70%. So we I said, you know, it's 2020.

Um this is post is a few years old. A little bit has changed Instagram since then. And you know, the question is like, what would this look like if we did it with more data, larger data and current data? And that's where this whole idea for for today's episode came from is the fact that we we don't have any of those things with with this older post. So the first place we want to start is where are we going to get the data?

Now, there's a couple different places you can get data, but the one that we use a lot is a service from Facebook called Crowd Tangle. Um we got we were grandfathered into this. Uh we used to be a paying customer until they they um turned it to a free tool, but then they closed the doors. It's only open now to like journalists and stuff. It's really intended for journalism and and academic research, which we actually do a uh fair amount for.

What's cool about it is you can load up a list of accounts and then export all the posts for that account for as you know, up to two years worth of back data, which is really, really handy. Uh, and as long as it counts as public, you can get the data. And it works for um Facebook, Instagram, and Reddit. They used to do Twitter, but then Facebook and Twitter had a little falling out. Um that's the first step.

Now, if you don't have access to this tool, um, your best bet is going to be working with Facebook's API directly. Um they have an Instagram uh graph API, you can write software to connect to it. It's kind of cumbersome and requires a lot of hoops to jump through. There's some pretty strict uh firm limits. But um, if you have access to a tool like Crowd Tangle, that's a good place to start.

And you can also do what Scott did in his post, which is you know, just run a bunch of tests. If you happen to work at an agency or something, you can run a bunch of tests on client accounts with their permission to get the same thing. But I like to have a large amount of data. So that's that's step one. The second step is we got to explore this data set.

Um, how large is it? So we pulled together a list, uh actually a couple lists inside the the tool, we have uh 4,200 brands, uh and all the stuff that they are are sharing, and then we have a uh a list from uh Crowd Tangle of influencers, and this is a crazy, crazy long list of like 11,190 uh influencers. And there's a lot of uh you know duplicates and stuff here, but there's so much data to be had. Um so the next step is in the process is we got to get the data out of here and start figuring out what it looks like. Now, John, when you do social media analysis, what is stuff that you typically look at?

Like when you're looking for you know uh conversations and things when you're looking at like marketing over coffee, what is it that you look for? Well, the biggest thing is that you know the stuff that you've done, uh, that Trust Insights does where you're not just looking at pure volume, you know, how much junk has been posted, you're not looking at just raw followers, likes you're starting to get to some actionable stuff. Um, but really the machine learning stuff you've done, if you can look at what content is being shared by others, that's the real measure of you know what's effective and what is moving the needle and where the true influencers are. You know, it's not just the millions of followers, it's that when they put something out there, it actually gets liked or shared. Yeah.

Now, the challenge with that is um Instagram doesn't have that whole lot of that data. Um, they do have handles, like every you know, the same as Twitter handles, but for the most part, we can't see what happens after somebody leaves Instagram. And that's true of everything except yeah, obviously, sites we own in Google Analytics. So that's uh not visible. So for this experiment, we're probably gonna want to look at just engagement and and kind of hope that engagement leads to results later on.

I mean, the uh the the challenge with engagement is you can get a lot of engagement for all the wrong things. Right. Well, and then uh you do, you know, the good news is that if you're measuring something you're caring about, you can't close the link. You actually do see how many folks come in the door and convert. So it's not uh it's not as if we're just kind of praying that large numbers equal success.

We can't at least see where they convert. Yeah. So let's go ahead and uh this is gonna be a bit like a cooking show because uh loading three million posts on Instagram uh into any kind of data processing software is a bad idea. Um it will take a really long time. So what we did is we pulled it in, we pulled in uh we have 1.3 million brand posts and 1.6 million influence posts for a total of three million posts in total.

Now, the first thing you got to do with all data, but especially with Facebook's data, because Facebook is not great about this, is you got to clean it up. Um, one of the things that we found is that their data tends to be very messy, and what users put in into like you know, comments and in just you know, uh posts, titles, things tends to be even more garbage. Like is it astonishing just how much garbage uh is in these things? So the first step in any of these processes is okay, let's go in, let's clean things up. So you have a really long function that basically says let's create things like comment rates, engagement rates, video rates, uh, mark things whether it's paid or not.

And critically for our purposes today, count the number of uh uses of the pound side as part of a hashtag, um, you know, with with the words and numbers after that. So, our first run through, go through and clean all this stuff up. Um, what we end up with is like I said, is three million posts, and we want to see we always ask ourselves a very naive question is there a relationship of any kind between hashtags and what do we say we're going after? Uh the original post was likes. Um let's do should we do likes or should we do engagements, like all engagements?

Hmm. No, your call, either way. Crushing the article is always interesting to me. All right, let's do likes, let's do likes, comments, and engagements, right? I think that seems like a safe and comprehensive look.

So we're gonna do a little bit of data processing here. Uh, all this is gonna do is put our data together and uh get those measures into one spot, and then we're going to uh do what's called a correlation plot. Now, a correlation plot is real straightforward. It uses simple correlation and it answers the question is there a relationship between uh two sets of variables? Let's see what we come up with here.

We have our numeric six thing with seven variables. We have our spearman correlation. And let's draw a plot. Ah, there we are. Okay.

Let's zoom in on this. So what we're looking for here is we want to know is there a relationship between all these variables? And this is this is where correlation plots are super handy because you can see across the board what the relationships are. So we see, for example, uh there's a perfect, almost perfect correlation between engagements and likes. That makes sense, right?

Because this is actually a case of what's called a um uh high correlation. Oh god. Um it's not linear dependency. It's basically saying you have one number, it's made up of other numbers. So of course you're gonna have a perfect correlation.

If you were doing machine learning, this would be a really bad thing. Uh, you'd want to knock out the things that are composed of each other. But so, yeah, we have a perfect correlation between engagements and likes. You have a uh very strong correlation between engagements and the number of followers you have at posting. You have a very strong correlation between engagements and comments.

Uh, you have a weaker relationship between engagements and engagement rate. Um, you have no relationship between engagements and views. And here's where we are. There's no real there's actually a slight negative relationship, not enough to be statistically relevant, uh, but there's they're basically saying no relationship between the number of hashtags and the number of engagements. The same is true for likes, uh, even less so for uh comments and things.

So our first pass at this, we're like, um we don't see it. We don't see that that experiment uh design showing up where yeah, there's a there's this negative relationship. So at this point, John, what do we do? Yeah, well, this is the it's a big negative there. Now I'm just wanted to be clear too.

So engagements and likes are even and engagements or comments are less though. Now every comment is an engagement, but that's because it's shorter because the majority of engagements are just likes with no comments. Is that right? That's correct. Yeah, the vast majority of engagements are people smashing the like button.

And then as far as where the statistically relevant mark is, is this everything is statistically relevant? You know, as like a P of two, or is it have to be over 0.2 here for us to consider it relevant? So this is a spearman correlation. Um, and that means that generally speaking, anything between minus 0.25 and plus 0.25 is is really weak. There's not much there.

Anything between 0.25 and 0.5 is moderate on both positive and negative. And anything above 0.5 is strong, saying, yep, there's a there, there of some kind, there's some kind of relationship. Now it's obviously does not uh look at all at anything like causality and stuff. It's just saying there's a mathematical pattern matching between these two things. And the first step in this kind of investigation would be like, is there a pattern match?

Like, do we see any relationship at all between hashtags and and uh these measures? And the answer really is no, there isn't. Yeah, and then not only we're not even at point two, like you said, it's actually negative, it could be reversed. So what it first order to the marketing team is just don't bother with hashtags. Well, I would say that the initial conclusion the article came up with that you know, um, you have to be using hashtags to get engagement.

The first pass we're coming up empty. So what we should do now is dig into this a bit more. Uh and the and the question is how would we test this a bit more to see you know is there something else at work here? One of the challenges that you run into with um with very naive tests like this is that there can be very, very un you know badly matched uh samples. If a million posts, uh if three million posts, there's three million posts and two and a half million don't use hashtags, um, and you know, only half a million use hashtags, then you've got a really unbalanced sample set, and that means that you may be drawing conclusions incorrectly, right?

The ideal uh and again, this is where hats off to the Agora Pulse team. The ideal way to do it is to do that controlled A B test. Like that's that's really a really good way of doing things. Um the challenge you run into is obviously if you only have a handful of accounts and they're not in you know different industries and things like that, then you're gonna still you're gonna come up with an underpowered test. So there's a way to um simulate an A-B test uh with a uh technique called propensity score matching.

So we're gonna go ahead and we're gonna basically say, even just split the group of all these posts into two groups. One group uses hashtags, one group doesn't, right? Let's worry about the the number of hashtags later. Just want to find out call the group that uses hashtags, we call it like a treatment, right? Because they've used the they've done the thing and call the group without hashtags uh the control.

They didn't do the thing. What we want to do is we want to do a sample. We can't do the whole data set because uh my computer will melt. Um, and this show will be like, well, 11 hours long, like you know, 10 and a half hours, which will be just staring at the screen. Um, so we're gonna take a sample of this of the three million now we'll uh again, this is where a lot of people tend to go wrong uh the question is how large a sample should you use um there are any number of really good simple sample size calculators on the web um this one I uh just you know this is like the first Google result I said I want a 99% confidence interval which means that if I repeat this test a hundred times 99% of the time we'll come out with the same uh answer and I want a confidence interval one percent meaning the every answer I get I'll be plus or minus one percent correct like that's a pretty solid if you want to get really really aggressive can do a uh what half of a percent what size population would I need to survey out of out of my three million people in order to get that half a percent margin of error uh I would need 65000 so let's go back here and in my sample I'm gonna do exactly that I'm going to sample my data set and choose 65 at random 6500 um samples here what do we got here it says object oh I ate my treatment file there we go okay so I now have a sample of 6500 randomly chosen that's gonna be a big enough sample to go okay there's some statistical reliability here let's go ahead and run um our matching and what we're doing is we're taking this sample um we're splitting it into two groups and then we're looking and the software's gonna go through and identify um and try and match up and create a subset of this of saying who uh which posts are similar.

So, like John, if you have a thousand Twitter uh uh Instagram followers and I have like 10,000 followers, um there's a good chance that we're not a good match, right? Because there's a we're different. But if you have a thousand followers on Instagram and I have twelve hundred, it's probably a closer match, right? We're more alike than we are different numerically. And so comparing our results would make more sense as opposed to comparing you know either one of us with like I don't know, uh pick a celebrity or choice.

I don't know. I'm trying to do that. We'll throw a Kardashian in there for a bit. You know what? Let's do Dwayne Johnson, right?

You know, Dwayne Johnson posts on Instagram, and and you know, millions of dollars of things get sold. Um what we want to do with this technique, and this is a technique that's super useful for all kinds of uh marketing data, is we want to do that, run that sample and figure out okay, what do the what does it look like? So we have here our balanced data set. Here's the control, here's the the treatment. What are the differences?

Actually, in our in our sample set, um we see that people who use hashtags have gotten substantially worse performance on stuff. So now is now we're really into head scratching territory, like what the heck is going on here? Uh, out of this, you know, uh a statistically valid sample, um, we're getting we're seeing some substantial differences between these two groups. Uh and the in particular, the engagement rate is is much lower uh between the treatment and control group for the hashtag. So now we're like, huh, I think we might have you know poked a another hole in this in this thesis that you know you must use hashtags for engagement.

Yeah, it's at least for where we are, that's that's not the case. We had a funny David chimed in with a question. Said he's worked with a statistician that liked an 80 that said 85 was still a coin toss and he wanted to have uh his clients do it 99%. I you know, if that was the case, I'd love to uh do it, have an 85% confidence coin toss and play for money, but that's you know, just me. I can see where he's coming from with that though.

Oh, yeah, totally, because an 85% confidence interval if you think about it, if you run a test 100 times and 85% of the time it comes out okay, and 15% of the time it you can't reproduce the results, you're like, uh that's a lot of variability, right? Um it's like you know, what a revolver has what six uh six uh things, right? One out of six is sixteen percent. So basically you're playing Russian roulette uh at that point. If you have a one in six chance of being wrong, um that's that's not great odds.

I would rather you have a have the you know the one chance in 90 out of 99 that uh that things have gone wrong, you know, certainly for stuff like this. So what we've done here is we've done a simple test of correlation and found that there was no relationship. And then when we did a a dug into the sample and tried to do basically a retroactive A B test, we found that actually for this sample set and this group of uh uh people, um, hashtags didn't help things. In fact, hashtags might have made things worse. So what do you think, John?

Should we just completely uh can using hashtags on Instagram? I don't know. Is there any way to flip it to find out if there's you know any things that did well that do have more hashtags? You know, is there any way to look at it from the other angle, or is it just better to call this dead? That's a good question.

Uh I would say the next step, because we're talking about a massive list here. This is across tens of thousands of accounts on Instagram, brands and and and individuals. I think then the next step, if if this was something that you know a client had engaged us to do would be, okay, let's get together a curated list of a couple hundred accounts just in you know the industrial concrete industry or whatever, whatever the client's industry would be, and say, okay, let's see how this looks specific to your industry and see if there's a different pattern. Because again, we we we've done here is we've lumped Toyota and JetBlue and the the Republican Party and you know the Smith and Wesson all together into the Bob's pizza too. Tiny geo.

Exactly, Bob's pizza, you know, with just one giant gumbo pot. Um and everything kind of gets you know mashed together. Uh whereas if you're looking at a specific industry, that industry may have very very tailored things that are going on that could be like, okay, we need to maybe focus in on on just this thing. In fact, um, yeah, we we've got a little time. Well, no, because we have to reload the whole data set, and that takes about 20 minutes.

Uh I would say, you know, one thing we could do in in the future is identify are there accounts that you know use hashtags sometimes, but not all the time? And if so, um, you know, then can we look at the hashtags they use and say, okay, can we now in on that industry? Like you know, women's shoes, for example. You know, it's our our our favorite go-to uh for for example is to say, okay, what about just this in women's shoes? Could we identify um just that just those people and then build ourselves a data set and test it just for women's shoes?

Let me think about this. Could we do that? How hard can we push R without it running for without without your laptop first community? Um let's try. Oh, that's a tough one.

Um, how about pound fashion? Fashion. I like fashion. Okay. Uh we want this to be all this fashion count.

And we're gonna need to replicate that for our influencers as well. We're gonna bet you we're gonna get a ton more influencers than let's see how long this takes if this utterly melts down. Yeah, is there any uh as far as I know, there's no public hashtag data. You can't just say, like, okay, here's the top 200 hashtags. So there are um data sets that say that.

Um I don't know how reliable they are. Like, I don't know where they're getting the data from. Right. I that that must be scraped. I cannot, it's not from the API.

Okay, let's see what we get here. Out of those three million posts in each, we're gonna get um 2900 29,000 brand and 17,000 influence. That's interesting. I would have expected the opposite to be true. Okay, let's uh put together ourselves a take our two fashion data sets here, trim those down.

And now let's rerun our correlation plot with just our smaller set. Let's see what we got for fashion. Interesting. So now hashtag count and engagement, there's a minor positive relationship. Um so in in just this, hashtags may work for you know for fashion.

Um but it's not gonna triple your huh? Well, it's not gonna triple your results. I mean, we see a slightly No, no, but there's a problem. We we filtered by a hashtag, so inherently every result contains a f hashtag. Right?

We don't have any any data in our our our set of accounts that were in this space that did not use a hashtag because by default we counted the fashion hashtag. Right. And so, but so we do see we don't see an uh correlation between hashtag and likes or um engagements. Right. So let's do this.

Let's take just the word fashion. That will get us, I think, a little bit closer to where we want to be. Because that could that would occur whether or not you use the hashtag. It's unfortunate I'm not strong enough in R to actually be able to do play-by-play here. So we are at now uh we now have 70,000 posts just using the word fashion or brands, 45,000 not.

So now we're much more we have a again uh uh a correctly balanced data set. Let me jump on in here to our chart, and we see still not rocking the relationship. So even here for just a subset, it there isn't a there. Yeah. So I think you know, we're it's you could obviously repeat this over and over again for you know wine or uh sports or politics, um, and probably get similar results.

But I'm starting to feel like this might be you know barking up a uh uh the wrong tree for for the hashtag side of things. Yeah, right. So that would be the last step for a company that you would take your the 10, 15 hashtags you think work, and you could actually see for those if they literally work or not. Exactly. And you'd use the generic term and then see if it there's the the skew between the generic and the thing.

Um let's pick one more. Um what do you want to use sev fashion? Well, that's I you know, um, oh, travel is huge insta. Uh travel's travel's gonna be too big. Pick something in niche, something like you know, a little bit obscure.

Hmm. Uh how about cosplay? Oh, yeah, that's a good one. Like that. All right, let's see what we got here.

Okay, so we ended up with sixty-eight posts by brands, two hundred thirty-four by people. It's a really small group uh yeah, that's a lot more niche than I thought it would be. Yeah. Um well, keep in mind, these are brands and influencers broadly, these are not people specific to cosplay. Like we did not put together a cosplay uh influencer list.

So you're not gonna get a ton of that. Um look in here, we actually have uh uh now an actual negative correlation, which means that as hashtags go up, uh in engagement goes down. So it's a negative for for cosplay, which is uh as a term. Uh this is really interesting because now we're at a point where like, huh, hashtags actually hurt more than help there. And again, we don't know why, because we're not looking at the underlying data to see if like these are just you know, maybe there are posts that were just you know bland or whatever.

Uh and obviously this is not addressing things like post quality is it to the right audience? Um, did you boost it? Is that this is just the the raw organic stuff, but definitely a case of there's data there there. Yeah, and uh just trying to, you know, as you start to think about what actually draws that. So my first thought would be that if it there's spammy posts, you know, that's why the hashtags are in there and those aren't doing well so that would be one theory um another and even to take it up another level too I was trying to think the only way hashtags can make a difference right is you you're assuming that users have standing searches for those hashtags right I mean is there any other use case where how else could hashtags help uh you know I guess it a hashtag is mainly a discovery function right to use a hashtag um because somebody taps on it because they want to follow it and see stuff in there so uh there may be some opportunities to to um use that for discovery but it doesn't it may or may not lead to engagement may just lead to people finding your stuff yeah yeah one of the things that is not in here is we did not attempt to track you know um changes in follower counts from post to post to see if like a certain if a certain numbers of posts generated more followers um so that's also not in here that's something you have that has to be computed separately but it's definitely a case of uh you know there's there's there's it's not looking good it's just not looking good so um so that pretty much wraps up today's show for uh what data is available for Instagram um how to explore it using a utility you know like our you can use other tools there are you know plenty of tools IBM's SPSS modeler is a fantastic tool you know disclosure we're an IBM business partner but it's a uh drag and drop clickable interface rather than writing code uh it's a lot easier for folks now on a small enough data set, you could do this with Excel.

There's nothing wrong with good old-fashioned Microsoft Excel. It gets the job done a lot of the time uh and then from here, once we do the analysis, we walk through a couple of different steps, a couple of different tests to look at the data and figure out, you know, what's the next step? What should we do? And then in this case, we've pretty much ascertained. There.

So if you got uh follow-up questions and stuff, by all means, you know, let us know. Uh, we'll put up some information about where you can follow for this. But John, any uh any parting thoughts on our great Instagram hashtag test? Uh yeah, I don't need to hashtag any of this stuff when I post it to Instagram. So we're good.

I would say if you're gonna do it, test it. You know, test it on your account because at the end of the day, the only account you really care about is yours. Right. So test it and see if it works for you. If it doesn't work for you, great.

Um, you're an outlier. If it doesn't work for you, well, we kind of found that out. So all right. Well, let's uh roll on out of here and we will talk to you folks next week. Thanks for watching today.

Be sure to subscribe to our show wherever you're watching it. For more resources and to learn more, check out the Trust Insights Podcast at TrustInsights.ai slash TI podcast and a weekly email newsletter at trustinsights.ai slash newsletter. Got questions about what you saw in today's episode? Join our free analytics for markers Slack group at trustinsights.ai slash analytics for marketers. See you next time.