You Ask, I Answer: Third Party Schedulers and Social Content Performance?

In today's episode, Ian asks factor fiction using third-party schedulers for social media posts get a kick in kick in the shins as they haven't come from the native platform tools. This is a good question because it highlights some of the issues that we have in uh marketing with uh analytics and statistics. Specifically, some companies have done some testing on this. Uh Agora Pulse has done some testing, Buffer has done some testing, a few other uh social media posting tools have done some testing, and obviously uh one of the challenges there, uh well, there's several challenges. One is uh there's an inherent conflict of interest if a uh third-party company is testing to see if third-party companies are penalized, but that's not reason enough to disqualify their efforts because as long as it's done in a statistically valid way and um methodologies are disclosed and data is made available, then it's totally fine, right?

It's the same with any kind of academic research. You disclose your funding, you disclose your conflicts of interest, if any uh so that people know that the study may or may not be as neutral as it could be. Here's the issue with the testing that's been done. It's not that it's been by third parties, it's that it's not been done in a statistically rigorous way, not rigorous enough. Um the pro the reason for this is that a lot of the data uh that's out there is It's not well structured.

So there's two ways that you can uh statistically test for something like this. One is to run uh true A-B tests, where you're running the same content, uh, one from one tool, one from whatever the the control is. Um, and it has to be the same content. Uh or has to be very similar to the the content so that uh a post that you know contains cats uh and people hit like on cats um does well and it shouldn't matter which method it you've posted by. So you have to post one with cats in your treatment group and one with cats in your control group.

That tends not to be um the case when a lot of testing goes on. And it understandably so, because if you're doing that, uh you're essentially double posting your content and uh it gets tricky. It gets tricky to manage that. The second methodology that you can use is a stats technique called propensity score matching, where you take a whole bunch of data and you group it, you cluster it, and then you try to find uh data that is similar in each of the groups. It's kind of like uh building a uh an A-B test retroactively.

The challenge there is you need a lot of data. You need uh you know, thousands of data points so that you can match control groups, uh create essentially a controlled uh a control group and a treatment group uh of similarly performing content so that you can see if the variant is because of the treatment or not, in this case using a third-party tool. So how would you go about doing this? Well, again, if you're going the A-B route, you create a bunch of social content, uh half of it, uh you you basically duplicate it, you make a copy of everything, right? And then you put one copy in your native platform and one copy in your uh scheduler, and you s after a while you run the numbers and you see which content performed better, excepting the fact that you're gonna have double posted content.

The second method is uh using again uh the data that you get out of your systems uh you will need to tag if if the social platforms and their analytics don't give it to you you'll need to tag which was third party scheduler and which was native and then run a propensity score match. So let's take a look at what this looks like here. So this is uh Twitter uh Twitter very kindly provides uh whether a post was well what tool a post was from which is super super handy and so if I look in my data frame here let's go ahead and take a look at our sources we have eighteen different possible sources we have a whole bunch of uh individuals and then we have the the Twitter family of official uh clients and then some more individuals there. So what I've done is I've extracted the last thirty two hundred tweets from my account removing retweets because retweets uh we're we're trying to test uh stuff that is fr native uh and then assigning if it was used if Twitter the native platform was the the source I want to go with uh in this case I'm gonna make that the treatment doesn't really matter just we just assign the groups and then if it was a third party tool make it uh a zero. You know what makes more sense to logically do this.

Let's let's back this up. So treatment's gonna be using the third party tool. Let's go ahead and run our matching test. Okay, and we have as a result the treated group, meaning a third-party tool on measures of favorites, retweets, and engagement. When we look at the mean difference, third-party content in for my account on Twitter does slightly better than uh the uh stuff from the native platform.

Now, again, this is really important. This is my account, so you can't generalize this to anybody else. You should do this on your own account. Um, and uh this is for Twitter only, so you'd want to perform similar tests with similar data on the social platforms you care about Instagram, YouTube, Facebook, uh whatever. But this is the methodology for how you do this.

You can either retroactively do it with a propensity score match, or you can do a true A-B test. Let's take a look at uh one more. Let's do I I know it's in the data set, so let's test this. There we go. So in this case, for this account, similar the treatment uh slightly better.

So is there a difference in third-party tools? Yes, and in actually in this case, it's uh a couple of uh tests here. We've seen that it it's a slight improvement. Should you generalize this to your social media strategy? Uh no.

No, I would test this with your own data. Make sure that it this idea applies to you specifically, because that's what you care about. You don't really care about my account. You care about your account. So you'd want to run this similar type of testing uh for you.

That said, based on the less statistically rigorous tests we've seen, based on doing something with a bit more rigor, uh, so far, haven't seen any strong evidence that third-party tools cause harm. If anything, scheduled content, at least in my tests, performs slightly better. So, does that mean that you should change your strategy? No. Um the test for yourself first.

Get the answer for yourself first, and then build your strategy on that. Um, and if you don't have the capacity to do this, then I would say frankly, it doesn't hurt to use third-party tools so far. You know, again, the various tests have been out there kind of proven just one degree or another, that there's not substantial harm. There's certainly nothing that makes you go, oh wow, that was that's terrible. Like, let's let's stop using these immediately.

It's not the case. Um, and again, you have to test this for each platform you're on because each platform has different algorithms. So, uh, you have follow up questions, leave them in the comments box below. Subscribe to the YouTube channel and the newsletter. I'll talk to you soon.

Take care. Want help solving your company's data, analytics, and digital marketing problems? Visit TrustInsights.ai today and let us know how we can help you.