You Ask, I Answer: Proving Algorithmic Understanding?

Jen asks a follow up question from a previous uh episode of the Trust Insights live stream on uh social media algorithms. She asks, how do we know that the conclusions, uh for example, hashtags do or don't matter for uh retweeting or for visibility? How do we know that that's a true statement? The answer is you test, you build models to test those assumptions. And the way to do that is using machine learning, using some data science tools to prepare the data and then machine learning to build uh models that are statistically valid, statistically accurate to understand the outcome.

I'll walk you through this pretty briefly. Let's say we want to test to see if using too many hashtags dampens our uh reach on Twitter. How would we know? Well, to start, you would download all of your Twitter data, right? Um, you would maybe spend a week or a couple weeks doing uh tweeting with extra hashtags.

Maybe instead of one hashtag, you put in three or four or five, you vary it up, but it's more than what you usually do. And then at the conclusion of that period, you would take all of your existing data, maybe six months worth of data, and using either causal inference or uh causal inference with Bayesian uh static time series modeling or propensity score modeling, you would essentially say, here's my treatment period. This is the period of time when I applied this treatment. I used double the number of hashtags or triple the number of hashtags, and here's my control period of the past six months' worth of data. These software packages will say, great, let's take a look at everything else that is in this data set and then look at our treatment period, denoise any uh variations or any variability that that is statistically present in both sets, you know, sort of the the things you would have happened anyway, business as usual.

And from the remainder say, okay, this is the impact of the treatment of the change that you made. These techniques, these data science techniques are very common. They are very common, particularly in things like biostatistics, where you are taking an actual medication and you're applying it to some people and you're applying a placebo to other people. In doing so, because you can't A-B test a person, right? You can't split that person in half and give half the person one treatment and the half the person not.

You try to find people who are similar to that person in age, uh, physical condition, maybe ethnicity, uh, things like that. You try and control for as many variables as possi as possible. And then you have a control group and you have a test group. The exact same thing applies when you're testing out a social media algorithm. Maybe you and five or six other uh social media managers all team up and you run a test together, or maybe you download very large data sets from uh bunches of different accounts if it's on a uh network where you can do so, like TikTok or Twitter, and you run those models, and you see, is that statistically valid?

Does is the number of hashtags, does that change the reach or the engagements of these different accounts? And that's how you draw those conclusions. Um when I ran those uh tests to see if the number of hashtags impacts uh the the level of engagements, which is a proxy for reach, because no one can engage if they don't you don't reach them. Um the answer was it had almost no effect. There was no statistically relevant effect over it was three and a half million tweets.

So it was certainly a large enough sample size to look at um people who used uh two or fewer hashtags or two people who use more than two hashtags in a in a tweet. And the answer is it didn't really matter. There was no difference. Uh the same thing was true for other things like time of day and day of week. So to answer Jen's question, you have to test your assumptions using the data that you have available to you.

Ideally, a lot of it. Um, ideally data that's well balanced, that is that has the ability to uh remove noise or remove uh effects in the data that could be causing uh accuracy issues, remove covariance, for example, um, and correlates. But in doing so, you can ask the questions of the data that you want to know. And what's critically important is you want to run this ideally on your data or your data may and maybe your competitors' data, because as we know for particularly things like social networks, where there are these graph networks, and and there are things can happen and phenomena that can happen locally within part of the network that is not true for the network as a whole. You want to run this with your data to see if there's something that is true about you and your peers and your competitors that might not be true about a Kardashian as an example.

Uh those people and and people outside your industry, maybe they're gonna have different types of interactions in their portion of the network. So build your models, test your assumptions, look for those statistical outcomes that tell you that something does or does not matter, and then apply the appropriate strategies. It's a good question. Thanks for asking. If you like this video, go ahead and hit that subscribe button.