You Ask, I Answer: Twitter Bot Detection Algorithms?

In today's episode, Joanne asks, in your investigation of automated accounts on Twitter, how do you define a bot? So this is a really important question. A lot of the time when we use software packages that are trying to do detection of something and are using machine learning in it, we have a tendency to just kind of accept the uh outcome of the software, especially if we're not technical people and it says like this is a bot or this is not. We just kind of accept it. And that's really dangerous.

It's really dangerous because it's not clear how a model is making its decisions, what goes into it, how does it make its decisions, uh, how accurate is it? And without that understanding, uh it's very easy for things like errors to creep in, for bias to creep in, uh, for all sorts of things to go wrong, and we don't know it because we don't know enough about what's going on under the hood to be able to say, hey, this is clearly not right. Um, except to uh inspect the outputs, and then again, if you're uh not technical, uh, you are kind of stuck in a situation of either I accept that the outputs are wrong or um I find another piece of software. So in our Saturday night data parties that we've been doing, identifying uh Twitter accounts that uh may be automated in some fashion, uh, there are a lot of different things that go into it. Now, this is not my software.

This is software by Michael Kearney from the University of Nebraska. Uh it's open source, it's free to use, it's part of the R, it's an R package, so it uses the R programming language. And that means that because it's free and open source, we can actually go underneath uh go under the hood and inspect to see what goes in the model and how the model works. So let's um let's move this around here. If you're unfamiliar with open source software, particularly uncompiled, which uh the R programming language is a scripting language, and therefore it is uncompiled, it's not uh uh a binary piece of code.

Um you can actually look at not only just the software itself, right, and explain uh the author goes through and explains uh how to use it uh the software, but you can if you're again if you're a technical person, you can actually click into the software itself and see what's under the hood, see what the software uses to make decisions. This and this is this is why open source software is so powerful, because I can go in as uh another user and see how you work, how do you work, Mr. Piece of Software? How are the pieces uh being put together, and do they use a logic that I agree with? Now we can have a debate about uh whether my opinions about how well the software works should be part of the software, but at the very least, I can know how this works.

So let's go into uh the features, and every piece of software is gonna be different. This is just this particular author's uh syntax, and he he's done a really good job with it. Um we can see the data it's collecting. If we scroll down here, uh like since the last time, time of day, the number of retweets, number of quotes, all these things, the different clients that it uses, uh, tweets per year, years on Twitter, uh, friends count, followers count, ratios, and all these are numeric, or many of these are numeric features uh that uh you're good the software's gonna tabulate and essentially create a gigantic numerical spreadsheet uh for it. And then it's going to use an algorithm called gradient boosting machines to attempt to classify uh whether or not an account is is likely a bot based on some of these features.

And there's actually two sets of features. There's that initial file, and then there's another another file that looks at things like sentiment, tone, uh, uses of different emotions and emotional keywords, and the range, the it's called emotional valence, the range of that within an author's tweets. So if you're sharing, for example, in an automated fashion, a particular point of view, let's say it's it's uh propaganda for the the fictional state of Wadiya, right? From uh the movie The Dictator. And you are just uh promoting uh Admiral General Aladin over and over and over again, and you're you're gonna have a very narrow range of emotional expression, right?

Uh and there's a good chance you're gonna use one of these pieces of scheduling software. There's a good chance that uh you will have automated on a certain time interval, and those are all characteristics that this model is looking for to say, you know what, this looks kind of like an automated account. The posts are at the same time every single day, the amount of time between tweets is the exact same amount each time. Um the emotional range, the context is all uh very narrow, almost all the same, probably a bot, as opposed to the way a normal user, a human user functions, where the the the space between tweets is not normal, it's not regular because you're interacting and participating in conversations. The words you use and the emotions and the sentiment of those words is gonna vary sometimes substantially because somebody may anger you or somebody may uh make you really happy, and that will be reflected in the language that you use.

And so the way this software works is it's essentially quantifying all these different features, hundreds of them, and then using this this machine learning technique, gradient boosting machines, to build sequential models of uh how likely is this a contributor to uh a bot-like outcome? How how regular is this this data spaced apart? Now the question is once you know the m how the model works, do you agree with it? Do you agree that all these different characteristics are relevant? Do you agree that all these are important?

In going through this, I have seen some things that like I don't agree with that. Now, here's the real cool part about open source software. I can take this software and what's called fork it basically make a a variant of it that is mine, and I can make changes to it. So there are, for example, some Twitter clients in here that aren't really used anymore, like the companies that made them or have gone out of business. So you won't be seeing those in current day tweets.

We still want to leave those in because for historical Twitter data. But I also would want to go into Twitter now and pull a list of you know the most common Twitter clients being used today and make sure that they're accounted for in the software. Make sure that we're not missing things that are features that could help us to identify those things. Um I saw in the model itself, they made a very specific choice about the amount of uh cross-validation folds in the in the gradient boosted tree. If that was just a bunch of words to you, cross-validation is basically trying over and over again.

How many times you rerun the experiment to see is the result substantially similar to what happened the last time, or is there a wide variance? You're like, hey, that seems like what happened these two times or three times or however many times uh was random chance and is not a repeatable result. They use a specific number of the software. I think it's a little low. I would tune that up in my own version.

And then what I would do is I would submit that back to the author as a um as like a pull request and say, hey, I made these changes. What do you think? And the author go, yep, I think that's a sensible change. Yep, I think that Twitter client should be included. No, I disagree with you about uh how many iterations we need, or how many trees we need, yeah, or how many cross-validation folds we need.

And that's the beauty of this open source software is that I can contribute to it and make those changes. But to Joanne's original question, this is how we define a bot, right? The software has an algorithm in it, and an algorithm, uh, as my friend Tom Webster says, is uh data plus opinions. Right? It's data plus opinions.

We are choices we make. And so by being able to deconstruct the software and see the choices that were made, the opinions that were encoded into code, and the data that it relies on, we can say, yes, this is a good algorithm, or no, this algorithm could use some work. So that's how we define a bot here. Maybe on another Saturday night data party, we'll actually hack on the algorithm some and see if it comes up with different results. I think that would be a fun, very very very, very technical uh Saturday night party.

But it's a good question, and it's a good question I would urge you to ask of all of the machine learning systems that you interact with on a regular basis, all the software you interact with on a regular basis. Is there a bias? Is there opinion being expressed by the developer? What is it, and do you agree with it? Does it fit your needs?

And if it doesn't, you may want to consider a solution like open source software where you can customize it to the way you think uh the system should function. So good question. If you have follow-up questions, leave them in the comments box below. Subscribe to the YouTube channel and the newsletter, and I'll talk to you soon. Take care.

Want help solving your company's data analytics and digital marketing problems? Visit trustinsights.ai today and let us know how we can help you.