You Ask, I Answer: Detecting Bias in Third Party Datasets?

In today's episode, Jim asks, are there any resources that evaluate marketing platforms on the basis of how much racial and gender bias is inherent in digital ad platforms? So, not that I know of, mostly because in order to make a determination about the bias of a platform, you need to look at three different things, right? You need to look at the data set that's gone in it, the algorithms that have been chosen to run against that, and ultimately the model that uh these ma these machine platforms use in order to generate their results. And no surprise, the big players like Facebook or Google or whatever have uh little to no interest in sharing their underlying data sets because that literally is the secret sauce. Um, their data is what gives their machine learning models value.

So, what do you do if you are concerned that the platforms that you're dealing with may have bias of some kind in them? Well, first, acknowledge that they absolutely have bias in them because they are trained on human data and humans have biases. For the purposes of this discussion, let's focus on um let's focus on the machine definition of bias, right? Because there's a lot of human definitions. Um the machine or statistical definition is that a bias is if something is calculated in a way that is systematically different than the population being estimated, right?

So if you have a population, for example, that is 50-50 and your data set is 6040 for any statistic, you have a bias, right? It is systematically different than the population you're looking at. Now, there are some biases that that's fine, right? Because they're not uh what are called protected classes. If you happen to cater to, say, people who own Tesla cars, right?

Not everybody in the population has a Tesla car. Um, and so if your database is unusually overweight in that aspect, that's okay. That is a bias, but it is not one that is protected. Um, this actually is a lovely list here of what are considered protected classes, right? We have race, creed, or religion, national origin or ancestry, gender, age, physical and mental disability, veteran status, genetic information, and citizenship.

These are the things that are protected against bias legally in the United States of America. Now, your laws in your country may differ depending on where you are, but these are the ones that are protected in the US. And because companies like Facebook and Google and stuff are predominantly US based, headquartered here, um, and are a lot of their data science teams and such are are located in the United States. There's it these are at the minimum the things that should be protected. Again, your country, your locality, like the EU, for example, may have additional things that are also prohibited.

So, what do we do with this information? How do we determine if we're dealing with some kind of bias? Well, there's some easy tools to get started with, right? Knowing that these are some of the characteristics. Let's take Facebook, for example.

Facebook's audience insights tells us a lot about um who our audience is. So there are some basic characteristics. Let's go ahead and bring up uh this here. This is uh people who are connected to my personal Facebook page and looking at age and gender relationship and education level. Remember that things like uh relationship status and education level are not protected classes, but it still might be good to know that there is a bias, that the the my data set is statistically different than the underlying data.

Right. So here we see, for example, um, in my data set, I have 0% uh males between the ages of 25 and 34, whereas the general population there is going to be like you know 45% or give or take. Um we see that my uh in the 45 to 54 bracket, I am 50% uh of my data of that group there. So there's a definite bias towards men there. There is a bias towards um women in the 35 to 5044 set, there's a bias towards women in the 55 to 64 set.

So you can see in this data that there are differences from the underlying all Facebook population. This tells me that there is a bias in my pages data. Now, is that meaningful? Maybe. Um, is that something that I should be calibrating my marketing on?

No, because again, gender and age are protected classes. Um, and I probably should not be creating content that or doing things that uh potentially could leverage one of these protected classes in a way that is illegal. Um that said, if your product is or service is aimed at a specific demographic, like if I sold, I don't know, um wrenches, right? Statistically, there's probably gonna be more men in general who would be interested in wrenches than women. Not totally, but enough that it would be a difference.

In that case, I'd want to look at the underlying population, see if I could calibrate it against the interests to see if it not the Facebook population as a whole, but the category that I'm in to make sure that I'm behaving in a way that is representative of the population from a data perspective. This data exists. Um is not just Facebook. So this is from I can't remember what iPlum stands for. It's the University of Minnesota.

They ingest population data from the U.S. Census Bureau, current population survey. It's microdata that comes out every month. And one of the things you can do is you can go in and use their little shopping tool to pull out all sorts of age and demographic variables, including industry and what you work, you know, and class of worker. You can use this information, it's anonymized.

Um, so you're not going to violate anyone's uh personally identifiable information, but it's anonymized. And what you would do is you would extract the information from here, it's free, look at your industry and get a sense for things like age and gender and race and marital status, veteran status, the uh disability. And for your industry, get a sense of what is the population. Now, you can and should make an argument that there will be some industries where there is a substantial skew already from the general population, for example, programming skews unusually heavily male. Um, and this is for a variety of reasons we're not going to go into right here now, but we acknowledge that that's a thing.

And so one of the things you have to do when you're evaluating this data and then making decisions on it is is the SKU acceptable and is the SKU protected, right? So, in the case of, for example, marital status, marital status is not a protected class. So, is that something that if your database skews one way or the other, does it matter? Probably not. Um is it material to your business?

Again, where we sell uh, for example, Trust Insight sells marketing insights completely immaterial. Um, so we can just ignore it. If you sell things like, say, wedding bands, marital status might be something you'd want to know. Um, because there's a good chance that some of your customers, you know, not everybody goes and buys new rings all the time. Typically it's uh a purchase happens very very early on in a long-lasting marriage.

Um on the other hand, age, gender, race, etc., those are absolutely protected classes. So you want to see is there a skew in your industry compared to the general population? And then is that skew acceptable? If you are hiring, that skew's not acceptable, right? You cannot hire for a specific race, not allowed.

You cannot hire for a specific age, not allowed. Um so a lot of this understanding will help you calibrate your data. Once you have the data from the CPS survey, you would then take it and look at your first party data in like your CRM software, your marketing automation software, if you have the information, and if you have that information, then you can start to make the analysis. Is my data different than our target population, which is the group we're drawing from? Is that allowed?

And is it materially harmful in some way? Um so that's how I would approach this. It's a big project, and it is a project that is you have to approach very, very carefully and with legal counsel, I would say. Uh, if you are if you suspect that you have uh bias and that that bias may be materially harmful to your audience, you should approach it with the legal counsel so that you protect yourself, you protect your customers, you protect uh the audience you serve, and you make sure that you're doing things the right way. Uh, I am not a lawyer.

So, good question. We could spend a whole lot of time on this, um, but there's there's a lot to unpack here. But this is a good place to start. Start with uh population population survey data, start with the data that these tools give you already, and look for a drift between your population and the population you're sampling from. You have follow-up questions, leave them in the comments box below.

Subscribe to the YouTube channel and the newsletter. I'll talk to you soon. Take care. Want help solving your company's data analytics and digital marketing problems? Visit trustinsights.ai today and let us know how we can help you.