So What? The AI-Ready Data Quality Audit

Well, hey everyone, happy Thursday. Welcome to So What, the Marketing Analytics and Insights Live Show. I am Katie, joined by Chris and John. Howdy, fellas. Well done.

Uh this week, we're talking about the AI ready data quality audit. We're gonna talk about the three levels of a data quality audit. We're gonna talk about the 6C Data Quality Audit Framework, and we're gonna talk about what AI will fail without good quality data, which I believe should be where AI will fail. But a human Katie wrote that, so here we are. So AI did not write that.

Human Katie did write that. Um with all of her poor grammar. So we're talking about data quality, and we've been talking about this for a couple of weeks. Well, let me back up. We always talk about data quality, but we've really kind of focused in on data quality in the past couple of weeks, starting with the LinkedIn algorithm paper that we put out uh that Chris compiled.

Um, was it just a week ago? Maybe a little bit more, like a weekend. Um, and so this was to help understand what was going on with the LinkedIn algorithm. And unsurprisingly, it's you know, at their whim, a bunch of large language models that they can tune up, tune down. And on Monday, not Monday, Monday, we recorded it on Wednesday on our podcast.

I'm getting all my days are just wrong this week. Um, Chris and I talked about data quality to help train your social media algorithm. Uh so you can get that at TrustInsights at AI slash TI podcast. Um, and so because we really want to help ourselves and our audience understand that data quality isn't just okay, I've exported something from Google Analytics. Is it missing anything?

Yes, no, okay, great. That's my data quality. Data quality extends far beyond just the output, it's also the inputs, it's it's what you're putting in, but it's not just into your web tracking system, it's not just into your CRM, it's into your social media, it's into the content that you're producing, it's how you're disseminating content, it's how you're making choices about where things go. So, Chris, where would you like to start this week? Well, we probably should start with perhaps even the the whole point of a data quality audit, right?

The whole point is to make sure that, like you said, what's going in makes sense. If you think of a data quality audit as the equivalent of you know, getting a product, taking out the glasses, and looking at the ingredients, uh buttermilk flavor, powder, buttermilk, natural butter flavor, maltodextrin, granulated garlic, granulated onions, sea salt, sugar, dried dill, and citric acid is a decidilant. So by looking at the ingredients on a product, I know what's in it. I know whether it's safe to use, right? If I have an allergy to one of these things, I need to know that before I go putting this on my food.

Because if I have an allergy to that really long word, uh and I consume it, uh it causes harm. You actually, Katie talked about this recently with uh conversation about data hygiene, right? Hygiene has like three levels. There's there's uh remediation, like you you bad things are happening, you need to get clean. There's preventative, like brushing your teeth so that bad things don't happen, and there's optimization, doing things to advance your health, right?

So that's that's like basic hygiene. Then with data, you have exactly the same thing. You have bad data that's screwing up your decision making. You have so you gotta fix that, right? That's that's the the the baseline level.

Then there's preventative. What can you do to prevent bad data from getting into your systems? And then there's optimization to say what new data or what additional data can we bring in to make our systems better. And so if we think about that data hygiene conversation in the context of a data quality audit, the data quality audit really is the diagnostic that tells you what level of hygiene you have. Well, and and we talked about this in the newsletter this week, that even small things like skipping flossing, for example.

If we're going in the hygiene analogy arena, um, let's say I skip flossing, you know, two of the seven days a week, no big deal, right? I've flossed the other five days. Well, that actually over time becomes compounded so that when I go to the dentist, they're like, hey, you actually now you have a cavity. And I'm like, what are you talking about? I've been brushing every day, I flossed most days.

Like, right, but you're supposed to be flossing every single day. And so really trying to think about it in terms very simple terms. Because when we think of a data quality audit, we're making assumptions that we have this like big wonking data set that has all of these issues and it's not clean and it's like messed up and it's got like strings where there should be numbers. Like it's not even that deep. It could be something very simple, but it's those small tiny things that we think, oh, that's not a big deal.

I'll deal with that later. Or it's just missing one day. That's fine, it's not gonna matter. Over time, those small issues compound to be very large things, and then you're making decisions based on what you think is really good data, but is actually become really poor quality data. And so I would say the data quality audit or the purpose of doing a data quality audit is to make sure that you're doing, you know, making the right decisions.

You may not see big issues, but you may have some things that over time, if not addressed, um, you know, it's like an injury or anything else. It might be okay today, but three months down the line, six months down the line, some of us going to PT 30 years down the line, it's a big deal, and you have a lot of work to do. Yep. So the three levels of a data quality audit is baseline human, which is I'm gonna go into Google Analytics and I'm gonna look at my data. I'm gonna run reports, I'm gonna examine the data along the six dimensions of good quality data to figure out is my data in good shape.

And that is something that everybody can and should do. Right. That's that's level one. Level two is to use automations to uh to to process the data and do things like spot anomalies, right? And so this is what I would call classical machine learning, where you're looking for, hey, what happened there?

Um, we see this, for example, in our CRM data. A lot, there's uh some forms just attract garbage submissions. And you know, you can always tell because it's you know it's NIC six and then a long string of numbers, and then some random you know domain on the other end, like, okay, clearly that's 280 of those is uh in a row, is is pretty clearly not good data. And then at the highest level is almost what you would call agentic data quality, right? Where you have AI, classical and generative, operating on your data in real time to alert you, like, hey, you're getting more bad data than you thought today.

Uh, and you already have experiences with some of these tools. Uh you should, for example, with Google Analytics, inside Google Analytics are the is the alerts facility. If you do not have an alert set up inside your Google Analytics analytics that does something as simple as, hey, I noticed you stopped sending data. Would you like it? Are you aware of that that you're your system is no longer sending data?

That's one of those things that you should know, right? And those alerts are part and parcel of the system. Well, you can do obviously a lot more than that now with generative AI, but those would be some of the very basic things that an agent would do to to increase the data quality. And if you think about it, that kind of maps to the levels of hygiene where you have we just got to get caught up. We gotta get that get the house in order to get caught up.

Now, okay, let's talk about preventing things, um, using automations to to clean data on the way in, perhaps, and then ultimately doing that proactive let's let's make everything better and know well in advance that something's going wrong. So before we get too deep into the AI effectation of data quality, John, as the person who I would argue owns our CRM, how much do you think about the data hygiene going in? Like, so you do a lot of the input into the system along with you know the automation that comes in. You know, how much do you think you think about the data hygiene and the process? And you can be honest, this is a safe space.

I'm not gonna hold it against you during the course of the live stream. We'll talk offline. I'm just kidding. No, it's a things have changed a lot so much. It used to be a critical thing that you know, in other organizations, I'd have to do it monthly because there were all these dependencies that had to be perfect below it, you know, like all the addresses had to be right because when you dump them to mail a catalog or to you know do other some kind of direct mail that you know your data had to be clean, or if you're you know basically doing anything else, email campaigns, whatever, anything where you're pulling the data out, it had to you know be in perfect alignment all the time.

And it's it's total the flossing analogy is perfect that you don't realize that just you know, like one out of seven, if you're missing two days, that's a huge percentage of the overall time, and these things compound, you know, it's like once they start going off the rails, it gets worse and worse. But there's so much going on now with AI that look because we've talked about like for our source codes, and for you know, it used to be like you would want to have a pick list so the source codes were straight, but now you can just grab raw text and have AI summarize it for you, and that actually even does a better job than the human pick lists and and you know can give you can find uh relationships that you didn't even realize existed and and a whole bunch of other stuff. So um, yeah, it's a you know, I'm not as worried as I was about data quality. Like things are are getting better and easier, but I'm completely interested in digging in here and finding out what kind of stuff you know can do a better job of cleaning it up and yeah, ultimately, you know, setting up at the front door so that you can stop the problems before they even start. You know, the pred preventative stuff is the highest value.

But um, yeah, interested to dig into the tool set. Yep. So the first place we should start is what data quality is, because you can't do an audit if you don't know what you're auditing. Um, this is the 60 data quality framework, which you can get on the trust insights website, go to trusted insights.ai, and it's under the insights section. The six aspects of data quality are clean, complete, comprehensive, calculable, chosen, and credible.

And each of these is a dimension of data quality. So clean data. You think like all, you know, that's the most important thing, although they're actually all important. Um, clean data is data that is prepared well, free of errors. This includes things like formatting errors, incorrect calculations, malformed text, text in the wrong language, right?

Uh, text in the wrong encoding. It's the the data itself is just not clean. So that's number one. Number two is complete. Is your data complete?

This is a data has no missing information, no field. If you it's a table. There's no cells within the table that are missing stuff that it shouldn't look like Swiss cheese. Um it's fully available based on the format, complete tables. Data is not missing at random, data is not missing, not at random.

Um thing that's really tricky about complete, it's what it's the hardest of the six C's to deal with because you may not know that something's missing if it's not there. For example, this lovely device, the iPhone, is one of the platforms that blocks a lot of trackers like Google Analytics. iPhone users are underrepresented in your data. When you look in your Google Analytics data, it won't show oh iPhone, I you know that was an iPhone, I have no idea. It's just not there.

And because it's not there and you don't know it's not there, you don't know that your data is incomplete. That's number two. Number three is comprehensive. Does the data answer the question that's being asked? Is it scoped correctly?

For example, if I had you know a food uh uh food information of some kind. If my question is, what is the calorie count on this food? And there's no calories, not just the list of ingredients, it doesn't answer the question. I all the other data on here, the ingredients are on here, but there's no calorie information, then it it is not comprehensive, it doesn't answer the question being asked. Uh calculable is the most important for AI.

Calculable means formatted in a structure that both machines and humans can use. And this is especially important for generative AI because generative AI has some languages that reads well and some languages it reads really poorly. CSV files, TSV files. Generative AI struggles with that. Markdown or JSON, generative AI can read that data much more easily.

So a big part of calculable is is the data if you're gonna be using AI on it, all the data needs to be converted to the appropriate format first before you can use with AI. The fifth is chosen. No irrelevant or confusing data. Data that's chosen well is comprehensive enough to answer the question, but then doesn't have distracting or irrelevant stuff. So if you're saying, I want to know the ROI of our LinkedIn paper, and you're handing me Twitter data, that might not be relevant, right?

That that might not be super helpful, or you're handing me pay-per-click advertising data, and we didn't run ads. One of the things that all of us, but marketers in particular have a tendency to do is when they're under stress or strain, they just tend to back the truck up and pour data everywhere and say, here, I'm giving you everything. Like, no. Statistically representative samples, data that's appropriately weighted if you're doing weighted averages and things, data that is de biased. It also means, and this is a really hard one, data that is sourced from credible sources of good quality data.

So, for example, say you were working with uh a data set, and the the entity that produces that data set suddenly had somebody come in and say, you know what? I want the data to say this. That data is no longer credible. Um my wife used to work at a survey company way back in the 2000s. They were they've been out of business for like 20 years now.

Um you'll find out why very shortly. Um, people would call them up and say, I need a survey that says this, so that I can go and run an ad saying surveys show that you know nine out of 10 people prefer the color blue on uh on their toothbrushes. And they would go out and conduct a survey to get that answer, which is the opposite of good survey practice. So credible is that. So that's the six C framework.

And you know, so you're talking about in terms of using AI, but I mean, it's a good gut check just for humans as well. And John, sort of, this is what you were talking about in terms of the data going into the, you know, the precursor to it going into the CRM. You know, if you think about, you know, things like chosen, um, it immediately makes me think of all of those dashboards that we've either created or seen where it's just everything and you don't know what to pay attention to, but it's like, well, if I put everything on there, then maybe somewhere along the lines that'll tell a story. Or, you know, credible. I think the survey example is a really good example.

We've, you know, probably all run into that of, you know, I need to say this about our product. Well, you can't go in telling me what the answer is gonna be. That's not how surveys work. Um, you know, and so a lot of this is just a really good, if nothing else, checklist for the humans before you even get to the machines. Exactly.

And the machines, the the the age old rule that's been true for what, 60 years now, is still true. Garbage in, garbage out. If you give AI bad data, it's not gonna magically make it better. Uh, if anything, it's gonna make it worse because that's what AI does. AI takes the good and makes it better, takes the bad and makes it worse, and and that's what's gonna happen.

So these are the six criteria that any data you want to audit, you have to go through each of these six criteria. So let's do a uh super simple example. Let's go into our Google Analytics and let's ask ourselves a question. So I'll go to the explore tab. Let's c start an exploration here, and let's say I want to know about the LinkedIn paper.

That seems like a the kind of thing that I might want to know about. So I'm gonna say, let's do the LinkedIn landing page, and the landing page contains uh linked in algorithm guide. And I just I all I really want to know is how is that doing. I get a uh some nice charts here. Now, the first step in the process is is the data clean?

And the answer here is yes. It's it's you know, it if I were to export this as in uh in a useful data format like CSV, um, the data is pretty clean. It's it's you have dates and you have numbers. Katie, you look flummoxed. Because I don't know what this chart is telling me.

To me, it's not calculable or comprehensive. Like what is so you have this, you know, uh you have this column of totals, and then you have set like I don't know what this is. To me, this is not good data quality. I don't know what it is. If we look at the export, it doesn't involve Google's very strange labeling, is the date and the number of sessions.

That's all the data is. But I think you raise a very good point. So for calculable for humans, the Google interface, not so useful because you're you were clearly struggling to figure out what the heck does this thing even say. I would not have guessed. If you go back to the Google Analytics screen, I would not have guessed that that was the date.

It might be that from my view it's a little bit small, and I can't see like slashes. I don't think those are even in there. But my first instinct was not, oh, those must be dates. Because it says totals. It does.

Because that's there's the there's the column header for date up here, and then totals it refers to this row because that's the way Google lays things out. And yeah, so I would like I would give that a D for uh calculable. Right. Which again, super important, right? So is the data clean?

When we if we were to look at this data here, is this clean. How do you know? What is what's you how do what are you looking at it against? So it looks clean in terms of you know, to the naked eye, to the KDI, it's rows and columns, and everything seems to be filled out. So now that I know it's a date, those are dates.

Now that I know those are sessions, those are sessions. Like it looks to be clean. I'm not seeing wingdings, I'm not seeing strings where there should be numbers. So to the naked eye, it looks clean. It is not.

Google adds five rows of garbage comments up top and then puts a total row in this. So if you wanted to use this and you just loaded this in as is, it wouldn't work. Um, you it would immediately go off the rail. So it you're uh one of things you'd have to do is delete the total row, which is an annoying that you have to you have to do that, um, and delete all their commentary uh to get rid of all that stuff too, so that you finally have an actual rectangular table. So automatically clean, no complete, yeah.

The data's there. Is it comprehensive? Uh does it answer the question being asked? Well, the question we're asking is what is the ROI of the LinkedIn paper? ROI is earn minus spent divided by spent.

How much did you earn from something minus what you spent on it, divide by spent? This data tells us none of that. It's just the number of sessions to that landing page. And Google Analytics won't have that data at all. Our CRM won't even have that data because we don't necessarily have a one-to-one of hey, this person visited this page and eventually became a closed one deal in nine months or what have you, because B2B takes a long time.

So we can't even answer this question, which should immediately stop the data quality audit here. If we're saying, like, you know what, we can't answer this question. And but yet, and yet it doesn't. What ends up happening is a couple of things. One, if your question is what is the ROI of the LinkedIn paper, and this data doesn't answer it, people will fudge the numbers.

People will go find other data sources, like, oh, well, let me go find revenue data and try to mush it together, and it doesn't have a correlation. Or and or they'll be like, Well, that wasn't the question I was trying to answer anyway. I just wanted to know about the awareness, and it does answer that question. Does it? I don't know.

But it's just like they're gonna make up the numbers, they're gonna mash numbers together, and they're gonna change their question because they're gonna change their purpose to fit the data versus finding data to answer the question. And the reality is this and the ugly this is the ugly reality in marketing, you may not have the data. Like we know iPhones block a lot of things from click streams. We know certain browsers just flat out block things. We know that about 40% of your data inside of Google Analytics is inferred.

So it's it's it's literally Google guessing based on usage patterns at the user level. So there's all these things that we know that could be wrong with this data, and that goes back to complete. We we know the data is incomplete, we don't know how incomplete it is. And and the only way to determine that is to look at companion data sets. So your next stop would be well, i if I don't have this information, where could I go get it?

How what would be the the the way to look at whether that data even exists? So the the good and bad news is that there are other data sources that are available to you, but you gotta go get them, and then you have to figure out are they clean? And the answer to that is also maybe uh possibly no. So for example, let's say at a very high level, we go to our Google Analytics, and Katie, you say, you know what, forget all this other crazy stuff. Just tell me how many people visited our website.

We can do that, clearly, right? And we're breaking my heart. The answer is no not really um and there's a and there's a reason for it there's a there's a good reason for that and the good reason for that is we so much of the data is excluded because it may not it just may not be there um so let's take a look let's go to our basic reports in GA4 let's look at the last 30 days make this as easy as possible traffic acquisition last 30 days that's just look at overview google says we got twenty seven thousand active users on our website right so that that's that's how many people we got how does that sound to you that sounds really high what if I told you in the last 30 days according to our web hosting company that bills us um on visits we actually had 158000 huh that also sounds really high a hundred and fifty eight thousand visits made it to our website and this is and and there's an there's an additional wrinkle if you want to get really really uh crazy about it what would it look like if you counted everything, including like all the bots and scrapers and all all that stuff, uh, what would your traffic look like then? One million visitors. But I like I don't even know where this is going because it's just those numbers just don't feel realistic based on historically because so we track the data on a monthly basis, have been for years, and when I look at traffic to the website getting into the five digits a month, which for some people that's no big deal, but for us, that's a pretty big deal.

Like we have a very small website, and that's not where people get information about us. They tend to get it from other places. So that to me feels off, and I don't have anything to prove it other than my instinct and historical data. So if we look at our CDN, Cloudflare, the Trust Insights website in last days has 147,000 unique visitors, 4.4 million requests to the website, of which it it has serves up you know a lot cash. That just on the unique visitors is almost 5x, what Google Analytics is seeing, which means one of two things, and this this is the heart of data quality audits right here.

This is why they are so painful. This is why they a lot of people choose to have someone do it for them. You gotta figure out what the actual answer is. If you're B2C and you're selling, you know, packs of chewing gum, this is super easy because you can say just how many packs of gum do I sell on any given day? And then I can do a straight up regression analysis and say which of all these numbers correlates to packs of gum.

We can't do that because we don't sell a data quality audit a day or a 10 a day or whatever. That's just not the way our business works. So what we'd want to do is find some slightly lower number somewhere in the funnel, and that could be contacts established in the CRM as an example on a day-to-day basis, and then say all that, because that number is actually, you know, contacts in the CRM is at least something John can work with. Based on that, which of these three numbers, Google Analytics, uh WP Engine, Cloudflare, which one of those correlates the most correctly to that number. That's how you would get to this very, very thorny issue, which again is the heart of data quality audits.

And it it's for someone like me who was trained more classically, that you don't guess at data, you don't look for the closest directional data set, like the data is or isn't accurate, period. This is such a hard thing, even this many years in, over a decade outside of the academic world into this world, I still struggle with that. I'm still not totally okay with all right. Well, it's the it's the closest to right, we think. Like that destroys me.

I have and I know I'm not alone in that, but that's where we get caught up trying to make the data perfect. And it's just not a reality. It's the more data sets you have to work with, the more systems you have, the more tech in your stack, the more likely these scenarios are to happen, even if you have a person 100% dedicated to setting these up, maintaining them, data in, data out, you know, John, your sole focus is business development and sales. You can't stay on top of the data that goes into the CRM 24-7 because a lot of it's out of your control, despite how we set it up. And that's just the reality.

And I think that's where people are struggling. Like, what do you mean I can't do individual level attribution analysis? What do you mean I can't do attribution analysis at all? What do you mean a company as small as Trust Insights can't tell who downloaded the LinkedIn paper and then converted into a customer? Like it's it's hard to wrap our brains around that you just can't do that because the data just it either doesn't exist or it's just poor data quality.

I this is second week in a row that I hate this episode. So, how would we go about doing that? That's I think that's probably the most important next question is how would we go about figuring out uh what corresponds to reality here? Like, is is what's in Google Analytics even real? Um how would you go about doing that?

Oh I mean, one of the steps that we skipped, quite honestly, is uh let me find it. Where is it's in here somewhere? Is the five piece? We didn't purposely skip this step, but that's where I would start is okay. Let me just slow down for a second and say, what is my purpose?

What is the question I'm actually trying to answer? You know, is it I want to know the ROI of the LinkedIn paper? Is it I want to know how much traffic was driven to our website by the LinkedIn paper? And not just like those are like good questions, but what's my why? Why do I care about those things?

Because that's gonna help me with the rest of the P's of okay, who has that data? Or, you know, where in this channel? So we're talking about people, where are those people coming from? How did they get the paper in the first place? And then who's actually gonna help us get the data out and do something with what is the process?

How are we extracting the data? How are we thinking about the data? How are we answering those questions? What are the platforms? This I think is getting to the heart of what you're asking, Chris, which is okay, so we're saying Google Analytics is going to tell us the answer, but that may not be true.

We may be looking at other platforms in order to understand the question being asked. So Google Analytics might be the wrong platform, but first we have to be clear on the purpose, and then our performance is did we answer the question in a satisfactory way? So, did I understand if the LinkedIn paper drove traffic to our website so that we can do more of it, we can do less of it, we can do more thought leadership things like that. You know, we have to have a reason for doing it. So I guess that's to it in a long-winded way of answering your question.

That's how I would start to think about if this isn't telling me what I need to know. Let me back up and check to make sure I'm even thinking about it the right way. And I think that's that is uh the the most sensible approach is to is to question the purpose itself. Now, in the interest of helping folks, how would they do actually do even something as a simple question like that? Let's go through an example using Google's Colab.

So if you're unfamiliar, Colab is a data science tool. Uh it is powered by Google's Gemini, uh, which is of course Google's large language model. And unlike uh Colab started out as a pure coding environment. It was intended for nerds uh to be able to just you know run Python and other code remotely. Nowadays, it's sort of seen this revival as a generative AI coding environment.

And that's super useful because we want to be we want to be able to use it and understand uh what's going on, but we may not necessarily know the coding language. The good news is now we don't have to. So let's do this. I'm gonna take what I've done is I've taken those those two files. I've taken two files from our website.

One is from the form fills on the actual website, I've exported the data out uh of the uh the exact form fills. And the second is I've gotten from Google Analytics and exported the number of thank you page visits um from the website. Those two numbers should be identical. Right. So the number of people who filled out the form should be identical to the number of people who got to the thank you page.

So the question we want to ask Google uh Google Colab is is a very straightforward one, which is we want to do a correlation analysis. We want to figure out is this these two data sets, are they are they well correlated? I've provided two data sets here. One is Google Analytics form completion data of the number of people per day that have filled out and completed a form on a website. I've also provided a date time stamped log of every individual entry in the downloads.csv file of those same form fills.

You're going to need to aggregate the downloads.csv file to day level counts first, and then perform a spearman correlation to identify how closely correlated these two data sets are, and then provide a week level summary of how much drift or variance there is from one data set to the other. So we've given it this nice, very long prompt. We're gonna hit go on it, and it's gonna come up with a work plan, and then from that work plan, it's going to start writing the actual Python code. Now, the nice thing is for you and I, uh, and for anyone who is not a particularly good Python coder, it's going to write the code, it's going to test the code, it's going to tell it's going to fix its own errors in the code, and then ultimately come up, hopefully come up with an answer to let us know of these two data sets, which one how reliable is Google Analytics? Because that's the question that I really want to know is we have WP Engine, we have Cloud Slayer, we have GA4, and we're like, none of these numbers make sense.

We have a known good number because you got to the form and you filled it out, and I have the form 8 data. So now what's gonna happen? Right. Um, John, I don't know about you, but I would say 99% of the time I forget that Chris can do uh speech to text in these prompts. And so he starts talking, like, wait, is he talking to us or is he talking to the machine?

Right, is it a driving kit to get him to get things done as we're writing here? It's only about like halfway through his statement. Then I'm like, oh no, wait, he's prompted. Okay. Don't interrupt.

Look at that. What? A 0.65 Spearman correlation. That is awful. So Spearman correlations go zero to one.

One is perfect correlation, zero is no correlation at all. That should be a one, not a.65. Where because I we are seeing smaller screens, where are you seeing that? Right here. Got it.

That should not be a.65. Like that is that and for folks who uh like me didn't do well in stats back in university. There's three different forms of correlation. There is uh Pearson, which is for parametric normal distributions, there's Spearman for non parametric distributions, and then there's Kendall Town. I can never remember what the heck you're even supposed to use KendallTal for.

Um and causation. And Spearman is generally the best for marketing data because marketing data more often than not is not a binomial distribution. It's not a bell curve. Marketing data more often than not is a power law curve, right? You know, 8020 will Pareto curves.

And so Spearman will can work with that, typically works with that better than Pearson. So we have here sessions, downloads, the drift. So it looks like Google Analytics is really messed up in terms of the of our data. Um it is it it's a moderately strong positive correlation, but it should be perfect. It should be a perfect correlation.

Instead, there's a negative drift. Google Analytics just isn't getting the data. Now we know that we're Google Analytics blocks is blocked by a lot of trackers, right? By iOS and stuff. I didn't know it was this bad.

Um this is bad enough to make me go, I don't know that we should even trust Google Analytics because it's again, it should be a perfect, should be one to one. You download, and Google says, hey, you downloaded. And believe it or not, this is after installing server-side tracking. So we installed Google Tag Manager's uh server-side tracking two months ago now, I think. Um it should have fixed this, and it clearly did not.

Well, and so this is where it starts to get tricky because someone will see this and then you know, lose their mind trying to fix the system when the system is unfixable. And I think that's just sort of a disclaimer of, you know, you really need to know the system enough to know like what is in and out of your control. You know, so Chris, you said we installed server-side tracking, it should have fixed it. So we can at least go back and see like, did we set it up incorrectly? Is it this?

Is it that? Whatever. But at some point, the system is out of your control. Mm-hmm. And the system is very clearly out of control in general.

Yeah. Um, now here's here's the thing to keep in mind for at least for us in this example. The closer you get to the bottom of the funnel, the more control over the data you have, right? Because you can't accidentally, you know, go have data, form-filled data go missing. If someone's filled out the form, and as long as your systems are working correctly, right, and you haven't blown up your website, that data is there.

Someone who is in your CRM that you're emailing back and forth, unless you're on drugs, you're not imagining those people, right? They are actually in your CRM and you're actually having conversations with them. So the closer you are to the bottom of the funnel, the more solid the is the higher up you go into things like website traffic or heaven forbid social media traffic, the more you may as well just go by vibes at that point because the data is so incomplete, it is so not comprehensive, it is so not well chosen, it's not credible that there's almost no point in trying to audit that level of data quality. So that goes back to that because is the fifth P in the five piece, which is performance. Is the data quality audit even worth doing?

If you know the higher up you go, the worse the audit's gonna be because you just don't have control of those systems and the systems are broken. Well, I would argue that it is worth doing to at least see so that you could have that data point to say this is why we should not be looking at this data. So one of the things that we are offering is the AI ready data quality audit, which if you want to learn more about it, uh you can go to trust insights.ai/slash AI ready data quality. And so what we do in there is we actually take a combination of the five Ps. So we try to outline first, why are you looking at this data?

You know, who is it for? What's so on so forth? We go through the five Ps. And then we go through the six C's to say, all right, here's all the questions we have about this data based on the five Ps. Let's just go ahead and check, you know, the quality of the data in general.

Is it usable? Should we even be looking at it? And you'll get a score out of 60. Um, in the example that I often talk through in engagements, is you know, as an SEO data set. And in the SEO data set, it's from something like Hrefs, but I don't state that, so it doesn't come up as credible.

You know, I don't give her, you know, the KPIs that we'd be looking for, those aren't in there. And so it comes back with a score of 15 out of 60, which is a really low score. And what you get back is uh, Chris, if you have Colab, if you can pull that up real quick, so it's not the exact data quality audit that Trust Insights would do for you, but it's a good example. If you scroll to the bottom, um, you can see you have the data analysis key findings, but also your insights and next steps. And so what you would get is basically your roadmap for okay, if you really do need to use Google Analytics data, here are some of the things you need to do to fix this data set in order to make it usable, which really sort of stems into the system as a whole, but at least it gives you some direction of where to start.

And so each of the six C's is scored on a scale of one to 10, you can see which ones are the most broken, which versus which ones are the lower priority. So if you're interested in learning more about auditing your data quality, you go to trust insights.ai slash AI ready data quality. You'll get a hold of John, our chief statistician, and he'll walk you through it. But basically, before you make big decisions with your data, before you put your data into something like a large language model, you probably want to do a gut check to say, is this even good data? So we went into this episode thinking, all right, our Google Analytics is in pretty good shape, right?

Like we know how to set it up. It's actually one of the foundations of how we started Trust Insights, which was helping people set up their Google Analytics. I would say, Chris, you are head and shoulders above a lot of people when it comes to understanding how the Google Analytics, Google Tag Manager, the Google ecosystem works. And yet we're still seeing now that the data quality that coming out of Google Analytics is pretty poor. And that says there's there a lot in the system that we don't have control over.

But unless we had done that audit, we wouldn't have known. We would have carried on saying, oh yeah, this is telling us we feel confident in how it's set up. It's not a knock against us the humans. It's stuff that's out of our control. Yeah, it's systems we don't have control over.

I would also suggest that for any given measure that you uh that is something that you get a bonus for, um, you probably should have some ideally alternate way of measuring it, right? So perhaps you have Google Analytics installed alongside Matomo, which is uh, or server log analysis or anything. If you are paid on, you know, leads generated, obviously your CRM is going to have some of that data, or should have that data. Um, but is there an additional system that runs in parallel that you can cross-check and validate? The reality is is that as these devices get smarter and as AI gets more and more of the in the way of us being able to function as marketers uh or as business folks, we have to work on building trust in our audience so that they give us the data voluntarily and then rely on the things that we know we are given, um, by things like form fills, people joining Slack communities or newsletter lists and things like that.

Relying on data that is provided by a third-party system that we don't own and control, you're at the whim of that system. And there, as many people have found out from the LinkedIn paper, they can change the engine at any time under the hood. And you're just like, why does the car feel different now? Why am I not going in the same direction? It's because it ain't your car.

So, John, you mentioned you were interested to see the tools and techniques. Anything, any revelations, anything you're gonna change, or just more questions? Yeah, no, a whole lot more questions. It's much more horrible than I had initially hoped. Um question I did want to ask though is one thing that has been constant when I've seen how marketing people do, you know, running reports and answering questions.

It's like there's always a black box in the system somewhere. And so the fallback is always relative reporting. Like even for Google Analytics, even okay, so you know you're off by this huge margin, still, you could still look at the past three months and be like, well, we were 4x what we were five weeks ago. And so that at least tells you something. It's it's kind of the classic uh, you know, one eyed king in the land of the blind.

Is that still relevant now, though, or are we coming to a point where the data is just so much that like even measurements like that don't help? So, yeah, that that is what our friend Tom Webster calls predictably wrong versus unpredictably wrong. Systems that are predictably wrong, like is like a car that always steers to the right. Like, okay, you know, you always have to tug a little more, but because it's predictably wrong, you can compensate, you can adjust for it. When it's unpredictably wrong, and say, hey, the car's veering to the left, and the car just doesn't operate the way you think it should, period, it's a lot more dangerous.

And the same thing is true of analytics. Now, with Google Analytics, it depends on what's going on under the hood. Um it has gotten progressively worse over time as it uses more of its own AI to try and make up for missing data, and you don't have control over how it does that imputation. Um with something like LinkedIn. When LinkedIn changes, the second pass ranker pulls the whole thing out, puts 360 brew in in its place.

Now all the data prior to that point is worthless because you've got a whole new engine that's is completely different. And so it to Katie's point, you have to know what the system is doing to even know whether it's predictably wrong or unpredictably wrong. And the reality is if it's a system that's not under your control, it's probably unpredictably wrong. I think the other side of that, John, that I would add is it's all about expectation setting. And so if your expectation is that I'm gonna get a crystal clear, completely accurate number, then that's what you have to chase.

But if you're like, I'm gonna use this data directionally to your question of like, are we at least moving in the right direction? So we looked at our Google Analytics website visits, and it was 27,000 sessions. That's way higher than it normally is for a variety of reasons, but directionally it tells me it's not zero. People aren't not coming to the website. People are coming to the website.

What for? We didn't even get into that. But there, but the website is doing something, so I can at least directionally know things are working, or we're getting spammed by bots every other day. It's one of the two. Exactly.

So uh this concludes the another week of disappointing Katie. Thanks for tuning in, and we will see you all on the next one. Thanks for watching today. Be sure to subscribe to our show wherever you're watching it for more resources and to learn more. Check out the Trust Insights Podcast at TrustInsights.ai/slash TI podcast at our weekly email newsletter at trustInsights.ai slash newsletter.

Got questions about what you saw in today's episode? Join our free Analytics for markers Slack group at trustInsights.ai slash analytics for marketers. See you next time.