So What? Setting Up AI Bot Tracking in Google Analytics

Well, hey everyone, happy Thursday. Welcome to So What the Marketing Analytics and Insights Live Show. Joined by Chris and John. Howdy, fellas. Hello.

There we go. Uh this week we are covering setting up AI bot tracking in Google Analytics, which is a continuation of what we talked about on last week's live stream, which you can catch that at TrustInsights.ai slash YouTube. Go to the So What Live Stream playlist. And we talked about how to measure how to, you know, and so it Chris was showing four parts of a measurement plan for looking at uh your AI bot tracking. And one of those pieces was specifically Google Analytics, which there's a lot more uh complexity in what people can and can't do with these things.

If you want to learn more, you can go to TrustInsights.ai slash AI bot analytics. That's AI bot with a T analytics. But today we're actually gonna walk through at a high level, a little bit of uh pre-baked stuff, because uh as Chris mentioned, this isn't something that you can just decide to do in if, like, oh, I have a spare hour. Like it's gonna take a little bit more than that. There's a lot of uh considerations, but we are gonna walk through it and we're gonna use my personal site, which is really just a test bed for trust insights.

Um, Chris, what else do we need to know before we get started? Before you get started, depending on the site size of your site and how much AI bots visit it, this may cost money. Um Cloudflare Workers is the underlying mechanism that this uses, and that is fancy for you're setting up a server on somebody else's architecture and you get a hundred thousand hits a day for free, and then you have to pay. Um that sounds like a lot. It's not a lot because AI bots can be 20x, 100x the number of humans that visit your site based on how much content you have.

Um, so that is that's that's our big caution here is that uh if you are on the free plan, uh pay very careful attention to your notifications and then expect at some point to get a bill. Well, and uh to be fully transparent, this happened to us at trustinsights.ai, our website, which is a decent sized website, but in terms of traffic, like we sort of thought, like, oh, you know, the bot traffic, it must not be that much, a hundred thousand hits. Like, can you sort of walk through to high level what we experienced this week? Yeah, we uh we hit those limits in like seven hours. But then as the result of that, the result is that we got a temporary notification on on Cloudflare saying, hey, your site has been rate limited.

Uh please insert credit card to continue. I'm like, oh well, crap, I guess we're we're we're turning this feature off. Mm-hmm. And I think that it's just important to note that, you know, it might feel like you know, a hundred thousand is a lot, but we hit it in seven hours, as Chris mentioned. And so it is something just in terms of your setup to be aware of, especially if you have it set to automatically roll over to payments.

Yeah, which again, that gets expensive. Yeah. All right. So there's three steps to this process. And the three steps are first we have to configure Google Analytics, then you have to configure uh the Cloudflare worker, and then you have to configure your reporting.

And uh as Katie mentioned, we will help you do this if you would like to hire us to do it. Uh but uh at a high level. The first um thing you have to do is you have to create a new property because by default, Google Analytics does not allow bots. Like they try to screen them out because they want to let they want to know like who are the humans uh that are visiting your site. In fact, uh if you're still using sessions as a metric in Google Analytics, stop uh because some it does allow some measurement bots to get through.

Uh you want to use engaged sessions for regular Google Analytics. But that's that's a different show. So for this, we're gonna set up a specific property that is just for bots, and we can't use the regular Google Analytics tracking code because it blocks them out. So we're gonna call this uh Katie's AI bot friends. That's uh the uh the account we're gonna go with.

Uh we're gonna go through and set it up. Uh this is a one uh let's call this a business and industrial site, one person site. And our goal is to understand web traffic and look at retention. So this is this is stock setting up a property. Next, we have to put uh create a data stream.

Uh this is Katie Robert.com, and this is Katie's site. We're hit create and continue. And now we hit next. Now it's gonna say, hey, do you want to set up a Google tag? Absolutely not.

We do not want to do this. Uh we want to, because we don't want to mix the human traffic from Katie's site in with the bot account. We have to see just the bot account. So we're gonna take note of the measurement ID. I'm gonna have a little notepad uh separately, and then we have to create the measurement.

Go away. Uh we have to create the integration to the measurement protocol. I'm gonna drag this off screen because we don't want uh that to be visible here. And while I do this, you and John should tell some funny stories. Well, and John, I think, you know, I mean, when we were getting set up for this when we were talking about it, we knew that there was gonna be some time that Chris was gonna have to be setting things up.

Like there's only so much that he could have done ahead of time, and I think it's a really good um demonstration of it's not just setting up an additional Google Analytics property, like there's a lot of additional steps that go into it. Um, and so like we were joking that you and I should have our shtick and have some stories to tell. But you know, in terms of you know what you're seeing, you know, because you you see from a different lens what's going on in the market, what's going on with AI? Like, what's your perception of you know, people struggling with bot versus real traffic? Oh, well, the first one right out of the gate is Google pulling an adobe here by wanting to survey Chris on how much the product is working for him while he's in the middle of doing something really important.

I could just go on for 25 minutes about it. And it always seems like those things pop. Like here we're on a webinar. Like that could have fired this morning when he was just routing around through GA, but no, it had to happen like right now. So there's that.

Um, just you know, nice one, Adobe. Uh, even though it's a Google problem, I'm throwing Adobe under the bus for that. Fair enough. Because they're the champs of it. Every time I fire up Acrobat or Photoshop, I get hit with that thing.

Um, but as far as the bot traffic, uh, yeah, this is I mean, this is a big ugly mess. I know you and I are both on the same page as far as Chris is like, hey, I'm just turning it on, it's easy, we're getting this rolling. And we're like, uh no, you're like going through 65 screens that have all changed in the past two years. Like we haven't seen any of these alleyways in GA. So um, yeah, I'm just like to be getting a new tour as far as the latest and greatest with this stuff.

I think in terms of, you know, if I were being held accountable for website traffic, you know, it's a I mean, it's a big conversation that you know a lot of marketers are having right now with leadership. Is it's a struggle because it looks like, you know, it could look like a lot of things. It could look like your traffic is up, it could look like your traffic is down. And the thing that I keep hearing is oh, it must be AI. But that is such a vague statement and such a blanket blame that being able to do something like this is helpful to understand really what that means.

Yep. So we also have to create the dimensions because bots are not people, they don't have things like devices, they don't have things like browsers. Um, so Cloudflare provides nine different categories of data in addition to events. They tell it the bot behavior, what kind of uh bot it is the category of behavior that it is. Is it a remember from last week's show?

Is it a crawler? Is it a trainer? Is it a search or is this is it as an assistant? Um they have for if you are on the enterprise plan, you'll also get a bot score, which is a confidence score. If you're not on, you don't get that.

The IPASN is an identifier, it's a numeric identifier. Cloudflare has a catalog of like 100,000 different bots, and each bot has its own unique identifier, which for our reporting we don't care about. But if you wanted to do like some deep analysis in BigQuery, you could absolutely do that. There's two fingerprints, which are encryption fingerprints that basically tell you sort of like it's like an MD5 hash of the payload without having to look at it. We don't pick care about that, we don't need it.

Um, request path. That is one of the most important fields. That is what URL did the bot hit. Um user agent, which is who is the bot? Is it is it GPT bot?

Is it Claude, etc.? And then is it a verified bot? That's another Cloudflare control field. So those are the custom definitions. The last thing we need to do is what we should do is we should create a big query link to link it to a BigQuery database because this is all um big query uh stores data for Google Analytics, and because what's coming in from bots is event level, Google Analytics, based depending on your settings for your property, will only store it for between two and 14 months.

So if you want to do like year over year analysis after the first year and you don't have the setup, that data is gone. So now Google Analytics is is set up and ready to go. All right. Well, and so a little bit of context in case you missed the live stream last week, um, which again, you can get at TrustInsit start AI slash YouTube, go to the playlist. So when we talk about bot traffic, and it's sort of we sort of tongue in cheek called it Katie's AI bot friends.

Um, you know, I think that we determined on last week's episode that we're talking about these agentic AI assistance. So when you're when you're saying, is it Chat GPT? Is it Gemini? Is it Claude? What's happening is I, the human, am typing instructions to Claude or Gemini saying, I want to find out more about trust insights, and it's saying, okay, cool, let me send out my team.

I'm gonna go find that information and come back to you. And so instead of getting Katie Robert hit trust insights, we're getting Katie we're not getting me specifically. I'm now anonymized. We're getting someone asked Gemini to go find this information. So you're getting that uh that's the kind of data that we're trying to figure out.

That's right. You are getting the intermediaries, which is an increasingly large percentage of every search, thanks to Google's AI mode, uh AI overviews, etc. So this is believe it or not, this was the easy part. Now comes the extremely painful hard part. Uh, you need to have a Cloudflare account.

Uh, it can be a free one, but you need to have a Cloudflare account, and you need to have uh set up uh an account so that you have permissions to talk to it to be able to make changes to it. So Katie's got a Cloudflare account already set up. We set it up when we set up her website. That is a prerequisite. So that we're gonna skip by past all of that.

To configure a server, because that's what we're doing. We have to build the server in Cloudflare. And the best way to do that is to not use the Cloudflare interface at all, which seems counterintuitive, but here we are. Instead, we're going to go to the command line and we're going to create a Cloudflare local server configuration. We're going to call this KatieBotTelemetry.

And this is going to go to Cloudflare's node package. And it's going to ask me some questions like what are we doing? I want to do a hello world example. We're going to use a this, I believe it is worker only, because we're only making a worker server here, which is a little server on their services. What language are we going to use?

We're going to use JavaScript because that's what Google Analytics speaks very well. And it's going to now create an entire setup, a system setup here with source code files. I'm going to put the index file in here. And now comes the uh next difficult part, which is we you have to configure the JSON uh system configuration for Katie's website. So uh we're gonna add a route in here at the end of this file, uh actually before observability.

And this is gonna have www.katierobare.com and the zone name is from your cloudflare account, katyrobare.com. And so now we've got the configuration file for this Cloudflare worker. Do I want to add an ages file? No, I don't want to add age spell. And now I think this is a good reminder that we can do this work for you because, Chris, to your point, it doesn't stop at just setting up a Google Analytics property, which a lot of people can do, but to actually get the data correctly, we need to be doing all these other things.

And you know, you talked about um last week. We had asked the question, does it have to be Cloudflare as the CDN specifically? Because there's other ones out there. And you said at the time, yes, because of the capabilities of what Cloudflare specifically can do in this context. Can you say just a little bit more about that and what some of the alternatives that people are using might be?

Yeah. So essentially what we're trying to do is we are trying to get ahead of our Google Analytics. Because you, like I said, if you use regular Google Analytics tracking code, it knocks away all the bots. It says, I don't want you here. I want to only track real people.

So Cloudflare is it's called a CDN, a content delivery network. And that means that it can it sits in front of your website. It sits in front of your domain name, depending on how you set it up. And what it does is it says, okay, here comes a request. Cloudflare looks at it and says, is this something that I handle?

Um and it the well, the worker evaluates it. And the worker says, no, I don't handle that, it passes it onto the website. And so uh if it's if it's not a bot, if it's a human, Cloudflare says, you know what? Go to the Trust Insights website, human, you're okay to go over here. If the worker says, hey, you're a bot, um, it then passes it into its workflow.

It says, What kind of bot are you? As are you an AI crawler? Are you, you know, one of these types? Great. I'm gonna now bring up Google Analytics through its API and say, hey, Google Analytics, I got a bot for you.

Uh and if it doesn't recognize it, it just says, okay, get out of here. Um, once it does that, then it goes to Google Analytics and says, Here's your here's the bot information I got for you. Google Analytics records it, puts throws it to BigQuery, and then we can report on it. So that's sort of the workflow of what's happening from a technological perspective here. I mean, so and we talk when we talk about it with our clients, we really call Cloudflare or CDN sort of the gatekeeper.

Uh, to put it in human terms. John Wall is the Cloudflare of Trust Insights. And so if you want to talk to anyone at Trust Insights, you have to get through the wall. Um, but I mean, but it is really sort of like it's an interesting way to think about it because if you have that one person who's vetting everything coming in, saying this is a legit deal, we should explore it, or this is spam, we're gonna ignore it. It's essentially the same thing of you have that head of business development who's vetting everything coming into the CRM before it gets to anyone else.

Your CDN is saying, is this trash? Is this useful? Is this spam? Let me go ahead and filter it and not even bother the website yet. Mm-hmm.

Exactly. How about from that first trigger where it does start to write GA data at that point too? Does the bot just get accessed? Does it get let through, or does it have to go through the whole chain before the bot can actually get to whatever content it was looking for? Um, it has to go through the chain.

So it has to pass because a lot of bots get bounced away. Right, right. So it's got to make it through the the end of that stack to actually access content. Exactly. Yeah.

If you if it doesn't have that there, then it it it gets sent uh sent packing. So like there's a lot of very hostile crawlers. Um, even things that are um malicious, like uh bots trying to inject code into your website, all that cloud flows, yeah, get out of here. Okay. So our next step is after this point, we have to so we've gone ahead, we've modified the configuration.

I'm just gonna double check to make sure the configuration is correct. Uh is the routes are installed, the compatibility flags are installed, uh the bindings are set. So I think the only other part I have to do here is now register. So remember we got the your Katie's API uh key and measurement property key for her website from Google Analytics. Cloudflare has to be told what those are, but you can't put it in the configuration because you you it's a best practice to never put configuration data, particularly valuable stuff, into a text file that you upload to a server.

Instead, you want to put it in a secrets file, and Cloudflare provides a mechanism for doing this. So while I do that, um So I actually have a question for you, John, about bot traffic while Chris is doing that. So uh for those who are new here, for those who don't know, uh let's see, where did it go? Here we go. John Um is the host along with Chris of Marketing Over Coffee, which is how many years now has this podcast been going?

It'll be 20 in 2027. Oh my goodness. You guys better do something. I'm gonna be mad if you don't. Um but you know, and so in terms of podcast, have you seen an inflation of bots skewing your listener numbers?

And so, you know, we're talking about a website. I know you have marketing over coffee.com, which people should go to to understand why they should be listening to marketing over coffee. If you're not, you should be listening to it. But have you seen this sort of same idea as a problem for you know, inflating um listener numbers or podcast download numbers? No, that's one thing that has changed a lot in the past five or six years is there's actually a number of competing bodies that monitor podcast stats, and all of them are militant about chopping out all bot traffic.

So that's it's actually been the opposite of we have a lot of situations where human traffic gets discarded and not considered part of actual traffic. Um, so yeah, unfortunately, I wish we had that problem that there was, you know, the numbers looked great, and you know, maybe there were some bots in there, but it's more the other end of all of the uh infrastructure and cruft around podcast analytics has really choked it off to the point where the the problem we'll have is um the network will uh will be doing stats uh ticking like for example, this one gate that we have for a bot, they'll actually for the podcast be three or four of them stacked up. And the problem is, you know, once it makes it for through bot one or two, sometimes or gate one or two, you'll have the third or fourth gate say, Oh, yeah, we're gonna ignore all the traffic that came from those previous gates. We're gonna consider that not human traffic. So you can, yeah, it's insane.

I I can literally see a window of anything from say 12,000 down to 30 downloads for an episode based on how those things end up being stacked up. That must be so frustrating as someone who like you it sounds like you have no control over those gates. Yeah, right, exactly. It's still but and and we ultimately just always fall on the side of look, you have to tell us how many leads you got last month, you know. Tell you tell us if you're getting the traffic because it's no longer us just selling by the thousand isn't really doing anybody any good because we can't prove much of anything.

Oh, you're talking about the people who sponsor marketing over body, yeah. No, and I think that that's a big consideration, you know. And so there's a lot of websites that do have sponsorship spaces on them. And so this is a consideration for that in terms of the type of metrics you're actually able to report on. You know, how inflated is it?

Is it true? Is it bots? Is it spam? Is it human? Because the bots don't really care about the ads, they're not gonna buy anything yet.

And you want to know that the human is seeing the ads, you know, and so I can see where this would be a challenge for someone selling sponsorship on a website as well. Yeah, and then the at the other end, there's just this whole gigantic bucket of, well, these bots are coming around. It's just like you said, well, they're maybe they're not buying today, but we already have situations where bots can make purchases, and there's geo as part of this too. Like having these bots in are going to get you to show up in other places, which we know are gonna be the future of uh of the web and of how information is shared, you know, getting into agents, getting into software tools via web MCP, whatever. Like this is all um, you know, how the future of the web is gonna look.

Have we filled enough airspace, Chris? Yes. So in the time while you were talking, we we authenticated to your website, we passed um your your API keys over. And effectively, what we've done is we've built a small server here on my computer, and then that gets pushed over to Cloudflare. You can see that the that Cloudflare now has that particular bot uh installed.

So the server exists um and it is ready to begin listening to what we've got to say. So the last part we have to do now is we have to go to Katie Robert, we have to go to the worker routes, and we have to make sure that it is plugged in. I'm gonna edit this to say anything from KDrobair.com, send it to that bot, the geo bot. The geobot telemetry is there, it looks good. Let's quit check our workers.

There's our worker. We have logs are enabled, the domain is there, and it is enabled. And now once this server spins up and and starts working, it's effectively gonna start every time a bot comes in, like we were talking through the workflow, it's gonna see those logs. It's gonna say, I know what to do, these logs, and I'm gonna send them to uh to Katie's website to K's Google Analytics for. And then so now the wiring is all done.

Amazing. Bing bang boom, done. Yeah, this has kind of blown me away because I didn't realize that there was like this full on dev and server environment as part of Cloudflare. But so is this grown out of, you know, so many people are using Cloudflare for just managing raw traffic for situations where they get overwhelmed. And so are those tools normally there for your building out, you know, what happens when things start to light on fire and prioritizing traffic and things like that.

So you can build Cloudflare has expanded substantially all kinds of crazy stuff that you can do. They have their own AI platforms. You can deploy models and workers and GPUs and stuff. So they've they've gone from, hey, let's manage your website traffic to hey, if you want to do anything with AI and you don't have the hardware for it. Um, let's say I wanted to build uh an AI chatbot quiz on the Trust Insights website.

I could stand up uh a language model on a Cloudflare server, connect it to our website, and then it would serve up the AI. Now, the moment you turn that on, the the meter would start spinning like uh like you know the electric meter during uh during Ghostbusters, but it is possible to do that. So they've added this whole AWS style compute infrastructure to your website anal, your your website, your CDN, which is it's really A, it's very impressive, and B, it's like holy crap, there's so many more things that you can do with it now. And even if you just want to gut check on if you have things set up correctly and if you're about to get like a $10,000 bill, just go to trust insights.ai slash AI bot analytics, and we can certainly work out something that you know lets us take a look at your configuration because you may have the skills in all of these different systems. You may just want a gut check to make sure that uh you're not about to cost yourself a heck of a lot of money.

One of the things that you should do on your Cloudflare account, I'm gonna come bring up here because for obvious reasons, is you should set up building alerts. And you should I mean for any cloud-based service, you should set up building alerts. But you should set up building alerts that say, hey, you know, here's the ceiling, and I'm required to notify the user when I exceed the ceiling. The best case scenario, and this is not a lot of companies do this, but the best case scenario is that it just cuts you off and says, okay, I'm gonna alert you, and then I'm not gonna spend above this point. Google is notorious for doing the opposite.

Google will send you alerts, but we'll keep the meter running. And you're like, well, crap, it took me an hour to get to my computer, and I got it, you know, eight thousand dollars worth of compute uh, you know, in that hour. Um, but definitely in your systems, make sure that you have alerts set up to say, like, hey, you're about to spend a whole bunch of money. The last and final part of this kind of setup is once, and there we see AI and AI bot has visited Katie's website. There you go.

Yay. Um, you always, always, always, and we had this experience with the Trust Insights live stream this week, actually, from our web MCP episode. Uh, we actually caused some damage. I caused some damage to the Trust Insights website this week because the plugin that we made to do Web MCP, I made too secure. I made it so secure that it was intercepting Google Analytics and saying, you're not running from a trust insights.ai domain, so I'm gonna not let you run.

Uh, which could have gone better. We definitely saw that in our data. And so, you know, we gave one of these to our data set and said, Hey, something's wrong. Uh, we should probably take a look, which Chris, to your credit, you did, and you trouble you were able to troubleshoot what was happening. But this is again why you want to consult with someone who knows what they're talking about.

Because, you know, yes, we missed a couple of weeks of traffic. It could have been a lot more devastating, you know, depending on you know how you had put the uh web MCP together. You could have vibe coded it, you could have included you know, a lot of privacy holes. We could have gotten a lot of you know unfriendly traffic to our site. Like there's a lot that could have gone wrong.

So you definitely want to make sure you're talking with someone and engaging with someone who knows what they're doing as you're setting these things up. Yeah. Uh a piece of software that was too secure is one of the less bad problems to have. It's um preferable to the other end of the spectrum for sure. Exactly.

Um, so we pull up the real-time view just to make sure that the telemetry is there. We can see that now. The last step would be creating a a dashboard to look at this actual data. And for that, um, you'd probably want to just use good old-fashioned uh data studio. Um, data studio is you know, is the is the preferred easy way to get it at things.

Let's pick Katie's website. There's Katie's AI bot friends. And what's gonna be different about this is remember all of those custom dimensions and metrics we put in, those are the only ones you can use because all the standard Google fields are going to be empty. So I might throw in like a little date control here, and then I might toss in a little table of some with some heat map stuff here, and in this table, we would have things like um our bot category. And then instead of views, we have to use events because the only measure we're gonna have, and there's gonna be no data for this because uh obviously it's a 24 hour delay.

Um you'll be able to see what uh the the bots have been doing. So I put in request path here, and then tomorrow, okay, we should be able to come into this dashboard and see what bots are hitting what pages on our website and how many times each. And this you can make you know charts and graphs and all these things and get data studios. Um, but it will give you a sense of here's where the bots are spending their time. No, I mean, and again, you know, like I I feel like I keep emphasizing it, but like we all saw how quickly you just put all of this together and set up a data studio dashboard.

I'm looking at that, I'm like, oh, that would the dashboard alone is gonna take me an hour to remember where all the things are. And so I want to make sure that people are clear that Chris is expertly proficient in these things. And so the time it takes him to set that up, take with a grain of salt. And I say that like, you know, completely respectfully, you know, unless you are someone who is as well versed in all of these systems as Chris, it's gonna take some time. So it's not gonna be a I have an hour, let me set it up.

It's gonna be maybe a few days. Yeah. Um, it's interesting, even just looking here now, we've we can see you've you've already had four bots. Now your site, like you said, is kind of a it's not a huge site, it's like five pages on it. Five pages.

There are no humans other than me on it at the moment. Uh, and yet we've already had four bots pop by uh and consume some resources. So even a very small site like Katie's, now she's not gonna hit the 100,000 events in a day uh mark, but it you know, for any site that's other anything other than the placeholder, you're gonna the this is gonna add up real fast. Yeah. So I mean, I would imagine, I mean, so I to your point, I might not hit it in seven hours like we did for Trust Insights, but I could very well hit it, you know, by the end of the weekend.

And with AI bots in particular, it's not a question of how many humans visit your site, it is how much content you have. So if I were to turn this on from my personal website, my personal website has 11 or 12,000 pages uh that goes back to 2007. Um AI bots love it because there's a lot for them to grab and train on. My Cloudflare Workers account, if I set it up, would be done in like probably an hour and a half, like because so many bots are hitting so many pages so often. Right.

And I think that that's a huge consideration for the size of your site as well. Um now, within Cloudflare, or I guess I I should say maybe within this setup, can you restrict what it's looking at? Like, can you say, I only want to look at, you know, bots that hit the home page of Katie Robert? So let's say you have a large site like yours. Could you segment it down and be like, I only care about bots hitting pages where I have something for sale?

So the answer to that is yes-ish. And the reason I say yes-ish is because you're going to need to pop open your coding tool of choice. And there's an index file that we created that contains the actual code for what it tracks. I could make a modification to this instead of tracking every page to have something to say just track you know the top level pages or uh you know first level pages and nothing beyond that. Um, this also has a bunch of different crawlers.

So like I custom built this script um for these specific things. You could also say, like, I I don't want to track certain bots. Like I maybe you don't care about Cohere. Throw it out. Um, so yes, you could I I would have to modify this to do that.

And I think again, that's you know, all custom work that someone who knows what they're doing, like Trust Insights could do for you. You know, I think about you know, some of our clients who have very large websites that are broken down into different parts of the site are literally different cost centers and business units. And I would imagine setting up something like this for each of those cost centers versus the site as a whole, especially a very large enterprise-sized site where it's gonna like give it maybe 30 minutes and you're gonna hit all your you know maximum bots, but it's also not really gonna be helpful if it's not broken down by which section of the site is getting the most bot traffic. You may find that, you know, my blog gets all the bot traffic versus my about me page as an example. Yep.

You could do that. Um, because it's just JavaScript and I have the routing, I have the routing set to go to the main Google Analytics um measurement protocol um account and the API key that we set up. There's nothing stopping me from adding, you know, supporting five or ten different measurement protocol APIs. So if you had a different if you had different Google Analytics accounts or properties for different lines of business, yeah, I could modify the script to route it based on you know what if it's this section of the website, send it to this measurement protocol API. If it's this section that, you know, if it's if it's your widgets, go to the widgets API.

If it is the podcasts section, go to that, you know. If it's the the left-handed smoke shifters, send it to this API. So you could modify that way. Could also modify it to go straight to a database. So let's say you're in your company that has like a data lake uh and you have something set up like Snowflake.

You could, as long as Snowflake has a publicly exposed endpoint that with it, you could send me the API key and I could rig up the script to say, okay, I'm gonna send data to Snowflake instead of Google Analytics. We have it set up with Google Analytics here because that's what most marketers are used to. But if you have a data science team and they want the raw goods, you know, themselves, we can absolutely repoint the script to to any database. Okay. I think that's also helpful context because you know, as much as Google Analytics is quote unquote user friendly, I can see where people want to go even deeper.

Yep. Or if you want to use it with like an Adobe Analytics account, uh using Adobe's measurement API if you're in Adobe Shop, you know, certainly rewire it for that. It just the one non-negotiable part here is the cloud flare up front in front of, because it has to intercept the bots before it routes them. Sure. Makes sense.

John, questions, comments? This is one of those things we just have to sit back and see what comes in, you know. Over the next couple months, we'll be trying to figure out what who's hitting us and what are they looking for and what's working, not working. It's uh kind of opening up a new Pandora's box. It really is.

Yep. One of the first analyses I would suggest people do if you set this up is to look at do a do a set of comparisons. So last week we told you about with Bing Webmaster Tools and the pages that you can get out of that. We also told you about Google Search Console and the pages you'll be able to get out of that. Um, this will give you the pages that the bots are hitting by bot type.

And you should grab all three data sets, maybe with also your regular Google Analytics for what your humans, the pages your humans hit the most, and say to your favorite coding tool, hey, what do all four of these data sets have in common? Like which pages are popular across the different surfaces? What are the pages that are popular only in one domain or two domains, or what pages are popular with the AI stuff, but not with the humans? And that delta can then tell you, huh? I wonder why.

Because if you think about it, when people talk to AI, they are talking, they're getting back the most probable answers. So the humans who are actually on your site are going to behave differently than the machines. So you this is a way for you to can compare and contrast those two data sets to say, do the machines behave differently on our site or with our data than humans do? And if so, if machines are 20x your human traffic, you might start optimizing your site differently for the machines. Mm-hmm.

It makes sense. It really depends on what your goals are. Um, you know, we haven't had a chance to do it yet, but it's a nice plug for the 5P framework by Trust Insights, which you can get at TrustInsights.ai slash 5p hyphen framework. What is your goal of understanding this traffic in the first place? That is a really great place to start before instituting all of these different protocols because you know, as Chris just showed, like he kind of went through it because we have a certain amount of time for the live stream, but like it's gonna take a bit to get them set up correctly.

So what's your goal for doing it in the first place? What is what is the question you're trying to answer with this data? Next, the people who needs this data? Who is going to continually use it, or is it a one and done? Because that may not be enough of a justification or business case to go through all of these hoops of setting this up.

So who is benefiting from having this data? Your process? Chris just showed what the process is at a very high level. Um it's not an easy process. I'm just gonna say it's not easy.

The platform we just showed there's what Google Analytics, Cloudflare, Big Query. Uh, you may have some coding, you may have, you know, some terminals set up, like there's at least four or five different platforms. Um, you need node installed on your machine. You need note, so okay. And then performance.

Did you answer the question that you were after by setting this up in the first place? Um, you know, did you find out here's what percentage of traffic to our site is bot traffic? But the question really is once you have that data, what are you going to do with it? Is it just nice to know? In that case, it's really not worth setting up because it's a lot of work.

If you're going to do something about it, as Chris mentioned, if you're to take that delta and think about a different way to optimize either for humans or for AI, then it is worth it. And that's when I really would recommend reaching out and we can help you do that. Yeah. Like the literal title of the live stream, so what? Like, well, what are you gonna do with the data?

Yeah. Because I think there's a lot of really cool, shiny objects, way to get data now that people are interested in. But that's just new. It's new duct tape to old problems. I guess is the way.

So like you have your zebra print duct tape, you have your pink duct tape, you have your blue duct tape. At the end of the day, it's still just duct tape. If you're not telling me what I'm doing with any of this data. And so there's a bunch of new ways to collect data. There's a bunch of new ways to look at the information.

But if you don't have a plan for what you're doing with it, why you do it in the first place? Yep. Katie, uh, 27 bots have come to your website now. Man, I feel popular. The robots are everywhere.

But for a website that one, barely anyone knows about, and two is only five pages, that's a lot. Yeah, that's a lot in what uh 12 minutes. We set this up 12 minutes ago. We turned it on 12 minutes ago. Yeah, that's a big deal.

It raises the very interesting question, and this is a strategy question. Maybe something we should debate on the podcast sometime. Is what happens when your largest audience isn't human? Mm-hmm. And I think that that, you know, to go back to not to belabor it, but but to belabor it, the five Ps, that's a great question to be asking.

And then you run through this sequence again of what do I do if the majority of my audience is not human? What do we do about it? I don't know. I have some thoughts, but I would want to go through, you know, a few different business cases before I actually start taking action because I don't want to alienate the humans that I do have. Right.

Because for now, humans still are still the ones buying things, but it also things that again, the five Ps help solve. A lot of people are spending a lot of time trying to get good at GEO. Once you're good at GEO, then you have more bots than humans. Now what? And if you want to learn how to get good at GEO, we get a course for that.

Go to trustinsights.ai slash G E O 101. Uh if you take that and you're like, cool, I'm ready for the next thing. We have a 201 course as well in our academy. Um, you should go to Academy.trust insights.ai. We're just we just got all the stuff for you.

We're gonna help you out. Exactly. Oh all right. So Katie, your your your bot family is continuing to grow. It's like Tamagotchi.

Um I understand that reference. Um, and I tried setting this up on Marketing Over Coffee, and holy crap, because that's 19 years of data. Yeah, it hit its limit like in 90 minutes. It was just done. Good news, bad news, John.

Uh you know, no surprise, yet another dumpster fire over at the marketing over coffee website. That's just uh price. No, no, it's it's the popularity of a with the bots. It is like they are our number one fans. It's bot crazy.

Oh, yeah, that I've seen other we've talked about this in previous casts, but that'll have to be something. You know, the thing that I'm really coming around to, the big idea is very soon there will be this is what your AI website needs to look like. Like there just needs to be now how much overlap there is, that's a whole another thing. But basically, what is your AI website? Like, that's a thing.

We're gonna be talking more and more about that over in coming months. For sure. That's a good idea. We could we could build that because I keep meaning, I haven't gotten around to it yet, but I keep meaning to build the savory oatmeal company website. We have we own the domain name, so that's for another day.

That's for another day. That's gonna do it for this episode of the live stream. Thanks for tuning in, folks, and we will see you all on the next one. Thanks for watching today. Be sure to subscribe to our show wherever you're watching it.

For more resources and to learn more, check out the Trust Insights podcast at TrustInsights.ai slash TI podcast, and a weekly email newsletter at TrustInsights.ai slash newsletter. Got questions about what you saw in today's episode? Join our free analytics for marketers Slack group at trustinsights.ai slash analytics for marketers. See you next time.