So What? Generative AI and Analytics

Well, hey everyone, happy Thursday. Welcome to So What the Marketing Analytics and Insides Live Show. I'm Katie, joined above me by Chris and John. How's it going, guys? Hello.

I love the enthusiasm. It's Thursday. Come on. It is Thursday. Thursday.

It's just cold. It's a frozen reaction. It is cold. Um excuse me. Oh my goodness.

This week we are talking about generative AI analytics. So how generative AI applies and doesn't apply to analytics, what tools to use for generative AI analytics applications, and the not-at-all secret to use generative AI for any analytics use case. And generative AI is the shiny new object. So how do we bring the two together to make all of our analyses more straightforward? And so Chris, I'm guessing you are going to blow our minds today and just make us dizzy.

So where would you like to No, I'm gonna irritate you. Well, I mean that's a day ending in Y. Um I think we should preface this by saying that there are definitely specialty vendors that are trying to build tools in this space, and we will not be talking about them today. Uh we'll be talking about using the base tools that people are familiar with, chat GPT, Claude, etc., and not individual vendors. So I just want to put that out there in front because otherwise someone will say, well, this company does like I'm I'm sure they do.

I have not tested their software, so I don't know. That's fair. Okay. So let's talk about analytics. Probably the uh the most common data source that everyone's used to is good old fashioned Google Analytics for the uh the bane of marketers all over.

By the way, a quick reminder um if you have not backed up your data from Google from Universal Analytics, you should do that sooner rather than later. That will Google at some point will be taking that away. I believe uh it was July 1 of this year. I think they moved that back to January 1 of 2025. But either way, you should export that data and save it because you it will go away permanently after a certain amount of time, and you will never ever get it back.

Um I would say if you have questions about that, reach out to us, trust insights.ai slash contact. So Google Analytics is probably the most common uh form of of digital marketing analytics that people are aware of. The challenge with it is that there are no direct hooks to getting its data from here into a gender AI system. So you need to essentially get your data out of here first, and then you can take it to different language models and ask them questions about your data. Now, here's the the other challenge.

Language models, you know, for example, our our friend ChatGPT or Claude or any of these things are really good at language. They're good at processing and predicting the next sequence of words or something. They are not good at math. They don't know how to do math. Um, what to the extent that they can do math in their bare bones version, it is because they have seen that math somewhere before.

So if you ask a language model, this is particularly true in the early days. If you ask it like two plus two, it could answer four, because it's seen that example a bunch of times. If you asked it 176 plus 12 to all 13, it would hallucinate because it wasn't sure what to do with those numbers because it can't add, it can't do math. Which is really weird. And I say that because I feel like that's the misunderstanding of what these tools are meant to do.

It, you know, if you ask, you know, a Google search bar what that same equation, it's gonna give you an answer. And I think that there's a misunderstanding that these tools are identical, if not more advanced. And I think that that's a really good distinction to call out. Yeah, they they're not, they are language machines. The core models themselves are just word prediction machines.

They're really good at predicting sequences of words that they have seen before. Sometimes math is expressed like that, um, but not that often. The work around that, some tools, chat GPT in particular, and many other tools in the analytics space, is that they write code. They write code behind the scenes that and because that is language, and then they run the code in a virtual environment and that can then do the math. So let's take um let's take a pretty straightforward example of some data.

So I'm gonna take some data out of my Google Analytics instance here. Uh, and this is real simple stuff. I said I just want the number of sessions uh by channel for the for the past two months and the two months prior. So this is from my personal website. And we can see here I had, for example, 18,000 visits uh sessions from email in that two-month period.

I had 37,000 in the the most recent two-month period, and so on and so forth. Relatively straightforward. The way that you would typically use this kind of data is you would you would put it into a prompt and you would ask it of language models. Um that seem clear so far, Katie. Yes.

I'm I'm curious to see where this is going, because in my mind, I'm like, well, you have everything you need. Why do you need to bring in generative AI? Because you wanted to know sessions for the past two months and sessions for the past 30 days or previous period, and you have that data, and you have percent change and you have um absolute change, I'm guessing is the next one. Um yeah, so columns E through I, I added. Those are those does not come out of Google Analytics.

But that's a basic calculation. So I'm so yes, it makes sense so far, but I'm curious to see where this is going because in my mind, I'm like, okay, you're done. You got the answer. Mm-hmm. And I think that's actually a good um sidebar to this is when you're when you're thinking about generative AI and analytics, you probably should be thinking about what your user story and use case is that you'd want to use this first.

Because if it's just to do math, you don't that you know, you don't need generative AI to do math. Um, and so as a friendly reminder, if you're unfamiliar, user story is a simple sentence comprised of three parts. As a persona, I want to so that the persona being who you are or who you're representing, the want to being the action, and the so that being the outcome. And so in this case, it sounds like you started with I want to know the sessions to my website for the past two months and then the previous periods and how much it is either grown or not grown. So that I can decide what do I need to do to make my website better.

Yeah, exactly. That's that is the the use case. Um the way that you would take this data in would be through using a prompt. Um, so I'll give you an example prompt that says uh you will act as a Google Analytics expert. You have knowledge of Google Analytics for Google marketing platform, blah, blah, blah, all these different terms.

Here's some background information. This is following, if you're not familiar, uh the what we call the race framework, uh, role action context execution. And you can get this uh for free on the Trust Insights website if you go to TrustInsights.ai/slash prompt sheet. Uh it's a PDF. And so I have my role, I have my action, I have my context, which is all this information, which uh things that I think are important about analytics.

So I have some some experience with analytics, and then here's the data itself. Here is the execution statement. Do this thing. So let's go ahead and take this. What I want to do is I'm gonna bring up a a very interesting service.

This is a service called chat uh chatbot arena by LM Systems. And I want to bring this up for a couple of reasons. One of the things chatbot arena allows you to do is it allows you to compare models side by side uh to see which models handle this sort of information best. So I'm gonna go to side-by-side arena. And let's say, let's start with OpenAI's GPT-4 Turbo, which is their biggest best model.

And then next to it, we have our choice of different models that we could look at here. Well, let's take Google's Gemini, the one that powers Bard, right? That's a that's a good choice. Um these are two very popular models. Actually, well, is Gemini a good Gemini is more of a well, let's leave it with that.

Well, it sounds like you first need to understand what all the different tools are. So, you know, as I'm looking at this list, I know what GPT4 is. So I'm obviously not a uh power user like you are, more of a casual user. I didn't know what half of those are. I recognize GPT, I recognize Llama, I recognize a couple of others, but I don't know that Gemini for the casual user is a common model.

That's true. Yeah, you would only know it uh as Google Bard, today's Google Bard. Um yeah, so there's a bunch of other ones. So there's an open source models like Mixdraw in there, there's OpenAI's GPT 3.5, there's Anthropic's Claude, uh, a few versions of that, and there's a bunch of open source models. Um I would say for for what we're doing today, we'll stick with the commercial providers.

Uh for example, GPT-4, uh, Gemini from Google, and Claude from Anthropic, because those are the same models that power the like very common interfaces that everybody knows. But you would use, you can and should use this particular system if you want to evaluate which of these different models would be good for a specific kind of prompt, which is exactly what we want to do here today. Okay. All right, so I'm gonna take my massive prompt that has all my data in it, because I took those those two tables right out of Google Analytics, and I said summarize these findings, taking into consideration the background information comparing the previous period to the current period. Include percentage and absolute numeric changes from period to period in the analysis.

So we're going to send these two. It says you have hit the hourly rate limit on this model for uh GPT4 Turbo. Um so let's see what that's a pretty weak Thunderdome. I mean, if it's over in like 0.2 seconds, you know, I'm here for the show. What about you, John?

Yeah, no, exactly. If you Tina Turner would not approve, obviously. All right, so Gemini says direct traffic saw a significant increase from 5696 to 8265 sessions, representing a 45% increase, could potentially indicate issues of tagging and data governance. Unassigned traffic uh increased to 1145 to 2270. Email traffic group substantially from 18 to 607 to 37294 was representing an impressive 100% increase.

Organic search traffic remained relatively stable. Uh referral traffic was a 17% disk decline, and so on and so forth. So this is what Gemini has said. Now, here's something funny. Let's see.

Gemini is saying, so Gemini is getting the math correct. Um 45% increase. Uh yeah, uh for direct traffic from period to period. Unassigned went up 98%. Yep, unassigned went up 98%.

So Gemini at least can do basic counting, which is good. Going email going from 18607 to 37294, which is 100% increase. So far, so good. Next, let's take our second follow-on prompt, which is now build a marketing strategy with recommendations based on this information. Because if you're right, just doing the math, yeah, you can absolutely do that in Excel.

Well, and so that's it. I've been sort of on the edge of my seat going, where is this going? Because that first prompt, now it's starting to make sense. Like, okay, you really just want to first validate that the model can read the data correctly. That's number one.

So you have to go through that step. You can't just say, here's some data, create a strategy. You need to validate first that it can actually read it. Then you can move on to what the heck do I do with this information? Exactly.

And this is there's a very subtle gotcha in there as well. Language models get better at generating results more verbose they are, the more time they have to talk out loud. And the reason for this is a mathematical reason for this, that's to do with in-distribution versus added distribution tokens. But the the simplified version is if you ask a model to to go through things step by step and talk through its thinking, it's going to assemble more words that are related to the initial prompt. And because of the way the Transformers architecture works, it takes into account all the words it's generated in addition to your original words.

And so it becomes stronger at writing about that topic the longer it goes on. If I gave it a table and said, just create a marketing strategy, there's not enough words there for it to come up with a great answer, as opposed to talking through everything first to create more words that are relevant that you can then refine. That's why short prompts deliver t typically deliver worse results than longer prompts, because the more longer prompt is, the more words there are for it to consider, and the easier it is to find the probabilities of what word to say next. Do you think that people who are using these systems to try to create strategies? This is what they're doing.

They're basically saying, here's some data, what do I do with this? And it's it's very generic and short-sighted. Yes, I'm not going to name any names, but a prominent marketing influencer got called on the carpet by a number of folks recently for very generic bland short prompts that were going to generate terrible content. Um and it's because fundamentally the person in question doesn't actually understand AI. They don't understand gender of AI.

And so the advice they're giving is very generic, and it's going to deliver subpar results compared to someone like you who knows the some of the architecture behind the systems to say, like, yeah, they need to talk. They're talkers. They need they need they need to for good or ill, they need to mansplain a bunch of self stuff, even to themselves, just so that they get the right words in context. Because you know, I need another thing mansplaining to me on a daily basis. Was this prominent influencer named John J.

Wall, chief statistician of Trust Insights? Not at all. Um that is okay. Long prompts. It's all about one.

So OpenAI's model just did decide to get working. You could see here GPT-4 Turbo uh has presented the same math. It can count, uh, which is good. Um just quickly comparing to the spreadsheet that I have. Yes, it is correctly counted, which is great.

Now let's see what OpenAI's marketing strategy recommendations. Email. Continue to optimize email campaigns with A B testing. Segment the email list more granically. That's not a word.

Direct traffic, review and correct any tagging or tracking issues, organic search, uh audit SEO practices, unassigned, check your tracking, strengthen relationships with referral sites, organic social, reassess your social media strategy, organic video, invest to higher quality video production. Three three tactical tips. Improve tracking and tagging, SEO optimization, and social media engagement. Now we head on over to Gemini, and it says your email marketing. So Gemini just went through my channel by channel as well.

Continue investing in email marketing traffic. Great. Conduct A B testing. Yep. Do segmentation.

SEO, do an audit, create relevant content, referral traffic, uh partner with more embedded websites, social media strategy, reevaluate your strategy because you suck at it. Uh, and develop a video marketing strategy. Three most important things conduct a comprehensive SEO audit, implement an email marketing campaign, and develop a video marketing strategy. So that's both these two models side by side and their assessment of the same data. I'm underwhelmed by both.

Um well, I guess uh it also looks like Gemini doesn't know uh where the space bar is. Yeah, I was gonna ask about that on the title, so that's bizarre. It is this is good though, like because I've seen this in a number of things where if you have a pile of stuff and you'll look at and you'll say, Okay, yeah, I think we need to do these 10 things, and then you run it through the model, and it'll give you a list of 20 things, and you will probably find like three or four. You're like, oh yeah, I I should have thought of that one. Like that's a good one.

Like it it casts a decent net. But yeah, it is funny too how these are all kind of like all the generic info. Like, I'm sorry, but A-B testing is not just gonna solve all your email problems, even though that's you know the number one option. Yeah, it feels very much like all right, what is the most bland vanilla advice that I can give that's not incorrect, but also not super specific because I only have this one tiny data set. So let me at least say something.

It feels very much like I brought my data to an overpriced consultant and this is what they gave me back, and I'm underwhelmed. Now, of these two results, which one is better? They feel the same. Uh I'd vote for A over B. I'd vote for even just for because open AI can find the space bar.

Yeah. Well, yeah, ease of readability, open AI is better. Um yeah, I mean, I honestly I feel like they're both equally underwhelming. They are. And again, this is partly because uh A, it's partly because it's a relatively simple data set.

Like if I hand this if I hand this spreadsheet to you as a human, I could do that. The first thing you're probably gonna say is I have some questions. There's not enough information to make a judgment here. Yeah. What do you want to do?

What you know, where do you have you know, budget, who are your resources, who's on your team. You know, I know, like I could ask you a bunch of questions, but I know you don't have the bandwidth to focus on social. And I know you don't have the bandwidth to focus on partnerships for better referral traffic. You have the bandwidth to focus on email and maybe some organic search through content creation. And so I would say, okay, so the other channels aren't terrible, or do you really not care about social and video, et cetera?

So where can we refocus and get you to boost some numbers? So let's do another example then. Let's go back into Google Analytics. Let's go into our explore hub this time and let's create an exploration. Um and let's do last 28 days is good for dimensions.

I want to do the page of the of my content. So let's do page title, import that. For our metrics, let's do sessions. And let's do events. Where's my event count?

Event count. And let's do conversions. While you're pulling that up, I will do a quick plug. I know that a lot of people have given the feedback that Google Analytics for reporting is a nightmare. And they're not wrong, but if you want help with that, we still our foundational services include Google Analytics.

Uh so if you want some support with your reporting, please give us a shout. And John will take care of you. Maybe depending on his mood. Bring a bag of money and no job too big, no fee too big. Thank you, Ghostbusters.

All right. So we now have a much more thorough piece uh pile of data. So this is the pages on my website, the number of sessions, events, and conversions for those pages. So let's take this now and let's feed this into a prompt and see what happens. We're gonna restart my thing here.

And I'm gonna take a prompt. I'm gonna take uh let's choose who do you want to choose for our let's let's go with GPT4 Turbo. Wanna do Claude this time? Claude, yeah. All right, Claude 2.1 this time.

Uh let's start with our prompt basics here. Now this one that you're using this chat Thunderdome thing. Is this a paid service? And is this frequently? Okay, and so I know that uh if you're using like chat GPT to use the analysis portion, that's a paid service.

So how does that work here? While that's so that's not in here. We're gonna get to that in just a bit. Okay. Because this is just using the language model portions.

Got it. Of pages and metrics associated with the pages on my website. Produce some useful insights about what content is working for me. Okay, so let's take that. Let's take our data set itself.

And go. So interesting. You didn't clean up the data at all. Nope. You didn't remove any of like the headers or the junk at the end.

You're just saying, here you go. Good luck. Yeah, and and part of the reason for that is that's what someone who maybe is not necessarily skilled at data analysis might do. The whole reason you might be thinking about using generative AI with your analytics is because you're like, I am not an analyst, so machine, you do the work for me. Which is fair.

Um, I do think that at the very least, if you're exporting your data from a system like Google Analytics or other systems, just give your data an eyeball first because you'll see that a lot of these systems will generate unnecessary information like headers. Um, yeah, so what Chris has up, like the first five rows. You don't need that in the analysis. It just basically says, you know, here is the account, here's the type of report, and here is the date range. You don't really in there.

One other thing that fouls up virtually every system with Google Analytics data is they always put a summary row in somewhere weird, they don't label it, and then that throws off everything. Thanks, Google. Super helpful. All right, let's see how our two contestants uh did. So we have GPT4 Turbo and we have Claude.

It's uh it says, Here's some insights I can provide your newsletter, uh top performing content. Your my account page has the highest number of sessions, events, conversions. Uh that's newsletter content. Yep. Uh newsletter attraction, pages related to the almost hand newsletter should you buy a custom GPT.

You're consistently seeing good engagement rates, content-related generative AI marketing is strategy of performing well. Well, no duh, that's all the content I've been doing lately. That's not surprising. The page, thanks for taking my poll, has comparatively high number of conversions. Yes.

Thank you, page. Uh the subscribe to has a new summit of sessions and we're not counting conversion rates lower, yes, because that's a landing page. Uh content with lower engagement, receiving fewer sessions and conversions may need to be optimized. Sure, presence is not set. There's no direct data set.

Well, of course that's it's only it's only page data. Uh strategic recommendations, enhance your newsletter sign-up process. Did that. Uh develop more AI and marketing content, doing that. Improved data collection.

Okay, so not bad from from GPT4. Let's see how Claude did. Your newsletter are doing 50% of your conversions are going to these two pages. Yeah. Blog content overall seems to be driving decent traffic.

The conversion rates are lower. That's not super specific. Social media traffic data is unavailable. Well, duh. Um several generative AI focus posts ranking well for traffic, have lower conversion rates.

Comments and account management pages have very high turn rates, and overall traffic conversion bars are healthy. Okay, so that's what the two different models spit out from this fairly long data set. Mm-hmm. How are you feeling, Katie, about this? I'm once again underwhelmed because I'm like, okay, is that this is the recommendations?

This again, I could look at the data and know all of that. I don't need to put it into uh a machine learning model to know that your email newsletter is the highest converting uh for your website, or that you know, the variety of different things that it says. Like I get all of that from the data when I'm looking at it in the spreadsheet. So I'm still sort of stuck on why I need this step. And maybe it's just me personally, maybe I'm an N of one, and because I know how to analyze data, uh, I'm sort of trying to figure out where this comes in useful.

So put yourself in the shoes of maybe uh the a junior person on our team. You just you're you're fresh out, fresh out of college, and Chris, the annoying you know, tech guy says, Hey, here's some website data, go analyze it. And that's all the direction Chris gives because Chris is a terrible manager. So in that scenario. I'm with you so far.

In that scenario, would this be an appropriate response for that junior person as opposed to just staring at the screen going, I don't know what to do? I see. Yeah, I guess it's a good starting place to at least get a baseline analysis of the data, and then you can take it further. So, you know, for example, if I'm looking on the left hand side of the GPT model and I'm seeing content audits in SEO, uh, perform content audits to identify refresh underperforming pages, use SEO best practices. So that's a good starting place.

What I would then say to the junior associate who is doing this, like, okay, let's go, let's take it a step further and come up with more specific recommendations for SEO. So, how can we dig into this data and figure out what is the most useful SEO, you know, set of tactics for Chris specifically? Um, so it is it's a good conversation starter to that point, but it's not the full picture, it's not the full analysis by any means. So our junior employee says, Wait, how do I do a content audit? Okay, yeah, you get remote.

And GPT is out again by open AI. Yeah. So Claude is saying set some goals, inventory your existing content, analyze individual performance traffic, engagement metric search rankings, content quality, depth, accuracy, formatting, optimization of top and bottom content items, look for content gaps and identify actionable recommendations. That's not again for that junior person who is not super skilled, that's not bad. Well, and I guess we should stop saying junior person because it could be any marketer who just this isn't their core competency.

Um, so for me, for example, like I'm not a content marketing expert. So it's helpful for me if you say, you know, Katie, our content is underperforming, where should we start? I'd probably be like, oh, that's a really good question. Not what I focus on. So let me think about, you know, what a content audit would look like.

So I can see how this is useful to at least get started. Because even that I'm looking at the Claude recommendations, they still feel really generic and not instructive. It's just more of like a, and theoretically, here's all the things you could do, but it's not saying, okay, when you say inventory all of your content, what does that mean? What do you actually need to know about the content to consider it inventoried? Like, do I need to say I have five pieces on this topic and six pieces on this topic?

Okay, now what? So I'm gonna say here that uh that GPT 4 is certainly uh more aligned with what we would consider best practices than Claude is here because we would start with the five Ps, the purpose being like, what are you trying to do? Like, what's the what's your goal? And here it says before diving into the audit, define what you want to achieve. Do you want better rankings, increase user engagement, and so on and so forth?

Whereas Claude's like, hey, just get going and do all this stuff. So it sounds like instead of just saying what should I do, a better prompt would have been for us to go through the five Ps, include that information with this data that we pulled from the website and say, okay, help us put together a strategy based on this information instead of just asking the system, what do you think I should do? Which I'm guessing a lot of people are still doing. Like, well, okay, that's great, but what's next? What else?

And it's just not really directive, it's still super generic. Exactly. And again, when they talk about the content uh inventory here, this is actually fairly good right here if from the GPT 4 Turbo model saying, hey, create a comprehensive list of your website. Tools like Screaming Frog, SEMRush, or a website crawl, will help you generate the list of your URLs and include these pieces of information. So who, you know, URL, page title, uh, the various metrics, collect and analyze your data, quant qualitative analysis, relevance to your current audience of strategy, categorize your findings, prioritize your actions.

That's actually pretty good. Whereas Claude just goes very generic and doesn't really dig into useful detail there. And I agree. I think that the GPT step by step, it's like, okay, you can now put that into a project plan and have someone start executing against it. So, John, look out.

This is gonna be on your plate uh in about an hour. That's right, I'll get the the punch list. Now, we've been talking and using the pure language model version of these tools for the last 31 minutes. The one of the tools on the market, uh, chat GPT, uh, has an advanced data analysis uh tool. What this tool does, which is better than pretty much anything else on the market for the beginning, the the entry level tool set, is that it it will process the data using actual math.

So let's do this. We're gonna take that exact same prompt. Let me go back to actually retrieve the original prompt here. This was the prompt that included the data and gave the background. Exactly.

So I'm going to include that prompt and now I'm going to attach the actual data file. So I'm going to load the spreadsheet itself. Unclean, same, same as we started with. And what it's going to do is behind the scenes, I had to click on little drop-down. It is going to do what language models do best, which is write.

They can write language. Aha. So it has discovered those stupid header rows. So let's see. I'm going to attempt to load the CSV with a more with a more flexible approach.

And what it's doing behind the scenes, it is writing Python. Says, hey, I found a bunch of crap here. I'm going to trim that off. Um I've now entered another error. So for those of you watching, uh, one of the things you may not be able to see on the screen because it's going by quickly, is for every code snippet, it has a little box in the top right-hand corner that says copy code.

And so this is a great way for you to start building your own code base. So, you know, let's say you know that this is a question that's going to come up over and over and over again, and you happen to know how to code in Python or whatever the system is, you can say, okay, let me use generative AI to help me write the code so that I don't have to start from scratch, just as a pro tip. Exactly. So here you can see it's skipping a certain number of rows. It's got the main head, and it says, I've successfully loaded the data.

Here's what I see. The title of the webpage sessions, event count, and conversions. I'll first clean the data by removing the summary row and converting the conversions column to a numeric format, handling non-numeric values appropriately. So it has it has gone through and done that. And now it says here's some key aspects we're going to explore popular pages, engaging content, conversions, and page efficiency.

Which page is the most efficient in terms of balancing traffic with engagement event count and conversions. So actually, in terms of things that you'd want to do with your data, this is pretty good because this is unguided. We have not reprompted the software yet. So it's creating these three different data sets. And as Katie said, I can copy this code.

Let's pull up uh VS Codium here. Start a new piece of code here. We're gonna call this GA4. This is that's not Ruby, it's it whatever. Um, we're it this is usable running code that you can run.

So let's see how it's done. Here's the analysis of the results. It says popular pages, my account is the most visited page of the top pages, engaging content by event count conversions, data is not recorded or not recorded properly, which is um I might need to take a look at that. Uh page efficiency, you can't do that. Here's some recommendations, improve conversion tracking, focus on high performance content, enhanced data quality.

So now we might say, okay, great. Give me based on what you've seen, give some content topic recommendations for future content that are likely to do well. And because again, this is Python-based data, it can then load the Python data back into its main memory, the context window. And now it's the leveraging the language portion of the language model to actually look at the words and phrases in column A of the spreadsheet. What are the topics and coming up with uh these answers?

So one of the things that's really nice about the open AI system is that it can flip back and forth because it's it got it's under the hoods basically four or five different models all working together. Um, but it can put all that stuff back together into one coherent stream, whereas with other tools you would have to hop around from tool to tool. What's interesting is it's giving recommendations that are a little different from what we got from the other head-to-head, because it could really do a more in-depth analysis of looking at the data. And so the very first recommendation is uh in-depth articles on marketing analytics and AI. So that's what you do already.

That's the majority of the content that you create. The rest of these aren't things that you necessarily focus on all that much. And so that to me is the interesting part of when you're looking at the overall inventory of your content and what's performing, there's a lot of things here that you don't have a lot of. And so that I think is a really solid recommendation of like, okay, you probably like you know email marketing inside out. Definitely write more about that.

Uh, things that are more interactive, things that are more educational. You do write educational content, but it tends to be at a 301 or a 501 level. So I think there would be definitely some uh exploration there. Uh, emerging trends in digital marketing, that's a question that comes up all the time. And so, what could you write about that?

So, I think that this is really interesting. And you're correct, because the the context window for this model now contains the quantitative analysis from the Python code that it ran, it can give better recommendations. Because again, the pure language models can't do math. Because they can't do math, they have no understanding of what it is they're looking at. They're just trying to predict words based off of numbers.

This is it has sort of distilled down all the math into specifics. So what we get out here is is pretty good. And of course, you can continue to have conversations from here. So this is the not so secret part of generative AI and analytics, which is that these things can't do math. So use tools that can create the code that can run against your data, and then take the results from the data analysis and put that in here, as opposed to just trying to put the raw data in itself.

Because the raw data itself, it's like ingredients, right? You can go to your your pantry and just like take handfuls of raw flour and eat it, and you won't you won't die. It's nutritious. It's just, yeah, exactly. It's going to taste like eating raw flour.

Or you can bake some bread with it and then eat the bread, which is the distilled down product through a series of processes. That's what's going on here under the hood. The advantage of this particular system is that it obviously can do that, does that for you without having you you having to to write code. Now, I will say the code part, as you mentioned, Katie, is really important if it's something that you have to do over and over again. If you can, if you have the the technical skill yourself, or you have someone on your team who has technical skill you can get that code out run it first and then skip right to the analysis part I feel like you're going to have to keep saying you know maybe even just to me that generative AI can't do math because for some reason I feel like that is such a hard reality to wrap my head around because it seems like it should be able to because of the you know qualifier artificial intelligence well duh it should be able to do math and these are language models is it and isn't math just another language and the answer is in this instance no and so I think that that is such an important piece.

So when we talk about generative AI analysis we're not talking about these systems doing the math to analyze the data it is taking the information and writing a narrative around what it's reading. Exactly they don't have any actual reasoning capability they don't have any actual cognitive abilities they they they're not thinking machines these are prediction algorithms and and they're very good at the things they're very good at and they're very bad at the things they're very bad at and to your point a lot of people assume they're kind of like magic boxes there's no magic here it is literally all just mathematics and and algorithms under the hood. So part of using gendered AI with your analytics is knowing when you're doing a task that is a language-based task versus a not language-based task the the smartest thing you can do is do the actual analysis in the tool of your choice, right? You don't have to use uh Chat GPT for that. As you as we started off, you know, the show today, you can do that here in just good old fashioned Excel.

There's nothing wrong with this. And in fact, you're pretty much guaranteed to get correct math out of this 100% of the time, right? Addition, subtraction, multiplication, division, Excel, amazing. Um, but then you can take the analysis portion, have the language model write the language around it. Say, here's my data.

What conclusions could you draw? Where is their weak spots? Um, I really liked in this content analysis. We did upload the spreadsheet of these different gaps and say, okay, well, this is great. Tell me about a the gaps in my content.

Assume I have an audience of senior marketers, VPs, CMOs, etc. What sorts of content based on the data we've analyzed so far would appeal most to this audience. So content gap analysis, again, that's language, and now it based on on the tools knowledge of our audience, and you might want to provide an actual ideal customer profile as part of this. It would then be able to give you more useful stuff. Like, hey, here's things that are not in your content.

You know, the advanced analytics data-driven decision making, case studies of successful campaigns, ROI and performance management, leadership and management and marketing. These are all things that are, yeah, in in my in because I think I used the last 28 days worth of page content from Google Analytics. This is not stuff I've written about in the last 28 days because it's all been generative AI. Right. But this is information that we could say, oh, guess who can write about that stuff?

The person at Trust Insights who knows all of those things inside and out. Exactly. And so the final takeaway here really is for these generative AI tools, they are word prediction machines. If you're predicting words, you're gonna be just fine. So the tasks that are their best predicting words at use them for the tasks that are mathematics or calculation based, do that in a different tool or use a tool that has a separate mathematics module so that it you're going to get a correct answers, and B, um the tool can then pass along that data to a language model to craft language around your data.

John, what data are you going to start to throw at your chat GPT system? Oh that was the first mistake I made when I was playing around with prompts like months ago. I was bothering Chris. I'm like, come on, I just asked it, you know, what are the five largest companies, you know, in this vertical and whatever. And he's like, yeah, no, they don't work like that.

That's not you're not gonna get anything like that. And so, you know, I tried like 20 more prompts, and yeah, I did not get any of that stuff. And so yeah, I think one um area to be aware of though that's really important, and we have this has been a use case for us, is a lot of times you just get to a technical answer and you need something to rewrite. You know, it's that rewrite function where you can say, okay, the data says these five things, write this in a way someone who has no marketing understanding, you know, can comprehend. And so you can simplify and make something more understandable to people who don't know, you know, can't do the calculations themselves or don't know what's going on marketing.

So there, you know, there's some opportunity there. But yeah, as far as doing like the heavy lifting and doing the regression analysis for you, that's not happening. Yep. The other part that and KD touched on this as well, the data prep itself, you probably should spend some time doing data prep because with any ecosystem, it's always garbage in, garbage out. When you look at what is inside that data file itself, you know, even just something as simple as what's in the the page titles, like, yeah, that's from my WordPress blog.

It's it's carrying over uh this that extra content, my name and and you know, the marketing keynote speaker, that's unnecessary. We don't need that uh in here. There's it's pulling in some, you know, comments pages and things. We don't want you we don't need that. So spending some time doing old good old fashioned data prep will help get better results.

And I think, well, so there's that, and then yeah, just making sure that the information is correct. Um, you know, I'm seeing, you know, you write about things that aren't related to marketing at all. So for example, uh, while you're probably wrong about lighter fluid, I know you use your personal blog for just sort of like here's everything that's on my mind. And so, you know, if you're looking to attract a certain audience, you would want to filter this data once you pull it, clean it up, but filter it by things that are relevant to the relevant audience. Um, or things that I guess I should rephrase that to say things that you feel will bring in a paying audience, um, that's you know, appropriate to your business.

Mm-hmm. For years, this the consulting billing rates thing was the one of the top pages of my website, and it brought in a whole bunch of folks that were not a good fit for my business. Um obviously that's changed since then, but uh but yeah, having the ability to look at that data and clean it up and remove stuff that is is not important is is is vital. So that's generative AI and analytics. It is not as with with the beginning with the you know initial tool set that you start with.

Um what's out there is capable in good in some areas, not capable in others, and your best bet is to not pretend it's a magic box that could do everything, but recognize the strengths and weaknesses. And if you want support with that, we do that. Chris literally just showed that we do that. So you can go to trust insights.ai slash AI services. If you want to learn to do it yourself, you can take our uh generative AI for marketers course at trustinsights.ai slash AI course.

If you have other questions, you can go to trust insights.ai slash contact, or if you want to just join the conversation and see what people are talking about, you can join our free Slack group at trustinsights.ai slash analytics for marketers. What did I miss? That's it for right now. I think uh that's good for today, and we will see you all next time. Thanks for watching today.

Be sure to subscribe to our show wherever you're watching it for more resources and to learn more. Check out the Trust Insights Podcast at TrustInsights.ai/slash TI podcast at our weekly email newsletter at trustInsights.ai slash newsletter. Got questions about what you saw in today's episode? Join our free Analytics for Marketers Slack group at TrustInsights.ai slash analytics for marketers. See you next time.