So What? Why should you be using Marketing Mix Modeling?

Well, hey, howdy everyone. Happy Thursday. Welcome to So What The Marketing Analytics and Insights Live Show. I am Katie, joined by Chris and John. Happy Thursday, guys.

Hello. Happy Thursday. This week we are talking about marketing mix modeling and why you should be using them. So we're gonna cover uh creating user stories for marketing mix modeling, how to use the five Ps for requirements gathering, uh, and most importantly, decisions that you can make with a marketing mix model. Um, and so Chris, you and I have been talking about marketing mix modeling.

I have to keep saying it's the I don't know, this I'm gonna stutter over it at some point. But you and I have been talking about marketing mixed modeling all week. And so uh when we recorded the podcast earlier this week, we talked about what it is. Um so if you're curious about that, you can go to trust insights.ai slash TI podcast and catch that episode where Chris and I break down what the heck a marketing mix model is. Um in yesterday's newsletter, which you can get at TrustInsights.ai slash newsletter, I talk through all of the front preparation you have to do before you can even do an analysis.

And so today what we want to do is cover, let's say you already have the pieces. How do you get started with a marketing mix model? What are the user stories that you would want to create to know what decisions you can make? So, Chris, where should we start? This is a big topic.

It is a big topic. And uh to give you a sense of the level of complexity, we're a three-person company, right? We we're not a huge company, and yet it took me two and a half days to pull together the data for a marketing mix model. Um, about a you know, a day to to write the code necessary to format it to be compatible with this each other, and then you know, uh literally just before the show is rolling, finishing up the code and running it so that we actually have a model to show on the air. Um but the the biggest challenges really are around around those five Ps because you you've got to have your purpose, which is the user story, right?

Um, what do you want what decisions do you want to make with this thing? Most of the time people will use them to make decisions about what should we do less of, what should we do more of, what should we spend more on, what should we spend less on? Um and we'll talk about that because you in the data, it actually does tell you very clearly uh what things impacts the outcome you care about. Um then obviously with any kind of model, there's the people side of things, which is who's gonna do the thing. Uh it may be someone in-house, if you have your own data science team that can put it together, it may be a vendor.

I will say this vendors in this space all fall in the category of reassuringly expensive. Um, these these models are uh require a lot of iteration. So we're gonna go through the model that I put together, and I have absolutely no doubt in my mind that Katie, you're gonna be like, well, what about this? And why is this in here? And why is that not in here, etc.?

So there's a lot of that. Um, and that and that's the the heaviest lifting in a model is really the process part, which is what goes in, who what are we gonna use information for? What stuff shouldn't be in here, which is a big set of discussions, um, and then interpreting the outcome. And then, of course, platform is you know, whatever software you want to use. And then the performances, did the model even work?

Um, and there's some technological answers to that, but then there's also the human answers to that. Uh, and then finally, uh, you know, what do you do with it? So I want to amend a little bit of your processing platform. And this, you know, as I mentioned, I covered this in the newsletter this week. And so, you know, as you were prepping for the live stream, we didn't talk through our own user story.

We're just putting together a proof of concept, a demo for the purpose of of this, but you know, had we gone through a user story, which is as a persona, I want to sew that persona being the people involved, want to being the process and platform, and so that being the purpose and performance of your five Ps. Um, you know, the want-to, as we're talking about process and platform, it's not just how you're running the analysis, but then factoring in all of the different platforms that you need to extract data from, bring that back into build a process around it to extract it, clean it, normalize it, blend it, all of those things. And so making sure that your user story is specific. So, you know, it could be I want to look at the past five years worth of data, which is very different from the past five months, which is very different from the past five days. You know, you're gonna have a different set of processes for each of those time frames, depending on what the decision is that you want to make.

And so, you know, there's a lot that goes into it. And so to say, Chris, that you know, you took about two and a half days is really underestimating the actual amount of time that even for a small company like ours, uh, would want to put into building a model like this. Oh, yeah. Like you said, this is a demo. This is not um, this is not the final.

Because again, that's the whole process of stakeholder interviews and building those user stories and validating, even validating that the data itself is correct. So I ran into some issues with our Google Analytics data, huge surprise. Uh and so even had to adjust for that. But if you think about it, two and a half days of consistent effort for a small company where I literally know every system inside and out means that in a bigger company with same double employees where we don't know the systems, you're talking about like two and a half months of work to do all these, to ask people the knowledge that's implicit in our heads, because we're working with the systems every day. Well, and a dedicated team just to figure out what the heck do you want to do with a marketing mix model?

And so, you know, John, I'll start with you. If you were to create a user story, if I said to you we can create a marketing mix model, uh, you know, specifically for you, what would your user story look like? Yeah, the big thing, you know, from the limited exposure I've had to this stuff, it's always about getting information that you can't even get from attribution reports. You know, for most of the stuff that I always worked with, attribution reports get you there. But when you start doing real large branding campaigns, so if you're spending a ton unpaid and getting a lot of unattributed traffic, those are the kinds of situations where you roll that out.

So it'd always be, you know, as a CMO, I want to measure the impact of our branding or you know, a public opinion of the brand, so that I can same thing you'd have on the attribution model, you know, figure out what programs are working, not working, spend X more, make more. Um, but the yeah, the big thing is just getting visibility beyond what people just get from an attribution model, which that everybody can kind of do from in-house stuff, but to to get up to the marketing mix model, now you've got to bring some, you know, heavy data and data crunching to the table. Exactly. So let's look at it just just to start what data we even used for this toy model. So this is our studio.

For those who are unfamiliar, our studio is a programming interface. Um I started with our Google Analytics data, right? So we we know that Google Analytics is one of our our better systems for keeping record of what's happening. Then I went to our friends over at Agora Pulse, uh, because we manage most of our social media channels from Agora Pulse, and I pulled out all of the data that we have in Agora Pulse for the the social networks we manage in there. Um so that's LinkedIn, Twitter, Instagram, TikTok, and I pulled in our personal ones where we have it as well as the company ones.

Cause either they you know, John and I have a podcast that's been on the air for like 17 years. Uh so that that does take uh some impact. Um and then we have our YouTube channels, right? We have the Trust Insights YouTube channel. We have my personal YouTube channel.

I pulled in email marketing data from our marketing automation system, the trust insights one and the almost timely newsletter one. I went over to Libsyn and grabbed our podcast data from uh again from the the feed that I have in there and the trust insights feed. And then finally, um, which is where John lives, I pulled in our HubSpot data to because the HarbSpot data really is um the outcomes that we're after. So in our user story for like a CMO, you know, I want as a CMO, I want to know what's working to generate deals, right? Just to generate sales opportunities so that I can allocate budget and time appropriately.

And so that was sort of my back of the envelope. These are all the different data sources that we're probably going to need. And there's stuff that's still not in here. There is uh public relations data that's not in here. I didn't have time to go and get our our brand un you know unlinked brand mentions from our social from our brand monitoring system.

I did not have time to pull in our speaking data. I did not have time to look at outbound emails and stuff that John does from inside the HubSpot system itself. So there is stuff still missing. Well, and we again as a small company aren't uh taking advantage of every available marketing channel either. And so um this is you know, we do what I would call lightweight marketing for ourselves, and even with that, you just covered about a dozen different data sources that we have, you know, good working knowledge of how they operate, but you know, we don't run paid ads, you know, we're not running Google ads and social ads and you know, other kinds of ad networks, you know.

So if you factor in all of those, you know, we're not doing SMS messaging. Um, you know, we're doing lightweight organic social to your point, but we're not doing you know campaigns on you know all the available platforms, and then you haven't you aren't able to factor in private social, where we do spend a lot of our time with our community on your Discord servers, those places, and so there's a lot of data missing just from this model, just from this small sample that you're doing as well. So sort of I wanted to bring that up to acknowledge that you know, again, as we're talking through, you know, we're a smaller company, we still have about a dozen data sources that we're trying to wrestle with, and it just it really only scratches the surface of what's possible. Exactly. So after you get the data, you then have to deal with the data.

Sounds terrible. They're all different, right? So you have different types of summarization you need to do. Sometimes, like for example, with Google Analytics. Um, I pulled it this in as default channel groupings because source medium would be even more hairy to try and deal with.

Um, but even there, you know, removing direct traffic and unassigned traffic because that's not something you can make a decision about. Um say, Oh, yeah, let's go get more traffic that has no attribution data. It's probably not going to be something uh you're gonna enjoy. You know, HubSpot data, we had um all sorts of we actually have uh a lack of the ability to count things natively in HubSpot. So we have to actually do the counting within our our our marketing mix modeling code to just count how many deals we had uh on a given basis.

Um we have of course all the hairy stuff that comes with the so different social networks. Um, and then with our email marketing data, that comes in. So again, there's no simple counter in our email system. So we had to go into the back end database of our email marketing system and export a couple of very large tables just to summarize them for this, which again is is a fairly heavy lift, and then of course our podcast data. So that's kind of we're we're still early in the process part of the five Ps.

This part at a small company with highly proficient technical resources took days to do. It reminds me at a very basic level, uh, when I was trying to look at more than one data source, like in an Excel spreadsheet, for example, just even the way that the date stamp comes out or how it's broken down, you know, because now we're talking about feature engineering of, you know, some of these come in, you know, hourly data, and some of them come in daily data, and some of them come in weekly data. But even then, how the date stamp is structured is going to be different. So some of them are gonna be, you know, day, month, year or month day year, or a long string of everything, including time frames and time zones and those things. And even just cleaning that up to all get it to be the same kind of data, especially when some of the data doesn't exist at that granular level, is I would imagine that's a big effort just to clean the data and normalize it.

It is, and and it gets even hairier when you start having to do manipulation of the data. So one thing that's not in here, because again, I knew I would not have time to do it, is Google Search Console data. There's branded organic search and unbranded organic search. You would want to be able to split both of those apart. Um, people searching for trust insights uh are uh more valuable to us than people are searching for, you know, uh what are the five Ps or what is a marketing uh what is the mark index?

Because those are folks who are still on their informational part of their customer journey, not the intent part where they're like, Yeah, I want to talk to John Wall right now. He is ready and waiting. That's right. Time life operator standing by. Exactly.

Or Ghostbusters, we're ready to believe you. So I feel like we're sort of, you know, giving all the bad news, um, which, you know, to be fair, we need to really sort of paint the picture of all of the work that goes in up front before doing, I mean, yes, a marketing mix model, but really any kind of analysis, you want to make sure that you're doing your due diligence up front because it will save you time on the back when you start to run the model. And one of the processes that you know, as we are building out this marketing mix model uh software, is that we for ourselves want to build the processes so that they're repeatable so that as we're extracting the data, it's being extracted regularly and consistently and being cleaned thoroughly in a you know standardized way so that we can rerun the model over and over again. Because what happens a lot of times, uh, and Chris, we've run into this with clients before, is that there's you know, this big six-month effort to gather all of the data. We run the model once, and then the client says, Okay, let's do it again.

We're like, okay, but you're gonna spend another six months, and you're already then a year out of date from the decisions you want to make. And so that's another factor is how timely do the decisions that you want to make need to be made. That will dictate how you set up your processes. Exactly. And there's a minimum amount of data that you want to have to be able to do the model, right?

So at a bare minimum, you want to have about 90 days. In this case, we are using 16 months of data. We're going back all the way back to the beginning of 2022 for for this particular model, because with less data, you will have the statistical software to say, I can't make anything of this, right? It's so that does happen a lot. Um, so after you get the data from every data source, clean it up and get it into a standard format, you then have to glue it all together, which is you know what this long section of code here does.

And then you actually get to this to the modeling part. So here's where you can get into philosophical differences about what kind of marketing mix model you want to build based on the type of regression. So uh for those who skipped stats or slept through it or just failed it like I did, which was embarrassing. Uh oh, I also failed it. Yeah.

I'm a stath guru here. This was ridiculous. Oh, John. Well, I know uh we're gonna do role switching now. Exactly, John.

So would you explain to the crowd what regression analysis is? Uh regression analysis is when you take a huge pile of data and you try and prove that there's some kind of correlation uh within the data, like there's a reason why it's going in a certain direction. Right, and there's what, 15 or 16 different types of regression analysis, right? There's linear, there's logistic, there's lasso, gradient descent, um, there is uh gradient boosting regression, stochastic regression um support vector machines. So there's so many different choices.

And a big part of marketing mix modeling is deciding what kind of regression you should use for the data. And this is again is where it helps to have a statistical background and a machine learning background to know which one to pick. It's kind of like picking an appliance in your kitchen. Like there's there's a bunch of them, they're all good, but yeah, I would not suggest putting your stake in the blender. Well, and this goes back to starting with a user story because knowing the different elements of what it is you are trying to understand and the decisions that you need to be able to make is going to help you choose the right kind of model uh that you're gonna run.

Uh, because to your point, Chris, you know, I would imagine a linear model and a logistical model and a gradient boost model, or you're gonna get three different sets of responses. And it really comes down to what is the decision you're trying to make by running this model, and therefore that's the kind of model we should be running. And we have to take into account the nature of marketing data itself. Um, that's a big part. So a standard linear regression model, you know, which is just a uh a literal linear regression, doesn't look at the individual variables and say, hey, this one doesn't really matter.

Like you've got you know, a number of tweets with poop poop emojis on Tuesdays as as a as a variable here. And just statistically, like, okay, this this doesn't make any sense. Um, so part of the modeling process is looking at the data set and saying what does it look like? So our just for this little toy model, we are at uh 107 different variables and 458 rows of data. So 458 days of data with 107 different variables from all these different sources.

So, in terms of regression, the that sort of hints at which models you should be looking at. So, in our case, uh the two best choices would be ridge regression or lasso regression. Um, ridge regression tries to deal with overfitting on lots of variables, so it's good for a high dimensionality, which means very wide tables. Um lasso regression does the same thing, but also allows sort of a tilting in the regression model to to knock out variables that don't matter. So it does sort of a uh reducing reduction of irrelevant stuff for you.

And that really helps to make things work better. So, John, as our resident statistician, does that all so that I'll check out? Well, no, I was gonna ask. That's actually a great question because you know, back when I was doing you know these models, it was a matter of you run one and then you run a second and maybe you run four or five, and then you actually just have to gut it out by hand, like go back and look and see which ones uh match reality. But I imagine it's gotta be ours powerful enough now that you can just run all the models, right?

And see the output. I mean, what's how do you actually discern which one is closer to the mark to what you're looking for? Um, there are automated suites like uh H2O that can do an ensemble. And in fact, for for a bigger project where where we didn't maybe know the data as well, you'd want to use probably an ensemble of models to get to an answer. So we've we've done this with uh a couple of our clients in the past where we've built a model ensemble, usually it was a combination of lasso regression up front to weed out um less important variables and then uh gradient boosting to to amplify the the values of the remaining variables to to give it you know uh more statistical oomph, if you will.

Um for us for this toy model, I stuck just with lasso regression because again, it's a very wide table, it's not a very long table, and lasso regression is really well suited for that. Question on this. Um, because as you're describing, you know, and John, you're mentioning like you would just you know keep running the different models until you get something that matches reality. I guess I have a couple of questions there. So, what if, you know, especially as you get into enterprise-sized companies, you don't know what reality is, and that's one of the reasons why you're on you're, you know, doing this analysis, but.

But also, you know, if you aren't sure or don't understand the different kinds of models, then you just kind of start picking and choosing, you know, do you run the risk of using the wrong model to answer the question? Because I don't know, the data kind of looks good. It makes me look great. It might not be reality, but I'm a shining star if I use this model over here. You know, is that a risk?

Um, which again speaks to why people should have a John Wall on their team who's a statistical superstar. No, we all stare at John. Yeah, well, you're that's a great question, though. So it's you know, are you using bias or is it basically you have to test that after the fact? You have to then get a second data set and actually see if it holds as far as what works.

Yeah, I think that's a that's a really good point, sort of running it against something that is mostly known and then test the using the same kind of model against something that's less known. Yeah, and there are statistical methods, there's there's technical methods for for determining which model performs best. And we'll talk about that in a bit because we'll we'll talk to the validation part in a little while. Um, but one of the things that's challenging here, you'll notice on line 330 is just this agreeing what the outcome is, right? So in this case, I said the outcome that we care about is the number of deals in our Hubspot instance.

Um if you're doing a marketing mix model, um, this is uh you're probably gonna be looking at deals or you're gonna be looking at sales qualified leads, uh particularly in B2B, because a marketer who does their job should be delivering leads to sales, and those leads should be good quality leads, right? So a sales qualified lead where sales has said, yep, we agree this lead is good. Um that's really where your responsibility ends uh as a marketer. You what happens after that is up to sales, right? Sales has got to get the deal.

They've got to get agree on a deal size, they've got to agree on a closing date. They've got to be able to close the deal. Marketing really has very little impact, particularly for B2B after that point. So even when you're building a model, you have to agree on well, what is the outcome that we are responsible for that we want to model against? Because I had the opportunity to use deal amount, but from a marketing perspective, that's not really something we have much control over.

Well, and this goes back to having one or I would advise multiple user stories, because the last part of the sentence, the so that the that is the outcome. And that's really going to help you understand when you start to get into developing this model, what it is, the outcome. So there's the outcome that you want overall, and then more specifically, there's the outcome that you want from the model that's gonna help you understand what's in the user story. So, you know, we keep talking about Chris, how we're a small business. And so, you know, in B2B, yes, marketing sort of has that cutoff point, and then they hand it to sales, but for us, we are all of the things.

And so we would want to know, you know, maybe deal amount or you know, more of the financial side of things because we are doing the marketing and the sales and the finance and the operations. Exactly. So what's happening here? First things we do some more cleaning. We remove columns that are empty, because that's uh silly to have them.

We remove models that are constant, right? If you know John Wall has the same number of followers for the entire period, that's that offers no predictive power, it just kind of screws things up. So we get rid of that. Um then we do uh four cleaning steps. So we have to remove zero values, right?

Because that again something at the zero has no predictor power. Um we remove highly correlated variables. Um, and this is important because uh in almost all forms of regression, variables that are highly correlated just screw up your predictive power, right? If the number of clicks on emails and the number of amount of email traffic that arrives at your website through GA, this they're kind of probably gonna be almost perfectly correlated. You don't need both of those variables, you just need to pick one.

Um and the software will pick uh the one that has more relevance. You also want to get rid of linear combinations, so these are our variables that are essentially um almost identical, and you want to then normalize your variables, so that means centering and then scaling them. All this, you know, back it back in the day, you would have had to do by hand. Thank goodness you don't have to do that anymore. But normalization is an important step because it makes things apples to apples that otherwise would not be.

So the number of likes I have on my tweets versus the number of likes uh maybe uh John has on his tweets, proportionally are might be the same, but uh in absolute terms can be very different because it I just have a large Twitter account. So we want to normalize that to make it uh apples to apples for comparison across all these columns in the table. I'm just like just the data cleaning part of it is exhausting, but so necessary because you know, let's say you have anomalies in your data and you didn't bother to clean it, you're gonna break the model every single time and not know why. And you know, you know, depending on how you're running this, you know, if you're using an agency or a very expensive piece of software, it could be a huge waste of money. Um, and that's why we keep going back to using the 5P structure to really make sure that you are gathering the requirements, but specifically focusing on your processes uh to really understand am I even ready to run the model?

Um and you know, until you do that cleaning, you're not, you are absolutely not ready. I like broken models better than I like models that appear to be uh good, but in fact are not. Yep. Like bad enough, like slightly rotten data is worse than broken data because slightly rotten data you'll make decisions with like it looks okay. Like it, you know, it's like you sniff the the milk and like it smells okay at the first class, and you pour it out in some chunks, like uh, I guess it wasn't okay.

Yeah, that's uh I think a better, maybe a better example here is uh when Chris and I were doing monthly reports earlier this week, we were looking at Google Analytics data and specifically Google Analytics 4, and we kept coming up with different numbers depending on how we were looking at it. And at the end of the day, it was all roughly the same, but not exactly the same. And so that's sort of where Chris is talking about the slightly rotten data that could have a big impact on the overall decisions versus data that's flat out broken. Right. So now we get to the where the sort of the quality checking of the models, right?

To to look at the outputs. And again, this is something that's automated, um, but it's something that uh as a human being, you can absolutely go and inspect. Um, what you're looking for essentially in these two charts here is when does the data substantially change? When does it reach a point where of these dozens of different candidate models inside this tool, uh, which is the best one? So you're looking for in this case here, where the pink line, the root mean squared error, uh, where that error gets as low as possible, and then you're looking at um R squared, which is the goodness of fit, uh, when that's high as possible.

So it's you know, this model here is the one that's machine will automatically choose because it says, hey, the green line is as high as it's gonna get, and the red line's as low as it's gonna get. It would never choose, for example, uh, one of the models early on in here, where it's like, okay, the red line's as high as it's gonna get, and the green lines as low as it's gonna get. That would be a model that has very low predictive power. So with that, uh, we finally actually get to the result. Yay, yay, and what does the result say?

Well, there's two different things that come out of the result. Uh, and we're gonna switch to full screen here. Um, you get a um a measure of variable importance, right? Um, and which is for for marketing mixed models are not super helpful, but when then you get uh what is called um the regression coefficients, and this is where uh this is where the the rubber hits the road. John, you want to explain a regression coefficient?

Uh as far as it, you know, from what I remember of R is if it's over.5, then that's considered statistically relevant, and anything below is basically not on marked. No, regression. So for R squared, uh that would be true. For regression coefficients, the regression coefficient is the change in the independent variable, um, and its effect on the dependent variable. So Katie's like, what's the measure of impact?

Right. So here's what this means. In a regression coefficient, it essentially is saying for every one of the outcome, um, this many of the thing is required. So for organic search, for if we want to um for every one um bit of traffic essentially in organic search will result in 0.13 of the outcome, right? That's sort of what a regression coefficient means.

The the longer bars going to the right mean those those variables have more impact. So this is now the actual marketing mix model. Um, so things that matter. Katie uh getting mentions, brand mentions on her Twitter account. Seems to matter with this model.

That's me. That's I matter. Woo! Organic search traffic matters, right? The uh impressions, the number of impressions that the trust insights posts get on LinkedIn matters.

Uh engagements uh likes specifically on John's tweets matter. And again, brand awareness mentions on my Twitter account matter. Um, the more we crank out of these things, the closer we will get towards convert the the outcome we care about, which is the HubSpot deals number. On the other side, when the bars go far to the left, it means those things matter very little. They they kind of have an almost uh uh a negative impact, right?

So uh you see total followers there on both uh from my Twitter account and the the company's Instagram account, that would effectively say to a CMO, you know, this would require a lot of messaging to explain this don't bother chasing followers, right? If you're saying that we need more followers, no. In fact, um depending on basically on this data, it looks like the quality of the followers on my Twitter account are are so irrelevant to the outcome that me spending time chasing followers would be would reduce the number of deals we get. Well, and you know, this is sort of the same thing we say about our newsletter numbers is I'm okay having a smaller number if it's the right number of people, if we're reaching the right audience, then it should be smaller. Uh, because we can't be everything to everyone.

And it's interesting how this model very much mimics the way that well, not interesting, but um, it's refreshing that the model mimics the way that we operate our content, where we're not trying to be everything to everyone, we're trying to reach the right people with the right information. Exactly. Um, other things that this model says, the number of impressions on TikTok really don't matter. Um not great. Now here is the gotcha.

Here's the gotcha. Um if you're sitting here thinking, okay, we we've got a marketing strategy, um, we we we know what to do. If I run this section of code here. Let's see, where am I? Oh, the wrong section.

Well, and to be fair, that wasn't a marketing strategy. That was just the initial data. There's still a lot of human intervention needed to put that together and to make a plan from that. And I think that that's the other piece of it. When you go back to the people in the five Ps, someone has to interpret this data and turn it into something actionable.

I would be surprised of any system that just hands you on a silver platter, here's exactly what you need to do. But this goes back to what John was saying earlier. The R squared on this model is 0.153. Which means the model is invalid. Oh, good.

That's helpful. So everything that you just saw does not in fact matter because the model has no statistical significance. Um, the R squared number effectively says how well the the regression line describes the data, describes the trend. In this case, it it describes it very poorly, right? Um generally speaking, you'd look at a minimum of 0.25.

What John said it was exactly right, is you're really looking for 0.5 or above. Right. At that point, you're saying this this model describes reality. So going back to what you were saying earlier, Katie, you know, does how do you know this describes reality? Well, you have a statistical measure that says this is reality.

And in for this particular model, this is not reality, does not meet that test. But this doesn't tell you why. And that's that becomes one of the challenges is it doesn't tell you why it's not statistically valid. Is it because of a lack of data? Is it the wrong data?

You know, whatever the thing is. And so without having that person or agency who can actually build and run the model for you, you're like, oh, I don't know, it gave me data. I can make decisions from this, can't I? And this is where, so generally speaking, this is a generalization with marketing mixed modeling. If you get a number that is um statistically invalid, it means that you're missing something, right?

There is missing data that would help describe reality better. So there's some aspect of our marketing, we've talked about brand mentions missing, public speaking is missing. Um, all those things are not in here. So what is in here, you can make a model, but it does not, it's it's missing those key ingredients. It's like you bake a cake, you taste it like okay, it looks like a cake, but it has no flavor, like, oh, because there was no salt, no sugar or anything in the in the recipe.

Like those key things are missing, and yet it looks like a cake. Um, you can put frosting on it, but it it's not gonna, it's it's not cake. Um, that's what's happened here. So we know now, looking at this based on that error message or that error metric, this isn't reality. So if you saw this and then you saw the R squared number, would you say, no, we can't make decisions based on this data because it is not statistically valid.

That's right. I would say this is this does not describe reality, right? So making decisions based on this would be a poor choice. Here's the problem. There are a lot of companies that offer a lot of software for marketing mix modeling, none of them tell you the statistical stuff happening on the back end.

So when it presents the answer, you don't know what if that actually is the answer or not, right? Because unless you say, show me the RMSE and the R squared for this model, and I can then judge for myself how good is this, right? How how how accurate is it? Um, but a lot of software packages, a lot of SaaS software packages will say, Here's the answer. And that's it.

And you're like, okay. And then you and no one says, Well, is it right? Well, and the other piece of data, the huge piece of data that's missing is uh any kind of uh costs from our side, our time, like time sheets, expenses, those kinds of things. And with a marketing mix model, that's really if you're if you're looking for where should I spend more time, where should I spend less time, where should I tune up and down budgets, you want to have that data included. And so you're gonna have to look in your time tracking systems, you're gonna have to look in your HR systems, you're gonna have to look in your ad systems and other soft dollar costs too.

And so that is not included here. Um, as we said, sort of in, you know, here's what we're covering. Let's just say for the sake of argument that that analysis that you showed was statistically valid. What kinds of decisions? I mean, I can sort of rattle some off, but I want to ask both of you, um, what kinds of decisions would you start to make, assuming that this out data was the correct data?

Yeah, but that you just march down from the top, right? Or you can go from the bottom. But the big one was, you know, look at the first three there that seem to be exponentially better than everything else that's out there, and just increase doing those. But like you said, you also layer price on top of that because you find you might find out that hey, the second one, you know, only takes four bucks to triple, and the other one, you know, we're not gonna get another Super Bowl ad. Um, and then cost can also help you prioritize.

I mean, actually, that's the right answer. If you can prioritize this by cost, that's the way to go. Because you look at the bottom four too. If pulling the plug on any of those is going to make a huge difference in your budget, that could make you know a world of difference in the stuff that actually works for you. Exactly.

So the John nailed it, there's there's three actions you take. You do more of what's at the top, uh, assuming that the costs work out, do less of what's at the bottom. And what's in the middle, that's kind of uh uh an iffy milgram. And by the way, this chart is truncated. This this table is 107 items long.

Um there's a whole bunch of metrics in the middle that don't seem to move the needle either way, they're not reducing performance, but they're not boosting performance. So those are the ones you look at and go, um, are we doing that wrong? Right? You know, are we doing something wrong there that the maybe the channel's valid and we're just we're just not good at that, right? You know, maybe we're just not good at TikTok.

We we just I can't dance, uh you know. Um and and so that's those were sort of the three big things you get from a market mixed model. What should we do more of? What should we do less of, and then what has potential, but maybe we're not approaching it the right way. No, and that makes sense.

And so, you know, uh hypothetically, if we were uh focusing a lot on getting that new followers through our social media accounts, that would be the first thing to get cut of all right. Uh, you know, we have a team of people who are you know trying to get followers and spending time on these, you know, influencer programs trying to figure out who to follow and who's gonna follow back, and you know, creating a lot of social posts of like if you follow me, I'll follow you, whatever the thing is, you know, that would be the first thing to be like, okay. Great, knock it off, because that's not helping us at all. So let's reallocate those resources into something more useful that is bringing us closer to that higher deal count, which as we go back to the user story, uh, you know, John, that you were sort of getting and Chris that you played out through the analysis, that's really what it comes down to is what are we doing that's gonna bring us to that higher deal count, that bigger revenue number, that more, you know, qualified leads in the pipe that someone like John can go chase and close uh the contracts on. That's really what this, for at least for us, is all about.

And so making sure we're not losing sight of that overall user story of as a CMO, I want to understand, you know, my digital marketing so that I know what to do more or do less of to get bigger deal counts. Yep. The other thing that's really important in here is if you look at those data sources, right? Those data sources are all over the map. We talked about that at the beginning of the show.

One of the things that, and then John John hit the nail on the head up with this, that's powerful about these models is that they can accommodate a lot more than digital marketing uh attribution models can, right? If you've got a billboard on 9995 and you have traffic, you know, you have you have up eastbound traffic uh or northbound traffic on that, that can go in here, right? So here's the number of impressions for that billboard, and then you have your billboard cost per day that can go in here too. Uh if you have uh radio spots on terrestrial radio and you know when that's uh when that's being played, you can incorporate that and build what's called an ad stock model for the those impressions on radio, um, where impressions are run by through a time decay. So, so like you have after a certain amount of number of impressions, you know, they lose their impact.

If you have a TV ad. So everything and anything that you can get data for can go into a marketing mix model. And as we found out with our model, there's some things that are based on the statistical invalidity of this. We're missing stuff. We are missing stuff that needs to be in here just to even make the model valid.

That's why these projects take so long, right? So for trust insights, yes, we slap this together very haphazardly to create a statistically invalid model in two days. Like no one's gonna say, oh, well, I would like that, right? Um to do this well would require us to now iterate. Um say, okay, well, what other data do we need?

What data, you know, are there other additional uh screening factors of stuff we need to take out of here? And this could this little even just for us uh again as that three-person company, this could take a good month, maybe two of seeing what's the big blind spot in this model that it makes it statistically invalid, and that's why a marketing mix modeling takes so long, and b, why it costs so much. Well, and making sure that you are not skipping over any part of the requirements gathering, and you know, we've talked about this in other episodes and pieces of content. It's the part that is skipped over because for a lot of people it's the boring part, they just want to get to the doing. But hopefully, we've demonstrated why requirements gathering is so essential, especially as you're dealing with multiple data sources and the way in which those data uh sources are brought into you know one central database, it's gonna be a mess.

Um, but with that, if you want help, you can contact our redis resident statistician John Wall, and he can tell you exactly how Trust Insights can help you put those pieces together. Um so, John, uh any stats facts that you want to leave us with before we uh close out this episode. Yeah, I'm gonna be doing a webinar on pre-internet statistics. We'll be talking about how to use a slide rule and the TI35 if you're cutting edge. So we're we're gonna get right to the heart of that.

That's perfect. I love it. How to do your marketing mix model after an EMP on paper. Get a big eraser. Love it.

I would I would show up for that. I'll be the goodbye dummy part of the show. All right. Um any final thoughts, Katie. I mean, I think we've covered it.

So if you want to hear more about what exactly a marketing mix model is, you can catch that episode of our podcast at TrustInsights.ai slash TI podcast. Um, and if you want to get more detailed information around using the five Ps to gather those requirements for a marketing mix model, uh, you can subscribe to our newsletter that came out this past Wednesday at TrustInsights.ai slash newsletter. Um and if you just have any general questions about marketing mix modeling um or want to share what you're working on, you can join our free Slack group at TrustInsights.ai slash analytics for marketers, where Chris and I and John are in there every day, you know, creating chaos and then trying to rein it back in. Exactly. I'll leave you with the with these thoughts.

Modeling is an iterative process. It's an it is almost never a one and done. It's something that even just to get a single model that you can make decisions with takes a lot of time. So if it's something that you're considering doing, if it's something you're being asked to do, go into it with those expectations that yeah, it's gonna take uh some effort to do, it's gonna take some mathematics to do. Uh, and if you don't have those resources and you don't have the ability to acquire them, then you have to let your stakeholders know.

Here's why this is probably not the best choice for us. Maybe we start off with something smaller, like you know, a standard digital attribution model until we get the revenue we need to pay for the the big model. Um, but expect it to take a while, expect it to take a few tries, uh, and and don't get discouraged if you run into all sorts of data quality landmines, uh, processed landmines and things. You may even run into, depending on on your company, you may even run into some people landmines where some people don't want their data included because it may not put the paint them in the best lights, which is always a delight. So thanks for tuning in.

We will see you all next time. Thanks for watching today. Be sure to subscribe to our show wherever you're watching it. For more resources and to learn more, check out the Trust Insights Podcast at TrustInsights.ai slash TI podcast. At our weekly email newsletter at trustinsights.ai slash newsletter.

Got questions about what you saw in today's episode? Join our free analytics for marketers Slack group at trustinsights.ai slash analytics for marketers. See you next time.