You Ask, I Answer: Scientific Method for Marketing Data Science?

In today's episode, Jessica asks, what is the most common scientific method to analyze data so that when I, a business person is working with data scientists, I can have an intelligent conversation. To my knowledge, there really is only one scientific method, which is you develop a question, uh, you define the problem and the data, you formulate a hypothesis, you create a test, you run the test, collect the data, analyze the results, you refine your hypothesis or throw it out, uh, and then you observe uh and uh repeat the process. Now, all that said, the application of the scientific method is where things differ uh from traditional uh science a little bit, not a lot, but a little bit. Let's say you're uh let's say you're testing a new vaccine, right? For uh for uh say a coronavirus.

You would have a question does this vaccine work, right? Does it create antibodies? You would uh define the parameters, you would do the formulation, uh, and you would run the test. Where data science is slightly different is you still have the question you want answered, but in the problem definition itself, that's where you're going to do a lot of what's called exploratory data analysis, and that is to understand the problem better, to define it better, uh, to experiment a little bit, not a lot, but a little bit, to analyze the data set itself uh if you have it, and to do a lot of of uh refinement to it, the cleaning of the data, etc., so that you can formulate a hypothesis and understand what it is that you want to ask and and define the parameters of the test. Let's say you want to know uh let's say you want to know the impact of Twitter on your lead generation.

That's a good question, right? I would you know what is Twitter's impact on my lead generation? What data you're gonna need? You'll need Google Analytics data, probably you'll need Twitter data, and you have a hypothesis, like you hypothesize that you tweet, if you tweet more, your conversions will go up, maybe by a certain amount. How much is that amount?

Do you know? Um this is where you take that step back into the defined stage of the process and go, okay, let's look at my Twitter data, let's look at my Google Analytics data. Is there a mathematical relationship there? Is there even a there there before we set up a test? Before we create a uh a hypothesis that is testable.

Are we even barking up the right tree? So you might run like a regression analysis and see if there is a mathematical association between the two, because without an association, there probably isn't going to be uh a causation. And you would explore your Twitter's data. Does Twitter give you enough data to build uh uh a good hypothesis? Or conversely, uh, does Twitter give you so much data that you're not even sure what to test, right?

Is it length of tweet? Is it the day and time of a tweet? Is it uh a certain emotion or sentiment or tone that has been working for you, images in the post, video in the post? What uh what kinds of stuff do you have access to? And then what kinds of things actually matter?

And this is where you would run things like feature selection or feature importance, uh, or uh prediction uh predict predictor strength on your Twitter data combined with Google Analytics data. So there is a fair amount in that defined stage that to our discredit, we tend to f to gloss over. Like, oh, just define the problem. There's a lot that actually goes into that. And that that it really is exploratory data analysis, which is almost a discipline unto itself of being able to explore data and understand this is what's in this thing, this is what's in the box.

Or, and this is where your subject matter expertise really is important, and your uh analytical expertise is really important, what happens when you see an association? Like, yes, it looks like Twitter data has a correlation to Google analytics conversions, but then when you run regression tests and things to try and f and isolate what are the most important features, you come up with nothing. You're like, I just tested a whole bunch of things. What what happened here? There's a there's an association, so there should be correlative variables that contribute to it, but none of these show any kind of of statistically valid predictor strength.

What happened? And so that at that point, the experienced marketing data scientist would say, Okay, what am I missing? I've got data, but I've got no statistical relevance. What's not in the box? What what else do I need to go and get?

And that's where uh you'll find your biggest challenges. Because it's tempting to just run the test and say, uh, okay, here's the the top thing. But if the top thing is, you know, a point zero eight, and you're looking for 0.25 or better for um some of these multiple regression tests, you'll be like, hmm, what do I do? The very junior or naive data scientist says, well, I'll just take the top three. That's good enough, right?

That's the the algorithm spit that's good enough, but it's not the case. That is very rarely ever the case. I'm trying to think of a situation where that is the case, and I'm not coming up with anything. And so for what we want to do as people who want to become experienced marketing data scientists, we have to say, okay, well, there's clearly something else that's missing. There's a variable that is we don't have that would glue these two data sets together, or a combination of variables.

On the flip side of that, you get things called like uh conflated variables, where there's something that is uh mixing the two up and creating stronger signal strength than there actually should be. That typically happens with highly correlated variables mixed together. So if you uh simple example, if it turns out that the length of a tweet is important, and you have the number of characters in the tweet and the number of words in a tweet, and that goes into your your algorithm to determine strength, that's gonna screw things up because those two are perfectly correlated, and it's gonna create an uh an unnecessary signal for the algorithms that the regression algorithms use to say, like, oh yes, this is an important contributor to the outcome. You again, as as a more experienced data scientist, you would look at that and go, okay, well, we've got to get rid of one of the correlates here because they both can't be in there. They're but there's they're gonna throw a wrench into the into the computation.

So in Twitter's case, again, you would use your social media expertise. Twitter counts at the character level. Right? You have two hundred and eighty characters to work with. Okay, so if that's the case, let's get rid of number of words in the tweet and just stick with the characters in the tweet, and that will be a better measure of uh whether the length of a tweet is relevant to the outcomes that we care about.

So to sum up, there is only one scientific method that I know of. Uh but it is the implementation that matters the most and how you do it within data science. Because there's a lot that goes into defining the problem. You will spend a lot of time, you should spend a lot of time defining the problem. If you don't, if you immediately jump into a hypothesis and you immediately jump into running a test, chances are something has gone missing.

Something has been omitted that will come back to bite you in the end. I can virtually guarantee that. Something will come back to haunt you, and you will not you will not enjoy the process of having to redo the experiment from scratch. Really good question. We could spend a whole lot of time on this, but that's a a good uh starting point.

If you have follow-up questions, leave them in the comments box below. Subscribe to the YouTube channel and the newsletter. I'll talk to you soon. Take care.