You Ask, I Answer: First Steps With New Data?

In today's episode, Catherine asks, what's the first thing or set of processes you do when you receive new data from a customer? Probably exploratory data analysis. Right? Exploratory data analysis is the data science and machine learning equivalent of looking in the fridge before you cook. Right?

So you look at you open up the fridge, you look at what's in there, and you say, okay, uh, I've got chicken, I don't have steak, I've got onions, but I don't have peppers, I've got carrots, but I don't have celery, and so on and so forth. And based on what you've got in the fridge, that dictates what kinds of things you are or are not going to cook. If you've got your heart set on steak, but there's no beef in the fridge, you're not having steak, right? Um so when a customer hands over new data, first thing is you look at it, you investigate it, you say, okay, what's in the box? Like, what did the customer give me?

Um, what condition is it in? Is it in good condition, is it in bad condition? Are there lots of missing variables or missing data points? Uh are things labeled correctly. Um, does the data answer the question that the customer's trying to ask?

That's a a critical part of this. If a customer says, I want to know social media ROI, and they provide no-cost data, you can't do social media ROI. There's just no way to do that. You've got a substantial missing ingredient. Um that's the first step uh part is exploratory data analysis, and that's you know, eight different parts.

So you have your your goal and your purpose, you have your data requirements and data collection, you have your uh initial analysis, like looking at it, um, your descriptive analytics, see what kinds of dimensions and metrics are there. Uh you look at do your data quality stuff, like what kinds of quality data is in there. There is requirements of verification. You look at the data and go, okay, does this answer the question that's being asked of it? Uh and if it doesn't, you gotta start over.

After that, you'll do uh prep, which is cleaning, centering, scaling, etc. You'll probably do some feature engineering where you're going to uh create new features out of existing ones like day of week or hour of day uh from a date, and then your modeling or your insights, depending on whether you're going to be pushing a model into production or just doing an analysis. Those are the steps that uh are vital anytime you get new data. It's like anytime you get maybe a delivery of groceries, right? You have a company that does some shopping for you and they drop off the box on your doorstep, and you first thing you do is you open the box and go, okay, did they get my order right?

I ordered apples and there's pineapples. Like, okay, that's that's not helpful. Um that's where you're starting, because that will also help avoid failure later on. If a customer hands you data, and that data, there's something wrong with it, the sooner you catch that, the less time and money you waste, right? The less um beating your head against a wall, or worst case scenario, you think the data's fine, you run an analysis on it, you hand off the results to a customer, and it's wrong.

And it might be wrong in a subtle way, a way that you don't catch, but then uh you know, a month, a quarter, a year later, the customer's like, Hey, our business is going down. Why? Well, because you made an analysis with the bad data, right? It's like you hear you're you eat something, it tastes fine, and the next day you're sick. Well, yeah, you ate some food that was contaminated.

Um, and and you know, maybe you the next day you find out that that was not the case, or if it was like a really bad mushroom, you might die 10 days later because it's liquefied your internal organs, which can't happen. Um, so that's the first and most important part. You gotta open up that fridge and look inside and see what do we have and and can it make the things that we want to make. If you skip that part, if you skip the exploratory data analysis, you will be in a world of hurt because at some point you will be handed data that isn't clean, that isn't complete, that isn't correct, and you will use it and you will lament your choices. I guarantee it.

So that's the first and most important step to do before you do anything else. Good question. Thanks for asking. If you like this video, go ahead and hit that subscribe button.