Gini asks, “OK! I have questions! Let’s assume the data is a mess or “dirty”, how can a communicator or marketer figure out what’s missing, where the holes are, or why something isn’t working?”
This is a process known as exploratory data analysis (EDA), and it’s a formal discipline within data science. Learn what EDA is, the steps involved in the process, what software data scientists typically use, and why marketers shouldn’t attempt to go it alone when doing rigorous statistical analysis. Watch the video for full details.
Can’t see anything? Watch it on YouTube here.
Listen to the audio here:
- Got a question for You Ask, I’ll Answer? Submit it here!
- Subscribe to my weekly newsletter for more useful marketing tips.
- Find older episodes of You Ask, I Answer on my YouTube channel.
- Need help with your company’s data and analytics? Let me know!
Machine-Generated Transcript
What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for watching the video.
In today’s episode, Ginny asks, I have questions. Let’s assume that data work exploring is a mess or dirty, how can a marketer figure out what’s missing where the holes are or why something isn’t working. So this is a process and it’s actually entire discipline is a part of data science is called exploratory data analysis. And in exploratory data analysis, what you were doing is you are taking a look at your data using data science tools to understand
what’s wrong with the data if there is anything wrong and also features and facts about your data. There are a number of packages within data science software. So if you use the statistical programming language, our or Python for example, there are entire libraries and packages that plugin to these the
The software that does what’s called eta exploratory data analysis and can automate some of it generally speaking there’s going to be
five or six steps to the ETA process. The first is you got to get the data in one place and some things like missing data very easy to spot because they will actually show up is not available
blanks within the data set and if your data set has a bunch of holes in it you’ll know really quickly you look at the types of data that are in your data set to see if they are consistent so when you load data in in our for example it will come in and are will do its best guests to say like this is character data this is numeric data, this these integers these are dates and if it comes in and it looks wrong you look at the data types to meet you go ooze, there’s there’s something not clean and there that’s throwing it off that it doesn’t know what to do software like our for example will default to text if it hasn’t if there’s numbers and letters and
Characters all mixed together. So real simple example of looking at your data types. The third step is to look at measures of what’s called central tendency. So you look at the mean the average the meaning and the mode, and you look up this in all the different columns and things and you look for you look for oddities, you look for things that don’t make sense. You look for one measure being substantially different than that could tell you that there’s a lot of outliers that there’s a lot of garbage in anything quantitative. Your fourth step is going to be measuring what it’s called dispersion, which is looking at things like ranges, standard deviation, the sigma is the probabilities of distributions and variances
in any kind of normal distribution you’re looking for, like a bell curve. So you’re looking for also things like skew newness, where, like, does the bell curve kind of squished in one end it doesn’t the variables not distributed evenly and then
You’re going to do a lot of visualization of that data
box plots and bar plots and scanner plots and, and all these things to look at your data to represent it and say, okay, does this data set look, okay? Does this data set look healthy. And there are plenty of cases where it won’t be when marketers are looking at their data,
figuring out those missing values is an important part of this process. And you’ll probably end up doing some form of what’s called amputation to try and restore the data if there’s not too much missing. So like, if less than 5% of the data says missing, you can use statistical technology to restore machines. best guess at what those those data points were assuming a normal distribution
there, you will look for anomalies to so you’ll see a database like 10 visitors, 10 visitors, 10 visits, 100 visitors, 10 visitors like what happened there and you either have to be able to account for the anomalies and say like, yep, that was the day that
A Russian Twitter bot attacked us or you may have to throw those anomalies out something that a lot of folks do when they’re doing data set analysis built, trim off anything outside of like three sigma say, okay, anything that’s that far off the standard deviation is clearly just a weird anomaly. Let’s discard it. And again, there are statistical packages within our Twitter actually has a couple of anomaly detection, breakout detection libraries that say, Yep, that’s an anomaly. You can see if we throw about or that’s a that’s a breakouts, a trend changed, and he can’t throw it out because the something a change happened and the change stuck. So
here’s the catch to Jenny’s question. Your
average marketer your average communicator is not going to be able to do this on assistant. A lot of this requires either very good statistical software
well it requires
Vegas statistical software, but it also requires actual data science and statistical knowledge. This is not something that can you just pour the data set in. And a nice clean data set pops out the other side. The for that first step about domain knowledge is so important. And here’s the gap. Here’s the opportunity for for savvy marketers. Your average marketer is not a good quantitative person, your average data scientist is not a good marketer. And so there’s these two gaps this this chasm between these two domains of excellence domains of expertise, and someone needs to sit in the middle of that gap and bridge that gap and say, Okay, I can talk to the marketer and understand where they got the data and and what their goals are with this data. And I can talk to the data scientist and say, okay, you process the data, here’s the outcome that we’re trying to get. So you can discard, you can safely discard X, Y, and Z variables because we don’t need them for this goal. And that person who sits
In the middle, we call them calling the marketing technologists for a number of years now. But it’s someone who’s who is a translator between the two disciplines who can help the data scientist understand the marketers needs and the mark, help the marketer understand what the data scientist needs to be able to do their job.
A marketer is probably not going to be able to do this on their own
looking at a data set, they’re probably not going to be able to ascertain anything other than like the basics. Of course, they can do the basics like yeah, there’s a bunch of columns here that are zeros or empty,
but anything beyond that the most obvious things to repair you’re going to need help with Now,
what’s changing is that there are a number of tools in the marketplace that are beginning to advance that are are doing some of this cleaning for you. And I emphasized that it is some of this cleaning because there is still no substitute for that domain expertise within data science there is there are tools like Watson studio that
Make the importation and cleaning easier or and can automate common obvious mistakes. But at the end of the day, you still need that human, several humans, the marketer and the data scientists working together to understand what’s an anomaly or a bug or mistake versus what’s, nope, there’s a real there that we need. They’re there that we need to investigate. And that is the hardest part. So great question.
marketers need to develop some level statistical proficiency. Because data scientists are in such demand right now, that’s unlikely data scientists going to become a marketer on the side, it’s probably not going to happen. So marketers need to begin developing those statistical mathematical and data science skills in order to make the most of their data if they don’t want to outsource it to someone else. Now, if you work in a large institution, you have a data science team on staff, there’s a good chance you could at least buy them a beer and ask them your questions.
If you work at a mid sized or small organization looks your agency partners to see if they have data science capabilities. And if they don’t. Well,
shameless plug my company trust insights does that so we’re happy to help. But most of all,
be aware that
one of my martial arts teachers calls it you’re reaching for something that isn’t there. don’t reach for something that isn’t there. Meaning if you know you’re not good at quantitative analysis, you know, you’re not good at statistics. Don’t try to fake it yet some help get some help to fix the problem so that it’s done right. So great question. We could spend hours talking about data quality and things and maybe we’ll do that in a webinar or something. But in the short term, pair up with a data scientist and explore your data together. As always, please subscribe to the YouTube channel on the newsletter and I’ll talk to you soon What help solving your company’s data analytics and digital marketing problems. This is trust insights.ai today.
Let us know how we can help you.
You might also enjoy:
- Almost Timely News, January 7, 2024: Should You Buy a Custom GPT?
- Mind Readings: You Need Passwords for Life in the Age of Generative AI Fraud
- Almost Timely News, January 28, 2024: Copyright Must NEVER Apply to AI-Made Works
- You Ask, I Answer: AI Works And Copyright?
- You Ask, I Answer: Retrieval Augmented Generation for Tax Law?
Want to read more like this from Christopher Penn? Get updates here:
 Take my Generative AI for Marketers course! |
Leave a Reply