You Ask, I Answer: How to Analyze Business Data?

In today's episode, Andrea asks, what is your go-to strategy to analyze your business data? The strategy is generally the same. It's a defined process regardless of what kind of business data you're looking at: marketing data, sales data, finance data, and so on and so forth. And it is a many, many, many step process that uh in the data science world is called exploratory data analysis. And what that means is exactly what it sounds like.

Explore the data to find out what's in the box. Like what are the things that are in this data set that we need to know about in order to make decisions about its quality? Can we use this thing? Is the data any good? And so on and so forth.

So let's look at uh those steps. Number one, what is your goal and strategy for the data? Uh and this is specific to the data itself. Do you have a data strategy around this data? Do you know what you're going to use it for?

Do you know why you have it? Do you know how you got it? That part is vital because there's this thing called lineage, uh, which is part of governance that is essential for understanding where the data came from and being able to, in an audit, document where you got it from and uh and that it was uh it maintained its integrity throughout your pipeline. Second step, of course, is data collection, gotta get the data from somewhere. The third step is attribute classification.

What have you got in this data? What are the dimensions? What are the metrics? What is the categorical continuous variables? Uh, and what are the data types?

If you think about something like a SQL database, a SQL database can have many, many different types of data, variable character length fields, numeric fields, integers, floating point fields, uh can have binary fields, large text blocks. So, what is what is the stuff, right? What what kind of data did you get? The next step is uh once you know what you have, uh doing what's called univariate analysis, and that means when you're doing basic statistics on uh the fields that you've got in this data. So for numerical fields, what's the average or the mean?

What's the median and central tendency, first, second, third quartiles? Um, what are the the summaries of uh uh numeric fields for non-numeric fields? What are the width or length of the fields? Um, what are the uh types of fields that they are, uh, the number of counts, number of distinct entities? If you get a bunch of, for example, first names, how many times is the name Julie appear uh within that field?

So you have to do this univariate analysis and create and collect statistics about the different types of data that you have. You may also at this point do some sorting just to get some a rough sense of you know what are the most prominent features. The second step is multivariate analysis where you start looking at relationships between fields. Um for example, uh you have first names, you have last names. Do certain first names go with certain last names, you have um birth dates, are there certain years, months, days that um have relationships to certain types of names?

So multivariate analysis lets you understand uh the different types of relationships in a data set for something like for example, commu uh uh consumer finance, you could look at uh say complaints. What are the uh the the strength of the complaints? How long are they, and is there a relationship between the length of the complaint and uh the strength of the complaint? Do people who are angrier leave longer complaints, for example? So that's a multivariate analysis.

After that, you have to do data quality analysis. What's missing? What's what things have errors, what things uh have values that are outside the bounds of what you would expect. Do you need to do uh any kind of data preparation? Um this will help you answer that question.

After that comes uh anomaly and breakout analysis, uh, which is okay, say the data that you do have. Are there weird spikes in it? Are there trends? Are there trends and changes? Uh, especially if there's time involved in the time series.

Um, so anomaly and breakout detection, very important. Then you get to a really important part, uh, which is called data requirement verification. In the goal and strategy section, you set out what the data was for, why you have it, and how you're gonna use it, and and where it came from. Data requirement verification. Now that you've an analyze the data set, you know what's in the box, you have to very be very thorough in matching up.

Did we get what we needed? Or are there pieces missing? Are there pieces that do not meet the requirements of the project? And this is a part that uh many, many many folks skip over. I have skipped over it in the past, and I have paid the price for it by uh not being able to do the work that I needed to do.

So that's super super important. Um the next step is if as long as you've met the requirements and the data is in good con uh condition, you're gonna do uh a whole bunch of data preparation, so centering and scaling the data if you need to, removing anomalies, uh cleaning up the data essentially. After that, you do a step called feature extraction, uh, where you pull out attributes from the data to make new types of data, uh, especially if you're gonna be using this data for uh any kind of machine learning. And feature extraction is is making new fields from existing fields. So, for example, if you have a date that when a an online form is filled out, you can extract at the day of the week, the week of the month, the month of the year, the quarter of the year.

You can st uh the day of the quarter, and so on and so forth. You can create new features from that because those features may be useful for modeling. At this point, you may also choose to do feature augmentation. Feature augmentation occurs when you've done the requirements gathering and you verified that the data meets those requirements, but you know through your own domain expertise, there might be additional data from third parties that could supplement, I could add new things, even though they're not part of the original requirement, they could add value. So you may uh choose to use uh augmentation again.

That requires a lot of domain expertise. If you're looking at I don't know, cardiac data, um what are the things that as a cardiologist you wouldn't you know other data exists out there? Someone like me who is not a medical professional would not know how to do that. I would know that there might be other data out there, but I wouldn't necessarily know what it is or where to go looking for it. The next step is feature selection.

Great, you've made this huge data set with a thousand columns, right? And you've extracted and augmented this thing, and it's super huge and robust. Now you need to remove anything irrelevant. So feature selection is what features are important. If you're doing any kind of supervised learning or uh looking at uh trying to analyze a specific outcome, say like uh you know customer value.

Are there features that simply have no mathematical relationship to customer value? If so, you can drop those. And finally, you would do either modeling or other applications of your data where you've done the analysis, you understand this data left, right, up down. Now, what are you gonna do with it? Are you going to uh take it right from this point as insights and and uh or generate insights from it uh manually?

Are you gonna turn it over to machines for further analysis? Uh, what are you going to do with the data? So that's the strategy for analyzing business data. It is extensive, it it requires a lot of time and it requires domain expertise. So, this is not something that you can do quickly, generally speaking, not if you want to do it well, and it's not something that you can do cheaply.

Again, not if you want to do it well. Now, if this is data that you're just doing I don't know, making a ridiculous like Instagram meme out of, you don't have to spend you know months to do that. But if you're gonna be betting your business health or your customers' health or wealth on this data, you'd better do this exploratory data analysis process really really really well, and that takes time and money. So, great question, Andrea. A complicated question.

You have to do it as well as you possibly can within the the limitations that you're given uh for the data and for the overall business. Uh, if you have follow up questions, please leave them in the comment box below. As always, please subscribe to the YouTube channel and the newsletter. I'll talk to you soon. What help solving your company's data analytics and digital marketing problems?

Visit trust insights.ai today and let us know how we can help you.