You Ask, I Answer: Data Democratization and AI?

In today's episode, Jim asks, I am skeptical of data democratization because the average decision maker does not understand data collection, transformation, integration, etc. Doesn't AI make this an even bigger problem? So let's first uh quickly define data democratization is the ability for anybody to be able to work with data, hence the term democratization, just like new media, a podcasting and blogging was the democratization of media, the ability for anybody to make a podcast, anybody to make a blog and such. And data democratization has been something that has been on uh technology companies radar for a really long time. Um my whole start working with IBM uh years ago was around uh Watson analytics, the idea of the citizen analyst, as you know, the average person who could pick up uh some good tooling, uh data set, and generate some usable insights.

Didn't work out so well. Um and to Jim's point, the reason it didn't work out so well is because data analysis, even though it sounds simple, isn't it uh there are a lot of pitfalls, there are a lot of gotchas. There are a lot of things that are not taught well and poorly emphasized when people start working with data. Things like margins of error and uh statistical uh relevance, statistical validity. All these things are things that in a layperson's uh experimentation with with you know data and math and and statistics and such, don't really get covered.

Uh people have a tendency to approach things in a very naive fashion, which is normal. Um they pick up a tool, they run uh a basic analysis and say, aha, I've got the answer. You can tell, you can tell how expert somebody is in the realms of mathematics and statistics by how many shades of gray their answers come with, right? The beginning person is like, this is the answer. The expert person is like, well, you know, there's a whole bunch of reasons why this may or may not be the answer, and we're not really sure.

Um we can give you probabilities, we can give you uh a sense of reliability or not in the data, but for the most part, uh it's not as cut and dried as you think. Right? And those are the answers people hate. Uh people hate those answers. And that's why data democratization hasn't worked out so well, because there are a whole bunch of people who want the answer, and the answer doesn't exist.

There could be a range of answers. So, does AI make this worse? Well, it depends on how abstracted the decision maker or the AI tool user is from the tools themselves. Certainly, the pandemic in particular has shown us that the general population is completely incapable of parsing even basic scientific data, like why to wear a mask. Sorry.

It's true. Um people couldn't can't understand even the most basic scientific facts and and data points and make good decisions from them. So the question then is does AI make this worse or an even bigger problem? It depends on who's working on the AI. It depends on who's working on the models.

If you hand somebody a piece of fully polished software, a model of some kind, and they can't really tinker with the innards. They can only use it for its intended purpose. The likelihood that it goes off the rails is lower if it's good software than somebody, say, uh picking up Python and just trying to hand at copying and pasting code randomly from Stack Exchange. If AI is left in the hands of legitimate experts, it could potentially improve things. Um IBM Watson Studio is introducing bias detection at multiple levels in the process, from data intake uh to model to model drift, right?

And it puts a big old warning saying, like, hey, your models drifted, you know, more than six percent, or hey, your model's drifting on this protected class. I should probably uh, you know, not do that. And so in those instances where the the person who's working on the system um has to stay within the boundaries of a a well-trained system and has to obey the warnings that it gives, yes, AI could potentially improve our work and potentially reduce some of the problems that come with data democratization. On the other hand, if you get that, you know, six-week crash course uh person who uh you know took their six-week crash course and AI uh certificate, uh yeah, that could make things a lot worse because that person doesn't have the background in data science, doesn't have the background in stats and probability, probably. I don't know, it's a generalization, but probably doesn't have that level of background and that level of experience of just having models go off the rails.

Um without a mentor, without uh somebody more experienced to guide them, it could make things a lot worse. I was having a conversation with uh uh a founder of a startup a few weeks ago, who's talking about uh how there's you've got all these sophisticated models they're working on building, and uh uh you know, you got a a friend of his from college, uh, they just both graduated from college, you know, they're they're gonna be engineering these models and have some experience in it. And they said, Okay, tell me about how you're doing bias detection. Tell me about who who on the team has the most experience with ethics, you know, data ethics. And he's like, uh, okay, so this is gonna go off the rails pretty quickly.

I said that you the you need to be building into your product. Things like monitoring for model drift, things like looking for ethical problems, things like that would, you know, uh fail you on a basic uh ethics audit. And this was news to the person. So in that instance where you have a bunch of people who are inexperienced with AI trying to deploy it, yes. AI is gonna make those problems of you know data democratization even worse because these are people who don't know what they're doing.

On the other hand, you get somebody who has you know five, ten, fifteen years of experience in working with data sets, knowing when a data set is imbalanced, knowing when a uh you know, a P value has gone off the rails, knowing uh how to do uh a two-tailed uh T uh test. In those cases, the person who's building the system will probably do okay and will make a system that is hard for other people to screw up. So it really comes down to which is gonna deliver um a good outcome, a good long-term outcome. One of the dangers in AI, particularly around bias, is that a system with no constraints will perform better. It will do things that you do not want it doing, it will it'll behave in unethical ways, but it will in the short term deliver better results.

A longer term a system built for the long term will make trade-offs and say, like, yeah, we don't want bias on gender identity. We don't want bias on race. And the system will have to sacrifice some levels of performance, uh, the model's ability to generate top line performance in order to meet those competing objectives. But that's exactly the kind of caution and care and uh attention to detail that you want. So, will AI make data democratization worse?

Potentially. Will it make it better? Potentially. It all depends on who's steering the ship. How do we help people uh steer the ship better?

Give them that list of questions. The uh pick up Dr. Hillary Mason's free book, uh Ethics and Data Science. It's got a ton of great checklists in it about uh questions you should ask before starting any project uh with data. Uh ethical questions, process questions, accountability questions.

Those will help make all of us better data scientists, better data analysts, better uh AI engineers. And if you work in a company and you have a leadership role, hold your company accountable to a lot of those those benchmarks. Say we're going to adhere to these basic processes so that we generate outcomes that will not get us sued or get us failing in audit um of some kind. So, really good question. We could spend a lot of time on this.

If you have follow up questions, leave them in the comments box below. Subscribe to the YouTube channel and the newsletter. I'll talk to you soon. Take care. Solving your company's data, analytics, and digital marketing problems?

Visit Trustinsights.ai today and let us know how we can help you.