Mind Readings: AI Bill of Rights, Part 4: Notice and Explanation

Welcome to part four of our examination of the AI Bill of Rights published by the United States White House as proposed regulations for essentially the use of AI and the rights that people should have when it comes to the use of AI. Today we're going to talk about notice and explanation. So let's dig into this one. You should know that an automated system is being used and understand how and why it contributes to outcomes that impact you. Designers, developers, and deployers of automated systems should provide generally accessible plain language documentation, including clear descriptions of the overall system functioning and the role automation plays.

Notice that such systems are in use, the individual or the organization responsible for the system, and explanations of outcomes that are clear, timely, and accessible. Alright, so this one is pretty straightforward, right? If a system is being used that's automated, you should know how it's how it works, right? You should know what's in the box, and you should be able to say, let's let's perhaps not use this if it's not working, right? Or at the very least be able to explain the outcomes.

Let's look at um a couple of the examples that they they give in the paper. Number one, a lawyer representing an older client with disabilities who had been cut off from medicated funded home health care assistance couldn't determine why, especially since the decision went against historical access practices. In a court hearing, the lawyer learned from a witness that the state in which the older client lived had recently adopted a new algorithm to determine eligibility. The lack of a timely explanation made it harder to understand and contest the decision. A formal child welfare investigation is opened against a parent based on an algorithm and without the parent ever being notified that data is being collected and used as part of an algorithmic child maltreatment risk assessment.

The lack of notice or an explanation makes it harder for those performing children uh maltreatment assessments to validate the risk assessment and denies parents knowledge that could help them contest a decision. Number three, a predictive policing system claimed to identify individuals at the greatest risk to commit or become the victim of gun violence based on an automated analysis of social ties to gang members' criminal histories, previous experiences of gun violence, and other factors, and led to individuals being placed on a watch list with no explanation or public transparency regarding how the system came to its conclusions. Both police and the public deserve to understand why and how such a system is making these determinations. A system awarding benefits changed its criteria invisibly. Individuals were denied benefits due to data entry errors and other system flaws.

These flaws were only revealed when an explanation of the system was demanded and produced. The lack of an explanation made it harder for errors to be corrected in a timely manner. So this is about black boxes, right? As we use more and more sophisticated decision systems, as we use more and more sophisticated AI like deep neural networks, there's more and more that we don't understand about what's going on inside of the machine. And this is part of the reason why there's a major push towards interpretability and explainability.

In the context of AI, interpretability means you have the ability to look at the code that is at use and diagnose it line by line. Here's what this line of code does, here's what this line of code does, and so on and so forth. Explainability is looking at the outcome and being able to explain the outcome. Here's how the machine arrived at these conclusions. The challenge that people are running into right now, and that tech companies in particular are very resistant to go the interpretability route is that interpretability is dramatically more expensive for companies to do because deep learning systems, you can audit them, you know, layer by layer, but it's computationally very, very expensive to do so.

So you have a lot of big tech companies saying, no, no, no, explainability is all you need, which is not true because again, these deep neural networks are basically if you don't build interpretability in, they're just big black boxes, and you don't know how the system is making its decisions. All you know is whether this the decisions make sense or not. The classic example of this is that researchers trained an image recognition algorithm to differentiate um uh a wolf from a dog, right? And they fed it hundreds of photos uh of you know wolves and dogs, and the system performed really well in in theory, and then they started feeding it real life stuff, and it failed all over the place. And when someone went back and built interpretability into the system, like there was no way to explain the outcome, but when they built interpretability into the system, at again, considerable performance penalty, it turns out the system was not looking for dogs or wolves or ears or jaw shape or stuff, it was looking for snow.

If there was snow in the photo, it was a wolf, at least in the training data set. And so the decision system behind the scenes was making decisions based on a non-relevant factor. That's fine if you know, obviously, if you're building an image recognition system for wolves. That's not so fine if you're building systems that impact people's lives, even within marketing, right? Who you market to has an impact.

I was talking to an insurance company a number of years ago, and they were building a system to identify ideal customers. Their ideal customers to them were people of certain affluent means. And the the ugly reality in the United States of America is that money tends to also have a very high correlation with race. And as a result, the system they built, even though theoretically it was not discriminating on race, in practice, it absolutely was. And so they effectively invented redlining.

Another company I saw in at uh one of the Martech shows built a predictive algorithm for ideal best customers for Dunkin' Donuts. I don't know if Duncan was actually a client of theirs, they were just using it as a demo. But they showed us this map of the city of Boston, said here's all the red dots, those are the areas where your your ideal customers are, and here the black dots are where there aren't ideal customers. And I looked at this map and I said, You invented redlining again. And they're like, what's that?

Like, oh, for God's sakes. Um they were essentially using, I believe it was income spending patterns, but it also perfectly replicated the demographics of Boston. Uh areas like Mattapan, Roxbury, Dorchester had no ideal customers, right? Because they're predominantly black areas of the city. They are also lower income areas of the city, but they're predominantly black areas of the city.

Places like Cambridge, Somerville, Boston, uh, you know, the financial district, all ideal customers. Now, if you know anything about Dunkin' Donuts coffee, uh, the only people in Boston who don't drink Dunkin' Donuts are dead. Everybody else, regardless of race, ethnicity, any any protected class, everybody, a certain significant portion of the population for every demographic drinks Dunkin' Donuts, right? So their algorithm was flat out wrong. It was there was discriminatory and wrong, and there was no explanation of how it worked.

And that's what this principle of AI Bill of Rights is is really all about. It is about can you explain how your system is making decisions? Think about this. Go into your marketing automation system, right? Or your CRM as a marketer.

Do you know how the lead scoring system works? Can you explain it? Can you explain to somebody, yes, you have a lead score of this, you were selected for this, you received this email because of this. I have had, even in my own stuff, just my personal newsletter, I've had to go digging around in my own system to figure out why somebody was getting an email from me when they said they didn't want it. And I dug into it, and there have actually been some i alarming cases of um bots submitting other people's data.

We were looking at this one person who's based in Amsterdam, and there was what looked like bot traffic from um uh uh uh a server farm somewhere in the USA here that submitted their information at a at a time that they wouldn't have been online, you know, subscribing to my newsletter. And I can't go back and hunt down exactly where that went, but I have at least the IP log to trace uh it down. But if I hadn't been able to explain, if I hadn't been able to dig into my system, I could have been held liable for a violation of uh of you know uh an international law. That's the thing is that for us as marketers, we've got to understand our systems, we've got to know what systems are what they're doing, what decisions they're making. You know, how does somebody uh an A lead or B lead in our system, right?

Because you don't want to discriminate. If you are uh adhering to the United States uh Title IX laws, and your system is saying someone's a better lead than uh someone else because of a protected class factor, like what gender they are, you're breaking the law. Right? Uh, and that's that's going to get you in a whole bunch of trouble. So you've got to know what's going on in your systems, be able to explain it, defend it, and then uh if there's a problem, deal with it.

Deal with it. So this is a very good principle. Um requiring explainability, requiring interpretability of AI systems is essential. And again, big vendors are going to resist this like crazy because it is expensive to do. But the first lawsuit they lose, you know, for a billion dollars might convince them otherwise.

So there may be some change on that front. But to protect yourself, know how your systems work, know how your vendor systems work, uh, require transparency from them, require technical details from them. If they're unwilling to provide those details, you may have to change vendors. Your legal department and your finance department certainly will advise that you change vendors if it is uh creating substantial risk to your company. So be aware of those risks as well.

In the uh in the final section of the AI Bill of Rights, um, we are going to talk about uh human alternatives. So stay tuned for that. If you like this video, go ahead and hit that subscribe button.