Mind Readings: AI Detectors are a Joke

In today's episode, let's talk about AI detectors, AI detection. Let's go ahead and bring up the most advanced and reliable chat GPT predictor in the world. According to their own marketing. Let's put in some text and see how well it does detecting AI. Remember, this is not a plagiarism detector.

This is an AI detector. Was this text generated by AI? Plagiarism is different. Detect the text. 96% AI generated.

Wow. That is incredible. What is the text? Yeah. The Declaration of Independence.

Which was made, what, 246 years before Chat GPT? Yeah. AI detectors are a joke. They're they're garbage. They're they're less than a coin flip in accuracy.

This is a very confident assessment from the self-proclaimed most advanced and reliable AI detector in the market. That's 246 years before chat GPT, I put in a document that we know was human written. Now, either space aliens or Skynet or somebody sent a terminator back in time to write the declaration of events, or this thing is just broken. It is just broken. Why does this happen?

Why are we getting this result? Well, a couple reasons. Number one, AI detectors, and this is true of most of them on the market, use a relatively small number of metrics. Two of them that are pretty well known. One's called perplexity, the other's called burstiness, to judge whether a document is AI generated or not.

Perplexity is the variance, the number of variances in language, the more uh the more variety you have in your language, the less likely it is that the tool thinks your your text is written by AI and burstiness, which is things like line length or paragraph length to see uh if everything is just even lines, the same number of lines over and over again, the tool thinks that it is AI generated. Now, why would it think that the declaration of independence, which is a very obviously um machine not machine generated, why would it think that? Well, if we look at the actual text, just looking at this huge section here where we list all of the complaints of the things that we did not like uh England doing to America back then, you can see there's a lot of very similar line length. That's gonna pick that up. You're gonna see a lot of similar words and similar tokens, right?

When you look at all these sentences, you know, he has done this, he has done this, he has done this over and over and over again. That's gonna get flagged. That's gonna get flagged by an AI detector because an AI detector is gonna say there's a lot of repetition here. There's a lot of chunks of text that are very, very similar, and therefore this must be generated by AI. No, it's a very specific kind of document.

The second and lesser reason is that a lot of these detectors either use very primitive metrics to begin with and have very, very straightforward fast code that is fast and cheap for them, or they're using very small language models that are again, very fast and and and cheap for them. And as a result, they're not very capable. They're they're kind of dumb. And so they're gonna look for there, they may have been given some examples and said, look for things like this. This is clearly they would give it uh things like purely generated text and say just learn the general characteristics that will learn things like perplexity and burstiness.

That's all worthless. All of it is worthless. AI detectors are worthless. They are garbage. Show this example.

Show this video, show the run the test yourself. When someone is loudly proclaiming, oh, you use AI to generate that. I posted this on LinkedIn originally, and the amount of feedback that I got from it, and the you know, a lot and a lot left feedback about this, saying how their kid was suspended from school or put on academic probation because a school was using one of these tools as though it were gospel and saying, Oh, well, you you you know violated our academic integrity standards. No, the tools are garbage. Stop using them.

Educational institutions should stop using them. Period. End of sentence. They're worthless. I saw another comment on the LinkedIn post.

Someone saying they were turned down for a job because they submitted their resume and cover letter and you know the hiring manager was using a uh a detector and says, oh, well, your application uh was AI generated, clearly you're not real, we're gonna we're gonna not extend a job offer to you. Again, that has real world consequences from someone who's using a tool who thinks it's good, but it's not, it's garbage. If you are a parent, use this example, please, to show your school board and your school administration just how garbage, how worthless these tools are. If you are at a company and the AHR person or the hiring manager's trying to use one of these tools, tell them to stop it immediately because they're worthless. They're no good.

They're less than a coin flip in accuracy. You want to judge whether something's AI or not, flip a coin. Are there giveaways that something has been generated by AI? Yes, but they're fewer and fewer every day as models advance as they get better, as they get smarter, as they're trained and tuned better. And the big tech companies that make AI models have way, way more budget and technical capabilities and people than people making the AI detectors.

So the models are accelerating far faster than their the detection abilities. What's the solution? Well, two things. Number one, does AI detection matter? Does it really matter whether the student wrote the paper or whether they wrote a good prompt to generate the paper?

And if it matters that the student wrote the paper, you ask yourself why. Because you want them to be able to think, writing is thinking. That's fair. There are other ways to measure that. Ask students live in a class.

Let's let's debate a point and see how much background knowledge you remember. We're not going to bother with term papers. It's a waste of everyone's time. Machines can write better term papers than humans. So we're going to have we'll do things like Socratic debate.

Or one example from Framingham State University near my house. Um they have students on purpose write papers with chat GPT. And then the class in small groups critiques the paper and says, What's wrong with it? What did it get wrong? What did it miss?

What could it have done better? What is an overgeneralization? And again, that reinforces that critical thinking and that gets at what you're trying to teach, which is you want students to think that's powerful. A second use case where, yeah, AI or not AI might matter, is on copyright. If you are trying, if you want to assert copyright over a piece of content, you have to be able to demonstrate that it was human-led.

Um, purely machine-generated content can't be copyrighted. Well, how do you do that? You show the lineage and provenance of the content, you show how it was made, you keep records, you keep an audit trail to say, like, here's how I'm making this piece of content. If you absolutely positively have to prove that I made this piece of content, you you keep an audit trail. I do some work for a client, I use AI as part of the process.

But it starts with me doing a voice memo, you know, for 15 minutes of me foaming at the mouth and recording it and then getting it transcribed, and then put applying the client style guide to it with AI things and giving them the final product. I can show them the chain of evidence from beginning to end, the original voice recordings and things, so I can say, yes, this is machine managed, but human-led. And because it's a derivative work of my original, which was human-led, it retains copyright, and therefore I can assign the copyright to the client under the terms of the agreement. So that's a way if you know where copyright matters. Like using it to disqualify someone from a job because they use AI in their resume or CV is stupid.

That is a dumb application of AI. Why? Because you want maximum productivity from your employees, and if that means that an employee is smartly using AI, penalizing them for that is stupid. That's a dumb thing to do. Unless you specify in the job description, this is a role in which you are prohibited from the use of AI in the job description itself.

If you put that in the job description, then yeah, you could theoretically make an argument. But again, there's ways to do this stuff that don't involve using broken, incompetent detection tools. If you are currently using or considering using, or you know a colleague who is using one of these AI detector tools, please show them this, share this with them, persuade them that these tools are worthless. They are garbage, they are unreliable, and they should under no circumstances be used for anything that is actually important, such as a student's academic standing or being hired for a job. This is your public service message.

Thanks for tuning in. Talk to you next time. If you enjoyed this video, please hit the like button. Subscribe to my channel if you haven't already. And if you want to know when new videos are available, hit the bell button to be notified as soon as new content is live.