Mind Readings: AI Content Detectors Deep Dive Part 3

Welcome back. This is part three. We're going to put some AI content detectors to the test. First, let's talk about methodology. How would you do this if you are an institution?

The first thing, and I said this in a previous post is you have to decide what your acceptable rate of false accusation is. How permissible is it to falsely accuse a student, a faculty member, a member of the administration at your institution? What is the acceptable rate of false accusation? Because when you use AI content detectors, you are going to have false positives. Remember, good old statistics, type one and type two errors.

Type one error is a false positive, in this case, saying something was made by AI when it was not. And a false negative. Something was made by AI, but you detected as not being made by AI. What is your acceptable rate? Is it are you willing to falsely accuse 5% of your students?

1%, 60%? What is your what is your acceptable rate of false accusation? This is something that your ethics committee has to decide. I would argue that with this technology, because it is so flawed, the acceptable false positive rate is zero. If you're talking about ending someone's academic career, kicking them out of school or removing their revoking their tenure or uh firing them, or not hiring someone.

Those are very high-risk decisions. And as a result, the margin for error for false positives should be zero. Now, how do we test this? You're going to need some samples. You will need, I would argue, again, if you're testing systems, if you want to buy a system, you need to have a robust testing library, a library of four types of content.

Number one: samples of content known to be human-led with no AI use at all, typically from historical databases. Now, one of the cautions is that AI tools that from which models were made ingested many of these documents. So they will naturally have some statistical correlations in their outputs that can sound a little bit like pieces of those original documents. Something to be aware of. So that's set of your first set of samples.

Your second set of samples should be holdout data. This is known human-led content that you know to be 100% human. Typically, this would be papers or things written by humans, maybe prior to 2022, because there was no generative AI back then, that will have never been published, never been put online, never vacuumed up into the tools that AI makers, uh the database that AI makers use to make their models. This is critical data. So you you want unpublished research, unpublished papers, things that have set on a hard drive, private emails, whatever the data set is that you will hold someone accountable to.

You need examples that you can test services and say, we want to make sure that the service is good. We want to make sure there's no false positives, no false accusations, or whatever you your acceptable rate of false accusal is at your comp at your institution, using data that these models have not trained on, and therefore we know it's a fair test. Third, you need samples that are 100% AI-led, made by machine. You know they were made by machine, you made them by machine, and there's just absolutely no content, no, no question whatsoever. This was made by machines.

And fourth, you probably want some samples that are human-led, that are human-led but AI modified. So for example, if you were to take a transcript of uh something that you know for sure was human read, like I read something aloud. Wow, the sun is really bright. Um, and then you had AI clean up the grammar. That would be an example of something that's human made and and machine modified.

So you need four different samples to do this. Let's go ahead and get started. I'm gonna do what most people would do. I'm gonna Google for AI detector, right? And we've got uh one, two, three, four, five, the top five results for AI detectors.

Uh there's one, two, three, four, five. So this is great. Now I'm gonna start with a paper. This paper is called uh Supreme Court Cases and the Economy. It was written by my kid.

Um, and it was written, let me double check the date on this file. This file was written April 28th, 2022. Chat GPT, of course, came out in November of that year. So this file is guaranteed, guaranteed um free of AI, because no one had access to it. So let's go ahead and paste.

Actually, no, let's start, let's start with our known good. I'm gonna take Federalist Paper number eight by Alexander Hamilton as our first test candidate. We're gonna put a few paragraphs of the Federalist Paper number eight by Alexander Hamilton in here. Let's go ahead and go machine by machine, service by service, and we got 500 words, so it's a reasonably fair test. Uh zero GPT, 64% of Federalist Paper number eight, written in 1787, is AI generated.

So if your false positive rate uh is is zero, this tool fails. This is a false positive. There's no way, I don't care how good Lynn Manuel Memoranda is, there is no way Alex and used AI to write Federalist paper number eight, right? Not a chance. Okay, Quillbot says 0% is likely AI generated.

CopyLeaks just says gives it a thumbs up. That's not super helpful. Zero GPT says highly. And writer says it's 99% human generated content. Okay.

So Federalist Paper number eight passes. Let's try something a bit more sophisticated. This is an op-ed from the New York Times from night April 27th, 1924 about immigration legislation. And boy, it's amazing. A hundred years ago, we were still talking about the same things.

I'm going to take the text of this article. And this is again 1924. Let's go ahead and put this in. Oh, we can only scan 5,000 characters. That's fine.

Let's see how we're doing. So this one, zero GPT came back to 0% AI, 0% on this one. Says on human text, 0%, 100% human. So interesting, Federalist paper got a 99%, and writer got 100%. So that is round two.

Now we've used two papers that are almost certainly content that was in training databases. Let's now take my kid's paper. Let's see how we do. 72% AI. Cool.

So this would get my kid kicked out of school. Right? If you were using this particular deck detector, this one came out with 0%. This one came out AI content detected. Look at that.

This one came out 2%. Doesn't say it's not highlighting well which ones it is. This one came out 94%. So saying, hey, there's this probably some AI content in here. That's really bad.

Right? If we have a holdout of the type of content that someone would submit, this is a paper that my kid wrote. And it came out. Several detectors said, hey, you probably use AI and parts of this. That's really barely bad.

So if your false positive rate is 0%, most of these tools fail. You would want to keep doing this over and over and over again. Let's take another one just for fun. I'm gonna take this was a paper, natural rights in the constitution. This was written in 1920 uh 2021.

So a full year before chat GPT. Here we go. And remember, these documents have never been seen before online. These are my kids writing. 49% AI.

Okay, this one says 0%. This one says human text this time, 2% AI in there, and 95%. So again, across the board, this should be 100%. 100% across the board. Because there's no way my kid went back in time and went forward in time to do this.

Now, let's create some AI content that we because we need some AI content to test. Let's go back to our close this up here. Let's take our Supreme Court cases from the previous episode. Let's clean up the markdown because markdown is a dead giveaway that you've used AI and format it nicely. And so we end up with a document that looks like this.

Pretty straightforward. This is 100% AI. Let's go ahead and restart our detectors and let's see how we did. 98% AI. Okay, that one did a good job.

100% AI generated. You've reached your daily limit. 100% AI, 61% human generated. Right. So now this is in the territory the territory of false negatives, right?

So this is a case where a tool erred on the side of it's human when it really isn't. In a scenario where you have substantial risk of people being kicked out of school things, I'm okay with false negatives where someone did something with AI and it doesn't get caught. And as we've seen in just the examples in this episode, pretty clear that this falls, these tools fall down. If your if your tolerance for false accusal, which by the way is probably a lawsuit, uh, is zero, none of these tools pass. Let's look at two more examples of this.

The first we're gonna use, we are going to use a human-led document that is AI modified. And so what does that mean? That means I'm gonna take a transcript from a previous episode of mine. This is my these are my words. I spoke these words aloud.

And what I had a generative AI tool do was clean up my grammar, clean up my my words without rewriting it, right? So let's see how just AI use on human-led content performs. So isn't that interesting? So this uh was modified by AI, but it is obviously uh a transcript. This one came out zero percent.

This one came out 40% AI detected, which again, zero percent of it was made by AI. Um still hit my daily limit there. Um, this one says one percent probability, 96% generated. So again, kind of all over the all over the place. One final test would be a section from my book, and my book is the the Intelligence Revolution.

This is a book that is derived by AI from my original newsletters. However, I gave it all my newsletters and YouTube videos and things like that, and said, I want you to write as though you were me. Plagiarizing well, not plagiarism copying uh from me as as much as possible. Let's see how this does. Again, these it's this is derived from my words.

I'm probably gonna hit the daily limit on that one. That's okay. And let's see how we did here. So 92% AI generated, 100% AI generated for that one, daily limit, 76% AI generated there, 76% human generated. So here we are at the end of our tests.

All these tools fall down, all of them, without exception. They don't get things right. Why don't they get things right? Depends on how the tool is engineered, whether it's looking at basic text uh statistics like perplexity and burstiness, or whether they're using an AI model underneath that's been trained on some sample data, whatever the case may be, all of them fell down historical documents, on holdout data. The holdout data was the really important one because again, uh, these are if you're using this in an academic context, and I'm using my kids' papers from 2020 to 20 early 2022.

None of this could be AI generated, period. And it has never been seen by an AI model, so AI could not train on it and replicate any portion of it, and these tools still fell down. Um, they fell down on detecting AI content, and especially on um both slightly modified and substantially modified. So the key message here is do not use these tools to detect AI content. Period.

End of story, unless you are okay with falsely accusing someone and reaping the consequences from it, which will include things like lawsuits against you. Right? If you accuse someone and they take this video, which clearly demonstrates these tools are highly flawed, you're gonna lose that case and you're gonna end up paying a whole bunch of money. So do not use these tools. In part four, let's talk about building the term paper of the future.

And the process that you might want to teach a student on how to use AI to write a term paper. Stay tuned. If you enjoyed this video, please hit the like button. Subscribe to my channel if you haven't already, and if you want to know when new videos are available, hit the bell button to be notified as soon as new content is live.