Mind Readings: Generative AI Evades Detection Better

In today's episode, Carol said uh said, not really question. Um, she feels insecure about AI and its capabilities and how fast it's progressing and um about detecting uh AI generated content seems to be getting harder. Yeah, there's a good reason for that feeling. It's it's valid, it's a valid feeling, it's warranted. It's true.

Let's look at a couple of examples. A lot of the assumptions people have about AI-generated written word pieces are based on OpenAIS model. Um GPT for Omni, the one that powers Chat GPT. Every model, whether it's Claude or Chat GPT or Gemini, uh, they write differently. They have their own inherent style, and and and that they will write in that style unless you give it instructions to the opposite.

Um, for example, ChatGPT loves the word delve. Uh Claude, not as much. Uh Gemini likes Delve a fair amount. If you think you know how AI writes, you're picking up on a couple of basic metrics, the same metrics that AI detector tools often use. They're called perplexity and burstiness.

Perplexity measures how diverse the language is. A model that is uh poor quality or poorly prompted will rely on a small base of knowledge and creates very generic writing with high probability words that are common and boring. And so a a piece of text that has high perplexity is a piece has a lot of linguistic variety. So if you go to open up um uh a really well-written piece of fiction, there'd be a wide range of words. Um different you it would not rely on on similar sounding words over and over again.

Uh a piece with low perplexity, like you get out of a model or that's person who's not a good writer, will have very low perplexity. They keep using the same word over and over again. Um there are there are people who speak that way. There are people who they their vocabulary is so limited that they can only they can only speak in in in broad generalities with very common language. There's nothing wrong with that but it's it's it's noticeable the second metric is called burstiness burstness is how div diverse things like line length are if every paragraph is you know exactly four sentences and every sentence is exactly 12 words that is low burstiness there's no variation in line length or paragraph length it looks and feels and reads very uniform regular human people generally don't write like that unless they have you know word counts they're trying to hit like students trying to to pad a paper um models tend to do that a lot but as models advance they have more natural variability they have more natural a wider vocabulary they today's models are starting to evade the very very primitive detection tools much more easily these days with today's state of the art models and and a good prompt the so-called ai detector tools are worse than a coin flip in terms of their abilities to tell whether something is machine generated or not because machines are advancing so much faster than the detection software it's an arms race and the detectors are losing second example um recently black forest labs released their flux model an image generation model it came out last week and it is incredible it runs on your laptop if you have a gaming laptop the it will run and it is amazing it is head and shoulders better than similar size models and competitive with the big paid services like if I was mid-journey or whatever, I'd be like, this is this is pretty good.

Um, and it's good enough that many of its images will pass casual inspection as real. Right? The the the good old days of you know someone waving with 14 fingers, uh those days are gone. That's how good models are getting now. And they can run on your on your computer.

They don't need a huge server room. Detection of AI content is getting harder. Right. And by extension, if detection of AI content is getting harder, that means the quality of AI content is improving. After all, if you can detect it easily, it's probably not great quality.

So what's the takeaway? Should Carol remain anxious. Here's the takeaway that I see with this stuff. If we stop thinking of things as human or machine, and we start thinking of human and machine, we're gonna get the most best results. Subject matter expert plus AI expert.

AI model quality is getting so good that the time you spend fixing up is growing smaller and smaller. The time it takes to write better prompts is getting smaller and smaller, and the quality is getting better and better, which means you can spend more time coming up with the most best ideas and then hand off that to for the models to produce. And that is that's how you win. That's how you win with AI. It is not you versus the machines, it is you plus the machines, gives you a substantial advantage over people who don't have machines, just can't be as productive, or people who have machines and aren't skilled at using them.

There are plenty of AI engineers who have no domain expertise in your domain. If you collaborate with a technical person and AI, assuming you're you're not a technical AI person yourself, you will you will win way more than anyone else because who wins in the age of AI? And I say this in all my keynotes. Who wins in the age of AI is whoever has the most best ideas and the most best data. When it comes to Carol's anxiety, you have to have the most best ideas, and you will win in the age of AI as long as you have a good uh partner to help you work the AI and you have good quality AI tools.

That's gonna do it for this episode. Thanks for tuning in. I'll talk to you on the next one. If you enjoyed this video, please hit the like button. Subscribe to my channel if you haven't already, and if you want to know when new videos are available, hit the bell button to be notified as soon as new content is live.