Mind Readings: How Large Language Models Really Work

In today's episode, let's talk about how language models work with a different explanation. Now, the way that I've typically explained this in the past, and I do this in my keynotes, is think about a prompt when you're prompting a tool like ChatGPT or Gemini or Claude or any of the tools that are out there as word clouds, right? And as you type words into your prompts, word clouds are appearing behind the scenes, and the intersection of those words is what the machine knows to spit out. And conceptually, that's more or less how they work. Mathematically, that's not completely wrong.

So I've been looking for a better explanation that is more aligned with the mathematics of how these things work. And here's what I've come up with. Have you ever read, as a kid or maybe an adult, uh, the choose your own adventure books? Right? We open this book and it's got the starting page of the story, and at the bottom of each page says, you know, turn to page 41 if you choose the red button, or you know, turn to page 43 if you choose the blue pill.

Uh that is a really good example of how generative AI models work, of language models. Right? You you keep reading and you choose the next page, you make a decision, you choose the next page, and you hop it around this book, uh, and eventually you you get the story told you want. Except that instead of reading a few paragraphs and then turning to the appropriate page to continue the story, a language model is choosing how the story continues after every single word. And the book is massive.

The book is as big as the English language, right? It is it's it's terabytes of data. And every word has a choice at the end for the for what the next word is going to be. Why this explanation is better is because, like a choose your own adventure book, a language model keeps track of the story that's already been told, right? It doesn't go backwards and make different choices.

It says, okay, well, you chose this word, so the problem the next set of probabilities are this. When you're reading a choose your own adventure story, you keep reading and you keep following these threads throughout the book. There aren't an infinite number of choices at the bottom of every page when you read a choose your own adventure book. This there's a handful, right? In the same way, when a language model is picking the next word to come up with, there's also not an infinite number of choices at the bottom of every page, if you will, as it as it predicts as it reads.

There's a handful of words that are most probable based on the story so far. That's the critical point. Because a language model can keep track of what's been written so far, it uses everything that's been written so far to predict the next word. Suppose the story in AI is processing has read the following words. Probably the word flag, right?

Because in American English, it's very common uh to hear people say, I pledge allegiance to the flag. If you're if you're English, you'll say, God save the whatever the next word is. Uh, it could be king or queen, depending on how old you are and and what's going on. But it's probably not rutabaga. In either example, right?

The next word is probably not rubber bigger. Statistically, it's unlikely to be that. And so a language model makes its choice based on probabilities, based on the number of previous things it's read in its training data, where flag is probably going to be the next word. That's a really important thing to understand. Because when we prompt these tools, we are giving them some words to start with.

We're giving them the first page of the story. And then they have to, from the words that we provided, read that guess the next word. And if it does a bunch of guesses and we like what it says, we like, you know, it's it wrote us a blog post or whatever, then it will continue to predict based on those choices. And it never goes back and changes things in the past, but it uses all the past to help decide what the next word is going to be, what what page it's going to turn to next. This is why models go off the rails sometimes, right?

When you're using a tool like ChatGPT and it's just starts spitting out nonsense or it writes really badly all of a sudden, it's because they go awry when they have not read enough story to choose the next word sensibly. Imagine, imagine you were reading a choose your own adventure book, right? And the first page of the book uh has one word on it, says today. You're like, uh there's there's a bunch of choices for the, you know, choose turn to page 82 if you want this. And you're like, I it just says today on the page.

How am I supposed to know what's a good what to choose for the next page? You'd have a nearly limitless choices, even if you knew that you wanted a romance story or thriller story, it's it's still just too vague. That's what happens when a model runs off the rails. It gets some word either that doesn't have enough words to make a decision, or it's got conflicting words, and it's like, I don't know what to choose next. So I'm just gonna pick a random word or a word that matches the what I know statistically, even if it doesn't make coherent sense.

This is why prompt engineering with detailed prompts is so important, because what you want to do is you want to give the model enough of the story so far so that the next part of the story as it chooses the next page will be much more sensible, right? If you give it a prompt, like write a blog post about B2B marketing, and then you're really unhappy with the generic swill that it comes up with, it's because you didn't give it enough story. So it's like, okay, I'll just I'll just pick something that's that seems sensible. If you give it a three or four paragraph prompt about the story so far, B2B marketing is this, and this is the things I care about, and don't mention this because we already know this, and so on and so forth. You will have it create better content because there's fewer choices behind the scenes for what page it's gonna go to next, what page it's gonna turn to next.

That's how these things work. And it's if you understand this, you will get better results. I promise you, you will get better results. The more relevant words you use, the better these tools will perform for you. So that's gonna do it for today's episode.

Thanks for tuning in. I'll talk to you soon. If you enjoyed this video, please hit the like button, subscribe to my channel if you haven't already, and if you want to know when new videos are available, hit the bell button to be notified as soon as new content is live.