Mind Readings: Retrieval Augmented Generation vs. Fine Tuning in Generative AI

In today's episode, let's talk about two techniques, two technical techniques that people use to improve the performance of generative AI systems like ChatGPT, for example. Although these techniques are more for geared for people who are running models like the GPT-4 model, but local ones within their uh servers, or using the more technical API versions of these tools. Why would you do this? Why would this be of interest to you? Well, you may want to have a model have very specific custom information, or you may be running a model in a controlled environment where uh security and data privacy is really important, and you you just can't use the public tools, uh, national security, protected healthcare information, etc.

So there's two general ways to change how a model behaves. One is called a retrieval augmented generation, where you connect a database of your data to a model, and the other is called fine-tuning, where you essentially give a model lots and lots of specific examples of what you want it to do and retrain it, re-weight it. And there's a variety of techniques within that school. When you hear these terms, when you hear people talking about these terms, they are talking about changing behaviors typically for a specific application, like a chat bot on your website. You wouldn't really do this for the consumer version of any of these tools.

You wouldn't do it, for example, to change how you use chat GPT on a day-to-day basis. There's really no point in doing that. What are these things? Here's a way to think about it. If you think of a language model, again, like the ones that power chat GPT or Anthropics Claude, etc., as a library, this huge library.

And there's a certain percentage of the model that in technical terms is called retrieval heads. Think of these as the librarians at the front desk in the library. And when you give them a prompt, they go back into the library and they try and uh pull all the stuff necessary to answer your prompt. Except instead of returning entire books, they return like one word at a time. And they're going through all these books to try and find the related words and you know bring back all the words to you.

Sometimes they get it wrong. Sometimes they this called hallucinating, where they try and get the uh they try and and and do what you tell them to do, but the information may not be in the library or they may not know how to do it. For example, uh, suppose it's a really small library, right? In this fictional example, you go to the library and say, Hey, I'd like a copy of the joy of cooking, and the librarian goes back into the those very small stacks, looks around, says, Don't have that. What's the closest thing I can find?

Pulls the book off the shelf and comes back and says, Here's the joy of sex. That's what you wanted, right? And you're like, no, no, that is I mean, linguistically, yes, it's close in title, but this is completely the wrong answer. Hallucination typically is something that is statistically correct or statistically close, but factually wrong. And that's a a sort of humorous example.

So in situations like that, you may want to say, like, yeah, we need more books in the library. So you would use a technique like retrieval augmented generation and connect a database of your data. Because one of the rules of thumb with AI is the more data you bring to the party, the less it's going to hallucinate, the less it's gonna make things up. So in retrieval augmented generation, it's like saying, Oh, the library doesn't have books about our company, about about trust insights. Let's give the model, let's connect a database of all the stuff that Trust Insights has ever done, all our newsletters, our blog posts, our YouTube videos, our live streams.

And suddenly, the next time I go back to the librarian, the library, I say to the librarian, hey, what do you've got for for stuff on trust insights? The librarian looks around and goes, Well, there's not much in the main library, but wow, there's this new this new wing you just built filled with stuff about trust insights. I can go and get that stuff from that section of the library, and here's the answers. And you, as the at the prompter, uh, are much happier. Fine-tuning, on the other hand, is when the librarian doesn't know how to do something.

You go to the library, and this is a silly example. You go to the library and you you say, Hey, I want some books and I want some DVDs. And the librarian's like, What's a DVD? Never heard of it. We got books, but I don't know what DVD is.

And you're like, okay, so let me let me show you some examples. This is a DVD, you pull one out and stuff like that, and it's filed like a book, but it's a little round disc and it's got a cover and it's got a barcode. And you essentially teach the librarian, here's what a DVD is, here's how to find a DVD, here's where they are in the library, and so on and so forth. And the librarian becomes smarter. Librarian now knows how to find books and how to find DVDs.

That's an example of fine-tuning. You're giving lots of examples to the model to say, I want you to learn how to do this specific task really well. Now, when you fine-tune, depending on the method you use, um, you have the potential of causing um some interesting chain reactions inside the model. Uh, it'll be like teaching the librarian uh how to find DVDs and they forget what a book is. Like, whoops, that that wasn't supposed to happen.

So that can happen. But these two techniques are important to understand the difference because a lot of people mix the two up in terms of what they're they're good at, and as a result, pursue the wrong strategy when it comes to customizing AI within your company at a at a company level, at an organizational level, at an enterprise level. Again, this does not apply to your personal usage of Chat GPT or Gemini or Claude. This applies to companies building applications based on AI models. In the big public stuff, you have the ability to fine-tune a KAT library and how to do something different.

Um, but to connect your own data is a little bit more work. So uh, although there are systems like Google's vertex that allow you to do both. You need to know the difference because you need to know what kind of problem you're trying to solve. Is generative AI in your company's application is it failing because it just doesn't have enough books in the library, then you want retrieval augmented generation. Is it failing because it doesn't know how to answer customers' requests that are specific to your business?

If that's the case, that's a behavioral issue, and you need to use fine-tuning. You need to gather up a thousand, two thousand, three thousand examples of a customer request and the appropriate answer, and you tune the model to say here's how to answer these types of customer requests. So knowing the difference between the two helps you set a better strategy for how you want to customize your use of AI in enterprise applications. But the analogy of a librarian who either just can't find the books in the back or doesn't know what a DVD is really helps to make tangible these two very technical concepts. That's gonna do it for today's show.

Thanks for tuning in. Talk to you on the next one. If you enjoyed this video, please hit the like button, subscribe to my channel if you haven't already, and if you want to know when new videos are available, hit the bell button to be notified as soon as new content is live.