You Ask, I Answer: Future of Retrieval Augmented Generation AI?

In today's episode, Jesper asks, how do you see the future for retrieval augmented generation AIs? When particularly news outlets shut out AI crawlers, scrapers, etc. Okay, so AI crawlers, uh scraping and crawling bots, typically are deployed by a a company. They've had an identified browser agent, right? OpenAI's crawler.

And you can, and and if you want to, you can block those specific crawlers. However, there's a bunch of other ones that are pulling the exact same information. In fact, if you look at Common Crawl, go to commoncrawl.org, you will see that they crawl the entire public internet. So even if a news outlet says you may not crawl us, you know, open AI bot, open AI just has to go to Common Crawl, pull the latest vintage from there, and then use that for processing. So that's kind of a fool's errand trying to block AI systems from consuming content.

Especially if you're already giving it to search engines, right? So if you are allowing Googlebot, well, sure, open AI might not then crawl your site, but Google will. And if Google's gonna do it, then guess where that information is gonna end up? It's gonna end up in one of Google's models. So you've really not accomplished anything.

To the question though about retrieval augmented generation and how that plays a role. It's important to understand the role of retrieval augmented generation. So let's let's go back to some basics. When you have an AI model like GPT-4, the model that powers the paid version of Chat GPT, there's a couple different ways to get a model to behave differently. One is prompting, the prompts you give, the instructions, the directions, the plain language coding.

The more sophisticated your prompting, the better the results you were getting you will get out of a big general model like that. So that's one area, is just being very good at prompting. And there's a whole bunch of ways to do that. There's some really advanced studies coming out now that are showing that good prompting can actually outperform some other methods of getting models to work in a certain way. Fine-tuning is sort of the second way, and this is where you condition a model to answer specific kinds of questions better than the model was originally trained on.

So if you fine-tune a model on, say, medical questions, and you just give it a whole bunch of questions and answers, the model may not get any new information that way, but it's going to get it's going to learn how to answer those questions better than whatever medical information was put in in the original model. I used I like to think of this as like the way you train a dog, right? If you train a dog to sniff for drugs, it's not going to be able to sniff for explosives or earthquake survivors. But it's going to be really good at what you trained it to do. That's what a fine-tune is.

Retrieval augmented generation is a library, it's a database, it's an add-on to a model, which gives the model more context, more information, new information that it wasn't trained on. So the model still has the same capabilities, it can still answer questions, but now it has a new place to look first before it goes to its um before it tries to go to its with the data it was trained on. And we see retrieval augmented generation popping up all over the place. So open AI's custom GPTs, for example, is an example of retrieval augmented generation. You give it some documents that maybe have updated information or very specific information, and the model knows to go to those first before going to its general knowledge pool and to prefer the knowledge it gains from that as well.

So the future of retrieval augmented generation is very strong because it allows us to change the context, the knowledge base of a model without having to rebuild the model itself. It's like if you had a kitchen full of appliances and your pantry full of ingredients, retrieval augmented generation adds more ingredients to the pantry, right? Your appliances don't change, but what you can cook now is greater variety because you you got some new stuff in the pantry that you maybe you didn't buy with the previous week's groceries. Fine-tuning upgrades the appliances, right? Uh maybe your your your crappy uh Hamilton Beach Blender gets replaced with a Vitamix or a Blend Tech, right?

Now you've got a much more powerful tool, but your ingredients in the pantry are the same. You just it does a better job now. So you you know, the smoothie you used to make with your Hamilton Beach is not going to be as good as the smoothie you can now make with a Vitamix. So that's kind of the difference between these different ways of approaching this these techniques for improving the performance of models. And if news outlets are shutting out AI crawlers and scrapers, okay.

Uh again, that data is available in other places, right? You today can build your own scraper and crawler. I've built dozens of these things that are very purpose built, and I can take their outputs and put it into something like a custom GPT from OpenAI, and that puts that news, that information I want back into the model. So even if the base model doesn't have it, I can use my own software plus you know, retrieval augmented generation to put that knowledge back in the model and make it available. When you get into open source, then you get to some real interesting stuff.

Open open weight models like Llama 2. You can tune those models and do retrieval augmented generation and change the alignment of the models to be like uncensored. So there are some topics, for example, with the the big public models like the ones that PowerChat GPT, there's some topics it won't talk about. If you ask it to build something harmful, it'll say, nope, can't do that. You can take an open weight model that has an unit censorship and say, Yeah, here's the directions for how to do that bad thing.

So even in cases where news outlets are trying to quarantine their information, unless they publish it in some format that people can't read, that information is eventually going to find its way into a model somehow. So I think it's kind of a uh a fool's errand there. Now, the real concern that they have, and this is a valid concern, I'm not saying it's not, is that their content is being used and they're not being compensated for it. And I think that's a valid concern. If you own property, content, data, you have the right to say how it is and is not used, right?

That's implicit in property rights. And so if you if you want to exert and enforce those rights, you should talk to an attorney and about what your options are. Like, can I sue them for using my stuff? And you know, your attorney will advise you as to what that what that looks like. But retrieval augmented generation and fine-tuning are still the paths forward for making models do stuff very specifically combined with really solid advanced prompting.

So there are all sorts of really advanced techniques that you can use that are not, they're not easy compared to you know just saying, hey, write me a blog post about this. Um but they deliver best in class results. Maybe another time we'll we'll dig into what that is. But it's a really good question. Um, and hopefully this answered the difference between those techniques and how they work.

So thanks for asking. I'll talk to you soon. If you enjoyed this video, please hit the like button, subscribe to my channel if you haven't already, and if you want to know when new videos are available, hit the bell button to be notified as soon as new content is live.