So What? Use cases for Retrieval Augmented Generation

Well, hey there everyone. Welcome to So What, the Marketing Analytics and Insights live show, which is looking a little orange this week. Um I was like, ooh, is that me? Nope. It is definitely orange for some reason.

But anyway, we move on. How's it going, guys? Going orange. Yeah. It's one of those things that, like, ugh.

Anyway. Um, so this week we are talking about use cases for retrieval augmented generation. And so if you missed it, Chris and I talked about what is retrieval augmented generation and why you should be thinking about it as a marketer or operations or any sort of other professional who has any kind of interaction with artificial intelligence at this point. So in a nutshell, if I can remember correctly, Chris, retrieval augmented generation is essentially you have to basically it's used in s in situations where you have a contained amount of information, sometimes proprietary information that you only want to pull, you know, answers from. Example, and I know we're gonna go over different use cases, but the way that you talked about it that really kind of stuck with me is if you have customer data, proprietary information that you don't want going out to these large language models to then use as their training data to search from, you would use a retrieval augmented generation system like a notebook LM, for example, put the information in there, and you can only search that.

So, in some ways, it's like think about old school like wikis and knowledge bases that we used to build and SharePoint, or people still build them, but I don't build them anymore. So, this old school to me. So, that's my high level understanding of retrieval augmented generation or a rag system. Am I close? You are close.

Yeah, let's review the system diagram to understand what a rag system is and where it belongs. In normal generative AI, when you fire up chat GPT or the system of your choice, you put in a prompt, and then once you put in that prompt, it goes and processes it, it thinks a bunch, and gives you an answer. When you use retrieval augmented generation, what that does is you put in a prompt, and then the system tries to figure out do I need to get additional information based on that prompt, and it goes into a retrieval system, which is located right here. That has a very uh traditional database style feel to it. Uh, it goes, it gets the data, it retrieves the data, it converts it into content that it can work with your your LLM, and then you can see here it merges the prompt and your external data together that goes on to the AI system, and then it comes out with a response.

So, this is this is where this is essentially how rag works today, and there are variants, uh, a number of variants. There's a gentic rag and all that wonderful stuff uh that we may or may not get to. However, the two considerations for when you want to use a rag system. Number one is privacy. There is some data you just can't let out of your control.

So as a reminder, and by the way, if folks you would like a copy of this chart, it is available on the Trust Insights website. Go to TrustInsights.ai/slash AI data privacy. You can download that this chart there. It's on our instant insights, no forms to fill out. Um it comes to privacy, there's differing levels of privacy with gender of AI.

The only system that is guaranteed private is systems that you run locally on your hardware, whether it's your personal laptop, whether it's servers inside the walls of your company, whatever the case may be. That is the only system where you are guaranteed that no one else will ever look at your information. There are systems that are mostly private, which means that they're they don't train on your data, but a human being can look at your stuff if you cause a terms of service violation. If you say, hey, let's, you know, give me step-by-step instructions for how to do this bad thing. Uh, the model will say, uh, you really shouldn't do that bad thing.

And if you do that several times in a row, it will trip a flag saying it looks like you're trying to do a really bad thing. And so we're gonna have a human come in and check to see what exactly it is, what bad thing you're trying to have it do. In those cases, because you've tripped a circuit breaker on the bad thing, um, a human being can look at your data, and so it is no longer guaranteed confidential. And we could probably do a whole AMA episode on what what is confidential, and you know, there are guidelines, there are certain laws that exist around confidential data privacy, but there's always a squishy, you know, uh a squishy place in those regulations, especially when it comes to terms of use and whether or not someone has read, because we could say, yes, you're giving us confidential information, but we're gonna use it. And I'm not saying that's what our you know documents say, but if someone just went ahead and checked, I agree to everything, and that's usually what happens with consumers, and then they're like, well, wait a second, what do you mean you're selling my data?

Well, you said it was okay. It's a lot like uh those silly folks who post on platforms like Facebook this copy and paste. I do not give this platform the you know authority to use my pictures, but you're already on there, they're already using it. You already said yes. The second you gave them a login information, you already said yes, it's too late to go back now.

Anyway, that's sort of a little bit of a rant. But um, I think it is important, Chris. Thank you, to cover data privacy and confidential data because that's a lot of why a use case for retrieval augmented generation exists. Exactly. Um, and so if you if the data you're working with has restrictions that require only mostly private, you and and it that's one of the two dimensions for determining reg, uh, that's a consideration.

Say, like, you know what? We're working with blog post content, right? We're not working with medical records, we're not working with national security secrets. Maybe in those cases, uh, you know, if you're just doing marking stuff, you might not need a rag. Uh so that's one.

The second consideration comes down to size. So many systems. Well, to pause there for a hot second, can you bring that chart back up for we have a question about it? Sure. Uh so Brian is asking on this particular data privacy matrix, where does Notebook LM fall on this chart?

Is it the same as Gemini? It depends on whether it's the free version or the paid version. So the free version contains the same general restrictions as Gemini free. The paid version contains the general same restrictions as Gemini paid. So I think that you know, essentially, if it's free, your data is the payment.

It's a fun tagline. Mm-hmm. Uh all right. So you were talking. Go ahead.

I was gonna say check the terms of service. Uh is the single best thing you can do is check the terms of service. It's that little tiny link at the very bottom of every screen that you can say, oh, yep, there it is. And if you're unclear about the terms of service, put it through a generative AI system and say, evaluate this terms of service. How private is my data on a scale of one to 10.

That's that's very cyclical because first you have to have a system, then to read the terms of service, and then if it's not the right terms of service, you go to a different system. That seems like a lot of work. We will put a link in our analytics for marketers Slack group to our custom GPT that we built to evaluate terms of service. You just copy and paste the either the URL or the terms of service, and it will give you a nice score as to how private your data is for any given system's terms of service. So go ahead and check that.

We'll put that in at the end of the show. Um so privacy is one aspect. The second thing, the second angle about which we've logged many generation, is size. How much data are we talking about? So every language model has what's called a context window.

It's short term memory of how much information it can remember in any given chat. Chat GPT holds 128,000 tokens, which is about 90,000 words. So it can hold one of these. Can hold one of these, right? So if your data is this is smaller than this, you might not need reg because you could just put all of the data into the context window.

Now that can be expensive, but it was it is highly functional. Meta's new Llama 4 Scout, which came out last week, has a 10 million token context window. So it can hold seven of these in its memory and be able to retrieve from it. So again, that's an example of if you have the hardware and you have the bandwidth, uh, you may not need rag. If your data is bigger than that, you would need a rag system.

So those are our two dimensions privacy and data size. If you need privacy, absolute privacy, uh, you want a local rag system. If you have really big data and you only need to fish out certain parts here and there. Say, like, I'm going to go into my medical record system and pull out every mention of Morphe Scleroderma. I would have uh just the a REX, so pull out just those pieces from my database and load them.

So those are the general situations as to when you would use RAG. Would there be a scenario? So let's say size is you know the consideration. And so like I have my whole EMR situation. That's all my electronic medical records.

Rather than bringing it into a third-party system, would you lay an interface on top of it or do you like an API call or something rather than move the data over? Does that kind of bypass the size issue? It could, depending on how good your your query system is on the underlying system. So systems like Epic, for example, the EMR system, uh, looks like it's straight out of Windows 95, and the backend data is about the same. Um, so unlike Mick Jagger, that data does not age well, and as a result, uh you could build I I mean that's subjective.

I don't know that he's aging well. Well is subjective, yeah. Well is better than Keith Richards. These are some rough examples. Anyway, moving on.

Um there is an entire set of systems for that that are um evolving rapidly, uh, one of which is called model context protocol, which is an entirely separate episode uh that we can go into at a later time. Uh so yes, you can interface with traditional databases and just query records, you know, with good old-fashioned SQL queries and pull back just those limited pieces. When we think about a rag system, um typically that purple external knowledge source is something like a vector database, but it doesn't have to be, it can be a traditional good old-fashioned SQL database as well. It's more a question of can your LLM system talk to it? And most LLM systems today are not designed for that.

You need some kind of retrieval system. Maybe it's just a Python script, uh, maybe it's a whole ecosystem to manage that storage and retrieval. So, which is why you have to have that size consideration, because the LLMs don't talk to your existing databases. So you do have to bring the data over. Yes, that's right.

Now, today's AI first rag systems typically have a special kind of database behind them. And the special kind of database is called a vector database. What happens when your query is uh when you put your prompt in is that behind the scenes, the machines turn that into just numbers, and those numbers get processed and matched with the internal database that the an AI is trained on, and potentially your external system here, and all that is blended together and then sent into the AI model. So a vector database has your data, but it's already pre-encoded for AI. So you don't need a translation layer on the fly, which can be very slow depending on the size of your data.

Um instead, it's all pre-sorted and embedded and processed, so it's it's AI first. It's in the same format as the language model itself, which is a bunch of statistics. There's no words, there's no identifiers, it's just statistics. That's what makes vector-based RAG systems so efficient and so fast is that they've taken all of your data and pre-crunched it down to a format that AI can natively understand. And so it can be very, very fast.

So I'm gonna pause here for a second because this has gotten very technical. It's under well, no, and understandably, like we want to make sure people understand the retrieval augmented generation system as we're talking about the use cases. So, John, how how is your brain computing this? Like, are you are you keeping up? Because I'm I'm like hanging on by a thread, but I think I'm there.

Yeah, I'm getting the thing that comes up for me though, is I don't really understand the state of the industry with this. Like I understand how not to do it because I've I've talked with Chris in the past about rag lock, you know, where you use notebook LM so that you're not going out to grab other data. But I kind of had no idea of like which of the tools out there are doing uh, you know, going and retrieving on other systems. Uh that's wide open. And then, or is this just all because I know a lot of this falls under agentic AI where you actually are building stuff to go retrieve things in other places.

So yeah, I guess the big question for me first is are there any currently available systems that do this that you know you could use without having to string it together yourself at all? There are, and we'll look at an example of one. Um actually we'll look at a couple examples. So um go ahead, Katie. And with that, we have another question from Brian.

Can you build a rag tool using a foundational model? Do you have to use API calls or are these completely different models used to build RAG. So um, when we get into the local example, we'll talk about we'll cover a little bit about this because yes, you can use everything local. However, there are some there's some gotchas with that. So we're always.

So the most familiar RAG tool, the one that pretty much everybody knows, is Notebook LM. And if you don't know it, we have covered Notebook LM on past episodes of the live stream, which you can find at TrustInsights.ai/slash YouTube. This one is super simple. You go into a new notebook, you add your sources, you drop your documents in, and Notebook LM, which is a Google product, um, will digest those data sources and then allow you to ask questions and chat with your data. And so what this allows you to do is load in lots of data.

So the free version can hold 50 documents, the paid version holds 300 documents. The free version, I believe, is constrained. I want to say it's like I think it's like 300 megabytes total, but either way, it's smaller than the the large version, which can hold a lot of stuff. And so if your data is under uh you know 500 megabytes, you can throw it all in here and then ask questions of it. And there are, of course, the many, many fun things you can do with notebook LM.

Uh, we talked about on previous episodes, like having the mind maps. So I can look, here's the uh inbox insights newsletters that we've been writing for this is five years worth of of newsletters, and we can look at the topics that we've talked about. There's personal professional growth, and then the two types of people that Katie wrote about, and so on and so forth. So you can explore this. It's a very it's convenient interface.

The downside of Notebook LM is that it is self-contained, it is its own little thing, and it's very difficult to get data in and out of it. Um, so it's good as a it's great as a research tool, it's great as a processing tool. It's terrible in a workflow because it you you can't get stuff out of it. There's no API. There's no um automated data export.

You have to manually spit things out from it. I was using notebook LM Jesse I actually use notebook LM uh this week to outline the four part series on creating a strategy for integrating AI um that we're doing for the newsletter because I know we've written about strategy a lot so I wanted to start to string together the pieces and so I was able to use notebook LM to say well what have we written before how have we done it what has it looked like you know what frameworks have we referenced and it was actually a really easy uh tool to use I mean I love it and the mind map I think is honestly just fun. Like that is such a great thing. The other thing I like is the FAQs. So one of the challenges that our clients tell us when they talk about introducing notebook LM is well it's great to have the information there but people don't know where to start they don't know what questions to ask and you can build the FAQs will sort of like self-populate for you and you can just click on them and say it will basically be like a seed starter of what questions should we be thinking about exactly so it's a great tool I I love it.

I think it's it's phenomenal however the data input and output issue is is a big deal if you want to integrate your private data or your too big data into other systems as part of a workflow. That's where things kind of blow up uh with this tool. So uh an one of the things you can do is you can download an install your own vector database. And there are options such as Postgres SQL, which has a PG vector, there's Pinecone, Weaviate, Milvis, MongoDB has a vector store, Superbase, ZEP, there's many, many of these vendors. And about half of them are open source vendors like QDrant, uh, Milvis, Weviate, they all have open source versions that you can run locally on your machine.

That starts to become more handy when you want to build on top of your data. So here's a very simple example of how you would deploy this. So this is the anything LLM app. Uh, let me make this full screen. This we've covered on the show in the past, in terms of setting up local models.

Inside here, if you go into the settings, there you can choose obviously your LLM. I'm going to be using uh Gemma 3, which is Google's uh model, and then there's a vector database. And you can either use the one that's built right into the software, which doesn't require any configuration, or you can choose from the gazillion different vendors if you have uh a vector database store. Here's the the if-then statement. If you're using this for yourself, use the built-in one.

It's kind of a no-brainer. If you're using this with a team inside of a company or within an organization, you'll want to have a central vector database that everybody has access to because it would be kind of silly to just make multiple copies of your data on everybody's laptop when you could have it in one place. Which makes sense, and that's one of the challenges people are having with generative AI in general right now, is it's kind of siloing things unless you have a shared account. Yep, exactly. So if you have, say, if you set up a Weavate server uh inside your organization, then everybody can use it.

And of course, many of these have cloud hosted options. But again, if the if you're dealing with stuff like, hey, these are national security secrets and you don't want to be added to the signal chat by accident, you want to have the the database centrally on your hardware inside your company and not in the cloud, because otherwise, what's the point of having it in the right? You want it local. So again, most of those services, there are options for having it on your hardware. And I would I would recommend for that, you know, if you're doing with like PHI public uh private uh protected health information, yeah, have it have it on servers you trust.

Um this is where things go wrong. The embedder. In order to convert your documents into a language into the statistics that uh an LLM can understand, you need to use an embedder. You need to have some kind of embedding tool that will convert it from words to statistics. And the catch here, and this is why so many RAG implementations go completely off the rails, is that your embedder has to match language model because everybody does things differently.

So if you are using open AI and you know, chat GPT and in that ecosystem, you'd better be using an open AI embedder. If you're using in Google's Gemma, you probably want to use the gem the Gemini style embeddings. If you're using Mistral, the French model, there's an embedder called Tekken, Tekken 7, um, that you use. What people screw up here a lot is they will pick a simple embedding system of some kind that is a mismatch for the model. It's kind of like pulling up to the gas station in your Prius and accidentally putting diesel in your car.

Like it's a mismatch for what the engine runs on. And so this is where I I have, you know, we've been to I forgot how many Maycon shows now, Katie, but I've seen and I've sat in on the different sessions people had about you, you know, building rag systems. Nobody mentions this part. And when I look at the implementations and I see that someone's using an open AI embedder on and then sending the data to Gemini or Claude, I'm like, ooh, that's not gonna go well. Well, and you know, this all speaks to you kind of have to have a handle on what you're doing as you're setting up tech, you know, which is a whole different, you know, episode and conversation.

But it kind of goes without saying that yeah, you can set up anything. Doesn't mean it's gonna be right. How will that melt down? Well is it basically that whole data set doesn't even get used, it just becomes useless. Um, it melts down by have by causing the the LLM to return poor quality results, right?

Because open AI's embedding might be at the word level. So you would say marketing over coffee, but then you use the wrong embedder and you know it it chunks that into mark and et and then ing and then over and then co and then f e and it would break the words up into different segments that don't match what's in the actual uh with the what's in the actual um target target LLM. So the LLM and the data store have to be using the same uh pieces of information to match them together. If that doesn't happen, then you're gonna get poor quality results that just don't make a lot of sense. So I know you're walking through the setup, but I also want to make sure that we spend some time on the actual use cases.

So is that where we're going next with this? We will get there, yes. Um, so let's go ahead and drop in some customer service emails into our embedding system. And we're gonna say, and now you can see the file has been embedded. So this has now been converted into the document format that the underlying model needs.

And say, summarize the negative emails we've received. And I'm by the way, I am using um synthetic data from our new generative AI use cases for marketers course, which is going to be available in five days. You go to trust insights.ai slash use cases course. Um this data is all synthetic because they don't want to be showing you know live uh customer data. But if we look at the very back end, we can see that the model is pulling in from the database just the relevant parts.

This is a very large file, right? This file is uh how many thousands of lines of of data is this? It's several thousand. Um eight thousand. Uh and you can see it only went through and pulled out a few things because it matched on the prompt that we gave it, which is the negative emails.

Now, that's a terrible prompt, by the way. But what happened is behind the scenes, so this tool took my prompt, broke it up into pieces, and then said, find the relevant things within the vector database that match this and bring it back and show it. And so we see Olivia Scott from SummerCamp's Inc. reported poor customer service to an unresponsive account manager, hindering the ability to manage food service effectively, requiring immediate contact and improvement to support, right? Because this is all about the savory oatmeal company.

So I can now query all my data that I've loaded in here and ask it for piece by piece those those little things from the database without having to load the whole document because that would exceed the amount of memory that the Gemma model is running on, and without spitting up a bunch of garbage. So as we're talking through use cases and the limitations of these systems. So let's say, you know, my boss says, I want to know all the negative, you know, customer service feedback emails from the past year, but we get you know 2,000 emails a day, you know. So because there's limitations to the systems, I would have to repeat this process over and over and over again. Is that correct?

In a chat interface like this, that is completely correct. So then does that make it a good or bad use case for a retrieval augmented generation system? That makes it a good use case, but this is the wrong tool. Okay. And I think that that's also important to point out.

Mm-hmm. So in a case like that, um, just to give you a sense, the underlying data looks like this because it's all synthetic, obviously. Um there's no dates on here, right? So the obviously the data itself has to have the required field, so it's a data governance thing. Uh if the if it this it's not in the training data, because that's effectively what we're doing is we're providing its own training data.

If it's not the training data, then it's not then we're not going to be able to fill fulfill that that request. And you have to say to the boss, hey, boss, we haven't been collecting date information. So I can't provide you that information, um, which would be in a very uncomfortable conversation. Well, you know what could help that? It's a 5P framework.

You set me up like that. Was a nice tee up for that. Um, you know, so we're going sort of a little bit out of order today, you know, as Chris is showing how to set up a retrieval augmented generation system, a rag system. I keep saying the full thing because I'm trying to make sure I don't forget the words. Um, but as you set up a rag system, but the use cases, so in that example, you know, that I just gave, yes, you can set up a rag system, but before you jump into, oh, well, I can set up something in anything LLM or I could set up something in notebook, you want to go through each of the five P's.

What is the purpose? What are you gonna do with this information that you're that you're trying to query? Who are the people? You know, you probably want to talk to your customer support team, they're the ones collecting the information. What is the process?

What data do you need? If date stamp is important, you should probably figure out if that exists first. Then you can pick the platform. So if you really need to do this, then you can figure out what platform is the correct one, versus hey, I set up a cool vector database and anything LLM. Let's start querying it and see what happens.

That's the wrong order of things. So I just needed to make a little bit of a plug for the 5P framework. And if you want to get your copy of that, you can go to trust insights.ai slash 5p framework. Okay, carry on. So what is the right implementation?

What is the right use case implementation for that instance? Anytime you're talking about retrieving a lot of data sequentially like that, you are talking about something that involves automation. And automation and AI dine at the same table, but they're not the same thing. So in a case like that, you'd probably want to go to a system uh something like uh N8N, which we've briefly talked about in the past, but uh we might want to do to dig in uh at another time in greater depth. What you would do is you would have your LLM, and then you'd want to attach uh some form of memory or tool.

So then this case, uh maybe I could use a simple memory model there, or I could also use my there's where my vector data store would be. So I'm gonna attach my vector data store to this. And here, again, based on the model we're working with, I have to choose what kind of embedding I would want to use. So because I'm working with Gemini, I would use the Gemini embeddings. So if we go back to this here, now I've got my Gemini embeddings, I've got my deck vector data store.

Now I could uh add in something like my original uh data. So let's put in file, and I could say, let's let me read my files from disk. I'm doing this out of order. It should it that's not actually the order should be in should be in the opposite order. Go away.

Uh delete that. So I would I would send my chat message uh to my AI agent. Actually, that's wrong. The AI agent would be feeding the V the data store, and then the agent would go and retrieve from the data store and ultimately repeat the process over and over again. Say, okay, is this email from 2014 or 2024 or whenever?

Is this email positive or negative? Is this email and so on and so forth? And you have multiple inter multiple flows within this setup that would sequentially filter your data until you got to the right curated data set to answer the question you had. Like show me the the emails, the all the emails that we sent to this client in the last year, broken out by month. You would need to build an automation that involves AI, but AI itself does not power it.

And I think you're just making a really good case for why you want to get yourself organized first. Because I mean, I would love to just go ahead and start making connections in N8N and play around with it, see what happens. But is that the best use of my time? Probably not, as much as I would love it to be. I don't have the luxury of that kind of a time.

And I know a lot of you know professionals don't. So getting yourself organized first to then if you have if you may find out you need to set up this automation first before even getting into a rag system, that's a big deal. That's but that's good information to know because then you're not wasting your time building a rag system that doesn't work for you. Exactly. So now going back to Brian's question about can you build a rag tool using a foundation model?

You can, but you're the first part is going to be getting the data, building the embeddings, storing them in the uh in a vector database, and then you can use the foundation model to talk to that database. Uh, and that can be locally, that can be in the cloud, that can be a hybrid approach, but the construction of the vector database is in some ways independent of the actual use of it in the same way that like in Notebook LM, yeah, you load you load all the data first. You don't load the data one piece at a time. You load all your data first, and then you ask questions of all your data. That's kind of the same process with a more advanced rag system where you say, like, I need to load all my customer service emails, all 28 terabytes into the database first before I can start asking questions for it.

It's not having something that that grabs and embeds data on the fly is going to be incredibly inefficient and also probably break a lot. So I want to spend a little bit of time talking about some practical use cases for when you should and shouldn't use a rag system. So something that springs to mind is a sales playbook. So that tends to be proprietary to any given company. We have one.

John has created a very expansive sales playbook. That to me seems like the perfect candidate, along with some of our customer information to put into a rag system that keeps it contained and proprietary, that we can ask questions of it. So that's a great starting use case. First two questions: how private does it need to be, and how large is the data? Great questions.

Thanks for asking. Don't know the answer. I don't know how big those particular things are, but I know that our customer data, we want to keep confidential, we want to keep it private. We don't want to share all of that identifying information for a variety of reasons. Um, and then the sales playbook is, for lack of a better term, the secret sauce that any sales team has.

And so, unless you are being paid to do a training on building a sales playbook, you probably don't want to be sharing it publicly. Right. So I do know that the sales playbook, even though it looks like a lot of text to us humans, it really isn't. It's actually a very small document in the grand scheme of things. It easily fits inside any LLM's memory.

So size is off the table. So already you got one out of two from a rag usage system, it's it's not a good fit there. The CRM data, on the other hand, that is a good fit because it's so large, particularly all of the transcripts of all of our sales calls. That definitely is a good candidate because of the size of it. And the second, of course, is the privacy issue, uh, which is for something like that, um, the question is, would you be okay enough with using a mostly private system, or is it so sensitive that it absolutely positively has to be uh locally hosted only?

Well, it depends on what we mean by conditionally private. Like what is it about it that's not private? You know, so am I giving away people's social security number, or is someone able to see that the following, you know, dozen companies are clients of ours. So in the conditionally private, uh, and this is this is unique to the the Microsoft slash open AI ecosystem, they have training turned on by default, but you can turn it off if you know where to look. Okay.

So it's conditionally private because it's not private out of the box, but you can make it private. That's in that case, I would be fine if I know how to configure things correctly. Yep. Whereas the ones that are in green, those are those are mostly private by default, and only if you trip the the safety breakers would a human being start to look. So in a case like the sales playbook, if you were just using the sales playbook by itself, I would say just use one of the green tools and don't worry about a rag.

Uh if you're using the sales playbook with the CRM data, then uh you could, yeah, and and if we if we look in the systems themselves, when you look at uh what model to attach here, if I get rid of Olama and just toss it in the bin, um, I can choose one of those APIs. I can say, like, I want to I want to use Deep Cha never use that one for private data. Uh I could use the Azure Open AI, right? That is governed by your Azure instance. I could use the Google Vertex uh edition of Google Gemini, and that it is governed by this the the uh SLA and on our account.

And so I would have my vector database hosted somewhere, right? That has all my sales data, and then I would have one of these language models, maybe one of the cloud-based models doing the the processing of the language data. So that would be a hybrid system. And for the in the in the case of maybe going through and extracting at scale, you know, the characteristics of all the calls that John's had with our prospects, I think that would be a great use case for Rag system where you'd say, okay, I want you to go through our sales CRM record by record and extract out the sentiment of that call. Uh we already know whether the deal closed or not.

What language, if the deal closed, what language, what adjectives and adverbs did John use on that call? You know, extract those out and put them in a database. And then we can build a classical AI model to say, like, hey, what do all of the calls that closed have in common? So what did John say? And you know, if you buy now, I'll throw in a right if he said that phrase in all these calls.

Use it a lot. We would know that's something that has to be in added to our sales playbook. Right. So that is a case where you're using RAG because you're you're pulling data out of a massive database. You're using a foundation model that is private based on what you know about the different models and their privacy settings.

And the ultimate outcome is going to be a an analysis of here's what works. And to, I think for a lot of sales leaders and marketing leaders, that would be a very interesting question for them to answer. It's like, oh, I can I you can you're telling me I can learn the language of what works for effective sales calls? The answer is yes. Yeah, that seems like an amazing use case.

I would add to that, you know, if we are building this theoretical rag system for the sales playbook, it would be the playbook itself, the call transcripts, but I would also put in there the proposals that we've written and closed. Um, because that would be just an extra layer of, and here's what's working, and really come up with that sort of master playbook. So I love that use case for a rag system. Where let's talk about another use case, uh, voice of customer data. And so voice a customer is exactly what it sounds like.

It's what are your customers telling you that you need to know about the things that you're selling, the things that they're getting, the benefits. And there's a lot of different places where you can get voice of customer data. And so I'm I'm wondering, yeah, so voice of customer. Um, can one, two, three, four, five, six, but let's call it a dozen. Um, I'm not gonna count all of them.

That to me seems like another really great place where a rag system might be a good option. It could be a very good option, especially if you want to blend it all down into a single source of truth. So, for example, in advertising and marketing reactions, that's going to be comments that people leave in your social media. So, if we if we start to step through this here, we can see um this would be you know, here's savory oatmeals, Facebook reactions. That's some of the things that that data, the information would be very useful to have, but it's in a weird format.

You could vectorize it and put it into a vector database, and then you go into your market research and have all the transcripts from your your focus groups and blend that into a single vector database. You could look at social media conversations like uh for this thread from Reddit Foodies, um, looking at at this. You could look at product and service reviews, you know, people leaving the the Amazon reviews, all of this. What you see here is these are all heterogeneous data sources, they are all data sources that are in different formats and kind of a mess, to be honest. And so, to your point, Kate, a use case would be to take all these different data sources, get them vectorized, put them in a single database, maybe with a big old pile of metadata, like Amazon review, Facebook comment, and so on and so forth.

And then from there, you can query just the single store of data and say, in general, what do people think about our chicken flavored oatmeal? And it would it would aggregate from across your different uh customer service interactions, your voice of the customer interactions, into one central concept. Like generally speaking, people like the bacon oatmeal better than the chicken oatmeal. I feel like this is it's one of the reasons why market research is so difficult, is because analyzing qualitative data is exponentially harder than analyzing quantitative data because you are comparing numbers and words. Now you have the opportunity to create a rag system that brings all of that qualitative data together and actually figure out what people want from you instead of just guessing.

Now, the downside is it has the same vulnerabilities as generative AI in general. So this is an example. This is a really stupid question, right? This customer named Chris uh says, Hey, do you have a kosher version of bacon oatmeal that's made with real bacon? Like that's just a stupid question.

Of course, it's not kosher, it's bacon. There's bacon, there's never been kosher bacon in the history of kosher. Yet, if this gets loaded into a vector database, the word associations will get blended together. And the the rag, even though it's locked to the data, could mistakenly interpret this as saying, oh, we have a kosher bacon option because the word associations in there are so strong, particularly if you get this question a lot. And so, even though a rag system is locked to your data, there are there is still the possibility of hallucination, depending on this the quality of the data going in, which is an and again going back to the five Ps and going back to governance the if the garb data's garbage going in, the data is coming out of the rag system, it's going to still be garbage.

And you still need the experts. I think that this is, you know, and we were talking about uh on last week's live stream, we were talking about vibe coding and vibe marketing and how there's a misunderstanding of what it means and how it can be seen as irresponsible to just let the systems do the thing. And so when we talk about agentic rag systems, you're talking about just letting the systems retrieve the thing and do the thing. There still needs to be human intervention in that process somewhere, definitely in the planning process, which again is where the five Ps come in really, really handy, but also at the output to make sure if someone is asking questions like this, that you can then correct the system to say if someone asks about kosher bacon, say we don't have it. You know, you have to inject that additional context.

Exactly. And that's something that we controlled in your system instructions. So that's that is also a big part of any rag system, is having solid system instructions that provide the guardrails on the data itself. So the data's got to be good, and the system has to have guardrails. And out of the box, as you saw with anything LLM, there isn't any.

There are no guardrails built into the system itself. You have to provide them, and you have to know that you have to provide them. So, John, what's a use case that you've been thinking about? I I always fall back to financial data. You know, it's always fun to think about can you finally spot that pattern that you know goes where things want to go.

It's a lot of the stuff that we do, we just don't have the volume of data. You know, it's like you really need to have like 10,000 people or 10,000 calls or you know, something like that. And so yeah, I just feel like I spend a lot of time without enough data to really take full advantage of what's going on. And financial data is one of the worst things you can use with generative AI because it involves math and they're bad at math. Uh and it's just uh classical AI, yes, generative AI, absolutely not.

Another use case where RAG is can be very helpful with a strong coding model is you can load your entire code base into a rag system. So maybe you've got a really big app like Microsoft Word that has like seven million lines of code. Uh, you're not fitting that in a context window, but you can absolutely create a rag system, load your code into it, and then with good metadata, a an AI coding system can know where to reference in the system. So if you're a company that has software, a rag system could be a great way to A audit your code base and B have the tool selectively know how to debug because one of the things that goes most wrong with debugging when you're using AI is it does not necessarily always understand the dependencies from file to file. If you have good code with good documentation in the code, it will then know where else to look and it can invoke that from the reg system and find those pieces and bring them, uh bring them back to the the model's uh context window.

I feel like you're talking about the unicorn. Good code with good documentation. My code these days has great documentation because AI writes it for me. That's one of the core requirements is you must provide robust documentation in the code. And I've been having it go back and add documentation to my old code.

Say, hey, read the code and figure out what documentation I should have included. Which is smart. And you know, that's probably a whole nother topic as well. It's very easy to sort of you know spin off on these things. But, you know, so as we are winding up this episode on use cases for a rag system, you know.

So we talked about financial data. As nice as it would be to do trend spotting in your finances, it probably helps to have a very large uh data set, but also don't ask it to do analysis. What are some other use cases that are just kind of off the table as marketers think about do I need a rag system? Again, it goes back to size and privacy. If you're using a model like Google's Gemini or Meta's Lama 4, and you can fit the data inside its working memory, a rag system is overkill.

It is it it's add unnecessary complexity to the system. It sounds cool. Sounds cool to add a gentic rag to your system. You'll you'll get that innovation award, uh, but it will it will not make things better. If you're a marketer using these things to do like marketing strategy, there's a good chance your information will fit inside the context window.

If it doesn't, like maybe you're it you're in a super competitive industry like chewing gum, and you have reams and reams and reams of of market data and customer data, then yes, a rag system there would make sense. To you know, hey, I need to to find all the comments online about bubblegum. Like, okay, that's gonna be a lot of information to dig through, and a rag system would make sense in those cases. So size and privacy are are the two factors. If you have both, you definitely want a rag system.

If you have one but not the other, reg might or might not be the right choice. And if you have neither, don't over don't overcomplicate things. And when in doubt, use the five Ps to start to sort out what your thoughts are. Try to figure out what the heck it is you're trying to do. And AI may or may not even enter the conversation.

Um so you know, to Chris's point, don't overcomplicate it by you know falling for the shiny object. Um, and one of the things that you can get in a mere five days, five days from this live stream recording is our Trust Insights use cases course where we talk through how many use cases, Chris? Do you know if you want? 21 use cases for using generative AI. So you go to trustinsights.ai slash use cases course, launches in five days, available for pre-sale right now.

Um, and if you want more information about that, you can find that in our analytics for marketers Slack group. That is free to join. Trust insights.ai slash analytics for marketers. What else do we need to say about this episode? That's it.

And I think uh we next week we are off because we're all scattered to the wind. So we'll be back in two weeks. That's right. All right, folks, we will see you on the next one. Thanks for watching today.

Be sure to subscribe to our show wherever you're watching it. For more resources and to learn more, check out the Trust Insights Podcast at TrustInsights.ai slash TI podcast. And a weekly email newsletter at TrustInsights.ai slash newsletter. Got questions about what you saw in today's episode? Join our free Analytics for markers Slack group at trustInsights.ai slash analytics for marketers.

See you next time.