So What? Generative AI in 2024

Well, hey, howdy everyone. Happy New Year. Welcome to So What the Marketing Analytics and Insights live show. I'm Katie, joined as always by Chris and John, who I can I can lift up today. We're never quite in the same configuration.

So when you guys are like the two top boxes, it's a little easier than me trying to figure out left from right. Happy New Year, guys. Welcome back. We haven't done this in a while. So glad holidays are over.

Our chief statistician, John is also our chief grumpy pants. Chief Chief Grinch. So we're gonna get right back into it. Um as expected, we're talking about generative AI. Specifically, we're talking about generative AI in 2024, and we're doing today's episode uh Ask Me Anything style.

So if you have questions, feel free to drop them in the comments and we'll try to get to as many as we can. We do have a bunch of questions already that we're gonna try to get to because that's that's the topic de jour. Generative AI. What do I need to do? How do I use it?

What does it mean for me? So we're gonna try to answer some of those questions today. And this I suspect uh guys will become something we do quite often on this show. So um let's just start at the top. Um is it possible, Chris, to sort of summarize what's new in generative AI?

Let's just say, like in the past 10 days. 10 days. A bunch of things have happened. And let's scope it to just before the holidays as well. The big one from a regulatory perspective was the EU parliament passing the EU AI Act, which is and we've talked about this in the past.

The EU tends to be the leader in the world of regulation on major technology and stuff, especially when it comes to individual rights. For those who uh may or may not remember, back in the the last decade, the EU was the one of the planetary leaders on data privacy, culminating in 2018 with an act called GDPR, the General Data Protection Regulation, which became sort of the gold standard planet wide for how to protect data. And many other legislative acts, uh California's CCPA and CPRA followed in the footprints left behind by GDPR. And so many of the other privacy legislation that that exists in the world is a derivative of that in some fashion. If you comply with GDPR, you're pretty much compliant with all of the other, you know, follow-on acts, if you will.

Well, the EU AI Act appears to be going the same general route. Uh, and that means that because it's setting the the the tone for and about the ways AI should be used, it's likely to be the sort of the gold standard for it, and then other uh places in legislation will will probably follow in its footsteps. There's an entire module in the Trust Insights course, the Gender of AI for Marketers course, which you can get at Trusted Sights.ai slash AI course. I believe it's chapter six. Um, um, uh that goes through all the the five major provisions, the six major provisions of the EU AI Act.

The one that is really important, well, this this we call it the 5D process, right? Um, which is uh number one is deter, don't use prohibited cases of AI. So there's a whole bunch of things that EU AI Act lists like, yeah, don't do this with AI. Um it's all the things you'd expect. Uh there is uh defend, defend the rights of individual users.

There is document, which is document the use of AI. There's disclosing the use of AI. I can't remember what the fifth one is, which is why it's in the course. Um the big one, the big one that affects most marketers is uh document and disclose. The EU AI Act makes disclosure of the use of AI mandatory.

We've been saying last two years, um you really should be disclosing the use of AI. It helps reinforce copyright and so on and so forth. And now with this new legislative act, it's not optional anymore. So when we if we if we were to put up uh you know the new theme music for this show, we would have to disclose generated by AI. We want to be compliant with the EU AI Act.

If you put up a blog post image on your blog and it's uh you know, a dog on a skateboard wearing a tutu, you need to have some disclosure saying, you know, made by AI so that you're compliant with the law. That's one of the really big developments. The other two things uh that have come out that I think are really interesting. One is in December, we saw uh Miss Stroll, the French company, release a new model called Mixed Rall, which is a mixture of experts model. And if you think about a tool like chat GPT and the GPT for model, that's like having an expert chef in your kitchen, right?

It's like having Gordon Ramsay living in your kitchen, completely with all the swearing. Uh Mixstral is kind of like having maybe a slightly lesser chef and like seven sous-chefs all in the kitchen at the same time. So that they're not as good as a single Gordon Ramsay talent-wise, but they are a lot more productive because you can have that many more people simultaneously working on things. So mixture of experts is just a new model architecture. I mean, it's it's not new, it's from 1991, but the implementation is new.

And that is it, it is proving to be best in class for open models that you can download and run on your laptop. That's number two. And then the third big thing is that we're starting to see really interesting multimodal use cases, things that cross a lot of really weird boundaries. Just today, uh, speaking of newness, just today, uh, Meta released an audio to uh image, audio to video uh generation model that is just mind-boggling. You give it sampled audio, like the conversation we're having right now, and it renders 3D avatars that are photorealistic of you know, synthetic people having that conversation.

So, Katie, if you and I were recording just an audio podcast, or if John and I wanted to resurrect like old episodes of marketing over coffee, this would create 3D realistic, synthetic people having that conversation. So we could like get some sampled video of us sitting in a coffee shop and recreate like the first 800 episodes of marketing over coffee. So that's but that's multimodality in ways that is absolutely crazy where you're taking audio, turning it into text, and then using that text to feed three different video models to create this synthetic stuff. So it's is just in the last 10 days, that's what's been happening. Well, that's a lot.

And I think that you know what we what we've seen so far is that the technology is changing so quickly. Um I totally think that you should absolutely take that number three point and do that with old episodes of marketing over coffee to, you know, well, you know, and I mean it in all seriousness, because one of the things we know is that content is king. Let's just say like people love that phrase content is king. Content is what people consume. That's how they learn about you, it's how they engage with you.

And I think that marketing over coffee would be a really good opportunity because you have so much content and you have so much high-quality content to really experiment with some of these tools and demonstrate the capabilities while also repurposing your content, maybe to a whole new audience. So, you know, thank you for that summary, Chris. Um, one of the questions that we have is what are some use cases for operations using AI. So before we get into that, I want to highlight one other use case that's come up actually in our Slack group. The uh, if you go to uh trust insights.ai/slash analytics for marketers, you can join our Slack group there.

Uh, it's free to join. Uh, was someone saying recently, hey, I love marketing over coffee. I would love to share it with our team, but they only speak to Portuguese. Um, what's the likelihood that we can get uh marketing over coffee in Portuguese? And the answer used to be um probably not soon.

So again, this is an open source library released by MetaCo. It's called Seamless Communications, a seamless M4T is the name of the model. I will give you the the upfront caveat. Getting this thing to work is a pain in the ass. It requires extensive technical knowledge and a willingness to uh roll up your sleeves and you know compile code install libraries, all that fun stuff.

But if you do it and you're willing to endure the pain of doing it or setting up the environment for it, what it does is pretty cool. You give it audio in one language and it translates the audio into another language. So you could have marketing over coffee in Portuguese or in ear insights in Ukrainian or any of the, I think it's like 73 different supported languages. When we talk about uh marketing and even marketing operations, just having your content available to other markets is pretty cool, especially if it does not require additional work from you as the human, right? So translation and translation technologies like this are pretty incredible.

Now, I will also say seamless M4T is buggy as hell. Um I I've been playing with it uh extensively and it it's got a little ways to go, but there have been some other new tools. Uh Unity AI uh is another uh package that exists that can do voice cloning, uh you know, again, running on your desktop. So I think from a marketing operations perspective, the being able to take stuff that you've already built and putting it into many more different formats is huge. I would agree with that.

I think, you know, we often talk about the video transmedia framework where you're basically starting with one piece of content, maybe it's recorded video, and then you are pulling it apart to repurpose it. So you're doing the work once and then getting a lot out of it. This almost feels like the opposite process where you're starting with something small, like maybe an audio clip, maybe even a blog post, and using generative AI to create a video file, create an audio file, translate it into multiple languages. Because you know, Lord knows we can't wait for John to finish learning Portuguese. That's a blind spot on my part there.

Not picking up any language. Testing has proven that that's a futile exercise. What do you think, John? About, you know, so I mean, we're using marketing over coffee as the example. Is this the kind of thing that you would be interested in using generative AI for?

Yeah, the translation thing is really interesting. I mean for marketing over coffee, but I just think in general, I think we're gonna there's gonna be some major societal shifts when people can just speak every language. We're very close to just real-time translation on a phone, you know, when you go around. So I yeah, that's amazing. That's fantastic.

The meta thing that Chris was talking about, where you know, you just give it an audio file and it creates, you know, fresh content. That's really interesting as far as in content production for movies and TV and stories and things like that. I mean, imagine they basically get a star to train the model once, and then that's it. You just feed it the scripts and the shows come out the other end. I mean, that's kind of insane to think about.

In fact, I did notice they had David Schwimmer as one of the audiophiles in there just to uh drive that point home for anybody that didn't, you know, didn't see that coming stealth. Um, yeah, the only other thing that has hit me recently, though, is you know, content is king has always been the mantra, but I've heard a lot of people with the dawn of AI going with, well, content, uh unique content is gang. You know, you have to have something that's at least better than the average stuff out there, or you're gonna be one of a million of the same thing out there. I feel like that sentiment has always been true. It's just never been outwardly stated.

Yeah. Um, you know, you have to have good, high quality, unique content in order for it to stand out. But now with generative AI, and we'll get into some of the uh SEO and search questions, you know, it's even more important to have really good high quality content. Otherwise, I think what's the term, Chris? The sameness of it all starts to happen.

Because if you're using generative AI to create your content, it's gonna look the same as everyone else because everyone's using the same training models. And that's a really important point because part of what will make you successfully using generative AI for content creation in particular, which is one of marketing's big use cases, is leveraging your own data. Something that came up in this week's um inbox insights, Trust Insights newsletter, which you can get a trust insights.ai slash newsletter. We talked about data is one of the differentiators for how your generative AI efforts will differ from others. So a couple really simple examples.

One if you are willing to invest the time and spend a whole lot of time digging around in the models themselves, you can get them to approximate writing style really well. Now, writing style is one of those things that is very challenging because of imprecision in language. So the reason why AI doesn't work as well as we would expect it to is because our language is shockingly imprecise, uh, even though we use it to communicate every single day. If I say he was wearing a red dress, that there's so little information in there about specifics to visualize that that it requires additional content to create a context. With generative AI, style is one of those things you can't describe because it is so ambiguous because and it's it's so difficult to say, and there's so many components of it.

There's word selection, vocabulary, sentence structure, pacing, grammar, et cetera, et cetera. What you can do is feed either examples to a machine to have replicate a certain type of writing style or do what's called priming, where you have you you take a we do this with our our synthetic version of KD, KD GPT. I've taken probably close to a hundred thousand words of stuff that Katie's written, and then asked uh the language model on each of the what is it, 22 aspects of writing style, describe Katie's writing style for this, and then taking that feedback, passed it back through a model, and said, now summarize each of these things into the specific keywords to create a priming representation. That allows me to have a very, very complex system prompt that's behind the scenes in KD GPT. So when you talk to it, it is going to sound more like Katie than just you get something you can do with prompts alone.

That has to come from your data. You have to have the data, and it has to be in good condition, and you have to have the techniques to do that to be able to create those priming representations of those few shot learning examples to generate really good output. But when you do, it's a lot more unique. It is not the same as lame, as our friend JBay likes to say, same as lame. Yeah, if you were just saying, hey, write me a blog post about B2B marketing influencers, yeah, you're gonna get the same as lame stuff.

If you spend the time to build to codify your writing style, and then you add in unique content that you have, maybe about B2B marketing influencers. Then when you ask the machine to create stuff drawing on your data, you're gonna get stuff that is unique to you that has never been done before. Does John GPT exist yet? Not yet. If we want a podcasting flavored version of me, I guess, yeah, that would be the way to go.

No, and I think I think go ahead, Chris. That's a really, really important point that John just made. Things like podcasts are super super valuable sources of information. One of the things that I did with Katie GPT as part of that writing style development was to take, I think it was 80 episodes of Ian Your Insights, take the transcripts. I used some Python code that chat GPT worked to split the transcripts by speaker.

So I took only Katie's stuff, uh, the only things Katie have said to put that in the knowledge base with 800 episodes of marketing over coffee with John being on every single one of them. We can take all 800 transcripts, split them up, and say this is the language of John GPT. Well, and I think that you know, we've we've talked about this before that, and this is my stance is that new tech doesn't solve old problems, and so your point about needing to have really good data and data quality is it's always been important, but now where generative AI is so much in the forefront of what everyone's trying to do, having really good data quality, data governance is vitally important in order for these systems to work correctly. You know, if you have a CRM system, depending on the size of your company, you can kind of go through and like fix things as you see the errors with a system like Chat GPT or any sort of generative AI system. It's harder to do that because the models, unless you're someone like Chris, are a bit of a black box to you.

You can't really go in and tweak it. You have to know that the data that you're putting in is really the good quality, and that's something, you know, just to put a little bit of a plug that we can help with. We really have a lot of deep knowledge and expertise on data quality and data governance to help you get to that point where the data that you have is really good and you can feel confident putting it into your generative AI model. Um, so another question. Um this is sort of a big question, but um, what is the best way to integrate different software systems with AI?

And I feel like there's gonna be a number one, it depends. Number two, uh, you know, what does that mean for you? And there's probably a number three that I'm not thinking of. So yeah, let's just let's just start with maybe even helping people think about how to answer that question for themselves. Maybe that's the way to approach it.

I'll say, what are you integrating? Um, here's the thing a language model. Uh, if we're talking language models, actually, well, we can we can break it. So language models and image models and video models, sort of the three big and audio models. Okay, so four four big things text, audio, video, um, image, they all have all these things have APIs, all these things have you know inputs and outputs.

And so the the question you have to ask is how does this appliance fit into the overall workflow? Right? It's like having a blender. I you know, where does the blender fit into the workflow? If you're making margaritas, it's pretty important.

If you're making steak, you're not going to use the blender. Not if you not if you're sane. Um, and so the first part you have to figure out is and this is uh unsurprisingly gonna mirror the the trust instance 5p framework. What are you trying to do? What's the purpose?

Um, you know, who's involved, what's what are the existing processes, and then when you're talking about the integration of gender of AI, it is what platforms will you be connecting and how will you be connecting them? Again, these things just have APIs, they're like any other piece of software. So if you've used tools like Zapier or if this then that or whatever, you know kind of how APIs work. You just it's like connecting Lego blocks together. The language model is going to be used for language purposes in this flow, and then ultimately you have some kind of outcome that you measure by your performance.

The one big warning for everyone to remember is that language models are good at language. Language models are not good at things that are not language. Um, and so people trying to use chat GPT for everything or DOLI for everything, it's just gonna give you a bad result because that's not what it's good at. It's good at language, it's not good at things that are not language. So if you're trying to have it do regression analysis for attribution modeling, a marketing example, it's not going to do well with that.

It can write the code in Python that will do the attribution modeling, but can do the math. And in fact, if you've used ChatGPT's advanced data analysis, that's what it's doing behind the scenes is it's just writing code and then the code is executing and it's looking at the outputs. So my my caution there is when you're talking about integrating different systems with AI, what are the language tasks that you're trying to do if you're using language models? And then are you connecting those things together? If you're doing image tasks, are you, you know, where does that image fit into your workflow?

You've got to have your people and processes and your purpose documented out before you do the integrations. Um, you know, you talked about how the advanced uh models are doing if they're writing the code. This is uh shout out to Andy Crestedina. One of the things that he showed when I saw him speaking at content marketing world was you can ask these systems like like generative AI to show your work or show me the code that you've written. And it's like, okay, here's the code I've written.

Um, even if that's not part of what you're trying to accomplish, so that, you know, again, if you have someone like a Chris Penn on your team, you can say, here's the code for all of this. Is there a way that we can then now put this together for our own script so that we can run this without having to bring it to generative AI? I just thought that that was such a really interesting pro tip is that you can ask it to show its work and it will. And then if you understand how to read code, you can say, Oh, that's not what I wanted it to do at all. Let me fix that.

And that takes a bit of that mystery out of what is generative AI doing when I'm asking it to do something. Mm-hmm. In fact, not only can you have it show it's your work, you can have it give you the actual code, right? So, this is something I was working on earlier today, Katie. One of the things Katie asked for help with this year was getting more data out of HubSpot.

We have HubSpot. Um, we have data in it, we know what we've done to set this thing up, but we also know that HubSpot's integrations now they have a robust Python package to connect to their API. Um that's not a language I I code in very well, but chat GPT sure does. So I know what to ask it for. I say, okay, let's write the code to get all the data out of HubSpot so that we can you know spit do the work that we need to do to build an attribution model.

This is language, this is a task that a language model is really good at. So when we're talking about integrating generative AI into different systems, questions we have to ask do do the existing systems have APIs? If they do, great. Guess what? Now you have a way for generative AI to talk to it.

Does the downstream output have some kind of API? In this case, you know, one of the things that we use a lot is we use Google's BigQuery because we want to be able to look at stuff in Data Studio. And so once this is done, it's going to write its data to BigQuery so that Katie can look at all the things she wants to look at in Data Studio without needing to code. So that's an example of this integration. But the first thing she had to do was that she actually had to sit down.

We had to sit down together and write out user stories for what this thing was supposed to be doing. I think we wrote probably a couple of dozen user stories, and we found the similarities between them once we wrote them, but we really needed to spend some time doing those requirements up front because you know, Chris, you're you're reviewing all of these things sort of at a ver at a very quick pace and at a high level, but the amount of time that actually goes into writing this code, you know, working with these APIs, doing this integration, it's a multi-day, multi-week, multi-month process. And 10 times out of 10, there's going to be errors the first time around. So you have to go back to trouble. It's software development, essentially.

And that there's nothing speedy about good qualify good, good software development. That's true. Uh, by the way, pro tip if you're doing uh development with chat GPT, uh, one of the easiest things you can do to keep your keep it and you on the rails is to have your requirements written out, preferably in like a bullet point checklist, and that gets pasted at the end of your code. You can see here my requirements for this code are in the code itself as a big comment. When chat GPT processes the code, each time you are refreshing its memory as to what the requirements are, and it generally develops much more coherent outputs if you do that.

When Chris Penn starts championing requirements and requirements documentation, I know that I've done my job correctly. So you need to have a sword you can like tap him as Sir Christopher. That's right. The champion of business requirements. Well, um it's a language model.

It and requirements are language. And so if you want it to generate good language, you need to provide it with good language, and there's no better language for keeping an AI on the Rails than requirements. Hmm. It's funny how the same could be said about humans. No, because humans can ignore stuff.

Machines are you can compel machines to pay much more close attention. Well, we're not going to get into this debate today, but I feel like that is something that we'll cover in the podcast at some point. Um, so here's an interesting question. So there's been a lot of chatter about SEO and search engines and how is it how is generative AI changing? So one of the questions that we got was uh how do search engines deal with AI hallucinations now and in the future?

So before we get into that, Chris, can you define what an AI hallucination is? This is a uh a very complicated question to answer because it depends on the model. So let's say it's let's say it's chat GPT, which is the one that most people are using. So this is a very, very complicated question to answer. I thought it was simplifying it, but perhaps not.

No. Um when you build when when a company like OpenAI or Google or whoever builds one of these models, what they are doing to put it in the simplest possible terms, is that's mathematically incorrect. So if you are if you know vectors and embeddings and stuff like that, this is completely wrong, but this is conceptually correct. Um, they're basically just building really big word clouds. And every word that they scan has a word cloud around it, and it is and what a model is is a giant series of word clouds.

That's conceptually what's going on. When you prompt a foundation model, um, which is the the basic raw model compiled from you know trillions of pages of text, it is essentially looking at how word the different word clouds intersect and overlap, and what it produces is a st the statistically most relevant representation of the words that belong together, right? Words and phrases and stuff. The Transformers architecture is just looking at contextually which words belong next to each other. So we can assemble these.

A pure foundation model generates what are called hallucinations, which are statistically relevant but factually incorrect associations. Foundation models generate hallucinations 100% of the time. When you look at a raw model, what's coming out of it, it is coming out with just complete hallucinations all the time because it's all mathematics. It's all you know, this word belongs next to this word because it's seen frequently. When you get to an instruct model, um, which is like what the first version of chat GPT was before it was open to the public called destruct GPT.

This is a model that has been tuned. Uh it's called supervised fine-tuning, where um a company like OpenAI uh gives it hundreds of thousands or maybe millions of examples. Uh, you know, what is the color of the sky? Blue. Who is the president of the United States in 1996, Bill Clinton, and so on and so forth.

And it's just gazillions of these things. And it essentially tells the model, I want you to change the associations of between all these words and phrases and sentences to match the inputs and the outputs. It's called supervised fine-tuning. At that point, you're starting to reduce hallucination rates because now you're saying, hey, the like something like who's president of the United States in 2001, George W. Bush, right?

That it's it's not George H. W. Bush, it's George W. Bush, and so on and so forth. So it's it's conditioning the word associations to follow these paths.

That's at the point where hallucination rates start to go down. Then the third step when you make these models is called reinforcement learning with human feedback. So as people use a tool like Chat GPT, it gets better over time because more and more it takes the inputs it gets from our usage of it. So if I'm writing Python code and I say, hey, check your work. I don't think you did this right, it knows that its previous response wasn't as good.

And so it gets tuned and shaped along those lines. When we're talking about hallucinations, it depends on the level of model that we're using. Most people are going to use a service like ChatGPT or Google Bard, and the hallucination rate is reasonably low. It's it is almost never going to be zero, because what comes out of the foundation model itself is 100% hallucination. These things are hallucination machines by design.

And so it's this fine-tuning that shapes the results. And awkwardly, the hallucinations are decreasing, but so is the utility, because the more rules you put on these models, the worse their outputs get in terms of quality. They get more sameness, less creativity, uh, the more rules you put on. So for example, if you talk to ChatGPT about a sensitive topic, right? Or a polit uh political opinion, whatever, it will oftentimes just say, nope, I can't do that.

Or if it does respond, it's very, very bland and boring. And again, that's that reinforcement learning human feedback has been tuning it to say, hey, uh, you should respond in these certain ways, and it it's not as creative. Hmm. Did you get that, John? Remember that kid in junior high school that would just lie all the time to try and make friends?

That's what's happening here. It's coming up with an answer that's considers most probable and throws it out there. And it doesn't really matter if it's true or not. So well, it's funny, uh, you know, to sort of continue along that thought process, John. I often say to my husband, like he is, he's all like he never gives you a straight answer the first time, but he says it with such authority that you can never tell if he's kidding or not.

And so he has his whole team believing certain things about him and his life because he says it with such authority that they don't think to question that it's not correct, including you know, his favorite bands and you know who he was in a past life and you know what his hobbies are. They're all lies, they're all complete fabrications, but he says it in such a way that you're like, oh, it must be true. And I feel like that sort of along the lines, you know, in a very, very simplistic way of what you're describing, Chris. Very simplistic. And and the challenge with language is language itself is extremely imprecise.

So part of that imprecision is why we have so us, we struggle sometimes with getting language models to do what we want. So I'll give you an example. Um, this let's take the sentence a friend of mine uh sent this to me the other day. I I laughed clerics. Let's take the sentence, I never said we we should kill him, right?

Based on that sentence, that's just text. But where you put the emphasis as a human changes the meaning. If I say, I never said we should kill him, I'm saying it's not my fault, right? If I said I never said we should kill him, as if you killed the wrong person. Um, I say, I never said we should kill him.

I thought we were gonna hire a hitman. Um that same string of text changes its meaning based on our our spoken emphasis, which is not reflected in just the words themselves. And so we have this ambiguity. And so when we talk about hallucination, a good part of hallucination is because we're dealing with ambiguity in language, language is inherently imprecise. That's one of the reasons why a tool like ChatGPT does so well with coding, because Python and other languages are precise language, it either runs or it doesn't run, and there's not a lot of ambiguity in the actual word assembly of the word, the tokens.

Whereas writing a blog post, yeah, there's a million different ways to talk about B2B marketing influencers. And if we don't provide a lot of specificity in our prompting, we are going to get ambiguous and or sometimes hallucinatory outputs. That's a killer example. I love that because it is, it plays differently totally differently with every reading. It does.

I mean, that's that's actually an exercise I've seen people do, like public speaking things. They'll take a sentence and say, now change the emphasis on every each time you read the sentence on a different word and see how it changes the meaning. Oh, even though link like English is not a tonal language, like Chinese, for example, um, that emphasis changes meaning and it's not reflected in text. So going back to your question, Katie, about how search engines deal with AI hallucinations, the number one way is they don't use models for knowledge, right? Uh, if you look at Microsoft Bing and its use of chat GPT, uh the GPT board model, if you look at perplexity AI, if you look at other tools like this, what they are doing is they are acting as a language interpreter.

And if you can watch this, when you use Microsoft Bing with um Copilot, you'll say, hey, you know, what is a good recipe for guacamole? You will see it translate your text into Bing search queries, make those queries, get results from Bing's search archive, and then aggregate and synthesize and answer. So very smartly, what Microsoft is doing is saying this language model is good at language, but we know it's bad at facts. So we're going to instead offload the fact part to our search engine. Bing comes back with some results, it synthesizes that those results into language again that is conversational in nature.

And you're like, oh, that's helpful. Compare that to Google Bard, which does try to use uh in previous in the previous versions, it does try to use the knowledge of the model and it comes up with hallucinations. In fact, so much so that you they've they've actually put in a button that says, you know, check Google's responses, and then it goes back and Googles the things it wrote and said, Oh, yeah, actually, I lied, I completely made that up. Um, and so that structure of going to a different source of truth is how search engines deal with that. That is contingent, obviously, on the search engine itself and how good its search engine results are, right?

So if you ask a search engine for something that is scientifically questionable, it will give it to you, right? You ask Google, I you know, show me videos or show me text that proves the earth is flat, like Google will find that for you. It's wrong, but it will still, it is factually wrong. But there's a lot of content on the internet about that, and Google will surface that. And so part of this is one of the big challenges with AI and with search itself, is just because a machine can find it doesn't mean it's true.

Mm-hmm. I think that one of my favorite uh descriptions of that is you have the entire solar system when every other planet is a sphere, and then the earth is just a flat like thing. It's like, how do you think this this is a thing? But okay, let's go with it. So I, you know, I don't want to, I think misinformation is it's a whole topic that we can cover.

Um, you know, but I think that that's a really important uh disclaimer is that there's a lot of it misinformation that's gonna be spread even more quickly than we were, you know, being able to sort of get our arms around it before. Um of the questions, Chris, because we are sort of uh we've been talking about this for quite a bit. Um, one of the questions that came up was why image generation prompts don't work as well as text generation prompts. And before you get into the weeds and the technical details, one of the things that you said to me the other day just totally blew my mind. We were recording the podcast, and you said something along the lines of generative AI can't see what it's showing you.

And it was like it's it was such a factual simple statement, but it just like blew my mind because it's a machine, it's numbers, it's code, it doesn't have eyeballs or a way to actually see. So if you say this is a picture of a chicken, it's taking your word for it. It's saying, okay, this is a picture of a chicken. It can't look at that and be like, John, you're lying. That's a donkey.

And that is like totally blew my mind. So image generation prompts, why don't they work as well as text generation prompts? Because they're different. Hey, they're two, there's two things. They're different mechanisms, right?

So image generation is done with a technique called diffusion, uh, whereas text generation is done with this process called transformers. They're similar mathematically under the hood, but they're not the same thing. And because of the way image generation models are trained, the prompting structure is going to be different. The general process for training an image diffusion model is to take a gazillion number of images and their associated captions or text or alt text or whatever, and create associations. So you would have, you know, picture of Jimmy Carter, uh, you know, picture of Jimmy Carter at Camp David, and so on and so forth.

Those captions from news sources, from image catalogs, Amazon, alt text for for visually impaired, all that becomes this matrix of here's a word and here's what's like associated with this type of type of image. And then when you prompt an image generation model, you give it words, and then what it does is it takes sort of a mathematical average of all the images it knows that contain all these different words and try to meld them together and chips away at all the pixels until it gets something that reaches a mathematical average of the words that you've used. So if you had a dog on you know wearing a pink tutu write driving writing a skateboard, it's going to have sort of a mathematical understanding of those different concepts from all the things it's trained on, which is why in in pure image generation systems, you have these really strangely worded prompts, like picture of uh an envelope ejecting through the cockpit of an F18 superhorn, photo realistic, highly realistic photo, 24 millimeter lens DSLR, right? That's an image generation prompt. Makes very little sense to us as spoken word, but it's because we know we're taking advantage of captions and highlights and you know descriptions from museum walls that that these image models were trained on.

Because of that, if you don't know the syntax of how an image generation model works, you're going to get subpar results. Combine that with the fact that what is typically put into image generation models are very short snippets of text. Again, they're captions or alt text. They're not paragraphs of description. Um, John, as an example, describe this.

That would be purple chicken with Santa hat and scarf. Right. But you could spend a lot of time talking about this, right? Like this is clearly a holiday chicken. It's probably a plastic holiday chicken.

It seems to light up, it's got a scarf, it's got a hat, there's a little bit of mistletoe on the hat. It's just it's a painted piece of plastic. Um, the chicken doesn't look super thrilled to be here. Right. There's so many different things you could say about this.

What's the old adage from from the classics, right? And uh a picture is worth 10,000 words. You could easily, you shouldn't, but you could easily write 10,000 words about this thing. And yet an image model's been trained on maybe 20, right? And so when you don't get good performance out of image models, it's because the words you're using don't occur in captions, in in image captions, is essentially what it boils down to.

Now, there are services like stable diffusion, which are trying to help build libraries where machines can auto-tag and create more enhanced captions on images they already have control over. Um, but that's a long complicated and expensive process. And so the reason why image generation prompts don't work as well if you're just trying to use natural languages because the models were not trained that way. And the best models that that are, you know, in terms of easy use and performance typically have very large data sets, but they still require a little bit of that tweaking. When you use the Dolly function inside chat GPT, what it's doing is very much what Bing is doing.

It's taking your words, translating it to a Dolly compatible prompt, and then pushing that into the system. And that's where that our conversation point came from earlier, uh, Katie, which is it's passing a prompt to Dolly, it can't see what Dolly's doing. It has no idea what Dolly is doing. So it just gets the result back from the API. It says, here's your thing.

Did you like it? And you're like, no. Uh I had this problem with a client. Um, I was doing a picture of four people in a car on a road trip, and it kept putting five people in the car. And I would keep saying, no, no, try again.

You have four people in the car. No, there's five people in this car. Again, that's a because of the caption data that was trained on. I what I end up doing is saying, make a picture of three people on a road trip in a car. You're surprised, I got a car with four people in it.

Like, I don't know what is wrong with this training data set, but it's clearly clearly got some problems counting. I feel like creative directors around the world are rejoicing, saying, Finally, people understand our pain when our clients say, I don't know, just make something pretty or I don't make it blue. Like, what does that mean? What shade of flu? What kind of flu?

What is it about it that you want to be blue? And so it's what strikes me, and you know, we've sort of talked about this uh quite a bunch as well, is that all of the struggles that people are having working with these generative AI systems are the exact same struggles that people have as managers, as communicators, as delegators. The difference here, and you know, Chris, this is where you get excited, is that the machine doesn't talk back and say, I need to take a break and go get a nice coffee. Um the machine says, Okay, what else do you want me to do? And it's very agreeable and it's very compliant, but you still have to know what it is that you're asking.

And so for someone like me, seeing this just sort of play out, I'm like, well, now all of the struggles that I've had trying to get people to document requirements, understand why they're important, think about specificity when you're delegating or you know, giving instruction. Like this is all coming to light, it's all surfacing of like, and this is why. And so now we can take all of these skills that we're learning working with generative AI systems and reapply that back to our teams. Like, oh, now I understand. If I'm clear in my instruction when I ask my team members to do something, they'll know what I'm asking, and there won't be frustration on both sides to be like, well, you didn't tell me what you wanted.

You just said fancy font. I don't know what fancy font means. Yep. And I think that's a really good example. Plus, um, what just came in in the comments as well.

Uh with a lot of these systems, we have to think about them. The way they work under the hood. And again, this is something actually going to be coming covering in an upcoming course uh in the Trust and Science Academy on advanced prompt engineering. But the way they work is by essentially selecting uh probabilities, right? Under the hood, there's they're selecting different probabilities.

When you give it a task that is entirely self-contained, like make a one-page opt-in site, the number of probabilities that can be invoked for that are so many that it that the machine will not be able to complete the job. It's like saying, Hey, drive from New York to San Diego, like just give me just give me three three sets of uh you know, three three bullet points on how to drive from New York to San Diego. Like, well, get in your car, start driving until you get there, right? It's it's gonna be unhelpful. If you say, give me turn by turn instructions on how to drive from New York to San Diego, you're gonna get a much better result, right?

The same is true of language models. It's a system called chain of thought or or um well, there's a variety of different ways to describe it, but essentially these models, when they're choosing probabilities, they need room to do so, they need to choose probabilities, then consider everything giving you to choose new probabilities and consider everything giving it and so on and so forth. It's recursive, it's the nature of the transformers architecture. And so when you say, instead of uh give me a one page opt-in site, say let's first talk through how would you do this, right? And have it write an outline and then say, Great, now let's talk about step one, right?

In the HTML, step two, writing the body copy, step three, writing the things, and if you follow those steps, it will generate uh really good results. When I'm doing something like the this code that we were working on earlier, I do not say we're gonna write um uh code to interface with HubSpot's API because I will get nothing useful back. What I'll say is first outline the steps that you need. Now frame out just the function names and placeholders for the functionality, and it will write that. And then I will say, okay, great.

Now let's write function insert to database, write that, and it does it. Okay, now let's write the safe int conversion function. Let's write that. And then you assemble the pieces because you're doing chain of thought, and you end up with very strong, coherent uh outputs, but you can't do it in one shot. It has to be done in multiple steps.

What you're saying, Katie, very much like working with a really smart intern. You can't say, hey, go make me a website. Like, no, no, no, you have to tell the intern step by step and work with the intern step by step to get the output that you want. So as we're wrapping up, because we're just about at time, Chris. Do you feel like working with generative AI systems is now going to change the way that you interact with human team members?

Um, yes, it makes me less tolerant of working with human team members because I have to deal with all of their needs and stuff, and you know, it's like I can't do that. I'm you know, I'm having a bad day. Like, no, I'm just gonna work with machines because machines don't have bad days. Um, I can work with them, I can I can give them this exact requirements. I don't have to worry about hurting their feelings, I don't have to worry about um all the frailties of of flesh and blood.

It is it to me, they are the perfect co-workers. So let me rephrase. Do you think it has changed your approach in terms of giving instruction to a human team member? Oh, absolutely. In fact, we had uh we were working with an intern last year.

Um, and I found myself in giving directions to the intern, just essentially writing prompts. Like I wrote prompts for the for the the human intern. Um yeah. John, any final thoughts? No, I think that's a great place to wrap it up there.

It's great for documentation. I think that's the right approach. And it is, you know, this iterative approach is better, and we just have to deal with the human inconsistencies. That's just part of the mix, but that's okay. I, you know, there's a lot of questions that we didn't get to.

So we'll definitely be doing more of these uh AMA style live streams over the next few weeks. The other thing I just want to note for those of you who have been watching the show is that Chris is a wealth of knowledge. You know, we haven't yet been able to stump you on the details. And so if you're interested in bringing Chris in to your organization or your agency, you can visit us at trustinsights.ai slash AI services. And we have a lot of different options for you to be able to work with, you know, the caliber of expertise, Chris, that you have on AI in general, but also more specifically on generative AI and the more deeply technical.

We've been doing, we're focusing this year a lot on education. Um so, of course, we have our uh new generative AI course that launched at the end of last year. You can get that at TrustInsights.ai slash AI course if you want something that's more self-paced. But if you're looking to bring in that expertise in-house and have someone really talk to your team about generative AI, then I think that, you know, bringing Chris in is a really good option. I think that's a a great place to end.

And I, you know, if you folks have other questions that you have about this stuff, you know, by all means let us know because we're always trying to we're trying to find convert the knowledge into useful practices, right? Because the the technology is really cool, and I really could spend all day talking about it. But at the end of the day, it has to convert into something useful. So that's going to do it for this week. We will talk to you all next time.

Thanks for watching today. Be sure to subscribe to our show wherever you're watching it. For more resources and to learn more, check out the Trust Insights Podcast at Trustinsights.ai slash TI podcast at our weekly email newsletter at TrustInsights.ai slash newsletter. Got questions about what you saw in today's episode? Join our free analytics for marketers Slack group at TrustInsights.ai slash analytics for marketers.

See you next time.