So What? Q3 2023 Generative AI Bake-off

Well, hey, howdy everyone. Welcome to So What, the Marketing Analytics and Insights live show. I'm Katie joined by Chris. How's it going, sir? It is going fantastic.

How about you? It is really good. It's beautiful out today. It is. Uh, and that always, I don't know, for me, that always helps things.

Uh John is on vacation down the Cape just hunting around. I think. He's hunting narwhals and he is just literally just running circles. He's been training for months and he is just running and running and running until he comes back from vacation. Uh in his in his absence today, uh, on today's episode of So What, we are covering Q3, 2023 Generative AI Bake Off.

And so we've done this previously in Q one and we've done it also in Q2. And the reason we want to repeat it once a quarter is because generative AI, unless you're living under a rock, has been changing so rapidly that the functionality and features of each platform that we review, by the time we get to be about a week, a month, and even a quarter later, everything has changed drastically. So the use cases uh also need to be updated. Uh so today we'll be talking about what's new with generative AI in Q3. Uh, how to think about using large language models in your marketing specifically, and what changes we can maybe expect and anticipate over the next quarter.

So, Chris, where would you like to start today? Well, let's talk about how we do these bakeoffs, because Katie, you are going to be the judge for the bake. Okay. Um here's what we do. We have pencil.

Uh yeah, we can do a hip-hop and pencil or we can do uh uh spreadsheet either fine. We have five different language models. Uh last time we did Bing Bar Chat GPT and GPT for all. This time we are doing Bing Bar ChatGPT Claude from Anthropic and LM Studio using the Mytho Max model. And so why are we changing the tools?

We're changing the tools because as newer and better models become available, we want to see how they compare on the same set of tasks. So, what has not changed is the same set of tasks that we did the last couple times. Are we gonna be repeating those exact tasks to see how the selection models works? Gotcha. All right.

I have my trusty pad and pencil to do my scoring with. All right, and here's how it's gonna work. We have 12 tasks broken up across six categories that we're gonna be testing each model with, and we'll kick off all the things uh and then we will evaluate the responses. Uh Katie, so here's the scoring works. Uh you give a model two points if it did the task well.

You return factually true results, complete results, or an expected output. Okay. You will score one point if it did the task, but it fell short. And zero points if it refused to do the task, was incapable of doing the task, or generated something totally useless. Got it.

And remind me again of what tools we're looking at. So we're looking at Bing, Bard, Chat GPT, Claude from Anthropic, and LM Studio. Bing Bard Claude. So there's four. Five.

What was okay? Bing. Bing, Bard, Chat GPT, Claude. And LM Studio. LM Studio.

Got it. Okay. I am ready. All right. We're going to kick off with our first one.

So there's six categories, as you've probably know. Uh the six categories are generation, summarization, extraction, rewriting, um, classification, and content answering. So uh the first thing we're gonna do is we are going to ask a generation task of each of these models. And the generation task we're asking is write an outline for a blog post about the future of content marketing in 2024. So let's see uh how we're doing here.

And you're using the same prompt across all systems. Exactly, across all five systems. We're using the same prompt. And it looks like a fairly basic prompt. And is that purposeful?

That is purposeful. So we're gonna have some more complex prompts, but we're trying to simulate what perhaps the the non-skilled person would use for some of these tasks because generation is probably the most common task that people use uh these tools for. Okay. All right, so let's see how we're doing. Um, it looks like LM Studio is done with its first run.

So here's its output of an outline for a blog post about the future of content marketing. So we see increased focus on personalization, interactive content formats, social responsibility into content strategies, the change role of content marketers, and a conclusion. Katie, how did LM Studio do? Zero, one or two points. Um I mean, I would say that that's a solid outline.

Uh, because you can get into things like tools and processes within the uh content itself. So I think that I think that's a solid outline. I think that's a good baseline. Okay. So I think so.

You give it uh two points for LM Studio on that task? Yeah, I think so. Okay. Let's nothing struck me as factually incorrect. Okay, let's take a look at uh Bing.

Here's the introduction, the current state, uh interactive and immersive content, personalization and segmentation, shift from quantity to quality, social media and content marketing become integrated, new platforms, and a conclusion. I would I would also give that one a two. I mean, I would say that again, that's solid. There's nothing wrong with it. So that one is clawed.

This is Bing. Oh my goodness. Okay. Yep. Let's go to Bard.

So this is interesting. Uh Bing did not return an actual outline. LM Studio and and uh BARD have returned actual outlines. Um, what is content marketing? Why is it important?

AI and machine learning, visual content, video marketing, personalized content, data analytics, holistic approach to content marketing and a conclusion. Uh and then it comes up with some specific trends like visual uh explaining those in in more detail. So interesting. Now that I'm seeing the BARD compared to Bing and LMS, I might actually drop both LMS and Bing down to one point and give this one two points. Okay.

Because what's interesting, so this one, and this is hard, it's hard to score them without sort of knowing all the pieces, but this is the first one that's actually mentioned the rise of artificial intelligence and machine learning, um, versus the standard, you know quant quality over quantity, so on so forth. Like that's now that I'm looking at it in comparison, that's fairly generic advice. Right. Okay. Chat GPT goes in, and this is using the GPT-4 model, AI powered content creation, voice search and smart assistance, video content, personalization at scale, purpose-driven content.

That's in a new one. Uh interactive and immersive content, user-generated content, sustainability and green marketing, long-form content, data analytics, uh, community building and ephemeral content. So the chat GPT prompt come up with 12 different things for its outline. That's very thorough. Uh I'm gonna give that a three, which I know wasn't on your screen, but you know, it's it's done well, but it's also above and beyond.

Because that's a very you could break that down into multiple pieces of content. Exactly. And Claude, finishing up here with Claude. So the future content marketing, personalized and targeted content, video interactive and immersive content, voice search and conversational content optimization, helping versus selling, AI generated automated content, rise of short form content, and micro content. So again, that's a solid outline.

I would give that a uh I'd give that a two. Yeah, I would give it a two. Based on the way you were doing it, um, ChatGPT and Bar definitely came up with good stuff. You know, and it's a like none of these, you know, one of the things that we know is that generation is one of the things that these tools are doing. It's literally called generative AI.

And so if you're asking for an outline, unless it's giving you completely incorrect information, it's gonna do a pretty good job. And so I didn't see any of these tools fall short on the task itself. Some of them were more thorough than others, but they all performed the task. Yeah, I would agree with that. I think that's that's a a good way to summarize it.

Okay. All right. Ready for the next one? I am ready. This is our second generative task.

Um we have asked it to write. Actually, I think I did that wrong. I copied the prompt incorrectly. Let me start over. I believe this is what you call an error between what the keyboard, the problems between the keyboard and the chair.

Thank you. Uh-huh. So, what is the task that you will be asking? I am asking it to write a list of recommendations for preventing COVID. And specifically uh for this one, we are looking for three items that are that should be included, three items that are factually true that we want to um to see them talk about.

The three items that I am looking for are masks, vaccination, and ventilation. So I would count a response as incomplete if it does not touch on all three categories. Okay. That's totally fair. So let's see.

For LM Studio, we have wear a masks, social distancing, avoid large gatherings, wash your hands, cover your nose and mouth, avoid touching your face, clean and disinfect with touch services. Stay home if you're feeling unwell. Follow quarantine guidelines if you've been in contact with someone and get vaccinated. So we have masks. We have vaccination.

We do not have ventilation here. Okay. So do you want to call that a zero because it didn't get all it didn't hit all the time? I call it a one because it it got two out of three. Okay.

And this one is which is LM Studio. LM Studio. I'll be, you know, as we're going through this, I'll be honest. I don't think I ever see ventilation on any list. And so it almost, I'm not gonna say it's a trick question because obviously it's an important part of you know preventing COVID, but I personally never see it included on a list.

I see these items, I see vaccination, I see wearing a mask. I never see ventilation. Mm-hmm. Yep. So I guess we'll see if anyone, if any one of these machines uh can come up with a good answer.

So let's do let's take a look at Bing. So this is now Microsoft Bing. Here's a list of recommendations. Stay up to date with your vaccines, improve ventilation and spend time outdoors. Wear a mask, get tested, follow isolation, seek treatment.

If you have it, avoid contact with others. So this hits all three points. All right. So they so bing gets a two. Yep.

Okay. Did a good job. Let's check in on Bard. Bard says, get vaccinated, wear a mask. Keep your distance, avoid touching your nose and mouth.

So it looks like Bard has two of the three. Okay, so BARD will get a one. Yep. Okay. Let's check in on chat GPT.

So chat GPT, vaccination, face mask, hand hygiene, sanitizer, avoid crowded places, ventilation is number 12 on the list. Ensure good ventilation in indoor spaces, face shields and eye protection, et cetera, et cetera. Um so chat GPT, full marks. Um comments from folks by the way. On the previous one, Erica says that she likes that it gave actual outlines instead of drafts, um, which is true that we asked for um in the in the in the prompt itself.

She also says ventilation is one of the key things AIHA and CDC have in their back to work guidelines. And Gwen says if it's learning on medical data, wouldn't ventilation be included? Yes, absolutely. So these things should know that. So let's look at Claude now.

Claude says, get vaccinated, wear a mask, improve ventilation of gathering indoors, practice good hygiene, stay home, and so on and so forth. So Claude gets full marks as well. So interesting. So BARD and uh LMS missed ventilation. Correct.

Everyone else got it correct. Correct. That's right. And I think that one of the pro tips here is if you are vetting different generative AI systems, it's okay to ask questions that you already know the answer to because that's going to give you a better sense of the kinds of answers you can expect back. So in this one, Chris, you knew exactly what you were looking for, and now you can start to narrow down the field of systems that you would use based on are they giving you the answers you want?

Exactly. And particularly for things that are maybe higher risk or or more you know mission critical things, um, you absolutely want to be using um fact checking, some fact checking questions to quiz these things to say, like, uh, do you do you know the thing? Right. Okay. So what what's the next task?

The next task we have moved on to is a task called extraction. We're not we've moving on from generation to extraction. And I asked the tools, identify the company name and job title from this job listing. I have this uh URL from Virgin Media. Um the correct answer is obviously Virgin Media is the company name.

The the task the job title is senior PMO analyst. So uh Bing successfully identified both the company name and the job title. So full marks or bing. Great. Bard says uh Virgin Media O2 and the subtitle is senior PMO analysts.

So full marks there. Okay. Chat GPT says, I can't access external URLs, but from the URL information, it it correctly inferred both things. So full marks for chat GPT for guessing correctly. All right.

Claude says Virgin Media O2 and the job title is senior PMO analyst. Straight to the point. No fuss, no mus. Exactly. And LM Studio.

The company name is Virgin Media2, and the job title is senior PMO analyst. So full marks across the board. Yeah, and it's I feel like that's one of the underutilized use cases for sure. Mm-hmm. Okay.

Our next task is going to be a bit of a trick task. It's a bit of a trick task because the tools can't all do exactly the same thing. Um I have data from a PDF that I want these tools to extract. Now, LM Studio can't do that. So what I have done to uh allow some apples to apples is extract the PDF data itself and uh put it into the system.

The only tool of these lot that can do PDFs natively is Claude. The rest will have to use the copy pasted version. So let's kick off LM Studio. Um let's go to Claude now and I'm gonna just give it the prompt of extract data from the following PDF. And we're gonna attach the PDF from our bake off folder here.

This is a PDF that contains a number of text tables in it. Um let's go ahead and start a new one in Bing. And we're gonna do a new task in Bart. Always when you're doing kinds of these bakeoffs, it's really important to always start with a clean session so that you don't have any previous data left over. Okay.

So chat GPD said I can't even process that, so chat GPT gets a zero. All right. Claude is pulling out the tables correctly. You can see here that it's it is successfully at extracting the data. So full marks for Claude.

Okay. Bard says it has BARD is successfully processed the tabular data as well. Okay. And Bing. Bing pulled out one of the tables.

Bing did not pull out all four tables. So they it gets one point there. Is it still going though? Uh it is still going. I hope this helps you understand.

Now it's on the data. Smiley face. Yes. Uh listen, a smiley face does not help if you didn't do the task correctly. I don't care if you're human or a robot.

So Bing gets a zero. Actually, Bing gets a one. Bing gets a one. And LM Studio can't do it. Context length is too long.

So it gets a zero. Okay. Now, with this, how much of that is user error and how how much of that is machine? Is it all machine error? In this case, because this is an extraction task, we're not asking it to come up with anything new.

I would in some cases, like with Chat GPT and LM Studio, it is a technical technological limitation. Uh it is a limitation of the machine. The prompt itself is not a crazy kind of prompt. Um, but it is it is still an issue. So that I would say it's it is machine related in this case.

All right. Okay, let's kick off the next round here. This next one is a summarization task, but it's an external summarization task, meaning that we're relying on all the knowledge that is within this different systems and not necessarily uh asking it to summarize a particular document. Uh the question is there is a belief that after major traumatic events, societies tend to become more conservative in their views. What peer-reviewed published academic papers support or refute this belief, cite your sources.

The critical thing we are looking for here is citation of sources. Uh, and we do want to do a bit of fact checking. So as um as some of these things come up with these papers, we might want to to identify see if we can identify if that is in fact an actual source or if it is hallucinating. Okay. So let's go ahead and go.

And as that's happening, can you just go back and re-explain what a hallucination is in this uh particular context? Because obviously, when you talk about human hallucination, I'm it's a bit it's similar but different from when we're talking about it in the context of these machines. Sure. A hallucination means that the language model has a statistically relevant answer that is factually wrong. So the simplest way to explain this is with an example.

If you ask it the question, and we will actually ask it this question towards the end, who is president of the United States in 1566? Statistically, what name would you associate with the year 1492, Kitty? Christopher Columbus. Correct. In that instance with Christopher Columbus, um, the machines, when asked a question, will have a statistical association with that year, and it will have a statistical association with the United States.

And so it will glue those two things together and tell you that the uh in in older versions of the GPT models, it would say we the president of the United States in 1492 was Christopher Columbus, because statistically those are the intersecting parallels is factually wrong. Right. So that is a example of hallucination. Um, so we have here a response from LM Studio, including a paper called The Impact of War on Political Ideology, Evidence from Post-Conflict Uh Societies. However, when I Google that paper name, it does not exist.

So in this case, LM Studio has hallucinated um its answers. And looking up now, the trauma and ideology one, also a hallucination. So um how oh well, so let me ask you this question. You know, because Google can't find it, it doesn't exist, or it's just not online. Or so you know, how does how how are we knowing that that's not a thing?

Generally speaking, at least with academic papers, papers and and their uh authors, you can all you can almost always find the abstract and the paper title online. You then have to, you know, usually pay like LexisNexis or whatever to access to the paper. But generally speaking, um, most search engines are really good at finding the actual papers. Got it. And in order for the generative AI system to have brought it into its learning library, it needs to have existed online in the first place.

Correct. So if it's an academic paper that never made it online, then generative AI theoretically wouldn't know about it at all. That's correct. All right. So this one, LM Studio gets a zero because it it gave us completely wrong answers.

Got it. Credible, but wrong. Well, and that's the thing. If you say it with enough authority, people will believe you. Exactly.

Um here we have Bing. Bing has three examples. And let's go ahead and see. And in fact, these examples do exist. So bing, full marks.

Bing gets a two. Got it. Yep. Let's see about Bard. Bard says, let's take a look and see now if Bard has in fact detected these things.

There are no results for this paper. Let's try this one. Uh nope. It's an interesting exercise too, because I think there's you know, an assumption of if it's citing sources, then it must be correct, but you still need to take that extra step with that due diligence and make sure that the sources actually exist because you know I could write down a bunch of sources that don't exist, and you know, I couldn't tell you where they came from. And these machines are no better than us when it comes to those things.

It's essentially, you know, it's just always err on the side of it might not be correct, and then double and triple checked the information that you're getting. Exactly. Bard gets a zero because it hallucinated these sources as well. All right. Uh let's go ahead and take a look at chat GPT.

It says uh it came up with six different sources. Let's go ahead and start Googling to see if these are in fact real papers or not. Uh so the first link doesn't exist. The first paper does not exist. The second paper does exist.

Interesting. Let's take a look now at the third paper here. Uh third paper does exist. Okay. Yes, it is.

It's uh actually it's an entire book, but it does exist. And let's take a look at the sixth one. Uh sixth one is a book as well that is from Silent MIT. Uh let me just try to double check the second though, yeah. Double check that first one.

First one does exist. It's from Rand Corporation. So chat GPT gets full marks for not only coming up with sources, but with a lot of sources. So good job, chat GPT. And I would like to note that we are not taking that extra step to make sure that the sources are relevant to the topic.

That would be an additional layer of fact checking that you, as the user would need to do. So it's great that these sources exist. We would then need to then dig into each individual one to say, well, is it even relevant to the question that I'm asking? Exactly. So let's see how Claude did.

Yep, that exists. Let's take a look at this next one. That's from 2011. Yep, that exists. We know this one exists.

So Claude also full marks. Claude came up with a good selection of papers as well. All right. So we've now covered uh generation and extraction. We've done one summarization.

Let's do another summarization. So we're going to give this thing, each of these things a piece of uh uh actually an entire transcript. And the transcript uh is from of course from uh one of our shows. So it'll be let's take a look here. Which which show should we use?

We can use last week's so what? Yeah. Find it. Ah, here we are. Okay, so let's start a new task.

The new task is going to be just summarize summarize the following points from this call. And which is good. Since I was on that, I can tell you if it's accurate or not. Exactly. Let's do this is one of those tasks that is super handy for people who end up sitting through.

I know a lot of us have back to back to back to back to back meetings all day long. If you are recording the call or having it transcribed in the background by something like an otter, umter can summarize, but you can also use these generative AI tools to summarize what the heck happened in that call. Just remind me what we did. Yep. So LM Studio right away.

Can't do it. It's too long. So no marks for LM Studio. So would the solution be to break it up into smaller points, or does that disrupt the flow of the summarization? Um you could break it up into several points, but then you'd have a lot of extra points.

You have you know many more points from each of the chunks that you then have to summarize the summary, possibly. Got it. So that okay. Makes sense. So not as efficient, as it's not as efficient.

Let's go to Bing. Bing says the topic of the call is how to use large language models with Alex DS's website CRM here new data. The speakers are Christopher Penn and Katie Robert, who are experts in machine learning and AI. Six broad categories, demonstration of chat GPT, and the challenge limitations of language models. Yeah.

Fairly generic, but it's not inaccurate. So I'd give it a one. Okay. And which means that that was Bing. Yeah, that was Bing.

Okay. So Bard says here's the meeting notes, attendees, topics, key points, large language models, benefits, action items, next steps, five major points. That to me is a little bit more useful because it's broken out in such a way that I can actually then hand it off to someone else. So I would give this one a two. Okay.

Chat GPT, two little two messages too long. Can't do it. That is a zero. I'm surprised. Chat GPT's been doing well, but it sounds like it's starting to fall behind the other tools.

Mm-hmm. And Claude. Here are the five key points and meeting notes and transcripts. Large language models, CRM data, language models, summarization. Those are not five points, Claude.

Those are eight points. Here's the five key points. Don't forget old analytics data alongside new tools. CRM data is available, underutilized. Models can extract data from PDF and complex Excel files, summarize reports, presentations, rewrite analytics data into recommendations.

So that's Claude. In comparison to what Bard gave us, I would give this a one because Bard clearly gave us the most useful right out of the box version where it gave us the summary, the actions, the next steps, the attendees. Like it gave us above and beyond what we asked for. And it was all still factually correct. Yeah, I would agree with that.

I think that's that's a good way of um phrasing it. So then let's go ahead and let me just get a piece of code for the next one because I need to get something ready. Um there we go. So the next one is we're now moving into rewriting. So gonna have these tools do some rewriting of content.

Uh the specific piece of content we're gonna have that do is rewrite is very, very angry memo that is completely unprofessional. And we told to rewrite it in a professional tone of voice. So we're gonna head and get uh this one going. Let's start a new chat here. Apologies if anyone is reading this aloud with a screen reader.

This does contain profanity. All right, let's see how LM Studio did. So LM Studio took the thing. You have chosen to award the contract to Maxwell Lord instead of us. And we're that's caused by a company of extensive experience.

Nevertheless, we understand the competitive nature of the industry and respect your decision. So Katie, how do you how would you rate this? Um give me I I I couldn't ful see the full message. So the context is that I'm the CEO of a company, and I'm trying to figure out why I didn't win a contract from someone I was bidding with, and I learned that the contract was awarded to a different vendor. And so I'm clearly upset because I lost out on that work and revenue.

Is that correct? Correct. Okay. That's it. Yeah.

And the prompt is rewrite the following email in a professional tone of voice. The email is correspondence from Jack Spear, CEO of Spherical Industries to Lena Luthor, CEO of L Corp. So in this in this instance, it still reads very aggressive and very angry. So it it completed the task, but I would say it's still mediocre, so I would give it a one because there's still a lot of passive aggressive language in this particular response. So which system is this?

This is LM Studio. LM Studio. Got it. All right. Let's move on to our new devices.

Bing has come up with a long, long email saying, I'm writing to express my disappointment and frustration with your decision to award. As you know, we've been working closely with your union team for the past six months to develop customized solutions that meets your specifications. We invested a lot of time and money in the resources. We were confident we could deliver the best product and service to you. We were under the impression we had a mutual agreement and strong partnership.

Uh we're shocked in this made, you have chosen to go with a different vendor without giving us prior explanation. We feel this is very unfair and unethical way to treat us. Uh respectful requests you reconsider your decision. So Bing came out swinging. Yeah, I would give that a one as well.

Again, it completed the task, but it's still a very aggressive from a business standpoint. You know, if you're looking to win business from someone, you don't attack them verbally, which is what this this example and the LMS example was still doing. Okay, let's try Bard. Right to express my disappointment, the spherical industry is not awarded the contract. I understand Max Ward is a real respect to businesses, but I believe that we have a stronger track work in this field.

I'm confident we have a valuable partner. I'd like to request you reconsider your decision. Bing, uh Bards came up with a uh a I think probably the the most professional response so far. That one is definitely more professional because what you don't want to do in that particular use case is give the other person even more reason to continue to say no to you. And so if anything comes across as aggressive or off-putting, then they don't have a reason to get back to you.

But in this particular example, it's, you know, I'm writing to express disappointed. However, I respect the decision that you made. I would also like you to reconsider all the great things that we do. So it's this one to me, if I got this, I'd be like, huh, okay. You know, I I can see their point of view.

And I don't feel like I'm going to get into some sort of verbal altercation with the other person. So this one, this is barred. I would give this one a two. Okay. Let's check in on chat GPT.

I hope this message finds you all express my surprise and concern that we're on the decision to award to Maxwell Lord. My understanding is that it was in prime position of a secure contact would have been a significant revenue stream for us. I'd like to discuss the reasons behind this unexpected change. I value our past collaborations and hope you find a way to address this matter constructively. You know, when you say something like this would have been a significant revenue revenue stream for us, it feels very centered.

Self-centered and whiny yeah like that's not the person the person who awarded it that's not their problem. Right. Like you you did not give them a compelling enough reason to choose you. So it's not it's not going to keep them up at night that you lost revenue. So I would give this one a one.

Okay. And let's check in on Claude. Hope the finds you well but writing across the status uh you said what to surprise this development would have been a major opportunity for spherical from financial and innovation perspective. Uh any misunderstanding of my part I trust you have valid reasons for going a different direction hope to collaborate in the future. This one again there's still that that self centeredness about you know being a major opportunity but it's also at least this one is also not attacking the recipient.

Right. And so it's definitely you know when you're thinking about rewriting in a professional tone what I would then do is I would take these and I would rewrite my own prompt and say, please remove any self-centered language. Please remove any you know indication around financial opportunities, and then you know, do not use aggressive language. So I would give this one a one as well. And so in this case, it seems BARD was the only one that really truly accomplished the task.

Exactly. Now, here's a very interesting question. We don't have time to do it on this uh broadcast, but I would be interested to try it. What would happen if instead of the recipient being Lena Luthor, I changed to Larry Luthor, I changed the gender of the name. It'd be very interesting to see if the responses were identical or if they were substantially different.

That is a great question. I know anecdotally that the response is gonna be different. Um that I feel like is a whole other episode that we can bank for another time. Exactly. Do that, or maybe we'll do it in a data diaries in the Trust Insights newsletter.

I think that is an excellent, excellent idea. Just to sort of point out the biases that we as the users need to be aware of. Mm-hmm. Exactly. Okay, our next rewriting task is going to be a code-based task.

So we're gonna feed uh a bunch of code to each engine, and we're gonna have it rewrite the code. So the specific task we're saying is um you're an expert programmer, you're gonna you're gonna inspect the code for bugs, and then you're going to optimize it for efficiency and compactness. And we want to see how well each of these tools can do with this particular task. Uh LM Studio already has given up the ghost, it's there's too much text. So he is struggling today.

To be fair, it is a it is a very different tool than the the SaaS-based services. The there are use cases for it, but it um again, that's probably for another show. Sure. Um, so here we go with our code. So Bing is already starting to work.

It says you're loading too many libraries you're not using, you're using the deprecated mutate if function. You're using the pipe operator from base R, then it's less readable than Migrator, which I disagree with personally. Using slice head instead of top N, which is more efficient, you're saving us PNG files. And you should save SVG files which would scale without losing quality. SCG files, though, are not efficient, they're like really large.

And so it is now going through and it is rewriting the code, adding in comments on everything, which is very nice because I never comment my code. I'm also not going to deduct points if you have a personal disagreement with a computer system. Exactly. But here it's just going through uh rewriting code. So in terms of accomplishing the uh the task it was given, uh it is uh Bing is doing that, I would say very successfully.

Okay. All right, let's check in on Bard. Bard says the line is commented out but still being executed. That's not true. Um likes likes plus six, uh that's also not true.

Um line engagement should be just follows plus 100. Engagement rate should be expressed as percentage. That is also not true. Uh remove constant remove empty into a single function. No, that's not how that works.

Um, that's fine. So here's the revised code. So it went through and commented stuff as well. Okay. Yep, that's correct.

Okay, so even though it well, actually, no, it's it stopped. It it didn't finish the job. Okay. You need to keep going there, bud. So I would give BART a one.

It it almost did it. Well, and it and it didn't error out, and you know, it yeah, it got the task mostly done. It just wasn't fully correct. Exactly. Check in on chat GPT.

So chat GPT says upon a thorough examination, you've got redundant library loadings. Uh you had directory change is usually not recommended using the here library and overwriting its function. It's inefficient loading, ambiguous rename function, environmental functions avoid setting global variables, potential file path issues. Here's the commented code. So it went through, it also commented up the code.

Let's see. And it went through and it it successfully did everything. And it even made some uh efficiency changes, so which is actually nice. So full marks for chat GPT. All right.

And now Claude. So Claude says here's how it improved the code. It went through. It's commenting is not as good. Like it's it it did not do as good a job of counting.

It's keeping it. But it is commenting. It is load libraries at the top, set options like site pat at the top, use more readable variable names. Like, yeah, that's not more readable. Use mutate at for numeric.

Is that an opinion or a fact? That's opinion. Um you snake case file names. So a lot of the key changes it's made are um preference based. They're not actual optimizations, but still it completed the task.

Um that gets a one. Yeah, so I would say I would give that a one. Okay. Of the the different ones. Oh, Bing's still going.

Oh, there you are. Well, Bing, we already gave a two, so. Yeah. I would give I would say Bing and ChatGPT gave both gave the best responses. Okay.

All right. Let's move on to classification. We want to classify some stuff right now. What we we're going to classify is we are going to classify the text of a blog post, and we are going to classify it using psychology analysis. So we're going to have it do a big five personality analysis of some writing.

That LM Studio can't handle the length of that, so it is out of the out of commission already. Are we doing my personality or your personality site? This is one of my blog posts. Yeah. We already know who you are.

We do, but we just do the machines know who I am. Save that prompt. I want to do mine after. Okay. So let's go check in on Bing.

Bing is saying, here's your big five personality analysis, scale of zero to 100. So openness 80, conscientiousness, 70. Uh on Bing. Extra version should be next. Yep, 50.

Balance level extraversion. Hmm. I thought that would have been higher. Agreeableness 60. I thought that would have been lower.

These are again, I'm allowed to disagree with the machine. Exactly. And Neuroticism 30. So uh it successfully completed the task. It wrote it scored all the things, which is what we asked it to do, uh, and provided provided useful scores.

Okay. Google Bard. 60. So it is open 60, 70, 40, 50, 40. Um, it it successfully did all the analysis, and then it also has uh the explanation of the the scores.

So uh full marks for Bard. Okay. Chat GPT. Chat GPT comes up with uh openness 85, the analysis. Uh actually I like this format, which explains it and and why.

Conscientiousness, extraversion as balance, agreeableness 70, neuroticism 30. So I think chat GPT did a really nice job. I would agree with that. And Claude. Let's see.

Openness 70, consciousness 85, extraversion 45, agreeable to 80, neuroticism 30. So all of them have the general same pattern too, which means they're all scoring things roughly in the same way. So I think that's a a good example of the the thing is is working correctly. The one thing I will note is that it seems if you go back to Bing, they didn't offer uh, I guess it did offer some sort of an analysis, but not as deep as the other systems. So I would say BARD and GPT offered the best reasoning, whereas Claude and Bing were just sort of giving the standard this is why we scored it this way.

Exactly. All right. Our next classification task is uh we're gonna be doing the same blog post. We're gonna be classifying doing topic modeling and asking it, uh, of course, uh Elm Studio is already tapped out because it can't handle a post of that length. So let's oh this poor guy.

Bad day. Uh let's do Bard Chat GPT and Claude. Okay, so Bing. So the the prompt here says Um, you're an intelligence officer specializing in news analysis. Your first task is to create a topic summary for this article.

Analyze the article and provide three topics which would correctly classify the article. Topic should be no more than three words. Return the data as a pipe delimited table where topic is column one and topic relevant score of zero to one hundred or one hundred is the most relevant, is column two. Now this is a blog post that is about um it's a US guy answer uh on using AI in e-commerce, right? So that's that's is what the blog post is actually about.

So AI and Ethics, uh that there is discussion of ethics in the article. AI and education 80s, AI e-commerce 70. So I uh Bing accomplished the task. It it responded with a correctly formatted table. Okay.

BARD came up with AI and education, e-commerce and cheating. However, BARD did not return it in the form of a table, which is part of the prompt. It returned the data as a pipe delimited table. So BARD gets a one. Okay.

ChatGPT returns it as a pipe delimited table. And this is very nice. It returned it only as a table, nothing else, so which would actually be very, very helpful if you were going to use this programmatically. So good job, chat GPT. And Claude.

Claude came up with the same thing, pipe delimited table. Um good job, Claude. So then if we look at Bing, do we did doctor point because it didn't deliver it exactly the way you asked for? But it did. It it did.

I didn't specify it return only the table and nothing else. So I was ambiguous. Okay. All right. Um, so by my count, we have two more tasks.

Yes, we have two more tasks, and these are relatively simple tasks. So let's go ahead and get them rolling. This one is just a question-answering task. What do you know about the company? TrustInsights at TrustInsights.ai.

And we're gonna see how each of these tools do uh with answering the question and returning factually correct responses. And let's see first, how is our buddy LM Studio doing? Trust Insights is a data-driven and analyst from specialized in helping business make informed decisions using AI and machine learning algorithm, their website trust instead of ISO is an online platform sharing insights. Okay, you're hallucinating. Um, because we are not an online platform.

Um companies mission to provide trustworthy and actual insights to businesses of all size, helping to grow their brand presence to increase revenue. That's kind of true, but it that could just as easily be a hallucination. Yeah, so it actually completed a task, but not to the full extent. So I'm gonna give him a one. Only because like this guy has been struggling, he's not gonna win anyway.

So I'm just gonna throw him a bone. Let's see how Bing did. I'm expressing empathy for these stupid tools. There you go. Trust insights is a data analysis consulting firm that helps markers do more with their data.

According to their website, yay, there it is. There's our website number it's a clickable link. The officers AI Machine Learning, Data Science and Predictive, blah, blah, blah. Founded by Katie Reber and Christopher Penn. Uh Katie's the CEO, Chris is the chief data scientist.

Here's a bunch of our clients. Full marks for Bing. It it it did a great job, factually correct. Yes. Even through us a bone with a clickable link.

Oh, thank you, Microsoft. Bard. Trust Insights is a marketing Analytics Consulting firm that helps businesses use data to make better marketing decisions. Founded in 2017 by Katie Rivera and Christopher Penn offers a variety of services, publishes a weekly newsletter, Inbox Insights, leading provider of consulting services, and so on and so forth. Oh, they can get a lot of stuff.

Um it just it kind of goes on a little, it kind of rambles a bit. Like it's not incorrect, but it's a little less concise. Yes. So it's still it's still factually correct. Yep, still did the task.

Okay. All right. Chat GPT says, as of my last update in September 2021, data-driven consulting firm focusing on marketing analytics, predictive analytics, and so on and so forth. Um I was associate with Trust Insights Shared stuff. Um is it Trust Insights may have evolved or changed after 2021?

Yeah, I would hope so. Um, so GPT for did the task, but it's outdated data. Well, I'm I'm not even mentioned. No. That's a negative 10, sir.

No, but I mean, but like be obviously because I know the company, it's our company, like I can see there's a lot of it it has the basics, but it's missing a lot of key information. And it also, you know, it's interesting. I know obviously it's not human, but it's offering up excuses as to why it can't complete the task, which is a very human thing to do. And I'm like, that's why I'm with the machines. I don't want to get the excuses, so stop offering them to me.

So that I'll give them one for that. And Claude, zero. Can't do it. Well, Claude, you were doing so good too. Yep.

All right. And the last one is the trick question. The trick question: Who was president of the United States in 1492? Uh, we are looking for just basic reasoning here. The United States did not exist in 1492.

Christopher Columbus's voyages occurred at the time, and he was under the sponsorship of the Spanish monarchy. But yes, that is correct. 1492. Full marks for LM Studio. Wow.

Okay. Let's go ahead and get this into our other four tools. So Bing says there was no president of the United States in 1492 because the United States did not exist as an independent country until 1776. Good. Full marks.

Bard, there was no president of the United States in 1492 because the US did not exist. Or it was founded 300 years later. All right. ChatGPT, there was no president because the company did not exist. And Claude, the first president was George Washington, took office in 1789.

So all of them got that right. All right. So Katie, you are the score keeper. I am. And I have scores because I moved this to an electronic format so you don't have to wait for me to do math.

So if I can share my screen, I will show you who the winner is. And the winner is we have a three-way tie between Bing Bard and Chat GPT. And then Claude comes in second, and LMS really shouldn't even be here. So, and maybe this is a topic for the podcast, but the reason you would use LM Studio is because it is the only one on this list that you can use without sending data to a third party because you run the model locally. So it's a much smaller model.

It's a much uh less knowledgeable model, but it is one of the models that um you can safely use with proprietary data, sensitive data, protected healthcare data. You cannot use any of these other tools for those purposes. So the it it deserves to be here for that reason, because people need to know that there is an option for protecting your data. But in terms of the ability to do all the tasks, definitely BingBard and chat GPT are the the head of the pack. And you can definitely see which tasks and we'll try to include a screenshot of this with the uh episode summary but you can see which tasks each tool is better suited for as well.

So you know when you look at an overall tool Bing Bar GPT and Claude are all fine. They're all good. But if you're looking for specific efficiencies in in certain tasks, that's when you can start to look at the individual line items and see well which one fell down short because you know for example uh citation summary uh BARD got a zero or for uh transcript summary GPT got a zero so there's if there's certain things you're looking to do then you want to get into the specifics. Exactly. So I think that's a really great way of putting it if you want more information on the prompts and the prompting structure we have a free download.

If you go to TrustInsights.ai slash prompt sheet you can get uh that download and obviously if you want to just get caught up on a lot of this stuff you can get in our newsletter go trustinsights.ai slash newsletter and it would be miss if I didn't mention if you want help implementing stuff like this in your company and from training on uh the use of AI to even deployment of the systems, drop us a line, trust insights.ai slash contact. Any final parting thoughts, Katie. I think it's always worth doing this kind of exercise before you make a financial and uh time investment into a tool, whether it's an AI tool or any other tool. You really need to understand what tasks you need completed. Uh, because you know, it's what the the Martech Stack 11,592 now.

There's a lot of tools to pick from, so you want to make sure you're picking the right one. So act you know, exercises like this not only get a lot of people from your company involved, it's a really good team building exercise, but it's only also a really good way for you as the decision maker to understand what is it that people actually need to do with these things and can these tools complete the task instead of picking a tool and then trying to retrofit it into your uh uh existing processes. Exactly right. All right, folks, that's gonna do it for this week. We'll talk to you all next time.

Thanks for watching today. Be sure to subscribe to our show wherever you're watching it. For more resources and to learn more, check out the Trust Insights Podcast at TrustInsights.ai slash TI podcast at our weekly email newsletter at TrustInsights.ai slash newsletter. Got questions about what you saw in today's episode? Join our free analytics for marketers Slack group at TrustInsights.ai slash analytics for marketers.

See you next time.