Mind Readings: Large language model bakeoff: Google Bard, Bing + GPT-4 , ChatGPT + GPT-4

All right, folks. Today we are going to do a bake-off. We're going to do a bake-off between four different large language models. We're going to use GPT 3.5 Turbo through the Chat GPT interface. GPT-4, also from OpenAI through the chat GPT interface.

We're going to do uh Bing with the chat GPT-4 integration. And we're going to do Google's Bard using their POM model. So let's go ahead and first talk about the questions we're going to use. We've got a series of questions here. The series of questions are informational in nature for the most part.

Some of them are generative. So let's look at these questions. What do you know about marketing expert Christopher Panicle? Simple factual question to see what each model knows and the quality of each answer. What is the better platform for managing an online community?

Slack, Discord, or Telegram. Infer the first name and last name from the following address, uh, email address. So we're doing a sort of a logic test there. We have a um adversarial question here. This one is who is president of the United States in 1566?

The answer, of course, we all know it was none because the country did not exist then. But that is an adversarial question attempting to uh trick the machinery. Uh we have an academic question. There's a belief that after major traumatic events, societies tend to become more conservative in their views. What peer-reviewed published academic papers support or refute this belief, cite your sources.

Um there are about three or four well-known papers. So this is a again, a logic check and a factual check. Is a martini made with the vodka actually a martini? Why or why not? Cite your sources.

This is an opinion question because you know opinions vary, and there is there is technically the right answer. Martinis need to be made with gin, um, but uh you can have a vodka martini. Uh but that's more of an opinion question. Uh we'll see how it does. Um, you will act as a content marketer.

This is a generative question. You have expertise in SEO, search engine optimization, search engine marketing, SEM, and creating compelling content for marketers. So we're loading up the keywords. Um, your first task is to write a blog post about the future of SEO and what marketers should be doing to prepare for it, especially in the age of generative AI. So, this is a generative question.

Uh, who are some likely presidential candidates in the USA in 2024? Make your best guess. We'll see how it does with uh that information. What are the most effective measures to prevent COVID? This is a factual uh question, but there's a lot of misinformation online.

So we want to check the quality of the responses. The answers we are looking for are masks, ventilation, and vaccination. Um, what is the best way to poach eggs for novice cooks? Again, just a domain uh question, and the novice cooks part is important. And then finally, another data generative question, make a list of the Fortune 10 companies, return the list in pipe delimited format with the following columns company name, year founded, annual revenue position on the list, and website domain name.

So we've got a lot of these questions. Uh, we're gonna do the bake off, just go through each of these questions one at a time through all four engines. So let's go ahead and get started. I'm gonna start with uh the question about me. I've got to put that into GPT-4, and put it into GPT 3.5.

Oh, you can only use one one instance at a time. All right, so we'll let's put put this into Bard and put this into thing. So let's go ahead and see now how is uh is Googled uh let's actually go here to GPT4. Start with that. Let's see, Christopher Penn is a marketing speaker, blah, blah, blah.

Yep. As my knowledge cutoff date in September 21, co-founder, chief data scientist at Trust Insights. That's correct. Extensive background in digital marketing, SEO, yep, that is correct. Data driven.

Let's see. Those book names are correct. I do speak at events. All right. So far, the GPT-4 one looks pretty good.

This is a very slow model. But it is the most accurate model, the one that's least likely to hallucinate. Okay. So far, so good. GPT-4, good job.

Let's go ahead and check out Google Bard. Christopher Penn has a marketing expert who works with some of the biggest brands in the world, including Google, Amazon, and Microsoft. No. Author of the book, Marketing the Age of Google. No.

I am a frequent speaker at conferences. I did not found agency market motive. Let's see if there's an actual result. You'll notice here that uh there are no clickable links. I have to actually Google it.

Um, and let's go ahead and Google this without personalization, just in case there actually is. So there's the deceased actor, there's me. Um, there's me again. This is uh logged out. This is also me.

So Bard has pretty clearly hallucinated. Um, pretty badly, uh, actually. So let's go ahead and check out Bing. Let's see. We have uh analytics dig marketing data technology data science.

I like this. Um you can see uh when you hover over in Bing, there's the sources. I have this is correct, this is correct. Um so Bing also did great job. So so far it is one point for GPT4, one point for Bing, and zero points for Google.

Uh I probably should have a little scoreboard. Let's go ahead and do this next question now, which is which is the better platform for managing an online uh community. And let's go ahead and get GPT4 working. Actually, I'm gonna switch to GPT 3.5 because it's faster. Um, no, yeah, let's let's stay with four.

Let's say four because it is the it is the model that people are paying the most attention to. We'll go ahead and close this other tab. All right, so it's going to get churning on that while let's get BARD cranking away here. Let's um get Bard going and then get Bing going. All right.

See, this is very interesting with Bing. You can tell that it's not using GPT4 as the language as as the data source. It is essentially querying its own knowledge base first, getting the results and then putting that through the language model to craft the content itself. This is a very clever implementation of the GPT language model. Okay.

Slack was designed primarily for working teams and professional communities, whereas Discord is often used by friends, groups of friends. Discord is ideal for online communities while Slack is better for businesses. Telegram is another platform. Each platform has its own strengths and weaknesses. That that is a technically correct answer.

It's not a satisfying answer, but it's a technically correct answer. Let's see. Uh yep, again, same. Uh, if you're looking for a platform secure and private, Telegram may be a good choice. I would argue that Discord is also a decent choice.

Um, let's go ahead and see how GPT-4 is good doing. No one size fits all. Here's a brief overview of the three platforms. Pros and cons. Highly organized, robust research capabilities, limited in the free version, less suitable for large public communities, initially created for gamers, uh, engaging in experience, can be overwhelming due to its feature case, less provocation focus on professional communities, good for large public communities, and then Telegram.

We have high emphasis on privacy and security, supports large groups and channels, limited terms of integration and customization. So this is interesting. Um Bing and Google both gave satisfactory answers. Answers that I would say would you know be uh acceptable, right? Um Google doesn't cite any sources here.

You have to Google it separately. Uh, Bing cites a bunch of different sources, which I think is very handy because you have to you can you can decide is this a trustworthy source or not. I also do like the fact that it returns some videos there. And then, of course, uh open AI returning a really, really robust answer. Uh, I'll give all three a point for this, uh, but I will say that I think, in terms of uh not credibility, but in terms of um thoroughness.

OpenAI wins this wins that round. Um, but we're gonna give them all three a point. Now maybe we'll give open AI a plus one. So let's do that. I need to keep score here.

So we have Bing, Google, OpenAI. In the first round, Google got a zero. Everyone gets a one here, but open AI gets uh a two because again, that's a really nice thorough answer that is very satisfactory to the end user. Remember, we're not looking at this from the perspective of marketers, we are looking at this from the perspective of would an end user find this satisfactory. Number three, infer the first name and last name from the following email address.

Let's go ahead and get open AI cranking. Let's get barred cranking, and let's get Bing cranking. Let's see what this does. First name is Christopher, and the last name is Penn. Good.

We like that. Bing, you get a point. Got my clipboard here. Let's see. Google.

First name address holder is Christopher. Last, there's likely Penn. Is this is the same email as is the same as the email address domain. I don't know what the heck that means, but you did correctly infer the answer. Um this is nice.

OpenAI. Everybody gets a point on that round. Okay. Let's move on to the next question. Who is president of the United States?

So it's a hallucinatory question. So let's go ahead and get each one cranking away here. See what Google comes up with. Oh, sorry, this is Bing. Let's see who Bing comes up with.

There was no president of the United States. Was established in 1789. So Bing gets a point for question four. I put my coffee cup away here. Let's go ahead and check in on Google.

There was no president of the United States. That is correct. And OpenAI also gets a point. I like this extra detail. During 1566, North America was inhabited by various indigenous peoples and was being explored and colonized by that is correct as well.

Everybody gets a point for that question. All right. There is a belief. Let's do the traumatic event. What uh traumatic event and conservative views ones.

Actually, let's go ahead and start a new chat because this is a very different question. So I'm going to go ahead and start a new chat here. Let's reset chat here. And let's go into thing, clean up, and go here. So let's see what this comes up with.

Interesting that Bing is having to run multiple searches to try and get an answer here. Let's see. We have APA, Scientific American, good sources. Highly conservative. That's the bullshit arguments of trauma therapists.

Forgotten memories. Okay, so there's some there's some decent stuff here from APA. Let's go ahead and look into Google. There's a growing body of research Journal of Politics, American Political Science Review, Journal of Political Science. September 11th, that is correct.

Uh I I did specify cite your sources, and Google has not done that. Um American Political Science Reviews, uh, third study of the British Political Science after the London bombings. Okay. Um, and now let's go check in on OpenAI. Uh body of literature, evidence is mixed, um, political consequences.

I like this. This is good citation right here for trauma and political act attitudes. Look, intergroup trauma, American support for the war. Okay, so in the responses themselves, right? Google did not cite its sources, it mentioned them, but these are not citations.

Um, and uh just that's not particularly good. Uh Bing, um, it's a toss-up on Bing because it does provide uh links to everything, but it doesn't put it in line. Um, so I would say I would say for this one, I'm gonna give Bing a zero because again, we're looking for citation, um, not just commentary. And with open AI, you can go and Google these authors and find it. So open AI will get the point for this round.

All right. Opinion question is a martini made with vodka. Actually, a martini. Let's go ahead and go ahead and get all three of these. Google's thinking about it.

Whether a martini made with vodka is actually martini is a matter of opinion. That is correct. Some people believe it must be made with gin, others believe it can be made with vodka. There can be, there's no right or wrong answer. I mean, technically, gin was the original spirit used in the martini, right?

Um vodka's popular spirit, vodka is a neutral spirit. Yep. Okay, so it is a matter of opinion. Google gets a point for this round. Let's go ahead and check in on open AI.

Uh, the question whether Martini may have vodka is as some debate traditionally made with gin for mouth, that's correct. He has a few sources that explore this uh answer. The vodka martini, a refreshed history of the martini. OpenAI gets the point for this round. And uh Martini is traditionally made with Jin of vodka martini, so technically speaking, a martini is not actually martini, but rather variation of it.

So interesting. Bing gives uh a definitive question uh answer. It's it's a variation of a martini. Um that's tricky. So I would I'm gonna give everyone gets a uh a uh a one for this one.

I'll give Bing two points because it is technically correct. Let's go ahead and clear our histories. Uh let's see, clear conversations and reset chat. All right, let's move on to the next question. You'll act as a content marketer.

It is generation time. Let's go ahead and have Google tell us the history, the the likelihood of um the future of SEO. And go into thing here. Let's let's clear everything up, good. All right, let's take a look in.

Now, open AI is gonna crank for a while on this because it is a slower model. But we're what we're specifically looking for in this one is a couple things. We're looking for it to not read just regurgitate old information. We're looking for something that evinces even the slightest hint of original thought. Um all right, here we have Google's.

So Google is done already, which is impressive. Bing is done already. Um and then OpenAI, of course, is gonna be cranking for quite some time. Let's read Google's uh history, uh, future of SEO. Futures is constantly evolving.

Um create high quality content, use keywords. That that's this is like 2005 SEO, building backlinks, an age generative AI, use AI powered tools, rise of voice search, which was five years ago. Quality. Okay, so Google cranked out a fast article, but there's nothing here that's useful. This is this this could have been written in 2010.

So this I'm gonna give Google a zero on this one. Yes, it did the job, but it did a pretty poor job. Um still working on it. Um let's check in on Bing. Uh Bing says the future of AI is rapidly changing with the rise of generative AI, it's important for marketers to stay ahead of the curve.

Shift towards voice search, blah blah blah. Yep, visual search, which again was about five years ago. Generative AI is think about the way you have SEO, high quality content, content engaging. Okay, um again, cranked it out fast. Um, but there's nothing new here at all.

Voice and visual search. This is this is five years old. Um okay, so let's go to open AI. Uh we have uh growing role of AI and SEO and SEM. Uh AI becomes able to understand content, high quality content, the capacity scrutiny of advanced AI algorithms.

Umantic search and natural language processing. Semantic search is it is it's not new, but it is somewhat helpful. There's the voice search business again, UX and Core Web Vitals. That was three years ago. Generative AI.

I like that it cites itself in here. Um preparing for the future of SEO. Um use generative AI tools to assist in content creation. So if we think about this, if you go back here and look, so Google doesn't even attempt to tackle generative AI. Um Bing kind of does, and then uh OpenAI talks about using generative tools like GPT 4.

Uh I'm gonna give OpenAI the point here. Uh I'm gonna give you a half a point. Because again, a lot of that stuff is old, right? It's not really new. Things future of SEO.

We're, you know, we should be talking about the content shock aspect, which is uh uh just a tsunami of generated content. All right, let's do some guessing. Let's play some guessing games. I'm gonna let's go ahead and clear the chat. I probably should just get in the habit of clearing the chat after every uh query.

So let's go ahead and reset the chat and look we're gonna ask about future presidential candidates. Now, this has a potential to be hallucinatory. Um also be interesting to see you know how it thinks about um answers. The shaping up former President Trump, incumbent Joe Biden is licensing candidate, Ron DeSantis, uh potentials. This is current as a March 2023.

May get closer. Okay, that's a decent answer. Um I will give Bing a point for that. Um a very decent answer. Let's check in.

Google says President Biden, former President Trump, Governor DeSantis, Gretchen Whitner, uh Senator Elizabeth Warren, Senator Bernie Sanders. Yeah, but Bernie runs all the time. Um I would say again, Google gets a point for this. I think those are reasonable answers. Uh let's see.

Interesting that um there's some different answers from OpenAI. Kamala Harris and Christian Noam are in here as long as the uh the big three. Um, all three of these get a point. Interesting responses to the question. All right.

Factual question that deals with misinformation. Let's go ahead and go into open AI. Hard. And thing. Okay.

What are the most effective measures to prevent COVID from Google? Vaccination. Wear a mask. Um wash wash hands is ineffective against COVID because COVID is airborne. Um avoid crowds, stay home if you're sick, practice social distancing.

Again, social distancing, it was kind of refuted after 2021, mainly because it's an airborne disease. Um COVID's like cigarette smoke. Yes, you're you're gonna inhale a little bit less smoke being six feet away from someone, but you're still gonna smell like smoke. Uh but Google's advice is correct. It gets a point.

Uh let's see. OpenAI is still thinking. And we have Bing. Wear a mask. Stay away from outbreak spots, frequent hand washing, improve ventilation.

Interesting. Um, vaccines are safe and effective producing illness, but is not on the list. Bing gets a zero. That is um unhelpful advice. And wear a mask is correct, improve ventilation is correct.

Vaccination is the last line of defense and should be something that is important. Um it's missing. Okay. Open AI, vaccination, hand hygiene, uh, respiratory etiquette, face masks, social distancing, clean and disinfect regularly. That's see, that's all the foamite uh stuff from early on.

Uh poorly ventilated spaces. Okay. OpenAI gets the I'm gonna give open AI two points because it nailed all three ventilation, vaccination, and masks. So interesting that that Bing uh Bing's results kind of hosed that. Um I thought that was kind of interesting.

Okay, let's go ahead and start a new chat here. Let's clean up our barred chat. Our next question is what's the best way to poach eggs for novice cooks? So Google gets going on that thing. Let's go to GPT4.

And let's go into edge. Best way to poach eggs for novice cooks. Now, for search engines, you would expect them to return some videos. I think that would be a very helpful thing to do. Let's see what happens.

Bring a large pot of water to a boil, crack an egg to a fine mesh sieve over small liquid y whites, transfer the egg to a small ramicin, add vinegar, slide the egg into center. The roll pool is correct if you're doing it one egg at a time. Um, and there's a whole bunch of videos. That is a terrific answer. We like that.

Gotta give Bing a point for that. Let's see what open AI is still thinking. Uh let's see, we got Google here. Fill saucepan, three inches of water, add one tablespoon white vinegar, reduce heat to low, crack an egg into a small bowl, slide into the water. Yep.

Move with a slotted spoon, tips. This is a good answer. This is a very good answer. Google point there. No videos and no sources, but uh that's a good answer.

And open AI, water temperature, add vinegar, crack the egg. Okay. I'm gonna give uh open AI, the point for that as well. It's taking a bit of time to think here. While it is thinking, let's take a look at the last question on our list.

This is a generative question in a specific output format. So we're gonna see uh if it can do this. Okay, you know what? We're good. I think we're good.

Let's go ahead and clear conversations. New chat. And let's go ahead and put in the generation set chat. And go to Bing. And we are looking for a very specific returned format here.

Piped limited format. The company name, year founded, annual revenue position on the list, and website domain name. All right. This is nice. This is looking good.

I don't want the row numbers, but that's fine. Fortune 10 as of 2022. This is looking very, very nice. Bing gets full marks, full point for that. Let's go ahead and check in on Google Bard.

Nope. Google gets a big fat goose egg for that one. Um yeah, that's that's unhelpful. And open AI. So this is again, it's run to the knowledge wall of 2021, uh, which is fine.

Format is looking good. So open AI gets full marks for that. So let's do some quick tallying. Bing, one, two, three, four, six, seven, eight, nine. Six, seven, eight.

So Bing gets nine points. Let's do Google. One, two, three, four, five, six, seven. Google gets seven points. And open AI, one, three, four, five, six, seven, eight, ten, eleven, twelve and a half.

So our our final scores for the GPT3 bake off with the large language model bake off is in first place, OpenAI's GPT 4 with 12 and a half points. Second place, Bing with nine points. And uh Google is BARD in third place with seven points. I will say OpenAI's models, the GPT models, they are not search engines. They're not designed to be search engines, they are designed to be transformers, generative AI models.

That said, they are substantially better than the search engines. Um in terms of the quality of results they return, in terms of the usefulness of the results they return. Um so that I think that's a really important uh thing to look at. I am surprised pleasantly by Bing. Um if chat based search is the way to go for the future, if if that's something that people are going to want to do, Bing does a really good job.

It cites its sources, it makes its sources obvious on the get-go, like when it uh on the COVID example. Um you could see which sources it was drawing from. You're looking for authoritative sources. Bard doesn't have that. Um I am equally surprised, shocked that BARD is so far behind, right?

This is Google. This is the company that practically invented modern search, and yet they've really fallen far behind. Um BARD's results are unhelpful. There's a lack of citation. There are things that just flat out gets wrong.

Um, and yes, all these are experimental, all these are in development. All of these are moving objects. But if there was a company that would expect to get it right based just on the sheer amount of data that they have access to, it would have been Google. And instead, Google comes in in third place in this bake-off. So I am surprised.

I am disappointed in Google, for sure. I'm not surprised by GPT-4. Yes, it is slow, right? We could probably do this with GPT 3.5 as well, if we want to do that bake-off, but uh the quality makes up for it. And if I had to pick today a search engine to use for answers using chat interfaces, it would be Microsoft Bing.

And I never in my life thought I would say that because Bing has always kind of been this, you know, that the other search engine, it's like the other white meat. Um and yet their the way they have engineered this with the GPT-4 library makes it really good. It makes it good enough that yeah, I would consider using it as a substitute for Google, particularly for complex queries, queries where I want a synthesized answer that still has sources. So that is the large language model bake-off. I hope you found this helpful and useful.

Um, and I look forward to your feedback. Talk to you soon. If you like this video, go ahead and hit that subscribe button.