So What? Setting up an A/B test for ChatGPT

Well, hey everyone, happy Thursday. Welcome to So What, the Marketing Analytics and Insights live show. I'm joined by Chris and John. How's it going, guys? Hello.

Uh this week we are talking about setting up A-B tests for Chat GPT. Chris and I were talking about this earlier this week because, you know, you can't, you know, take a left turn without hearing someone talking about chat GPT. You can't go online on a social platform, you know, in the news, uh, without someone talking about chat GPT. And so we started talking about the effectiveness, the performance. How do you measure chat GPT?

So everyone, you know, everyone and their mother has signed up for some kind of account and are playing with it, using it, you know, pretending that it's gonna revolutionize their company, their marketing, and in some cases that's true. Um, but for most of us, we really need to understand how effective chat GPT is, what it can do for our marketing, but how more importantly, how do we measure it? And so we wanted to do today is set up some A B testing. Um so we'll go into what all of it is, how to set it up, but what we really want to understand is are what we doing as marketers, is it better or worse than what chat GPT can do for us? And we're gonna start with some very simple examples.

Uh so before we get into it, John, are you do you have you been playing with chat GPT? Or is it something that you've you know thought, you know what? I can have this do the podcast for me. I don't even need to show up anymore. No, I haven't I have I at least you know dug into it once, but I have not spent a ton of time at all.

And so I'm looking forward today to kind of figure out what's going on. And I am I really want to issue a chat GPT NFT later this week. So I'm hoping that'll be a big moneymaker for me. That's my You're fired. That is, I mean, if you're gonna uh talk about it on Clubhouse, then we're good.

Even better. Check out my Google Plus page. We'll be here to go. Um all right, Chris. So, you know, we've talked a lot about what chat GPT and what chat GPT isn't.

Can you give a very quick rundown on the basics of you know, some use cases of for chat GPT and why you know setting up an A B test is a good idea? Sure. So very briefly, ChatGPT is a web interface to a large language model. Um the large a large language models is a piece of software that has been trained on enormous amounts of text. And what it does is it generates or rewrites text um based on uh instructions you give it called prompts.

So that's you know, the the the dimes to work explanation of it. This is what it looks like. We're using the uh plus edition, which is the paid edition. Uh yesterday, OpenAI just announced that um the API is now available for people who want to do development with it. And it's a pretty straightforward uh piece of software.

So you give it some instructions and it will do some things. Let's do analyst. I'll post it. And so Chris, what you're demonstrating is how we've seen the majority of people using Chat GPT, which is to create net new content. Chat GPT can obviously do a lot of other things, but I would say this is probably statistically the most common use case that people are using uh Chat GPT for is to generate something new to generate an outline, to generate, you know, a first draft.

And that's how you're demonstrating the use of it right now. Exactly right. So what we wanted to talk about today was for the process of rewriting content or um re remixing content, how to judge whether or not chat GPT does a better job than you know the content you've already got. So the first place we should probably um start is with the five Ps. Um, what is the the thing we're trying to do?

So the purpose for a lot of marketers of content creation is is attracting organic search traffic or organic social traffic, but uh mostly organic search traffic. So because that's the case, then the first place we should probably look is within an organic search tool. And the one that uh everyone has access to for free is Google Search Console. The place to look in here is under search results, go to pages, and you will see um the number of pages on your website, the clicks they get and the impressions they get. And what I'm looking for is I'm looking for pages that get decent impressions, which means that Google thinks that the page is relevant, but just doesn't get the clicks.

It means that the when a human being reads the search listing, they're like, eh, no, I'm that's not what I'm and if you, the viewer is interested in learning more about how to use Google Search Console for yourself to see if it's set up on your website. We do have a course. Go to TrustInsights.ai slash search search console. It is a free tool, you just need to make sure that it is connected to your website so that it can collect uh this kind of data. So, Chris, what you're showing uh is the Trust Insights website, and so these are the different pages, uh, primarily probably uh blog posts from our website, and you can see uh we have our top pages, we have the clicks and the impressions, and so it sounds like what you want to see is the pages that get both a lot of impressions and clicks.

Is that correct? Or are you going the reverse order? I'm uh well, I'm looking for pages that get good impressions because that means Google thinks the page is relevant, but doesn't get it, it doesn't get the click. So there's something wrong with that page that you know people look at the summary or the synopsis and go, eh, that's that's not what I was looking for. So here's one.

This is uh a blog post from four years ago now called AI and SEO in the post burden that marketers need to know. Now this is actually a fairly long page. Um let's take a good chunk of this here. And now let's take this and toss it over into chat GPT or create a conversation and say right. Following.

Okay. And so what you're seeing Chris doing is what we've talked about on other shows. It's called prompt engineering. And so Chris is writing it as specifically as possible. And you can check out that episode over on TrustInsights.ai slash YouTube.

The reason Chris is getting so specific is because he doesn't want to have to go through this process multiple times. There may be some refinement that needs to be done. But as the more specific you can get in giving Chat GPT instructions, the better the results are going to be. And so, you know, the example that we were giving earlier this week was Chris. If I said to you, bake me a cake.

I haven't told you what kind of cake, what shape cake, what flavor, if it needs to be gluten free, dairy free, any of that stuff. So you're gonna say, okay, let me bake a cake. And so you're gonna hand me a cake, and then I'm gonna get frustrated because it wasn't a gluten free, dairy-free, vegan, you know, only beet sugar, you know, chocolate with a unicorn on top. But had I told you those things, you would have done your best to deliver that. And so using prompts in chat GPT is very similar to you know, asking someone to do something.

Get as specific as you can, and then you're likely to get the result that you're after. Exactly. And I will never bake you that cake. I don't actually want that cake. That sounds awful.

It really does. It's like a just a block of sugar. Um, so what you see here is what you see here is the revised content. Now, what when we're talking about A-B testing, we want to do things as apples to apples as possible. Let me show you a couple of examples.

This is an original blog post of mine from 2016. Um, way out of date. And I had Chat GPT rewrite it. I dated it the exact same um, you know, date and from a promotional perspective. Um, I would promote each post.

The similarly, same for you know, this one change improvement, changes improvement. Again, following the exact process we just did, I've created these posts in parallel. And so what I'd be looking to do then is give each page appropriate amounts of promotion and see which page performs better. Does the does the text that's machine written perform better than the text that is purely human written or not? And that's that's what we want to ascertain.

And so, you know, is this something remind me, is Google sunsetting Google Optimize? Yes. Okay. Um, it's Google's well, Google Surveys. Wait, no, actually, they have yeah, they have they are sunsetting Google Optimize.

Yeah, optimized it's going away, yeah. Um so this is a way to do an A-B test without relying on a tool such as Google Optimize, which a lot of us have been using for years, but Google has decided that it's no longer going to be a standalone piece. I think what's actually happening, and we can double check on this, um, is they are rolling it into Google Analytics for so that you can you have to use it through that system. Um so it's no longer something you can just standalone. So the reason I bring that up is because I think when you know we as marketers think about A-B tests, we immediately think we have to be using a tool such as Google Optimize.

But Chris, what you're demonstrating is that you don't. You can do this without having uh a third-party tool running the test for you. Exactly right. And I mean, this is as simple as it gets. You just create additional pages and you promote each of the pages.

One of the things I would suggest that you do is as part of the experiment, you put together a very simple spreadsheet, right? You know, here's the URLs, and then do your promotion, and then after 30 days of promoting both pages, you know, super simple, uh compare the page views and say, well, we promoted both equally. Um, which page performed better? And you can look at there's a whole bunch of different metrics you can use to to judge this pages would certainly be the simplest. Well, and I think that that goes back to your original point, Chris, about the five Ps is understanding why you want to do this in the first place.

And so if your goal with the content on your site is to drive traffic through organic search, if you're it's to build awareness, those are the things that you'd looking to be you would be looking to measure in terms of the effectiveness of one version of this versus the other. Exactly um now that was the simple version one of the things that is challenging about search console is that you don't get the query terms alongside with the pages I'm not sure why they don't let you have at that but you can only get that through the API so if you're familiar with uh the API or or APIs in general you can uh this should look familiar if you're not one of the things you can get is programmatically get at your data to see what pages um and the associated search term um are are being uh impacted so here at AI and SEO post the the query term being searched for is BERT SEO so that is very helpful for um when you're writing your prompts to be able to say make sure to include um concepts about BERT and SEO so with the content that you just uh had chat GPT rewrite um you know it would you suggest that we take what was just rewritten by Chat GPT and stand it up against the original post so don't optimize the original post but stand it up against and have them running in parallel on our site to see which one performs better. For the purposes of an A B test, yes okay um and I would I mean I would imagine, and I think I already know the answer to this question, like you don't have to have existing content to do this A B test with. You could just take, okay, I'm looking at this and I could see, oh, Bing Webmaster Tools. That's a topic that I know enough about that I could write.

I could do in some ways, Chris. I think you like to call it a bake off, like a head-to-head, have ChatGPT write a piece of content about Bing Webmaster Tools, and then I'll write a piece of content about Bing Webmaster Tools. We'll put them both up, have the same promotion plan around them, and see which one does better. Um is that another way to approach this A B test? Absolutely.

And in fact, if you if you have it set up in your account, um, you have to opt into it. Um, Google will give you content ideas. It says these are things. This is content ideas are based on searches that might lack good results, which is Google's a nice way of saying, hey, we need you to do some free work for us. Um, you know, there's some really good, interesting questions in here, kind of the kinds of questions that some you might want to legitimately have on your website.

Um easily copy and paste right over to chat GPT. Exactly. So here's a good one. What can't you measure in Google Analytics? And so, John, so at this at the start of the show, you were talking about wanting to learn more.

Is this giving you some ideas around how you know, for the marketing over coffee podcast, you could utilize chat GPT or even within the business development stuff uh that you do for trust insights? Now you're always sort of mentioning A B testing subject lines for emails. Would you use Chat GPT to do this? Yeah, because it's just it's so efficient in cranking out copy, right? Because that example you talked about before where it's like, okay, the two of us do this, we're all kind of used to that thing where you take one piece of copy and you sit there and you bang out a second version of it in you know 10 minutes or whatever, but to be able to just cut and paste five times and have it spit back five versions and then you can pick and choose what you like.

And then the thing with that is really the additional insight. You know, this is pulling from such a huge corpus of data that you're gonna get new sentences or new ideas that you would never have thought of, you know, by by generating 10 different angles, you'd be sure to find two or three things that you you know hadn't even thought about or considered, and that's really it's kind of like uh you know, no charge brainstorming. I mean, it's a great way to just continue to crank out stuff without too much pain. That makes sense. Um, and I'm wondering, I think there's probably two different ways that you could approach it in terms of getting that content to A B test.

Um, you know, Chris, in terms of prompt engineering, would you actually ask Chat GPT, you know, write 10 email subject lines that I can do A B testing with? So that there would chat GPT know to make to slightly vary them so that you could start to see what's working, what's not? Yep. Okay. Done that um with writing headlines, writing tweets, for example, is is a very another very popular one.

So what are we looking at here? So this was this content idea from Google Search Console. We've now had Chat GPT draft an answer to the question, what can't you measure in Google Analytics? Now, to your point, Katie, you would want to have a human subject matter expert also write the same article, ideally without looking at the ChatGPT version. You know, keep uh church and state separate, and then put both of them up and see which one performs better.

I think it it's an interesting exercise, especially for companies that are really feeling like chat GPT is the answer to a lot of issues or resources uh that they might be facing right now. Uh, you know, there's a lot of companies that are cutting back on budgets, and so they're looking to chat GPT to fill some of those gaps, especially around writing and their content marketing, which is all well and good, but I think that you know, running an experiment like this before making that decision is probably a good way to gauge whether or not Chat GPT can, you know, write to the quality that your brand is accustomed to. Um, is there a way to train Chat GPT specifically on your voice, your style of writing? Like the more that you use it, will it uh learn who you are and what you want to see? Not yet.

In uh OpenAI's Discord server, they were saying that fine-tuning is not available for chat GPT yet. And so that's another consideration, uh, and which is why running an A-B test is a really good idea before making those hard and fast decisions to have a tool like ChatGPT stand in for a human. Because, you know, Chris, to your point, what we're talking about is you can't fine-tune it. You can't train this large learning model to your specific criteria. Basically, the tone and the content is what you're going to get, is what you're going to get.

You can add into the prompt, make it warm, make it friendly, make it stern. You know, you can sort of give it those cues, but it will never truly mirror the way that you write or that you you know, your brand uh tends to speak about things. And so that's a consideration as well. Exactly right. Um, however, that said, uh, if there are plenty of cot uh ideas you know in the content ideas section, if there's stuff that you want to take a crack at, um having a chat GPT version of it might even if you don't do the A B test, might be better than not having anything at all.

No, and I think that that makes a lot of sense. And so, you know, what we had talked about in terms of the risks is making sure that you have a human who's editing it to make sure that it's not saying anything egregiously incorrect. Um, you know, so back to the notion of uh A B testing. So, Chris, if we go to Chat GPT again, um, you know, what would it look like for you to ask ChatGPT to write, you know, five email, you know, subject lines for a sales pitch for our Google Analytics 4 course that we want to A B test. I feel like we've have some of that written down somewhere.

Well, let's pretend we don't. What would how would someone get started with that if they want to start an A-B test for email subject lines? So first thing we have to do is provide guardrails for the system. So we need to one of the more popular things to do is in your prompts to structure it, you know, tell it what kind of person it is. Not because it understands that, but because the the words and phrases that you use in that will contextual give it context for that.

So let's do this. It says you will act as email. I think writing highly effective subject. And so one thing to note as Chris is writing this out is he's not naming an individual person. He's giving the characteristics of the person who is writing uh as chat GPT.

So an email marketing expert writing highly effective subject lines for emails that get opened more than others. That is a very specific type of person, and that is something that chat GPT can start to understand of like, okay, you're looking for the following things. Got it. Right. So that is that is our first set of guardrails.

In doing that, it helps it understand um what it is that we want to be doing, but more specifically, um it's in the context of email marketing. Right. I did, I finally did find my okay. First task need to read compelling subject lines. So John, I hope you're taking notes because I'm gonna make you start doing this for our emails.

No, this is it. I've got I just threw in a random subject line and said generate five versions. And it yeah, it goes, it pulls, you know, because it the line was first look seven weeks to show time. And it gave me countdown to show time, get your first look in seven weeks. Don't miss out first look in seven weeks, get a sneak peek, first look in seven weeks.

So yeah, it just, you know, it's a great way to just fire up synonyms. Nice. I like that. And I would imagine, Chris, um, you know, especially since it's a subject line, you could probably put in the guardrails too of you know, write this no longer than X number of characters or something like that, too. Yes.

What I've done is I provided it now, all the relevant facts to help with the construction of the subject lines as well. Um this is important. Um, you can't just, you know, again, you you don't want to leave very much up to chance. You want to have as much as possible in the prompt so that it knows uh what it is you're talking about. So let's go ahead and put this in the system.

Okay. Yep. As doing that all from the the requirements. So if we think about this, this is nothing more than requirements gathering. Right.

And then it does the thing based on what we provided it. And so now in terms of an A-B test, you know, if you're running an email marketing, your system likely has some kind of an A-B test function built in. So what you would do is you would have the body of your email and then load up all of these different headlines to then run, set up your test in that system, and then you can start to see, you know what? The headlines that ChatGPT wrote actually doing pretty well. Or I do pretty well on my own, and I don't need ChatGPT to be writing headlines for me.

So that's where you start to measure the effectiveness of using a tool like this in your marketing. Is it performing better? Is it performing the same? Is it performing worse? Um, you know, if you have a large email campaign, probably not the best time to be running A-B tests.

You know, you kind of want things to go well and be predictable. Um, you know, but that all comes down to your level of risk and what you can accept. Is it a time to experiment? That really depends on if you're okay with some of the emails not doing as well. Exactly.

So again, to what you're saying, if you want to do another test was now switched up to you are going to do some Instagram stuff. Yeah, and it does make me feel bad for the times in the past where I've sat there and had to like crank out 15 versions of Google ads to be able to do variants on copy there. It just makes life so much easier. I mean, it's like the early settlers, John, they had to churn their own butter and now machines do it for you. I know this is it.

This is the uh advent of the machines. And so what Chris is doing is now you're changing context where there's other elements included in an Instagram post versus an email subject line. So you've included things such as suggested caption, suggested image. And then that way, so it's not going to give you an image, but it will describe the type of image that you could use alongside the caption. I like that it has all the emoji in it.

And we just we just have a difference of opinion and it's a respectful difference of opinion. It is just not my thing. It's got all the analytics emoji. So uh just like any other system, um, you know, you can now put these, you know, you can go through these and decide which of these do I want to put up on our company Instagram page to see does it drive more traffic to our GA4 course, or does not, you know, are people not engaging? And so this is where, you know, as Chris was showing his simple spreadsheet, you'd want to do something similar for any of the other A B tests that you're running to make sure that you're actually gathering the results, because otherwise, what's the point of doing it in the first place?

Um, because what you want to understand is can Chat GPT create something that is just as effective, if not more effective, than what a human can create, you know, and so there's going to be a lot of you know things that you still want humans to be creating. But for something like an Instagram caption, that might not be a bad idea, especially if you have a lot of social media posts that you want to start to get out there and increase the frequency in which you're posting. Exactly right. Now, if you want to do a more advanced system of measurement, um what we've shown with the spreadsheet is is good. It's a good place to start.

Um, there is a technique from biostatistics called uh survival testing. Um we have all it's also been called propensity score matching, um, things like that. And what that does is you designate um which are the treatments, aka the chat GPT driven, you know, made posts or emails or whatever, and then you designate uh everything else as your control, and it will go through your data. Um, try to do apples to apples comparisons uh across the data and eventually come out with an assessment that says here is um here is the the change, the lift from the treatment. Now, this is synthetic data.

This is not uh because we have not obviously run this yet, but this is an example of in this case for the for the treatment factor, the treatment resulted in 117% more conversions, 62% more engaged sessions, 34% more sessions in general, 17% more scrolled users, meaning users scrolled all the way down the page, um, and a lower bounce rate. So in this case, the treatment that was applied really lifted this you know, this set of pages up uh compared to other kinds of pages on our website. And so this treatment would be considered highly successful. So if you start with a catalog of the just the the URLs or whatever. And then you can get the relevant data out of Google Analytics.

You can build some very advanced models for understanding wow this this did or did not you know really lift these posts up or creator them. So what do you think John are you gonna record an episode of marketing over coffee and then have chat GPT record an episode of marketing over coffee and see which one performs better. See what it can do. Well the easy hit for this is transcripts. You know we have a bunch of old transcripts that were automatically generated just to paste them in here and say hey you know proofread these and clean them up I'm gonna A B test that we'll see how accurate the thing gets because if it can get to 85 90% accuracy on cleaning up transcripts then that just becomes a new way to do business.

I think it's actually really good at that. Mm-hmm. Well and you know when we started at the beginning talking about use cases um you know it's one of the use cases that isn't used as frequently as net new content generation. So definitely you know consider using chat GPT to clean up transcripts to clean up copy um to do some editing and then you can A B test that you know the A B test can also be something in terms of time savings and accuracy. It doesn't have to be something externally facing.

But what you really want to be doing is measuring the success and understanding is Chat GPT making our lives easier and doing things better and enhancing and supporting, or is it just a shiny object that we're all getting distracted by and it's no better than what our highly trained marketing team is already is already doing. And so you know, we will continue to keep you know challenging you and uh expecting you to be testing these things because unfortunately people make these snap impulse decisions about these new tools, and you really need to put it through its paces before you say, you know what, this this replaces my whole content marketing team. Um that's probably not a great way to be approaching it. Exactly. So this is an example to what John was just saying.

Um spelling creation. Oh, and you've got proactivated over here, so it's not gonna fight back on the size of anything. So this is from this week's in your insights, and what's it's doing that's very nice is it's fixing a lot of the weirdness that transcripts create. So it's it's adjusting the punctuation and making it a little bit cleaner. It doesn't dramatically change the text.

In fact, you know, the wording is is pretty much word for word, but just the weirdness of the way transcripts come out, the automated transcripts, um this does a terrific job with them. I think that makes a lot of sense. Um any other thoughts on how to be A B testing chat GPT. I mean, that's it's like everything. Um you've got you you want to sit down and figure out what are the five Ps.

What do you want? What's the purpose? Who's gonna do it? What's the process for doing it? We talked a lot about that today with the spreadsheets and things.

Um the platform, is it email, social media, text messaging, is it a blog post? Um deciding where you're gonna test, and then choosing the good measures for measuring the performance. So, like what constitutes good measures. It might not be the analytics we chose today, right? If you were concerned about search, it might be search impressions that is your your measure of success.

But whatever it is, you need to make sure that you're clear about it in order for the software and all of your tests um to assess them correctly. Makes sense. John, final thoughts. Yeah, a big thing was I when we were talking about A-B up front, that pretty much pigeon-hold me. I was thinking about email and maybe some web stuff, but being able to use this across all the ad platforms, you know, especially when you're doing campaigns where you have five, 10, 15 variations.

It's a great way to make your life a whole lot easier instead of having to wordsmith it all yourself. Oh, yeah, for sure. I mean, crank out a couple hundred Google ads. Yeah. And I think that that's the really good takeaway is that you know, it doesn't have to be that boxed-in traditional thinking of what an A-B test is.

There's a lot of different ways that you can be testing the effectiveness of your content across multiple platforms. Um so hopefully from this episode, you'll get some ideas on how to be testing your own content using Chat GPT, especially if it's something that your company or agency is looking to start utilizing even more. The one other thing I would strongly suggest is if your team has technical skills, um, you should be using the chat GPT API. This, the the web interface is kind of like the AI playground. It's a great place to test things out, but then when you want to scale, use the API because you can that you can get a known prompt that works for you, like you know, write a hundred Google AdWords ads, you know, or write add Google ads about this topic like we did with GA4, and just run that code over and over and over again, and you're gonna end up with hundreds or thousands of email subject lines or Google Ads copy, whatever the case may be.

Um, you know, 10 at a time in chat GPT is is a time saver, but having systems that can do it, you know, by the hundreds of the thousands is really a big time saver. I think that's an excellent pro tip, Chris. Thank thanks for that. Um I think that's it for this week. We'll see you all next time.

Thanks for watching today. Be sure to subscribe to our show wherever you're watching it. For more resources and to learn more, check out the Trust Insights Podcast at TrustInsights.ai slash TI podcast, and our weekly email newsletter at trustinsights.ai slash newsletter. Got questions about what you saw in today's episode? Join our free analytics for marketers Slack group at TrustInsights.ai slash analytics for marketers.

See you next time.