So What? Using Generative AI for Survey Analysis

Well, hey there everyone. Happy Thursday. Welcome to So What The Marketing Analytics and Insights Live Show. I am Katie, joined by my trusty chief statistician John. Here are the numbers for you.

We got them. Chris is once again traveling this week. We're keeping him busy. We're keeping him on the road. We're mostly keeping him out of trouble.

Um so John, it got cold this morning, like really cold in New England. Daylight savings time and weather both on the suck list. I'm sorry. I just can't deny that. It's I don't know.

I agree. All of it. I I wholeheartedly agree. Um, so what doesn't suck is what we're talking about today. Uh we are talking about using generative AI for survey analysis.

So we'll sort of go over the basics of how you can use tools like Chat GPT to do some uh qualitative uh data analysis. Um this came up because every quarter we ask a one question survey. And so the one question survey we ask people, we give them a text box, they can add in as much um or as little as they want in terms of responses, but then we have to do something with that data. And we use it to inform our educational tools, our resources, our content, uh any upcoming uh webinars or services. And so it's really valuable data, but getting to the summary of what's in the data is something that we often, you know, don't build in time for, or we forget, or we just sort of keep pushing off.

And so I actually want to do this exercise with all of our past surveys as well to see what comes up because I think it would be really interesting uh exercise. And so John, uh we recently asked if you had money left over in your marketing budget this year, what would you spend it on? What kind of responses were you hoping that we were going to get you know for us, the big thing with this quarter is we've seen a lot of bad signs, you know, and and interestingly enough, Chris has seen some stats as far as things possibly turning around within the last month or two, like things not being as bad as they have been. But I a big part of it was just to um you know validate that data whether that's really the case or not like do people actually have budget and if they did what would they spend it on? And then yeah, for us it's obviously you know it's a blind promotion stunt for us.

I mean we're just like hey what what will people throw money to because we have different services we can provide we have training we have um live events you know there's all kinds of different things we can do. So as we're looking at year-end, you know, burn your budget before it's gone offers, we want to be able to offer you know what our customers are most likely to jump on. So it helps us with some make some product decisions and then yeah give us really a lot more feel for just how the market is doing and what's going on. And historically we've asked questions around more specific topics like Google analytics or um you know what kind of marketing challenges or education. And so this time we wanted to leave it a little bit broad because John to your point we really wanted to make sure that if we had offers or services or content or resources that they were the right ones.

We weren't just sort of throwing everything at the wall and hoping that it would stick or resonate with people. So let's go ahead and get into it. So the first step was to actually get the data. So I wasn't going to have everybody watch me download the data, but so I just I downloaded our data from our website. We use Gravity Forms and I downloaded all the responses into a single CSV file.

And so what I'm going to do next is I'm actually going to bring it that file into uh chat GPT. So I've already so you guys don't have to watch me uh stumble through typing because I John I don't know if this happens to you but I can type fine until somebody's watching oh yeah that's classic uh you know performance anxiety that's you know you're supposed to be a great typer and when it's the stakes aren't there it's fine but yeah it becomes sweaty quickly when the whole world's watching it really does I make most of my mistakes when people are watching which is I know there's a whole psychology behind that but we that's probably a different show. So in chat GPT 4 uh you have to be using chat GP4 and you have to be using the paid version so that you can have the advanced settings of being able to upload your data. So if you have that you'll see this little plus sign right here that's where you can add your attachment so let me go ahead and do that. So first first of all you have to have data you have to have it in some sort of a readable form.

I would probably recommend a CSV file uh for ease of use and so let me go ahead and add that we go great all right so you can see it just added my spreadsheet um and then we have to build the prompt we actually have to tell the system, what we wanted to do. So when I did this a couple of weeks ago, I built a very straightforward prompt. But John, what I want to do is get from you what you think I missed so that we can make this a better, more uh valuable prompt. And so I started with uh the persona. So you are a market research analyst.

Do you think that that's accurate? Yeah, that's good enough for this sure for a first run. Okay. Um, and then I said, and you asked your audience if you had money left over in your marketing budget this year, what would you spend it on? Which is the exact question that we asked in the survey.

Um, I then said you are analyzing the data to be easily understood. I realized in reading through this, I didn't add in a purpose. Like, what are here's what we're planning to do with the data to give a little bit more context. Um, so what would you say? We uh you want to be able to use this data to tailor content, promotion, services.

Yeah, well, how about and just run down after this is what you need to answer. What are you asking it down there? What's the rest of the um I basically say I say show your work and provide a download link. And I'll get into what these two uh pieces of the prompt mean in just a second. Okay, cool.

So yeah, so we can go through the questions, but I want to make sure I sort of get the full context. Yeah, and so you're asking for a chart. Okay, so that's good. So as far as um I don't know, for our audience, you could say that it's primarily focused on business to business. Um you could also say, yeah, we're going to use this data to determine uh what products and services to offer in the fourth quarter.

Um also identify any trends that would reflect um positive or negative economic growth. Okay. Yeah, it's too bad Chris is on the road. He's the master of this stuff. Written 10 million prompts.

So this is good. We're gonna see how we how we stack up and if we can crash the bus. Well, you know, and I feel like this is a good exercise to not have Chris included on because Chris is an expert in artificial intelligence and generative AI and prompt engineering. Whereas I like to think that I represent everybody else who isn't an expert, but I know enough to get the system running. So I want to show what it looks like from a non expert, a non-data scientist perspective, where a lot of us are just sort of like, huh, what can this system do?

What do I need to be able to make it do? Um, so you would said identify any trends that indicate economic dedicate. Or you even say indicate the state of the economy. The perfect. Attached are the responses.

Um, I like to add that in just in case. Like I like to cut down on any ambiguity between the prompt and the system. And so I don't want this stuff to come back and be like, is this the is the data that's attached the data like the data that you want to use? To me, this just sort of cuts down a step of saying like attached are the responses that you'll be analyzing. Um you could also add in into the sentence, like, are you able to read the data?

So that's also a good QA check if you're unsure of how your data is structured. You could say, I'm attaching some data. Are you able to read it? And the system will say either yes or no. Uh, if they say if the system says no, you can say, 'What do I need to do in order for you to be able to read it?

Or can you help me? ' And then you can jump into your prompt. Um, you know, so that's just another way to approach it. I happen to know my data is literally one column of plain text with a bunch of responses. So I'm not too worried about the data not being readable.

Um, so we've given it the data, we've given the prompt, and we said, this is what you need to answer. What are the themes? Can you put these responses into some kind of chart? What doesn't make sense in the responses? What will people primarily spend their budgets on?

Identify any trends uh that indicate the state of the economy. Anything else that you would add to that, John? Anything? I mean, so we can keep asking it questions, but is there anything off the jump that you would want to know? No, yeah, I'm all about iterative, you know, it's let's pull the lever and see what it spits out.

Okay. Um, so I always like to include show your work. When you include this when you're doing some kind of data analysis, what it's going to give you back is Python code. And so if you're someone like Chris who likes to build their own code sets to do data analysis, this is something that would help you start to piece together that code so that you could do it in a repeatable way offline from using uh a generative AI system. So I don't actually need this.

I just like to include it just because I'm curious. And then something that I learned from watching Andy Crestedina was uh adding in provide a download link. And it's such a simple thing, but it was so genius to me of like, oh, that's actually really smart because how else are you going to get the information out of this system? And it's like provide a download link, and the system's like, okay, here you go. So let's see uh what happens.

So I'm gonna do this. It should move pretty quickly because it's a very short data set, a very short prompt. And so the first thing that it does is load the CSV file and examine the contents and structure. Once we understand the data, we can proceed to analyze it. So again, this is what you want to make sure is that it's a readable data set.

And because I said show your work, the system is actually telling me step by step what it's gonna do, which I personally find to be very helpful. Um, you know, you can absolutely skip this step, but for me, it's a good quality assurance check of is it doing what I expected? Was I too vague? Do I need to go back and rethink how I'm approaching the prompt. So that's Grandon.

The other show work that's done, is that actually will that give you a download for the Python? Or what is that? So that is the Python. Oh, that's interesting. So you can just grab that code and you have a copy code button up here.

And so if you wanted to start to build your own code set, this is how you would do it by asking the system to show its work. Okay, so now it's going through to analyze responses. I will take this six-step process based on the information we gave. And so again, we said show your work. So you can see all of the different uh pieces of code that it's starting to assemble, which is really interesting.

So again, you could use that to start to assemble your own uh code set. Uh and so now we're starting to get some information. So it says the categorization of the responses into themes has revealed the following. Uh 28 responses that did not fit into predefined categories were too unique to categorize. So we didn't give it the categories ahead of time.

And I purposely didn't want to do that because I feel like if we said it has to fit into these categories, it would be too biased or it would sort of be too narrow. But if you're a company that has like a set of products or a set of services, you could say which responses fit into these following categories. Yeah, so that's interesting. It's really granular because we've got you know that 28 is far and away the largest bucket of stuff. Mm-hmm.

Um, 18 responses related to data analytics, machine learning and AI, seven responses to content creation, five to SEO and web, three to training and development, uh, three to technology tools, and so on and so forth. And it keeps going down the list. So, what we're starting to get is topics, categories of topics. We have data analytics, content creation, SEO and web, training and development. These are the things that our audience have told us are the most important to them.

If they had extra money, this is where it would go. So that's incredibly valuable. So let's see. Okay, next, I will create a chart to visualize these counts, which will help in understanding what products and service to offer in the fourth quarter. So you can see again, you start to get the code that you can start to piece these things together.

So this is a chart. So if you had to put this into a presentation, a PowerPoint deck, the visualization is already done for you. It says here's the chart showing the frequency of each theme based on survey responses. You can see that data analytics is the most common theme followed by content creation, SEO and web. And this is the key.

You can download the chart image from the following link. And it gave me a link to go ahead and download it. So I'm gonna get a JPEG right into my downloads folder that I can just drop into a presentation or an email and say, this is what the survey found. And it says, next I will take a closer look at the other category to identify any additional themes. And so we know that this was the largest bucket, but we also know that there's probably gonna be some uh hidden gems in here.

And again, you know, because I asked for it to show the work, you see how the code is starting to come together. I like this one. Some responses are quite unique or off topic, such as MDMA, shoes, puppies, and unicorns, which don't seem to make sense in the context of the question. It could be jokes or errors. Um, so that's actually really interesting.

So we can start to say, all right, so if we were gonna do a more in-depth, we could ask the system to like pull out anything with references to the following. Other responses suggest activities not covered by the initial themes, such as sponsorships, social media marketing assistant, and sending small tangible presents to would-be clients, which could which could suggest new categories related to engagement or direct marketing. So that's really interesting. And so we could go back and look at who uh responded with like sponsorships. And you know, John, that could be an opportunity to say if you find X number of dollars, here's some things if you wanted to sponsor.

Yeah, that's uh you know, stuff to jump on there. That's interesting. And yeah, it is funny. I hadn't thought about how, you know, over in the Slack community, there's a lot of frank discussion and a lot of joking. And of course, that translated right over to some of the quick.

Um, so to your question about the economic indicators, there are responses that are indicative of caution due to economic concerns, like I think the economy is going is going to shit and it's going to hide, and it's time to hide away nuts, which may align with savings and investments. Um, you know, so we have to take that with a grain of salt, of course. Um, and again, asking it to show its work. And so now we're seeing after refining the themes and recategorizing other responses, here are the updated counts. And of course, we got a network error.

So that's unhelpful. So we would have to, you know, start this all over again. But so far, John, do you think that this is a valuable exercise? Yeah, it is. I, you know, not surprisingly, data analytics is huge.

I think the another step would be have it output a CSV file, you know, by category. So then we could go back and read the the specific ones and see get a better feel for what's going on there. But and yeah, I don't know. I mean, what's your thought? My gut was, you know, there was a whole long tail of ones that are three or less.

I would just cut that off unless there was something else going on. Well, and I think that those are questions, like if you're purely using generative AI for this analysis, I think that it's good to add that into your prompt. And so, you know, if you sort of give an initial look to the CSV file and be like, it looks like there might be some junk in there, or it looks like there might be some really short answers. Uh, you could probably include in the prompt um only look at responses that are five words or longer, or disregard any responses that are one-word answers, or disregard any responses that include the following terms, such as puppies and unicorns, which I'm pretty sure is was my response when I was testing the survey. Yeah, it's funny.

It's like any import export of data. Like every time you do it, you end up with a grocery list of like, okay, next time we run this, these are the 12 things we gotta add. So now what's interesting is so because we got the network error, we are regenerating it. And the way that it's putting together its responses is different from what we just got five minutes ago. So I'm gonna let it see if it will finish, and then we can go through what the responses are.

But I find that to be really interesting that it's presenting the data a different way this time. Yeah, that's really interesting for you know long term testing and the way you do stuff. And I know some models have a seed value that you can set so that you could get, you know, closer aligned. But yeah, it is interesting how it's really a randomizer in some ways, right? I mean, you're you're getting a different set of probable answers.

So um, yeah, that that that's a little bit shaky for folks pursuing the truth. You know, it's a little bit um creepy that it comes up with a different take every time. Um, you know, and it's funny when earlier this year when Chris and I were going over a lot of these systems, one of the questions that got came up a lot was if I ask a prompt and you ask a prompt and it's the exact same prompt, will we get the exact same answer? And the answer is no. Um we're two different people on two different computers, and even if it's the exact same prompt, I think we actually did this exercise on one of the live streams, you're gonna get different responses even if you hit go at the exact same time, because the amount of data being collected by these systems at any given second is constantly changing the backend database.

And so there's no possible way for you to get identical answers every single time. Okay, so it did finish this time without an error. So that means that we'll be able to ask it new questions. So let me go back up to the top. We see the chart again.

So let's see what it did this time. All right, so again, it sort of went through. Let's start by loading and examining the contents provided in the data. So that's great. And you can see the code is there.

Um and then once it did that, it said, Here are the first five responses from the data set. Um, so it's validating that it can read the data. So we could look at our data set and be like, yep, those are the first five responses. Got it. Um to analyze this data, I will do the following things, which is just repeating what we asked it to do, which is great, because then it means that we have structured the prompt in such a way that the system understood what we were asking.

So it says, first I will proceed with a qualitative analysis to identify the themes. Then we'll qualify these themes and visualize the results, which is interesting. It didn't do this last time. And so if you're wondering why, I couldn't tell you why we got a different, you know, set of responses this time than we did a few minutes ago. Um, but that may be something to definitely look into when you're testing this on your own is if you're not happy with the first uh set of results, try it again, see what happens.

Um we see the most common words found in the responses include terms like marketing, AI tools, training and money. Makes sense. However, the frequency analysis includes many common English words that are not indicative of specific themes. Do you get a clear picture? I'll need to read through the responses and categorize them based on the context context in which the words are used.

So it then it says, I will now do this. So the thematic categorization revealed the following distribution AI and machine learning, marketing, tools and technology, financial caution, training and education, and other. So this again, we start to get these categories which can turn into topics where we could create content or take a look at our services and see what aligns with AI and machine learning or marketing or tools and technology, tech stacks, those kinds of things. Um, and again, it says the other category did not clearly fit. So that's the same as last time, but it's approaching it differently this time.

Uh so the other category, responses to other included the following things. So this time it gave us a better list of other things. And these this list actually helps us think through like what else are we missing? So we have creating promo videos, uh, start a community, sending small tangible presents, uh, conference registration. Uh, I would throw it back into the company, which makes sense.

Um, and then based on the response, it seems there could be additional themes such as client relations, research, traditional advertising, corporate savings, and professional development. Um see, so we got more the refined categories. So it took all of the data that it had above and refined the categories and said, these are the categories that I want to give you that I have found in your data. This is my answer to you. And then it created a visual representation.

And again, uh you can see it's a very clear chart of the frequency. So if you had to bring this into a presentation, um, you'd be able to say, like, based on the responses, this is the category that we need to be focusing on. And you get the download link for the chart, which is great. Um, and it says, based on this analysis, it seems that people would primarily spend their budgets on AI and machine learning enhancements, followed by various marketing efforts, tools, and technology are also significant but to a lesser extent. So this is something we can do something with, John.

Yeah, it's interesting though. I don't like the categories as much as the first round, you know. So I think that would be one thing that you know, we could take the categories from the first prompt and force those to to get a take on it. But um yeah, it's because you know, the interesting it's interesting to have just marketing as a general topic and tools. I mean, obviously those show up usually in everything.

Um, but then having AI and machine learning as its own category. That's interesting because that talks to a lot of the stuff that we've been seeing as far as you know, public speaking for AI requests that have been through the ceiling, and you know, there's a lot of stuff going on on that front. So that does match that, that does kind of validate that the stuff we've been seeing over there is the way the market's gonna go, at least for the next quarter or two, probably. I would agree with that. Um you had asked about the economic trends.

So this time it said the trends indicating the state of the economy, presence of categories like financial caution suggests that there is a sense of economic uncertainty among the respondents. Now, I want to give the caveat that um we have less than a hundred responses in this data set. So I would not say that it is just statistically significant in terms of accurately representing our audience to say this is exactly what they want. For us, this is a guide. If we wanted a more statistically significant representative sample, we would obviously need more responses in order to say yes.

We feel confident saying that these are the things that people want. Um and at the end, it says, would you like to proceed with any further analysis or have any questions on this data? So, John, one of the things that you would ask was can it provide a CSV file of the counts, right? Of the topics? Yeah, and or uh a list of all the responses and the categories that they're in.

File of the responses and the categories. See the whole people watching me type thing that they are in. Please provide a download link to this file. Um, John, when you write prompts, do you say please and thank you? Yes, I want the robot overloads to pass the house over when they come killing.

This guy was nice to me. He wasn't a jerk. So it says I have saved the categorized responses to a CSS. Ooh, a CSV file. You can download it from the following link.

So I'm going to download it. And let me pull it up. So okay, that's interesting. Oh, you'll have to share that one if you want to. Yeah, I'm going to see if I can uh share this one instead.

Share screen. Share window. Categorized responses. All right. So I got exactly what I asked for.

I got a very basic spreadsheet with here are the responses and here are the categories. And so I can just go ahead and uh sort these and see everything that falls into what Chat GPT thought was AI and machine learning, um, client relations and so on and so forth. So that's actually really helpful too because you can then go back to your original file. So we have emails and contacts in our files, but knowing that we're using a public system like ChatGPT, it's not firewalled in our own uh server. I didn't want to provide any contact information.

Any personally identified identifiable information does not belong in a system like Chat GPT. So I purposely cut all of the emails out. So I would not, I would caution you do not use systems like this in order to do that kind of analysis with that data. Um, our data set is small enough that we can then go back and re-poll it again and say who responded to what. Yeah.

And I I mean, I really would love to just for a second, like shine a light on this because this is amazing. Like it because we're dealing with, you know, 40 odd responses here, people are like, all right, whatever. But if you had 5,000 responses, just to be able to do this, this is fantastic. You would be able to chew this up. And then, yeah, privacy is totally concerned.

But the way you can always deal with that is, you know, whether in your Salesforce or HubSpot or whatever you're in, your records do have a unique key. So when you're doing your export, you take the key, and then that way you can run this report and you can match it back and upload it back in. And so now you can run full reports and do all kinds of call lists or whatever based on these categories. So that's a really powerful way to get through a project that could be a massive pain, you know. Like a job like this could really suck if you have 10,000 responses, you know, or if you were hitting Chris's newsletter and we've got 25,000 responses, you know, this could really be a game changer for you to identify what the top 10 things are that need to have action taken.

Mm-hmm. And no, and I think that that's huge because it really is the goal of artificial intelligence, especially generative AI, is to make things more efficient. And so I feel like this is a really good example of instead of me fumbling through Excel and trying to figure out, you know, pivot tables and creating columns and copying and pasting, you know, we've spent, I mean, less than 30 minutes twice now going through Chat GPT, having it do the work. And I now have a usable spreadsheet. I have a bar chart that I can present.

I have a rough summary analysis, all of which I would say go ahead and, you know, make sure you do your QA and edit it yourself. But I feel like it is really helpful to get you, you know, 90% of the way with this kind of analysis. Yeah, and it's fantastic for exception analysis too. You know, as you're looking at those categories, if you agree with those, you can spot check and match them up. And now you only have to deal with the other ones.

You know, and so this would be a great pass you could take and go through the others, come up with the next batch of categories or see if stuff needs to be, you know, properly aligned or whatever. Well, and you know, so one thing that you could do, you know, with that as you're going through it is if you say, you know, this doesn't really fit under uh so a direct mail campaign. That doesn't really fit under AI and machine learning. You could re-categorize it, re-categorize all these, even change the names of the categories and then re-upload the file into chat GPT and do the exercise again, but it's still not going to take you as long as if you were trying to do this manually. And chat GPT gave you the code so that you could start to build it yourself.

Yeah, right. You know, worst case you'd be saving an updated uh prompt. But yeah, if you really wanted to get crazy, you could go into the Python and write that. Mm-hmm. And you know, and for people like Chris, that's exactly what they want to be doing.

And so he uses it to build code for uh different analysis projects. And he's found that it, you know, I think he stated it gets him like 85 to 90% of the way there. And then he just has to sort of put all the pieces together. Um, so for this analysis, John, are there other questions that you would want to ask based on this data? Like what haven't we covered?

Yeah, you're accurately categorizing is really 99% of the work, right? Because ultimately we do get to the point where we have to, you know, read through the analysis and come up with some hypothesis of what we want to test. Yeah, that's always a great question. Just ask what it thinks it should do with it, see if it comes back with any wisdom or stuff that we had forgotten. Um I personally have found that these responses, like when you ask it a question like that, the responses tend to be fairly generic.

Um, but to your point, it is sort of like a good starting place of like, oh, okay, I hadn't even thought of that, for example. Um, so it's saying strategic planning, marketing and sales initiatives, investment decisions, customer engagement, economic insight, content development, RD prioritization, feedback loop, product building, sales training, measure ROI. Like there's a lot of things that Chat GPT thinks we can do with this data set, which is really interesting. Are any of these surprising to you? No, that's a you know, it's funny.

Uh a couple of weeks ago I was running a similar set of queries with um Jenny on crisis communications, and it was the same deal. You know, it came up with a list of like 16 different topics. And so, yeah, it's a great way to cast a net and make sure that you're not missing some major category that you could have been taking advantage of. It's a great safety net for uh your analysis. Um, the other kind of question that I think doesn't get asked a lot to these systems is what haven't we done with this data?

Or you could also say, what else could you do with this data? Um, and so this will be interesting to see what the system thinks we could be doing. Um, and then if you wanted to dig into any one of these, you could say, what you could ask Chat GPT, what do I need to do to get started? Yeah, and it's you know, for this one question thing, it it's you know, a lot of this stuff doesn't apply. But man, if you had some kind of huge data set, or if you just wanted to dump some uh anonymized data out of your CRM system and run some of these reports, it would be killer, you know, as far as sentiment analysis across the board.

Um and longitudinal too. That's you know, we have the past 10 quarters of you know, one cues. So that can definitely be done too. We can see what the trends are on that front and where that goes. Mm-hmm.

Uh, predictive modeling. It's something John, you and I talked about last week. So to your point, if we had a much larger data set, we could do predictive modeling with uh that data to say, okay, what does this look like? Uh, if we forecast it out. We have customer journey mapping, content gap analysis.

That one makes uh sense, competitive analysis, resource allocation, workshop and training, community engagement. So some of these do to your point do feel like a little bit of a stretch. But uh it is again to your point, it casts a really wide net of, you know, what haven't we thought about, or that's a really interesting, or maybe it sparks another idea. Yeah, it runs in this is funny. Shout out to the ninja that you're talking about new custom GPT.

We were just talking about that earlier this week, as far as uh price drops, huge price drops across the board. It's actually much cheaper now. And then some vision stuff, being able to upload images. Um, and yeah, there's there's a whole bunch of stuff rolling out in the new release that's uh Chris is very excited about. He got all wound up on it on Monday.

Anything else we should know about this data set. And here's the thing you can keep asking as many questions as you want to chat GPT, and it won't get exhausted with okay, stop asking questions. Like it is there specifically to do this kind of work for you. So you can exhaust it, like you can try to exhaust it with all of the different questions. Like any questions you have, this is a great place to ask all of them.

You can anticipate the questions that might come up from your steering committee, your decision makers suite, ask those questions and see what ChatGPT gives back so that you can at least be formulated and answered, knowing that those questions are likely to be asked. Uh so let's see. Uh anything else we should know about this data set. Uh it says response bias. Okay, the data might be subject to response if the survey was not distributed evenly.

That's a good one for us to think about. Sample size and representativeness. We did talk about how it is a smaller data set. Uh, survey context, the way the question is phrased, uh, making sure that uh, you know, we don't have that survey bias, data granularity, qualitative data complexity, potential for misinterpretation, qualifiers and non-response, temporal relevance, ethical and private considerations, need for validation, missing contextual data, all really good things that we should be thinking about. So you could say uh, can you write a methodology for this analysis that I can use publicly?

Yeah, that's the heartbreak list there. All researchers can't hate that. That's throwing all their hypotheses into the dumpster. And it says, certainly, here's a sample methodology. And again, you want to make sure that you don't just copy and paste these things into, but you know, John, as you were saying that if you have a much larger data set, this is the type of information you would also want to include.

And having the system do this for you, especially since the system did the analysis, having the system also write the methodology is a really good idea so that you can say, okay, this is what it says it did. Does this align with the way that you would expect it to work? It's actually going pretty well in depth, which is great. Yeah, and this is just classic, like nobody wants to write this document, right? Like having this put together the first draft for you is huge time saver.

All right. So we have all the things. Objective, the primary goal is to understand the preferences of B2B audience regarding their allocation of a marketing budget. That's accurate. We aim to identify key themes, that's accurate.

Data collection, yep, that's accurate. Data processing, that's accurate. So again, this is sort of a good way to also check to make sure that chat GPT analyze the data the way that you would expect it to. So this isn't just even if you don't plan to use this publicly, um, it's a good way to uh approach it so you can say, okay, this is what we did. Because especially if you're presenting this data to someone else, they're likely to ask, like, well, how did you get from A to B data export analysis limitations?

That's great. Conclusion. Um can you put this methodology into a down loadable doc for me? Please provide download link. What I've learned through doing these exercises is there's really no limit to what you can ask the system to do to help and assist you to taking it things farther.

So let's see if this is the thing that breaks it. Yeah, and does it actually will it do a dot DOC file? I don't know. I know it'll do a CSV. I know it will do images.

Yeah, so I don't know. Interesting to see if it's afraid of Microsoft. They shouldn't be given the ownership status of OpenAI, but I mean, even if it's a plain text file, I mean that's a good point. You could say, you know, please create a.txt file that you can just open within Word or some other uh processing software. But yeah, it looks like this one it may have been thinking a little bit more.

So it's also a good way to sort of test the limitations of the system that you're working with. There we go. Oh, here we go. The methodology document has been created. You can access it through this.

Downloaded. Let me just pull it up and I will. Oh, hey, look at that. Let me share my screen, show everybody what we got. I think this is a pretty good place to start to wrap up this conversation because I'm actually really impressed by this.

So yeah, there. And it's our it's uh pretty well formatted, uh, very easy to copy and paste. I could just make my edits directly in here. And so, you know, I could go back through and say, you know, can you put all the pieces of code together into one, you know, file so I can download that? So there's a lot of different ways you can approach um getting the information out of Chat GPT.

Uh, you could ask it to put the summarizations, the analysis into a document and download that. So again, you don't have to like copy and paste everything. Just tell the system what you want it to do in very clear language, and it should be able to do it for you. But in less than 45 minutes, John, we have analyzed our data. We have started to understand what our audience wants to spend their money on if they had it.

Um, we have a methodology statement. We know that there are limitations with the data. This is all within 45 minutes. I mean, that's pretty good. Yeah, being able to clean up and categorize the data in you know, five minutes is just game changing.

Um, so what else would you add before we start to let to give chat GPT a break? Anything else? No, I think that's a good you know place to run with. I think the the thing with this too is you know, you see that, and then over the next three days you come up with more ideas for like, okay, how could we use this and where could it go? And so this is definitely it's great to play around with it.

And uh anything that can speed up chewing of the data, formatting it in different file types, all those headaches that slow everything down. It's great to find ways to make that happen faster. Um, I would say too, and so what what we saw was like limitations, representativeness. You can start to ask the system those questions of how many more responses do I need to get in order to have a representative sample? And it can start to outline those plans for you of if I have 70 responses, but my audience is 10,000 people, how many responses do I need to get in order to have it to be statistically significant?

Yeah, 70 responses if you're if you're getting you know under 30 in the individual categories, yeah, you're nowhere near where you need to be. You gotta go back to the drawing board. Um, so we covered um bringing your qualitative data into chat GPT to do a basic straightforward analysis. Uh I have my charts, I have my uh summaries, I have my methodology. I think I'm good to go.

So if I had to take this to a meeting, I would feel pretty confident to say I at least have talking points to say this is what we've learned from this data. We can certainly do a deeper dive, but now we can create a content calendar, we can, you know, think about promotional emails, highlighting certain products. We have one-to-one outreach that we can do. So there's a lot of next steps that we can now take from this short amount of time that we just spent uh doing this exercise. Sounds good.

Oh, yeah, you know, we had one other question asking about security risk for getting the data. And yeah, it's absolute security risk. You don't want to be just randomly loading, you know, stuff lying around your company. And we have clients that we've worked with on this stuff as far as being able to set up something in-house on your own server, so then you can upload all the goodness you want, and then you have the added benefit of that's your training set too, so that you can adapt the model to your business specifically and not share it with anyone. So uh yeah, good question.

Thanks for that. Yeah. So if you're using uh public systems, if you're subscribing to Chat GPT and it's not firewalled with on your own service, you're not building your own own large language models as much as you can, don't include any of that information. John, you made a really good point earlier about systems like Hubspot and Salesforce have unique identifiers attached to each record. So use those numbers instead when you're doing the analysis so then you can re-import the data into your CRM system or whatever system you're using, and it matches back to those individual customers rather than using their names, their emails, and their companies.

Yeah, don't do that. Um any final words? Get back to work. There's plenty of data to churn through. Get at it.

All right. Well, uh, we're gonna be on hiatus for the next two weeks. We'll be back here on November 30th. Until then, you know, stay classy, San Diego. Thanks for watching today.

Be sure to subscribe to our show wherever you're watching it. For more resources and to learn more, check out the Trust Insights Podcast at TrustInsights.ai slash TI podcast and a weekly email newsletter at TrustInsights.ai slash newsletter. Got questions about what you saw in today's episode? Join our free analytics for marketers Slack group at TrustInsights.ai slash analytics for marketers. See you next time.