Fireside Chat: Consumer Research and AI with Stefanie Francis

Well, good morning, good afternoon, good evening. Uh, this is Fireside Chat. Today I am chatting with Stephanie Francis, founder and CEO of Hootology. Stephanie, how are you? I am well, Christopher.

So glad to be here. All right. Uh let's start easy. We already know who you are. What do you do?

And what in the world is a hootology? A hootology is a mashup word that we created in 2016 when we started, and it's reflective of three different things. The hoot stands for two parts. It stands for in cahoots or in partnership with our clients. But yada yada yada, what agency doesn't say they're in partnership with our clients.

So the next layer is going to be a hoot nanny or a celebratory good time. And got a lot of pushback in the beginning about calling combining business and fun and thought a lot about it and took took the dear mentors whose whose advice was to stay away from that to heart and realize that we love what we do. It's a lot is at stake. It's challenging, but it's also fun. And so we didn't want to want to run away from that.

And then the last part is ology or the ongoing evolution. And so it's really our commitment to ourselves to constantly be wondering and questioning is the way that we're going about things today the thing that makes the most sense for today, right? So ongoing learning and that internal challenge. Got it. And your focus is market research.

Yes. Yeah, we we grew out of I I was a market researcher, and we've grown uh through and uh with market research over the last probably 25 years. Okay. Today you're trying you are using generative AI as part of that. Talk to me a bit about how that works.

Yeah, yeah, there's it um it didn't happen overnight, right? It wasn't a bandwagon that we jumped on um uh quite recently. Uh but it was it was something that in my personally in my years um just felt this angst around um understanding the beauty of quantitative research, or what what we would all know on this call is more surveys that we all see probably uh more than we want to see in our inboxes these days, and qualitative research or really that depth of psychological knowledge that comes from a one-on-one interview to understand thinking and rationale or something like a focus group. And the the angst over the decades for me was was gosh, there there has to be a way that we can quantify or have that statistical assurance of qualitative information or that irrationality of the human mind. And so that was just a drive of mine for for so long.

And I was teaching at NYU. I was a I was an adjunct professor in a graduate program um for a course called Global Market Research. And um, I don't know if you remember your market research professors or or anyone listening in at some point remembers theirs, but I can tell you that my graduate students when they walked in on day one didn't expect their uh statistics or market research professor to look like me. Um, and so it became kind of a fun, a really kind of fun collaborative process where um was really around changing thinking and opening up minds to possibilities. That was my number one goal as a professor.

And I realized in my in my seventh year that I didn't, I was I was spending so much time helping my students be prepared to doubt the the quantitative and qualitative research that they that they would be receiving in their decision making roles post graduation that that it was the moment where I was like, I know we at Hootology could create something today. And and before the first time we all heard about generative AI, we were already starting to work on solving for that, like actively solving for it. And then what seems like we all woke up one day and Gen AI was everywhere with OpenAI's announcement. Um it was the unlock. Um, and so it was really the piece that now we're using to be able to have that quantification or that statistical reliability around qualitative or kind of just like just just psychological thinking that we all have.

Talk to me more in depth about what how that works. Like how are you picking apart qualitative data? Like what's the what's the the thinking there? Yeah. Well, Christopher, maybe it's in the interest of like knowing sort of which direction for me to go.

Tell me in terms of of your understanding about about qualitative data and quantitative data and how I can better, I want to get the framework for how I can best sort of uh tee it up so I don't kind of go down the research uh, you know, nerd rabbit hole. Oh, the the nerd rabbit hole is the best place to live. No, um, quantitative research is how much of anything, and then qualitative research is is uh what the thing is. So the way that we typically talk about it when we're talking with people is to say you do qualitative research to figure out what the questions are, and then you do quantitative research to figure out to quantify uh the answers to those questions. Because it may turn out, yeah, like some you have one weirdo like me, like, oh yeah, this is the most important thing and I'm like literally the only person uh in the world that cares about that thing and so it's it's sort of this iterative process of qual quant qual quant you uh the second round you look at the answers to the quantitative service like why did this happen then you go back and you ask more qualitative questions to dig deeper and so on and so forth that's that's generally the framework that I've that I've operated on that I was taught.

Yeah yeah thank you you you communicated that so much more simply than I could have um and so I think the very um technical answer to what we've done is we've mushed all that together and so we've made all of that possible in one moment in time so let me pick apart um a little bit and go a little bit deeper into like what you beautifully laid out in terms of the difference between qual and and note the first thing you said which was so cool which was we do qual first to figure out the questions right and so and so what that means is um we gather a group of people who are the the type of people that we're looking to understand. So this could be um it could be soda drinkers if we're looking to understand kind of the big soda decisions it could be tech users it could be B2B decision makers in a certain industry um and we ask them basically how they're how they're thinking about something to uncover the topics that that in the past we would then design a questionnaire around right um and that was a beautiful time um because both of those things were fantastic ways about going going about it. But the difference the the the point the the difference changed when there was a moment where like there wasn't the luxury of time to do these two very kind of in-depth studies, and the costs got more and more expensive. And so at the speed the brands were needing to move, a lot of times they started having to choose one or the other, right? So kind of in a dream world, what we would do would be a qualitative conversation to, as you said, to uncover the questions, the quantitative to be able to figure out the what or the how many, but then qual again to be able to figure out, okay, we've made the decisions off of these two, let's make sure those are right before we go live.

And so what we've been able to do is be able to have this conversation in a real-time virtual environment that has a quantitative camp uh sample size. So the the ability to have that statistical assurance because we're talking to enough people, but allowing every part of that conversation to start with people's thoughts and ideas. So that's really just the launching point. But there's something else that you said that I think is worth kind of talking about before we dig into kind of the the intricacies of hooks or the platform that we created, is if you if you take both um quant and qual in isolation, um, you mentioned kind of that the the the one weirdo, right? I also in my life am the one weirdo.

So this is why I think that we've been come together uh today. And if we're if we're talking to a group of maybe eight people in a conversation that looks like a focus group, um, and one person has an idea, we we have no idea if that's a fantastic idea if we if we take that to the entire population with with just doing qual. And we have no idea what that idea is if we just do quant, right? And so by starting a conversation with a statistically representative sample, so whether that be 30 people or 2000, we're then able to have that group of people tell us what's on their mind as that launching point. So the generative AI role in this, is that in the interviewing?

Is it acting as the interviewer or is it acting as the respondent? Yeah, neither, um, in our case. Um, so the so the the there is a lot of conversation out there, and I think a lot of really good applications around um generative AI uh serving as a moderator, right? And doing a whole series of one-on-one conversations and tailoring each conversations as the moderator based on the responses that the individual person um is providing. The the concern that we had with that is um for for the purposes of what we're looking to do, I think that there's there's great applications of that, is then we still have a hundred, five hundred, fifty, whatever it might be, um, individual conversations that are not unique that we can't then therefore quantify and compare to each other.

So in our case, um, we're able to start with one, the same question. And um everyone in the conversation who is the profile of who we're looking to understand for for some pretty big decisions, um, all respond to that one question. And so the the first element of our gender of AI is employed is it's looking at those responses in real time and uh deducing the themes and coming up with those themes to be able to continue the conversation. So, in essence, Christopher, we want to think about it like with a survey, um, the creator of that survey decides what the five or six options are for that question. And in the hooks conversation, the people we're looking to understand tell us what the five or six things are that are most important to them.

Got it. So you're essentially using it as an NLP engine. As we we are at at the first step, yes, right, to be able to synthesize um uh and find the information to be able to continue and and go deeper. Have you looked at the work from Stanford um from last summer about using generative AI as the respondent? Yeah, yeah.

This is what I'm excited to hear from you because I've heard a lot about um the your interest in sort of getting past the human component. Is that is that accurate? Because this is a fun, this is gonna be fun. Yeah, there was a paper, and the paper was titled, let me actually bring it up here, um predicting results of social science experiments using large language models. This is from Hewitt et al.

August of 2024. And what they found was they basically took a bunch of existing social science experiments where they had data sets and things like that, and they created generative AI, essentially avatars to behave like the respondents to the surveys, and they basically had the uh the language models respond to the surveys as though they were those people, and then looked at the level of accuracy. How accurate was the were the respondents for uh to to simulate human beings? And what they found was for training for data and studies that were not published and therefore could not have been in the training day of these models, it had an accuracy range for about 90%. So that's obviously you know, not point, you know, point uh a p-value of 0.95 or or or whatever.

So there's still some wiggle room there. But in terms of being able to scale, it particularly if you're trying, as you mentioned at the beginning, if you are a brand and then you're looking at like spending three million dollars on a massive market research campaign. Um if you can get 90% of the accuracy of real people, but at a substantially lower cost, depending on the model, uh that would I would imagine that would be fairly appealing to to a budget-minded exec. Absolutely. Um I um kind of back to the theology part of hootology, the um I'm at my core driven to believe the next big thing is possible.

Um, so I'm I'm wholeheartedly with you that at some point, um, and that might be tomorrow, it might be today, um, or or my personal hypothesis, I'm a 52 year old, 52 years old, uh, squarely Gen X in every single way. And uh I do believe that that's probably gonna come um after my lifetime in terms of that true level of accuracy for the types of decisions that large scale brands, global organizations and agencies need to make and need to get right, right? So um, what I would would have loved to do is to be able to prep and dig into the I'm I'm the nerdy person who um I will gladly send you this stuff afterwards, yeah, yeah, yeah. Um, but being but being a little bit unprepared right now in the moment where I don've been. Oh no, sorry is no, I love this.

This is fine. The the methodology, right? So I was the person who at the start of any research uh presentation or conversation, this is what I did with my students, is like, let me go to the methodology and understand how how we got to this and what they're what they're saying it could be used for. So I want to know what has the 95% accuracy, what types of topics? And so my my question and concern would be your I am 100% sure it could predict um a lot of things that are really important that would save big companies and big organizations money by being able to bypass the human piece, right?

So it might be um purchase behavior, near-term purchase behavior, it might be um purchase intention and things of that nature. What I would wonder and want to learn more about is it yet able to understand the future? Is it yet able to, if it's trained on past data, how, as we all know in terms of again, I'm gonna I'm gonna disclaim my own irrationality, right? Um, the the ability to have aggregate irrationality um with accuracy. So let's think about this, right?

Let's pull the example of maybe um a past example sometimes is easier than kind of like uh imagining. Salad water. Well, no, let's go, let's go back to the iPod, right? Uh 2001. Okay.

If this was um we were all as humans craving the iPod based on our behavior, but we couldn't have said this is what I want, right? So so our past behavior, right? From the time I was in high school, it was making mixed tapes off the radio and hoping we we get the recording right. Then it became um a hundred CD players where we would spend time programming, blah blah blah. We were we were looking to get all of our favorite songs in one place, and we were maximizing the possibility of what was available today with our behavior because we couldn't yet say, oh, I want this.

And so what we have created is a way that can allow us to look at the current day moment of a particular group psychology and dig into um the nuance of what we're asking for and what we're really saying about our needs and wants for the future. So to restate that we're looking for low probability. Because until Steve Jobs came up with the idea with Johnny Ive, that was basically that idea existed in two people's heads out of you know six billion people at the time. It's extremely low probability. Whereas today's generative AI models are are by definition high probability models.

So they will come up with an expression like person has severe gastric distress, right? That's a high probability phrase. They would probably not engineer the phrase, it looks like you power washed your restroom with Nutella, right? Which is a very low probability phrase you probably would not see or hear often, but the low probability is inherently more creative and and more interesting to read or listen to because it hasn't existed yet. Is that a fair recap of that?

Yeah, you took me though from iPod to to to Nutella to gastrointestinal problems, and so I think I think I can thread a needle, Christopher, but but but boy, oh boy, tell me what your point is. Well the point is that it what you were trying to say is that the generative AI models can't predict what has never happened because it's not in their training data sets. And therefore if you were using the synthetic respondents they would not be able to articulate that they really desperately wanted an iPod because they didn't know it existed in the same way that uh Henry Ford famously said if we asked people what they wanted, they'd say wanted a faster horse. Right. Yeah.

And so it if we think about um we are completely in alignment that that I do think bypassing the human eventually is going to be possible, right? And it certainly I believe is possible today. It's also a matter of what's at stake of getting it right. So let's um let's think about like if we're sitting in the seats with a big pressure of of a of a billion dollar 100 million dollar decision whatever it might be on our shoulders are we sure enough that the model that doesn't include humans is going to get it right so maybe what comes to mind and we we did no work on this particular project um but if we think about the Kendall General commercial for for PepsiCo right um conceptually um it it might have come from a from a decent place and a good intention but it or if we had maybe um been able to generate it as a high probability that this would be a great idea and not have humans in the loop right look at how we could have saved the impact on on brand reputation if we had taken a moment to ensure that that this was on point in all of the ways it it it was clearly missing the mark. Um, so I just think that the the decisions are are too big for the types of things that in this moment we're trusting that reliability to be right without without involving the human piece, especially when now we can quantify that qualitative component or quantify the psychology of the why behind what people's thinking is.

So I'll play devil's advocate and say, well, humans were responsible for the the Kendall general thing. So clearly a machine probably could not have done worse. Yeah, maybe, right? So let's let's let's devil upon devil here then because the the humans that were in charge of that were experts, right? And I and I say that in a way with with air quotes not as a not as shade, but as acknowledgement that they're so in it.

And I think that that we always forget when we're the experts, we're the ones we're thinking about this we're every single day, eight hours, arguably 12 hours, possibly sometimes more every single day, and we forget how far away we we are from the end user of whatever it is. So a couple examples from the past, um, we did a lot of work with the NFL. And I remember there was a moment where um we were talking about younger generations and and one of the super senior executives said, Well, well, my teenage son doesn't think that, right? And it's like, and so therefore it is true, right? About the NFL.

My teenage son doesn't think that about the NFL, right? And you you like this this teenage son is probably a fantastic human, but but cannot be representative of teenage male fan base of the NFL for the future, right? And and so that bias um is is so easy in so many ways to miss. We also were working with a um a major league baseball team uh in a moment where the spring training location was changing for the first time, I think in 20 years. And it was three weeks before uh the first spring training was ever supposed to be kind of in the in the local market that the team actually played.

And uh it was it was a traditional focus group day. So I was literally in a traditional focus group facility behind the the triple silencing glass and the the senior execs were behind that glass. Uh and I remember that I was sitting talking to eight eight fans, and and fans by the definition that we had all decided that for this particular conversation was a fan, because obviously a fan can mean a lot of different things, and no one around the table even knew that spring training was moving, and this had been a 10-year process. And I could hear screaming behind that triple plane glass and things hitting it, right? Because it was like, how can they not know?

And it's like, how can how can we forget that for the last 10 years, this has been one of the number one objectives for everybody within that organization? And so that bias and that expertise, and so it's like that is where we have to listen and remember that the true experts are the ones who who who are gonna end up use our use our product services, and not just from a from a capitalism standpoint, but how can we understand to give people and create for people meaningful connection? So whether that be with new products that actually matter and enhance their lives, whether it be with campaigns that make them feel connected and that they truly share values with the brand, like there's ways to do this in a way that is both profitable and meaningful for that for the human condition that we're all stuck with for arguably uh you know 80 years, unless unless those folks who are trying to crack that code for 150 get it right. They're not gonna do it in time for me, which I'm okay with. Yes, well, there's uh there's a whole rat hole we could dive down there, but we will we will give that a miss for the moment.

Um you took us on the Natella Gratz gastrointestinal track. I believe, I believe in you, Christopher. You can do it. Well, so what I want to come back to though is because I like the the framing that yes, we have to ask people and get away from those experts and things like that. One of the things that's interesting about generative AI models is how quickly they have advanced.

When that paper that I showed you was written, they were using a model called GPT4, which was OpenAI's big, big model in uh 2023. GPT4, by today's standards, would be a face-rolling moron. Um it really is uh quite possibly, you know, now one of the dumbest models there is. So I'm gonna bring up this chart from uh this is from a site called Artificial Analysis. Artificial analysis is uh I find it uh very useful for understanding what's what's going on with AI models.

And this particular chart here, the GPQA diamond chart, shows the scores of various models. There is GPT4 that that model is was uh in that study. That had it would have scored a 36%. A human PhD on this chart scores at about a 70% 70 to 80 percent, depending on where you are, if it's within their expertise. So an aster astrophysicist PhD will answer about an 80 percent.

What we see is that today's foundation models are all at or above human PhD levels, which means that they have the ability to answer very, very difficult questions correctly and thoughtfully. And so to your point about how humans in particular can bring a whole bunch of of questionable data baggage with them, or are it it with a at the stakeholder side are making decisions that are uh a little too insulated. Uh I could see a a case where you have something that's a hybrid, where you have either generative AI as a moderator in a system like what you're describing, or having humans be the moderators and genitive AI being the respondents, so that there's still people involved, but it's either scales faster or it shrinks the response times. What do you make of that? Yeah, well, I think you I think you bring up an interesting point with your chart and your emphasis on the PhD piece, right?

So let's let's start there and then let's bring it back to a little bit deeper in terms of um our platform hooks and and how we are using it, right? Because we we only stopped at the first the first step. So what's interesting is you mentioned PhD level, right? And that was that was the emphasis of that chart um that you repeated a few times. And so I think what we're what you're emphasizing and and reiterating in in uh for me is exactly the point I was trying to make, which is like we can get to understanding if we're talking about facts of like black and white facts of a PhD level knowledge of any particular topic, the models can now give us that.

And that is all proven, and that is all like patented, locked and loaded, right? And so so yes, it can get there. The the the reasoning piece, which which the the most latest edition from open II is starting to look at that, that reasoning component. And just just now, and again, it's still still very early days, we all know that. It's is it going to be able to replicate the reasoning, the rationale, the thinking of humans who are not PHD level, right?

Who are who are who are thinking about things like we all think about in our daily lives across, but but thinking about it from across all different um elements that make up the 350 million people just in this this particular country, not to mention um around the world, is what's on our daily minds, right? It's it's it's our bills, it's our relationships, it's our ability to be able to connect and get everything done that we have to do. And and so we're possibly bypassing the understanding by focusing just on PhD being the goal, right? So if we flip back into what you mentioned about moderating, right? So generative AI, you're right, in our world, which um I I've sort of bristled around calling us market researchers for a while, and that's not as a as a shade to the industry because I am squarely, I have the badge.

I um I was the most I I earned it hard and true with the global market research agencies. I was, I was the most I try and tell people who are new to my life, I'm like, I used to be so uptight, so rigid, nothing was possible. It was always it had to be this certain way, else it wasn't right and you couldn't consider it. To being like, okay, you know, these these traditional modalities, your surveys and focus groups and IDIs and ethn ethnographic research, they deserve to be in the hall of fame. Like, and that means honoring that they were fantastic for the time, but wondering how we keep that sound research principle and bring it in.

So the ways that generative AI is are mostly being used so far, is going to be that that that moderation component with the caveat that the folks who are in the conversation are each having a different conversation because it's tailored to them. Therefore, it's challenging to combine. Being able to talk to, it's what it's being called quantitative data, right? To pull in lots of different data sets and to be able to create an agent that is able to kind of go and look across. Um, and this comes in my like uptight uh professor component, is it's going to be darn near impossible to calibrate those methodologies to ensure that we can connect these data points, right?

That's going to be hard. And I would love to sit in a room with whiskey and a dark moody environment and talk to somebody who's saying I can do this. I would love, I want to be, I want to, I want to, I want to, I want to see it, I want to believe it. I it's it's scary, right? It's scary because I've seen people do that with secondary research for so many decades to now say um this other thing that doesn't show their work is actually spitting out the right answer, right?

Or then the ability to be able to do what what we're doing, which is basically um go back and forth between quant and qual, kind of like you said in the beginning, right? And but but you talked about doing qual first to get the questions and then quant so the way that we're deploying generative AI is is starting with those that open ended question being able to have all for the sake of this conversation I'll say 800 people um 800 people all respond with no bot with no bias the meaning I don't see a list I don't see what another person in the conversation said all I see all I see is tell and tell us you know tell us tell us how you feel about this topic so so all 800 people put in their responses generative AI goes to work just on this conversation right so we're not bringing in stuff that we can't be sure about I can't put my name on I can't ensure that it's worth the hootology brand right because they don't know how they're gathering it and what they're doing with it saying okay there's there's these 800 responses and they're rich and they're they're deep because there's an electricity of this synchronous real time conversation. And it pretty quickly because this is a pretty basic feature at this point of generative AI, it finds those themes. So now it's the people telling us what's important to them. And then we flip immediately into that quant right so now we can kind of toggle back and forth between going deeper and then being able to figure out the how many or how what so those those those themes or what would have been in the quant survey those list of options right um that the expert created now the people are telling us and now they're going to vote on.

So they're gonna have a select all so we can have that actual quantification percentage around what's truly on the on the hearts and minds of of people so so think about um political surveys are are are always are always uh compelling for for lots of reasons. Um, and and if we think about what's truly important to to, if you will, uh everyday, everyday folks, more more moderate folks, or the folks who aren't necessarily the ones um that have a representative talking head on on cable news cycle, is hearing hearing from them what's important and listening to that versus kind of being told what's important and responding to those things. Um, so we're able to then have those quantitative features, be able to rate things, and then be able to flip back in, Christopher, to say, okay, you've now told us, irrespective, Christopher, what you individually told us in the beginning was important to you on this topic. You've now been able to see what some other people think in terms of that aggregated themes. You've been able to clarify how important each each of the ones are that are important to you.

And now I know what of this list of what's most important to you. Now, Christopher, I can flip back in to qualitative and say, tell me why. Tell me, I have now have your first response, and now I have with some other awareness and interest from other people, I now understand what now you're saying is your most important, and now I want to hear why. And so let's say there's 800 people in the conversation, and a hundred of them all agreed with you that this one particular theme that you all told us was important is the most important. Now a hundred of you tell us why that's important.

We can now validate that and summarize that in the in the moment, and then have you tell us if it's right. So the the difference uh of how we're deploying a genitive AI is we never have it work without checking in with that representative sample, so we can say this is actually true statistically, because the the folks that we've pulled together, who we already know is a representative sample say it's true. So there's no guessing. So we don't get into a place where we go we go live with a very, very expensive campaign and risk having missed something, or we never go live go spend hundreds of million dollars developing something that seems like a really good idea without having missed something. Got it.

So mentally I see this. So there's the person, there's the questions, they respond. That's parsed and themed, then quantified by the because the panels in operation in in uh presumably near real time or something something like that. And then it goes back to qual, they get to talk about why, and so on and so forth. The process repeats, where essentially the language model is doing summarization extraction, uh, and then you have more traditional processing, just doing all the the stats on the back end.

Is that rough approximation of the system? That is a rough approximation. Okay. One thing you mentioned, and I would I really wanted to hear a bit more of this, is how you mentioned bias. Um, and there's obviously different kinds of bias, there's statistical bias where it differs from the population, there's cognitive bias, and then there's you know we call belief bias, like, oh, those damn Koreans, uh, you know, which is is more belief.

It would classify as things like racism or sexism, um, which are are not biased statistics, but they're biased that they're ways of alter our thinking. In this system and in in the research field in general, how are you? How have you seen um belief-based bias show up? And then how have you mitigated it to the extent that you can without removing humans? Um it's accepted in our world or has been accepted that there's going to be some level of bias baked into every study.

But the fact that you're actually doing a study is creating bias, right? Right. Um, and so and so it becomes along the way and over the years, how you can take out as much as possible. So the types of bias that come in that are that are just baked into uh humans based on factors I think that that we we hopefully can all agree upon are are true. There's there's bias in the quant side of who's writing the questionnaire, right?

Um so most of the people writing questionnaires uh have uh at least a master's degree, which is a a small percentage of the US population. Um they're probably from a majority uh racial or ethnic background, and so they're baking in um likely unintentionally uh their own bias that they may or may not be aware of. Um and that all then seeps out into what the the findings are if we if we aren't careful enough to understand the difference and the uniqueness of people to be able to write a quantitative questionnaire in a way that that as is as free of bias as possible. And it's it's unfortunately uh very uncommon that that bias is is is not baked in. And you can't see that in the numbers on the on the qualitative side, the bias gets baked in by who the moderator is, and same thing in terms of their own self-awareness around how they appear and interact in the world and the benefits and privileges that they've had or not had.

And then the bias of grabbing eight strangers, if we're talking about a traditional focus group, and the social dynamics that appear in terms of status and power that impact a conversation, right? So impact, depending on the dynamics and the psychology of each of the people in the group, who speaks up first, and then what that does to impact the other people in the conversation, that that doesn't show up in the findings for qual. So by no means uh is there a solution that has no bias in it for today, um, even including bypassing the people, right? Because we've we all know that um at this moment in time um who the large language models were were created by and and this particular moment for, there's there's bias baked into that system if we're bypassing actual humans. So where I personally stand as it relates to what we've created is we have tried to eliminate that in in several ways, right?

Is ensuring that the people who are invited into any particular conversation are fully reflective of the people, the total population that we're looking to understand. Um, and that by starting the conversation without a list or without um an expectation of response, now we can start with as um raw and and and clean, pure using research y type words that that don't necessarily translate in in a way that makes sense, but information that is as true as possible so we can understand and listen in a way that as a human, we sometimes can't. Do you do any bias testing on the AI models that you use within the system for the summarization and extraction? If so, how? Yeah.

I believe, Christopher, we're getting around that by just looking at the information that's in this particular conversation. By by closing it off to any sort of open source additional information, we're able to just look at who's in the conversation and we control who's in the conversation. What we did do in our nine months of development is we had the benefit of having um several longitudinal qualitative studies with some some very large clients and where we had kind of had a history of human analysis. And so we went into that testing of our of our prompting strategy to continue to work um to ensure that testing what the themes that were being generated and the insights that were being generated in our early stages of development to then be able to see if we were close to the human, like the human assessment or the human analysis and being willed to do that over time. But but one thing that's important to me is we had the mindset of we need to be open-minded, right?

Ology. Um we need to be open-minded to the fact that as humans, we may not have gotten it right. So we weren't just trying to program this to replicate us because us has our own bias baked into things. And so it was like it was that this constant tension for almost nine months of testing to be sure um which one's actually more right, right? And until we get into the place of what we feel good about.

And the piece that also um mitigates the bias is that we are never taking what generative AI turns out, whether it be summation of the themes, whether it be identification of those outliers, which I want to talk about, um, or whether it be um the summarization of of the insight without asking the folks that we're talking to if it's right, right? So so the bias is gonna catch itself in every single conversation and in every single touch point because we don't assume it's right. We then can say, hey, here's the insight that you and the people that um chose this theme, why you told us is this is this accurate? So every time we have an insight, we can have that accuracy score that says this is 95% of folks said this was right, this particular insight. This these six themes that were just generated a hundred percent of folks said that they see their theme in there, and so therefore we feel confident that it's reflective.

Um, but we actually don't want it to be 100%, and then that that will get us to the outliers piece because we know that the next great idea is not gonna show up in a in a generated theme from the population. That's right, it's going to be that one weirdo that has that's a great idea. Yeah, you know what's great though is now we have hundreds of weirdos, right? And and so that's that's beautiful because if we have 800 people in the conversation, there's gonna be now, I'm gonna just just wildly pull uh a non-scientific number from all of my years of working in understanding outliers, right? Um, there's gonna be 80 weirdos in that conversation, and you and I are gonna be one of them, right?

And so now we have let's say that's so there's 10% of people who who have a different idea than the masses. So we ask, how do you what are your thoughts on this particular topic? Again, could be about politics, could be about the future of healthcare, could be about the future of education, could be about um needs and wants or hopes for their next um tech device. And probably 90% of people are gonna think one of four or five, six things, right? And we're gonna be able to classify those pretty quickly and bring those forward as our as our list to start with.

But what about these 80, right? What do we do with these 80? Because there's either magic in these 80 or there's or there's madness. I say that lovingly. Um so what we're now able to do is hold those aside and continue to work on those while the conversation goes.

And when I say work on those, what I want you to think about is there probably would have been a seventh theme and an eighth theme if we had if we had gone down. And so there's probably of those 80, there's probably 60 that are saying maybe one, two, or three of the same next things. So what we want to do is pick the best ones of those, um, and and bring those, hold those aside, and then there's those 20 that don't fit anywhere, right? And so now we have a point in the conversation later when we've done all of those different steps that we've talked about, and we say, Hey, thanks, we're almost done with this topic. Here's an idea that someone else said.

Now we're bringing it back to all 800. And we say, What do you think about this? So, okay, and imagine we had 100 people in this in this in this large focus group, and one person had this wacky idea. Now all 800 get to have a quantitative response to that. So we can now separate trend versus noise.

So if we're dealing with some of those large companies, those large agencies that we work with, or the large brands or large global organizations that have a lot at risk, they now can say, oh, this idea, we don't need to spend 50 million dollars to bring this to life because actually people don't want it. And it might look like on social media that it's that idea is blowing up, but we can't quantify what percentage of the of the of the population is actually talking about that, not to mention bots and all of the other things that we know about. So this allows us to have that statistical rigor of that of that survey with that crazy idea from qual and be able to say, you know what, only 10% of people actually think that this is a good idea. Let's let that go. But hey, look at these five things that only five people mentioned, but 80, 90, 100% of your of this of the of your target market thinks this is a great idea.

Heck, we just saved a lot of money in that in that part of the conversation, and now we can get into the why of those things in future conversations. Which is very cool. My last question is a philosophical one of sorts. Um market research and the research field in general is predicated on the premise that facts and truth and data actually matter. When we look at the behavior of human beings, particularly in the last five years, um it in certain circumstances, it becomes clear that facts and truth and data are completely irrelevant to their belief when it conflicts with their beliefs.

And how do you see this impacting the research field? And what, if anything, do you see as a uh either an antidote or a way for stakeholders to recognize, like, yeah, guess what? This thing that you might have thought was irrelevant, like I don't know, redesigning your cracker barrel logo, um, is in fact going to uh have no relationship to reality, no relationship to truth, but people are going to believe it anyway, and it's going to have a material impact on your business. Cool. I love this conversation.

Um let's talk about what we're talking about, though. Are are you specifically talking about politics if you're talking about the the, or are we talking about just um generically? Because we can go down, I can go down either way, and I'm ready to. I just want to be sure where if we're if we're it's hard to say that it's hard to delineate because it seems like uh like an S T D politics seems to be infecting everything. Yeah.

The so I I guess I would want to talk and and and question your first statement, which is like uh I'm I'm paraphrasing, but market research is about is about facts and and people don't necessarily care about facts. I think what you're actually saying with that market research about that market research piece, and this is someone who has been has fought against kind of market researchers being market researchers, right? I understand there's there's there's there's like how how can we loosen up so we can be more relevant to how the information's used, which is our is just something that it's taken me a while to get to. Um but I think we're saying that those those those we're we're we're kind of like wiggling back in, wigging back in to like a quantitative kind of consistency versus listening to what's true from people and strategizing from there. And so it's like then it becomes what is truth, right?

If if we can understand what the the that irrationality, right, that that we all have, but we judge others for, right? Especially on the macro level right now, if we can understand, which which hooks does do that, it uncovers what is that thinking, and now we can figure out what to do about it, right? Versus solving a problem with the the history and the ways I was listening to a podcast yesterday, I think it was with Joy Reid and Karis Wisher, who were saying we, you know, Joy Reeb is saying we can't take the historical political policy and the way things have all that have always been solved and apply it to today. But what if we listened in a different way, in a truly open way to figure out what the thinking was, to then figure out what the strategy and path forward could be because I have to still believe that there's a there's a there's a strategy forward that that brings us back from the edge. Well, uh I am less optimistic than you are, but I appreciate it.

I don't know exactly. Um where can people find more uh about more about you? Um I'd say come to Hutology. We're at www.hootology.com. Um just launched um hooks, which is uh that platform we've been talking about, about new ways to quantify qualitative information to make some better decisions.

All right. Thank you very much. And we will catch you all on the next one.