You Ask, I Answer: Determining Sample Sizes for Surveys?

In today's episode, Phil asks, how do you determine a large enough sample size for things like a survey? I always thought 10% sample would be enough, but you seem to think that's not true. Is not is not true. Here's why. Surveys and statistical validity depend on the size of the population you're surveying.

The smaller the population, the larger sample you're gonna need to deal with outliers and uh discrepancies. And it's tough to explain, you know what? Let's do this. I'm gonna take this I have five colored blocks here, right? Three green three blue, two yellow.

I'm gonna put them in this hat. Now I'm gonna pull one block out of this hat. Remember, three three blue, two yellow. This is a twenty percent sample of a population of five. If I conclude then, based on this sample, that every block in this hat is blue, we know that's not true, right?

There's two yellows and three blues in here. And so from a very small sample sample size, I have to be able to randomly draw, you know, so I pull out two here, yep, still blue, right? If I pull out three here, okay, now we're starting to get somewhere. Now there's there's a yellow in there. Pull out four in our 80% sample, three blue and one yellow, and then a hundred percent sample, five.

So if you have a very small population, one outlier can really ruin the survey size, right? Now, if yes, I do keep blocks and other creative things at my desk. If I have a a box full of these, right, and I start pulling out, you know, a handful, this is probably about 10%, you're gonna see there's because there's so many more blocks, as long as they are properly mixed, when I pull out samples, I can start to see that I'm getting a more representative sample of the population as a whole. Now, if this block box were 300 million bricks, we wouldn't be doing this video because my basement would be full. Uh, but if this if if I had 300 million, I could pull out a thousand of these.

And as again, as long as it was well mixed, I would have a pretty good idea of what the entire sample would look like, or what the entire population would look like based on that sample of a thousand, because there's so many that as long as it's stirred, I'm getting a representation. That's what we're trying to figure out is can we get a group, a cluster that is representative of the whole, that we can extrapolate to the whole. When you have a small group, you can't do that because there's such a much greater chance of of variation of um uh variability that you could end up drawing some really wrong conclusions. Uh, even something as simple as say like uh I'm at a uh a conference and I get speaker reviews back, and there's 500 people in the room, and ten people left reviews, and uh you know, fifteen uh ten people after reviews, five of them said I was a great speaker, five of them said I was a terrible speaker. Is that representative?

No. It's not even close. Because there's a self-selection bias even there. Those ten people felt strongly enough to leave comments, and the other four hundred and ninety people didn't. And there's a very good chance that those four hundred and ninety people felt differently than the ten people who did decide to respond.

So there's a whole bunch of different ways that you have to tackle surveys in particular. I would refer you to there's there's three reading sources I think are great. One is Edison Research and my friend Tom Webster, who uh so go to Edison Research.com and also brand savant.com is a good place to go. Um and then there are uh organizations, the American Association American Association of Public Opinion Researchers, APOR, AAPOR.org, and CASRO, the Coalition of Americans oh gosh, I don't remember what it stands for. Um both of those are great organizations that have detailed best practices about public opinion research and surveys that will give you some really good starting points for understanding how to do surveys well, how to avoid many of the biases and the traps that that you run into.

Uh non-response bias, meaning that the people who don't respond are different than the people who do respond is a big one. If you're doing a survey of uh, say your email newsletter list, and you only send it to people who have opened emails in the past, what about all those people who don't open your emails? Do they feel differently about your brand or your company? You bet they do. So you have to keep in mind all these different things that can go wrong.

Your best bet for doing a sample, determining sample size, is to use one of the many, many sample size calculators out there on the web. Uh SurveyMonkey has one, SurveyGizmo has one, pretty much every surveying company has one. And they're going to ask you for two major numbers. They want to know your confidence level and your confidence interval. Confidence level means that if you repeat a process 100 times, what number of times will you get the same results?

So when I have this five blocks in the hat business, right? How many times, so I repeat this draw a hundred times in a row, how many times am I going to get the same result? That is your confidence level. Most surveys operate at uh 95% confidence level. That's the general best practice.

So if you repeated the survey 100 times, 95 of those times, you get the same result. That's a that is that will help you determine the sample size. How large of the population do you need to survey in order to get that reliability of 95 times out of 100, you get the same results in your survey. The second is confidence interval or margin of error. Um this is how granular do you need the results to be in order to be able to judge that's accurate.

So let's say there's a yes or no question, right? And 49% of people said no, and 51% of people said yes. If you have a margin of error of 3%, meaning any answer could go either way, plus or minus 3%, then if 49% of people said no, and 51% of people said yes, there's a large enough margin of error there that you can't tell which answer is correct, right? Because the 49% could be as low as 46%, could be as high as 52%, and the 51% could be as low as 48%, as high as 54%. And they overlap, and that means that your confidence interval is too wide.

The catch is the narrower you make the confidence interval, the larger your sample has to be in order to have it be representative. The same is true of confidence level. The the higher your confidence level, 90%, 95%, 99%, the larger your sample has to be. If you incur a cost of you know for sending out a survey, then you have to make that balance between how much do I want to spend and how accurate do I need my survey to be? And it's it is a it is a balancing game uh to make that determination, especially if you ever want to ask questions we have to drill down to a subset of your population, then it's gonna get really expensive.

So keep that in mind. These are good questions to ask before you do a survey because they dic dictate this the uh type of survey you're gonna do, they dictate the cost of it, they dictate what you can and can't do with the information. So it's a really good question. Again, use many of the calculators, but spend some time learning about surveys, in particular the biases that go into them, because that is what will ruin them more than anything else is it's having you know, doing a survey and saying it's representative and then it not be. Because if you make a decision based on a a sample that's too small and therefore skewed, you could really throw off every decision you make from that.

Like, you know, do you spend money on this? Is this a focus of yours? Is this something that people care about? If the answers are skewed because you you didn't get a good enough sample, you could spend a lot of time and money, waste a lot of time and money on something that's not going to work. So get the surveying basics down first before you run the survey.

Because the other thing that's tricky about surveys is there's no going back. There's no rewinding, you can't fix a data uh the survey data after you've done it. Great question to leave your follow the questions here uh in the comments box. Subscribe to the YouTube channel on the newsletter. I'll talk to you soon.

Want help solving your company's data analytics and digital marketing problems? Visit trustinsights.ai today and let us know how we can help you.