You Ask, I Answer: Sample Sizes for Web Analytics?

In today's episode, Courtney asks, how many web sessions or users would you need to send to a website for the results to be significant? It's interesting. At first I thought this question was about A B testing, which is most of the time where we start getting into discussions about statistical significance when it comes to your analytics, but uh they're actually asking about how many visitors you need for your web analytics to be useful in general. And there's a a a few different answers for this. First, if you're just getting started or you're just trying to get a sense of your website's performance, then it's really only a question of time.

You need most of the time uh a minimum of 30 days of traffic data to collect anything useful with web analytics. More you have the better. Um 90 days is better. Years is obviously better because you can do year over year comparisons. That's especially true if your um uh business has uh a high level of seasonality to it, of cyclicality to it.

So that time part is is probably the most important part. When it comes to measuring your website, the number one thing to measure is improvement against yourself, right? There's that's really the uh the best way to judge. Are you getting better results? Are you getting you know even one percent more conversion this month than you were getting last month?

Are you getting one percent more traffic from you know qualified sources this month than last month? That kind of measurement is is the best because it takes away all the external factors and focuses only on what you're doing to grow your website. Now, if we want to get fancy, um way to think about your web analytics is how much of your obtainable market are you getting? This is the market that you could be getting of people in your space. So your website should be attracting more of your obtainable market as more time goes on, as your marketing gets better, as your search engine optimization gets better, and so on and so forth.

So, what is your obtainable market for a website for most websites? This is going to be um a tiny piece, a tiny tiny little sliver, a wafer thin, of what your uh audience is searching for in your uh SEO topics. So, say digital marketing analytics, how many people search for that? Um it's like 10 million people, right? What's the likelihood that you can obtain any of that through SEO?

Assuming you're really good at it, you're still talking like probably like a tenth of one percent. Uh so it's a tiny little slice, so it's gonna be a very, very thin slice of that obtainable market. Once you've figured out what is obtainable, what's what's you're likely to be able to attract, um I use I I typically use a benchmark of uh cost per click for that. Um if you're uh if you're a company and and you take all your major topics and you add up the cost per click and you add up uh the number of clicks, and you can get this out of almost any SEO tool, and you will see it's costs 27 million dollars a month to to capture 100% of the traffic. Great.

What percentage of budget are you willing to throw into that? Right? If you got like if you're saying, well, I got a hundred bucks a month I could spend on on paid search. Oh, great. So to add that 27 million, your your hundred bucks is is the the percentage you're you're committed to obtaining.

Now, I'm not saying go out and spend that. I'm saying that's a good way to just kind of get a a feel, a sense, a benchmark of what percentage of the obtainable market you're likely to get, because that's how much you're willing to commit in terms of resources. Do add in the time spent as money uh for organic SEO for content marketing, because that that is an important part of trying to obtain that audience. So that's part one. Part two is then you need to calculate a sample size, right?

What sample size, assuming you can get a representative portion of that overall audience that's searching for the things that you do, uh what sample size are you comfortable with that would tell you, yes, I'm getting enough of the audience that I can uh I should be getting, you know, uh uh a a good idea of what people are interested in for the purpose of analyzing that audience in your web analytics. Um most marketers, most of the time, uh, because we're not doing anything life-saving, right? This is not the ER, um, are comfortable with uh a confidence level of 95% and a confidence interval of plus or minus three percent. What that means is confidence level in statistics is if you repeated this test a hundred times, 95% of the time, if you have a 95% confidence level, you would get the same results, right? If you were doing something with like pharmaceutical trials, you'd want a confidence level like 99% of the time, or 99.9 or 0.999.

Uh, you want to crank that number up so that you get reliable results. For marketing, where no one's gonna die, 95%'s just fine. Confidence interval says that when you repeat that test over and over again, um, your results should be plus or minus, whatever that number is, um, in this every time you repeat it. So a plus or minus three percent says that result should have a margin of error, roughly of of plus or minus three percent. So uh if you're running that test over and over and over again, uh you should not vary much more than that number outside.

The higher you crank up the confidence level, and the lower you crank the confidence interval, the larger a sample size you need. The first time somebody does that statistical test, they'll crank those numbers like to their extremes, and like, oh, you need a hundred percent of your sample or your audience to get a reliable sample. Like, oh, oh that's not helpful. Um plug those numbers plus your obtainable market into the survey sample tool of your choice, and you'll get the required sample size for any kind of survey. So, you know, for say 10,000 people with 95% confidence level and plus or minus three percent confidence interval, you're talking about 964 people, right?

That's in effect what Courtney was asking about. That's again, that's only useful if you're trying to figure out you know how much of the audience do I need to have as traffic on my website to do statistically relevant judgment about them, their age, their demographics, their behaviors, their search terms, and things like that. For the purposes of managing your marketing, you don't need that at all. You don't need that at all. All you need to focus on is your numbers for awareness, consideration, evaluation, and conversion, and are you improving them day after day, week over week, month over month, year over year.

If you're if you are improving, that's all that really matters. And I would say that is true for your Google analytics, your web analytics, your marketing automation, your email marketing, your social media metrics, any metric at all that you're measuring, focus on the improvement. Focus on what can I do to get the improvement. A number of years ago, uh Julian Smith, the CEO of Breather, said when you want to go into growth mode for a startup, uh, the number one thing you can do is pick one relevant metric and come hell or high water, make it grow eight percent week over week. He's like, that's if you want to grow, and you're willing to throw time, you're willing to throw energy, you're willing to throw all your money at this at something to make your business grow.

Pick one thing that's relevant and force that thing to grow eight percent week over week, and it will have uh uh spillover effects into everything else. So that's a uh uh an interesting way to think about it. Now you may not be able to get eight percent a week, but could you get five percent month over month? Something to think about. Great question, Courtney, complicated question.

There's a lot of statistical uh stuff to dig in here, and it really shows that even for something as simple as a uh a simple question, a seemingly simple question about web analytics can have a lot of stats and data science behind it in order to really dig into what it is you're after. So uh good question, complicated question, more complicated than you thought. Uh, as always, please uh leave your comments below, subscribe to the YouTube channel and to the newsletter. I'll talk to you soon. Take care.

Want help solving your company's data, analytics, and digital marketing problems? Visit TrustInsights.ai today and let us know how we can help you.