Rich asked:

On another note, what is the value of very large quantities of anecdotal research/surveying? Does the large quantity redeem its unreliability and inexactitude? No cherry picking of course.

eg. If 10 people you ask, 9 say that X cured them of disease Y = unreliable anecdotal
vs asking 10,000,000 people, 9,000,000 say X cured them of disease Y = reliable anecdotal evidence?

There’s an unasked but important question in here – what is anecdotal evidence? After all, a super-large sample size can’t possibly be anecdotal, can it? The answer is yes, it can. Here’s why.

Anecdotal evidence is defined as evidence collected unscientifically. It is data that can be riddled with multiple forms of bias. The population surveyed anecdotally may not be representative of the population you’re ultimately trying to get answers for. Most importantly, anecdotal evidence can be so distorted that you can’t even prove correlation, and without correlation, proving causation is impossible. (all correlation is not causation, but all causation also has correlation)

To Rich’s question, if you substitute water or oxygen in the air we breathe as X in his questions, would that be valid evidence? No, of course not.

Here’s another example of how anecdotal evidence, even at large scales, can be flawed. Imagine you’re tackling a hot button political issue. 9,000,000 people say that this issue is important to them. That’s a lot of people? Unless you know how the data was collected, you might have a case of severe selection bias, as the people who care about the issue would respond to a voluntary survey. You’ve got a lot of anecdotes, but no objective data – even if you can report in the media that 9,000,000 people can’t possibly be wrong about this issue.

PodCamp Party at the Grand Canal

That said, there is a place for anecdotal evidence, and that place is in the formation of the hypothesis for rigorous testing. Anecdotes are great places to start creating more questions to find scientifically-proven answers to. Suppose in the example above, 9,000,000 people swore that chocolate-covered jalapeños were what cured them of their disease. That anecdote is a place to start research. What chemical compounds exist in chocolate-covered jalapeños? Can we test those compounds in a reliable, repeatable fashion in a laboratory? Can we study the chemical interactions with disease Y and see how the food might interact with it? Absolutely – and this is the value of anecdotal evidence. It spurs questions that we can then prove or disprove objectively.

Thanks for the question, Rich! I think it might be worthwhile to do a multi-part blog post series on the different ways we can screw up data. What do you think? Leave your answer in the comments.

You might also enjoy:

Want to read more like this from Christopher Penn? Get updates here:

subscribe to my newsletter here

AI for Marketers Book
Get your copy of AI For Marketers

Analytics for Marketers Discussion Group
Join my Analytics for Marketers Slack Group!

Subscribe to My Free Weekly Newsletter

Subscribe to My Free Weekly Newsletter

Sign up now to the free Almost Timely Newsletter, released every weekend with the latest news about marketing, technology, analytics, data science, and AI.

  • This field is for validation purposes and should be left unchanged.

You have successfully subscribed to the Almost Timely Newsletter!