What is anecdotal evidence?

Rich asked:

On another note, what is the value of very large quantities of anecdotal research/surveying? Does the large quantity redeem its unreliability and inexactitude? No cherry picking of course.

eg. If 10 people you ask, 9 say that X cured them of disease Y = unreliable anecdotal
vs asking 10,000,000 people, 9,000,000 say X cured them of disease Y = reliable anecdotal evidence?

There’s an unasked but important question in here – what is anecdotal evidence? After all, a super-large sample size can’t possibly be anecdotal, can it? The answer is yes, it can. Here’s why.

Anecdotal evidence is defined as evidence collected unscientifically. It is data that can be riddled with multiple forms of bias. The population surveyed anecdotally may not be representative of the population you’re ultimately trying to get answers for. Most importantly, anecdotal evidence can be so distorted that you can’t even prove correlation, and without correlation, proving causation is impossible. (all correlation is not causation, but all causation also has correlation)

To Rich’s question, if you substitute water or oxygen in the air we breathe as X in his questions, would that be valid evidence? No, of course not.

Here’s another example of how anecdotal evidence, even at large scales, can be flawed. Imagine you’re tackling a hot button political issue. 9,000,000 people say that this issue is important to them. That’s a lot of people? Unless you know how the data was collected, you might have a case of severe selection bias, as the people who care about the issue would respond to a voluntary survey. You’ve got a lot of anecdotes, but no objective data – even if you can report in the media that 9,000,000 people can’t possibly be wrong about this issue.

That said, there is a place for anecdotal evidence, and that place is in the formation of the hypothesis for rigorous testing. Anecdotes are great places to start creating more questions to find scientifically-proven answers to. Suppose in the example above, 9,000,000 people swore that chocolate-covered jalapeños were what cured them of their disease. That anecdote is a place to start research. What chemical compounds exist in chocolate-covered jalapeños? Can we test those compounds in a reliable, repeatable fashion in a laboratory? Can we study the chemical interactions with disease Y and see how the food might interact with it? Absolutely – and this is the value of anecdotal evidence. It spurs questions that we can then prove or disprove objectively.

Thanks for the question, Rich! I think it might be worthwhile to do a multi-part blog post series on the different ways we can screw up data. What do you think? Leave your answer in the comments.

You might also enjoy:

• Almost Timely News: Principles-Based Prompt Engineering (2024-02-25)
• Fireside Chat: Geraldine Deruiter on Food, Feminism, and Fury
• Mind Readings: You Need Passwords for Life in the Age of Generative AI Fraud
• You Ask, I Answer: Retrieval Augmented Generation for Tax Law?
• Almost Timely News, January 7, 2024: Should You Buy a Custom GPT?

Take my Generative AI for Marketers course!

Join my Analytics for Marketers Slack Group!