Rich asked:
On another note, what is the value of very large quantities of anecdotal research/surveying? Does the large quantity redeem its unreliability and inexactitude? No cherry picking of course.
eg. If 10 people you ask, 9 say that X cured them of disease Y = unreliable anecdotal
vs asking 10,000,000 people, 9,000,000 say X cured them of disease Y = reliable anecdotal evidence?
There’s an unasked but important question in here – what is anecdotal evidence? After all, a super-large sample size can’t possibly be anecdotal, can it? The answer is yes, it can. Here’s why.
Anecdotal evidence is defined as evidence collected unscientifically. It is data that can be riddled with multiple forms of bias. The population surveyed anecdotally may not be representative of the population you’re ultimately trying to get answers for. Most importantly, anecdotal evidence can be so distorted that you can’t even prove correlation, and without correlation, proving causation is impossible. (all correlation is not causation, but all causation also has correlation)
To Rich’s question, if you substitute water or oxygen in the air we breathe as X in his questions, would that be valid evidence? No, of course not.
Here’s another example of how anecdotal evidence, even at large scales, can be flawed. Imagine you’re tackling a hot button political issue. 9,000,000 people say that this issue is important to them. That’s a lot of people? Unless you know how the data was collected, you might have a case of severe selection bias, as the people who care about the issue would respond to a voluntary survey. You’ve got a lot of anecdotes, but no objective data – even if you can report in the media that 9,000,000 people can’t possibly be wrong about this issue.
That said, there is a place for anecdotal evidence, and that place is in the formation of the hypothesis for rigorous testing. Anecdotes are great places to start creating more questions to find scientifically-proven answers to. Suppose in the example above, 9,000,000 people swore that chocolate-covered jalapeños were what cured them of their disease. That anecdote is a place to start research. What chemical compounds exist in chocolate-covered jalapeños? Can we test those compounds in a reliable, repeatable fashion in a laboratory? Can we study the chemical interactions with disease Y and see how the food might interact with it? Absolutely – and this is the value of anecdotal evidence. It spurs questions that we can then prove or disprove objectively.
Thanks for the question, Rich! I think it might be worthwhile to do a multi-part blog post series on the different ways we can screw up data. What do you think? Leave your answer in the comments.
You might also enjoy:
- Almost Timely News, 17 October 2021: Content Creation Hacks, Vanity Metrics, NFTs
- Retiring Old Email Marketing Strategies
- You Ask, I Answer: Google Tag Manager and Google Analytics Integration?
- The Biggest Mistake in Marketing Data
- Marketing Data Science: Introduction to Data Blending
Want to read more like this from Christopher Penn? Get updates here:
 Get your copy of AI For Marketers |
This is exactly why scientific research is so important- can we isolate variables and separate people into reliable and compliant treatment arms so we know for sure whether the treatment was the cause for improvement or whether confounding factors were at play?
Some of my husband’s research is actually easier to do in India, because they have a system of local public health nurses that can go to people’s house every day and ensure they take medication as prescribed- something that would be almost impossible to do in the US.
Isolating variables and tracking results is critical. It’s not always easy to do. I hope you do do a full series on this- I’ve had to get my kids to listen to the BBC’s More or Less podcast to help them learn numerical literacy.