Warning: this content is older than 365 days. It may be out of date and no longer relevant.


The most powerful revelation from the Digital Marketing Summit for me came from master data storyteller Tom Webster, who effectively wrecked a lot of people’s perceptions of A/B testing in his talk (those who were paying attention, anyway). One of the most powerful ideas he delivered was that A/B testing in its current form is broken. It automatically discards the opinion, preferences, and inclinations of a significant minority of the audience in order to maximize the results in the biggest segment of the audience.

When you think about that, he’s right that we are crazy for doing that. If 60% of our test audience likes an email and 40% doesn’t, then when we ship the email to our entire audience, assuming the test audience is a representative sample, we’ve basically told 40% of our audience that their preferences are unimportant to us. Tom Webster’s prescription for fixing this is elegant and yet simple (but not easy): segment out the 40% and figure out why they liked “the losing choice” better. There may be a market opportunity there to make those people much happier with you (and earn more revenue from them) rather than ignore their wishes and marginalize them.

How we’ll do that will be tricky. Certainly, in things as simple as subject lines, if you’re testing to two formats, then segmenting folks into different formats is simple enough. You could easily imagine a “snarky subject line” segment and a “cup of soup marketing” segment. What will be tougher for many of us as marketers is to adapt our content (or automate the adaptations) to conform to those segment’s desires for personalization. Imagine going to a website, having a colleague visit the same website, and having radically different experiences from the start because you fall into two different segments that have different needs.

We see this happening already to some extent; those folks who have LinkedIn Pages for their companies can customize the order in which products are shown based on profile information. C-Level executives or people from large companies can be shown different offerings. On LinkedIn, even status updates can be targeted at specific audience subsets, rather than broadcast generally:

SHIFT Communications: Overview | LinkedIn

But these kinds of customizations are canned generalizations and are just the very tip of the iceberg compared to what Tom Webster was explaining.

This is the future of marketing – being able to give people what they most want that makes them happy, regardless of whether they are in the majority or not. Now we just have to build the tools, technologies, and infrastructure to make that happen. Special thanks to Tom for sharing his wisdom and insights.

You might also enjoy:

Want to read more like this from Christopher Penn? Get updates here:

subscribe to my newsletter here

AI for Marketers Book
Get your copy of AI For Marketers

Analytics for Marketers Discussion Group
Join my Analytics for Marketers Slack Group!