Christopher S. Penn – Marketing AI Keynote Speaker

Why we’re doing A/B testing wrong according to Tom Webster

—

by

Christopher S Penn
in , , ,
Warning: this content is older than 365 days. It may be out of date and no longer relevant.

IMG_1038

The most powerful revelation from the Digital Marketing Summit for me came from master data storyteller Tom Webster, who effectively wrecked a lot of people’s perceptions of A/B testing in his talk (those who were paying attention, anyway). One of the most powerful ideas he delivered was that A/B testing in its current form is broken. It automatically discards the opinion, preferences, and inclinations of a significant minority of the audience in order to maximize the results in the biggest segment of the audience.

When you think about that, he’s right that we are crazy for doing that. If 60% of our test audience likes an email and 40% doesn’t, then when we ship the email to our entire audience, assuming the test audience is a representative sample, we’ve basically told 40% of our audience that their preferences are unimportant to us. Tom Webster’s prescription for fixing this is elegant and yet simple (but not easy): segment out the 40% and figure out why they liked “the losing choice” better. There may be a market opportunity there to make those people much happier with you (and earn more revenue from them) rather than ignore their wishes and marginalize them.

How we’ll do that will be tricky. Certainly, in things as simple as subject lines, if you’re testing to two formats, then segmenting folks into different formats is simple enough. You could easily imagine a “snarky subject line” segment and a “cup of soup marketing” segment. What will be tougher for many of us as marketers is to adapt our content (or automate the adaptations) to conform to those segment’s desires for personalization. Imagine going to a website, having a colleague visit the same website, and having radically different experiences from the start because you fall into two different segments that have different needs.

We see this happening already to some extent; those folks who have LinkedIn Pages for their companies can customize the order in which products are shown based on profile information. C-Level executives or people from large companies can be shown different offerings. On LinkedIn, even status updates can be targeted at specific audience subsets, rather than broadcast generally:

SHIFT Communications: Overview | LinkedIn

But these kinds of customizations are canned generalizations and are just the very tip of the iceberg compared to what Tom Webster was explaining.

This is the future of marketing – being able to give people what they most want that makes them happy, regardless of whether they are in the majority or not. Now we just have to build the tools, technologies, and infrastructure to make that happen. Special thanks to Tom for sharing his wisdom and insights.

You might also enjoy:

Want to read more like this from Christopher Penn? Get updates here:

subscribe to my newsletter here

AI for Marketers Book
Take my Generative AI for Marketers course!

Analytics for Marketers Discussion Group
Join my Analytics for Marketers Slack Group!

Comments

5 responses to “Why we’re doing A/B testing wrong according to Tom Webster”

  1. JonDiPietro Avatar
    JonDiPietro

    Thanks for sharing this, Chris. It is an important reminder. We tend to think of A/B testing as a decision for deploying 100% of resources toward a single option. Instead, in some cases it may make more sense to deploy 60% of those resources in a particular direction, to use your email example.

    As such, instead of calling it “A/B” testing, I think I’m going to referring to “A/b” testing.

    Reply
  2. Tom Webster Avatar
    Tom Webster

    Thanks for the post, my friend. I knew you’d run with this one–and maybe it’ll shake up the whole ESP industry 😉

    Reply
  3. jaybaer Avatar
    jaybaer

    The easier way to get at Tom’s recommendation would be to not do A/B at all, but rather full MVT which, when done correctly, will allow the specific differentiating element to surface mathematically. So instead of testing one variant, you’re testing 8 or 10, and you can hone in on which one of those 8 or 10 is having the impact on recipient behavior. It’s definitely not easy to build a full MVT in email, and you have to have a pretty big list to make the math even conceptually reliable, but it can work.

    Reply
    1. Tom Webster Avatar
      Tom Webster

      Actually, the tool I might use for the job with a modest email list would be a kind of conjoint analysis by proxy–a discrete number of overt trade-offs, rather than a full MVT–which can identify drivers. But again, those drivers are designed to please segments, but not necessarily individuals. Maybe ESP software can move beyond that someday.

      Reply
  4. Chris Faber Avatar
    Chris Faber

    Tom’s theme of using tests as opportunities to learn something about the audience is intriguing and definitely challenges the still prevailing thinking about testing. The more knowledge we have about our audience the better we are able to serve them. But I must be missing some of the ingredients in this process. How have you inferred 60% “liked” one version over the other? Isn’t the available information limited to the “success” of the competing alternatives – open rates, click through and other similar metrics? Also, isn’t it possible some of those that “voted” for the less successful alternative did that despite the limitations of the format being tested? How do we isolate them?

    Thanks for presenting yet another idea to challenge my thinking.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Pin It on Pinterest

Shares
Share This