Why Marketing Tests Feel Terrible (And Why That's Fine)

I spend a lot of time on this blog telling you to test your marketing. Measure to make sure it works. Follow the data. Run an experiment. That sort of thing.

I believe every word of it. And I’ll keep saying it.

But I feel obligated to tell you something I usually don’t. Namely, empirical market testing sometimes sucks.

Nobody warns you about that. Testing evangelists, myself among their number, make testing sound clean and satisfying. So you walk away with the idea that you’ll run a tidy experiment and get a tidy answer. Then you’ll stride confidently toward profit.

I wish. Don’t get me wrong: testing is the surest path I know to success.

But a lot of the time, the test will either fail or, even more often, it’ll come back inconclusive.

So you’ll be left deciding what to do next. And much of the time, that means killing work you were proud of.

Also, you can’t always tell why ideas don’t work. The data can tell you they don’t, and that’s a very important thing to know. But charts and metrics can’t necessarily tell you why your idea went wrong.

But I’ll tell you one thing I believe fully. This discomfort is the price of knowing instead of pretending to know. When testing feels good, you should probably be a little suspicious.

So just in case you needed it, here’s a permission slip. You’re allowed to hate testing and feel bad about it even as you do it. You’re allowed to resent your dashboards. And that doesn’t mean you’re doing anything wrong—and I’ll explain why I say that in five points.

You wrote a headline you loved. The whole team loved it too. It was clever, it had a little music to it, and you were sure it would win.

Then you tested it against some boring, workmanlike alternative, and the boring one won. And it wasn’t close.

Now if you’ve done marketing for a while, you might have heard someone trot out that old Jay Abraham quote that: “sometimes the best copy to sell a horse is ‘Horse for Sale.’”

And yeah, that’s true. Of course, it’s true. It’s just not any fun.

This happens to marketers every day, and even if you’ve been in the business for a while, it still smarts. There's a famous piece of writing advice for exactly this feeling: "murder your darlings." Most people credit it to William Faulkner. They're wrong—it comes from Arthur Quiller-Couch, in a 1914 Cambridge lecture—but the point doesn’t change regardless of who said it. The writing you're proudest of might not be the writing that best serves your reader.

The same thing goes in marketing. Your favorite ad is not automatically your best ad. Frequently it's your favorite because it flatters your taste, which is a different job than moving a prospect to act.

This is why I’m a fan of running ten or more headlines and seeing what happens. Doing this also makes it easier to separate your ego from your ideas. Because all you have to do then is run one good headline out of ten and you’re a winner.

You’re not your ad. You’re not your copy. You’re not your offer. You’re the person testing those things and seeing what happens.

It’s easy to think about Newton getting bonked on the head by an apple and suddenly inventing physics. And when running a test, I always hope I get to be a bit like Archimedes, getting out of the bathtub, solving some thorny problem and running down the streets yelling “eureka!”

But here in the real world, most tests produce: nothing much.

Seriously, I have data to prove how hard data is to get. Most tests don’t produce clear winners. Across 28,304 experiments analyzed by Convert, only about 20% reached 95% statistical significance. Data from VWO and Optimizely lands in the same neighborhood showing that roughly 22% of A/B tests produce a significant winner. Another audit of e-commerce tests found over 40% flatly inconclusive.

And yes: all three of these stats come from companies that sell testing tools. But I can also personally vouch for these numbers as being “about right.”

That means you'll spend real money, wait real weeks, and a solid majority of the time the answer will be some version of "we can't tell" or "those two are basically the same." That can take the wind out of your sails. Case studies don’t talk about that.

But you should also know: inconclusive is not the same as wasted. A test that says "this change does nothing" just stopped you from rolling a nothing-change out to your whole audience and calling it strategy. It also narrows the search for the something-change. "Not that" is real information. It's just not the kind of information that feels like a win.

Statistical significance is, to put it bluntly, a rich person's game. It wants traffic. Lots of it. To get to the magical 95% confidence level, you need the kind of volume that lets you do that in a matter of weeks or months.

But for those of us without private jets: you're left making decisions on directional, ambiguous data, and it feels deeply irresponsible. Except it isn't irresponsible. It's the actual job at your scale.

Better than chasing certainty you can’t afford is to commit to thresholds in advance. I call these tripwire criteria: the pre-set lines that say "if we hit this number, we scale, and if we hit that one, we stop." You decide them while you're calm and thinking clearly, because you will not be calm and thinking clearly once the money's in and your judgment is tangled up in wanting the bet to pay off.

Oh, here’s a fun one: your marketing campaign dies. Why?

Offer? Creative? Targeting? Landing page? Price? Seasonality? Some unholy combination of all of these and another six or seven things you didn’t even think about?

Tests tell you what happened. They seldom tell you why.

This problem is so tricky to solve that philosophers have a name for it: the Quine-Duhem thesis. You don’t need to be up on your philosophy of science canon to get the idea, though. When you test a whole campaign, you're testing a stack of assumptions all at once, and when the stack falls over, the rubble doesn't come labeled.

The ambiguity that comes from this can be dangerous. It's an open invitation to believe whatever you already wanted to believe. The channel underperformed? Must be the channel, not the thing I’m selling itself. No, sir.

But as Richard Feynman said: “you must not fool yourself—and you are the easiest person to fool.” And to that end, you have one defense and it’s as effective as it is boring. Isolate variables, change one thing at a time, and refuse to accept the flattering explanation just because it's flattering.

Empirical marketing is not a slot machine. There's no lever or lights, and there’s no instant payout. You launch, and then you wait. That means waiting weeks for a paid test to gather enough data, and six to nine months for SEO to say much of anything. So much for dopamine.

And then there's the worst outcome of all (at least emotionally): the test works perfectly and tells you the offer is the problem. Not the copy. Not the targeting. The thing itself. The market doesn't want this, at this price, right now.

That's expensive to hear and humbling to accept. Sometimes it means going back to the drawing board. I've told clients to stop spending entirely because our small tests were clear that the offer wasn't landing, and that no amount of clever media was going to paper over a mismatch between what they were selling and what people wanted to buy.

No one wants to hear that. But good marketing has a way of humbling its practitioners, and pretending otherwise just makes the lesson more expensive when it finally shows up.

If you kind of skim this post, you might think: he’s telling me I’m bad at my job. And if that’s what you’re thinking, I’d like to gently reframe it.

Failed tests are not failed marketing.

This is what science feels like from the inside. And I mean that literally, as in with professionals with PhDs and lab budgets and careers. Those folks spend most of their time getting null and ambiguous results. We know this because even though what we see publicly is that between 80% and 95% of published papers report positive findings, all the other findings just go unpublished. Researchers call it the "file drawer problem."

The demoralizing part of empiricism is so real that an entire reward system grew up around avoiding it. It’s called “publication bias” and it might even be part of why science—as a broad concept—has had to fight for its reputation as of late.

When you dig into the replication crisis, what you’ll find isn’t that science is broken. It's that these failures "reflect the normal operation of science—a mechanism by which unsupported hypotheses are eliminated."

If you read that again as a marketer, it translates well. Null results are not failures. They’re successes. They’re supposed to happen.

So when a test comes back inconclusive or ugly and you feel that little drop in your stomach, that feeling is not evidence you're incompetent. It's evidence you ran a real experiment instead of a piece of theater designed to tell you what you wanted to hear, but with numbers.

I don't want to leave you with a pep talk and no plan. A few things that I find helpful:

Separate the feeling from the decision. You're allowed to be disappointed the darling lost. But be extremely careful about overruling the number because of it. Trust the data over your feelings.

Pre-commit your thresholds. Decide what you'll do at each outcome before you have skin in the game. Systems beat willpower, and you don’t want to make hard calls when emotional.

Detach your ego from your ideas. The goal is to be wrong fast and cheap, not slow and expensive. An idea that dies in a $500 test is a bargain next to one that dies in a $50,000 launch.

Expect "meh," and budget for it. Most tests won't resolve cleanly. Plan for a pile of inconclusive results, and learn to value the ones that just say "don't ship this."

Remember the alternative. Not testing feels wonderful. Then it fails, and often quite hard. Vanity metrics feel good precisely because they aren't telling you the truth. If your marketing never makes you uncomfortable, that's not a sign you've transcended the struggle. It's a sign you're not looking hard enough.

Testing won’t always feel good. But the success that comes with sticking to it does.

Being uncomfortable is a damn good moat, after all. Most people hate that feeling. They’ll try to get away from it. So if you can tolerate the ambiguity, the boredom, and the slings and arrows, you’ll probably outperform the ones who can’t.

If you can watch your favorite ad lose and run the boring winner anyway, you’ll end up knowing things others don’t. That’s your competitive edge.

Market testing sucks sometimes. Do it anyway. You’ll be glad you did.

My company helps B2B service businesses generate qualified leads through data-driven SEO. We do the work and we build the tracking to show you what's producing results.

If you're interested, book 30 minutes of my time and we can talk about whether it makes sense for your business.