What Measurement Can't Do (And Why I Measure Anyway)

I am, by reputation and by temperament, the measurement guy.

I build multi-touch attribution. I score leads with an API. I put UTM parameters on links I'm fairly sure no one will ever click. When a client tells me a campaign is "doing great," my first question is nearly always some variant of “OK, but how do you know?”

My instinct is Popperian. If a claim can't be tested, and can't even in principle be proven wrong, I get mighty suspicious.

So it should mean something when I tell you: the longer I do this work, the more respect I have for what measurement can't do.

There's a line I keep coming back to. You can usually find it on a corny-looking image made for a bygone era on Instagram, often wrongly attributed to Einstein, but in actuality from a sociologist named William Bruce Cameron: "Not everything that can be counted counts, and not everything that counts can be counted."

That sentence is the whole problem with marketing, compressed. Your analytics dashboard is a map. It is not the territory. And the difference between a good marketer and a dangerous one is whether they know that.

Here, I want to tell you about four things measurement can't do. Because knowing them has made me better at measurement, not worse.

When a campaign works, you almost never know what worked.

Think about what changes between a bad month and a good one. Say revenue jumps the month you launch new ad creative. But you also changed the offer. And a competitor ran out of stock. And your best salesperson finally hit her stride. And probably seasonality is playing some kind of role too, but good luck figuring out what. Any one of those could be the reason, or all of them, in proportions you will never recover.

This isn't a tooling problem. You can’t fix it with tools. It’s an older, deeper problem than that. Philosophers of science call it the Duhem-Quine problem: you can't test a single idea in isolation, because every test secretly leans on a stack of other assumptions. When the result comes back, you can't be sure which assumption it was really testing.

In a lab, you control for that with a clean experiment. In marketing, you almost never get one. You run a hundred experiments at once and call it a campaign.

Someone on r/SEO put it better than any textbook. Early on, he'd change titles, content, and links all in the same week, and then he "had no idea what actually helped."

Attribution isn't broken because the software is bad. It's hard because reality wriggles around like a toddler that doesn’t want to be carried (a fact I discuss at length in my post on the absolute state of conversion attribution).

The things easiest to count are usually the least important. The things that matter most are the hardest to count. So guess which ones get the budget.

This has a name: the McNamara fallacy, after the defense secretary who ran the Vietnam War on enemy body counts, because body counts were a number he could put in a chart. The marketing researcher Daniel Yankelovich described the path you go down when you make a mistake. His quote in full:

“The first step is to measure whatever can be easily measured. This is okay as far as it goes. The second step is to disregard that which can't be easily measured or give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can't be measured easily really isn't very important. This is blindness. The fourth step is to say that what can't be easily measured really doesn't exist. This is suicide.”

The data-driven crowd lives this and mostly doesn't notice. Les Binet and Peter Field spent decades inside the IPA's effectiveness database and found something that ought to make you real uncomfortable. In short: the so-called measurability of digital is a trap.

Don’t get me wrong, short-term clicks are easy to track. But long-term brand growth is not, so money flows to the measurable and starves the thing that drives most of the growth. The most efficient thing to measure is rarely the most effective thing to do.

Or, as Rory Sutherland, vice chairman of Ogilvy, likes to say, "the human mind does not run on logic any more than a horse runs on petrol." The moves that shift real people are often the ones a spreadsheet would have killed in the meeting.

You can measure only what's easy, sure. But you’ll also optimize yourself to death.

For more on watching the things that resist measurement: how to know your marketing is working before revenue shows up.

This one grinds my gears. The act of measuring something can ruin it as a measurement.

A metric is a proxy. It stands in for something you care about but can't see directly. Clicks stand in for interest. Reviews stand in for quality. Rankings stand in for demand. The trouble starts the moment you start aiming at the proxy instead of the thing behind it.

This is Goodhart's Law, in the anthropologist Marilyn Strathern's tidy phrasing: "When a measure becomes a target, it ceases to be a good measure."

There’s a decent chance you’ve seen this happening in your marketing too. Chase impressions and you buy junk traffic that converts on nothing. Chase engagement and you breed outrage bait. Chase five-star reviews and someone starts handing out discounts for them. Chase domain rating and you wind up paying for a number to brag about while the phone stays quiet.

The better your team is, the faster this happens, because capable people are very good at hitting whatever target you set, even when everything is made up and the points don’t matter. Which is the whole reason I'm so picky about which SEO metrics matter.

Numbers don't talk. Numbers are just numbers. Someone has to walk up and decide what they mean, and that someone is never neutral.

Let me ask you this: Who picks which metrics land on the dashboard? Who decides what counts as "success"? Who narrates the monthly report?

Each of those is a choice, and meaning doesn't rise off the data on its own. Somebody assigns the meaning. Whoever builds the dashboard quietly frames the whole conversation, which is a kind of power most people never think to question.

And make no mistake: people respond to incentives. You can distill a good portion of Charlie Munger’s philosophy to one line. "Show me the incentive and I will show you the outcome."

This is why I’m suspicious of agencies that are paid to look busy. It’s why I try very hard not to be the guy running the agency with a client who is paying him to look busy. The temptation to show numbers that look like action is enormous when you’ve got invoices outstanding in QuickBooks.

Even clean numbers don't speak for themselves. As one marketer told a guy on Reddit who was panicking over a campaign with zero leads, there's "no way of knowing from raw numbers alone" without seeing the offer and the page behind them.

So the question is never just "what does the data say." It's "who decided what to measure, and what do they get when the number goes up?" Often the answer is hiding in whether the leads were any good to begin with.

You might expect this to be the part where I tell you to trust your gut and stop tracking things.

Definitely not.

Gut feel is worse. The gut is confident, unfalsifiable, and remembers only its wins. Measurement is still the best discipline we have for not fooling ourselves, and that’s the first rule after all, right?

The point isn't to stop measuring. It's to measure without worshipping the dashboard, because that’s data idolatry.

Measurement is a flashlight, and if you point it at something, it can show you that thing in sharp detail. But it tells you nothing about the rest of the dark room, and the worst mistake you can make is to assume the room is empty everywhere you didn't aim the beam.

So you measure relentlessly, and then you bring judgment, and craft, and a healthy suspicion of your own incentives, to everything the numbers can't reach. Rigor narrows the uncertainty. It does not erase it. Pretending it does is its own kind of malpractice, the data-driven version of believing your own press.

Not everything that counts can be counted. The best marketers I know hold both halves of Cameron's sentence at once: they measure more rigorously than anyone, and they stay humble about the parts of the picture no instrument will ever pick up.

So measure everything you can. Interpret it carefully. Build the dashboard, use the dashboard, defend the dashboard.

Just don’t tell yourself that data alone is truth.

My company helps B2B service businesses generate qualified leads through data-driven SEO. We do the work and we build the tracking to show you what's producing results.

If you're interested, book 30 minutes of my time and we can talk about whether it makes sense for your business.