About a third of well-designed experiments lose. Another share come back flat. If a program only produces value from the winners, two thirds of the spend is waste, and the pressure to declare victories quietly corrupts the whole practice.

The fix is not optimism. It is designing tests whose failure is informative before you ship them.

Separate the four kinds of loss

  • The mechanism was wrong: the doubt you targeted was not the doubt holding people back.
  • The mechanism was right, the execution was not: the idea never actually reached the customer clearly.
  • The audience was wrong: the change helped one segment and hurt another, netting to nothing.
  • The test was wrong: instrumentation, traffic, or duration made the read meaningless.

These four demand different next moves, and only the first is real evidence about your customer. Most teams collapse all four into 'it did not work', throw away the hypothesis, and lose the one asset the test produced.

A loss is only expensive when it fails to narrow the next question.

Write the loss down before you run it

In the test brief, next to the hypothesis, we write what a negative result would prove. If the answer is 'nothing much', the test is not worth a build cycle, and that decision costs nothing to make in advance. This single habit removes more waste from a program than any tooling change.

Read the segments, then stop

Segmented analysis is where honest programs go to die. Cut a flat result finely enough and something will look significant. Decide the two or three segments that matter before the test starts, look only at those, and treat anything else you notice as a hypothesis for a future test rather than a finding.

Losses that changed a roadmap

A stronger guarantee that did not move conversion told us risk was not the blocker, and pushed the work toward the sizing question the reviews had been shouting about. A simplified checkout that lost told us the removed field was carrying reassurance, not friction. In both cases the loss redirected budget faster than a marginal win would have.

Report losses in full detail

Negative reads get one-line summaries in most decks. That asymmetry teaches the organisation that only wins deserve attention, which is exactly how teams start retesting the same failed idea in a new colour eighteen months later. Write the loss up like a win: what we believed, what we changed, what the customer did, what is now less likely to be true.

A program that compounds is not one with a high win rate. It is one where nothing has to be learned twice.