# Variant B is clearly losing, can I kill the test before the thresholds clear?

Asked by Fabien Leroux on 2025-09-30. Tags: ab-testing, methodology, bayesian.

Client experiment I am consulting on, 9 days in, variant B is converting well
below control and the gap has not narrowed once. Every day it runs the client
loses real revenue on the B traffic.

I know the tradeoff. My old client-side tool let me peek at p-values daily and I
called winners that later reversed, so I am not asking for permission to peek. I
am asking what the defensible move is under Croct's Bayesian stats. Do I have to
let it run until every threshold clears, or is there a metric that tells me the
cost of stopping now is acceptable

## 2 answers

### Accepted answer from Henrik Dahl (2025-09-30)

Potential loss is the metric built for exactly this question. It estimates the
cost of making the wrong call, so instead of asking "is B significantly worse"
you ask "if I stop now and B was secretly the winner, how much would that
mistake cost". Croct's recommended-winner bar combines both sides: Probability
to Be Best above 95% AND potential loss below 0.1%.

For a clear loser, look at it from B's perspective. If B's probability to be
best is near zero and control's potential loss is tiny, the expected cost of
stopping is negligible and stopping becomes a business decision, not a
statistical sin. The dashboard supports it, you can pause the experiment and
roll out the winner without any code changes.

Two cautions. First, early data exaggerates effects. The per-variant minimums
(1000 visitors, 25 conversions, 1 week) exist because day-2 leads routinely
evaporate, so make sure your 9 days of data is not just a thin sample dressed
up as a trend. Second, whatever you do, do not reallocate traffic mid-flight
to "reduce the bleeding". Changing the traffic split during a running
experiment invalidates the statistics. Stop it cleanly or let it run.

#### Reply from Fabien Leroux (2025-10-01)

That reframing helps. Control's potential loss is showing 0.02% so I can put
an actual number on the risk when I talk to the client. Stopping it today.

### Answer from marta_k (2025-10-01)

One thing I would add from running a lot of these: distinguish "kill the
experiment" from "kill the variant". If the test had three or more variants
you could not just drop the loser and keep going, adding or removing variants
mid-experiment is discouraged for the same reason as reallocating traffic.
With a plain A/B like yours the clean options are exactly two, pause the whole
thing and ship control, or let it finish. There is no middle path that keeps
the stats valid.