Variant B is clearly losing, can I kill the test before the thresholds clear?
Client experiment I am consulting on, 9 days in, variant B is converting well below control and the gap has not narrowed once. Every day it runs the client loses real revenue on the B traffic.
I know the tradeoff. My old client-side tool let me peek at p-values daily and I called winners that later reversed, so I am not asking for permission to peek. I am asking what the defensible move is under Croct's Bayesian stats. Do I have to let it run until every threshold clears, or is there a metric that tells me the cost of stopping now is acceptable
2 answers
Potential loss is the metric built for exactly this question. It estimates the cost of making the wrong call, so instead of asking "is B significantly worse" you ask "if I stop now and B was secretly the winner, how much would that mistake cost". Croct's recommended-winner bar combines both sides: Probability to Be Best above 95% AND potential loss below 0.1%.
For a clear loser, look at it from B's perspective. If B's probability to be best is near zero and control's potential loss is tiny, the expected cost of stopping is negligible and stopping becomes a business decision, not a statistical sin. The dashboard supports it, you can pause the experiment and roll out the winner without any code changes.
Two cautions. First, early data exaggerates effects. The per-variant minimums (1000 visitors, 25 conversions, 1 week) exist because day-2 leads routinely evaporate, so make sure your 9 days of data is not just a thin sample dressed up as a trend. Second, whatever you do, do not reallocate traffic mid-flight to "reduce the bleeding". Changing the traffic split during a running experiment invalidates the statistics. Stop it cleanly or let it run.
That reframing helps. Control's potential loss is showing 0.02% so I can put an actual number on the risk when I talk to the client. Stopping it today.
One thing I would add from running a lot of these: distinguish "kill the experiment" from "kill the variant". If the test had three or more variants you could not just drop the loser and keep going, adding or removing variants mid-experiment is discouraged for the same reason as reallocating traffic. With a plain A/B like yours the clean options are exactly two, pause the whole thing and ship control, or let it finish. There is no middle path that keeps the stats valid.