My test has run a week and still says "in progress", when does it actually conclude?
My experiment has been live for exactly a week now and the status still reads "in progress". I am trying to work out whether that is expected or whether something is stuck on my end.
My reasoning is that a week should be enough time for a result, but I would rather understand what the tool is actually waiting for before I assume it is ready. What has to be true before I can trust the outcome
2 answers
A week on its own is not the bar. An experiment stays "in progress" until every variant has cleared three thresholds at once: at least 1000 visitors, at least 25 conversions, and at least one week of runtime. All three, per variant, not any one of them.
So the calendar week you have hit is only the third condition. If a variant is still short on visitors or conversions, the status will sit at "in progress" no matter how long it runs. Once all three are met for each variant, a recommended winner appears when probability to be best passes 95% and potential loss drops under 0.1%.
The upside of that design is you can read the numbers whenever you like. They are Bayesian, real time, and unsampled, so nothing breaks by peeking. The thresholds just stop the tool from calling a winner on thin data.
That lines up, my variants are well past the visitor count but sitting around 15 conversions each. So it is the conversion floor holding it. Makes sense, thanks
In my experience the one that keeps a test lingering is almost always the conversion threshold, especially if your primary goal is something that fires deep in the funnel. Plenty of traffic, plenty of visitors, but the goal itself is slow to accumulate 25 per variant. If it is a purchase or a signup, that floor can take a while on lower-traffic pages.