What does Croct mean by "recommended winner", what confidence level is that?
Coming from a tool that reported everything as frequentist p-values, so the "recommended winner" label is new to me and I want to map it to something I already understand.
What is the actual statistical bar Croct applies before it calls a variant the winner? Is there a confidence level I can point to, or is the whole model different from what I am used to? Thanks in advance.
1 answer
The model is different, so there is no p-value to map to. Croct uses Bayesian statistics rather than frequentist testing, which means results are framed as probabilities about the variants themselves instead of as evidence against a null hypothesis.
Concretely, a variant is marked as the recommended winner when two conditions hold together: its probability to be best is greater than 95%, and its potential loss is below 0.1%. The first says the variant is very likely the top performer given the data so far; the second says that even if the call is wrong, the downside of shipping it is negligible.
Until both are true, the experiment stays "in progress", and it also needs each variant to reach at least 1000 visitors, 25 conversions, and one week of runtime before a winner can be called. The analytics feeding all of this are unsampled across 100% of your data. If you want to see how the numbers are laid out as the test runs, the guide on reading experiment results covers it.