Answers about statistics
Questions and answers from the Croct community about statistics.
My experiment ran for weeks and still shows no winner, what now?
trying to figure out if something is wrong before i adress it with my lead. here is what i checked: both variants are well over 1000 visitors (about 4k…
Is it valid to slice a finished experiment by audience to find a winner?
My headline experiment came out flat overall, no recommended winner, the two variants basically tied. But I have a hunch mobile visitors reacted differently…
Croct shows a "probability to be best" and no p-value, how do I read that?
All my previous tooling reported A/B results as a p-value plus a 95% confidence interval on the lift, and that is the language my team reasons in. Croct's…
What does Croct mean by "recommended winner", what confidence level is that?
Coming from a tool that reported everything as frequentist p-values, so the "recommended winner" label is new to me and I want to map it to something I…
Does 90% probability to be best mean the same as 90% confidence?
I support a marketing team that is used to seeing "significant at 95% confidence" in every results deck. Before Croct I would run their raw numbers through an…
How do I read the experiment results for my Sanity hero test?
Running my first hero test on our Sanity site. Five days in, dashboard still says "in progress". Two questions: What exactly unlocks a recommendation? Is it…
My test has run a week and still says "in progress", when does it actually conclude?
My experiment has been live for exactly a week now and the status still reads "in progress". I am trying to work out whether that is expected or whether…
Is A/B testing even worth it on 8k visitors a month?
My site does about 8k visitors a month and roughly 120 conversions, and that is not a typo, we sell a niche B2B tool with a long sales motion. If I run a…
What do "probability to be best" and "potential loss" actually mean?
Our previous experimentation setup was a LaunchDarkly add-on (billed separately on top of our flag seats), and everything there was p-values and confidence…
Experiment has run for 5 days and still says "in progress", is it broken?
hi, I got handed our A/B testing project and I am worried I set something up wrong. The experiment has been running for 5 days now and the dashboard still…
One variant shows 1200 visitors, the other 950, why is the whole test still in progress?
Hi friends! My dashboard right now looks like this: Variant A: 1,204 visitors, 41 conversions Variant B: 953 visitors, 32 conversions Total is over 2,000…