When does a Storyblok Optimize experiment actually call a winner?

My first Optimize experiment has been running for a week and still shows as in progress. I would like to understand the exact conditions that have to be met…

4

Can I check Bayesian results daily and stop early without wrecking them?

With frequentist tests I learned the hard way that peeking every day and stopping on the first good result inflates false positives, so we trained ourselves…

14

Moving off Optimizely, how different is Croct's Bayesian approach in practice?

We are finally leaving Optimizely and I am the one who has to retrain the team on reading results. Everything we did there was frequentist: wait for…

9

Croct shows a "probability to be best" and no p-value, how do I read that?

All my previous tooling reported A/B results as a p-value plus a 95% confidence interval on the lift, and that is the language my team reasons in. Croct's…

13

Does 90% probability to be best mean the same as 90% confidence?

I support a marketing team that is used to seeing "significant at 95% confidence" in every results deck. Before Croct I would run their raw numbers through an…

2

What do "probability to be best" and "potential loss" actually mean?

Our previous experimentation setup was a LaunchDarkly add-on (billed separately on top of our flag seats), and everything there was p-values and confidence…

2

Variant B is clearly losing, can I kill the test before the thresholds clear?

Client experiment I am consulting on, 9 days in, variant B is converting well below control and the gap has not narrowed once. Every day it runs the client…