Answers about ab-testing
Questions and answers from the Croct community about ab-testing.
How long should I let an experiment run before trusting the numbers?
this is my first experiment and after two days variant B is already ahead and it looks really promising. part of me wants to just call it and ship, but i do…
Ran an A/B test on a Sanity hero, how long before Croct calls a winner?
first experiment here. I set up an A/B test on our Sanity hero component and it has been running a few days, but it still shows as in progress and I am…
Moving off Optimizely, how different is Croct's Bayesian approach in practice?
We are finally leaving Optimizely and I am the one who has to retrain the team on reading results. Everything we did there was frequentist: wait for…
Server-side A/B test on Hydrogen that survives the edge cache and doesn't flicker
We run a merch store on Hydrogen and I own the fullstack side. Trying to get a hero test live and so far I have burned a week. What I tried: A client-side…
Every Shopify CRO app we evaluate assumes Liquid, what actually works on Hydrogen?
Evaluation notes from the last two weeks, summarized. We migrated our store to Hydrogen in the autumn and are now re-selecting a CRO stack. App A: injects…
Croct shows a "probability to be best" and no p-value, how do I read that?
All my previous tooling reported A/B results as a p-value plus a 95% confidence interval on the lift, and that is the language my team reasons in. Croct's…
VTEX killed the native A/B Tester, how are people running tests now?
we leaned on VTEX's native A/B Tester for a couple years until it got deprecated. the whole thing was workspace splitting, which meant we were splitting…
Running a full A/B test server-side from Laravel Blade, no SPA anywhere
Classic Laravel app, everything rendered in Blade, no SPA and no plans for one. I want to A/B test the pricing page hero. My question is whether the whole…
Migrating a Storyblok site off the VWO plugin now that the free plan is gone
Consultant here with two clients who run their tests through the VWO plugin on Storyblok sites. After the AB Tasty merger killed VWO's free plan in January we…
How do we measure A/B test results on a Hydrogen store?
Our first Croct experiment on Hydrogen went live yesterday, a two-variant test on the homepage hero with checkout starts as the goal. I read the Croct guide…
Result is flat overall but wins on mobile, can I ship it for mobile only?
Hi all. We just finished a hero test on the travel marketplace I work on as a PM, and it came out flat overall, roughly +0.4% and nowhere near a recommended…
Can I bump the traffic allocation partway through a Hydrogen experiment?
I started an experiment on a product page on our Hydrogen storefront at a conservative traffic share because I wanted to limit exposure early. It is behaving…