Answers about ab-testing
Questions and answers from the Croct community about ab-testing.
How do I prove the industry-specific hero actually lifts conversion?
Leadership likes the idea of firmographic personalization but wants proof it moves a number before we roll it out everywhere. Reasonable ask, honestly, I…
Could our content team own A/B tests end to end on Storyblok?
I lead a small content team on a Storyblok site and I am trying to figure out whether we could genuinely own experimentation ourselves rather than depending…
My experiment ran for weeks and still shows no winner, what now?
trying to figure out if something is wrong before i adress it with my lead. here is what i checked: both variants are well over 1000 visitors (about 4k…
Should I start an experiment at 10% traffic and ramp it up over time?
Coming from a feature-flag background I am used to rolling changes out slowly, 10% then 25% then 50% then everyone, so if something goes wrong the blast…
Is it valid to slice a finished experiment by audience to find a winner?
My headline experiment came out flat overall, no recommended winner, the two variants basically tied. But I have a hunch mobile visitors reacted differently…
Multivariate test for four elements, or a series of A/B tests?
I want to test four things on our landing page: the headline, the hero image, the CTA colour, and the subheading. I am torn between one multivariate test that…
How long should I let an experiment run before trusting the numbers?
this is my first experiment and after two days variant B is already ahead and it looks really promising. part of me wants to just call it and ship, but i do…
Can I check Bayesian results daily and stop early without wrecking them?
With frequentist tests I learned the hard way that peeking every day and stopping on the first good result inflates false positives, so we trained ourselves…
Moving off Optimizely, how different is Croct's Bayesian approach in practice?
We are finally leaving Optimizely and I am the one who has to retrain the team on reading results. Everything we did there was frequentist: wait for…
Server-side A/B test on Hydrogen that survives the edge cache and doesn't flicker
We run a merch store on Hydrogen and I own the fullstack side. Trying to get a hero test live and so far I have burned a week. What I tried: A client-side…
Every Shopify CRO app we evaluate assumes Liquid, what actually works on Hydrogen?
Evaluation notes from the last two weeks, summarized. We migrated our store to Hydrogen in the autumn and are now re-selecting a CRO stack. App A: injects…
Ran an A/B test on a Sanity hero, how long before Croct calls a winner?
first experiment here. I set up an A/B test on our Sanity hero component and it has been running a few days, but it still shows as in progress and I am…