Answers about experiments
Questions and answers from the Croct community about experiments.
How do I prove the industry-specific hero actually lifts conversion?
Leadership likes the idea of firmographic personalization but wants proof it moves a number before we roll it out everywhere. Reasonable ask, honestly, I…
Does the 15 experiments limit on Growth count paused and archived ones?
I am planning our 2027 experiment roadmap and Growth lists 15 experiences/experiments. Before I organise the roadmap I need the counting rule pinned down. An…
My experiment ran for weeks and still shows no winner, what now?
trying to figure out if something is wrong before i adress it with my lead. here is what i checked: both variants are well over 1000 visitors (about 4k…
Croct says variant A won, how do I use Clarity to understand why?
I have a clear Bayesian winner in Croct, variant A. Before we roll it out, my team wants to understand the behaviour behind the result rather than just trust…
Should I start an experiment at 10% traffic and ramp it up over time?
Coming from a feature-flag background I am used to rolling changes out slowly, 10% then 25% then 50% then everyone, so if something goes wrong the blast…
With several experiments live at once, can I isolate replays for just one?
we run multiple concurrent experiments on different slots, and a single visitor can land in more than one of them at the same time. When I open Clarity I want…
When does a Storyblok Optimize experiment actually call a winner?
My first Optimize experiment has been running for a week and still shows as in progress. I would like to understand the exact conditions that have to be met…
Multivariate test for four elements, or a series of A/B tests?
I want to test four things on our landing page: the headline, the hero image, the CTA colour, and the subheading. I am torn between one multivariate test that…
How long should I let an experiment run before trusting the numbers?
this is my first experiment and after two days variant B is already ahead and it looks really promising. part of me wants to just call it and ship, but i do…
Starting my first experiment on a Strapi hero, what traffic split should I begin with?
I wired our Strapi hero block to a slot and built two variants, so now I have to actually set the experiment up. I am not sure whether to send all my traffic…
Can I check Bayesian results daily and stop early without wrecking them?
With frequentist tests I learned the hard way that peeking every day and stopping on the first good result inflates false positives, so we trained ourselves…
Can a HubSpot form submission be the primary goal of an A/B experiment?
I am setting up an experiment on a landing page and I want the conversion to count when someone actually submits our HubSpot demo form, not just when they…