Huge lift in the first three days, now shrinking daily. Novelty effect?
Launched a flashy new hero variant last Tuesday. Days 1-3 it was showing about +30% on our signup goal and yes, my CEO has already screenshotted the day-2 dashboard and put it in a slide (I have been burned by this exact move before).
Day 4 it was +19%. Day 6, +11%. Today it is hovering around +6% and still drifting down.
Is the early window the "real" effect that competition is eroding, or is the trend the truth and the first days were noise? And is there anything in how Croct reports results that could exaggerate teh early numbers?
2 answers
Classic novelty effect shape. Returning visitors react to newness itself, they notice that something changed and engage with it because it changed, not because it is better. The early window of any test over-represents returning visitors relative to the long-run mix, so the first days are systematically biased toward whatever is new and shiny. This is one of the reasons the 1-week minimum runtime exists before Croct resolves anything, it forces the test past the initial burst.
To your last question: no, the reporting is not exaggerating anything. Croct evaluates results in real time on unsampled data, so the decay you are watching is real behavior, not a sampling or reporting artifact. Use the time-window drill-down to inspect the trend directly, if you slice the results by week you will likely see the effect stabilize somewhere well below the day-2 headline.
Tell your CEO the +30% slide has a shelf life.
Adding the practical stopping rule: do not call this until the dashboard shows a recommended winner, which requires Probability to Be Best above 95% and potential loss below 0.1%. A decaying lift often still ends up as a genuine winner, just a smaller one than the launch spike suggested. The +6% you see today might be close to the true effect, or it might keep drifting to zero. The winner rule is what separates those two cases, not the screenshot date.