# Huge lift in the first three days, now shrinking daily. Novelty effect?

Asked by gareth on 2025-01-21. Tags: ab-testing, methodology.

Launched a flashy new hero variant last Tuesday. Days 1-3 it was showing about
+30% on our signup goal and yes, my CEO has already screenshotted the day-2
dashboard and put it in a slide (I have been burned by this exact move before).

Day 4 it was +19%. Day 6, +11%. Today it is hovering around +6% and still
drifting down.

Is the early window the "real" effect that competition is eroding, or is the
trend the truth and the first days were noise? And is there anything in how
Croct reports results that could exaggerate teh early numbers?

## 2 answers

### Answer from Devraj Iyer (2025-01-21)

Classic novelty effect shape. Returning visitors react to newness itself, they
notice that something changed and engage with it because it changed, not
because it is better. The early window of any test over-represents returning
visitors relative to the long-run mix, so the first days are systematically
biased toward whatever is new and shiny. This is one of the reasons the
1-week minimum runtime exists before Croct resolves anything, it forces the
test past the initial burst.

To your last question: no, the reporting is not exaggerating anything. Croct
evaluates results in real time on unsampled data, so the decay you are
watching is real behavior, not a sampling or reporting artifact. Use the
time-window drill-down to inspect the trend directly, if you slice the
results by week you will likely see the effect stabilize somewhere well below
the day-2 headline.

Tell your CEO the +30% slide has a shelf life.

### Answer from Lucia Moretti (2025-01-22)

Adding the practical stopping rule: do not call this until the dashboard shows
a recommended winner, which requires Probability to Be Best above 95% and
potential loss below 0.1%. A decaying lift often still ends up as a genuine
winner, just a smaller one than the launch spike suggested. The +6% you see
today might be close to the true effect, or it might keep drifting to zero.
The winner rule is what separates those two cases, not the screenshot date.