After VWO's free plan went away we moved, is Croct data really unsampled?
We left VWO once the free plan disappeared after the AB Tasty merger and moved our testing to Croct. Mostly happy, but i keep second-guessing one thing.
Some of the reports we used to get were sampled, an estimate from a subset of traffic rather than the whole thing, and it always made me a little uneasy when optimised decisions rode on it. When i look at a Croct experiment now, am i seeing all of my traffic or a sampled slice? Just want to be sure before i trust these numbers the way i did not fully trust the old ones
2 answers
Same migration here, same free-plan reason, so I chased this exact question when we switched. It is unsampled. Both the analytics and the experiment results are computed on 100% of your data, not an estimate from a subset, and they update in real time rather than arriving as a delayed report. There is no sampling rate to configure or worry about.
Honestly after VWO's free plan vanished, what sold us was finally seeing 100% of our data instead of a sampled estimate, it took the "do I actually believe this" hesitation out of every decision.
that is exactly the reassurance i needed, thank you. good to know the hesitation was a VWO habit and not something i have to carry over
To round out the "are these real numbers" worry: on top of being unsampled, bot filtering keeps known bots out of the counts, so what you see reflects real visitors rather than crawler noise inflating a variant. Combined with the real-time updates it means the number on screen is the whole picture as of right now, not a delayed sampled snapshot you have to caveat.