GA4 says variant B lost, Croct says it won, which number do I trust?
Croct's experiment results show a 4 percent lift for variant B with a recommended winner badge. GA4, looking at the exported variant dimension against the same conversion event, shows the conversion rate basically flat.
I have been doing attribution long enough to know that when two tools disagree, the usual answer is "both, in their own special way". But I have to report this to leadership next week and "the numbers are vibes" is not a slide I can present.
How do people here reconcile the two? Is one of these numbers actually wrong or am I comparing populations that were never going to match
2 answers
You are comparing populations that were never going to match, but the gap has identifiable causes.
The two pipelines process different data. Croct's experiment results are unsampled Bayesian statistics with bot filtering applied. GA4 applies its own processing, drops events beyond 100k per user per day, can sample explorations, and groups high-cardinality dimension values under "(other)", which silently removes rows from a variant breakdown. On top of that, ad blockers block analytics endpoints differently per tool, so each tool sees a slightly different slice of your traffic to begin with.
For the slide: the number built for this decision is the experiment-scoped one. The recommended winner badge means variant B crossed Probability to Be Best above 95% with potential loss below 0.1%, computed on 100% of the experiment's data. GA4 is a general-purpose tool answering a broader question on a differently filtered population; flat-ish there does not contradict a 4 percent lift measured on the experiment population.
Related discussion on the export-side caveats: GA4 Croct dimensions caveats.
A pattern that has saved me in these meetings: agree on the decision source before the experiment starts, not after the numbers disagree. We write it into the experiment brief, results in the experiment platform decide ship/no-ship, analytics tools are for diagnosis and follow-up questions. When the tools inevitably differ, nobody relitigates the decision because the hierarchy was agreed upfront.