How do I keep a holdout group to measure our personalization program overall?
I run growth at a media group in the Nordics, and individual test results are not my problem. The board question is: does the personalization program as a whole pay for itself. For that I need a slice of visitors who see none of it, a global holdout, so I can report program-level lift rather than a stack of individual test wins that may overlap.
Does Croct support a holdout group as a concept, or do I need to construct one myself? And if I construct it, what is the recommended mechanism so the comparison holds up in an org-level report
1 answer
Traffic allocation is the built-in lever for this. When you configure an experiment you set what share of traffic participates, for example allocating 30% to the experiment means the remaining 70% sees the default content. That untouched share is functionally your holdout: a randomly assigned group that experiences none of the variants, giving you the comparison baseline.
Allocation is set as part of experiment configuration, alongside the primary Goal ID and the 2 to 5 variants. One rule matters a lot for the kind of report you are writing: do not change the allocation once the experiment is running. Adjusting the split mid-flight invalidates the statistics, so pick the holdout size you can defend and leave it alone for the full run.
For the program-level question specifically, look at journey-level experiments. Croct can run experiments across full multi-page journeys rather than a single slot, so your holdout comparison can span the whole funnel a visitor moves through instead of measuring one hero at a time. That maps much more cleanly onto "does the program pay" than aggregating per-slot results, because it captures how the experiences interact along the path.