Croct says variant A won, how do I use Clarity to understand why?
I have a clear Bayesian winner in Croct, variant A. Before we roll it out, my team wants to understand the behaviour behind the result rather than just trust the number.
We use Clarity for replays and heatmaps. How do people combine the quantitative result with the qualitative view so the two actually inform each other?
2 answers
The pattern you are describing is exactly the one we designed the Clarity integration around: quantitative to know which variant won, qualitative to understand the behaviour behind it.
On the quantitative side, Croct gives you the Bayesian, unsampled result, and it recommends a winner once the probability to be best is above 95 percent. That is your decision signal. How that number is derived is explained in reading Bayesian experiment metrics, and the mechanics are in the experiment analytics reference.
On the qualitative side, because each Clarity session carries the variant, experiment, and audience it belonged to, you filter Clarity down to the winning variant and watch how those visitors actually moved through the page. The recommended flow is to let Croct tell you A won, then open the variant A recordings and heatmaps in Clarity to see what about A worked. The connection that makes this possible is on the Clarity integration page.
This is our exact workflow now. We wait for Croct to call the winner, then spend maybe an hour in Clarity on the winning variant to write up the why for the rollout doc. Quant first so we do not go fishing in the replays for a story that is not there.
Echoing the quant-then-qual order, with one caution. Use Clarity to understand the winning variant, not to re-litigate the result by slicing replays into ever-smaller segments until you find a story. The number is the number. The thread on the trap of post-hoc segment drilldowns is worth keeping in mind before you start segmenting the replays too finely.