First A/B test on a VTEX product page, what does the flow look like end to end?
The tag is installed and events are flowing on our VTEX store, so I am ready to run my first real A/B test on a product page. I have never set one up before and I would love a clear step-by-step, from picking the slot all the way to reading the result.
Sorry if this is obvious, I just want to get the sequence right the first time.
2 answers
Not obvious at all the first time, so here is the whole sequence.
Start by wiring the slot on the product page, the specific placeholder where the variant content will render. Then build an experience for that slot, and attach an experiment to it. On the experiment you set the primary goal; because your VTEX order placement records a conversion automatically as Goal Completed, that is the natural conversion goal for a product-page test. You also set your traffic allocation and add your variants there.
Then let it run. The experiment stays in progress until each variant reaches at least 1000 visitors, at least 25 conversions, and at least one week, so do not read into it before those thresholds. The results are Bayesian and unsampled, and a recommended winner appears once a variant's probability to be best passes 95%.
The experiment setup steps walk through attaching it to the experience, and reading experiment results covers the analytics view once traffic is in.
For anyone else running their first one: do not touch traffic allocation or add and remove variants once it is live. Changing those mid-flight invalidates the stats and you effectively restart the reads. Set them at the start and leave them.
One more piece of reading that helped me the first time, the how to run A/B tests on VTEX write-up walks the same flow with a store example. And if it looks stuck later, the experiment stuck in progress thread and the probability to be best explainer are the two I keep going back to.