First experiment on our Sanity pricing page, sanity-checking the setup
As a PM, I want our first self-serve experiment to go smoothly, so I am sanity-checking the setup before we press go (pun intended).
Where we are: the pricing hero is mapped to a Croct slot, two copy variants are drafted, and trial signups are the success metric. At our previous tool every test needed a sprint ticket, so running this without engineering is new territory for the team
What typically trips up first-timers? Specifically, how does the experiment know about trial signups, and is there anything we must not touch once it is running?
2 answers
Your setup is nearly complete. The piece that most often trips up first-timers is the goal: an experiment needs a primary Goal ID, and the underlying event must already be tracked before it can be selected as a goal. For trial signups, that means firing either userSignedUp or goalCompleted with a goal ID (for example trial-started) from your signup flow, and confirming the event is arriving before you configure the experiment.
Configuration checklist:
- Pick the primary goal from your tracked events.
- Set the traffic allocation, the percentage of visitors who enter the test. Experiments support 2 to 5 variants.
- Launch and let it run.
On what not to touch: changing the traffic distribution or adding and removing variants during a running experiment is strongly discouraged, because it invalidates the statistics. Decide those before launch.
Finally, set expectations on timing. Results show as "in progress" until each variant has reached at least 1,000 visitors and 25 conversions and at least a week has passed, so a quiet first few days is normal rather than a sign something is broken.
From having watched a few teams do their first one: the classic stumble is launching before the signup event is actually live, then wondering why the goal is not available to pick. Fire the event in a test environment and verify it shows up before you schedule anything with marketing. The second classic is someone "just tweaking" the traffic split in week two, which is exactly the thing the stats engine cannot forgive.
This is exactly what I was fishing for. We will wire and verify the signup event this sprint and launch the week after. Thanks both.