Answers about methodology
Questions and answers from the Croct community about methodology.
Result is flat overall but wins on mobile, can I ship it for mobile only?
Hi all. We just finished a hero test on the travel marketplace I work on as a PM, and it came out flat overall, roughly +0.4% and nowhere near a recommended…
How do I keep a holdout group to measure our personalization program overall?
I run growth at a media group in the Nordics, and individual test results are not my problem. The board question is: does the personalization program as a…
Price testing vs content testing on Shopify, do I need separate tools?
CRO specialist at an agency, scoping a testing program for a Shopify client. The client was quoted $499 a month for a CRO app to run price and shipping tests,…
What traffic allocation for an experiment on a niche audience?
We run ABM personalization and I want to experiment on an audience of enterprise accounts that represents roughly 4% of total traffic. The general advice I…
Running two experiments on different slots at the same time, do they interfere?
I ran experimentation programs at my previous company where we serialized everything out of interaction-effect paranoia, and it throttled the roadmap badly.…
Is A/B testing even worth it on 8k visitors a month?
My site does about 8k visitors a month and roughly 120 conversions, and that is not a typo, we sell a niche B2B tool with a long sales motion. If I run a…
Ad blockers silently drop a chunk of visitors from our client-side tests, how do you keep samples honest?
Framing this as a sampling bias problem because that is what it is. Our client-side testing tool's script is on common blocklists, so visitors running ad…
Variant B is clearly losing, can I kill the test before the thresholds clear?
Client experiment I am consulting on, 9 days in, variant B is converting well below control and the gap has not narrowed once. Every day it runs the client…
Should the experiment goal be purchases or add-to-cart clicks?
Hello everyone. I am a CRO specialist at an agency and I keep having the same debate with clients. They want "purchase" as the primary goal on every test,…
How much do anonymous users switching devices distort experiment results?
I do data science at an ecommerce group. I understand that anonymous visitors cannot keep a variant assignment across devices, there is no identifier to link…
Huge lift in the first three days, now shrinking daily. Novelty effect?
Launched a flashy new hero variant last Tuesday. Days 1-3 it was showing about +30% on our signup goal and yes, my CEO has already screenshotted the day-2…
How many variants can one experiment have?
Planning an A/B/n headline test for a client and we brainstormed seven candidate headlines. Before I start cutting, what is the actual ceiling on variants per…