Rolling our own variant logic on Strapi vs using a platform, what am I missing?
We are a 5-person startup running Strapi, and as the CTO I estimate roughly a weekend to hack together variant fields on our content types plus a cookie-based split in the frontend. Before I commit either way I want the hidden costs spelled out by people who have been down both roads.
Specifically: what does a homegrown setup end up needing beyond the initial bucketing code? And for the platform route, what does a free tier actually cover, because at our size I am not signing a contract to run one hero test.
2 answers
I built exactly this at my last company and the weekend estimate is accurate for the happy path. What it does not cover is everything that made us rip it out a year later:
- Bucketing that stays deterministic across deploys. Our first hash function reshuffled users when we changed variant counts.
- Sticky assignment. A cookie split falls apart the moment a user logs in on another device and sees the other variant.
- Statistics. Counting conversions per bucket is easy, knowing when the difference is real is not. We shipped a "winner" that was noise.
- Bot filtering. Crawlers landed in buckets and skewed everything until we noticed.
None of these are hard individually, but together they are a product, not a weekend. Whether that trade is worth it at 5 people is your call, just do not price it as a weekend.
On the free tier question, since you mentioned Croct-style platforms: their forever-free plan is $0 up to 10k MAU, 3 content slots, and 1 experience or experiment, no credit card required. That covers exactly the one-hero-test scenario you described, so you can validate the approach before any money moves.
The stats side is also worth weighing against a homegrown version: results are Bayesian and unsampled, and identified users get consistent variant assignment across devices. Reproducing that with a cookie split is where Sam's point 2 and 3 bite hardest. Details are on the Croct pricing page if you want the exact limits.