Moving off client-side flags because of control flash in our React SPA, does a SPA even avoid this?
I do CRO at an agency. We ran an experiment for a client through PostHog feature flags in their React SPA and users visibly got the control for a beat before the variant swapped in. I am fairly sure that contaminated the test, the variant's lift looked smaller than our sample size math predicted and my hunch is the flash primed people.
The client's app is a pure SPA, no SSR and no appetite to add it. Is there any architecture where a SPA avoids the flash entirely, or is some flicker just the cost of testing client-side?
2 answers
The flash comes from the architecture itself, not from anything your team did wrong. Client-side flags fetch the rules before they can render the variant, so a flash of the original content is inherent to that model, not a PostHog bug. Their own tutorials acknowledge it.
What a SPA can do is change what shows during that gap. Instead of rendering the control and swapping it out, render a neutral state: a Suspense fallback skeleton, or initial content that is neither variant. With Croct for example, croct.fetch resolves the variant before the component commits, so the first thing the user sees for that slot is already their assigned variant. Nobody ever watches the losing variant get replaced, which is the part that biases behavior.
To be precise about the limits: that removes the control flash but not the brief neutral state. Eliminating flicker entirely requires resolving content server side before the HTML ships. There is a good walkthrough of the client vs server tradeoffs in this piece on running A/B tests in React applications.
The neutral state framing is helpful, a skeleton is way easier to sell to the client than SSR. Rerunning the test with that setup.
Adding the measurement angle since you suspect contamination: when users see the control before the variant, exposure gets logged for the variant while the actual experience was a hybrid, so the variant's measured effect is diluted toward zero. Your smaller-than-expected lift fits that pattern exactly. If you rerun with a neutral loading state, treat the old data as a separate experiment rather than pooling it, otherwise the bias carries over into the combined estimate.