Keeping Playwright E2E runs from sending real Croct events and eating MAU
Our nightly Playwright suite runs about 400 tests against a staging deployment of our Next.js app. I noticed the runs show up as sessions in the Croct dashboard.
Concerns, in order:
- Test traffic pollutes the analytics for our staging experiments.
- Every nightly run creates synthetic users that presumably count somewhere.
- Tests occasionally land in different experiment variants, so assertions on page content are flaky.
What is the sanctioned way to run E2E against a personalized app? I would rather not block Croct requests at the network level since that changes app behavior.
1 answer
There is a first-class switch for exactly this. Set NEXT_PUBLIC_CROCT_TEST in the environment your Playwright target is built with. It enables the mock event transport: all the app code paths stay identical, track calls resolve normally, but no real events are sent. That solves both the polluted analytics and the synthetic sessions, without network-level blocking that would change how your app behaves.
Since it is a NEXT_PUBLIC_ variable, it is inlined at build time, so set it on the build for your E2E target specifically, not on production.
For your third concern, flaky assertions from variant assignment: pair the test transport with fallback content in your fetchContent and useContent calls, and assert against those deterministic defaults. That way tests verify your components render content correctly without depending on which experiment variant a fresh test browser happens to land in. Experiment behavior itself is better verified with the preview links from the dashboard, which pin a specific variant.