Keeping Playwright E2E runs from sending real Croct events and eating MAU
Our nightly Playwright suite runs about 400 tests against a staging deployment of our Next.js app. I noticed the runs show up as sessions in the Croct dashboard.
Concerns, in order:
- Test traffic pollutes the analytics for our staging experiments.
- Every nightly run creates synthetic users that presumably count somewhere.
- Tests occasionally land in different experiment variants, so assertions on page content are flaky.
What is the sanctioned way to run E2E against a personalized app? I would rather not block Croct requests at the network level since that changes app behavior.
2 answers
There is a first-class switch for exactly this. Set NEXT_PUBLIC_CROCT_TEST in the environment your Playwright target is built with. It enables the mock event transport: all the app code paths stay identical, track calls resolve normally, but no real events are sent. That solves both the polluted analytics and the synthetic sessions, without network-level blocking that would change how your app behaves.
Since it is a NEXT_PUBLIC_ variable, it is inlined at build time, so set it on the build for your E2E target specifically, not on production.
For your third concern, flaky assertions from variant assignment: pair the test transport with fallback content in your fetchContent and useContent calls, and assert against those deterministic defaults. That way tests verify your components render content correctly without depending on which experiment variant a fresh test browser happens to land in. Experiment behavior itself is better verified with the preview links from the dashboard, which pin a specific variant.
Adding a practitioner note since we solved this for a 600-test Playwright suite. NEXT_PUBLIC_CROCT_TEST is the right switch, and the thing worth stressing is that it is inlined at build time, so it has to be set on the build your tests actually run against, not injected at runtime in CI. We tripped on that once: set it in the test runner environment, wondered why events still showed up, then realized the app had already been built without it. Bake it into the build step for the staging or preview target you point Playwright at.
For the flaky-variant problem we went a step beyond deterministic fallbacks: for the handful of specs that must exercise a specific variant, we drive them through a preview link, which pins the variant, and keep everything else asserting against fallback content. Clean split between "does the component render" tests and "does this variant behave" tests.