Cross-Browser Testing

CI Matrix Strategy

PR pipeline: Chromium only for fast feedback. Nightly or pre-release: full chromium + firefox + webkit matrix.

Shard large suites across jobs: npx playwright test --shard=1/4 per CI node.

# GitHub Actions matrix example
strategy:
  matrix:
    project: [chromium, firefox, webkit]
steps:
  - run: npx playwright test --project=${{ matrix.project }}

Parallel matrix jobs finish faster than one sequential job.

Coverage Tiers

Tier 1 PR:     chromium, @smoke grep
Tier 2 merge:   chromium + firefox full
Tier 3 nightly: all engines + visual

• Tag smoke tests for fast PR gate.
• Full matrix on main branch nightly.
• Track pass rate per engine separately.
• Quarantine flaky tests per engine.

Cross-Browser CI Patterns

Practical tiering.

Sharding Across CI Nodes

When your suite exceeds a few minutes even with parallel workers, split it across CI machines using --shard=i/n. Each job runs a disjoint subset of tests.

npx playwright test --shard=1/3
npx playwright test --shard=2/3
npx playwright test --shard=3/3

Combine with matrix.project for browser × shard parallelism.

Merging Reports from Matrix Jobs

Each matrix cell produces a blob report. Upload artifacts from all jobs, then run npx playwright merge-reports to produce a single HTML dashboard stakeholders can browse.

npx playwright merge-reports ./all-blob-reports --reporter html

Essential when cross-browser runs span multiple CI nodes.

Common Mistakes

• No cross-browser testing until production bug report.
• Same failure threshold ignoring WebKit-only flakes.
• Running full matrix on every commit — slow PRs.
• Not merging blob reports from sharded jobs.

Key Takeaways

• Tier testing: fast PR, full pre-release.
• Projects + CI matrix scale cross-browser runs.
• Sharding splits suite across machines.
• WebKit is essential for Safari user coverage.