Cypress test metrics: spot flaky tests before they bite
Turn Cypress test metrics into Prometheus time series with a Pushgateway and Grafana Alloy, and see the flake and duration drift a single CI run hides.
Cypress test metrics turn a pass/fail log into a time series, and that is the whole difference. A single CI run tells you which specs failed today. A trend tells you that one spec has been getting 200 milliseconds slower every week and fails once in every twenty runs — the flake that is already costing your team reruns, before anyone opens a ticket about it.
Grafana published a method for this in September 2026, and the notable thing is that it invents nothing. Every component is existing open source: Cypress lifecycle hooks, a Prometheus Pushgateway, Grafana Alloy, and Prometheus-compatible metrics storage. If your team already runs Grafana, the missing piece is about forty lines in a config file.
What Cypress test metrics actually look like
The instrumentation lives in two Cypress lifecycle hooks. Grafana's walkthrough uses before:run, which fires once before any spec and stamps a single run_id for the whole suite, and after:spec, which fires after each spec with the results object carrying test counts, states and durations.
Those hooks produce a small, deliberate set of series. From Grafana's implementation:
| Metric | What it holds | What it answers |
|---|---|---|
cypress_tests_total | Test counts by outcome — passed, failed, pending, skipped | Is the suite getting greener or just smaller? |
cypress_tests_run_total | Tests executed per spec | Did someone quietly skip a file? |
cypress_spec_duration_seconds | Spec execution time | Which file is eating the pipeline? |
cypress_spec_success | 1 when the spec had zero failures, otherwise 0 | Spec-level flake rate over time |
cypress_test_duration_seconds | Individual test duration by result state | The ten slowest tests, ranked |
The labels are what make it useful: spec for the filename, run_id for the suite execution, and github_run_id read from the GITHUB_RUN_ID environment variable that GitHub Actions provides. That last one is the join key. Without it a worsening metric is a curiosity; with it, you click from a spike in a Grafana panel to the exact CI run and the commit that caused it.
Why a Pushgateway, and why Prometheus warns against one
Prometheus scrapes. A Cypress run does not sit still long enough to be scraped — by the time a scrape interval comes round, the process has exited. So the results are pushed to a Prometheus Pushgateway, which holds them until Alloy collects them. Grafana's implementation accumulates spec metrics in memory and re-posts the complete body after each spec, under the grouping key path /metrics/job/{job}/instance/{instance}, so the gateway always holds a current snapshot.
Here is the part the coverage does not mention, and it matters before you roll this out. Prometheus's own documentation is openly discouraging about the component: it says the Pushgateway is only recommended in certain limited cases, warns that it becomes a single point of failure when many instances push through it, that it never forgets series and will expose them forever unless deleted manually, and that you lose the automatic up metric that tells you an instance is alive.
None of that invalidates the approach — service-level batch jobs are precisely the case Prometheus does endorse, and a CI suite is one. But two consequences follow directly. First, stale series are your problem: rename or delete a spec file and its old series lingers, quietly polluting any average you compute. Second, an absent metric no longer means a broken test run, because there is no up to tell the difference between "the suite passed and pushed nothing new" and "the gateway is down".
Grafana's own guidance points the same way: treat publication as a side effect of the test rather than part of the test outcome, so a failure to publish metrics never fails an otherwise successful run. Wrap the push in a try/catch and mean it. A monitoring outage that turns a green pipeline red destroys trust in the pipeline faster than any flaky test.
Wiring it up: hooks, gateway, Alloy, Grafana Cloud
The pipeline is four hops: Cypress builds the metrics in a hook, posts them to the Pushgateway, Grafana Alloy scrapes the gateway, and Alloy remote-writes to Grafana Cloud. In Grafana's example, a prometheus.scrape component targets the gateway on its default port, honours the labels already attached, scrapes on an interval, and forwards to a prometheus.remote_write component that ships to Grafana Cloud with basic auth and an external label marking the source.
Two implementation details are worth copying exactly. Escape your label values — a test title containing a quote or a newline will break the Prometheus text format and silently drop the push. And stamp one run_id per suite rather than per spec, because without it you cannot group a run's specs back together afterwards.
One thing to watch that Grafana does not flag: cardinality. A label per test title, multiplied by a run_id per execution, multiplied by a busy CI pipeline, is how a metrics bill grows without anyone deciding it should. This is the same trade-off that makes bucket and cardinality choices worth thinking about before the data lands, not after. Keep per-test series to the suites where you genuinely need them, and hold the rest at spec level.
What to alert on — our judgement, not Grafana's
Grafana's post describes dashboards; what follows is ours, from the test suites we inherit. Dashboards get looked at after someone is already annoyed. Alerts are what change behaviour, and the wrong alert is worse than none — the thresholds below are starting choices for you to tune, not findings from a study.
- Spec flake rate, not test failures. Alert when
cypress_spec_successfor a single spec drops below one over a rolling window of the last N runs while the branch stays green overall. A spec that fails sometimes and passes on rerun is the definition of a flake, and it is invisible in any single run. - Duration drift, as a ratio to the baseline. Alert when a spec's
cypress_spec_duration_secondsexceeds its own median over the previous fortnight by a set multiple. An absolute threshold ages badly; a ratio to the spec's own history does not. - Test count falling. Alert when
cypress_tests_run_totalfor a spec drops between runs with no corresponding deletion. This catches the most common silent regression in a CI suite: a skip added to unblock a release and never removed. - Nothing arrived. Because the Pushgateway removes the
upsignal, alert on the absence of fresh metrics for a branch that had a run. Otherwise your monitoring failing looks exactly like everything being fine.
Deliberately not on that list: total failure count. It fires on every genuine breakage, which CI already told you about, and it trains people to ignore the channel. The metrics are worth having because they answer questions a build log cannot — and because flakiness is a measurement problem before it is a fix problem, which is the same reason running a thing five times tells you more than running it once.
What this will not tell you
Metrics locate the flake; they do not explain it. cypress_spec_success dropping to 0.8 tells you a spec fails one run in five. It does not distinguish a race condition in your application from a fixture that depends on the time of day from a CI runner under memory pressure. That investigation is still yours, and it is still the expensive part.
These numbers also say nothing about whether the tests are any good. A suite can be fast, green and stable while asserting almost nothing — the gap that mutation testing exposes and coverage hides. Stability metrics measure the health of the suite you have, not whether it is the suite you need.
The practical next step is smaller than it looks. Instrument one suite, push to a local gateway, and watch it for two weeks before you write a single alert — you need a baseline before a threshold means anything. If the result is that your pipeline is slow rather than flaky, that is a different problem with a different fix, and CI throughput under AI-era load is where to take it. If it is flaky, you will now be able to name the spec, which is further than most teams get. Either outcome is worth two weeks of a QA and test automation engineer's attention.
Frequently asked questions
Record each spec's outcome as a metric across many runs rather than reading one CI log. A spec whose success metric sits between 0 and 1 over a rolling window is flaky by definition: it passes sometimes and fails sometimes on the same code, which no single run can reveal.
Prometheus pulls metrics on a schedule, but a Cypress run is a short-lived job that exits before a scrape arrives. The Pushgateway accepts pushed metrics and holds them so a collector can scrape them afterwards. Prometheus recommends it specifically for service-level batch jobs like this.
In our experience four are worth an alert: per-spec flake rate over a rolling window, spec duration drift measured against that spec's own recent median, test counts falling without a deletion, and the absence of metrics after a run. Total failure count is not, since CI already reports it.
Sending test results to Grafana puts suite health beside application and infrastructure metrics, so you can correlate a failure spike with a deployment or an environment change. It is worth doing when your suite is large enough that trends matter more than individual failures.
Attach the GitHub Actions run identifier as a label. Grafana's implementation reads the GITHUB_RUN_ID environment variable and adds it as github_run_id alongside a per-suite run_id, so any spike in a dashboard panel can be traced to the exact CI execution and commit behind it.
Prometheus's documentation warns that a Pushgateway becomes a single point of failure, never forgets series unless they are deleted manually, and removes the automatic up metric that signals an instance is alive. Renamed or deleted specs leave stale series that quietly distort averages.
