A test flakiness score is a metric that measures how often a specific test produces inconsistent results, passing and failing across repeated runs against unchanged code. It's typically calculated as a ratio or percentage, how many of the last N runs disagreed with the majority result, and it turns a vague, anecdotal complaint ("that test seems unreliable") into a specific, trackable number a team can monitor, compare across tests, and act on. A test with a flakiness score of zero has been perfectly consistent; a test with a high score is actively undermining confidence in the suite it belongs to, and everything in between tells a team roughly how much attention that particular test deserves.
The reason a flakiness score exists is that flaky tests are genuinely hard to spot by instinct alone in a large test suite. A test that fails once every fifty runs looks, on any single day, exactly like a test that never fails, right up until it fails at the worst possible moment, blocking a release or getting dismissed and re-run without investigation. Without a number tracking this pattern over time, flaky tests hide in plain sight, and teams develop a habit of just re-running failed builds instead of asking why they failed, which quietly erodes the entire point of having an automated test suite. A suite that gets re-run automatically until it passes is barely different, in practical terms, from a suite that doesn't run at all.
What distinguishes a flakiness score from a simple pass/fail rate is that it isolates inconsistency under unchanged conditions specifically, rather than measuring how often a test fails overall. A test that reliably fails every time because the code is genuinely broken has a low flakiness score and a high failure rate; a test that fails unpredictably, sometimes passing and sometimes failing on the exact same commit, has a high flakiness score regardless of its overall pass rate. That distinction matters because the fix for each is completely different: a consistently failing test points to a real bug, while a flaky test points to a problem in the test itself or its environment.
By 2026, flakiness scoring has moved from something a handful of large tech companies built internally into a standard feature of mainstream CI platforms and test reporting tools. Growing test suite sizes, more parallel execution, and heavier reliance on integration and end-to-end tests, all of which introduce more opportunities for timing issues and shared state problems, have made flakiness a bigger practical problem than it used to be, and tooling has caught up by automatically tracking historical pass and fail patterns per test rather than requiring teams to build that tracking themselves. What used to require a dedicated internal tooling team to build from scratch is now something a mid-sized engineering organization can turn on with a configuration change.
This page covers how flakiness scores are calculated, what actually causes flaky tests, concrete ways to fix flakiness at its source rather than just working around it, how to use a flakiness score to prioritize fixes instead of just documenting a problem, where flakiness scoring is worth the investment and where it's unnecessary, and how to build a practice around driving the score down instead of just watching it. The point underneath all of this is straightforward: test reliability is measurable, and once it's measurable, a team can manage it deliberately instead of treating flaky tests as an unavoidable cost of automated testing. That framing turns "our tests are flaky" from an eternal complaint into a specific, addressable engineering problem with an actual owner and an actual number to track.
The most common approach tracks a rolling window of recent runs for each individual test, typically the last twenty, fifty, or hundred executions, and calculates what fraction disagreed with the majority outcome. If a test passed forty-eight times and failed twice in its last fifty runs, with no corresponding code change around those two failures, its flakiness score reflects that two-out-of-fifty inconsistency rate. The rolling window matters because it keeps the score current, a test that used to be flaky but got fixed should see its score drop as clean runs accumulate and push older, inconsistent runs out of the window.
A more refined version of this calculation controls for actual code changes, since a test that fails after a genuine code change isn't flaky, it's correctly catching a regression. This requires correlating test results with the specific commit or change being tested, and only counting a result as "inconsistent" if the exact same code produced different outcomes on different runs. Simpler flakiness tracking that ignores this distinction can misclassify tests, especially in codebases with frequent small commits, so more mature tooling explicitly separates "failed because the code changed" from "failed because the test itself is unreliable."
Some scoring systems weight recent runs more heavily than older ones, on the reasoning that a test's current reliability matters more than its reliability six months ago, especially after a known fix has been applied. Others incorporate failure clustering, noting whether failures tend to happen at specific times (like right after a deployment, or specifically during parallel execution but not serial execution), which gives a much stronger diagnostic signal than a flat percentage alone, since it points toward what kind of environmental issue is actually causing the inconsistency.
Aggregate flakiness metrics, rolling the individual test scores up to a suite-wide or project-wide number, help teams see overall trends and set organizational goals, like reducing overall flaky test rate by a certain amount over a quarter. But these aggregate numbers can hide serious individual problems: a suite with a low average flakiness score can still contain a handful of severely flaky tests that happen to sit on critical paths, so most mature practices track both the aggregate trend and a ranked list of the worst individual offenders, since the two numbers answer different questions and neither replaces the other. A quarterly aggregate number is useful for a leadership dashboard; a ranked list of the ten worst offenders is what an engineering team actually works from week to week.
It's also worth tracking flakiness separately by test type, since unit tests, integration tests, and end-to-end tests tend to have very different baseline flakiness profiles for structural reasons that have nothing to do with how carefully any individual test was written. Unit tests, which typically don't touch a real database, network, or file system, should have flakiness scores close to zero almost by default, and a rising trend there usually points to a specific, isolated bug in test isolation. End-to-end tests, which often exercise a full running system including real timing and real network calls, naturally carry more inherent flakiness risk, so comparing an end-to-end test's score against a unit test's score on the same dashboard without that context can lead a team to draw the wrong conclusions about where the real problems are concentrated.
Shared state between tests is one of the most common root causes: a test that writes to a database, a file, or a global variable without properly cleaning up afterward can leave behind state that silently affects whatever test runs next, especially when tests run in a different order between executions or in parallel across multiple workers. This kind of flakiness is maddening to diagnose because the failing test often has nothing wrong with it directly, the actual problem lives in a completely different test that ran before it and left the environment in an unexpected state.
Timing and asynchronous assumptions cause a huge share of flakiness in modern systems, particularly anything involving network calls, background jobs, or UI rendering that completes at slightly variable speeds. A test that assumes an operation finishes within a fixed, short delay will pass reliably on a fast, quiet CI machine and fail intermittently on a slower or more loaded one, not because the underlying feature is broken but because the test's patience ran out before the real system finished doing its actual work. This category gets worse as systems become more distributed, since more network hops means more opportunities for variable latency to trip up a test that wasn't built to tolerate it.
External dependencies, real third-party APIs, shared test databases, live network services, introduce flakiness that has nothing to do with the code being tested at all. A test that calls a real payment gateway's sandbox environment will fail whenever that sandbox is slow, rate-limited, or briefly unavailable, and the test suite ends up reporting a false failure against code that never actually had a problem. This is exactly the kind of flakiness that erodes trust fastest, since developers quickly learn that a failure from this particular test usually means nothing, and they start ignoring it, which is dangerous the one time it's actually reporting a real issue.
Test order dependency and resource contention round out the most common causes: tests that only pass when run in a specific sequence, or tests that compete for a limited shared resource, a fixed port, a single test account, a rate-limited API key, when run in parallel. Both of these problems tend to be invisible during serial, single-threaded local development and only surface once a team starts running tests in parallel or on CI infrastructure that runs many jobs concurrently, which is part of why flakiness often seems to appear "out of nowhere" right after a team makes an unrelated infrastructure change like enabling parallel test execution to speed up CI.
The most durable fix for shared-state flakiness is strict isolation, giving each test its own database instance, its own transaction that rolls back afterward, or its own containerized environment, so no test can leave behind state that affects another. This is often more work upfront than the alternative, adding retries or increasing timeouts, but it addresses the actual defect rather than papering over its symptom, and it tends to stay fixed permanently instead of resurfacing every time the system changes in some small way that shifts the timing just enough to expose the same underlying race again.
For timing-related flakiness, the fix usually isn't a longer fixed delay, that just shifts the problem to a slower failure mode instead of removing it, it's replacing a fixed wait with an explicit poll or wait-for-condition pattern that checks repeatedly until the expected state is reached or a generous timeout expires. This makes tests both faster on average, since they don't wait longer than necessary, and more reliable, since they no longer fail just because a system happened to be a little slower than usual on a particular run.
External dependency flakiness is best solved by removing the real dependency from the test entirely wherever practical, replacing a live third-party API call with a mock or fake service that behaves predictably and doesn't depend on a network connection, a rate limit, or another company's uptime. Where a live integration genuinely needs to be tested, that kind of test usually belongs in a smaller, separately tracked suite that isn't relied on to gate every single deployment, precisely because its reliability partly depends on infrastructure the team doesn't control.
Retrying a failed test automatically before marking it failed is sometimes treated as a fix, and it does reduce the immediate annoyance of a build failing for a known-flaky reason, but it should be treated as a temporary mitigation rather than a resolution. A test that needs three attempts to pass reliably is still telling you something is wrong; automatically retrying it without ever addressing the root cause just changes flakiness from a visible, tracked problem into an invisible, silent one that continues to cost time and CI resources indefinitely.
A flakiness score only earns its keep once it changes what a team actually does, and the most direct use is triage: sorting flaky tests by score and business criticality so effort goes to the worst offenders on the most important paths first, instead of whichever flaky test happens to have failed most recently and annoyed someone. A test with a modest flakiness score sitting on a critical release-blocking path deserves attention before a badly flaky test covering a rarely used, low-risk feature, even though the second one has a worse raw number.
Setting a threshold that automatically flags or quarantines tests above a certain flakiness score keeps the problem from being purely a manual review exercise. Many teams automatically move a test into a separate "known flaky" suite once its score crosses a set point, still running it and still tracking its score, but no longer letting its failures block a deployment. This isn't a way to hide the problem, the test's score keeps getting tracked, and a genuinely fixed test can be moved back once its score drops and stays down for a meaningful stretch, but it does stop one unreliable test from blocking legitimate, unrelated releases.
Correlating flakiness scores with recent changes to the test itself, or to the shared infrastructure it depends on, often points directly at the fix. A test that was stable for months and suddenly developed a rising flakiness score right after a shared fixture or a CI configuration change almost certainly broke because of that change, not because of some newly discovered fundamental instability in the test's logic. Teams that treat a sudden flakiness spike as a signal to check recent related changes, rather than immediately assuming the test itself is poorly written, tend to find and fix the real cause faster.
Tracking flakiness trends over time, rather than looking at a single snapshot, reveals whether a team's broader engineering practices are actually improving or quietly getting worse. A rising trend across many tests, rather than one isolated problem test, usually signals something systemic, growing shared test infrastructure under strain, increasing parallelization without corresponding isolation work, a test database that's gotten slower as it accumulated more historical data. Treating flakiness as a portfolio-level metric worth reviewing regularly, rather than a per-test annoyance to whack down one at a time, catches these systemic issues before they get much worse. A team that only ever looks at individual test scores in isolation can miss a slow-building infrastructure problem until it has already spread across dozens of unrelated tests, at which point the fix is far more expensive than it would have been if caught early.
Flakiness scoring earns its investment fastest in large, mature test suites with heavy CI usage, many parallel test runs a day, and multiple teams depending on the same suite's results to make deployment decisions. In these environments, the collective cost of investigating individual flaky failures manually, multiplied across dozens of developers running the suite many times daily, is large enough that automated tracking and a clear prioritization signal pay for themselves quickly, often within weeks of the tooling actually going live.
It matters less, though it's rarely completely useless, for small projects with a handful of tests run occasionally by a small team, where a person can reasonably keep a mental model of "oh, that one test is a little flaky" without needing formal tracking. Building a dedicated scoring system for a five-person project running fifty tests a handful of times a day is solving a problem at a scale that doesn't quite exist yet, and the tooling overhead isn't obviously worth it until the suite and the team both grow.
The judgment call in between, a mid-sized, growing suite, usually comes down to whether flaky failures are already causing real friction: developers re-running builds without investigating, releases getting delayed by tests nobody trusts, or a general sense that "the tests are unreliable" without anyone being able to say which ones or how bad it actually is. That friction is the signal that a team has crossed from "flakiness is a minor annoyance" into "flakiness is an active drag on delivery speed," and that's the point where formal scoring starts paying for itself rather than adding process for its own sake.
Flakiness scoring doesn't replace fixing the underlying causes, isolation problems, timing assumptions, external dependencies, it just makes those problems visible and prioritizable. A team that tracks flakiness scores meticulously but never actually assigns engineering time to fix the worst offenders has built a very precise dashboard of a problem it isn't solving, which is arguably worse than not measuring at all, since it creates the appearance of managing the issue without any of the actual benefit. The measurement only pays off once it's tied to an actual commitment, a fixed amount of time each sprint dedicated to driving the worst scores down, rather than a report someone glances at occasionally.
A test flakiness score is a metric that measures how often a specific test produces inconsistent pass and fail results across repeated runs against the same, unchanged code, typically expressed as a percentage or ratio over a rolling window of recent runs on that individual test.
A failure rate measures how often a test fails overall, which can be high for a consistently, correctly failing test that's catching a real bug; a flakiness score specifically measures inconsistency, disagreement between runs on the same unchanged code, regardless of the test's overall pass or fail balance.
The most common causes are shared state leaking between tests, timing assumptions that don't hold under variable system load, reliance on real external dependencies like third-party APIs, and resource contention or ordering issues that only appear once tests start running in parallel.
Deleting removes the immediate annoyance but also removes whatever real coverage the test provided; a better first step is usually quarantining it from blocking releases while investigating and fixing the actual cause, and only deleting it if it turns out, after honest investigation, to provide no meaningful, salvageable coverage at all.
There's no universal number, but most practical implementations use a rolling window of somewhere between twenty and a hundred recent runs, balancing having enough data to be statistically meaningful against keeping the score responsive to recent fixes or regressions in the underlying test.
Not necessarily. Flakiness often comes from shared infrastructure, environment issues, or timing assumptions that have nothing to do with how well the assertion itself was written, so a high score should prompt investigation into the test's environment and dependencies first, rather than assuming the fault lies in the assertion logic itself.
Combine the flakiness score with how business-critical the test's coverage area is; a moderately flaky test on a release-blocking checkout flow generally deserves attention before a severely flaky test covering a low-traffic, low-risk feature that rarely affects real deployment decisions.
Most mainstream CI platforms and test reporting tools now track flakiness automatically by recording historical pass and fail results per test and calculating rolling scores, which removes the need for teams to build this tracking themselves from scratch the way large tech companies had to years ago.
A team can build a precise, well-maintained flakiness dashboard and still never actually fix the underlying causes, which creates a false sense of managing the problem without delivering any of the real benefit, since the flaky tests keep undermining trust in the suite regardless of how well they're measured. The metric only earns its cost once fixing the worst offenders becomes an actual, scheduled piece of engineering work rather than a report people glance at occasionally.
