flaky-test-triage
Use when a test passes on rerun, fails only in CI, fails only when the whole suite runs, fails after midnight or across a DST change, fails under parallel execution, fails on a loaded machine, or intermittently times out waiting for a message or event, or a scripted UI run fails half the time while the feature works by hand; or when someone proposes a retry, a skip, or a longer timeout to make the build green, or declares a flaky test fixed because it stopped failing after a change. Covers intermittent failures, order dependence, shared state, races, and quarantine.
Overview
A flaky test is a defect report whose owner has not been determined yet. The defect is in the test roughly half the time and in the product the other half — and the half that lives in the product is a race that users are already hitting at the same rate the test fails.
When to use
- A test failed, was rerun, and passed, with no code change in between.
- A test passes alone and fails in the suite, or passes in the suite and fails alone.
- Failures cluster on CI, on one worker, at a time of day, or after a DST change.
- Someone is about to add a retry, a skip marker, or a larger timeout.
- The suite is rerun as a matter of routine, and nobody reads the first result.
- Not for: a test that fails every time — that is a deterministic bug, owned by
diagnosing-before-fixing. - Not for: a check on a language model's output, which varies by design; its failure rate is the measurement, owned by
evaluating-llm-output.
The arithmetic that forces the issue
Independent per-test reliability compounds. At 99.9% per test, a 500-test suite is green 61% of the time; a 2,000-test suite, 13%. To keep a 2,000-test suite green on 95% of runs, each test must be 99.997% reliable. There is no per-test flake rate that is acceptable at scale, which is why "it only fails 1 in 50" is not a defense.
Cause taxonomy
| Signature | Cause | First diagnostic |
|---|---|---|
| Passes alone, fails in suite | Order dependence: shared mutable state, cached singleton, unrolled DB write | Bisect the test order to find the polluter |
| Fails only under parallel execution | Contended resource: fixed port, shared temp path, same DB row, global config | Run the same subset single-threaded |
| Fails near midnight, month end, or DST | Wall clock read directly; date computed twice across a boundary | Rerun with the clock offset by +6 months and a +05:45 timezone |
| Fails only on slow or loaded machines | Sleep or timeout tuned to the author's laptop, or a genuine race | Run under one core or artificial load |
| Fails at a stable rate with no pattern | Unseeded randomness in data, ids, or iteration order | Seed everything; check hash and map iteration order |
| Starts failing after N runs, then always | Resource leak: file descriptors, connections, disk, ports in TIME_WAIT | Watch handle and connection counts across the run |
| Fails only on the first run of the day or in a clean environment | Test depends on a warm cache, existing data, or a previously-created account | Run against a freshly provisioned environment |
| Order-dependent assertion on a callback-built list | The callback only preserves delivery order under a real dispatcher/event-loop context; without one — the normal case in a test harness — the guarantee disappears, whether the runtime falls back to unordered dispatch, throws, or no-ops | Check whether the code normally runs under a real dispatcher or event loop that the test doesn't supply |
| Fails at a stable rate, always on a wait's timeout; passing runs are fast, and a longer timeout only makes the failures slower | An earlier wait read the message and threw it away: where reading takes a message off the stream (a socket, queue, channel, or a child process's output), a helper that skips what it is not waiting for deletes it for every later wait, and two messages with no guaranteed order sometimes arrive the other way round | Log every message each wait reads, including the ones it skips; ask whether the first wait skips the kind the next one waits for, and whether anything guarantees their order |
| An automated driver reports a step failing about half the time; done by hand the feature always works | The driver, not the feature: its own calls have side effects and timing (an input sent right after a state dump is lost; a "back" pressed with nothing open leaves the app) | Do the failing step by hand, then again with the driver's instrumentation removed or spaced out |
| A different assertion fails each time | A real race in the product, not in the test | Stop triaging the test; treat it as a production incident |
The last row is the one that gets misfiled most often, and it is the one that matters most.
The driver row is the case where the test is wrong in the least visible way. A scripted UI run, a debugger, a tracer, a screenshot or state-dump call all act on the system they observe, with timing and side effects of their own, so a red from one is evidence about the pair (driver, system), not about the system. Settle which half failed before debugging the feature: repeat the failing step by hand, then through the driver with the instrumentation removed or spaced out. If it passes both ways, the fault is in the harness. Then make every step idempotent against the state it finds: press "back" only if the thing it closes is open, wait for the control to exist before acting on it, and assert the state each step expects before the next one runs. A step that assumes the previous one worked turns one lost event into a run of unrelated-looking failures, because everything after it ran against the wrong screen.
The order-dependent-callback row has the opposite prescription: the fix is asserting an order-independent invariant — max value reached, set membership, monotonic non-decrease — instead of position, or giving the test a synchronous stand-in for the missing dispatcher context, never a retry.
Like the callback row, the wait-timeout row is usually the test's defect; it is the product's only if the protocol promises the order that flipped. Wait once for the whole set of expected messages, in any order, under one deadline, or make the helper keep what it skips for the next wait to read first. A longer timeout cannot bring the message back: it is not late, it was read and thrown away, and the failure rate is how often the order flips.
Diagnosis
- Capture before rerunning. Full output, seed, test order or shuffle key, worker index, host, timestamps, and any artifacts. A rerun destroys the only evidence; a flake report lacking seed and order is unactionable, so make CI print both on every run.
- Quantify in isolation. Run the single test 100 times alone. Never failing alone means the cause is environmental or order-related, and further solo debugging is wasted.
- Quantify in context. Run the full suite with the recorded order 10–20 times. Establish a reproduction rate before attempting any fix; without one, the fix cannot be evaluated.
- Bisect the order. With a reproducing order fixed, binary-search the prefix of tests run before the victim. Roughly 10 runs isolates one polluter out of 1,000. This finds shared-state bugs no amount of reading finds.
- Vary one axis at a time. Threads: parallel versus single. Clock: shifted date and non-hour-offset timezone. Load: constrained CPU. Randomness: fixed seed. Each axis that changes the failure rate names the category.
- Attribute. Write down whether the defect is in the test or the product before writing any fix. A fix applied without attribution usually just moves the flake. When the defect is in the test — it's exercising something inherently non-deterministic rather than a real product race —
testing-the-untestablecovers making that source of non-determinism controllable instead of retried around.
A clean run count after a race fix is evidence only if the race still happened. A fix can make the bad interleaving rare instead of harmless — it moves the timing and the race window closes — and the count then comes back clean for the wrong reason. So while measuring the fixed build, also count the interleaving itself, with a throwaway log line recording which order each run saw. The fix is verified when that order still turns up at about the old failure rate and every run that saw it passed; the matching rate confirms the attribution too. If the order has stopped turning up, the clean count proves nothing either way, since the fix or the log line itself may have moved the timing: force the order with debugging-concurrency's injected delay and test the fix under that.
Quarantine, with an expiry
Quarantine is a scheduling decision, not a verdict. Every quarantined test carries four things or it is not quarantined, it is abandoned:
- A named owner.
- A linked defect describing the observed failure, with the captured evidence.
- An expiry date, 14 days or less. On expiry the test is fixed or deleted — no third option.
- Continued execution: it still runs and still reports, it just does not block the build. A quarantined test that stops running is deleted with extra steps, and the code it covered is now untested without anyone deciding that.
Cap the pool. Past ~1% of tests in quarantine, the suite no longer signals what people believe it signals; stop feature work until it drains.
Why blanket retry is the wrong response
Retrying converts an intermittent product race into a green build. The rate the retry hides is the rate users experience — the test was measuring it, and the retry deletes the measurement. Two conditions make a retry defensible, and both must hold:
- The failure is attributed to infrastructure outside the code under test (a provisioning timeout, a registry fetch), not to product behavior.
- Every retry is counted, reported, and alarmed on a threshold. Uncounted retries always grow, because nothing pushes back.
Retrying at the whole-suite level is worse than at the test level: it hides which test is unreliable and multiplies CI cost against a failure nobody will ever look at.
Common mistakes
| Symptom | Real cause |
|---|---|
| "It passed on rerun, so it's fine" | The rerun destroyed the evidence; the failure rate is unchanged |
| Fix applied, flake returns weeks later | No reproduction rate was established, so the fix was never verified |
| Timeout raised repeatedly over a year | A race is being masked; each increase buys a shorter reprieve |
| Same test flakes after being "fixed" three times | The defect is in the product; the test keeps being adjusted instead |
| Nobody investigates because the report has no detail | CI does not print seed, order, or worker; every report is a dead end |
| Suite green, users hit intermittent errors | Retries hid a real race that the test was correctly detecting |
| A skipped test's feature breaks in production | Quarantine without expiry silently removed coverage |
| Flakes only appeared after enabling parallelism | Pre-existing shared state that serial execution was accidentally hiding |
| Failure moves to a different test after a fix | The polluter was never found; only the victim was patched |
| A scripted UI or end-to-end run fails half the time and the feature works every time by hand | The driver's own calls lost an event or left the app in the wrong state, and later steps assumed otherwise |
| Test asserting the last item of a callback-built list is intermittently flaky | Looks like a product race, but it's the test harness missing the dispatcher context the callback needs to stay ordered |
Red flags
- "Just rerun it."
- "It's flaky, ignore it" — said about a test nobody has attributed.
- "Add
@retryand move on." - "Bump the timeout to 30 seconds."
- "Skip it for now" with no date attached.
- "It only fails in CI, so it's a CI problem."
- Marking a test flaky without ever having reproduced it.
- A test file with more skip markers than assertions.
- Debugging the feature when a driver's red has never been repeated by hand.
- Asserting a specific position — especially "the last item" — in a collection built by an async callback, with no real dispatcher or event loop under test.