choosing-test-scope

Overview

Test each behavior at the highest layer that can fail for exactly one reason. Higher than that and a red test names no cause; lower than that and a green test proves nothing about the shipped system.

When to use

Adding coverage and unsure whether it belongs in a unit, integration, contract, or end-to-end test.
The end-to-end suite takes long enough that people rerun it instead of reading it.
A defect reached production while every layer of tests was green.
A double has grown enough conditional logic that it is a second implementation of a dependency.
Tests break during a refactor that changed no observable behavior.
Someone is proposing a coverage percentage or a pyramid ratio as a target.
Not for: how to make a chosen layer deterministic — that is testing-the-untestable.
Not for: output from a language model, whose correctness is a rate rather than a verdict — that is evaluating-llm-output.

What each layer alone can catch

Layer	Only it catches	Typical cost	When red, it points at	Survives internal refactor
Unit (in process, no I/O)	Branch logic, edge cases, arithmetic, error paths hard to trigger from outside	<10ms	One function	Only if it tests the module's boundary, not its internals
Integration (your code plus one real dependency)	Query dialect, migrations, transaction and isolation behavior, serialization, driver semantics, index and collation effects	10ms–2s	One seam	Yes
Contract (consumer expectations verified against the provider)	Schema drift and breaking changes between independently deployed parties	<1s per pair	One party	Yes
End-to-end (whole stack, real transport)	Wiring, configuration, authentication, deployment topology, cross-service ordering	5s–minutes	Nothing specific — diagnosis is the dominant cost	Yes
Manual and exploratory	Qualities nobody can specify: comprehensibility, aesthetics, surprise	Human time	A conversation	n/a

The "cost" column is the smallest part of the real cost. Maintenance and diagnosis time dominate: an end-to-end failure typically consumes a person's afternoon before anyone knows which component moved, and that cost recurs on every flake.

Applying the one-reason rule

Too high if a red test has more than one plausible cause. Split it or move it down until a failure names a component.
Too low if the test could stay green while the shipped system is broken. Every behavior needs at least one test that exercises real wiring, even if only one per deployable.
Match the layer to where the behavior is decided. If uniqueness is enforced by a database constraint, a unit test asserting uniqueness is a fiction about code that does not enforce it. If it is decided in the domain, an integration test for it is slow and redundant.
Pick the layer per behavior, not per file. One class routinely deserves fast unit tests for its branching and one integration test for the query it emits.

Contracts beat integration combinatorics

Cross-testing every consumer against every provider grows as the product of both. Six services that all call each other are 30 directed integration paths; the same coverage as contracts is 6 consumer expectation suites plus 6 provider verifications — 12 runs, each of which can execute without the other side deployed.

Use contract tests when the two sides deploy independently. That independence is the whole reason schema drift can happen.
Skip them when both sides ship as one unit — an integration test is cheaper and catches the same thing.
The contract must be verified against the real provider, not against a shared mock. A mock both sides agree on is a shared belief, not a check, and it drifts with neither side noticing.

Test the boundary you own

Do not test that the HTTP client sets headers, that the ORM emits valid SQL, or that the framework parses configuration. Those have their own suites, and duplicating them makes upgrades harder without making the system safer.

Do test:

Your adapter's translation of the library's errors, timeouts, and edge results into your domain's vocabulary. Nobody else tests this, and it is wrong more often than any other code in the boundary.
One pinned assumption per behavior you would want to hear about on a version bump: that this query returns rows in this order, that this call retries, that this serializer omits nulls. These are the tests that make a dependency upgrade a five-minute job instead of a week.
Configuration you wrote, at the layer that loads it.

Where a double's content comes from

The two-party version is above: a mock both sides agree on is a shared belief, not a check. The one-party version is worse, since no second party exists to disagree — a double and the parser that reads it, written from one document by one person, encode the same misreading and pass forever.

grounding-test-doubles covers fixture provenance and the live test that breaks the tie.

Budgets, and the end-to-end death spiral

Suite	Budget	What breaks past it
Unit	Whole suite under 60s	Stops running on save; feedback moves to CI
Integration	Under 5 minutes	Stops running before push
End-to-end	Under 10–15 minutes, 10–30 scenarios	Nobody reads failures; reruns replace diagnosis

Every end-to-end scenario must be a user-visible journey no other layer can cover — sign in and complete a purchase, not "the validation message reads correctly". The spiral is mechanical: scenarios accumulate, runtime grows, flakes appear, reruns become routine, real failures get rerun away, and the suite is now a tax that catches nothing. It is cheaper to delete two thirds of it than to maintain the version nobody trusts.

Ratios are an output, not a target. A codebase that is mostly glue is legitimately integration-heavy; a codebase with a rich domain model is legitimately unit-heavy. Coverage percentage is a floor detector — it can prove a module has none — and stops being informative as a target above roughly 80%, where teams start writing tests without assertions to reach the number.

Code whose job is to prevent something is tested at its call path

The table puts wiring in the end-to-end row, and for ordinary code that is unremarkable: an unwired feature does nothing, somebody notices within the day, because the thing they asked for is visibly missing.

Guard code inverts it. A redaction step, a permission check, an input validator, a rate limiter — when one of those is not wired in, nothing is missing. The output still appears, the request still completes, and the check simply never ran. A thorough unit suite over the function goes on passing, and it never could have said otherwise: "is anything calling this" is a question about the caller, and no quantity of coverage of the callee answers a question about its callers.

The failure also tends to err safe, which removes the last mechanism that would have surfaced it. A stripping step that runs on an empty input strips nothing and lets nothing through either; a filter that is never consulted, in a pipeline that defaults to rejecting, rejects. Nobody is harmed, so nobody reports it, and the documentation describing the check keeps being true about the code and false about the system. A safety property that is accidentally too strong produces no symptom at all — only the too-weak direction ever complains, which is why an audit that checks one direction finds nothing and concludes the right thing is happening.

Two consequences worth making explicit:

The test that carries the weight feeds a real input the guard should act on through the real call path, and asserts on what the pipeline actually emitted. The unit tests stay; they stop being the evidence.
When auditing a safety property, deliberately check whether it is too strong as well as whether it is too weak. Both are defects. Only one of them will ever come to find you.

A guard that matches on a value has one more place to be silently inert. A block list, a quota, a tenant rule or a per-user limit keys on an identifier — a header, a token claim, a field — and the test that drives it sets that identifier by hand. The hand-set value is the one step production does not take. The real callers may send something else under that name, or something else entirely: their own installation's key where the guard holds the end user's, a proxy's address where it expects the client's, an internal id where the list holds the public one. The test goes green and the guard matches nothing real, failing in the permissive direction, which looks exactly like an empty list. The call-path test above does not catch it, because the path it drives starts at the guard's door, after the step that decides what the key is.

So trace the key backwards from the check to the code that sets it in production, and name that code. If the test supplies the value itself, add a test of that caller proving it sends this key under this name — or have the guard's test consume what the caller's test captured, so one artifact crosses both. Each end of the join needs its own evidence, and neither proves the two meet.

A setup too small to show the difference has not tested for it

Layer is one half of scope. The other is whether the setup you run the check in can express the difference between passing and failing, and a setup that cannot produces a green result that means nothing — not a weak signal, no signal.

It is not the same failure as testing at the wrong layer, and it survives getting the layer right. The check runs where it should, exercises what it should, reads the value it should, and the value it reads would have been the same either way:

A layout measured in a frame barely larger than the element, where "centred" and "pinned to a corner" differ by a few pixels and both look correct.
A concurrency test at a parallelism the race needs more of.
A truncation test whose fixture fits.
A pagination test with one page of data.
A timeout test whose operation returns before the timeout could fire.
A cache test whose run is shorter than the TTL.

The question that catches all of them, asked while the setup is being written rather than after it goes green: what would the broken version have measured here? If the honest answer is "about the same", the number is a property of the setup and not of the code, and the setup has to grow before the result is worth reading.

Getting this wrong is expensive in a particular way: it produces confident evidence. A green check in an under-sized setup is indistinguishable, from the outside, from a green check that proves something — so it does not get revisited, and the defect ships behind a passing test that everyone believes.

The same question is owed to a guard running against real traffic, and it is almost never asked of one, because a rule that always passes reads as a clean population rather than as an inert rule. This is the second of the two ways a guard can be silently absent, and it is not the one further up: there the check was never called, here it is called on every request and cannot reach a verdict other than the one it always gives. An audit looking for the first finds a live call path and stops. A validator, a policy check, or a safety ruleset carried from the environment it was written for into a different one can land somewhere its condition cannot be constructed at all — the field it keys on is never populated, the shape it rejects is unreachable here, the identifier it matches is formatted differently. It then returns the same verdict on every record, forever, and is credited with protecting the thing it is structurally incapable of noticing. A check whose output has never varied across a real population is unproven, not passing. Sample the distribution of its verdicts rather than their value, confirm some adjacent field does vary so you know the input reached it at all, and treat any check ported into a new environment as inert until it has been seen to discriminate there.

When a bug escapes every layer

For each defect that reached production, ask one question: what is the lowest layer that could have caught this? Add the test there, and only there.

If the honest answer is "only end-to-end", the behavior exists solely in the wiring, and that is a design finding rather than a testing one — extract a seam so the decision lives somewhere addressable. Answering the question with "add another end-to-end scenario" every time is how the end-to-end suite reaches the size at which it stops working.

An exclusion names a mechanism, not a justification

Every suite grows a list of things deliberately left out — skipped tests, ignored lint rules, allowlisted inputs, rows marked not-applicable — and each entry carries a reason. The reason is what decides whether the list stays honest, and there are two kinds that read identically at the moment of writing.

A justification says why leaving it out was reasonable. A mechanism says what makes the case unreachable. The difference only shows up later: a justification stays true after the thing it justified stops being true, so the entry survives the change that should have deleted it, and the exclusion quietly starts hiding a live defect.

Kind	Example	What happens when the world changes
Justification	"not applicable here", "covered elsewhere", "this path is theoretical"	Still reads as true. Nothing prompts a re-check
Mechanism	"the harness constructs the other implementation, which never reads this field"	Becomes visibly false the moment the harness changes

The test for an entry you are about to write: state the reason so that it would become false if the case became reachable. If no edit to the system could falsify the sentence, it is a justification and it will outlive its own truth.

This is enforceable rather than merely advisable — require every entry's reason to name a concrete component, and reject the ones that only name a feeling about the case. And when one entry in a list turns out to be wrong this way, read the rest of the list rather than fixing the one: the entries were written by the same person on the same day under the same misunderstanding, and they fail in batches.

That enforcement is worth having and it is not sufficient — worth knowing before you lean on it. Requiring the reason to name a component makes an entry well-formed; it does not make it true, and the two come apart the moment the named component changes. A list whose every reason named a real mechanism — "the generator derives all five of these flags from one dirty bit" — stayed green straight through the change that gave each flag an independent draw, because the check read the sentence's shape and nothing evaluated it. Five entries were false and the suite went on reporting a healthy list, inside the module written to prevent exactly this.

So stop storing the reason as prose. An exclusion is a claim that perturbing this thing cannot change the outcome, which means it is executable: store the perturbation beside the sentence and let the suite apply it — mutate the excluded field, run, assert the result is unchanged. A stale reason becomes a red test rather than a paragraph nobody re-reads. It costs a few lines per entry and it pays on the first run: the same list, once its claims were executable, went red on six entries where the change that prompted the work had predicted five — and the sixth sat on the most reachable field in the whole configuration.

Rewording the false entries instead is the move to refuse. It restores the appearance of correctness and leaves intact the thing that let them rot.

Automating that check is worth doing and audits half the entry. A guard that perturbs each excluded case and asserts nothing changes will find the entries that have gone stale, in batches, on its first run — but what it verifies is the claim, not the explanation. An entry whose reason names entirely the wrong component passes forever, so long as the case really is unreachable for some other reason.

That is a worse position than it sounds, because of what the guard removes. Before it, a wrong reason and a wrong exclusion tended to fail together, and the second one was eventually noticeable. After it, the reason is the only part of the entry that can still be wrong, and nothing is looking at it — while the green run reads as coverage of both halves. The reason is also the half that gets acted on: it is what the next reader consults when deciding whether the entry still applies, and they will not re-derive it from the assertion.

So the guard buys permanence for the list, not correctness for its prose. The cheap repair is to make the reason findable from the thing it talks about: require it to name its component by the identifier the codebase actually uses, so that anyone grepping for that component lands on the exclusion alongside the code. Then a sentence that is true of a neighbouring component — the one shape that survives both the reader and the test — gets read by the person touching the component it wrongly names.

An accumulator is checked against a second derivation

Sometimes the honest answer is "no layer", because the escape was never about layers. A systematic offset — a per-unit figure that omits a fee the total includes, a rate applied at the wrong precision throughout — makes every individual number wrong in the same way and every individual number plausible, and anything that accumulates carries that error into every contribution it adds up. A test checking one of those numbers passes whenever the expectation holds the same mistake, and the expectation usually does, because it came from the same reading of the same specification by the same person. This is the one-party problem from "Where a double's content comes from" arriving as arithmetic rather than as a fixture: one belief, written down twice, agreeing with itself.

Adding more point-checks does not escape that. A hundred of them agree a hundred times, at whichever layer they run. What breaks the tie is a quantity computed by a different route and asserted against the first: a running total accumulated from individual events against the same figure derived from opening and closing balances; inventory on hand against receipts minus shipments; a queue's processed count against enqueued minus remaining. Neither route is the authority. The assertion is that they agree, and a disagreement names a defect without yet saying which side holds it.

This is cheap and it is normally one test per accumulator — money, inventory, quotas, counters, capacity, anything that adds up over time. The layer follows the one-reason rule as usual, with a condition on top of it: the two routes have to be genuinely independent, so a layer at which both of them run through the function that holds the error buys nothing. That test agrees with itself for the same reason the point-checks did.

Common mistakes

Symptom	Real cause
Green suite, broken deployment	Everything tested with doubles; no layer exercised real wiring or configuration
Every refactor breaks dozens of unit tests	Tests written against internals rather than the module boundary
A double grew conditionals and its own bugs	A real dependency was needed; the double is now an unversioned second implementation
End-to-end failure takes an afternoon to attribute	Behavior tested above the layer that decides it
Contract tests pass, integration breaks	Contracts verified against a shared mock instead of the real provider
Every mocked test passes; the first real call returns nothing	Fixtures were written from a reading of the provider, so the suite tests the reading
A shape change surfaces as an empty result rather than an error	The boundary parses leniently; an unrecognized payload must raise, not yield nothing
Coverage is 90% and bugs still escape	Coverage measures execution, not assertion; branches are run, not checked
Every figure in a report is individually plausible and the total is wrong	Each expectation came from the same understanding of the rules the code did; nothing computed the answer by a second route
A well-tested guard function that nothing calls	Coverage of the callee cannot answer a question about its callers, and a guard that never runs produces no symptom to notice
A block list or limit passes every test and never matches real traffic	The test set the key by hand; the real callers send a different identifier from the one the check reads
Integration suite has thousands of cases	Business-rule permutations tested through the database instead of in the domain
Dependency upgrade needs a week of manual testing	No pinned-assumption tests; the boundary's behavior was never written down
The team reruns CI as a first response	A suite past its wall-clock budget; rerunning is now cheaper than reading

Red flags

"We'll catch it in end-to-end."
"Add a browser test for that validation rule."
"Mock the database, it's faster" — for a behavior the database decides.
"We need 100% coverage."
"Every line item has a test" — said about a total that nothing recomputes independently.
"The redaction/permission/validation function is thoroughly tested" — said without having checked that the pipeline calls it.
A guard's test sets the header, claim or id the guard keys on, and nobody has named the production code that sets it.
"The pyramid says we need more unit tests" — stated with no failing behavior in mind.
Writing a test at a given layer because the harness there was already set up.
"I built the mocks from their reference implementation, so they're accurate."
A fixture never once compared against a real response.
An end-to-end scenario added for every escaped bug, regardless of cause.