confirming-before-claiming-done

Overview

"It should work now" and "it works" are different claims. The first is free — it falls out of having made the edit and having a plausible story for why the edit fixes things. The second costs one command: run the actual check, read what it actually printed, and only then use the word "done." Being careful, understanding the bug deeply, or having been right the last ten times does not substitute for that command. It only means the claim hasn't been checked yet.

The rule survives paraphrase. "Should be passing," "looks good," "that should do it," and "I'm fairly confident this is fixed" are the same unverified claim in different clothes. If the check hasn't run since the last change, none of them are earned.

When to use

• About to say a fix, a feature, or a test suite is done, fixed, working, or passing.
• About to commit, push, open a PR, or hand work off on the strength of that claim.
• Reporting what a subagent, a CI run, or a tool said about itself, instead of what you independently checked.
• Moving on to the next task because this one feels finished.
• The thing corrected has a canonical source and copies published or mirrored by hand.
• The only thing behind the claim is that the diff looks right and nothing has actually been executed.
• Not for: judging whether a past decision was actually correct in hindsight (see revalidating-decisions).
• calibrating-confidence is the adjacent concern: marking your own uncertainty honestly as verified, inferred, or assumed. This skill is the concrete act that earns the verified label in the first place — running the check before the claim.

Name the check, then run it

Before typing the claim, name the exact command whose output would prove it. No candidate command means the claim isn't checkable yet — say that, instead of skipping straight to the confident version.

Run it in full: not the fast subset, not just the file that changed, the whole thing the claim is actually about. Read the whole output, not just the line that flatters the claim — a summary can say "34 passed" while the same output shows a separate failure the summary doesn't count. Then check what the output actually says against what the sentence is about to claim: a build that compiles is not a test suite that passes, and a linter with zero complaints hasn't touched runtime behavior at all.

Only once that comparison holds does the claim get made — and it gets made with the evidence next to it, not implied. "34/34 tests pass" carries different information than "tests pass."

One mechanical way to lose that output is worth naming on its own, because it costs nothing to hit and leaves no trace when you do. Verification commands are noisy — a build, a full suite, a deploy — so the natural move is to pipe one into a filter and read only what matters. A shell pipeline reports the exit status of its last stage, which is now the filter. The build fails, tail succeeds, set -e sees nothing wrong, and a && echo "OK" chained onto the end prints OK. The step that made the output readable is the step that threw the verdict away, and what remains on screen is a plausible-looking excerpt of a failed run with a success line under it. portable-shell-scripting has the mechanics and the portability caveats; the discipline here is that nothing standing between a check and your reading of it may be able to swallow the answer — so either keep the status explicitly, or verify the artifact the command was supposed to produce rather than the pipeline that ran it — with the caveat the next section owes it, which is that the artifact was very likely already there.

Keeping the status explicitly repairs the verdict and does nothing for the evidence, which is the second half of the same loss. head and tail choose what to discard by position, and which end holds what you need depends on the claim you will end up making — which is not chosen yet. A failure prints last, so head throws it away. The lines that support a success print early — the gate that passed, the input the run resolved, the version it selected, the branch it took — so tail throws those away instead, and keeps the progress spam a long build spends its final minutes on. Either way the survivors read as complete. Nothing in an excerpt marks it as an excerpt.

The cost lands at report time, far enough from the decision that it is rarely traced back to it: you reach for the line that supports the sentence you are writing, it scrolled off, and the exit code is all that is left to argue from. "It exited 0" and "I watched it succeed" are different claims, and only the second one requires having kept the output. So capture first and filter second. cmd > run.log 2>&1 puts none of the run in context, and a grep against that file afterwards answers whichever question turns out to matter — the version it resolved, the check that gated the step, the one warning that was not boilerplate — as many times as you need to ask, whereas a capped pipe answers a single question you had to choose before knowing which one it would be. The same holds anywhere a long output reaches you through a bounded window: a scrollback buffer, a job page that renders only a run's last N lines. Never truncate by position a record you will later have to quote from — filter it by pattern instead.

When the check itself is unreliable — an intermittent bug that only reproduces some fraction of the time — a single clean run doesn't carry the same weight it would for a deterministic one. That's a reason to run it enough times to get real signal, or to report status honestly as still-in-progress, not a reason to fall back to a hedged claim instead. "Should be fixed, let me know if you still see it" spends the same unearned confidence a flat "it's fixed" would; softer wording doesn't make one weak attempt add up to evidence.

The claim and what actually proves it

Claim	Real evidence	Doesn't count
Tests pass	This session's run, fresh, zero failures	A run from before the last edit; "should still pass"
Lint is clean	This session's run, zero warnings	Spot-checking only the file you just touched
Build succeeds	Fresh build, exit code checked	Lint passing; no red squiggles in the editor
It starts up	Started the way it will actually be started, watched past the point the framework calls the hook	It compiles and the suite is green; the same function passes when a test calls it directly
It is live	A signed-out, cache-busted fetch of the published URL	It renders in your authoring session; the upload exited 0
The correction is published, not just made	Every copy enumerated from the host, and each one's bytes fetched	The source was corrected and the record says resolved
It is not installed	The whole target tree searched for the artifact's own name; the running process's open files and loaded modules read	The one location you knew to check is untouched
The encoded form is this	One real value put through the real encoder, and the output printed	The type's declaration read, annotations and all; a grep for the spelling
A bug is fixed	Reproduced the original symptom on the new code, and it's gone	The diff looks like the right fix
A regression test guards it	Red on the old code, green on the fix, both watched	Passes once, never run against the broken version
It comes back after a restart	A real restart, then the start time within seconds of boot	It is running; the registration reported success
It works from a clean checkout	A clone into an empty directory, built there	It builds in the tree you have been working in
It works on the target	The check run on the target itself	A green run on an emulator, container, or staging stand-in
The backup is good	A restore performed from it	The backup job exited 0 and the file is the right size
A subagent finished the task	The diff it actually produced, read	Its own summary of what it did
Requirements are met	Checked line by line against the spec	The tests pass, so it must be done

Evidence goes stale the instant code moves

A test run is a claim about the exact code that existed the moment it ran. Change one more line afterward — even a line that "shouldn't touch" the part under test — and the run now describes a version of the code that no longer exists. "It passed ten minutes ago" and "it passes" stop being the same sentence the instant anything lands in between.

This is what makes "I fixed it" and "I confirmed the fix" different acts, not just different phrasings. Declaring something fixed the moment the edit is typed, without re-running the check against the post-edit code, fails for the identical reason a stale test run fails: the evidence on offer was gathered before the thing it's supposed to prove even existed.

Evidence does not travel to another environment

Evidence expires along a second axis, and this one fails far more quietly than time does, because nothing about the run looks weakened. The command was named, run in full, and read; the output says everything passed. It simply ran somewhere that is not the place the claim is about — an emulator rather than the device, a container rather than the host, a staging tenant rather than production, your operating system rather than the one it ships on.

A stand-in is not a smaller version of the real thing. It is the real thing minus a particular set of services, it does not publish the list, and everything that does not need what is absent behaves perfectly — which is why the result comes back green rather than flaky. routing-around-capability-gaps makes that argument for whoever chose the stand-in in order to get unblocked. It holds unchanged for whoever merely inherited one as the ordinary place work runs, and that reader is the likelier of the two to be caught by it, because they never made a decision they could think back to.

So the environment travels with the claim, or the claim is wrong: an unqualified "it passes" is read as a claim about the real place, whatever you privately meant by it. "Passing" and "passing on the emulator" are different sentences, and the second is not a hedge — it is the measurement, stated. Where the requirement names a place, the evidence has to come from that place: the real device, when what is claimed involves a driver, a sensor, a permission model, or a display; the real operating system, when it involves a path, a signal, a file lock, or line endings; the real deployment, when it involves scale, latency, or a neighbour's load.

An acceptance line closed on a stand-in is therefore marked with where it was proven rather than with done, and downgrading one already written as verified is the ordinary move at that moment rather than an embarrassment — finishing-what-you-started covers surrendering a criterion visibly instead of quietly deleting it.

A report is not a check

A subagent reporting success, a CI badge sitting green, a teammate saying it should be fine — none of these are verification, they're claims, and repeating one as your own confirmed status launders someone else's unchecked belief into something that sounds checked. Read the diff the subagent actually produced instead of its summary of the diff. Open the CI log instead of trusting the badge. Run the command yourself instead of describing having run it. The report may well be accurate — that's a separate fact from whether it's been checked.

A green you produced by moving the check is not a pass

The check is part of what changed. A run that has been trying to turn a suite green has two ways to succeed — change the code until the tests pass, or change what the tests demand — and from the outside the second looks exactly like the first: same command, same exit code, same word in the report. It is also the one taken most often when the honest route is hard. Benchmarks built in 2025 and 2026 to catch it found some capable coding agents doing it in most runs where the tests could not legitimately pass, and agents passing every visible test while failing held-out ones at a rate that grew with the size of the code. Making the test files read-only stopped edits to them; the special-casing moved into the code.

So before the claim, diff the checks as well as the code, against where the work started, and look for the moves that make a check pass without the behavior:

• A test deleted, skipped, marked expected-to-fail, or renamed so the runner no longer collects it.
• An assertion loosened: equality to containment, an exact value to a tolerance, a specific error to any error, a count to "not empty".
• An expected value edited to whatever the code now prints.
• A branch in the code keyed on a literal that appears in the test's input, or a return value that is simply the test's expected answer.
• Equality or comparison overridden so that everything compares equal.
• An exception caught and discarded on the path the test was there to exercise.
• The unit under test replaced by a double, so the test checks the double.
• Suite configuration changed: files excluded, a coverage or score threshold lowered, a job no longer run, a timeout raised until the slow path fits.

Any of these can be right — a requirement did change, a test really was wrong. Then it is a finding to report, with its reason, not an edit to fold into "tests pass". A test that looks wrong is reported, not rewritten, by the run that is trying to pass it — unless the task itself changed what the test checks, and then the test changes first, is watched failing, and the change is named in the report. finishing-what-you-started covers surrendering a criterion openly rather than editing it away. And one input the tests never named — a property generated rather than listed (property-based-testing) — is the cheapest held-out check there is: code that learned the test cases fails it at once.

The same diff is the first thing to read in a subagent's work, before its summary: what it did to the tests says more about whether it finished than anything it says about the code.

The suite is not the contract

Where code has to satisfy something defined elsewhere — a wire protocol, a file format another tool reads, the shape a caller expects back, a vendor's API — a test suite can only compare the codebase against itself. It will confirm that the manifest parses, that the version matches everywhere it is written down, that the process exits 0, that the two registries agree. Every one of those can be true while the thing on the other side receives nothing it recognises, because not one of them ever consulted the document that says what it expects. The suite proves consistency, and correctness is not a fact about consistency.

The tell is mechanical, and worth running over any suite that guards an integration: read the assertions and ask which of them would fail if the other side changed its mind. If none would, the suite cannot fail for the reason that matters most — and its green is not weak evidence, it is evidence about a different claim. It will go on meaning "these parts agree with each other" for as long as they do, which is not the question.

Two things close that gap, and neither is a bigger suite. Exercise the real other side wherever you can: one round trip against the actual consumer settles what a hundred internal assertions cannot. Where you cannot, because there is no licence or no hardware or no account, make the test carry the external fact rather than a paraphrase of it — pin the exact shape the other side documents, and cite where that came from in the test's own name or docstring. A checked fact and a copied assumption are indistinguishable six months later, and which one it is decides whether the next person can trust it or has to re-derive it. Where the other side documents nothing at all, that is worth writing down too: say in the test that the shape is a guess, and prefer the guess whose failure is survivable.

The exit code is the most persuasive form this takes, and deserves naming on its own. A process that ran and returned 0 has told you it did not crash. It has said nothing about whether anything on the other end read what it wrote, and a consumer that silently drops a shape it does not recognise is the ordinary case rather than the exotic one. An integration that runs is not an integration that arrives.

The push is not the reach

Where a check ran is part of what it proves. Confirming a published thing from the seat that published it — the authoring session, the signed-in browser, the tool that did the upload — establishes that the artifact exists and that you, specifically, can reach it; the claim is that its audience can. The two contexts differ along axes invisible from inside the authoring one, authentication and caching chief among them, and the two failures they produce run in opposite directions.

A default-private artifact is indistinguishable from a public one at the owner's seat, so a page that renders perfectly for its author returns nothing to anyone else — worse once the URL has already gone out as public. A stale cache is the mirror image: it serves the previous version after a genuinely successful deploy, so a correct check reads as a failure and invites a pointless re-push. One makes a broken thing look fine, the other makes a fine thing look broken, and a signed-out, cache-busted fetch from outside the publishing tool settles both.

Both of those assume a push happened at all. Where the deliverable has a canonical source and copies kept in step by hand — a document mirrored onto a site, a notice repeated in a store listing, a policy filed with a registrar or an authority — "fixed" is a claim about the source and nothing else. The edit is real, the review is real, the ticket closes, and every published copy goes on serving the old text, because no act of publishing was ever part of the fix. Nothing detects that: there is no failing check, the source reads correctly to anyone who opens it, and the record says resolved. A record saying resolved is evidence about an intent, not about a deployed state — so the check has to land on the bytes a consumer is actually served.

Which copies, then, is a question memory cannot answer. The number you can name is the number you knew about the last time you looked, which makes it a floor and not a count — and the copy nobody remembers is, necessarily, the copy nobody updates. A look is not a search, below, is the move: it is written for claims of absence and holds here unchanged. Enumerate from the side that does the serving — what the host actually publishes, what the registry or the store lists, what a search for the document's own title returns — rather than from a note or a recollection, and fetch each one. writing-durable-docs's A mirror with no generation step is a fork owns the repair: stop hand-mirroring, and generate one copy from the other or diff them in CI.

An artifact is not this run's artifact

The remedy above sends you to the artifact instead of the pipeline, and on its own it swaps one weak check for another: an artifact is usually already there from last time. A build directory holds the previous binary, a report holds yesterday's numbers, and neither goes missing when a step fails to run. Presence proves the step ran at some point in the artifact's history — a claim nobody was making.

The artifact is stale. A step never ran — wrong directory, missing dependency, an error the output filter swallowed — and the previous artifact sat there answering for it. Everything downstream verifies the old thing and reports a pass, and the log it emits reads exactly like a fresh one, because it is a real log of a real earlier run. The check is freshness: a modification time later than the change, or an identifier the run stamped in that the previous artifact could not be carrying. There is also a tell worth knowing on its own, because it fires before you think to check anything — a verification that finished faster than the work it claims to have done.

The artifact is fresh and came from somewhere else. This is the one a digest cannot see. A hash answers did these bytes arrive intact and is silent on are these the bytes I meant: a correct hash over an artifact built from the wrong source passes every check in the chain, because each stage compares against the one before it and none of them compares against the change you made. Integrity and identity are different questions, and only the first is being asked. The check is a positive probe — search the artifact for content unique to the source you expect produced it, a string or symbol or constant the change itself introduced. Finding it is a claim no hash can make.

Build the probe from long, distinctive content, because one built from short content fails open. Short strings get folded into the surrounding code and leave no searchable trace, so a wrong build sails through a probe made of them, silently and in the costliest direction. This is the mirror of the rule that a negative result from an artifact proves nothing: there absence was uninformative, here absence is the whole signal — so the probe has to be built from content substantial enough that its absence means something.

The capture is fresh and shows somebody else's surface. A screenshot or recording taken from a shared instrument — a test phone, a browser profile, a display, a staging box — is the same case in a different medium: it arrived on time, it is real, and it can be a correct picture of whatever owned the surface at that instant, because another session, a dialog, or a hand on the device can take the foreground between two steps of a scripted pass. The positive probe here is the owner: read what had the foreground at capture time, record it beside the capture, and treat a capture without that line as an artifact of unknown source. formidable's audit reference carries the addressing rule — capture by window id, package, or tab, never by whatever has focus.

The capture predates the change it was taken to show. A screenshot says what was last composited, not what the layout or the state is, and the compositor keeps its own clock. Taken within a frame or two of a resize, a navigation, or a click, it can hand back the frame from before the reflow: content filling two-thirds of a window whose every element measures full width. It can also be stale by region, with the content area already showing the new tab while the tab strip still highlights the old one. When the claim is about geometry or state, measure that property directly (element bounds, the active attribute) and let the picture corroborate. When a picture disagrees with a measurement taken in the same second, suspect the picture first, and capture again once the surface has settled before calling anything a defect.

The run is not the return

Some requirements are not about the present at all. Comes back after a restart. Survives a cold cache. Works from a fresh clone. Persists across a session. Recovers when the network drops. Each of those is a claim about a condition that has not occurred yet — and the check that is conveniently available almost always measures the present instead: it is running, it answers, it is registered to start, the cache has entries in it.

Those checks pass. They would also pass, identically, in the world where the requirement is entirely unmet. That is the whole problem, and it is invisible from inside the check, because nothing about a green result advertises which of the two worlds produced it.

So put the check to a specific question before trusting it: would this still pass if the condition the requirement names had never once occurred? If the answer is yes, what you are holding is a proxy, and no amount of re-running it converts it into proof. A service started by hand reports itself running and answers on its port exactly as convincingly as one the machine brought up by itself, for as long as nobody restarts the machine.

The fix is not another check of the same kind. It is to induce the condition once — to actually make the thing happen, deliberately, in a window you choose: reboot the host, delete the cache, clone into an empty directory, kill the process, pull the network. That costs a real disruption, which is why it keeps being deferred — but the condition arrives on its own schedule eventually, and that schedule is reliably worse than the one you would have picked.

Which makes inducing it a decision that is not yours alone to take. On anything shared, live, or depended on by someone else, the disruption is the whole cost of the proof, so it needs the owner's go-ahead and a window agreed with them rather than the next convenient gap — and deciding-reversibility is the call to make first, because a restart you cannot undo from where you are sitting is a different act from one you can. Being unable to get that go-ahead is not a reason to fall back on the proxy and call it proven. It is a reason to say plainly that the requirement is configured but unverified, and to name the one test that would settle it.

Two things read the result more honestly than any status string. Timestamps say who caused something; a status string only says that it is so. A service's start time sitting seconds after the host's boot time means the machine started it; the same service reporting itself healthy with a start time hours after boot means a person did, and the boot path has never been exercised at all. And a restart counter separates coming up cleanly from being caught and retried — a supervisor configured to restart on failure makes a binary that crashes on every start look permanently healthy from outside, so the count of zero is part of the evidence, not a detail.

Away from services the shape is the same one every time: something was configured to happen later, and the configuring got recorded as the happening. A setting that says a thing will occur is a plan, and the only evidence it was a correct plan is the thing having occurred once. A backup is not a backup until a restore has been performed from it — the job exiting 0 attests to a file being written, which is a different claim entirely.

A look is not a search

Not every claim is that something is there. It is not installed. The patch never landed. That hook is not registered. The migration never ran. A check can be fresh, first-hand, complete, and run from exactly the right seat and still answer a narrower question than the one asked — because the obvious way to check an absence is to open the one place the thing would live, and finding that place untouched establishes one fact about one location. The claim being made is about the whole target.

The failure is dangerous precisely because it feels like rigor: a real command, a real file, evidence gathered first-hand a minute ago, and a wrong answer at the end of it. What keeps the wrong location plausible is that it genuinely was the right one at some point — the thing being looked for moved, and the old location stayed behind, present and untouched, because nothing writes there any more. Installers relocate their hook between major versions; a patch targets a path that has since been renamed; a migration records itself in a ledger the current version no longer writes. The check has not become unreliable, which would at least be visible. It returns the same clean answer whether the thing is installed or not — the proxy problem one section up, moved from the moment a check runs to the place it points at.

A negative claim needs a different move: search the target surface for the artifact itself, rather than interrogating the one location you can name. Sweep the whole target tree for the artifact's own name, which is cheap and is what finds a relocated install. Ask the running process what it actually has open and loaded, the only one of these that separates present-on-disk from actually in effect. Read which candidate the runtime's resolution order really selects, rather than the path you remember it using. A location you know about settles the question the moment the thing turns up in it and can never settle it the other way, so an absence is only ever earned by a search — and the claim should name what was searched, because that is the part a reader can check.

That search can be defeated by the surface rather than by the location, and that failure survives getting the location right. A built artifact is a lossy encoding of the source it came from. An optimizer is free to materialize a short string as two immediate operands the program assembles at runtime rather than as contiguous bytes; a minifier renames the identifier being searched for; a stripped build keeps no symbol to match; a packed or compressed payload preserves none of it. Searching one of those for a short literal and finding nothing is not weak evidence of absence — it is no evidence at all, and it looks exactly like the code having been eliminated, which is the conclusion it will be used to support. This is not the unreliable-checker problem under The declaration is not the payload: the pattern is correct, the command is correct, and the artifact simply does not contain the thing in the form searched for. Probe with something the transformation has to preserve — a long literal, a message the program prints, a symbol or export table, a version banner — or, where the question is whether a code path shipped at all, make the program take it and watch.

The same fact runs the other way and is easier to miss, because it arrives as good news. If the artifact does not preserve the form you searched for, then a hit in it is not evidence of presence either — something else produced those bytes, and a short pattern can occur by coincidence in a large one. The asymmetry people apply here is backwards: a miss gets treated as suspicious and a hit gets treated as settled, when both are claims about a surface that does not preserve the thing being asked about. Before a hit counts, say what produced it — which section, which surrounding context, which offset — or find a second, longer pattern that must co-occur with it. A hit whose source cannot be named is the same non-result as a miss, and it will be cited as proof. The lossy surface is the sharpest case of a general one: calibrating-confidence's A search result is the input to a check, not the check owns the ordinary version, where the surface preserves everything and the hits still go unread.

The call site is not the context

The usual complaint about a green build is that nothing actually ran. This failure survives the fix for that, because a test that runs the code does not necessarily run it where the code will run. Anything placed inside a hook a framework calls — a setup or init function, a registration callback, a plugin entry point, an installer's post-install step — is written and compiled as ordinary code and does not execute under the conditions the rest of the program executes under. It runs at the moment and in the surroundings the framework picked: possibly before the async runtime is up, before there is a window to draw into, on a thread that does not own the thing the line touches, or before configuration the identical line would have found in the program's entry point has been read.

Nothing at the call site carries any of that. The hook's signature looks like every other function's, so no static check can see the difference — and neither can a test that calls the hook directly, because calling it supplies the test's surroundings instead of the framework's. That is what makes this failure quiet: the suite is not wrong about what it measured, it measured a context the program will never be in. Compile, typecheck and every last test can be green on something that fails on every single start.

There is a tell, and it costs nothing to look for. If the framework exports its own version of the thing you were about to call — its own spawn, its own timer, its own way onto the main thread, its own handle to the running application — that wrapper exists because the general-purpose one does not work here. Reaching past that wrapper is the shape of this bug, and the only check that observes the framework's context is the expensive one it was tempting to skip: start the program the way it will actually be started, and watch it get past the point the hook runs.

The safe mode is not a rehearsal of the real one

A dry run, a simulation, a paper mode, a --no-op flag: the same code path, run with the consequences switched off, so it can be exercised without cost. The implicit claim is that everything except the final effect was genuinely performed. That claim is almost never audited, and where it fails it fails silently and for months, because the safe mode's whole job is to produce a clean result.

Two divergences account for most of it, and they are different sizes of the same question — where is the branch?

• The branch is too early. The safe mode synthesizes its answer before the work it was supposed to rehearse is constructed at all. Nothing downstream of that point has ever run. A mode that "only changes what happens at the end" frequently changes what happens near the beginning, because that is the cheapest place to implement it.
• The consequences differ, so the outcomes are not comparable. A gate that fails is tolerated in safe mode and terminal in the real one; a retry is free in one and rate-limited in the other; a partial result is discarded here and persisted there. The run completes in both, and only one of them tells you what the other will do.

So the safe mode is worth designing rather than adding: put the branch at the last boundary that actually effects something, and record every gate's outcome identically on both sides of it. A gate that failed for all candidates should be as visible in a simulated run as in a live one; if the simulated run is green where the live one would have stopped, the mode is not conservative, it is uninformative.

And say what it does not reach. A safe mode's coverage claim is bounded by where its branch sits, so the honest form is this exercises everything up to X, and nothing past it has been run — which is a sentence somebody can check, unlike "tested in simulation". Where a system has never once completed end to end in the real mode, the count of defects past that frontier is not low. It is unknown, and those are different claims.

The harness is not the product's path

A problem gets investigated with something written for the investigation — a script, a probe, a diagnostic command, a one-off runner that reproduces the complaint. It imports the product's real code, which is what makes the next step feel safe: the harness now shows the problem cleared, so the problem is fixed. What it has actually shown is that the data exists and that some arrangement of the product's own parts produces the right answer. Whether the arrangement a user's action goes through is that one is a separate question, and a green harness does not touch it.

It survives every check aimed at a green build that never ran, because both halves are real code out of the same repository. A field missing at one caller gets fixed in the parser, confirmed in the harness, and stays missing in the product — the harness reached it through a merge helper written for the harness, while the path a user takes calls an entry point that merges nothing. Every test passes. The product, exercised the way a user exercises it, still shows the bug.

So before the claim: name the entry point the user's action reaches, and confirm the fix sits on that path, not merely inside a module that path imports. Two tells that this check is owed, and both are cheap to notice — the evidence comes from a file that did not exist before the investigation started, and nothing in the verification ever launched the product the way a user launches it. Where a helper genuinely has to serve both, move it into the product and let the harness borrow it rather than the reverse. A helper only the harness owns is a standing gap between what gets tested and what ships.

A name in a build's output directory is not an identification

A build tool's own lock was in the way, so a test suite got launched by finding a plausible-looking file directly: a hash-suffixed binary in the build's output directory, name matching the test target. It ran repeatedly before anyone noticed, because "the tests hung" read as a slow suite rather than as a wrong program — the build tool places the product binary in the same directory under the same naming shape, and the lookup that had always landed on the test binary before landed on the shipped application this time. The tell was sitting in plain view the whole run: a hung "test" whose threads carried GUI names.

The general shape is that a build output directory is not a namespace the artifact's role is encoded into — it is where the tool put things, and more than one kind of thing can match the name you go looking for. Locating an executable by pattern is a guess about which executable it is, dressed as a lookup. Nothing about a clean match tells you it guessed right; a wrong guess returns a real, running, plausible-looking process, not an error.

So ask the build tool which artifact is the harness rather than finding one yourself — most build systems that separate product and test binaries have a command that lists or names the test artifact directly, rather than making you locate it by pattern in their shared output directory. Where that is unavailable, treat a "test run" that opens windows, binds sockets, or polls the network as evidence about which binary ran, not about test flakiness — a unit test doing any of those is the same tell as a hang that never resolves.

The declaration is not the payload

Some questions are about a thing the system produces rather than about whether it works: what key this ends up under, what that enum looks like once it is serialized, what is actually in the column, what the header says. The reflex is to read the code that produces it, and for this class of question the code is the wrong place to look. An attribute or annotation renames and re-cases fields, a custom encoder overrides the declaration entirely, an inherited default supplies a policy written nowhere near the type, and a library changes its own default between releases. The declaration is a request. The bytes are the answer.

So produce one payload and read it. Serialize a single real value and print it; read one stored row; dump the header off one real request. That is usually one command; it costs less than the reading it replaces; and it settles the question outright instead of raising confidence in a guess. Reading the code is a hypothesis; producing the payload is evidence.

There is a second reason, and it is the sharper one. A grep, a regex, or a throwaway script written to answer the question is code that has existed for a minute and has never been tested, and it can be wrong in the direction that costs the most: reporting the absence of what is there. diagnosing-before-fixing covers separating a broken instrument from a refuted hypothesis once a result is in hand, and its discriminator is that a broken apparatus usually fails in several ways at once. A pattern that is quietly too narrow fails in exactly one way: it returns nothing, cleanly, and nothing about that looks like a malfunction. So the cheaper move is not to build the checker at all — and where one has already run, treat a negative result from it as unconfirmed until something independent agrees.

A neighbouring number is not a baseline

A claim that something helped — the cache, the index, the retry policy, the compression step, the tuned parameter — is a comparison against a world where it was not applied. That world does not exist in the run being measured, so its number has to be constructed. It is never lying around waiting to be read.

What is lying around is a different number, produced by the same treated run, about roughly the same thing, and sitting close enough to the result to look like the other half of a ratio. Divide by it and the arithmetic works, the units cancel, and the figure that comes out is a confident measurement of nothing. The direction it fails in is the expensive one: it reports "no effect" for an intervention that is working, and the reasonable response to no effect is to remove the intervention.

One question settles it, and it costs nothing to ask before writing the ratio: if the intervention were switched off, would this denominator change? If it would not, it is not a baseline — it is a second reading off the treated run. Construct the real one instead: run the workload with the intervention disabled, replay the same inputs through the previous version, or derive analytically what the untreated result would have been. performance-profiling's interleaved A/B runs are the same discipline applied to timings, and for the same reason.

A number you did not predict is not a check

A gate that reports a figure — tests collected, files changed, rows migrated, warnings emitted — gets read as a check, and on its own it is not one. Whatever it printed, you were going to accept, because you had not committed beforehand to any value that would have looked wrong. The number does the work of evidence while carrying none.

Saying the number first is the whole of the fix, and it costs one sentence before the gate runs: this change adds six checks, so the total should land six above the last measurement. The reading becomes a comparison rather than an observation, and a mismatch is a finding you had no other route to. What a mismatch usually means is that the inputs are not the ones you think you have — a concurrent session writing into the same checkout, a generated file rebuilt underneath you, a dependency update that brought fixtures of its own. None of it turns the run red. The summary line is green, and nothing in any output you were already reading mentions it. Ask for both halves of what moved separately: git status for whatever is sitting uncommitted in the tree, and a log since the commit this pass started from for whatever landed already committed.

Two limits, because the move is easy to perform and get nothing from. A match is worth exactly what the arithmetic behind it is worth — two changes that cancel produce one as readily as an undisturbed tree does. And a prediction taken from a record rather than from a measurement, the total the last pass wrote down or the figure quoted in a previous report, is only as fresh as that record: everything that legitimately landed since then surfaces as an unexplained delta, that reads as something being wrong now, and the pass goes off investigating its own stale bookkeeping. Predict from the current state plus the arithmetic of this change, never from the last number anybody recorded. § Evidence goes stale the instant code moves is the same decay one step earlier, about the run rather than about the note describing it.

Where a figure genuinely cannot be predicted — a duration, a byte count, anything carrying real variance — the rule does not soften, it inverts. That number is not a check either, and it belongs in the report as an observation rather than standing behind a claim.

Common mistakes

Symptom	Real cause
"Should pass now" in a commit message or handoff note	The verification step was replaced with confidence in the fix
Tests declared passing based on a run from before the last edit	Evidence treated as durable when it's only valid for the code it ran against
"The agent said it completed the task" reported as the task being complete	A tool's self-report repeated as independently checked fact
A command "passed" and the artifact it should have produced is untouched	Its verdict was read from a pipeline whose last stage was a filter, not from the command
At report time the only thing left behind a claim is the exit code	The run's output was capped by position, so the lines it would have been argued from were the discarded ones
A search of the built artifact says the feature never shipped	The artifact does not store what was searched for; the pattern was sound and the surface was not
The suite went green and the behavior still fails on an input nobody listed	The run changed what the tests demand rather than what the code does — an assertion loosened, a case skipped, a branch on the test's own literal
A regression test added and trusted without ever seeing it fail	Never run against the broken code, so it's unknown whether it tests anything
Build green, shipped, runtime error in the first minute	Compilation was checked; behavior never was
"Looks right" standing in for "ran and confirmed"	Review of your own diff mistaken for verification of its behavior
One failing test out of many waved off as unrelated	A partial pass rate treated as a pass
Everything reports healthy for months, then nothing comes back after a power cut	Liveness checked repeatedly; durability never once induced
"It is set to start automatically" offered as evidence that it starts automatically	Configuration read as behavior, with nothing having exercised it
A correction closed weeks ago, and the published copy still serves the old text	The fix landed in the source; publishing the copies was never part of it
An improvement measured at "no change" and abandoned, then found later to have worked	The denominator of the ratio was another quantity from the treated run, so switching the intervention off would not have moved it
Another session's work shipped inside your green run and nobody noticed	The suite's total was read rather than predicted, so no value it could have printed would have looked wrong

Red flags

• "Should," "probably," "looks like it," or "I'm fairly confident" appearing anywhere near a completion claim.
• Satisfaction expressed — "great," "done," "that's it" — before the check has run in this message.
• About to commit, push, or open a PR, and the last thing that ran was the edit, not a check.
• A claim that would change if reworded, because the wording was doing the work the evidence should be doing.
• "It's probably fine, I'm confident in the fix" as a reason to skip the command that would confirm it.
• Tired, near the end of a long task, and tempted to call it done to stop working rather than because it's verified.
• A requirement worded with "after," "across," "survives," or "from scratch," answered by a command that only observes right now.
• "It's enabled, so it'll come back" — enabling is the plan; coming back is the thing being claimed.
• "It is not in the built artifact, so it never shipped" — said after grepping a release build for a short string.
• Reluctance to induce the condition because it would be disruptive, on a system where the condition will occur anyway, unattended.
• A before/after ratio whose two terms were both produced by the "after" run.
• "Fixed" said about something with published copies, on the evidence of the source file and a remembered count of the copies.
• A success line printed by something other than the command being verified — an echo after a pipeline, a summary the runner emits regardless.
• A long check launched through | tail -n NN, with a report that will have to quote from its output.
• "Tests pass" reported by a run whose diff touches the tests, and the report does not say how.