checkpointing-long-runs

Overview

A long run does not fail by forgetting. Forgetting would be survivable, because a blank is visible and a blank makes you go and look. It fails by remembering fluently and partially: after a compaction, the account you hold of your own run is coherent, confident, and missing exactly the things that were never restated — the dead end that cost an hour, the ruling that settled an argument, the three files already committed.

So the controller re-dispatches work that is already done, re-explores a path already proven empty, and re-opens a decision already made, all while feeling completely oriented. That is the most expensive mistake in this class of work and the one it is most confident about while making it.

Knowing this is not the hard part, and neither is intending to write things down. What fails is the writing itself: a note taken at the moment it would help is always the least urgent thing available, and the note taken later is taken by the memory it was supposed to guard against. So this skill is mostly about what a written checkpoint has to contain — because a checkpoint with the wrong fields in it costs the same to write and buys nothing.

recovering-agent-context is the receiving end of this problem — what to do when you arrive cold to someone else's half-finished work. This is the sending end: what to leave so that arrival is cheap, whether the arriver is another agent, another session, a person, or you in two hours with a compacted context.

When to use

The work will take more sessions than one, or more context than one.
A sweep is running across enough files that its own progress is state.
The conversation is long enough that a compaction is plausible.
Work is about to be handed to a subagent, a colleague, or a later session.
A compaction has already happened and the run must decide what it knows.
Not for: arriving at work someone else started — recovering-agent-context.
Not for: composing what a reviewer needs to judge a finished change — handing-off-for-review.
Not for: a run that is merely unsupervised. That is bounding-autonomous-work, and the two compose: an autonomous campaign needs both.

The rule that makes everything else work

The record is written when the thing happens, not when the run ends.

A record assembled at the end is a record written by whatever survived, which is precisely the memory that cannot be trusted. Every rule below is a specialization of this one.

What one checkpoint has to contain

A checkpoint is not a status update, and the difference is three fields. A line missing any of them is a sentence about the past rather than something the next actor can act on.

Field	What it is	Without it
Anchor	Something outside the conversation — a commit hash, a path, a command and the output it printed	"Task 3 done" is a claim. "Task 3 done, a1b2c3d, 48/48" is a checkpoint. Only one of them survives being doubted
Establishes	What is now known that was not known before, stated narrowly	A line that records activity rather than knowledge, so the next actor cannot tell what they may build on
Leaves open	What this specifically does not settle	The most expensive omission here, and it has its own section below

A negative result has two halves, and only one gets written

"I tried X and it didn't work" is half a finding. It is the half everyone writes, and on its own it is worse than nothing, because it reads as closing a door it did not close.

The missing half is the boundary: what the attempt actually ruled out, and what it left standing. Those are almost never the same size as they look.

Pinning the clock and seeing no change reads as time is not involved. What it establishes is far narrower — the clock in this process is not involved. A cache with its own expiry, a database with its own now(), a broker with its own visibility timeout each keep their own clock, and none of them was pinned. The next actor reads "not a timing issue," skips the whole family, and pays for it a day later.

There is a second boundary that gets lost the same way: what the null actually rests on. A negative measured against an intermittent failure needs enough trials to mean anything at all, and three clean runs against a one-in-six failure rate is not evidence of anything. A negative result that does not say how hard it was tested will be read as conclusive, because that is how a flat statement reads.

So the required form of a dead end is four parts, and it is barely longer than the one part usually written:

Tried:       what was done, concretely enough to not repeat it
Ruled out:   the narrow thing this actually eliminates
Still open:  what it looks like it eliminated and did not
Rests on:    how many trials, against what base rate — or "not measured"

Deprioritized is not eliminated, and the difference belongs in the file rather than in the head of whoever ran it.

What goes in, in order of what it costs to lose

Most runs write the cheap half and lose the expensive half, because the cheap half is the part that feels like progress.

Write	Cost of losing it	Usually written?
Constraints someone stated in conversation — "don't push", "ask me before deploying", "stay out of that directory"	Highest, and uniquely invisible. Losing it does not cost an hour; it means the run does the forbidden thing, with nothing anywhere reporting that the bound is gone	Almost never — it feels like it was already said
Negative results — what was tried and did not work, and why	Highest. The next actor pays the same hour to learn the same thing, and has no way to know they are repeating it	Almost never
Rulings — decisions that could have gone the other way, with the losing option	High. Resurfaces as a re-argued question with nobody remembering the reasons	Rarely
Assumptions — what was settled without an answer	High. Silently becomes a fact, and the fact outlives the run	Rarely
What is genuinely finished, with the evidence	Medium. Causes duplicate work, which is at least visible	Sometimes
What is left to do	Low. Usually recoverable from the plan and the code	Nearly always

The ordering is the point. If a checkpoint has room for one line, it is a negative result, not a status update.

The first row earns a paragraph of its own, because most runs assume its failure cannot happen. A constraint stated in prose is not stored anywhere. Whatever enforces it re-reads it out of the run's own context on every check — so anything that shortens that context can delete the bound while every part of the machinery goes on reporting normally: a compaction, a summarised handoff, a fresh session resuming from a transcript, a subagent that never received the message it was stated in. And because a bound is precisely the thing your own judgment is not allowed to lift, a bound that has silently vanished and a bound you have decided was satisfied are indistinguishable from the inside. Copy every stated constraint into the record verbatim at the first checkpoint, and where the system offers a durable form — a configuration rule, a deny entry, a file the enforcer reads rather than infers — put it there too. A sentence is how a bound gets communicated. It is not how a bound gets kept. And where the bound has to live in prose after all, put it at the top of the file that carries it — every mechanism that shortens context does so from the end, which writing-durable-docs treats as an ordering rule rather than a style one.

Where the state lives

One file, appended to, in whatever the run already uses — the ledger from finishing-what-you-started, the plan document, or the run record from orchestrating-work-end-to-end. A second file competing with the first is two accounts of progress that will disagree, and the disagreement will be discovered at the worst moment.

Three properties make it durable rather than decorative:

Append, never rewrite. A rewritten log agrees with whatever you now believe, which makes it useless for the one job it has.
Anchored to something outside the conversation. A commit hash, a file path, a test command and its output. "Task 3 done" is a claim; "task 3 done, a1b2c3d, 48/48" is a checkpoint.
Readable by someone with no context at all. Names spelled out, paths absolute enough to find, no pronouns pointing at things only this conversation saw.

The seams worth a checkpoint

Not every N minutes. Checkpoint at the places where state becomes expensive to reconstruct:

After anything irreversible or externally visible — a commit, a push, a deploy, a sent message. Before it, the world and the record agree by default; after it, only the record says so.
At every phase or task boundary, including one that failed.
The moment a dead end is confirmed dead. This is the one people skip, because a dead end feels like nothing happened. It is the most valuable line in the file.
Before dispatching anything in parallel, so that a partial fan-out is legible rather than a mystery.
A deliberate pause. Stopping on purpose is a boundary like any other, and the one most often crossed with nothing written. Stop dispatching, then record the resume point, the branch or worktree holding unfinished work, and everything still running or awaiting a human, each with how you would tell it has landed. A "where are we" or "what is still running" from a person mid-run is answered from that record, not from the last notification, since a late completion for a task already handled would otherwise read as live work.
When the context is visibly filling. Not after — the checkpoint written under compaction pressure is written from a memory already degrading.

That last one is a discipline, which makes it the weakest entry on the list: it asks the part of the run least able to judge its own state to notice its own state. Where the harness exposes the boundary as an event — a hook that fires before compaction, at session end, when a subagent returns, when a worktree is removed — wire the checkpoint to it and the discipline becomes a mechanism. Harnesses have accumulated a lot of these, and pre-compaction in particular is the exact moment this section is asking you to remember, offered as a callback that cannot forget. automating-repetition owns the general judgment about when a habit is worth turning into machinery; a checkpoint whose entire purpose is surviving an event the harness already announces is near the top of that list.

After a boundary, read; do not recall

At the far side of a compaction, a session restart, or a handoff, the order of trust is fixed and it is not close:

The repository — git log, git status, the diff, the tests as they run now.
The record — the ledger, the plan's checkboxes, the phase log.
Your own recollection of the run.

Read the first two before acting. A fluent memory of having done something is not evidence of having done it, and this is exactly the situation that produces fluent memories of things that never happened. Re-measure counts rather than restating them; confirming-before-claiming-done owns why a recalled number is not a measured one.

One input belongs with the repository in that order and rarely gets treated as if it did. The document the remaining work is measured against — a specification, a design, a contract, a reference rendering — is an artefact, and a boundary replaces it with a summary of itself, which reads exactly like the spec to whatever consumes it. It is not one. A summary cannot be diffed against an implementation, and a build checked against it diverges structurally — parts in the wrong order, labels in the wrong case, half of each component missing — with a confident rationalisation attached to every divergence. Re-open the source, end to end, before measuring anything against it. Then ask the same question about the run before the boundary: a document that was only ever searched — grepped for identifiers and copy — was never read, so "I know this spec" may have been false from the start. Grep is a lookup, not a read. In the run that taught this, re-reading the source took one command and produced a dozen concrete defects at once, every one of them previously explained away.

The order above says what to trust. What it does not say is that a boundary also strips the tier off everything crossing it. A summary carries claims, not their provenance: the thing you verified by running a command, the thing you inferred from one file, and the thing you assumed because it was plausible all arrive on the far side in the same voice — indistinguishable, and equally fluent. calibrating-confidence calls that confidence laundering at a boundary between two parties. A compaction performs it on you, at a boundary you never noticed crossing, against your own earlier caution.

That is what makes rows 1 and 2 a rule rather than a preference. Anything still checkable gets re-checked, because re-running the command is cheaper than working out which recollection was earned. Anything no longer checkable — a tool output that is gone, an observation about a state that has since moved — is downgraded to assumed on arrival rather than carried at whatever confidence it seems to have. And anything expensive to re-derive gets its tier written into the record while you still hold it, because the record is the one thing a compaction cannot paraphrase.

An obligation crosses that boundary too, and the rule above is wrong for it in a way worth stating on its own. A prohibition, a stop condition, a "check with me before X" loses its provenance exactly like a claim does — but downgrading one to assumed on arrival is not caution, it is the failure itself, because a constraint held at low confidence has already stopped constraining. There is no honest tier for an obligation. It is either in force, in which case it gets re-read from wherever it was written down, or it is gone and the run is now doing something nobody authorized. So obligations are written to the record before the work, and every boundary that re-reads the plan re-reads them as well. bounding-autonomous-work's pre-committed stop conditions are the case that matters most here and the likeliest to be quietly summarized into a general sentence about being careful — which reads like the constraint surviving, and is not.

One thing crossing a boundary carries no tier at all while reading as though it does: a background process started before the boundary and never seen to end. The transcript records that it began, and it has no mechanism for recording an end that happened outside it — so "finished cleanly", "was killed by a teardown, a monitor timeout, or the interface", and "still running" all arrive identically, as silence. Absence of a completion marker is evidence of nothing. Resolve it the way rows 1 and 2 resolve everything else — from a durable thing you can go and look at — by reading what the process was writing to: the output file and its size, the mtime, the lock, the pid. The durable state is the artifact; the transcript only ever recorded the intention to produce one.

A resume restores your record, not the world

The paragraph above found one thing the record cannot hold: a process whose ending happened outside the transcript. That is one instance of a general shape — the record and the world are two different things, saved by two different mechanisms — and the shape has a second failure running the other way, worse than the first for being clean. Here the record crosses intact. A verdict, an approval, a "this one was validated" survives in perfect condition and re-attaches itself to whatever now occupies the slot it named.

The mechanism is that a checkpoint captures your record and nothing else. The thing it was a record about went on moving. So a restored state can hold two halves that are no longer jointly true — a reviewer who approved and has since left, a token verified and since expired, a resource permitted and since deleted — and in the worst case two that were never true at the same instant, a conversation half saying the suite passed against a workspace half predating the patch that was tested. Each half is exactly what was saved. Nothing saved the relation between them.

Three rules, and the first does most of the work:

A recorded verdict names what it judged, never the role it played. "The artifact is clean", "the branch is approved", "the fixture was validated" all re-bind on restore to whatever is currently called that. Anchor it the way the Anchor field above already requires, with one addition: the anchor has to stop being true when the object changes, so a bad re-bind fails loudly instead of succeeding quietly against the wrong thing.
Anything the record asserts about the outside world gets re-observed, not trusted. The world was not in the checkpoint. Row 1 of the trust order says this for the repository; it holds for every external fact — who still has access, what still exists, what has not expired.
A check that passed on replay is evidence about the replay — still the best evidence available and still better than recall, so this does not soften the re-check rule above. It is a caution about one case: where the original run's value came from catching something intermittent, a replay does not inherit that, and the same command typed twice is not the same event. Re-obtain the judgement there rather than carrying the old one forward.

The asymmetry is what makes this hard to notice: a boundary makes claims less reliable and verdicts more dangerous. A claim at least arrives as a claim, still open to being re-checked. A verdict arrives as a question already closed, and invites no check at all.

Handing to a successor

A brief, not an archive, and a brief is a set of named sections rather than a short essay — a successor scanning for one of these should find it as a heading, not as a clause inside a paragraph:

The goal, in the requester's own words, quoted rather than paraphrased.
Where it is now — anchored to commits and paths.
The single next action, specific enough to start.
What is known not to work, each one in the four-part form above. This is the section that is worth the whole brief, and the first one cut for length.
Every open assumption and ruling.
What would make this run stop if the successor is also unattended.

Length is a feature here, not a cost, and crouton decides the budget when there is one. What is never acceptable is a brief that is short because compressing it was easier than writing it.

Common mistakes

Symptom	Real cause
The record's last entry is always just before the gap, never inside it	Checkpointing was left to a judgment call at the boundary, where the harness offered an event that fires there on its own
A guess made before the compaction is acted on after it as settled fact	The boundary carried the claim and dropped its tier; verified, inferred, and assumed arrive in the same voice
A second session redoes committed work	Recollection trusted over `git log`, which is the default after a compaction
The same dead end explored twice	Negative results were never written; only forward progress was
A whole family of causes skipped, wrongly	A dead end recorded without its boundary, so "pinning the clock changed nothing" was read as "not a timing issue"
A negative result quoted later as settled fact	It never said what it rested on, and a flat statement reads as conclusive
The plan's checkboxes are all empty at task six	Progress lived in context. A compaction now costs the entire run
A decision gets re-argued with nobody remembering why	Rulings never recorded, so the reasoning left with the context
The handoff brief is accurate and useless	Written for someone who was there — pronouns, shorthand, unstated referents
The record was written up at the end and reads beautifully	It was written by the surviving memory, which is the one thing it was meant to guard against
A counted claim in the final report is wrong	The count was recalled from an earlier checkpoint rather than re-measured
A restored run acts on an approval whose subject has since changed	The verdict named a role rather than an object, so it re-bound on restore

Red flags

"I'm holding it all fine" — said by a context about to be compacted.
"I'll write it all up at the end."
"That didn't work, moving on" — with nothing written down about what didn't.
"Ruled that out" — said about an attempt whose boundary was never established.
"It's not a timing issue" — from one clock, in one process.
"I think task 4 was done?"
"The summary can just say what I remember doing."
"Checkpointing is overhead on a run this size" — the runs that need it are exactly the ones where it feels like overhead early.
Relying on noticing the context filling, in a harness that will announce the compaction to a hook you never wired.