DeepSeek Harness · Sessions & Loops

What Happens After Esc: Cancel, Crash Recovery, and Re-entry

One cancel signal per turn; balance the books even on interrupt; after kill -9, restart and keep going—every persisted result stays.

Course goalBy the end you can explain three things: how the cancel signal fans out from one AbortController across the model stream and tool execution after Esc, and why cancel power covers only the current turn; why an interrupted turn still writes a turn/end to the log; and after kill -9, what lets a restarted DSH salvage a half-finished session—and what it keeps.
Interactive demo · Interrupt drill ground

Here's a turn mid-work: the model is streaming, and it called a long-running tool. Left is the session log; right is runtime state. Three scenario buttons map to three accidents: Esc, kill -9 then restart, and a hypothetical truncate-style recovery for contrast. Hit Play—the caption tells you what each step adds or drops in the log.

Session event log (append-only · durable once on disk)
Runtime state
ProcessRunning
This turn's cancel signalnot created
Inbox queued messages0 msgs
Log stats show here: which events were kept, which were lost.
Pick a scenario and hit Play, or scroll here to auto-play Scenario A.
Teaching simulation: event rows are simplified. Mechanisms map to packages/core/agent-loop/src/agent.ts (cancel and turn/end logging) and packages/core/session/src/repair.ts (crash-recovery synthesized closers). Scenario C's truncate recovery is a teaching hypothesis—DSH does not implement it—used to contrast data loss.
Design idea 1 · Cancel power follows the turn

What problem it solves

Imagine cancel as a global switch: a boolean on the agent, anyone can flip it, and code peeks when it feels like it. Things go sideways eventually: a timeout callback from the last turn wakes at midnight and cancels the new turn mid-flight; or cancel is Promise.race, the losing tool call gets no cleanup, and keeps editing files and writing state in the background—zombie work.

The hard parts of cancel are stopping cleanly, and stopping only what should stop. A global switch gives you neither.

What the idea is

DSH cancel is an explicitly passed wire: one end tied to the turn, the other to every busy boundary. Each time the driver wakes to work, it creates a new AbortController; when a turn finishes and the queue still has work, it swaps in a fresh one. At most one controller is live at a time—cancel power lives exactly as long as the turn.

Esc just tugs that wire. The UI maps the key to agent.cancel({ kind: 'user' }); a parent interrupting a child uses { kind: 'parent' }—cancel carries identity. The cancel entry is small enough to memorize—two moves: by default clear the inbox (queued unrun messages die; pass keepInbox to keep queued work and only interrupt the current activity); then abort(cause) on the current controller. Calling cancel while idle is a no-op—it won't pre-bury cancel state for future work.

The same signal is passed explicitly into pre-step, prompt assembly, model requests, stream reads, tool execution, and approvals—even bash can follow it to kill the whole process group. Delivery is cooperative: the loop checks abort around every await boundary, without Promise.race discarding a still-running Promise mid-flight, so zombie work can't quietly mutate state.

One explicit wire: from Esc to every busy boundary Esc pressed Mapped to identity-bearing cancel cancel does only two things Clear inbox, then abort(cause) This turn's only signal Lives and dies with the turn Model stream stops reading Tools & bash process group Assembly & approval stand down Reclaim cancel power before publishing turn/end; next turn gets a fresh signal
The signal is an explicit parameter, not global state; each boundary checks around await and stands down cooperatively.

Finally, the handoff. Before publishing turn/end, the loop clears this turn's cancel holder—after that, even if persistence flush hasn't settled, nobody can cancel finished turn work; the next turn gets a brand-new signal, and old callbacks that try to overreach can't even find the handle.

Source:Per-turn AbortController creation is in packages/core/agent-loop/src/agent.ts line 187; swap-after-finish at line 325; cancel entry (optional inbox clear, then abort with typed cause) at lines 134–140; design notes on not leaking cancel across turns: Agent Note 2026-07-16.

Why it lasts

Explicit tokens plus scope binding is the structured-concurrency rule of thumb—Go's context and .NET's CancellationToken take the same path. The creator reclaims the token; it doesn't outlive its scope, so overreach has nowhere to stand. As long as a system must stop streaming IO and external processes together, rewrite it in another language and you still need this explicit wire.

Design idea 2 · Balance the books even on interrupt

What problem it solves

Suppose a canceled turn writes no terminal state: the log stops mid-cut, replay can't tell how the turn ended; the UI can't honestly say which queued work was dropped; downstream readers can't tell orderly stop from accidental death. Interrupt is normal business—unbooked interrupt is the accident.

What the idea is

The turn's main logic sits in try. The catch branch sees signal.aborted and sets the outcome to aborted; finally always writes a turn/end to the log. Already-persisted stream chunks and tool outputs are never deleted.

So the log has only two ways to die, each with its own signature. aborted is written by the loop itself: someone called cancel, the turn closed orderly. interrupted is never emitted by the loop—only synthesized by the persistence backend on crash recovery, the only non-loop-authored ending. One glance at reason tells you how the turn ended.

Two companion details. The log keeps only coarse aborted—not who pressed it; user vs parent is runtime info, replay neither needs nor should know. Cleared queued messages get no turn/end; foldConsumedWork single-passes the log and derives droppedUnrun from inbox records with outcome: 'canceled', so the UI can honestly say work was thrown away unrun.

A clean goodbye and an accidental death—one glance at the log tells them apart.

Source:catch sets the outcome in packages/core/agent-loop/src/agent.ts lines 302–305; finally writes turn/end at lines 316–323; droppedUnrun folding is in consumed-work.ts line 87.

Why it lasts

Every exit path writes a terminal state—the floor for any system that treats the log as authoritative, same idea as commit/abort records in a DB transaction log. Exception and happy paths produce the same kind of ledger, so rebuilders don't guess. Model generations change, languages get rewritten—this discipline stays.

Design idea 3 · Crash recovery completes; it doesn't truncate

What problem it solves

Cancel at least gets finally cleanup; kill -9 gets none. The instant the process dies, the log stops mid-cut: turn/start is open, a tool call recorded tool/call, but tool/result never arrives. Cold-loading that log after restart faces a choice: delete the unfinished turn, or complete it?

Deleting looks clean; it costs far more. A single turn in a long job can be huge—dozens of steps, tons of tool output, already append-persisted before the crash. Truncation buries the user's labor with it—Scenario C in the demo is that bill.

What the idea is

DSH chooses completion. Recovery emits a few deterministic synthetic events to close the tail: first an error-placeholder tool/result for each dangling tool call, then close any open step, finally synthesize turn/end with interrupted. Order matters: writing turn/end while a step is still open violates the log invariant—so close step boundaries first, then turn.

Persisted real events stay untouched; only three synthetic events append at the tail kill -9 Turn start User message Model output Tool start Fill placeholder result Fill step closer Fill turn ending Keep going Persisted before crash · all kept Synthesized on recovery · ending marked interrupted
Synthetic event timestamps reuse the last real event's—never invent a future time.

Placeholder text for dangling tool calls splits two cases. If the tool recorded a start but no result on disk, the placeholder tells the model the result is unknown—only read-only or idempotent ops may retry; side-effecting ones must verify external state or ask the user first; the source stresses “Do not retry blindly”. If the tool never started, it simply says retry if needed.

“It does not truncate the log: in long-running tasks a single turn can be huge (many steps, lots of tool output), and those events were already append-persisted before the crash. Instead the backend closes the leftover turn with a synthetic turn/end { reason: { kind: 'interrupted' } }, balancing interrupted execution without changing any independent events before or after.”

docs/subsystems/persistence.zh.md · crash recovery keeps interrupted turns

One easy-to-miss boundary: this repair only applies to cold sessions. While a session is still live, load waits for the authoritative in-memory snapshot to flush and returns only when the log balances; an unclosed active turn is refused outright—never inject synthetic boundaries into a running turn. The write side has discipline too: the persistence plugin batches to disk, and the loop checkpoints with session/flush before claiming the next turn, watching both order and write errors.

Source:Two phrasings for placeholders: packages/core/session/src/repair.ts lines 91–124; closer synthesis order (step then turn) at lines 126–132; cold-session-only repair in docs/subsystems/persistence.zh.md line 17; flush checkpoints in the matching section of the same doc.

Why it lasts

Append-only logs plus repair-on-read is the DB WAL idea: after a crash, don't rewrite history—only append the minimal events that let the state machine continue. As long as authoritative state lives in an event log, recovery rewritten a hundred times still looks like this. Conversely, systems willing to truncate the log assume a single turn is cheap enough to throw away—that assumption fails in the long-task era.

Side-by-side · How two systems face interrupt and crash

Grok Build

Process-level · dedicated crash crate

Grok has a dedicated crate xai-crash-handler: sigaction catches SIGSEGV and SIGBUS; at the crash site only async-signal-safe ops write a binary snapshot to last-crash.bin, and precomputed escape sequences restore the terminal; on next start it resolves symbols and writes a crash report, keeping the latest 5 (crate README and handler.rs).

It answers how the process died and how not to leave the terminal a mess. DSH's repair.ts answers how the session log survives. Each owns a layer; head-to-head, DSH's distinctive move is baking recovery semantics into the persistence contract: load's interface comments promise to complete an interrupted tail without rewriting committed events. In verified Grok Build materials, no equivalent session-log balancing mechanism showed up—based on publicly available source.

Claude Code

Task-level · AbortController on the task

Background-agent cancel goes through killAsyncAgent: pull abortController from task state, abort, mark the task killed (restored-src LocalAgentTask.tsx lines 283–298, cited in manuscript ch. 6). Controller lifetime follows the task—one per task.

DSH cuts one grain finer: the controller follows the turn; the same agent's next turn automatically gets a new signal, and old callbacks that overreach can't find the handle. DSH also explicitly does not persist cancel cause—the durable log keeps only aborted; Claude Code's session-recovery and interrupt-record details aren't expanded in the verified manuscript chapters—left as unknown here.

Classroom Exercise
01

Work through log differences at two moments

Same turn, cancel at two different moments: a) mid model stream; b) while a bash tool is running.

Q1: For each case, list the events between step/start and turn/end, and what turn/end's reason is.

Q2: Swap both accidents for kill -9—how many closers does repair synthesize after restart? Hint: stream interrupt has no dangling tool/call; tool-execution interrupt has one started tool/call, with unknown-result placeholder text.

Takeaway:Cancel is an explicit wire: one AbortController per turn; cancel clears the inbox then aborts; interrupted turns still write turn/end { aborted }; cancel power is reclaimed before turn/end publishes—never across turns. Crash recovery doesn't truncate: cold load fills error placeholders for dangling tools, synthesizes turn/end { interrupted } to balance, and keeps every event persisted before the crash. A clean goodbye and an accidental death—one glance at the log tells them apart.