LLM Adapter Layer: Single Attempt, Explicit Retry, Dual-Stream Persistence

Play first, then talk. Left: SSE chunks on the wire (raw frames the provider emits). Right: the session log (each chunk lands immediately as assistant/chunk; when the stream ends, a derived assistant/message is appended). Top: what the user sees. Three scenarios: A is a clean dual-stream write; B drops mid-stream so you can watch retries appear as explicit timeline events; C is the cautionary tale—SDK silent retry, same failure, nothing in the log.

First, the rule. DSH’s adapter contract is blunt: one adapter call is one provider attempt, and adapters must disable library-built-in retries (docs/subsystems/llm-streaming.zh.md, adapter-contract section). HTTP libraries love to retry quietly for you—looks helpful, actually hides information: why it was slow, how many retries, why each failed, all vanish inside the library loop. DSH peels that off. The adapter does one job: fire one request, emit a uniform StreamChunk stream, and on failure normalize a serializable LlmFailure (with a stable error code). Nothing else.

Hang protection lives here too: both shipped remote adapters carry a streamIdleTimeoutMs watchdog (default five minutes); a stalled provider maps to TIMEOUT. Easy to miss: empty replies are errors. A stop with no content blocks becomes EMPTY_RESPONSE, not a silent success—so the retry layer can still save it.

Then persistence. While the agent loop consumes the chunk stream it does two things: append each chunk verbatim as an assistant/chunk event to the session log, and feed it to BlockAssembler. When the stream finishes cleanly, the assembled result is written again as an assistant/message. That’s dual-stream: the chunk stream is the raw recording—replay tests rebuild the response frame by frame from it; the message stream is derived history—the next request’s conversation context comes from it. Reasoning blocks and text blocks are different types in the chunk protocol; each assembles and persists on its own.

      const assembler = new BlockAssembler()
      const chunkSeqs: number[] = []
      const stream = preparedCall?.stream(request) ?? this.loopCtx.llm.stream(request)
      signal.throwIfAborted()
      for await (const chunk of stream) {
        signal.throwIfAborted()
        chunkSeqs.push(this.session.append('assistant/chunk', { turn, step, chunk }).seq)
        assembler.push(chunk)
      }

These nine lines are the heart of dual-stream: persist first, assemble second, no chunk exempt. After the stream ends, the fork: if finish is error or abort, hand the failure to agent/request-error to decide retry (lines 354–371); only on success does line 381 append assistant/message and record this batch’s chunk seqs in sourceEventSeqs, marking which chunks the message came from. So failed half-output has a clear home: it stays in the chunk stream as evidence and never enters derived history. When the next request rebuilds context, that half never happened.

The assembler deserves a pause too. BlockAssembler is the repo’s only chunk-folding implementation—adapters just emit well-formed chunks by index; nobody reimplements block reassembly. On malformed streams it’s firm: on block-end, if the block already closed, return immediately and ignore later duplicate closes. The comment calls it “First close wins”—only the first close counts, so streamed output and the final assembled block stay consistent. That one defensive line is exactly where a property test caught a real bug (postmortem in Testing a Nondeterministic System).

Source:packages/llm/llm/src/assembler.ts block-end branch, lines 75–82; verified on 2026-08-13.

Retries live one layer up. The dsh-llm-retry plugin listens for agent/request-error and decides whether to save the call using the policy captured when each provider route was registered. Default normal policy: retry only EMPTY_RESPONSE, RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT—at most twice, backoff 500 ms to 10 s with 10% jitter; a valid Retry-After from the provider replaces local backoff (packages/llm/llm-retry/README.zh.md).

The key is how it books the work. Before waiting out backoff, the plugin appends an llm/retry event—retry id, provider, policy mode, full failure info, planned delay; when backoff ends and it actually fires, it appends llm/retry-started. Neither enters the model-visible surface; the model knows nothing about retries. UI and postmortems lean on them: the UI withdraws failed half-output and shows “retry 1/2 in 3s”; when debugging, every attempt and every wait has a timestamp. The retry itself opens a new numbered turn, rebuilds the same request from persistent history, and leaves the old turn’s records untouched.

One cross-adapter detail: replayState. A successful finish chunk may carry adapter-private replay state (e.g. DeepSeek’s native reasoning representation) and store it with assistant/message. Before the next request sends history to an adapter, LlmRuntime.forAdapter() checks each history message: state is forwarded only if the history provider and target provider share the same adapter instance; switch adapters and the state is stripped—the peer gets provider-agnostic content only (packages/llm/llm/src/index.ts lines 822–836). One house’s private data never feeds another.