Testing a Nondeterministic System

Play first, then talk. Scenario A is a real HTTP fault server: behaviors queue up, each request consumes one, watch how the client responds. Scenario B is deterministic replay: take a real session log, derive a replay script in one shot and re-run it, then tamper with one line and watch the diff testify on the spot.

There’s only one way to test a nondeterministic system: fence off the uncertain parts and make everything else deterministic. DSH’s test layers (docs/testing.zh.md) are built around that idea. Unit tests hunt edge cases, error paths, event ordering, and concurrency races; CI’s coverage gate demands 100% per file under packages/*/*/src (AGENTS.md line 65, Commands section). The docs also say it flat: line coverage is necessary, never sufficient — unrun lines are usually dead code to delete, not tests to add.

Real API tests with keys are another layer. Here’s a line with real identity:

Inference is cheap for us, so smoke tests go real: boot a real example, send a prompt, check the outside world. Assertions matter too: e2e should re-read files and re-run commands to verify results — keyword sniffing on the agent’s own output lets a cheating agent pass. Environments without keys skip automatically and block nobody.

The main act is replay. Last lesson covered this (see LLM Adapter Layer): every streaming chunk lands as an assistant/chunk event in the session log as-is. The dsh-llm-replay plugin flips that around: take a recorded session.jsonl, group chunk events by (turn, step), and each group is the full chunk sequence from one model call back then. In tests the real agent runs as usual — only the model end is swapped for a replay adapter that emits the recorded chunks frame by frame. Nondeterminism lives only in that one recording; every re-run is byte-identical afterward, no API Key needed.

That’s what “logs as test assets” means: fixtures aren’t hand-written mocks — they’re production-format session logs themselves. Snapshot tests pin the whole assembled behavior with them; change one line of code and fork the behavior, and the diff lights up red on the spot. A neat detail sits in fork (forked sessions): a child session’s log starts by inheriting the parent’s seed events, so when deriving the replay script you must slice after the seedLength boundary — otherwise the parent’s chunks get replayed as if they were the child’s calls:

    const text = readFileSync(childFile, 'utf8')
    const header = parseSessionHeader(text)
    // Derive the child's script from its own events only — events AT OR after the seed
    // boundary.
    const ownEvents = parseSessionLog(text).slice(header.seedLength)
    children.push({
      recordedId: header.id,
      createdAt: header.createdAt,
      entries: deriveReplayScript(ownEvents),
      primary: false,
    })

Cross-platform discipline follows from here too: checked-in fixtures must replay on both macOS and Linux; if a recorded snapshot fails on either platform, fix the fixture itself. The exact words on AGENTS.md line 123 are “fix fixtures, not normalizers”: fix the fixture, don’t write a normalizer. Normalizers pad cotton between tests and reality — pad enough and you stop testing reality.

Replay tests that behavior stays put — one piece still missing: the transport layer’s creative ways to die. Connection refused, socket reset mid-send, clean close without [DONE], rate limit with Retry-After, or just hanging still — each looks different to the adapter and recovery layers. In-process mocks miss them all, because mocks bypass real boundaries like fetch, SSE framing, socket teardown, and idle watchdogs. So DSH built dsh-llm-mock-server: a real Node HTTP server speaking OpenAI dialect, behaviors fully script-controlled, one behavior consumed per request, explicit error when the script runs out (design motive in Agent Note 2026-07-25-scriptable-llm-wire-fault-server.zh.md). Developers who want to reproduce faults by hand just point any app at a new base URL and key.

It also has a random mode that draws faults by weight for stress testing, with a public reproducible seed. The default weight table is itself a checklist of what LLM clients meet in the wild: plain success 48, slow success 10, mid-stream cut (partial_disconnect) 10, then 5 each for connection reset, disconnect, empty reply, and rate limit, server error 4, hit max_tokens / hang / 503 at 2 each, and the two meanest at 1 each: partial_eof (stream ends cleanly but unfinished) and malformed_json (bad JSON). Source comments remind you: this is tunable test pressure, not an estimate of production incident rates.

Source: DEFAULT_MOCK_LLM_RANDOM_WEIGHTS in packages/test-support/llm-mock-server/src/index.ts lines 56–70, verified on 2026-08-13.

The server’s discipline is restrained: it only reports protocol-layer facts, never whether to retry — policy belongs to the harness. Real combination tests send requests through the DeepSeek adapter, agent loop, and retry plugin in order, and check concrete things: exact request counts, numbered retry steps, failed half-chunks never leaking into history, and half-output with clean EOF classified as STREAM_CLOSED with no retry by default.

The last weapon fights interleavings nobody thought of. Protocol-shaped code (chunk streams, event logs, inbox scheduling) has a combinatorially exploding input space; example tests can only pin cases you already imagined. DSH gives every protocol-shaped package a fast-check–driven property test: generators build realistic but adversarial inputs (duplicate indexes, lagging chunks, malformed streams missing block-start), and assertions target invariants, not concrete outputs — e.g. assembled block count can’t exceed distinct indexes seen, and repeated calls must be stable. Failures automatically print a reproducible seed.

Its track record opens the Agent Note’s first line:

This infrastructure has a side product: the official benchmark path in BENCHMARK.md is the Python SDK plus the minimal variant, each task in its own workspace. Once the test system is solid, running evals is just swapping the input.