Grok Build · Memory Engineering

The Real Mechanism of Dream

Dream merges recent session logs with the existing MEMORY.md to form long-term memory. It is triggered by session end, optional periodic checks, or a manual command, and is constrained by both gating and a best-effort lock.

Learning Objective

Understand the complete chain from Dream's trigger to index rebuild, and explain what problems DreamGate, idempotency requirements, lock contention, and write-failure rollback each solve.

TEACHING DIAGRAM

Dream State Machine

The diagram below is a teaching visualization. Node names come from the source code; layout and labels have been reorganized for instructional clarity.

Dream flow from entry to index sync Entry passes through three gates, builds messages, calls the model, acquires lock, writes memory, then cleans up and syncs the index. Entryend / timer / slash DreamGateenabled / hours / sessions build message32K input cap model call30 min timeout try_acquirebest effort lock write MEMORYfailure → rollback cleanup + indexonly deleted stems
Entry & Gating

1. Config Gate

MemoryDreamConfig.enabled defaults to true. Sub-Agent sessions skip Dream entirely.

2. Time Gate

min_hours defaults to 4. The lock file mtime records the last successful consolidation time.

3. Session Gate

min_sessions defaults to 3. Counts session Markdowns modified since the last consolidation, excluding the current session.

Trigger fact: The default check_interval_secs = None means periodic checks are disabled. The source explicitly supports session end and /dream; only after configuring a check interval will the session actor gate-check periodically. Therefore it is incorrect to say "Dream always runs automatically whenever idle."
crates/codegen/xai-grok-memory/src/dream.rs crates/codegen/xai-grok-memory/src/dream_lock.rs crates/codegen/xai-grok-memory/src/storage.rs crates/codegen/xai-grok-memory/src/index.rs crates/codegen/xai-grok-config-types/src/memory.rs crates/codegen/xai-grok-shell/src/session/acp_session_impl/memory_dream.rs check_dream_gates maybe_run_dream
Lock, Idempotency & Failure Recovery

DreamLock is Best-Effort Coordination

.dream-lock stores the PID and uses mtime as the last-success timestamp. Returns Ok(None) while a live process holds a non-expired lock. Dead processes or stale locks can be reclaimed.

Source code comments explicitly note it is not strictly mutual-exclusive. Read-after-write reduces contention probability, but two processes can still both believe they won, so Dream must tolerate duplicate consolidations.

Success Boundary Defines Cleanup Boundary

  • If the model returns empty, NO_REPLY, or no Markdown heading, nothing is written and no session is deleted.
  • If writing MEMORY.md fails, rollback(prior) restores the old lock state.
  • Sessions are cleaned only after a successful write; files still active within 5 minutes are skipped.
  • The search index only removes actually deleted paths, then rebuilds the index and embedding for the new MEMORY.md.
Real Source Code Snapshot
crates/codegen/xai-grok-memory/src/dream.rsREAL SOURCE · abridged
pub fn check_dream_gates(
    config: &MemoryDreamConfig,
    lock: &DreamLock,
    sessions_dir: &Path,
    current_session_sid8: Option<&str>,
) -> DreamGate {
    if !config.enabled { return DreamGate::Disabled; }
    // Time gate, then session gate
    ...
    DreamGate::Open { sessions }
}

Snapshot note: Code retains the real function signature and return type; the intermediate implementation is compressed with ellipsis. The state machine SVG in this page is a teaching visualization, not an architecture diagram generated from the repository.

Classroom Exercise: Locate System State After Failure

Scenario: Dream has completed the model call, but writing to MEMORY.md has failed. Answer: what state should the lock file be restored to? Which session files can be deleted? Does the index need updating? Provide evidence from the branches of execute_dream.

Takeaway: Dream's reliability comes from its success-boundary design. Gating reduces unnecessary calls, best-effort locking reduces concurrency, idempotency absorbs a small duplicate risk, and rollback plus deferred cleanup ensure that failures can always be retried.