Grok Build · Memory Search

From File Changes to Hybrid Ranking

Sync dirty files before querying, then combine FTS5 BM25 with optional sqlite-vec KNN. Merged scores are adjusted by time decay, source weight, and access boost; MMR diversity re-ranking is optionally enabled at the end.

Learning Objective Describe in correct order: sync-on-search, FTS, embedding, KNN, weighted merge, and MMR. Be able to explain what happens when embedding fails and when MMR is disabled.

Core Diagram · Complete Retrieval Path

Pedagogical diagram: failure branch falls back to FTS-only; MMR is opt-in and does not re-rank by default.

Real Mechanisms in the Pipeline

01 · SYNC

Sync Before Query

MemoryFileWatcher accumulates changed Markdown paths. The backend re-indexes new or modified files at the start of each search, and deletes stale chunks for removed files.

02 · FTS

BM25 Candidates

First run standard FTS, then supplement with global and workspace source queries to reduce crowding-out caused by too many sessions.

03 · VECTOR

Optional KNN

Embeds the query only when sqlite-vec and the provider are available. Embedding errors are logged as warnings; None is passed to continue FTS-only.

04 · SCORE

Normalize & Merge

BM25 scores and vector L2 distances are normalized separately. Dual-path hits are merged by weight, while ensuring results are not below the chunk's FTS score.

05 · WEIGHT

Time & Source

Sessions decay exponentially with a half-life; global and workspace sources are treated as evergreen. Then multiply by source weight and a moderate access boost.

06 · DIVERSITY

Optional MMR

When enabled, performs greedy re-ranking by relevance and Jaccard diversity of snippets. Finally truncates to max_results.

Two Commonly Misread Switches

Embedding Failure

The vector path stops, but FTS results still enter hybrid_search_merge. The page or caller does not need to treat an embedding failure as a total search failure.

fallback = FTS-only

MMR Default State

MmrConfig::default() sets enabled: false and lambda: 0.7. The 0.7 value only takes effect when MMR is explicitly enabled.

enabled = false

Real Source Code Evidence

crates/codegen/xai-grok-memory/src/search.rs · Lines 146–190 (excerpt)

pub async fn hybrid_search(
    index: &MemoryIndex,
    embedding_provider: Option<&dyn EmbeddingProvider>,
    query: &str,
    config: &MemorySearchConfig,
) -> Result<Vec<SearchResult>, Box<dyn std::error::Error>> {
    let candidate_limit = config.max_results * 3;
    let mut fts_results =
        index.search_fts(query, candidate_limit).unwrap_or_default();
    /* source for supplementing evergreen FTS candidates is here */

    let vec_available = index.vec_available();
    let query_embedding = if vec_available {
        if let Some(provider) = embedding_provider {
            match provider.embed_batch(&[query]).await {
                Ok(embeddings) if !embeddings.is_empty() =>
                    Some(embeddings.into_iter().next().unwrap()),
                Ok(_) => None,
                Err(e) => {
                    tracing::warn!(error = %e,
                        "embedding query failed, falling back to FTS-only");
                    None
                }
            }
        } else { None }
    } else { None };

    hybrid_search_merge(index, fts_results, query_embedding.as_deref(), config)
}

crates/codegen/xai-grok-memory/src/backend.rs: search() — executes watcher sync and query crates/codegen/xai-grok-memory/src/watcher.rs: MemoryFileWatcher crates/codegen/xai-grok-memory/src/mmr.rs: mmr_rerank crates/codegen/xai-grok-config-types/src/memory.rs: MmrConfig defaults

Source Snapshot Note: Based on the local repository grok-build-main, verified on 2026-07-17. Code blocks retain real functions and branches; the only omitted section is explained by a comment; the flow diagram is explicitly labeled as a pedagogical diagram.

Classroom Exercise

Trace a Degraded Query

Assume the watcher detects a modified file, query embedding then fails, and MMR stays at its default configuration. Write out — in order — index update, candidate generation, weighted ranking, and final truncation, and mark the two steps that did not occur.

Takeaway: Memory retrieval has two key guarantees: graceful degradation and sync-on-search. FTS always provides baseline candidates while vector search enhances based on availability; time decay and source weight adjust ranking; MMR requires explicit activation; the watcher ensures external Markdown changes are indexed before the next query.