Skip to content
DimpleDimple

Dream (consolidation)

What it does

Section titled “What it does”

The write-path rules already propose structure on every write (identity collisions → supersedes, near-duplicates, contradictions — all deterministic, all pending). The dream job narrows and confirms what the rules propose:

  write-path rules (deterministic, every write)
        │  identity collisions / near-duplicates / contradictions
        ▼
  pending candidate edges
        │
        ▼
  ┌──────────────────────────────────────────┐
  │              dream job (LLM)             │
  │   classify candidates + cluster concepts │
  └──────┬──────────────┬─────────────┬──────┘
         │              │             │
         ▼              ▼             ▼
  memory_supports  memory_super-  memory_contra-
  confirmed        sedes          dicts
                   confirmed      confirmed / rejected
         │              │
         ▼              ▼
  provenance for    old memory:
  derived memories  active → superseded


  concept clustering (deterministic k-means over the corpus)
        │
        ▼
  memories annotated with concepts ──▶ query filters
  1. Edge classification — each pending candidate is classified by the LLM into memory_supports / memory_supersedes / memory_contradicts and confirmed or rejected. Classification is LLM-mandatory — there is no rules-mode fallback: if the LLM is absent, the job fails typed rather than silently guessing. A confirmed supersede flips the old memory active → superseded.
  2. Concept clustering — deterministic k-means over the corpus (the same CF-triplet primitives as the topic-tree splitter); each cluster becomes a concept row and members get concept_annotates edges — the “what is this about” labels queries can filter by.
  3. Provenance rule — derived memories require ≥ 1 confirmed memory_supports edge: if it can’t show its work, it doesn’t get to claim the thought. Without a supporting edge, a derived memory’s claim stays unconfirmed.

Gates & safety

Section titled “Gates & safety”
  • Requires the llm config section; without it dream() fails typed (LLM not configured) — never a silent no-op
  • The job is durable and watermark-gated: the watermark pair advances only on success, so a crash mid-pass re-runs exactly the unprocessed range — nothing is double-applied
  • pinned memories are respected: a writer-set pin (“this memory is deliberately stable”) exempts the memory from automatic graph lifecycle moves
  • Every dream(modelKey) call gets a fresh run id — it always executes; results are deterministic where the pipeline is deterministic (clustering), LLM-confirmed where judgment is required (edges)

The deterministic floor

Section titled “The deterministic floor”

Everything works without an LLM: structural rules, the topic tree, hybrid retrieval, and the graph all run locally with only embeddings configured. dream is the optional layer that consolidates — turn it off and nothing breaks; turn it on and the corpus starts organizing itself. Tune the clustering with the graph config: clusterK (default 8) and clusterMinSize (default 3).

Running it

Section titled “Running it”
Terminal window
dimple dream minilm          # CLI — waits for completion
# or
await dimple.dream("minilm");  // SDK — enqueues + waits (durable)

The sibling repair(modelKey) runs the graph invariant repair pass (graph.repair job) — the two consolidation surfaces are dream (structure from content) and repair (structure from invariants).