Concepts & consolidation

What concepts are

A concept is a cluster of similar memories (an emergent tag, bottom-up from the corpus), complementary to the hierarchical topic tree (top-down deterministic routing). Together they enable the classic two-step: retrieve from the correct topic leaf, then filter by concept tags.

  corpus of memories (embeddings)
        │
        ▼
  deterministic k-means (CF-triplet, corpus seeding pass)
        │
        ▼
  concepts (k groups, min size clusterMinSize)
        │  concept_annotates edges (confidence + assigned_at)
        ▼
  each memory tagged ──▶ query filters (profile.conceptFilter)

Clustering

Algorithm — deterministic k-means (the same CF-triplet primitives as the topic-tree splitter — see Topic tree → CF-triplets):

1. L2-normalized member vectors
2. corpus pass: seed selection over the whole corpus (deterministic)
3. k-means refinement (CF-triplet centroids, stable tie-breaks)
4. clusters ≥ clusterMinSize members become concept rows
5. members get concept_annotates edges (confidence + assigned_at)

Reproducible by construction. Tune with the graph config: clusterK (how many groups, default 8) and clusterMinSize (minimum members for a concept, default 3). Concepts are graph nodes too — explain concept <id> works.

The dream job (consolidation)

dream(modelKey) is the durable consolidation pass that turns raw memories into structure — see the dedicated Dream (consolidation) page. In short: it classifies the write-path’s pending edge candidates with an LLM (confirm/reject), runs the concept clustering above, and enforces the provenance rule — watermark-gated, crash-safe, fresh run id per call.

The deterministic floor: everything works without an LLM — the structural rules, tree, retrieval, and graph run locally with only embeddings configured. The dream is the optional layer that narrows and confirms what the rules propose; without an llm config section it fails typed (LLM not configured).

Query sovereignty

Concepts annotate; retrieval merely prefers or filters by them — profile.conceptFilter: ["<concept-id>", …] restricts results to the annotated set (vector leg). Concepts never move or delete memories, so turning the dream off changes nothing except the labels.