Skip to content
DimpleDimple

Durability & determinism

The job runtime

Section titled “The job runtime”

Every structural mutation (split, merge, rebalance, backfill, consolidation, audit write) runs on a durable queue (a libSQL-backed workflow engine):

Job Trigger Work
topic.split / topic.merge / topic.rebalance automatic per write keep every leaf within its size budget (deterministic splits, donor-min borrow, nearest-sibling merge, uneven-split rebalance)
memory.embed model backfill re-embed a memory with a target model + route into its tree (idempotent)
concept.cluster dream deterministic concept clustering
memory.dream dream() LLM edge classification + clustering (fresh run id per call)
graph.repair repair() graph invariant repair
audit.write query path async audit-log rows (never blocks queries)

Durability mechanics:

  write commits ──▶ enqueue structural job (split / merge / dream / …)
                          │
                          ▼
                  worker claims it (lease)
                          │
              ┌───────────┼────────────┐
              │           │            │
        completes      crashes     another worker
              │           │            │   (any process can claim)
              ▼           ▼            ▼
        watermark    lease expires   handler re-runs idempotently
        advances   → re-claimable    (txn rollback, watermarks
        (exactly                    advance only on success)
         once)
  • Jobs are leased: a handler interrupted by a crash or restart has its lease expire and the job is re-claimed by any worker (cross-process resume)
  • Handlers are idempotent by design: transactions roll back on failure, watermarks advance only on success, re-runs converge
  • Retries are bounded (maxAttempts, clamped to [1, 100]); every CLI process / makeDimple boots a worker, so pending work drains across runs
  • A burst that outruns the worker self-heals: still-over-budget children enqueue their own splits until every leaf converges

Storage model

Section titled “Storage model”

Two SQLite files (or one Turso remote database — the same engine over the wire):

Database Holds
dimple.db (storeUrl) memories + lifecycle journal, topic tree, graph edges, concepts, retrieval audit
jobs.db (jobs.dbUrl) durable queue + workflow state

file:-only behaviors (parent-dir creation, migration lock) skip for remote URLs. The bundle is worker-safe: node:fs loads lazily, so the same stack builds for edge runtimes with hosted embedding models.

Determinism — the reproducibility contract

Section titled “Determinism — the reproducibility contract”

The same input must produce the same tree, the same clusters, and the same audit hashes — regression-tested across runs and databases:

  1. Fixed embedding model version per routing run (per-model trees; model_key-scoped everywhere)
  2. Vectors normalized consistently; one distance function (cosine)
  3. Canonical ordering — IDs/vectors sorted before every batch operation; split child keys hash canonical vectors, never memory ids
  4. Stable tie-breaks — lexicographic topic_key / member order, never insertion order or random ids
  5. No random clustering — farthest-pair seeds, CF-triplet math
  6. Append-only change log for structural mutations
  7. manual_lock memberships never auto-moved
  8. Retrieval audit rows carry FNV-1a query hashes — reproducible query identity