Argus memory pipeline — what gets collected, injected, and how we manage it¶
This is the operator + developer reference for the Argus memory data flow: every stage, the data that moves through it, where it lands, and — because trust is the whole point — the audit surface that lets you verify the stage is actually working. Silent failure is the cardinal sin; every stage below writes an audit row or a log line so "is this working?" is always answerable from data, never a guess.
Source of truth is the code (internal/memory/*.go, migrations/*.sql). This
doc is generated from a code read on 2026-06-20; constants/columns can drift, so
trust the code over this when they disagree.
CAPTURE → SEGMENT → DISTILL → EMBED → DEDUP → RECALL → (EXPORT)
raw_items episodes distilled_notes ↑judge ↑two paths Obsidian
(append-only, immutable) confidence inject log
The spine is immutable: raw_items is append-only (DB trigger), and every
downstream view (episodes, notes, embeddings) is regenerable from it. Nothing
below mutates the raw record.
1. CAPTURE — conversation turns → raw_items¶
Client: cmd/capture-hook · Server: internal/memory/capture.go, hook.go
Endpoint: POST /capture (bearer ARGUS_CAPTURE_TOKEN)
Driven by Claude Code hooks (wired in each machine's settings.json →
~/.argus/bin/argus-capture-hook): Stop (after every response), SessionEnd,
and PreCompact (auto). The hook reads the session JSONL transcript, extracts
only the visible user + assistant text, and posts it. It filters out:
tool blocks, sidechain traffic, compaction digests, and poison signatures
(API-error text, assistant refusals) — the latter is the explicit fix for the
memory-os self-poisoning loop.
No LLM on the capture path — it is pure transport + dedup. This matters: the hot path stays fast and deterministic; the only judgment happens later in distill.
Idempotency: a per-session byte offset in ~/.argus/hook-offsets/{session}.offset
(client side) plus a server dedup_key = md5(session | original_timestamp | body)
(migration 0014). A re-post of the same turn returns the existing row id with
duplicate=true — never a second copy.
Lands in raw_items: source (claude-code/codex/telegram/task/bookmark),
kind (turn/document/event/synthesis), body (verbatim), cwd, session,
project, metadata (role, host, repo, original_timestamp), captured_at.
Append-only, enforced by a DB trigger.
Verify it's working: capture failures are loud on the hook's stderr (fail-open
for the session, but never silent). Row count in raw_items per session; the
duplicate flag rate tells you idempotency is holding.
2. SEGMENT — turns → episodes¶
File: internal/memory/segment.go · Endpoint: POST /segment (via /process)
Groups a session's turns into coherent episodes using three deterministic-first
boundaries (a topic boundary fires only after the cheap checks):
- size-cap — 20 turns (SizeCap)
- time-gap — >30 min between turns (TimeGap)
- topic-shift — rolling-window cosine drop below threshold (needs MinTurns=4)
Lands in episodes: session, project (+ project_method/confidence/cwd
for audit), first_raw_id..last_raw_id, turn_count, boundary_reason,
boundary_score, distilled_at (NULL = pending).
Verify it's working: segment_checks records every boundary check — even
the ones that didn't fire — with the score and threshold. That's how you calibrate
the topic threshold from real data instead of guessing.
3. DISTILL — episodes/items → distilled_notes (write-gated)¶
Files: internal/memory/distill.go, gate.go, process.go
Endpoint: POST /process (Ofelia-scheduled) → ProcessReport
Two tracks: conversations distill per closed episode (whole-conversation
context); documents/events/syntheses distill per raw item. The LLM is asked
for atomic, durable facts as constrained JSON ({notes:[{title, content, importance,
keywords, tags, context, source_ref}]}).
The write-gate is the trust boundary. Every candidate note must pass:
1. poison check — content/source_ref must not match API-error/refusal regex
2. grounding check — source_ref must be an actual whitespace-normalized span
of the raw body (the model can't invent a fact with no source)
3. title ≤ 80 chars
Survivors are written atomically (old notes for that raw/episode tombstoned,
new ones inserted) with model + model_real attribution. Rejected candidates
are logged to gate_rejections (reason + detail) — never dropped silently.
Lands in distilled_notes: content, importance (0–1), confidence
(starts 0.5, earned via dedup), keywords, tags, source_ref (grounding),
valid_from/valid_to (bitemporal), shown_count, deleted_at (soft tombstone),
embedding (set next stage). raw_id/episode_id link back to the source.
Resilience: a distill failure is classified infra-vs-content; infra failures
(timeout/5xx/429) don't burn the streak. After DistillFailureCap=5 a unit is
parked in distill_failures (dead-letter) — visible + retryable, not starved.
Verify it's working: gate_rejections (grounding-rejection rate, poison hits),
distill_failures / /ui/parked (dead-lettered units), ProcessReport counts
(episodes_distilled, notes_created, parked, transient_failures).
4. EMBED — notes → vector(768)¶
File: internal/memory/embed.go (LiteLLM embed tier) · via /process
Live notes with embedding IS NULL, batched 64 (embedNotesBatch), embedded via
LiteLLM /v1/embeddings. Strict 768-dim validation (EmbedDim); per-batch
transactions so partial progress survives a failure (this fixed the M2 embed
death-spiral).
Verify it's working: count of embedding IS NULL live notes should trend to 0;
a non-zero floor means the embed tier is down.
5. DEDUP — corroborate confidence, hold risky merges for review¶
Files: internal/memory/dedup.go, review.go · via /process
For each newly embedded note: exact-duplicate pre-check, then cosine kNN over older
live notes (≥ dedupSimilarityFloor = 0.80, up to 3 candidates). If neighbors
exist, an LLM judge returns duplicate | refine | supersede | distinct:
- duplicate → tombstone the new note
- refine → keep new, tombstone old, bump confidence (diminishing returns)
- supersede → keep new, tombstone old — UNLESS the old note's confidence
≥ dedupFlagConfidence = 0.70, in which case it is HELD for human review
(status=pending), not auto-applied
- distinct → keep both
Lands in dedup_log: every verdict with similarity, model, status
(applied/pending/approved/rejected), reasoning.
Verify it's working: dedup_log (verdict mix, pending holds), /ui/review
(approve/reject the held collisions — hold-until-approved means a confident fact is
never silently overwritten by an LLM).
6. RECALL — two paths, both audited¶
Both paths use the same "live note" predicate everywhere:
deleted_at IS NULL AND valid_from <= now() AND (valid_to IS NULL OR valid_to > now()).
6a. SessionStart context recall (deterministic)¶
Files: context.go, inject.go · Endpoint: GET /recall/context · hook: SessionStart
No embedding, no LLM. Returns live notes ordered by importance DESC, valid_from
DESC, where project = {cwd basename} OR project = 'global' (own project first,
then global), limit 12. This is what seeds a fresh session.
6b. UserPromptSubmit prompt recall (hybrid)¶
Files: hybrid.go, prompt_hook.go · Client: cmd/recall-hook · Endpoint: /recall/notes
Three gates run before any work, each logged (never silent), in order:
1. system-turn gate — skip if the prompt leads with <task-notification or
<system-reminder (harness-injected turns are not user queries). (ARGUS-24)
2. substance gate — skip if < 5 meaningful tokens (an "ok"/"yes" ack carries no query)
3. relevance floor — applied after recall: drop hits below the floor
(default PromptRelevanceFloor = 0.015, overridable per-machine via
ARGUS_RECALL_FLOOR in ~/.argus/env). (ARGUS-18/19)
Hybrid recall (hybrid.go): two tracks — semantic (pgvector cosine) and keyword
(Postgres FTS) — fused by Reciprocal Rank Fusion: each track contributes
1/(60+rank) (rrfK=60), a note in both tracks sums both. Then a type-prior
multiplier (per raw.kind), an optional recency decay, and a same-project
boost ×1.25 (a nudge, not a filter). Top 5 (promptHookMaxNotes) injected.
Lands in injections (the recall ledger): session, query, note_ids
(rank order), note_scores (the fused scores), project, prompt_mode
(true=prompt, false=SessionStart context). Every recall logs a row — even a
zero-hit one ("recall ran and injected nothing" is a signal, not a gap).
Verify it's working: /ui/injections — per-recall log + the score-distribution
histogram (prompt-mode only) with the floor marked, so you can calibrate the floor
against real data. (ARGUS-20/21/22) The gate skips are at slog.Debug.
Two score scales live in
injections: prompt-mode recalls carry RRF scores (~0.016–0.033, what the floor gates); SessionStart context recalls carry the importance scale (0.6–0.9). The calibration histogram filters to prompt-mode only.
7. EXPORT — notes → Obsidian vault (optional, scheduled)¶
File: internal/memory/export.go · Endpoint: POST /export
One markdown file per live note (YAML frontmatter + content), byte-identical files skipped (no write, no livesync churn — this fixed a vault-revert incident). Adds a Related section from kNN neighbors (cosine ≤ 0.35) and cluster hub pages from the mutual-neighbor graph (stricter ≤ 0.22 — the giant-component guard). Published to devices via livesync-bridge → CouchDB.
Verify it's working: ExportResult (written/unchanged/removed/linked/clusters/
largest — the giant-blob detector).
8. Project attribution¶
metadata.project if explicit → else path.Base(cwd) → else (with a topic threshold)
classify the episode's mean embedding against project centroids in the projects
registry, recording project_method/confidence + a cwd-mismatch audit. The
global pseudo-project (migration 0021) has no centroid — it is never
auto-assigned, only explicitly attributed, and is appended to every SessionStart
recall.
9. Observability surfaces (the whole point)¶
| Layer | UI | Audit table | "is it working?" |
|---|---|---|---|
| Capture | — | raw_items (+duplicate) |
rows per session; idempotency rate |
| Segment | /ui/episodes |
segment_checks |
every boundary check + score |
| Distill | /ui/episodes/{id} |
gate_rejections |
grounding-reject / poison rate |
| Distill DLQ | /ui/parked |
distill_failures |
parked units, retry |
| Embed | — | (embedding IS NULL count) |
unembedded floor → embed tier down |
| Dedup | /ui/review |
dedup_log |
verdict mix, pending holds |
| Recall | /ui/injections |
injections (+histogram) |
per-recall log, score vs floor |
| Resurface | /ui/feed |
feed_state |
weighted decay draws |
| Registry | /ui/projects |
projects |
attribution + centroids |
| Tasks | /ui/tasks |
task_items/task_events |
evidence-derived verified status |
Every arrow in the pipeline writes to one of these. That is the design contract: no stage moves data without leaving a queryable trace.
Key constants (verify in code)¶
| Constant | Value | File |
|---|---|---|
EmbedDim |
768 | embed.go |
embedNotesBatch |
64 | embed.go |
rrfK |
60 | hybrid.go |
projectScopeBoost |
1.25 | hybrid.go |
PromptRelevanceFloor |
0.015 | prompt_hook.go |
promptSubstanceFloor |
5 | prompt_hook.go |
promptHookMaxNotes |
5 | prompt_hook.go |
dedupSimilarityFloor |
0.80 | dedup.go |
dedupFlagConfidence |
0.70 | dedup.go |
DistillFailureCap |
5 | process.go |
SizeCap / TimeGap |
20 / 30m | segment.go |
DefaultRelatedMaxDist / DefaultClusterMaxDist |
0.35 / 0.22 | export.go |
See CLAUDE-CODE-HOOKS.md for the hook surface that drives capture/recall, and
research/2026-memory-best-practices.md for how this design maps to the 2026 field.