Export: related-note links + derived clusters (vault tier 2)

Goal: the Obsidian vault export gains a deterministic ## Related section per note (kNN over existing note embeddings) and a derived cluster frontmatter key (mutual-kNN connected components), so the graph view shows real structure instead of 2,818 disconnected dots.

Status: IMPLEMENTED 2026-06-12 (same day; spec reviewed by Aaron in conversation). Re-runnable: every export pass recomputes links and clusters from the live note set; same data in, byte-identical vault out.

Non-goals (explicit): - The distilled_notes.links BIGINT[] column stays untouched — it is reserved for the future LLM synthesis pass (typed links: supersedes / elaborates / contradicts). Tier 2 is computed at export time, render-only. - Episode/project hub pages (tier 1) — deferred, Aaron passed for now. - Cluster naming/labeling — clusters are anonymous component IDs; naming them is LLM-judgment work that belongs to the synthesis pass. - ANN index: brute-force kNN at 2.8k notes is milliseconds. Add HNSW (CREATE INDEX ... USING hnsw (embedding vector_cosine_ops)) only when live notes exceed ~30k; note it here, do not build it now.

Verified data facts (2026-06-12)

• distilled_notes.embedding vector(768) (nomic via LiteLLM embed tier).
• 2,818 of 2,818 live notes have non-NULL embeddings — full coverage, no backfill step needed.
• No vector index exists (only GIN fts) — acceptable at this scale.

Design

1. Neighbor query (one pass, lateral join)

For every live note (same liveness predicate Export already uses), fetch up to K=5 nearest other live notes by cosine distance, gated by a hard threshold:

SELECT n.id, k.id AS neighbor_id, k.dist
FROM distilled_notes n
CROSS JOIN LATERAL (
    SELECT m.id, (n.embedding <=> m.embedding) AS dist
    FROM distilled_notes m
    WHERE m.deleted_at IS NULL
      AND (m.valid_to IS NULL OR m.valid_to > now())
      AND m.id <> n.id
      AND m.embedding IS NOT NULL
    ORDER BY n.embedding <=> m.embedding
    LIMIT 5
) k
WHERE n.deleted_at IS NULL
  AND (n.valid_to IS NULL OR n.valid_to > now())
  AND n.embedding IS NOT NULL
  AND k.dist <= $1   -- ARGUS_EXPORT_RELATED_MAX_DIST
ORDER BY n.id, k.dist;

• Threshold, not just top-k: kNN always returns something; the floor is what keeps a loner note about the water heater from "relating" to five Postgres notes. Config: ARGUS_EXPORT_RELATED_MAX_DIST (cosine distance, float). Default 0.35 (calibrated against the real corpus, see below).
• No exclusions beyond self (KISS). If Related lists turn out to be dominated by same-episode siblings, add an episode exclusion then — with evidence, not preemptively.
• Notes with NULL embedding (future partial states) simply get no Related section and no cluster; they must not fail the export.

2. Rendering

Export becomes two-pass (it already collects all notes before writing):

1. Compute every live note's filename (existing note-<id>-<slug>.md logic) into the liveFile map FIRST.
2. Render each note appending, when it has kept neighbors:

## Related

- [[note-123-some-slug]] (0.31)
- [[note-456-other-slug]] (0.38)

• Wikilinks use the actual computed filename minus .md — links must resolve in Obsidian, so they come from the same liveFile map the cleanup pass uses, never re-derived.
• The cosine distance is rendered to two decimals deliberately: it lets the operator eyeball whether the threshold is right every time they read a note. Verifiability over prettiness.
• Notes with zero kept neighbors get no ## Related section at all (no empty headers).

3. Clusters (derived, mutual-kNN union-find)

• Edge set for clustering = mutual pairs only: A→B and B→A both survived the threshold. One-directional edges still render as Related links but do not merge clusters — this is the standard guard against transitive chaining collapsing the graph into one blob.
• Union-find over mutual edges → connected components. Component ID = cluster-<smallest note id in component> (stable across runs as long as that note lives).
• Frontmatter: cluster: cluster-123 on every note in a component of size >= 2. Singletons get NO cluster key (absence means unclustered — honest, queryable via Obsidian search -["cluster"]).
• Fallback noted, not built: if mutual-kNN still yields a giant component on real data, upgrade clustering to label propagation. Decide on evidence from the first real run, printed by the stats below.

FALLBACK TRIGGERED (first real run, 2026-06-12): at 0.35 the mutual edge set chained 2,400 of 2,564 clustered notes into ONE component (largest_cluster in the /export response is exactly the detector that caught it). The fix was NOT label propagation — a union-find sweep over the real top-5 edge list showed a clean phase transition:

cluster ceiling	mutual	clusters	largest	notes clustered
0.35	3,882	56	2,400	2,564
0.30	3,203	129	1,868	2,307
0.25	1,798	288	263	1,593
0.22	1,106	293	82	1,137
0.20	730	248	32	825

So clustering gets its own tighter ceiling, ARGUS_EXPORT_CLUSTER_MAX_DIST (default 0.22), while Related links keep 0.35 — topical links are useful even when too loose to imply a group. Mutual in ExportResult now counts the clustering edge set (pairs within the cluster ceiling).

4. Observability (ExportResult extension)

ExportResult gains:

Linked   int // notes that rendered a Related section
Edges    int // directed neighbor pairs kept (post-threshold)
Mutual   int // mutual pairs (clustering edge set)
Clusters int // components of size >= 2
Largest  int // size of the largest component (giant-blob detector)

All surfaced in the POST /export JSON response — so the nightly task's run detail in the ledger shows them, and a threshold misconfiguration (Linked==0, or Largest==Linked) is visible in /ui/runs without opening the vault.

5. Threshold calibration (one-time, before merging)

Run against the real corpus, eyeball, then write the chosen value into the compose env AND record it in this doc:

-- distribution of top-1 neighbor distances (what "related" looks like)
SELECT width_bucket(dist, 0, 1, 20) AS bucket, count(*),
       round(min(dist)::numeric, 2) AS lo, round(max(dist)::numeric, 2) AS hi
FROM ( SELECT (SELECT n.embedding <=> m.embedding
               FROM distilled_notes m
               WHERE m.id <> n.id AND m.deleted_at IS NULL
                 AND m.embedding IS NOT NULL
               ORDER BY n.embedding <=> m.embedding LIMIT 1) AS dist
       FROM distilled_notes n
       WHERE n.deleted_at IS NULL AND n.embedding IS NOT NULL ) t
GROUP BY bucket ORDER BY bucket;

Then sample ~10 note pairs at each candidate threshold (e.g. 0.30 / 0.40 / 0.50) and judge relatedness by hand. The chosen default goes in docker-compose.yml and is recorded here with the date.

CALIBRATED VALUE: 0.35 (2026-06-12, run against the live corpus of 2,818 embedded notes). Evidence: top-1 neighbor distance distribution peaks at 0.20-0.30 and the tail is empty past 0.42; hand-sampled pairs were unmistakably related at ~0.20, mixed at ~0.32, and still mostly topical at ~0.36 (the UniFi WiFi-tuning notes found each other; the duds were pairs like "task CRUD PR" <-> "reflection cron"). 0.35 keeps topic-level clusters while cutting the weak tail. Revisit only with new evidence (the rendered distances make marginal links auditable in-vault).

TDD plan (extends export_test.go, dockertest + pgvector)

Fixtures: insert notes with HAND-SET embeddings (e.g. unit vectors along distinct axes for "far", small perturbations of one vector for "near") so distances are exact and assertions deterministic — no LLM, no nomic.

1. TestExportRelatedSection — two near notes link to each other with the expected rendered distance; a far note renders no Related section.
2. TestExportRelatedThreshold — a pair just above the threshold renders nothing; just below renders. (Set threshold per-test, not global.)
3. TestExportRelatedLinksResolve — rendered wikilink text equals the neighbor's actual exported filename minus .md (slug drift guard).
4. TestExportClustersMutual — A↔B mutual => same cluster id cluster-<min>; C with only a one-directional edge to A keeps the Related link but NOT the cluster.
5. TestExportSingletonNoClusterKey — far note's frontmatter has no cluster: line.
6. TestExportDeterministic — two Export passes over the same data produce byte-identical files (sort order locked: neighbors by dist then id; map iteration never reaches the renderer).
7. TestExportNullEmbeddingSkipped — a live note with NULL embedding exports fine, no Related, no cluster, no error.
8. TestExportResultCounts — Linked/Edges/Mutual/Clusters/Largest match the fixture by hand-count.

Existing export tests must pass unchanged (the new section is additive; notes without neighbors render exactly as today).

Implementation notes

• Branch feat/export-related-links, PR per convention, merge only when Forgejo CI is green (merge-commit push, MCP merge tool is broken).
• After merge + deploy: trigger POST /export, verify counts in the run ledger, pull the vault on a device, and screenshot the graph view — the acceptance test for this whole feature is literally "the dot cloud grew edges and colorable clusters".
• The livesync-bridge will fan the rewritten files out to devices automatically; expect every note file to change once (Related sections appear) — a one-time ~2.8k-file sync burst, same size class as the initial sync the phone just survived.