Skip to content

Context-Injection Ranking — what actually helps (2025-2026)

Date: 2026-06-24 Trigger: ARGUS-76 cross-encoder re-rank shipped behind a flag, then the on/off eval (ARGUS-87) measured a wash: ~+6% precision@12 but ~-10% recall@12 with bge-reranker-base. This doc is the research into why, and what to do instead.

TL;DR — the reframe

The problem is set selection, not ordering. Modern LLMs (Claude included) are increasingly order-insensitive — they reason over multi-chunk context robustly regardless of rank order. A cross-encoder optimizes order / precision@K, which is the thing the consuming model cares least about. So reranking is the wrong lever for injection-into-an-agent. The field has moved to adaptive selection of a small, high-precision set + lost-in-the-middle mitigation.

(a) Why our reranker washed — expected, not a config bug

  • Domain mismatch + model size. Off-the-shelf cross-encoders degrade out-of-distribution; OOD generalization needs larger models. bge-reranker-base is the small end, and our corpus (CC sessions, dev decisions, syntheses) is OOD vs its web/QA training. Reranking a noisy candidate pool produces exactly our symptom — recall decline + high-scoring false positives. (Déjean et al. 2024, arXiv:2403.10407.)
  • Rerankers optimize the wrong objective. They optimize ranking metrics (NDCG/MRR), not downstream decision correctness (CalibRAG). A listwise LLM reranker matched but did not beat a simpler approach on a real downstream metric, at +40% latency / +15% cost (Fin.ai A/B report).
  • Order-insensitivity. GPT-4o / Claude 3 / Gemini 1.5 handle multi-chunk reasoning regardless of order, undercutting reranking's main benefit.

(b) Lost-in-the-middle — this challenges maxNotes=12

Injecting more notes (high recall) actively hurts generation: relevant info in the middle of a long context is under-attended — a U-shaped accuracy curve (Liu et al. 2023). A benchmark showed RAG accuracy drop 33% when the relevant chunk sat in the middle of 10. Current guidance: fewer, higher-precision chunks (3-5), don't dump 12-20.

This is in direct tension with our maxNotes 8->12 bump, which optimized retrieval recall. 12 notes may be past the point where the consuming model loses the middle ones. Retrieval wants recall; generation wants few-and-precise. We optimized only the retrieval side. Re-examine maxNotes; let adaptive-K set it per query.

What actually solves "precision without losing recall" (priority order)

  1. Adaptive-K (top recommendation). Replace the fixed cap with a dynamic cutoff from the score distribution / a confidence gap — 3 notes on a focused query, more on a broad one. Solves the exact problem: no force-drop of relevant (it's adaptive), no junk dump (it's thresholded). Plug-and-play, single-pass, no training, runs on the pgvector scores we already compute. (Adaptive-k, Taguchi et al. 2025, arXiv:2506.08479; Cluster-based Adaptive Retrieval, arXiv:2511.14769; CRAG confidence evaluator.)
  2. Lost-in-the-middle reordering (cheap). Place the top notes at the start and end of the injected block, not the middle. Trivial; our injection is already an ordered block.
  3. Structural signals. Shared-entity / same-project / same-file co-retrieval (Mem0's real lever). We have project-boost; file-path + entity matching are reliable dev-memory cues that cosine misses.
  4. Temporal validity, not recency decay. Suppress superseded facts (a reversed decision, a changed preference) — Zep's win. Distinct from recency decay, which we tested and which hurt (dev knowledge is durable). About correctness, not freshness.
  5. HyDE as a recall fallback for short/vague prompts, gated on low first-stage confidence only (it adds +25-60% latency + hallucination risk; not always-on). (Gao et al. 2022; medical fine-tune +4.9% NDCG@10, Li et al. 2024.)
  6. Cross-encoder: keep OFF. Correct from the eval, now backed by the literature. If ever revisited, fine-tune on our own grade labels rather than a bigger off-the-shelf model.

(c) What production agent-memory systems actually do (2025-2026)

None rank by cross-encoder. - Mem0: semantic + BM25 fusion (≈ exactly our RRF) + shared-entity co-retrieval. - Zep/Graphiti: + temporal validity windows + graph traversal + community summaries (beats Mem0 on temporal benchmarks). - Letta/MemGPT: agent-driven search (the agent decides what to page in), tiered store, no fixed ranker.

The field ranks by structural + temporal signals + LLM/agent selection, not rerankers. External confirmation our energy belongs in (1)-(4).

(d) Evidence caveats (honesty)

Much of the downstream-outcome evidence is industry blogs, not peer-reviewed — treat percentages loosely. Reranking does help some QA tasks; task type is the decisive variable. Lost-in-the-middle is being mitigated by newer order-robust models, shrinking its severity. But the direction — adaptive selection over reranking — is consistent across academic and practitioner sources.

Decision for Argus

Pivot from "make the reranker work" to set selection: ship adaptive-K (biggest, best-evidenced win) + start/end reordering, then structural signals; re-examine maxNotes=12. Leave the reranker built-but-off (ARGUS-85/86 are merged and flag-gated, so they cost nothing idle). Higher-leverage and pgvector-native vs chasing a bigger cross-encoder.

Sources

  • Déjean et al. 2024, cross-encoder/LLM/SPLADE comparison — arXiv:2403.10407
  • Lost in the middle (Liu et al. 2023) — via arXiv:2512.14313 (Dynamic Context Selection)
  • Adaptive-k (Taguchi et al. 2025) — arXiv:2506.08479
  • Cluster-based Adaptive Retrieval (CAR) — arXiv:2511.14769
  • HyDE (Gao et al. 2022); medical iterative fine-tune (Li et al. 2024)
  • Production framework comparisons (Mem0/Zep/Letta) — particula.tech, agentmarketcap.ai, 2026
  • CalibRAG; Fin.ai listwise-rerank A/B; CRAG; Self-RAG (practitioner + arXiv)