Ten Layers Between a Question and Its Context: Anatomy of a Unified Retrieve Call

When an agent asks one question about one customer, the vector search is only one of ten layers that decide what it gets back. Most of the quality lives in the layers around it.

One question, one call

An agent is about to draft a follow-up to a prospect. It makes one call:

{
  "mode": "brief",
  "record": { "email": "jane@acme.com" },
  "message": "What pricing did we discuss, and what discount rules apply?",
  "budget": { "totalTokens": 8000 },
  "synthesize": true
}

What it needs back is spread across four kinds of storage: typed properties on Jane's record (deal stage, plan), atomic memories from past calls ("asked about annual billing in September"), a section of the pricing guideline that governs discounts, and graph edges (who Jane reports to). I wrote about why those four sources need to come back in one call. This piece is about what happens inside that call.

The common mental model of retrieval is "embed the question, run a nearest-neighbour search, return the top k." In our retrieve_unified call, that nearest-neighbour search is one step in one of four sources. Most of the quality lives in the layers around it: which record you are talking about, which part of a document actually answers the question, what the caller is allowed to see, how one budget is shared, and what survives when a step fails.

Here are the ten layers, in order, with the real thresholds from the code.

1. One tool, five modes

The tool has five modes, and each mode turns a different set of sources on by default:

A caller can switch any source on or off explicitly, and an explicit false always wins. External sources are opt-in in every mode. If the caller leaves out the mode, it is inferred from the request shape: a fetch list means fetch, filters mean filter, a continuation id means expand, anything else is scout.

Depth is a separate knob (fast, standard, deep). The point worth copying is not the knob itself but the reporting around it. When a deep request times out or errors, the response says so: the document route carries the requested depth, the effective depth, the backend that served it and a fallback reason such as deep_timeout or deep_error. An agent that asked for deep and silently got fast will reason as if it had the better answer. Reporting the downgrade costs a few bytes and removes that whole class of confusion.

2. Resolve who you are talking about

Before any search, the target is resolved to a record. An email or phone number means a contact, a person-profile URL means a contact, and any other website means a company. Identifiers are normalized first: emails lower-cased, websites reduced to their domain, phone numbers to digits. Merged records are followed to the surviving record. A record that is pending deletion comes back as a bare tombstone with no properties, so the agent learns the record existed without getting its contents.

There is one quiet expansion here: other live records that share the same company domain are added as aliases, capped at 20. If that lookup fails, the call continues without them.

3. Understand the question (cheaply)

Three small things happen to the question before it touches an index:

• A keyword classifier assigns a coarse intent bucket. No model call.
• If the caller gave no explicit filters, a small model infers document types and tags from the question. It has a 5-second timeout and a rule-based fallback, and its filters are only applied when its confidence is at least 0.7. A wrong filter is worse than no filter, because it removes the right answer before ranking ever sees it.
• For document search, the question is widened with three rephrasings generated from different angles, appended to the embedding query. A caller can supply its own augmentation instead.

4. Recall from four sources

Each source answers a different kind of question, and each has its own recall strategy:

• Properties. The typed values on the resolved record. If no record resolves but there is a question, records are discovered by searching property meaning instead (minimum similarity 0.5, up to 10 records), and each hit says which property matched.
• Memories, in two tiers. Tier 1 is deterministic: the 25 most recent memories on the record. Tier 2 is semantic: a pgvector cosine search over memory embeddings (HNSW index, minimum similarity 0.5, up to 10), with tier-1 items removed. Tier 2 sits behind a feature flag and an organization allowlist.
• Documents. Embedding candidates, optionally fused with Postgres full-text search using reciprocal rank fusion with the standard k=60 (Cormack, Clarke and Büttcher, 2009). Rank fusion avoids pretending that a cosine similarity and a text-rank score live on the same scale. An optional cross-encoder rerank runs after fusion. Both hybrid search and reranking are behind flags.
• Graph. Active edges only, stub records excluded by default, one to three hops out, ordered by confidence and then weight.

Recall is deliberately generous at this layer. The next layer is where precision comes from.

5. Pick the section, not the document

Organizational documents are long. A pricing guideline might have twelve sections, and the agent needs one. Returning the whole document wastes budget and buries the rule that matters.

So the document path does something closer to what a person does with a table of contents:

1. A clear winner skips the model. If the top candidate scores at least 0.55 and leads the second by at least 0.12, it is returned directly. No model call, no added latency.
2. Otherwise a model picks sections. The selector sees the candidate documents and their section structure and returns at most three picks: the first needs a relevance of 3 out of 5, every additional pick needs 4. Only the chosen sections are cut out of each document.
3. A more specific sibling can replace the pick. If a candidate's name ends with the picked document's name and is longer ("Pricing Guidelines" versus "EMEA Pricing Guidelines"), and it scores at least 80% of the pick, the more specific document replaces it, keeping the same sections.
4. Fallbacks, in order. If the model picks nothing, the best non-system candidate scoring at least 0.30 is returned as a lower-confidence result. Up to three unpicked candidates scoring at least 0.65 come back as supplementary context. If everything is still empty, the best candidate at 0.40 or above is used.

That 0.65 floor has a history worth sharing. It was 0.50. The code comment that explains the change reads: department guidelines scored 0.50 to 0.64 on nearly every query and flooded results with the wrong documents. Thresholds in a retrieval pipeline are not constants you pick once; they are claims about your corpus, and the corpus changes.

6. Decide what not to return

This layer is the one that never shows up in demos. Every query is scoped to the organization. Records pending deletion, and everything attached to them, are hidden. Every document candidate is re-checked against fresh rows and filtered per caller, and the visibility check fails closed: if it cannot decide, the document is dropped. Documents attached to a specific record stay out of organization-wide search, so a note on one customer never surfaces as a "relevant guideline" for another. Explicitly requested documents are still visibility-checked and capped at 20.

Before anything reaches the synthesis model, personal data in the packed context is masked, and the values are restored in the answer afterwards. The audit log records the shape of the request, not the bodies.

7. One budget for everything

The caller sets one budget (up to 50,000 tokens). It is split across sources by default: memories 35%, documents 30%, external 20%, properties 10%, graph 5%. Shares of sources that came back empty are redistributed to the ones that did not. The response reports how much each source used and how many items each source had to drop, so an agent can tell "there was nothing" apart from "there was more, and it did not fit."

One detail I like: the budget counts UTF-8 bytes, not tokens. Bytes are a deliberately conservative upper bound that does not depend on any tokenizer, so a context that fits the byte budget always fits the model.

8. Synthesis, only when asked

The answer step is opt-in. Without synthesize, the agent gets the packed, cited context and does its own reasoning. With it, a model is told to answer using only the context, cite memories as [M#] and documents as [D#], and say plainly when the context does not contain the answer. It returns JSON with the answer, its reasoning, a confidence and the sources it used. A deferred option returns the context immediately and runs synthesis in the background.

9. Degrade, and say so

Every source reports ok, empty, error or skipped. If semantic memory search fails, the deterministic tier still comes back and the result is marked partial. If hybrid fusion fails, document ranking falls back to embeddings. If the reranker fails, the original order stands. If the section selector fails, a heuristic picks. If synthesis fails, the answer is null with an error, and the context the agent paid for is still returned.

A retrieval call that fails loudly is annoying. One that fails quietly and returns a confident partial answer is dangerous. Most of this layer exists to turn the second kind into the first.

10. Sessions, feedback and the paper trail

expand mode pages through a result using a session that remembers what was already returned (one-hour idle expiry, scoped to the organization). Agents can rate a result, and the rating is joined to the call that produced it. Every call writes an audit row, and billing is charged up front and refunded if the call throws.

Five ways it was silently losing evidence

Writing down layers is easy. Checking that evidence actually survives all ten of them is harder. When we built a measurement harness for this pipeline, it found five places where the right evidence was retrieved and then lost before the agent saw it:

1. Keyword-only matches dropped during fusion. A document found only by full-text search (an exact product code, say) could disappear when candidate lists were merged. Now lexical-only candidates are kept, re-checked for visibility, and the clear-winner shortcut is skipped when they are present.
2. Recent memories lost when semantic search failed. An error in the optional semantic tier could discard the deterministic tier with it. Now tier 1 survives and the failure is reported.
3. Evidence dropped before synthesis. Some retrieved items (secondary records, supplementary documents, late properties) were returned to the caller but never reached the synthesis prompt. Now everything authorized is packed, or its omission is reported.
4. Two budgets that disagreed. The context returned to the caller and the context sent to the model were sized separately. Now one packed budget feeds both.
5. Caller settings ignored. Some explicit controls, such as turning filter inference off, did not propagate on every path. Now they do, and the route that actually ran is reported per request.

None of these produced an error. Each one produced a slightly worse answer with full confidence. That is the case for measuring retrieval end to end instead of trusting that each layer works on its own. All five fixes are merged into our integration branch and moving through release.

What I have not measured yet

I would rather show the measurement before the number. So far we have run small, paid diagnostics on one layer: the section selector, over a public catalog of 25 guideline documents. On three paraphrases of one question it recovered the right document and section 11 out of 11 times. That is a smoke test, not an accuracy claim.

The diagnostics did teach us something about cost. Repeated identical requests hit the provider's prompt cache and cost roughly an eighth of a cold call. But a different question reused only about 45% of the prompt, because the prompt placed the question before the document catalog, so the cached prefix ended early. Moving the variable part to the end of a prompt is an old rule; it is easy to forget when the prompt grows by accretion.

The real benchmark is next: the 25 governance documents from our Governed Memory paper, extended with records, graph edges, distractors and cross-tenant traps, and 120 blind questions. It will report document and section accuracy, discovery of documents nobody named, leakage, latency and cost, per layer. I will publish the numbers, and the dataset, when they exist.

The principle

Retrieval is usually described as a search problem. In production it is closer to an editorial one: decide who the question is about, find more than you need, cut it down to the part that answers, remove what the reader may not see, fit it to the space, and be honest about what is missing.

Memory is just a polite word for a model of someone. Retrieval decides which part of that model an agent is looking at when it acts.