# The AI Personalization Handbook

A practitioner's reference for AI personalization, from the first decision to production.

By Hamed Taheri. Source: https://hamedtaheri.com/handbook/

---

# Introduction: Hi {first_name}

*The question: Why this handbook, and how should I read it?*

Larkspur Systems (a fictional company, a composite of many real ones) sells field-service and fleet software. At nine on a Tuesday its quarterly newsletter goes out to 250,000 contacts. The first line reads "Hi {first_name}". For a few thousand contacts the field is empty, so the fallback fires and they get "Hi there".

On that list, one customer's renewal is thirty days out, and the account has not logged in to the reporting module since spring. Another customer's operations director opened three support tickets this week about route sync failing overnight; she is, reasonably, angry. A third account just hired a new head of operations, whose name the CRM picked up from a signature line in a support thread. A fourth has doubled its fleet and is quietly outgrowing its plan.

Larkspur knows all of this. The renewal date is in the CRM, the tickets in the support desk, the usage drop in product telemetry, the new hire in an email thread.

And every one of those people receives the same four paragraphs about the spring product release.

You have sent this email. So have I. Nobody at Larkspur is careless, and the newsletter is well written. The problem is not that the company does not know. It is that knowing and saying have never been connected at the scale of a customer list.

![Larkspur (fictional) already knows four customers' situations: one account renews in 30 days and has not used the reporting module since spring; an angry operations director opened three tickets this week about route sync failing overnight; a new head of operations was picked up from a support signature; and one account has doubled its fleet and is outgrowing its plan. These facts sit in the CRM, the support desk, product telemetry, and email threads. Yet all four, along with 250,000 contacts, receive the same Tuesday 9:00 a.m. newsletter: "Hi {first_name}" (or "Hi there" when the field is empty) and the same four paragraphs about the spring product release. The gap is not data; knowing and saying were never connected at the scale of a list.](/images/handbook/ch00-knowing-not-saying.svg)

*Figure 0.1. Larkspur knows each customer's situation, and every one of them still gets the same email.*

## The argument in one paragraph

Larkspur is not unusual. Most companies know far more about each customer than they ever say to that customer. The gap is not laziness. It is the old cost of understanding: going deep on one person took a skilled human's hours, so depth was rationed to a few accounts and everyone else got a template. AI is the first technology that can do deep and wide at once. The winners will not have the cleverest prompts. They will build a memory holding an accurate model of each customer, a decision layer that chooses the right next action for one person (including doing nothing), and governance that runs inside the system rather than beside it. This handbook explains how to build those, how to prove they worked, and where the honest limits are. Use the reading paths below rather than reading front to back.

> **Questions this introduction answers**
> - We already know a lot about our customers. Why do our messages not show it?
> - What does AI actually change about personalization, in one paragraph?
> - Which chapters should I read first, given my job?
> - How do I hand this handbook to my own AI and get something useful back?

## Nobody wants to be a merge field

Customers can tell when they are a row in a list. In Salesforce's 2023 survey of 14,300 consumers and business buyers across 25 countries, 61% said most companies treat them as a number, and 73% of business buyers said most sales interactions feel transactional [2]. These are survey responses, not behavior. They are still a fair description of what "Hi {first_name}" feels like from the other side.

The fix is not to stop personalizing, and I want to be precise about that. In randomized field experiments, adding the recipient's name to an email subject line raised open rates from 9.05% to 10.80% and reduced unsubscribes [1]. The merge field earns its keep. But notice what it cannot do. It cannot tell the angry operations director that the sync problem is known and fixed. It cannot tell the renewing account what it stopped using and why that matters. The first name was never the problem. It was the ceiling.

So here is the sentence this whole handbook turns on. Most companies do not have a data problem. They have a depth problem.

Depth is the ability to turn what you already hold about customers into an understanding of each one, reason about what that person needs and plan what to do, then act on it with a message they actually care to read. The customer cares because the message is about their situation. You care because that is where the business value is: renewals that close, churn that does not happen, loyalty, referrals, and revenue.

When personalization disappoints, the instinct is to blame the data. Some of that blame is earned, and Part III is about fixing it. But look again at Larkspur. The facts that matter most to each of those four people already exist. What is missing is the depth to read them for each person, work out what they mean this month, and decide what to say or do. Until now, that took a person.

## Real personalization never scaled, until now

Real personalization has always been the most expensive thing a business does, and the least scalable. To write something that genuinely fits one customer, someone had to understand that customer: their situation, their language, what they were trying to get done. That took hours, and skilled hours are finite. The account manager who knows forty clients well cannot know four thousand. The marketer who reaches four thousand cannot know any of them. So attention was rationed, and most customers got the template.

That is the frame this handbook is built on (Chapter 4 develops it). Picture a grid: depth of understanding on one axis, scale on the other. Human one-to-one service (account managers, advisors, concierge desks) is deep and few. First-era machine personalization (merge fields, segments, recommenders, propensity scores) is wide and shallow. The corner that stayed empty was deep and wide, because the arithmetic did not allow it. AI changes the arithmetic. For the first time there is a path to scale what used to need a person: reasoning about one customer's situation, empathy in how something is said, clear communication, even the right touch of humor. Not the feeling itself, but the work those qualities used to require. Reading a support thread, a call transcript, and years of usage history, then reasoning about what they mean for one account, is now machine work, across an entire customer base.

The idea itself is not new, and I should say so. Don Peppers and Martha Rogers argued in 1993 that companies should compete on share of customer, building relationships "one customer at a time" [3]. Much of this handbook's spirit owes them an acknowledgment. What they could describe but not deliver was the cost of understanding each person. That cost is what has fallen.

## What this handbook is, and is not

It is a practitioner's guide, written by someone who builds these systems. I built and run a governed personalization engine that turns one record of a person, an account, and a campaign into a web page, an email sequence, and a seller brief, with code checks, safe fallbacks, and delivery recovery wrapped around the model. I also built a self-hosted memory system that keeps that record in the customer's own database and tidies it while idle, proposing its own corrections before an operator lets it write them. The memory architecture behind both is described in a public paper [4]. So the handbook shows the mechanism, the cost math as ranges, and the failures.

It is not an argument for maximal personalization. More detail about a person often makes a message worse, and one correct fact that changes what should be said is worth more than a hundred that only decorate it. The standard throughout is simple: be specific and true, or be honestly general.

It is not a vendor catalog, not a promise of revenue, and not a manual for pressuring people. Any system precise enough to serve a person is precise enough to pressure them, which is why governance gets an entire Part rather than a closing paragraph.

## Who this is for, and where to start

Seven readers, one text. Every chapter opens with a short answer a leader can stop at, then goes deeper for the people who build.

| If you are | Your Monday | Start with |
|---|---|---|
| **CEO** | "We have ten years of customer data. Why doesn't it make us money?" | Ch 1, 4, 6, 14 (short answer), 21, Closing |
| **CMO** | The board wants "our AI personalization story"; the team runs merge fields | Ch 1, 2, 4, 6, 15, 16, 21; Playbooks M1 to M3 |
| **CRO** | AI SDR tools produce plausible spam and reply rates fall | Ch 1, 4, 10, 15; Playbooks S1, S2, and One Customer, One Conversation |
| **Product leader** | An experience should adapt, and it is unclear whether rules, a model, or an LLM should drive it | Ch 1, 3, 4, 14, 16, 17, 21; Playbooks Product and Software Companies |
| **CTO or architect** | Seven teams, seven personalization stacks, no shared memory | Ch 3, 9 to 13, 16, 18, 20, 22 |
| **AI builder or GTM engineer** | You can wire agents in a weekend but were never trained in personalization | The companion repo, then Ch 8 to 17, 20, 22 |
| **Software founder or customer success leader** | Most trials never convert, and the health score is green on accounts that leave | Ch 1, 6, 10, 14, 21; Playbooks Software Companies and Customer Success |

The handbook has two layers. **Foundations** (Parts I to VII, twenty-two chapters between this introduction and the closing) holds the durable ideas: the case, where to start, customer data, memory, decisions, trust, and proof. **Playbooks** (Part VIII, eleven playbooks) applies them by function (marketing, sales, support, customer success, product, research, and the coordination across them) and, in one case, by business model (software companies). That layer changes as channels and platforms change.

![The handbook has two layers. Foundations, Parts I to VII, hold the durable ideas in twenty-two chapters framed by the Introduction (Chapter 0) and the Closing (Chapter 23): Part I, the case (Chapters 1 to 5); Part II, where to start (Chapter 6); Part III, customer data (Chapters 7 to 9); Part IV, memory (Chapters 10 to 13); Part V, decisions (Chapters 14 to 17); Part VI, trust (Chapters 18 to 20); and Part VII, proof (Chapters 21 and 22). Playbooks, Part VIII, apply those foundations by function and business model in eleven playbooks: marketing (M1 to M3), sales (S1, S2), support (SP), customer success (CS), product (PR), research (RS), one customer, one conversation (P9), and software companies (SW); they change as channels and platforms change.](/images/handbook/ch00-handbook-map.svg)

*Figure 0.2. Seven parts of durable foundations, applied by function in playbooks that change with the channels.*

## How to use this with your AI

This handbook has two readers: you, and the AI you will ask to help apply it. Every chapter ends with a block called **For your AI**: structured YAML holding the chapter's concepts, its decision rules, the questions an assistant should ask about your company, and its named patterns and anti-patterns. Concept names stay stable across chapters, so an assistant can connect them. Every chapter is also published as clean Markdown, and the site offers an `llms.txt` index and a single full-text file, so "give the whole handbook to your AI" is one link: the index at [hamedtaheri.com/llms.txt](https://hamedtaheri.com/llms.txt) and the full text at [hamedtaheri.com/handbook/full.md](https://hamedtaheri.com/handbook/full.md).

Two skills in the companion repo turn reading into a plan. **Assess** interviews you about your company and scores you on six maturity dimensions, with the evidence and the gaps. **Plan** takes that assessment and the handbook and drafts your first three decisions to personalize, a starter memory schema, governance rules, a measurement plan, and a 90-day build sequence. It is built to make gaps visible, not to flatter.

## The company you will watch change

Larkspur will reappear in most chapters. It has about 40,000 customer accounts, 250,000 contacts, ten years of CRM history, a warehouse, product telemetry, events, a support desk, and a small direct-mail program. It starts where most companies start: five segments, merge fields, a quarterly newsletter, and an AI SDR pilot that is making reply rates worse, not better.

Each chapter moves it one step: choosing its first decisions, fixing identity, building memory, adding a decision layer, governing, measuring, deploying. The closing chapter shows Larkspur a year later.

The goal is not a better newsletter. It is that the renewing customer, the angry operations director, the new head of operations, and the growing fleet each hear something true and useful about their own situation, or hear nothing at all when nothing is the right answer.

Personalization was never about the first name. It is about knowing one person well enough to decide what they need, and doing that for everyone.

## References

1. Sahni, N. S., Wheeler, S. C., and Chintagunta, P. (2018). "Personalization in Email Marketing: The Role of Noninformative Advertising Content." *Marketing Science* 37(2). https://pubsonline.informs.org/doi/abs/10.1287/mksc.2017.1066
2. Salesforce (2023). *State of the Connected Customer*, 6th edition. Double-blind survey of 14,300 consumers and business buyers in 25 countries, fielded May 3 to July 14, 2023. Report PDF (hosted copy): https://event.hbrturkiye.com/storage/uploads/state-of-the-connected-customer-655b114557dfa.pdf
3. Peppers, D., and Rogers, M. (1993). *The One to One Future: Building Relationships One Customer at a Time.* Currency/Doubleday.
4. Taheri, H. (2026). "Governed Memory: A Production Architecture for Multi-Agent Workflows." arXiv:2603.17787. https://arxiv.org/abs/2603.17787

---

# 1. What Personalization Is, and What It Is Worth

*The question: What is personalization, and why should a leader care now?*

> **Questions this chapter answers**
> - What do we actually mean by "personalization," and how is it different from segmentation or customization?
> - Is it worth the investment, and how much is it worth for us?
> - What should we be optimizing for: clicks, revenue, or something else?
> - Why does some personalization feel helpful and some feel creepy?
> - What is the minimum quality bar before we let a machine speak to a customer as an individual?

## The short answer

Personalization is not a feature, a merge field, or a data asset. It is a decision: choosing what to say, show, offer, or do for one specific person, based on what you know about them, instead of what you would do for everyone. Holding a lot of personal data is not the same as making personalized decisions. Most companies have the first and very little of the second.

The evidence says it is worth doing, within honest limits. In one randomized field experiment, adding a recipient's name to an email subject line raised opens by about 20% and leads by about 31% [4]. McKinsey reports typical revenue lifts of 10 to 15% for companies that do it well [1]. The evidence also shows the downside. In a 2024 Gartner survey, 53% of customers reported a negative personalization experience, and those customers were 3.2 times more likely to regret a purchase [3]. Personalization shifts the odds; it does not command outcomes. Done badly, it shifts them the wrong way.

So judge every personalized decision on four separate tests: relevance, usefulness, timing, and appropriateness. Measure its worth per decision, as lift over a real baseline, net of cost and risk. And hold one standard above the others: every personal claim is either **specific and true, or honestly general**. A message confidently wrong about a customer's own life costs credibility you had before you sent it.

Why now: the cost of understanding one person deeply has fallen far enough that depth no longer has to be rationed to your largest accounts. That changes which decisions are worth personalizing, and raises the price of getting them wrong at scale.

---

You have sent both kinds of message.

On a Tuesday morning at Larkspur Systems (a fictional mid-market software company that runs through this handbook), a customer success manager writes one email to one fleet operations director. She names the thing the director is actually dealing with this month: a new depot opening in two weeks, and the dispatch rules nobody has configured for it yet. She offers a thirty-minute setup call before the opening.

The same morning, the quarterly newsletter from the introduction goes out to 250,000 contacts. It produces the usual open rate and a few unsubscribes. The single email gets a reply within the hour that says thank you.

You already know the difference, because you enact it on both sides. As a customer, you delete the message written for everyone without quite deciding to, and you answer the one written for you. As an operator, you know which of the two your company can afford to send 250,000 times, and which it can afford only a few dozen times a week, by hand.

That gap is what this handbook is about. This chapter defines it, prices it, and sets the bar for crossing it.

## Personal data is an input. Personalization is a decision.

The word "personalization" is used for at least five different things, and the confusion is expensive, because teams buy tools for one and expect the results of another.

| Term | Who chooses | What varies | Example at Larkspur |
|---|---|---|---|
| **Segmentation** | The company, per group | Treatment per segment; everyone in a segment gets the same thing | Five segments by fleet size, one email version each |
| **Customization** | The person | What the person configures for themselves | A dispatcher sets their own dashboard layout |
| **Personalization** | The company, per person | The treatment for one individual, based on what is known about them | The renewal email mentions the depot this customer is opening |
| **Adaptive** | The system, over time | The treatment adjusts as the person responds | Onboarding steps reorder based on which features a new customer actually uses |
| **Agentic** | An AI system, within bounds | Research, decision, action, and follow-up, over several steps, for one person | An agent researches the account, drafts the note, schedules the follow-up, and waits for the reply before deciding what comes next |

These are not rungs you must climb in order; a mature program uses all five. But only the last three are decisions made about an individual. Segmentation decides for a group. Customization lets the customer decide.

A company can hold ten years of CRM history, product telemetry, support tickets, and call transcripts and still make no personalized decisions at all. Larkspur is exactly this company. It knows which accounts are renewing, which are angry, which just hired a new operations lead. It says the same thing to all of them. Data in a warehouse is potential. Personalization happens only at the moment a decision about one person uses it.

That tells you where to look for value: not in "our data," but in the repeated decisions your company makes about individual customers every day. What to send, what to show, what to offer, whom to call, when to wait. Chapter 6 turns that into an inventory. For now, hold the definition: **personalization is choosing the next action for one person, using what you know about them, where the choice would have been different for someone else.** If the choice would have been the same for everyone, it was not personalized, whatever the template says.

## A good personalized decision passes four separate tests

Teams tend to ask one question: "Is it relevant?" That question hides four different failures, with different causes and different fixes.

1. **Relevance.** Does it concern something this person has, does, or cares about? Fleet electrification pitched to a customer with no electric vehicles fails here. Usual cause: bad or missing data.
2. **Usefulness.** Does it help them do, decide, or avoid something? "We noticed you use our dispatch module!" is relevant and tells the customer nothing. Usual cause: no point of view about what the person needs.
3. **Timing.** Does it arrive when it can still matter? The setup offer two weeks before the depot opens is service; two weeks after, it is proof you noticed too late. Usual cause: batch schedules that ignore the customer's clock.
4. **Appropriateness.** Would this person expect, and accept, you knowing and using this here? Usual cause: a system that treats "we have it" as "we may use it."

![Four side-by-side panels show the four tests a personalized decision must pass, each with its question and its usual cause of failure: relevance (is it about something they have, do, or care about; bad or missing data), usefulness (does it help them do, decide, or avoid something; no point of view on need), timing (does it arrive when it can still matter; batch schedules that ignore their clock), and appropriateness (would they expect and accept you using this here; treating "we have it" as "we may use it"). All four feed one gate: the decision ships only when all four pass, scored separately with one owner per test.](/images/handbook/ch01-four-tests.svg)

*Figure 1.1. "Is it relevant?" hides four independent tests; a decision ships only when it passes all four.*

The tests are independent, and the failed one is often the one the customer remembers. Gartner's 2025 research found that personalization relevant at one stage of a purchase became irrelevant when the buyer moved to a harder task, and that customers receiving personalization were twice as likely to feel overwhelmed by information [3]. Relevance without usefulness and timing produces more messages, not better ones.

## What the evidence actually shows

Personalization statistics are among the most recycled numbers in business writing, and many of them cannot be traced to a method. So sort the evidence by what kind of claim it is before deciding how much weight it carries.

**Surveys of what customers say.** McKinsey's 2021 research reported that 71% of consumers expect personalized interactions and 76% get frustrated when they do not get them [1]. Salesforce's 2022 survey of more than 13,000 consumers and nearly 4,000 business buyers found 73% expect companies to understand their unique needs [5]; its 2023 edition found 61% feel treated as a number [6]. These are real findings about stated expectations, not measurements of behavior. A widely quoted 2018 figure, "80% of consumers are more likely to purchase" when brands personalize, came from a vendor survey of 1,000 US consumers asked about their intentions [7]. It is a fair example of how industry numbers are made, and a poor basis for a budget.

**Reports of what companies experienced.** McKinsey also reported that personalization most often drives a 10 to 15% revenue lift, with company-specific results from 5 to 25%, and that faster-growing companies drive 40% more of their revenue from personalization than slower-growing peers [1]. These come from a consultancy's client experience, not controlled experiments, and the second is a correlation: growing companies may personalize more because they are growing. Treat the range, not the midpoint, as the finding.

**Field experiments.** This is the strongest evidence, and it is narrower. Sahni, Wheeler, and Chintagunta ran randomized email experiments with real companies: adding the recipient's name to the subject line raised the open rate from 9.05% to 10.80%, raised sales leads by 31%, and cut unsubscribes by 17% [4]. The name told the reader nothing new. It shows how much attention the first signal of "this is for you" can buy, and also the ceiling of shallow personalization: the absolute rates stay low.

**Experiments on deeper tailoring.** Matz and colleagues reported in 2017 that ads matched to inferred personality, reaching about 3.5 million people, produced up to 40% more clicks and 50% more purchases [8]. Critics pointed out that ad-platform delivery was not fully randomized, which can inflate the effect [9]. A 2026 meta-analysis of 41 studies found that digital footprints predict personality only weakly and that personality-tailored messages have negligible average effects on behavior [10]. The honest reading: tailoring on a thin, inferred trait is a weak lever, and effects depend on what is inferred, how accurately, and what outcome is measured.

**Evidence of the downside.** Gartner's late-2024 survey of 1,464 B2B buyers and consumers found that 53% had experienced negative personalization, which made them 3.2 times more likely to regret a purchase and 44% less likely to buy again [3]. Twilio Segment's 2024 report found 61% of consumers worried that inaccurate data would undermine AI personalization [11].

Here is what it adds up to. Personalization does not pull a lever inside a specific person. It shifts the odds, a little, across a great many people, and a small shift across many decisions is worth a great deal. It is a game of rates, not of certainties. And the shift can go either way: the same machinery that earns a reply can earn a regret.

## The worth of personalization is measured per decision

No company buys personalization in aggregate. It buys better decisions, one decision type at a time. So price it that way.

**Value per decision = (outcome value) × (lift over baseline) − (cost to make the decision) − (expected cost of getting it wrong)**

- **Outcome value**: what one improved result is worth (a renewal saved, a meeting booked, a support contact avoided), in revenue, retention, cost to serve, or team time.
- **Lift over baseline**: the incremental change against your current generic treatment, not against doing nothing. Chapter 21 covers how to measure it.
- **Cost to decide**: data, compute, review time, delivery. It has fallen sharply for research and drafting (Chapters 4 and 5).
- **Cost of being wrong**: the term most business cases leave out. The Gartner numbers above are the reason to put it back.

Work one example with the assumptions visible (all figures illustrative). Larkspur has roughly 6,000 at-risk renewals a year. Suppose a saved renewal is worth $6,000 to $15,000 in annual contract value, and a researched, account-specific renewal plan improves the save rate by 1 to 4 percentage points over the generic sequence. That yields 60 to 240 additional saved renewals, worth roughly $0.36 million to $3.6 million a year. The range is wide, and it should be. The assumption carrying the weight is the lift, and only a test narrows it. Subtract research, drafting, and the expected cost of a damaging message, and you have a number you can defend, because every term can be checked.

![The formula for value per decision: outcome value times lift over the generic baseline, minus cost to decide, minus expected cost of being wrong, with the last term highlighted as the one most business cases leave out. Below it, the illustrative Larkspur (fictional) renewal example: 6,000 at-risk renewals a year times a 1 to 4 point save-rate lift gives 60 to 240 more saved renewals, and times $6,000 to $15,000 per saved renewal gives $0.36 million to $3.6 million a year before costs. A to-scale bar on a $0 to $4 million axis shows how wide that range is; the lift carries the weight, and only a test narrows it.](/images/handbook/ch01-value-per-decision.svg)

*Figure 1.2. Price each decision type as a range; the lift assumption drives the width, and only a test narrows it.*

Run the same arithmetic on the quarterly newsletter. The volume is large, but the value per recipient is small and the achievable lift modest. This is the most common error in first programs: personalizing the highest-volume touchpoint because it is easiest to automate, not the highest-value decision.

What should the decision optimize for? Five objectives, held together: **customer value** (did the person get something useful?), **business value** (did the outcome move, incrementally?), **long-term trust** (will they be more willing to hear from you next time?), **cost**, and **risk** (the worst credible outcome if the system is wrong). Optimizing only business value, usually through a proxy like opens or clicks, is how programs produce rising short-term numbers and falling relationships. Personalization that serves the customer tends to serve the business. The reverse is not guaranteed.

## The standard: specific and true, or honestly general

Here is the sentence this chapter turns on. **Dependable, high-quality personalization depends on an accurate model of the individual person.** Not a first name dropped into a template. A working understanding of who this person is, what they are dealing with, what they are trying to do, and what a message has to look like to belong in their situation.

Shallow personalization can create measurable lift, as the subject-line experiment shows. Depth changes the ceiling and the reliability. A shallow model produces the uncanny near-miss: the email that uses the customer's name and gets their role wrong, leaving the sender worse off than if it had stayed general. When I built systems to do this across hundreds of thousands of people at once, the property that decided whether the whole thing worked was never how impressive one generated output looked. It was whether the average output was relevant enough to earn a real response, and how rarely it said something confidently wrong about the reader's own business. The model is the machine. The message is just its exhaust.

A plain, hand-written template that names a real need will often beat a fluent machine-written message built on a shallow understanding. The reader does not reward eloquence. They reward being understood.

Accuracy does not mean the largest possible profile. More detail makes a message worse when it is irrelevant, stale, or included to prove the sender found it. What matters is decision-level fit: the real constraint, priority, or unfinished problem that makes the message useful now. A fact earns its place in a message by changing what the message should say; facts that only decorate it add length, cost, and risk.

Hence the standard this handbook holds every system to: every personal claim is either specific and true, or honestly general. When evidence is strong and recent, speak to the specifics. When it is thin or old, drop to what is true about the company or the industry, and say nothing you cannot support. A well-written general message is a success. A specific-sounding message built on an invented or outdated detail is a failure even when it reads better. Accuracy is not the goal; it is the floor. Usefulness is the goal, and you cannot be useful and wrong at the same time.

In the governed personalization engine I built for go-to-market teams, this standard is a ladder enforced by the system, not a line in a prompt. As evidence thins, output steps down from person-and-account copy to account level, then segment level, then approved static copy, and finally to sending nothing. The rule the design serves: specificity degrades before trust does. The near-miss it catches most is not an invented fact but an assumed relationship: "your renewal is coming up," "you already use our dispatch module," "your current contract." Unless the relationship is actually recorded, the claim is blocked and the copy is reframed as an opportunity to evaluate. A renewal reminder sent to someone who never bought tells them you do not know who they are.

![A three-tier ladder of specificity: person (speak to the specifics, when evidence is strong and recent), company (what is true about the company), and industry (what is true about the industry, when evidence is thin or old). A green downward arrow says thin or old evidence drops a tier; a crossed-out red upward arrow says never climb a tier by inventing a detail. Below, a well-written general message is marked a success, and invented or outdated specifics are marked a failure even when they read better.](/images/handbook/ch01-honestly-general.svg)

*Figure 1.3. Choose the specificity tier from the evidence; dropping a tier is a success, inventing to climb one is a failure.*

## Known, not watched

Appropriateness is where most personalization anxiety lives, and the research on it is clearer than the anxiety suggests.

Researchers call the tension the personalization paradox: people want to be recognized, and they feel exposed when they learn how much is known. It resolves once you look at *how* the knowledge was obtained. Aguirre and colleagues found in 2015 that personalized ads raised click-through when information was collected overtly and lowered it when collection was covert, because covert collection raised customers' sense of vulnerability [12]. Kim, Barasz, and John found in 2019 that explaining why someone saw an ad helped when the information flow matched their expectations, and hurt when it revealed flows they considered unacceptable, such as tracking across other sites or inferring what they had not shared [13].

The common thread is expectation. Customers do not object to being known. They object to being watched. A Larkspur customer expects Larkspur to know their fleet size and their open support tickets. They do not expect it to know about a personal detail found on another site, or an inference about their life drawn from usage patterns. The same fact can be service in one context and surveillance in another. Interface designers learned the same lesson about expectation decades ago: people accept adaptation they can predict, see the reason for, and correct, and they switch off what moves without warning (Chapter 3).

The most retold example is Target's 2012 pregnancy-prediction score, reported by the *New York Times* [14]. The score is reported fact; the famous anecdote about a father learning of his daughter's pregnancy from coupons is single-source and should not be repeated as confirmed. Either way, the lesson stands: accuracy and appropriateness are separate tests, and passing the first does not pass the second.

One honest line, once. Any system precise enough to serve people well is precise enough to pressure them. The difference is not in the model. It is in the objective and the governance, which is why later chapters treat both as engineering, not policy.

## What this does not do

This is where a chapter like this can lose the right to be believed by exaggerating, so here are the limits.

Personalization does not fix a weak offer. If the product does not solve the customer's problem, a message that describes the problem precisely only makes the gap clearer.

It does not improve every decision. Often the generic treatment is already near the best available, and personalizing it adds cost without lift. Part of a good program is deciding where not to personalize.

It does not scale with data. More data, especially inferred data, raises the risk of a stale, wrong, or inappropriate fact faster than it raises relevance.

The popular claims deserve pushback from both sides. In 2019, Gartner predicted that 80% of marketers who had invested in personalization would abandon it by 2025 for lack of ROI and data problems [2]. It is often quoted as a finding; it was a forecast. Its underlying point holds: programs built on data volume instead of decision value fail on ROI. At the other end, the "80% more likely to purchase" statistics measure intention, not purchases. I do not accept either as a planning number. The planning number is your own lift, on your own decisions, against your own baseline.

## At scale

At a few dozen accounts, a skilled person passes the four tests by judgment. At 250,000 contacts, judgment has to become architecture.

Errors become rates. A system wrong about a customer 2% of the time is wrong 5,000 times in one Larkspur send, so quality is measured on the population, not the demo account. Consistency becomes visible: when marketing, sales, and support each personalize from their own copy of the customer, the customer receives contradictions, which read as not being known at all. And a creepy or false message at scale is not a bad email; it is a brand event. "Specific and true, or honestly general" has to be enforced in code, before and after generation, not left to a prompt. Generative does not mean uncontrolled: use the model where judgment adds value, and code where certainty is available.

Success also splits into more states than a demo shows. In that engine, one record can succeed at generation (an output exists) and still fail on content (incomplete, unsafe, or inappropriate), on persistence (the next process cannot yet read it), or on delivery (the customer never received the right artifact). Above those sits a fifth: across hundreds of records, is the program behaving as agreed? A personalized decision has been made only when all five hold. Chapters 17 and 20 build that machinery; a demo shows only the first.

## Failure story: The Creepy Reveal

Larkspur's AI SDR pilot starts from thin list data (name, title, industry, company size) and adds whatever an ungoverned web search returns for the prospect's name and company. One prospect, a fleet manager at a regional utility, receives an opening line that congratulates her on "your recent move from Denver" and mentions a conference talk she gave three years earlier under her previous employer's name.

Every fact is true. None came from a relationship with Larkspur. She did not expect a vendor she had never spoken to to know where she lived, and the old conference detail felt like digging. She replies with one line asking to be removed and forwards the email to a colleague: "this is why I don't take vendor calls."

The system passed relevance and failed appropriateness. The fix was not better data. It was a rule: personal facts from outside the relationship may inform whether and when to reach out, but are not quoted back unless they are professional, recent, and something the person would expect a vendor to know.

## Patterns

**Four-Test Gate.** *Problem:* teams judge on relevance alone. *Forces:* each test has a different owner (data, strategy, operations, governance). *Solution:* score each decision type on all four tests separately; ship only when all pass. *Tradeoff:* slower first launches, far fewer brand events.

**Value per Decision.** *Problem:* business cases rest on aggregate industry statistics. *Forces:* volume, value, and lift vary enormously by decision. *Solution:* price each decision type as a range with the load-bearing assumption named; fund the highest-value decisions first. *Tradeoff:* needs a baseline and a test before the number is trusted.

**Honestly General.** *Problem:* systems invent specificity when evidence is thin. *Forces:* specific copy demos better. *Solution:* choose the specificity tier (person, company, industry) from the evidence; never climb a tier by inventing. *Tradeoff:* some outputs are plainer; none are false.

## Leader questions

1. Which three repeated decisions about individual customers are worth the most, and are they personalized today?
2. For each personalized program, what is the measured lift over our generic treatment?
3. What would a customer be surprised to learn we know and use? Who decides whether we use it?
4. When our system is unsure about a customer, what does it do?
5. Are we optimizing for a proxy (opens, clicks) or for the outcome and the relationship?

## Build checklist

- [ ] Inventory repeated per-customer decisions before choosing tools.
- [ ] Define the generic baseline for each decision so lift can be measured.
- [ ] Score each decision type on the four tests, one owner per test.
- [ ] Price decisions as value-per-decision ranges, including cost of error.
- [ ] Tag every customer fact with its source and whether the customer would expect you to hold it.
- [ ] Implement evidence-chosen specificity tiers (person, company, industry).
- [ ] Block quoting of out-of-relationship personal facts by default.
- [ ] Block relationship claims (renewal, current product, contract) unless the relationship is recorded; reframe as an opportunity instead.
- [ ] Sample real outputs across the population, not showcase accounts.

## Metrics to watch

- **Incremental lift** per decision type, against a holdout on the generic treatment.
- **Negative-signal rate**: unsubscribes, complaints, "remove me" replies, and spam reports per thousand personalized messages.
- **Factual error rate** in personal claims, from sampled review.
- **Specificity mix**: share of outputs at person, company, and industry level, watched for sudden shifts.

## Reader Q&A

**Is a first name still worth using?** It can lift opens [4], and it is the shallowest personalization there is. Use it where name data is clean; do not mistake it for a program.

**We have little data on most prospects. Should we wait?** No. Start where you hold the most first-party evidence, usually existing customers, and stay honestly general elsewhere.

**How do we know if something is creepy before sending it?** Ask whether the customer would expect you to know the fact, given how they have dealt with you. If the honest answer is "only if they knew what our enrichment vendor does," do not quote it.

## For your AI

```yaml
chapter: 1
title: "What Personalization Is, and What It Is Worth"
concepts:
  - name: Personalization
    definition: "Choosing the next action for one person, using what is known about them, where the choice would differ for someone else."
  - name: Personal data vs. personalized decision
    definition: "Holding customer data is potential; personalization happens only when a decision about one person uses it."
  - name: Four tests
    definition: "A personalized decision must pass relevance, usefulness, timing, and appropriateness, evaluated separately."
  - name: Value per decision
    definition: "Outcome value x lift over the generic baseline, minus cost to decide, minus expected cost of being wrong; expressed as a range."
  - name: Specific and true, or honestly general
    definition: "Every personal claim is either supported by evidence or replaced by a true, less specific statement."
  - name: Known, not watched
    definition: "Customers accept use of facts they expect the company to hold; facts from unexpected flows reduce trust and response."
decision_rules:
  - if: "a personalized decision fails any one of the four tests"
    then: "do not ship it; fix the failing dimension or use the generic treatment"
  - if: "evidence about a person is thin, stale, or single-source"
    then: "drop to company-level or industry-level specificity; never invent to stay specific"
  - if: "a fact was obtained outside the customer relationship and is personal rather than professional"
    then: "may inform whether or when to act; do not quote it to the person"
  - if: "copy asserts a customer relationship (renewal, product in use, current contract) that is not recorded"
    then: "block the claim and reframe as an opportunity or evaluation; specificity degrades before trust does"
  - if: "an output was generated"
    then: "do not count it as done until content, persistence, delivery, and program-level checks also pass"
  - if: "a business case cites aggregate industry statistics"
    then: "replace with value-per-decision ranges and a plan to measure lift against a baseline"
  - if: "a program optimizes opens or clicks only"
    then: "add incremental outcome, negative-signal rate, and error rate before scaling"
assessment_questions:
  - "Which repeated per-customer decisions carry the most value, and which are personalized today?"
  - "What is the generic baseline for each, and has lift been measured against a holdout?"
  - "What customer facts do you use that a customer would not expect you to have?"
  - "What happens in your system when evidence about a customer is thin?"
  - "Which objectives (customer value, business value, trust, cost, risk) does each program track?"
patterns: [Four-Test Gate, Value per Decision, Honestly General]
anti_patterns: [The Creepy Reveal, Volume Over Value, Fake Specificity, Proxy Optimization]
maturity_dimension: understanding
```

## References

1. Arora, N., et al. "The value of getting personalization right, or wrong, is multiplying." McKinsey & Company, 2021-11-12. https://www.mckinsey.com/capabilities/growth-marketing-and-sales/our-insights/the-value-of-getting-personalization-right-or-wrong-is-multiplying
2. Gartner. "Gartner Predicts 80% of Marketers Will Abandon Personalization Efforts by 2025." Press release, 2019-12-02. https://www.gartner.com/en/newsroom/press-releases/2019-12-02-gartner-predicts-80--of-marketers-will-abandon-person
3. Gartner. "Gartner Survey Reveals Personalization Can Triple the Likelihood of Customer Regret at Key Journey Points." Press release, 2025-06-03 (survey of 1,464 B2B buyers and consumers, November to December 2024). https://www.gartner.com/en/newsroom/press-releases/2025-06-03-gartner-survey-reveals-personalization-can-triple-the-likelihood-of-customer-regret-at-key-journey-points
4. Sahni, N. S., Wheeler, S. C., and Chintagunta, P. "Personalization in Email Marketing: The Role of Noninformative Advertising Content." *Marketing Science* 37(2), 2018. https://pubsonline.informs.org/doi/10.1287/mksc.2017.1066
5. Salesforce. *State of the Connected Customer*, 5th edition, 2022. https://www.salesforce.com/content/dam/web/en_ie/www/PDF/state-of-connected-customer-fifth-ed-comp.pdf
6. Salesforce. *State of the Connected Customer*, 6th edition, 2023.
7. Epsilon. "New Epsilon research indicates 80% of consumers are more likely to make a purchase when brands offer personalized experiences." Press release, 2018-01-09. https://www.epsilon.com/us/about-us/pressroom/new-epsilon-research-indicates-80-of-consumers-are-more-likely-to-make-a-purchase-when-brands-offer-personalized-experiences
8. Matz, S. C., Kosinski, M., Nave, G., and Stillwell, D. J. "Psychological targeting as an effective approach to digital mass persuasion." *PNAS* 114(48), 2017. https://www.pnas.org/doi/10.1073/pnas.1710966114
9. Eckles, D., Gordon, B. R., and Johnson, G. A. "Field studies of psychologically targeted ads face threats to internal validity." *PNAS* 115(23), 2018. https://www.pnas.org/doi/10.1073/pnas.1805363115
10. Perla, et al.. "The (In)Effectiveness of Psychological Targeting: A Meta-Analytic Review." *Psychology & Marketing*, 2026. https://onlinelibrary.wiley.com/doi/10.1002/mar.70073
11. Twilio Segment. *State of Personalization Report 2024*. https://www.twilio.com/en-us/report/state-of-personalization-report
12. Aguirre, E., Mahr, D., Grewal, D., de Ruyter, K., and Wetzels, M. "Unraveling the Personalization Paradox: The Effect of Information Collection and Trust-Building Strategies on Online Advertisement Effectiveness." *Journal of Retailing* 91(1), 2015. https://www.sciencedirect.com/science/article/abs/pii/S0022435914000669
13. Kim, T., Barasz, K., and John, L. K. "Why Am I Seeing This Ad? The Effect of Ad Transparency on Ad Effectiveness." *Journal of Consumer Research* 45(5), 2019. https://academic.oup.com/jcr/article/45/5/906/4985191
14. Duhigg, C. "How Companies Learn Your Secrets." *The New York Times Magazine*, 2012-02-16. https://www.nytimes.com/2012/02/19/magazine/shopping-habits.html

---

# 2. The First Era

*The question: How did we personalize before language models, and what did it teach us?*

> **Questions this chapter answers**
> - How did companies personalize before large language models?
> - Which of those methods actually worked, and where did each one stop working?
> - Are rules, segments, recommenders, and A/B tests obsolete now?
> - Why did a model that predicted our customers well still lead to bad decisions?
> - How do I choose between a rule, a model, and an LLM for a given decision?

## The short answer

Before language models, companies personalized with four tools: merge fields, machine learning, experiments, and people. Each one worked, measurably, and each one hit a ceiling that the others could not lift.

Merge fields were cheap and wide, and said almost nothing. Machine learning (segments, recommenders, propensity scores) predicted what tens of millions of people would do, but could only choose among options someone had already written. Experiments made personalization honest by measuring it against a control, but a classic A/B test picks one winner for everyone. People understood customers deeply and wrote for them individually, and there were never enough hours to do that for more than a few.

So the first era split the work. Machines knew a little about everyone. Humans knew a lot about a few. Nobody could know a lot about everyone and act on it in words.

Three lessons survive intact, and a leader should insist on them before funding anything new. **Simple first**: a rule or a well-tuned model beats an LLM when the decision is narrow and the data is structured. **Prediction is not decision**: the customer most likely to buy is often the one who needs no push. **Nothing counts without a control**: if you cannot say what would have happened without the personalization, you do not know that it worked.

Language models did not make the old tools stop working. They supplied the missing piece: turning a model of one person into words for that person, without a human writing every message.

---

On September 21, 2009, Netflix handed a check for one million dollars to a team of seven researchers calling themselves BellKor's Pragmatic Chaos. [1] Three years earlier the company had released 100,480,507 movie ratings from 480,189 customers and offered the prize to anyone who could beat its own recommendation algorithm, Cinematch, by ten percent on prediction error. The winners cleared the bar with a 10.06 percent improvement. A second team matched their score and lost because it submitted twenty minutes later.

The winning solution was never put into production.

Netflix had already adopted two simpler methods from an earlier stage of the contest. The final blend of more than a hundred models was different. Xavier Amatriain, who led the algorithms team, later wrote that it "would have taken a large engineering effort for a small gain in accuracy that was most likely not worth it." [2] By then Netflix had moved from mailing DVDs to streaming, and predicting what people would actually watch mattered more than predicting the stars they would give.

That story holds most of what the first era learned. The model got better. The objective moved. The accuracy that won the contest was not the accuracy the business needed. The most famous personalization competition in history was, in the end, a lesson in choosing the right problem.

## The first era had four tools, and each hit a different ceiling

Everything companies did to personalize before large language models falls into four families, taken here from shallowest to deepest. The running example is Larkspur Systems, a fictional mid-market company selling field-service and fleet software, with about 40,000 customer accounts, 250,000 contacts, and ten years of CRM history. Larkspur runs every tool in this chapter. It still says roughly the same thing to everyone.

![Four panels ordered from shallowest to deepest show the first era's personalization tools. Merge fields are cheap and wide and say who, but never why. Machine learning (segments, recommenders, propensity scores) predicts tens of millions of people, but can only choose among options, not write the message. Experiments measure lift against a control, but a classic test picks one winner for everyone; people understand and write for each customer, but are capped by hours. The result: machines knew a little about everyone, and humans knew a lot about a few.](/images/handbook/ch02-four-tools-four-ceilings.svg)

*Figure 2.1. Each first-era tool worked, and each hit a ceiling the others could not lift.*

## Merge fields: the cheapest personalization, and the easiest to get wrong

Mail merge arrived with the first personal-computer word processors around 1980; WordStar's companion program, MailMerge, was among the earliest. [3] The idea has not changed since. A template holds slots, a list holds values, and the software fills one from the other: `Dear {first_name}`, `your team at {company}`.

It is easy to dismiss, and the evidence says not to. In randomized field experiments, Navdeep Sahni, Christian Wheeler, and Pradeep Chintagunta added the recipient's name to email subject lines. Open rates rose from 9.05 percent to 10.80 percent, leads rose by 31 percent, and unsubscribes fell by 17 percent. [4] The name carried no new information. It still changed behavior, because it signaled that the message had been addressed rather than broadcast.

The ceiling is just as clear. A merge field says *who*, never *why*. It cannot tell a customer in the middle of a renewal from one who signed last week. And when the field is wrong it does worse than generic: the uncanny near-miss, the email that uses your name and gets your company or role wrong, leaving the sender worse off than if it had stayed general. Larkspur's quarterly newsletter opens with `Hi {first_name}` for 250,000 contacts, and some of those fields hold a company name, an initial, or the word "Accounts."

## Machine learning: the era that learned to predict people

The second family is where most of the first era's real value was created. It is a stack of techniques, each built to fix the weakness of the one before, and every layer still runs in production somewhere. For each, a practitioner needs three facts: what it needs, what it is good at, and where it breaks.

| Family | What it needs | Good at | Where it breaks |
|---|---|---|---|
| Segmentation and clustering | A few descriptive attributes | Planning, creative briefs, explaining the base | Treats everyone as the segment's average |
| Propensity and churn models | Labeled outcomes (who bought, who left) | Ranking a whole base cheaply | Predicts the outcome, not the effect of acting |
| Content-based filtering | Item attributes and a person's history | New items; explainable ("because you read...") | More of the same; only as good as the tagging |
| Collaborative filtering, matrix factorization | A large matrix of interactions | Tastes nobody labeled | Cold start; optimizes the target it was given |
| Item-to-item | Purchase or view histories | Real time at catalog scale | Similar is not the same as useful |
| Two-stage deep recommenders | Very large logs and serious infrastructure | Millions of items, many signals | Cost, opacity, offline wins that fail live |
| Learning to rank | Graded labels or clicks | Ordering a whole list, not scoring items alone | Clicks carry position bias |
| Contextual bandits | Live traffic and a reward | Learning while serving | Optimizes whatever reward you give it |

**Segmentation.** Group customers who look alike, then treat each group as one: first with hand-built rules (recency, frequency, and monetary value in direct marketing), later with clustering. Segments are cheap and easy to explain. Their weakness is built in: everyone is treated as the segment's average, and the average customer does not exist.

**Propensity, churn, and next-best-offer.** These predict an action. Who is likely to buy, upgrade, or leave in the next ninety days? Score every customer, rank, and send the top of the list to a campaign or a salesperson; next-best-offer systems do the same per offer. This is still the backbone of enterprise personalization, and it works. Its blind spot is this chapter's failure story: a score says who will act, not who will act *because* you did something.

**Recommender systems.** Here the first era reached individual scale. Content-based systems recommend items whose attributes resemble what the person already liked. Collaborative filtering ignores attributes and uses behavior: you work out how similar users are to one another, and how similar items are to one another, and you use the two together to match people to things. I have built features of this kind; it is one of the most ordinary techniques in software, and one of the most effective.

The Netflix Prize made the next step famous. Yehuda Koren, Robert Bell, and Chris Volinsky, three members of the winning team, summarized it in 2009: matrix factorization, which represents each person and each item as a short vector of learned "latent factors," beat the classic nearest-neighbor methods and could absorb implicit feedback, changes over time, and confidence levels. [17] Nobody names those factors in advance; the model finds the dimensions of taste that best explain the ratings.

The landmark industrial paper is Amazon's. In 2003, Greg Linden, Brent Smith, and Jeremy York described item-to-item collaborative filtering, built because existing methods could not scale to a store with "more than 29 million customers and several million catalog items." [5] Their key move was architectural, and it is worth a builder's attention: compute the expensive similar-items table offline, so that the online step depends only on how many items this customer has bought, not on the size of the catalog or the customer base. The result ran in real time and worked from as few as two or three purchases. The same paper observed that clustering customers into segments, the cheaper alternative, produced recommendations of "relatively poor" quality.

At Netflix, Carlos Gomez-Uribe and Neil Hunt reported in 2015 that about 80 percent of hours streamed came from recommendations, and estimated their retention value at more than one billion dollars a year, a company self-estimate rather than audited revenue. [6] A more careful 2025 Netflix study swapped the production system for simpler alternatives: engagement fell about 4 percent against basic matrix factorization and about 12 percent against popularity rankings, with the largest gains on titles that were neither blockbusters nor obscure. [7]

**Two stages: candidates, then ranking.** Deep learning did not replace this architecture; it formalized it. Paul Covington, Jay Adams, and Emre Sargin described YouTube's 2016 recommender as two networks. [18] The first, candidate generation, retrieves "a small subset (hundreds) of videos" from a corpus of millions using the person's history. The second ranks those hundreds with richer features, predicting expected watch time rather than click probability. They added a sentence every model team should pin to the wall: "live A/B results are not always correlated with offline experiments." A later YouTube system made retrieval a *two-tower* model: one network embeds the person and context, another embeds each item from its content features, and retrieval becomes a nearest-neighbor search over tens of millions of videos. [19] Ranking borrowed from search, where *learning to rank* trains on the order of a whole list rather than each item alone; an ensemble of LambdaMART rankers won Track 1 of the 2010 Yahoo! Learning to Rank Challenge. [20]

**Contextual bandits.** A bandit learns while it serves: it shifts traffic toward what is winning while still exploring, and a contextual bandit chooses per person from their features. On the Yahoo! front page, Lihong Li and colleagues reported a 12.5 percent click lift from a contextual bandit over a context-free one. [11] Chapter 14 treats it as a decision-maker.

**Features, served fresh.** Every family above is only as current as its inputs. When Uber described its machine learning platform in 2017, the centerpiece was a shared feature store, about 10,000 features that teams could reuse, built so that "the same data is used for training and serving." [21] That sentence names a common silent bug: a feature computed one way for training and another at serving time, so the model scores a customer it never learned. Chapter 17 covers when a feature must be fresh.

**Cold start.** Collaborative methods share one structural weakness. A model built from a population is a summary of what a population did, and with no history there is nothing to summarize. These systems have always struggled with the new arrival, the account that signed yesterday: the cold-start problem. There is nobody yet to be similar to. The standard patches are content features (the item tower in a two-tower model exists partly for this), popularity, stated preferences at signup, and a bandit's deliberate exploration.

The ceiling of all of these is the same. Machine learning could *choose* among things someone had already made (a segment's message, a catalog item, one of five offers), but it could not *write* the thing for the person. The model of the customer could be individual. The message could not.

What carries forward is the architecture, not just the models. The shape that scales is still candidate generation, then ranking, now followed by a language layer that writes for the one person the ranker chose. And the scores this era learned to produce (propensity, churn risk, similarity, predicted uplift) do not disappear. They become inputs to the decision layer in Chapter 14, which weighs them against constraints, cost, and the option of doing nothing.

## Experiments: the discipline that made personalization measurable

The third family is not a way to personalize. It is the way to know whether personalization worked, and it is the first era's most durable contribution.

In December 2007, Dan Siroker ran an experiment on the Obama campaign's splash page, testing combinations of images and button text. The winning variant raised sign-ups from 8.26 percent to 11.6 percent, a 40.6 percent relative lift. [8] (The widely repeated "$60 million" attached to that test is an extrapolation of donations from the extra sign-ups, not a measured figure.) At Bing in 2012, a change to how ad headlines were displayed sat in the backlog for more than six months before an engineer finally tested it. It raised revenue by 12 percent, worth more than $100 million a year in the US alone. [9]

The deeper lesson from experimentation is humility. Ronny Kohavi, who ran Microsoft's experimentation team, and his colleagues put the success rate of ideas at Bing at about 10 to 20 percent, and note that most progress comes from small improvements of 0.1 to 1 percent, "after a lot of work." [10] The confident plan in the meeting is usually wrong. The only defense is a control group.

**A/B and multivariate tests.** An A/B test randomizes people between a control and one change. A multivariate test, like Siroker's, crosses several factors at once. Kohavi, Randal Henne, and Dan Sommerfield's practical guide recommends single-factor tests for incremental changes and factorial designs only when factors are suspected to interact strongly, because every added combination splits the audience and cuts power; in their experience, interactions are less frequent than people assume. [22]

**Power, in plain terms.** Power is the chance that a test detects an effect that is really there. The required sample grows with the square of the precision you ask for: halve the effect you want to detect, and you need four times the people. The same guide works an example: on a site where 5 percent of visitors buy, detecting a 5 percent relative change in conversion takes under 500,000 users, while a 20 percent change takes about 30,400. [22] Personalization effects usually sit at the small end, so many tests are underpowered; Chapter 21 does this arithmetic for Larkspur.

**Peeking and sequential tests.** A classic test fixes its sample in advance; checking daily and stopping at the first significant result inflates false positives. Ramesh Johari and colleagues described "always valid" p-values that stay correct under continuous monitoring, deployed in a commercial testing platform since January 2015. [23] The rule for a team is simple: either fix the horizon and do not stop early, or use a sequential method built for looking.

**Variance reduction.** The cheapest power is the noise you remove. CUPED, from Alex Deng, Ya Xu, Ron Kohavi, and Toby Walker, adjusts each person's outcome using their own behavior before the experiment began. On Bing it reduced variance by about 50 percent, "effectively achieving the same statistical power with only half of the users, or half the duration," and the best covariate was usually the same metric from the pre-period. [24] Personalization programs have exactly this data: the customer's history is the covariate.

**Interleaving for rankers.** When the thing under test is a ranking, you can show one person both versions at once. Interleaving merges two rankers' lists and credits whichever one supplied what the person chose; Olivier Chapelle, Thorsten Joachims, Filip Radlinski, and Yisong Yue validated it at scale on two commercial search engines and a scientific-literature search system. [25] Netflix uses a variant of team-draft interleaving: the two rankers take turns picking titles like captains choosing sides, and viewing hours are credited to whichever supplied each title. Its team reported in 2017 that interleaving "requires >100× fewer users than our most sensitive A/B metric to achieve 95% power," and that its verdicts correlated strongly with that metric. [26] The catch is that interleaving measures relative preference, not retention, so Netflix uses it as a fast pruning stage and runs a conventional A/B test on the survivors.

**Bandits and uplift.** Bandits trade proof for speed; Chapter 21 explains when that trade is right and why a bandit cannot tell you whether a program was worth running. The second idea that pushed experiments toward the individual was **uplift modeling**, introduced by Nicholas Radcliffe and Patrick Surry in 1999 and named "true lift" by Victor Lo in 2002. [12] Uplift models do not predict who will buy. They predict who will buy *because* you contacted them, which is a different question and, for spending decisions, the right one. Holdouts, incrementality, and misleading proxies belong to Chapter 21.

The ceiling of classic experimentation is that it finds one winner for everyone: the best page on average, not the best page for this visitor. Bandits and uplift models narrowed that gap, but still chose among variants a person had written.

That ceiling is also why experiments are the first era's best gift to the next one. When a language model writes a different message for every person, there is no variant B to test against variant A, and the power arithmetic rules out testing message variants within any one context. So the unit of the experiment moves up a level. You test what stays fixed while the words change: the policy that decides who is contacted and when, the template and the zones a model may fill, the decision rules, the model and prompt version as a treatment arm, all against a holdout. You cannot test every message. You can test the system that writes them.

## Humans by hand: deep, and capped by hours

The fourth family is the oldest and still the deepest. An account manager who has handled a customer for years knows the politics inside the account, the failed project from two years ago, and that the new operations lead prefers a phone call. A good researcher reads a company's filings, job posts, and the champion's public writing, and comes back with the one fact that changes the conversation.

In 1993, Don Peppers and Martha Rogers argued in *The One to One Future: Building Relationships One Customer at a Time* that companies should compete for share of customer, not share of market. [13] The subtitle is close to this handbook's title, and the lineage is deliberate. They described the goal. What the first era lacked was a way to reach it without a person in every relationship.

That is the ceiling: hours. Salesforce's 2022 survey of 7,775 sales professionals found reps spending 28 percent of their time actually selling. [14] Larkspur has twelve account managers. They cover the top 400 accounts deeply. The other 39,600 accounts get segments, merge fields, and a propensity score.

## What the first era got right, and what the new era keeps

Here is the sentence this chapter turns on. **Every first-era method was good at one of three things (knowing the customer, deciding what to do, or saying it well), and none could do all three for the same person at scale.** Models knew, a little, about everyone. Experiments decided, on average. People said it well, for a few.

![A grid with three jobs as columns (knowing the customer, deciding what to do, saying it well in words) and three methods as rows. Models fill only the knowing cell, a little about everyone; experiments fill only the deciding cell, on average; people fill only the saying cell, for a few. A final row for one person at scale shows knowing built at scale, deciding only on average, and saying never built: nothing turned one customer's model into a message for that customer.](/images/handbook/ch02-knowing-deciding-saying.svg)

*Figure 2.2. Knowing, deciding, and saying: each first-era method covered one, and none covered all three for the same person at scale.*

Knowing a person and speaking to that person are separate capabilities. The first era built the first at scale and never the second: nothing could turn one customer's model into a message for that customer, and read the reply, without a person at each end. That is the subject of Chapter 5.

It does not follow that the old tools are obsolete. Three of them are more important now, not less.

**Rules still govern.** Contact limits, eligibility, consent, and suppression stay deterministic. A language model should never decide whether a customer who unsubscribed gets an email.

**Models still rank.** A propensity score for 250,000 contacts costs almost nothing. A language model reasoning about each contact is justified only where the reasoning changes the decision.

**Experiments still judge.** Every claim about lift should end in a controlled comparison (Chapter 21). They prove what moved, not why; Chapter 3 pairs them with qualitative research and models in one loop.

The split survived contact with production when I built a governed personalization engine for B2B campaigns. The rule it settled on is one line: use AI where judgment adds value, use code where certainty is available. A headline limit, a ban on non-HTTPS links, and the rule that copy may not say "your renewal is coming up" unless the relationship is confirmed are all enforced in code. Choosing the most relevant approved proof point is left to the model. Scoring followed the same line: a language model combines weak and strong signals into a score with a rationale, inside the campaign's criteria, but it does not replace the deterministic qualification rules, and code checks that its outputs agree with each other (a score of 88 labeled "low" is flagged, not shipped). Population questions, such as every contact scored above 80 or every record missing an artifact, are answered by a deterministic filter with no model call at all. And when someone thinks a message would perform better, that idea goes to a controlled test, not into another paragraph of prompt.

This is the **Simple First** principle: use the least expensive technique that makes the decision well, and add depth only where depth changes the outcome.

![A three-step ladder rising left to right along an axis of cost, speed, and difficulty to explain. Tier 1 is rules, which govern eligibility, consent, and contact limits; tier 2 is classic models, which rank with propensity scores and recommenders; tier 3 is language models, which read unstructured evidence and write for one person. Arrows show moving up a tier only when the cheaper tier demonstrably fails, and a band beneath all tiers shows experiments judging every program against a control group or holdout.](/images/handbook/ch02-simple-first-ladder.svg)

*Figure 2.3. Simple First: rules, then classic models, then language models, each tier earned by the failure of the one below.*

### Choosing a technique

Three questions sort most decisions: how much structured data you have, how costly a wrong call is, and how much someone needs to understand why the decision was made.

| Data available | Stakes of a wrong call | Need for explanation | Start with |
|---|---|---|---|
| Little or none | Low | Low | Rules, or popularity plus a bandit |
| Rich behavioral history | Low | Low | Recommender or propensity model |
| Rich history | High (money, contract, compliance) | High | Rules for eligibility, uplift model for targeting, human approval |
| Mostly unstructured (calls, emails, documents) | Medium | Medium | Language model reading the evidence, with a deterministic check after |
| Unstructured and deep, one account at a time | High | High | Language model research plus a human decision |

In words: structured data and low stakes favor classic models; high stakes demand rules and human judgment around any model; unstructured evidence is where language models earn their cost, because nothing in the first era could read it.

## What this does not do

The first era's record is distorted by numbers repeated past the evidence, some now used to sell the second era.

The most famous is that "35 percent of what customers buy on Amazon comes from recommendations." It is usually traced to a 2013 McKinsey article; Amazon has never published it. Linden, Smith, and York themselves wrote only that click-through and conversion rates "vastly exceed those of untargeted content." [5] Netflix's billion dollars is a retention estimate. Obama's $60 million is an extrapolation. None of these is invented, but none is the measured fact it is usually presented as.

A quieter overreach is the belief that a better model means a better business result. The Netflix Prize says otherwise, and so does Booking.com, which reported after deploying some 150 models that a better offline model did not necessarily bring more business value. [15] A recommender that looks better in offline testing routinely loses to a worse-looking one on real people. The same mistake is being made now with language models, scored on benchmarks and deployed on customers.

And the first era did not fail. In the companies I work with, recommenders and propensity models remain the working core of personalization (an observation, not a survey). What the era could not do was reach depth for most customers. That is a ceiling, not a failure.

## At scale

At Larkspur's size, each team owns one first-era tool: marketing owns segments and merge fields, product owns recommendations, sales owns the propensity score, customer success owns the account notes. Each holds a partial model of the same customer, and none reads the others. The churn model does not know about the open support escalation. The account manager's knowledge leaves when they do.

At millions of customers the pattern is the same, only more expensive. The first era's scale problem was never compute. It was that understanding stayed fragmented across tools that could not share it.

## Failure story: The Discount Mistake

Larkspur's data science team builds a good upgrade-propensity model. Marketing takes the top decile, about 4,000 accounts, and sends them a 20 percent discount on the next tier. Upgrades in that group are strong. The campaign is declared a success.

A year later someone asks what those accounts would have done without the discount. A small holdout, kept by accident, answers: they upgraded at almost the same rate. The model had correctly found the customers most likely to upgrade, and the campaign paid them to do what they were already going to do. (Illustrative numbers; the pattern is well documented.)

Uplift research calls these customers "Sure Things," alongside "Persuadables," "Lost Causes," and "Do-Not-Disturbs," the last being customers whom contact makes worse. [12] Studies of retention campaigns have found that targeting the highest-risk customers can be ineffective, and that the better target is sensitivity to the intervention, not risk. [16]

The error is reading high propensity as need for an incentive. Prediction is not decision.

## Patterns

**Simple First.** *Problem:* teams reach for the most capable technique by default. *Forces:* capable techniques are costlier, slower, harder to explain. *Solution:* start with rules, then classic models, then language models, moving up only when the cheaper tier demonstrably fails. *Tradeoff:* some early lift left on the table, in exchange for cost control and a baseline to beat.

**Keep a Holdout.** *Problem:* results are reported without a counterfactual. *Forces:* holdouts cost short-term revenue and feel wasteful. *Solution:* a random control group for every personalized program, plus a small permanent global holdout. *Tradeoff:* a small real cost, against knowing whether anything worked.

**Target Uplift, Not Propensity.** *Problem:* incentives go to customers who would act anyway. *Forces:* propensity models are easier to build and look accurate. *Solution:* for any action that costs money or attention, estimate the effect of the action, not the likelihood of the outcome. *Tradeoff:* uplift needs experimental data and is noisier.

## Leader questions

1. For our three largest personalization programs, what is the control group, and what did it do?
2. Which of our decisions are made by rules, which by models, and which by people? Is that split deliberate?
3. Where do we give incentives to customers most likely to buy, and have we measured whether they needed them?
4. Before we fund an LLM for this decision, what simpler method did we try, and how did it do?

## Build checklist

- [ ] Inventory every personalization in production and label its technique: rule, segment, recommender, propensity, experiment, human.
- [ ] Audit merge fields for bad values (company names in first-name fields, placeholders, stale titles) and define a fallback.
- [ ] Attach a control group to every personalized program; keep a small permanent global holdout.
- [ ] For every incentive-based campaign, compare treated and untreated outcomes before scaling.
- [ ] Define a cold-start path for new customers that does not depend on history.
- [ ] Keep eligibility, consent, and contact limits in deterministic code, separate from any model.
- [ ] Compute every model feature through one path for training and serving, and use pre-period behavior to reduce variance in tests.

## Metrics to watch

- **Incremental lift versus holdout**, per program, not raw conversion.
- **Share of decisions with a counterfactual**: the percentage of personalized actions covered by a control.
- **Merge-field error rate**: records whose personalization fields are missing or malformed.
- **Incentive efficiency**: incremental conversions per discount dollar, not conversions per discount.

## Reader Q&A

**Are A/B tests obsolete now that AI can personalize each message?**
No. When every message is different you cannot test messages against each other, but you can and must test the system against a holdout. The unit of the experiment moves from the variant to the policy.

**We have a propensity model that works. Do we need an LLM?**
Maybe not for ranking. A language model adds value where the decision depends on evidence your model cannot read (call notes, emails, documents) or where a message must be written for one person. Keep the propensity model; use it to decide where deeper work is worth paying for.

**Why do our recommendations work for long-time customers and fail for new ones?**
That is cold start: with no history, there is nobody to be similar to. Rules, popularity, stated preferences, and now a language model reading what little a new customer has said can cover the gap.

## For your AI

```yaml
chapter: 2
concepts:
  - name: First-era personalization
    definition: "Personalization before LLMs, using merge fields, machine learning (segments, recommenders, propensity models), experiments, and human effort."
  - name: Simple First
    definition: "Use the least expensive technique that makes the decision well; add depth only where depth changes the outcome."
  - name: Knowing, deciding, saying
    definition: "The three functions of personalization; each first-era method did one well, none did all three for one person at scale."
  - name: Cold start
    definition: "Failure of history-based models for new customers with no behavior to compare."
  - name: Uplift
    definition: "The change in outcome caused by an action for a person, as opposed to the probability of the outcome."
  - name: Candidate generation and ranking
    definition: "The two-stage recommender architecture: a cheap stage retrieves hundreds of plausible items from millions, a richer stage orders them; in the LLM era a language layer follows and ML scores feed the decision layer."
  - name: Training-serving consistency
    definition: "The requirement, served by a feature store, that a feature is computed the same way when a model is trained and when it is used."
  - name: Variance reduction (CUPED)
    definition: "Adjusting each person's experiment outcome by their own pre-experiment behavior to reach the same power with fewer people or less time."
  - name: Interleaving
    definition: "Comparing two rankers by merging their lists for the same person and crediting the ranker whose items were chosen; far more sensitive than A/B for rankers, but measures relative preference only."
decision_rules:
  - if: "the decision is narrow, the data is structured, and the stakes are low"
    then: "use rules or a classic model before a language model"
  - if: "an action costs money or attention (discount, call, retention offer)"
    then: "target on estimated uplift, not propensity, and keep a holdout"
  - if: "a personalized program has no control group"
    then: "treat its reported lift as unproven"
  - if: "a merge field value is missing or malformed"
    then: "fall back to an honestly general greeting"
  - if: "the decision depends on unstructured evidence (calls, emails, documents)"
    then: "consider a language model to read it, with deterministic checks after"
  - if: "two ranking algorithms must be compared"
    then: "prune with interleaving, then confirm business outcomes with an A/B test on the survivors"
  - if: "results will be checked while a test is running"
    then: "use a sequential method with always-valid inference, or fix the horizon and do not stop early"
  - if: "each person receives a differently generated message"
    then: "test the policy, template, zones, decision rules, or model version against a holdout, not individual messages"
  - if: "a requirement can be checked exactly (length, link scheme, confirmed relationship, score consistency)"
    then: "enforce it in code, not in the prompt; reserve the model for judgment within approved evidence"
assessment_questions:
  - "Which personalization programs run today, and which technique drives each?"
  - "Which programs have a control group or global holdout?"
  - "Where are incentives given, and has incremental effect been measured?"
  - "How are new customers with no history personalized?"
  - "What customer knowledge exists only in account managers' notes or heads?"
patterns: [Simple First, Keep a Holdout, Target Uplift Not Propensity]
anti_patterns: [The Discount Mistake, The Uncanny Near-Miss, Offline Accuracy Worship, Peeking, Training-Serving Skew]
maturity_dimension: decisioning
```

## References

1. Netflix Prize: launch October 2, 2006; 100,480,507 ratings from 480,189 users on 17,770 movies; Grand Prize awarded September 21, 2009, to BellKor's Pragmatic Chaos, 10.06% RMSE improvement. https://en.wikipedia.org/wiki/Netflix_Prize ; award announcement: https://www.netflixprize.com/community/topic_1537.html
2. Amatriain, X. "On the 'Usefulness' of the Netflix Prize." https://amatria.in/blog/on-the-usefulness-of-the-netflix-prize-403d360aaf2/ ; original: Amatriain & Basilico, "Netflix Recommendations: Beyond the 5 stars (Part 1)," Netflix Tech Blog, April 2012. https://netflixtechblog.com/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429
3. "Mail merge" and "WordStar," Wikipedia (MicroPro's MailMerge, c. 1980). https://en.wikipedia.org/wiki/Mail_merge
4. Sahni, N. S., Wheeler, S. C., & Chintagunta, P. (2018). "Personalization in Email Marketing: The Role of Noninformative Advertising Content." *Marketing Science* 37(2). https://pubsonline.informs.org/doi/abs/10.1287/mksc.2017.1066
5. Linden, G., Smith, B., & York, J. (2003). "Amazon.com Recommendations: Item-to-Item Collaborative Filtering." *IEEE Internet Computing* 7(1), 76 to 80. https://dl.acm.org/doi/10.1109/MIC.2003.1167344
6. Gomez-Uribe, C. A., & Hunt, N. (2015). "The Netflix Recommender System: Algorithms, Business Value, and Innovation." *ACM TMIS* 6(4). https://dl.acm.org/doi/10.1145/2843948
7. Zielnicki et al. (2025). "The Value of Personalized Recommendations: Evidence from Netflix." arXiv:2511.07280. https://arxiv.org/abs/2511.07280
8. Siroker, D. (2010). "How Obama raised $60 million by running a simple experiment." Optimizely. https://www.optimizely.com/insights/blog/how-obama-raised-60-million-by-running-a-simple-experiment/
9. Kohavi, R., & Thomke, S. (2017). "The Surprising Power of Online Experiments." *Harvard Business Review*. https://hbr.org/2017/09/the-surprising-power-of-online-experiments
10. Kohavi, R., Tang, D., & Xu, Y. (2020). *Trustworthy Online Controlled Experiments.* Cambridge University Press. https://experimentguide.com/ ; Kohavi et al. (2014), "Seven Rules of Thumb for Web Site Experimenters." https://exp-platform.com/Documents/2014-08-27ExperimentersRulesOfthumbKDD.pdf
11. Li, L., Chu, W., Langford, J., & Schapire, R. E. (2010). "A Contextual-Bandit Approach to Personalized News Article Recommendation." WWW 2010. https://arxiv.org/abs/1003.0146
12. Radcliffe, N. J., & Surry, P. D. (1999). "Differential Response Analysis: Modeling True Response by Isolating the Effect of a Single Action." *Credit Scoring and Credit Control VI*; Lo, V. S. Y. (2002). "The True Lift Model." *SIGKDD Explorations* 4(2); Radcliffe & Surry (2011), "Real-World Uplift Modelling with Significance-Based Uplift Trees." https://www.research.ed.ac.uk/en/publications/real-world-uplift-modelling-with-significance-based-uplift-trees/
13. Peppers, D., & Rogers, M. (1993). *The One to One Future: Building Relationships One Customer at a Time.* Currency/Doubleday.
14. Salesforce (2022). *State of Sales*, 5th ed. (n = 7,775). https://www.salesforce.com/news/stories/sales-research-2023/
15. Bernardi, L., Mavridis, T., & Estevez, P. (2019). "150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com." KDD 2019. https://dl.acm.org/doi/10.1145/3292500.3330744
16. Ascarza, E. (2018). "Retention Futility: Targeting High-Risk Customers Might Be Ineffective." *Journal of Marketing Research* 55(1). https://journals.sagepub.com/doi/10.1509/jmr.16.0163 ; Radcliffe, N. J., & Simpson, R. (2008). "Identifying who can be saved and who will be driven away by retention activity." https://stochasticsolutions.com/pdf/SavedAndDrivenAway.pdf
17. Koren, Y., Bell, R., & Volinsky, C. (2009). "Matrix Factorization Techniques for Recommender Systems." *IEEE Computer* 42(8), 30 to 37. https://dl.acm.org/doi/10.1109/mc.2009.263
18. Covington, P., Adams, J., & Sargin, E. (2016). "Deep Neural Networks for YouTube Recommendations." RecSys 2016. https://research.google/pubs/deep-neural-networks-for-youtube-recommendations/
19. Yi, X., et al. (2019). "Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations." RecSys 2019. https://dl.acm.org/doi/10.1145/3298689.3346996
20. Burges, C. J. C. (2010). "From RankNet to LambdaRank to LambdaMART: An Overview." Microsoft Research Technical Report MSR-TR-2010-82. https://www.microsoft.com/en-us/research/publication/from-ranknet-to-lambdarank-to-lambdamart-an-overview/
21. Hermann, J., & Del Balso, M. (2017-09-05). "Meet Michelangelo: Uber's Machine Learning Platform." Uber Engineering Blog. https://www.uber.com/us/en/blog/michelangelo-machine-learning-platform/
22. Kohavi, R., Henne, R. M., & Sommerfield, D. (2007). "Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO." KDD 2007. https://ai.stanford.edu/~ronnyk/2007GuideControlledExperiments.pdf
23. Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2017). "Peeking at A/B Tests: Why It Matters, and What to Do about It." KDD 2017, 1517 to 1525. https://www.semanticscholar.org/paper/b8b8c4627bf9bea8118a098f7dcd1612603f4795
24. Deng, A., Xu, Y., Kohavi, R., & Walker, T. (2013). "Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data." WSDM 2013. https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf
25. Chapelle, O., Joachims, T., Radlinski, F., & Yue, Y. (2012). "Large-Scale Validation and Analysis of Interleaved Search Evaluation." *ACM TOIS* 30(1). https://dl.acm.org/doi/10.1145/2094072.2094078
26. Parks, J., Aurisset, J., & Ramm, M. (2017-11-29). "Innovating Faster on Personalization Algorithms at Netflix Using Interleaving." Netflix TechBlog. https://netflixtechblog.com/interleaving-in-online-experiments-at-netflix-a04ee392ec55

---

# 3. What Designers Knew

*The question: What did UX research and HCI learn about adapting interfaces to people, and what does it teach AI personalization?*

> **Questions this chapter answers**
> - Why did adaptive interfaces like Clippy and Office's personalized menus frustrate the people they were built to help?
> - Adaptive or adaptable: who should control the change, the system or the person?
> - What do personas, jobs-to-be-done, and field research add that behavioral data cannot?
> - How do qualitative research, machine learning, and A/B testing work together?
> - Can synthetic users replace testing with real people?

## The short answer

Long before language models, designers and HCI researchers ran the experiment AI personalization is running now: software that learns a person and changes itself to fit. They learned three things that still hold.

First, **where** a system adapts matters as much as **how well**. Interfaces that moved things around (menus that reordered themselves, toolbars that promoted and demoted buttons) made people slower and less sure of themselves, even when the predictions were decent. Interfaces that kept a stable frame and adapted inside a clearly marked area did better. Accuracy matters too, and a lab study found it mattered more than predictability; but inaccurate adaptation in the places people rely on is the combination that fails.

Second, **when to act is its own decision.** Microsoft's Office Assistant rested on a respectable probabilistic model of what users were trying to do. It failed on a cruder rule about when to interrupt them.

Third, **control has to be real.** People value seeing and steering what a system believes about them, and they stop trusting controls that do not visibly work.

UX research also brings a method the data team usually lacks. Qualitative research finds *why* people behave as they do; machine learning finds *who* behaves that way at scale; experiments prove *what* worked. A personalization program that skips any of the three is guessing somewhere.

For AI personalization, the translation is five rules: a stable frame, adaptive slots, predictable change, visible control, and explanation on request. Synthetic users can help draft research. They cannot validate it.

---

In 1999, a person opening the Format menu in Word 2000 saw a short menu. The commands they used most were there; the rest were hidden behind a chevron at the bottom, or appeared after a few seconds. As they worked, the menu learned: items they used were promoted to the short menu, items they ignored were demoted. The idea, as Jensen Harris of the Office user-experience team later put it, was "that eventually you'd have a fully-tuned, auto-customized UI that would show you only what you need and use." [1]

In 2006, explaining why Office 2007 would abandon menus altogether, Harris wrote the post-mortem. "Adaptive Menus were not successful." Scanning took two passes, because pressing the chevron reflowed the menu and reset the eye. And the prediction was never quite right: "Auto-customization, unless it does a perfect job, is usually worse than no customization at all." What people experienced was "a sense [of] randomness and unpredictability: one time, a menu item would be in a certain place, and then two days later it wasn't there anymore." His verdict on the feature and its toolbar twin: "most customers, especially those in corporate environments, turn both of these features off." [1]

The same suite had shipped an even more ambitious learner two years earlier. The Office Assistant, the paperclip most people remember as Clippy, drew on Microsoft Research's Lumière project, which used Bayesian models to infer a user's goals from their actions and queries. [2] Eric Horvitz, who led Lumière, recorded what happened on the way to the product: the Office team put "a relatively simple rule-based system on top of the Bayesian query analysis system to bring the agent to the foreground," and his group "had been concerned upon hearing this plan that this system would be distracting to users." [2] Microsoft made the Assistant off by default in Office XP in 2001 and removed it in Office 2007. [3]

Two lessons, and neither is "the AI was bad." The menus adapted the one thing people needed to stay put. The Assistant's inference was reasonable; its decision about *when to speak* was not.

## Designers modeled need before anyone modeled data

Personalization teams tend to start from the data they hold. Designers started from the person.

**Personas.** Alan Cooper popularized personas in *The Inmates Are Running the Asylum* (1999): design for a specific, archetypal user, not an elastic "user" who can be stretched to justify any feature. [4] The critique arrived quickly. Christopher Chapman and Russell Milham argued in 2006 that nobody can tell how many real users a persona represents, and that personas cannot be verified or falsified. [5] A persona is a communication device that keeps a team arguing about a person rather than an abstraction. It is not a segment, and it is not data.

**Jobs to be done.** Clayton Christensen and colleagues argued that people "hire" a product to make progress in a circumstance. In their best-known example, milkshake research stalled on demographics and moved once researchers watched who bought and when: nearly half were bought before 9 a.m. by lone commuters, hiring the shake to make a long, boring drive more tolerable. [6] Nielsen Norman Group's reading is the useful one: jobs and personas are complements, because two groups can share a job and still want different things from it. [7]

**Field research.** Contextual inquiry (watching and interviewing people where they actually work), journey maps (the steps, thoughts, and emotions of someone pursuing a goal), and diary studies (behavior logged over weeks) are how designers find the moments that matter. [8][9][10]

Here is why a personalization team should care. These methods do not produce predictions. They produce **decisions worth personalizing**, which is the input Chapter 6 asks every leader to find. A Larkspur Systems researcher (Larkspur is the fictional field-service software company used throughout this handbook) shadows three dispatchers for a morning and sees each of them print the day's route sheet at 6:40. The tablets lose signal at rural job sites, and paper is the backup. Telemetry records only a print event. The personalization decision that follows is not "which tip to show a dispatcher." It is "which jobs to preload for offline use, per technician, before the van leaves." Telemetry would have ranked that decision nowhere. One morning of fieldwork put it first.

The data knows what people did. Only watching them tells you what they were trying to do.

## Five users is a method, not a sample size

Usability testing gave the industry its most quoted rule of thumb, and its most misquoted one. Jakob Nielsen's 2000 argument that you only need to test with five users rests on a model in which each user reveals a share of the problems, about 31 percent on average in the projects he and Tom Landauer studied, so five users find roughly 85 percent. [11] Read the rest of the article and the rule narrows. Nielsen recommends three rounds of five, with a redesign between each, rather than one round of fifteen; three or four users per group when user groups differ; and about twenty users for quantitative measures. [11]

The empirical record narrows it further. Across 49 users on four production websites, Jared Spool and Will Schroeder found the first five revealed about 35 percent of the problems. [12] Laura Faulkner resampled a 60-user study and found that some sets of five found 99 percent of known problems while others found 55 percent. [13] The 85 percent is an average over the *obvious* problems for *one* population doing *one* set of tasks.

That caveat is the personalization problem in miniature. A personalized surface is a family of interfaces, one per combination of person, evidence, and fallback, and the version the design team tests is rarely the one a thin-data customer sees. So test small rounds per variant class (rich record, thin record, fallback fired, each role), and test what was rendered, not what was stored. In the governed personalization engine I built for go-to-market teams, quality review inspects the live page, because stored fields can be correct while the experience is wrong: a stale template, a hidden zone, the wrong page identity, a stale cache, a broken link. [14]

Five users find the obvious problems in one experience. Personalization ships thousands.

## Adaptive or adaptable: who controls the change?

Nielsen Norman Group draws the distinction that organizes the rest of this chapter: personalization is done by the system; customization is done by the user. [15] HCI research uses *adaptive* and *adaptable* for the same pair, and it has tested both for three decades.

The early result favored stability. In 1994, Andrew Sears and Ben Shneiderman proposed **split menus**: the few most frequently used items move to a top section, and the rest keep their order. In two field studies, selection times fell 17 to 58 percent. [16] The split was set from usage data, but it did not keep shifting under the user. A decade later, Leah Findlater and Joanna McGrenere compared static, adaptive, and adaptable split menus in the lab: the static menu was faster than the adaptive one, and the abstract reports that most users preferred the control of the adaptable version. [17] The Product playbook applies this to in-app surfaces; I will not repeat it here.

Then the story gets more interesting, and it is where I part company with Harris's rule that imperfect auto-customization is "usually worse than no customization at all." In 2008, Krzysztof Gajos and colleagues at the University of Washington and Microsoft Research ran 23 participants through a modified Word interface with an adaptive toolbar, crossing two accuracy levels (50 and 70 percent) with two strategies (a predictable most-recently-used rule, and random). Both predictability and accuracy raised satisfaction. Accuracy also raised performance and use of the adaptive toolbar, and, "contrary to our expectations," it had the stronger effect. [18] Notice the design of that toolbar: an adaptive area, the authors wrote, "clearly designated as hosting changing content," sitting beside the familiar toolbars, which did not move. The user could always take the familiar route.

The strongest evidence for adaptation comes from the same research line. Gajos's SUPPLE system generates interfaces automatically; its ability-based version, SUPPLE++, built interfaces from short motor tests of people with motor impairments. They were 26.4 percent faster and made 73 percent fewer errors than with the manufacturers' default interfaces, and strongly preferred the generated ones. [19] In an in-vehicle system, Tal Lavie and Joachim Meyer found adaptivity helped most in routine situations. [20]

Put the evidence together and the defensible claim is narrower than either camp's slogan. Adaptation helps when it is accurate, when it rests on something measured rather than guessed, and when it happens somewhere the person expects change. It hurts when it is inaccurate and rearranges what the person has already learned. Harris was describing the worst corner of that space, and he described it accurately.

That gives a spectrum, and the useful question at each point is who controls the change.

![A horizontal spectrum of five interface designs, ordered from the person controlling more to the system controlling more. 1, static: the designer decides once and nothing learns. 2, adaptable: the person changes it with settings, pins, and favorites. 3, suggested: the system proposes and the person accepts, as in a split menu or a recommended row. 4, adaptive in a slot: the system changes a marked area while the frame stays fixed and the person can override. 5, adaptive everywhere: the system changes the frame itself, as Office 2000's personalized menus did, which is where trust collapses and customers turned it off. Designs 3 and 4 are outlined as the working zone (stable frame, accurate change in marked slots), and a closing line says accuracy matters most, but only inside a place the person expects to change.](/images/handbook/ux-adaptive-adaptable-spectrum.svg)

*Figure 3.1. The adaptive-to-adaptable spectrum: the working zone is a stable frame, accurate change in marked slots, and a person who can override.*

## A system that learns should know when to stay quiet

In 1999, Horvitz published "Principles of Mixed-Initiative User Interfaces," twelve factors for systems that share initiative with a person. [21] Among them: consider uncertainty about the user's goals; time the service to the user's attention; weigh the expected costs and benefits of acting; minimize the cost of poor guesses; and scope the precision of the service to match uncertainty, doing less but doing it right. His LookOut prototype chose among three actions for each incoming email: do nothing, ask, or act, according to how confident it was about the user's intent. [21]

Clippy, as shipped, appeared on a rule rather than on the model's uncertainty, and interrupted rather than waited.

Twenty years later, Saleema Amershi and colleagues at Microsoft consolidated more than 150 design recommendations into 18 **Guidelines for Human-AI Interaction**, tested by 49 design practitioners against 20 AI products. [22] Eight of the 18 map to Horvitz's principles. For personalization, the essential ones are G10 (scope services when in doubt), G14 (update and adapt cautiously), G15 and G17 (granular feedback and global controls), G18 (notify users about changes), and G11 (make clear why the system did what it did). G11 drew one of the highest counts of violations. One participant, testing an e-commerce recommender: "I have no idea why this is being shown to me. Is it trying to sell me stuff I do not need?"

The decision layer in Chapter 14 includes "do nothing" as a first-class action for the same reason Horvitz's LookOut did. Personalization is two decisions, not one: what fits this person, and whether acting now is worth their attention. Clippy got the first roughly right and the second wrong, and the second is the one people remember.

## Control that does not work is worse than no control

Recommender research reached the same conclusion from the other side. Nava Tintarev and Judith Masthoff listed seven aims an explanation can serve, among them transparency, trust, and **scrutability**: letting a person tell the system it is wrong. [23]

What people want from control is specific. In focus groups on news recommenders, Jaron Harambam and colleagues found that participants valued an intelligible profile (what the system thinks they read and like) and the ability to steer the algorithm toward their own goals. Filtering results after the fact was "not sufficient to make users feel that they have control," and participants openly doubted whether controls did anything at all. [24]

That doubt was measured in 2022. Mozilla analyzed data from more than 20,000 volunteers and over 500 million YouTube videos and found that "Don't recommend channel" prevented 43 percent of unwanted recommendations, and "Not interested" only 11 percent. In a survey of 2,758 participants, 62.3 percent said the controls were ineffective or gave mixed results. [25] The button existed. It barely worked. People noticed.

Shipped products have since added controls that change the model rather than the list. Spotify lets listeners exclude playlists (sleep sounds, a child's music) from their taste profile, so context stops masquerading as preference. [26] TikTok lets people reset their For You feed. [27] Those are the right shape: they act on what the system believes, and the effect is visible.

A control is a promise. If it does not visibly change what happens next, it is a broken one.

## Qual finds the why, ML finds the who, experiments prove the what

Each discipline has a blind spot. The research team knows why a dispatcher prints the route sheet and cannot say how many dispatchers do. The data team can count the print events across 40,000 accounts and cannot say why they happen. The experimentation team can prove a change moved a metric and cannot say for whom or why. Chapter 2 covers the models and the experiment design in depth; the point here is how the three disciplines hand work to one another.

The experimentation canon says this itself. Ronny Kohavi and colleagues popularized "HiPPO" (the highest-paid person's opinion) in arguing for testing over conviction [28], and the same authors' textbook devotes a chapter to the complementary methods (user experience research, surveys, focus groups, log analysis) that generate hypotheses and explain why metrics moved. [29] Experiments tell you what moved. They do not tell you why.

So run the three as a loop, which I call the **Triangulation Loop**:

1. **Qualitative research finds the why.** Fieldwork and interviews surface a job and the moment it is at stake: dispatchers need tomorrow's jobs on the tablet before the van leaves the yard.
2. **Machine learning finds the who.** Telemetry and memory size it: which accounts work rural sites, which technicians lose signal, how often.
3. **Experiments prove the what.** Preloading jobs for those technicians is tested against a holdout (Chapter 21), and the measured effect decides whether it ships.
4. **Anomalies go back to the start.** When the experiment surprises you, or a segment behaves differently, the next round of interviews goes to those people.

![Three stations joined by arrows, with a return path. Qualitative research (fieldwork, interviews, diary studies) finds the why; in the Larkspur (fictional) example, dispatchers print route sheets at 6:40. Machine learning over telemetry and memory finds the who by sizing the need: which accounts work rural sites, and how often. Experiments prove the what: preloaded offline jobs are tested against a holdout, which decides whether it ships. Surprises and segments that behave differently return to qualitative research, and a closing line says that skipping a station leaves the program guessing the why, the who, or the what.](/images/handbook/ux-triangulation-loop.svg)

*Figure 3.2. The Triangulation Loop: each discipline answers the question the other two cannot.*

The loop also works inside a running AI system, and this is where designers' habits pay off most. In the engine I built, a campaign's first outputs are read the way a researcher reads interview transcripts: a person, helped by AI, reviews the first ten to twenty-five real outputs, then the first fifty to a hundred, then a few hundred, and asks questions across the population. What errors repeat? Where is the content too generic? Which fallbacks fire too often? What belongs in code rather than in another guideline? Every finding is classified by root cause before anyone proposes a fix, and a hunch that some content would perform better is routed to a controlled test rather than into the prompt. [14] That is qualitative research on the machine's own output: small samples, read closely, with the numbers held for the experiment.

## Deception is the boundary, not a setting

Designers also named what personalization must not become. Harry Brignull coined "dark patterns" in 2010 for interface tricks that make people do things they did not mean to; he now prefers "deceptive patterns." [30] In 2019, Arunesh Mathur and colleagues crawled about 53,000 product pages across roughly 11,000 shopping sites and found 1,818 instances of such patterns, 183 sites using outright deceptive practices, and 22 third-party vendors selling them as ready-made components. Their definition is the one to keep: design that benefits a service "by coercing, steering, or deceiving users." [31]

Personalization makes this more dangerous, not less, because a pattern tuned per person is harder for anyone else to see. The boundary for a builder is structural. Adapt what helps someone decide; never adapt the exit, the price disclosure, the consent, or the cancel path. Those stay fixed, identical, and outside every adaptive slot. Chapters 18 and 19 make this enforceable.

## Synthetic users draft; people validate

The newest temptation in UX research is to let language models play the participants.

In Nielsen Norman Group's comparisons, synthetic users produced shallow, unprioritized lists of needs, were sycophantic about concepts, and reported finishing every online course they started, where real participants admitted quitting after three of seven. [32] A 2025 review of three studies found that synthetic users capture the direction of effects but underestimate their size and show less variance than people, and that "digital twins" built from interviews with real individuals do better than personas built from demographics. [33] The strongest result points the same way. Joon Sung Park and colleagues built agents from two-hour interviews with 1,052 people; agents given the interviews and survey answers reproduced held-out survey answers at 86 percent of the consistency people showed with themselves two weeks later, against 74 percent for agents given demographics alone. [34] The interviews were the input. The research was not replaced; it was reused.

Synthetic users are a drafting tool (piloting an interview guide, pretesting wording, listing edge cases), not a validation tool. The Research and Strategy playbook applies the same rule to market research.

## What it means for AI personalization

Everything above converges on one design for AI personalization. Here is the sentence this chapter turns on: **people forgive a system that guesses wrong inside a place they know will change; they do not forgive one that moves the place.**

That yields five rules, which I group as **Stable Frame, Adaptive Slots**:

1. **Stable frame.** Navigation, core controls, legal copy, prices, and exits stay identical for everyone.
2. **Adaptive slots.** Change happens only in declared, bounded places, each with a purpose, allowed evidence, limits, and an approved fallback.
3. **Predictable change.** A slot changes on meaningful events, not on every visit, and follows rules a person could describe.
4. **Visible control.** People can see and correct what the system believes, and the correction visibly changes what happens next.
5. **Explanation on request.** "Why am I seeing this?" gets a true, short answer grounded in what the system actually used.

The engine I built follows the first two by construction. A page is a governed static shell with, in a typical first implementation, about 8 to 12 text zones; brand, navigation, and legal copy are never zones. Each zone carries a contract: its purpose, the context it may and may not use, a length limit, its validations, and an approved fallback. [14] Generative does not mean uncontrolled. Chapter 15 covers zone contracts in the build lane, and the chapter on generative experiences (Beyond the Message) extends the same rules to whole pages, apps, and notifications, where Nielsen Norman Group has already named the risk: generated interfaces can cost people the consistency and learned efficiency they rely on. [35]

The rules are old. What is new is that a language model can fill an adaptive slot with something written for one person. The frame is what lets them trust it.

## What this does not do

The evidence here is older and smaller than the confidence people bring to it. The menu studies are lab studies with a few dozen participants, run on 2000s desktop software. Extending them to generated content in AI products is an inference, not a finding; I think it is a sound one, but I am labeling it.

The evidence also cuts against pure stability. Gajos found accuracy mattered more than predictability [18], and SUPPLE++ showed large gains from adapting the whole interface when the adaptation rested on measured ability [19]. "Never adapt the frame" is a default for general audiences, not a law. Accessibility may be the strongest case for adapting the frame itself.

Personas remain unfalsifiable [5], journey maps can become wall art, and qualitative research is slow and costly per person. That is its limit, and it is the gap that customer memory (Chapter 10) partly fills: memory read closely can show patterns across thousands of accounts. It cannot replace watching someone work.

Explanation is not a cure. Tintarev and Masthoff note that explanations can be built to persuade rather than to inform [23]; a "why am I seeing this" written to sell is a dark pattern with better manners.

## At scale

At scale, controls must be global and shared. A "not interested" pressed in the app has to reach the email program and the account manager's brief, or the control is only local, and the person experiences it as broken, which is the Mozilla finding reproduced across channels. One memory, many surfaces (the Customer Memory chapters and Playbook P9) is what makes a control mean the same thing everywhere.

Proxies also show up at scale. Netflix, which personalizes title artwork per member from viewing history, faced complaints in 2018 that some Black subscribers were shown artwork foregrounding minor Black cast members; the company said it did not use race, gender, or ethnicity, only viewing history. [36] A proxy can reproduce a sensitive attribute nobody collected, and people judge the output, not the input policy. Only a review of rendered outputs across groups catches it.

## Failure story: The Helpful Interruption

Larkspur (fictional) ships an in-app assistant for dispatchers. When the product sees three reassignments of the same job within five minutes, a rule opens a panel over the dispatch board offering to optimize the route. The trigger is accurate in its way: repeated reassignment does signal trouble. It fires most at 7 a.m., when dispatchers are reassigning jobs as technicians call in sick, and the panel covers the board they are working. Within a month most dispatchers have found the setting that turns the assistant off, including for its useful features. (Illustrative composite.)

The inference was fine. The interruption policy was the failure, as it was for the Office Assistant. The fix follows Horvitz and Amershi: move the suggestion into a non-modal slot beside the board, suppress it during the morning peak, scope it to one clear action, add "why am I seeing this," and let a dismissal teach the system something.

A good guess at the wrong moment is still a bad decision.

## Patterns

**Stable Frame, Adaptive Slots.** *Problem:* adaptation erodes people's learned map of the interface. *Forces:* personalization pays in specific places; people rely on everything else staying put. *Solution:* fix the frame (navigation, controls, legal, price, exit); adapt only in declared slots with contracts and fallbacks; change slots on events, not visits. *Tradeoff:* less expressive than whole-screen adaptation; much easier to test, support, and trust.

**Working Controls.** *Problem:* controls exist but do not visibly change outcomes, and people stop trusting them. *Forces:* model-level controls are harder to build than list filters. *Solution:* controls act on what the system believes (the profile, the evidence), apply across surfaces, say when they take effect, and are measured for effectiveness. *Tradeoff:* engineering cost; some users will exclude signals that were predictive.

**Triangulation Loop.** *Problem:* each discipline optimizes alone and misses what the others see. *Forces:* research is slow, models are opaque about why, experiments are narrow. *Solution:* qual finds the why, ML finds the who, experiments prove the what, and surprises return to qual. *Tradeoff:* slower first cycle; far fewer confident mistakes.

## Leader questions

1. Which parts of our product and pages are allowed to change per person, and who decided that?
2. When a customer tells us "stop showing me this," how many surfaces honor it, and how do we know it worked?
3. What was the last personalization decision that came from watching customers rather than from a dashboard?
4. Can a customer, or our support team, see why the system showed something?
5. Where are we using synthetic users, and is any of that output being treated as validation?

## Build checklist

- [ ] Map every surface into frame (fixed) and slots (adaptive); keep exits, prices, consent, and cancel paths in the frame.
- [ ] Give each slot a contract: purpose, allowed and forbidden evidence, limits, validations, fallback.
- [ ] Define the events that may change a slot; do not re-adapt on every visit.
- [ ] Ship "why am I seeing this" for every adaptive slot, grounded in the evidence actually used.
- [ ] Make controls act on the profile, propagate across channels, and state when they take effect.
- [ ] Measure control effectiveness: after a "not interested," how often does similar content return?
- [ ] Run small usability rounds per variant class (rich data, thin data, fallback, each role), and review rendered surfaces, not stored fields.
- [ ] Put a qualitative step in every release: read a sample of real outputs closely and classify findings by root cause.
- [ ] Tag synthetic research output as synthetic and keep it out of customer memory.

## Metrics to watch

- **Control effectiveness**: share of "not interested" or exclusion actions after which similar content does not return within a set window.
- **Adaptive-feature opt-out rate**: how many people turn personalization off, per surface; a rising rate is the Office 2000 signal.
- **Task time on core flows**, before and after adaptation, for the frame's key tasks (they should not get slower).
- **Explanation-request rate and follow-up**: how often people ask why, and how often they then correct or dismiss.
- **Share of personalization decisions with a qualitative origin**: how many shipped slots trace to research, not only to data.

## Reader Q&A

**Should we let users customize instead of personalizing for them?**
Offer both. Adaptable controls give people a sense of ownership and a way to correct the system; adaptive slots help people who do not know what they need. The evidence favors systems that adapt accurately inside a stable frame and let the person override.

**We ran five usability tests on the new personalized page. Are we covered?**
For the version you tested, on its obvious problems. Test small rounds per variant class, especially thin-data and fallback versions, and review a sample of what real customers were actually shown.

**Do personas still matter if we have customer memory?**
Yes, as a communication device and a source of hypotheses, not as data. Memory tells you what each customer did and said. A persona, grounded in research, keeps a team designing for a person rather than for a metric.

**Can we use AI-generated users to test personalization?**
To draft and pilot, yes. To decide, no. Simulated users flatten variance and agree too easily; decisions need real people or a controlled experiment.

## For your AI

```yaml
chapter: 3
title: "What Designers Knew"
concepts:
  - name: Adaptive vs adaptable
    definition: "Adaptive interfaces change themselves based on the system's model of the user; adaptable interfaces are changed by the user. Most working designs sit between: the system suggests or adapts in marked slots, and the person can override."
  - name: Stable Frame, Adaptive Slots
    definition: "Keep navigation, core controls, legal copy, prices, and exits fixed for everyone; allow change only in declared, bounded slots with contracts and fallbacks."
  - name: Predictability vs accuracy
    definition: "Two separate properties of adaptation; lab evidence shows both raise satisfaction and accuracy has the stronger effect, but inaccurate adaptation of what people rely on fails worst."
  - name: Mixed-initiative interaction
    definition: "Systems that share initiative with the person, choosing among doing nothing, asking, or acting according to uncertainty about the person's goal and the cost of interrupting."
  - name: Triangulation Loop
    definition: "Qualitative research finds the why, machine learning finds the who, experiments prove the what; surprises return to qualitative research."
  - name: Working Controls
    definition: "User controls that act on the system's model of the person, apply across surfaces, state when they take effect, and are measured for effectiveness."
  - name: Synthetic users
    definition: "LLM-simulated research participants; useful for drafting and piloting research, not for validating decisions."
decision_rules:
  - if: "an element is navigation, a core control, legal copy, a price, consent, or an exit path"
    then: "keep it in the stable frame; never personalize it"
  - if: "the system is uncertain about the person's goal or the cost of interrupting is high"
    then: "do less: stay quiet, or offer a non-modal suggestion in a slot rather than interrupting"
  - if: "a user control exists"
    then: "verify it changes the system's model and the next outputs across all surfaces; measure its effectiveness"
  - if: "a slot shows personalized content"
    then: "provide a short, true explanation on request grounded in the evidence actually used"
  - if: "a personalization idea comes only from data"
    then: "check it against qualitative research before building; if it comes only from research, size it with data; in both cases prove it with an experiment"
  - if: "testing a personalized surface"
    then: "run small usability rounds per variant class, including thin-data and fallback versions, and review rendered output"
  - if: "research output was produced by synthetic users"
    then: "treat it as a draft or hypothesis; validate with real people or an experiment; never write it into customer memory"
assessment_questions:
  - "Which surfaces change per person, and which elements are guaranteed not to change?"
  - "Which user controls exist for personalization, and has their effect been measured?"
  - "Can customers and support staff see why a personalized element was shown?"
  - "When did qualitative research last change a personalization decision?"
  - "How are personalized variants usability-tested, including fallback versions?"
  - "Where are synthetic users or simulated panels used, and for what decisions?"
patterns: [Stable Frame Adaptive Slots, Working Controls, Triangulation Loop]
anti_patterns: [The Helpful Interruption, The Shifting Map, Broken Controls, Synthetic Validation, Five-User Certainty]
maturity_dimension: experience_design
```

## References

1. Harris, J. (2006-03-31). "Combating the Perception of Bloat (Why the UI, Part 3)." Office User Interface Blog, Microsoft. https://learn.microsoft.com/en-us/archive/blogs/jensenh/combating-the-perception-of-bloat-why-the-ui-part-3
2. Horvitz, E., Breese, J., Heckerman, D., Hovel, D., & Rommelse, K. (1998). "The Lumière Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users." UAI 1998. https://arxiv.org/abs/1301.7385 ; project page with the Office Assistant account: http://erichorvitz.com/lum.htm
3. "Office Assistant." Wikipedia (off by default in Office XP, announced April 2001; removed in Office 2007). https://en.wikipedia.org/wiki/Office_Assistant ; Sinofsky, S. (2005-09-12). "More about learning from the past: Office Assistant." https://learn.microsoft.com/en-us/archive/blogs/techtalk/more-about-learning-from-the-past-office-assistant
4. Cooper, A. (1999). *The Inmates Are Running the Asylum.* Sams.
5. Chapman, C. N., & Milham, R. P. (2006). "The Personas' New Clothes: Methodological and Practical Arguments against a Popular Method." *Proc. HFES Annual Meeting* 50(5), 634 to 636. https://journals.sagepub.com/doi/10.1177/154193120605000503
6. Christensen, C. M., Hall, T., Dillon, K., & Duncan, D. S. (2016). "Know Your Customers' 'Jobs to Be Done'." *Harvard Business Review*, September 2016. https://hbr.org/2016/09/know-your-customers-jobs-to-be-done
7. Laubheimer, P. (2017-08-06). "Personas vs. Jobs-to-Be-Done." Nielsen Norman Group. https://www.nngroup.com/articles/personas-jobs-be-done/
8. Beyer, H., & Holtzblatt, K. (1998). *Contextual Design: Defining Customer-Centered Systems.* Morgan Kaufmann. https://dl.acm.org/doi/10.5555/286067
9. Gibbons, S. (2018-12-09). "Journey Mapping 101." Nielsen Norman Group. https://www.nngroup.com/articles/journey-mapping-101/
10. Flaherty, K. (2024-03-29). "Diary Studies: Understanding Long-Term User Behavior and Experiences." Nielsen Norman Group. https://www.nngroup.com/articles/diary-studies/
11. Nielsen, J. (2000-03-18). "Why You Only Need to Test with 5 Users." Nielsen Norman Group. https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ ; history in Sauro, J., "A Brief History of the Magic Number 5 in Usability Testing," MeasuringU. https://measuringu.com/five-history/
12. Spool, J., & Schroeder, W. (2001). "Testing Web Sites: Five Users Is Nowhere Near Enough." *CHI '01 Extended Abstracts*, 285 to 286. https://dl.acm.org/doi/10.1145/634067.634236
13. Faulkner, L. (2003). "Beyond the Five-User Assumption: Benefits of Increased Sample Sizes in Usability Testing." *Behavior Research Methods, Instruments, & Computers* 35(3), 379 to 383. https://doi.org/10.3758/BF03195514
14. Author's governed personalization engine: zone contracts (static shell, about 8 to 12 zones), rendered-surface review, and human-supervised review of early campaign cohorts with root-cause classification. Internal capabilities reference, 2026-08-25 revision, sections 5.1, 9.4, 10.1 to 10.3 (status LIVE; root-cause workflow partly productized). [PRODUCT-NAMING: name the engine here if the product decision allows]
15. Schade, A. (2016-07-10). "Customization vs. Personalization in the User Experience." Nielsen Norman Group. https://www.nngroup.com/articles/customization-personalization/
16. Sears, A., & Shneiderman, B. (1994). "Split Menus: Effectively Using Selection Frequency to Organize Menus." *ACM TOCHI* 1(1), 27 to 51. https://api.drum.lib.umd.edu/server/api/core/bitstreams/f53cafcd-2b8d-46d0-a8fe-87420f708f72/content
17. Findlater, L., & McGrenere, J. (2004). "A Comparison of Static, Adaptive, and Adaptable Menus." *CHI 2004*, 89 to 96. https://dl.acm.org/doi/10.1145/985692.985704
18. Gajos, K. Z., Everitt, K., Tan, D. S., Czerwinski, M., & Weld, D. S. (2008). "Predictability and Accuracy in Adaptive User Interfaces." *CHI 2008*. https://dx.doi.org/10.1145/1357054.1357252
19. Gajos, K. Z., Wobbrock, J. O., & Weld, D. S. (2008). "Improving the Performance of Motor-Impaired Users with Automatically-Generated, Ability-Based Interfaces." *CHI 2008*. https://dl.acm.org/doi/10.1145/1357054.1357250 ; SUPPLE: Gajos, K., & Weld, D. S. (2004), IUI 2004. https://dl.acm.org/doi/10.1145/964442.964461
20. Lavie, T., & Meyer, J. (2010). "Benefits and Costs of Adaptive User Interfaces." *International Journal of Human-Computer Studies* 68(8), 508 to 524. https://doi.org/10.1016/j.ijhcs.2010.01.004
21. Horvitz, E. (1999). "Principles of Mixed-Initiative User Interfaces." *CHI '99*, 159 to 166. https://erichorvitz.com/chi99horvitz.pdf
22. Amershi, S., Weld, D., Vorvoreanu, M., et al. (2019). "Guidelines for Human-AI Interaction." *CHI 2019*. https://www.microsoft.com/en-us/research/publication/guidelines-for-human-ai-interaction/
23. Tintarev, N., & Masthoff, J. (2007). "A Survey of Explanations in Recommender Systems." ICDE Workshops 2007. https://dl.acm.org/doi/10.1109/ICDEW.2007.4401070 ; extended in *UMUAI* 22 (2012). https://link.springer.com/article/10.1007/s11257-011-9117-5
24. Harambam, J., Bountouridis, D., Makhortykh, M., & van Hoboken, J. (2019). "Designing for the Better by Taking Users into Account: A Qualitative Evaluation of User Control Mechanisms in (News) Recommender Systems." *RecSys '19*, 69 to 77. https://doi.org/10.1145/3298689.3347014
25. Mozilla Foundation (2022-09-20). Blog post on the "Does This Button Work?" investigation of YouTube user controls. https://www.mozillafoundation.org/en/blog/mozilla-investigation-youtubes-dislike-button-other-user-controls-largely-fail-to-stop-unwanted-recommendations/
26. Spotify (2023-02-08). "'Exclude From Your Taste Profile' Will Make Your Personalized Recommendations Even Better." https://newsroom.spotify.com/2023-02-08/exclude-from-your-taste-profile-will-make-your-personalized-recommendations-even-better/
27. TikTok (2023-03-16). "Introducing a Way to Refresh Your For You Feed on TikTok." https://newsroom.tiktok.com/en-us/introducing-a-way-to-refresh-your-for-you-feed-on-tiktok-us
28. Kohavi, R., Henne, R. M., & Sommerfield, D. (2007). "Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO." KDD 2007. https://ai.stanford.edu/~ronnyk/2007GuideControlledExperiments.pdf ; HiPPO origin: https://exp-platform.com/hippo/
29. Kohavi, R., Tang, D., & Xu, Y. (2020). *Trustworthy Online Controlled Experiments.* Cambridge University Press, ch. 10, "Complementary Techniques." https://www.cambridge.org/core/books/abs/trustworthy-online-controlled-experiments/complementary-techniques/D65F23223FD2B9F88929BA3FF579E48C
30. Brignull, H. (2023). *Deceptive Patterns: Exposing the Tricks Tech Companies Use to Control You.* Testimonium. https://deceptive.design/about-us/
31. Mathur, A., Acar, G., Friedman, M. J., Lucherini, E., Mayer, J., Chetty, M., & Narayanan, A. (2019). "Dark Patterns at Scale: Findings from a Crawl of 11K Shopping Websites." *Proc. ACM HCI* 3 (CSCW), Article 81. https://arxiv.org/abs/1907.07032
32. Rosala, M., & Moran, K. (2024-06-21). "Synthetic Users: If, When, and How to Use AI-Generated 'Research'." Nielsen Norman Group. https://www.nngroup.com/articles/synthetic-users/
33. Budiu, R. (2025-08-15). "Evaluating AI-Simulated Behavior: Insights from Three Studies on Digital Twins and Synthetic Users." Nielsen Norman Group. https://www.nngroup.com/articles/ai-simulations-studies/
34. Park, J. S., Zou, C. Q., Kamphorst, J., et al. (2024, rev. 2026). "LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals" (earlier title: "Generative Agent Simulations of 1,000 People"). arXiv:2411.10109 (current abstract). https://arxiv.org/abs/2411.10109
35. Moran, K., & Gibbons, S. (2024-03-22). "Generative UI and Outcome-Oriented Design." Nielsen Norman Group. https://www.nngroup.com/articles/generative-ui/
36. *The FADER* (2018-10-22), report on Netflix artwork and race, with Netflix's statement. https://www.thefader.com/2018/10/22/netflix-target-users-race ; Chandrashekar, A., Amat, F., Basilico, J., & Jebara, T. (2017-12-07). "Artwork Personalization at Netflix." Netflix TechBlog. https://netflixtechblog.com/artwork-personalization-c589f074ad76

---

# 4. Shallow to Deep

*The question: Why was deep personalization always limited to a few people, and what does "deep" actually mean?*

> **Questions this chapter answers**
> - Why could we only ever afford to really understand a handful of customers?
> - What is the difference between shallow and deep personalization, in terms my team can act on?
> - Is more depth always better, or can knowing more make things worse?
> - What does it cost to understand one customer now, compared with a person doing it by hand?

## The short answer

For most of commercial history, personalization forced a choice. You could understand a few customers deeply, through account managers and advisers, or treat millions shallowly, through merge fields, segments, and scores. Not both, because understanding a person took a skilled person's time, the most expensive input in the building.

Picture the choice as a grid. One axis is **depth**: how much of a customer's situation a decision reflects. The other is **scale**: how many customers receive that treatment. Human service lives deep and few; first-era machine personalization, wide and shallow. Deep and wide stayed empty. Shallow and few is where failed pilots tend to live: expensive effort, a small audience, nothing learned.

AI changes the price of the input that kept the deep-and-wide corner empty. Reading an account, interpreting its situation, and drafting something specific now costs cents to about a dollar instead of tens of dollars of skilled time. That moves the frontier; it does not make every decision deep. Depth is a dial: build to the depth each decision requires, grounded in facts that are true, and nothing decorative beyond that.

The line to remember: a fact earns its place only if it changes what should be said.

---

Two prices, and the distance between them is the subject of this chapter.

The first is the postage on a presorted commercial marketing letter in the United States: between 37 and 47 cents a piece as of July 2026, depending on how finely it is sorted [1]. Add printing and it is still one of the cheapest ways ever devised to reach a stranger, cheap for a simple reason: it is substantially the same letter to everyone.

The second is the price of a skilled person paid to understand one customer. The median US wholesale and manufacturing sales representative earned between $72,080 and $104,920 in May 2025, depending on whether they sell technical products [2]. Wages are about 70% of what a private employer spends on a worker [3], so an hour of that person costs roughly $50 to $72 before tools, management, or unsold hours. Senior account managers cost more.

Attention that was cheap was not personal. Attention that was personal was not cheap.

*Larkspur Systems, the running example in this handbook, is a fictional composite.* It sells field-service and fleet software to about 40,000 customer accounts. It knows which accounts are renewing, which have an open escalation, which just hired a new operations lead. Its quarterly email says the same thing to all of them. Not laziness: arithmetic.

## For most of history, understanding had to be rationed

To write a message that fits one person, someone had to understand that person: their situation, their language, their goal. That took time and attention, both finite. Deep understanding at large scale was not forbidden by any law or policy. It was ruled out by arithmetic.

Peppers and Rogers argued for building relationships "one customer at a time" in 1993 [4], and they were right about the destination. They could not change the price of the trip, so the industry built what it could afford: a few deeply served accounts, everyone else handled by rules. The price of understanding was the policy.

## The Shallow-to-Deep grid

**The vertical axis is depth of understanding:** how far up the depth ladder (next section) a decision reaches. Low: a name or a segment label. High: the customer's situation, goals, and likely next need.

**The horizontal axis is scale:** how many people receive decisions at that depth. Low: dozens or hundreds, what a human team can hold. High: tens of thousands to millions.

The four quadrants:

| | **Few people** | **Many people** |
|---|---|---|
| **Deep** | **The Craft.** Account managers, private bankers, concierge desks, the seller who knows all thirty accounts. Excellent, capped by hours. | **Deep at Scale.** Situation-level understanding for every customer. Historically empty. Reachable now for the first time. |
| **Shallow** | **The Empty Quadrant.** Expensive effort applied to shallow data for a small audience. Neither the economics of scale nor the payoff of depth. Where failed pilots tend to live. | **The Broadcast.** Merge fields, segments, recommenders, propensity scores. Cheap per person, measurable, and blind to situation. |

For most of history the achievable options formed a line from top-left to bottom-right. You could slide along it, trading depth for reach, but you could not get above it, because every step up in depth multiplied the human hours per customer. Think of that line as the **cost frontier**.

![A two-by-two grid with depth of understanding on the vertical axis and scale on the horizontal axis. The Craft (deep, few people) is excellent but capped by hours; The Broadcast (shallow, many people) is cheap per person but blind to situation; The Empty Quadrant (shallow, few people) pays Craft prices for Broadcast depth; Deep at Scale (deep, many people) was historically empty. A dashed old cost frontier runs from The Craft down to The Broadcast, and cheap reading moves a new frontier up and to the right so that it passes through Deep at Scale.](/images/handbook/ch03-shallow-to-deep-grid.svg)

*Figure 4.1. The Shallow-to-Deep grid: AI moves the cost frontier, not the axes, and Deep at Scale becomes reachable.*

Two clarifications. The Broadcast is not stupid: a good recommender reads real behavior and matches items to people at a scale no human could. But it reaches rung three of the ladder below at most; it knows what a customer did, not what they are dealing with. And nobody chooses the Empty Quadrant. Teams arrive by pointing an expensive tool at a pilot group and feeding it the Broadcast's thin data. They pay Craft prices for Broadcast depth.

AI changes the frontier, not the axes. When reading a customer's record costs cents, the line moves up and to the right, and Deep at Scale stops being a contradiction. It becomes an engineering problem: memory, decisions, and governance, which is what the rest of this handbook is about.

## What "deep" means: the depth ladder

Here "deep" means how much of the customer's actual situation shapes the decision. Six rungs, on one Larkspur account.

1. **Name.** "Hi Jordan." Decides nothing.
2. **Firmographic or segment.** Mid-market logistics company, 120 vehicles, Pro plan. Decides which of five templates to send.
3. **Behavior.** Dispatch-module usage fell 40% over two months; two support tickets about route sync. Decides timing and topic.
4. **Situation and goals.** A new operations lead started six weeks ago; the fleet is expanding into a second region; renewal is in 90 days. Decides what is worth saying.
5. **Reasoning about this customer's problem.** The usage drop and the tickets point at one cause: the new region's depots were never configured, so dispatchers route by hand. Decides the recommendation.
6. **Anticipating the next need.** A second region usually means a second dispatcher team to onboard before peak season. Decides whether to act now, later, or not at all.

The first era computed rungs one to three from structured data. Rungs four to six lived in call notes, emails, tickets, and an account manager's head. Reaching them meant a person reading: the rationed input.

Two warnings. The ladder measures understanding, not data volume; an account with ten thousand events can sit on rung three. Events record what people did, not what they were trying to do; Chapter 3 shows how field research reaches the rungs telemetry cannot. And confidence falls as you climb: rung six holds the most value and the most confident mistakes.

![The depth ladder drawn as six ascending rungs, each with the decision it drives: 1 Name decides nothing; 2 Firmographic or segment decides which of five templates; 3 Behavior decides timing and topic; 4 Situation and goals decides what is worth saying; 5 Reasoning about this customer's problem decides the recommendation; 6 Anticipating the next need decides whether to act now, later, or not at all. Rungs one to three were computed from structured data in the first era; rungs four to six lived in call notes, emails, tickets, and an account manager's head, so they needed a person reading. Confidence falls as you climb, and rung six holds the most value and the most confident mistakes.](/images/handbook/ch03-depth-ladder.svg)

*Figure 4.2. The depth ladder: each rung decides more, and the top three once required a person reading.*

## Depth is a dial, not a goal

The grid does not say "move everything to the top-right." Here is the sentence this chapter turns on. **One correct fact that changes what should be said is worth more than a hundred facts that only decorate it.**

Depth costs money, time, and risk. Detail makes a message worse when it is irrelevant, stale, or included to prove the sender found it. What matters is decision-level fit: the real constraint, priority, timing, or unfinished problem that makes the communication useful now. A model can be wrong in several places and right at the one place where the decision turns.

So set the dial per decision. A password reset needs rung one at most. A strategic renewal needs rung five. A reactivation offer might need rung three and nothing more.

The dial also turns by reader. In the governed personalization engine I built, one account's research compiles into an analytical seller brief and a short customer email. The seller may see a rung-five hypothesis about the account's problem that is never allowed into the buyer's email. A guess shown to a colleague is context; the same guess stated to a customer is a claim.

Customers report the cost of getting it wrong. In a Gartner survey of 1,464 B2B buyers and consumers, fielded in late 2024, 53% reported negative experiences with personalization, and they were 3.2 times more likely to regret a purchase [5]. Gartner locates the misfire where buyers switch tasks and their problem is more complex than the offer. In this frame, that is a depth failure: a rung-three offer to someone living at rung four. (Self-reports; read the direction, not the magnitude.)

A shallow model is not a failed model. It is a model built to the specification the decision required. When the facts to go deeper are missing or unverified, do not invent specificity. Become honestly general.

In the engine I built, honestly general is a ladder the system walks down on purpose: person and account, then account only, then segment, then approved static copy, then fail closed. A lead with a freemail address does not inherit the email provider's firmographics; it gets lower-specificity copy. A headcount that contradicts other evidence is hidden, because a missing number costs less than a confident wrong one. And "your renewal is coming up", a rung-four fact, is blocked unless the relationship is actually known; otherwise the copy is reframed as an evaluation. Specificity degrades before reliability does.

## The binding constraint was the cost of understanding

Put numbers on it, as ranges with the assumptions visible.

**The human pass.** Assume a competent person needs 20 to 60 minutes to read an account's history, tickets, usage, and public news, and draft a first touch that reflects the situation. That range is my working assumption, not a measured figure. At $50 to $72 an hour loaded, one pass costs about $17 to $72 per account. For Larkspur's 40,000 accounts that is 13,000 to 40,000 hours: six to nineteen people doing nothing else for a year, before a single reply. And sellers never had the whole year: in Salesforce's 2022 survey of 7,775 sales professionals, reps reported spending 28% of their time actually selling [6].

**The machine pass.** As of September 2026, one major provider lists a mid-tier model at $2 per million input tokens and $10 per million output tokens, halves both for batch jobs, and charges $10 per 1,000 web searches [7]. Assume a research run reads 30,000 to 300,000 tokens (record, tickets, notes, a few web pages), writes 3,000 to 15,000, and runs 5 to 20 searches. That is roughly $0.15 to $1 per account at standard prices; a top-tier model costs several times more. For 40,000 accounts: about $6,000 to $40,000 per pass.

![A log-scale comparison of the cost to research one account. By hand, 20 to 60 minutes at $50 to $72 an hour costs $17 to $72 per account, or 13,000 to 40,000 hours for 40,000 accounts. By machine, one research run with a mid-tier model at standard prices costs about $0.15 to $1 per account, or about $6,000 to $40,000 per pass over 40,000 accounts. The gap between them is one to two orders of magnitude.](/images/handbook/ch03-two-prices-per-account.svg)

*Figure 4.3. Understanding one account costs tens of dollars by hand and cents to a dollar by machine.*

The model call is not the whole system; identity, memory, and review cost real money (Chapter 12). But on the rationed input, the gap is one to two orders of magnitude and widening. The price of reaching a fixed capability level has fallen between 9x and 900x per year depending on the task [8]; GPT-3.5-level performance fell from $20 to $0.07 per million tokens between November 2022 and October 2024 [9].

The bigger change is where the money lands. The old model spent the expensive hour first, on a guess. The new model spends cents on every account, watches for a signal, and brings in a skilled person or a stronger model only when a customer engages or an exception needs judgment. **The expensive hour is no longer spent on the guess. It is reserved for the signal.**

The calculation underneath is one every growth team already runs: what a better decision is worth, times how likely the approach is to produce it, minus what it costs. Spend where that is positive. When the cost term falls fiftyfold, the set of customers worth deep treatment grows quietly. Accounts that never cleared the bar for an hour of human attention now clear it easily.

## What this does not do

The easiest way to oversell this chapter is to treat a falling price as a solved problem. Here is what it does not claim.

**Cheap reading is not accurate understanding.** A one-dollar pass can produce a fluent, wrong account summary. The price of depth fell; the price of verified depth fell less. Chapters 11 and 20 close that gap.

**Shallow still works, a little.** In randomized field experiments, adding the recipient's name to an email subject line raised open rates from 9.05% to 10.80% and reduced unsubscribes [10]. Rung one is not worthless. It is a low ceiling.

**"Deep for everyone" is overreach.** AI does not make every customer a strategic account. The Craft still wins where the relationship is the product. The frontier moved; the top-left corner did not disappear.

**Failure was never only about depth.** In 2019, Gartner predicted that 80% of marketers invested in personalization would abandon it by 2025, citing lack of ROI, the perils of customer data management, or both [11]. A prediction, not a finding; I suspect much of what got abandoned lived in the Empty Quadrant. But the data half is real, and cheaper models do not fix it. Bad identity makes deep personalization confidently wrong about the wrong person.

**Falling token prices do not mean falling bills.** Research agents consume far more tokens than a chat reply. Budget per decision.

**A large design space is not a large experiment.** Cheap depth multiplies what a system could say. In an illustrative campaign from the engine I built, 4 personas, 6 industries, 5 regions, 8 message themes, and 3 offers yield 2,880 distinguishable contexts, and a page with 10 zones of 5 eligible options each has 5^10, nearly 9.8 million, possible compositions per context. That is expressive capacity, not experiment volume. Nobody generates or tests those pages, and no cohort is large enough to. The practical move is a few strong hypotheses and learning where the good regions are (Chapter 21).

## At scale

At a few hundred accounts, depth is a research problem; at a few hundred thousand, a systems problem. Understanding must be stored, not recomputed (Chapters 10 and 11); in the engine I built, one account's research feeds the page, the email, and the seller brief, so it is paid for once and the surfaces are less likely to contradict each other. Depth must be chosen per decision by policy, not by whoever wrote the prompt (Chapter 14). And errors become rates: a 2% rate of confidently wrong specifics is invisible in a demo and is 800 bad messages across Larkspur's 40,000 accounts. Deep at Scale is not the Craft multiplied; it fails differently.

## Failure story: The Empty Quadrant pilot

Larkspur's first AI move was an AI SDR pilot aimed at 300 prospect accounts. The tool was not cheap. The data it received was a name, a title, an industry, and a company size: rung two, plus whatever an ungoverned web search returned for the name and company. The model did what fluent models do with thin data: it wrote warm, specific-sounding paragraphs about challenges "companies like yours" face, and occasionally about challenges the prospect did not have. Reply rates fell below the plain template the team had used before.

The post-mortem blamed the model. The pilot paid for depth, supplied none, and reached too few accounts to learn from. Craft prices, Broadcast depth.

## Patterns

**Depth to Decision.** *Problem:* teams inject every available fact. *Forces:* more context feels safer; each fact adds cost and error risk. *Solution:* per decision, name the required rung and the few facts that would change the output; retrieve and verify those, ignore the rest. *Tradeoff:* you must define the decision first (Chapter 6).

**Wide, Then Deep.** *Problem:* top-tier depth on every account is still too expensive. *Forces:* signals are uneven. *Solution:* a cheap pass over everyone to rung three or four; escalate to deeper reasoning or a human only on signal or exception. Put it in configuration, not discipline: in a self-hosted memory system I built, each function and tier routes to its own model (a small one for extraction on every record, a strong one only for the top tier), retrieval has a no-model mode that answers "do we know anything?" before a synthesized brief is paid for, and bulk loads can go through a provider's batch interface at about half price. *Tradeoff:* the cheap pass sets the ceiling; a missed signal is invisible downstream.

**Honestly General.** *Problem:* no verified fact exists at the depth the decision wants. *Forces:* generation fills gaps fluently. *Solution:* a defined fallback to the rung the evidence supports, treated as correct behavior. *Tradeoff:* a lower peak on some messages for far fewer confident errors.

## Leader questions

1. For our five most valuable repeated decisions, what rung of the depth ladder does each one actually need?
2. Where are we paying Craft prices for Broadcast depth today?
3. What does one researched account cost us today, by hand and by machine?
4. When our system lacks a fact, does it fall back to honestly general, or does it guess?

## Build checklist

- [ ] Every personalized decision declares its target rung and the facts it depends on.
- [ ] Rung four and above facts carry a source and a date.
- [ ] A cheap wide pass runs on everyone; escalation rules define when deeper work or a human is triggered.
- [ ] Research is stored and reused, and cost is tracked per account and per decision.
- [ ] An honestly general fallback exists for every generated field.
- [ ] Pilots are sized large enough to measure, with a plain-template control.

## Metrics to watch

- **Depth reached per decision:** share of decisions resting on a verified rung-four-or-higher fact.
- **Unsupported specificity rate:** share of personal claims that cannot be traced to a source.
- **Signal-first ratio:** share of expensive hours spent after a customer signal, not before.
- **Lift over honestly general:** the deep version against the best generic version, not against nothing.

## Reader Q&A

**Should we skip segments?** No. Segments are cheap, explainable, and often right. Add depth where a segment gives the wrong answer for enough customers to matter.

**Our AI pilot underperformed the old template. Is this overhyped?** Check the quadrant first. Rung-two data for a few thousand people tests the Empty Quadrant, not deep personalization.

**How deep is too deep?** When a fact would surprise the customer that you know it, or does not change the decision, leave it out. Chapter 19 covers the first case; this chapter the second.

## For your AI

```yaml
chapter: 4
title: Shallow to Deep
concepts:
  - name: Shallow-to-Deep grid
    definition: "A 2x2 of depth of understanding (vertical) against scale (horizontal). Quadrants: The Craft (deep, few), The Broadcast (shallow, many), The Empty Quadrant (shallow, few), Deep at Scale (deep, many)."
  - name: Cost frontier
    definition: "The historical line from deep-and-few to shallow-and-many; companies could trade along it but not exceed it because depth required human hours. AI moves it outward."
  - name: Depth ladder
    definition: "Six rungs of understanding a decision can reach: 1 name, 2 firmographic or segment, 3 behavior, 4 situation and goals, 5 reasoning about this customer's problem, 6 anticipating the next need."
  - name: Depth is a dial
    definition: "Set depth per decision to what the decision requires; decision-level fit beats volume of detail."
  - name: Honestly general
    definition: "The correct fallback when facts at the required depth are missing or unverified: a message at the rung the evidence supports. Implemented as a specificity ladder: person and account, account, segment, approved static copy, fail closed."
  - name: Expressive capacity is not experiment volume
    definition: "The combinatorial space of contexts and zone compositions a governed system could produce (for example 2,880 contexts x 5^10 compositions) describes what it could say, not what has been tested; learn through a few strong hypotheses."
  - name: Signal-first spending
    definition: "Spend cheap machine attention on everyone and reserve expensive human or model attention for customers who show a signal or exception."
decision_rules:
  - if: "a decision's output would not change given a fact"
    then: "do not retrieve or include that fact"
  - if: "the decision needs rung 4 or higher and the supporting fact has no source or date"
    then: "fall back to honestly general at the highest verified rung"
  - if: "identity is weak (for example a freemail address) or a firmographic value contradicts other evidence"
    then: "suppress company-specific facts, hide the value, and step down the specificity ladder"
  - if: "an output goes to the customer rather than an internal seller"
    then: "exclude hypotheses and unconfirmed relationship statements; keep them in internal surfaces only"
  - if: "a pilot uses rung 1 to 2 data on fewer accounts than needed to measure lift"
    then: "flag it as an Empty Quadrant pilot; add data depth or scale before judging the approach"
  - if: "expected value of deep treatment for an account (value x likelihood minus cost) is positive at current machine cost"
    then: "include the account in the wide pass and escalate on signal"
  - if: "an action is high-stakes or relationship-defining"
    then: "keep a human (The Craft) in the loop regardless of machine depth"
assessment_questions:
  - "What are your five most valuable repeated customer decisions, and which depth rung does each need?"
  - "Which rungs can your current data support for most customers?"
  - "What does one researched account cost today by hand, and by machine?"
  - "Where do your pilots sit on the grid?"
  - "What happens when a generated message lacks a verified fact?"
patterns: [Depth to Decision, "Wide, Then Deep", Honestly General]
anti_patterns: [The Empty Quadrant Pilot, Decorative Depth, Paying Craft Prices for Broadcast Depth]
maturity_dimension: understanding
```

## References

1. United States Postal Service, Price List Notice 123, effective 2026-07-12. USPS Marketing Mail commercial automation letters: $0.374 (5-digit, DSCF entry) to $0.467 (mixed AADC). https://pe.usps.com/text/dmm300/notice123.htm
2. US Bureau of Labor Statistics, Occupational Outlook Handbook, "Wholesale and Manufacturing Sales Representatives" (median wages, May 2025). https://www.bls.gov/ooh/sales/wholesale-and-manufacturing-sales-representatives.htm
3. US Bureau of Labor Statistics, "Compensation costs for private industry workers averaged $46.89 per hour worked in June 2026," The Economics Daily. https://www.bls.gov/opub/ted/2026/compensation-costs-for-private-industry-workers-averaged-46-89-per-hour-worked-in-june-2026.htm
4. Peppers, D., and Rogers, M. (1993). *The One to One Future: Building Relationships One Customer at a Time.* Currency/Doubleday.
5. Gartner, "Gartner Survey Reveals Personalization Can Triple the Likelihood of Customer Regret at Key Journey Points," 2025-06-03. https://www.gartner.com/en/newsroom/press-releases/2025-06-03-gartner-survey-reveals-personalization-can-triple-the-likelihood-of-customer-regret-at-key-journey-points
6. Salesforce, State of Sales, 5th edition (2022 survey, n = 7,775). https://www.salesforce.com/news/stories/sales-research-2023/
7. Anthropic, API pricing (list prices, batch discount, web search), accessed 2026-09-26. https://platform.claude.com/docs/en/about-claude/pricing
8. Epoch AI, "LLM inference prices have fallen rapidly but unequally across tasks," 2025-03-12. https://epoch.ai/data-insights/llm-inference-price-trends
9. Stanford HAI, AI Index Report 2025. https://hai.stanford.edu/ai-index/2025-ai-index-report
10. Sahni, N. S., Wheeler, S. C., and Chintagunta, P. (2018). "Personalization in Email Marketing: The Role of Noninformative Advertising Content." *Marketing Science* 37(2). https://pubsonline.informs.org/doi/abs/10.1287/mksc.2017.1066
11. Gartner, "Gartner Predicts 80% of Marketers Will Abandon Personalization Efforts by 2025," 2019-12-02. https://www.gartner.com/en/newsroom/press-releases/2019-12-02-gartner-predicts-80--of-marketers-will-abandon-person

---

# 5. The New Era

*The question: What exactly changed with AI, and where is it going?*

> **Questions this chapter answers**
> - What can AI actually do for personalization today that it could not do three years ago?
> - Can it really understand a customer, and can it write as well as a person?
> - How cheap is it now, and why is my AI bill still going up?
> - How long can an agent work without a person, and where will that be in two or three years?
> - What is real, what is a demo, and what is a forecast?

## The short answer

What changed is the price of understanding one person. For the whole history of business communication, reading a customer's situation closely enough to say something specific and true required a skilled human's hour. That hour was the binding constraint, so depth went to the few accounts that could pay for it and everyone else got the template.

Language models removed most of that constraint for six kinds of work, which I call the capability stack: read, reason, decide, write, act, learn. Each layer is real today, each is uneven, and none is free.

Three facts should shape your planning. First, the price of a fixed level of model capability has fallen by roughly 10x to several hundred times per year depending on the task, but agents consume far more tokens per task, so your spend per decision is an architecture choice, not a market price. Second, the length of task an agent can complete without a person has been doubling roughly every six to seven months on software benchmarks, faster since 2024; that is a measured trend on narrow tasks, not a guarantee about your customers. Third, fluency is not truth. The system that writes a brilliant account brief can invent a detail with the same confidence.

The practical conclusion: deep personalization for every customer is now an engineering and governance problem rather than a staffing problem. Everyone rents the same models. The winners will be the companies that build the memory, the decisions, and the checks that turn cheap understanding into accurate action.

---

Three hours, and then four minutes.

Larkspur Systems (a fictional composite company used throughout this handbook) sells field-service and fleet software to about 40,000 accounts. Priya, an account manager, has a renewal call Thursday with Brightwater Fleet Services (also fictional), a regional operator with 300 vehicles. On Tuesday she reads eighteen months of CRM notes. She skims fourteen support tickets and notices that nine of them came from the same dispatcher. She listens to half of a recorded quarterly review in which the operations director says, almost in passing, that the board wants fuel costs down before the next budget cycle. She finds a job posting for a "telematics analyst," a role Brightwater has never had. Her brief: the dispatcher is struggling with routing, the director is under cost pressure, and the new hire suggests they are about to take their data seriously. Lead with fuel-cost reporting, offer routing training, and do not open with the price increase.

That brief took about three hours. At the loaded cost of skilled selling time from Chapter 4, roughly $50 to $72 an hour, it cost Larkspur at least $150 to $220, and more in practice, because senior account managers cost more than that median. It was worth it, and impossible to repeat for more than a few hundred of 40,000 accounts. Doing all of them would take 120,000 hours (a full three-hour brief, not the 20-to-60-minute first touch Chapter 4 priced), roughly sixty full-time people for a year. So Larkspur does what every company does: deep work for the top accounts, a segment email for everyone else.

Now run the same job through an agent. It reads the same notes, tickets, transcript, and posting, plus public news, and writes a comparable brief in a few minutes. A run of this shape consumes somewhere around a hundred thousand to a million tokens, on the order of $0.15 to a few dollars per account at current prices for capable models. The low end matches Chapter 4's $0.15 to $1 research run; the high end is higher because this is agentic, multi-step reading of a transcript and a long history, not a single research pass. Across all 40,000 accounts, that is roughly $6,000 to $120,000, depending almost entirely on how the system is built.

The ratio is the story: the price of a first-pass understanding of one account fell by two to three orders of magnitude. It is also the change most easily misread.

![A log-scale bar chart comparing the cost of one account brief. A person working about three hours costs $150 to $220, more in practice; an agent working a few minutes costs $0.15 to a few dollars, two to three orders of magnitude less. Below, across 40,000 accounts at Larkspur (fictional), people would need 120,000 hours, roughly sixty full-time people for a year, while agents would cost roughly $6,000 to $120,000 depending almost entirely on how the system is built.](/images/handbook/ch04-cost-of-one-brief.svg)

*Figure 5.1. On a log scale, a first-pass account brief fell from hundreds of dollars of skilled time to cents or dollars of model time.*

## Understanding got cheap; being right did not

Here is the sentence this chapter turns on: **the cost of understanding a customer used to decide who got understood; now the decision moves from the budget to the architecture.**

When understanding was expensive, the budget did the rationing; nobody had to decide which of 40,000 accounts deserved depth, because the arithmetic of human hours decided. When understanding is cheap, that implicit policy disappears, and three explicit questions replace it. Which decisions deserve depth? How do we know the understanding is true? What happens when it is wrong at scale? Most of this handbook answers those questions.

## The capability stack: read, reason, decide, write, act, learn

Six layers arrived, at different speeds, and they fail in different ways.

| Layer | What it does for personalization | Maturity (as of 2026-09) |
|---|---|---|
| **Read** | Extracts facts and signals from calls, emails, tickets, documents, images | Ships now, widely |
| **Reason** | Interprets ambiguous situations; forms a view of the account | Ships now, uneven |
| **Decide** | Chooses the next action for one person, including doing nothing | Ships now inside designed systems; weak when left to the model alone |
| **Write** | Generates language for one person at human quality | Ships now; accuracy is the constraint |
| **Act** | Runs a bounded loop of tool calls without a person between steps | Ships now for bounded tasks; long unattended runs are early |
| **Learn** | Carries what happened into the next decision | Ships now as memory and feedback you build; not as the model changing itself |

### It reads more than the customer said

Start with reading, because leaders underestimate it. These systems do not only recover stated facts. They infer relationships among the facts and propose what those relationships imply. Give a model a customer's support history, a call transcript, and a job posting, and it may infer, as Priya did, that a team is struggling with one module or that a new hire signals a change in how the customer uses data.

Those are inferences, not knowledge. Some will be wrong, and fluency can make a weak inference sound much stronger than it is. But the most valuable thing in a model of a customer is often not a fact they stated. It is a pressure, preference, relationship, or unfinished problem inferred across facts. Until recently that required a thoughtful person reading closely. A machine can now attempt it across a whole customer base.

The first-era systems in Chapter 2 could not. A propensity model reads columns, and the director's aside about fuel costs never became a column. Most of what a company knows about its customers sits in calls, emails, notes, and documents, and for the first time it is readable at the scale of the customer base.

In a governed personalization engine I built, account research lands as structured fields, not one long paragraph: industry, buying triggers, leadership changes, technology initiatives, stated priorities, challenges. Each field can be trust-checked on its own, and each surface renders only the fields it needs instead of asking a model to restate them. The next step I am building attaches confidence, source, and freshness to every field.

### It works through an account the way an expert would

The deeper change is that the same medium can be used to work through a problem. A capable system can take an objective, break it into parts, compare routes, use evidence and tools, inspect a result, and revise its next move. It does not have to resemble human thought internally to reproduce some of the work expert reasoning performs externally.

Expertise was scarce not only because experts knew more facts, but because they could structure a difficult problem: identify what mattered, discard weak approaches, notice missing evidence, choose the next step. The best evidence that this transfers comes from the field. In a study of 5,172 customer-support agents, access to a generative AI assistant raised issues resolved per hour by 14% on average and by 34% for the least experienced workers, largely by spreading the practices of top performers to everyone else ([Brynjolfsson, Li, Raymond, 2025](https://academic.oup.com/qje/article/140/2/889/7990658)). That is the pattern that matters for personalization: the reasoning of your best account manager becomes available on every account.

One caution matters in every later chapter. A displayed explanation feels like a judgment you have inspected, but it is not an audit trail. When researchers slipped hints into questions, reasoning models that used the hint mentioned it in their reasoning only 25% (Claude 3.7 Sonnet) to 39% (DeepSeek R1) of the time ([Chen et al., 2025](https://arxiv.org/abs/2505.05410)). The explanation is an output to evaluate, not a record of why the system acted. That record has to be built separately (Chapter 18 calls it the decision trace).

### It decides, but only as well as the system around it

Reasoning produces a view. A business needs a decision: what to do next for this person, across thousands of people, under rules. Models can propose a next action with a rationale for every account overnight. What they cannot do alone is hold the organization's objectives, contact rules, and constraints consistently across 40,000 decisions. That comes from a designed decision layer where deterministic policy stays deterministic (Chapter 14). The capability to decide arrived; the reliability of deciding is something you build.

In the same engine, a model scores each lead against the campaign's criteria and returns a score, a band, a rationale, and a suggested approach, weighing weak and strong signals together the way a good account manager would. Code then checks that the pieces agree: a score of 88 labeled LOW is flagged, not sent. The model supplies judgment; the system supplies consistency.

### It writes like a person, for each person

Software can now write in the idiom, length, and tone a moment calls for, cheaply enough to do it separately for every customer. In a 2025 three-party Turing test, GPT-4.5 prompted to adopt a persona was judged to be the human 73% of the time, more often than the real humans it was compared against ([Jones and Bergen, 2025](https://arxiv.org/abs/2503.23674), preprint). In marketing, a set of studies found AI-generated images matched or beat human-made ones, and a field test with more than 173,000 impressions reported click-through rates up to 50% higher than stock imagery ([Hartmann, Exner, Domdey, 2025](https://www.sciencedirect.com/science/article/pii/S0167811624000843)).

That does not mean every output passes. It means the old assumption, that volume must reveal a template, no longer holds. The two halves of personalization, forming a useful account of someone and speaking to them as an individual, can now be joined in software. In the engine I built, one account context feeds a seller playbook, an email sequence, and a landing page, and the hypotheses a seller may see never reach the buyer.

It also means the capability cuts both ways. In a 2025 debate experiment, GPT-4 given six demographic facts about its opponent was more persuasive than a human in 64.4% of the pairs where one side was more persuasive; a 2026 author correction adds that its advantage over un-personalized GPT-4 did not reach statistical significance ([Salvi et al., 2025](https://pmc.ncbi.nlm.nih.gov/articles/PMC12367540/)). That is one reason this handbook spends a whole part on governance.

### It acts: the loop that used to need a person

Running an account well is not one message. It is a loop: observe the customer's world, plan, act, read what comes back, adjust. A good account manager does this over weeks, and every turn needs their attention.

Software can now run a bounded stretch of that loop itself. You give it a goal, particular tools, and limits. It observes, decides, acts, reads the result, and decides again without returning to you between steps. The human moves from driving each step to setting the objective, granting the tools, defining the stopping conditions, and handling the exceptions. Traditional automation acted only along routes written in advance, which is why it was reliable. An agent is useful where the route cannot be fully written beforehand, which is also what makes it harder to predict and control.

In the systems I run, "bounded" is literal. In a self-hosted memory system I built, an outside agent reaches customer memory through a tool interface (MCP) limited to its role, and autonomous agents are never offered deletion. A long job can run on a model provider's own background service, carrying a credential minted for that job alone: three memory tools, revoked when the job ends, and the job cancelled after 24 hours by default. Autonomy is a set of limits written down in advance.

### It learns, in a narrower sense than the word suggests

*Update* is the more accurate word. An agent carries new information from one step into the next without retraining the underlying model. Across sessions, it remembers only what you built a memory to hold. That is not a limitation to wait out. It is where most of the durable value sits (Part IV).

In my systems, learning ships today in two forms. Memory is curated while the system is idle, by passes that only propose changes until an operator promotes them (Chapter 11). And a person uses AI to read across the first cohorts of real output, asking which errors repeat and which fixes belong in code rather than in another instruction. AI that implements and validates its own changes before a person approves them is a design I am pursuing, not a result.

## The letter became a conversation

Put reading, writing, and reasoning together in real time and personalization changes shape. The old message was a letter. It was written once, said the same thing to everyone in a segment, and could not respond. If the customer had a doubt the letter did not anticipate, the letter had no answer.

What arrived is a conversation, and a company can hold a different one with each customer. A support exchange can pick up where the last one ended. A website can answer the question this visitor is actually asking rather than the question the page was designed for. An onboarding flow can notice that a user is stuck and ask why, instead of sending day-three email number two. The same shift reaches pages, product screens, and notifications, which can now be composed for one person; Chapter 16 sets out how far that ships. A static message has to anticipate everyone. A conversation only has to answer this person, one turn at a time, and each reply tells the system more about what matters.

A letter is judged once, when it is written. A conversation is judged at every turn, so the rules, the memory, and the checks have to be present at every turn too. Personalization used to be a property of a message. It is becoming a property of a relationship.

## The price fell; the bill did not

Now the economics, where most leadership conversations go wrong. The price of a fixed level of capability has collapsed. The Stanford AI Index reports that the cost of querying a model scoring at GPT-3.5's level on the MMLU benchmark fell from $20.00 per million tokens in November 2022 to $0.07 in October 2024, a more than 280-fold reduction ([Stanford HAI, AI Index 2025](https://hai.stanford.edu/ai-index/2025-ai-index-report)). Epoch AI, tracking the cheapest model to reach fixed thresholds across six benchmarks, found prices falling between 9x and 900x per year depending on the task; GPT-4-level performance on MMLU went from $37.50 to $0.18 per million tokens between March 2023 and February 2025, about 40x a year ([Epoch AI, 2025-03-12](https://epoch.ai/data-insights/llm-inference-price-trends)). Epoch cautions that the fastest drops were the most recent and may not persist.

So why are AI bills rising? Because nobody buys a fixed capability for a fixed task. Agents read more, reason longer, and call tools repeatedly. Anthropic reported that in its own systems, agents typically used about 4x more tokens than chat interactions and multi-agent systems about 15x more, and that token usage alone explained 80% of the performance variance on the evaluation it studied ([Anthropic, 2025-06](https://www.anthropic.com/engineering/multi-agent-research-system)). Better results are bought with more reading.

![Two linear-scale charts side by side. Left: the price per million tokens for GPT-3.5-level performance on MMLU fell from $20.00 in November 2022 to $0.07 in October 2024, more than 280-fold cheaper. Right: in Anthropic's own systems, agents used about 4x the tokens of chat and multi-agent systems about 15x. Beneath them, price per token is set by the market, while tokens per decision are set by your architecture.](/images/handbook/ch04-price-fell-bill-did-not.svg)

*Figure 5.2. The market sets the price per token; your architecture sets how many tokens each decision consumes.*

This is where I part company with the popular reading of "LLMflation," a16z's observation that the cost of a given capability falls about 10x a year ([Appenzeller, 2024](https://a16z.com/llmflation-llm-inference-cost/)). The observation is right. The conclusion people draw, that AI cost will take care of itself, is wrong. Price per token is set by the market. Tokens per decision are set by your architecture. A pipeline that researches the same company once per employee, or loads the whole customer history into every call, makes cheap tokens expensive. On a production pipeline I run, per-record research bought the same company's research close to a hundred times in one day for one popular employer. Moving research to the company, cached for seven days, cut that spend and raised the share of records with rich research from 60.3% to 93.5%. The budget stopped paying for answers it already had.

Where the model runs is an architecture choice too. In the self-hosted memory system I built, each function gets its own model: a cheap one for extraction, a strong one only where judgment pays. Bulk loads can go through a provider's batch interface at roughly half the per-token price. A local model removes the per-token bill entirely and trades it for hardware and lower concurrency.

The unit to manage is cost per decision, not price per token. Chapter 12 does that accounting.

## How long it can go without you

How far does autonomy go today? The honest answer is a moving line. One useful measure is not how smart a system seems but how long a task it can complete before it needs a person. METR estimates this "time horizon" by comparing an AI system's success on tasks with the time skilled people take to do them. In its original 2025 analysis, the length of task frontier systems could complete with 50% reliability had been doubling roughly every seven months for six years ([METR, 2025-03-19](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/)). An updated methodology published in January 2026 put the long-run doubling at about 196 days, and at about 89 days for models since 2024; its top measured model at the time, Claude Opus 4.5, had a 50% horizon of about 320 minutes, with a confidence interval from 170 to 729 minutes ([METR Time Horizon 1.1, 2026-01-29](https://metr.org/blog/2026-1-29-time-horizon-1-1/)). By May 2026, METR's tracking page warned that measurements above 16 hours are unreliable with its current task suite, which tells you where the frontier had reached.

Three qualifications keep this honest. The tasks are mostly software engineering, machine learning, and cybersecurity; METR itself describes capability as jagged across domains. The horizon is measured against people with little prior context, not experienced staff who know the account. And the success threshold matters as much as the length. A task completed half the time is impressive in a demonstration and unusable in a process that must work every night; require eight successes in ten and the horizon becomes far shorter. Capability changes what is possible. Reliability decides how the work must be supervised.

Field evidence agrees. In a randomized trial of 16 experienced open-source developers working on 246 real issues, early-2025 AI tools made them 19% slower, while the developers believed they had been sped up by about 20% ([METR, 2025-07-10](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/)). Gains are real and uneven, and the difference is in where you point the system.

For personalization, full autonomy is not required for the operating model to change. If a system handles the repeatable middle of the account loop and returns only the ambiguous or consequential moments, one person can supervise far more accounts than they could personally work. When I build these systems, keeping them on task is most of the work, and the gain comes from deciding in advance which failures are tolerable and which uncertainty sends the case back to a person.

## Ships now, lab demo, forecast

Here is the map I would give a leadership team, with each claim labeled by kind. Dates matter: this is as of September 2026.

![Three columns sorting AI capabilities by kind of claim, as of September 2026. Ships now: reading calls, emails, and tickets; a brief for every account; individually written messages; conversational personalization; and bounded agent tasks running for hours, with accuracy depending on checks. Lab demo or early production: agents managing a relationship for weeks, and interfaces and offers composed per person, not yet common practice. Forecast: reliable day-long tasks and per-decision cost that keeps falling, both conditional and not guaranteed.](/images/handbook/ch04-ships-now-demo-forecast.svg)

*Figure 5.3. Much of the stack ships now; long, lightly supervised autonomy is still a lab demo or a forecast.*

| Capability | Status | What the evidence supports |
|---|---|---|
| Extracting facts and signals from calls, emails, tickets, documents at customer-base scale | **Ships now** | Routine in production; accuracy depends on schemas and checks (Chapter 8) |
| First-pass account research and briefs for every account | **Ships now** | Minutes and cents to dollars per account; quality depends on source reconciliation |
| Individually written messages at human quality | **Ships now** | Fluency is solved for most business writing; factual accuracy is not (see limits) |
| Conversational personalization in support, onboarding, and web | **Ships now** | Deployed widely; continuity depends on shared memory most companies lack |
| Agents running bounded multi-step tasks for hours | **Ships now, bounded** | Measured on software tasks; business-process reliability varies by design; in my systems, bounded by role-scoped tools, a one-job credential, and a time cap |
| Agents managing a customer relationship for weeks with little supervision | **Lab demo / early production** | Possible with durable memory and checkpoints (Chapter 13); not common practice |
| Interfaces and offers composed per person at request time | **Lab demo / early production** | Per-account pages generated before the visit, inside declared zones, run in production in my engine; composing at request time for anonymous visitors is pilot design; measured lift at scale is thin (Chapter 16; Closing chapter) |
| AI improving its own rules and memory | **Early production, propose-only** | Ships as curation that proposes until an operator promotes it and as AI-assisted cohort review; AI implementing and validating its own changes is design direction |
| Agents reliably completing day-long business tasks at high success rates | **Forecast** | Inference from the METR trend, if it holds and transfers beyond software; neither is guaranteed |
| Per-decision cost continuing to fall | **Forecast, conditional** | Per-token prices likely keep falling; per-decision cost falls only if architecture controls tokens |

Adoption is moving too. The 2026 AI Index reports that generative AI reached 53% population adoption within three years, faster than the personal computer or the internet ([Stanford HAI, 2026-04-13](https://hai.stanford.edu/news/inside-the-ai-index-12-takeaways-from-the-2026-report)). That is observed adoption of the tools, not evidence that companies are personalizing well with them.

## What this does not do

A chapter about new capability is most tempting to oversell, so here are the limits, stated plainly.

**Fluency is not truth.** Models trained through prediction can infer something useful about a customer and confidently invent something false. They can produce a strong analysis and then accept a bad premise, or construct an explanation that sounds complete because sounding complete is part of what they learned. Research on why models hallucinate argues that standard training and evaluation reward guessing over admitting uncertainty ([Kalai et al., 2025](https://arxiv.org/abs/2509.04664)). On Vectara's grounded summarization benchmark, where the model only has to summarize a document it was given, hallucination rates ranged from about 1.8% for the best models to over 24% for the worst ([Vectara leaderboard](https://github.com/vectara/hallucination-leaderboard), snapshot 2026-09-22). Open-ended personalization is harder than summarizing a supplied document. At a million messages, a 2% error rate is 20,000 messages that say something false to a customer about their own business.

**Lab accuracy is not production accuracy.** Benchmarks use clean inputs. Your inputs include duplicate records, stale titles, and two people with the same name at the same company. A single mistaken identity can contaminate every later step, because the system treats its own earlier conclusion as context. Autonomy compounds both useful work and error.

**The persuasion studies are weaker than the headlines.** Matz and colleagues reported that messages generated to fit a person's inferred psychology were more influential ([Matz et al., 2024](https://www.nature.com/articles/s41598-024-53755-0)), and that finding's robustness has been publicly disputed ([PNAS commentary](https://www.pnas.org/doi/10.1073/pnas.2418817121)). The Salvi correction above narrows what can be attributed to personalization itself. The honest claim is that personalization shifts odds under some conditions, not that it commands outcomes.

**Cheap understanding does not mean you should use all of it.** More detail can make a message worse when it is irrelevant, stale, or included to prove the sender found it. The capability to infer is not permission to say what was inferred (Chapter 19).

**Nobody knows the slope in 2029.** Anyone presenting the time-horizon trend as a schedule is selling something. I plan for continued improvement and design so the system still works if it stalls.

## At scale

Across 40,000 accounts and 250,000 contacts, three things change that a single brief hides.

Errors stop being anecdotes and become rates. One wrong detail in Priya's brief she catches on the call. The same error rate across every account is a population of wrong sentences, measured as a population (Chapter 20).

Research multiplies unless it is shared. Five teams running five agents on one customer buy the same understanding five times and reach five conclusions. Write it once to a memory everyone reads (Chapter 10).

The bottleneck moves to supervision. When research is cheap, the scarce resource is human attention for review and exceptions. Design where it goes, or it goes everywhere and nowhere.

## Failure story: The Fluent Guess

A composite of a pattern I see often: a team at a company much like Larkspur switched on agent-written renewal briefs for every account. The briefs were excellent to read. One of them told an account manager that the customer was "expanding into Canada," based on a job posting for a remote role that listed Toronto among several cities. The manager opened the call with Canadian pricing. The customer had no such plan and asked, reasonably, where that had come from.

Nothing in the system distinguished what the account had said from what the agent had inferred. **The Fluent Guess** is the anti-pattern: inference presented with the confidence of evidence. The fix is not a better model. It is a system that records where each claim came from and how strong it is, and lets that strength decide how the claim may be used.

## Patterns

**1. Evidence-Weighted Voice.**
*Problem:* outputs state weak inferences as confidently as verified facts. *Forces:* readers trust fluent prose; inferences are often the most valuable content. *Solution:* attach source and confidence to every claim before generation; state only corroborated facts as fact; phrase inferences as hypotheses; fall back to honestly general when evidence is thin. *Tradeoffs:* fewer striking sentences; more work at extraction.

**2. Research Once per Entity.**
*Problem:* per-record pipelines re-research the same company or person repeatedly. *Forces:* the record is the natural unit of work; the entity is the natural unit of knowledge. *Solution:* key research to the entity, cache it with a freshness window, and have downstream steps read the cache. *Tradeoffs:* needs identity resolution first (Chapter 9) and an invalidation policy (Chapter 11).

**3. Human at the Boundary.**
*Problem:* either every step waits for a person, or no step does. *Forces:* agents handle the repeatable middle well and consequential edges poorly. *Solution:* automate the middle of the loop; route to a person on low evidence, high consequence, or irreversible action. *Tradeoffs:* requires explicit thresholds and a review queue people can actually use.

## Leader questions

1. For which decisions would a three-hour human brief change the outcome, and how many of those decisions do we make per year?
2. What is our cost per decision today, not our price per token, and who owns that number?
3. When our AI states something about a customer, can we show where it came from?
4. Which of our AI claims to the board are ships-now, which are demos, and which are forecasts?
5. Where does human attention go in our current AI workflows, and is it on the consequential cases?

## Build checklist

- [ ] Inventory unstructured sources (calls, emails, tickets, documents) and who can read them.
- [ ] Run one account-research pipeline on 50 accounts and have a person grade the briefs.
- [ ] Record source and confidence for every extracted claim.
- [ ] Measure tokens and cost per decision, not per call.
- [ ] Key research to the entity with a freshness window.
- [ ] Define the evidence threshold below which output becomes honestly general.
- [ ] Define which actions always return to a person.
- [ ] Give every agent a role-scoped, expiring credential and a maximum run time.

## Metrics to watch

- **Cost per decision** (all model, tool, and data costs divided by decisions made).
- **Unsupported-claim rate** in a graded sample of outputs.
- **Research reuse rate** (share of research reads served from existing entity research).
- **Human touches per hundred decisions**, split by reason.

## Reader Q&A

**Can AI really understand a customer?**
It can build a useful working account of one from what you give it. Whether that is understanding philosophically does not matter operationally. What matters is whether its claims are true and traceable.

**Does it write as well as our best people?**
For fluency and tone, often yes. For knowing what not to say, and for being right about the customer, only when the system around it supplies accurate memory and checks.

**Should we wait for the next model?**
Waiting for a better model is waiting for the part everyone will rent. The memory, decisions, and governance you build now carry over to every model you use later.

## For your AI

```yaml
chapter: 5
title: "The New Era"
concepts:
  - name: Capability Stack
    definition: "Six layers AI now performs for personalization: read, reason, decide, write, act, learn; each matures and fails differently."
  - name: Cost of Understanding
    definition: "The cost to form a usable, specific account of one customer; historically a skilled human hour, now a model run whose cost is set by architecture."
  - name: Ships now / Lab demo / Forecast
    definition: "Labeling discipline for AI capability claims: deployed in production, demonstrated but not common, or projected from trends."
  - name: Letter to Conversation
    definition: "Personalization shifts from a static message judged once to an interactive exchange judged at every turn."
  - name: Time Horizon
    definition: "Length of task (in skilled-human time) an AI completes at a given success rate; METR measures it mainly on software tasks."
  - name: Cost per Decision
    definition: "Total model, tool, and data cost divided by personalized decisions made; the unit to manage instead of price per token."
decision_rules:
  - if: "a claim about a customer is an inference without corroboration"
    then: "phrase it as a hypothesis or question, or fall back to an honestly general statement"
  - if: "a capability is cited in planning"
    then: "label it ships-now, lab demo, or forecast, with a date"
  - if: "research about an entity already exists within its freshness window"
    then: "reuse it; do not re-research per record"
  - if: "an action is irreversible or evidence is below threshold"
    then: "route to a person"
  - if: "the AI bill rises while per-token prices fall"
    then: "measure tokens per decision and find repeated or oversized reads"
  - if: "an agent is granted tools to act"
    then: "limit them to its role and job, give the credential an expiry and the run a time cap, and withhold destructive operations"
  - if: "a model score and its label or band disagree"
    then: "flag the record in code before anything is sent"
assessment_questions:
  - "Which unstructured sources about customers exist, and does any system read them today?"
  - "What is the current cost per personalized decision, and who measures it?"
  - "Can the team trace any AI-stated customer fact to its source?"
  - "Which AI workflows run unattended, and what returns them to a person?"
  - "How many accounts receive human-depth research today, out of how many?"
patterns: [Evidence-Weighted Voice, Research Once per Entity, Human at the Boundary]
anti_patterns: [The Fluent Guess, Per-Token Illusion, Horizon-as-Schedule]
maturity_dimension: understanding
as_of: 2026-09-26
```

## References

1. Brynjolfsson, E., Li, D., Raymond, L. (2025). "Generative AI at Work." *Quarterly Journal of Economics* 140(2). https://academic.oup.com/qje/article/140/2/889/7990658
2. Jones, C. R., Bergen, B. K. (2025). "Large Language Models Pass the Turing Test." arXiv:2503.23674 (preprint). https://arxiv.org/abs/2503.23674
3. Hartmann, J., Exner, Y., Domdey, S. (2025). "The power of generative marketing." *International Journal of Research in Marketing* 42(1). https://www.sciencedirect.com/science/article/pii/S0167811624000843
4. Salvi, F., Horta Ribeiro, M., Gallotti, R., West, R. (2025). "On the conversational persuasiveness of GPT-4." *Nature Human Behaviour* 9:1645-1653. https://pmc.ncbi.nlm.nih.gov/articles/PMC12367540/ ; Author Correction, 2026-09-03: https://www.nature.com/articles/s41562-026-02588-0
5. Stanford HAI (2025). *AI Index Report 2025*. https://hai.stanford.edu/ai-index/2025-ai-index-report
6. Epoch AI (2025-03-12). "LLM inference prices have fallen rapidly but unequally across tasks." https://epoch.ai/data-insights/llm-inference-price-trends
7. Anthropic (2025-06). "How we built our multi-agent research system." https://www.anthropic.com/engineering/multi-agent-research-system
8. Appenzeller, G. (2024-11). "Welcome to LLMflation." a16z. https://a16z.com/llmflation-llm-inference-cost/
9. METR (2025-03-19). "Measuring AI Ability to Complete Long Tasks." https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/
10. METR (2026-01-29). "Time Horizon 1.1." https://metr.org/blog/2026-1-29-time-horizon-1-1/ ; tracking page (updated 2026-05-08): https://metr.org/time-horizons/
11. METR (2025-07-10). "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity." https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
12. Stanford HAI (2026-04-13). "12 Takeaways from the 2026 AI Index." https://hai.stanford.edu/news/inside-the-ai-index-12-takeaways-from-the-2026-report
13. Kalai, A. T., Nachum, O., Vempala, S., Zhang, E. (2025). "Why Language Models Hallucinate." arXiv:2509.04664. https://arxiv.org/abs/2509.04664
14. Vectara. Hallucination Leaderboard (HHEM), snapshot 2026-09-22. https://github.com/vectara/hallucination-leaderboard
15. Chen, Y., Benton, J., et al. (2025). "Reasoning Models Don't Always Say What They Think." arXiv:2505.05410. https://arxiv.org/abs/2505.05410
16. Matz, S. C., et al. (2024). "The potential of generative AI for personalized persuasion at scale." *Scientific Reports* 14:4692. https://www.nature.com/articles/s41598-024-53755-0 ; critique: https://www.pnas.org/doi/10.1073/pnas.2418817121
17. H. Taheri, first-party system documentation: capabilities reference for a governed personalization engine (2026-08-25; only capabilities marked live are described as built) and release documentation for a self-hosted governed memory system (release 0.8.2, verified 2026-09-16); unpublished. [PRODUCT-NAMING: name the systems here if Hamed decides to]

---

# 6. Choosing Your First Decisions

*The question: Where is the highest-ROI personalization in my company?*

> **Questions this chapter answers**
> - We have forty AI personalization ideas. How do we pick three?
> - How much is a personalization idea actually worth before we build it?
> - Which channel should go first, and where does generated content need a human to approve it?
> - What should we deliberately not do yet?

## The short answer

Do not start from use cases. Start from decisions. A decision is something your company chooses again and again for one person at a time, with real options (including doing nothing) and an outcome you can measure: which at-risk account gets which kind of help, what a new inbound lead hears first, whether a customer sees an expansion offer at all.

Price each decision before you build anything. Value at stake is the number of times the decision is made in a year, times the value of making it better once, times the lift you can realistically achieve over what you do today, minus the cost of running it and the risk of getting it wrong. Every term is a range, not a point. Whatever range you write down, rank on its low end, because a decision that only pays at the top of its range is a bet, not a first decision.

Then score the survivors on three things value hides: whether the data exists, whether a wrong decision can be undone, and how long it takes to learn whether it worked. Keep those scores separate. Collapsing them into one "feasibility" number is how companies end up funding whatever is easiest.

Choose three: a fast win, with ready data and evidence in weeks; a memory-building bet, high in value, whose preparation builds the customer memory every later decision reuses; and a measurement foundation, the holdouts and decision logs that prove any of it. Human-assisted channels go first for high-value decisions. Cold, paid, and irreversible channels wait.

The most common mistake is not choosing the wrong technology. It is personalizing the touchpoint that was easiest to automate, which is usually the one that mattered least.

---

Larkspur Systems (a fictional company, a composite of many real ones) holds its leadership offsite in a hotel meeting room with a whiteboard that fills by eleven. Forty sticky notes, each labeled "AI personalization." AI-written newsletter intros. A website hero that changes by industry. An AI SDR for every prospect in the market. Renewal-risk alerts. A support bot that knows the account. Birthday emails for fleet managers. Personalized event agendas. A pricing page that adapts.

The CMO wants the newsletter: it goes to 250,000 contacts, the tooling exists, it could ship in two weeks. The CRO wants the AI SDR expanded, even though the pilot is making reply rates worse, because the vendor says the next version fixes it. The head of customer success says nothing until the end, then points at one note: renewal risk. "I have six people rebuilding account history by hand before every save call."

Nobody is wrong about their own note. The room simply has no way to compare them.

## A use case is a wish; a decision is a unit you can price

"Personalized website" cannot be estimated, funded, or tested. It is a wish. Hidden inside it are several decisions: which of three proof points a returning visitor from a renewing account sees; whether a visitor from an open support escalation sees a sales banner at all; which case study a prospect from a fleet of 500 trucks gets instead of a fleet of five.

A decision qualifies for this chapter when it has three properties:

1. **It repeats.** It is made hundreds or thousands of times a year, each time for one person or account. A one-off strategic choice is not a personalization decision.
2. **It has options.** At least two things could happen, and one of them is usually "nothing." If there is only one possible action, there is nothing to personalize.
3. **It has a measurable outcome.** Renewed or churned, replied or not, resolved in one contact or three. If you cannot name the outcome, you cannot know whether personalizing it helped.

![A use case such as "personalized website" is a wish: it cannot be estimated, funded, or tested. Hidden inside it are several decisions, such as which of three proof points a returning visitor from a renewing account sees, whether a visitor from an open support escalation sees a sales banner at all, and which case study a fleet of 500 trucks gets instead of a fleet of five. A decision qualifies only when it repeats (hundreds or thousands of times a year), has options (at least two, usually including nothing), and has a measurable outcome (renewed or churned, replied or not).](/images/handbook/ch05-use-case-to-decisions.svg)

*Figure 6.1. Break every use case into the repeated decisions hiding inside it; only those can be priced and tested.*

Name the objective each decision serves, for which person: acquire, convert, expand, retain, reactivate, serve, or reduce the cost to serve. "Increase engagement" is not on the list; engagement is a measurement, not an outcome anyone pays for.

The **decision inventory** is a table with one row per decision: the person, the moment, the options, the outcome, the annual volume, and the owner who would act on a better answer. Larkspur's forty sticky notes collapsed into eleven rows once duplicates merged and wishes were translated; three had no outcome anyone could name and were dropped. Writing the inventory is most of the work: it forces each idea to say what would be chosen differently, for whom, and how you would know.

Write the options with their depth, not only "something or nothing." In the governed personalization engine I built, each campaign declares its personalization mode before anything runs: company-level only, personalized to the named person, or a score and brief for the rep with no customer-facing copy at all. Quality review then judges each record against the declared mode, so a company-level campaign is never faulted for not naming the individual. The row should also say what happens when the evidence for one record is thin: fall back to the account, then the segment, then approved static copy (Chapter 20). A decision whose only options are "fully personalized" and "nothing" fails on every record the data cannot support.

## Value at stake is volume times the value of changing the decision

Every growth team already runs a version of this arithmetic. You estimate what a good outcome is worth. You estimate how likely a particular approach is to produce it. You multiply, subtract what the approach costs to run, and spend where the number is highest, stopping where it turns negative. The personalization version adds one discipline: price the change, not the customer.

> **Value at stake** = annual decision volume × value per improved decision × achievable lift, minus cost to run, minus risk.

Three of those terms are routinely misread.

**Value per improved decision is not the value of the customer.** It is the value of the outcome that changes because the decision changed. A retention offer to a customer who would have renewed anyway is worth nothing, however large the account. Eva Ascarza's field experiments showed that targeting the customers at highest risk of churning can be ineffective, and that the right target is the customers whose behavior the intervention actually changes [1]. Uplift modelers have a name for the worst case, "sleeping dogs": customers a retention contact pushes toward leaving [2]. So the value term is about the margin. It is not "who is this account worth," but "what is this decision worth when we get it right instead of wrong."

**Lift is measured against what you do today, not against nothing.** Larkspur already sends every at-risk account a generic check-in. A personalized save play has to beat that, not beat silence. And lift should be written in absolute points. A 20% relative lift on a 2% click rate is 0.4 points, which sounds very different in a budget meeting.

**Achievable lift is smaller than hoped.** Ron Kohavi, who ran experimentation at Microsoft, reported that only about 10 to 20% of ideas tested at Bing improved the metric they targeted, and that most winning changes moved it by 0.1% to 1% [3]. Personalization ideas are ideas. Plan on the low end.

Here is Larkspur's inventory, priced. Every number is illustrative for a fictional company; the ranges and the reasoning are the point.

| Decision | Annual volume | Value per improved decision | Achievable lift (absolute) | Gross value at stake |
|---|---|---|---|---|
| Renewal save: what each at-risk account gets (CSM call, tailored plan, or nothing) | 6,000 at-risk accounts | $6,000 to $15,000 (one retained contract) | 1 to 4 points | $360K to $3.6M |
| Inbound response: who answers a demo request, how fast, with what | 3,000 requests | $6,000 to $15,000 (first-year contract) | 1 to 3 points of close rate | $180K to $1.35M |
| Expansion: whether and how to offer the fleet-tracking add-on | 40,000 accounts | $1,500 to $4,000 per year | 0.2 to 1 point | $120K to $1.6M |
| Support context: next step on a ticket, with account history | 60,000 tickets | $5 to $20 in handling cost | 10 to 30% of tickets handled differently | $30K to $360K |
| AI SDR first touch to new prospects | 100,000 emails | $300 to $1,500 per meeting | 0.1 to 0.5 points (could be negative) | $30K to $750K, before reputation risk |
| Newsletter intros, per contact | 1,000,000 sends | $0.50 to $5 per extra click | 0.1 to 0.5 points | $500 to $25K |

Two things jump out. The ranges are wide, often tenfold, and that is honest: the lift assumption carries most of the weight, and a defensible model shows it instead of hiding it inside one number. And the newsletter, the easiest idea on the whiteboard, sits one to three orders of magnitude below everything else. At a fraction of a cent to two cents of generation per contact (an assumption), its cost could exceed its value.

![Log-scale range chart of annual gross value at stake for six decisions at Larkspur, a fictional company, with illustrative numbers. Ranked by the low end of each range: renewal save $360K to $3.6M, inbound response $180K to $1.35M, expansion offer $120K to $1.6M, support context $30K to $360K, AI SDR first touch $30K to $750K before reputation risk, and newsletter intros $500 to $25K. Each range is volume times value per improved decision times achievable lift, and the newsletter, the easiest idea, sits one to three orders of magnitude below the rest.](/images/handbook/ch05-value-at-stake-ranges.svg)

*Figure 6.2. Ranked on the low end of each range, the easiest idea to automate is worth the least.*

Here is the sentence this chapter turns on: **rank decisions by the value of changing them, not by the value of the customer and not by the ease of automating them.**

## Four scores, kept separate

Value decides what deserves attention. Three other properties decide what goes first.

**Data readiness.** Can you make this decision well with what you have today? Larkspur's renewal decision needs support history, usage telemetry, and call notes, which exist in three unjoined systems. The inbound decision needs a form, a web session, and a CRM record, all ready. Readiness is not a reason to skip a high-value decision; it tells you how much memory work sits between you and the first result.

**Reversibility.** What happens when the decision is wrong? An in-app suggestion can be withdrawn tomorrow. An email cannot be unsent. A quoted price can become an obligation. A cold email that lands in spam damages a sending domain every other team relies on. The less reversible the action, the more governance it needs before it runs unattended (Chapter 18), and the worse it is as a first decision.

**Time to evidence.** How long until you can tell whether it worked? This is arithmetic, not intuition. To detect a click rate moving from 2.0% to 2.4% with a standard two-sided test (5% significance, 80% power) you need about 21,000 people in each arm. The newsletter has that in one send. To detect a renewal rate moving from 85% to 88% you need about 2,000 accounts per arm, and the outcome arrives only at the renewal date, months after the decision. Chapter 21 covers the methods; for choosing, the point is that measurability is a cost you pay, not a value you get.

One kind of evidence does not wait for a test. Whether the output is correct and appropriate shows up in the first few dozen real records. In the engine I built, an operator reviews the first cohorts of a new campaign (the first dozens, then the first hundred, then the first few hundred), with AI reasoning across the whole population for repeated errors, weak outputs, and fallbacks that fire too often (Chapter 22). Plan that review for every first decision, and do not mistake it for lift.

Here I part ways, partly, with the most quoted enterprise AI report of 2025. MIT's Project NANDA found that about half of generative AI budgets in its sample went to sales and marketing while the clearest returns came from the back office, and said the allocation "reflects easier metric attribution, not actual value" [4]. The diagnosis is useful, within the report's own limits: 52 interviews, 153 surveyed leaders, and a six-month window it concedes may understate success [4]. The lesson I take is not "leave the front office." It is that when time to evidence stands in for value, the easy-to-measure decision always wins the budget.

The same objection applies to value-versus-feasibility grids such as Gartner's "Prism" [5]. "Feasibility" blends readiness, reversibility, and time to evidence into one axis, so a decision that is ready, quick to measure, and irreversible scores as feasible while being the most dangerous thing on the board.

## The channel sets how much the machine may say on its own

The channel decides how much generated content can reach a person without review.

- **Human-assisted** (a rep, CSM, or support agent receives a researched brief and a draft, and decides what to send): the highest tolerance for generation, because a person who knows the account reads it first. High-value, low-readiness decisions start here, and every human edit is evidence of what the system got wrong.
- **Owned and automated** (email, in-app, website, support replies): generation is fine inside typed zones with grounding checks (Chapter 15). Anything that states a price, a contract term, a commitment, or a regulated claim should require approval or come from a fixed template. So should any statement about the customer's own relationship with you: "your renewal is coming up" or "you already use the fleet module" appears only when memory confirms it; otherwise the copy speaks of an opportunity. In the engine I built, that rule lives in the guidelines and in a code check behind them (Chapter 20).
- **Paid** (ads and sponsored placements): you control the creative; the platform controls much of the targeting and delivery, under its own policies. Per-person claims have little room here; it is usually a later step.
- **Cold outbound to people with no relationship**: the least tolerance of all. The recipient has no context for why you know what you know, and the reputation cost is shared by the whole company.

That is why Larkspur's AI SDR pilot is hurting reply rates. The writing is not the problem. The most skeptical channel was chosen first, for the decision with the least memory behind it.

## The first three: a fast win, a memory-building bet, a measurement foundation

One first decision is a single point of failure. Forty is a portfolio nobody can staff. Three works, if each plays a different role.

**The fast win** has ready data, reversible actions, and evidence within weeks. At Larkspur it is inbound response: an agent researches each demo request's account and drafts a brief and first reply; a rep approves and sends. Measurable in a quarter, and visible to the sales team. Known identity is most of what makes it ready. In the engine I built, the flow that runs in production is the known-lead flow: a form or a source says who the person is, and one research pass feeds a score, a brief for the rep, a personalized page, and an email sequence. Personalizing to an anonymous visitor before the form is still pilot design there. Start where identity is handed to you.

**The memory-building bet** has high value and low readiness, and the work of making it ready builds something reusable. At Larkspur it is renewal save. Joining support history, telemetry, and call notes into one account record is slow, but once it exists, expansion, support context, and every later playbook read from the same memory (Chapter 10). The first decision pays for infrastructure the next five use for free. The CSMs send; the system prepares.

**The measurement foundation** is less a decision than the plumbing that makes every decision provable: a randomized holdout on each new decision, a small permanent holdout across the whole program, and a log of what was known, decided, and done for each person. If a decision must carry it, pair it with expansion, where 40,000 accounts give enough volume to learn within a quarter and "no offer" is a real option to test.

What Larkspur chose not to do yet matters as much. The AI SDR pilot paused until account memory exists and the decision can be made for fewer, better-researched prospects. The newsletter kept its merge fields. The adaptive pricing page stayed on the whiteboard: high stakes, irreversible, and measurable only slowly.

![Larkspur's (fictional) first three decisions each play a different role. The fast win is inbound response: ready data, reversible actions, evidence within weeks, and a rep approves and sends. The memory-building bet is renewal save: high value and low readiness, joining three systems into one account record that later decisions read. The measurement foundation is holdouts and logs: a holdout on each new decision, a permanent program holdout, and a log of what was known, decided, and done, run on expansion's 40,000 accounts. A written not-yet list holds the AI SDR pilot (paused until account memory exists), newsletter intros (kept their merge fields), and the adaptive pricing page (high stakes, irreversible, slow to measure).](/images/handbook/ch05-first-three-portfolio.svg)

*Figure 6.3. Three roles, not three favorites: a fast win, a memory-building bet, a measurement foundation, and a written "not yet" list.*

Choosing three is not the same as saying the other thirty-seven are bad. It says they are later.

## What this does not do

The value-at-stake calculation is a way to rank, not a forecast. It is only as good as the lift assumption, and at the start nobody has a measured lift for their own decisions. Treat it as a disciplined argument about which assumption matters, and replace it with measured results as soon as you have any.

Do not import industry averages as your lift. McKinsey's widely quoted finding that personalization "most often drives" a 10 to 15% revenue lift, with a company range of 5 to 25%, describes company-level results in its research, not the lift of any single decision [6]. Gartner's 2019 prediction that 80% of marketers would abandon personalization by 2025 was a prediction, cited far more often than it was ever checked [7]. Neither belongs in your spreadsheet.

Personalization is not a fix for a bad default. If the generic onboarding is broken for everyone, fixing it for everyone will usually beat personalizing it for some. The inventory sometimes reveals that the highest-value decision is not a personalization decision at all.

And some decisions should not be made by a system at all, whatever they score (Chapter 19).

## At scale

With one team, the inventory is a spreadsheet. With seven, it is a contested asset: marketing, sales, success, and support all want the same person's attention in the same week, so the inventory must record which decisions compete, and contact policy becomes a decision in its own right (Chapter 14 and the *One Customer, One Conversation* playbook).

Value per decision also falls as you work down the list, while the cost to govern each one does not. What keeps a program alive at scale is not better lift per decision but lower marginal cost per decision, from shared memory, governance, and measurement. That is why the memory-building bet belongs in the first three, not the fifth.

## Failure story: The Streetlight Pilot

A composite of several companies. The team personalized the newsletter because it was easiest to automate: a contact list, an owned channel, a vendor who could ship in a sprint. Open rates rose, and the dashboard celebrated, though some of that rise was not people at all: since 2021, Apple's Mail Privacy Protection preloads content for users who enable it, which registers as an open whether or not anyone reads the email [8]. Clicks moved slightly. Pipeline did not. Meanwhile six CSMs kept rebuilding account history by hand before every save call, on a decision worth far more.

Nothing failed technically. The pilot looked under the streetlight because that is where the light was.

## Patterns

**Decision Inventory.** *Problem:* ideas cannot be compared, because each team frames its own as a use case. *Solution:* translate each idea into repeated decisions with person, moment, options (including nothing), outcome, volume, and owner; drop anything without a nameable outcome. *Trade-off:* it slows the first meeting down, and it kills some favorite ideas.

**Low-End Ranking.** *Problem:* value estimates are wide, optimistic, and politically loaded. *Solution:* estimate every term as a range, rank on the low end, and show which assumption carries the weight. *Trade-off:* genuinely high-upside bets rank lower; take them deliberately, as bets.

**First-Three Portfolio.** *Problem:* one pilot is fragile; many are unstaffable. *Forces:* leaders need a visible result, infrastructure needs time, and nothing counts without proof. *Solution:* one fast win, one memory-building bet, one measurement foundation. *Trade-off:* the bet shows little for a quarter and needs a sponsor willing to wait.

## Leader questions

1. For each idea we fund, what decision changes, for whom, and how many times a year?
2. What lift over today's treatment are we assuming, and is that the assumption carrying the estimate?
3. If this decision is wrong, can we undo it, and how many weeks until we know it worked?
4. Which of our first three builds something the next five decisions will reuse?

## Build checklist

- [ ] Every idea translated into a decision with person, moment, options (including nothing), outcome, volume, owner.
- [ ] Personalization depth declared per decision, with the fallback when a record's evidence is thin.
- [ ] Value at stake estimated as ranges, with lift in absolute points over today's treatment.
- [ ] Cost per decision estimated: compute, human review minutes, governance.
- [ ] Readiness, reversibility, and time to evidence scored separately, never merged.
- [ ] Channel chosen by generation tolerance; approval rules written for prices, terms, and commitments.
- [ ] Sample size, holdout, and decision log designed before the first send.
- [ ] A written "not yet" list, with the reason for each item.

## Metrics to watch

- **Measured lift per decision** against a holdout, replacing the assumed range as soon as data exists.
- **Cost per decision**, including human review time.
- **Time to evidence**: weeks from launch to a readable result, per decision.
- **Reuse**: how many decisions read from the memory the first bet built.

## Reader Q&A

**We are a small team. Is three too many?** Then keep the roles and shrink the scope. A fast win and a measurement foundation can be the same decision if it has volume. Do not drop measurement to make room.

**Should the first decision be customer-facing?** Not necessarily. A human-assisted decision, where the system prepares and a person acts, often has the best ratio of value to risk.

**What if the highest-value decision has the worst data?** That is the memory-building bet. Start it now, in a human-assisted channel, and let the fast win carry the visible results while the memory fills.

## For your AI

```yaml
chapter: 6
concepts:
  - name: Decision
    definition: "A choice made repeatedly for one person or account, with at least two options (usually including doing nothing) and a measurable outcome."
  - name: Decision Inventory
    definition: "One row per decision: person, moment, options, outcome, annual volume, owner, objective served."
  - name: Value at Stake
    definition: "Annual volume x value per improved decision x achievable lift over current treatment, minus cost to run, minus risk; every term a range."
  - name: Value per Improved Decision
    definition: "The value of the outcome that changes because the decision changed; not the value of the customer."
  - name: Time to Evidence
    definition: "Time until a test can detect the target lift, set by sample size per arm and outcome delay."
  - name: First-Three Portfolio
    definition: "One fast win, one memory-building bet, one measurement foundation."
  - name: Personalization Mode
    definition: "The declared depth of a decision (company-level, person-level, or rep-only brief and score), against which each record is judged."
decision_rules:
  - if: "an idea has no nameable, measurable outcome"
    then: "drop it from the inventory until one is named"
  - if: "a decision is attractive only at the top of its value range"
    then: "treat it as a deliberate bet, not a first decision"
  - if: "lift is stated in relative terms"
    then: "convert to absolute points over current treatment before ranking"
  - if: "an action is irreversible or states prices, terms, or commitments"
    then: "require approval or a fixed template; do not choose it as a first unattended decision"
  - if: "a high-value decision has low data readiness"
    then: "run it human-assisted as the memory-building bet"
  - if: "a record's evidence is too thin for the declared depth"
    then: "fall back to account, then segment, then approved static copy; do not guess"
  - if: "copy states the customer's relationship with you (renewal, product owned, contract)"
    then: "allow it only when memory confirms it; otherwise frame it as an opportunity"
  - if: "the generic experience is broken for everyone"
    then: "fix the default before personalizing it"
assessment_questions:
  - "List the repeated decisions you make about individual customers. Which include doing nothing as an option?"
  - "For your top three, what is annual volume, what happens today, and what absolute lift would matter?"
  - "Which decisions produce actions you cannot undo?"
  - "How many people per test arm could you assign to each decision in a quarter, and when does the outcome arrive?"
  - "Which data sources would the highest-value decision need, and are they joined today?"
patterns: [Decision Inventory, Low-End Ranking, First-Three Portfolio]
anti_patterns: [The Streetlight Pilot, Feasibility Blending, Customer Value Mistaken for Decision Value]
maturity_dimension: decisioning
```

## References

1. Ascarza, E. (2018). "Retention Futility: Targeting High-Risk Customers Might Be Ineffective." *Journal of Marketing Research* 55(1). https://journals.sagepub.com/doi/10.1509/jmr.16.0163
2. Radcliffe, N. J., and Simpson, R. (2008). "Identifying who can be saved and who will be driven away by retention activity." *Journal of Telecommunications Management* 1(2). https://stochasticsolutions.com/pdf/SavedAndDrivenAway.pdf
3. Kohavi, R., et al. (2014). "Seven Rules of Thumb for Web Site Experimenters," KDD 2014 (slides). https://exp-platform.com/Documents/2014-08-27ExperimentersRulesOfthumbKDD.pdf ; see also Kohavi, R., Tang, D., and Xu, Y. (2020). *Trustworthy Online Controlled Experiments.* Cambridge University Press.
4. Challapally, A., et al. (2025). *The GenAI Divide: State of AI in Business 2025.* MIT Project NANDA. Copy read: https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Business_2025_Report.pdf
5. Gartner. "Toolkit: Discover and Prioritize Your Best AI Use Cases With a Gartner Prism." https://www.gartner.com/en/documents/4000998
6. Arora, N., et al. (2021-11-12). "The value of getting personalization right, or wrong, is multiplying." McKinsey & Company. https://www.mckinsey.com/capabilities/growth-marketing-and-sales/our-insights/the-value-of-getting-personalization-right-or-wrong-is-multiplying
7. Gartner (2019-12-02). "Gartner Predicts 80% of Marketers Will Abandon Personalization Efforts by 2025." https://www.gartner.com/en/newsroom/press-releases/2019-12-02-gartner-predicts-80--of-marketers-will-abandon-person
8. Litmus. "Apple's Mail Privacy Protection Is Here: What It Actually Means for Email Marketers." https://www.litmus.com/blog/apple-mail-privacy-protection-for-marketers

---

# 7. Getting the Data

*The question: What data can we use, where does it come from, and are we allowed to use it?*

> **Questions this chapter answers**
> - First, second, third party, public, inferred: what is each one actually worth, and what does each one cost us in risk?
> - When our systems disagree about a customer, which one wins?
> - Is buying enrichment data still a good idea, and how accurate is it?
> - What happened to third-party cookies and mobile tracking, and should we care?
> - How do we know we are allowed to use a fact for the thing we want to use it for?

## The short answer

Customer data arrives by six routes: what the person told you (declared), what you saw them do (observed), what a partner shared (second party), what you bought (third party), what is public, and what a model concluded (inferred). Each route makes a different kind of claim. Declared data is strongest about what a person wants. Observed data is strongest about what they did. Purchased data is a vendor's estimate, and you inherit the vendor's consent problems with it. Inferred data is a hypothesis, never a fact until confirmed.

The common mistake is not collecting too little. It is flattening all six routes into one column, so that a guess, a stale purchase, and a sentence the customer wrote last week look equally true to the system that reads them.

So carry the route with the value: source, time, confidence, and permitted purposes on every fact you personalize with. Then rank conflicts with a Signal Ladder: gate by purpose, prefer the source with authority over that kind of fact, let freshness override authority for things that change, and abstain when nothing clears the bar.

Build on what customers give you and what you observe in your own relationship. The web's tracking layer changes on other companies' schedules, and bought and inferred data are evidence to check, not truth to copy.

## Six answers to one question

Open one contact record at Larkspur Systems (a fictional company, a composite of many real ones): Maya Okafor, at a regional utility contractor, also fictional. Ask the record one question. What is her job?

- The CRM says **Fleet Manager**, typed by a sales rep in 2022.
- A webinar registration form says **Director of Operations**, which Maya selected from a dropdown in March 2025.
- Enrichment vendor A says **Operations Manager**, with no date on the value.
- Enrichment vendor B says **VP, Operations**.
- The signature on a support email three weeks ago says **Director, Field Operations**.
- An AI summary of her support threads, written by Larkspur's own pipeline, says she is **"likely the economic buyer for the renewal."**

Six sources, six answers, each true or believed at some moment. No typos, no duplicates. The record simply cannot say where each answer came from, how old it is, or which one Larkspur may quote back to her. The newsletter template reads the CRM field. Maya gets an email addressed to a Fleet Manager, a job she left two years ago.

Here is the sentence this chapter turns on: **where a value came from is part of the value.** Strip the source off and you do not have a cleaner fact. You have a rumor with a column name.

## Every signal arrives by a route, and the route decides what it can claim

The six routes are not a legal taxonomy. They describe what kind of claim each piece of data makes.

| Route | What it is | What it is good for | How it fails |
|---|---|---|---|
| **Declared** (zero party) | What the person deliberately told you: preferences, forms, stated goals | What they want; how to contact them | Goes stale; answers the question you asked |
| **Observed** (first party) | What they did with you: purchases, usage, tickets, replies | What happened, in context | Circumstance read as character |
| **Partner** (second party) | Another company's first-party data, shared under agreement | Gaps you cannot observe | Consent given to someone else, for something else |
| **Purchased** (third party) | Vendor-assembled records and enrichment | Coverage, firmographics, cold start | Unknown age and method; inherited consent risk |
| **Public** | Company sites, filings, announcements, profiles | Current account events | Public is not permitted; wrong-entity matches |
| **Inferred** (derived) | What a model or rule concluded | Choosing what to do | Fluent, confident, and sometimes about nobody |

![Six data routes shown as cards, each with what it is good for and how it fails. Declared data (zero party) is good for what the person wants and fails by going stale; observed data (first party) is good for what happened and fails when circumstance is read as character. These two are grouped as the routes to build on. Partner, purchased, public, and inferred data are grouped as enrichment and evidence to check: partner data fills gaps you cannot observe but carries someone else's consent, purchased data gives coverage and cold start with unknown age and method, public data shows current account events but public is not permitted, and inferred data helps choose what to do but is sometimes about nobody.](/images/handbook/ch06-six-routes.svg)

*Figure 7.1. Six routes, six kinds of claim: build on what customers declare and what you observe, and check everything else.*

Forrester's definition of zero-party data is "data that a customer intentionally and proactively shares with a brand" [1]. The definition is useful. I disagree with the way the category is often sold, as the clean answer to everything else. Intentional is not the same as current or complete. Maya's dropdown choice was intentional in March 2025. It was also the closest option on a list somebody else wrote.

### Declared data records what people say they want

When a person hands you something on purpose (a preference, a stated problem, a timeframe), they do it for a reason: a better answer, a faster service, a discount. When relevant, it is the strongest material you have, because the person chose it. It records what they say they want, not what can be worked out about them.

Its limits are real. A declared answer answers a question somebody asked, framed the way they framed it. And its purpose travels with it: disclosed for one purpose, it is not automatically available for every other.

### Observed data records what people do, in the situation they were in

Observed data moves the question from *what did they say they want* to *what do they repeatedly behave as though they want*, and the answers often differ. What people say answers a question. What they do responds to the situation they were standing in. That is also the limit: behavior is evidence of circumstances as much as of character. A customer who stopped using a reporting module may be unhappy, or may have hired an analyst who exports everything. The observation is a fact. The meaning is an inference, and should be stored as one.

Observation is also eroding at the edges you do not own. Since 2021, Apple's Mail Privacy Protection has loaded remote email content in the background and hidden the recipient's IP address, so senders cannot reliably tell whether a message was opened [2]. Observations from inside your own product, support desk, and billing system are the ones that hold.

### Inferred data is a claim about a person, not a fact about them

Inference is anything a system concludes that the person never told it: a statistical model lending someone what their group tends to do, or a language model reasoning about who wrote a set of emails. The methods differ in cost and in how often they are right. They produce the same thing: an understanding of a person with a confidence attached. How deep does it go? Three degrees, each a different kind of thing.

The first is estimating attributes nobody stated. In 2013, researchers predicted personal attributes from the Facebook Likes of about 58,000 volunteers, accurately for some and weakly for others [3]. A decade later, language models inferred private attributes from ordinary forum posts with up to 85% top-1 and 95% top-3 accuracy on the benchmark tested, at about 1/100th of the cost and 1/240th of the time of human labelers [4]. What changed was not mainly accuracy. It was price.

The second degree is explanation. Given a few true things, a capable model proposes a theory: what this person is responsible for, what is weighing on them this quarter, whose opinion they defer to. Larkspur's "likely the economic buyer" is this kind of claim.

The third degree is simulation. A 2024 Stanford-led study interviewed 1,052 people for two hours each and built an agent of each; on General Social Survey questions the agents matched participants' answers at about 85% of the consistency participants showed with themselves two weeks later [5]. That is the top of the data range: long, consented interviews. A CRM record and a few emails do not get anyone there.

Now the ceiling. Inference about personality, mood, beliefs, or intent is uncertain, carries demographic bias, and fails often, and fluency makes a weak inference sound like a finding. **Fluency is not validity.** So the rule is structural: inferred values live in their own labeled lane, used to choose what to do and never asserted to the customer until something first-hand confirms them.

## Bought data arrives with its supplier's consent attached

Purchased enrichment is still useful: firmographics you would never collect yourself, and a starting point for accounts you have never talked to. Two things about it deserve more attention than procurement usually gives them.

The first is accuracy. Nearly every enrichment accuracy comparison I could find was published by a company that sells data. As of September 2026 I have not found an independent, method-disclosed study of B2B enrichment accuracy that I would cite. Run your own bake-off instead: send each vendor a sample where you know the truth, and measure accuracy per field, not in aggregate.

Decay is real, but the usual number is not: "B2B data decays 22.5% a year" has no primary source I can trace. Job tenure is a defensible proxy: in January 2026, median tenure among US wage and salary workers was 4.1 years, and 20.6% had been with their employer a year or less [6]. Every job-linked field inherits that churn. A purchased title with no date is a guess about when it was true.

The second is consent. When you buy data, you inherit the question of how your supplier got it. In May 2026 the FTC settled with a Kochava subsidiary over location data; the order requires a supplier assessment program "designed to confirm that consumers have provided consent" [7]. Read it as a direction of travel: buyers are increasingly expected to know the provenance of what they buy. California came at it from the other end. Through the Delete Act's DROP platform, consumers could send one deletion request to every registered data broker from January 2026, and brokers had to start processing them on August 1, 2026, at least every 45 days, at $200 per day per unprocessed request [8]. Bought data about Californians now expires on a schedule. (As of September 2026; not legal advice.)

Enrichment is worth more as a cross-examiner than as a filler: **use it to challenge the data you hold, not only to add more of it.** Chapter 8 turns that into trust rules. In the personalization engine I built, the cross-examination runs on headcount and revenue: a provider value that is implausible, looks like a bucket artifact, or contradicts current research is hidden rather than used, because a missing number does less damage than a confident wrong one.

## The tracking layer shifted, and did not collapse

In January 2020 Google announced a path toward making third-party cookies obsolete in Chrome [9]. In July 2024 it said it would not deprecate them [10]; in April 2025 it dropped the planned cookie-choice prompt [11]; in October 2025 it retired most replacement technologies, including Topics, Protected Audience, and the Attribution Reporting API [12]. As of September 2026, Chrome still allows third-party cookies by default, while Safari and Firefox block them [13].

Cookies did not die. The replacement did.

On mobile, Apple's App Tracking Transparency has required apps since iOS 14.5 (April 2021) to ask permission before tracking users across other companies' apps and websites [14]. Reported opt-in rates range from single digits to around half, depending on the denominator [15].

The lesson is not about cookies. It is about dependency. Identifiers rented from a browser, an operating system, or an ad platform change on someone else's schedule. Build the model of the customer on what customers declare to you and what you observe in your own relationship, and treat everything else as enrichment.

## Provenance and consent are part of the data

A lawyer cares enormously how a fact was obtained. The model reading your customer record does not care at all. By the time a declared preference, a purchased title, and a model's guess reach a prompt, they are indistinguishable unless the record tells them apart. So provenance and consent belong inside the data, not in a separate system nobody queries at read time.

In practice, each fact you personalize with carries a small provenance envelope:

```yaml
property: job_title
value: "Director, Field Operations"
route: observed              # declared | observed | partner | purchased | public | inferred
source: support_email_signature
source_ref: ticket-88213
observed_at: 2026-09-02
method: extracted            # entered | selected | extracted | vendor | model
confidence: high
corroborated_by: [public_profile_2026-08]
permitted_purposes: [service, account_management, marketing]
allowed_surfaces: [internal, seller, email]
review_after: 2027-03-01
```

Three pairs do most of the work. `route` and `method` tell a fact from a guess. `observed_at` and `review_after` let freshness be computed rather than assumed (Chapter 11 builds the freshness policy on top). `permitted_purposes` and `allowed_surfaces` turn consent from a checkbox elsewhere into a filter applied before any fact reaches generation, so a confidential support note can shape a strategy without ever being quoted.

![A provenance envelope drawn around one value, job_title "Director, Field Operations", with three pairs of fields beneath it. Route and method (observed, extracted) tell a fact from a guess. Observed_at and review_after (2026-09-02 and 2027-03-01) make freshness computed rather than assumed. Permitted purposes and allowed surfaces say which use and which surface the value may serve, acting as a filter before generation.](/images/handbook/ch06-provenance-envelope.svg)

*Figure 7.2. The Provenance Envelope: three pairs of fields turn source, freshness, and consent into part of the value.*

Law points the same way. Under the GDPR, when personal data was not obtained from the person, you must tell them its source [16], and where you rely on consent you must be able to demonstrate it [17]. Neither is possible for a value whose source was discarded at import. (Not legal advice; Chapter 19 and the dated companion notes cover jurisdictions.) Inferences need their own flag, too: the Court of Justice of the EU has held that data "liable indirectly to reveal" a special category, such as health or sexual orientation, falls under the stricter rules for that category [18]. An inference can be sensitive even when every input was ordinary.

## The Signal Ladder: what overrides what

Provenance says where each value came from, not which one wins. For that, rank conflicting values on four dimensions, in a fixed order.

1. **Purpose is a gate, not a weight.** First remove every value that is not permitted for this use on this surface. A value can win on every other dimension and still be unusable here.
2. **Authority depends on the kind of fact.** For preferences (topics, channels, frequency), the person's own declaration has authority. For circumstances (current role, employer), the person's own recent artifacts have authority: a signature, a reply, a changed login domain. For transactions (what they bought, what they use), your system of record has authority. For facts nobody stated, inference may fill in, but only to decide, never to assert.
3. **Freshness can override authority for things that change.** A declared title from 2025 loses to an observed title from last month. A declared channel preference does not lose to an inferred one, however recent.
4. **Confidence breaks ties, and corroboration raises it.** Two independent sources agreeing beat one source alone. Two vendors that resell the same upstream file are one source.

When nothing clears the bar, abstain. Hide the value, fall back to account-level or segment-level framing, or ask. A specific but false message is worse than a general but useful one.

Run Maya's record through the ladder. Every title is permitted for account management. Job title is a circumstance, so her own recent artifact has authority: **Director, Field Operations**, three weeks old, corroborated by a public profile. The CRM value becomes history, not deleted. The vendor values are set aside until they agree with something first-hand. The "economic buyer" inference stays in its lane: it may route the renewal conversation to her, and never appears in a sentence she reads.

![The Signal Ladder as four numbered rungs, each paired with its result for Maya's job title at Larkspur (fictional). Rung 1, purpose is a gate: every title is permitted for account management. Rung 2, authority by kind of fact (preference, circumstance, transaction, inferred): title is a circumstance, so her own recent artifact rules. Rung 3, freshness can override authority for things that change: her support email signature is three weeks old. Rung 4, confidence breaks ties: it is corroborated by a public profile. The winner is "Director, Field Operations"; the CRM value is kept as history, vendor values are set aside, and the "economic buyer" inference routes the conversation but is never asserted. When nothing clears the bar, the system abstains: hide, fall back, or ask.](/images/handbook/ch06-signal-ladder.svg)

*Figure 7.3. The Signal Ladder applied to Maya: purpose, authority, freshness, and confidence, in that order, or abstain.*

## What this does not do

A ladder does not fix identity. If the support email came from a different Maya Okafor, the freshest, most authoritative value belongs to the wrong person. Resolving who a record is about comes first (Chapter 9).

Permitted is not appropriate. A fact can clear every purpose gate and still feel invasive in a message (Chapter 19).

More routes do not mean more depth. The usual response to thin personalization is another data vendor, yet the facts that would change what Larkspur says to Maya were already inside Larkspur: a ticket, a signature, a usage log. Depth comes from the fact that changes the message, not from the number of facts on file.

Nor is the envelope a finished product, including mine. In the personalization engine I built, account research is increasingly stored as structured fields so each one can be trust-filtered and rendered on its own; attaching confidence, provenance, and freshness to every one of those fields is the step still in progress. The envelope in this chapter is the design I am building toward, not a description of a system that already carries it everywhere.

And be wary of confident claims. "Third-party cookies are dead" is wrong as of September 2026. "Only 4% allow tracking" is a real number with a narrow denominator. "Our data is 95% accurate" is usually a vendor measuring itself.

## At scale

With 250,000 contacts, conflicts stop being edge cases: even a small share of multi-valued fields means thousands of silent decisions per send. The ladder makes them consistent rules, and the envelope makes each one explainable afterward.

Scale also changes the economics of verification. Re-checking a field by hand was rationed to top accounts. Re-checking it by extraction from the customer's own emails and tickets is cheap enough to run on every record, on a schedule. The constraint moves from the cost of checking to the discipline of storing what the check found.

And many teams means many writers: marketing imports a vendor file, sales edits the CRM, a support agent extracts a title. Without a shared envelope, the last writer wins. With one, the ladder wins.

More sources also means one of them is always failing. In the engine I built, research runs across several providers and tolerates partial failure: if one source errors and two succeed, the record is built from the two instead of failing, and a provider that keeps failing is skipped for the rest of the batch rather than called again on every record (Chapter 22 shows what happens without that). The consequence for this chapter is provenance. A record assembled from two of three sources should say which one was missing. **An unanswered source is not a source that agreed**, and corroboration counted across sources that were never asked is not corroboration.

Intake needs the same distinction between kinds of absence. The engine rejects a lead whose required fields are invalid, such as an unusable email, and accepts one with no title, with a warning: a missing title and an invalid email are not the same class of problem. Reject what breaks identity or permission, warn on what only reduces depth, and never fill a gap with a default that looks like data. The load date in the failure story below was exactly that kind of default.

## Failure story: The Undated Purchase

A composite, at a company much like Larkspur. Before a renewal campaign, marketing operations loads a vendor enrichment file to fill blank titles and refresh old ones. The vendor's titles carry no date, so the import job stamps each value with the day it was loaded. One contact's support-email signature, six weeks old, read **Director, Customer Experience**. The vendor file said **Dispatch Manager**, a role she had left more than a year earlier. Stamped with the load date, the purchased title looked fresher than her own signature, and the freshness rule meant to protect the record did the damage: the vendor value won. The campaign invited her to a workshop "for managers who run dispatch every day." She forwarded it to her successor with one line: "I think this is yours."

The vendor's value was true once. It became wrong when the import gave an unknown age a date it never had. `observed_at` is when the world was seen, not when the file arrived. A purchased value without one ranks as undated, below any first-hand artifact, and waits for corroboration.

## Patterns

**Signal Ladder.** *Problem:* sources disagree about one property. *Forces:* they differ in authority, age, and permission; templates read one field. *Solution:* gate by purpose, rank by per-property authority, let freshness override for volatile properties, break ties by corroborated confidence, abstain when nothing qualifies. *Tradeoffs:* needs provenance everywhere; authority rules take real thought.

**Provenance Envelope.** *Problem:* facts lose their source at import. *Forces:* schemas were designed for dashboards; the new reader is a model. *Solution:* every personalization-relevant value carries route, source, method, time, confidence, purposes, and surfaces. *Tradeoffs:* larger records; import pipelines must keep what they used to discard.

A third pattern, **Enrichment as Cross-Examination**, belongs with these two; Chapter 8 defines it in full.

## Leader questions

1. For our top five personalization fields, can we say where each value came from, and when?
2. Which suppliers could show how their data subjects consented, if a regulator asked?
3. When two systems disagree about a customer, which wins today, and is that written down?
4. How much of our targeting depends on identifiers a browser or platform controls?
5. Where are AI conclusions about customers stored, and can anyone tell them from facts?

## Build checklist

- [ ] Tag every source feeding customer records with its route.
- [ ] Store route, source, method, timestamp, and confidence on every value used in personalization.
- [ ] Enforce permitted purposes and allowed surfaces before context reaches generation.
- [ ] Write per-property authority rules (preference, circumstance, transaction, inferred).
- [ ] Keep inferences in a labeled lane; never overwrite a fact with one; keep superseded values as history.
- [ ] Bake off enrichment vendors per field; keep supplier consent attestations and review them yearly.
- [ ] Define the abstention path: hide, fall back to account or segment framing, or ask.
- [ ] Record which sources were asked and which answered; never count a failed source as agreement.
- [ ] At intake, reject what breaks identity or permission, warn on missing depth, and never default a gap into data.

## Metrics to watch

- **Provenance coverage:** share of personalization values with a known source and timestamp.
- **Conflict rate per property:** share of records with competing values, and how many the ladder resolves.
- **Abstention rate:** how often a value is withheld for lack of trust; watch the trend.
- **Enrichment agreement:** per-field agreement between each vendor and first-hand evidence.
- **Inference leakage:** customer-facing claims whose only source is an inference (target: zero).

## Reader Q&A

**Is first-party data always better than third-party data?** For what happened in your relationship, yes. For firmographics about an account you have never spoken to, a vendor may be all you have. That is why authority is defined per property type.

**Can we use publicly available information freely?** Public is not permitted. Under the GDPR, public personal data is still personal data, and you still owe notice of the source [16]. Check with counsel.

**How do we store an AI's conclusions about a customer?** As inferences: with the model, the inputs, the time, and a confidence, in a lane the generator is not allowed to quote from without confirmation.

## For your AI

```yaml
chapter: 7
concepts:
  - name: Data Route
    definition: "How a value entered the record: declared, observed, partner, purchased, public, or inferred. The route determines what kind of claim the value makes."
  - name: Signal Ladder
    definition: "Precedence for conflicting values: gate by purpose, rank by authority for the property type, let freshness override authority for volatile properties, break ties by corroborated confidence, abstain when nothing qualifies."
  - name: Provenance Envelope
    definition: "Per-value envelope carrying route, source, method, time, confidence, permitted purposes, and allowed surfaces."
  - name: Inferred Lane
    definition: "Separate, labeled storage for model or rule conclusions, usable for decisions but not asserted to the customer without first-hand confirmation."
  - name: Enrichment as Cross-Examination
    definition: "Using purchased and public data to challenge held values, not only to fill blanks. Pattern defined in Chapter 8."
decision_rules:
  - if: "a value is not permitted for the intended purpose or surface"
    then: "exclude it before ranking, regardless of authority, freshness, or confidence"
  - if: "the property is a preference (topic, channel, frequency)"
    then: "the person's own declaration has authority over observed or inferred values"
  - if: "the property is a circumstance (role, employer) and a first-hand artifact is fresher than the declared or stored value"
    then: "the fresher first-hand value wins; keep the older value as history"
  - if: "the only source for a customer-facing claim is an inference"
    then: "do not assert it; use it only to choose the action, or confirm first"
  - if: "a purchased value conflicts with first-hand or corroborated public evidence"
    then: "surface the conflict and do not use the purchased value in customer-facing content"
  - if: "no candidate value clears the trust bar"
    then: "abstain: omit, fall back to account or segment framing, or ask the person"
  - if: "a research source failed or was not consulted for a record"
    then: "record it as not checked; do not count it toward corroboration"
  - if: "an intake record is missing a non-essential field such as title"
    then: "accept it with a warning; reject only values that break identity or permission, and never fill the gap with a default"
assessment_questions:
  - "Which sources feed your customer records, and which route does each represent?"
  - "Do stored values keep their source and timestamp after import?"
  - "Where are AI-generated conclusions about customers stored, and are they labeled?"
  - "Can each data supplier document how consent was obtained?"
  - "Which personalization depends on third-party cookies or mobile advertising identifiers?"
  - "Which regions' customers are in scope, and has counsel reviewed notice-of-source and consent records?"
patterns: [Signal Ladder, Provenance Envelope]
related_patterns: [Enrichment as Cross-Examination (Chapter 8)]
anti_patterns: [The Undated Purchase, Flattened Routes, Rented Identity]
maturity_dimension: understanding
as_of: "2026-09"
```

## References

1. Khatibloo, F., et al. (Forrester, 2018), zero-party data definition, as quoted in *Frontiers in Big Data* (2022). https://www.frontiersin.org/journals/big-data/articles/10.3389/fdata.2022.943372/full
2. Apple Support, "Use Mail Privacy Protection on iPhone." https://support.apple.com/guide/iphone/use-mail-privacy-protection-iphf084865c7/ios
3. Kosinski, M., Stillwell, D., and Graepel, T. (2013). "Private traits and attributes are predictable from digital records of human behavior." *PNAS* 110(15). https://doi.org/10.1073/pnas.1218772110
4. Staab, R., Vero, M., Balunović, M., and Vechev, M. (2024). "Beyond Memorization: Violating Privacy via Inference with Large Language Models." ICLR 2024. https://arxiv.org/abs/2310.07298
5. Park, J. S., et al. (2024). "Generative Agent Simulations of 1,000 People." arXiv:2411.10109. https://arxiv.org/abs/2411.10109
6. US Bureau of Labor Statistics, Employee Tenure news release (January 2026 data), released 2026-09-24. https://www.bls.gov/news.release/tenure.nr0.htm
7. FTC, "FTC to Ban Kochava Subsidiary from Selling Sensitive Location Data...," 2026-05-04. https://www.ftc.gov/news-events/news/press-releases/2026/05/ftc-ban-kochava-subsidiary-selling-sensitive-location-data-settle-charges-they-sold-location-data
8. California Privacy Protection Agency (CalPrivacy), "Data Brokers" and DROP. https://privacy.ca.gov/data-brokers
9. Google, Chromium Blog, "Building a more private web: A path towards making third party cookies obsolete," 2020-01-14. https://blog.chromium.org/2020/01/building-more-private-web-path-towards.html
10. IAPP, "Google ends third-party cookie phaseout plans," 2024-07-22. https://iapp.org/news/a/google-ends-third-party-cookie-phaseout-plans
11. Google, "Next steps for Privacy Sandbox and tracking protections in Chrome," 2025-04-22. https://privacysandbox.google.com/blog/privacy-sandbox-next-steps
12. Google (A. Chavez), "Update on Plans for Privacy Sandbox Technologies," 2025-10-17. https://privacysandbox.google.com/blog/update-on-plans-for-privacy-sandbox-technologies
13. MDN Web Docs, "Third-party cookies." https://developer.mozilla.org/en-US/docs/Web/Privacy/Guides/Third-party_cookies
14. Apple Developer News, App Tracking Transparency requirement with iOS 14.5, 2021. https://developer.apple.com/news/?id=ecvrtzt2
15. Flurry Analytics, iOS 14.5 ATT opt-in rate reports, 2021. https://www.flurry.com/blog/ios-14-5-opt-in-rate-att-restricted-app-tracking-transparency-worldwide-us-daily-latest-update/
16. GDPR (Regulation (EU) 2016/679), Art. 14(2)(f): information on the source of personal data not obtained from the data subject. https://eur-lex.europa.eu/eli/reg/2016/679/oj
17. GDPR, Art. 7(1): controller must be able to demonstrate consent. https://eur-lex.europa.eu/eli/reg/2016/679/oj
18. CJEU, Case C-184/20, *OT v Vyriausioji tarnybinės etikos komisija*, 2022-08-01. https://curia.europa.eu/juris/liste.jsf?num=C-184/20

---

# 8. Preparing It

*The question: How do we clean, enrich, and use unstructured data?*

> **Questions this chapter answers**
> - How clean does our customer data actually need to be before AI can use it?
> - Our best knowledge lives in calls, emails, and documents. How do we turn it into something a system can act on?
> - What does enrichment really add, and when does it make things worse?
> - Where does personal data get handled, and why does the order of the pipeline matter?

## The short answer

Most companies prepare data for the wrong reader. They clean it for reports and dashboards, where a field is either filled or empty. The new reader is an AI system that has to decide what to say to one person, and it needs something different: facts it can trust, with a source, a date, and a clear meaning.

Four moves do most of the work. **Clean for the decision, not for perfection**: normalize the fields your decisions actually read, and let the rest wait. **Preserve, then process**: keep every raw source before any model touches it, so every derived fact can be traced, re-extracted, and corrected. **Extract with a contract**: turn calls, emails, and documents into typed facts using property descriptions precise enough to act as instructions to the extractor. **Make enrichment argue with what you have**: use outside data to cross-examine existing values, not only to fill blanks.

Sensitive data that has no business in memory is removed on the way in, before extraction and embedding, because what reaches a vector cannot be cleanly taken back out. The payoff is not tidier data. It is the ability to be specific when the evidence supports it and honestly general when it does not.

---

Here is a record from Larkspur Systems, a fictional composite company we follow through this handbook: a mid-market maker of field-service and fleet software with about 40,000 customer accounts. One account, Oakhurst Mechanical (also fictional), looks like this in the CRM: industry "Construction," employees "51-200," stage "Customer," renewal in November, three contacts, last activity 19 days ago.

Now read the transcript of Larkspur's 42-minute check-in call with Oakhurst's operations director. In it she says they are adding about forty vans in the spring. The dispatcher who championed Larkspur left in June. Finance is consolidating vendors this year. The mobile app's offline mode fails at rural job sites. And the renewal decision has moved from her to the CFO.

Five facts that change what Larkspur should say, to whom, and when. The CRM record contains none of them. The transcript sits in a call-recording tool, searchable by date and little else. That gap is the subject of this chapter.

## The record was never the whole customer

CRM fields hold what someone decided in advance was worth a column. What customers actually tell you (what changed, what worries them, who now decides) arrives as language: calls, emails, tickets, notes, documents.

You will often read that 80 to 90 percent of enterprise data is unstructured. I do not use that number. Its usual origin is a 1998 Merrill Lynch report that said "some estimates run as high as 80%," with no source, and Seth Grimes traced and debunked the rule in 2008 [1]. IDC's forecast that 80 percent of the global datasphere would be unstructured by 2025 counts video and sensor data, which says nothing about your customer records [2]. You do not need the statistic. Put one transcript next to its CRM record and count.

This is observed, not measured at population level: in the deployments I have worked on, the language channels held the facts that changed decisions, and the fields held the facts that routed them.

## Clean for the decision, not for perfection

Data quality is genuinely poor almost everywhere. In the best-documented study, Nagle, Redman, and Sammon had 75 executives each inspect the last 100 records their teams created; on average 47 percent of new records had at least one critical error, and only 3 percent of the resulting scores met a basic acceptable standard [3]. Gartner's widely quoted figure that poor data quality costs organizations $12.9 million a year on average comes from self-reported estimates by data-quality vendors' reference customers, so treat it as a signal of scale, not a measurement [4].

The trap is concluding that everything must be clean before anything can be personalized. That project never finishes. Clean what your first decisions (Chapter 6) read:

- **Identity keys**: emails lowercased, domains reduced to the registrable domain, and personal email providers never treated as employers. A Gmail address does not make someone an employee of Google.
- **Enumerations**: one vocabulary for industry, stage, and region, mapped from every source's variants.
- **Dates and units**: one time zone convention, one currency, explicit ranges instead of provider buckets that masquerade as numbers.
- **Nulls with meaning**: "unknown," "not applicable," and "checked, found nothing" are three different states. Collapse them and you lose the difference between ignorance and evidence.

Cleaning is a cost; spend it where a wrong value would change what someone receives.

One rule belongs here, before any model runs: **structured facts stay structured.** In the personalization engine I built, a value already known exactly (a campaign ID, a score computed in code) is written as a value and skips extraction, because a model pass adds cost and turns a certain fact into a probable one. The reverse holds too: in the memory system I built, fields such as a person's name can be declared non-extractable, so they arrive only from a system of record.

## Preserve, then process

Before extraction, there is a rule that sounds like plumbing and decides whether the system can ever be trusted: **preserve, trace, then derive.**

When a transcript arrives, store the original first, with a stable source ID, system of origin, the entities it concerns, observation and retrieval times, and a processing state. Then run the models. A retry must not create a second, apparently independent memory, and lineage must point to the original record, not to the first agent's prose. One call may concern three people and two companies; that is one source with five associations, not five sources.

Only then can you re-extract when your schema improves (it will), explain any statement by walking back to the words someone actually said, and delete properly, because you know every derivative of a source. And an empty check is not evidence: "nothing new in this pull" is not "this account has no activity." Update when you last looked; create no memory.

Derivation must never gate preservation: a document is committed before property extraction runs, and a failed extraction leaves it stored and says so.

![A pipeline of six steps: a source arrives, the raw source is preserved first, content is redacted before the model, then extracted and embedded, the output is redacted again, and derived facts are written with lineage. The preserved original is stored with a stable source ID, system of origin, the entities it concerns, processing state, and observation and retrieval times, and a dashed lineage line runs from every derived fact back to that original rather than to an agent's prose. A retry creates no second memory, and one call about three people and two companies is one source with five associations. Preserving first is what makes it possible to re-extract when the schema improves, explain any statement by walking back to the words actually said, and delete properly because every derivative of a source is known.](/images/handbook/ch07-preserve-then-process.svg)

*Figure 8.1. Preserve, trace, then derive: store the original before any model runs, and every fact can be traced, re-extracted, and deleted.*

Estimate the feed before you turn it on. A useful planning shape:

`source checks x material-change rate x processing cost + entities x analytical tasks x review cycles x activation rate`

Providers may charge for every check, including empty ones; models only consume tokens for records that survive filtering. Model inference is often not the largest cost: data rights, subscriptions, identity resolution, and correction can cost more. Collecting everything is not intelligence. Start with sources that change a defined decision, measure their yield, and turn off the ones that change nothing.

## Redact on the way in

A transcript like Oakhurst's can contain a card number read aloud, a personal mobile number, and an integration key pasted into the meeting chat. The question is not whether to remove sensitive data but where removal actually protects anything.

The answer is before extraction and before embedding. Once personal data is embedded, its signature is in the vector and can influence similarity search even if you filter the text at retrieval time. Retrieval-time filtering hides the problem; it does not remove it.

The rule I recommend is tiered. Secrets, financial identifiers (card numbers validated with the Luhn checksum, IBANs), and government identity numbers are always removed, because no personalization decision needs them; memory keeps "identity was verified," never the passport number. Contact data is configurable: a sales team needs email addresses to match records, a healthcare deployment must strip them. Custom patterns cover identifiers no built-in tier knows, such as policy numbers or internal case IDs. Where linking is needed without the value, a consistent hash says "these five documents mention the same address" without keeping it.

Redaction has a price: what it removes, the extractor cannot extract. Reversible tokenization resolves that for outside models. In the memory system I built, emails, phone numbers, and a supplied list of names become format-preserving stand-ins before the call and are restored locally before anything is written. It is an egress control, not an at-rest one: the record holds real values, so the data is pseudonymized, not anonymized. And a token the model mangles stays a lost value, never guessed back to the wrong person.

The embedder is a separate channel, and tokens cannot protect it because they are not stable between a save and a later query: redact before embedding, or embed locally for regulated data. Map every point where content leaves (extraction, embedding, query embedding, answer synthesis) and log each external call with a hash of what was sent, never the text.

Redaction runs twice: before the model sees the content, and again on the extracted output, to catch what the pattern matcher missed or the model reconstructed. Three production lessons. Early extractors sometimes wrote the placeholder itself into a field, setting an email property to `[EMAIL]`; validate for placeholders. A privacy option that silently does nothing is worse than an error: where a save path could not apply redaction, we made it refuse the option rather than store the body unredacted. And make the guarantee a test: every build of the memory system I built proves that no reversible token can reach a stored record, memory, or graph edge.

## Extract with a contract

Here is the sentence this chapter turns on: **the property description is the extractor's instruction.** What you write to define a field is what the model reads when it decides what to pull from a transcript. A vague description produces vague extraction, and no model upgrade fixes that.

The first extraction prompt is written by hand and works for ten fields. By thirty it contradicts itself, and teams break each other's results by editing one shared prompt. The fix is to move the instructions into each property and compose the prompt at run time from the properties relevant to that content.

A property description that works as a contract has four parts:

1. **What to capture**: the meaning in this organization, and why it is used.
2. **Where to find it**: the signals and sources (transcripts, signatures, support tickets, order forms).
3. **What it is not**: the boundary with neighboring properties and common look-alikes.
4. **Examples**: one clear case, one that needs transformation, one where nothing should be extracted.

For Larkspur, one property looks like this:

```yaml
name: fleet_expansion_plan
type: text
update: replace          # newest plan supersedes older plans
description: >
  Capture: a stated, dated plan to add vehicles, technicians, or service
  regions, with size and timing when given. Sales uses it to time expansion
  conversations; Customer Success uses it to plan onboarding capacity.
  Where to find it: renewal and check-in calls, emails from operations or
  fleet managers, purchase requests, job postings for dispatchers.
  What it is NOT: general optimism ("we are growing"), a competitor's plans,
  a vendor's forecast, or plans the speaker attributes to someone else
  without confirming them.
examples:
  - input: "We're adding about forty vans in the spring."
    output: "Adding ~40 vans, spring (stated by operations director)"
  - input: "Business is good, we'll probably grow."
    output: null
```

![A four-row table showing the parts of a property description that works as a contract, filled in for Larkspur's (fictional) fleet_expansion_plan property. What to capture: a stated, dated plan to add vehicles, technicians, or service regions, used by Sales for timing and by Customer Success for capacity. Where to find it: renewal and check-in calls, operations emails, purchase requests, and job postings for dispatchers. What it is not: general optimism, a competitor's plans, a vendor's forecast, or unconfirmed plans attributed to someone else. Examples: "We're adding about forty vans in the spring" becomes "Adding ~40 vans, spring," while "Business is good, we'll probably grow" becomes null. When extraction is wrong, fix the description and examples before reaching for a bigger model.](/images/handbook/ch07-property-contract.svg)

*Figure 8.2. Extract with a contract: four parts turn a field definition into an instruction the extractor can follow, including when to extract nothing.*

Ownership becomes local: the team that owns a field owns its definition, and editing one cannot silently degrade another. Diagnosis becomes local too: when extraction is wrong, fix the description and examples before reaching for a bigger model, which buys reasoning depth but not clarity. The research agrees: Sainz and colleagues showed that models trained to follow detailed annotation guidelines generalize to unseen extraction tasks, and that detailed guidelines are key to good results [5].

A schema only catches what its designer anticipated, so the same pass should also extract open-set observations: short, sourced statements the schema had no field for. In a controlled test across 20 samples on a system I built, 38 percent of the valuable information was captured only by open-set extraction, 12 percent only by schema extraction, and 16 percent by neither [6]. Neither form alone was enough, which is why the architecture I published pairs open-set atomic facts with schema-enforced typed properties in one memory model [10]. Telling the extractor what kind of entity it is reading matters too: the same company web pages, read "as a company record, like an analyst," produced products, pricing, positioning, and named customers instead of a list of names and titles [7]. This is the quiet decision inside every pipeline: what you choose to extract is the kind of understanding of a person you are building.

## Enrichment should argue with what you have

Enrichment is usually sold as filling blanks. That is half the job. The other half is cross-examination: using new sources to challenge values you already hold.

If a provider reports 100 employees and several current sources describe a global enterprise, the number deserves suspicion; the conflict should surface before any message is written, not be settled by whichever value makes the copy more interesting. I prefer deterministic trust rules: reject counts outside a plausible range, reject values that look like a provider's bucket floor, reject a value that differs from explicit research by a factor of five or more. Once a value is rejected, the model must not be allowed to reason its way back into using it.

Cross-examination is easier on fields than on prose. The engine I built increasingly stores account research as structured fields (headcount, leadership changes, buying triggers, competitors) rather than one paragraph, so each field is trust-filtered on its own and a known fact is rendered, not regenerated. The next step, confidence, provenance, and freshness on every field, is still being standardized.

Uncertainty needs its own state: known, likely, uncertain, rejected. For anything customer-facing, the policy is **hide when uncertain**. That does not freeze personalization; it moves it down a ladder from person-and-account to account to segment, without treating the downgrade as a failure. A specific but false message is worse than a general but useful one.

![Conflicting evidence, a provider reporting 100 employees against current sources describing a global enterprise, flows into deterministic trust rules: reject counts outside a plausible range, reject values that look like a provider's bucket floor, and reject values five times or more off explicit research, with no reasoning back into a rejected value. The result is an explicit state: known, likely, uncertain, or rejected. Uncertain and rejected values are hidden from customer-facing output, which steps personalization down a ladder from person and account, to account, to segment, and stepping down is not a failure.](/images/handbook/ch07-enrichment-cross-examination.svg)

*Figure 8.3. Enrichment as cross-examination: trust rules settle conflicts before any message is written, and uncertain values move personalization down a level instead of into the copy.*

Decay is why enrichment is continuous. The usual claim that B2B data decays 22.5 percent a year has no primary source I can find. A defensible proxy is job tenure: the U.S. Bureau of Labor Statistics reported median tenure with the current employer of 4.1 years in January 2026, with 20.6 percent of wage and salary workers at one year or less [8]. Titles, teams, and authority move on roughly that clock, which is why Oakhurst's champion could leave in June while the CRM still listed her.

## What this does not do

Extraction does not make a fact true. It makes a claim structured. Structured outputs guarantee shape, not accuracy: a perfectly typed field can hold something the speaker never said. Provenance is what lets you check.

Pattern-based redaction and tokenization are high precision, not complete coverage: spelled-out numbers, unusual formats, and unlisted names escape them. Even dedicated tools say so: Microsoft's Presidio documentation states there is no guarantee it will find all sensitive information [9]. Redaction is one control among several (access scoping, retention limits, deletion), never the whole answer.

Single-pass extraction has a ceiling. In the test above, 16 percent of valuable information was missed by both methods, often implications that span several sources. And the standard memory benchmarks measure whether a system recalled a fact correctly, not whether it extracted something an agent can act on [7]. Evaluate on your own decisions.

I also disagree with the popular framing behind the "80 percent unstructured" line [1]. It makes unstructured data sound like a volume problem solved with storage. It is a meaning problem, solved with definitions.

## At scale

At 250,000 contacts, four things change. **The schema becomes shared infrastructure**: dozens of properties across teams stay workable only when each carries its own contract, each call includes only the properties relevant to that content, and names are canonicalized on save so "Deal Stage" and "deal stage" never become two silent keys. **Duplicates dominate**: when five overlapping sources were processed for one entity in a controlled test, 83 percent of candidate memories duplicated something already stored [6], so write-time deduplication is not optional. **Replay becomes routine**: every schema, redaction, or model change raises the question of what to re-extract. Without preserved originals, the answer is "all of it, and hope." **Loads go through a queue, not a loop**: a CRM export, a spreadsheet, or a year of history enters as one bulk job, never as thousands of looped single saves. In the memory system I built, that queue lives in the database, so jobs survive a restart; each row retries with backoff and failures are reported row by row; and ingest dedupes by a content and entity hash, so re-running an import after a timeout does not write twice. A provider's batch interface cuts the model price roughly in half for up to a day of latency, and a failed batch falls back to the normal worker, so no record is lost.

## Failure story: The Quality Trap

We once ran extraction over a set of company websites. The first pass returned 12 memories for one company, five of them names and titles. We added aggressive splitting to "capture more," and the count jumped to 101. The dashboard looked like a breakthrough.

Then we read the output: 149 near-duplicate pairs, 14 junk entries such as copyright notices and "has a careers page," and a product's feature list shattered into ten fragments. Two rules fixed it: keep lists together unless an item carries its own detail, and filter boilerplate. The count dropped to between 18 and 22, and every remaining memory was usable [7]. More facts in the store means more noise in every context window that reads them. Count is not quality.

## Patterns

**Extract with a Contract.** *Problem*: one hand-written prompt cannot scale past a few dozen fields owned by many teams. *Solution*: each property carries the four-part description; prompts are composed per call from relevant properties; open-set extraction runs alongside. *Tradeoffs*: authoring work per property, and an evaluation harness to prove it.

**Preserve then Process.** *Problem*: derived facts cannot be traced, replayed, or deleted. *Solution*: store the raw source with identity, origin, timestamps, and processing state before any model runs; derive idempotently; keep lineage. *Tradeoffs*: raw content carries its own retention and redaction obligations.

**Enrichment as Cross-Examination.** *Problem*: enrichment layers conflicting values onto records and the model picks one. *Solution*: compare sources with deterministic trust rules, give uncertainty an explicit state, and hide uncertain values from customer-facing output. *Tradeoffs*: less specific output for some records, by design.

## Leader questions

1. Which three decisions are we preparing data for, and which fields do they actually read?
2. When our system states a fact about a customer, can we show the original words or record it came from?
3. Where in our pipeline is personal data removed: before or after it reaches a model and an index?
4. When two sources disagree about a customer, who or what decides, and is that rule written down?
5. How many of our extracted facts were used in a decision last month?

## Build checklist

- [ ] Clean identity keys, enumerations, dates, and units for your first decisions; treat personal email domains as non-employers.
- [ ] Store every raw source before processing, with source ID, origin, entities, timestamps, and processing state.
- [ ] Make extraction idempotent; retries never create new memories; load backfills and imports as bulk jobs, not looped saves.
- [ ] Write known values directly, never through extraction; declare the fields that must never be inferred.
- [ ] Redact secrets, financial, and identity numbers before extraction and embedding; configure contact data per use case; tokenize identifiers bound for an outside model; treat the embedder as its own channel; scan outputs again, including for placeholder values.
- [ ] Write every property description in four parts, with a negative example.
- [ ] Run open-set extraction in the same pass as typed extraction.
- [ ] Encode trust rules for enriched values and a hide-when-uncertain policy.
- [ ] Deduplicate at write time; estimate the feed cost before switching a source on.

## Metrics to watch

- **Traceability rate**: share of customer-facing facts that link to a preserved source.
- **Extraction precision per property**: sampled against human labels, using each property's own criteria.
- **Noise and duplicate ratio** in newly written memories.
- **Enrichment conflict rate**: share of enriched values rejected or flagged, by provider.
- **Source yield**: decisions changed per thousand source checks.

## Reader Q&A

**Do we need a data-cleansing project before we start?** No. Clean the fields your first decisions read, and let the rest wait until a decision needs it.

**Should we store transcripts, or only what we extract from them?** Both, under retention rules. The extraction is what agents read; the original is what lets you verify, re-extract, and delete.

**Can a larger model replace good property descriptions?** Rarely. A stronger model reasons more; it does not know what your organization means by "decision maker." Fix the description first.

## For your AI

```yaml
chapter: 8
concepts:
  - name: Extract with a Contract
    definition: Each property carries what to capture, where to find it, what it is not, and examples; this description is the extractor's instruction.
  - name: Preserve then Process
    definition: Store the raw source with identity, origin, timestamps, and processing state before any model derives facts from it.
  - name: Enrichment as Cross-Examination
    definition: Outside data is used to challenge existing values as well as fill blanks, under deterministic trust rules.
  - name: Redact on the Way In
    definition: Remove sensitive data before extraction and embedding, and scan extracted output again.
decision_rules:
  - if: "a field is not read by any prioritized decision"
    then: "defer cleaning it"
  - if: "an enriched value conflicts with current evidence beyond a set threshold"
    then: "reject it and hide it from customer-facing output"
  - if: "extraction for a property is wrong"
    then: "revise the property description and examples before changing model tier"
  - if: "a source check returns nothing new"
    then: "update last-checked time; create no memory"
  - if: "a value is already known exactly from a system of record or code"
    then: "write it as a structured value; do not route it through extraction"
  - if: "content is sent to a model or embedder outside your infrastructure"
    then: "redact always-removed tiers, tokenize identifiers for the chat model, and redact or run locally for the embedder"
  - if: "loading a backfill, CRM export, or spreadsheet"
    then: "submit it as one durable, idempotent bulk job with per-row status, not a loop of single saves"
assessment_questions:
  - "Which systems hold calls, emails, and documents, and is any of it extracted into typed fields today?"
  - "Are raw sources retained with lineage to every derived fact?"
  - "At which pipeline step is personal data redacted, and is the embedding call covered?"
  - "How are conflicts between enrichment providers and CRM values resolved?"
patterns: [Extract with a Contract, Preserve then Process, Enrichment as Cross-Examination]
anti_patterns: [The Quality Trap, Retrieval-Time Redaction, The Placeholder Value, Clean Everything First, The Silent Privacy Option]
maturity_dimension: understanding
```

## References

1. Seth Grimes, "Unstructured Data and the 80 Percent Rule," Breakthrough Analysis / Clarabridge Bridgepoints, 2008-08-01. http://altaplana.com/articles.html (catalog: http://metadatace.cci.drexel.edu/omeka/items/show/14556). Traces the claim to Shilakes and Tylman, Merrill Lynch, 1998.
2. VentureBeat, "Report: 80% of global datasphere will be unstructured by 2025" (IDC forecast). https://venturebeat.com/data-infrastructure/report-80-of-global-datasphere-will-be-unstructured-by-2025
3. Tadhg Nagle, Thomas C. Redman, David Sammon, "Only 3% of Companies' Data Meets Basic Quality Standards," Harvard Business Review, 2017-09-11. https://hbr.org/2017/09/only-3-of-companies-data-meets-basic-quality-standards
4. Gartner, "Data Quality: Why It Matters and How to Achieve It." https://www.gartner.com/en/data-analytics/topics/data-quality
5. Oscar Sainz et al., "GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction," ICLR 2024. https://arxiv.org/abs/2310.03668
6. Hamed Taheri, "Dual Memory: Free Text and Typed Properties," hamedtaheri.com. /articles/dual-memory-free-text-and-typed-properties
7. Hamed Taheri, "Beyond Fact Count: Measuring What Actually Matters in Agent Memory Extraction," hamedtaheri.com, 2026-04-08. /articles/beyond-fact-count-agent-memory
8. U.S. Bureau of Labor Statistics, "Employee Tenure in 2026," released 2026-09-24. https://www.bls.gov/news.release/tenure.nr0.htm
9. Microsoft Presidio, FAQ. https://microsoft.github.io/presidio/faq/
10. Hamed Taheri, "Governed Memory: A Production Architecture for Multi-Agent Workflows," arXiv:2603.17787, 2026-03-18. https://arxiv.org/abs/2603.17787

Further reading on the site: "Schema-Guided Extraction," "Guided Memory Extraction," "4-Tier PII Redaction," "Two-Phase Redaction," and "Building an Enterprise Personalization Engine" (sections on enrichment and data trust).

---

# 9. One Customer, Many Systems

*The question: How do we unify a customer across CRM, warehouse, and tools without merging two people?*

> **Questions this chapter answers**
> - Our customer lives in the CRM, the warehouse, the product database, and the support desk. How do we make them one customer?
> - What is identity resolution, and how accurate does it have to be?
> - Do we need a CDP, or is the warehouse enough? Where does AI memory sit relative to both?
> - When should two records that look alike stay separate?
> - Which system wins when two of them disagree?

## The short answer

Every personalization system eventually has to answer a question that sounds trivial: is this the same person? The CRM has a contact. The product has a user. The support desk has a requester. The event pipeline has an anonymous browser that later filled in a form. Each system is right about its own records and silent about everyone else's.

Unifying them is identity resolution, the least glamorous and most consequential step in the stack. Memory, decisions, generation, and measurement all inherit its mistakes. A missed match makes you forgetful: you ask a customer something you already know. A false match makes you reckless: you tell one person something that belongs to another. The second error costs far more, and most systems are tuned as if the two were equal.

Three decisions matter more than any tool choice. Match deterministically where strong shared keys exist and probabilistically only where you must, with explicit confidence. Treat a merge as a reversible, logged event, not an overwrite. And let identity confidence control how specific you are allowed to be: when the system is unsure who someone is, it should say less, not guess more.

On the stack: CRM, warehouse, CDP, and event pipelines each do one job well. None of them alone is the customer's memory. Decide which system is the source of record for each field, write it down, and make every integration obey it.

The sentence to take away: **if identity confidence falls, personalization specificity should fall with it.**

---

## Two Sam Patels

*Larkspur Systems is a fictional composite company used throughout this handbook.*

Larkspur sells field-service and fleet software to about 40,000 customer accounts. One of them is a transport group with two subsidiaries that buy separately: a freight business and a rail-maintenance business. Each has its own contract, renewal date, and negotiated price. Both use the parent's email domain.

Both also employ someone named Sam Patel.

A cleanup job, written to reduce duplicates before an AI renewal pilot, matched contacts on first name, last name, and email domain. The two Sam Patels became one contact, and one email address survived. Three weeks later the renewal agent, working exactly as designed, pulled the freight subsidiary's contract, discount, and usage history and wrote a clear, well-grounded renewal note. It went to the Sam Patel at the rail subsidiary, who forwarded it to his procurement lead with one line: "Why are they charging us more?"

Nothing in the generation was wrong. Every fact in the email was true, sourced, and current. It was true about someone else.

I call this **The Wrong Merge**, and it deserves study because it defeats every safeguard downstream of it. Grounding checks pass, because the facts are in memory. Tone checks pass. A human reviewer approves, because the email is good. The only layer that could have stopped it is the one that decided two records were one person.

## This problem is older than the CRM

In 1946 the US public-health statistician Halbert Dunn put it in terms that still hold: "Each person in the world creates a Book of Life. This Book starts with birth and ends with death. Its pages are made up of the records of the principal events in life. Record linkage is the name given to the process of assembling the pages of this Book into a volume." [1]

Your CRM, warehouse, and support desk each hold loose pages. Newcombe and colleagues ran the first computerized probabilistic linkage of vital records in 1959 [2], and in 1969 Ivan Fellegi and Alan Sunter gave the field the formal theory most modern matchers still descend from [3].

The failure mode is old too. On February 27, 2011, Sergio Ramirez went to a Nissan dealership in Dublin, California, to buy a car. His TransUnion credit report carried an alert that his name matched a name on the US Treasury's sanctions list. The dealer refused to sell to him; his wife bought the car in her own name. The Supreme Court's opinion records that TransUnion "did not compare any data other than first and last names" and that the product "unsurprisingly" generated many false positives [4]. A class of 8,185 people sued; in 2021 the Court held that the 1,853 whose reports had gone to third parties had standing to recover damages [4].

The stakes there were higher than a renewal email, but the mechanism is the same as the Sam Patel merge: attributes that feel like identity, treated as proof. Names are not identifiers. Domains are not identifiers. They are evidence, and evidence has weight.

## Identity is a claim with a confidence, not a key

**Entity resolution** is the general problem: deciding which records refer to the same real-world thing, whether a person, company, product, or address. **Identity resolution** is the customer-data case: which records, identifiers, and events belong to the same person or organization, including stitching anonymous browsers and devices to known people. The mathematics is shared; the evidence differs.

Before matching anything, decide what kinds of things exist. It is tempting to create types called Customer, Lead, Partner, and Employee. That becomes brittle quickly. One person can be a product user, the billing contact on an account, a champion on an open deal, and next year a buyer somewhere else. Mixing identity with role creates duplicates and hides change.

Use a small set of durable entities instead:

| Entity | Larkspur example | Why it is separate |
|---|---|---|
| Person | Each Sam Patel | A person persists while jobs and roles change |
| Organization | The group; each subsidiary | Contracts belong to legal entities, not people |
| Account | The freight subsidiary's Larkspur account | Billing, entitlements, and renewals attach here |
| Role | Fleet operations lead at the freight subsidiary, since March | A dated link between a person and an organization |
| Household | Rare in B2B; central in consumer businesses | Shared addresses, devices, and payment |

"Lead", "customer", and "champion" then become dated relationships among entities rather than containers. A person **works at** an organization; an organization **owns** an account; a person **is the billing contact for** an account from one date to another. This is what lets a system notice that someone left instead of writing to their old job, and it would have told Larkspur it had two people with two roles at two organizations, not one person with a messy history.

Model entities, not dossiers. A dossier is a pile of facts attached to a name. An entity has stable identity, typed relationships, and a history.

## Deterministic first, probabilistic where the evidence is thin

**Deterministic matching** links records that share a strong identifier exactly: a product user ID, a CRM ID carried into the warehouse, a verified email address, a billing account number. It is fast, explainable, and auditable. It only works when a clean shared key exists, and often one does not: people use two addresses, company names appear with and without legal suffixes, trials start from personal email.

**Probabilistic matching** weighs many weaker signals together. In the Fellegi-Sunter model [3], each compared field (name, email, phone, company, address) has two probabilities: how often it agrees when two records truly match (*m*), and how often it agrees by coincidence when they do not (*u*). Agreement on a rare surname is strong evidence; agreement on "John Smith" is weak. The weights add up to a score compared against two thresholds: above the upper, link; below the lower, non-link; in between, a possible link that goes to review.

That three-way outcome is the most useful idea in the field, and most customer-data pipelines throw it away by forcing every pair into "merge" or "don't."

![A record pair's match score is compared against two thresholds. Below the lower threshold is a non-link: the records stay separate entities. Between the thresholds is a possible link: a candidate link only, with no facts crossing it, high-consequence pairs routed to review, and any model reasoning logged as evidence. Above the upper threshold is a link. Field agreements carry different weights: agreement on a rare surname is strong evidence, agreement on "John Smith" is weak.](/images/handbook/ch08-match-zones.svg)

*Figure 9.1. Fellegi-Sunter gives three outcomes; the possible-link band is where most pipelines lose information by forcing a yes or no.*

Open-source tools make this practical at company scale. Splink, built by the UK Ministry of Justice's data linking team, implements Fellegi-Sunter with parameters estimated by expectation-maximization; its authors report linking a million records on a laptop in about a minute, with users including the UK Office for National Statistics and the Australian Bureau of Statistics [5]. Zingg takes an active-learning approach on Spark, Databricks, and Snowflake [6]; Senzing sells a commercial engine [7]. That is landscape as of September 2026, not a recommendation.

Two mechanics decide whether any of this runs. **Blocking:** Larkspur's 250,000 contacts make about 31 billion possible pairs (250,000 squared, halved). You compare only within blocks of plausible matches (same domain, same postcode, similar surname), and a pair that never shares a block can never match, so test your blocking rules for lost recall. **Normalization:** lowercase emails, strip legal suffixes, reduce domains to their registrable form, and enforce one rule that is violated constantly: **a personal email provider is not an employer.** A Gmail address should not inherit Google's firmographics. In the governed personalization engine I built, a freemail address is a flag with consequences: company research suppressed, firmographics hidden, lower-specificity copy, extra QA checks. The same over-trust of domains merged the two Sam Patels.

Language models are good in the gray zone, reading two records and explaining whether "Harrow Rail Services Ltd" and "Harrow Rail" are the same operating company. They are poor as the primary matcher: expensive, hard to reproduce, and confidently wrong. In the self-hosted memory system I built, that split is enforced. A deterministic finder proposes pairs from three probes: a strong key already pointing at a different record (two imports disagreeing about who owns it), the same normalized name *and* the same parent organization (a name alone never qualifies), and a shared secondary identifier such as a phone number. A group of more than 50 records sharing one name and parent, or one phone number, is treated as a placeholder value (a switchboard number, a default), not 50 duplicates. A model then judges each pair and must return merge, with a confidence and a reason, or keep distinct. Merges below a 0.9 confidence floor are refused, and an applied keep-distinct verdict stops the pair from being proposed again: knowing two Sam Patels are two people is evidence too.

## The two errors are not symmetric

A **false merge** joins two different entities. A **missed merge** leaves one entity split across records. Tighten thresholds and false merges fall while missed merges rise.

Most teams tune for the error they can see. Duplicates show up on every dashboard; false merges are invisible until they hurt someone. For personalization that is backwards:

- A missed merge makes the system **forgetful**: a repeated question, a welcome email to someone already onboarded, a long-standing account treated as cold. Annoying, recoverable, usually private.
- A false merge makes the system **wrong about who someone is**: one person's data disclosed to another, one account's pricing applied to a different account. It can breach a contract or a privacy obligation, and it contaminates every memory, score, and summary built on the merged record.

So the rubric I use for merge confidence is expressed as what the system is allowed to do:

| Merge confidence | Typical evidence | What the system may do |
|---|---|---|
| **Certain** | Shared system-issued ID, or a verified email the person controls | Merge automatically; person-level facts allowed |
| **High** | Several independent strong signals agree | Link automatically; person-level facts on owned channels |
| **Possible** | Mixed evidence: the Fellegi-Sunter gray zone | Candidate link only; no facts cross it; review high-consequence pairs |
| **Weak** | Name only, or name plus a shared domain | Do not link |

The Sam Patel merge was weak evidence treated as certain.

Here is the hinge of this chapter: **if identity confidence falls, personalization specificity should fall with it.** A system sure of the person can use person-level facts. A system sure of the company but not the person should personalize to the account and role. A system sure of neither should be honestly general. Identity confidence is not a data-quality footnote. It is an input to every decision about what the system may say.

![A three-step ladder in which identity confidence falls from top to bottom. When the system is sure of the person, it may use person-level facts. When it is sure of the company but not the person, it personalizes to the account and role. When it is sure of neither, it stays honestly general. The resolution service returns one of four permitted levels: person, account, segment, or general.](/images/handbook/ch08-confidence-specificity.svg)

*Figure 9.2. Confidence controls specificity: the less sure the system is of who someone is, the less specific it is allowed to be.*

In the governed personalization engine I built, this is code, not a prompt instruction. Relationship statements ("your renewal is coming up", "your current contract") are blocked unless the relationship is known, and the copy is reframed as an evaluation. Weak evidence steps a record down a ladder: person and account, account, segment, approved static fallback, fail closed. It would not have caught the Sam Patel email, though: the bad merge made the relationship look known.

## A merge is an event, not an overwrite

The common implementation picks a survivor record, copies some fields over, and archives the loser. It is simple, and it destroys the evidence you need when the merge is wrong. Once the two Sam Patels were one row, there was no clean way back: whose phone number, whose tickets, whose call notes?

The alternative is to **link, not overwrite**. Keep every source record intact with its source system and native ID. A separate identity layer says "these source records resolve to entity E-1234", with confidence, evidence, the deciding rule or model, and a timestamp. A merge is a log entry; an unmerge is another. Views are rebuilt from links, so reversing a bad decision restores the prior state instead of requiring archaeology.

My own memory system uses a close cousin: a redirect with a journal. Both IDs stay valid and reads of the absorbed one land on the survivor; properties fold only where the survivor has no value or the absorbed one is strictly newer; the journal keeps a pre-image, so unmerge restores the old state. The limits: memories written after the merge stay with the survivor, and the journal expires (90 days by default). Merges also start propose-only; an operator promotes the judge, not each verdict, once its proposals hold up.

This matters more once AI memory sits on top. Extracted facts, account summaries, and scores were all written against an entity ID. When the entity splits, each memory must follow the source record it came from, which is only possible if every memory points to its source. Provenance is what makes identity mistakes repairable. W3C's PROV model offers a vocabulary to borrow (an entity *was generated by* an activity, *was attributed to* an agent) [8].

## Anonymous to known is a promotion, not a merge

Much behavior arrives before you know who someone is. Event pipelines give a browser or device an anonymous ID and link it to a known one when the person logs in or fills a form. Treat that transition as a promotion with its own rules:

- **A browser is not a person.** Shared laptops and office machines mean one anonymous ID can be several people. Attach anonymous history only for the session that led to identification, or within a window you can defend.
- **Anonymous identifiers are fragile.** As of September 2026, Chrome still allows third-party cookies by default while Safari and Firefox block them [9], and WebKit has capped script-set first-party cookies since 2019 (seven days under ITP 2.1) [10]. Do not build identity that needs a cookie to survive for months.
- **The identifying moment carries evidence.** A verified login is certain. A typed-in email on a form is weaker: people mistype, and people enter a colleague's address.

Packaged tools encode similar protection. Segment, for example, lets teams set per-identifier limits and a priority order; by default a profile holds one `user_id`, and when an event would push a profile past a limit, the lower-priority identifier is demoted and resolution retried rather than profiles being merged past the limit [11]. Whatever your tool, find the merge protection and configure it deliberately.

## Where each system is good, and where memory sits

**The CRM** (Salesforce, HubSpot, and peers) is the system of record for relationships and commercial workflow: accounts, contacts, deals, owners, activities. It carries the scars of every import and is rarely the best place to resolve identity at scale.

**The warehouse or lakehouse** (Snowflake, Databricks, BigQuery, and peers) holds every source side by side, which makes it the natural home for matching jobs.

**The customer data platform.** The CDP Institute defines a CDP as "packaged software that creates a persistent, unified customer database that is accessible to other systems" [12]. Packaged CDPs bundle event collection, identity, profiles, and activation. The **composable CDP** builds the same capabilities on the warehouse and uses **reverse ETL** to push modeled fields and audiences back into operational tools. The market has moved both ways: Twilio paid $3.0 billion for Segment in 2020 [13], Salesforce launched a "zero copy" partner network in 2024 to use warehouse data without moving it [14], and in 2025 mParticle merged with Rokt and Fivetran agreed to acquire Census [15]. Read that as consolidation, not a verdict.

**Event pipelines** (Segment and peers) collect behavior and handle anonymous-to-known stitching.

**Where AI memory sits.** None of these was designed for the reader that now matters most: an agent that needs, in one call and under a token budget, the typed facts, evidence, and synthesized understanding of one resolved entity (Chapter 10). The placement rule is simple: **memory sits downstream of identity resolution and upstream of decisions.** It consumes resolved entity IDs and never invents its own. If each agent keeps its own idea of who a customer is, you have rebuilt the duplicate problem inside your AI layer, only faster.

### The system-of-record map

Unification fails less on algorithms than on one unanswered question: when two systems disagree, which wins? Answer it field by field.

| Field | Source of record | On conflict |
|---|---|---|
| Contract terms, renewal date | Billing / contract system | Billing wins; flag CRM mismatch |
| Account owner, deal stage | CRM | CRM wins |
| Product usage, active users | Product data via warehouse | Warehouse wins |
| Job title, role | Most recent verified source | Enrichment may challenge, never overwrite |
| Consent, communication preferences | Preference center | Always wins |
| AI-extracted facts | Memory layer | Written to namespaced fields; never overwrites human entries |

Two write-back rules keep integrations from corrupting what they integrate. **Agents write to namespaced fields**, never on top of fields people maintain. **Every write is logged** with writer, time, and diff, and bulk write-backs start in dry-run. I apply both in an open-source CRM operations library I maintain, and they are the difference between a revenue leader trusting agent writes and banning them.

Enrichment gets one more rule: it should challenge what you hold, not only fill blanks. When a vendor's headcount disagrees with three current sources, surface the conflict instead of letting the newest value win.

## Identity as a first-class subsystem

Put the pieces together and identity stops being a cleanup job. It becomes a subsystem with a data model, an owner, and a service contract. I call the shape the **Identity Spine**:

1. **Stable internal IDs** for every person, organization, and account, never changed or reused. Source IDs are attributes, not keys.
2. **An identifier table**: every email, phone, CRM ID, product ID, anonymous ID, and domain with its source, first-seen and last-seen times, and verification status.
3. **Typed, dated links**: person to organization, person to account, account to parent, each with type, dates, and source. A role is a link, not a column.
4. **A merge log**: confidence, evidence, rule version, and timestamp for every link. Unmerge is a first-class operation.
5. **One resolution service**: memory, decisions, generation, and analytics all ask it "who is this?" and receive an entity ID, a confidence, and a permitted specificity level.

![Five source systems (CRM, warehouse, product database, support desk, event pipeline) feed the Identity Spine, which holds four layers: stable internal IDs that are never changed or reused; an identifier table with source, first and last seen, and verification status; typed, dated links where a role is a link rather than a column; and a merge log with confidence, evidence, and rule version. A single resolution service sits on top and answers "who is this?" for memory, decisions, generation, and analytics. Every answer carries an entity ID, a confidence, and a permitted specificity.](/images/handbook/ch08-identity-spine.svg)

*Figure 9.3. The Identity Spine: every consumer asks one resolution service, so "confidence controls specificity" is enforced in one place.*

The fifth element is the one teams skip. Without it, marketing resolves identity one way, support another, and each new agent a third.

Resolve by every identifier the business uses (phone, device ID, a CRM or deal ID), with one normalization shared by writes and reads. In one release of my memory system, records saved under a CRM key could not be read back by it, so a CRM-keyed sync minted duplicates. Silent splits come from asymmetry as often as from bad matching.

The spine also makes relationships queryable. Store every relationship identifier at write time (a contact carries both the person's email and the company's domain) and "every open ticket at this contact's company" becomes a lookup rather than a guess.

Identity also has a scope: the same account can sit in two campaigns, the same record in two tenants. In that engine, organization and campaign are resolved before any research or generation, and page identity is scoped by campaign so one program cannot overwrite another's page for the same company. Correct content in the wrong tenant or campaign is still a critical failure.

## What this does not do

**There is no perfect match rate.** Accuracy depends on the data, not just the algorithm. In healthcare, where investment is large and errors are dangerous, research cited by Pew (by the contractor Audacious Inquiry, for the federal health IT office) found match rates "as low as 50 percent" between organizations, even ones on the same electronic health record vendor [16]. Your CRM is not cleaner than a hospital's records. Plan for residual error and design downstream systems to survive it.

**"Golden record" and "360-degree view" are aspirations, not deliverables.** I agree with the CDP Institute's goal of a "persistent, unified customer database" [12]. Where I part company is with how the idea is usually sold: as one overwritten row per person that everyone can trust equally. A unified customer is a set of linked source records with confidence and provenance. A flattened profile that hides which fields are certain and which are guesses is less trustworthy than the fragments it replaced.

**Zero-copy does not resolve identity.** Querying the warehouse in place removes a copy; it does not decide whether two rows are one person.

**Models do not replace matching discipline.** A model asked "are these the same person?" answers fluently every time; measure it against labeled pairs like any other matcher.

**Resolution is not permission.** Linking product usage to a marketing profile is easy; whether you may use one for the other depends on consent and purpose (Chapter 19).

## At scale

- **Resolution becomes incremental.** New records match against existing entities as they arrive; periodic full runs catch drift. Keep both on the same rules, or they will disagree.
- **The gray zone outgrows human review.** Route only high-consequence pairs (open deals, active contracts) to people; leave the rest unlinked.
- **Identity is a production API.** It needs an owner, versioned rules, and change control. A threshold change is a release.
- **Corrections must propagate.** An unmerge has to reach memory, audiences, CRM fields, and cached summaries, so track which consumers read which entity.
- **Measurement depends on it.** Holdouts (Chapter 21) assume one person sits in one group. Duplicates leak people across groups; false merges mix them.

## Failure story: The Wrong Merge

- **What happened:** two people with the same name at sibling companies sharing a domain were merged; one subsidiary's pricing went to the other.
- **Root cause:** name plus domain treated as certain; parent and subsidiaries modeled as one organization; survivor-record merge with no log.
- **Fix:** one organization entity per contracting subsidiary; name plus domain demoted to weak; contract facts limited to contacts linked to that account at high confidence or better; link-and-log merges so the split was reversible.

The anti-pattern in one line: **deduplicating for tidiness before deciding what a false merge costs.**

## Patterns

**Identity Spine.** *Problem:* each system and agent resolves identity differently. *Solution:* stable internal IDs, identifier table with provenance, typed dated links, merge log, one resolution service. *Tradeoff:* a real subsystem to own; consumers give up private matching logic.

**Link, Don't Overwrite.** *Problem:* merges destroy the evidence needed to undo them. *Solution:* keep source records intact, resolve through logged links, make unmerge an operation. *Tradeoff:* more complex queries; views must be materialized.

**Confidence Controls Specificity.** *Problem:* downstream systems personalize equally hard whatever identity confidence is. *Solution:* the resolution service returns a permitted level (person, account, segment, general) and decisions and generation obey it. *Tradeoff:* some messages become generic; reduce that share by improving evidence, not by lowering the bar.

## Leader questions

1. Which system is the source of record for each customer field, and is that written down?
2. If two records are merged by mistake today, can we undo it, and how long does it take?
3. Do we measure false merges, or only duplicates?
4. Do all our AI agents get the same answer to "who is this?"
5. What share of personalized messages go out at person, account, and segment level?

## Build checklist

- [ ] Durable entity types defined; roles modeled as dated links.
- [ ] Internal IDs issued, never reused; source IDs stored as attributes.
- [ ] Normalization rules, including personal-email-is-not-an-employer and legal-suffix handling.
- [ ] Deterministic rules for strong keys; Fellegi-Sunter-style matching for the rest, with link / possible / non-link thresholds.
- [ ] Blocking rules tested for lost recall on a labeled sample.
- [ ] Merge log with confidence, evidence, and rule version; unmerge tested end to end.
- [ ] Model judges only finder-proposed pairs, returns merge or keep distinct, and starts in a propose-only stage.
- [ ] Resolution by every business identifier, with one normalization shared by write and read paths.
- [ ] Anonymous-to-known promotion rules, including shared devices.
- [ ] System-of-record map by field with conflict precedence.
- [ ] Write-backs to namespaced fields, logged with diffs, dry-run first.
- [ ] One resolution service returning entity ID, confidence, and permitted specificity.

## Metrics to watch

1. **False merge rate** (precision) from a regularly labeled sample of links, reported beside the **missed merge rate** (recall).
2. **Gray-zone backlog:** possible links waiting, and their age.
3. **Specificity mix:** share of decisions at person, account, segment, and general level.
4. **Time to unmerge** across every downstream consumer.
5. **Source-of-record conflicts** by field, and their trend.

## Reader Q&A

**Do we need a CDP?** Not necessarily. If data already lands in a warehouse and you can model it, a composable approach does the same work. A packaged CDP makes sense when you need real-time collection and activation without building them. Either way, the rules, thresholds, and system-of-record map are yours.

**How accurate must matching be?** Accurate enough for the action you take. For a generic newsletter, missed merges cost little. For anything carrying contract, pricing, health, or financial facts, false merges must be near zero, which means refusing weak links and accepting more duplicates.

**Should we merge a contact who changed companies?** The person is the same; the role is not. Keep one person, end-date the old role, and never carry the old employer's account facts into messages to the new one.

## For your AI

```yaml
chapter: 9
title: "One Customer, Many Systems"
concepts:
  - name: Identity Resolution
    definition: "Deciding which records, identifiers, and events across systems belong to the same person or organization."
  - name: Entity Resolution
    definition: "The general problem of deciding which records refer to the same real-world thing; identity resolution is its customer-data case."
  - name: Fellegi-Sunter Model
    definition: "Probabilistic record linkage that weighs field agreements by m and u probabilities and classifies pairs as link, possible link, or non-link."
  - name: Identity Spine
    definition: "Stable internal IDs, an identifier table with provenance, typed dated links, a merge log, and one resolution service returning entity ID, confidence, and permitted specificity."
  - name: Merge Confidence Rubric
    definition: "Certain, high, possible, weak; each level defines whether records may be linked and which facts may cross the link."
  - name: System-of-Record Map
    definition: "A per-field table naming the authoritative source and which source wins on conflict."
  - name: Finder-Judge Split
    definition: "A deterministic finder proposes candidate pairs; a model only judges each as merge or keep distinct, under a confidence floor and an operator-gated stage."
  - name: Anonymous-to-Known Promotion
    definition: "Rules for attaching anonymous device or browser history to a known person, handled separately from ordinary merges."
decision_rules:
  - if: "two records match only on name, or on name plus a shared email domain"
    then: "do not link; treat as separate entities"
  - if: "an email address is from a personal email provider"
    then: "do not infer employer or firmographics from the domain"
  - if: "identity confidence for the recipient is below high"
    then: "personalize at account, segment, or general level; never use person-level or contract-level facts"
  - if: "a pair falls between the link and non-link thresholds"
    then: "record a candidate link, share no facts across it, and route high-consequence pairs to review"
  - if: "two systems disagree on a field"
    then: "apply the system-of-record map; enrichment may challenge but not overwrite"
  - if: "an AI agent or integration writes back to the CRM"
    then: "write to namespaced fields, log writer, time, and diff, and start in dry-run"
  - if: "a merge is found to be wrong"
    then: "unmerge through the merge log and propagate the correction to every consumer, including memory"
  - if: "a language model is used to judge a candidate pair"
    then: "judge only finder-proposed pairs; require merge or keep distinct with a reason; refuse merges below a confidence floor; start propose-only"
  - if: "a customer-facing statement asserts a relationship (renewal, contract, product ownership)"
    then: "block it unless the relationship is known at high confidence; otherwise reframe as an evaluation"
assessment_questions:
  - "Which systems hold customer identity today, and which one wins on conflict for each field?"
  - "How are duplicates found and merged today, and can a merge be undone?"
  - "Do you measure false merges separately from duplicates?"
  - "Do all agents and channels use one resolution service, or does each resolve identity itself?"
  - "How are anonymous visitors linked to known contacts, and how are shared devices handled?"
  - "Are subsidiaries and parent companies modeled as separate organizations with their own contracts?"
patterns: [Identity Spine, Link Don't Overwrite, Confidence Controls Specificity]
anti_patterns: [The Wrong Merge, Survivor-Record Merge, Domain Equals Employer, Per-Agent Identity]
maturity_dimension: identity_and_memory
related_chapters: [7, 8, 10, 11, 19, 21]
```

## References

1. Dunn, H. L. "Record Linkage." *American Journal of Public Health* 36(12): 1412-1416, 1946. https://pmc.ncbi.nlm.nih.gov/articles/PMC1624512/
2. Newcombe, H. B., Kennedy, J. M., Axford, S. J., James, A. P. "Automatic Linkage of Vital Records." *Science* 130(3381): 954-959, 1959. https://www.science.org/doi/10.1126/science.130.3381.954
3. Fellegi, I. P., Sunter, A. B. "A Theory for Record Linkage." *Journal of the American Statistical Association* 64(328): 1183-1210, 1969. https://www.tandfonline.com/doi/abs/10.1080/01621459.1969.10501049
4. *TransUnion LLC v. Ramirez*, 594 U.S. ___ (2021), decided June 25, 2021. https://www.supremecourt.gov/opinions/20pdf/20-297_4g25.pdf
5. Linacre, R., et al. "Splink: Free software for probabilistic record linkage at scale." *Real World Data Science* (Royal Statistical Society), 2023-11-22. https://realworlddatascience.net/applied-insights/case-studies/posts/2023/11/22/splink.html
6. Zingg (open-source entity resolution). https://github.com/zinggAI/zingg (accessed 2026-09-26)
7. Senzing (entity resolution software). https://senzing.com/ (accessed 2026-09-26)
8. W3C. "PROV-DM: The PROV Data Model." W3C Recommendation, 2013-04-30. https://www.w3.org/TR/prov-dm/
9. MDN Web Docs. "Third-party cookies." https://developer.mozilla.org/en-US/docs/Web/Privacy/Guides/Third-party_cookies (accessed 2026-09-26)
10. WebKit. "Intelligent Tracking Prevention 2.1." 2019-02. https://webkit.org/blog/8613/intelligent-tracking-prevention-2-1/
11. Twilio Segment Docs. "Identity Resolution Settings." https://www.twilio.com/docs/segment/unify/identity-resolution/identity-resolution-settings (accessed 2026-09-26)
12. CDP Institute. "What is a CDP?" https://www.cdpinstitute.org/what-is-a-cdp/ (accessed 2026-09-26)
13. Twilio Inc. Form 10-K for fiscal year 2020 (Segment acquisition). https://www.sec.gov/Archives/edgar/data/1447669/000144766921000070/twlo-20201231.htm
14. Salesforce. "Salesforce Launches Zero Copy Partner Network." 2024-04-25. https://www.salesforce.com/news/press-releases/2024/04/25/zero-copy-partner-network/
15. Rokt and mParticle merger announcement, 2025-01-16, https://www.prnewswire.com/news-releases/rokt-and-mparticle-merge-to-redefine-real-time-relevance-302352650.html ; Fivetran, agreement to acquire Census, 2025-05-01, https://www.fivetran.com/press/fivetran-signs-agreement-to-acquire-census-delivering-the-first-end-to-end-data-movement-platform-for-the-ai-era
16. The Pew Charitable Trusts. "Enhanced Patient Matching Is Critical to Achieving Full Promise of Digital Health Records." 2018-10-02. https://www.pew.org/en/research-and-analysis/reports/2018/10/02/enhanced-patient-matching-critical-to-achieving-full-promise-of-digital-health-records

---

# 10. Memory Built for Agents

*The question: What changes when the main reader of our customer data is an AI agent, not a person at a dashboard?*

> **Questions this chapter answers**
> - Our CRM works fine for our people. Why does it fail our agents?
> - We already have RAG. Is that memory?
> - What should customer memory actually look like, and what goes where?
> - Context windows keep growing. Why not give the agent everything?
> - How do several agents share what they learn about the same customer?

## The short answer

For forty years, customer data was designed for one kind of reader: a person looking at a screen, who already knew most of the story and needed a reminder. That reader could interpret a status field because they were in the meeting where it was set. They could skim 200 activity entries because they knew which three mattered.

The main reader is now an agent. It was not in the meeting, it reads thousands of records a day, it pays for every token, and it cannot tell which field is authoritative or which note is stale. Hand it a dashboard-era record and it reads too much, reads the wrong slice, or acts on a field whose meaning it guessed.

The answer is not a bigger context window or a better search index. It is a different model of the customer. I call the pattern **Unified Record Memory**; in this handbook, where the record is almost always a customer, it appears as **Unified Customer Memory**. It is one record per entity that unifies everything an agent can know about it, kept in four forms that answer four different questions:

- **Typed properties** answer *which facts qualify, count, and govern?* (stage, renewal date, consent status).
- **Evidence-level memories** answer *what is relevant here?* (small, sourced observations: "the ops lead said route sync failed twice in June").
- **Long-form synthesis** answers *what is understood?* (an account brief an agent can read in one pass).
- **The graph** answers *what is connected?* (who reports to whom, which site belongs to which account).

All four are written in one save and read back in one call under a token budget. Every agent reads the same record before it acts and writes back what it learned.

For a leader, three things follow. This is where agent quality is decided: the same model writes a generic email or a specific, true one depending on what memory hands it. It is where cost is decided, because agents are billed by what they read. And it does not fix bad identity or bad objectives; it makes those mistakes faster.

## The reader changed

Open a Larkspur Systems account record the way its account managers have for ten years. (Larkspur is the fictional company that runs through this handbook: field-service and fleet software, about 40,000 customer accounts and 250,000 contacts.) Harlow Fleet Services, 400 technician seats. Tier: Gold. Health: Yellow. Renewal: March 31. Below that, 212 activity entries, most of them logged emails.

Priya, the account manager, reads it in thirty seconds. She knows Yellow means the route-sync failures in June, because she set it after the call. She knows the renewal date moved once, and why. She skips the activity log, because she was there for most of it.

Now watch the other reader. This quarter Larkspur runs a renewal agent, a support copilot, and an outbound pilot. Between them they read Harlow's record dozens of times a week, and across the book of business, several hundred account records a minute at peak. None of them was on the call. The renewal agent sees Yellow and does not know who set it, when, why, or whether the problem was fixed. It sees 212 activity entries and either reads all of them, paying for that on every call, or reads the five most similar to its question and misses the one that matters. It sees a renewal date and cannot tell whether it came from the contract, the billing system, or a rep's guess.

The CRM did not get worse. The reader changed.

## Dashboard-era data fails agents in four specific ways

This is not a data quality problem; Larkspur's data is as clean as any mid-market company's. The data was designed for a reader who carried the missing context in their head. Take the head away and four gaps appear.

**Fields without meaning.** "Health: Yellow" is defined by a team's habits. Two agents asked to update it will write two different concepts into the same field. A person resolves the ambiguity by asking a colleague. An agent resolves it by guessing, fluently.

**No provenance.** Without source, observation time, and confidence written down, an agent cannot tell a contract date from a rep's estimate. (Chapters 7 and 11 build the provenance envelope.)

**No narrative.** The activity log records that 212 things happened, not what they mean. The two-minute story Priya would tell a colleague exists nowhere the agent can read it.

**No budget.** A dashboard costs nothing to glance at. An agent pays for every token, and its accuracy degrades as the input grows. Chroma's 2025 study of 18 models found that "performance grows increasingly unreliable as input length grows," even on simple tasks [1]. Earlier, Liu and colleagues showed that models use information in the middle of long inputs worst of all [2]. Anthropic's engineering guidance puts it plainly: context "must be treated as a finite resource with diminishing marginal returns" [3].

That last gap changes the objective of data modeling. Relational design optimized for storage integrity: store each fact once, return exactly the rows that match. Memory for an agent optimizes for **the most task-relevant signal per token**. The discipline of modeling entities, relationships, and constraints transfers almost entirely. The objective does not.

## A prompt is not a memory

The first response most teams try is a bigger prompt: paste the account record, the last twenty emails, and the open tickets, and let the model sort it out. It works in a demo, because a demo asks one question about one customer once.

A prompt is temporary material assembled for one call. Durable memory is organizational state: it persists, can be retrieved selectively, carries source and time, and can be corrected, expired, or deleted. When the support copilot learns on Tuesday that Harlow's route-sync problem is fixed, the prompt the renewal agent assembled on Monday does not change. Nothing in a prompt compounds.

A prompt helps one call. A memory changes the next one.

## Model entities, not dossiers

The first design choice is what the system remembers. The tempting answer is a profile per customer: one big document with everything in it. That becomes brittle quickly, because the things we call "customer" are several different things.

Use a small set of durable entities instead, and let roles and relationships connect them:

| Entity | Larkspur example | Why it is separate |
|---|---|---|
| Person | Dana, Harlow's fleet operations lead; Priya, Larkspur's account manager | A person persists while roles change |
| Organization | Harlow Fleet Services; a regional reseller | Contracts often belong to organizations, not people |
| Role | "Operations lead at Harlow, 2023 to present" | Connects a person to an organization for a period and a purpose |
| Account | Harlow's subscription, 400 technician seats | Billing, usage, and entitlements attach here |
| Deal or case | The 2027 renewal; the June route-sync ticket | Work has its own status, participants, and evidence |
| Contract | The master agreement and its renewal clause | Commercial terms, and what is actually promised, live here |
| Event | A call, a ticket, a week of product usage | Time and sequence change what information means |

"Customer," "champion," "admin," and "former contact" become typed, dated relationships rather than containers. Dana **works at** Harlow; Harlow **owns** an account; Dana **is the admin of** it. When Dana changes jobs, the person persists, the role gets an end date, and the account needs a new admin. In a dossier model, Dana keeps receiving messages written for a job she no longer has.

Every one of these entities needs a resolved identity first (Chapter 9). Memory built on a wrong merge is a very efficient way to tell one customer another customer's renewal terms.

## One record, four ways of knowing

Here is the sentence this chapter turns on. **Memory for agents is not a place to store what you know about a customer; it is the working model of the customer that every agent reads before it acts and writes back to after.**

That model needs more than one shape, because knowledge about a customer comes in more than one shape. A single sales call about Harlow contains all four:

**Typed properties: what qualifies, counts, and governs.** Dana's role is "Fleet Operations Lead." Renewal stage is "Evaluating." Competitors under evaluation: 2. These values have one current state, and other systems need to filter, count, and trigger on them. `stage = 'evaluating' AND renewal_date < '2027-04-01'` must return the right accounts with no model in the loop.

**Evidence-level memories: what is relevant.** "On the September 12 call, Dana said the June route-sync failures cost her dispatchers two days of manual scheduling." That is a self-contained, dated, sourced observation. No schema anticipated it, and it is exactly what a renewal conversation should acknowledge.

**Long-form synthesis: what is understood.** The account brief: history, open risks, stakeholders, what we have promised, what is unresolved. An agent reads it in one pass instead of reconstructing the story from 212 fragments. It is a generated view, not self-authenticating truth: the underlying evidence and the system of record still win on conflict.

**The graph: what is connected.** Dana reports to Harlow's COO, who signed the original contract. Harlow shares a reseller with four other Larkspur accounts. The route-sync ticket is linked to a known defect affecting 60 other accounts. Edges should be typed, directional, dated, and sourced: `reports_to` is not `approves`, and `previously_admin` is not `admin`.

The division of labor is simple: **evidence preserves, properties operationalize, synthesis explains, the graph connects.**

![One piece of raw content, such as a call, email, or ticket, goes through a single save that embeds it for recall, extracts typed properties and atomic memories, and applies write-time gates (resolve pronouns, anchor time, deduplicate). The save writes onto one record per entity, kept in four forms: typed properties answer what qualifies, counts, and governs; evidence memories answer what is relevant here; long-form synthesis answers what is understood; the graph answers what is connected. Evidence preserves, properties operationalize, synthesis explains, and the graph connects, and every agent reads the record in one budgeted call and writes back what it learned.](/images/handbook/ch09-one-save-four-forms.svg)

*Figure 10.1. One save, four forms on one record, each answering a different question.*

This is related to the research taxonomies but not the same. CoALA splits an agent's long-term memory into episodic, semantic, and procedural stores [4]; MemGPT, which became Letta, pages information between context and an external store the way an operating system pages memory [5]. Both concern an agent's memory of its own experience. Unified Customer Memory is many agents' shared memory of a customer: a system of record read by models, not a model that keeps notes.

## If you would put it in a WHERE clause, it is a property

The hardest design question in memory is not which database to use. It is deciding, per piece of knowledge, what to crystallize into a typed property and what to leave as a memory. The rule I use:

> **The Boundary Rule.** If you would put it in a WHERE clause or a chart, it is a property. If you would put it in a paragraph, it is a memory.

Crystallize when the fact is filterable or countable, when it governs a workflow (consent, stage, tier), when it is stable and enumerable, or when another system needs it synced. Leave it as a memory when it is nuanced, explanatory, one-off, or "the why."

Both sides fail when the rule is ignored. Keep everything as prose and "which Gold accounts renewing in Q1 have an open severity-one ticket?" means reading every record with a model, at a model's cost and error rate. Force everything into fields and you get a property called `insights` that agents fill with paragraphs no workflow can test: the dossier again, wearing a schema.

![The Boundary Rule as two columns. Knowledge you would put in a WHERE clause or a chart becomes a typed property: it is filterable or countable, governs a workflow such as consent, stage, or tier, is stable and enumerable, or must sync to another system, for example stage = 'evaluating' AND renewal_date < '2027-04-01'. Knowledge you would put in a paragraph stays an evidence memory: nuanced, explanatory, one-off, or the why, for example Dana's comment on the September 12 call that June route-sync failures cost her dispatchers two days of manual scheduling. Ignoring the rule fails both ways: everything as prose means every filter reads every record with a model, and everything as fields produces an insights property full of paragraphs no workflow can test.](/images/handbook/ch09-boundary-rule.svg)

*Figure 10.2. The Boundary Rule: filter or chart it, make it a property; explain it, keep it a memory.*

Two design details carry most of the weight.

**The property description is the extractor's instruction.** When a model fills a property from a transcript, the description you wrote is the prompt it follows. A good description has four parts: what to capture, where to find it (signatures, call transcripts, order forms), what it is not, and one to three examples. "Renewal risk" with no description gets you a model's mood. "Renewal risk: the customer's stated or strongly implied intent not to renew, from calls, emails, or tickets; not general dissatisfaction with a feature; example: 'we are putting the renewal out to tender'" gets you a field two agents fill the same way.

**Each property declares how it changes and who may write it.** Some values replace (current role); some accumulate (pain points raised). Some may be written by a model (buying stage); some belong to a system of record and must never be overwritten by extraction (contract dates, consent). A model that "corrects" a contract date from a rep's offhand comment is a governance failure that looks like helpfulness.

## One save, several representations

If four forms have to be maintained by four pipelines, they drift apart. The fix is a **unified write**: one save of raw content produces several representations at once. The transcript is stored and embedded for semantic recall. In the same pass, a model extracts typed property values against the schema and atomic memories that no schema anticipated. Documents saved as documents can optionally feed the same extraction.

In the system I built, a save runs one model call per collection of properties plus one for free-text memories: three collections, about four calls per save. That is the price of thorough extraction.

In our own measurements the two extraction modes cover different ground. Across 20 annotated samples, 34 percent of the valuable information was captured by both typed and free-text extraction, 38 percent only by free-text memories, 12 percent only by typed properties, and 16 percent by neither, for combined recall of 82.8 percent [6]. Treat those as internal numbers from a small sample, not a benchmark. The shape of the result is the point: the schema catches what you anticipated, the free text catches what you did not, and neither alone is enough.

My paper reports a much higher figure, 99.6 percent fact recall with complementary dual-modality coverage [20], and the two numbers are not in conflict because they measure different things. The 82.8 percent is the share of the valuable information in 20 annotated samples that either extraction mode captured in the internal experiment above. The 99.6 percent is the paper's fact recall in controlled experiments (N=250, five content types), with typed and free-text extraction combined. Read the first as a small-sample estimate and the second as a controlled-experiment result; neither is a benchmark of your data.

Write time is also where quality is cheapest to buy. Before a memory enters the store it should be self-contained, have its pronouns resolved ("she" becomes "Dana, Harlow's fleet operations lead"), and have relative time anchored ("recently" becomes "September 2026"). It should also be checked against what is already stored. When we ran five overlapping sources about the same entity through a shared store, 83 percent of the candidate memories duplicated something already there [7]. Without deduplication at write time, the store fills with echoes and each echo makes a fact look more important than it is.

Write-time quality gates are a one-time cost. Read-time noise is a tax you pay on every call, forever.

## Documents are memories too

Short-form memory (atomic facts and properties) is precise and cheap to retrieve. Some knowledge is bigger than a fact and smaller than a database: an account brief, a renewal plan, a record of what was promised. Agents want to write these as sectioned notes and revise one section at a time, the way coding agents keep a notes file.

Long-form memory is not a separate store. It is another form on the same record, with the same identity, permissions, and retrieval call. Its value is compression. In one of our experiments, 1.5 million tokens of source material was compacted into about 27,000 tokens of long-form notes, and the agent still had what it needed [8]. That is one internal run, not a general ratio, but it shows why synthesis earns its place: it is the only form an agent can read whole.

Keep synthesis subordinate to evidence. A beautifully written brief that says "renewal is safe" is an opinion with good formatting. It should cite what it rests on and be regenerated when that changes. (Chapter 11 calls that ongoing work dreaming.)

## Retrieval is a conversation, not a query

Once memory is shaped this way, reading it changes too.

The usual approach hands knowledge to an agent by similarity: chunk everything, embed it, and return whatever looks most like the question. That finds text that resembles the query. It does not assemble what the task needs. The fact that matters often shares no words with the question: "should we offer Harlow a multi-year renewal?" needs the open severity-one ticket, the stakeholder change, and the contract's price-protection clause, none of which look like the word "renewal." Similarity is not relevance.

Retrieval for agents has three properties the chat era did not need.

**One call, several sources, one identity.** A request for Harlow should return typed properties first (they are authoritative), then the most relevant memories, document sections, and graph neighbors, all resolved to the same entity, with citations to specific facts. An agent that calls four stores and joins the results itself rewrites that plumbing in every workflow and, sooner or later, joins two slightly different Harlows.

**A token budget, not a result count.** More is not better. In our end-to-end evaluation of personalized sales emails (10 prospects, 3 runs each, scored on a 100-point rubric), quality rose from 79.5 with no memory to about 86 with governed memory, and most of that gain came from roughly the first seven well-chosen memories per entity; adding more did not help [9]. Small sample, one task, internal rubric. The paper's larger controlled experiments found the same shape: output quality saturated at approximately seven governed memories per entity [20]. The durable lesson matches the context research above: the bottleneck is curation, not capacity. Store generously; inject sparingly.

![A budgeted read is one call scoped to one entity that returns typed properties first because they are authoritative, then the most relevant memories, relevant document sections, and linked graph neighbors, stopping at the token budget while everything else stays stored; the next step in the session returns new items, not repeats. Below, drawn to scale, the chapter's cost arithmetic for 40,000 accounts read 5 to 20 times a month: the full history at 30,000 to 80,000 tokens per read is 6 to 64 billion input tokens a month, while a curated payload of 3,000 to 6,000 tokens is 0.6 to 4.8 billion.](/images/handbook/ch09-budgeted-read.svg)

*Figure 10.3. Store generously, inject sparingly: a budgeted read costs about an order of magnitude less than reading the full history.*

**Continuation across steps.** An autonomous agent does not retrieve once. The first read about Harlow reveals that the admin changed; that makes the contract relevant; the contract reveals a notice period; the plan changes. Retrieval becomes an action reasoning selects, so the memory layer should remember what it already delivered about this record and return the next most useful items, not the same ones again. In one research agent we instrumented, 23 percent of output tokens went to the agent explaining to itself why it had already seen repeated content [10]. The fix was not a better ranker. It was a session that knew what had been shown. In my paper's controlled experiments, progressive delivery across steps cut delivered tokens by 50 percent [20].

The retrieval I built exposes this as modes, and the rule is the cheapest mode that answers: a record scan with no model call, a deterministic property filter, a semantic brief that cites the memories it used, or a session that continues where the last read stopped. "Which Gold accounts renew in Q1?" should never cost a model call.

## Four forms do not need four databases

None of this needs four databases. The self-hosted memory system I built keeps all four forms, plus the governance documents, in one Postgres database with the pgvector extension. Every prior property value is kept with the dates it was valid, so a superseded renewal date is retrievable, not lost (Chapter 11). One database means one backup, one permission model, and one place where "Harlow" is resolved, which is most of what makes a single budgeted read possible.

The schema installs as a bundle: entity types, property collections with descriptions and update semantics (replaceable or append-only), relation types, rules that turn properties into edges (a saved competitor list becomes `competes_with` links), and seed guidelines. The Boundary Rule becomes a reviewable artifact instead of a habit. Reinstalling is harmless, but in my system graph rules cannot be edited after install, so test the bundle in a throwaway namespace first.

Agents reach it through the Model Context Protocol. A session opens with an orientation call that returns the schema, the index of governance documents (names, never contents), and only the tools this agent's key may call. A read-only agent's write tools are absent, not merely refused, and autonomous agents are never offered delete. Each turn then runs one loop: recall, retrieve the rules, act, write back.

A governed personalization engine I built on memory like this used a short list: exact properties for campaign IDs, scores, and status flags; free-text memories for research; batch ingestion; semantic recall; a compiled digest per account; property history for audit; deterministic filters for QA and exports; and bulk updates to write scores back. The hard part is keeping it on one record.

## Memory is one of three questions

Memory is often asked to do jobs that belong elsewhere. Three questions keep them apart. *What is relevant?* is retrieval over a corpus (product docs, past proposals). *What do we know about this customer?* is memory: entity-scoped, persistent, written by many agents. *What are the rules?* is governance: pricing, brand, consent, contact limits, applied whether or not the agent thought to ask.

The common mistake is storing rules inside customer memory; when Larkspur's discount policy changes, it should change in one place, not in 40,000 records. The opposite mistake is serving rules by similarity search, where a draft policy outranks the approved one because it uses the right words. Retrieval ranks by similarity; governance has to rank by authority (Chapter 18). In my paper, tiered governance routing reached 92 percent routing precision [20]. That is a different measure again: of the governance documents routed to a task, the share that were the relevant ones. It says nothing about recalling customer facts.

## What this does not do

An argument for memory that lists only its strengths should not be trusted, so here are the limits, my own numbers included.

**Memory does not fix identity or objectives.** If Chapter 9's identity resolution merged two Danas, unified memory will faithfully give both of them one history. If the decision layer is optimizing for sends instead of outcomes (Chapter 14), better memory produces better-targeted versions of the wrong message.

**The benchmarks do not measure what you need.** In 2025, memory vendors fought publicly over the LoCoMo benchmark [11]. Mem0's paper claimed a 26 percent relative improvement over OpenAI's memory, with large latency and token savings [12]. Zep replied that a simple full-context baseline, just feeding the whole conversation to the model, scored about 73 percent against Mem0's best of about 68, that LoCoMo "lacks questions designed to test knowledge updates," and that Zep's own corrected score was 75.14 [13]. Mem0's CTO in turn alleged an error in Zep's earlier headline figure [14]. Then Letta ran a plain agent with file tools and no specialized memory system and reported 74.0 percent with a small model [15]. All of these are vendor claims, and I report them as claims. The architecture I published scores in the same band: 74.8 percent overall on LoCoMo [20], which is part of why I stopped leading with the number. When a filesystem, a full-context baseline, and several dedicated products land within a few points of each other, the benchmark is mostly measuring context management on one conversation. It is not measuring what breaks in customer memory: many writers, facts that change, and questions that filter across thousands of accounts. LongMemEval is closer, because it tests knowledge updates and abstention, and it found that commercial assistants and long-context models showed a 30 percent accuracy drop on sustained interactions [16]. Build your evaluation from your own freshness and contradiction cases.

**The strongest opposing view deserves an honest answer.** Context windows reached a million tokens in 2024 [17] and keep growing, and Letta's result suggests a filesystem is all you need; so skip the architecture, keep good files, and let the model read. I agree with Letta's conclusion that "memory is more about how agents manage context than the exact retrieval mechanism used" [15]. For one agent on one conversation or one codebase, files and a long window are often right, and I would not overbuild. The argument breaks on the customer. "Which Gold accounts renewing next quarter mentioned a competitor?" is a filter across 40,000 records, not a reading task. A file has no field-level permissions, no system-of-record protection, and no wall between one customer and the next. And a long window does not decide which of two contradictory facts is current; it hands the contradiction to the model on every call, in a context the research [1][2] says it reads less reliably as it grows.

Cost makes the same point with arithmetic. Suppose each of Larkspur's 40,000 accounts is read by agents 5 to 20 times a month. Reading the full history at 30,000 to 80,000 tokens per read is 6 to 64 billion input tokens a month. A curated payload of 3,000 to 6,000 tokens is 0.6 to 4.8 billion. The assumptions carry the weight (read frequency and history length vary widely), and prompt caching changes the price of repeated reads but not the attention cost of a bloated context. The order-of-magnitude gap is the durable part.

**In-model memory may change the picture.** Research such as Titans explores models that learn to memorize at test time [18]. That is a lab direction, not a production option for shared, governed customer records, and I label it a forecast. Even if it matures, an organization will still need a record it can inspect, correct, permission, and delete.

## At scale

At one agent and a hundred customers, almost any memory approach works. At Larkspur's scale, three things change.

**Writers multiply.** Enrichment, support, sales, telemetry, and humans all write about the same account. Without deduplication and precedence rules, the record becomes an argument. Every property needs an owner and a rule for who wins on conflict. The customer is a writer too: a "not interested" pressed in one surface must land here so every surface honors it, or the control reads as broken (Chapter 3). Synthetic research respondents never write here at all.

**Isolation becomes architectural.** Similar accounts in the same industry have similar embeddings. Keeping Harlow's memories out of another customer's retrieval must be enforced by scoping every read and write to a resolved entity identifier, not by semantic distance. In my paper's controlled test, entity scoping leaked nothing across 500 adversarial queries [20]: one test, not a guarantee.

**Schemas become living documents.** Properties nobody fills, or that two agents fill differently, show up only across thousands of records. Measure extraction per property and revise descriptions the way you revise code.

A memory precise enough to serve a customer this well is precise enough to pressure them, which is why governance and privacy sit on this layer rather than beside it (Chapters 18 and 19). Memory is, in the end, a polite word for a model of someone.

## Failure story: The $450K email

The deal was worth $450,000. (The scenario is a composite, told here at Larkspur; the pattern is one I have seen repeatedly.)

In month one, Larkspur's enrichment agent found that a customer's COO was evaluating two competing platforms and had complained publicly about route-sync reliability. It stored this in its own workspace. In month two, the outbound agent sent the account a segment-level email about "modernizing your fleet operations." In month three, the support copilot resolved a severity-one route-sync ticket for the same account and closed it. In month four, the scoring model saw no recent engagement and lowered the account's priority. In month five, a renewal agent pitched the fixed route-sync feature as new. In month six, the customer chose a competitor, and nobody at Larkspur could say why.

Four agents. One account. Zero shared context. Each learned something true; none of it reached the others, because each workflow's memory belonged to the workflow, not to the customer. The difference between the email the COO deleted and the one she would have answered was not model quality. It was whether the agent could read what the organization already knew. The anti-pattern: **Siloed Agent Memory**.

## Patterns

**Memory Layers.**
*Problem:* one memory shape cannot serve filtering, recall, explanation, and connection. *Forces:* agents need exact answers and nuance; tokens are expensive. *Solution:* keep four forms on one record: typed properties, evidence-level memories, long-form synthesis, and a graph, each answering its own question. *Tradeoffs:* more design work up front; some intentional redundancy between forms (overlap is cheaper than loss).

**Boundary Rule.**
*Problem:* teams either trap facts in prose or force nuance into fields. *Forces:* filterability versus expressiveness; schema maintenance cost. *Solution:* WHERE clause or chart, property; paragraph, memory. Every property carries a four-part description, update semantics, and write permission. *Tradeoffs:* schemas need owners and revision; some facts will live in both forms for a while.

**Unified Write, Budgeted Read.**
*Problem:* separate pipelines per form drift apart; agents over-read. *Forces:* extraction cost, consistency, context limits. *Solution:* one save produces all representations with quality gates and deduplication; one read returns properties first, then curated memories, documents, and graph neighbors under a token budget, with session continuation. *Tradeoffs:* extraction cost moves to write time (usually a good trade, since each record is written once and read many times).

## Leader questions

1. When two of our agents act on the same customer, can the second one see what the first one learned? Show me.
2. For our five most important customer fields, who owns each one, where does its value come from, and can a model overwrite it?
3. How many tokens does an agent read to answer one question about one account, and how do we know that number is not mostly noise?
4. Where do our rules live: in one governed place, or copied into prompts and customer records?
5. How do we test whether memory stays correct when facts change, not just whether it recalls them?

## Build checklist

- [ ] Every entity has a resolved identity before anything becomes memory (Chapter 9).
- [ ] Entities are modeled separately from roles; roles and relationships are typed and dated.
- [ ] Each property has a type, a four-part description, update semantics (replace or append), and a write permission (model-writable or system-of-record only).
- [ ] One save produces properties, atomic memories, and (optionally) document content in a single pass.
- [ ] Write-time gates: coreference resolved, time anchored, self-contained, deduplicated against the store.
- [ ] Every memory carries source, observation time, and confidence.
- [ ] One retrieval call returns properties first, then memories, documents, and graph neighbors, scoped to one entity, under a stated token budget.
- [ ] Multi-step agents continue a retrieval session instead of re-reading the same items.
- [ ] Organizational rules live in governance, not in customer records.
- [ ] Reads and writes are isolated by entity identifier, not by embedding similarity.

## Metrics to watch

- **Tokens read per decision**, and the share that were cited or used.
- **Cross-agent reuse:** the share of decisions that used a memory written by a different workflow.
- **Property fill rate and agreement** per property (null rates, disagreement between extractors or with human labels).
- **Duplicate rate at write time**, trending over time.
- **Freshness accuracy** on your own test set of changed facts (role changes, fixed issues, moved dates).

## Reader Q&A

**We already have RAG over our CRM notes. Isn't that memory?**
It is retrieval. It makes the agent well read. It does not maintain a customer's current state, decide which fact is current, or give other systems typed values. Once you add extraction, write-back, deduplication, and supersession, you have built memory that uses retrieval as one component.

**Do we need a graph database to start?**
No. Start with typed, dated, sourced relationships on the record. Move to a dedicated graph when traversal questions become frequent. GraphRAG shows graphs help most on questions spanning a whole corpus [19]; most early customer questions are about one account and its neighbors.

**How many properties should we define?**
Fewer and better than you think: the fields that drive your first three decisions (Chapter 6). Free-text memories will catch what you did not anticipate.

**Is this just a CDP with a vector index?**
A CDP unifies identity and events for activation. Unified Customer Memory sits on that identity and adds extracted evidence, typed properties with extraction instructions, synthesis, and budgeted retrieval. Many companies will keep both.

## For your AI

```yaml
chapter: 10
title: "Memory Built for Agents"
concepts:
  - name: Unified Customer Memory
    alias: Unified Record Memory
    definition: "One record per entity that unifies everything agents can know about it, kept as typed properties, evidence-level memories, long-form synthesis, and graph relations, written in one save and read in one budgeted call by every agent."
  - name: The reader changed
    definition: "Customer data was designed for humans who carry missing context in their heads; agents do not, and pay per token, so data must carry meaning, provenance, narrative, and a budget."
  - name: Memory Layers
    definition: "Properties answer what qualifies and governs; memories answer what is relevant; synthesis answers what is understood; the graph answers what is connected."
  - name: Boundary Rule
    definition: "If you would put it in a WHERE clause or a chart, it is a property; if you would put it in a paragraph, it is a memory."
  - name: Unified write
    definition: "One save of raw content produces embeddings, typed property values, and atomic memories in a single extraction pass, with quality gates and deduplication."
  - name: Property description as extractor instruction
    definition: "Each property's description (what to capture, where to find it, what it is not, examples) is the prompt the extractor follows."
  - name: Retrieval as conversation
    definition: "Multi-step agents continue a retrieval session anchored to an entity, receiving new items rather than repeats, under a token budget."
  - name: Four forms, one database
    definition: "All four forms, plus governance documents and property history, can live in one Postgres database with pgvector, giving one backup, one permission model, and one identity resolution."
  - name: Three questions
    definition: "What is relevant (retrieval), what do we know about this entity (memory), what are the rules (governance) are separate problems with separate infrastructure."
decision_rules:
  - if: "a fact will be filtered, counted, charted, synced, or used to gate a workflow"
    then: "model it as a typed property with a description, update semantics, and write permission"
  - if: "a fact is nuanced, explanatory, or one-off"
    then: "store it as an evidence-level memory with source, time, and confidence"
  - if: "a field is owned by a system of record (contract dates, consent)"
    then: "mark it not model-writable; extraction may propose but never overwrite"
  - if: "an organizational rule would be copied into many customer records"
    then: "store it once in governance and route it to agents by authority, not similarity"
  - if: "an agent retrieves more than once about the same entity in one task"
    then: "continue the retrieval session so it receives new items, not repeats"
  - if: "a question can be answered by filtering typed properties"
    then: "use a deterministic filter read with no model call; reserve model synthesis for questions that need it"
  - if: "memory quality is being evaluated"
    then: "test on your own changed-fact and contradiction cases, not only public recall benchmarks"
assessment_questions:
  - "Which agents or workflows act on the same customers today, and can each see what the others learned?"
  - "Which customer fields drive your first decisions, who owns each, and which may a model overwrite?"
  - "How many tokens does an agent read per decision, and what share is used?"
  - "Where are organizational rules stored, and how many copies exist?"
  - "Is identity resolved before content becomes memory?"
patterns: [Memory Layers, Boundary Rule, Unified Write Budgeted Read]
anti_patterns: [Siloed Agent Memory, The Dossier, Modality Monoculture, Prompt as Memory, Rules in Records]
maturity_dimension: identity_and_memory
```

## References

1. Hong, Troynikov, Huber. "Context Rot: How Increasing Input Tokens Impacts LLM Performance." Chroma, 2025-07-14. https://www.trychroma.com/research/context-rot
2. Liu et al. "Lost in the Middle: How Language Models Use Long Contexts." TACL 2024; arXiv:2307.03172. https://arxiv.org/abs/2307.03172
3. Anthropic. "Effective context engineering for AI agents." 2025-09-29. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
4. Sumers, Yao, Narasimhan, Griffiths. "Cognitive Architectures for Language Agents" (CoALA). TMLR 2024; arXiv:2309.02427. https://arxiv.org/abs/2309.02427
5. Packer et al. "MemGPT: Towards LLMs as Operating Systems." arXiv:2310.08560, 2023. https://arxiv.org/abs/2310.08560
6. Taheri, H. "Dual Memory: Why You Need Both Free-Text Facts and Typed Properties." hamedtaheri.com, 2026-03-14. Internal experiment, 20 samples. https://hamedtaheri.com/articles/dual-memory-free-text-and-typed-properties/
7. Same source as 6 (five-source deduplication experiment). Internal.
8. Taheri, H. "Unified Record Memory, part 3" (LinkedIn, 2026-07). Internal experiment, single run.
9. Taheri, H. "Seven Memories Per Entity Is All You Need." hamedtaheri.com, 2026-03-14. Internal evaluation, 10 prospects x 3 runs. https://hamedtaheri.com/articles/seven-memories-per-entity/
10. Taheri, H. "Retrieval Is a Conversation, Not a Query." hamedtaheri.com, 2026-06-03. Internal instrumentation. https://hamedtaheri.com/articles/retrieval-is-a-conversation/
11. Maharana et al. "Evaluating Very Long-Term Conversational Memory of LLM Agents" (LoCoMo). ACL 2024; arXiv:2402.17753. https://arxiv.org/abs/2402.17753
12. Chhikara et al. "Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory." arXiv:2504.19413, 2025. Vendor claims. https://arxiv.org/abs/2504.19413
13. Zep. "Lies, Damn Lies, & Statistics: Is Mem0 Really SOTA in Agent Memory?" 2025-05-06. Vendor claims. https://blog.getzep.com/lies-damn-lies-statistics-is-mem0-really-sota-in-agent-memory/
14. Mem0 CTO, GitHub issue getzep/zep-papers #5, 2025-05-08. Contested claim. https://github.com/getzep/zep-papers/issues/5
15. Letta. "Benchmarking AI Agent Memory: Is a Filesystem All You Need?" 2025-08-12. Vendor claims. https://www.letta.com/blog/benchmarking-ai-agent-memory
16. Wu et al. "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory." ICLR 2025; arXiv:2410.10813. https://arxiv.org/abs/2410.10813
17. Google. "Our next-generation model: Gemini 1.5." 2024-02-15 (1 million token context in limited preview). https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/
18. Behrouz, Zhong, Mirrokni. "Titans: Learning to Memorize at Test Time." arXiv:2501.00663, 2025. https://arxiv.org/abs/2501.00663
19. Edge et al. "From Local to Global: A Graph RAG Approach to Query-Focused Summarization." arXiv:2404.16130, 2024. https://arxiv.org/abs/2404.16130
20. Taheri, H. "Governed Memory: A Production Architecture for Multi-Agent Workflows." arXiv:2603.17787, 2026-03-18. Author's own controlled experiments (N=250, five content types) and LoCoMo result. https://arxiv.org/abs/2603.17787

---

# 11. Keeping Memory True

*The question: How does memory stay accurate, fresh, and traceable?*

> **Questions this chapter answers**
> - Our AI remembers our customers. How do we stop it remembering things that are no longer true?
> - When two systems disagree about a customer, which one wins?
> - How do we know where a fact came from, and whether we may still use it?
> - What should expire, and what should never have been stored?
> - Everyone is talking about "dreaming." What is it, and does it apply to customer data?

## The short answer

Memory does not decay on its own. It accumulates by default and is corrected only on purpose, so a memory nobody maintains does not fade into irrelevance. It gets larger, more confident, and more wrong at the same time.

Keeping memory true takes four disciplines, and none of them is a model upgrade. Every property gets a **freshness policy**: how long a value can be trusted before it must be re-verified or stop being used. Every fact carries a **provenance envelope** (introduced in Chapter 7, extended here): where it came from, when it was observed, when it was valid, which extractor produced it, and whether it is a fact or an inference. **Contradictions are resolved by precedence**, not by whichever write arrived last. And the memory is **curated continuously in the background**: duplicates merged, contradictions surfaced, summaries rewritten, stale facts expired.

The field now calls that fourth discipline "dreaming." OpenAI, Anthropic and Letta all use the word, building on research into sleep-time compute, and they apply it to an agent's or an assistant's own memory. This chapter applies it to the customer: shared memory that many agents and teams read, sitting on resolved identity, holding typed facts with their sources, and changed only through reviewable, non-destructive steps.

For a leader, the test is simple. Pick one customer fact your AI used this week and ask: where did it come from, when was it last true, and what would have told us if it stopped being true? If nobody can answer the third, the memory is not being kept. It is being hoarded.

## The customer who changed jobs and the memory that did not

Larkspur Systems (a fictional company, a composite of many real ones) sells field-service and fleet software. For three years its main contact at a regional HVAC contractor was Elena, the fleet operations manager. Her calls with Larkspur were about technician scheduling, route sync, and the dispatch board.

Last spring Elena was promoted to run procurement, and the Larkspur renewal became hers to sign. An enrichment feed noticed the new title in May, and the CRM's job-title field changed.

Nothing else did. The account summary Larkspur's agents read before every touch still described Elena as "the operations lead who cares about dispatch efficiency." Forty call-note memories said the same thing, correctly, about the past. The assistant that drafted her renewal outreach in November weighed forty consistent memories against one changed field and wrote her a warm, specific, well-grounded email about the new dispatch board.

Every sentence was traceable to a real source. It was still wrong: the person it was written for no longer exists. I call this **The Stale Self**: the system stays loyal to a version of the customer that has expired.

Identity resolution, extraction and retrieval all worked. A correction landed in one representation, never propagated, and no process existed whose job was to notice.

Propagation also fails in milliseconds. In a governed personalization engine I built, a successful write did not always mean the next step could read it, so important state gets a bounded read-after-write check before anything renders from it. A correct value that arrives late still produces a wrong page. The self-hosted memory system I built taught the harsher version: under a saturated database, a save once reported success while writing nothing, and a failed embedding once returned success with an empty vector. Both now fail loudly. Success must mean landed.

## Memory goes wrong in four ways, and only one is the model's fault

When personalization gets a fact wrong, the instinct is to blame hallucination. That is usually not the reason. In customer memory, errors enter through four doors: **the world changes** (the fact was true when stored); **sources disagree** (CRM, enrichment vendor and support transcript each say something different); **extraction errs** (a model stored the wrong value, or an inference as a fact); and **derived views lag** (summaries, scores and embeddings were computed from facts that have since changed).

Only the third involves a model misreading anything, and its fix is usually a better property description. The other three are maintenance, which memory tooling tends not to ship. Anthropic's memory tool documentation is candid: the model can create, update and delete memory files, while expiry appears under security considerations as your job, with the advice to "periodically delete memory files that haven't been accessed in a long time" [1]. LongMemEval, the benchmark that explicitly tests *knowledge updates*, found assistants losing roughly 30% accuracy across sustained interactions [2]. And OpenAI's own illustration, from the launch of ChatGPT's Dreaming feature, is a memory still saying "You're going to Singapore in July" after the trip is over [3].

The writing is automatic. The forgetting is your job.

## Every fact has a half-life

A freshness policy states, per property, how long a value can be relied on and what happens when that time runs out. Different facts lose trustworthiness at different rates, and the rate belongs to the kind of fact, not to the database.

People ask me for the "decay rate" of B2B contact data. The often-quoted 22.5% a year has no primary source I can find, so I do not use it [4]. The defensible anchor is labor data: the U.S. Bureau of Labor Statistics reported median employee tenure of 4.1 years in January 2026, with 20.6% of wage and salary workers at their employer for a year or less [5]. Tenure counts employer changes, not role changes. Elena never changed employer, so she would not appear in that statistic at all. Treat it as a floor on how fast contact records go stale.

A starting freshness table for Larkspur. The windows are illustrative, to be calibrated against your own correction data:

| Property | Volatility | Trust window | When the window expires |
|---|---|---|---|
| Legal company name, domain | Low | 12 to 24 months | Re-verify on next enrichment; keep using |
| Contact's employer | Medium | 6 to 12 months | Re-verify before outreach that assumes it |
| Contact's role | Medium to high | 3 to 6 months | Stop naming the role; write for the account |
| Stated priorities, current initiative | High | 60 to 90 days | Phrase as history ("in spring you mentioned...") or drop |
| Open support escalation | Very high | Hours to days | Re-read from the system of record at decision time |
| Contact preferences, opt-outs | None | Until the person changes it | Never expire silently |

The last column is a *behavior*, not a deletion: most expired facts change how they are used before they are removed. And some facts never go stale on a timer; an opt-out does not weaken because nobody has looked at it for a year.

That is where I part ways with access-based expiry. Deleting what has not been read in a long time is a sensible default for an agent's working notes [1]. For customer memory it is backwards in both directions: a rarely read fact (a do-not-call request) can be the most important thing on the record, and a frequently read fact can be the stalest. Expire by the kind of fact and its evidence, not by traffic.

The rule that falls out is the handbook's accuracy standard in operational form: **if a fact used to personalize is older than its freshness policy, re-verify it or fall back to an honestly general statement.**

## A fact without a source is a rumor with a timestamp

Here is the sentence this chapter turns on: **you cannot keep a memory true if you cannot trace each fact back to where it came from.** Freshness, precedence, correction, deletion and curation all depend on it.

Each material fact already carries the **provenance envelope** from Chapter 7: route, source and source reference, observation time, method, confidence, corroboration, permitted purposes, allowed surfaces, and a review date. Keeping memory true adds these fields to it:

| Field | What it answers |
|---|---|
| `recorded_at` | When did it enter memory? (Chapter 7's `observed_at` says when we saw it.) |
| `valid_from`, `valid_to` | When was it true in the world? |
| `extractor` | Which model, prompt and schema version turned the source into this value? |
| `kind` | Is this value a fact, or a model's inference or summary derived from other facts? (Chapter 7's `route` says how it arrived.) |
| `sensitivity` | Does it need stricter handling than its purposes alone imply? |
| `status` | Current, superseded, disputed, or under correction? |

![The provenance envelope from Chapter 7 carries route, source and source_ref, observed_at, method, confidence, corroborated_by, permitted_purposes, allowed_surfaces and review_after: where a fact came from and where it may be used. Keeping memory true adds six fields to it: recorded_at (when it entered memory), valid_from and valid_to (when it was true in the world), extractor (model, prompt and schema version), kind (fact, inference or summary), sensitivity (whether it needs stricter handling), and status (current, superseded, disputed or under correction).](/images/handbook/ch10-provenance-envelope.svg)

*Figure 11.1. Chapter 7's envelope says where a fact came from; the extension says when it was true, who extracted it, and whether it still stands.*

Separating when a fact was true from when the system learned it is not pedantry: Zep's temporal knowledge graph is built on exactly this bi-temporal model, and when a new fact contradicts an old one it *invalidates* the old edge by closing its validity window rather than deleting it [6]. W3C's PROV model offers a standard vocabulary for the same idea [7]. Truth has a time, and so does knowledge of it.

The `extractor` field is the one teams skip and the one that pays back first. When a model or schema change degrades one property, the extractor version tells you exactly which values to replay. Without it, every correction is a full re-ingestion or a guess.

## Append or replace is a decision about truth

Every property in a typed memory schema answers a quiet question: when a new value arrives, does it replace the old one or join it? **Replace** fits properties with one current value: role, account owner, plan tier, renewal date. **Append** fits histories: objections raised, products evaluated, events attended.

The mistake is treating replace as overwrite. In a customer system of record, replace should mean *supersede*: the old value's validity closes and it stays as history. Elena's old role is not wrong. It is last year's truth, and it lets the system say "you ran dispatch, so you know what the board costs to operate," which is what makes a renewal conversation feel continuous rather than robotic.

![A timeline of Elena's role at Larkspur (fictional) under two update rules. Under overwrite (last write wins), the new value "leads procurement" arrives in May and the old value "fleet operations manager" is deleted, leaving no history and no rollback. Under supersede, the old role's validity window closes at spring and it is kept as history, while the procurement role is current from spring, even though memory only recorded it in May, ahead of the November renewal outreach.](/images/handbook/ch10-supersede-not-overwrite.svg)

*Figure 11.2. Supersede closes the old value's validity window instead of deleting it: the old role is last year's truth, not an error.*

Decide the semantics per property, in its definition. A property whose update rule nobody chose fails silently in one direction or the other.

Documents need the same rule. In the memory system I built, an agent's document update that named no mode once replaced the whole body, and the history the documentation promised held only the mode, not the old text. Now agent updates append by default, the replaced body is snapshotted in the same transaction as any overwrite, and a failed snapshot fails the update.

## When sources disagree, precedence decides, not the latest write

Last-write-wins is the default in most pipelines and the wrong default for customer memory. The latest write is often the least authoritative: a nightly enrichment job, an agent's inference from an email signature. Resolve contradictions with Chapter 7's Signal Ladder, applied per property, rather than a fixed ranking of sources:

1. **Purpose gates first.** A value not permitted for this use is out, however good it is.
2. **Authority depends on the kind of fact.** The system of record owns transactions and contract terms (billing for plan tier, contracts for renewal date). The person's declaration owns preferences. The person's own recent artifacts (a signature, a reply, a changed login domain) own circumstances such as role and employer. Third-party values wait until something first-hand agrees with them, and an inference may inform a decision but never outranks a first-hand source.
3. **Freshness can override authority for properties that change.** This is where the freshness windows above do their work: authority holds until its window expires, then it must be re-verified or yield. A fresh first-hand artifact can beat a stale declared title; an inferred channel preference never beats a declared one, however recent.
4. **Corroborated confidence breaks ties.**

Three rules keep this honest. **A contradiction is a signal**, not only a conflict: a fresh third-party title contradicting a two-year-old declared role most likely means the person's situation changed, which is exactly when the summary should be rewritten. **Count independent sources, not agreeing writers**: three agents repeating one unverified claim are one source echoing. **Generated content is never its own source**: a summary the system wrote must not be re-ingested as evidence for the facts it summarized.

How well does automated conflict handling work? In the evaluation I published of the governed memory system I build, across 30 synthetic conflict pairs (a company changing its database, a contact switching roles), the fresh fact appeared in 83.3% of answers, only the fresh fact in 33.3%, and the stale fact alone once [8]. Most of the gap was acceptable: answers led with the new fact and mentioned the old one as trajectory. It is a small synthetic set, an upper bound, not a production rate. The lesson: detection is tractable; what the answer *should* say about history is a product decision.

## Dreaming: curating the customer while nobody is asking

Policies say what is true. Something still has to find the entities whose memory changed, reconcile them, and rewrite the views that depend on them. The field has converged on a name for that work, and neither the name nor the mechanism is mine. Credit, in order:

- **Reflection.** Generative Agents (Park et al., 2023) periodically synthesized an agent's memory stream "into higher-level reflections" [9].
- **Sleep-time compute.** Lin et al. (2025) let models "think offline about contexts before queries are presented," cutting test-time compute about 5x at equal accuracy on two reasoning benchmarks and average cost per query 2.5x when amortized across related queries; the benefit tracked how predictable the queries were [10].
- **Sleep-time agents.** Letta (2025) made the primary agent unable to edit its core memory and gave that job to a background agent [11], later adding git-versioned memory and the phrase "agent dreaming" [12].
- **Dreams.** Anthropic's Managed Agents research preview (announced May 19, 2026) reads a memory store and up to 100 past sessions and writes a *new* store, duplicates merged and stale or contradicted entries replaced; the input is never modified, so the output can be reviewed or discarded [13]. Anthropic reports Harvey's completion rates rose about 6x in its tests [14].
- **ChatGPT Dreaming** (June 4, 2026) synthesizes across a user's conversations and rewrites what the assistant remembers, turning the Singapore plan into "You went to Singapore in July 2026" [3].

The biological metaphor is only a metaphor: complementary learning systems theory pairs fast capture of episodes with slow extraction of structure, and sleep appears to help consolidate memories [15][16]. The useful part is the division of labor: capture fast, reorganize when nobody is asking.

Every system above dreams about *one agent* or *one assistant user*: one memory, one owner, one reader. Customer memory is a shared system of record that dozens of agents and several teams read and write, about people who never open the tool. **The field taught the agent to dream about itself. The customer is the harder subject, and the one where the value is.**

Applying dreaming to the customer adds five requirements:

1. **It runs across writers**, not one agent's notes.
2. **It sits on identity resolution.** Consolidating memory for a wrongly merged entity makes the error permanent and fluent. Curation must respect merge confidence (Chapter 9). In mine, deterministic rules propose duplicate pairs, a model only judges them, verdicts below 0.9 confidence are refused, and every merge can be undone for 90 days by default [21].
3. **It writes typed properties with provenance**, not only prose. A superseded `role` with a closed validity window is what the decision layer and governance can act on.
4. **It is governed.** What a pass may change is policy: merge duplicate observations freely, propose a role change with evidence, never touch consent, contract terms or system-of-record fields.
5. **It is non-destructive and reviewable.** It proposes a new state as a diff that can be reviewed and rolled back.

The **dreaming cycle** as I recommend building it, and as I built it into a self-hosted memory system (dreaming shipped there on July 30, 2026, after the releases above; the controls that make it safe to leave unattended followed on August 11 [21]):

1. **Select** entities with new evidence, facts past their window, or open contradictions. Never re-dream the whole base by default. Mine runs on a schedule when the job queue is idle, yields to live requests, and re-dreams a record only when it changed or the job's objective did. Curation's own writes never trigger curation. A scope filter must fail closed: mine once silently ignored an unrecognized collection filter and ran against the whole organization; an unknown scope now fails the run.
2. **Gather** the entity's properties, recent memories, summary, and their provenance, plus a preamble: which company it works for (the organization's own context documents), today's date, which is what lets it write "ran fleet operations until spring" instead of "runs," and the stage it is running at. A missing context document does not stop the pass; the miss is reported.
3. **Consolidate** duplicate observations into one memory with all its sources attached.
4. **Reconcile** contradictions with the Signal Ladder, per property; mark the unresolvable as `disputed`.
5. **Re-synthesize** the summary from current facts, phrasing superseded facts as history.
6. **Expire** what is past its window by superseding it. Deletion does not happen here: the curator I built is never given a delete tool, and erasure runs through its own path.
7. **Propose, gate, apply.** A new curation job only proposes until an operator promotes it. After that, low-consequence changes apply automatically; high-consequence ones queue for review or sit outside the curator's reach; every change is logged with the cycle that made it.

![The dreaming cycle for customer memory runs in seven steps. It selects entities with new evidence, expired facts or open contradictions; gathers their properties, memories, summary and provenance; consolidates duplicates while keeping all their sources; reconciles contradictions with the Signal Ladder per property, marking the unresolvable as disputed; re-synthesizes the summary from current facts with superseded facts phrased as history; and expires what is past its window by superseding it, leaving deletion to a separate erasure path. The final step proposes a diff and gates it: new jobs only propose, low-consequence changes apply automatically, high-consequence ones queue for review or stay out of reach, and every change is ledgered and attributed. Superseded values stay as history, so every pass can be reviewed or rolled back.](/images/handbook/ch10-dreaming-cycle.svg)

*Figure 11.3. Dreaming about the customer curates only changed entities and ends in a gated, reviewable diff, never an overwrite.*

Building it taught me three things about step 7. **A job earns the right to write.** A new one only proposes, five records a pass; after a completed pass an operator may promote it to a pilot (ten records, three writes each), then to full. You approve the judge, not each verdict. An emergency stop drops a job back to propose-only at once, but it disarms writes, not spend: a job still on a schedule keeps running, and consuming model tokens, until someone also disables it. **The gate lives in code, not in the prompt.** Write permission is checked in the call path on every write, so a model cannot talk its way past it and a stop lands mid-run. Close every path around the gate, not only the main one: a pass scheduled through the general scheduler without a job definition would have run with none of the controls, so that path now refuses it. And the curator writes through identity, never raw ids, after an early pass let a model put an email in an id field and create an orphan record. The system, not the model, stamps each write with its job and run, and a ledger records every applied, proposed and refused write by property name, never value. **Correct, do not pile.** A curator that appends a note beside a wrong fact makes memory longer, not truer. Mine corrects the current value and closes the old one's validity window: the supersede rule, applied by the curator.

Run the Stale Self through it. The May title change selects Elena. Reconcile finds her declared role past its window, supersedes it, and flags the change as material. Re-synthesize writes: "Elena ran fleet operations until spring and now leads procurement; she owns the renewal." The November email is written for the person who exists.

## Forgetting is a feature you have to build

The GDPR's storage limitation principle requires personal data to be kept in identifiable form no longer than its purposes need [17]; Chapter 19 covers the law. The engineering point: **deletion must cascade through every derived representation.** When a source is deleted or corrected, the properties extracted from it, the memories citing it, the summary built from them and the embeddings of all three must follow. A system that deletes the transcript but keeps the summary sentence it produced has not forgotten anything. Derivation lineage in the provenance envelope is what makes that cascade testable. The curator's run records count: in the system I built, erasing a person also scrubs them. And the cheapest deletion is the write you refused: sensitive inferences and details with no authorized purpose should never enter memory.

The schema needs the same maintenance: descriptions that extracted well under one model drift under the next. Treat it as a living document: replay extraction on a sample, classify each property as correct, missed, low-confidence or inaccurate, revise the weak ones, validate, then backfill using the `extractor` field [8].

## What this does not do

Curation cannot create truth. If the only evidence about Elena is forty notes from her old job, no dreaming pass discovers her promotion; a fresh source has to arrive. Curation also introduces its own errors: a merge can fuse two different facts, and a summarizer can promote a hedge into a certainty. That is why proposals are reviewable. Nor does curation referee itself: two jobs can fight over one property, one retiring a value the other re-derives every night. Mine detects that oscillation and reports it; which job stands down is a person's call.

Be careful with numbers here. Most memory results are vendor-run, and they disagree, as the 2025 LoCoMo dispute among Mem0, Zep and Letta showed (Chapter 10) [18][19][20]. Popular benchmarks mostly measure context management, and almost none measure whether a system notices that a fact stopped being true. Harvey's 6x and OpenAI's reported internal recall gains are claims from the companies involved. Evaluate on your own freshness cases: real customers whose facts you know changed. I have published no correction rate for my own curator, so I quote none; its ledger of applied, proposed and refused writes is where yours should come from.

I also disagree with rewriting memory in place, even correctly. For a personal assistant, "went to Singapore" is a fine rewrite. For a customer system of record, the old state is evidence: it explains past decisions and lets you roll back a bad pass. Supersede; do not overwrite.

## At scale

At Larkspur's size (250,000 contacts) the question is how to avoid curating everything. If 1 to 3% of contacts receive material new evidence on a given day, that is 2,500 to 7,500 entity cycles, not 250,000. At an assumed $0.005 to $0.05 per cycle (illustrative; it depends on model, memory size and caching, and should be recomputed with Chapter 12's cost model), the daily bill runs from about $12 to $375. The selection rate carries the weight: a nightly full-base pass multiplies cost roughly 30 to 100 times for little gain. It is sleep-time compute's amortization logic, applied per customer [10]. Granularity matters too: when my curator moved from one run per page to one run per record, the first nightly passes were the ones to watch, bounded by the stage caps until re-dreaming only what changed settled them [21].

Scale also changes ownership. With six teams (marketing, sales, customer success, support, product, events) writing to one memory, the per-property authority rules are an organizational agreement, not a config file, and changes to it deserve the review a schema migration gets.

## Failure story: The Echo Summary

An agent reading a sales email inferred that another Larkspur customer "may be evaluating competitors," and stored it correctly labeled as an inference. The nightly summary included it, hedged. Two weeks later a curation pass rebuilt the summary from the previous summary plus new notes, and the hedge fell away. A retention agent read "the customer is evaluating competitors," found it matched the stored inference, counted two sources, and sent a discount to an account that had never considered leaving.

Nothing lied. A guess was re-read by the system that wrote it until it sounded like a fact. The fix: rebuild summaries from source-level facts, never from prior summaries; keep inference labels through every rewrite; count only independent sources. **A memory that reads its own summaries will eventually believe them.**

## Patterns

**Provenance Envelope.** *Problem:* facts cannot be corrected, expired or deleted because nobody knows where they came from. *Solution:* every material fact carries Chapter 7's envelope (route, source, method, observation time, confidence, purposes, surfaces), extended with recorded and valid times, extractor version, kind, sensitivity and status. *Tradeoff:* heavier writes; pays back on the first correction cascade or model change.

**Freshness Policy.** *Problem:* old facts are used with the confidence of new ones. *Solution:* per-property trust windows with an explicit behavior at expiry. *Tradeoff:* windows need calibration from real corrections; too short wastes enrichment spend, too long produces Stale Selves.

**Non-destructive Dream (Propose, Gate, Apply).** *Problem:* memory accumulates duplicates, contradictions and stale views. *Solution:* event-selected, entity-scoped background curation that emits a reviewable diff; new jobs start propose-only and an operator promotes them; the write gate lives in code; low-consequence changes auto-apply, high-consequence ones queue; history is kept and the curator holds no delete tool. *Tradeoff:* review capacity becomes the bottleneck if the gate is too tight.

## Leader questions

1. Pick one customer fact our AI used this week. Where did it come from, and when was it last verified?
2. When a customer changes role, how many days until every agent that writes to them knows?
3. For our ten most-used properties, who owns the rule when sources disagree?
4. If a customer asks us to delete their data, can we show the summaries and embeddings built from it are gone?
5. What does our background curation change without a human, and how would we roll it back?

## Build checklist

- [ ] Every property declares its update semantic (supersede or append) and a freshness window with an expiry behavior.
- [ ] Every material fact carries a provenance envelope, including extractor version and fact kind.
- [ ] Authority rules (Chapter 7's Signal Ladder) are defined per property; last-write-wins is off for customer facts.
- [ ] Unresolved contradictions are stored as `disputed`, not silently resolved.
- [ ] Summaries are rebuilt from source-level facts, never from prior summaries.
- [ ] Curation outputs are diffs with a review gate and a rollback path; new curation jobs start propose-only and an operator promotes them; no path starts a pass without those controls.
- [ ] Deletion and correction cascade through every derived representation, with a test that proves it.
- [ ] A freshness evaluation set of real, known changes runs on every pipeline or model change.

## Metrics to watch

- **Stale-use rate:** share of personalized outputs that used a fact past its trust window.
- **Time to propagate:** days from a change in a source to every representation reflecting it.
- **Contradiction backlog:** open `disputed` facts, and their age.
- **Curation acceptance rate:** share of proposed changes applied after review; a falling rate means the curator is drifting.
- **Per-job digest:** stage, records covered, writes applied, proposed and refused, failures, quarantined records, and whether a pass awaits review. In the system I built, this is the operator's review surface.
- **Cascade completeness:** share of corrected or deleted sources with zero surviving derived references.

## Reader Q&A

**Is dreaming just a batch job with a new name?** Partly, and it is fine to say so. The new part is that language models can read unstructured evidence, reconcile it, and rewrite narrative views. The old part (scheduling, idempotency, review, rollback) is ordinary engineering, and skipping it is how curation becomes a new source of errors.

**Should the curator use our strongest model?** For consolidation, usually not. For reconciling high-consequence contradictions and rewriting account summaries, often yes. Route by step (Chapter 12).

**How often should we re-verify?** Let the freshness table set the schedule and your corrections tune it. If corrections to a property cluster at 90 days, a 180-day window is too long.

## For your AI

```yaml
chapter: 11
concepts:
  - name: Freshness Policy
    definition: "Per-property trust window plus the behavior when it expires (re-verify, rephrase as history, generalize, delete)."
  - name: Provenance Envelope
    definition: "Chapter 7's per-value envelope (route, source, source_ref, observed_at, method, confidence, corroborated_by, permitted_purposes, allowed_surfaces, review_after), extended for keeping memory true with recorded_at, valid_from/valid_to, extractor version, kind (fact, inference, or derived summary), sensitivity, and status."
  - name: Per-property precedence
    definition: "Contradictions are resolved by applying Chapter 7's Signal Ladder to each property: purpose gate, authority by kind of fact, freshness overriding authority for volatile properties, corroborated confidence for ties; a fresh first-hand artifact can beat a stale declared value."
  - name: Supersede, not overwrite
    definition: "Replace semantics close the old value's validity window and keep it as history rather than deleting it."
  - name: Dreaming (applied to the customer)
    definition: "Continuous, event-selected background curation of shared customer memory: consolidate, reconcile, re-synthesize, expire; governed, identity-aware, provenance-preserving, non-destructive and reviewable. Builds on reflection (Park et al. 2023), sleep-time compute (Lin et al. 2025), Letta sleep-time agents, Anthropic Dreams and ChatGPT Dreaming."
  - name: Correction Cascade
    definition: "A correction or deletion of a source propagates to every derived property, memory, summary and embedding, including the curator's own run records."
  - name: Curation Stage Ramp
    definition: "Each curation job's stage caps what it may do (propose-only, pilot, full); only an operator promotes, one stage at a time, after a completed pass; an emergency stop always works; the gate is enforced in code, not the prompt."
decision_rules:
  - if: "a fact used to personalize is older than its freshness window"
    then: "re-verify it or fall back to an honestly general statement"
  - if: "two sources disagree about a property"
    then: "apply Chapter 7's Signal Ladder for that property (purpose gate, authority by kind of fact, freshness override for volatile properties, corroborated confidence for ties); store unresolved conflicts as disputed"
  - if: "a higher-authority value is past its freshness window and a lower-authority source contradicts it"
    then: "treat the contradiction as a likely change; re-verify and trigger re-synthesis of dependent summaries"
  - if: "several agents report the same claim"
    then: "count independent original sources, not agent writes"
  - if: "content was generated by the system (summary, inference)"
    then: "never ingest it as evidence for the facts it was derived from"
  - if: "a curation pass would change a high-consequence property (consent, contract, identity merge)"
    then: "queue the change for review; do not auto-apply"
  - if: "a source is deleted or corrected"
    then: "cascade the change to all derived representations and verify none survive"
  - if: "a new background curation job is created"
    then: "start it propose-only; promote one stage at a time after reviewing a pass; give it no delete tool"
  - if: "a downstream step renders from state that was just written"
    then: "await the write and run a bounded read-after-write check before rendering"
  - if: "a curation job is stopped in an emergency"
    then: "also disable or unschedule it; a stop disarms writes but a scheduled job keeps spending"
  - if: "a curation scope filter does not resolve"
    then: "fail the run; never widen the scope to the whole base"
assessment_questions:
  - "Which customer properties have a defined trust window, and what happens when it expires?"
  - "Can you trace any stored customer fact to its source, observation time and extractor version?"
  - "What rule decides which system wins when CRM, enrichment and conversation data disagree?"
  - "Does any process rewrite account summaries when underlying facts change? How is it selected, reviewed and rolled back?"
  - "Can you prove a deleted source no longer influences summaries or embeddings?"
patterns: [Provenance Envelope, Freshness Policy, Non-destructive Dream]
anti_patterns: [The Stale Self, The Echo Summary, Last-Write-Wins, Access-Based Expiry for Customer Facts, Overwrite in Place, Silent Success, Oscillating Curators]
maturity_dimension: identity_and_memory
```

## References

1. Anthropic. "Memory tool." Claude Platform documentation (accessed 2026-09-26). https://platform.claude.com/docs/en/agents-and-tools/tool-use/memory-tool
2. Wu, D. et al. "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory." ICLR 2025. arXiv:2410.10813. https://arxiv.org/abs/2410.10813
3. OpenAI. "Dreaming: Better memory for a more helpful ChatGPT." 2026-06-04. https://openai.com/index/chatgpt-memory-dreaming/ (primary page not directly retrievable in this research pass; details confirmed via secondary coverage)
4. The "22.5% per year" B2B data decay figure is commonly cited (for example by HubSpot, attributing MarketingSherpa); no primary study located. Not used as fact.
5. U.S. Bureau of Labor Statistics. "Employee Tenure in 2026." Released 2026-09-24. https://www.bls.gov/news.release/tenure.nr0.htm
6. Rasmussen, P. et al. "Zep: A Temporal Knowledge Graph Architecture for Agent Memory." arXiv:2501.13956, 2025. https://arxiv.org/abs/2501.13956
7. W3C. "PROV-DM: The PROV Data Model." W3C Recommendation, 2013. https://www.w3.org/TR/prov-dm/
8. Taheri, H. "Governed Memory: A Production Architecture for Multi-Agent Workflows." arXiv:2603.17787, 2026. https://arxiv.org/abs/2603.17787 ; results summarized in "99.6% Fact Recall, 74.8% on LoCoMo: What the Numbers Actually Mean," hamedtaheri.com, 2026-03-14.
9. Park, J. S. et al. "Generative Agents: Interactive Simulacra of Human Behavior." UIST 2023. arXiv:2304.03442. https://arxiv.org/abs/2304.03442
10. Lin, K. et al. "Sleep-time Compute: Beyond Inference Scaling at Test-time." arXiv:2504.13171, 2025-04-17. https://arxiv.org/abs/2504.13171
11. Letta. "Sleep-time Compute." 2025-04-21. https://www.letta.com/blog/sleep-time-compute
12. Letta. "Context Repositories: Git-based Memory for Coding Agents," 2026-02-12, https://www.letta.com/blog/context-repositories/ ; "Memory Models: Towards Agents That Learn," 2026-06-25, https://www.letta.com/blog/towards-agents-that-learn/
13. Anthropic. "Dreams." Claude Managed Agents documentation, research preview (accessed 2026-09-26). https://platform.claude.com/docs/en/managed-agents/dreams
14. Anthropic. "New in Claude Managed Agents." 2026-05-19. https://claude.com/blog/new-in-claude-managed-agents
15. Kumaran, D., Hassabis, D., McClelland, J. L. "What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated." Trends in Cognitive Sciences 20(7), 2016. https://web.stanford.edu/~jlmcc/papers/KumaranHassabisMcC16CLSUpdate.pdf
16. Diekelmann, S., Born, J. "The memory function of sleep." Nature Reviews Neuroscience 11:114-126, 2010. https://www.nature.com/articles/nrn2762
17. Regulation (EU) 2016/679 (GDPR), Article 5(1)(e). https://eur-lex.europa.eu/eli/reg/2016/679/oj
18. Chhikara, P. et al. "Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory." arXiv:2504.19413, 2025. https://arxiv.org/abs/2504.19413
19. Zep. "Lies, Damn Lies, & Statistics: Is Mem0 Really SOTA in Agent Memory?" 2025-05-06. https://blog.getzep.com/lies-damn-lies-statistics-is-mem0-really-sota-in-agent-memory/ ; Mem0 response, GitHub issue getzep/zep-papers #5, 2025-05-08. https://github.com/getzep/zep-papers/issues/5
20. Letta. "Benchmarking AI Agent Memory: Is a Filesystem All You Need?" 2025-08-12. https://www.letta.com/blog/benchmarking-ai-agent-memory
21. Author's release notes for the self-hosted memory system in the text: dreaming 2026-07-30; stages, promotion gate, stop, ledger and per-record runs 2026-08-11; merge pass 2026-09-07; save and document-history fixes 2026-08-04 and 2026-09-16; checked against the 2026-09-16 release. [PRODUCT-NAMING: link the public changelog if the product is named.]

---

# 12. The Cost of Remembering

*The question: How do we keep memory affordable at scale without losing accuracy?*

> **Questions this chapter answers**
> - Where does the money in an AI memory system actually go?
> - Our pilot was cheap and production is not. Why does cost climb with adoption?
> - Should we just use a cheaper model?
> - What is the right unit for budgeting AI personalization?
> - Which savings are safe, and which quietly cost us accuracy?

## The short answer

A memory system is not expensive because it stores a lot. Storage is the cheap part. It is expensive because of reading: a model reads raw sources to build the memory, and then agents read the memory, plus instructions, plus their own earlier steps, on every call they make. In the pipelines I instrument, the input tokens outnumber the output tokens by roughly 33 to 1 on enrichment and 77 to 1 on generation. The writing is a rounding error. The reading is the product.

That gives leaders four levers, and none of them is "pick the cheapest model":

1. **Read raw data once, at write time, on the cheap path.** Extraction that can wait an hour can run through provider batch APIs at half price, with prompt caching on top.
2. **Serve compact memory, not raw history.** An agent that needs an answer should receive a cited paragraph, not forty documents.
3. **Lay out every prompt so the stable part is cached.** Instructions and policy first, this record last.
4. **Route each step to the smallest model that passes it.** Judgment on the strong model, extraction and classification on small ones.

Budget in **cost per accepted decision**, not cost per token or per seat. That unit includes retries, the strong-model reading, background curation, and the human review that is often larger than the model bill. Metered AI behaves like electricity, not like software licenses: cost rises with use. If you budget it like seats, success will look like an overrun.

The caveat: each saving can also cut accuracy. Measure cost and acceptance together, per step.

## The bill is the reading

I meter input and output tokens separately on every step of the pipelines I run. The first time I saw the split, I assumed a bug. An enrichment step read about 10,551 tokens to write 322. A content-generation step read about 54,161 tokens to write 704. That is simply what the work looks like.

![Bar chart drawn to scale comparing input and output tokens for two pipeline steps. An enrichment step reads about 10,551 input tokens to write 322 output tokens, roughly 33 to 1; a content-generation step reads about 54,161 input tokens to write 704, roughly 77 to 1. A dashed marker shows the floor of about 7,700 input tokens that even a six-word test prompt bills. The output bars are barely visible: the writing is a rounding error and the reading is the product.](/images/handbook/ch11-the-bill-is-the-reading.svg)

*Figure 12.1. Drawn to scale, output is a sliver; what a model reads is the bill.*

Three structures showed up, and each is a design decision. There is a **floor**: a six-word test prompt still billed about 7,700 input tokens, because instructions and guidelines load before the question does. There is **re-sending**: inside one multi-step generation, per-step input climbed from 10,033 to 13,749 tokens while each step added only 700 to 850 new ones; about 86 percent of that call's billed input was text the model had already read. And the **records rhyme**: across one campaign, shared instruction context ran near 5,900 tokens per record while the genuinely unique data ran near 190.

Now picture *Larkspur Systems* (a fictional composite company used throughout this handbook: field-service software, about 40,000 customer accounts and 250,000 contacts). Its renewal program started as a pilot: a renewal agent on 300 customer accounts, costing a few hundred dollars a month. The team extends it to all 40,000 customer accounts and adds a research agent and a review agent. The month-two invoice is not 130 times the pilot; it is larger, because every agent re-reads the same account history, at every step, on the strongest model available. Nobody misused anything. Call it the re-read tax.

## Where the money actually goes

A memory-backed personalization system spends in four places.

**Write time: extraction.** A model reads calls, emails, tickets and documents and turns them into typed properties, evidence-level memories and summaries (Chapter 10). Per record, this is the largest single model call the memory layer makes. In my own systems, the LLM cost of extracting a memory is on the order of fifty times the cost the memory layer spends serving that record back over its life.

**Read time: the consuming agent.** This is where the 50:1 above can mislead you. Serving memory from a database is cheap; what is expensive is the agent that receives it and reads it, with its instructions, at premium rates, on every step. Across the whole system, including the agents, this read side usually dominates, and it grows with every agent you add. Anthropic reports that its agents use about 4 times the tokens of a chat interaction and multi-agent systems about 15 times [1]. Both statements are true at once: the memory layer's own bill is dominated by writes, and the system's bill is dominated by what agents read.

**Background: curation.** Consolidating, deduplicating and re-synthesizing (the "dreaming" cycle in Chapter 11) is model work nobody waits for, so it can run on the cheapest path.

**Outside the model.** Data subscriptions, identity resolution, storage, and above all human review, which (as the worked example shows) can cost more per decision than all the tokens combined.

Here is the sentence this chapter turns on: **memory saves money not by storing more, but by being read less: read raw once, cheaply, and then served compact to every expensive reader after that.**

![Flow diagram of where memory cost goes. Raw sources (calls, emails, tickets, documents) are read once at write time by extraction, the largest call per record, on the batch path at half price with a cached prefix. Extraction produces a compact, cited entity brief of 2k to 3k tokens with raw sources one lookup away, kept current by background curation on the cheapest path. At read time, multiple agents receive the brief at premium rates on every step; this is usually the largest share and grows with every agent. The naive alternative, the re-read tax, has every agent re-read raw history at every step. Outside the model sit data subscriptions, identity resolution, storage and human review, which can cost more per decision than all the tokens combined.](/images/handbook/ch11-read-once-serve-compact.svg)

*Figure 12.2. Memory saves money by being read less: read raw once on the cheap path, then serve a compact brief to every expensive reader.*

## Cost per accepted decision is the unit

Tokens are the wrong unit because nobody knows what a token is worth. Seats are wrong because a metered system costs more exactly as it becomes useful. Microsoft reportedly wound down most internal Claude Code licenses in one large engineering group in 2026 and moved engineers to its own tool; the same reporting cites Uber engineers spending $500 to $2,000 a month each on tokens and exhausting a year's AI coding budget in four months [2]. Nothing there was misuse. It was a metered tool on a seat-shaped budget.

The unit that survives is the decision that shipped: one person, one chosen action (including "do nothing"), accepted by your quality bar. The planning shape:

`cost per accepted decision = (write share + read cost + curation share + review share) / acceptance rate`

Dividing by acceptance rate matters more than it looks. On one production batch I ran, fewer than a thousand records became about 3,700 generation runs once retries were counted. Every failed attempt is paid for and produces nothing.

**Worked example (Larkspur, ranges, list prices as of September 2026).** One decision per account per month: "what, if anything, should happen next on this renewal?" That is 40,000 decisions a month. Prices are one provider's published list rates: a small model at $1/$5 per million input/output tokens, a mid-tier at $2/$10, a frontier model at $4/$20, batch at 50 percent off, cache reads at 0.05 to 0.1 times the input rate [3][4]. Every volume below is an assumption, stated so you can replace it.

| Component | Assumption | Naive design | Engineered design |
|---|---|---|---|
| Write (per account-month) | 5k to 20k tokens of new source material, 4k-token schema prefix, mid-tier model | Sync, no cache: $0.03 to $0.06 | Batch plus cache: $0.01 to $0.03 |
| Read (per decision) | 4 agent steps on the frontier model | Raw history re-read each step, 60k to 200k input tokens: $0.26 to $0.84 | 2k to 3k-token memory brief, stable prefix cached, 30k to 50k input mostly cache reads: $0.04 to $0.10 |
| Curation (per account-month) | Weekly pass on the 30 percent of accounts that changed, batch | None (memory decays instead) | $0.01 to $0.04 |
| **Token cost per decision** | | **$0.29 to $0.90** | **$0.06 to $0.17** |
| Human review (per decision, averaged) | 2 to 10 percent sampled, 2 to 4 minutes each, $60 to $100 loaded hourly cost | $0.04 to $0.67 | $0.04 to $0.67 |

At 40,000 decisions a month, the naive design spends roughly $12,000 to $36,000 on tokens and the engineered one roughly $2,400 to $6,800. Before dividing by acceptance rate.

![Range chart drawn to scale on a zero to one dollar axis for the Larkspur (fictional) worked example, per decision, at list prices as of September 2026. Naive design token cost spans $0.29 to $0.90; engineered design token cost spans $0.06 to $0.17; human review spans $0.04 to $0.67 and is the same in both designs, so it can exceed the whole engineered token bill. At 40,000 decisions a month, tokens cost $12,000 to $36,000 naive versus $2,400 to $6,800 engineered. Cost per accepted decision is write plus read plus curation plus review, divided by acceptance rate.](/images/handbook/ch11-cost-per-decision.svg)

*Figure 12.3. Architecture shrinks the token cost per decision; human review stays the same and can become the largest line.*

Two assumptions carry the weight: **tokens read by the expensive model per decision**, and **the review rate**, which can exceed the whole token bill and does not fall when models get cheaper.

Note what the table does not say. A B2B renewal decision is worth far more than $0.90; token cost rarely makes one decision unprofitable. What kills programs is the curve: cost equals entities, times decisions, times agents per decision, times steps, times tokens per step, times price. Every factor is rising at once. That growth is multiplicative, not exponential (I have called it exponential in my own writing; the word is wrong, and the multiplicative version is bad enough).

## Pay again only for what failed

Retries are the factor you control most directly. The rule: a retry buys back the smallest thing that failed. In a governed personalization engine I built, that is running code [21]:

- **Conditional best-of-two.** The first email set is scored deterministically (blank, bad length, repeated openings, missing approved proof). A clean set ships; only a flawed one triggers a second generation, and the lower-penalty set wins. Better quality without always paying twice.
- **Targeted recovery.** Failed sequence slots are regenerated, not the sequence; a missing page is rebuilt from the preserved research and score, not the whole record.
- **Delivery retry without regeneration.** If the endpoint is down, resend the exact completed payload. Regenerating pays twice and returns a different answer.
- **Circuit breaking and idempotency.** A failing research source stops being called for the rest of the batch; a duplicate request claims the existing work instead of paying again.

## Read once, at write time, at a discount

Extraction is the easiest place to save: CRM imports, nightly ingest, backfills and re-extraction after a schema change can all tolerate an hour.

Providers price for that. As of September 2026, Anthropic's Message Batches API and OpenAI's Batch API both charge 50 percent of the synchronous rate, with results within 24 hours and most Anthropic batches finishing in under an hour [4][5]. Prompt caching stacks on top: the schema, extraction instructions and examples are identical across thousands of calls, so after the first write they are read from cache at around a tenth of the input price [3][6]. On extraction workloads where the cacheable prefix is 60 to 85 percent of input, I measure the combined path at roughly 35 to 40 percent of the synchronous price, for the same extraction [7].

The cost is latency variance. "Within 24 hours" is a ceiling, not a promise, and Anthropic's own docs say cache hits inside batches are best effort, ranging from 30 to 98 percent depending on traffic [4]. So batch belongs to queued work, and sync to writes a live agent is waiting on (Chapter 17). The engineering job is making the discount reachable through the same call: one flag on the write, not a second integration. In a self-hosted memory system I built, a failed batch is also requeued to the normal worker, so the discount never costs a lost write [21].

The cheapest extraction is the one you skip. A value already known exactly (a campaign ID, a score, a renewal date from billing) is written as a typed property, never passed through a model: no call, and no probabilistic error added to data that was right [21]. Then scope the rest: in that system a save runs one extraction call per property collection plus one for free-form memories, so limiting a save to the collections its source can inform is a direct cut.

## A prompt is a layout

Caching turns a prompt from a message into a layout: a cache only matches an identical prefix, so anything that varies early invalidates everything behind it.

So order by volatility. Stable first: system instructions, governance and brand policy, the extraction schema, examples. Then slower-moving context: the account brief. Last: this step, this question, this new email. On the workload described earlier, that ordering alone put the cacheable ceiling between 85 and 94 percent of billed input, and the model never saw a different problem.

A timestamp or a user name in the first line of a system prompt silently defeats the whole scheme. A short prompt is not a cheap prompt; a well-ordered one is. Fan-out makes this concrete: in my memory system the shared curation preamble sits in the system prompt so every per-record pass re-sends an identical prefix, and some providers cache only where the request marks an explicit breakpoint, so verify the hit rate rather than assuming it.

## The context window is a budget, not a bucket

The strongest argument against stuffing context is not cost. It is accuracy. Liu and colleagues showed that models use information at the start and end of a long context better than information in the middle [8]. Chroma's 2025 study of 18 models found performance "consistently degrades with increasing input length," even on simple tasks, and that "even a single distractor reduces performance" [9]. A bigger window is a bigger invoice and, past a point, a worse answer.

Treat context as a budget per step. A workable allocation for a personalized decision:

| Slot | Content | Typical size | Cached? |
|---|---|---|---|
| Instructions and policy | Role, rules, output contract | 3k to 6k | Yes |
| Entity brief | Synthesized memory with citations | 1k to 3k | Per record, across steps |
| Task evidence | The few items this step needs | 0.5k to 3k | No |
| Working history | Compacted prior steps | Capped | Partly |
| Output reserve | The answer | 0.5k to 2k | n/a |

Three techniques keep inside it.

**Compaction and synthesis.** Summarize prior steps and raw history into short, cited state. Anthropic reports that context editing cut token use by 84 percent on a 100-turn internal evaluation [10]; vendor-internal, but directionally consistent with what I see. Chapter 10's finding that quality saturated at around seven well-chosen memories per entity points the same way.

**Progressive delivery.** Do not re-send what the agent already has. Tracking which policies were delivered in a session, and sending only what is new, cut token use by about half across our multi-step workflows with no measured quality loss, entirely by removing repetition [11].

**Intent over query.** When an agent asks memory a question, let the memory layer answer it on a cheap model and return a cited paragraph, rather than returning twenty raw items for the premium model to compress. In one worked comparison, the premium model's cost per call fell to 5 to 10 percent of the query pattern, and over a 30-call loop the agent carried about 7,500 tokens of retrieved context instead of about 90,000 [12].

## Right model, right step

The price spread is the reason routing works. As of September 2026, one provider's list prices run from $1 to $10 per million input tokens between its smallest and largest general models, a tenfold gap, with the same gap on output [3]. FrugalGPT reported matching the best single model with up to 98 percent cost reduction by cascading cheaper models first [13]. RouteLLM reported cost cuts of more than two times without quality loss, using routers trained on preference data [14].

Take those as existence proofs measured on public benchmarks, not forecasts for your steps. What transfers is the principle:

- Small models for narrow, well-specified steps: classification, extraction, routing, change detection.
- The strong model where judgment lives: the ambiguous decision, the contradiction, the irreversible action.
- The more precisely a step is specified, the smaller the model it can run on.
- Measure success rate per step, not per agent; an average hides the step that is quietly failing.

Make it configuration, not code. In my memory system each function (extraction, recall, synthesis, generation, embeddings) names its own model, and so does each curation job: cheap for high-volume deduplication, stronger for deciding whether two records are one person [21]. Any function can run on a local model, which removes the per-token bill, not the cost: you pay in hardware and in the concurrency one local instance cannot sustain.

A cheap model that fails half the time is the most expensive option you have: you pay for the call, the retry, and the human who cleans up. Also check the unit you are comparing. Anthropic notes that its newer tokenizer produces about 30 percent more tokens for the same text [3]; a lower price per token is not automatically a lower price per task.

## Curation runs on cheap time

Curation should run on the cheapest path you have: batch, cached, smaller models, only on entities that changed. Lin and colleagues' sleep-time compute work reports about five times less test-time compute at equal accuracy, and about 2.5 times lower amortized cost per query, when predictable questions are prepared offline [15]. The benefit scales with how predictable the questions are, and customer memory is unusually predictable: the same agents ask about the same accounts every week. Curation also pays back at read time, because a curated brief is shorter than the history it replaces.

Cheap time still needs a budget, because nobody watches background work run. Most controls in my memory system's idle-time consolidation (Chapter 11) are cost controls [21]. It runs only when the queue is quiet and yields to live requests. A record is reviewed again only if it, or the job's objective, changed. A record that fails three passes in a row is quarantined for a week, so one poison record cannot burn tokens nightly. One lesson: moving to one pass per record gave each record its own write budget and audit trail, and made the first nightly runs cost more until convergence settled them. Another: the emergency stop disarms a job's writes, not its schedule. A still-enabled job keeps running in propose-only mode, paying for model calls on every pass, until someone also disables it. A kill switch for writes is not a kill switch for spend.

## What this does not do

**Token savings are not accuracy.** Memory vendors advertise compaction. Mem0's paper reports more than 90 percent token savings against full context [16], but the LoCoMo results behind such claims are disputed in both directions, and a plain full-context baseline held its own (Chapter 10) [17][18]. The lesson I take is narrower: when histories are short, full context is a perfectly good design, and compaction is an accuracy risk you must measure, not a free win.

**Cheaper tokens do not mean a smaller bill.** Epoch AI estimates that the price of reaching a fixed capability has fallen between 9 and 900 times a year, depending on the task [19]. Spend still rises, because agents and fleets use more tokens. Planning on price declines to fix an architecture is planning on the wrong curve.

## At scale

Across millions of entities and many teams, three things change. Cost becomes an attribution problem: without per-step, per-agent metering, nobody knows whose agent is re-reading the account history. Shared memory becomes the main saving: six teams reading one curated brief beats six teams re-deriving it. And concurrency becomes a quality variable. Past a provider's limits, calls often come back marked successful with degraded work; in our pipelines visible abort rates rose from about 2 percent at forty concurrent generations to about 19 percent at eleven thousand [20]. Ramp in steps and hold the acceptance bar constant.

## Failure story: The Cheap-Model Swap

Larkspur's renewal program (fictional) triples its budget in a quarter. The Friday-meeting fix: move every step to the smallest model. The token bill drops by about 80 percent. Over the next six weeks the acceptance rate falls, retries double, the review queue grows, and two renewal emails cite a contract term the account never had. Measured per accepted decision, the program now costs more than before, and it is less trusted.

The mistake was optimizing the price of a token instead of the cost of a decision. The fix was boring: route per step, batch the extraction, serve compact briefs, keep the strong model where judgment lived.

## Patterns

**Read Once, Serve Compact.** *Problem:* every agent re-reads raw history. *Forces:* agents need depth; depth is expensive to read repeatedly. *Solution:* extract at write time on the batch path, curate in the background, serve a cited brief; raw sources stay one lookup away. *Tradeoffs:* write-time latency; compaction can drop what mattered, so measure accuracy against a full-context baseline.

**Stable-First Layout.** *Problem:* repeated instructions billed at full price. *Solution:* order prompts by volatility, stable to volatile, and keep variable content out of the prefix. *Tradeoffs:* discipline across teams; cache lifetimes are short.

**Right Model, Right Step.** *Problem:* one model for the whole job overpays or underperforms. *Solution:* specify steps tightly, route narrow steps to small models, escalate on low confidence, measure success per step. *Tradeoffs:* more moving parts; routers need evaluation on your data.

## Leader questions

1. What is our cost per accepted decision today, including retries and human review?
2. What share of our input tokens is text the model has already read?
3. Which of our AI writes are genuinely waiting on a user, and which could run in batch?
4. Is our AI budget shaped like seats while the bill is shaped like a meter?

## Build checklist

- Meter input and output tokens separately, per step, per agent, per decision.
- Route every non-interactive write through batch.
- Order every prompt stable-first; lint for variable content in the prefix.
- Serve entity briefs with citations instead of raw retrieval to premium models.
- Assign a model per step, with a per-step success metric.
- Write known structured values as typed properties; never send them through extraction.
- Retry the smallest failed unit: the slot, the artifact, or the exact completed payload.
- Run curation on batch, only for changed entities.
- Ramp batch sizes in steps with a fixed acceptance definition.

## Metrics to watch

- Cost per accepted decision (and its spread by decision type).
- Cache-read share of billed input tokens.
- Input tokens read by the strongest model per decision.
- Retry amplification: runs per accepted output.
- Review cost as a share of total cost per decision.

## Reader Q&A

**Won't million-token windows make all this unnecessary?** No. Long windows are billed per token, and accuracy still degrades with length and distractors [8][9]. Long windows are useful for rare, deep reads, not as a default.

**Is batch safe for customer-facing work?** For the write that prepares memory, usually. For a reply a customer is waiting on, no.

**Why not just negotiate a volume discount?** Do that too. It lowers the price; architecture lowers the quantity, and quantity is the part that grows.

## For your AI

```yaml
chapter: 12
concepts:
  - name: Cost per accepted decision
    definition: "Total cost (write share, read, curation share, human review, retries) divided by decisions that pass the quality bar."
  - name: The bill is the reading
    definition: "In LLM systems, input tokens (what models read) dominate cost, usually by one to two orders of magnitude over output."
  - name: Context budget
    definition: "A per-step allocation of tokens across instructions, entity brief, task evidence, working history and output, treated as a limit rather than a container."
  - name: Right model, right step
    definition: "Assign the smallest model that passes each well-specified step; reserve the strongest model for judgment."
  - name: Read once, serve compact
    definition: "Extract raw sources once at write time on the cheapest path, then serve curated, cited briefs to every consuming agent."
decision_rules:
  - if: "a write is not awaited by a user or live agent"
    then: "route it through a provider batch API with prompt caching"
  - if: "prompt content varies per record or step"
    then: "place it after all stable instructions and policy"
  - if: "a premium agent needs an answer about an entity"
    then: "request a cited synthesized brief, not raw items"
  - if: "a model change lowers token cost"
    then: "accept it only if acceptance rate per step is unchanged or better"
  - if: "a record partly fails or its delivery fails"
    then: "retry only the failed unit or resend the completed payload; never regenerate the whole record"
  - if: "a value is already known exactly from a system of record"
    then: "write it as a typed property without model extraction"
  - if: "histories per entity are short"
    then: "prefer full context over compaction until measured otherwise"
assessment_questions:
  - "What is your cost per accepted decision, and does it include human review and retries?"
  - "What share of input tokens is cached or previously read?"
  - "Which steps run on which models, and what is each step's success rate?"
  - "What share of writes run synchronously that could run in batch?"
  - "Is your AI budget seat-shaped or usage-shaped?"
patterns: [Read Once Serve Compact, Stable-First Layout, Right Model Right Step]
anti_patterns: [The Cheap-Model Swap, The Re-Read Tax, Seat-Shaped Budget, The Infinite Context Problem, Replay the Whole Pipeline]
maturity_dimension: identity_and_memory
prices_as_of: "2026-09"
```

## References

1. Anthropic, "How we built our multi-agent research system," June 2025. https://www.anthropic.com/engineering/multi-agent-research-system
2. The Next Web, "Microsoft's Claude Code retreat" (citing Windows Central, The Verge, The Information), 2026. https://thenextweb.com/news/microsoft-claude-code-retreat-ai-cost
3. Anthropic, Pricing (model, caching, batch, tokenizer note), accessed 2026-09-26. https://platform.claude.com/docs/en/about-claude/pricing
4. Anthropic, Batch processing docs, accessed 2026-09-26. https://platform.claude.com/docs/en/build-with-claude/batch-processing
5. OpenAI, Batch API guide, accessed 2026-09-26. https://developers.openai.com/api/docs/guides/batch
6. OpenAI, Prompt caching guide, accessed 2026-09-26. https://developers.openai.com/api/docs/guides/prompt-caching
7. H. Taheri, "65% Off Memorization," 2026-06-03. https://hamedtaheri.com/articles/65-percent-off-memorization/
8. Liu et al., "Lost in the Middle: How Language Models Use Long Contexts," TACL 2024. https://arxiv.org/abs/2307.03172
9. Hong, Troynikov, Huber, "Context Rot," Chroma, 2025-07-14. https://www.trychroma.com/research/context-rot
10. Anthropic, "Managing context on the Claude Developer Platform," 2025-09-29. https://claude.com/blog/context-management
11. H. Taheri, "Progressive Context Delivery," 2026-03-14. https://hamedtaheri.com/articles/progressive-context-delivery/
12. H. Taheri, "Think and Execute For Me," 2026-06-03. https://hamedtaheri.com/articles/think-and-execute-for-me/
13. Chen, Zaharia, Zou, "FrugalGPT," 2023. https://arxiv.org/abs/2305.05176
14. Ong et al., "RouteLLM," ICLR 2025. https://arxiv.org/abs/2406.18665
15. Lin et al., "Sleep-time Compute," 2025. https://arxiv.org/abs/2504.13171
16. Chhikara et al., "Mem0," 2025. https://arxiv.org/abs/2504.19413
17. Zep, "Lies, Damn Lies, & Statistics: Is Mem0 Really SOTA?" 2025-05-06. https://blog.getzep.com/lies-damn-lies-statistics-is-mem0-really-sota-in-agent-memory/
18. Letta, "Benchmarking AI Agent Memory," 2025-08-12. https://www.letta.com/blog/benchmarking-ai-agent-memory
19. Epoch AI, "LLM inference prices have fallen rapidly but unequally across tasks," 2025-03. https://epoch.ai/data-insights/llm-inference-price-trends
20. H. Taheri, first-party production measurements (fan-out abort rates, retry amplification, token anatomy), 2026-08 to 2026-09; unpublished.
21. H. Taheri, first-party system documentation: capabilities reference for a governed personalization engine (2026-08-25, live capabilities only) and release documentation for a self-hosted governed memory system (release 0.8.2, verified 2026-09-16); unpublished. [PRODUCT-NAMING: name the systems here if Hamed decides to]

---

# 13. Agents That Run for Weeks

*The question: How do long-running agents use and maintain memory?*

> **Questions this chapter answers**
> - Our agent works fine in a demo. Why does it fall apart on a job that takes six weeks?
> - What exactly should an agent remember between sessions, and where should that live?
> - If the agent crashes on step nine, does it come back at step nine or step one?
> - How do agents hand work to each other, and to people, without losing what they learned?
> - Should we split the job across many subagents to make it faster?

## The short answer

An agent that works a customer over weeks is not a long conversation. It is a case: a goal, a plan, commitments made along the way, and a customer who keeps changing. The context window cannot hold any of that for long, because every session, crash, deploy, and handoff empties it. So the design question is not "how big is the context" but "where does the case live when the agent is not running?"

The answer is three kinds of memory with three lifetimes. Working memory is the context window, and it lasts one session. Task memory is the case file: the plan, which steps are done, what was promised, what is still open. It lasts as long as the job. Entity memory is what the organization knows about the customer, and it outlives every agent that touches it. Most teams build the first, borrow the third, and forget the second entirely. That missing middle is why agents restart from zero.

Three rules follow. Persist the outcome of every step, so a failure resumes instead of restarting. Hand off state, not reports: agents and people read and write the same case file. And fan out to isolate work, not to go faster, because each extra agent multiplies cost and each shared write can multiply one error.

For a leader, the test takes a minute. Ask your team to kill a running agent in staging. If it comes back at step one, state lives in the wrong place, and every crash is billed twice.

## Week four, and the agent starts over

Larkspur Systems (a fictional company, a composite of many real ones) runs a renewal agent on its larger fleet accounts. Take one of them, a regional haulage operator we will call Kestrel Haulage (also fictional). The renewal is six weeks out.

In week one the agent does good work. It sees in the telemetry that route optimization has barely been touched since a spring outage, and in the support history that Kestrel's new head of operations opened two tickets about overnight sync. It drafts a check-in; the customer success manager sends it. The operations head replies: the sync issue is fixed, thanks, but please do not offer a discount again; last quarter finance took it as a sign the product was worth less. The success manager promises to hold the price and schedule a route-optimization session before the renewal call.

In week four Larkspur deploys a new version of its agent service, and the renewal agent's session ends mid-run. It restarts with the CRM record and nothing else. It re-researches the account, finds the old sync tickets, and drafts an apology for a solved problem. It sees low route-optimization usage and, following its at-risk playbook, proposes a discount.

The success manager catches it, this time. Notice what was lost. Not data: every fact is still in an email thread somewhere. What was lost is the case. The agent had learned three things no system of record held: the discount was off the table, a promise had been made, and the sync issue was closed. None of it was written anywhere the next session would read.

## A long job is many short sessions linked by state

The instinct is to treat this as a context problem: a bigger window, or a summary when the window fills. Both help inside one session. Neither survives a crash, a deploy, a second agent, or a person picking up the case.

Anthropic's engineering team described the same problem for coding agents in terms any operations leader will recognize: "Imagine a software project staffed by engineers working in shifts, where each new engineer arrives with no memory of what happened on the previous shift" [1]. Their fix was not a larger window. It was a progress file read at the start of every session, a JSON feature list with a pass or fail field, and git history as the record of what was done. They say plainly that "compaction isn't sufficient" [1]. Compaction summarizes a conversation near the limit and restarts with the summary [2]. It is a way to keep going, not a way to remember.

The window is also the wrong home for cost reasons. Anthropic treats context as "a finite resource with diminishing marginal returns" [2], and Chroma's tests across 18 models found recall degrading as inputs grow [3]. An agent carrying six weeks of history pays for all of it on every step and reads it worse the longer it gets.

Here is the sentence this chapter turns on. **An agent that runs for weeks does not think for weeks. It thinks for minutes at a time, many times, and the weeks live in the state between.**

## Three memories, three lifetimes

Separate what the agent must remember by how long it must last.

| Memory | What it holds | Lifetime | Where it lives | Who may write |
|---|---|---|---|---|
| Working | The current step's instructions, retrieved context, tool results | One session | Context window | The running agent |
| Task | Goal, plan, completed steps and their outcomes, commitments made, open questions, next action, deadline | The job | A durable case record outside the model | The agents and people assigned to the case |
| Entity | What the organization knows about the customer: properties, evidence, synthesis (Chapter 10) | The relationship | Unified Customer Memory | Governed writers, with provenance |

Most frameworks handle working memory, and Chapters 10 and 11 cover entity memory. Task memory is the one teams skip, because in a demo the task fits in one session. In production it is the difference between an agent and an expensive restart loop.

![A six-week timeline with three rows. Working memory appears as many short sessions, one of which a week-four deploy cuts off. Task memory is a single case record spanning the whole job, so it survives the deploy. Entity memory, the customer memory, extends before and after the job and across every agent; the boundary test is that a fact which still matters after the job closes goes to entity memory with source and date, and otherwise to the case record.](/images/handbook/ch12-three-memories-three-lifetimes.svg)

*Figure 13.1. Working memory lasts a session, task memory lasts the job, entity memory lasts the relationship.*

The boundary matters. "The customer asked us not to offer a discount" is about Kestrel; every agent that touches Kestrel should see it, so it belongs in entity memory with source and date. "Check-in drafted, waiting for the success manager to send it" is about this job, so it belongs in the case record. The test: if the fact would still matter after the job closes, it is entity memory. Write each to the right place and a new session, agent, or person can pick up the case in one read.

## Resume, do not restart

The oldest part of this problem is not about AI. Payment systems and order pipelines have survived crashes for decades by persisting the outcome of each step as it completes, so a failure replays from the last completed step. The name is durable execution.

What changed is the price of redoing a step. A re-run workflow step used to cost milliseconds. A re-run agent step costs a model call, retrieval, perhaps a web search, and the tokens to re-read everything accumulated so far. In a May 2026 interview, Temporal's SVP of Engineering, Preeti Somal, described customers returning to build version 2.0 of agents they had already shipped because they "didn't take care of the plumbing" [4]. Anthropic's multi-agent research system was built the other way: because agents "maintain state across many tool calls," it resumes "from where the agent was when the errors occurred," and the lead agent saves its plan to external memory before its context is truncated [5].

Three properties make resume real:

1. **Checkpoint outcomes, not transcripts.** Record what each step produced (the draft, the score, the decision, the message ID) and the evidence it used. A transcript says what was said; a checkpoint says what is true.
2. **Make external actions idempotent.** If step seven sent an email and crashed before recording it, a replay must not send it again. Give every outbound action a key the downstream system can deduplicate on.
3. **Re-ground on resume.** The checkpoint says where the job was; entity memory says where the customer is now. Read both before acting.

In the memory system I built, a job is a database row, not a process. If its worker dies, the job outlives its timeout, goes back in the queue, and runs again. Running again is the recovery path, which is why property two is not optional.

The personalization engine I built journals finer states (accepted, processing, delivered, artifact pending, recovery attempted, terminal error), so recovery rebuilds the smallest missing unit. Page missing but score delivered: regenerate the page, keep the score. Output right but destination down: resend the exact payload rather than generate a second, different answer. The rule: **retry faults, not decisions.**

A crash that costs one step is an incident. A crash that costs the whole job is a business model you did not choose.

![A job of nine steps where steps one through eight each checkpoint their outcome and evidence into a case record, and the agent crashes on step nine. Restart brings the agent back at step one, so every completed step runs again and the crash is billed twice. Resume brings it back at step nine from the case record, so the crash costs one step. Three properties make resume real: checkpoint outcomes rather than transcripts, make external actions idempotent so a replay never sends twice, and re-ground on resume by reading the case record and entity memory.](/images/handbook/ch12-resume-or-restart.svg)

*Figure 13.2. With checkpointed outcomes, a crash on step nine resumes at step nine instead of step one.*

## Hand off state, not reports

Long jobs involve more than one worker. The renewal agent hands research to a specialist, waits on a person, and eventually passes the case to the account executive. Every handoff is a place where understanding leaks.

The default is the report: agent A writes a summary and passes it to agent B. The summary is lossy by design. It drops the queries tried and abandoned, the weak evidence, the caveat that seemed minor. Chain four reports and the fourth agent reasons from a summary of a summary.

When I rebuilt our own agent API, we replaced one long prompt with an ordered list of steps sharing one accumulated context, so step three sees step one's raw tool results rather than a paraphrase. We got specialist behavior without handoff loss. Across weeks and agents, the same principle has to move outside the model: every worker reads and writes one shared record and passes a reference, not a retelling. Anthropic's research system does the same: subagents "store their work in external systems, then pass lightweight references back to the coordinator" [5].

Shared state changes what the system can see. When a usage analyst records that route optimization stalled after an outage, and a support analyst records that the outage tickets are closed, the renewal strategist can reason over both without either specialist being asked to connect them. The capability lives in the shared record, not in any one agent. The same holds for people: the success manager who opens the Kestrel case reads the same record the agent reads. A handoff that requires someone to write a briefing is a handoff that will eventually be skipped.

![Two panels compare handoffs. In the report chain, agent A passes a summary to B, B to C, and C to D, losing abandoned queries, weak evidence, and minor caveats at each hop, so the fourth agent reasons from a summary of a summary. In the shared-state design, a usage analyst, a support analyst, a renewal strategist, and a success manager all read and write one shared case record with typed outputs and provenance, passing references instead of retellings.](/images/handbook/ch12-state-not-reports.svg)

*Figure 13.3. Reports lose evidence at every hop; a shared record lets every worker, human or machine, reason over the same facts.*

## Fan out for isolation, not for speed

Every major agent platform can now spawn subagents in bulk. The question is when you should.

The cost is documented. Anthropic reports that agents use about 4 times the tokens of a chat interaction and multi-agent systems about 15 times [5]. Each subagent carries its own instructions, tool definitions, and baseline context, and re-reads them every turn. Parallelism feels faster; it is not always faster end to end, and it reliably costs more.

The benefit is real but different. A subagent can read forty documents and return two paragraphs, keeping forty documents out of the main context for the rest of the job. Anthropic describes subagents exploring "tens of thousands of tokens or more" and returning "often 1,000-2,000 tokens" [2]. That is the justification: isolation. A fan-out that exists because two things could happen at once is usually paying twice for one answer.

Isolation also means scope. Each specialist sees only the memory its job needs and holds no more authority than its task requires. The usage analyst does not need contract terms; the drafting agent cannot change who the customer's signer is. Every retrieval is scoped to the entity before any similarity search runs. In an adversarial test I published (100 entities in the same industry with similar roles, overlapping names, and similar deal sizes; 500 queries, 3,800 results), filtering by entity key before vector search produced zero true cross-entity leakage; every flagged result was a false positive from a shared name token [9][10]. Embedding distance alone cannot promise that, because similar customers produce near-identical embeddings.

Authority needs the same scoping when the agent runs on someone else's infrastructure. When my memory system hands a long job to a model provider's background service, the hosted agent never sees the organization's key; it gets a credential for that one job, limited to three memory tools, expiring in a day, revoked when the job ends. What the hosted agent reads is tokenized, and the arguments it sends back are detokenized on the way in, so a real value echoed by the model lands on the right record instead of an orphan keyed by a fake identity. One constraint pulls the other way: those tools work only if the provider's cloud can reach the memory endpoint. A deployment kept on a private network gets a hosted job with no memory access, so the job records a warning rather than running blind without saying so.

For work repeated across thousands of records, the right unit is smaller still: one structured subagent per record, running the same authored instruction chain, grounded in that record's memory, writing typed outputs back. Autonomy is the right default for one open-ended task. It is the wrong default for the same task repeated fifty thousand times.

## Drift is the default; re-grounding is the design

Left alone long enough, agents drift. They misread a situation and commit to it. They treat their own earlier conclusion as fact. They report progress with more confidence than their results earn. Keeping long-running agents on task is most of the engineering work, and anyone who says it is solved is selling something.

Some drift is mechanical: the research agent in Chapter 10 that spent almost a quarter of its output tokens on content it had already seen. The fix is retrieval that remembers what it already delivered within a run and reports when a source is exhausted, so the agent stops asking. The same applies to rules: in my published evaluation, tracking which governance context a session had already received cut re-sent context by about half over five steps [9].

Most drift is semantic, and it is fixed by re-grounding at fixed points rather than hoping the agent notices:

- **At every resume**, re-read entity memory and the case record.
- **Before any consequential action**, re-check recorded commitments and policy. The Kestrel discount dies here.
- **At every handoff**, read the shared record, not the sender's summary.
- **On a schedule**, compare the plan against what changed. A week-one plan can be wrong by week four without any agent making an error.

The context window forgets by accident. A well-designed agent forgets on purpose and remembers on schedule.

## Shared memory multiplies learning, and error

A shared record multiplies in both directions. One unverified write, read by three agents, looks like three confirmations (the failure story below shows how). That is not corroboration. It is one claim echoing through the architecture.

Deliberate corruption of agent memory, usually called memory poisoning, is an active research area. AgentPoison (Chen et al., 2024) showed that a very small fraction of tampered entries in an agent's memory or knowledge base could steer its behavior while leaving normal performance nearly unchanged [6]. For a builder the lesson is integrity controls, and they are the same controls that stop honest mistakes:

- **Separate observation, inference, and authoritative fact.** An agent may propose that someone is the new signer; only a verified source may make it a property.
- **Treat content an agent reads as data, never as instructions** about what to remember or do.
- **Count independent sources, not agreeing agents.** Provenance (Chapter 11) makes echoes visible.
- **Mark derived content** so a generated summary never becomes its own evidence.
- **Write through identity, never raw IDs.** Validating autonomous writes in my own system surfaced a model that put an email address in the record-ID field, creating an orphan, and an update that quietly turned a contact into a generic entity. The fix was code: give the agent each record's identity (email or domain), coerce ID-shaped mistakes, and keep the existing type.
- **Scope write access by role**, stamp each write with a job identity the model cannot see or forge, and keep rollback history for high-impact properties.

Long-running systems also accumulate duplicates, stale entries, and contradictions, which is why Chapter 11's background curation (dreaming) matters here. Anthropic's Managed Agents Dreams, a research preview, reads a memory store plus up to 100 past session transcripts and writes a new, reorganized store, leaving the input untouched for review [7]. That is the right instinct for customer memory: curation should produce a reviewable proposal, not a silent overwrite. One rule from running it: a curation pass's own writes must never wake the next pass, or the loop feeds on itself.

## What this does not do

It does not make agents reliable over long horizons by itself. METR finds the length of task frontier models complete at 50% success has doubled roughly every seven months for six years, and projects week-long software projects within two to four years if the trend holds [8]. Read it carefully. It is measured mostly on software tasks, at a 50% threshold, and a process that must work every night needs far more than one success in two. It also answers a different question. A six-week renewal is not a six-week task. It is dozens of short tasks linked by state, and the reliability that matters is the linkage.

Nor does multi-agent design guarantee better results. Anthropic's widely quoted 90.2% gain for its multi-agent research system was measured on its internal research evaluation, strongest on breadth-first questions, and the same post reports that token usage alone explained 80% of the variance in its browsing evaluation [5]. Much of the gain was spending more tokens. A real result, and a narrow one.

Where things stand, as of September 2026:

| Capability | Status |
|---|---|
| Durable execution, checkpointing, idempotent actions | Ships now; decades-old engineering |
| Case records, progress files, external agent memory | Ships now (vendor tools and in-house builds) |
| Background curation of agent memory with human review | Early production and research previews |
| Agents that manage a multi-week customer relationship with no human checkpoints | Not production practice; forecast at best |

## At scale

At one account, task memory is a file. At 40,000 accounts it is a system of thousands of open cases, most of them waiting on a condition rather than running. The architecture that holds up treats cases as data: a worker is dispatched against a case, checkpoints, and exits; a trigger (a reply, a usage change, a date) wakes the next one. Nothing sits in memory waiting.

Sleeping cases need their own guards. In my systems a schedule fires each occurrence once across the cluster, and an invalid one is disabled rather than retried forever. A job handed to a provider stays a row I own: a sweep reconciles jobs nobody polled, and anything still running after 24 hours is cancelled. A record that fails three passes in a row is quarantined for a week, so one poison record cannot burn tokens nightly. Background work yields queue slots to live requests, so a fan-out cannot starve the customer waiting now.

Three things change with volume. Cost is predictable only if each worker runs a bounded instruction chain in a known token envelope; open-ended autonomy across 40,000 cases is a budget you discover after spending it. Contradictions between agents become statistical certainties, so coordination rules (one owner per customer-facing action, one contact policy) must be enforced by the system, not remembered by each agent. And every error in shared memory reaches every reader, so write gates and provenance matter more than any single agent's accuracy.

## Failure story: Echoed Corroboration

At Larkspur, a support agent logs a caller's claim that he is "the new fleet director"; an enrichment job promotes it to his title; a renewal agent routes the pricing proposal to him, and the actual fleet director hears about it from a colleague. Three systems agreed, and it was one unverified sentence promoted twice. It is Chapter 11's Echo Summary traveling between agents instead of through a summarizer, and the fix is the same: count independent sources, and let no observation become a property without an authoritative one.

## Patterns

**Resume-or-Restart.** *Problem:* long jobs die mid-run and restart from zero. *Forces:* agent steps are expensive to repeat; external actions must never repeat. *Solution:* checkpoint each step's outcome and evidence in a durable case record; key outbound actions for idempotency; re-ground on resume. *Tradeoff:* state placement is decided before step one; retrofitting is a rebuild.

**Shared State over Messages.** *Problem:* reports lose evidence at each hop. *Solution:* all workers, human and machine, read and write one shared record with typed outputs and pass references. *Tradeoff:* shared state needs write permissions and provenance, or it spreads one error to every reader.

**Scope Isolation.** *Problem:* agents see too much, cost too much, and contaminate each other. *Solution:* fan out only to isolate verbose work; give each worker the minimum memory slice and authority; scope retrieval by entity before similarity search. *Tradeoff:* more boundaries to design and test.

## Leader questions

1. If our agent crashes on step nine, where does it come back? Show me.
2. Where does the case live when no agent is running, and can a person read it?
3. Which agents can write to customer memory, and what can each one change?
4. When three agents agree on a fact, how do we know they did not inherit it from one source?
5. Why is this job split across subagents: isolation or speed?

## Build checklist

- [ ] Task memory exists as a durable case record, separate from the context window and from entity memory.
- [ ] Each step's outcome and evidence are checkpointed on completion.
- [ ] Every external action carries an idempotency key.
- [ ] Resumed and handed-off workers re-read entity memory before acting.
- [ ] Commitments made to a customer are written to entity memory with source and date.
- [ ] Agents pass references to shared state, not free-text reports.
- [ ] Retrieval is scoped by entity before similarity search; a leakage test exists.
- [ ] Write permissions are scoped by role; observations cannot promote themselves to properties.
- [ ] Every agent has a turn ceiling and a context budget; every job has a timeout and a maximum age.
- [ ] Faults are retried; completed outputs are resent, not regenerated.
- [ ] The kill test (stop a running agent in staging) runs before every release.

## Metrics to watch

- **Resume point after failure:** share of interrupted jobs that resume at the last completed step.
- **Re-execution cost:** tokens spent redoing steps already completed, as a share of total.
- **Commitment violations:** actions that contradict a recorded customer commitment (target: zero).
- **Single-source agreement rate:** facts cited by several agents that trace to one source.
- **Cost per completed case**, not per call.

## Reader Q&A

**Is a bigger context window a substitute for task memory?** No. It delays the problem inside one session and makes each step more expensive. It does nothing for crashes, deploys, or handoffs.

**Should the transcript be the memory?** Keep transcripts for audit and background curation, not as the first thing the next session reads. A case record tells the next worker what is true; a transcript makes it work that out again.

**How many subagents is too many?** There is no universal number. If you cannot state a subagent's boundary (what it reads, what it writes, what it isolates) in one sentence, it probably should not exist.

**Where should a human sit in a six-week job?** At the consequential moments: before irreversible or customer-facing actions, and wherever evidence is weak. Chapter 18 covers making that review usable.

## For your AI

```yaml
chapter: 13
concepts:
  - name: Working memory
    definition: "The context window: current instructions, retrieved context, and tool results; lasts one session."
  - name: Task memory
    definition: "A durable case record for one job: goal, plan, completed steps and outcomes, commitments, open questions, next action; lasts as long as the job."
  - name: Entity memory
    definition: "What the organization knows about the customer, with provenance; outlives every agent (see Chapter 10)."
  - name: Durable execution
    definition: "Persisting each step's outcome so a failure resumes from the last completed step instead of restarting."
  - name: Re-grounding
    definition: "Re-reading entity memory, the case record, and policy at resume, handoff, and before consequential actions."
  - name: Echoed corroboration
    definition: "Several agents appear to confirm a fact because they inherited one unverified write; the multi-agent form of Chapter 11's Echo Summary."
decision_rules:
  - if: "a fact will still matter after the current job closes"
    then: "write it to entity memory with source and date; otherwise write it to the case record"
  - if: "an agent resumes after a crash, deploy, or handoff"
    then: "load the case record, re-read entity memory, then act; never rebuild state from the transcript"
  - if: "a step performs an external action"
    then: "attach an idempotency key so a replay cannot repeat it"
  - if: "a subagent is proposed only to finish sooner"
    then: "reject it unless it also isolates verbose work from the main context"
  - if: "several agents cite the same fact"
    then: "count independent sources via provenance, not the number of agents"
  - if: "an agent's observation would change a high-impact property"
    then: "require an authoritative source or human review before promotion"
  - if: "a step fails after its output or decision already exists"
    then: "retry the fault (resend the completed output); do not regenerate the decision"
  - if: "a job runs on infrastructure you do not control"
    then: "issue a job-scoped, expiring, least-tool credential and revoke it when the job ends"
assessment_questions:
  - "Where does an agent's in-progress job state live when no agent is running?"
  - "What happens, concretely, when a running agent is killed mid-job?"
  - "Which agents can write to customer memory, and what can each change?"
  - "Are commitments made to customers recorded somewhere every agent reads?"
  - "Is retrieval scoped by entity before similarity search, and is there a leakage test?"
patterns: [Resume-or-Restart, Shared State over Messages, Scope Isolation]
anti_patterns: [The Amnesiac Restart, Echoed Corroboration, Fan-out for Speed, Report Chains]
maturity_dimension: identity_and_memory
```

## References

1. Anthropic Engineering. "Effective harnesses for long-running agents." 2025-11-26. https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents
2. Anthropic Engineering. "Effective context engineering for AI agents." 2025-09-29. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
3. Hong, K., Troynikov, A., Huber, J. "Context Rot: How Increasing Input Tokens Impacts LLM Performance." Chroma, 2025-07-14. https://www.trychroma.com/research/context-rot
4. VentureBeat. "AI agents are entering their rebuild era as enterprises confront the reliability problem." May 2026 (interview with Preeti Somal, Temporal). https://venturebeat.com/orchestration/ai-agents-are-entering-their-rebuild-era-as-enterprises-confront-the-reliability-problem
5. Anthropic Engineering. "How we built our multi-agent research system." 2025-06-13. https://www.anthropic.com/engineering/multi-agent-research-system
6. Chen, Z., Xiang, Z., Xiao, C., Song, D., Li, B. "AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases." arXiv:2407.12784, 2024-07-17. https://arxiv.org/abs/2407.12784
7. Anthropic. Claude Managed Agents, "Dreams" (research preview), documentation, accessed 2026-09-26. https://platform.claude.com/docs/en/managed-agents/dreams
8. Kwa, T., et al. (METR). "Measuring AI Ability to Complete Long Tasks." 2025-03-19; arXiv:2503.14499. https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/
9. Taheri, H. "Governed Memory: A Production Architecture for Multi-Agent Workflows." arXiv:2603.17787, 2026-03-18. https://arxiv.org/abs/2603.17787
10. Taheri, H. "Zero Cross-Entity Leakage Across 3,800 Results." hamedtaheri.com, 2026-03-14. Source of the 100-entity and 3,800-result detail. https://hamedtaheri.com/articles/zero-cross-entity-leakage

---

# 14. The Decision Layer

*The question: What should happen next for this person, including nothing?*

> **Questions this chapter answers**
> - Our models score every customer. Why do the actions they trigger still feel wrong so often?
> - When is "do nothing" the right answer, and how does a system choose it on purpose?
> - Where should an LLM make the call, and where should plain code or a rule make it?
> - How much confidence do we need before the system acts on its own?
> - How do we plan across many steps, many channels, and many teams without burying people in messages?

## The short answer

Most personalization programs are built to answer the wrong question. They ask "how likely is this person to buy, churn, or click?" and then act on the answer. That is a prediction. The question that creates value is different: "what should we do next for this person, and what changes because we did it, compared with doing nothing?" That is a decision, and it needs its own layer in the architecture.

The decision layer sits between memory (what we know about the person) and generation (what we say). It does four things. It lists the actions that are actually available, and that list always includes waiting, asking, routing to a human, and doing nothing. It applies hard constraints (consent, contact limits, open escalations, legal rules) before any model has an opinion. It ranks what remains against objectives that include the customer's long-term trust, not only this quarter's conversion. And it matches the level of autonomy to the consequence of being wrong: cheap, reversible actions can run automatically; irreversible ones get a gate that can say no.

Three findings should shape how a leader funds this. The customers a model scores as most at risk are not reliably the ones an intervention will save, and some retention campaigns measurably increase churn. Sending less can be the better decision: LinkedIn removed four of every ten emails and reported member complaints cut in half. And most decisions do not need a language model; rules and scores are cheaper, faster, and auditable.

If you fund one thing from this chapter, fund a decision layer that can choose silence on purpose, log why it chose it, and learn whether it was right.

## The upsell that should not go out

Larkspur Systems (a fictional company, a composite of many real ones) has a propensity model it is proud of. On a Monday morning it scores Dana, the fleet operations lead at Harlow Fleet Services (also fictional), a regional fleet customer, at 0.81 for the route-optimization add-on: the highest score in her segment. Her usage fits the pattern. Her fleet grew by a third this year. The model is not wrong. She probably would buy it, eventually.

The campaign system reads the score and queues a personalized email. It is well written. It mentions her fleet size and a feature that would save her dispatchers an hour a day.

What the campaign system cannot see is in the support desk. Nine days ago, a sync failure started dropping routes from Dana's dispatch board. Her ticket is marked priority one. She has escalated twice, and her last message ends with "we are evaluating whether this platform is reliable enough for us." Her renewal is in eleven weeks.

Send the upsell and Larkspur tells Dana, in its own voice, that it knows exactly who she is and has chosen to sell to her while her dispatch board is broken. The correct decision for Dana this week is not a better email. It is no email: suppress the campaign, tell her account manager, and put the offer back in the queue for thirty days after the ticket closes, conditional on the fix holding.

The score was accurate. The decision was wrong. That gap is the subject of this chapter.

## A score predicts; it does not decide

A propensity score answers "how likely is this outcome?" A decision needs "how much does this outcome change if we act?" Those are different quantities, and confusing them is the most expensive habit in personalization.

The uplift-modeling literature, which Nicholas Radcliffe helped shape and Gutierrez and Gérardy review, makes the distinction concrete with four groups. **Persuadables** do the desired thing only if you act. **Sure Things** do it either way, so acting wastes money. **Lost Causes** do not do it either way. **Sleeping Dogs** (also called Do-Not-Disturbs) respond badly because you acted. A response model ranks all four groups together, and the customers at the top of a churn score are often the most dissatisfied, which means the list is rich in Sleeping Dogs.

![A two by two grid of what a person does if we act against what they do if we do nothing. Sure Things do it either way, so acting wastes money; Lost Causes do not do it either way; Sleeping Dogs, also called Do-Not-Disturbs, respond badly because we acted; only Persuadables do it only if we act, and they are the people worth targeting. A response model ranks all four groups together, and the top of a churn score, often the most dissatisfied customers, is rich in Sleeping Dogs.](/images/handbook/ch13-uplift-four-groups.svg)

*Figure 14.1. A score finds who is likely to act; only Persuadables are moved by the action.*

This is not a thought experiment. In a paper on mobile operators, Radcliffe and Simpson describe two real retention campaigns. One cut churn in its target group from 30% to 25%. The other *increased* churn from 9% to 10%. Their explanation is plain: a dissatisfied customer is often held in place by inertia, and an intrusive retention call gives the dissatisfaction a moment to crystallize. Their first management implication reads: "Retention programmes can increase churn as well as reduce it."

Eva Ascarza tested the idea with two field experiments, at a wireless provider and a membership organization (*Journal of Marketing Research*, 2018). The customers at highest risk were not the best targets; firms should target on sensitivity to the intervention, not on risk.

Here is the sentence this chapter turns on. **The question is never how likely this person is to act. It is what changes because we acted, compared with doing nothing.** Every other idea in the decision layer follows from taking that sentence seriously.

It has an uncomfortable corollary, the one Chapter 2's Discount Mistake showed: the customer with the highest purchase propensity is often the one who least needs an incentive.

A score tells you who is likely to move. It does not tell you who you will move.

## The action space always includes nothing

Before you can choose well, you have to list what you can choose between. Most systems quietly assume one action (send the message) and optimize only its content and timing. A real decision layer starts with a wider menu:

| Action | What it means | Typical use |
|---|---|---|
| Show | Change what the person sees in a surface they already visit | In-product tips, site content |
| Recommend | Rank options for the person to choose from | Content, products, next steps |
| Send | Push a message into a channel | Email, SMS, direct mail |
| Wait | Hold the action until a condition or date | After a ticket closes; after a trial milestone |
| Ask | Request missing information from the person | Preference, intent, correct contact |
| Route | Hand the situation to a person or a specialist agent | Account manager, support lead |
| Escalate | Route with urgency and authority to override | Risk to the relationship or to compliance |
| Suppress | Block a category of action for this person | Open escalation, legal hold, opt-out |
| Do nothing | Take no action and record that choice | Nothing clears the bar |

Two items deserve emphasis.

**Do nothing is an option with a score, not an absence.** Pega's documentation for its next-best-action engine says the system weighs customer needs against business needs and may "do nothing at all (for example, if no offer is relevant enough to warrant the customer's attention)." The design is simple: give "no action" a value (the expected outcome without intervention, plus the attention you did not spend) and make every other action beat it. When nothing does, the system stays silent and logs why. HCI reached the same rule decades earlier, in Horvitz's mixed-initiative principles and the Office Assistant's failure to follow them (Chapter 3).

The strongest public evidence that silence is a decision comes from LinkedIn. In 2015 the company rebuilt email sending as an optimization problem and announced: "For every 10 emails we used to send, we've removed 4 of them," and "member complaints have been cut in half." The engineering work (Gupta and colleagues, KDD 2016) treats each candidate email as a cost-benefit decision and drops those whose value does not justify the attention. An earlier experiment found 45% more negative responses (unsubscribes and spam reports) among members who received every email than among members from whom a random share was withheld.

**Ask is the most underused action in AI systems.** When evidence is thin, one good question often beats acting on a guess. Language models do not do this by default. Su and Cardie (2026) found that models recognize an ambiguous request when asked to judge it, but rarely ask a clarifying question in normal use, and that retrieved context makes them *less* likely to ask. A model with rich customer memory will be more confident, not more careful. If "ask" matters, make it an explicit option the system can select.

Silence is a decision. Make it on purpose.

## Hard constraints first, objectives second, time horizon always

Every decision mixes three kinds of input, and most failures come from treating them as one.

**Hard constraints** are not traded off. Consent withdrawn, an open priority-one escalation, a legal hold, a contact cap reached: none is weighed against expected revenue. They remove options before ranking begins. "Open P1 escalation suppresses promotional sends" should live in code, run on every path, and be impossible for a model to reason around.

**Objectives** are traded off, and there are always several: value to the customer, value to the business, the cost of the action, the risk of being wrong, and trust. An objective that counts only conversion will find the most efficient way to annoy people, because annoyance is not in the function. Write the tradeoff down, with weights someone signed off on.

**Time horizon** is where good-looking decisions go bad. Rewards arrive in days (an open, a meeting); costs arrive in months (unsubscribes, ignored channels, a renewal that starts from irritation). Hohnhold, O'Brien and Tang (2015) showed Google that users learned to ignore search ads as ad load rose, and the company cut mobile search ad load by half. Chapter 21 covers measuring this. The decision layer's job is to carry a cost for attention even when this week's metric cannot see it.

## Six ways to decide, and when each earns its cost

"AI decisioning" hides several different machines. They have different strengths, different costs, and different ways of failing. Choosing among them is the first design decision, and it is made per decision, not per company.

| Mode | What drives the choice | Strength | Main risk | Use it for |
|---|---|---|---|---|
| Rule | Explicit logic and policy | Transparent, fast, deterministic | Brittle across nuanced cases | Constraints, eligibility, compliance, known branches |
| Data-driven | Scores, probabilities, rankings | Learns patterns from history | Prediction mistaken for intervention value | Ranking candidates at volume |
| Optimized | Objectives plus constraints | Balances competing goals | Bad objectives pursued efficiently | Contact budgets, offer allocation, channel mix |
| Reasoning | Evidence, interpretation, strategy | Handles ambiguity and novel situations | Cost, nondeterminism, false confidence | Ambiguous cases, account strategy, conflicting signals |
| Agentic | Reasoning plus tools, plans, permissions | Executes multi-step work | Errors become actions, not suggestions | Research, multi-step follow-up under guardrails |
| Reflective | Outcome evaluation and feedback | Improves over time | Feedback loops, reward hacking, drift | Tuning policies and weights from observed results |

The table reads left to right as an escalation in capability and in risk. The rule I follow: **use the least powerful mode that handles the decision well, and keep deterministic policy deterministic.**

In the prospecting agents I build, the pattern that survived production is code-orchestrated. Code assembles the context. One model call returns a typed classification (for a reply: positive, negative, question, out-of-office, referral). Code executes the branch. The model never writes to the record or sends anything on its own authority, and opt-out, rate-limit, and sequence-status checks run on every path regardless of what it returned. When someone asks why the system did something, the answer is two artifacts: the JSON the model produced and the branch the code took.

Scoring follows the same split. In a governed personalization engine I built, a model combines weak and strong signals into a score, a band, and a rationale; code checks that they agree, so a score of 88 labeled LOW is flagged rather than shipped.

Reasoning earns its cost in a narrower band than vendors suggest. Dana's case is in it: a high score, a live escalation, a renewal approaching, a thread whose tone matters. A rule catches the escalation; reasoning helps decide what the account manager should do about it. Whether a person who unsubscribed may be emailed has one answer, and it should come from code every time.

Let the model reason; let the code decide what is allowed.

## Candidate, rank, decide

At scale, the decision layer is a pipeline with three stages, each with a different job.

**Candidate generation** assembles every action that could apply to this person now: open campaigns they qualify for, follow-ups owed, service actions triggered by events, and the standing options (wait, ask, route, do nothing). Eligibility rules and hard constraints run here, so ineligible options never reach the ranker.

**Ranking** estimates the value of each remaining candidate for this person, ideally as incremental value over doing nothing, net of cost. Where you have control-group data, rank on predicted uplift, not predicted response. Where you do not, say so, and treat the ranking as a hypothesis.

**Deciding** applies the things a ranker should not own: the per-person contact budget, cross-team arbitration (sales and marketing wanting the same person the same week), confidence thresholds, and the consequence gate described in the next section. The output is one action (or none), a reason, and a trace.

![The decision layer as a three-stage pipeline. Candidate generation lists every action that could apply now (campaigns the person qualifies for, follow-ups owed, event-triggered service actions) plus the standing options wait, ask, route, and do nothing, and hard constraints run at this stage. Ranking estimates each remaining option's incremental value over doing nothing, net of cost, on predicted uplift rather than predicted response where control data exists, and as a labeled hypothesis where it does not. Deciding applies what a ranker should not own (per-person contact budget, cross-team arbitration, confidence thresholds, the consequence gate) and outputs one action or none, a reason, and a trace.](/images/handbook/ch13-candidate-rank-decide.svg)

*Figure 14.2. Candidate, rank, decide: constraints filter, the ranker values, the decision stage governs.*

The cleanest instance I have shipped is duplicate-record resolution in a self-hosted memory system I built. A deterministic finder proposes candidate pairs from hard evidence (a shared strong identifier, or the same name at the same employer; a name alone never qualifies). A model judges one pair and may call exactly one of two tools: merge, or keep distinct. Code owns the rest: a merge below the confidence floor (0.9 by default) is refused, a new judge only proposes until an operator promotes it, and every merge is journaled so it can be reversed for 90 days. The operator approves the judge, not each verdict.

Ranking needs to learn, and learning needs exploration: sometimes trying an option the model is less sure of. The contextual bandit is the classic tool. Li, Chu, Langford and Schapire (2010) framed news recommendation on the Yahoo! Front Page as a sequential decision problem; their LinUCB algorithm, evaluated on more than 33 million logged events, reported a 12.5% click lift over a context-free bandit. The lesson is not the algorithm. It is that exploration is a deliberate, budgeted part of the decision, best spent on low-consequence actions.

## Confidence is judged against consequence

A system that asks "am I confident enough?" without asking "for what?" will be reckless or useless. The same 70% confidence is fine for recommending an article and unacceptable for cancelling a contract.

Picture a grid. The horizontal axis is confidence, low to high. The vertical axis is consequence, from low and reversible (a recommendation, a draft, a flag for review) to high and irreversible (a message reaching a customer at a sensitive moment, a price commitment, a deleted record). The four cells call for four behaviors:

- **Low consequence, high confidence: automate.** Act without review, log the decision, and sample the logs.
- **Low consequence, low confidence: explore or ask.** Cheap mistakes are data. Try the option, or ask the person, and learn from the result. This is where exploration budget belongs.
- **High consequence, high confidence: act through a gate.** A deterministic check sits in the execution path and can refuse. For the most serious actions, a person approves.
- **High consequence, low confidence: do not act.** Gather evidence, ask, or route to a human. The default here is nothing, and it should be the easiest path in the system.

![A two by two grid with confidence on the horizontal axis, low to high, and consequence on the vertical axis, from low and reversible (a recommendation, a draft, a flag for review) to high and irreversible (a price commitment, a deleted record). Low consequence with high confidence: automate, log the decision, and sample the logs. Low consequence with low confidence: explore or ask, because cheap mistakes are data and the exploration budget belongs there. High consequence with high confidence: act through a deterministic gate that can refuse, with a person approving the most serious actions. High consequence with low confidence: do not act; gather evidence, ask, or route to a human, with nothing as the easiest default path.](/images/handbook/ch13-confidence-consequence-grid.svg)

*Figure 14.3. Autonomy is set by confidence relative to consequence, not by confidence alone.*

Reversibility is the axis most programs ignore. They apply uniform oversight, which becomes light oversight everywhere because heavy oversight everywhere is unaffordable. Sort actions by whether they can be undone, and put a real gate (a component in the path that can say no, and that the model cannot argue past) in front of the irreversible ones only. Fewer controls, carrying more weight.

Confidence also sets how specific an action may be. In the personalization engine, every record ends in one of three states: full quality, safe degraded, or fail closed. When identity is weak (a personal email address must not inherit the email provider's firmographics), deterministic policy suppresses company research and steps the copy down toward the segment, without waiting for the model to notice. Chapter 15 names the rungs; the bottom one, fail closed, is the do-nothing option reached because the evidence ran out.

The same logic applies when the system changes its own behavior. An agent that proposes is useful; one that applies its own proposals is a different capability, and the line must be structural. In the memory system, a background consolidation recipe starts propose-only, and an operator promotes it one stage at a time, only after a pass has completed at the current stage. The limit lives in the tool path, not the prompt, so the model cannot talk its way past it, and one call demotes a recipe mid-run. For campaign policy, the design I am building toward adds two human approvals: before a change is implemented and before it reaches production. Chapter 18 covers this in full.

## The decision architecture in ten steps

Every decision, from a one-line rule to a multi-step agent, fits the same ten steps. Writing them down per decision is the most useful hour a team can spend.

1. **Goal.** What outcome are we improving for this person, in this situation? (Dana: keep a strategic account through renewal.)
2. **Policies.** What is allowed, required, prohibited, or subject to approval? (Open P1 suppresses promotion.)
3. **Evidence.** What signals and memory are relevant? (Score 0.81, P1 ticket, two escalations, renewal in eleven weeks.)
4. **Interpretation.** What does the evidence imply, and how uncertain is it? (Buying intent is real; relationship risk is higher.)
5. **Options.** What actions are available, including nothing and asking? (Send, wait, route, do nothing.)
6. **Decision.** Which option best satisfies objectives within constraints? (Route to the account manager; hold the offer.)
7. **Plan.** If multi-step, what sequence follows? (Revisit thirty days after resolution if the fix holds.)
8. **Execution.** Carry it out with the right permissions and channel controls.
9. **Observation.** Capture the result and new evidence. (Ticket closed; tone of next reply.)
10. **Learning.** Judge the outcome and update future behavior. (Did holding the offer change renewal or later conversion?)

Steps 2 and 5 are where most programs cut corners, and they are where Dana's email should have died.

## Planning across steps, channels, and people

A single decision is the easy case. The hard case is a sequence (a six-week onboarding, a quarter-long renewal) or an account four teams all want to talk to.

Three rules keep planning sane. **Plan as a policy, not a script.** A sequence states its goal and exit conditions ("stop if they reply, if a ticket opens, if the champion leaves") and re-decides at each step with fresh evidence, rather than firing step four because step three happened. **Treat attention as a shared budget per person.** Every team and channel spends from the same balance, and the decision layer is the one place that can see it. **Arbitrate centrally.** When two teams want the same person the same week, one decision point chooses against the same objectives.

Contact policy and fatigue are not settings in an email tool. They are decisions the layer makes per person. Internal alerts too: the layer I am designing splits each seller or operator notification into four decisions (whether, what, when, and through which channel), with "not at all" scored like any other option. Chapter 16 works through that notification decision in full, including interruption levels and permission as a budget.

## What this does not do

Decisioning is a field that oversells itself easily, so here are the limits, stated plainly.

**Uplift modeling is harder than the four-group chart suggests.** You cannot observe one person both treated and untreated, so uplift is estimated from randomized control groups and needs far more data than a response model. Many companies lack clean control groups for most actions. Fernández-Loría and Provost offer a useful qualification: accurate causal-effect estimates are not required for good causal decisions, because what matters is ranking people correctly on whether to treat. That helps. It does not remove the need for experiments.

**A next-best-action engine is not a decision layer by itself.** Pega calls its hub an "always-on brain," a single centralized decision authority. Central arbitration is the right idea, and its do-nothing option is exactly right. But an authority is only as good as the evidence it sees. If support tickets never reach it, it will send Dana the upsell with full confidence. The hard part is wiring every signal and every team's actions into it.

**Bandits need volume and fast feedback.** LinUCB learned from tens of millions of events with clicks arriving in seconds. A renewal decision with a quarterly outcome and a few thousand accounts gives a bandit little to learn from. For slow, rare, high-value decisions, careful rules, human judgment, and periodic experiments beat online learning.

**Reasoning does not make decisions safer by default.** It makes them more flexible. Reasoning belongs inside constraints, never in place of them.

**Doing nothing has a cost too.** A customer in trouble who hears nothing may conclude nobody noticed. "Do nothing" must beat the alternatives on the same scale, not win because it avoids blame.

## At scale

With a few thousand customers, a thoughtful team can make many of these decisions by hand. With Larkspur's 250,000 contacts, six teams, and five channels, three things change.

First, **small error rates become large numbers.** A 1% error in a daily decision over 250,000 people is 2,500 wrong actions a day. Consequence tiers and log sampling become essential.

Second, **coordination becomes the main problem.** Each team's logic may be locally sound; the customer experiences the sum. Without one place that sees every pending action for a person, you get contradictions and fatigue no single team caused.

Third, **cost per decision becomes a design constraint.** A reasoning call on every person every day is rarely affordable or necessary. The economical shape is rules and scores for the bulk, reasoning for the cases they flag as ambiguous or high-stakes. If reasoning handles most decisions, the design is probably wrong.

## Failure story: The Fatigue Spiral

Before its decision layer, Larkspur (fictional) ran five programs that each made sense: a weekly nurture email, in-app tips and a monthly feature digest, customer-success check-ins and health alerts, a sales expansion sequence, and event invitations.

Every program had a good open rate. A mid-sized customer's operations lead, whom all five targeted, received eleven messages in two weeks from four senders, two announcing the same feature in different words. She set up a filter. Three weeks later the renewal notice, which actually mattered, went to the same folder.

No single message was bad. Nobody owned the sum. Each team optimized its own channel against its own metric, and the cost (a person who stopped reading Larkspur) landed on a metric none of them tracked. The fix was not better copy. It was a per-person contact budget, one arbitration point, and "do nothing" as a ranked option in every program.

Every message spends from an account whose balance nobody on the sending side can see.

## Patterns

**Do-Nothing Option.** *Problem:* the system acts whenever a candidate exists, so volume grows until people tune out. *Forces:* each team wants its action taken; silence is invisible in most metrics. *Solution:* model "no action" as a scored candidate every action must beat; log each silent decision with its reason. *Tradeoffs:* needs a counterfactual estimate, usually from holdouts; send volume drops before the benefit shows.

**Candidate, Rank, Decide.** *Problem:* one model filters, scores, and governs at once, and governance loses. *Forces:* rankers need freedom to learn; constraints must never bend. *Solution:* constraints in candidate generation; value (uplift where possible) in ranking; budgets, arbitration, and consequence gates in the decision stage. *Tradeoffs:* more components; far easier to debug and audit.

**Consequence-Gated Autonomy.** *Problem:* uniform oversight is unaffordable or too thin. *Forces:* cost pushes toward automation; irreversible mistakes cannot be recovered. *Solution:* tier actions by reversibility; automate the low tier, gate the high tier with a component that can refuse, require human approval for the irreversible, and separate "propose" from "change" with different credentials or operator-promoted stages. *Tradeoffs:* the tier list needs an owner, and approvals must be usable or they get rubber-stamped.

## Leader questions

1. For our three most important personalized actions, what is the "do nothing" option, and how often does the system choose it?
2. Are we targeting people who are likely to act, or people our action will change? How would we know the difference?
3. Which of our automated actions cannot be undone, and what component in the path can stop each one?
4. Who can see every message a single customer will receive from us next week, across all teams?
5. What share of our decisions go to rules, scores, and reasoning models, and what does each share cost?

## Build checklist

- [ ] Every personalized decision has a written ten-step spec, including policies and options.
- [ ] "Wait," "ask," "route," and "do nothing" are explicit, selectable actions, and each silent decision is logged with a reason.
- [ ] Hard constraints run in code before any ranking or model call, on every path.
- [ ] Rankers estimate incremental value where control data exists, and are labeled as hypotheses where it does not.
- [ ] A per-person contact budget is enforced across all teams and channels at one arbitration point.
- [ ] Every action is tiered by reversibility; irreversible actions pass a gate that can refuse.
- [ ] Model outputs are typed; code, not the model, executes actions.
- [ ] Weak identity lowers specificity by deterministic policy; "fail closed" is a logged outcome.
- [ ] Every decision writes a trace: evidence used, options considered, choice, reason, mode, and outcome when known.

## Metrics to watch

1. **Incremental lift over a do-nothing holdout**, per decision type (not response rate).
2. **Suppression and silence rate**: the share of eligible candidates the layer chose not to act on, with reasons. Near zero means the option is not real.
3. **Negative-response rate per person per window**: unsubscribes, spam reports, and channel disengagement, tracked against contact volume.
4. **Decision mix and cost per decision** by mode (rule, score, optimization, reasoning).

## Reader Q&A

**We already have a next-best-action tool. Do we need a decision layer?** Maybe not a new product. Check your tool against the build checklist; the gaps are usually in the wiring (support and sales signals that never reach it), not the engine.

**We have no control groups. Can we still do this?** Start with rules and scores, and start holding out a small random share of eligible people from each program today. Every week you delay a holdout is a week of decisions you cannot evaluate later.

**Should an LLM make our decisions?** Some of them: where interpreting messy, ambiguous evidence changes the answer. Keep eligibility, consent, and contact limits in code, and have the model return a typed choice that code executes.

**How do we set the confidence threshold?** Per consequence tier, not globally. Start strict on irreversible actions and loose on reversible ones, then adjust from the observed error rate in each tier.

## For your AI

```yaml
chapter: 14
title: The Decision Layer
concepts:
  - name: Decision Layer
    definition: "The architectural layer between memory and generation that chooses the next action for one person, including waiting, asking, routing, or doing nothing, under explicit constraints and objectives."
  - name: Prediction vs decision
    definition: "A score estimates how likely an outcome is; a decision requires estimating how much the outcome changes because of an action, compared with no action."
  - name: Uplift groups
    definition: "Persuadables (act only if treated), Sure Things (act anyway), Lost Causes (never act), Sleeping Dogs (react badly to treatment)."
  - name: Do-Nothing Option
    definition: "No action modeled as a scored candidate that every other action must beat; silent decisions are logged with reasons."
  - name: Confidence x Consequence grid
    definition: "Autonomy is set by confidence relative to consequence: automate low-consequence confident actions, explore or ask on low-consequence uncertain ones, gate high-consequence confident ones, do not act on high-consequence uncertain ones."
  - name: Decision modes
    definition: "Rule, data-driven, optimized, reasoning, agentic, reflective; use the least powerful mode that handles the decision well."
  - name: Decision architecture
    definition: "Goal, policies, evidence, interpretation, options, decision, plan, execution, observation, learning."
decision_rules:
  - if: "a candidate action conflicts with a hard constraint (consent, open escalation, legal hold, contact cap)"
    then: "remove it before ranking; never let a model weigh it against value"
  - if: "targeting is based on propensity or risk score alone"
    then: "treat it as a hypothesis; add a randomized holdout and move toward ranking on estimated uplift"
  - if: "no candidate beats the value of doing nothing"
    then: "take no action and log the reason"
  - if: "evidence is thin and the action is low consequence"
    then: "explore or ask the person rather than guess"
  - if: "the action is irreversible or high consequence and confidence is low"
    then: "do not act; gather evidence or route to a human"
  - if: "the decision has one correct answer given known facts (eligibility, opt-out)"
    then: "decide in deterministic code, not with a reasoning model"
  - if: "more than one team has a pending action for the same person in the same window"
    then: "arbitrate at a single decision point against a shared per-person contact budget"
  - if: "identity confidence is low or evidence is thin"
    then: "lower specificity by deterministic policy (full, degraded, fail closed); do not rely on the model to notice"
  - if: "a model returns a score and a band or label that contradict each other"
    then: "flag the record in code before any output ships"
assessment_questions:
  - "Which personalized actions run today, and what is the do-nothing option for each?"
  - "Do you have randomized holdouts for your main programs, and for how long have they run?"
  - "Which signals (support tickets, product usage, sales activity) reach the system that decides outbound actions?"
  - "Which automated actions are irreversible, and what component can stop each one?"
  - "Is there a per-person contact budget shared across teams and channels?"
  - "What share of decisions are made by rules, scores, optimization, and LLM reasoning?"
patterns: [Do-Nothing Option, "Candidate, Rank, Decide", Consequence-Gated Autonomy]
anti_patterns: [The Fatigue Spiral, The Reasoning Everywhere Problem, The Autonomous Error]
maturity_dimension: decisioning
```

## References

1. Radcliffe, N. J., and Simpson, R. (2008). "Identifying who can be saved and who will be driven away by retention activity." *Journal of Telecommunications Management* 1(2). Author PDF: https://stochasticsolutions.com/pdf/SavedAndDrivenAway.pdf
2. Radcliffe, N. J., and Surry, P. D. (2011). "Real-World Uplift Modelling with Significance-Based Uplift Trees." https://www.research.ed.ac.uk/en/publications/real-world-uplift-modelling-with-significance-based-uplift-trees/
3. Ascarza, E. (2018). "Retention Futility: Targeting High-Risk Customers Might Be Ineffective." *Journal of Marketing Research* 55(1). https://journals.sagepub.com/doi/10.1509/jmr.16.0163 ; preprint: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2759170
4. Fernández-Loría, C., and Provost, F. (2021). "Causal Decision Making and Causal Effect Estimation Are Not the Same... and Why It Matters." arXiv:2104.04103. https://arxiv.org/abs/2104.04103
5. Pega Academy. "One-to-one Customer Engagement paradigm." https://academy.pega.com/topic/one-one-customer-engagement-paradigm/v2 (accessed 2026-09-26); "Working with the always-on brain (outbound)." https://academy.pega.com/topic/working-always-brain-outbound/v1 (accessed 2026-09-26)
6. Awan, A. (2015-07-27). "Less is more: You're about to receive less email from LinkedIn." LinkedIn Blog. https://www.linkedin.com/blog/member/product/less-email-from-linkedin
7. Gupta, R., Liang, G., Tseng, H.-P., Holur Vijay, R. K., Chen, X., and Rosales, R. (2016). "Email Volume Optimization at LinkedIn." KDD 2016, 97 to 106. https://dl.acm.org/doi/10.1145/2939672.2939692
8. Gupta, R. "Less Is More: Optimizing Email Volume, Part 1." LinkedIn Engineering Blog. https://www.linkedin.com/blog/engineering/archive/less-is-more-optimizing-email-volume-part-1
9. Su, J., and Cardie, C. (2026). "Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions." arXiv:2605.25284. https://arxiv.org/abs/2605.25284
10. Hohnhold, H., O'Brien, D., and Tang, D. (2015). "Focusing on the Long-term: It's Good for Users and Business." KDD 2015. https://dl.acm.org/doi/10.1145/2783258.2788583
11. Li, L., Chu, W., Langford, J., and Schapire, R. E. (2010). "A Contextual-Bandit Approach to Personalized News Article Recommendation." WWW 2010. https://arxiv.org/abs/1003.0146
12. Gutierrez, P., and Gérardy, J.-Y. (2017). "Causal Inference and Uplift Modelling: A Review of the Literature." PMLR 67. https://proceedings.mlr.press/v67/gutierrez17a.html

---

# 15. Writing for One

*The question: How do we generate for each person without inventing?*

> **Questions this chapter answers**
> - Can AI really write for each customer as well as a good person would?
> - How do we stop it from making things up about the people we write to?
> - How do we keep brand, tone, and claims consistent across a million generated messages?
> - What should the system say when it does not know enough?
> - Does one piece of generation logic have to be rebuilt for every channel?

## The short answer

Writing is no longer the hard part. A current model can produce a fluent, well-toned message for every person in your database, cheaply enough that the cost of the words barely registers. What it cannot do on its own is tell the difference between what it knows about a person and what merely sounds like something it could know. Language models are trained and evaluated in ways that reward a confident guess over an honest "I don't know" [3]. In a personalized message, a confident guess about the reader's own business is the most expensive kind of error you can make, because the reader is the one person guaranteed to catch it.

So the design goal changes. You do not ask the model for "a personalized email." You give it a contract: which claims it may make, from which evidence, in which tone, at what length, with what fallback. You break the message into small typed zones, each with its own job and evidence rule. You make every personal claim carry a receipt back to memory. When the evidence is thin, the system says less, on purpose, and stays useful by being honestly general. Then code, not the model, has the final word on the rules code can enforce, before one governed context is compiled into email, web, seller notes, or print.

For a leader, the test is simple. Ask your team: "Show me, for any sentence we sent, the evidence that allowed it." If they can, the program can scale. If they cannot, every additional message adds risk faster than it adds value.

The standard for this chapter, and for the handbook: **specific and true, or honestly general.**

## A beautiful email about an acquisition that never happened

*Larkspur Systems is a fictional composite company used throughout this handbook.*

Larkspur's AI SDR pilot had been running for six weeks when the reply came in. The message the pilot had sent to the operations director of a regional HVAC service company was, by any stylistic measure, excellent. It was short. It sounded like a thoughtful account executive. Its second sentence congratulated her on the company's recent acquisition of a competitor and suggested that merging two dispatch operations was exactly the moment to revisit fleet scheduling.

There had been no acquisition. A company with a similar name, two states away, had announced one. The pilot knew only rung-two facts about her (name, title, industry, company size), plus whatever an ungoverned web search on her name and company returned. That search returned the article, the prompt asked the model to "reference a recent company event," and the model did what it was asked: it found the most relevant-looking event and wrote about it with the confidence of someone who had checked.

Her reply was one line: "You clearly have no idea who we are."

Nobody at Larkspur had written a false sentence. The prompt was reasonable. The model was good. The article was real. The failure lived in the space between them, where nothing required the claim to be about the right company, to be sourced, or to be checked after it was written. That space is what this chapter is about.

## The writing got cheap; the right to be specific did not

Software can now produce natural language in the idiom, length, and tone a moment calls for, cheaply enough to do it separately for an enormous number of people. The old assumption that enough volume would reveal a template no longer holds. The evidence, read carefully, supports a narrower claim than the vendor pitch. In a study of 195 patient questions posted to a public forum, evaluators preferred chatbot answers to physicians' answers in 78.6% of evaluations and rated them more empathetic [7]. In a controlled three-party test, a persona-prompted GPT-4.5 was judged to be the human 73% of the time [8]. In marketing, AI-generated images matched or beat human-made ones, with up to 50% higher click-through than stock photography in a field test of more than 173,000 impressions [9]. And a Marketing Science study of machine-drafted SEO landing pages found the machine could produce unique, human-like content while concluding that the human editor remained essential [10].

Read together: in narrow, well-specified tasks, machine writing is at or above the quality most organizations could afford per record. None of those studies measured whether the writing was true about a specific reader, because none of them had to.

That is the gap. Fluency is not a guarantee of truth. A model can infer something useful about a person and confidently invent something false in the same paragraph, in the same assured voice; a 2020 study of summarization systems, asked only to condense a given article, found substantial unsupported content from all of them [1]. Retrieval [2] puts evidence in front of the model; it does not oblige the model to stay inside it. Kalai and colleagues argue the invention is not a mysterious defect: models are "optimized to be good test-takers, and guessing when uncertain improves test performance" [3]. A human writer who invents a customer's acquisition has a reputation to lose. The model has none. The accountability has to live in the system around it.

Measured rates make the stakes concrete. On Vectara's leaderboard, where models summarize a provided document and are scored on whether they added anything unsupported, rates ran from 1.8% for the best model to 24.2% for the worst, as of September 22, 2026 [4]. That is the easy case: one document, an instruction to use only it. A personalized message asks for more (combine facts, infer relevance, sound natural), which gives a model more room to fill gaps. My inference, not a measured number, is that open personalized writing without controls sits well above the summarization floor.

Here is the arithmetic, labeled as illustration. Larkspur has about 250,000 contacts. Suppose each message makes two personal claims and 2% are unsupported before checking: 10,000 unsupported claims per send. If checks catch 90%, 1,000 still reach readers. Change the assumptions and the number moves; it does not go to zero. The question is which errors escape, and what they cost.

## The model may write; it may not decide what is true

Here is the hinge of this chapter: **a generated sentence may be only as specific as the evidence behind it, and something other than the model decides how much evidence there is.**

Everything else follows. The model does semantic work: selecting, connecting, phrasing. Deciding which facts are trustworthy, which surfaces may carry them, and how specific this message is permitted to be are policy outputs computed before the model is called. I build this as a **generation contract**: a machine-readable specification per zone that turns "personalize this" into something testable.

```yaml
zone: account_relevance
purpose: "One sentence on why this offer matters to this account now."
max_words: 35
tone: "experienced operations advisor; no flattery; no urgency language"
allowed_evidence:
  - memory.account.industry          # confidence >= high
  - memory.account.recent_signals    # observed_at within 90 days, source attached
  - memory.contact.role
prohibited_context:                  # never enters the prompt at all
  - seller_internal_notes
  - unverified_relationships
forbidden_claims:
  - customer_status_unless_confirmed
  - unverified_numbers
  - events_without_source_and_date
  - internal_notes
fallback:
  tier: industry
  source: approved_static_copy
validation: [grounded_personal_claims, no_forbidden_claims, max_words]
```

`allowed_evidence` is the only material the zone may be specific about. `prohibited_context` is stronger than a prompt instruction: the zone never sees it. `forbidden_claims` names the claim classes that are expensive when wrong; with the relationship unknown, "your current contract" becomes an evaluation, not an ownership claim. `fallback` defines the safe version in advance. `validation` names the post-generation checks, so contract and QA agree on what "correct" means.

Had Larkspur's pilot run under that contract, the acquisition would have been an `event` without a verified entity match, filtered out before the model saw it. The model cannot misuse a fact it was never given.

## A message is a set of typed zones, not one long generation

An email looks like one piece of writing. Operationally it is several decisions: subject, opening relevance, the value claim, proof, call to action. A seller brief has an account snapshot, recent signals, conversation openers, and source references.

I decompose each into **typed zones**, and each zone carries its own job, evidence rule, length, risk level, and fallback. A generator producing a twelve-word headline cares about compression. A generator summarizing recent account signals cares about sources and dates. Asking one prompt to do both makes each worse and makes failures impossible to locate. With zones, a defect has an address: "the signal card cites events older than 90 days" is fixable; "the email feels off" is not.

In the governed personalization engine I built, a landing page is a governed shell of roughly 8 to 12 text zones (headline, value proposition, industry framing, proof, call to action); legal copy and navigation never vary. The value-claim zone is not asked what would sound good; it picks from the campaign's approved capability map, and in selected engines numbers come only from customer-approved statistics.

![An email is broken into five typed zones: subject, opening relevance, value claim, proof, and call to action. The opening relevance zone is expanded into its generation contract: its purpose is why this offer matters to this account now; its allowed evidence is industry, role, and sourced signals within 90 days; it forbids unverified numbers, unsourced events, and internal notes; it runs 35 words at most; its fallback is the industry tier with approved static copy; and it is validated for grounded claims, no forbidden claims, and length. Because each zone has its own contract, a failure has an address.](/images/handbook/ch14-typed-zones.svg)

*Figure 15.1. Each zone of a message carries its own contract, so a defect points to one zone instead of "the email feels off."*

Zones also make optimization honest. A cherry-picked win shows you the ceiling; at scale you live on the floor. Tune each zone against a spread of real records (the thin-data account, the company whose name collides with a product) until the output holds across all of them.

Decomposition invites a trap: a separate autonomous agent for every zone. A specialized worker earns its existence only with a distinct combination of context, instruction, output schema, tools, evaluation, and fallback. Often several zones come from one call and are split deterministically, with an expensive reasoning model reserved for the one high-risk synthesis step. The goal is clean responsibility boundaries, not the largest number of agents.

## Every personal claim carries a receipt

Governed memory (Chapters 10 and 11) stores each fact with its value, source, confidence, recency, and the surfaces it may appear on. Generation should preserve that chain rather than dissolve it into prose. The practical mechanism is a **claim grounding check**: extract the concrete claims from the output (every named company, event, number, date, product, and relationship) and test each against the evidence the zone was allowed to use.

This is the production form of an idea from research. FActScore breaks generated text into atomic facts and measures what fraction a reliable source supports; in its original evaluation ChatGPT's biographies scored 58% [11]. Chain-of-Verification has a model draft, plan verification questions, answer them independently, and then revise, which reduced hallucination across several tasks [12]. In a personalization engine, the knowledge source is not Wikipedia. It is the person's own record, and every claim should cite it.

Not every claim needs the same scrutiny. "Field-service companies are under pressure on technician utilization" is a segment-level truth. "You acquired a competitor last month" is a high-risk personal claim. Named events, amounts, customer-status claims, exact headcounts, and executive changes get the strictest rule: grounded in allowed evidence, or removed.

In the engine I built, deterministic grounding runs today for selected high-risk classes (acquisitions, funding, certain numeric assertions) and flags, rewrites, or holds the output. It is not a universal fact checker. Full provenance on every customer-facing statement (claim, source, date, confidence, surfaces, content version) is the next step, not a result I can claim.

When a zone fails, replace the whole zone with its approved general version. Do not edit phrases in place. Surgical edits to generated prose wreck its grammar and leave sentences that half-assert the removed claim. Replace the section; do not repair the sentence.

## Specific and true, or honestly general

When the evidence is thin, the system should say less, on purpose. I use three tiers, which I call the **Specificity Ladder**: the mechanism by which a message steps down, rung by rung, toward honestly general.

| Tier | What the evidence supports | What the copy does |
|---|---|---|
| **Full** | Trusted identity, corroborated account facts, a recent verified signal | Speaks to the specifics: the real event, the role, the named detail |
| **Account** | Solid company facts; the person or the recency is uncertain | Talks about the company and its situation; nothing framed as "recent" without a date |
| **Industry** | Little that can be verified for this account | Says what is genuinely true for companies like this one; no personal claims |

![The Specificity Ladder shown as three descending steps. At the full tier, evidence includes trusted identity and a recent verified signal, and the copy speaks to the real event, role, and detail. As evidence thins, the message steps down to the account tier, where company facts are solid but the person or recency is uncertain, and the copy talks about the company and its situation. It steps down again to the industry tier, where little can be verified, and the copy says what is true for companies like this one with no personal claims. The tier is set in code by evidence, never by ambition, and the copy never fabricates to climb a rung.](/images/handbook/ch14-specificity-ladder.svg)

*Figure 15.2. The Specificity Ladder: the message steps down as evidence thins, and honestly general is a valid place to land.*

The tier is chosen by evidence, never by ambition, and the copy never fabricates to climb one. It is also chosen in code. Research suggests models are partly calibrated about what they know when asked [5], which makes self-assessment a useful signal. It is not a policy. Chapter 9's rule applies here: if identity confidence falls, personalization specificity should fall with it, and that rule should not depend on the model noticing.

This reverses a common instinct. Teams treat the industry tier as failure and push to raise the "personalization rate." But a well-written industry message is a success; a specific-sounding message built on an invented detail is a failure even when it reads better.

## The model reviews; code decides

Before a zone's output is accepted, the model re-reads it against the context: does each specific claim appear in the evidence, does it answer the zone's job, did it invent a relationship or capability, did a source or the recipient's name leak where it should not? This is cheap and catches a lot, because the context is still in attention.

It is also a soft control. Huang and colleagues found that without external feedback, models do not reliably correct their own reasoning [13]. I never let the same probabilistic component be the sole judge of whether it followed the rules. After self-review, deterministic validation runs: required fields, length, prohibited phrases, formatting leakage, internal provider names, unsafe relationship claims, and the claim grounding check. If a rule can be deterministic, it should not depend on the model remembering to obey it. Deciding where each rule lives is a design act:

| Requirement | Where it lives |
|---|---|
| Sound like an experienced advisor | Guideline |
| Headline at most 14 words; HTTPS links only | Code |
| No ownership claims unless confirmed | Guideline plus a code relationship guard |
| Lower specificity when identity is weak | Code policy |
| Pick the most relevant proof point | Model, within approved evidence |

Use the model where judgment adds value. Use code where certainty is available.

One check is easy to forget: a call that returns "completed" with an empty load-bearing section has not succeeded. **Technical completion is not semantic completion;** retry boundedly, then fall back or hold.

![A five-step pipeline alternates between code and the model. Code supplies evidence with source, date, and confidence, and code sets the tier (full, account, or industry). The model writes the zone inside its contract and then self-reviews, which is only a soft control. Code then validates the output with deterministic rules and the claim grounding check. For each extracted personal claim there are three outcomes: grounded claims that match allowed evidence let the zone ship; an ungrounded claim replaces the whole zone with its approved general fallback; and if the fallback also fails, the system fails closed and does not send.](/images/handbook/ch14-model-writes-code-decides.svg)

*Figure 15.3. The model writes and reviews, but code sets the tier and has the final word: replace the section; do not repair the sentence.*

Structured outputs belong in this layer, understood correctly. Constrained decoding guarantees that a response parses against your schema; Anthropic's documentation describes "schema-compliant responses through constrained decoding," with some JSON Schema features unsupported [6], and OpenAI reported its 2024 model scoring 100% on a complex-schema evaluation [14]. That solved a real engineering problem. It did not solve truth. A perfectly valid `{"recent_event": "Acquired Northwind Services"}` is still false. **Structured outputs guarantee the shape of a claim, not its truth.** On the way back into memory, the self-hosted memory system I built treats a generated value bound to a record property as optional by default, so an empty or invented answer is not written; requiring it surfaces the miss instead.

One more boundary: treat model output as untrusted input wherever it is rendered. Escape it, parse and allowlist its links, normalize structured values before templates see them. A generated sentence that reaches HTML is a security surface as well as a content one.

## One context, compiled into many channels

A mature engine does not build a separate customer model per channel. Channels compile from one governed context, and a **channel compiler** applies each surface's policy. A verified facility expansion might become one sentence in an email, a card on a landing page, a sourced talking point in a seller brief, and nothing on the outside of a mailer, because that surface has a different privacy contract. The channel changes the expression, not the underlying truth, which is why the same facts never contradict each other across surfaces (Playbook P9).

The seller brief is the dangerous neighbor: its hypotheses must never be copied into a buyer email. So whatever leaves passes an explicit serializer that allowlists fields; a new internal property is never exposed merely because someone added it.

The same logic extends beyond text. Personalized pages, offer selection, and adaptive product interfaces are zones too: the model chooses among approved components, offers, and layouts and fills their slots; it does not emit free-form HTML or invent a discount. Template-and-zone pages ship today. Fully generated interfaces composed per person exist as demos and early products; treat broad production use as forecast, and apply the same contract when it arrives (see the Closing). Chapter 16 carries the contract beyond the message: zones across whole pages and product screens, a render gate in front of every generated value, and notifications as a decision; Chapter 3 gives the usability evidence for keeping the frame fixed.

## What this does not do

**It does not make generation true in general.** A grounding check covers only the claim classes it can detect. It catches the invented acquisition; it may miss a wrong implication ("as you scale your fleet" to a company that is shrinking). Semantic review and human sampling still matter (Chapter 20).

**It does not rescue bad memory.** If the record holds the wrong employer, a perfect grounding check faithfully grounds the wrong claim. Identity and freshness (Chapters 9 and 11) come first.

**It will not win every test.** Honestly general copy can open worse than fake-specific copy in the short run. I accept that trade, because the fake-specific version spends trust the business needs later; but measure it rather than assume it (Chapter 21).

**It does not settle "as good as a human."** The studies above show preference and indistinguishability in narrow tasks [7][8][9]. The same Marketing Science study that found machine drafts human-like kept a human editor in the loop [10]. The honest comparison is not your best account executive on her best day. It is the attention you could actually afford for each of 250,000 people.

A popular claim deserves naming: that constrained decoding or "JSON mode" prevents hallucination. It prevents malformed output. Anyone who tells you schema compliance is a factuality control is selling the wrong guarantee.

One more use for the contract: its `forbidden_claims` list is also where you write down what the system must never use, even when it would predict a response.

## At scale

At a hundred messages, a person can read every one. At 250,000, the system must answer three questions without a person in the loop: what did we say, what evidence allowed it, and which tier did each message run at. That requires the claim ledger to be stored, not just computed, and the tier distribution to be a dashboard metric. A sudden rise in industry-tier messages usually means an upstream research source failed quietly. A sudden fall usually means a check stopped running.

Cost scales with fan-out: zones times channels times retries times review passes. Run cheap deterministic checks first, spend model intelligence on judgment, and sample semantic review across apparently clean records rather than reviewing everything. Two patterns from the engine I built keep retries cheap. Conditional best-of-two: score the first set deterministically (blank, wrong length, repetitive opening, missing approved proof); if clean, ship it; if not, generate once more and keep the lower-penalty set, paying double only where the first draft failed. Partial retry: when two emails of a sequence fail, regenerate those two, not the sequence. Beyond those two, treat guidelines, contracts, and validators as production artifacts: version them, test them against replay, held-out, and regression sets, and promote them with approval. In my engine, a person diagnoses and decides each fix today; three-set validation with approval-gated promotion is the design I am building toward (Chapters 18 and 20).

## Failure story: The Hallucinated Detail

Back to Larkspur's acquisition email, diagnosed by layer rather than blamed on "the AI."

- **Identity:** the research result was matched to the account by name similarity alone. No entity confidence, no domain agreement.
- **Context:** the article entered the prompt as a bare string: no source, no date, no confidence, no allowed surfaces.
- **Instruction:** "reference a recent company event" demanded specificity without permitting abstention. It asked the model to climb a tier the evidence did not support.
- **Validation:** nothing extracted the claim or checked it. The self-review, had there been one, would have found the article in context and approved it.

The fix was not a better prompt. It was an entity-confidence gate on research, provenance on every fact, a contract that forbids unsourced events, a grounding check, and an account-tier fallback. The anti-pattern: **specificity demanded, abstention forbidden.**

The failure class is not invented for the story. In the engine I built, acquisitions and funding are exactly the classes that earned deterministic grounding checks, and unsupported relationship language ("your renewal is coming up") earned its own guard. The strongest controls came from observed failures, each turned into a guideline, a code backstop, and a regression test.

## Patterns

**Generation Contract.** *Problem:* "personalize this" is not testable. *Forces:* campaigns differ; hard rules must not depend on prompt wording. *Solution:* per-zone, machine-readable spec of purpose, allowed evidence, forbidden claims, tone, length, fallback, and validation. *Tradeoffs:* upfront design work; contracts must be versioned like code.

**Claim Ledger (claim grounding check).** *Problem:* fluent text hides unsupported claims. *Forces:* not every claim can be checked; high-risk ones must be. *Solution:* extract atomic personal claims, match each to allowed evidence with provenance, store the result, replace the failing zone with its general fallback. *Tradeoffs:* detector coverage is partial; extraction adds latency and cost.

**Channel Compilers.** *Problem:* each channel rebuilds its own view of the customer, and surfaces contradict each other. *Forces:* channels have different lengths, audiences, and privacy contracts. *Solution:* one governed context; per-channel compilers that enforce surface policy and serialize only what may leave. *Tradeoffs:* compilers need their own tests; the rendered artifact, not the stored text, must be QA'd.

## Leader questions

1. For any sentence we sent last week, can we show the evidence that allowed it?
2. What share of messages ran at full, account, and industry tier, and who decided those thresholds?
3. Which claim classes are forbidden, and are they enforced in code or only in a prompt?
4. When a message fails validation, what does the customer receive instead?
5. Do our channels read from one customer context or several?

## Build checklist

- [ ] Every zone has a generation contract with allowed evidence, forbidden claims, and a fallback.
- [ ] Personalization tier is computed from identity and evidence confidence before generation.
- [ ] Facts enter the prompt with source, date, and confidence, never as bare strings.
- [ ] Rejected facts are removed from context, not flagged for the model to weigh.
- [ ] Self-review runs inside each zone; deterministic validation runs after it.
- [ ] High-risk personal claims are extracted and grounded; failures replace the whole zone.
- [ ] Structured outputs enforce shape; nothing in the design treats them as a truth check.
- [ ] Model output is escaped and link-allowlisted at every rendering boundary.
- [ ] Channel compilers enforce per-surface policy from one governed context; anything that leaves passes an explicit field allowlist.
- [ ] Contracts and guidelines are versioned and regression-tested before promotion.

## Metrics to watch

- **Ungrounded personal claim rate**, measured on sampled output after all checks (the escape rate, not the catch rate).
- **Tier distribution** over time (full / account / industry / fallback / fail closed).
- **Fallback trigger rate by zone**, which points to the zone or data source that needs work.
- **Cross-surface contradiction rate** from semantic QA.
- **Reader-flagged errors** ("you have the wrong company") per 10,000 messages.

## Reader Q&A

**Can we just use a bigger model and skip the checks?** Better models invent less, and the leaderboard spread shows the difference is large [4]. None is at zero, and at scale small rates become thousands of messages. Keep the checks; enjoy the lower fallback rate.

**Won't honest generality make us sound like everyone else?** Only if it is written badly. A sharp industry-level message that names a real pressure in the reader's world beats a fake-specific one that names the wrong event. Write the fallbacks as carefully as the full tier; many recipients will see them.

**Should a human approve every message?** Not at volume, and approval fatigue makes it unreliable anyway. Put people where judgment is scarce: approving contracts, reviewing samples, and signing off on high-consequence or physical sends (Chapter 18).

**Does this apply to chat, not just outbound?** Yes. In a conversation, each turn is a zone with the same contract. A tribunal held Air Canada responsible for what its chatbot told a customer [15]; the company owns what its generator says, whatever the channel.

## For your AI

```yaml
chapter: 15
concepts:
  - name: Generation Contract
    definition: "Machine-readable per-zone specification of purpose, allowed evidence, forbidden claims, tone, length, fallback, and validation."
  - name: Typed Zones
    definition: "Decomposition of a message or experience into small units, each with its own job, evidence rule, risk level, and fallback."
  - name: Claim Grounding Check
    definition: "Extract atomic personal claims from generated output and verify each against allowed, provenance-carrying evidence; replace the zone on failure."
  - name: Honestly General
    definition: "A lower personalization tier, chosen by evidence, that makes no personal claim it cannot support and remains useful."
  - name: Specificity Ladder
    definition: "The full, account, and industry tiers; the mechanism by which a message steps down toward honestly general as evidence thins. Distinct from Graceful Degradation (Chapter 17), which handles dependency and system failures."
  - name: Channel Compilers
    definition: "Per-surface transformers that express one governed context under each channel's length, audience, and privacy policy."
decision_rules:
  - if: "identity or evidence confidence for a zone is below its contract threshold"
    then: "drop to the account or industry tier before generation; never ask the model to decide"
  - if: "a generated high-risk personal claim has no matching allowed evidence"
    then: "replace the whole zone with its approved general fallback; do not edit the sentence"
  - if: "a rule can be expressed as code (length, links, forbidden phrases, internal fields)"
    then: "enforce it deterministically after generation, not only in the prompt"
  - if: "the fallback itself fails validation"
    then: "fail closed and do not send"
  - if: "a fact is allowed on one surface but not another"
    then: "enforce it in that surface's channel compiler"
  - if: "a call reports completion but a load-bearing section is empty, or the first set fails deterministic scoring"
    then: "retry boundedly (one alternative, lower penalty wins; only failed slots), then fall back or hold"
assessment_questions:
  - "Can you show, for any sent message, which evidence allowed each personal claim?"
  - "Is the personalization tier computed in code before generation, or left to the prompt?"
  - "Which claim classes are forbidden, and where is each enforced?"
  - "Do all channels read one governed customer context?"
  - "What does a recipient receive when generation or validation fails?"
patterns: [Generation Contract, Typed Zones, Claim Ledger, Channel Compilers, Specificity Ladder]
anti_patterns: [The Hallucinated Detail, Specificity Demanded Abstention Forbidden, Schema As Truth, Agent Zoo, Sentence Surgery]
maturity_dimension: experience_generation
```

## References

1. Maynez, J., Narayan, S., Bohnet, B., McDonald, R. (2020). "On Faithfulness and Factuality in Abstractive Summarization." ACL 2020. https://aclanthology.org/2020.acl-main.173/
2. Lewis, P. et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS 2020. https://arxiv.org/abs/2005.11401
3. Kalai, A. T., Nachum, O., Vempala, S. S., Zhang, E. (2025). "Why Language Models Hallucinate." arXiv:2509.04664. https://arxiv.org/abs/2509.04664
4. Vectara Hallucination Leaderboard (HHEM), accessed 2026-09-26; README updated 2026-09-22. https://github.com/vectara/hallucination-leaderboard
5. Kadavath, S. et al. (2022). "Language Models (Mostly) Know What They Know." arXiv:2207.05221. https://arxiv.org/abs/2207.05221
6. Anthropic, "Structured outputs," Claude Platform documentation, accessed 2026-09-26. https://platform.claude.com/docs/en/build-with-claude/structured-outputs
7. Ayers, J. W. et al. (2023). "Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum." JAMA Internal Medicine. https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2804309
8. Jones, C. R., Bergen, B. K. (2025). "Large Language Models Pass the Turing Test." arXiv:2503.23674 (preprint). https://arxiv.org/abs/2503.23674
9. Hartmann, J., Exner, Y., Domdey, S. (2025). "The power of generative marketing: Can generative AI create superhuman visual marketing content?" International Journal of Research in Marketing 42(1). https://www.sciencedirect.com/science/article/pii/S0167811624000843
10. Reisenbichler, M., Reutterer, T., Schweidel, D. A., Dan, D. (2022). "Frontiers: Supporting Content Marketing with Natural Language Generation." Marketing Science 41(3):441-452. https://pubsonline.informs.org/doi/abs/10.1287/mksc.2022.1354
11. Min, S. et al. (2023). "FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation." EMNLP 2023. https://arxiv.org/abs/2305.14251
12. Dhuliawala, S. et al. (2023). "Chain-of-Verification Reduces Hallucination in Large Language Models." arXiv:2309.11495. https://arxiv.org/abs/2309.11495
13. Huang, J. et al. (2024). "Large Language Models Cannot Self-Correct Reasoning Yet." ICLR 2024. https://arxiv.org/abs/2310.01798
14. OpenAI (2024-08-06). "Introducing Structured Outputs in the API." https://openai.com/index/introducing-structured-outputs-in-the-api/
15. Moffatt v. Air Canada, 2024 BCCRT 149 (2024-02-14). https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html

---

# 16. Beyond the Message: Generative Experiences

*The question: What happens when the page, the app, and the notification, not just the message, are composed for one person?*

> **Questions this chapter answers**
> - Should we personalize whole pages, or sections inside a page we control?
> - What is generative UI, and is it ready for our product and our website?
> - How do we personalize notifications without wearing people out?
> - How do we keep an interface usable when it is different for every person?
> - What must never be generated, and what happens when generation fails on screen?

## The short answer

A message is read once. An experience is used: a page someone navigates, a product screen they work in every morning, an alert that reaches their lock screen at 6:40 a.m. When AI composes those for one person, three things change. The output has a layout, so it can be broken, inaccessible, or unsafe to render. It is repeated, so a change the person did not expect costs them time every day. And a platform increasingly stands between you and the person, filtering, bundling, even rewriting what you send.

The practical path climbs a ladder. First, personalized zones inside a page you govern: this ships today and holds most of the value. Second, whole pages and product surfaces composed per person from a constrained vocabulary: this ships at large consumer companies, with small, real gains. Third, interfaces generated per person or per request: preferred to plain text by raters, and still too slow and unpredictable for the core of most products as of September 2026. Beside the ladder sits the notification, which is not content at all. It is a decision: whether to interrupt, with what, when, and how, with "not at all" as a scored option.

For a leader, two tests. "When the page changes for one customer, what stays exactly the same for everyone, and who decided that?" And: "How many people turned our notifications off last quarter, and do we count that as a cost?" If nobody can answer the first, the experience will drift. If nobody tracks the second, the channel is being spent without a budget.

## The summary that rewrote the news

In December 2024, iPhone users with Apple Intelligence, whose notification summaries had launched that October [30], saw a notification attributed to BBC News reporting that Luigi Mangione, the man charged with killing UnitedHealthcare's chief executive, had shot himself. He had not, and the BBC had never said so. Apple's on-device model had condensed a stack of BBC alerts into one line and invented a fact on the way. Other summaries had declared a darts final won before it was played and turned a warrant into an arrest [1].

Apple first promised clearer labels. Then, in the iOS 18.3 betas of January 2025, it switched summaries off for news and entertainment apps, italicized the rest, let people disable them per app from the lock screen, and warned in Settings that the feature "may contain errors" [2]. Six months later, in the iOS 26 betas, news summaries returned, labeled "Summarized by Apple Intelligence," with per-category controls [3]. When Google shipped notification summaries on Pixel phones in November 2025, it scoped them to long chats and group conversations, not news [4].

Notice who carried the error. The notification said BBC News. The BBC wrote none of it. A platform had composed a personal experience on the BBC's behalf, one lock screen at a time, and the sender's name was the one attached to the invention.

Personalization used to end at the message you wrote. Now the unit is the experience a person moves through, and parts of it are composed by you, parts by your model, and parts by someone else's.

## An experience is a message the person has to live inside

The Decision Layer (Chapter 14) chooses what should happen, including nothing. Writing for One (Chapter 15) turns a message into typed zones with contracts, so every personal claim is grounded or left out. That machinery carries over. What changes is the object. An experience has four properties a message does not.

**It has a layout.** A generated sentence can be false. A generated interface can also be unreadable to a screen reader, unsafe to render, or visually broken.

**It is interactive.** The person acts inside it, so the system makes decisions while someone is waiting.

**It is repeated.** A product screen is used daily, and every change the person did not predict costs them the time it takes to find what moved. A novel message is a small pleasure; a novel interface is often a tax.

**It is mediated.** Browsers, operating systems, assistants, and app hosts rank, bundle, summarize, and sometimes render your content inside their own frames.

Here is the sentence this chapter turns on: **the more of an experience you generate, the more of it you must hold still.** Everything that follows decides which parts may vary, how much, when, and with what fallback, so the parts that vary serve the person and the parts that stay put keep the experience learnable, safe, and accountable.

## The Generative Experience Ladder

Three rungs compose what the person sees. They differ in what varies, who owns the frame, how fast the result must arrive, and how much evidence of production readiness exists; climb only as far as the decision needs. A fourth surface, the notification, reaches the person uninvited. It sits beside the ladder because it is a decision before it is content.

![The Generative Experience Ladder, three rungs from most governed to least. Rung 1, zones inside a governed page: the model selects or fills declared zones while brand, navigation, and legal copy stay fixed; ships today. Rung 2, whole pages and product surfaces per person: a model composes the page from a constrained vocabulary such as rows or approved modules; ships at large consumer companies with small, real gains. Rung 3, interfaces generated per person or request through agent-to-UI protocols: preferred to plain text by raters but slow and still ranked below expert-built pages; an experiment in consumer assistants and a lab demo at website latency. Beside the ladder, the notification is a decision rather than a rung: if, what, when, and how, with not at all as a scored option. Each step up varies more of the experience and demands more of the frame, the renderer, and measurement.](/images/handbook/gx-generative-experience-ladder.svg)

*Figure 16.1. The Generative Experience Ladder: each rung varies more of the experience, and each one needs a stronger frame, renderer, and measurement to stay trustworthy.*

### Rung 1: zones inside a governed page

The first rung is a page designed once, with declared places where content varies per person. It predates language models. Netflix treats each title's artwork as a personalized element of a fixed slot, chosen per member by a contextual bandit, and shows one image per title per member at a time so titles stay recognizable [5]. A 2016 Yahoo paper, best paper at WSDM, learned positions, image sizes, and styles for a whole results page "as long as variations are within business and design constraints," and beat the leading presentation algorithm on real traffic [6]. Commercial testing platforms expose the same shape as named placements: the page asks for a slot's variation, the server resolves targeting and allocation, the page renders the result [7].

Language models add a second verb: a slot once filled by selection can now be filled by generation inside a contract. Chapter 15 covers the contract for one zone; the experience question is how the page holds together. In the governed personalization engine I built, a landing page is a branded template with roughly 8 to 12 declared zones, and legal copy and navigation never vary. Three properties make it an experience rather than a long email:

- **The link persists.** A personal page has an address the person can return to or forward. It must render the same governed content each time, or change only for a stated reason.
- **The identity is scoped.** The same company can sit in two campaigns, so the engine scopes page identity by campaign; one program cannot overwrite another's page for the same account.
- **Every zone has a fallback.** A zone that fails its checks renders approved segment copy. The page never shows a hole or a half-repaired sentence.

Select first, generate second. For each zone, ask whether choosing among five approved variants would serve the person almost as well as writing a sixth. Usually it would, and selection is cheaper, faster, testable by ordinary methods, and impossible to hallucinate. Reserve generation for the sentence that connects this person's stated problem to the offer.

### Rung 2: whole pages and product surfaces, per person

The second rung composes the entire page or screen for one person, from a vocabulary that is still constrained. The clearest public example is Netflix. In a paper accepted at RecSys 2026, its engineers describe GenPage, a single transformer that treats the member and request context as a prompt and generates the whole multi-row homepage, replacing a multi-stage recommender stack. Against a mature production system, the A/B test showed +0.24% on the core engagement metric, with serving latency 20% lower [8].

Three details matter more than the headline. The gain is small because the baseline was excellent; that is what real gains on mature surfaces look like. The model generates rows and titles, not free-form interface; the vocabulary is a grid. And offline, enriching the prompt with more context improved the model more than scaling it from 120 million to 900 million parameters [8]. What the system knew about the person mattered more than how big the model was: the memory argument of Part IV, made by a homepage.

For most companies this rung is less exotic than it sounds. At Larkspur Systems (the fictional field-service software company used throughout this handbook), the renewal page an account manager sends is composed from six approved modules. Which four appear, in which order, and what their zones say depends on the account's memory: offline sync for accounts whose tickets mention rural sites, the route-sync fix for the roughly 60 accounts that hit the defect. The frame, the pricing table, and the contract terms are the same for everyone.

### Rung 3: interfaces generated per person

The third rung generates the interface itself: layout, components, and interaction, composed for this person or request, with no designer having drawn it.

It exists. In November 2025, Google shipped an experiment in the Gemini app ("dynamic view") that designs and codes a custom interface in real time for a prompt [11]. Google's paper found generated interfaces "overwhelmingly preferred" to text-only answers and at least comparable to expert-made websites in about half of comparisons, but pages built by human experts still ranked first, and generation time was excluded [9]. Stanford researchers reported "up to" a 72% improvement in human preference over chat, a best case rather than an average [12].

For builders, the protocols matter more than the demos, and they point one way. **A2UI**, an open Google project, has the agent send a declarative JSON description, "not executable code," which the client renders from its own catalog of trusted, pre-approved components [14]. **MCP Apps**, co-developed by Anthropic, OpenAI, and the MCP-UI maintainers, lets a tool return a pre-declared interface template rendered in a sandboxed frame, with user consent for actions the interface initiates; it became an official extension in January 2026 [15], and OpenAI's apps in ChatGPT build on the same foundation [16]. The precedent is Microsoft's Adaptive Cards: "Purely declarative: No code is needed or allowed," with the host owning the look and feel [17]. I state the convergence as a rule: **catalog, not code.** The model chooses components and fills their fields. The host owns the components, the rendering, and the permissions.

The limits are specific.

**Latency.** Google's researchers note generation can take a minute or more [10]. Chapter 17 put an in-product interaction at about 0.1 seconds to feel instant, and Booking.com measured that about 30% more latency cost more than 0.5% in conversion [19]. A screen that arrives in 40 seconds is a different product, not a slower one.

**Learnability.** Nielsen Norman Group, broadly sympathetic to generative UI, names the cost: users rely on stable conventions and must relearn an interface that changes [13]. The chapter What Designers Knew traces the adaptive-interface research behind that warning.

**Accessibility.** Models routinely omit alt text, landmarks, and form labels. One team cut the inaccessibility rate of generated interfaces by 87.5% and 58.1% under two auditors, but only by training for it [18]. Accessible is engineered, not default.

**Testing.** You can QA a template. You cannot QA in advance an interface that did not exist until the person asked. Quality moves to the catalog, the renderer, and samples of what was actually shown (Chapter 20).

My labels, as of September 2026: interactive interfaces inside AI assistants **ship now**; generated interfaces in consumer apps **ship as experiments**; fully generated pages per visitor on a company's own site, at web latency, are a **lab demo** (the Closing gives the forecast). Vercel's 2024 SDK for streaming generated React components has since had that line of development paused [31]: the first architecture for a new capability is rarely the lasting one.

## The notification is a decision, not a message

A notification is the most personal surface most companies own and the one they treat most carelessly. It interrupts, in a space the person did not open, and the person can revoke it for good. When Android made notifications a runtime opt-in in 2022, one push provider's data across more than 16 million devices found gaming apps lost nearly a third of opted-in users over two years and news apps 19%, while finance and transport lost least [28]. Vendor data, but the direction is clear: people close a channel that is spent carelessly.

So I treat a notification as four decisions, in order, with a legitimate "no" at each.

![The notification decision as four questions in order, each with a legitimate no. IF: should this person be interrupted at all? Weigh value to them against their attention budget; not at all is scored like any other option. WHAT: which message, chosen from approved templates with a recency penalty so novelty does not wear out, and written to survive being summarized. WHEN: at a time the person can act, outside quiet hours, or held for a digest. HOW: which channel and which interruption level, with time sensitive reserved for what truly is. Every outcome, including silence, is logged, and disables and revoked permissions count as costs.](/images/handbook/gx-notification-decision.svg)

*Figure 16.2. The notification decision: IF, WHAT, WHEN, HOW, with "not at all" scored like any other option and every opt-out counted as a cost.*

**IF: should this person be interrupted at all?** LinkedIn's team wrote that sending on every eligible event "would result in too many notifications which would in turn annoy members," and built a near-real-time system that decides per member under a volume constraint [22]. Pinterest's system sets each user's weekly volume to optimize long-term engagement; it reduced volume while improving notification click-through and site engagement [23]. Both teams arrived at Chapter 14's do-nothing option, applied to interruption.

**WHAT: which message?** Duolingo's reminder system is the best-documented answer. Yancey and Settles treated the choice among hand-written templates as a bandit with two twists: some templates are ineligible for some people at some times ("sleeping" arms), and a template's effect recovers only after it rests, because novelty wears off. Against a random-template system that years of A/B tests had already made strong, a two-week experiment raised daily active users 0.5%, lessons 0.4%, and new users' recurring retention 2% [20]. Small numbers, and real ones. They came with a rule: the team could tune timing, templates, images, and localization but "could not increase the quantity of notifications without strong justification and CEO approval" [21]. It won by choosing better, not by sending more.

**WHEN: at a time the person can act.** A small 2014 study logged an average of 63.5 notifications a day per person [24]. In a within-subject experiment with 221 people, a week with alerts on raised inattention and hyperactivity symptoms compared with a week with them off [25]. People who went a day without push notifications were less distracted and more productive, and also more anxious about missing something [26]. The person is already managing a flood; sometimes the right "when" is tomorrow's digest.

**HOW: which channel, and how loudly.** Since iOS 15, each notification carries an interruption level: passive (delivered silently), active (the default), time sensitive (can break through Focus and the scheduled summary, if the person allows it), and critical (bypasses the ringer, by entitlement only). Apple's guidance is blunt: "Do not overuse their interruptive nature," and people can turn off time-sensitive alerts or all of an app's notifications [27]. The channel (push, in-app, email, or the account manager instead) is arbitrated against the same per-person budget as every other program (Chapter 14; Playbook P9).

Two consequences follow from the platform in the middle. **Write notifications that survive summarization:** fact first, self-contained, entity and verb in the same sentence, no meaning carried by tone or by sequence with other alerts. The platform can rewrite your words; it cannot take your name off them. **Treat permission as the budget:** the balance that matters is how many people still let you interrupt them, and every disable or revoked permission is a withdrawal you cannot easily reverse.

In the engine I built, notifications are not yet a shipped part of the personalization product; the design I am pursuing applies these four decisions to seller and operator alerts, drawing on the same account memory and governance as the pages and emails.

## Hold the frame still

The Product playbook applies this to in-product slots. The general form, for any rung:

**The frame is fixed.** Navigation, control locations, pricing, legal terms, and every control for cancelling, downgrading, consent, or turning something off stay out of the adaptive layer. They are the map.

**Change happens at predictable moments.** A part may update when a page loads or a session begins, never mid-task. Personalization that swaps a block after first paint moves things under the person's cursor (Chapter 17).

**The person can see why and undo it.** A generated module can carry a short, honest reason ("Because your team works rural sites") and a way to hide or reset it.

**Identity is stable.** Netflix's one-image-per-title rule [5] exists so a person recognizes yesterday's title today. Products, prices, and people need the same stability.

**What was shown is logged,** with the decision and the template, catalog, and guideline versions. Support cannot help with a screen nobody can reproduce.

Generate the parts; govern the frame.

## Rendering is where content becomes a security boundary

A generated sentence in an email can be false. A generated value that reaches a renderer can also be dangerous: a script, a link to somewhere it should not go, markup that breaks the page. Treat every model output as untrusted input at the point where it becomes an experience.

![A generated value passes five checks in order before it can be rendered. First, coerce it to the zone's type. Second, escape it so it is shown as content, never run as markup. Third, parse any URL and allow only approved schemes, refusing javascript: and data: links however confidently the model returned them. Fourth, confirm the component exists in the governed catalog. Fifth, validate it against the zone's contract. A value that passes renders; a value that fails any check is replaced by the zone's approved fallback, and if the fallback also fails, the zone renders nothing and the event is logged.](/images/handbook/gx-render-gate.svg)

*Figure 16.3. The render gate: generated output is untrusted input, and anything that fails a check becomes the approved fallback, never a repaired fragment.*

In the engine I built, this runs before any content reaches a rendered page: each value is coerced to its zone's type, HTML is escaped, URLs are parsed, only approved schemes pass, and a failure renders the zone's safe fallback. A `javascript:` or `data:` link is refused however confidently the model returned it. The protocols build the same idea into the platform: A2UI sends data to a catalog the client owns [14]; MCP Apps renders pre-declared templates in a sandbox and lets hosts block suspicious content before render [15].

Then review the rendered surface, not only the stored data. Fields can be correct while the live page is wrong: a stale template or cache, the wrong page identity, a hidden zone, malformed markup. In my engine, inspecting the actual page is part of the operator's QA workflow for that reason (Chapter 20).

The strictest version of this boundary is paper, which cannot be patched after it is mailed. The direction I am taking the engine, not yet a shipped product, uses the same governed context for print: an approved template with declared zones, a deterministic preflight stricter than for web copy, and a QR code that continues into a personalized page whose visit becomes a signal for the next decision (Playbook M3). The cost of an unrecallable error sets how strict the gate must be.

## Measure the experience, not the message

Chapter 21's tools apply, with three adjustments.

**The outcome is the task, not the click.** A page or screen succeeds when the person finishes what they came for: a form completed, a route dispatched, a renewal signed. A novel generated module attracts clicks for a while whether or not it helps. Measure task completion and time on task, and watch experienced users, who pay first when the frame moves.

**Effects are small and variants many.** GenPage's +0.24% [8] and Duolingo's +0.5% [20] were detectable because the populations were enormous. Test a few strong hypotheses about the frame and the catalog, let a bandit choose within slots, and keep a permanent holdout on the static experience. Do not trust offline wins: Booking.com found essentially no correlation between offline model gains and business gains across 23 comparisons [19].

**Engagement is the constraint.** In a two-year randomized trial in 18 Tennessee middle schools, 96% of students tried an AI tutor, but the median student used it in only 17% of practice sessions where they made a mistake; "the binding constraint appears to be engagement" [29]. A personal experience nobody uses has no effect. Measure use before lift.

For notifications, count the costs the channel hides: disables, quiet deliveries, revoked permissions, and unsubscribes, per program. A program that lifts this week's sessions while raising the disable rate is borrowing from next quarter, the ads-blindness shape from Chapter 21.

## Cost and latency: generate ahead, select live

Experiences are visited more than once, and someone is usually waiting. Take an illustrative known lead who opens a personal page four times. Generating per visit costs four generations, each on the waiting path. Generating once when the lead is accepted costs one, off the waiting path, and turns each visit into a fetch. That is how the known-lead page runs in the engine I built: the submission is acknowledged at once, the page is generated into zones in the background, validated, and ready before the person clicks.

The same logic prices the rungs. Selection inside zones is nearly free per view. A catalog-composed page costs one generation per person per meaningful change in their record. A generated interface costs a generation per request, plus the latency and the rendering risk. Notifications are the cheapest to generate and the most expensive to get wrong, because the cost is paid in attention and in permissions you cannot buy back.

## What this does not do

**It does not show that whole-page generation beats sections.** I found no public controlled comparison, on a commercial website, of zones in a fixed layout against generating the whole page. GenPage is the closest, and it generates a constrained grid [8]. My preference for zones inside a governed frame is a design judgment from building these systems and from the usability evidence, not a measured result.

**It does not make generated interfaces better than designed ones.** You will hear that users prefer generative UI. Google's evaluation, the most careful public one, found generated interfaces beat text, and expert-built sites still ranked first, with speed excluded [9]. The "72%" is a best case [12]. I share NN/g's direction toward interfaces shaped around outcomes [13]; I do not accept the stronger claim that every page will soon be generated fresh for every visitor.

**It does not fix a bad default or a bad volume.** If most users need the same adaptation, change the default for everyone. And a clever notification sent too often is still too often; Duolingo's gains came inside a volume its CEO guarded [21].

**The evidence on onboarding is thin.** I found no rigorous public study of LLM-generated or per-person onboarding flows. Treat large vendor-claimed activation lifts as unverified until your own holdout confirms them.

## At scale

At a few hundred customers, a designer can look at every personalized page. At 250,000 contacts the questions become statistical: what share of renders ran full, degraded, fallback, or empty; which zone falls back most and which data source is behind it; how many values the render gate refused, and why. A sudden rise in fallbacks usually means an upstream source failed; a sudden fall usually means a check stopped running.

Scale also brings the platform in. At millions of notifications, operating-system models summarize, group, and demote your content, and the signal you get back is partly theirs. Log what you sent, at which interruption level, and what you intended, so a platform change can be told apart from yours. And build accessibility checks into the render gate, not a quarterly audit: at scale nobody reviews a render before the person sees it.

## Failure story: Borrowed Urgency

*Larkspur Systems is a fictional composite company; the numbers here are illustrative.*

Larkspur's mobile app launched "smart alerts" for dispatchers. A model wrote each alert from the job's context, and a well-meant rule marked any alert about a customer job as time sensitive, so it would break through Focus and the scheduled summary. In testing, alerts were read faster. In the first month, the median dispatcher received about fourteen time-sensitive alerts a day, most of the "technician running ten minutes late" kind.

Dispatchers did what the operating system lets them do in one step: they turned off time-sensitive alerts for Larkspur, and many moved the app into the scheduled summary.

Five weeks later, the route-sync defect recurred overnight for a group of accounts. The alert that mattered, "Re-sync routes before 7 a.m. or technicians will see yesterday's jobs," was generated correctly, grounded correctly, and marked time sensitive. It arrived at 8:00 a.m., inside a summary, under three late-technician notices.

No single alert was wrong. The system had spent an interruption level it did not own, and the people it interrupted took it back. The fix was a per-dispatcher interruption budget, time sensitive reserved for a short approved list (a sync failure, a safety issue, a job cancelled within the hour), and "not at all" in every alert decision. The anti-pattern: **borrowed urgency**, using the loudest channel for ordinary news until the person turns the volume down for everything.

## Patterns

**Catalog, Not Code.** *Problem:* a model that generates interface markup can produce anything, including unsafe, inaccessible, or off-brand screens. *Forces:* generation adds real value in composition; the host must stay accountable for what renders. *Solution:* the model selects components from a governed catalog and fills typed fields; the host owns rendering, styling, accessibility, and permissions; every output passes a render gate with a fallback. *Tradeoffs:* expressiveness is limited to the catalog, which must be designed, versioned, and extended deliberately.

**The Four-Question Notification.** *Problem:* notifications are written as messages and sent whenever an event fires. *Forces:* interruptions have value and cost; permission is revocable; platforms summarize and filter. *Solution:* decide IF, WHAT, WHEN, and HOW in order, with "not at all" scored at every step, a per-person interruption budget shared across programs, a recency penalty on templates, and the loudest levels reserved for an approved list. *Tradeoffs:* fewer sends and slower-looking short-term metrics; requires counting disables as costs.

**Rendered-Variant Log.** *Problem:* per-person experiences cannot be reproduced, debugged, or measured without knowing what each person saw. *Forces:* storage cost; privacy of logged content. *Solution:* log, for every render, the variant, the deciding policy, the template, catalog and guideline versions, the tier (full, degraded, fallback, empty), and any refusals by the render gate, under the same retention rules as the memory it came from. *Tradeoffs:* log volume; requires discipline about what is stored and for how long (Chapter 19).

## Leader questions

1. When a page or screen changes for one customer, what stays exactly the same for everyone, and who owns that list?
2. Which rung of the ladder are we on for each surface, and what evidence says we should climb?
3. Does any model output reach a renderer without type coercion, escaping, URL checks, and a fallback?
4. How many people disabled or quieted our notifications last quarter, per program, and who counts that as a cost?
5. Can support see exactly what a given customer was shown yesterday, and why?

## Build checklist

- [ ] Every personalized surface has a written frame: the parts that never vary, including all cancel, downgrade, consent, and turn-off controls.
- [ ] Every zone or slot has a contract (Chapter 15) and an approved fallback; selection is tried before generation.
- [ ] Generated interfaces use a governed component catalog; no model output is rendered as code.
- [ ] A render gate coerces, escapes, checks URLs and schemes, confirms catalog membership, and falls back on failure.
- [ ] Accessibility checks run on rendered output, not only on templates.
- [ ] Per-person pages are generated ahead of the visit and served as a fetch; the live path only selects.
- [ ] Personal pages have persistent, campaign-scoped identity so one program cannot overwrite another.
- [ ] Notifications go through IF, WHAT, WHEN, HOW, with "not at all" logged as an outcome and a per-person budget shared across programs.
- [ ] Time-sensitive and critical levels are reserved for an approved event list.
- [ ] Every render and every notification decision is logged with versions and tier, and the rendered surface is reviewed, not only stored data.

## Metrics to watch

- **Task completion and time on task** for the person's job, with experienced users tracked separately.
- **Tier distribution per surface**: full, degraded, fallback, empty, and render-gate refusals.
- **Notification permission health**: disable, quiet-delivery, and revocation rates per program, against a holdout.
- **Do-nothing share** of notification decisions, watched for sudden drops.
- **Accessibility defect rate** on sampled rendered output.

## Reader Q&A

**Should we start with generative UI in our product?** Start with zones and slots. Generated interfaces are worth prototyping for exploratory tasks where people are already waiting for an assistant (analysis, planning, configuration), not for the screen someone uses fifty times a day.

**Can we let the model decide when to send a notification?** Let a model score the value and the timing; keep the budget, quiet hours, approved interruption levels, and the do-nothing option in deterministic policy. The model proposes; policy decides.

**What about summaries the platform writes of our notifications?** You cannot control them, so write for them: fact first, self-contained, unambiguous. If a notification is safety- or money-critical, say it in a way that cannot be summarized into its opposite, and consider a second channel.

## For your AI

```yaml
chapter: 16
concepts:
  - name: Generative Experience Ladder
    definition: "Rungs of composing an experience per person: zones inside a governed page; whole pages and product surfaces from a constrained vocabulary; interfaces generated per person or request. Notifications sit beside the ladder as a decision."
  - name: Stable Frame, Generated Parts
    definition: "Navigation, controls, pricing, legal terms, and all cancel, downgrade, consent, and turn-off controls stay fixed; only declared parts vary, at predictable moments, with explanation and undo."
  - name: Catalog, Not Code
    definition: "Models select and fill components from a host-owned, governed catalog; the host renders, styles, and permissions them. No model output is rendered as code."
  - name: Render Gate
    definition: "Coerce to type, escape, parse URLs and allow approved schemes, confirm catalog membership, validate against the zone contract; on failure render the approved fallback, then nothing."
  - name: Notification Decision
    definition: "IF, WHAT, WHEN, HOW, decided in order, with not at all scored as an option and a per-person interruption budget shared across programs."
  - name: Permission as Budget
    definition: "The notification channel's real balance is how many people still allow interruption; disables and revocations are costs."
  - name: Summary-Proof Notification
    definition: "A notification written so it stays true when a platform model bundles or condenses it: fact first, self-contained, unambiguous."
decision_rules:
  - if: "a zone can be served nearly as well by selecting an approved variant"
    then: "select; generate only where no approved variant fits the person's situation"
  - if: "a proposed adaptation would move navigation, a control, pricing, legal terms, or a cancel or consent control"
    then: "do not adapt it; keep it in the fixed frame"
  - if: "any model output is about to be rendered"
    then: "pass it through the render gate; on failure render the approved fallback, never a repaired fragment"
  - if: "an interface is generated for a surface with an interactive latency budget (about 0.1 s)"
    then: "do not generate at request time; precompute or select from a catalog"
  - if: "a per-person page will be visited more than once"
    then: "generate it once ahead of the visit, store it with campaign-scoped identity, and serve each visit as a fetch"
  - if: "an event could trigger a notification"
    then: "decide IF, WHAT, WHEN, HOW in order; score not at all; check the person's interruption budget"
  - if: "a notification would use a time-sensitive or critical interruption level"
    then: "allow it only for events on an approved list"
  - if: "notification disable or quiet-delivery rates rise for a program"
    then: "treat it as a cost against that program's lift, and reduce volume before rewriting copy"
  - if: "the surface is physical or otherwise unrecallable"
    then: "apply stricter deterministic preflight and approval than for web copy"
assessment_questions:
  - "Which surfaces are personalized today, and on which rung of the ladder is each?"
  - "What is fixed in each surface's frame, and is that list written down?"
  - "Does model output ever reach a renderer without coercion, escaping, URL checks, and a fallback?"
  - "Are per-person pages generated ahead of the visit or on request?"
  - "How are notification decisions made, and is not sending an explicit, logged outcome?"
  - "What are the notification disable and revocation rates per program?"
  - "Can support reproduce what a given person was shown and why?"
patterns: [Catalog Not Code, Four-Question Notification, Rendered-Variant Log, Stable Frame Generated Parts, Generate Ahead Select Live, Generation Contract, Do-Nothing Option]
anti_patterns: [Borrowed Urgency, The Shifting Map, Generated Frame, Sentence Surgery, Novelty as Lift, Summarizable Alert]
metrics: [task completion and time on task, tier distribution per surface, notification permission health, do-nothing share, accessibility defect rate]
links: {decision_layer: 14, generation: 15, latency: 17, governance: 18, privacy: 19, accuracy: 20, measurement: 21, product: PR, inbound: M1, physical: M3, coordination: P9, forecast: closing, adaptive_interfaces: 3}
maturity_dimension: experience_generation
```

## References

1. The Register (2025-01-07). "Apple responds to BBC complaint about AI-generated notification summaries." *The Register*. https://www.theregister.com/2025/01/07/apple_responds_bbc_complaint/
2. TechCrunch (2025-01-16). "Apple pauses AI notification summaries for news after generating false alerts." https://techcrunch.com/2025/01/16/apple-pauses-ai-notification-summaries-for-news-after-generating-false-alerts
3. MacRumors (2025-07-22). "iOS 26 Beta 4 Reintroduces Notification Summaries for News Apps." https://www.macrumors.com/2025/07/22/ios-26-beta-4-notification-summaries/
4. 9to5Google (2025-11-13). "Pixel AI Notification Summaries start rolling out." https://9to5google.com/2025/11/13/pixel-notification-summaries/
5. Chandrashekar, A., Amat, F., Basilico, J., and Jebara, T. (2017-12). "Artwork Personalization at Netflix." Netflix Technology Blog. https://netflixtechblog.com/artwork-personalization-c589f074ad76
6. Wang, Yin, Jie, Wang, Yamada, Chang, and Mei (2016). "Beyond Ranking: Optimizing Whole-Page Presentation." WSDM 2016, pp. 103-112. https://dl.acm.org/doi/10.1145/2835776.2835824
7. Dynamic Yield developer documentation, "Choosing Variations" (Choose API), accessed 2026-09-28. Cited as a representative example of the slot model, not an endorsement. https://dy.dev/docs/choose
8. Wang, L., Pan, J., and Baltrunas, L. (2026). "GenPage: Towards End-to-End Generative Homepage Construction at Netflix." arXiv:2606.31031; ACM RecSys 2026. https://arxiv.org/abs/2606.31031
9. Leviathan, Valevski, Kalman, Lumen, Segalis, Molad, Pasternak, Natchu, Nygaard, Venkatachary, Manyika, and Matias (2025-11). "Generative UI: LLMs are Effective UI Generators." Google Research. https://generativeui.github.io/static/pdfs/paper.pdf
10. Leviathan, Y., Valevski, D., Natchu, V., and Matias, Y. (2025-11-18). "Generative UI: A rich, custom, visual interactive user experience for any prompt." Google Research blog. https://research.google/blog/generative-ui-a-rich-custom-visual-interactive-user-experience-for-any-prompt/
11. Woodward, J. (2025-11-18). "Gemini 3 brings upgraded smarts and new capabilities to the Gemini app." Google blog. https://blog.google/products-and-platforms/products/gemini/gemini-3-gemini-app/
12. Chen, Zhang, Zhang, Shao, and Yang (Stanford SALT) (2025, rev. 2026). "Generative Interfaces for Language Models." arXiv:2508.19227; ACL 2026 Findings. https://arxiv.org/abs/2508.19227
13. Moran, K., and Gibbons, S. (2024-03-22). "Generative UI and Outcome-Oriented Design." Nielsen Norman Group. https://www.nngroup.com/articles/generative-ui/
14. Google A2UI Team (2025-12-15). "Introducing A2UI: An open project for agent-driven interfaces." Google Developers Blog. https://developers.googleblog.com/introducing-a2ui-an-open-project-for-agent-driven-interfaces/ ; project site https://a2ui.org/ (accessed 2026-09-28)
15. Model Context Protocol blog (2025-11-21). "MCP Apps: Extending servers with interactive user interfaces." https://blog.modelcontextprotocol.io/posts/2025-11-21-mcp-apps/ ; and (2026-01-26) "MCP Apps: Bringing UI Capabilities to MCP Clients." https://blog.modelcontextprotocol.io/posts/2026-01-26-mcp-apps/
16. OpenAI (2025-10-06). "Introducing apps in ChatGPT and the new Apps SDK." https://openai.com/index/introducing-apps-in-chatgpt/ (confirmed via secondary reports; page not fetched)
17. Microsoft Learn. "Adaptive Cards Overview," updated 2025-12. https://learn.microsoft.com/en-us/adaptive-cards/
18. Yoon et al. (2025, rev. 2026). "A11yn: Aligning LLMs for Web Accessibility-Aware UI Generation." arXiv:2510.13914. https://arxiv.org/abs/2510.13914
19. Bernardi, L., Mavridis, T., and Estevez, P. (2019). "150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com." KDD 2019. https://dl.acm.org/doi/10.1145/3292500.3330744
20. Yancey, K. P., and Settles, B. (2020). "A Sleeping, Recovering Bandit Algorithm for Optimizing Recurring Notifications." KDD 2020. https://research.duolingo.com/papers/yancey.kdd20.pdf
21. Mazal, J. (2023-02-28). "How Duolingo reignited user growth." Lenny's Newsletter. https://www.lennysnewsletter.com/p/how-duolingo-reignited-user-growth
22. Gao, Gupta, Yan, Shi, Tao, Xiao, Wang, Yu, Rosales, Muralidharan, and Chatterjee (LinkedIn) (2018). "Near Real-time Optimization of Activity-based Notifications." KDD 2018. https://dl.acm.org/doi/10.1145/3219819.3219880
23. Zhao, Narita, Orten, and Egan (Pinterest) (2018). "Notification Volume Control and Optimization System at Pinterest." KDD 2018. https://dl.acm.org/doi/10.1145/3219819.3219906
24. Pielot, M., Church, K., and de Oliveira, R. (2014). "An In-Situ Study of Mobile Phone Notifications." MobileHCI 2014. https://dl.acm.org/doi/10.1145/2628363.2628364
25. Kushlev, K., Proulx, J., and Dunn, E. W. (2016). "'Silence Your Phones': Smartphone Notifications Increase Inattention and Hyperactivity Symptoms." CHI 2016. https://dl.acm.org/doi/10.1145/2858036.2858359
26. Pielot, M., and Rello, L. (2017). "Productive, Anxious, Lonely: 24 Hours Without Push Notifications." MobileHCI 2017; arXiv:1612.02314. https://arxiv.org/abs/1612.02314
27. Apple (2021). "Send communication and Time Sensitive notifications." WWDC21 session 10091. https://developer.apple.com/videos/play/wwdc2021/10091/
28. Pushwoosh (2024-10-07). "Android 13 impact on opt-in rates." Vendor research. https://www.pushwoosh.com/blog/android-13-opt-in-rates-research/
29. Oreopoulos and Low (2026-08). "One Click Away: AI Tutoring with Khanmigo in a Two-Year School Experiment." EdWorkingPaper 26-1551, Annenberg Institute at Brown University. https://doi.org/10.26300/kner-hv33
30. Apple Newsroom (2024-10-28). "Apple Intelligence is available today on iPhone, iPad, and Mac." https://www.apple.com/newsroom/2024/10/apple-intelligence-is-available-today-on-iphone-ipad-and-mac/
31. Palmer, Ding, Leiter, Shadcn, Grammel, and Philemon (2024-03-01). "Introducing AI SDK 3.0 with Generative UI support." Vercel. https://vercel.com/blog/ai-sdk-3-generative-ui ; note current Vercel docs mark AI SDK RSC development as paused.

---

# 17. Now or Later

*The question: When must personalization be real-time, and when is async better?*

> **Questions this chapter answers**
> - Which of our personalization actually has to happen in real time?
> - Is it worth making the customer wait for a smarter answer?
> - How do we get deep, researched personalization onto a page that loads in under a second?
> - What breaks when we move work into queues and background jobs, and how do we stop it?
> - What should the customer see when the smart path fails?

## The short answer

Most personalization does not need to be computed in the moment. It needs to be *served* in the moment. Those are different engineering problems.

Speed matters where a person is waiting: controlled experiments at Google, Bing, and Booking.com measured real losses from added delay on pages people were using. But the deep work (researching an account, deciding what matters to this person) can almost always be done earlier, stored, and read back in milliseconds.

So the design rule is: precompute depth, add live context at the last moment. Keep the real-time path to a lookup plus a little fresh signal. Both major model providers now price patience: their batch interfaces charge half the synchronous rate.

Async is not free. Queues trade slowness for duplication, retry storms, and stale results. The cures are boring: idempotency keys, bounded retries with jitter, freshness windows, and a fallback ladder that makes output less specific when something fails, never less true.

What to fund: a latency budget per channel, a precompute layer keyed to the account or person rather than the request, and a degradation ladder agreed in writing.

---

Tuesday, 10:14 a.m. An operations director at a regional plumbing and HVAC contractor, a customer of Larkspur Systems (a fictional composite: field-service and fleet software, about 40,000 accounts), clicks a renewal email and lands on the pricing page. Larkspur knows her company opened a second depot this spring and added about twenty vans, that her team opened four tickets about route exports, and that her contract renews in 104 days. The page has well under a second to reflect any of that. If it stalls to think, she scrolls past a blank block, or watches it jump when the personalized version finally paints.

In the same minute, in a background worker, an agent is writing the renewal plan for that account. It reads eighteen months of tickets, call notes, telemetry, and news about her company. It will take most of an hour. It should. Nobody is waiting, and every minute of reading makes the plan better.

Same customer, same company, same morning. One piece of personalization is wrong if it is slow. The other is wrong if it is fast.

## Latency is a budget, and each channel sets its own

Start with what is measured. In 2009, Jake Brutlag at Google injected 100 to 400 ms of server delay into search results and saw daily searches per user fall 0.2% to 0.6%; after a 400 ms delay was removed, affected users stayed 0.21% below control for five weeks [1]. Ronny Kohavi's team at Bing isolated speed and estimated every 100 ms of speedup at about 0.6% of revenue [2]. At Booking.com, about 30% more latency cost more than 0.5% in conversion, which pushed the team to precompute and cache predictions where it could [3].

Small per interaction, large in aggregate. One qualification matters more than the headline: slowing Bing's right-hand pane by 250 ms produced no significant change in key metrics [2]. Delay hurts where the person is waiting. Delay nobody is waiting for is nearly free.

A **latency budget** is the time a channel allows before the delay itself changes the outcome. The channel sets it, not your architecture.

| Channel or moment | Working budget (design guidance) | What runs inside it |
|---|---|---|
| Web page, first paint (waiting) | A few hundred ms of server time; "good" Largest Contentful Paint is 2.5 s at p75 [5] | Lookup of precomputed content plus live context |
| In-product interaction (waiting) | About 0.1 s to feel instant; 1 s before flow breaks [4] | Lookup, small model score, rules |
| Chat or agent reply (waiting) | First visible output within about 1 s; the rest streams | Memory retrieval plus streamed generation |
| Triggered email (not waiting) | Minutes | Retrieval, generation, validation |
| Rep brief, account plan (not waiting) | Hours to a day | Multi-step agentic research |
| Direct mail (not waiting) | Days | Everything, plus human review |

The first rows are anchored in human-factors limits (Miller 1968, restated by Nielsen: 0.1 s feels instantaneous, 1 s keeps the flow of thought, 10 s holds attention) [4] and Google's page-experience thresholds [5]. The rest are my working targets, not standards. Write yours down before anyone argues about architecture.

![Six channels grouped by whether a person is waiting. Where someone waits, the budgets are tight: a web page's first paint gets a few hundred ms of server time for a lookup plus live context; an in-product interaction has about 0.1 s to feel instant and 1 s before flow breaks, enough for a lookup, a small score, and rules; a chat or agent reply shows first output in about 1 s and streams the rest. Where nobody waits, budgets widen to minutes for a triggered email, hours to a day for a rep brief or account plan, and days for direct mail, which is enough for retrieval, generation, validation, multi-step agentic research, and human review. The lower rows are working targets, not standards.](/images/handbook/ch15-latency-budget-by-channel.svg)

*Figure 17.1. Where a person is waiting, only lookup and selection fit; everything deeper belongs where nobody is.*

Watch layout shift too; Google's "good" threshold is 0.1 or less [5]. Late personalization that replaces a default block makes the page jump. Late personalization is not just slow. It is visibly unstable. Generated experiences stretch the budget furthest: a generated interface can take a minute or more, so Chapter 16 generates pages ahead of the visit and leaves the live path only to select.

## Do the work before they arrive

If the page has a few hundred milliseconds and account research takes minutes, the page need not be shallow. The research must already be finished.

Netflix described this split in 2013 as three tiers: **offline**, with few limits on data or complexity but prone to staleness; **online**, fresh but bound by a tight service level; and **nearline**, computed on events but stored for later serving. The tiers are not either/or: precompute part of a result offline and leave the cheaper, more context-sensitive parts online [6].

Language models make that split more valuable, because the expensive step got more expensive and more useful at once. The sleep-time compute research makes the economics explicit: processing a known context before the question arrives cut test-time compute about 5x at equal accuracy in its benchmarks, and the benefit grew with how predictable the queries were [7]. Personalization is unusually predictable. You know which accounts renew next quarter, and which facts a pricing-page visit will need.

I call the pattern **Precompute-then-serve**. The expensive understanding runs asynchronously and is written to memory as a stored artifact: an account synthesis, a ranked list of what matters now, pre-generated content zones. The real-time path reads it, adds what could only be known in the moment, and renders.

![Two lanes. In the async lane, where nobody is waiting, an event or a schedule triggers deep research keyed to the entity rather than the request, and the result is written as a stored artifact (account synthesis, ranked facts, content zones) with provenance and a freshness window. In the real-time lane, where a person is waiting a few hundred milliseconds, the system looks up that artifact (a fetch, not a generation), combines it with live context such as the pricing page, the email click, and the phone, selects among precomputed options, and renders. Only the part that depends on the last few seconds runs live.](/images/handbook/ch15-precompute-then-serve.svg)

*Figure 17.2. Precompute-then-serve: the slow understanding happens before the visit, so the visit is a lookup plus a selection.*

In the governed personalization engine I built, the inbound request validates the lead, persists workflow state, claims an idempotency key, dispatches the work, and returns 202 Accepted. Research, scoring, and the personalized landing page are produced in the background and the page link goes back with the callback, so the click is a fetch, not a generation. A production burst taught me not to hold that request open [16]. And research is keyed to the entity, not the request. When the unit was the individual lead, one popular employer was researched close to a hundred times in a day. Keyed to the company with a seven-day freshness window, the first record paid and the rest read, and records classified as research-rich rose from 60.3% to 93.5% as budget moved from known answers to open gaps [8]. The fastest personalization is the one you finished before they arrived.

## Event, schedule, or request

Async work still needs a reason to start. There are three.

- **Event-triggered**: something happened (a cancellation click, a fourth ticket in a week, a champion changing jobs) that changes what should be said. Seconds to minutes.
- **Scheduled**: the calendar is the trigger (renewals in 90 days, a weekly account refresh, the curation pass Chapter 11 calls dreaming). For predictable value and slow-changing inputs.
- **On-request**: someone asks, and the work runs inside their wait. Reserve it for what genuinely depends on the request.

Most teams default to on-request because it is easiest to build, which puts the most work inside the tightest budget.

## Depth is decided early; relevance is decided now

Here is the hinge. The question is never "real-time or batch?" It is: **which part of this decision depends on something that happened in the last few seconds?** That part runs live. Everything else runs earlier.

For the operations director, the new depot, the tickets, and the renewal date were all known yesterday. What is new is that she is on the pricing page, from the renewal email, on a phone. That live context selects among precomputed options: the renewal variant, the mobile layout, the route-export zone. Selection is fast. Understanding was slow, and it already happened.

Real-time is where you use what you know. Async is where you come to know it.

## Async is where the failures move

Moving work out of the request path changes the shape of failure. A slow request fails in front of the customer; a queued job fails silently, repeatedly, or twice.

Many model-call failures are not wrong answers at all. In Datadog's analysis of customer LLM traffic, 2% of model call spans errored in March 2026 and rate limits were almost a third of those; in February, 5% errored and 60% of errors were rate limits [9]. Capacity, not intelligence, is the first thing to design for. When a fleet of workers is refused in the same second and retries on the same interval, the retries land as a synchronized wave on a provider that is still saturated.

The disciplines are dull and non-negotiable:

1. **A concurrency cap** per provider, so parallelism is a number you chose, not one your traffic discovered.
2. **Bounded retries with exponential backoff and jitter**, at one layer of the stack, not several [10].
3. **Retry faults, not decisions.** Retry a timeout, a rate limit, a provider 5xx. Do not retry missing context, an invalid contract, or a policy rejection; that pays for the same "no" twice. Know where it failed, too: in the self-hosted memory system I built, a 429 comes before any write and is always safe to retry after its Retry-After interval; a bare 500 mid-save may already have written, so it is investigated first [17].
4. **Idempotency keys** on every mutating step (intake, generation, send, callback) [11], derived from the work (lead, campaign, step) rather than the request, so a redelivery carries the same key. Queues redeliver and networks fail after the server finished; without a key, each becomes a second generation and possibly a second, different message to the same person.
5. **Recover the smallest failed unit.** If the output is correct but the callback failed, resend the exact payload; regenerating costs money and produces a different message. If only the page is missing, regenerate the page alone. Both need a durable journal of each step's state (accepted, callback delivered, page pending, terminal error), not logs [16].

Refuse at your own front door too. The memory system rejects work it cannot run, on separate budgets so a write overload does not stop reads, rather than accepting everything and silently falling behind [17]. Only bulk work lands in its durable queue; a refused single save is not queued on the caller's behalf, so the caller retries or moves the load to the bulk path.

A queue turns a busy minute into a slow minute. Without idempotency, it turns a slow minute into a duplicate.

## Degrade specificity, never truthfulness

Netflix's 2013 advice for the online tier was to always have a fast fallback, such as a precomputed result [6]. AI personalization needs more than one cached answer, because some of its failures produce plausible output.

The name is deliberately narrow: **Graceful Degradation** is only for dependency and system failures (a throttled provider, a timeout, a missing upstream artifact). It is not Honestly General, the content principle that decides what to say when the evidence is thin (stepped down through Chapter 15's Specificity Ladder); this ladder decides what to serve when the machinery fails. The ladder I use follows the decision modes of Chapter 14:

1. **Agentic**: live reasoning over memory and fresh context. Most specific, slowest, most expensive.
2. **Predictive**: a precomputed or model-scored choice among approved variants. Fast, still personal.
3. **Rule-based**: segment or lifecycle-stage content selected by deterministic rules.
4. **Approved static**: the known-safe default for that zone.
5. **Fail closed**: publish nothing, when even the default would break the contract or the experience.

A missed deadline counts as a system failure. When rich enrichment fails close to its execution deadline, the engine I built emits a deterministic, lower-specificity brief rather than inventing detail or leaving the record stuck [16].

Each zone declares its own ladder, and every output is labeled full, degraded, or failed so it can be counted. A system that falls back silently looks healthy while serving generic content to a large share of its audience. A specific but false message is worse than a general but true one, so the ladder moves toward less specific, never toward less verified.

![A five-rung staircase triggered only by a dependency or system failure (throttled provider, timeout, missing upstream artifact). From most to least specific: agentic, live reasoning over memory and fresh context, labeled full; predictive, a precomputed or scored choice among approved variants; rule-based, segment or lifecycle content chosen by deterministic rules; approved static, the known-safe default for the zone, these three labeled degraded; and fail closed, publishing nothing when even the default would break, labeled failed. Verification is held on every rung. Each zone declares its own ladder, and every output is labeled so fallbacks can be counted.](/images/handbook/ch15-degradation-ladder.svg)

*Figure 17.3. Graceful Degradation steps down in specificity, never in truthfulness, and labels every step so silent fallback shows up in the numbers.*

## What this does not do

It does not make latency statistics portable. Kohavi's own rules of thumb include "your mileage will vary" [2]. Search engines and a travel marketplace do not tell you what 300 ms costs on your B2B pricing page. Run your own slowdown test first.

Two popular numbers should not be used at all. "Every 100 ms cost Amazon 1% of sales" is usually traced to Greg Linden's 2006 slides, with no published method. "Google lost 20% from a 500 ms delay" came from a test that also raised the results per page from 10 to 30; when Bing isolated speed, 500 ms cost about 3% of revenue, "not 20%" [2].

It does not make real-time better whenever you can afford it. Rewriting a page on a single click is often worse than a stable page informed by the last month; the adaptive-menu research in Chapter 3 shows why. Fresh does not mean important.

And a disagreement with a respected source. Jay Kreps argued against the Lambda architecture's separate batch and streaming code paths [12]: use one streaming path and reprocess by replay [13]. I agree with the first half: one code path and one generation contract for both routes. I disagree that replay is cheap here. Replaying a language-model pipeline costs money per record and does not reproduce the same outputs. Store generated artifacts with provenance, and recover missing pieces instead of replaying history.

## At scale

At 250,000 contacts, precompute has its own waste. A page for every contact, when a small fraction will ever visit, is content nobody reads. Precompute where reuse is high (the account synthesis, the ranked facts) and generate the person-specific layer on the event that signals interest.

Staleness scales too. Every artifact needs a freshness window and invalidation events: a job change, a cancellation, a closed ticket. A cache that only accumulates serves a confident wrong answer until it expires.

Background work must not starve the foreground. In the memory system, overnight curation claims queue slots only after waiting live work and is admitted only up to the ceiling minus a reserve (20 slots by default), so it backs off instead of crowding out a live request [17]. The bulk worker yields the same way but never stops: under a sustained burst it still takes a small slice, so an import slows rather than stalls.

Admission limits have their own scale traps, all from sizing that system [17]. **Limits are per process**: each replica counts only its own traffic, so three replicas admit roughly three times every number, and a team that forgets the multiplier sizes wrong. **Tune before you enforce**: a dry-run mode admits everything but records what would have been refused, so ceilings come from real traffic, not guesses. Raising admission without database connection headroom only moves the wait from the front door to the database. And a per-tenant **backlog ceiling** on the bulk queue caps what one runaway producer can claim, but it governs new intake, not work already accepted: when a provider's batch fails and its items are requeued, they go back even past the ceiling, because the caller was already told they were queued.

And cost follows timing. Anthropic's Message Batches API and OpenAI's Batch API each list a 50% discount for work that can wait up to 24 hours (as of 2026-09-26) [14][15]. The async decision is also a pricing decision. Make it per job, with a way back: a batch the provider fails is requeued to the normal worker, so nothing is lost [17].

## Failure story: The Double Send

A composite at Larkspur, built from failure modes I have debugged. At 9:00 a.m. a renewal campaign enqueues 40,000 accounts at once. The provider starts refusing calls; every worker retries on the same fixed interval, keeping it saturated. Some jobs time out after generation but before the send recorded success. The queue redelivers them. The send step has no idempotency key, so a few hundred customers receive two renewal emails, each personalized, each different, one citing a discount the other does not. Nothing in the model was wrong. The orchestration decided, in unison, to try again.

## Patterns

**Precompute-then-serve.** *Problem:* deep personalization does not fit the real-time budget. *Forces:* research is slow; the moment is short; stored results go stale. *Solution:* compute understanding asynchronously per entity, store it with provenance and a freshness window, serve by lookup plus live context. *Tradeoffs:* precompute wasted on people who never arrive; invalidation logic to own.

**Latency Budget.** *Problem:* teams argue architecture without agreeing what "fast" means. *Solution:* a written budget per channel, at a percentile, listing what may run inside it. *Tradeoffs:* needs re-measuring as channels change.

**Graceful Degradation.** *Problem:* the smart path fails in many ways, some plausible. *Solution:* a declared ladder per zone, every output labeled. *Tradeoffs:* more variants to approve and maintain.

## Leader questions

1. Which personalized experiences have a person waiting, and what is each written budget?
2. How much of our real-time path is computation that could have happened yesterday?
3. When the model provider throttles us, what does the customer see?
4. Can any step send the same person two different messages?
5. Under a burst, do we refuse early and say so, or fall behind silently?

## Build checklist

- [ ] Latency budget per channel, measured at p75 or p95.
- [ ] Precomputed artifacts keyed to the entity, with freshness windows and invalidation events.
- [ ] Real-time path limited to lookup, scoring, and selection among approved variants.
- [ ] Concurrency caps; bounded retries with backoff and jitter at one layer; faults retried, decisions not.
- [ ] Idempotency keys on intake, generation, send, and callback.
- [ ] Intake that returns 202 after validate, persist, claim, dispatch; a durable journal so recovery targets the missing step.
- [ ] Admission limits that refuse (429 with Retry-After) before any write; background work yields to live traffic; limits sized per replica and tuned in dry run first.
- [ ] Degradation ladder per zone; every output labeled full, degraded, or failed.
- [ ] Batch interfaces for all work nobody is waiting for.

## Metrics to watch

- **p75 and p95 latency of the personalized path**, against the channel budget.
- **Degraded-output rate** by zone and by cause.
- **Precompute hit rate and staleness**: share of real-time requests served from a fresh artifact.
- **Duplicate-action rate**: sends or writes that share an idempotency key.
- **Stuck and backlog counts** (claims, pending pages, recovery), with any new error signature treated as a warning [16].

## Reader Q&A

**Can we just use a faster model on the live path?** For selection and short generation, sometimes. No model does an hour of research in 200 ms; move the research.

**Is a streamed chat reply real-time personalization?** It is real-time delivery. The personal part should come from memory curated before the conversation started.

**How fresh is fresh enough?** Per fact. A job title can be a week old; a cancellation click cannot be five minutes old. Freshness is set per property (Chapter 11).

## For your AI

```yaml
chapter: 17
concepts:
  - name: Latency Budget
    definition: "The time a channel allows before delay itself changes the outcome; set per channel and measured at a percentile."
  - name: Precompute-then-serve
    definition: "Compute understanding asynchronously at the entity level, store it with provenance and freshness, serve by lookup plus live context."
  - name: Graceful Degradation
    definition: "A declared fallback ladder per zone (agentic, predictive, rule-based, approved static, fail closed) that reduces specificity, never truthfulness. Used only for dependency and system failures; thin evidence is handled by Honestly General and the Specificity Ladder (Chapter 15)."
  - name: Retry faults, not decisions
    definition: "Retry transient infrastructure failures; never retry deterministic rejections."
  - name: Admission control
    definition: "Refuse work you cannot run (429 with Retry-After, before any write) instead of falling behind; background work yields to live traffic."
decision_rules:
  - if: "an inbound request triggers research or generation"
    then: "validate, persist workflow state, claim an idempotency key, dispatch async work, and return 202"
  - if: "the output is complete but delivery to the destination failed"
    then: "retry the exact completed payload; do not regenerate"
  - if: "a person is waiting and the work takes longer than the channel budget"
    then: "move the work to an event-triggered or scheduled job and serve the stored result"
  - if: "part of the decision depends on signals from the last few seconds"
    then: "run only that part live, as selection among precomputed options"
  - if: "a step mutates state or sends to a person"
    then: "require an idempotency key before it runs"
  - if: "a call fails on rate limit, timeout, or provider 5xx"
    then: "retry with bounded exponential backoff and jitter at one layer"
  - if: "a call fails on missing context, invalid contract, or policy rejection"
    then: "do not retry; take the next rung of the degradation ladder"
  - if: "no one is waiting for the result"
    then: "use a batch interface"
  - if: "work already accepted is requeued after a provider failure and the backlog is over its ceiling"
    then: "re-admit it anyway and log a warning; the ceiling governs new intake, not work already promised"
assessment_questions:
  - "Which personalized experiences have a person waiting, and what is each latency budget?"
  - "What is computed on request that could be precomputed on an event or schedule?"
  - "Is research keyed to the request or to the entity?"
  - "What does the customer see when the model provider throttles you?"
  - "Which steps can send or write twice?"
patterns: [Precompute-then-serve, Latency Budget, Graceful Degradation]
anti_patterns: [The Double Send, Synchronized Retry, Silent Fallback, Everything On-Request]
maturity_dimension: governance_and_reliability
```

## References

1. Brutlag, J. (2009-06-22). "Speed Matters for Google Web Search." Google. https://services.google.com/fh/files/blogs/google_delayexp.pdf
2. Kohavi, R. et al. (2014-08-27). "Seven Rules of Thumb for Web Site Experimenters" (KDD 2014 slides). https://exp-platform.com/Documents/2014-08-27ExperimentersRulesOfthumbKDD.pdf ; see also Kohavi et al. (2013), "Online Controlled Experiments at Large Scale," KDD 2013. https://dl.acm.org/doi/10.1145/2487575.2488217
3. Bernardi, L., Mavridis, T., Estevez, P. (2019). "150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com." KDD 2019. https://dl.acm.org/doi/10.1145/3292500.3330744
4. Nielsen, J. (1993, updated 2014). "Response Times: The 3 Important Limits." Nielsen Norman Group, drawing on Miller (1968). https://www.nngroup.com/articles/response-times-3-important-limits/
5. Google, "Web Vitals" (last updated 2024-10-31; accessed 2026-09-26). https://web.dev/articles/vitals
6. Amatriain, X., Basilico, J. (2013-03-27). "System Architectures for Personalization and Recommendation." Netflix Tech Blog. https://netflixtechblog.com/system-architectures-for-personalization-and-recommendation-e081aa94b5d8
7. Lin, K. et al. (2025). "Sleep-time Compute: Beyond Inference Scaling at Test-time." arXiv:2504.13171. https://arxiv.org/abs/2504.13171
8. First-party measurement on a production enrichment pipeline operated by the author, 2026-08 (entity-level research cache).
9. Datadog (2026-07-30). "State of AI Engineering." https://www.datadoghq.com/state-of-ai-engineering/
10. AWS Well-Architected Framework, "Control and limit retry calls." https://docs.aws.amazon.com/wellarchitected/2025-02-25/framework/rel_mitigate_interaction_failure_limit_retries.html
11. AWS Well-Architected Framework, "Make mutating operations idempotent." https://docs.aws.amazon.com/wellarchitected/latest/framework/rel_prevent_interaction_failure_idempotent.html
12. Marz, N. (2011). "How to beat the CAP theorem." https://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html
13. Kreps, J. (2014-07-02). "Questioning the Lambda Architecture." O'Reilly Radar. https://www.oreilly.com/radar/questioning-the-lambda-architecture/
14. Anthropic, "Batch processing" (accessed 2026-09-26). https://platform.claude.com/docs/en/build-with-claude/batch-processing
15. OpenAI, "Batch API" guide (accessed 2026-09-26). https://developers.openai.com/api/docs/guides/batch
16. First-party: capabilities reference for a governed personalization engine built by the author (reliability, delivery, and recovery sections), 2026-08-25. Unpublished.
17. First-party: operator documentation for a self-hosted governed memory system built by the author (capacity and limits, features, changelog), release 0.8.2, verified 2026-09-16.

---

# 18. Governance

*The question: How do we govern what the AI says and does, at the level of the organization and at the level of each customer?*

> **Questions this chapter answers**
> - Who decides what our AI may say and do, and where does that decision actually live?
> - How do company-wide rules and one customer's wishes both reach the agent at the moment it acts?
> - If a customer complains about something our AI did, can we reconstruct what it knew, what it decided, and who allowed it?
> - Which actions need a human, and how do we keep human review from becoming a rubber stamp?
> - Do we need NIST, ISO 42001, or the EU AI Act to get started?

## The short answer

Governance for AI personalization is not a policy document, a review board, or a dashboard. None of those can stop anything, because none of them sits in the path the agent takes when it acts. **Governance is a code path**: rules are executed inside the system, before the model reasons, around the tools it can call, and after it writes, so a rule holds whether or not the model remembers it.

It works at two levels. At the **organization level**, it carries the company's rules (brand, claims, pricing authority, compliance) with explicit precedence, so a campaign instruction can never override a legal constraint, and with versions, so every rule change is a release with an owner and a rollback. At the **customer level**, it carries each person's consent, preferences, contact rules, the promises made to them, and sensitive context, and treats those as hard constraints on every agent that touches that person.

Three disciplines make it real. **Traceability**: every consequential action leaves a decision trace of what was known, which rule versions applied, what was decided, and who allowed it. **Risk tiers**: put real gates only in front of the short list of actions that cannot be taken back (sending to a customer, moving money, making a commitment, changing a system of record, letting data leave). **Usable human review**: people approve the few decisions that deserve it, with the evidence in front of them, in a flow where saying "no" is fast and saying "yes" without looking is hard.

If you fund one thing, fund the irreversible-action list and the gates around it. It fits on an index card, and it is where a mistake stops costing time and starts costing the thing itself.

---

In November 2022, Jake Moffatt's grandmother died. He went to Air Canada's website to book a flight and asked the airline's chatbot about bereavement fares. The chatbot told him he could buy a regular ticket and apply for the bereavement discount afterward. That was not the policy. A page elsewhere on the same website said bereavement fares could not be claimed retroactively, and when Moffatt applied, Air Canada refused.

He took the airline to British Columbia's Civil Resolution Tribunal, and in February 2024 he won [1]. The damages were small: roughly the fare difference, plus interest and fees. The airline's argument is what makes the case worth remembering. In the tribunal's words, "Air Canada suggests the chatbot is a separate legal entity that is responsible for its own actions. This is a remarkable submission." It "makes no difference whether the information comes from a static page or a chatbot," and the airline had not explained why customers "should have to double-check information found in one part of its website on another part" [1][2].

Notice what failed. Air Canada had the correct policy. It was written down, published, and owned by someone. It simply never reached the component talking to the customer at the moment it mattered, and nothing checked the answer before it left. The company had a policy. It did not have governance.

## Governance is a code path, not a document

Here is the sentence this chapter turns on: **if you cannot name the component that enforces a rule, and say what happens when that component says no, you do not have a rule. You have a preference.**

Most AI governance programs produce a policy, a committee, and a dashboard, and all three stand outside the execution path. By the time any of them has an opinion, the email is in the inbox.

The instinctive fix is to paste the rules into the system prompt. That holds for about a week. Prompts are unversioned, invisible to the teams who own the rules, and multiplied: ten agents across five teams become ten private interpretations of the company. When legal updates a policy, the change reaches the agents legal knows about and misses the rest. And a rule inside a prompt is a request, not a boundary. The model will usually honor it. "Usually" is a fine standard for tone and a poor one for a refund limit.

So split every rule into two kinds. **Rules code can check** (maximum discount, required unsubscribe link, no internal notes in customer output, no contact during a hold, no numeric claim without evidence) become deterministic checks. **Rules only judgment can apply** (tone, what counts as an overclaim, whether a message fits the customer's situation) go into the model's context as governed guidance and are then checked by semantic review and sampling. If a rule can be deterministic, it should not depend on the model remembering to obey it. On adaptive pages and screens the most important such rule is structural: exits, prices, consent, and cancel paths sit in a fixed frame no personalized slot can change (Chapter 3), and every generated value passes a render gate before anyone sees it (Chapter 16).

That split gives the first pattern, the **Policy Sandwich**: deterministic before, probabilistic in the middle, deterministic after. Before the model runs, code decides eligibility (may we contact this person, through this channel, about this?) and assembles the governed context. The model reasons and writes. Afterward, code validates the output and gates the action. The probabilistic layer is where the value is. The deterministic layers are where the guarantees are.

Add tools and you get **three-layer guardrails**: *before* (eligibility, consent, the right rules routed into context), *during* (the capability boundary: an agent can call only the tools, with only the parameters, its role allows), and *after* (output validation and an action gate in proportion to consequence). The middle layer is the one teams skip, and the one that holds when everything else fails. An agent that drafts retention offers should hold no credential that can issue a refund. Its worst case is the sum of its permissions, not the quality of its instructions.

In a self-hosted memory system I built, each agent's key carries a role, and the agent sees only the tools that role may call: a reader agent's write tools are not refused, they are absent. Autonomous loops are never offered hard delete. A job handed to an outside provider gets a credential minted for that job alone, limited to three memory tools and revoked when the job ends.

![The Policy Sandwich with three-layer guardrails. Before the model runs, deterministic code decides eligibility and consent and routes the right governed rules into context. In the middle, the probabilistic model reasons and writes, but during that step a capability boundary lets it call only the tools and parameters its role allows. After, deterministic code validates the output and gates the action in proportion to consequence; the code layers hold the guarantees and the model layer holds the value, so a rule holds whether or not the model remembers it.](/images/handbook/ch16-policy-sandwich.svg)

*Figure 18.1. Deterministic before and after, probabilistic in the middle, with a capability boundary around the tools.*

## Organization level: precedence, owners, versions

A legal constraint, a brand preference, a campaign strategy, a channel limit, and an instruction for one paragraph are different kinds of statement. Mixed into one prompt, they compete on equal terms, and the model resolves the conflict however it resolves it that day. I treat them as a **governance hierarchy**, where a lower level can never override a higher one:

1. Hard constraints (legal, safety, privacy; enforced in code where possible)
2. Enterprise policy (claims, pricing authority, brand)
3. Program or campaign contract
4. Channel policy (length, format, required elements)
5. Task instruction
6. Customer context
7. Model preference (what the model would do if left alone)

![The governance hierarchy as a seven-level ladder, where a lower level can never override a higher one. From top to bottom: hard constraints (legal, safety, privacy, enforced in code where possible), enterprise policy (claims, pricing authority, brand), program or campaign contract, channel policy (length, format, required elements), task instruction, customer context, and model preference (what the model would do if left alone). For example, an enterprise rule that keeps internal notes out of customer output beats a campaign instruction to use every available insight.](/images/handbook/ch16-governance-hierarchy.svg)

*Figure 18.2. Rules of different authority do not compete on equal terms; precedence decides, and hard constraints always win.*

Suppose the enterprise rule is that internal CRM notes never reach a customer, and a campaign manager writes "use every available insight to make this specific." The campaign must lose, and it should lose in code: the one serializer every customer-facing payload passes through strips internal fields whatever any instruction said. That is the general move at scale: find the last shared function before an important boundary and make it protective (the serializer, the sender that enforces suppression, the parser that rejects unsafe links), so every new program inherits the protection instead of relying on each author to remember it.

The most useful enterprise rules are allow-lists, not prohibitions. In a governed personalization engine I built, each campaign carries a **capability map** of what the AI may recommend, so the model is asked not "what would sound good here?" but "which approved capability fits this buyer?" Selected engines limit numeric proof to customer-approved statistics. A **customer-status rule** forbids "your renewal is coming up" unless that relationship is known; otherwise the copy is reframed as an evaluation. Air Canada's chatbot had no such list: it stated a policy nobody had approved.

Three practices keep the hierarchy honest.

**Every rule has an owner.** Marketing owns voice, sales operations owns discount authority, legal owns claims and consent language, support owns service promises. The governance layer does not make policy. It makes sure every agent, on every platform, receives the current approved version of each owner's policy by reference, and nothing else. In the layer I built, hard constraints are marked always-on and critical, so they are never trimmed when the context budget is tight, and authoring quality turned out to decide discoverability: in our internal tests, well-written rules were 20 to 50 percentage points more likely to reach the right task than poorly written versions of the same rules [3], and the architecture paper reports 92 percent governance routing precision in controlled experiments [16]. A rule titled "Q3 update" is governance in name only. Not every rule is for every agent, either: in the self-hosted memory system I built, a guideline stays org-wide by default but can be scoped to its author, a group, one person, or whoever can already see a given record.

**Every rule has a status and a version.** Draft, approved, deprecated; agents see only approved. A rule change is a code change in everything but syntax, so its record carries the previous and new version, the reason, the evidence, the approver, the validation results, and the rollback target. Then you can say "unsupported claims fell after v8" instead of "we tuned the prompt last week." Reference each program's rules by a stable identifier, never a loose name: "Q3 enterprise" and "Q3 enterprise EMEA" are different contracts. And make the rollback real. My memory system once let a document update silently replace the prior body; now the old version is snapshotted in the same transaction as the change, and a restore can itself be undone.

**Model changes are releases too.** In January 2024 the parcel company DPD disabled the AI element of its customer chatbot after it swore at a customer and wrote a poem criticizing the company. DPD's explanation: "An error occurred after a system update" [4]. A new model, prompt, or guideline that reaches customers deserves the same regression checks as a code deploy.

## Customer level: every person carries constraints

Organization-level rules say what the company will do. Customer-level governance says what the company will do *to this person*. It is the half most programs forget, because it looks like data rather than policy. Each record should carry, as typed fields with provenance rather than free-text notes:

- **Consent and lawful basis** per purpose and channel, with source and date.
- **Contact rules**: opt-outs, frequency caps, quiet periods, preferred channel, relationship owner.
- **Stated preferences**: "email only," "not before our fiscal close," "talk to my deputy."
- **Promises made**: a price hold, a service credit, "no outreach until the review is done." A promise is a constraint on every future action, not a line in a ticket.
- **Sensitive context**: an open escalation, a dispute, a bereavement, a legal hold. These often change whether to act at all, not only the tone.
- **Suppression**: an explicit do-not-contact or do-not-personalize state that overrides every program.

The precedence rule is simple: **a customer-level constraint outranks a campaign**, and it is checked in the "before" layer by code, so the agent is never invited to write the message. This is where governance meets memory (Chapters 10 and 11). A promise that lives as prose in a support ticket will not be found by the agent that needs it, or will be weighed as one fact among many. The Boundary Rule applies with force: if you would put it in a WHERE clause, it is a property. "Do not contact before March 1" is a WHERE clause.

The same layer should hold what the system declines to use: facts that are accurate and lawful but wrong to act on (Chapter 19). Appropriateness rules need a governed home, or they exist only in someone's judgment.

## Traceability: can you answer in thirty seconds?

When a customer, a regulator, or your CEO asks "why did the AI say that?", there are two possible answers: a week of log archaeology, or a lookup that takes thirty seconds because every consequential action wrote a **decision trace** when it happened. A useful trace records:

1. **What was known**: the customer facts used, each with source, time, and confidence, and the customer-level constraints checked.
2. **What governed it**: rule versions, program contract, channel policy, model and prompt versions.
3. **What was decided and why**: the candidate actions considered (including doing nothing), the one chosen, the stated rationale, every validator result.
4. **Who allowed it and what happened**: automatic or approved, by whom and when; what was actually sent or changed; what the customer did next.

Three cautions. A model's explanation of its own choice can be plausible and wrong, so the trace must hold the inputs and checks that make the rationale verifiable. The trace is itself customer data, with the same access, retention, and deletion obligations as the memory it points to (Chapter 19). And the trace is evidence only if the model cannot write it: in the memory system I built, the system stamps who made each automated change, every agent tool call that changes something writes an audit row with the caller and the shape of its arguments (never the content), and audit rows can be cryptographically chained so an edited history is detectable.

The trace turns incidents into lookups and lets you correlate quality with versions (Chapter 20). Outside parties increasingly expect it. As of September 2026, the EU AI Act's Article 50 transparency duties apply from August 2, 2026, including telling people they are interacting with an AI system unless that is obvious [5]. A disclosure is a string your agent emits; a record of what it did and who authorized it is a log line with an actor, an action, and a timestamp. Both are cheap to design in and expensive to retrofit, exactly like authentication. (Timing for other obligations moved with the 2026 Digital Omnibus [6]; keep dates in the companion note. Not legal advice.)

## Risk: sort actions by whether they can be undone

Heavy review everywhere is unaffordable, so uniform oversight quietly becomes light review everywhere, including on the actions that cannot be undone. Governance with stakes makes the opposite trade.

| Tier | What it covers | Default control |
|---|---|---|
| 0. Internal, reversible | drafts, notes, summaries, proposed record changes | automatic; logged |
| 1. Customer-visible, low stakes | website modules, in-product recommendations, anything replaced on next render | automatic with validation and sampled review |
| 2. Customer communication | messages that make claims or cite facts about the person | validation plus semantic review; humans review pilots and flagged cases |
| 3. Irreversible or binding | the list below | hard gate: limits in code, human approval above thresholds |

The **irreversible-action list** for an AI acting on customers has five groups:

- **Send**: a delivered message cannot be unread.
- **Money**: refunds, credits, discounts, price changes, write-offs.
- **Commitments**: statements of policy, terms, eligibility, or promises. Air Canada's chatbot made one its owner then had to honor [1]. In April 2025 the support bot of the coding tool Cursor told users a "one device per subscription" policy existed; it did not, users cancelled, and a co-founder apologized for an "incorrect response from a front-line AI support bot" [7][8]. In December 2023 a car dealership's chatbot agreed to sell a new Tahoe for one dollar and called it "a legally binding offer" [9]. Different causes, one lesson: an agent that can state terms needs a boundary on which terms it may state.
- **Records**: deleting, merging, or overwriting a system-of-record field; changing consent or opt-out state.
- **Data leaving**: exports, partner sharing, one customer's data surfacing in another's context.

That list fits on an index card. Everything else is recoverable by definition.

![Four risk tiers with their default controls. Tier 0, internal and reversible work such as drafts, notes, and summaries, runs automatically and is logged; tier 1, customer-visible low-stakes output replaced on next render, runs automatically with validation and sampled review; tier 2, customer communication that makes claims or cites facts about the person, gets validation plus semantic review, with humans on pilots and flagged cases. Tier 3, irreversible or binding actions, sits behind a hard gate with limits in code and human approval above thresholds; it is the irreversible-action list of send, money, commitments, records, and data leaving.](/images/handbook/ch16-irreversible-action-tiers.svg)

*Figure 18.3. Sort actions by whether they can be undone, and spend real gates only on the five that cannot.*

## The control hierarchy: approval is the fourth control, not the first

For each item on the list, apply these in order:

1. **Remove the authority.** The cheapest control is the capability that does not exist.
2. **Make it reversible.** Irreversibility is often a default setting, not a law of nature. Delay the send by fifteen minutes; stage the record change as a proposal; soft-delete with a restore window; cap and expire the offer.
3. **Bound it in code.** Thresholds, allow-lists of claims and offers, per-customer rate limits, the customer-level constraints above.
4. **Require approval where consequence justifies it**: above a discount threshold, for a new claim class, for a first send in a new program.
5. **Monitor and test.** Log every gate decision, override, and bypass; replay past cases against new rules before release.
6. **Support the reviewers.** Rehearse the approval path and reward the person who says "not yet."

The order prevents a common mistake: asking a person to compensate for authority the organization could have removed upstream. A human approving two hundred refunds an hour is not a control. That human is a latency.

The fix for volume is to **approve the judge, not each verdict**. The memory system merges duplicate customer records, a records action. Code finds candidate pairs, the model judges each, and every merge is journaled and undoable for ninety days by default. There is deliberately no per-merge approval: the routine starts by only proposing, and an operator promotes it one stage at a time on reviewed runs. Demotion is always allowed and takes effect on the next write. The stage is a property of the call path, not an instruction in the prompt, so a model cannot talk its way past it (Chapter 11). Close the side doors too. In the same system, every safety control on a background curation routine (stage, write ceiling, change ledger) hangs off its recipe, and for a while a generic scheduling path could start one with no recipe, and so with none of those controls. That path now refuses it. A control that one path can skip is a control on the main path only.

The same logic should govern changes to the system itself. In the personalization engine, that loop is operator-run today: a person uses AI to reason across the first real cohort, classifies each finding by root cause, and decides the fix. The design I am productizing is **two approvals**. A routine proposes a fix for a recurring problem (illustratively: "11 of 83 pages use unsupported relationship language; add one guideline change, one deterministic guard, six regression examples"). Approval one: the diagnosis is reasonable, implement and prove it outside production against replayed, held-out, and regression sets. Approval two: it passed, promote it. What should hold is the credential underneath: the analysis worker gets read-only production access and a branch, and no deployment or data-mutation rights, so nothing it holds can ship anything. That is **human-supervised autonomy**: autonomy to observe, diagnose, propose, implement outside production, and validate; production authority held by people.

## Engaging humans: the hard part is usability, not design

A human can sit **in the loop** (approves before the action; right for tier 3, and tier 2 during pilots), **on the loop** (watches behavior in near real time and can pause, override, or roll back; right for mature tier 1 and 2), or **over the loop** (sets rules, reviews samples and traces, changes the system; right for everything).

The EU AI Act's human-oversight article is a good design template even where it does not legally bind. (As of September 2026, Article 14 applies to systems the Act classes as high-risk; most marketing personalization falls outside that class, though uses such as credit decisions or life and health insurance pricing may fall inside [10].) It asks that overseers can understand the system's limits, "remain aware of the possible tendency of automatically relying or over-relying on the output" (automation bias), interpret its output, "override or reverse the output," and stop it "in a safe state" [10]. Read those as five requirements for a review screen.

Oversight rarely fails because the principle is wrong. It fails because it is too slow, adds friction to routine work, and gets waived for important accounts at quarter end. A review step the organization sets aside whenever it is in a hurry is not a control. It is a policy with a known bypass built in. So design review like any product for busy users:

- **Show the evidence, not just the output**: the facts the message relies on, with source and date, and which validators flagged.
- **Make "no" cheap**: one click with a reason code that feeds the learning loop (Chapter 20).
- **Put friction only where the consequence is.** If people route around review because it is slow for routine work, the tiers are wrong.
- **Watch for rubber-stamping.** Approval rates at or near 100 percent, or review times too short to read the evidence, mean the humans have become part of the automation. Seed known-bad cases and check they are caught.
- **Never let urgency waive the gate.** The path for "this must go now" is a faster approver, not no approver.

A control that survives contact with a real deadline is the only kind that counts.

## The frameworks are maps, not mechanisms

Your customers' procurement teams will ask about these, so know them.

- **NIST AI RMF 1.0** (January 2023) is voluntary and organizes the work into four functions, Govern, Map, Measure, Manage, with Govern cutting across the rest [11]. Its **Generative AI Profile, NIST AI 600-1** (July 2024), lists twelve generative-AI risks, including confabulation (NIST's term for fabricated output), data privacy, information integrity, and human-AI configuration [12].
- **ISO/IEC 42001:2023** is the first certifiable management-system standard for AI: policies, roles, risk assessment, continual improvement [13].
- **The EU AI Act** draws legal lines. Article 5 prohibits manipulative or deceptive techniques that cause significant harm; the Commission's guidelines say preference-based personalization is not inherently manipulative unless it uses such techniques [14]. Article 50 sets disclosure duties [5].

My qualification of all three: they tell you *what* to manage and are silent on *where the control sits*. An organization can map every NIST subcategory to a policy document and still run an agent that can issue any refund it likes. Aligning or certifying does not, on its own, settle any privacy or AI-law obligation (Chapter 19). Use the frameworks to structure the program; use this chapter's patterns to decide what runs in the path.

## What this does not do

Governance lowers the rate and cost of mistakes. It does not eliminate them.

**Deterministic checks cover only what they detect.** A grounding check catches unsupported revenue figures because someone wrote a detector for them. It will not catch the claim class nobody anticipated.

**Tests are snapshots.** When I tested our governance layer against fifty designed scenarios meant to push agents past ten kinds of policy (discount limits, confidential figures, unbacked SLA claims), compliance held in all fifty [15][16]. That is a signal about the architecture, not a guarantee about the world. A policy that does not cover a situation offers no protection, and a model update can shift behavior at the margins. Treat any 100 percent as the start of a larger test set.

**"A human will approve it" is not automatically a safeguard.** Without evidence on the screen and time to read it, it produces automation bias with a signature attached.

**Governance can strangle value.** A program that routes every tier 1 recommendation through approval will be abandoned within a quarter, and the abandonment will take the tier 3 gates with it. Proportion is the whole discipline.
## At scale

Larkspur Systems (a fictional composite company used throughout this handbook) runs agents across its six teams (marketing, sales, customer success, support, product, and events), touching about 40,000 accounts and 250,000 contacts. At that scale, two things change.

**Customer constraints must be global.** A quiet period set by support must stop marketing's agent, the sales sequence, and the renewal bot, which works only if the constraint lives in shared customer memory, not one team's tool. The most common failure at scale is not a rogue model. It is several well-behaved agents that never heard each other's promises.

**Oversight shifts from items to populations.** Nobody reviews 250,000 messages. You review the pilot cohort, then distributions: flagged-claim rates, suppression hits, overrides, complaints per thousand sends, by program and rule version. The trace makes this possible; the tiers make it affordable.

## Failure story: The Broken Promise

After a painful outage, Larkspur's account manager called a fleet customer's operations director and promised two things: a service credit, and no sales or marketing outreach until the post-incident review closed. She logged both in the support ticket, as a note.

Two weeks later the expansion agent ran its weekly scan. Usage on the account had spiked (the customer was re-running jobs the outage interrupted), and a model scored it as a strong upsell candidate. The agent checked consent: valid. Frequency caps: clear. It wrote an accurate, grounded email about capacity add-ons, passed every validator, and sent it. The operations director forwarded it to the account manager with one line: "So much for your word."

Every fact was true and every component behaved. The agent chose a locally reasonable action that broke a customer promise, because the promise lived in prose in another team's system and never became a constraint. The fix was not a better model. It was a typed customer-level field (outreach hold, reason, owner, expiry), written by the support workflow, read by every agent's "before" layer, outranking every campaign.

## Patterns

**Policy Sandwich.** *Problem:* rules in a prompt are followed usually, not always. *Forces:* the model's judgment is the value; some rules cannot tolerate "usually." *Solution:* deterministic eligibility and context before the model; deterministic validation and gating after. *Tradeoffs:* rules moved into code must be written and maintained; judgment rules stay probabilistic and need sampling.

**Irreversible-Action Gate.** *Problem:* uniform oversight becomes light oversight everywhere. *Forces:* review is expensive; permanent mistakes cost more. *Solution:* enumerate send, money, commitments, records, and data leaving; apply the control hierarchy to each. *Tradeoffs:* latency on gated actions; tiers must stay honest or gates creep into everything.

**Governance Hierarchy.** *Problem:* rules of different authority collide in one context. *Forces:* many owners, many programs, frequent change. *Solution:* explicit precedence, customer constraints above campaigns, owned and versioned rules served by reference, hard constraints enforced at shared choke points. *Tradeoffs:* up-front ownership work; conflicts surface and must be resolved by people instead of silently by the model.

## Leader questions

1. Name the component that enforces our discount limit. What happens when it says no?
2. If a customer asked why our AI said something last Tuesday, how long until we can show what it knew, which rule version applied, and who allowed it?
3. Where does a promise one team makes to a customer live, and which agents can read it?
4. Which irreversible actions can any agent take today without a gate?
5. What is the approval rate in our review queues, and how do we know reviewers are reading?

## Build checklist

- [ ] Write the irreversible-action list for every agent.
- [ ] Remove tools and credentials agents do not need; separate read, propose, and execute rights; mint per-job credentials that expire.
- [ ] Add reversibility: send delays, staged record changes, soft deletes, expiring offers.
- [ ] Move every code-checkable rule out of the prompt into validators or gates.
- [ ] Define the governance hierarchy; enforce hard constraints at shared choke points.
- [ ] Give every rule an owner, status, and version, with change records and rollback targets.
- [ ] Model consent, holds, promises, preferences, and sensitive context as typed customer fields checked before any agent acts.
- [ ] Emit a decision trace for every tier 2 and tier 3 action.
- [ ] Build review screens with evidence, one-click rejection with reasons, and seeded test cases.
- [ ] Treat model, prompt, and guideline changes as releases with replay, held-out, and regression checks.

## Metrics to watch

- **Gate outcomes by tier**: blocked, approved, overridden, bypassed per thousand actions; every bypass explained.
- **Trace completeness**: share of tier 2 and 3 actions with a full decision trace.
- **Customer-constraint violations**: actions that contradicted a hold, opt-out, or promise.
- **Review health**: approval rate, median review time, catch rate on seeded bad cases.
- **Rule freshness**: share of agent calls served the current approved version of each rule.

## Reader Q&A

**Does every customer-facing message need human approval?** No. It needs a tier. Pilots, new programs, new claim classes, and tier 3 actions get humans; mature, validated work runs with sampling and population monitoring.

**Is our AI vendor responsible if its agent says something wrong?** Do not count on it. Air Canada argued its chatbot was responsible for its own actions, and the tribunal called that "a remarkable submission" [1]. Customers and tribunals tend to hold the deploying company to what its agent said. Not legal advice.

**Who should own AI governance?** Domain owners own their rules; a platform owner owns the layer that serves and enforces them; a named executive owns the irreversible-action list.

## For your AI

```yaml
chapter: 18
concepts:
  - name: Governance is a code path
    definition: "Rules are executed inside the system (before the model, around its tools, after its output), so a rule holds whether or not the model remembers it."
  - name: Policy Sandwich
    definition: "Deterministic eligibility and context assembly before the model, probabilistic reasoning and writing in the middle, deterministic validation and action gating after."
  - name: Three-layer guardrails
    definition: "Before (eligibility, consent, routed rules), during (capability boundary on tools and credentials), after (output validation and action gate)."
  - name: Governance hierarchy
    definition: "Explicit precedence: hard constraints > enterprise policy > program contract > channel policy > task instruction > customer context > model preference; customer-level constraints outrank campaigns."
  - name: Customer-level governance
    definition: "Per-person consent, contact rules, preferences, promises, sensitive context, and suppression stored as typed fields and enforced as constraints on every agent."
  - name: Decision trace
    definition: "Record per consequential action of what was known (with provenance), which rule and model versions governed it, what was decided and why, who allowed it, and what happened."
  - name: Irreversible-action list
    definition: "The short list of actions that cannot be undone: send, money, commitments, records, data leaving."
  - name: Human-supervised autonomy
    definition: "Design pattern: agents may analyze, diagnose, propose, and validate outside production; production changes require two human approvals, and the agent holds no production write credentials."
  - name: Allow-list governance
    definition: "Enterprise rules written as what the agent may say (capability map, approved statistics, known customer relationships) rather than only what it may not."
  - name: Approve the judge, not each verdict
    definition: "A human promotes an automated routine's stage (propose-only, limited, full) on evidence instead of approving each item; demotion is always available."
decision_rules:
  - if: "a rule can be checked by code"
    then: "enforce it in a validator or gate, not only in the prompt"
  - if: "an action is on the irreversible-action list"
    then: "apply the control hierarchy in order: remove authority, make reversible, bound in code, require approval above threshold, monitor"
  - if: "a customer-level constraint (hold, opt-out, promise) conflicts with a campaign"
    then: "the customer-level constraint wins, evaluated before the agent is invoked"
  - if: "a rule, prompt, or model version changes"
    then: "treat it as a release: record owner, reason, approver, validation results, and rollback target"
  - if: "human approval rate is at or near 100 percent or review time is too short to read evidence"
    then: "suspect rubber-stamping; seed known-bad cases and redesign the review screen or the tiering"
  - if: "an agent may name capabilities, statistics, or a customer's relationship with you"
    then: "restrict it to approved allow-lists; if the relationship is not known in the data, reframe as an evaluation"
  - if: "a routine takes many similar consequential actions (merges, record changes)"
    then: "make each action journaled and undoable, start the routine propose-only, and promote it one stage at a time on reviewed evidence"
  - if: "a lower-level instruction would override a hard constraint"
    then: "the hard constraint wins, enforced at a shared choke point"
assessment_questions:
  - "Which component enforces each of your hard rules (discount limit, internal-data exposure, suppression), and what happens when it refuses?"
  - "Where do promises and preferences stated by customers live, and can every agent read them before acting?"
  - "Which irreversible actions can any agent take today without a gate?"
  - "Can you reconstruct, for a given customer message, the facts used, rule versions, model version, and approver?"
  - "Who owns each governance domain, and how does a rule change reach every agent?"
  - "What are the approval rate and median review time in your human review queues?"
patterns: [Policy Sandwich, Irreversible-Action Gate, Governance Hierarchy, Three-layer guardrails, Two Approvals, Shared Choke Points, Allow-list governance, Approve the judge not each verdict]
anti_patterns: [The Broken Promise, Governance by Document, Prompt as Policy, Uniform Oversight, Rubber-Stamp Review]
maturity_dimension: governance_and_reliability
```

## References

1. *Moffatt v. Air Canada*, 2024 BCCRT 149, British Columbia Civil Resolution Tribunal, February 2024. https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html
2. McCarthy Tétrault, "Moffatt v. Air Canada: A Misrepresentation by an AI Chatbot" (quotes from the decision). https://www.mccarthy.ca/en/insights/blogs/techlex/moffatt-v-air-canada-misrepresentation-ai-chatbot ; see also American Bar Association, Business Law Today, February 2024. https://www.americanbar.org/groups/business_law/resources/business-law-today/2024-february/bc-tribunal-confirms-companies-remain-liable-information-provided-ai-chatbot/
3. H. Taheri, "14 Agent Configs, 3 Teams, Zero Source of Truth," hamedtaheri.com, 2026-03-14 (internal routing experiments: 25 governance variables, 5 categories, 20 task types). https://hamedtaheri.com/articles/fourteen-agent-configs-zero-source-of-truth
4. ITV News, "DPD disable AI chatbot after it swears at customer and calls company 'worst delivery service'," 2024-01-19. https://www.itv.com/news/2024-01-19/dpd-disables-ai-chatbot-after-customer-service-bot-appears-to-go-rogue ; TIME, https://time.com/6564726/ai-chatbot-dpd-curses-criticizes-company/
5. EU AI Act (Regulation (EU) 2024/1689), Article 50, transparency obligations; Commission FAQ. https://artificialintelligenceact.eu/article/50/ ; https://digital-strategy.ec.europa.eu/en/faqs/transparency-obligations-under-article-50-ai-act (status as of September 2026)
6. Regulation (EU) 2026/1744, "Digital Omnibus on AI." https://eur-lex.europa.eu/eli/reg/2026/1744/oj/eng (status as of September 2026; details in companion note)
7. AI Incident Database, Incident 1039 (Cursor support bot, April 2025). https://incidentdatabase.ai/cite/1039/
8. Fortune, "Cursor's AI glitch triggers viral fallout," April 2025. https://fortune.com/article/customer-support-ai-cursor-went-rogue/
9. AI Incident Database, Incident 622 (Chevrolet of Watsonville chatbot, December 2023). https://incidentdatabase.ai/cite/622/ ; GM Authority, https://gmauthority.com/blog/2023/12/gm-dealer-chat-bot-agrees-to-sell-2024-chevy-tahoe-for-1/
10. EU AI Act, Article 14, human oversight. https://artificialintelligenceact.eu/article/14/
11. NIST, *AI Risk Management Framework 1.0* (NIST AI 100-1), 2023-01-26. https://www.nist.gov/itl/ai-risk-management-framework
12. NIST, *Generative Artificial Intelligence Profile* (NIST AI 600-1), 2024-07-26. https://doi.org/10.6028/NIST.AI.600-1
13. ISO/IEC 42001:2023, Information technology: Artificial intelligence: Management system. https://www.iso.org/standard/42001
14. EU AI Act, Article 5, and Commission Guidelines on prohibited AI practices, 2025-02-04. https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-5
15. H. Taheri, "Adversarial Governance Compliance: Our Methodology and What Near-Perfect Accuracy Tells Us," hamedtaheri.com, 2026-03-14. https://hamedtaheri.com/articles/adversarial-governance-compliance
16. H. Taheri, "Governed Memory: A Production Architecture for Multi-Agent Workflows," arXiv:2603.17787, March 2026. https://arxiv.org/abs/2603.17787

---

# 19. Privacy and Regulation

*The question: What should we never know, keep, or use, and what does the law require?*

> **Questions this chapter answers**
> - The data is ours and the inference is correct. Why does the customer still feel watched?
> - What should our systems never infer or store, even where the law would allow it?
> - What does the law actually require of AI personalization, in principles that will still hold next year?
> - How do we build privacy into the memory layer instead of wrapping a policy around it?
> - When a customer says "forget me" or "stop marketing to me", what has to happen, and where?

*This chapter is not legal advice. It sets out principles that have been stable for years. Jurisdiction-specific rules and dates change often; they live in the dated companion note, [Regulation as of September 2026](../notes/regulation-as-of-2026-09.md). Check it, and check with counsel, before you ship.*

## The short answer

Personalization runs on two tests, and most teams only run one. The first is accuracy: is the fact true? The second is appropriateness: would this person expect us to know it, did it come to us for this purpose, and does using it here serve them? A message can pass the first and fail the second, and when it does, the customer does not experience a clever system. They experience surveillance.

The law encodes a floor under the second test. Across the EU, the UK, California, a growing list of US states and Canada, the same ideas recur: collect for a stated purpose and use for that purpose; keep only what you need; treat inferences about health, sexuality, religion, politics and ethnicity as the sensitive data they reveal; let people stop direct marketing, absolutely; and put a human and an explanation behind automated decisions that seriously affect someone. Regulators also treat claims like "anonymized" and "we don't sell data" as promises they will check.

For a leader, three moves matter more than any single statute. Write a never-store and never-infer list and make the system enforce it. Build privacy into the write path of customer memory, because nothing downstream can fully undo what was stored or embedded. And make deletion and objection reach every representation of a person, including the tools you synced them to. If you cannot prove a deletion, you have not done one.

Compliance is the floor. The customer's expectation is the standard they will actually hold you to.

## One set of facts, two messages

Larkspur Systems (a fictional company, a composite of many real ones) sells field-service and fleet software. Two messages go to the same fleet operations manager in the same week. Both rest on the same two facts: idle time at her Denver depot is up this month, and she has looked at the pricing page for Larkspur's idle-reduction add-on.

The first appears inside the product, on the dashboard she opens every morning, signed by the account manager she talks to monthly: "Idle hours at Denver are up this month. You were looking at the idle-reduction add-on; here is what it would have flagged over the last thirty days, using your own data."

The second arrives at 11:40 p.m., at a personal email address the enrichment vendor supplied, from a sales rep she has never heard of: "Saw you checking our pricing late last night. Looks like Denver is having a rough month."

Same facts, same company. One reads as service; the other reads as being watched. Nothing in the second message is false. What changed is expectation (she knows her telematics feed the dashboard; she did not expect a stranger to narrate her browsing), purpose (she shared depot data to run her fleet, not to be prospected), channel (her personal inbox, near midnight), and who is speaking (a relationship, versus a system that has clearly been watching).

![Two messages rest on the same two true facts: idle time at her Denver depot is up this month, and she viewed pricing for the idle-reduction add-on. The in-product dashboard message matches her expectation (she knows her telematics feed it), serves the purpose the data came for (running her fleet), arrives in the dashboard she opens every morning, and comes from her account manager. The late-night sales email has a stranger narrate her browsing, uses depot data to prospect her, lands in her personal inbox at 11:40 p.m., and comes from a sales rep she has never heard of. The first reads as service; the second reads as being watched.](/images/handbook/ch17-one-fact-two-messages.svg)

*Figure 19.1. Nothing in the second message is false; expectation, purpose, channel and speaker are what turn service into surveillance.*

Aguirre and colleagues tested this with personalized ads and found that when data collection was overt, personalization raised click-through; when it was covert, the same personalization raised feelings of vulnerability and lowered click-through [1]. People do not object to being known. They object to discovering they were watched.

## Accurate is a property of the fact; appropriate is a property of the use

Here is the sentence this chapter turns on: **accuracy belongs to the fact, appropriateness belongs to the use, and your system has to check both.** A true fact can be the wrong thing to say, to this person, in this channel, today.

Stacks that check anything check accuracy: freshness, provenance, confidence (Chapters 11 and 20). Appropriateness is left to the copywriter's taste, which does not scale to a quarter of a million messages. Make it a gate, with four questions a decision layer can evaluate:

1. **Expectation.** Would this person expect us to know this? Declared data and data from the product they use with us pass easily. Browsing, enrichment, and anything inferred need a reason.
2. **Purpose.** Did the fact come to us for a purpose this use serves? Depot telemetry shared to run a fleet serves an operational insight. It does not obviously serve a cold prospecting email.
3. **Benefit.** Does using it here help the person, in this moment, more than it helps us? If the only beneficiary is the conversion rate, treat that as a warning.
4. **Say-how-we-know.** If they asked "how do you know that?", would the honest answer make the message better or worse? If worse, do not use the fact. Use it in a way where the source is obvious, or fall back to something honestly general.

The fourth question is the one teams skip, and the most useful: it turns a vague feeling into a test anyone can run. The dashboard message passes it; the midnight email fails it.

![A personal fact passes through two tests. Test 1, accuracy, is a property of the fact: is it true, judged by freshness, provenance and confidence; most stacks stop here. Test 2, the appropriateness gate, is a property of the use and asks four questions: expectation (would they expect us to know it?), purpose (did it come to us for this use?), benefit (does it help them more than us?), and say-how-we-know (would the honest answer help?). If the fact passes, use it; if it fails, make the source obvious or stay honestly general, because if you have to disguise how you know something, you should not be using it.](/images/handbook/ch17-appropriateness-gate.svg)

*Figure 19.2. Accuracy is checked on the fact; appropriateness is a four-question gate on the use, and a fact must pass both.*

A caution on one historical fix. In Charles Duhigg's 2012 account of Target's pregnancy-prediction score, the company reportedly learned that customers reacted badly to obviously targeted baby coupons, and responded by mixing them among unrelated offers so the targeting looked random [2]. (The widely retold story of a father discovering his daughter's pregnancy is single-sourced; I do not rely on it.) Hiding the inference is not the fix. It fails the say-how-we-know test by design. If you have to disguise how you know something, you should not be using it.

## The law draws a few lines, and they are stable

Statutes change; the principles under them have been steady. I state them in GDPR terms, the most developed text, and note where other regimes converge.

**Purpose, minimization, storage limits.** GDPR Article 5 requires that personal data be collected for specified purposes and not used in ways incompatible with them, that it be limited to what is necessary, and that it be kept no longer than needed [3]. Article 25 requires data protection by design and by default [3]. For a memory system: every fact should know its source and permitted uses, and some should expire.

**A lawful basis, and an absolute right to stop marketing.** GDPR Article 6 lists the lawful bases; Recital 47 says direct marketing "may be regarded" as a legitimate interest, which is permission to weigh, not a blank check [3]. Article 21 is sharper: a person can object to processing for direct marketing, including profiling for it, and then the data "shall no longer be processed for such purposes" [3]. There is no balancing test. Build objection as a hard suppression, not a preference score.

**Inferences about sensitive traits are sensitive data.** Article 9 restricts processing of data revealing health, sex life or sexual orientation, religious or political beliefs, racial or ethnic origin, trade union membership, and genetic and biometric data [3]. In the *OT* case (C-184/20, 2022), the EU Court of Justice held that data "liable indirectly to reveal" such traits falls under that protection [4]. That is the legal anchor for the rule practitioners need most: if your model infers it, you have processed it. US regulators arrive at a similar place through enforcement: the FTC's 2023 GoodRx order banned the company from sharing health data for advertising [5].

**Automated decisions that matter need a human and an explanation.** GDPR Article 22 restricts decisions based solely on automated processing that produce legal or similarly significant effects [3]. The CJEU has read it broadly: in *SCHUFA* (C-634/21, 2023) a credit score computed upstream counted as the decision when the lender relied on it decisively [6], and in *Dun & Bradstreet Austria* (C-203/22, 2025) "meaningful information about the logic involved" meant an intelligible account of how the person's data led to the result, not source code [7]. Most marketing personalization is not an Article 22 decision. Pricing, credit terms, eligibility, and account restrictions can be. California's automated decision-making rules, the UK's rewritten Articles 22A to 22D, and Colorado's replacement statute all regulate the same zone, with different scopes and dates (see the note).

**Personalization is lawful; manipulation is not.** The EU AI Act's Article 5 prohibits AI that uses subliminal, purposefully manipulative or deceptive techniques, or exploits vulnerabilities due to age, disability or social or economic situation, to materially distort behavior and cause significant harm [8]. The Commission's February 2025 guidelines say that advertising which personalizes content based on user preferences is "not inherently manipulative" unless it uses those techniques [8]. That is the legal line between the subject of this handbook and the thing it refuses to teach.

**Claims about privacy are themselves regulated.** The FTC's position is that hashing an identifier does not make data anonymous, because a hash is still a persistent identifier [9]. In the Rite Aid order (2023), the FTC held a company accountable for the error rate of its AI system and required deletion of data and models derived from it [10]. Say only what your architecture guarantees.

## Inference is now where most sensitive data comes from

The privacy conversation still pictures sensitive data as something a customer types into a form. In AI personalization it mostly arrives by inference, and the capability is not new; its price is. In 2013, Kosinski, Stillwell and Graepel used the Facebook Likes of more than 58,000 volunteers to predict private attributes; they distinguished gay from straight men 88% of the time and Democrats from Republicans 85% of the time [11]. Fewer than 5% of the gay users had liked anything explicitly about it. In 2023 and 2024, Staab and colleagues showed that general-purpose language models reading ordinary forum posts could infer personal attributes such as location, income and sex with up to 85% top-1 and 95% top-3 accuracy, at roughly 100 times lower cost and 240 times the speed of human labelers, and that common text anonymization did not reliably stop it [12]. Those are best results on labeled benchmarks, not a claim about any individual. Across the decade between them, the headline accuracy figures look similar; what fell was the cost and time of getting them.

For a builder, the consequence is concrete. Your extraction pipeline will infer sensitive traits whether or not anyone asked. Point a language model at a support transcript, ask for "anything relevant to the account," and it will record that the contact is on medical leave, going through a divorce, or observing a religious holiday next week. All of it is now in memory, retrievable by every agent that reads the record.

So enforce the never-infer list at extraction, not at generation. Say what not to capture in the property schema itself (Chapter 8's extract-with-a-contract), then check the output, because instructions reduce the problem and do not remove it. Better still, drop forbidden properties from the extraction spec before the model sees it. Store the operational fact, not the sensitive one: "unavailable until May 12; route to deputy" serves the account; "recovering from surgery" does not.

Inference is also often wrong, and fluency hides it. A wrong sensitive inference is a double failure: accuracy and privacy at once.

## Privacy belongs in the write path

The most important architectural decision in this chapter is where privacy runs. My answer, after getting it wrong once in the memory system I build, is: on the way in, before anything is embedded or extracted [13].

The reasoning is mechanical. In a memory layer, content goes to a language model for extraction and to an embedding model for retrieval. Once an identifier is in a vector, filtering text at read time hides it from the screen; it does not remove its influence on similarity search. Retrieval-time filtering is cosmetic. Only write-time controls are guarantees.

![The write path runs in four steps after content arrives from calls, emails and tickets. First, redact: secrets, financial numbers and identity numbers are removed by default, and contact data is configurable. Second, extract under a schema that says never infer health, sex life or orientation, religion, politics, ethnicity, union membership, genetic or biometric data, or personal-life circumstances. Third, check the extracted output for reconstructed identifiers and never-infer traits. Fourth, embed and store, scoped by entity key, with an audit record on provenance. The system stores the operational fact ("Unavailable until May 12; route to deputy"), not the sensitive one ("Recovering from surgery"), and it cannot do this at read time, because once an identifier is in a vector, filtering text hides it from the screen but not from similarity search.](/images/handbook/ch17-write-path-privacy.svg)

*Figure 19.3. Privacy runs on the way in: redact, extract under a never-infer contract, check again, then store; only write-time controls are guarantees.*

The mechanics are in Chapter 8 ("Redact on the way in"): secrets, financial and identity numbers removed by default; contact data configurable per deployment (a consistent hash is pseudonymous, not anonymous); two phases, before the model sees the content and again on what it extracted [13][14]. In the system I built, an operator can switch redaction off globally, per tier or per save, so keeping it on is a deployment rule, not a product guarantee. Three properties matter to a reviewer. Detection is a swappable layer you can audit, extend or replace; enforcement (which calls are intercepted, the restore step, the audit record) stays in the core. On a detection error, the system falls back to minimal redaction, or can be set to refuse the call; a pass-raw mode exists and should stay off [19]. And each external model call can log the entity types and counts removed plus a hash of the text sent, never the text, so "what did we send outside last month?" is a query.

Pattern-based redaction is precise on identifiers and blind to meaning; it will not catch a diagnosis described in plain words. Redaction handles identifiers; the extraction schema handles meaning.

Scoping is the other half, and it runs on two axes. Every memory belongs to one entity, and retrieval filters by entity key before similarity search runs; Chapter 13's leakage test shows why [15][18]. The other axis is the reader. In the self-hosted memory system I built, each person and agent can hold its own key, bound to a role and a namespace, enforced by row-level security inside Postgres and failing closed; a read-only agent is not even offered write tools. Test the policy with every key type: one release let a namespace-scoped admin key read every namespace, because the role check ran before the namespace gate. Isolation must be enforced by storage, never by semantics.

The same holds on the way out: the governed personalization engine I built passes customer-facing payloads through a serializer that allow-lists fields, so raw research or seller-only notes cannot leak just because someone added an internal property.

## Every place content leaves is a privacy decision

When I mapped egress for that self-hosted system, it was a closed list: up to three model calls on a save (extraction, embedding, a document rewrite), up to three on a retrieve (query embedding, intent classification, answer synthesis), and two for hosted agent jobs. Nothing is stored off the box, but protection is not uniform, and a builder has to know where it thins [19]:

- **The embedder is its own channel.** Stand-ins would not match between a save and a later query, so its inputs can only be redacted: names and emails still reach it. For regulated data, embed locally.
- **Query intent classification is redact-only**, since tokenizing would corrupt the search terms. Route it to a local model for full query privacy.
- **Hosted agent jobs are covered only where the system is in the path.** The job's text fields and tool results are tokenized, and echoed stand-ins are restored before any write, so a save never keys on a fake identifier; the rest of the provider-native request passes through as written.
- **Two paths are not covered:** a locally run curation pass sends its tool calls to the configured provider unprotected, and an interactive assistant connected to memory receives real values.

Tokenization, reversibly swapping identifiers for stand-ins, is an egress control, not an at-rest one: the provider never sees the real values, but they are restored and stored, so the data is still personal data (a partner's diligence review caught that our setting's name implied otherwise).

The choices form a ladder, most private first: fully local models, air-gapped if needed (no content leaves, at a cost in model size and speed); a cloud model with tokenization; a cloud model with redaction only (names and emails leave); raw content, acceptable only when the provider sits inside your own compliance boundary. On the cloud rungs, provider order matters; for regulated deployments I rank: a model in your own cloud account, reached through the cloud's workload identity so there is no provider key to leak; then a provider direct under a signed data processing agreement with no training and short or zero retention; never an aggregator router. The processing agreement, and a health-data business associate agreement where eligible, is generally part of the provider's terms at no extra cost. Verify current attestations and get retention terms in writing. None of this is legal advice, and none of it makes a deployment compliant by itself.

## Deletion is a test of your architecture, not your intentions

"Forget me" sounds like one operation. In an AI memory system it is many: typed properties, extracted memories, vectors, a long-form synthesis, cached context, drafts in a queue, decision traces, logs, backups, and copies synced to the CRM, the email platform and the ad audience. Delete the row and leave the synthesis, and the next agent still knows everything.

Three rules make deletion real:

1. **Keep a deletion manifest per entity.** Every representation written for a person registers itself, so erasure has a list to walk instead of a guess. Propagate to downstream systems and record their confirmation.
2. **Regenerate what was derived.** Summaries and syntheses that mentioned the person, or were built from deleted facts, get rebuilt, not just edited. If your design fine-tunes on customer data, know that the EDPB's view is that a model trained on personal data cannot always be considered anonymous [16], and the FTC has ordered deletion of models along with data [10]. The simplest defense is not training on customer records at all.
3. **Suppression survives deletion.** Objection to marketing must outlive the record, or the next enrichment import will bring the person back and the next campaign will email them. Keep a minimal suppression entry (for example a keyed hash of the address, flagged for this sole purpose) that every send path checks.

In the memory system I built, erasure is never metered or license-gated, also scrubs background consolidation summaries that cited the person, and is never offered to autonomous agents.

Deletion is where you find out whether "privacy by design" was design or décor.

## Channels have their own rules

In outline: US commercial email is opt-out (CAN-SPAM, which covers B2B mail); Canada is opt-in (CASL); the EU requires consent for most electronic marketing to individuals, with a narrow existing-customer exception. In the US, AI-generated voices count as "artificial" for telephone consent rules. Mailbox providers add their own bulk-sender rules: authentication, one-click unsubscribe, low complaint rates. The practical point: consent state and channel rules are inputs to the decision layer (Chapter 14), checked per person and per send. Thresholds, penalties and dates are in the companion note.

## What this does not do

It does not make you compliant; sector rules (health, finance, children) and contracts add obligations not covered here. It does not fully solve detection: regex finds identifiers, not meaning; name protection built on a supplied list misses unlisted names in free text, so pattern tools alone fall short of US Safe Harbor de-identification, which removes all 18 listed identifier types, free-text names and dates included; model-based detection adds its own errors. And it does not remove a real tradeoff. Less data sometimes means a less specific message. When the only specific thing you could say fails the appropriateness gate, the right output is honestly general. That costs some lift. It buys the thing lift depends on.

And two popular claims deserve pushback. The first is that consumers "expect personalization," usually supported by McKinsey's 2021 finding that 71% of consumers expect it [17]. That is a survey answer about wanting relevance, not consent to any use of any data; the same research base shows people punishing covert collection [1]. The second is the "privacy paradox" argument that because people share data despite saying they care, they do not really care. Aguirre's result points the other way: people share readily when collection is visible, and pull back when it is hidden [1]. The paradox is mostly a transparency problem.

## At scale

At a quarter of a million people, rare harms stop being rare: a sensitive inference that slips through once in ten thousand messages is dozens of people a month. The appropriateness gate has to run in code, on every decision, with its reasons in the decision trace (Chapter 18).

Scale multiplies jurisdictions, so policy is resolved per person (residence, consent state, channel) at decision time, not assumed per campaign. And scale multiplies copies: every new agent, cache and integration is one more place a deleted person can survive. Treat "can we find every copy of this person within a day?" as an operational metric, not an audit finding.

## Failure story: The Get-Well Email

A Larkspur support call runs long. The customer's fleet lead mentions, in passing, that she will be out for six weeks after surgery and that her deputy will handle the route sync issue. The extractor, asked for "anything relevant to the relationship," stores a memory: *contact recovering from surgery, returns mid-May.* Every fact is accurate.

Two weeks later the renewal agent, grounding its draft in memory exactly as designed, writes to her: "Wishing you a smooth recovery. When you're back in May, let's talk about expanding to your Phoenix depot." It passes every accuracy check. She forwards it to her CEO with one line: "Why does our software vendor know about my surgery?"

Nothing was hallucinated or hacked. The system stored health information nobody needed, then used it in a sales message. The fix had three layers: an extraction contract that captures "unavailable until May 15; route to deputy" and never the reason; a never-infer check on extracted output; and an appropriateness gate that blocks health, family and personal-life facts from generated sales copy.

The anti-pattern is **The Accurate Intrusion**: a true fact, used where the person never expected it to travel.

## Patterns

**Write-Path Privacy.** *Problem:* sensitive data reaches models, vectors and memory before any filter runs. *Forces:* extraction needs context; some identifiers are needed for linkage. *Solution:* tiered, two-phase redaction before extraction and embedding; always-on tiers for secrets, financial and identity numbers; configurable contact tier; audit record on provenance. *Tradeoffs:* false positives on numeric strings; misses meaning-level sensitivity, so pair with extraction contracts.

**Purpose-Tagged Memory.** *Problem:* facts collected for one purpose leak into another. *Forces:* agents read everything they can retrieve. *Solution:* every property and memory carries source, lawful basis or consent state, allowed purposes and expiry; the decision layer filters by purpose before a fact reaches a prompt; the appropriateness gate runs on the draft. *Tradeoffs:* more metadata per fact, and a schema discipline most teams have to learn.

**Deletion Manifest.** *Problem:* erasure misses derived and downstream copies. *Forces:* many representations, many systems. *Solution:* register every representation per entity at write time; erasure walks the manifest, regenerates derived syntheses, propagates downstream, records confirmations; keep a minimal suppression entry. *Tradeoffs:* write-path overhead, and discipline for every new integration.

## Leader questions

1. What is on our never-store and never-infer lists, and where in the system are they enforced?
2. If a customer asked "how do you know that?" about our last campaign, would every answer make us look better?
3. How long does a full erasure take, and how do we prove it reached every copy, including vendors?
4. Which of our automated decisions could significantly affect someone (price, terms, eligibility), and who reviews them?
5. What do our privacy claims to customers promise, and does the architecture guarantee each one?

## Build checklist

- [ ] Redaction runs before extraction and embedding; secrets, financial and identity numbers are always removed.
- [ ] Post-extraction scan for reconstructed identifiers.
- [ ] Extraction schemas state what never to capture (special categories, personal life) and store operational facts instead.
- [ ] Every fact carries source, consent or lawful basis, allowed purposes and expiry.
- [ ] Every model and embedding call that can send content out is listed with its protection; the embedder counts as its own channel.
- [ ] Retrieval is scoped by entity key before similarity search; per-person and per-agent keys are enforced in the database; customer-facing payloads pass an allow-list serializer.
- [ ] An appropriateness gate (expectation, purpose, benefit, say-how-we-know) runs on generated drafts, with reasons in the decision trace.
- [ ] Objection to marketing is a hard suppression checked by every send path and survives deletion.
- [ ] A per-entity deletion manifest covers properties, memories, vectors, syntheses, caches, queues, logs, document history and downstream systems.
- [ ] No training or fine-tuning on customer records unless deletion from the model is solved.
- [ ] Consent state and channel rules are inputs to each send decision, resolved per person.

## Metrics to watch

- **Sensitive-inference rate:** share of extracted memories flagged by the never-infer check (should trend to near zero, then stay there).
- **Appropriateness-gate blocks:** count and reasons per thousand drafts; a sudden rise usually means a new data source.
- **Time to complete erasure,** end to end, including downstream confirmations.
- **Suppression leaks:** messages sent to people who objected (target: zero; every one is an incident).
- **"How did you know" complaints:** replies and tickets questioning how you knew something, tagged and reviewed monthly.

## Reader Q&A

**Is B2B data covered by privacy law?** Yes, where it identifies a person. A named contact's work email, title and behavior are personal data under GDPR and in scope for CASL and CAN-SPAM. Company-level facts about an organization generally are not.

**If the data is public, can we use it?** Public is not the same as expected. Public data still needs a lawful basis in the EU, and inferences drawn from it can be sensitive (the *OT* principle). Run the appropriateness gate regardless.

**Is AI personalization "manipulation" under the EU AI Act?** Not by itself [8]. Any system precise enough to serve a person is precise enough to pressure them; the law, and this handbook, draw the line at the second.

## For your AI

```yaml
chapter: 19
concepts:
  - name: Appropriateness Check
    definition: "Four-question gate on any personal fact before use: expectation, purpose, benefit, say-how-we-know. Separate from accuracy."
  - name: Never-Store List
    definition: "Data the memory layer removes on ingestion by default, kept always on as a deployment rule because the system allows it to be switched off: secrets, financial account numbers, identity document numbers."
  - name: Never-Infer List
    definition: "Traits the system must not extract or derive: health, sex life or orientation, religion, politics, ethnicity, union membership, genetic or biometric data, and personal-life circumstances; store operational facts instead."
  - name: Write-Path Privacy
    definition: "Tiered, two-phase redaction before extraction and embedding, because stored vectors and memories cannot be reliably filtered afterward."
  - name: Deletion Manifest
    definition: "Per-entity registry of every representation and downstream copy, walked on erasure; derived syntheses are regenerated; suppression survives deletion."
decision_rules:
  - if: "a fact is accurate but the person would not expect us to know it, or its source would make the message worse if disclosed"
    then: "do not use it; make the source obvious or fall back to an honestly general statement"
  - if: "an extracted fact reveals or implies a special-category trait"
    then: "discard it and store only the operational consequence (for example availability or routing)"
  - if: "a person objects to direct marketing"
    then: "hard-suppress across every channel and agent, with no balancing, and keep a minimal suppression entry after deletion"
  - if: "an automated output sets price, terms, eligibility or access for a person"
    then: "treat it as a possible regulated automated decision: human review path, explanation, contest route; check the companion note for jurisdiction"
  - if: "a privacy claim (anonymized, not sold, deleted) is made to customers"
    then: "verify the architecture guarantees it; hashed identifiers are not anonymous, and tokenization protects what a model provider sees, not what is stored"
assessment_questions:
  - "Where does redaction run relative to extraction and embedding?"
  - "Which traits does your extraction schema explicitly forbid capturing?"
  - "Does every stored fact carry source, consent or lawful basis, allowed purposes and expiry?"
  - "How many systems hold a copy of a customer, and can erasure reach all of them within a set time?"
  - "Which jurisdictions do your customers live in, and is policy resolved per person at decision time?"
  - "Do any automated outputs affect price, terms or eligibility?"
patterns: [Write-Path Privacy, Purpose-Tagged Memory, Deletion Manifest]
anti_patterns: [The Accurate Intrusion, Retrieval-Time Filtering as Privacy, Hashing as Anonymization, Tokenization Sold as At-Rest Protection, Hide the Inference, Deletion That Leaves the Summary]
maturity_dimension: governance_and_reliability
companion_note: regulation-as-of-2026-09
not_legal_advice: true
```

## References

1. Aguirre, E., Mahr, D., Grewal, D., de Ruyter, K., Wetzels, M. (2015). "Unraveling the personalization paradox: The effect of information collection and trust-building strategies on online advertisement effectiveness." *Journal of Retailing* 91(1). https://www.sciencedirect.com/science/article/abs/pii/S0022435914000669
2. Duhigg, C. (2012-02-16). "How Companies Learn Your Secrets." *The New York Times Magazine*. https://www.nytimes.com/2012/02/19/magazine/shopping-habits.html
3. Regulation (EU) 2016/679 (GDPR), Arts. 5, 6, 9, 17, 21, 22, 25; Recital 47. https://eur-lex.europa.eu/eli/reg/2016/679/oj
4. CJEU, Case C-184/20, *OT v Vyriausioji tarnybinės etikos komisija*, judgment of 2022-08-01. https://curia.europa.eu/juris/liste.jsf?num=C-184/20
5. FTC (2023-02-01). "FTC Enforcement Action to Bar GoodRx from Sharing Consumers' Sensitive Health Info for Advertising." https://www.ftc.gov/news-events/news/press-releases/2023/02/ftc-enforcement-action-bar-goodrx-sharing-consumers-sensitive-health-info-advertising
6. CJEU, Case C-634/21, *SCHUFA Holding (Scoring)*, judgment of 2023-12-07. https://curia.europa.eu/juris/liste.jsf?num=C-634/21
7. CJEU, Case C-203/22, *Dun & Bradstreet Austria*, judgment of 2025-02-27 (press release 22/25). https://curia.europa.eu/site/upload/docs/application/pdf/2025-02/cp250022en.pdf
8. Regulation (EU) 2024/1689 (AI Act), Art. 5; European Commission Guidelines on prohibited AI practices, 2025-02-04. https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-5 ; summary: https://www.insideprivacy.com/artificial-intelligence/european-commission-guidelines-on-prohibited-ai-practices-under-the-eu-artificial-intelligence-act/
9. FTC Office of Technology (2024-07-24). "No, hashing still doesn't make your data anonymous." https://www.ftc.gov/policy/advocacy-research/tech-at-ftc/2024/07/no-hashing-still-doesnt-make-your-data-anonymous
10. FTC (2023-12-19). "Rite Aid Banned from Using AI Facial Recognition After FTC Says Retailer Deployed Technology without Reasonable Safeguards." https://www.ftc.gov/news-events/news/press-releases/2023/12/rite-aid-banned-using-ai-facial-recognition-after-ftc-says-retailer-deployed-technology-without
11. Kosinski, M., Stillwell, D., Graepel, T. (2013). "Private traits and attributes are predictable from digital records of human behavior." *PNAS* 110(15). https://www.pnas.org/doi/10.1073/pnas.1218772110
12. Staab, R., Vero, M., Balunović, M., Vechev, M. (2024). "Beyond Memorization: Violating Privacy Via Inference with Large Language Models." ICLR 2024. https://arxiv.org/abs/2310.07298
13. Taheri, H. (2026-03-28). "4-Tier PII Redaction: How We Built Privacy Into the Memory Layer, Not Around It." https://hamedtaheri.com/articles/four-tier-pii-redaction
14. Taheri, H. (2026-03-14). "Two-Phase Redaction: Scrubbing PII Before and After LLM Extraction." https://hamedtaheri.com/articles/two-phase-redaction-pii
15. Taheri, H. (2026-03-14). "Zero Cross-Entity Leakage Across 3,800 Results." https://hamedtaheri.com/articles/zero-cross-entity-leakage
16. EDPB (2024-12-17). Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models. https://www.edpb.europa.eu/documents/opinion-of-the-board-art-64/opinion-282024-on-certain-data-protection-aspects-related-to_en
17. McKinsey & Company (2021-11-12). "The value of getting personalization right, or wrong, is multiplying." https://www.mckinsey.com/capabilities/growth-marketing-and-sales/our-insights/the-value-of-getting-personalization-right-or-wrong-is-multiplying
18. Taheri, H. (2026-03-18). "Governed Memory: A Production Architecture for Multi-Agent Workflows." arXiv:2603.17787. https://arxiv.org/abs/2603.17787
19. First-party: operator documentation for a self-hosted governed memory system built by the author (data flow and PII handling, access control, hosted jobs, partner brief), release 0.8.2, verified 2026-09-16.

---

# 20. Accuracy and Quality

*The question: How do we achieve and keep accuracy at scale?*

> **Questions this chapter answers**
> - How accurate is "accurate enough" for personalization a customer will read?
> - Which checks should be code, and which need a model or a person?
> - Can a model check its own work, and can AI reliably supervise AI?
> - How do we find the errors no single review will ever see?
> - When something goes wrong, how do we fix the right thing instead of adding another paragraph to the prompt?

## The short answer

Accuracy in personalization is not "the model rarely hallucinates." A message is accurate when it is about the right person, uses facts trustworthy enough for the surface, makes only claims that follow from those facts, follows the campaign's rules, renders correctly, and arrives. A perfect sentence delivered to the wrong account is inaccurate. So is a correct page that never loads.

That makes accuracy a property of the system, built in layers, cheapest and most certain first:

1. **Deterministic controls** for everything code can decide: schemas, validators, allowlists, data-trust rules, and an abstention policy that makes the system say less when it knows less.
2. **Self-review** inside each generation step: cheap, useful, never the final word.
3. **An AI judge** for meaning code cannot check, calibrated against human labels, because judges have measurable biases.
4. **A stratified human sample**, so reviewers see the clean-looking records where blind spots hide.

The standard throughout: *specific and true, or honestly general.*

Keeping accuracy is a different job from reaching it. The first real cohort is part of system design, because it exposes defect patterns no test set anticipated. Every defect is diagnosed to the layer that caused it before anything changes, and every fix is validated against three sets: the failures that exposed it, a held-out sample, and previously good cases.

For a leader, the test is simple. Ask for the deterministic defect rate across the whole last cohort, and the semantic defect rate on a random sample of records that passed every automated check. If your team can give you both numbers, you have a quality system. If they show you five excellent examples, you have a demo.

## Two thousand messages, and the pattern no one could see

*Larkspur Systems is a fictional company used throughout this handbook. Its numbers are illustrative.*

Larkspur's first production cohort was 2,000 renewal-season messages to fleet operations leaders: an email, a personalized landing page, and a one-page brief for the account manager. Before launch, the team had read 25 of them closely. All 25 were excellent.

Then the engine ran a deterministic scan across all 2,000 rendered artifacts. It found 42 pages showing a headcount of exactly 10,001, a data provider's bucket floor that a trust rule should have rejected. It found 17 landing pages whose hero displayed a literal fragment of JSON, all from one template version that mapped the wrong zone key. It found 9 emails thanking a former customer, whose contract had lapsed, "as a valued customer." None of it was visible in the 25-record review, because none of it lived in those 25 records.

The more instructive finding came from the semantic pass. An AI reviewer read every flagged record plus a random sample of 200 records that had passed every check, and it noticed something no single record revealed. For accounts with thin data, the engine framed the message around "scaling your fleet" in almost four cases out of ten, including accounts whose telemetry showed a shrinking vehicle count. Each message was fluent on its own. Together they were a pattern, and it traced back to one worked example in the campaign guideline that happened to describe a growing fleet.

The fix was not a better model and not a longer prompt. It was a rejected-value rule, a repair in the shared rendering function, a customer-status check former customers could not pass, and a new guideline example. Four defects, four layers, four owners.

Larkspur's direct-mail team had caught the same 10,001 floor a year earlier, but that rule lived in the print workflow. A rule that lives in one channel protects one channel.

Accuracy is found in populations, fixed at the layer that caused it, and proved on records the fix was not designed around.

## Accurate means right person, true claims, right contract, delivered

Model benchmarks teach a narrow definition of accuracy: did the output contain a false statement? In an enterprise engine the test is the one in the short answer, plus two clauses: hard controls override softer preferences, and failures are visible, bounded, and recoverable.

Each clause belongs to a different layer: identity (Chapter 9), memory and freshness (Chapters 10 and 11), the generation contract (Chapter 15), governance (Chapter 18), and rendering and delivery. Accuracy breaks at whichever layer nobody tested. A second axis sits beside truth: appropriateness. A correct fact can still be wrong to use on this channel, for this person (Chapter 19). Truth checks and appropriateness checks are different checks, and both need an owner.

The standard that holds this together is **specific and true, or honestly general.** Verified, current, account-level evidence earns specificity. Thin evidence falls back to what is true for the company, then the industry, then an approved static version. Moving down that ladder is a designed outcome, not a failure.

> Specificity should degrade before reliability does.

## The accuracy stack runs cheap and certain first

Here is the sentence this chapter turns on: **never ask a probabilistic component to guarantee something a deterministic one can check.** Models reason. Guidelines express judgment. Code enforces invariants. Put as questions: AI asks whether this is semantically good and relevant; code asks whether it is structurally allowed and safe; operations asks whether the intended artifact actually arrived. Quality works when each control does the job it is good at, and fails when one control is asked to do all of them.

| Layer | What it answers | Cost per record | Coverage |
|---|---|---|---|
| Deterministic controls | Is this permitted and structurally safe? | Near zero | 100% of records |
| Self-review | Does this draft hold up against its context? | One extra pass, context already loaded | 100% of generations |
| AI judge | Is this semantically right for this person and contract? | A model call per record reviewed | Flags plus a stratified sample; full census for high-risk launches |
| Human sample | Is the judge right, and what is everyone missing? | Expert time | Small, stratified, recurring |

Anything the first layer can catch should never consume the attention of the fourth.

![The accuracy stack has four layers that run in order. Deterministic controls ask whether a record is permitted and structurally safe, at near-zero cost, on 100% of records. Self-review asks whether a draft holds up against its context, for one extra pass on 100% of generations. An AI judge checks semantic fit for this person and contract at a model call per record reviewed, on flags plus a stratified sample; a small, stratified, recurring human sample spends expert time checking whether the judge is right and what everyone is missing.](/images/handbook/ch18-accuracy-stack.svg)

*Figure 20.1. Cheap and certain controls run first, so scarce expert attention goes only to what code cannot decide.*

## If a rule can be deterministic, it should not depend on the model remembering it

Deterministic controls sit on both sides of generation.

**Before generation, constrain the context.** Data-trust rules decide which values the model may see. An employee count that matches a provider's bucket floor, falls outside a plausible range, conflicts five-fold with explicit research, or sits at 100 beside every marker of a multinational is rejected and removed from context entirely; the model must not be able to reason its way back into using it. An abstention policy turns uncertainty into an explicit state (known, likely, uncertain, rejected), and the rule for customer-facing copy is short: hide when uncertain.

**During generation, constrain the shape.** OpenAI reported its August 2024 model scored 100% on a complex schema-following evaluation in strict mode, against under 40% for an earlier model ([OpenAI, 2024](https://openai.com/index/introducing-structured-outputs-in-the-api/)); Anthropic documents the same schema guarantee ([Anthropic docs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs)). But a schema guarantees shape, not truth. A field typed as a string will always contain a string, not always the right one.

**After generation, check the claims.** A validator extracts concrete claims (named products, numbers, dates, events, relationship statements) and tests each against the verified context and campaign rules: no customer-status claim for a prospect, no leaked internal source name, no unsafe URL, every high-risk claim grounded in allowed evidence. "Your industry faces margin pressure" and "your company acquired a competitor for $480 million" are not the same kind of sentence, and the second deserves the strictest check.

Two lessons from operating this layer. When a section fails, replace the whole section with a true, general fallback; surgical edits to generated prose wreck its grammar. And put checks at shared choke points, the last function every output passes through before a boundary, so a new campaign inherits the protection without anyone remembering to add it.

The same rule applies to the plumbing under the checks: success must mean landed. In my memory system, a saturated database once accepted saves and returned success while writing nothing, and a failed embedding once came back as a success with an empty vector. Both now fail loudly, at the cost of a slower save. A quality stack reading from memory that silently did not update is checking the wrong facts.

A grounding check is not a universal factuality oracle; it covers only the claim classes it knows. In the engine I built it covers acquisitions, funding, and certain numeric assertions; an acquisition the evidence does not contain is flagged, rewritten, or held. But even a narrow check makes the most expensive mistakes impossible rather than unlikely.

## Self-review is a draft, not a verdict

Each generation step reviews its own output before the deterministic layer runs: re-read every specific claim, confirm it appears in the supplied context, remove it if it does not, and fill any gap with something true and general. It is cheap because the context is already loaded.

The research explains why it helps and why it cannot be the last word. Self-Refine had a model critique and revise its own output and reported roughly 20% average improvement across seven tasks ([Madaan et al., 2023](https://arxiv.org/abs/2303.17651)). Huang et al. found that on reasoning tasks "LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction" ([Huang et al., ICLR 2024](https://arxiv.org/abs/2310.01798)). A critical survey resolves much of the tension: self-correction "works well in tasks that can use reliable external feedback," and the authors found no prior work showing success from prompted-LLM feedback alone except in exceptionally suited tasks ([Kamoi et al., TACL 2024](https://arxiv.org/abs/2406.01297)).

I read that as a design instruction. Self-review works in personalization when it is not purely intrinsic: the model checks its claims against an external source, the verified customer context, rather than its own sense of what seems right. Two techniques make that explicit. **Chain-of-Verification** drafts, plans verification questions, answers them independently "so the answers are not biased by other responses," then revises ([Dhuliawala et al., 2023](https://arxiv.org/abs/2309.11495)). **SelfCheckGPT** samples several responses and treats disagreement between them as a fabrication signal, with no external database needed ([Manakul et al., EMNLP 2023](https://arxiv.org/abs/2303.08896)).

Self-review remains a soft control. The same probabilistic component should never be the sole judge of whether it followed the rules.

> A model grading its own homework has produced a better draft, not a review.

## AI can supervise AI, if you supervise the supervisor

Code cannot judge whether a sentence implies a relationship that does not exist, or whether a brief is too generic for an account the company knows well. For those questions, an independent AI reviewer is the only thing that scales.

The evidence that it can work is real, and so are the caveats. Zheng et al. found strong model judges "achieving over 80% agreement" with human preferences, "the same level of agreement between humans," and named position, verbosity, and self-enhancement biases ([Zheng et al., NeurIPS 2023](https://arxiv.org/abs/2306.05685)). Wang et al. showed that by changing only answer order, a much smaller model "could beat ChatGPT on 66 over 80 tested queries with ChatGPT as an evaluator" ([Wang et al., 2023](https://arxiv.org/abs/2305.17926)). Panickssery, Bowman, and Feng found evaluators score their own outputs higher than others' while humans rate them equal ([Panickssery et al., 2024](https://arxiv.org/abs/2404.13076)).

A judge is an instrument, and instruments need calibration:

- **Judge against the contract, not against "good."** The reviewer first resolves what the campaign promised (company name only, fully personalized, or a brief and score with no customer copy), then gets the trusted context and the final customer-facing output. A company-level experience should not fail for omitting a name; a brief promised as deeply personalized should not pass with generic industry copy. Written principles as the rubric is the move Constitutional AI made at training time ([Bai et al., 2022](https://arxiv.org/abs/2212.08073)).
- **Score atomic claims.** FActScore splits a generation into atomic facts and computes the share supported by a knowledge source; ChatGPT scored 58% on biographies ([Min et al., EMNLP 2023](https://arxiv.org/abs/2305.14251)). Here, the knowledge source is the customer's verified memory.
- **Evidence before verdict, positions swapped.** Ask the judge to cite supporting or contradicting context before rating, and swap order when comparing variants.
- **Different model family** from the generator where practical, to blunt self-preference.
- **Measure the judge** against a human-labeled set, as a first-class metric.
- **Challenge the blockers.** When the judge finds an issue serious enough to stop a release, a second reviewer tries to refute it first. False blockers cost campaigns.
- **Approve the judge, not each verdict.** In the memory system I built, an AI judge decides whether two records are the same person. Verdicts below a confidence floor are refused outright; a new judge only proposes until an operator reviews its proposals and promotes it, one stage at a time.

The output is a release verdict (GO, GO WITH FIXES, NO GO) with versioned thresholds a person can read. That is how QA ends in the personalization engine I built. For high-value launches, a domain expert also reads the first outputs before scale: a risk-based calibration gate, not a human review of every record.

## QA the thing the recipient sees, across the whole population

**Check the rendered artifact.** A perfect zone in a database can still become a broken page: wrong template version, wrong zone key, a hidden field, a stale cache, a page identity shared with another campaign, a missing URL, a static label that contradicts the generated copy. So QA runs on what the customer receives: the rendered page, the assembled email, the brief the rep opens, the print-ready PDF, the exact payload a downstream system accepts.

**Check everything cheap, sample everything expensive.** Deterministic checks run on 100% of records, which gives a real baseline. My population scan looks for classes code can name: a missing artifact, an empty stub brief, a suspicious firmographic, a personal email address that inherited a corporation's profile, a risky claim, a score of 88 labeled LOW. Semantic review then runs on every flag plus a stratified random sample of records that looked clean, stratified by evidence tier, persona, industry, template, and channel. The flags confirm known risks. The clean sample estimates what automation is missing, the number most teams never know.

> Sample the clean records, because that is where your blind spots live.

## The first real cohort is part of system design

A campaign is not finished when it passes its test records. The first few hundred real records reveal behavior the design team did not anticipate, because production data is stranger than any test set.

Site reliability engineering solved the release half of this long ago. Google's SRE workbook defines canarying as "a partial and time-limited deployment of a change in a service and its evaluation," judged on a few attributable metrics, "perhaps no more than a dozen" ([Google SRE Workbook](https://sre.google/workbook/canarying-releases/)). A pilot cohort is a canary with one addition: the reviewer looks for patterns, not only thresholds. I use AI as an analyst across the population, asking what no single record can answer: What errors repeat? Which personas expose a guideline gap? Where is the engine too generic for what we know? Which fallbacks fire too often? Which facts are consistently unreliable? Which strong outputs share a pattern worth keeping?

The protocol: define the contract; run a small real cohort; deterministic QA on all of it; semantic review on flags plus a stratified clean sample; population pattern analysis; diagnose each pattern to a layer; fix, validate, expand. My steps: synthetic records, then the first 10 to 25 real ones, then 50 to 100, then a few hundred, each read as a population before the next opens.

![The pilot cohort protocol runs in seven steps: define the contract, run a small real cohort, run deterministic QA on all of it, and run semantic review on every flag plus a stratified clean sample. Then comes population pattern analysis, asking questions such as which errors repeat and which fallbacks fire too often, followed by diagnosing each pattern to a layer, and finally fixing, validating, and expanding. It is a canary with one addition: the reviewer looks for patterns, not only thresholds.](/images/handbook/ch18-pilot-cohort-protocol.svg)

*Figure 20.2. The first real cohort is a canary: find population patterns and diagnose their layer before expanding.*

## Diagnose the layer before changing the system

The most common failure in AI operations is solving every problem by adding a paragraph to the prompt. The prompt grows, rules conflict, and nobody can say which instruction caused which behavior. Classify the cause first:

| Root cause | Typical response |
|---|---|
| Factual or safety defect | Deterministic prevention, a guideline rule, a regression test |
| Identity or data-quality edge case | Trust rule or classification, plus a safer fallback |
| Guideline defect | Update the guideline or its examples |
| Generator defect | Change the output contract or prompt architecture |
| Template or render defect | Fix the shared rendering function |
| Delivery defect | Retry, recovery, routing, or observability |
| Platform pattern (recurs across campaigns) | A shared platform mechanic, not a local patch |
| Campaign preference | Local configuration or guideline, not a platform change |
| Quality opportunity (correct, but could be better) | Messaging or guideline tuning, not a defect fix |
| Performance hypothesis | A controlled experiment, not a safety rule |

In my own memory system, validation before a release caught a weaker model putting an email address into the record-id field, creating orphan duplicates. A prompt paragraph would have lowered the rate; a code guard that coerces identifier-shaped values into the identity field removed the class.

## Two loops, three sets

Quality learning and performance learning share telemetry but must not share authority. The **quality loop** observes a defect, classifies its cause, fixes the guard, guideline, or fallback, and confirms recurrence drops. The **performance loop** forms a hypothesis about a content pattern, runs a controlled variation, and measures the outcome (Chapter 21). A system must never relax a factual guard because a bolder message earned more clicks.

> Correctness defines the feasible region. Optimization happens inside it.

Every change is validated against three sets, because a fix tested only on the records that exposed the problem overfits:

- **Replay set:** the records that exposed the defect.
- **Held-out set:** similar records not used to design the fix.
- **Regression set:** previously good records and known edge cases.

A change is ready when it reports: known failures fixed, held-out improved, good cases unchanged, no new blockers. Illustratively: 11 of 83 records show unsupported relationship language; the fix is one guideline update, one guard, six regression examples; replay 11 of 11, held-out 29 of 30, regression 40 of 40. Guidelines are versioned like code, with reason, evidence, validation results, approver, and rollback target.

Every QA run also writes a trend-ledger row (error rates, sample size, verdict, issue classes, versions). That turns "is today's sample okay?" into "are we improving or regressing across versions?"

![Two learning loops draw on shared telemetry but hold separate authority. The quality loop observes a defect, classifies its cause, fixes the guard, guideline, or fallback, and confirms recurrence drops; the performance loop forms a hypothesis, runs a controlled variation, and measures the outcome. Every change is validated on three sets: a replay set of the records that exposed the defect, a held-out set of similar records not used to design the fix, and a regression set of previously good records and known edge cases.](/images/handbook/ch18-two-loops-three-sets.svg)

*Figure 20.3. Quality and performance learning share data, not authority, and no change ships until it passes replay, held-out, and regression sets.*

Autonomy fits here only under Chapter 18's two approvals: a scheduled routine may run QA and propose a change with evidence, but a person approves the diagnosis and, separately, the proven release. In my engine, QA, the verdict, and the ledger run today and a person diagnoses; the loop that proposes and validates on the three sets is the next step, not a result.

## Quality is also maintenance

Accuracy decays even when nothing in the engine changes, because the world changes under the memory. Continuous background curation of customer memory, which this handbook calls dreaming, is the maintenance side of quality (Chapter 11). The quality system's job is to measure whether it works: how often generation abstains for stale evidence, and how often a curated fact is later contradicted. In my memory system, every curation pass reports writes applied, proposed, and refused; a record that fails three passes in a row is quarantined for a week; and two routines that keep rewriting the same property are reported as an oscillation rather than left to fight silently.

## What this does not do

A chapter about accuracy that overstated its own case would refute itself, so here are the limits.

**Hallucination is not solved.** Vectara's grounded-summarization leaderboard, a narrow and favorable task, showed rates from about 1.8% to 24.2% across listed models as of September 2026 ([Vectara](https://github.com/vectara/hallucination-leaderboard)). Open generation about a specific person is harder. The stack contains errors; someone still owns the residual.

**A judge is not an oracle.** Eighty percent agreement is the level at which humans agree with each other, so a judge will be confidently wrong on a meaningful share of hard cases. A judge never measured against human labels is an opinion with a number attached.

**Deterministic checks are silent about unnamed errors.** They make known classes impossible and say nothing about the rest, which is why the clean sample exists. On my own system, a configuration name mismatch once disabled semantic ranking, so retrieval quietly returned the newest memories instead of the most relevant, and every check passed. My paper on governed memory names silent quality degradation as a structural problem for this reason ([Taheri, 2026](https://arxiv.org/abs/2603.17787)).

**Small evaluations are coverage tests.** I have published a result I am proud of and careful about: 100% policy compliance across 50 adversarial governance scenarios in our own tests (the paper reports 100% adversarial governance compliance for the same architecture). It shows the architecture held under the pressure we designed, not under pressure we did not imagine. Treat any vendor's accuracy number the same way, including mine.

**Accuracy is the floor, not the goal.** A message can be specific, true, and useless. Whether it was worth sending is a decision problem (Chapter 14) and a measurement problem (Chapter 21).

I also disagree with a popular shortcut: "add a reflection step and the model will fix itself." Reflection earns its place only when it has evidence to reflect against (Huang et al.; Kamoi et al.).

## At scale

At scale, rates become counts. A 2% defect rate is 40 messages in a 2,000-record pilot and 20,000 across a million. The reader who gets one does not experience a rate; they experience a company that got them wrong.

Three things change with volume. The human sample stays small while the population grows, so stratification matters more than size. Cost pushes semantic review toward sampling, which makes judge calibration more important. And every model update is a release: in January 2024 a parcel carrier disabled its support chatbot after a system update left it swearing at a customer ([ITV News, 2024](https://www.itv.com/news/2024-01-19/dpd-disables-ai-chatbot-after-customer-service-bot-appears-to-go-rogue)). Model and guideline changes go through the same three sets and canary as code. Across many teams, protections in shared functions are inherited by every new campaign; protections in individual prompts are inherited by none. Once several campaigns run at once, the next step I would take (a design, not yet a result) is scheduled portfolio QA: the cheap deterministic scan across every active campaign, with semantic review aimed at flags, random clean samples, and any campaign whose trend-ledger row regressed.

## Failure story: The Self-Graded Exam

*A composite of a pattern that recurs.*

A team used the same model that generated its messages, with a "review this for accuracy" prompt, as the quality gate. The gate passed 97% of outputs, and leadership approved a full launch on that number. The first complaint came within days: a message congratulating an account on an expansion that had been cancelled. A human review of 100 passed records found unsupported claims in a double-digit share, clustered in accounts with thin data. The judge shared the generator's blind spot: it had no evidence to check against, and a measurable tendency to favor text like its own.

The fix was the stack: trust rules and abstention before generation, claim-level grounding after it, a judge from another model family scoring atomic claims against memory, and a human-labeled set to measure that judge. **The Self-Graded Exam** is a quality number produced by the component being graded.

## Patterns

**1. The Accuracy Stack.** *Problem:* one control, usually the prompt, is asked to guarantee truth, compliance, and quality at once. *Forces:* code is cheap and certain but narrow; models are broad but probabilistic; humans are accurate but scarce. *Solution:* deterministic controls, self-review, a calibrated independent judge, and a stratified human sample, in that order. *Tradeoffs:* more components; trust rules need upkeep; calibration needs a labeled set.

**2. The Pilot Cohort Protocol.** *Problem:* test records lack production's defect classes, and demo reviews hide the long tail. *Forces:* speed to launch versus exposure. *Solution:* treat the first real cohort as a canary, with population-level pattern analysis and layer diagnosis before expanding. *Tradeoffs:* a slower first launch; tooling to render and inspect final artifacts.

**3. Three Validation Sets.** *Problem:* fixes tuned on the failures that exposed them quietly break other cases. *Forces:* pressure to ship; few failing examples. *Solution:* report replay, held-out, and regression results for every change, and promote only with human approval and a rollback target. *Tradeoffs:* the regression set is ongoing work; small held-out sets are noisy.

## Leader questions

1. What was our deterministic defect rate across 100% of the last cohort, and the semantic defect rate on a random sample of records that passed every check?
2. Which quality checks are code, and which depend on a model remembering an instruction?
3. How well does our AI judge agree with human reviewers, and when did we last measure it?
4. When we fixed the last quality problem, which layer did we change, and how do we know nothing else broke?
5. Can any automated process change production behavior without a person approving it?

## Build checklist

- [ ] Data-trust rules reject implausible or conflicting values, and rejected values leave the context.
- [ ] An explicit specificity ladder (person, account, segment, approved static, fail closed).
- [ ] Structured outputs everywhere, treated as shape guarantees only.
- [ ] Claim-level validators for high-risk claims at shared choke points; failed sections replaced whole.
- [ ] Self-review that checks claims against supplied context.
- [ ] An AI judge scoring atomic claims against contract and memory, ideally from another model family.
- [ ] A human-labeled set measuring judge-human agreement, refreshed on a schedule.
- [ ] QA on rendered, delivered artifacts.
- [ ] Root-cause taxonomy for every defect; replay, held-out, and regression sets for every change.
- [ ] A trend ledger row per QA run (error rates, sample size, verdict, issue classes, versions).
- [ ] Versioned guidelines with two-approval promotion for automated proposals.

## Metrics to watch

- **Deterministic defect rate** across 100% of each cohort, by defect class.
- **Semantic material-issue rate** on the stratified clean sample.
- **Judge-human agreement**, tracked across model updates.
- **Abstention and fallback rate** by evidence tier (too low may mean invented specificity; too high, weak data).
- **Recurrence rate** of fixed defect classes.

## Reader Q&A

**How accurate is accurate enough?** It depends on the consequence of the surface. A wrong claim in an internal seller brief is caught by a person; the same claim in a customer email is not. Set thresholds per surface and claim class, and version them.

**Should the judge review every record?** For high-risk launches, a full census can be justified. In steady state, review every flag plus a stratified clean sample large enough per stratum to notice a pattern.

**Who owns accuracy?** Each layer has an owner, and one person owns the release verdict. If nobody owns the verdict, everyone owns the incident.

## For your AI

```yaml
chapter: 20
concepts:
  - name: Enterprise accuracy
    definition: "End-to-end property: right person, trustworthy facts, claims that follow from them, contract followed, correct render, confirmed delivery, bounded and recoverable failure."
  - name: Honestly General
    definition: "The accuracy standard (specific and true, or honestly general): use specificity only where evidence supports it; otherwise degrade to a less specific, true message."
  - name: Accuracy Stack
    definition: "Deterministic controls, then self-review, then a calibrated AI judge, then a stratified human sample; cheap and certain first."
  - name: Abstention policy
    definition: "Uncertainty is an explicit state (known, likely, uncertain, rejected); customer-facing copy hides what is uncertain."
  - name: Pilot Cohort Protocol
    definition: "Treat the first real cohort as a canary: deterministic QA on all records, semantic review on flags plus a stratified clean sample, population pattern analysis, layer diagnosis, then expand."
  - name: Three validation sets
    definition: "Every change is tested on replay (exposing failures), held-out (similar unseen records), and regression (previously good and edge cases) sets."
  - name: Two learning loops
    definition: "A quality loop that fixes defects and a performance loop that runs experiments; they share telemetry but not authority."
  - name: Trend ledger
    definition: "One row per QA run (deterministic and semantic error rates, sample size, verdict, issue classes, guideline and template versions), so quality is compared across versions, not judged per sample."
decision_rules:
  - if: "a rule can be checked by code"
    then: "enforce it deterministically; do not rely on a prompt instruction"
  - if: "a value fails a data-trust rule"
    then: "remove it from the model's context entirely"
  - if: "evidence for a specific claim is missing, stale, or conflicting"
    then: "degrade to account, segment, or approved static content; never invent to stay specific"
  - if: "a generated section fails validation"
    then: "replace the whole section with a true, general fallback"
  - if: "an AI judge is used for release decisions"
    then: "measure its agreement against human labels and use a different model family from the generator where practical"
  - if: "the judge raises a release blocker"
    then: "run a second reviewer that tries to refute it before stopping production"
  - if: "an AI judge makes decisions that write to production state"
    then: "refuse verdicts below a confidence floor, start the judge in propose-only mode, and promote it stage by stage on reviewed evidence"
  - if: "a defect is found"
    then: "classify the responsible layer before proposing a fix"
  - if: "a change is proposed"
    then: "validate on replay, held-out, and regression sets; promote only with human approval and a rollback target"
  - if: "a model or provider version changes"
    then: "treat it as a release and run the same validation and canary"
assessment_questions:
  - "What is your deterministic defect rate across 100% of the last cohort?"
  - "Do you review a random sample of records that passed every automated check?"
  - "Is QA run on the rendered, delivered artifact or on stored text?"
  - "Which model judges your outputs, and how often does it agree with human reviewers?"
  - "Can any automated process change production prompts, guidelines, or rules without human approval?"
patterns: [Accuracy Stack, Pilot Cohort Protocol, Three Validation Sets, Shared Choke Point, Specificity Ladder, Trend Ledger]
anti_patterns: [The Self-Graded Exam, The Prompt Paragraph Pile, The Demo-Account Sign-off, Fixes Tuned on the Failures Alone]
maturity_dimension: governance_and_reliability
```

## References

1. Madaan, A., et al. (2023). "Self-Refine: Iterative Refinement with Self-Feedback." NeurIPS 2023. https://arxiv.org/abs/2303.17651
2. Huang, J., et al. (2024). "Large Language Models Cannot Self-Correct Reasoning Yet." ICLR 2024. https://arxiv.org/abs/2310.01798
3. Kamoi, R., Zhang, Y., Zhang, N., Han, J., Zhang, R. (2024). "When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs." TACL. https://arxiv.org/abs/2406.01297
4. Dhuliawala, S., et al. (2023). "Chain-of-Verification Reduces Hallucination in Large Language Models." https://arxiv.org/abs/2309.11495
5. Manakul, P., Liusie, A., Gales, M. (2023). "SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection." EMNLP 2023. https://arxiv.org/abs/2303.08896
6. Zheng, L., et al. (2023). "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS 2023 Datasets and Benchmarks. https://arxiv.org/abs/2306.05685
7. Wang, P., et al. (2023). "Large Language Models are not Fair Evaluators." https://arxiv.org/abs/2305.17926
8. Panickssery, A., Bowman, S. R., Feng, S. (2024). "LLM Evaluators Recognize and Favor Their Own Generations." https://arxiv.org/abs/2404.13076
9. Bai, Y., et al. (2022). "Constitutional AI: Harmlessness from AI Feedback." https://arxiv.org/abs/2212.08073
10. Min, S., et al. (2023). "FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation." EMNLP 2023. https://arxiv.org/abs/2305.14251
11. Vectara. Hallucination Leaderboard (snapshot as of September 2026). https://github.com/vectara/hallucination-leaderboard
12. OpenAI (2024-08-06). "Introducing Structured Outputs in the API." https://openai.com/index/introducing-structured-outputs-in-the-api/
13. Anthropic. "Structured outputs" documentation (accessed September 2026). https://platform.claude.com/docs/en/build-with-claude/structured-outputs
14. Google. *The Site Reliability Workbook*, Chapter 18, "Canarying Releases." https://sre.google/workbook/canarying-releases/
15. ITV News (2024-01-19). "DPD disables AI chatbot after customer service bot appears to go rogue." https://www.itv.com/news/2024-01-19/dpd-disables-ai-chatbot-after-customer-service-bot-appears-to-go-rogue
16. Taheri, H. "How I Build an Enterprise Personalization Engine That Can Be Trusted at Scale." hamedtaheri.com, 2026-08-23. https://hamedtaheri.com/articles/building-enterprise-personalization-engine
17. Taheri, H. "Enterprise-grade accurate personalization at scale." hamedtaheri.com, 2026-07-13. https://hamedtaheri.com/articles/accurate-personalization-at-scale
18. Taheri, H. "Adversarial Governance Compliance: Our Methodology and What Near-Perfect Accuracy Tells Us." hamedtaheri.com, 2026-03-14. https://hamedtaheri.com/articles/adversarial-governance-compliance
19. Taheri, H. (2026). "Governed Memory: A Production Architecture for Multi-Agent Workflows." arXiv:2603.17787. https://arxiv.org/abs/2603.17787

---

# 21. Did It Work?

*The question: How do we prove personalization caused the result?*

> **Questions this chapter answers**
> - Our dashboards say personalization is working. How do we know it caused anything?
> - Which metrics lie, and how do we pick ones that do not?
> - Do we really need to hold customers back from the good version?
> - How long before we know whether a change helped or hurt?
> - Our system learns from customer behavior. Can that make it worse?

## The short answer

Personalization is a targeting decision, and targeting decisions are very good at finding people who were going to act anyway. That is why the default report (people who got the personalized message converted more) proves almost nothing. The only defensible answer to "did it work?" is a comparison with a counterfactual: similar people, chosen at random, who did not get the treatment.

Teams that know, rather than hope, do four things. They measure outcomes, not proxies. They keep holdouts, including a small permanent one, so "compared to what?" always has an answer. They measure uplift (the change they caused) instead of response (the outcome they predicted). And they wait long enough to see slow effects, because some short-term wins are long-term losses.

Learning systems add one more trap: they train on behavior they caused, and can grow more confident and less useful at once.

The leader's test is simple. Ask for the holdout. If there is none, you do not yet know whether your personalization works; you know that it runs.

---

Larkspur Systems (a fictional composite company used throughout this handbook) put a model in charge of its nurture emails in the spring, writing subject lines per contact from product usage and role. Within six weeks the open rate had climbed from the low twenties to the mid thirties, and the marketing team presented the chart at the quarterly business review.

Two quarters later the sales team raised a different chart. Meetings sourced from nurture email were up, but the share that became qualified opportunities had fallen. More people were clicking "book a call"; fewer had a real reason to buy.

Nothing was broken; the model did what it was rewarded for. Part of the open-rate gain was not even human: Apple's Mail Privacy Protection downloads remote content, including tracking pixels, in the background whether or not the person reads the message, so some "opens" were machines [1]. The rest was curiosity, bought by subject lines that promised more than the email delivered. The metric had stopped measuring what it was chosen to measure.

That is the Proxy Trap, and every personalization program walks into it at least once.

## An open rate is not an outcome

A proxy is a fast metric you hope moves with the outcome. Once a system is optimized against it, the relationship that made it a good proxy starts to break.

The economist Charles Goodhart put it in 1975: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." The line most people quote, "When a measure becomes a target, it ceases to be a good measure," is the anthropologist Marilyn Strathern's 1997 paraphrase, not Goodhart's own wording [2].

YouTube lived this at scale. In August 2012 the company explained that its discovery features "were previously designed to drive views. This rewarded videos that were successful at attracting clicks, rather than the videos that actually kept viewers engaged," and it moved to watch time [3]. Its 2016 recommender paper made the same point in engineering terms: ranking by click-through rate promotes deceptive videos the user does not finish, so the ranker predicts expected watch time instead [4]. Watch time is a proxy too. There is no perfect metric; choosing the proxy is a design decision that deserves the same scrutiny as the model.

Weights become policy. Facebook's 2018 "meaningful social interactions" metric, as internal documents reported by the *Wall Street Journal* described it, gave a like 1 point, a reaction or plain reshare 5, and a significant comment or reshare 30 [5]. Outrage provokes long comment threads. Nobody wrote "amplify anger" into the objective; the weighting did it. The metric, not the intention, decides what a system learns to do.

A metric you optimize is a metric you have stopped observing.

## The only question is: compared to what?

Here is the sentence this chapter turns on: **personalization selects people, and the people it selects were often going to act anyway.**

A propensity model picks the contacts most likely to renew, and you send them a personalized message. They renew at 80 percent; everyone else at 55. The message may have done nothing. You chose the renewers; you did not create them.

The only way to separate selection from effect is a counterfactual: a randomly chosen group who were eligible and did not receive the treatment. Randomization is the one method that balances everything you did not think to measure.

It also protects you from intuition, including senior intuition. At Amazon, Greg Linden built a prototype that recommended items in the shopping cart. A marketing senior vice-president "was dead set against it," arguing it would distract people from checking out, and Linden was told to stop. He ran a controlled experiment anyway, and the "feature won by such a wide margin that not having it live was costing Amazon a noticeable chunk of change" [6]. The reverse is more common. Kohavi and colleagues report that most successful changes at large web companies move key metrics by fractions of a percent, and that at Bing only a minority of ideas improved the metrics they targeted [7][8]. You cannot tell which ideas work from the meeting. And when a result looks spectacular, apply Twyman's law: any figure that looks interesting or different is usually wrong [7]. A 40 percent lift from a subject line is more likely a logging bug or bot traffic than a discovery.

## Hold some people back, permanently

A holdout is a random group excluded from a treatment so its outcomes stand in for "what would have happened." A mature program runs two kinds. A **per-decision holdout** asks a narrow question (does this renewal message beat the generic one, or beat sending nothing?) and ends with the test.

A **global holdout** answers the board's question: across everything we personalize, what did personalization add? A small share of customers, a few percent, receives the generic experience on every personalized surface for a long period. It is the only way to see the combined effect of fifty launches that each "won" their own test, including wins that cannibalize each other or add up to fatigue.

Netflix gave a clear public example of stating value as a counterfactual. In a 2025 paper, its researchers compared the recommender with simpler alternatives: replacing it with matrix factorization reduced engagement by about 4 percent, and replacing it with popularity-based recommendations reduced it by about 12 percent [9]. That is far smaller and more believable than the "more than $1 billion a year" figure the company offered in 2015, which was a retention-value self-estimate, not a measured revenue effect [10].

Holdouts cost something; show the cost as a range. If personalization lifts a decision's value by L and you hold back h of the population, the forgone value is roughly h × L × (the value flowing through that decision). With a 5 percent holdout and a true lift somewhere between 2 and 10 percent, you forgo between 0.1 and 0.5 percent of that decision's value. That is the price of knowing. The price of not knowing is running a program for years that may do nothing, or harm.

Size it with arithmetic, not instinct. Detecting a move in conversion from 2.0 to 2.4 percent (a 20 percent relative lift) with the standard 80 percent power and 5 percent significance needs roughly 21,000 units in each group. Larkspur has 40,000 accounts. If the outcome is an account-level opportunity, a 90/10 split leaves 4,000 in the holdout, too few for an effect that size. The honest options: measure a more frequent outcome, pool quarters, accept that only larger effects are detectable, or say the decision cannot yet be proven.

The holdout is the only customer who tells you the truth.

## Measure the change you caused, not the outcome you predicted

Holdouts make a better targeting question possible: not "who is most likely to convert?" but "whose behavior does this action change?" Uplift modeling estimates that difference per person or segment. Radcliffe and Surry's four groups frame it [11]. **Persuadables** act only if treated. **Sure Things** act either way. **Lost Causes** do not act either way. **Do-Not-Disturbs** (sometimes called Sleeping Dogs) are less likely to act if treated. A response model targets the first two groups indiscriminately and cannot see the fourth.

![A two-by-two grid sorts people by whether they act if treated and whether they would act if not treated. Persuadables act only if treated; Sure Things act either way; Lost Causes do not act either way; Do-Not-Disturbs are less likely to act if treated, so the best action for them is not to send. A response model targets both Persuadables and Sure Things, while uplift modeling targets Persuadables and needs randomized holdout data, the only way to see Do-Not-Disturbs.](/images/handbook/ch19-four-uplift-groups.svg)

*Figure 21.1. Only Persuadables are changed by the action; a response model cannot tell them from Sure Things or see Do-Not-Disturbs.*

The fourth group is real. Retention activity can drive away some of the customers it targets [12], and Ascarza's field experiments found that targeting customers at highest churn risk was less effective than targeting those most sensitive to the intervention; they are not the same people [13]. Uplift models need randomized data to train on, one more reason holdouts are not optional. For Do-Not-Disturbs, not sending is the best action, and only a counterfactual can show it. Chapter 14 treats doing nothing as a decision; this is how you prove it was the right one.

## Bandits learn faster and prove less

A bandit shifts traffic toward whichever option is winning while the test runs; a contextual bandit does it per person. Li and colleagues at Yahoo! showed the approach on the front-page news module, reporting a 12.5 percent click lift over a context-free bandit across more than 33 million events [14].

Bandits suit options that are many, short-lived, and cheap to be wrong about: subject lines, content blocks, send times. They are the wrong tool for proving a program's value, because they stop exploring what looks worse, and proof needs a stable comparison. They also optimize whatever reward you give them, which is the Proxy Trap at machine speed. So: bandits choose among variants inside a decision; holdouts measure whether the decision is worth making. Keep an exploration floor, a small share of traffic that is always randomized.

Generation makes the gap wider. In an illustrative campaign, 4 personas, 6 industries, 5 regions, 8 message themes, and 3 offers give 2,880 distinguishable contexts, and ten page zones with five eligible options each give 5^10, about 9.8 million possible compositions per context. That is expressive capacity, not experiment volume: at the roughly 21,000 people per arm the power calculation above needed, you cannot test one context's variants. Form a few strong hypotheses instead.

## Short-term wins can be long-term losses

Most experiments run for days or weeks. Some effects take months, because people learn. At Google, Hohnhold, O'Brien, and Tang built experiments to measure "ads blindness," users changing their inherent propensity to click on ads based on the quality of ads they had seen [15]. Showing more ads raised revenue in a short test. Over months, users learned to ignore them. The team reduced the ad load on Google's mobile search interface by 50 percent over two years. It was, in the paper's words, "a substantially short-term revenue negative change," yet "the long-term revenue impact was shown to be neutral," with a clear improvement in user experience [15].

Personalized outreach has the same shape: frequency that lifts this month's replies can train contacts to ignore the sender next quarter. Long-running holdouts observe slow effects directly. Surrogate indices predict them: Athey, Chetty, Imbens, and Kang showed how to combine several short-term outcomes into an index that predicts the long-term one, estimating a job-training program's nine-year employment effect from the first six quarters of data [16]. Validate fast metrics against the slow outcome once, carefully, then use them with a known error bar.

## The system trains on the behavior it caused

A personalization system shows people things, observes their response, and retrains on it. Items it never showed generate no clicks, so it concludes they are unwanted. Chaney, Stewart, and Engelhardt simulated this loop and found that recommenders trained on data from users already exposed to recommendations increase homogeneity without increasing utility [17]. The system becomes more certain about a narrower world.

The corrections are mechanical: log what was shown and its probability of being shown, so analysis can reweight; train partly on exploration and holdout data the system did not choose; track coverage alongside accuracy; treat rising model confidence with flat outcome lift as a warning.

![A loop of three steps: the system shows what it chose, observes clicks, and retrains on those clicks; items never shown get no clicks and are read as unwanted. The warning sign is model confidence rising and variety shown narrowing while outcome lift stays flat. Four corrections break the loop: an exploration floor that always randomizes a share, propensity logging of shown items and their odds, reweighted training that includes data the system did not pick, and tracking coverage alongside accuracy.](/images/handbook/ch19-feedback-echo.svg)

*Figure 21.2. The Feedback Echo: a system trained on behavior it caused grows more confident about a narrower world.*

## Measure the layers separately

A business outcome is the product of several systems: identity, memory, retrieval, decision, generation, delivery. A single lift number cannot tell you which layer moved it, so measure each against its own standard. Retrieval: did the agent get the facts it needed? Decision: would a good operator have chosen it, including "wait"? Generation: were claims specific and true, or honestly general (Chapter 20)? Delivery: did it arrive once, on time? For pages, screens, and notifications, add the experience itself: task completion, time on task, and notification disables (Chapter 16). Only then: did the outcome move against the holdout?

The glue is the decision trace: for every action, what was known, which policy applied, what was chosen and why, what was sent (and from which template and guideline version), and what happened. With it, you can find that a "losing" variant lost because many of its messages fell back to generic copy on stale data.

The campaign report needs the same columns. In the engine I built, a trend ledger already tracks deterministic and semantic error rates, sample size, and verdict per version, so the question becomes "are we improving or regressing?" rather than "is today's sample fine?". The columns I would add next are the ones that explain an experiment: delivery rate, fallback rate, recovery rate, active template and guideline version, and conversion by context. Without them, a comparison cannot tell a weaker message from one that mostly never ran. And a count you could not read must show as unknown, not zero.

Chapter 20 describes the two learning loops I run in these engines, today as a human-supervised operator workflow [18]. The quality loop defines what is allowed; the performance loop (hypothesis, controlled test, measurement) optimizes inside it. An experiment can change strategy, never a factual guard. When a finding is classified, only one class, a performance hypothesis, goes to an experiment. A factual defect gets a guard and a regression test; a delivery defect gets retry and recovery. You do not A/B test whether a page may claim a contract the customer never signed.

Keep the two kinds of evidence apart. The ready-to-promote report that workflow is being extended to produce looks like this (illustrative numbers, not a result):

```text
Cohort:      83 leads; 11 with unsupported relationship language
Proposal:    1 guideline update, 1 deterministic guard, 6 regression examples
Replay:      11/11 fixed
Held-out:    29/30 pass
Regression:  40/40 pass
```

Every line says the fix is correct and broke nothing. None says one more customer responded. A validation report answers "is it safe to ship?"; only a counterfactual answers "did it work?".

### The Measurement Ladder

Each rung depends on the one below:

1. **Reporting.** Counts and rates of what happened. Necessary, never proof.
2. **A/B tests.** Randomized comparison of variants for one decision.
3. **Holdouts and incrementality.** Personalized versus generic versus nothing, per decision and globally.
4. **Uplift.** Targeting on estimated treatment effect per person.
5. **Continuous learning.** Bandits and retraining inside guardrails, watched by a global holdout.

Most programs claim rung 5 while standing on rung 1.

![A staircase of five rungs, each resting on the one below: rung 1, reporting (counts and rates, never proof); rung 2, A/B tests (randomized variants for one decision); rung 3, holdouts and incrementality (personalized versus generic versus nothing); rung 4, uplift (targeting on estimated effect per person); rung 5, continuous learning (bandits and retraining watched by a global holdout). Rung 5 is marked as where most programs claim to be, and rung 1 as where most stand.](/images/handbook/ch19-measurement-ladder.svg)

*Figure 21.3. The Measurement Ladder: each rung depends on the one below, and most programs claim the top while standing on the bottom.*

### The KPI tree and the proxy checklist

Before launch, draw a tree: the root is the outcome you care about (qualified pipeline, retention, cost to serve), branches are drivers, leaves are fast metrics, and every leaf names the branch it should move. Then ask of every proxy: Can the system move it without moving the outcome? Can machines or privacy features inflate it? Has it been validated in a holdout? Which guardrail (unsubscribes, complaints, qualification rate) catches gaming? When is it re-validated?

## What this does not do

Experiments tell you *whether*, not *why*; decision traces and cohort review fill that gap.

Randomization is not always available. Some decisions are too rare, some effects too slow, and in B2B the unit that matters (the account) is small in number. When the arithmetic says a decision cannot be proven, decide on judgment and label it as judgment. That is more honest than a p-value from an underpowered test.

I disagree with two popular shortcuts. The first is benchmark ROI. McKinsey's widely cited 2021 finding that personalization most often drives a 10 to 15 percent revenue lift [19] is a cross-company aggregate, not a randomized comparison of your program. It says where companies believe value lies. It is not evidence that yours works. The second is attribution dashboards that assign credit to every touch a buyer saw. They describe paths, not what would have happened without a touch.

Synthetic panels need their own caution. LLM personas asked how customers would react are useful as a pre-test for defects: confusing copy, a claim that reads as intrusive, a tone that misses a role. As a measure of lift they are not. When Verasight built personas from 1,500 real respondents and compared simulated with actual answers, average error ranged from 4 to 23 percentage points depending on the model, with weak subgroup estimates [20]. Peer-reviewed work finds synthetic samples show artificially low variance, which makes wrong answers look certain [21]. A rehearsal can catch mistakes. It cannot replace the customer. Chapter 3 reaches the same verdict for UX research (synthetic users draft; people validate), and adds the other half: a holdout proves what moved, and only qualitative research with real people finds why.

Finally, offline gains are not business gains: Booking.com and YouTube both reported offline model improvements that did not carry into live tests [22][4]. Ship to a holdout, not to a leaderboard.

## At scale

With many teams running hundreds of personalized decisions, measurement becomes infrastructure.

**Randomize at the unit of influence** (see Patterns): contacts at one account talk to each other.

**Coordinate experiments.** Concurrent tests interact when several teams message the same person. One assignment log, one global holdout, and one contact policy stop fifty teams from each claiming the same renewal.

**Protect the global holdout.** Every quarter someone will ask to release it. Its cost is visible and its value invisible, which is why it gets cut. Put its size and end date in writing.

**Make traces queryable.** At millions of decisions, the trace answers "what happened to customers like this one?" in minutes, not in a data-science project.

## Failure story: The Feedback Echo

Larkspur's content recommender picked the next nurture asset for each contact: guide, webinar, case study, product tour. Webinars had the highest click rate at launch, so the model showed more webinars, which produced more webinar clicks, which became the next training set. Within two quarters most contacts saw little else, and the model's offline accuracy rose every month.

An analyst compared it with a small random-exploration slice someone had left on by accident. Contacts in that slice saw a mix and were more likely to reach a product tour, the asset that best predicted a qualified opportunity. The model had not learned what contacts wanted. It had learned what it had been showing them.

The fix was ordinary: a permanent exploration floor, propensity logging, exposure-weighted training, and qualified opportunities in place of clicks. The lesson was not. A system that learns only from its own choices will eventually be very sure of very little.

## Patterns

**Permanent Global Holdout.** *Problem:* many decisions each "win," but nobody can state the combined effect. *Forces:* holdouts cost visible value; leaders need a program-level answer. *Solution:* a small random share of customers gets the generic experience everywhere for a fixed, long period, with documented size and end date. *Tradeoffs:* forgone value (h × L); pressure to release it; needs volume to be powered.

**Randomize at the Unit of Influence.** *Problem:* treatment leaks between people who talk to each other, contaminating the holdout. *Forces:* contact-level tests have more units; account-level tests are cleaner. *Solution:* assign treatment where influence travels (account, household, buying group). *Tradeoffs:* fewer units, lower power: the honest price of a valid answer.

**Quality Before Performance.** *Problem:* optimization erodes accuracy or trust. *Forces:* performance gains show this week; trust losses surface later. *Solution:* separate loops with separate authority; correctness and policy define the feasible region, and optimization happens inside it. *Tradeoffs:* slower iteration on bolder variants.

## Leader questions

1. Where is our holdout, how big is it, and when was it last read?
2. What outcome is our headline metric a proxy for, and has anyone validated the link?
3. Are we targeting people likely to act, or people whose behavior our action changes?
4. Which of our wins could be long-term losses?
5. What does our system train on, and how much of it did the system not choose?

## Build checklist

- [ ] KPI tree drawn, with every proxy linked to an outcome.
- [ ] Randomized assignment logged per decision, at the unit of influence.
- [ ] Per-decision holdouts (with a "send nothing" arm where plausible) and a documented global holdout.
- [ ] Power calculation before each test; underpowered decisions labeled as judgment.
- [ ] Guardrail metrics on every test; machine-generated opens excluded from outcomes.
- [ ] Exploration floor and propensity logging for every learning component.
- [ ] Decision traces for every action: known, policy, chosen, sent (with template and guideline version), outcome.
- [ ] Quality and performance loops with separate authority; only performance hypotheses become experiments.

## Metrics to watch

- **Incremental lift versus holdout** on the root outcome, per decision and globally, with confidence intervals.
- **Proxy-outcome agreement:** how often a proxy win is also an outcome win.
- **Exploration share and catalog coverage** for every learning component.
- **Long-term delta:** lift at 90 days or more compared with lift at launch.
- **Delivery and fallback rate per arm**, so a losing arm can be told apart from an arm that mostly did not run.

## Reader Q&A

**We are B2B with a few thousand accounts. Can we test at all?**
Yes, within limits. Test frequent outcomes (replies, meetings), pool across quarters, accept that only large effects on rare outcomes are provable, and label judgment calls.

**Is it ethical to withhold personalization from a holdout?**
The holdout gets your previous standard experience. Until you measure, you do not know the personalized version is better; sometimes it is not.

**Can we use bandits instead of A/B tests?**
For choosing among variants, often. For proving that a program is worth its cost, no. Use both, and keep a holdout the bandit cannot touch.

**How long should a test run?**
At least one full business cycle, longer for anything that could cause learning or fatigue. Decide the duration before launch, not when the chart looks good.

## For your AI

```yaml
chapter: 21
concepts:
  - name: Counterfactual
    definition: "What would have happened to the same kind of people without the treatment, estimated by a randomized control group."
  - name: Holdout
    definition: "A randomly selected group excluded from a treatment so its outcomes stand in for the counterfactual; per-decision or global."
  - name: Global Holdout
    definition: "A small, long-running random share of customers who receive the generic experience across all personalized surfaces."
  - name: Uplift
    definition: "The difference in outcome caused by treatment for a person or segment; distinct from the probability of responding."
  - name: Proxy Trap
    definition: "Optimizing a fast metric until it moves without the outcome it was chosen to predict."
  - name: Feedback Echo
    definition: "A learning system trained on behavior it caused, growing more confident about a narrower set of options."
  - name: Measurement Ladder
    definition: "Reporting, A/B tests, holdouts and incrementality, uplift, continuous learning; each rung depends on the one below."
decision_rules:
  - if: "a personalization result is reported without a randomized comparison group"
    then: "treat it as reporting, not evidence of effect"
  - if: "a metric is optimized by the system"
    then: "validate it against the root outcome in a holdout and add a guardrail metric"
  - if: "people in the same account or household influence each other"
    then: "randomize at the account or household level"
  - if: "a power calculation shows the effect cannot be detected at available volume"
    then: "use a more frequent outcome, pool time, or label the decision as judgment"
  - if: "a learning component trains on its own served outcomes"
    then: "keep an exploration floor, log serving propensities, and reweight training data"
  - if: "a change improves performance metrics but weakens a factual or policy guard"
    then: "reject it; quality defines the feasible region"
  - if: "a finding is a defect rather than a performance hypothesis"
    then: "fix it with a guard, guideline, or recovery and a regression test; do not run it as an experiment"
  - if: "a change passes replay, held-out, and regression validation"
    then: "treat it as safe to ship, not as evidence of lift; measure effect against a holdout"
assessment_questions:
  - "Do you have a holdout for your main personalized decisions, and a global one?"
  - "What is your headline personalization metric, and what outcome is it meant to predict?"
  - "Do you target on likelihood to respond or on estimated uplift?"
  - "At what unit (contact, account, household) do you randomize?"
  - "What share of your recommender or bandit traffic is randomized exploration?"
  - "Can you trace any single action to what was known, the policy applied, and the result?"
patterns: [Permanent Global Holdout, Randomize at the Unit of Influence, Quality Before Performance, Measurement Ladder, KPI Tree, Proxy Risk Checklist]
anti_patterns: [The Proxy Trap, The Feedback Echo, Response-as-Uplift, Survey ROI as Evidence, Offline Wins as Business Wins, Validation as Lift]
maturity_dimension: learning
```

## References

1. Apple, "Mail Privacy Protection & Privacy" and "Use Mail Privacy Protection on iPhone." https://www.apple.com/legal/privacy/data/en/mail-privacy-protection/ ; https://support.apple.com/guide/iphone/use-mail-privacy-protection-iphf084865c7/ios (accessed 2026-09-26).
2. Goodhart, C. (1975), "Problems of Monetary Management: The U.K. Experience"; Strathern, M. (1997), "'Improving ratings': audit in the British University system," *European Review* 5(3).
3. YouTube Official Blog (2012-08-10), "Why We Focus on Watch Time." https://blog.youtube/news-and-events/youtube-now-why-we-focus-on-watch-time/
4. Covington, P., Adams, J., Sargin, E. (2016), "Deep Neural Networks for YouTube Recommendations," RecSys 2016.
5. Hagey, K., Horwitz, J. (2021-09-15), "Facebook Tried to Make Its Platform a Healthier Place. It Got Angrier Instead," *Wall Street Journal*. House reprint: https://docs.house.gov/meetings/IF/IF16/20211201/114268/HHRG-117-IF16-20211201-SD012.pdf
6. Kohavi, R., Longbotham, R. (2007), "Online Experiments: Lessons Learned," *IEEE Computer* 40(9). https://ai.stanford.edu/~ronnyk/2007IEEEComputerOnlineExperiments.pdf ; Linden, G. (2006), "Early Amazon: Shopping cart recommendations." https://glinden.blogspot.com/2006/04/early-amazon-shopping-cart.html
7. Kohavi, R., Tang, D., Xu, Y. (2020), *Trustworthy Online Controlled Experiments*, Cambridge University Press. https://experimentguide.com/
8. Kohavi, R. et al. (2014), "Seven Rules of Thumb for Web Site Experimenters," KDD 2014. https://exp-platform.com/Documents/2014-08-27ExperimentersRulesOfthumbKDD.pdf
9. Zielnicki, K. et al. (2025, rev. 2026), "The Value of Personalized Recommendations: Evidence from Netflix," arXiv:2511.07280. https://arxiv.org/abs/2511.07280
10. Gomez-Uribe, C., Hunt, N. (2015), "The Netflix Recommender System," *ACM TMIS* 6(4). https://dl.acm.org/doi/10.1145/2843948
11. Radcliffe, N., Surry, P. (2011), "Real-World Uplift Modelling with Significance-Based Uplift Trees." https://www.research.ed.ac.uk/en/publications/real-world-uplift-modelling-with-significance-based-uplift-trees/
12. Radcliffe, N., Simpson, R. (2008), "Identifying who can be saved and who will be driven away by retention activity," *J. Telecommunications Management* 1(2). https://stochasticsolutions.com/pdf/SavedAndDrivenAway.pdf
13. Ascarza, E. (2018), "Retention Futility: Targeting High-Risk Customers Might Be Ineffective," *Journal of Marketing Research* 55(1). https://journals.sagepub.com/doi/10.1509/jmr.16.0163
14. Li, L., Chu, W., Langford, J., Schapire, R. (2010), "A Contextual-Bandit Approach to Personalized News Article Recommendation," WWW 2010. https://arxiv.org/abs/1003.0146
15. Hohnhold, H., O'Brien, D., Tang, D. (2015), "Focusing on the Long-term: It's Good for Users and Business," KDD 2015. https://research.google.com/pubs/archive/43887.pdf
16. Athey, S., Chetty, R., Imbens, G., Kang, H. "The Surrogate Index: Combining Short-Term Proxies to Estimate Long-Term Treatment Effects More Rapidly and Precisely," NBER w26463. https://www.nber.org/papers/w26463
17. Chaney, A., Stewart, B., Engelhardt, B. (2018), "How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility," RecSys 2018. https://arxiv.org/abs/1710.11214
18. Taheri, H., "Building an Enterprise Personalization Engine," hamedtaheri.com (sections "Build two learning loops, not one" and "Validate every improvement against three sets").
19. McKinsey (2021-11-12), "The value of getting personalization right, or wrong, is multiplying." https://www.mckinsey.com/capabilities/growth-marketing-and-sales/our-insights/the-value-of-getting-personalization-right-or-wrong-is-multiplying
20. Verasight (2025), "Your Polls on ChatGPT." https://www.verasight.io/reports/synthetic-sampling
21. Bisbee, J. et al. (2024), "Synthetic Replacements for Human Survey Data? The Perils of Large Language Models," *Political Analysis*. https://www.cambridge.org/core/journals/political-analysis/article/synthetic-replacements-for-human-survey-data-the-perils-of-large-language-models/B92267DC26195C7F36E63EA04A47D2FE
22. Bernardi, L., Mavridis, T., Estevez, P. (2019), "150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com," KDD 2019. https://dl.acm.org/doi/10.1145/3292500.3330744

---

# 22. Deploying It

*The question: How does a company actually build and roll this out?*

> **Questions this chapter answers**
> - What do we build first, and in what order?
> - Should we build, buy, or own the personalization layer?
> - What team does this take, and who owns what?
> - What does the first 90 days look like, and how do we know we are ready to widen?
> - How do we tell where we are, and what to improve next, without chasing the newest tool?

## The short answer

A personalization engine is not a model with a prompt. It is a production system in which the model is one function among many: intake, identity, governed context, planning, generation, validation, channel compilation, delivery, recovery, quality review, and governance. Most of the deployment effort goes into the parts around the model, and most first-week failures happen there too.

So deploy it in the order the failures arrive. Write the contract before touching data: what the experience must do, which claims are allowed, what a fallback looks like, which rules code enforces. Build the shared mechanics once (identity, trust rules, idempotency, choke-point protections, workflow state) so every later campaign inherits them. Run a real but small first cohort as part of the build, not after it. Widen only when the release thresholds you wrote down in advance say so, and let automation propose improvements while people keep the right to ship them.

On build versus buy: rent what is commodity, own what compounds. The memory, the routing decisions, and the reviewed examples get more valuable with use, and whoever holds that layer holds the economics.

For a leader, three questions decide most of the outcome. Is there a written contract for the first decision? Can the team say, for any record, what state it is in and what was delivered? Will the second campaign cost clearly less to launch than the first? If not, you have built a pilot, not a platform.

The phases, in order: contract, foundations, shadow and first cohort, supervised expansion, a second campaign on the same mechanics. Only then does turning the engine into a product begin.

## What broke first

*Larkspur Systems is a fictional composite company used throughout this handbook. The failures below are composites of failure classes documented in my published engineering write-up [1]; the counts are illustrative.*

Larkspur switched on its engine for renewal season on a Monday. After the inbound fast win (Chapter 6), this was the engine's first renewal campaign: an email, a landing page, and a brief for the account manager, for each fleet-operations contact whose contract renewed within 120 days. Chapter 20 tells what the content review found. This is what the operations dashboard found, and it found it first.

By Monday afternoon, contacts who had signed up with personal email addresses were receiving pages that described "operations at your scale" using the headcount of their email provider. The domain-to-company step had trusted a free-mail domain as an employer.

By Tuesday, one research provider had exhausted its credits. Nothing stopped it, so every record sent it the same failing request, waited for the timeout, and moved on with thinner context. Quality dropped all day without an error anyone was watching.

By Wednesday, the CRM showed every record as "complete." A reconciliation script found that a few dozen landing pages had been generated, validated, and stored, but never received a public URL. The callback had reported success because the callback succeeded. The artifact the customer needed did not exist.

None of these was a model failure. The model wrote well all week. The failures lived in identity, dependency handling, and delivery state: the parts of a deployment that demos never exercise. Each has a standing guardrail in the engine I built: free-mail domains suppress firmographics, an exhausted provider trips a circuit breaker, and a missing page is regenerated alone, with the score and research untouched.

## The model is the smallest part of the deployment

In 2015 a group of Google engineers argued that machine learning's quick wins are not free: real-world ML systems accumulate "massive ongoing maintenance costs" through entanglement, hidden feedback loops, and data dependencies [2]. The model is a small box inside a large system. Generative personalization makes the box more capable and the system around it more important, because the box now writes what customers read.

Here is the architecture I use, top to bottom [1]: the **campaign contract**; **input validation and tenant routing**; **identity resolution**; **research and first-party context**; **data trust and provenance**; the **governed customer context** (facts with source, confidence, recency, and allowed surfaces); a **content plan** that splits the experience into typed zones and picks the personalization tier; **generation**, **self-review**, and **deterministic validation**; **channel compilation** for email, web, seller playbook, paid, print, and events; **delivery state**, **retry**, and **targeted recovery**; and finally **population QA** and **human-supervised optimization**, which feed the next version.

![The reference architecture drawn as one flow of twelve steps. It runs from the campaign contract through input validation and tenant routing, identity resolution, research and first-party context, data trust and provenance, the governed customer context, and a content plan of typed zones and tier. Only then comes the model step, generation and self-review, highlighted as one box among many. After it come deterministic validation, channel compilation, delivery state with retry and recovery, and population QA with supervised optimization, which loops back to feed the next version of the contract.](/images/handbook/ch20-reference-architecture.svg)

*Figure 22.1. The model is one step in the middle; everything before and after it decides whether its output may reach a person.*

Each layer exists because a failure there is different from a failure anywhere else. A wrong company match is not a writing problem. A missing page URL is not a research problem. If the system cannot tell these classes apart, every defect becomes another sentence in the prompt, and that does not scale.

In code, the model is one call in the middle. Identity is resolved first, trust policy filters context before the model sees it, the tier is a policy output, deterministic validation has the final word on hard rules, and fallback is an explicit state [1]. Chapters 9, 10, 15, and 20 build those pieces. This chapter puts them into production in the right order.

Here is the sentence this chapter turns on: **the model is a function call; the deployment is everything that decides whether its output is allowed to reach a person.**

## Start with the contract, not the customer record

Before touching data, I write the contract. It turns "personalize this" into something testable: who the audience is, which surfaces are active, what level of personalization is expected, which capabilities and offers are approved, which claims are prohibited, which facts need corroboration, what a valid fallback looks like, and which rules are advisory versus enforced in code [1].

Where does the contract come from? In the engine I built, it starts as a customer intake: the ICP, positioning, the economic story, the claims allowed and their sources, banned language, examples, legal copy, the CTA, and brand behavior. The intake becomes an approved guideline, the guideline a content-zone schema, the schema a template and channel configuration, and only then does anything get generated. That chain runs today as an operating process; turning it into a standardized, repeatable workflow is still in progress.

The contract is the only thing QA can judge against. A company-level page that never names an individual is correct for one campaign and a defect for another. Without a contract, the release decision falls back on a vague sense of good writing, and vague senses do not produce GO or NO GO.

Make it machine-readable where possible (surfaces, tier, prohibited claim classes, length limits, risk tier, fallback policy) and let a prose guideline carry the rest. Put the release thresholds in it too, written before the first record is generated, so the team cannot talk itself into a launch.

## Shared mechanics are platform; campaign judgment is configuration

The decision that most determines what the second campaign costs is where each requirement lives [1].

**Shared mechanics** behave the same everywhere: sanitizing malformed output, escaping HTML, normalizing identifiers, idempotency, classifying retryable failures, hard length limits, stripping internal metadata, blocking unsafe URL schemes, preventing cross-tenant routing, enforcing wire schemas. They live in code, at shared choke points, and get stronger over time.

**Campaign judgment** belongs in versioned guidelines and configuration: voice, approved capabilities, persona priorities, CTA strategy, recency windows, examples, proof hierarchy.

The reason is blast radius. A campaign wanting a more technical tone should not need a global engine deploy; a rule every campaign needs should not depend on twenty prompt authors remembering. For every requirement, ask: *what kind of control is this?* Models reason, guidelines express judgment, code enforces invariants.

![Three campaigns sit on top of one shared platform. Each campaign carries its own judgment in versioned guidelines and configuration: voice, approved capabilities, persona priorities, CTA strategy, recency windows, examples, and proof hierarchy, changed per campaign without a global engine deploy. Every campaign inherits the same shared mechanics, enforced in code at shared choke points: sanitizing output, escaping HTML, normalizing identifiers, idempotency, retry classification, hard length limits, stripping internal fields, blocking unsafe URLs, tenant routing, and wire schemas. Models reason, guidelines express judgment, code enforces invariants.](/images/handbook/ch20-shared-mechanics-campaign-judgment.svg)

*Figure 22.2. Put campaign intent in configuration and invariants in shared code, so each new campaign inherits the protections instead of forking the engine.*

The cheapest reliability investment in the deployment: find the last shared function before each important boundary (renderer, customer serializer, URL parser, tenant check) and make it protective. New campaigns inherit the protection without anyone remembering to add it.

## Delivery is half the job

A model call completing is not an experience succeeding. The engine needs explicit workflow state for every record: accepted, context ready, generated, validated, persisted, rendered, delivered, confirmed [1]. With that state, Larkspur's missing pages are a rendering defect found by a query, not a mystery found by a customer.

The state keeps five kinds of success apart, because each fails on its own: **generation** (did a step produce output?), **content** (is it complete and safe?), **persistence** (can the next step read it yet?), **delivery** (did the destination get the right artifact?), and **campaign** (does the population behave as the contract says?), and each needs a different recovery. Persistence is the one teams skip: a successful write is not always readable milliseconds later, so the engine I built confirms state that feeds a customer page with a bounded read before rendering.

Chapter 17 argues the operational disciplines in detail; here they are as a deployment checklist:

- **Idempotency keys** on intake, generation jobs, callbacks, and recovery, because exactly-once delivery is hard in distributed systems [3].
- **Bounded retries with backoff and jitter**, at one layer [4]. Retry a fault. Never retry a decision.
- **Circuit breakers per provider**, so an exhausted dependency trips once instead of failing on every record.
- **Three result classes**: full, degraded, and fail closed. Specificity should degrade before reliability does.
- **Recovery that targets the missing artifact.** If the page exists and the callback failed, resend the callback. Do not rewrite the account.
- **Separate internal and customer wire types**, so adding an internal field never publishes it. The wire schema also carries the destination's field limits: where a downstream system caps a field, generate to a budget below the cap, then validate and truncate before the payload leaves.

None of this is specific to AI. That is the point. Once the model is treated as an unreliable dependency, ordinary distributed-systems engineering does most of the work.

## Watch business state, not only logs

Logs catch exceptions. They do not catch an API that returns 200 while a required artifact is missing, a callback that completes with degraded content, or a provider that quietly returns worse data [1]. Larkspur's Tuesday failure produced no errors at all.

So I monitor two classes of signal: **system signals** (errors, timeouts, queue depth, provider failures) and **workflow signals** (stuck records, missing artifacts, fallback rate, recovery saturation, defect classes). The second class matters more, because silent degradation is the common failure. And a monitor needs a heartbeat: one that silently stops must not look like a healthy system.

Cost belongs in the same view. Personalization gets expensive through fan-out, not any single call: several providers, a dozen zone generations, an audit, three emails, a page and a playbook per record, multiplied by retries. The rule I deploy with: spend model intelligence on uncertainty and judgment, and use ordinary software for everything deterministic. Cheap checks run first; a failed callback never triggers regeneration; semantic QA runs on a worklist and widens only when risk justifies it.

Evaluation modes need the same discipline. In a notification engine I built for my own product, separate from the personalization engine (Chapter 16 sets out the notification decision), full AI evaluation cost about $0.004 per event, mostly spent deciding not to notify anyone. Right for calibration, wrong for production, where a daily per-user digest cost about $0.01 per user per day [5]. Deployment is where calibration mode should end.

## One campaign cannot corrupt another

Once the engine serves more than one campaign, a new class of defect appears: correct content in the wrong context. A company that appears in two campaigns must not share one page. A guideline found by a loose name search can be the wrong version or the wrong tenant. A callback can belong to a different downstream organization.

The protection is boring and complete: scope every durable artifact by the keys that define uniqueness (tenant, campaign, account or person, artifact type, version), and resolve authentication and tenant routing before any enrichment starts [1]. No model evaluation will catch a cross-campaign defect. Only keys will.

## The first cohort is part of the build

A campaign is not ready because it passed a handful of test records. The first few hundred real records reveal behavior nobody anticipated. In the engine I built, an operator widens the cohort in steps (synthetic records, then 10 to 25 real ones, 50 to 100, a few hundred) and at each step uses AI to reason across the population, asking what repeats and what belongs in code rather than another guideline. Chapter 3 frames this as qualitative research on the machine's own output. For high-value campaigns, a domain expert reviews the first outputs; that is calibration, not a person reading every record. Chapter 20 gives the method: population QA, semantic review on a worklist, root-cause classification by layer, and every fix validated on replay, held-out, and regression sets.

For deployment, three decisions follow. First, staff the first cohort as a build phase with engineers attached, not as a launch with a support rota. Second, gate widening on explicit verdicts (GO, GO WITH FIXES, NO GO) against thresholds written in the contract. Google's SRE practice of canarying, comparing a small slice against a control before widening, is the right mental model [6], judged here on content quality and delivery completeness as well as errors. Third, decide in advance who may promote changes. Today, in my engine, a person diagnoses and decides; the design I would adopt as automation grows is Chapter 18's human-supervised autonomy, with two approvals and no production credentials on the analysis worker. NIST's AI RMF treats governance as continuous across the lifecycle rather than a single gate [7]; this is that idea as an operating model.

## Deployment phases

Here is the order the architecture implies. It is an ordering, not a promise of duration.

| Phase | What gets built or decided | Exit criterion |
|---|---|---|
| 0. Decide and contract | The first decisions to personalize (Chapter 6); the contract; each requirement sorted by control type; release thresholds | A contract a reviewer could fail a record against |
| 1. Foundations | Intake with idempotency; identity and governed context; shared mechanics at choke points; workflow journal; wire schemas | Any record's state and provenance can be answered by query |
| 2. Shadow and first cohort | Synthetic records, then real ones in widening cohorts, delivered to internal reviewers or a small live slice; expert review of first outputs where the stakes justify it; population QA; layer diagnosis | GO verdict against the written thresholds |
| 3. Supervised expansion | Wider cohorts against a control or holdout (Chapter 21); business-state monitoring; cost ceilings; recovery tested on purpose | Stable defect, fallback, and cost rates at the wider volume |
| 4. Second campaign and supervised improvement | A second campaign on the same shared mechanics; scheduled analysis with two approvals | The second campaign launches for clearly less effort than the first |

A 90-day sketch under this order, for a mid-market company with one decision in scope: roughly the first three weeks on Phase 0 and discovery, the next five on Phase 1, then about three weeks of first cohort, and the final two on the expansion decision and the second campaign's contract.

### After the second campaign: the productization sequence

Deploying an engine and turning it into a product are different jobs. The sequence I am following for the engine I built is: **standardize, measure, productize operator workflows, expose safe agent control, broaden channel compilation, increase supervised autonomy**. Most of it is ahead of me; read it as direction, not results.

Standardize means one machine-readable campaign manifest (surfaces, personalization mode, templates, guidelines, research mode, quality policy, delivery requirements) that runtime, QA, onboarding, agents, and reporting all read. Measure means every report says what was expected, delivered, degraded, and recovered, and which version produced it. The human-run population review becomes a repeatable, read-only step that ends in a proposal. Agents get stable domain operations instead of raw cloud access. New channels come next, print first with stricter preflight, and autonomy widens last, with production behind a human approval (Chapter 18).

The six-month target: **repeatable** (a new campaign does not need the original architect), **measurable**, **AI-operable**, **human-controlled**, **multi-channel**. A target, not a result.

A warning about discovery: in the manual process I used to run, understanding a client's systems well enough to design an integration took six to ten weeks [8], mostly rediscovering what the code already knew. Point an AI assistant at the codebase and data model first, and spend the meetings on decisions.

## Build, buy, or own the layer

Most AI decisions in customer operations get made as a feature comparison: which vendor's agent looks best. That question hides the one that shapes the architecture. Which parts of this are commodity enough to rent, and which are strategic enough that you should own the intelligence and the economics behind them?

Renting is frequently right: the vendor carries the risk, the support burden, and the reliability commitment. The published evidence leans toward buying, with caveats. MIT's Project NANDA reported that, in its interview sample, external partnerships with "learning-capable, customized tools" reached deployment about 67% of the time, against about 33% for internal builds [9]. I take the direction seriously and the number lightly. The sample is 52 organizations, outcomes are self-reported, success definitions varied, and the authors themselves warn that the gap "may reflect organizational capabilities rather than implementation approach alone." The same report names the real barrier as a learning gap: tools that do not retain feedback, adapt to context, or improve over time [9]. That finding cuts across build versus buy. Whatever you choose, the learning has to accumulate somewhere you can use it.

That is my test. Three loops compound inside a personalization layer. **Memory**: every identity resolved and fact written down once is work the next operation does not repeat. **Routing**: every step priced to its real risk, a small model on narrow work and a large one where judgment lives, is a decision made once. **Acceptance**: every output a person reviews becomes a labeled example of what good looks like in your business. A rented layer can reset all three at the contract boundary. Rent the commodity; own what compounds.

Either way, ask five questions, in order of what a wrong answer costs [10]: What can it do that cannot be undone? Who approved each action, and can you prove it six months later? Who sets its ceilings? Does it work per step, not just on average? When it fails, does it degrade or collapse? Building does not exempt you from answering them.

If you own the layer, keep workers replaceable: the model underneath will change fast (GPT-3.5-level inference fell from $20 to $0.07 per million tokens between November 2022 and October 2024 [11]). And consider where it runs. Deploying into the customer's own infrastructure, with their keys and audit logs, lets their security team inspect the controls instead of taking a vendor's word for them.

The self-hosted memory system I built shows the shape (architecture in my paper [16]). It is one stateless container on the customer's own Postgres with pgvector, and that database holds everything: memories, governance documents, schema, job queue, relationship graph, usage counters. It runs on any major cloud, Kubernetes, one on-prem machine, or air-gapped with a signed offline license and local models. Models are chosen per function (small for extraction, stronger for generation, or local for all of it). The vendor's only call is a license check carrying usage counts, never content; the offline license removes even that.

That keeps the memory and routing loops the customer's by construction, and moves work to them: their database is the single point of failure unless run highly available, and the embedding dimension is fixed at first boot, so a different-sized embedding model means re-embedding everything. Nor does self-hosted mean nothing leaves: a hosted model still receives the content it processes. Redaction and tokenization shrink that (Chapter 19); only local models send nothing. [PRODUCT-NAMING: named worked example of a governed engine and a self-hosted memory system deployed in the customer's own infrastructure, if Hamed decides to name them.]

## The team is small, and the roles are specific

A lean team can run this; it cannot skip the roles. When agents do the work, coordination people once carried becomes schemas, memory, evaluations, and permissions [12]. Responsibility moves; it does not disappear.

| Role | Owns |
|---|---|
| Product or system owner | Purpose, the decisions in scope, success criteria, the go decision |
| Campaign owner | The contract, guidelines, and examples for each campaign |
| Identity and memory owner | Matching, provenance, freshness, correction, deletion |
| Platform engineer | Shared mechanics, delivery state, recovery, cost controls |
| Evaluation owner | QA method, release thresholds, the three validation sets |
| Governance approver | Irreversible-action list, the second approval, policy precedence |
| Channel owners | Each surface's compiler policy and final-artifact checks |

In a small company, one person holds several of these. The separations that must survive are the ones between proposing and approving, and between the person who writes a guideline and the evaluation that judges it.

## The same design at three scales

The design does not require enterprise complexity. It requires the separations.

| Organization | Scope | What matters most | Practical shape |
|---|---|---|---|
| Enterprise | Many campaigns, tenants, regions, and channels | Governance plane, campaign isolation, recovery at volume | Separate services per the reference pattern; request-serving and worker pools scaled separately; dedicated owners per role |
| Mid-market (Larkspur) | A few decisions across CRM, warehouse, product telemetry, support | One identity spine and one governed context feeding two or three channels | One pipeline with clear modules; two replicas on a managed database; a handful of people holding several roles |
| Small business | One owner, one CRM, a few channels | Fallbacks and one human approval before anything irreversible | One process performing separated steps, on one machine and one database |

The small version may run in one process. Evidence, inference, generation, validation, and authority still stay separate.

Scaling out stays boring when processes hold no state. In the self-hosted system, replicas coordinate only through the database (workers claim jobs with row locks that skip rows already held; the scheduler advances each schedule with a conditional update), so every job fires once however many replicas run. Request-serving and background workers can be split and scaled on their own signals (workers on queue depth), and in a container, set job concurrency explicitly: auto-sizing reads the host's cores, not the container's limit, and over-provisions. The container is rarely the ceiling: provider rate limits bind first, and as replicas multiply, database connections run out before CPU.

## Score maturity on six dimensions, not one ladder

A single ladder misleads: a company can have excellent decisioning and fragmented identity. Score each dimension on its own, then fix the weakest one that blocks the decision you care about.

![A grid of six maturity dimensions (understanding, identity and memory, decisioning, experience generation, learning, governance and reliability), each with its own five levels, from Generic to Real-time and uncertainty-aware for understanding, and from Ad hoc to Consequence-aware autonomy for governance. The example from the text is marked: the same company shows excellent decisioning at the top of its row while identity sits at Fragmented, Level 1. Each dimension is scored on its own, and the weakest one blocking the target decision is fixed first.](/images/handbook/ch20-six-dimension-maturity.svg)

*Figure 22.3. Maturity is a profile, not a ladder: one company can be excellent at decisioning and fragmented on identity.*

| Dimension | Level 1 | Level 2 | Level 3 | Level 4 | Level 5 |
|---|---|---|---|---|---|
| Understanding | Generic | Segment | Individual | Contextual | Real-time, uncertainty-aware |
| Identity and memory | Fragmented | Unified profile | Graph and context | Durable memory | Governed, adaptive memory |
| Decisioning | Static | Rules | Predictive | Optimized | Reasoning within constraints |
| Experience generation | Static | Variants | Dynamic selection | Generative | Interactive, adaptive |
| Learning | Manual reporting | A/B tests | Causal measurement | Bandits, online learning | Governed continuous improvement |
| Governance and reliability | Ad hoc | Policy checks | Observable decisions | Automated guardrails | Consequence-aware autonomy |

The companion Assess skill interviews a team and scores these six dimensions with evidence; the Plan skill turns the scores into a build sequence and a 90-day plan. Both are built to show gaps, not to flatter.

## What this does not do

**It does not make the decision worth making.** A reliable engine pointed at a low-value decision delivers low value reliably. Chapter 6 comes before this one for a reason.

**The phases are an ordering, not a guarantee.** I can show why each step comes before the next. I cannot promise how long each takes in your company, and neither can anyone quoting a universal timeline.

**The failure-rate headlines are weaker than they sound.** Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025 [13], and that over 40% of agentic AI projects will be canceled by the end of 2027 [14]. Those are forecasts. MIT's "95% of organizations are getting zero return" comes from the preliminary report above [9]. The causes they name (unclear value, cost, weak risk controls, systems that do not learn) fit this chapter; the percentages should not be quoted as findings.

**My evidence is my own deployments,** not a controlled comparison of architectures. I describe what repeatedly broke and what fixed it; I cannot give you a population-level success rate for this approach. The productization sequence and its six-month target are a plan, not an outcome. And a vendor with strong memory, governance, and export rights can beat an owned build: the test is what accumulates and who can use it, not who wrote the code.

## At scale

Across millions of people and many teams, four things change. Campaign count, not record count, becomes the growth axis, so the share of work done by shared mechanics decides cost per launch. Recovery becomes a standing service, because some records are always mid-failure. Governance needs its own versioned plane, so a policy change is a reviewed release with a rollback target [1]. And the person who can answer "why did this customer receive this?" must exist, per campaign and surface, before the question is asked.

## Failure story: The Campaign Fork

This is Larkspur's history from before the engine (fictional, as before). An early AI campaign was a success, built fast: one long prompt, a few trust rules written inline, a renderer with the campaign's CTA hard-coded. For the second campaign, a different team copied the repository and edited the prompt. The third copied the second. By the fifth, a fix for free-mail domains existed in two forks and not in three, a tone change requested for one campaign had leaked into another through a shared template, and nobody could say which version of which guideline had produced a given email.

Nothing about the model had changed. Every campaign had become a new engine, and every engine had to learn every lesson again. That history is the reason the engine in this chapter starts from shared mechanics.

## Patterns

**Contract First.** *Problem:* QA and release decisions have nothing to judge against. *Solution:* a machine-readable contract per campaign and surface, with release thresholds, written before generation. *Tradeoff:* slower first week; much faster every week after.

**Shared Mechanics, Configured Judgment.** *Problem:* each new campaign forks the engine. *Solution:* invariants in code at shared choke points; campaign intent in versioned guidelines and configuration. *Tradeoff:* discipline about where each requirement lives, enforced in review.

**Own What Compounds.** *Problem:* the layer that accumulates memory, routing decisions, and reviewed examples resets at every contract boundary. *Solution:* rent commodity capabilities; own, or secure export and control of, the compounding layer. *Tradeoff:* more responsibility for reliability and support.

## Leader questions

1. Show me the contract for the first decision. Could a reviewer fail a record against it?
2. For any record, can you tell me its workflow state, exactly what was delivered, and which version produced it?
3. Which protections live at shared choke points, and which live only in prompts?
4. Who may promote a change to production, and what credentials does the automated analysis hold?
5. What will the second campaign cost to launch compared with the first?

## Build checklist

- [ ] Contract per campaign and surface, including release thresholds and fallback policy
- [ ] Every requirement sorted by control type: guideline, deterministic, or both
- [ ] Idempotency keys on intake, generation, callbacks, and recovery
- [ ] Workflow journal with per-record state from accepted to confirmed
- [ ] Circuit breakers and bounded retries per provider
- [ ] Full, degraded, and fail-closed result classes, labeled in telemetry
- [ ] Artifacts scoped by tenant, campaign, entity, artifact type, and version
- [ ] Business-state monitoring with a heartbeat
- [ ] First cohort scheduled as a build phase with a GO / GO WITH FIXES / NO GO gate
- [ ] Two approvals for production changes; no production credentials on analysis workers

## Metrics to watch

1. **Delivery completeness**: share of records whose expected artifacts were confirmed delivered.
2. **Fallback and fail-closed rates** by surface, trended against identity and evidence quality.
3. **Cost per accepted artifact**, including retries and recovery, not just model calls.
4. **Time and effort to launch the next campaign**, the clearest sign of platform versus pilot.
5. **Defect recurrence** after a fix, by root-cause layer.

## Reader Q&A

**Can we start without identity resolution and add it later?** You can start with a narrow scope, but not without the rule that personalization specificity falls when identity confidence falls. Without it, the engine personalizes confidently to the wrong entity from the first day.

**Do we need a multi-agent system?** Rarely at first. A focused function per zone is often enough. Anthropic's engineering guidance says the same: find the simplest solution possible and add complexity only when it demonstrably improves outcomes [15].

**What if our vendor will not let us export memory or review data?** Then price that in. You are renting the compounding loops as well as the software.

## For your AI

```yaml
chapter: 22
concepts:
  - name: Experience Contract
    definition: "Machine-readable specification of audience, surfaces, personalization tier, approved and prohibited claims, fallback policy, enforcement type per rule, and release thresholds."
  - name: Shared Mechanics
    definition: "Invariants enforced in code at shared choke points that every campaign inherits: sanitizing, escaping, idempotency, retry classification, internal-field stripping, tenant routing, wire schemas."
  - name: Campaign Judgment
    definition: "Voice, priorities, examples, and strategy held in versioned guidelines and configuration, editable per campaign without an engine deploy."
  - name: Workflow State
    definition: "Per-record status from accepted to confirmed delivery, used to diagnose failures and target recovery."
  - name: Own What Compounds
    definition: "Rent commodity capabilities; own or control the layer where memory, routing decisions, and reviewed examples accumulate."
  - name: Five Kinds of Success
    definition: "Generation, content, persistence, delivery, and campaign success, tracked separately because each fails on its own and needs a different recovery."
  - name: Productization Sequence
    definition: "After the second campaign: standardize (one campaign manifest), measure, productize operator workflows, expose safe agent control, broaden channel compilation, increase supervised autonomy."
  - name: Six-Dimension Maturity
    definition: "Independent scores for understanding, identity and memory, decisioning, experience generation, learning, and governance and reliability."
decision_rules:
  - if: "no written contract with release thresholds exists for a decision"
    then: "do not generate for real records; write the contract first"
  - if: "a requirement must hold for every campaign"
    then: "enforce it in code at a shared choke point, not in campaign prompts"
  - if: "a requirement expresses one campaign's intent"
    then: "put it in versioned campaign guidelines or configuration"
  - if: "a failure is a deterministic outcome (missing context, invalid contract, policy rejection)"
    then: "do not retry; fall back or fail closed"
  - if: "an artifact exists but delivery failed"
    then: "redeliver the artifact; never regenerate or re-research"
  - if: "a capability accumulates memory, routing, or reviewed examples"
    then: "own it or secure export and control before renting it"
  - if: "a write feeds a customer-facing artifact"
    then: "confirm it with a bounded read before rendering; a successful write is not yet a readable one"
  - if: "policy or residency forbids vendor storage of customer data"
    then: "run the layer on the customer's infrastructure and database; use local models if no content may leave"
  - if: "the second campaign costs about as much to launch as the first"
    then: "stop adding campaigns and extract shared mechanics"
assessment_questions:
  - "Which decision is first, and is there a written contract for it?"
  - "Can you query any record's workflow state and delivered artifacts?"
  - "Which protections live at shared choke points versus in prompts?"
  - "Who approves production changes, and what credentials do automated workers hold?"
  - "Which parts of the stack are rented, and can you export memory and review data?"
  - "How do you score on each of the six maturity dimensions, with evidence?"
patterns: [Contract First, Shared Mechanics Configured Judgment, Own What Compounds, Recover The Smallest Unit, Human-Supervised Autonomy]
anti_patterns: [The Campaign Fork, Whole-Pipeline Replay, Logs-Only Observability, Demo-Driven Launch, Autonomous Production Mutation]
maturity_dimension: governance_and_reliability
handoff:
  assess_skill: "score the six maturity dimensions"
  plan_skill: "produce build sequence, phase plan, team roles, and 90-day plan from this chapter"
```

## References

1. Taheri, H. (2026-08-23). "How I Build an Enterprise Personalization Engine That Can Be Trusted at Scale." https://hamedtaheri.com/articles/building-enterprise-personalization-engine
2. Sculley, D. et al. (2015). "Hidden Technical Debt in Machine Learning Systems." Advances in Neural Information Processing Systems 28 (NIPS 2015). https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
3. AWS Well-Architected Framework, "REL04-BP04 Make mutating operations idempotent," accessed 2026-09-26. https://docs.aws.amazon.com/wellarchitected/latest/framework/rel_prevent_interaction_failure_idempotent.html
4. AWS Well-Architected Framework, "Control and limit retry calls," accessed 2026-09-26. https://docs.aws.amazon.com/wellarchitected/2025-02-25/framework/rel_mitigate_interaction_failure_limit_retries.html
5. Taheri, H. (2026-03-07). "Dogfooding governed memory: building smart notifications for our own product." https://hamedtaheri.com/articles/dogfooding-governed-memory
6. Google, *The Site Reliability Workbook*, Chapter 18, "Canarying Releases." https://sre.google/workbook/canarying-releases/
7. NIST (2023-01-26). *Artificial Intelligence Risk Management Framework (AI RMF 1.0)*, NIST AI 100-1. https://www.nist.gov/itl/ai-risk-management-framework
8. Taheri, H. (2026-03-10). "Encoding Solution Architecture Into an AI Skill." https://hamedtaheri.com/articles/solution-architect-skill-deep-dive
9. Challapally, A., Pease, C., Raskar, R., Chari, P. (2025-07). *The GenAI Divide: State of AI in Business 2025*. MIT Project NANDA (preliminary findings). https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Business_2025_Report.pdf
10. Taheri, H. (2026-08-10). LinkedIn carousel on five agent-platform questions.
11. Stanford HAI (2025). *AI Index Report 2025*. https://hai.stanford.edu/ai-index/2025-ai-index-report
12. Taheri, H. (2026-06-29). "The Lean Agentic Company." https://hamedtaheri.com/articles/the-lean-agentic-company
13. Gartner (2024-07-29). "Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025." https://www.gartner.com/en/newsroom/press-releases/2024-07-29-gartner-predicts-30-percent-of-generative-ai-projects-will-be-abandoned-after-proof-of-concept-by-end-of-2025
14. Gartner (2025-06-25). "Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027." https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
15. Schluntz, E., Zhang, B. (2024-12-19). "Building effective agents." Anthropic Engineering. https://www.anthropic.com/engineering/building-effective-agents
16. Taheri, H. (2026-03-18). "Governed Memory: A Production Architecture for Multi-Agent Workflows." arXiv:2603.17787. https://arxiv.org/abs/2603.17787

---

# Closing: The Generative Experience, and Your Plan

*The question: Where is this going, what will still be true when it gets there, and what do I do on Monday?*

> **Questions this closing answers**
> - What does a company look like a year after it takes depth seriously?
> - Will websites and products be generated for each person? What ships now, and what is still a demo?
> - If models keep changing every few months, what should we build that will not expire?
> - How do I hand this handbook to my own AI and get a plan back?

## The short answer

The next step in personalization is not a better sentence. It is a whole experience composed for one person: the page, the offer, the product screen, the conversation. Parts of that ship today; much of it is still a lab demo, and I label which is which below, as of September 2026. What will not change with the next model is the work underneath: a memory that holds an accurate, current model of each customer, a decision layer that chooses what should happen next (including nothing), governance that runs inside the system, and measurement that proves cause. A generated interface only raises the stakes on all four, because every component on a composed page is a claim about the person looking at it. The practical move is to give this handbook to your AI, let the Assess skill score where you stand, and let the Plan skill draft your first three decisions and a 90-day build. Then do the unglamorous part yourself.

## Tuesday, 9:00 a.m., a year later

Larkspur Systems (the fictional composite company that has run through this handbook) still sends a quarterly newsletter. It goes out at nine on a Tuesday, as it did a year ago. It goes to fewer people now, and it is honestly general: a product release note for contacts about whom Larkspur knows nothing that should change the message. Nobody pretends it is personalization. (All Larkspur details and numbers here are illustrative.)

The four people from the first page of this handbook got something else.

The customer whose renewal was thirty days out, and who had stopped opening the reporting module in spring, did not receive a renewal email first. Priya, her account manager, received a brief overnight: the drop in reporting use, the two tickets that preceded it, the name of the admin who had built the original dashboards and left in April. The brief cited its sources. Priya called. The email that followed was written after the call, and it said what Priya had promised.

The operations director with three open tickets about route sync received nothing from marketing that week. The decision layer suppressed the campaign she had been scored for, flagged the escalation to her account manager, and requeued the offer for thirty days after the fix held. When the offer came, it opened with the fix, not the feature.

The new head of operations, whose name had surfaced from a support signature, was not added to a nurture track. Memory resolved him to the account and marked the fact's source and date. His first message came from his account manager and explained what his team already used, because a new leader's first question is usually what he has inherited.

The account that had doubled its fleet was not sent a discount. The Discount Mistake from Chapter 2 was now a rule in the decision layer. It got a plan-fit review and a conversation about capacity.

None of this came from a smarter model. Larkspur's year looked like the table of contents. It fixed identity after a wrong merge. It built one memory that every agent reads, with a curation cycle that keeps it true. It put a contact budget and one arbitration point above six teams' programs. It turned its guidelines into checks that run before and after generation. Its first cohort of 2,000 messages failed a population-level scan in ways no reviewer caught, and the fixes were in four layers, none of them "a better prompt." Its open-rate chart went up while pipeline quality fell, and that is when it started a holdout. Some decisions still cannot be proven at Larkspur's volume, and the team says so.

A year in, the most visible change is not what Larkspur sends. It is what Larkspur no longer sends.

![A year later, each of the four Larkspur (fictional) customers from the introduction got a decision instead of the newsletter. The renewing account's account manager was briefed overnight with cited sources, called first, and sent the email after the call. The angry operations director's campaign was suppressed, her escalation flagged to her account manager, and the offer requeued for 30 days after the fix held, opening with the fix. The new head of operations was resolved to the account with the fact's source and date, skipped the nurture track, and heard first from his account manager; the account that doubled its fleet got no discount, because the Discount Mistake is now a decision-layer rule, but a plan-fit review and a capacity conversation. The newsletter still goes out, to fewer people and honestly general.](/images/handbook/ch21-four-people-a-year-later.svg)

*Figure 23.1. A year later, each of the four got a decision rather than the newsletter; the biggest change is what Larkspur no longer sends.*

## The page is next

A year ago, Larkspur's unit of personalization was the message. The unit is moving to the experience.

The old message was a letter. It was written once, it said the same thing to everyone, and once sent it could not respond. What has been arriving for a few years is a conversation, one that can be different with each person and can answer the question the letter never heard. The same shift is now reaching interfaces. A page, a dashboard, a form, or an onboarding flow can be composed at the moment a specific person asks for it.

Here is where that stands, labeled, as of September 2026 (Chapter 16 works through the mechanics).

| Capability | Status | Evidence |
|---|---|---|
| Per-person text and images in campaigns | **Ships now** | Routine in production. In a field test of more than 173,000 ad impressions, AI-generated images earned up to 50% higher click-through than stock images [1]. |
| Interactive UI returned inside AI assistants | **Ships now** | MCP Apps became an official Model Context Protocol extension on January 26, 2026: tools return dashboards, forms, and workflows that render in the conversation, with support announced in Claude, ChatGPT, Goose, and VS Code Insiders [2]. |
| Interfaces generated per prompt in consumer search | **Ships, as an experiment** | Google rolled out generative UI in the Gemini app and, for Pro and Ultra subscribers in the US, in Search AI Mode on November 18, 2025 [3]. |
| Declarative component catalogs for agent UIs | **Early specification** | Google's A2UI (v0.9 in a June 17, 2026 post) has agents send JSON describing what to render; the host renders only trusted components from a predefined catalog [4]. |
| Fully generated pages per visitor, at web latency, on a company's own site | **Lab demo** | Google's own researchers report generation can take "a minute or more" with "occasional inaccuracies", and raters still preferred pages made by human experts [3]. |
| Most customer-facing surfaces composed per person from memory, governed components, and a decision layer | **Forecast (mine)** | An inference from the rows above, not a finding. |

Two cautions about the forecast. The first is speed. METR's measure of how long a software task an AI agent can complete at 50% reliability has doubled roughly every seven months since 2019 [5], and METR itself warned in January 2026 that a 50% horizon of X hours "does not mean we can delegate tasks under X hours to AIs" [6]. Capability curves are real; they are not delivery dates.

The second is the claim you will hear most often: that every website will soon be generated fresh for every visitor. I do not accept it as stated. Google's evaluation, the most careful public one I know, still ranked expert-built pages first [3]. My expectation is narrower and, I think, more useful: a stable frame, with the parts that should vary composed per person from a governed catalog. The industry is converging on the same shape; the declarative approaches above restrict agents to allow-listed components for safety and consistency [4]. It is also the shape I ship. In the governed personalization engine I built, a landing page is a fixed shell: brand, navigation, and legal copy never vary, and roughly 8 to 12 declared text zones do. Each zone carries a contract (the context it may use, the context it must never use, a word limit, the checks it must pass) and approved segment copy to fall back on.

That shape is the whole handbook in miniature. The depth is precomputed and stored (Chapters 10 and 17). The component is chosen by a decision (Chapter 14). Every claim a component makes about the person must be grounded or left out (Chapter 15). The catalog is governed like brand guidelines, because it is brand guidelines (Chapter 18). The generated page does not replace the architecture. It is the architecture's newest output.

Where I am taking it next is direction, not results, as of September 2026. The same governed context should reach paper: an approved print template with declared zones and a deterministic preflight, whose QR code continues into a personalized page (Playbook M3). And it should start before identity: personalize to the campaign context a visitor arrived with (the audience, industry, and offer the ad was built for), then to the known person and account once a form tells you who they are. The second half runs today; the first is a pilot design.

## What stays true when the model changes

Here is the sentence this closing turns on. Models will change every few months; the model of your customer is the asset you keep.

Four things in this handbook do not depend on which model you run.

**Memory.** An accurate, current, sourced model of each person, resolved to the right identity and curated continuously. A better model reading bad memory writes better-sounding mistakes.

**Decisions.** What should happen next for this person, chosen from options that include waiting and doing nothing, under hard constraints. Generation answers "how do we say it"; the decision layer answers "should we say anything."

**Governance.** Rules executed as code paths before and after the probabilistic step, with a trace of what was known, decided, and why. An interface composed per person needs this more than an email does, because it has more places to be wrong.

**Measurement.** A counterfactual, a holdout, and outcomes that are not proxies. When every surface adapts, the only customer who tells you the truth is the one who did not get the adaptive version.

In the self-hosted memory system I built, that separation is literal. Memory and governance documents live in the customer's own Postgres; the model is a configuration entry per function, from several providers or a fully local model. Swapping the generation model is a config edit, and the memory does not move (changing the embedding model does mean re-embedding). Memory gives the next model continuity; governance gives it boundaries.

![Models are swappable: today's model gives way to the next one and the one after, every few months. Four layers do not depend on which model runs. Memory is an accurate, current, sourced model of each person; decisions choose what happens next, including waiting and doing nothing; governance runs rules as code paths before and after generation, with a trace; measurement uses a counterfactual, a holdout, and outcomes that are not proxies. The standard, specific and true or honestly general, does not expire either.](/images/handbook/ch21-durable-four.svg)

*Figure 23.2. Models change every few months; memory, decisions, governance, and measurement are the durable four you keep.*

The same boundary holds when AI improves the system itself. Today, in systems I run, AI proposes: a memory-curation pass starts by only proposing changes, and an operator promotes it stage by stage, with a stop that works mid-run. The direction I am building toward, not yet shipped, lets AI diagnose a campaign, fix it outside production, and validate the fix on replayed failures and held-out examples; production stays a separate human approval. High reasoning autonomy, bounded execution authority, explicit production control.

There is one more thing that stays true, and I will say it once. A system precise enough to compose an experience that serves one person is precise enough to compose one that pressures them, and which one it builds is set by an objective that a builder chooses.

Specific and true, or honestly general. That standard does not expire either.

## Your plan: give this handbook to your AI

You do not need to read this handbook front to back, and you do not need to translate it into a plan by hand.

**Step 1. Give it the text.** Point your assistant at the full-text file or the chapters you care about. Every chapter ends with a *For your AI* block: stable concept names, decision rules, assessment questions, and named patterns and anti-patterns. Those blocks are what the skills read.

**Step 2. Run Assess.** Install the Assess skill from the companion repo. It interviews you about your industry, channels, data sources, identity situation, stack, regions, team, and current personalization. It scores you on six maturity dimensions (understanding, identity and memory, decisioning, experience generation, learning, governance and reliability), each with evidence and gaps. Bring real artifacts: one actual customer record, one live campaign, one recent incident. Answers from artifacts are more honest than answers from memory.

**Step 3. Run Plan.** It takes your assessment and the handbook and drafts:

- your first three decisions to personalize, scored by value, data readiness, reversibility, and time to evidence (one fast win, one memory-building bet, one measurement foundation);
- a starter memory schema and freshness policy;
- starter governance rules and an irreversible-action list;
- a privacy checklist for your regions (dated, and not legal advice);
- an accuracy stack and a first-cohort protocol;
- a measurement plan with a sized holdout;
- a 90-day build sequence and the playbooks that fit your functions.

It is built to make gaps visible, not to flatter. If your identity layer cannot support the decision you most want, the plan will say so and put identity first. It will not promise revenue; it will show value as ranges, with the assumption carrying the weight made visible.

**Step 4. Argue with it.** The plan is a draft for your team, not a decision. Take it to the people who own the data, the channels, and the risk. The best use of the output is the meeting it starts.

## One at a time

In 1993, Peppers and Rogers asked companies to build relationships "one customer at a time" [7]. The phrase described something the economics could not yet deliver: understanding each person cost a skilled human's hours. That cost has fallen. The standard has not moved. Being right about one person still takes a sourced memory, a decision that is allowed to say "not now," and rules that hold when nobody is watching.

You will build this for hundreds of thousands of people. Each of them will meet it alone, one at a time.

## Leader questions

1. Which of our customer-facing surfaces is closest to being composed per person, and which of its components makes a claim we cannot source?
2. If we switched model providers next quarter, what would we lose? If the answer is "a lot", where does our customer understanding actually live?
3. What did we stop sending this year because a decision said not to?
4. Which personalization result would survive a holdout?

## For your AI

```yaml
chapter: 23
concepts:
  - name: Generative Experience
    definition: "An interface, page, offer, or flow composed for one person at request time, from memory, a decision, and governed components."
  - name: Governed Component Catalog
    definition: "An allow-listed set of UI components an agent may compose, each with a contract for what it may claim; governed like brand guidelines."
  - name: Stable Frame, Variable Parts
    definition: "Keep navigation and structure stable; compose only the parts that should vary per person."
  - name: Durable Four
    definition: "Memory, decisions, governance, and measurement: the layers that outlast any model choice."
decision_rules:
  - if: "a capability is described as the future of personalization"
    then: "label it ships-now, lab demo, or forecast, with an as-of date, before planning on it"
  - if: "a surface is to be generated per person"
    then: "precompute depth, choose components by decision, ground every personal claim, and render only from a governed catalog"
  - if: "a generated component would state a fact about the person without a sourced memory"
    then: "omit it or render the honestly general version"
  - if: "a model change is proposed"
    then: "confirm memory, decisions, governance, and measurement are provider-independent before switching"
  - if: "an AI agent proposes a change to live rules, memory, or a campaign"
    then: "let it reason, implement outside production, and validate; require an explicit human approval to promote to production"
assessment_questions:
  - "Which customer-facing surfaces could vary per person, and what decision would choose each variant?"
  - "Where does customer understanding live today: in prompts, in one vendor's tool, or in a memory you own?"
  - "Is there a holdout on each adaptive surface?"
  - "Which claims about a person can a generated interface make, and who approves the component catalog?"
patterns: [Stable Frame Variable Parts, Governed Component Catalog, Precompute-then-serve]
anti_patterns: [Every Page Generated, The Proxy Trap, The Hallucinated Detail]
maturity_dimension: experience_generation
skills_handoff:
  assess: "interview -> scores on six maturity dimensions with evidence and gaps"
  plan: "assessment + handbook -> first three decisions, memory schema, freshness policy, governance and irreversible-action list, privacy checklist, accuracy stack, measurement plan, 90-day sequence, playbooks"
```

## References

1. Hartmann, J., Exner, Y., and Domdey, S. (2025). "The power of generative marketing." *International Journal of Research in Marketing* 42(1). https://www.sciencedirect.com/science/article/pii/S0167811624000843
2. Model Context Protocol Blog, "MCP Apps: Bringing UI Capabilities To MCP Clients," 2026-01-26. https://blog.modelcontextprotocol.io/posts/2026-01-26-mcp-apps/
3. Leviathan, Y., Valevski, D., Natchu, V., Matias, Y., "Generative UI: A rich, custom, visual interactive user experience for any prompt," Google Research, 2025-11-18. https://research.google/blog/generative-ui-a-rich-custom-visual-interactive-user-experience-for-any-prompt/
4. Google Developers Blog, "A2UI + MCP Apps: Combining the best of declarative and custom agentic UIs," 2026-06-17. https://developers.googleblog.com/a2ui-and-mcp-apps/
5. METR, "Measuring AI Ability to Complete Long Software Tasks," 2025-03-19. https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/
6. METR, "Clarifying limitations of time horizon," 2026-01-22. https://metr.org/notes/2026-01-22-time-horizon-limitations/
7. Peppers, D., and Rogers, M. (1993). *The One to One Future: Building Relationships One Customer at a Time.* Currency/Doubleday.

---

# Playbook M1. Inbound and the Website

*The question: They came to us. What should they see and receive next?*

> **Questions this playbook answers**
> - A demo request just arrived. How fast do we need to answer, and with what?
> - Should our website change for each visitor, and how much can it know about someone who has not told us who they are?
> - What can AI generate for an inbound lead without inventing things about them?
> - Who should get a person, who should get an automated reply, and who should get nothing new?
> - How do we prove any of it worked with B2B volumes?

## The moment

*Larkspur Systems is a fictional composite company used throughout this handbook: field-service and fleet software, about 40,000 accounts and 250,000 contacts.*

Tuesday, 4:52 p.m. Rosa Alvarez, operations manager at Cardinal Mechanical (also fictional), a heating and plumbing contractor, has spent eleven minutes on Larkspur's website. She read the route-optimization page twice, opened a case study about a contractor with two branches, and filled in the demo form. In the free-text box, the one most forms make optional, she wrote: "38 techs across two branches. Our dispatcher quit last month and I am doing it myself from a spreadsheet."

Here is what old Larkspur did with that. An auto-reply thanked "Rosa" for her interest. The lead entered a round-robin queue. On Thursday a sales development rep called with the standard discovery script and asked how many technicians she had. On Friday she was enrolled in the same five-email nurture track as everyone else who had ever downloaded a guide. Nobody read the sentence about the dispatcher.

That is not an unusual failure; it is the median one. In an audit of 2,241 US firms published in HBR in 2011, Oldroyd, McElheran and Elkington sent each a web lead: 37% replied within an hour, 24% took more than a day, and 23% never replied at all. The average response among firms that replied within 30 days was 42 hours [1].

Here is the new Larkspur, the "fast win" chosen in Chapter 6. Within minutes of Rosa's submission, an agent checks whether Cardinal is already known, researches the account (or reads research another lead from Cardinal already paid for), and drafts two things: a brief for the rep and a first reply. The rep reads both, edits one sentence, and sends before 5:30. The reply answers what she wrote: it offers a 20-minute session on running dispatch for two branches without a dedicated dispatcher, links a page built for that situation, and asks one question about how jobs arrive today. Nothing in it claims to know anything she did not tell Larkspur or that is not public.

## The decision

This is the one place in personalization where the customer went first. Rosa told Larkspur who she is, what she has, and what hurts, in her own words. Here is the sentence this playbook turns on: **an inbound lead is not a prospect to be researched; it is a request to be answered, and the answer is personalized first by what the person told you.**

Most inbound programs decide the wrong things. They decide how to score the lead and which sequence to enroll it in. The decisions that matter, for each inbound moment, are these:

| Decision | Options (always including nothing) |
|---|---|
| Who responds | A rep, a specialist, customer success (if they are already a customer), support (if the "lead" is a problem), self-serve, no one new |
| How fast | Now, within the hour, next business morning (a considered answer beats a fast wrong one) |
| With what | An answer to their stated need, a question, a booked time, a resource, nothing further |
| What the site shows next | The default page, a variant chosen by context, a variant chosen by account, a page built for this lead |
| What to ask | One question that changes the next decision, or nothing |

"Nothing new" is a real answer more often than inbound teams admit. The form fill from a current customer with an open escalation belongs to their account manager, not to a sales sequence (Chapter 14's upsell that should not go out). The student, the job seeker, the competitor and the vendor pitching you need a polite routing or silence, not a nurture track.

Speed deserves an honest paragraph. The HBR data found that firms trying to contact a lead within an hour were nearly seven times as likely to qualify it as firms that waited even one more hour, and more than 60 times as likely as firms that waited a day or longer [1]. Two cautions. The data came from 1.25 million leads at 42 firms, and one of the authors was then CEO of a sales-acceleration software company, so read it as strong observational evidence, not an experiment. And the popular versions ("respond in five minutes or lose 21x" and "100x") trace to vendor-sponsored work, not to this paper; I do not use them. What survives is enough: interest decays in hours, and many companies still lose leads they paid for to plain neglect.

But speed is table stakes, not the product. An instant reply that ignores what Rosa wrote is just a faster way to sound like everyone else. The design target is the fastest *specific and true* answer, which in practice means minutes, with a human approving until the system has earned more autonomy (Chapter 18).

## What you need to know

Inbound signals come in four kinds, and they deserve very different trust.

1. **Declared**: form fields and, above all, free text. The most valuable personalization input in marketing is a sentence the customer typed, and most funnels throw it away. Extract it into typed facts with provenance (Chapter 8): technician count 38, branches 2, dispatcher vacancy, current tool "spreadsheet".
2. **Session context**: the pages read, the referrer, the campaign, the device. Useful for choosing what to show next; weak evidence about intent, because one visit is noisy.
3. **Inferred company**: reverse-IP lookup and enrichment. This names an organization, not a person, and is often wrong for remote workers, VPNs, mobile networks and shared offices. Treat it as a hypothesis with a confidence, never as identity (Chapter 9). A freemail address names no company and must not inherit the mail provider's firmographics; in the engine I built, a freemail lead suppresses company research and lowers the reply's specificity.
4. **Memory**: what Larkspur already knows. Is Cardinal a customer, a former customer, an open opportunity? Did another Cardinal employee fill a form last quarter? Is there an open ticket, an opt-out, a consent record? Match on strong keys: in the memory system I built, records resolve by email, website or a CRM id, a matching name alone never joins two records, and a merge stays reversible.

Buyers increasingly arrive late and informed. 6sense's 2025 buyer survey (vendor research) reports that buyers initiate first contact in about 79% of cases, at roughly 61% of the way through their journey [2]. Gartner's 2025 survey of 632 B2B buyers found 61% preferred a rep-free buying experience, 73% actively avoid suppliers that send irrelevant outreach, and 69% reported inconsistencies between a supplier's website and what its sellers told them [3]. Read together: by the time someone fills your form, they have done their homework, and the fastest way to lose them is to ignore it or contradict your own website.

The memory shape follows from that. A form fill is an **event attached to a person and an account**, not a new lead record. The anonymous session that preceded it is promoted to the known person only for the session that led to identification (Chapter 9: anonymous to known is a promotion, not a merge). The next step in the engine I built, a design rather than a result, carries one first-party session identifier from anonymous visit to form submit, so the pages read before the form join the lead's context. It links what one browser did on your own site; it never reveals who an anonymous visitor is. Account research is keyed to the company with a freshness window, so the second and third leads from Cardinal read the research the first one paid for (Chapter 17). Every fact carries its source, date, confidence and the surfaces it may appear on (Chapters 10 and 11), because the same fact that belongs in a rep's brief may not belong on a web page.

Identity confidence then sets the ceiling on what each surface may do:

| What you know | Evidence | What the website may do | What a message may say |
|---|---|---|---|
| Nothing but context | Referrer, campaign, page | Choose among approved variants by context | Not applicable |
| A probable company | Reverse IP, enrichment | Industry-level variant at most; never name the company | Not applicable |
| A person who told you | Form, typed email | Use what they declared, in pages linked to them | Their stated need; public account facts, sourced |
| A logged-in customer | Authenticated session | Account-specific content, within their permissions | Account history, within policy |

The table is Chapter 15's specificity ladder applied to the web: before identity, personalize to the campaign context; after it, to the person and the account. Enforce the row in code, not in a prompt, so the model cannot talk its way past it. The row decides how specific you may be; ambition does not.

![Four rising steps of identity confidence. Step 1, nothing but context (referrer, campaign, page): choose among approved variants by context. Step 2, a probable company (reverse IP, enrichment): an industry-level variant at most, never naming the company. Step 3, a person who told you (form, typed email): use what they declared, in pages linked to them. Step 4, a logged-in customer (authenticated session): account-specific content within their permissions. Steps 1 and 2 are anonymous and may use only public facts or what the visitor said this session; the row decides how specific you may be, not ambition.](/images/handbook/pb-m1-identity-ceiling.svg)

*Figure M1.1. The identity-confidence ceiling: each step of certainty about the visitor unlocks a little more, and anonymous pages never show account data.*

## The action

Three things happen after an inbound moment, on three clocks.

![A request arrives with form fields and free text and is first matched to memory: is this a customer, an open ticket, an opportunity, an opt-out? A customer or an open escalation goes to the account owner or support; a student, job seeker, competitor or vendor gets polite routing or silence. A real request is answered on three clocks: a first reply within minutes that answers the stated need and that a rep approves; a rep brief within minutes to an hour, where deep research lives, read by the rep and never sent; and a linked page built on lead acceptance, before the click, under automated checks. The reply and the page compile from the same context.](/images/handbook/pb-m1-inbound-three-clocks.svg)

*Figure M1.2. Match first, then answer: route or stay silent where that is right, and give a real request its reply, brief and page on three clocks.*

**The first reply (async, minutes).** An agent drafts it under a generation contract (Chapter 15) with a small number of typed zones: an opening that answers the stated need, one relevant proof point selected from an approved library, one question, one clear next step. The allowed evidence is what the person declared plus sourced, dated account facts; forbidden claims include prices, discounts, commitments, and anything learned from browsing ("I saw you on our pricing page"). A rep approves and sends it. Human approval at the start is not a bottleneck to remove quickly; it is where the system learns, because every edit is a labeled example of what it got wrong (Chapter 6). Loosen it by consequence tier as the edit rate falls.

**The rep brief (async, minutes to an hour).** Account snapshot, what the person said, what Larkspur already knows, what is unknown, and two questions worth asking. Deep research belongs here, where a person reads it before anything reaches the customer: research that takes minutes is a design decision, not a performance problem. Keep the boundary explicit: hypotheses and competitive context a rep may see must never reach the reply. In the engine I built, an explicit serializer decides what leaves the system, so a new internal field is never exposed just because someone added it.

**The website (real time, and mostly precomputed).** A page has a latency budget of a few hundred milliseconds of server time and a layout-shift budget of almost nothing (Chapter 17). So the live path selects; it does not think. For an anonymous visitor it selects among approved variants using context: the campaign that brought them, the industry page they came from. Campaign parameters already declare a persona, industry, region, message and offer; that alone picks a far better page than the default without identifying anyone. In the engine I built that mode is the next pilot: design, not result.

For Rosa, the page linked in her reply is generated and validated when the lead is accepted, before she clicks, so the click is a fetch. That is how the known-lead flow runs in the engine I built. The submission is acknowledged at once; in the background the page is generated into typed zones (hero, "for teams your size" section, relevant case study, next step), each with an approved fallback, inside a static shell for legal copy and navigation. Its inputs are read back before rendering, because a successful write is not always a readable one, and its link is checked before delivery. If only the page failed, only the page is rebuilt. It shows only what she declared plus public facts. Pricing stays the same for everyone; an adaptive pricing page is a separate, high-stakes decision Larkspur deliberately left on the whiteboard (Chapter 6). Chapter 16 places this page on the first rung of its Generative Experience Ladder and covers the render gate every zone passes before it is shown.

The page and the reply compile from the same context (one brain, many surfaces: Chapter 15's channel compilers). If the email says "two branches" and the page says "enterprise fleets," you have built one of Gartner's 69% [3].

The idea of a page that adapts to each visitor is older than language models. In 2009 Hauser, Urban, Liberali and Braun described "website morphing": inferring a visitor's cognitive style from clicks and switching the site's look and feel, estimating an almost 20% gain in purchase intentions from their model and data [4]. The newer version generates the interface itself. Google Research described generative UI in November 2025, deployed in experiments in the Gemini app and in AI Mode in Search: raters strongly preferred the generated interfaces over standard text answers, though human expert-designed websites still ranked first, and the authors note generation "can sometimes take a minute or more" with "occasional inaccuracies" [5]. For a marketing site, label it plainly: template-and-zone pages ship today; per-visitor generated pages are forecast. When they arrive, the same contract applies, because a generated page is a message with a layout.

Where this sits in the stack:

| Surface | What is generated | Timing | Approval |
|---|---|---|---|
| First reply | Zones of an email from a rep | Minutes after submission | Rep approves (initially all; later by tier) |
| Rep brief | Research synthesis, questions | Minutes to an hour | Read by the rep; never sent |
| Linked landing page | Zones filled from declared and public facts | On lead acceptance, before the click | Automated checks; sampled QA |
| Anonymous visit | Nothing; selection among approved variants | Real time | Variants approved in advance |
| Nurture | Next resource or nothing, per person | Scheduled | Contact budget arbitration (Chapter 14) |

## What can go wrong

**Failure story: The Billboard.** Larkspur's first web personalization vendor offered "account-based" pages: resolve the visitor's IP to a company, then tailor the hero. Someone configured it to use CRM fields. For a month, anyone whose connection resolved to Harlow Fleet Services, an existing Larkspur customer, saw "Built for Harlow's 400 technicians: see what's new in your plan." Harlow's office shared a building network with other tenants, so the page greeted a visiting competitor's sales engineer with Harlow's seat count and customer status. Nothing was false. It was customer data published to an unauthenticated surface on the strength of an IP address. The rule that fixes it is short: **an anonymous page may use only what is public or what the visitor told you in this session.** Account-specific content belongs behind a login or in a link sent to a verified person.

The other risks, in rough order of frequency:

- **Narrating the visit.** "Saw you were checking our pricing" is accurate and still fails Chapter 19's say-how-we-know test. People do not object to being known; they object to discovering they were watched. Aguirre and colleagues found overt data collection raised click-through on personalized ads while covert collection raised feelings of vulnerability and lowered it [6].
- **The confident wrong detail.** A first reply that invents a company event, or confuses Cardinal Mechanical with a similarly named firm (Chapter 15's failure story). Or one that tells a former customer "your current plan includes this." Entity-confidence gates on research, grounding checks on every personal claim, no relationship statement unless memory confirms it, and an honestly general fallback that steps down: person and account, account, segment, approved static copy, nothing.
- **Two voices.** Marketing enrolls the lead in nurture while sales is mid-conversation, or a customer's form fill starts a new-logo sequence. One arbitration point, one contact budget (Chapter 14; Playbook P9).
- **The broken page nobody read.** Late-arriving personalization that shifts the layout, or a hero showing a raw template fragment, the kind of defect Chapter 20 found only by scanning every rendered page rather than a sample. Stored fields can be right while the live page is wrong (a stale template or cache, the wrong page identity), so review the page a visitor sees.
- **Pressure dressed as personalization.** Countdown timers, invented scarcity, "three people from your company are viewing this." Excluded by policy, not by taste: write the prohibition into the contract (Chapter 18). Chapter 3 traces these deceptive patterns and why tuning them per person makes them harder to see.
- **Consent and channel rules.** Whether you may email the person at all, and how, depends on where they are and what they agreed to (Chapter 19). A demo request is consent to answer the demo request, not to a newsletter everywhere.

**What this does not do.** A faster, truer reply does not create demand; it stops you losing the demand that already arrived. And it does not make the anonymous website personal: until a visitor tells you who they are, choosing among approved variants by context is the ceiling.

## How you will know

Pick one primary outcome that is close to value but frequent enough to measure: **meetings held per inbound request**, or qualified opportunities per request. Not replies and not open rates (Chapter 21: an open rate is not an outcome). Track **time to first specific answer** as a process metric, and three guardrails: reader-flagged errors, unsubscribes and complaints from inbound contacts, and page Core Web Vitals on personalized pages.

Test design, with Larkspur's illustrative volumes. Chapter 6 put inbound at about 3,000 demo requests a year. Randomize at the request level: half get the new flow (agent-drafted, rep-approved, linked page), half the old one. To detect a close-rate change from 20% to 23% with standard settings (5% significance, 80% power) you need roughly 2,900 requests per arm: two years of Larkspur's volume. Detecting meetings held moving from 30% to 36% needs roughly 960 per arm, about eight months at a 50/50 split. So measure the nearer outcome, keep the test running long enough, and follow the treated cohort through to pipeline for the long-run check. Say plainly which effects your volume cannot detect.

For anonymous website personalization, randomize visitors (or accounts, when account-level variants are involved) and keep a small permanent holdout that always sees the default site. Set expectations with Kohavi's warning from Bing: only about 10 to 20% of tested ideas improved their target metric, and most winners moved it by 0.1% to 1% [7]. A page variant that "lifts conversion 40%" in week one usually has a bug, a novelty effect, or a tiny sample (Twyman's law, Chapter 21).

## Reader Q&A

**Should we buy a website personalization tool first?** Not first. Fix the inbound response path, which has the most declared evidence and the clearest outcome. Web personalization for anonymous visitors works on the weakest evidence you have.

**Is putting the first name in the subject line worth it?** It measurably helped in Sahni, Wheeler and Chintagunta's randomized field experiments (Chapter 1) [8]. It is also the shallowest thing you can do. Do it, then answer what they wrote.

**Can an AI reply instantly without a human?** For acknowledgment and scheduling, yes. For anything that makes a claim about the person's business, start with approval and remove it by consequence tier as the edit rate falls.

## For your AI

```yaml
playbook: M1
title: "Marketing I: Inbound and the Website"
question: "They came to us. What should they see and receive next?"
foundation_links: [6, 8, 9, 10, 11, 14, 15, 16, 17, 18, 19, 20, 21]
concepts:
  - name: Customer went first
    definition: "An inbound request is a stated need to be answered; personalization starts from what the person declared, not from what was inferred about them."
  - name: Identity-confidence ceiling
    definition: "What a surface may personalize is capped by how well the visitor is identified: context only, probable company, declared person, authenticated customer."
  - name: Declared-first evidence
    definition: "Form fields and free text are extracted into typed facts with provenance and outrank inferred company data."
  - name: Select live, generate ahead
    definition: "Website personalization on the live path selects among approved or precomputed variants; generation happens asynchronously before the click."
  - name: Context before identity
    definition: "Before identity, personalize to campaign context (persona, industry, region, message, offer); after identity, to the person and account."
decision_rules:
  - if: "the lead uses a freemail address or company identity is otherwise weak"
    then: "suppress company research and firmographics; use lower-specificity copy"
  - if: "a customer-facing zone states a relationship (current plan, renewal, product owned)"
    then: "block it unless the relationship is known in memory; otherwise reframe as an evaluation"
  - if: "only the personalized page failed"
    then: "rebuild the page alone and verify its link; do not rerun the pipeline"
  - if: "a visitor is not authenticated or verified"
    then: "use only public facts and what the visitor told you this session; never display account data or an inferred company name"
  - if: "an inbound request comes from an existing customer or an account with an open escalation"
    then: "route to the account owner or support; do not enroll in a new-customer sequence"
  - if: "the request contains free text"
    then: "extract it into typed facts with provenance and make the first reply answer it"
  - if: "a first reply makes any claim about the person's business"
    then: "require rep approval until the measured edit rate for that zone falls below the agreed threshold"
  - if: "company identity comes only from reverse IP or enrichment"
    then: "treat it as a hypothesis; allow industry-level variants at most"
  - if: "a proposed page element uses urgency, scarcity, or narrates the visitor's browsing"
    then: "reject it at the contract level"
  - if: "annual inbound volume cannot detect the target effect on close rate"
    then: "measure a nearer outcome (meetings held) and follow cohorts to pipeline"
assessment_questions:
  - "What is your median and 90th-percentile time to a first reply that addresses what the person wrote?"
  - "Do form free-text answers reach the rep and the reply, or are they discarded?"
  - "Is a form fill matched to existing accounts, open opportunities, customers and open tickets before routing?"
  - "What does your website show an anonymous visitor based on reverse IP, and could it expose customer data?"
  - "Do the reply, the landing page and the rep read the same customer context?"
  - "How many inbound requests do you receive a year, and what effect size can that volume detect?"
patterns: [Declared-First Reply, Identity-Confidence Ceiling, Precompute-then-serve, Generation Contract, Do-Nothing Option]
anti_patterns: [The Billboard, Narrating the Visit, Fast and Generic, Two Voices, The Hallucinated Detail]
metrics:
  primary: "meetings held per inbound request (or qualified opportunities per request)"
  process: "time to first specific answer"
  guardrails: ["reader-flagged errors", "inbound unsubscribes and complaints", "Core Web Vitals on personalized pages"]
maturity_dimension: experience_generation
```

## References

1. Oldroyd, J. B., McElheran, K., and Elkington, D. (2011-03). "The Short Life of Online Sales Leads." *Harvard Business Review*. https://hbr.org/2011/03/the-short-life-of-online-sales-leads
2. 6sense (2025). "2025 Buyer Experience Report." Vendor research. https://6sense.com/science-of-b2b/buyer-experience-report-2025/
3. Gartner (2025-06-25). "Gartner Sales Survey Finds 61% of B2B Buyers Prefer a Rep-Free Buying Experience." https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-sales-survey-finds-61-percent-of-b2b-buyers-prefer-a-rep-free-buying-experience
4. Hauser, J. R., Urban, G. L., Liberali, G., and Braun, M. (2009). "Website Morphing." *Marketing Science* 28(2):202-223. https://pubsonline.informs.org/doi/10.1287/mksc.1080.0459
5. Leviathan, Y., Valevski, D., Natchu, V., and Matias, Y. (2025-11-18). "Generative UI: A rich, custom, visual interactive user experience for any prompt." Google Research. https://research.google/blog/generative-ui-a-rich-custom-visual-interactive-user-experience-for-any-prompt/
6. Aguirre, E., Mahr, D., Grewal, D., de Ruyter, K., and Wetzels, M. (2015). "Unraveling the personalization paradox." *Journal of Retailing* 91(1). https://www.sciencedirect.com/science/article/abs/pii/S0022435914000669
7. Kohavi, R., and Longbotham, R. (2007), "Online Experiments: Lessons Learned," *IEEE Computer* 40(9), and Kohavi et al. (2014), "Seven Rules of Thumb for Web Site Experimenters" (KDD slides). https://exp-platform.com/Documents/2014-08-27ExperimentersRulesOfthumbKDD.pdf
8. Sahni, N. S., Wheeler, S. C., and Chintagunta, P. (2018). "Personalization in Email Marketing: The Role of Noninformative Advertising Content." *Marketing Science* 37(2). https://pubsonline.informs.org/doi/abs/10.1287/mksc.2017.1066

---

# Playbook M2. Accounts, Ads and Events

*The question: We chose them. How do we reach them as individuals?*

> **Questions this playbook answers**
> - We have a target account list. What should each person in each account see, hear, and receive?
> - How personal can paid ads be, and what do we actually control when the platform's AI runs targeting and creative?
> - What should happen before, during, and after a conference so the follow-up reflects the conversation that actually took place?
> - When is the right move to show nothing, invite no one, and send nothing?
> - How do we prove account-based work caused pipeline when the list is only a few hundred accounts?

## The moment

Larkspur Systems (a fictional company, a composite of many real ones) sells field-service and fleet software. This year it chose 300 target accounts for a mid-market push: large field-service operators that are not yet customers, picked by sales from the CRM and a purchased firmographic file. For eight weeks, LinkedIn ads ran against the list with one creative, "Field service, simplified." Then came the big moment of the quarter: a three-day booth at an industry trade show (fictional, like everything else here).

Marcus runs operations at a utility contractor with about 600 vehicles, one of the 300. He stopped at the booth on day two and spent forty minutes with Larkspur's solutions engineer. He described a dispatch migration from a competitor that had failed overnight, the two weeks his technicians spent on paper, and the reason he now distrusts any vendor that promises a "seamless cutover." The engineer promised to send a reference customer who had migrated at similar scale. She typed three lines into a notes app on her phone.

The show produced 1,400 badge scans. Two days later, all 1,400 received the same email: "Thanks for stopping by booth 412! Here is our e-book on field-service efficiency." Marcus received it. So did a student who scanned for a tote bag. The reference customer never arrived, because the promise lived on one phone. The ads kept running "Field service, simplified" to a man who had just explained why simplicity claims make him leave the room.

Larkspur had chosen Marcus's account by name, paid to reach it, and heard its most important problem in person. Then it treated him like everyone else.

## The decision

Account-based marketing (ABM) is the practice of choosing specific accounts and coordinating marketing and sales around each one. The industry benchmark distinguishes three depths: one-to-one programs, typically covering 5 to 100 accounts; one-to-few, 20 to 200; and one-to-many, 500 to 2,500 [1]. Most programs now run more than one type, and the median program covers about 250 accounts [1].

Here is the sentence this playbook turns on: **in account-based work, the account is the unit of decision and the person is the unit of message.** You decide to pursue a company. You never speak to a company. You speak to Marcus, to his CFO, and to the dispatcher who will use the product, and each of them needs a different reason to care.

![Left, the account as the unit of decision: pursue now, nurture later, or leave alone, plus a tier that is a budget for depth. At Larkspur (fictional), 30 accounts get a named rep, full research and human-approved plays; 120 get clustered, generated briefs reviewed by sampling; 150 get programmatic ads and honestly general content. Right, the person as the unit of message: Marcus in operations, his CFO, and the dispatcher who will use the product each get a different reason to care, and a buying-group member whose colleague is already talking to us hears nothing for now. Every moment (ad exposure, event invitation, meeting request, follow-up) includes silence as an option.](/images/handbook/pb-m2-account-decides-person-hears.svg)

*Figure M2.1. The account is the unit of decision and the person is the unit of message; "nothing" is a real option at every layer.*

So the decisions come in layers, each with "nothing" as a real option (Chapter 14):

- **Account:** pursue now, nurture later, or leave alone. An existing customer in an open escalation, or an account in live negotiation with sales, may need less marketing, not more.
- **Tier:** how much depth each account earns. Tiers are a budget for research and human time, not a status symbol.
- **Person:** who in the buying group hears from us, in which channel, and who hears nothing because a colleague is already in conversation.
- **Moment:** ad exposure, event invitation, a meeting request, a follow-up, or silence.

The decision is not "which accounts get the campaign." It is what changes for this person because we chose their account, compared with doing nothing (Chapter 6 prices that change).

## What you need to know

ABM runs on account memory: one record per company with its people and their relationships, not a list of contacts sharing a domain (Chapter 9). Three shapes matter.

**The account and its buying group.** Typed properties on the account (tier, fit score, customer status, open opportunities, active holds) and dated relationships to people: who is the economic buyer, who champions, who uses, who blocks. Roles change; a buying-group map with no dates is a guess about the past (Chapter 11). In the self-hosted memory system I built, the buying group is a graph, not a column: "works at" and "reports to" edges are inferred from saved content, so an agent can ask who else sits at this account before writing to anyone. Records resolve by whatever identifier a source carries (email, CRM id, or a key you define), because the badge scanner, the ad export and the CRM each name Marcus differently. A scan with a personal email address is weak identity; it must not inherit the email provider's firmographics.

**Evidence of what each person cares about.** Here the signals differ sharply in authority, and ranking them is most of the work (the Signal Ladder in Chapter 7):

| Signal | What it proves | Authority |
|---|---|---|
| A conversation (booth, call, meeting) | What this person said they are dealing with | Highest, if captured with source and date |
| First-party behavior (site visits, content, product trials) | Someone at the account engaged, sometimes who | Medium |
| Ad engagement | Someone at the account saw or clicked an ad | Low; account-level at best |
| Third-party intent data | Some unnamed people at the domain read related content somewhere | Lowest; treat as a hint to research, never as a fact to quote |

**Commitments we made.** Marcus was promised a reference customer. That promise is not a note; it is a typed obligation with an owner and a due date, readable by every agent and person that touches the account (Chapter 18). Conversations produce promises, and promises decay silently unless the system treats them as state.

The richest fact Larkspur held about Marcus was the forty-minute conversation. It was the only fact that never reached the system. Events generate the highest-authority signal in account marketing and usually store it in the lowest-fidelity place. Capture is a design problem: a short voice note after each conversation, transcribed and extracted against a schema (pains, current vendor, timeline, promises made, who was present), with the original preserved (Chapter 8). In the memory system I built, the transcript is saved first and properties are extracted onto the linked records afterward; if extraction fails, the transcript survives.

## The action

### Tiers: AI moves depth down the list

The benchmark data describes the old economics. One-to-one programs stay small because deep account research takes skilled hours. In the same 2023 study, practitioners rated their own use of generative AI at 1.9 on a five-point scale, the lowest of any tool category [1].

What changes (inference, not yet measured at scale): researching an account and drafting a brief per buying-group member now costs minutes of compute rather than hours of a person, so a one-to-one-style brief becomes affordable for one-to-few and even one-to-many accounts (Chapters 4 and 5). What does not change is the channel's tolerance for personal claims. Depth of understanding can move down the list; depth of what you say out loud should follow the channel.

At Larkspur the tiers became: 30 accounts with a named rep, human-approved plays, and full research; 120 accounts in industry-and-situation clusters with generated briefs reviewed by sampling; 150 accounts in programmatic coverage with ads and honestly general content until they show first-party engagement.

Write each tier's depth down as a contract. In the governed personalization engine I built, every campaign declares its mode before generation (company name only, personalized to the person, or an internal scored brief), and quality review judges output against that contract. When evidence is thin, output steps down: person and account, account, segment, approved static copy, nothing. Specificity degrades before trust does.

### Paid ads: you control the list, the exclusions, the creative set, the destination, and the signal

Paid media is where personalization claims most often outrun reality. Be precise about who controls what.

**The platform controls delivery.** LinkedIn will not serve an ad set to an audience smaller than 300 matched members [2]. That floor is a privacy boundary as much as a technical one: you cannot run an ad to Marcus alone. Meta and Google are pushing further toward automation. Google's AI Max for Search generates headlines and descriptions from your landing pages and routes people to the page it judges most relevant; Google reports advertisers "typically see 14% more conversions or conversion value at a similar CPA/ROAS," based on its internal data [3]. Meta says Advantage+ shopping campaigns average $4.52 return on ad spend, with 22% higher returns than advertisers not using its targeting [4]. Both are vendor-reported figures. In mid-2025 the *Wall Street Journal* reported Meta's aim to fully automate ad creation and targeting by the end of 2026, with Mark Zuckerberg describing a future in which "you don't need any creative, you don't need any targeting demographic" [5].

**You control five things.** The account and contact lists you upload. The exclusions. The set of creative you allow (by tier, industry, and stage, never by person). The page the ad lands on, which can be personalized per account far more safely than the ad itself (the *Marketing I* playbook; Chapter 16 on governed pages). And the conversion signal the platform optimizes toward. The engine I built is deliberately not an ad-buying system; it works on this owned side and leaves delivery to the platform. Choosing the landing page from the campaign context carried in the ad's link (persona, industry, region, theme, offer) is the design for my next pilot, not a result; a tracking parameter identifies a campaign, never a person (the *Marketing I* playbook).

That last one matters most. An ad platform personalizes to its model of the person, not yours. The most important personalization input you give it is not the creative. It is the definition of success. Optimize to form fills and the platform will find people who fill forms. Send back qualified opportunities from your CRM, and it learns something closer to what you want, with a lag and less volume. This is the Proxy Trap from Chapter 21, running inside someone else's model.

**Keep personal facts out of ads.** An ad cannot explain how you know what it implies you know. Aguirre and colleagues found that personalization raised click-through when data collection was overt and lowered it when collection was covert, with recipients reporting more vulnerability [6]. A banner saying "Still recovering from that failed migration?" to Marcus's colleagues is accurate and unacceptable. Relevance in ads comes from situation and industry, not from what someone told you at a booth.

### Events: before, during, after

The same decision logic applies at three moments, with different latency (Chapter 17).

![Three event moments on three clocks. Before (async, days): decide whom to invite starting from target accounts, send a meeting request with one researched reason, and give booth staff a brief the night before covering role, stage and what not to raise. During (real time, seconds): when a badge is scanned, answer three questions before anyone speaks (target, customer, or neither; open escalation or negotiation; a colleague already in conversation), then record a voice note after each real conversation. After (async, hours to days): follow-up is ranked on what was captured; a conversation with a target account gets a rep follow-up that keeps every promise, a conversation with a customer goes to the account manager, and a scan with nothing else gets an honestly general note or nothing.](/images/handbook/pb-m2-event-before-during-after.svg)

*Figure M2.2. Before, during, after: a badge scan is proof of proximity, not of interest.*

**Before (async, days).** Decide whom to invite and whom to leave alone, starting from target accounts and open opportunities rather than the whole database. For registered attendees at target accounts, the rep sends a meeting request with one researched reason that is specific and true, or honestly general. The night before each day, booth staff receive a one-page brief per expected target-account visitor: role, stage, open issues, what not to raise. In the engine I built this is a seller playbook (snapshot, why-now signals, talking points) compiled from the same account context as the landing page and the follow-up, so they do not contradict each other. Rep-internal intelligence is not customer copy: a guess about the incumbent belongs in the brief, never in the email.

**During (real time, seconds).** When a badge is scanned, the system answers three questions before anyone speaks: is this a target account, a customer, or neither; is there an open escalation or active negotiation; is there a colleague already in conversation? A customer with a priority-one ticket should hear "how is the fix holding?" and not a pitch. After each real conversation, the staff member records the voice note.

**After (async, hours to days).** The follow-up is a decision ranked on what was captured, not on the fact of a scan:

- **A real conversation with a target account:** the rep sends a follow-up drafted from the note, honoring every promise made, grounded only in what was said (Chapter 15).
- **A conversation with a customer:** routed to the account manager, not the campaign.
- **A scan and nothing else:** an honestly general note, or nothing. Never "great chatting with you," which asserts a conversation the system cannot prove.

A badge scan is proof of proximity, not of interest.

Speed matters, but the evidence for event follow-up timing is thin. The well-sourced result on response speed concerns inbound web leads [7]; I have not found a rigorous study of trade-show follow-up timing. The widely repeated line that "80% of trade show leads are never followed up" is commonly attributed to the Center for Exhibition Industry Research, but I could not locate the original study.

## What can go wrong

### Failure story: The Ad That Could Not Hear

A composite. After a painful outage, Larkspur's account manager promised a customer's operations director no sales or marketing outreach until the post-incident review closed. Larkspur had learned this lesson before (Chapter 18), so the promise became a typed outreach hold that every email and in-app agent read before acting. The agents went quiet.

The ads did not. The customer was also on an expansion audience, uploaded to LinkedIn from a weekly CRM export that did not include holds. For three weeks the operations director and her team saw "Routes that never drop" in their feeds, during an outage about dropped routes.

Nothing malfunctioned. The paid channel was simply never treated as a send path. The fix: exclusion audiences generated from the same suppression state as email, synced on every change rather than weekly, with a check that the platform has processed the update. Every send path is a send path, including the one you pay for.

### Governance risks

- **Invented familiarity.** Generated follow-ups that imply a conversation, a need, or a meeting the record cannot support. Ground every personal claim in captured evidence or go general (Chapter 15).
- **Five teams, one person.** ABM plays, an AI SDR sequence, event follow-up, and customer-success check-ins can all target Marcus in the same week. A per-person contact budget and one arbitration point decide (Chapter 14; the *One Customer, One Conversation* playbook).
- **Audience files are personal data.** Hashing emails before upload does not make them anonymous; the FTC has said so directly [8]. Deletion and objection must propagate to every uploaded audience (Chapter 19).
- **Regulatory lines on targeting.** In the EU, the Digital Services Act bars online platforms from showing ads based on profiling that uses special categories of personal data (Article 26(3)) and requires them to show users the main targeting parameters (Article 26(1)(d)); profiling-based ads to minors are barred under Article 28 [9]. Your audience definitions should never encode a special category, including by inference (Chapter 19). This is not legal advice; check current rules for your markets, as of your launch date.

### Honest limits

AI does not fix account selection. A beautifully researched brief for an account that will never buy is waste at a lower unit cost.

The most cited ABM evidence is self-reported. In the 2023 Momentum ITSMA benchmark, 81% of respondents who answered said ABM's ROI beats other marketing, yet only 52% said their company measures ABM ROI at all, and only 21% reported significant (over 10%) revenue growth they attributed to ABM [1]. I read that as a discipline with strong conviction and weak measurement, not as proof. Figures like "171% more pipeline" circulate without a traceable source; leave them out.

Platform performance claims deserve the same treatment. They are measured by the seller, on the seller's definitions.

## How you will know

Start with how badly ad measurement goes without experiments. In large randomized experiments at Facebook covering 1.6 billion impressions, Gordon and colleagues found that observational methods usually overestimated advertising effects relative to the randomized result, sometimes underestimated them, and did not reliably recover the truth [10]. At eBay, Blake, Nosko and Tadelis found brand-keyword search ads had no measurable short-term benefit, and non-brand ads paid mostly for frequent buyers who would have come anyway [11]. And Lewis and Rao showed why even good experiments struggle: across 25 large field experiments, the median confidence interval on ROI was more than 100 percentage points wide [12].

So the design follows Chapter 21, randomized at the unit of influence, which in ABM is the account:

- **Account-level holdout.** Before launch, randomly hold back a share of the target list from ads and plays (sales still works them normally). Compare opportunities created, win rate, and deal size between treated and held-out accounts. Never compare engaged accounts with unengaged ones; engagement is selection, not effect.
- **Be honest about power.** Detecting an opportunity rate moving from 10% to 15% at conventional thresholds (5% significance, 80% power) needs about 680 accounts per arm. A 300-account program cannot prove a five-point lift in one quarter. Pool across quarters, measure larger effects, or say plainly that the program runs on judgment plus directional evidence.
- **Choose hypotheses before you generate.** Four personas, six industries, five regions, eight message themes and three offers make 2,880 contexts (an illustrative campaign) a generative system can write for. That is expressive capacity, not experiment volume. A 300-account list supports two or three tested hypotheses a quarter.
- **Platform lift studies are useful and not neutral.** Use them, and label them as the platform measuring itself.
- **Events:** among scans with a captured conversation, randomize follow-up treatment (researched rep follow-up versus the standard one) and compare meetings and opportunities within 90 days. Track promise fulfillment: the share of commitments made at the booth that were met on time.

**Metrics to watch:** opportunities and pipeline per target account versus holdout; win rate in the treated tier; promise fulfillment rate; share of follow-ups grounded in captured evidence; suppression propagation lag to ad platforms.

## Reader Q&A

**Should we let the platform's AI write and target our ads?** For one-to-many coverage, often yes, inside an approved creative set and with a conversion signal you chose deliberately. Keep your exclusions, your claims policy, and your definition of success in your own hands.

**Can we name the account in the ad?** Naming a company can read as surveillance, and the platform floor means colleagues and others will see it too. Put account-level specifics on the landing page, where the visitor chose to arrive, and never name a person.

**Is third-party intent data worth buying?** As a hint for where to research, sometimes. As a fact to quote to a buyer, never. Measure it like any vendor claim: does it predict opportunities in your holdout better than your own first-party signals?

**Do we have to label AI-generated ad creative?** As of September 2026, the EU AI Act's Article 50 transparency duties apply from 2 August 2026: generative providers must mark synthetic output in machine-readable form, and deployers must disclose deepfakes [13]. Ordinary generated marketing copy is not generally caught, but platforms have their own labeling rules. Check both, per market; not legal advice.

## For your AI

```yaml
playbook: M2
title: "Marketing II: Accounts, Ads and Events"
question: "We chose them. How do we reach them as individuals?"
concepts:
  - name: Account as Decision Unit, Person as Message Unit
    definition: "Pursuit and tier are decided per account; every message is written for one role in the buying group."
  - name: ABM Tier as Depth Budget
    definition: "One-to-one, one-to-few, one-to-many tiers allocate research and human time, not status; AI lowers research cost so depth can move down tiers while channel tolerance limits what is said."
  - name: Conversion Signal as Personalization Input
    definition: "On automated ad platforms, the success event you send back shapes targeting more than the creative does."
  - name: Proximity Is Not Interest
    definition: "A badge scan proves someone was near the booth; only a captured conversation justifies a personal follow-up."
  - name: Promise as State
    definition: "Commitments staff make to a person at an event or meeting are typed obligations with owner and due date, readable by every agent."
  - name: Personalization Mode as Tier Contract
    definition: "Each campaign declares its depth before generation; QA judges against it, and thin evidence steps output down (person, account, segment, static, nothing)."
decision_rules:
  - if: "an event contact's email address is a personal (freemail) domain"
    then: "treat identity as weak; never attach the provider's firmographics; write at lower specificity"
  - if: "an account has an open escalation, legal hold, or active negotiation"
    then: "suppress acquisition and expansion ads and plays; sync the exclusion to every ad platform on change"
  - if: "an event contact has only a badge scan and no captured conversation"
    then: "send an honestly general note or nothing; never imply a conversation"
  - if: "a captured conversation includes a promise"
    then: "create a typed commitment with owner and due date; block generic follow-up until it is honored or reassigned"
  - if: "a personal fact was learned in conversation or inferred"
    then: "never use it in ad creative; ads carry industry and situation relevance only"
  - if: "the ad platform optimizes to a proxy (form fill, click)"
    then: "send qualified-opportunity conversions back instead, accepting lag and lower volume"
  - if: "the target list is too small to detect the expected lift in one quarter"
    then: "pool across quarters or state that results are directional; do not present engaged-vs-unengaged comparisons as lift"
assessment_questions:
  - "How many target accounts are in each ABM tier, and what research does each tier receive?"
  - "Where do booth and meeting conversations get recorded today, and can an agent read them within a day?"
  - "Which conversion event do your ad platforms optimize toward?"
  - "Do suppression holds and opt-outs propagate to uploaded ad audiences, and how fast?"
  - "Is there a random holdout of target accounts, and how many accounts per arm?"
patterns: [Account-Level Holdout, Suppression-Synced Audiences, Evidence-Ranked Event Follow-Up, Tier as Depth Budget]
anti_patterns: [The Ad That Could Not Hear, Badge Scan as Intent, Engaged-vs-Unengaged as Lift, Personal Facts in Ad Creative, Promises in a Notes App, Design Space as Test Plan]
links: {data: [7, 8, 9], memory: [10, 11], decisions: [14, 15, 17], trust: [18, 19], proof: [21], playbooks: [M1, S1, P9]}
maturity_dimension: decisioning
```

## References

1. Momentum ITSMA and ABM Leadership Alliance (2023-11). *Rethinking ABM: Outperforming the Market in the World of AI*, 2023 ABM Benchmark Study (7th annual). https://5356237.fs1.hubspotusercontent-na1.net/hubfs/5356237/MomentumITSMA_ABMLA2023_GlobalBenchmarkReport.pdf
2. LinkedIn Marketing Solutions Help, "Upload company and contact targeting lists for LinkedIn Ads" (accessed 2026-09-26). https://www.linkedin.com/help/lms/answer/a421822
3. Google (2025-05-06). "AI Max for Search campaigns." https://blog.google/products/ads-commerce/google-ai-max-for-search-campaigns/ (vendor-reported, internal data).
4. Meta, "Advantage+ shopping campaigns" (accessed 2026-09-26). https://www.facebook.com/business/ads/meta-advantage/advantage-plus-shopping-ads (vendor-reported).
5. Marketing Dive (2025-06), reporting the *Wall Street Journal* of 2025-06-02: "Meta plans to enable fully AI-automated ads by 2026." https://www.marketingdive.com/news/meta-plans-to-enable-fully-ai-automated-ads-by-2026/749613/
6. Aguirre, E., Mahr, D., Grewal, D., de Ruyter, K., Wetzels, M. (2015). "Unraveling the personalization paradox." *Journal of Retailing* 91(1). https://www.sciencedirect.com/science/article/abs/pii/S0022435914000669
7. Oldroyd, J., McElheran, K., Elkington, D. (2011-03). "The Short Life of Online Sales Leads." *Harvard Business Review*. https://hbr.org/2011/03/the-short-life-of-online-sales-leads
8. FTC Office of Technology (2024-07-24). "No, hashing still doesn't make your data anonymous." https://www.ftc.gov/policy/advocacy-research/tech-at-ftc/2024/07/no-hashing-still-doesnt-make-your-data-anonymous
9. Regulation (EU) 2022/2065 (Digital Services Act), Articles 26 and 28. https://www.cms-digitallaws.com/en/dsa/article-26/ ; https://www.cms-digitallaws.com/en/dsa/article-28/
10. Gordon, B. R., Zettelmeyer, F., Bhargava, N., Chapsky, D. (2019). "A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook." *Marketing Science* 38(2):193-225. https://pubsonline.informs.org/doi/10.1287/mksc.2018.1135
11. Blake, T., Nosko, C., Tadelis, S. (2015). "Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment." *Econometrica* 83(1):155-174. https://onlinelibrary.wiley.com/doi/abs/10.3982/ECTA12423
12. Lewis, R. A., Rao, J. M. (2015). "The Unfavorable Economics of Measuring the Returns to Advertising." *Quarterly Journal of Economics* 130(4):1941-1973. https://academic.oup.com/qje/article-abstract/130/4/1941/1914592
13. EU AI Act (Regulation (EU) 2024/1689), Article 50, and Commission FAQ. https://artificialintelligenceact.eu/article/50/ ; https://digital-strategy.ec.europa.eu/en/faqs/transparency-obligations-under-article-50-ai-act

---

# Playbook M3. The Physical Channel

*The question: When each touch costs real money, who deserves one and what should it say?*

> **Questions this playbook answers**
> - Is personalized print and direct mail worth doing in an AI-first program, or is it a legacy channel?
> - When every piece has a print and postage cost, how do we decide who gets one, and who gets nothing?
> - What can AI generate for a printed piece without inventing things, and what should never go on paper?
> - A printed mistake cannot be recalled. What has to happen before anything goes to the printer?
> - How do we measure a channel whose responses arrive by phone, by typed URL, and weeks later?

## The moment

*Larkspur Systems is a fictional composite company used throughout this handbook: field-service and fleet software, about 40,000 accounts and 250,000 contacts, and a small direct-mail program.*

September, the planning meeting for renewal season. About 10,000 Larkspur accounts renew in the next quarter. Marketing proposes a printed "Fleet Year in Review": a short booklet for each account, showing its own year on Larkspur (vehicles dispatched, routes optimized, hours of idle time flagged), with a personal note from the account manager and a QR code to a page that continues the conversation. A digital press can print every copy differently. A model can write every note.

Finance asks the question email never forced anyone to ask. At an assumed $3 to $6 all-in per booklet (digital print, a mailing envelope, postage, and proofing time; my working range, not a quote), the full quarter costs $30,000 to $60,000. The email version of the same idea costs almost nothing per send. So: who is each booklet for, and what will it change?

That question is the gift of the physical channel. Postage alone on a presorted US marketing letter runs 37 to 47 cents as of July 2026 [1]. Every piece is a line item.

## The decision

Here is the sentence this playbook turns on: **in print, every decision to act has an invoice, and every mistake has a shelf life.** Both halves change how the decision layer (Chapter 14) should behave.

The first half makes "who is worth it" concrete. Chapter 4 reduced the economics of attention to one calculation: what a better decision is worth, times how likely the approach is to produce it, minus what it costs. In email the cost term is near zero, so teams skip the arithmetic. In print you cannot. The rule for each account:

`send if (expected change in outcome because of this piece) x (value of that outcome) > (cost of the piece, including review time)`

Run it for two Larkspur accounts, with illustrative numbers. A two-truck plumber paying $1,200 a year: at $5 a piece, the booklet must raise the chance of renewal by more than about 0.4 percentage points to pay for itself. A 400-vehicle fleet paying $60,000: the same booklet needs less than a hundredth of a percentage point. The cost term barely registers.

That second result is the honest surprise. In high-value B2B, postage rarely decides who is worth it. **The sign of the effect does.** The expected change can be negative. Chapter 14 described Dana, whose dispatch board was broken while a campaign queued an upsell. A glossy booklet celebrating her "year on Larkspur," landing on her desk during a priority-one escalation, is the same mistake in a heavier format. Uplift modeling calls her a Sleeping Dog: someone an intervention makes worse [2]. Accounts that will renew anyway are Sure Things; a booklet to them is a thank-you, worth budgeting as one, not crediting as lift. In consumer programs with $40 orders, the cost term binds hard and the same formula prunes most of the list.

So the decision for each account has more than two options:

| Option | When it wins |
|---|---|
| Send the personalized piece | Expected change is positive and exceeds cost; facts for the content are verified and appropriate for paper |
| Send the general piece | The account qualifies, but the evidence supports only an honestly general version (Chapter 15) |
| Route to a person instead | High-value account with a live issue: the account manager calls; nothing is mailed |
| Hold | Something unresolved (open escalation, pending change of contact) makes this the wrong week |
| Do nothing | No option beats the cost, or the person has objected to direct marketing |

Larkspur's final list for the quarter was about 3,500 booklets, not 10,000, plus 400 accounts routed to account managers and a randomized holdout (see "How you will know"). The number was not chosen by budget. It fell out of the rule.

![A banner states the rule: send a printed piece only if the expected change in outcome times the value of that outcome exceeds the cost of the piece, including review time. At an illustrative $5 a piece, a two-truck plumber paying $1,200 a year needs a renewal lift above about 0.4 points, so cost binds, while a 400-vehicle fleet paying $60,000 needs less than a hundredth of a point, so cost barely registers. The choice therefore has five options: personalized piece (positive effect, verified facts), general piece (evidence supports only general), route to a person (high value with a live issue), hold (open escalation or contact change), and do nothing (nothing beats cost, or the person objected). For Larkspur (fictional), 10,000 renewing accounts became about 3,500 booklets and 400 calls.](/images/handbook/pb-m3-priced-decision.svg)

*Figure M3.1. Price every piece, then let the sign of the effect, not the postage, choose among five options, including do nothing.*

Email made it easy to forget that attention has a price. Print puts the price on the invoice.

## What you need to know

Print needs everything a digital message needs, plus an address that is current, a person who is still there, and facts that will still be true when the piece arrives.

**Address, with provenance and a date.** In the US, mailers claiming presorted First-Class or USPS Marketing Mail prices must update their lists against change-of-address records within 95 days before the mailing date, using an approved method such as NCOALink [3]. That is a floor, not a standard. In B2B the harder problem is that an office address tells you where a company receives mail, not where your contact works. Store the mailing address as a property with its source, its last verification date, and whether the person confirmed it (Chapters 10 and 11). If the only address is one an enrichment vendor supplied, treat it as a hypothesis (Chapter 7).

**The right person at that address.** Identity resolution matters more on paper because the piece outlives the error. A booklet addressed to a contact who left in the spring does not bounce; it sits in someone else's inbox tray, addressed to a former colleague, full of account data (Chapter 9). Role freshness (Chapter 11) is a selection criterion, not a nice-to-have.

**Facts that survive the trip.** A print piece is decided days or weeks before it is read (Chapter 17 gives direct mail a latency budget of days). The data is frozen at a print cutoff. So choose content facts with a long half-life: the year's totals, the plan they are on, the region they expanded into. Avoid anything framed as recent ("last week your team...") and anything volatile (open ticket counts, usage this month). "As of" dates belong on the page.

**Suppression state, twice.** Consent and objections, open escalations, legal holds, and contact budgets are hard constraints (Chapter 14), checked at selection and checked again at the drop date, because the list you selected on Monday is not the list that should be mailed on Friday.

The memory shape follows the Boundary Rule from Chapter 10. Selection runs on typed properties: renewal date, account value, open P1, address verified date, objection flag. Those go in a WHERE clause. Content draws on evidence-level memories with provenance: what the account achieved, which region they added, what the account manager wants to say. Those go in a paragraph. At the print cutoff, the system writes a **print manifest**: for each piece, the exact facts used, their sources and dates, the template version, and the decision reason. When a customer calls about something printed, the manifest is how you answer. In the self-hosted memory system I built, every superseded property value is kept with its valid-from and valid-to dates, so "what did we believe on the cutoff date" is a query, not archaeology.

## The action

**Generate zones, not pages.** A booklet is a set of typed zones (Chapter 15): the cover line, the account's year in numbers, one insight, the account manager's note, the call to action. Numeric zones are not generated at all; they are pulled from the warehouse by code and formatted by the template. In the governed personalization engine I built, a provider's bucket floor or a headcount that contradicts other evidence is marked untrusted and hidden before any surface sees it. Hide when uncertain: a blank beats a confident wrong number. The model writes only the prose zones, under a generation contract: allowed evidence, forbidden claims (prices, discounts, commitments, anything about the person rather than the account), length, tone, and a general fallback per zone. When evidence thins, specificity steps down (account, then segment, then approved static copy) before trust does; if nothing verifiable remains, the piece is not printed. The account manager's note is drafted for the manager to edit and approve; it carries their name, so it carries their approval.

**Compile for paper.** Chapter 15's channel compiler applies the surface's contract, and paper has a strict one. The outside of an envelope or a postcard is read by a mailroom, a receptionist, and anyone at the desk. Nothing but name, title, company, and address goes there. Inside, use only facts the recipient would expect Larkspur to know from running their account (Chapter 19's say-how-we-know test). Operational data from their own dashboard passes. Anything inferred about the person, anything from enrichment, and anything sensitive does not go on paper at all, because paper travels.

**Render to a print-ready file, then check that file.** Variable data printing produces one file in which every page can differ; PDF/VT (ISO 16612-2) is the standard exchange format for it [4]. QA runs on the rendered PDF, not on the database rows (Chapter 20), because the defects that matter live in rendering: an overflowed text box, a missing image, a literal `{fleet_size}`, a chart on the wrong account. Once checked, the file is frozen with its manifest. If the transfer to the printer fails, resend that exact file; never regenerate, because a second generation is a different set of pieces from the one you proofed.

**Make the response path a continuation.** A QR code or personalized URL (PURL) should open a page that continues from the booklet, not a generic homepage (Playbook M1). Two rules. The link must use an unguessable token, not the person's name: a URL like `/j-morales` lets anyone who guesses a name see a page built from someone's account data. And the page should show nothing beyond what was printed until the visitor signs in or confirms who they are. The chain should continue past the page: a scan is an engagement signal, written back to the account's memory, that can trigger the account manager's alert or the next email.

**Reserve the expensive hour for the signal.** The booklet is the cheap wide pass. The account manager's call is the expensive one, and it goes first to the accounts that scan, call, or reply, and to the 400 routed accounts that never got a booklet. The expensive hour is not spent on the guess; it is reserved for the signal (Chapter 4).

### The design: print as one more compiled surface

The engine I built already runs, for web pages, email, and seller briefs, the parts print needs: account memory, campaign governance, typed zones with fallbacks, trust rules, rendering safety, and campaign-wide QA. Print on top of it is a design, not a shipped product. The first version I would build:

1. **An approved print template**, designed by people. Not AI graphic design.
2. **Declared zones:** headline, account framing, the letter, approved proof, call to action, QR code.
3. **The same context and governance** as the digital surfaces, so the booklet cannot contradict the landing page.
4. **Generate or select zone content** under tighter character budgets and paper's stricter privacy rules.
5. **Deterministic preflight** with stricter thresholds and approvals than web copy, which can be fixed tonight.
6. **A print-ready render** (PDF/VT), checked as rendered.
7. **Fulfillment** of the frozen file, after the release gate.

Print is not a separate marketing tool. It is one more surface compiled from the same governed context.

### From the field: a personalized print deployment

## What can go wrong

**Failure story: The Unrecallable Batch.** Larkspur's first attempt, a year earlier, was a printed renewal letter to 2,400 accounts. The team reviewed 20 proofs; all were good. The file went to the printer on a Thursday. By the following week, three things were in customers' hands. On 212 letters, the "vehicles in your fleet" figure read 10,001, a data provider's bucket floor that no rule had rejected. Thirty-one letters went to contacts who had left their companies; one congratulated an operations lead who had left after a public dispute on "your team's best year yet," and her former colleagues opened it. And six went to accounts with open priority-one escalations, because the suppression list was pulled at selection and never refreshed before the drop.

Every one of those defects would have been caught by the checks in Chapter 20. In email, the team would have fixed the template and sent a correction within the hour. On paper there is no correction. The letters sat on desks for weeks.

An email error is a correction. A print error is a keepsake.

The pre-print protocol that replaced it:

1. **Deterministic checks on 100% of rendered pages:** placeholder text, overflow, rejected values (bucket floors, impossible ranges), numbers matched against the warehouse, links and QR codes resolved and tested.
2. **Semantic review** by an AI judge on every flagged piece plus a stratified sample of clean ones, against the generation contract (Chapter 20).
3. **Human proof of a stratified sample**, including the thin-data accounts and every template variant, on paper, because color, cropping, and legibility look different printed.
4. **Suppression and identity refresh at the drop date,** not only at selection. Anything that changed is pulled.
5. **Seed pieces** addressed to internal staff in several regions, to see what actually arrives and when.
6. **A release gate.** Sending the file to the printer is the irreversible action in this channel. It passes a gate that can say no, owned by a named person (Chapter 14's Consequence-Gated Autonomy; Chapter 18). Make the verdict explicit (GO, GO WITH FIXES, NO GO); in my engine's campaign QA, a blocker gets an independent, skeptical second check before it can force NO GO.

![An email error can be fixed and corrected within the hour; a print error has no correction and sits on desks for weeks. So five checks run before release: deterministic checks on 100% of rendered pages, semantic review by an AI judge of flagged pieces plus a stratified sample, human proof on paper including thin-data accounts and every variant, a suppression and identity refresh at the drop date, and seed pieces to internal staff. All of them feed a sixth step, the release gate: sending the file to the printer is the irreversible action, and a named owner who can say no approves it.](/images/handbook/pb-m3-pre-print-gate.svg)

*Figure M3.2. Print cannot be recalled, so every check runs before a named owner releases the file.*

Other risks, briefly:

- **Consent is different, not absent.** In the EU, postal direct marketing can rest on legitimate interests (GDPR Recital 47), unlike most email marketing, but the right to object to direct marketing is absolute (Article 21), and an objection must suppress print as fast as it suppresses email [5]. Keep one objection flag that every channel reads. Not legal advice; see Chapter 19's dated notes.
- **The handwritten illusion.** Machine "handwriting" signed by a manager who never saw the note is a trust debt. If it carries a person's name, that person approves it.

## How you will know

**Hold out at random, before selection runs out.** From the accounts the rule selected, Larkspur held back a random group that received nothing in print (the account manager could still call if the account signaled). The comparison is selected-and-mailed versus selected-and-held-out. Comparing responders with non-responders, or this year with last year, measures selection, not effect (Chapter 21).

**Measure outcomes, not scans.** QR and PURL tracking is now the norm: in the ANA's 2023 survey, 82% of those who track direct-mail response use QR codes, personalized URLs, or another online mechanism [6]. That is attribution, not incrementality. It undercounts (people call, type the main address, or mention it to their account manager a month later) and credits the piece for Sure Things who would have renewed anyway. Use renewal and expansion rates by arm, with matchback of all responses, and track complaints and objection requests as a guardrail.

**Size it honestly.** With 1,500 accounts per arm and an 85% baseline renewal rate, the smallest difference you can reliably detect (5% significance, 80% power) is roughly 3.5 percentage points. That is a large effect for one mailing. Measure a nearer, more frequent outcome (renewal conversations within 30 days), pool quarters, or say plainly that the program cannot yet prove its effect.

**What the evidence does and does not say.** Print is easy to oversell, and the most quoted numbers are weaker than they look.

- The ANA 2023 Response Rate Report ranked direct mail to house lists first on ROI among the media it covered, at 161%, against 34% for mail to prospect lists, with average response rates of 15.6% (house) and 10.8% (prospect) [6]. It is a self-reported online survey of 250 respondents, fielded July to November 2023, and each of those direct-mail figures rests on 22 to 26 answers, several flagged "small base" by the ANA itself, with many respondents giving estimates rather than actual metrics. Read the direction: mail to people who already know you tends to outperform mail to strangers. Do not plan a budget on the level.
- Postal neuromarketing studies are supportive, not decisive. Canada Post's 2015 study (270 participants, mock brands, commissioned by Canada Post) reported that direct mail took 21% less cognitive effort to process and produced 70% higher brand recall than digital [7]. The USPS Office of Inspector General's 2015 study with Temple University (56 participants in the exposure phase, 39 in the brain-imaging phase) found that digital ads captured attention faster while physical ads were remembered better and triggered more activity associated with valuation; it also found that participants' self-reported attitudes showed little difference between the formats [8]. Small samples, laboratory settings, and sponsors with an interest in the answer.
- I found no peer-reviewed field experiment measuring the lift of variable-data printing or PURLs themselves. Claims such as "PURLs lift response 500%" have no traceable primary source; I do not use them.

The defensible evidence comes from adjacent field experiments, and it points in a useful direction. Adding a recipient's name to an email subject line raised opens and reduced unsubscribes in randomized trials (Chapter 1) [9]: shallow personalization works a little. More instructive for print is a fundraising experiment by Adena and Huck, who mailed 10,004 letters for an opera house's youth program. Setting each recipient's matching threshold relative to their own past or predicted giving outperformed uniform thresholds; for past donors, random uniform thresholds had no detectable effect on returns [10]. What worked was not the name on the letter. It was that the letter carried a decision made for that person.

Personalize the decision the piece carries, not the decoration on it.

## Reader Q&A

**Is print worth it for a B2B software company?** For a selected set of existing customers at a meaningful moment (renewal, a milestone, a region launch), it can be, and the evidence, weak as it is, favors house lists over prospecting. Prove it with a holdout before scaling.

**Should the cover say the customer's name in large type?** Name personalization is the cheapest rung (Chapter 4) and helps a little. On paper it also tells everyone at the desk who the piece is for. A verified fact about their year does more work than their name in 72-point type.

**Can AI design each piece?** Let it choose among approved layouts and modules and fill prose zones, never free-form layouts or images: you cannot proof what you cannot predict, or recall what you printed.

**Does this apply to other traditional channels?** Yes: packaging inserts, statements, printed event materials, outbound calls. Wherever a touch is costly or cannot be taken back: price the decision, verify before release, hold some people out.

## For your AI

```yaml
playbook: M3
title: "Marketing III: The Physical Channel"
question: "When each touch costs real money, who deserves one and what should it say?"
foundation_links: [4, 7, 9, 10, 11, 14, 15, 17, 18, 19, 20, 21]
concepts:
  - name: Priced decision
    definition: "Send a physical piece only if expected incremental change in outcome times its value exceeds the full cost of the piece, including review time."
  - name: Sign over cost
    definition: "In high-value B2B the print cost rarely binds; whether the piece helps or harms (Sleeping Dogs, open escalations) decides who should receive it."
  - name: Shelf life of an error
    definition: "A printed mistake cannot be recalled or corrected, so verification must be complete before release."
  - name: Print manifest
    definition: "A per-piece record written at the print cutoff: facts used with sources and dates, template version, and decision reason."
  - name: Paper is a public surface
    definition: "Printed content can be read by people other than the addressee; the privacy contract for print is stricter than for email."
  - name: Long half-life facts
    definition: "Print content should rely on facts that remain true between data freeze and delivery, with as-of dates."
  - name: Print as a compiled surface
    definition: "Design, not shipped: approved template, declared zones, shared context and governance, zone content, deterministic preflight, print-ready render, fulfillment of the frozen file."
  - name: Hide when uncertain
    definition: "A suspicious value (bucket floor, implausible range, contradicted by evidence) is left off the piece."
decision_rules:
  - if: "a numeric value is suspicious or contradicts other evidence"
    then: "hide it and step the zone down to segment or static copy; if nothing verifiable remains, do not print"
  - if: "transfer of an approved print file to the printer fails"
    then: "resend the exact frozen file; never regenerate content after proofing"
  - if: "expected change x outcome value <= cost per piece"
    then: "do not mail; consider email, a general piece, or nothing"
  - if: "the account has an open escalation, legal hold, or recent objection"
    then: "suppress print; route to the account owner if the account is high value"
  - if: "the mailing address or contact role has not been verified within the freshness policy"
    then: "do not print account-specific content; verify, fall back to a general piece, or skip"
  - if: "a fact is volatile (current usage, open tickets, 'recent' events)"
    then: "exclude it from print content"
  - if: "content would appear on the outside of the piece"
    then: "allow only name, title, company, and address"
  - if: "a print file is ready for release"
    then: "require 100% deterministic checks on rendered pages, semantic review of flags plus a stratified sample, human proof of a sample, suppression refresh at drop date, and a named release approver"
  - if: "a QR code or PURL links to a personalized page"
    then: "use an unguessable token and show nothing beyond the printed content until the visitor is verified"
  - if: "a note carries a person's name or signature"
    then: "that person approves it"
assessment_questions:
  - "What does one physical piece cost you all in, and how is the recipient list chosen today?"
  - "When was each mailing address last verified, and against what source?"
  - "Is suppression (objections, escalations, departed contacts) refreshed between selection and drop date?"
  - "What checks run on the rendered print file, and on what share of pages?"
  - "Do you hold out a random share of selected recipients, and what effect size can your volume detect?"
  - "Which facts are allowed on paper, and who decided?"
patterns: [Priced Decision, Print Manifest, Pre-Print Release Gate, Do-Nothing Option, Wide Then Deep, Channel Compilers, Frozen Print File]
anti_patterns: [The Unrecallable Batch, Mail-Everyone-Eligible, Scan-Rate-as-Lift, Guessable PURL, The Handwritten Illusion]
metrics:
  primary: "incremental renewal or expansion rate, mailed vs randomized holdout"
  process: ["pre-print defect rate by layer", "undeliverable rate", "share of pieces falling back to general content"]
  guardrails: ["complaints and direct-marketing objections", "post-drop defects reported by recipients or staff"]
maturity_dimension: experience_generation
```

## References

1. United States Postal Service, Price List Notice 123, effective 2026-07-12. USPS Marketing Mail commercial automation letters: $0.374 (5-digit, DSCF entry) to $0.467 (mixed AADC). https://pe.usps.com/text/dmm300/notice123.htm
2. Radcliffe, N. J., and Simpson, R. (2008). "Identifying who can be saved and who will be driven away by retention activity." *Journal of Telecommunications Management* 1(2). https://stochasticsolutions.com/pdf/SavedAndDrivenAway.pdf
3. USPS PostalPro, "Move Update" (accessed 2026-09-26). https://postalpro.usps.com/address-quality/moveupdate
4. ISO 16612-2:2010, "Graphic technology: Variable data exchange, Part 2: Using PDF/X-4 and PDF/X-5 (PDF/VT-1 and PDF/VT-2)." https://www.iso.org/standard/46428.html
5. Regulation (EU) 2016/679 (GDPR), Recital 47 and Article 21. https://gdpr-info.eu/recitals/no-47/ ; https://gdpr-info.eu/art-21-gdpr/
6. ANA, "Response Rate Report 2023: Performance and Cost Metrics," published 2024-02-22. Self-reported online survey, n = 250, fielded July to November 2023. https://www.ana.net/miccontent/show/id/rr-2024-02-ana-response-rate-report-2023 ; PDF copy: https://theworldsgreatestmarketing.com/wp-content/uploads/2025/04/rr-2024-02-ana-response-rate-report-2023.pdf
7. Canada Post, "Direct mail beats digital advertising in driving consumers to act: neuromarketing study," 2015-08-27. Conducted by True Impact; commissioned by Canada Post. https://www.canadapost-postescanada.ca/cpc/en/our-company/news-and-media/corporate-news/news-release/2015-08-27-direct-mail-beats-digital-advertising-in-driving-consumers-to-act-neuromarketing-study
8. USPS Office of Inspector General, "Enhancing the Value of Mail: The Human Response," RARC-WP-15-012, 2015-06-15, with Temple University's Center for Neural Decision Making. https://www.uspsoig.gov/sites/default/files/reports/2023-01/rarc-wp-15-012.pdf
9. Sahni, N. S., Wheeler, S. C., and Chintagunta, P. (2018). "Personalization in Email Marketing: The Role of Noninformative Advertising Content." *Marketing Science* 37(2). https://pubsonline.informs.org/doi/abs/10.1287/mksc.2017.1066
10. Adena, M., and Huck, S. (2022). "Personalized fundraising: A field experiment on threshold matching of donations." *Journal of Economic Behavior & Organization* 200:1-20. https://www.sciencedirect.com/science/article/abs/pii/S0167268122001585 ; working paper (WZB, verified text): https://www.econstor.eu/bitstream/10419/256795/1/1801895139.pdf

---

# Playbook S1. New Customers

*The question: How do we open a relationship without sounding like every other AI message?*

> **Questions this playbook answers**
> - Why does most AI-written outreach read as fake, even when the writing is good?
> - What should an AI SDR actually decide, and what should it never decide?
> - What do we need to know about a prospect before we are allowed to be specific?
> - Which deliverability and consent rules shape cold outreach now?
> - How do we prove researched outreach beats volume, and how long will it take?

## The moment

*Larkspur Systems is a fictional composite company used throughout this handbook. Its numbers are illustrative.*

Six weeks after Larkspur paused its AI SDR pilot (Chapter 6), the CRO put two emails on the conference-room screen.

The first was the pilot's best-rated message from the spring: 140 words, warm, fluent, with a second sentence congratulating an HVAC operations director on an acquisition her company never made (Chapter 15). The second was a note one of Larkspur's senior SDRs had written by hand two years earlier to a regional utility. Four sentences. It said that the utility had just posted three dispatcher roles for a new service territory, that bringing a second dispatch team online before storm season is where most fleets lose a month, and asked whether that was on her list. No flattery, no "companies like yours." She had replied the same afternoon.

The pilot had sent to 300 prospect accounts with a name, a title, an industry, and a company size (Chapter 4). Reply rates fell below the plain template the team had used before. The vendor's proposed fix was to triple volume across the full prospect list.

"Which of these," the CRO asked, "do we want a machine to write 20,000 times?"

The answer turned out to be neither. What Larkspur wanted a machine to do was the part of the second email that happened before any writing: noticing that the utility had a reason to talk now, and that the SDR could say so truthfully.

## The decision

### Why most AI outreach sounds fake

The usual diagnosis is the writing: too polished, too long, too many adjectives. That is almost never the reason. Readers do not detect a model. They detect a message that was not written for their situation, and three structural causes produce it.

**Decorative specificity.** Thin data plus a model instructed to "personalize" yields details that prove research was done without changing what is said: the podcast you appeared on, the post you liked, your alma mater. The detail decorates. It does not give a reason to talk. Detail included only to prove the sender found it makes a message worse, not better.

**No reason now.** Most outbound exists because a sequence reached step three, not because anything changed for the buyer. The model is asked to invent relevance for a send that had none. It obliges, fluently.

**The most skeptical channel, with the least context.** A stranger's inbox is where tolerance for error is lowest (Chapter 6). And buyers arrive late: 6sense's 2025 survey of nearly 4,000 B2B buyers found first contact with sellers happening about 61% of the way through the journey, with buyers initiating 79% of those engagements and choosing from their day-one shortlist 95% of the time [1]. Gartner, surveying 632 B2B buyers in late 2024, found 61% preferred a rep-free buying experience and 73% actively avoid suppliers who send irrelevant outreach [2]. Its analyst put it plainly: "Bad prospecting actively damages relationships with potential customers" [2]. These are vendor and analyst surveys of stated behavior; read the direction, not the decimal.

Taken together, the first message's job is small, the penalty for a bad one is lasting avoidance, and fluency does nothing to fix either. Put more bluntly: a plain template, written by hand, that names a real need a specific person actually has will often beat a fluent, machine-written message built on a shallow understanding of them. The reader does not reward eloquence. They reward being understood.

### The decision is whether, not how

Here is the sentence this playbook turns on. **In prospecting, research is not there to make the message sound personal. It is there to decide whether to send one at all.**

For each account, and then each person, the system chooses among:

| Option | When it is right |
|---|---|
| **Do nothing** | Weak fit, no reason now, or no lawful basis to contact. The default. |
| **Watch** | Good fit, no trigger. Keep the account in memory and re-check on a schedule. |
| **Hand to marketing** | Fit is plausible but no individual has a reason to hear from sales; account-level programs (Playbook M2) do better. |
| **Send at account or industry tier** | Fit and a verified account-level reason; the person's own situation is unknown. |
| **Send at person tier** | Fit, a verified reason now, and a corroborated fact about this person's role or remit. |
| **Route to a human** | High-value account, or evidence complex enough that judgment matters more than speed. The system prepares a brief and a draft; a seller decides. |

Five inputs drive the choice: **fit** (does the account resemble customers who succeed with you, not merely customers who bought), **a reason now** (a dated, sourced trigger), **the right person** (a role that owns the problem the trigger creates), **permission** (jurisdiction, suppression, consent basis, existing relationship), and **budget** (how many touches this account and this domain can absorb this month). No reason now, no message. That single rule removed about three quarters of Larkspur's planned sends. It is the rule most AI SDR deployments do not have.

![Five inputs on the left: fit (resembles customers who succeed), a reason now (a dated, sourced trigger, highlighted), the right person (owns the problem the trigger creates), permission (jurisdiction, suppression, consent), and budget (touches the account and domain can absorb). They choose among six options, per account then per person: do nothing (the default when there is no reason now), watch, hand to marketing, send at account or industry tier, send at person tier, and route to a human with a brief and draft for high-value accounts. The rule underneath: no reason now, no message, which removed about three quarters of Larkspur's (fictional) planned sends.](/images/handbook/pb-s1-send-decision.svg)

*Figure S1.1. Research decides whether to send; without a reason now, the answer is do nothing.*

## What you need to know

### Research belongs to the account, not the lead

Most enrichment pipelines are built around the record: a lead arrives, it is researched, a message is drafted, and the same company is researched again for every employee in the funnel. Chapter 5's Research Once per Entity fixes that: keyed to the company with a seven-day freshness window, research-rich records rose from 60.3% to 93.5% on a pipeline I run while spend fell [3].

Research also has a second job most stacks skip. It adds what you lack, and it challenges what you already believe. A field saying fifty employees next to current sources describing a national operation is not a fact; it is a disagreement. The working rule: **a fact from a single source is a hypothesis; corroborated by an independent source, it is something you can put in front of a stranger** [4]. Chapter 8 covers the mechanics.

### Signals, ranked by what they are allowed to do

Not every signal may appear in the message. Some may only decide timing.

| Signal | Example at Larkspur | May decide | May be quoted? |
|---|---|---|---|
| Firmographic (rung 2) | Regional utility, 300 vehicles | Fit, template family | Rarely; it is not news to them |
| Technographic | Uses a legacy dispatch tool | Fit, value claim | Only if public and corroborated |
| Company trigger event | Three dispatcher roles posted for a new territory | Whether and when to send | Yes, with date and source |
| Role and remit | Director of field operations, owns dispatch | Who to contact, what to say | Yes, if current |
| Engagement with you | Two people from the account read the routing guide | Timing, routing to a human | No. Nobody expects a stranger to narrate their browsing (Chapter 19) |
| Personal, non-professional | Moved cities, old conference talk under a former employer | Nothing | Never (the Creepy Reveal, Chapter 1) |

The last two rows are where accurate outreach becomes invasive outreach. Chapter 1's rule applies: facts from outside the relationship may inform whether and when you reach out; they are quoted back only when they are professional, recent, and something the person would expect a vendor to know.

### The memory shape

A prospect is a pair of linked records in the same Unified Customer Memory that serves customers (Chapter 10), not a row in a sequencer. Typed properties hold what code must filter and enforce; memories hold the evidence; a short synthesis holds the "why now" hypothesis.

```yaml
account: regional-utility-0417
properties:
  customer_status: prospect          # checked against CRM before every send
  icp_fit: {score: 0.82, model: fit-v3, computed_at: 2026-09-20}
  fleet_size_band: 250-500
  trigger_events:
    - type: territory_expansion
      evidence: "3 dispatcher postings, new service territory"
      sources: [careers_page, job_board]   # two sources: corroborated
      observed_at: 2026-09-18
      entity_match_confidence: 0.97
  contact_policy: {last_touch: null, suppressed: false, jurisdiction: US}
synthesis:
  why_now: "Standing up a second dispatch team before peak season."
  confidence: medium
people:
  - role: director_field_operations
    remit_verified: true
    consent_basis: none_required_us_b2b_email   # set by policy, not the model
```

Every trigger carries source, date, and an entity-match confidence, because the acquisition email came from a real article about the wrong company matched by name alone (Chapter 9). Freshness is per signal (Chapter 11): a job posting is stale in weeks, a role in months. And `customer_status` is read before every send, because the fastest way to look automated is to prospect an existing customer, or to thank a former customer "as a valued customer" (Chapter 20).

Two more identity rules from the governed personalization engine I built belong in the record. A prospect writing from a personal mailbox does not inherit the firmographics of the mailbox provider: an address at a freemail domain is flagged, company-specific research is suppressed, and the tier drops. And a firmographic value that fails plausibility (an invalid value, a provider's bucket artifact, a tiny headcount beside strong enterprise markers, a contradiction with current research) is marked untrusted and hidden, not quoted. The policy is hide when uncertain. In a first touch, a missing number costs less than a confident wrong one [18].

## The action

### What the system does, and what it does not

An AI SDR done right is not one agent with an inbox. It is a pipeline in which code orchestrates, models read and write, and people own the conversation.

1. **Research, asynchronously.** Account research takes minutes, not milliseconds, and should (Chapter 17). It runs on a schedule and on triggers, writes to memory, and never runs at send time.
2. **Decide in code, with a model's help.** Fit scoring and trigger detection can use models; the send decision, suppression, jurisdiction, and volume caps are deterministic. Let the model reason; let the code decide what is allowed (Chapter 14).
3. **Draft under a contract.** The message is a few typed zones (subject, reason now, one value claim, one question), each with allowed evidence, forbidden claims, and a fallback (Chapter 15). The tier (person, account, industry) is computed from evidence before the model is called.
4. **Route by value.** For Larkspur's top 300 accounts, the output is a researched brief with sources and a draft; a seller edits and sends. Below that, the account and industry tiers send automatically within caps. Every human edit is logged as a label for what the system got wrong.
5. **Classify replies, then hand off.** One model call classifies each reply (positive, question, not now, wrong person, unsubscribe, out of office); code executes the branch. A positive reply goes to a human the same day. The machine opens; it does not negotiate.

![A five-step pipeline. One, research runs on a schedule and on triggers, takes minutes, writes to memory, and never runs at send time. Two, decide in code with a model's help: send, suppression, jurisdiction, and caps are deterministic. Three, draft under a contract of typed zones (subject, reason now, one claim, one question), with the tier set before the model is called. Four, route by value: for the top 300 accounts a seller edits and sends, the rest send automatically at account or industry tier within caps, and every human edit is logged as a label. Five, one model call classifies replies, code runs the branch, and positive replies go to a human the same day.](/images/handbook/pb-s1-sdr-pipeline.svg)

*Figure S1.2. Code orchestrates, models read and write, and people own the conversation.*

Here is the rewritten Larkspur note at account tier, where the person's situation is unknown but the company's is verified:

> Subject: dispatch for the new territory
>
> I saw the three dispatcher openings for your new service territory. Standing up a second dispatch team is usually where routing gets done by hand for a while, which is the problem we work on. Is that on your list this quarter, or is it already handled?

It is plain. It is true. It makes one claim, cites a public fact the reader knows is public, and asks a question the reader can answer in one word. That is the whole craft.

### The brief is for the seller; the email is for the buyer

For the top tier, the artifact that matters is the seller's brief, and it needs a boundary most stacks do not enforce. In the engine I built, one account's research and score (with rationale, priority, and a suggested approach) feed a brief, a seller playbook, and an email sequence, all from one governed context. The playbook may carry hypotheses and competitive context that must never reach the buyer, so rep-internal intelligence and customer-facing copy are separate outputs, and an explicit serialization layer decides what leaves the system.

Four checks earned their place. Code flags a score that contradicts its own label (88 with a band of LOW). A playbook whose load-bearing sections are empty is not done because the generation step reported success; it is retried with bounded backoff. When rich enrichment fails near a deadline, the engine writes a deterministic, lower-specificity brief instead of inventing detail or leaving the record stuck. And in some of its content engines, a first email set is scored in code (blank, wrong length, repetitive opening, incomplete ending, missing required proof); only a flawed set triggers a second generation, and the lower-penalty set wins, so quality improves without doubling model cost. If one message in a sequence fails, only that slot is regenerated [18].

The next step I am building, a design and not yet a result, is one reusable brief template set up by configuration rather than a code fork per campaign. Verified facts render straight from structured research, each with its source and a freshness date; the model writes mainly the customer-safe drafts. The brief confirms which account it describes, shows a thin-data state instead of padding when research is sparse, and lets a seller flag a wrong fact [18].

### What the contract forbids

The governance list for outbound matters as much as the prompt. Larkspur's contract forbids, in code: implying a prior relationship or conversation that did not happen; fake reply threads ("Re:" on a first touch); invented mutual connections; urgency or scarcity language; any number about the prospect that is not sourced; customer-status claims; and follow-ups that add no new information. Two follow-ups at most, each carrying something the first did not, or the sequence ends.

It also bounds what the model may offer. In the engine I built, each campaign carries a capability map, so the model is asked which approved capability fits this buyer, not what solution would sound good; that stops it inventing a feature because it fits the problem. Some of its content engines also restrict numeric proof to an approved list of figures. And without a recorded relationship, "your renewal is coming up" is reframed as an evaluation, never stated as ownership [18].

### The rules that shape the channel

As of September 2026, and not legal advice (details belong in dated companion notes; Chapter 19):

- **Gmail** treats any domain sending 5,000 or more messages a day to personal Gmail accounts as a bulk sender, permanently. Bulk senders need SPF, DKIM, and DMARC with alignment, one-click unsubscribe for marketing messages, and spam rates reported in Postmaster Tools below 0.30% [5]. Google recommends honoring unsubscribes within 48 hours and, from November 2025, began rejecting non-compliant traffic [6]. Yahoo and Microsoft have similar rules [7]. Cold volume from your primary domain puts every invoice and renewal email on that domain at risk.
- **CAN-SPAM** (US) is opt-out: honest headers and subject lines, a working opt-out honored within 10 business days, and penalties up to $53,088 per violating email [8].
- **CASL** (Canada) is opt-in: commercial email generally needs express or implied consent, with implied consent time-limited, and penalties up to CAD $10 million for organizations [9].
- **EU and UK** B2B cold email rules vary by country under the ePrivacy framework, and the right to object to direct marketing under GDPR Article 21 is absolute [10]. Treat jurisdiction as a typed property, not a guess.
- **AI voice.** The FCC ruled in 2024 that AI-generated voices are "artificial" under the TCPA, so AI-voiced calls fall under its consent rules [11].
- **LinkedIn's** user agreement prohibits bots or unauthorized automated methods to "send or redirect messages" or add contacts [12]. Draft there if you like; a person sends.

## What can go wrong

### Failure story: The Volume Rescue

Larkspur's vendor had a fix for falling reply rates: send three times as many, across the full prospect list rather than the 300 pilot accounts. In the fictional telling, the tripled volume went out from Larkspur's primary domain, spam complaints climbed past Gmail's limit within a week, and the first casualty was not the pilot. It was a batch of renewal notices and invoices from the same domain that landed in customers' spam folders. Meetings booked per week rose for a month. Replies saying "take me off your list" rose faster. The anti-pattern: **volume as a fix for irrelevance.** A message that should not have been sent does not improve by being sent more often.

### Evaluate the category on your own data

The AI SDR market is young and its claims outrun its evidence. In March 2025, TechCrunch reported allegations that 11x, a venture-backed AI SDR company, listed customers it did not have, including ZoomInfo, which said its trial "performed significantly worse than our SDR employees," and that former employees disputed how its revenue figures were counted. 11x said it had removed inaccurate customer mentions, attributed the rest to human error, and defended its revenue metric as one investors reviewed [13]. These are allegations, and the company disputes them. The lesson does not depend on who is right: logos and demos are not evidence. A holdout on your own accounts is.

### Governance risks, named

- **The Hallucinated Detail** (Chapter 15): an unsourced or mismatched event. Control: entity-match threshold, grounding check, whole-zone fallback.
- **The Creepy Reveal** (Chapter 1): true, invasive, unexpected. Control: signal-use table in code.
- **The customer prospected as a stranger**: identity failure between CRM and sequencer (Chapter 9). Control: status check at send time, not at list build.
- **Contradicting yourself**: Gartner found 69% of buyers see inconsistencies between a supplier's website and its sellers [2]. Control: one governed context and one claims library for every surface (Playbook P9, Chapter 18).
- **An open promise ignored**: a contact at a customer's parent company under an outreach hold (Chapter 18). Control: holds are typed and read by every sender.

### What this does not do

Research does not create demand. If 6sense is even roughly right that most buyers choose from a shortlist formed before they talk to anyone [1], researched outbound works mainly by reaching the minority of accounts with a live reason, and by not burning the rest. It will send fewer messages and may book fewer meetings in the first month. That is the cost of stopping the damage, and you should measure it rather than argue about it.

It also qualifies something I have written before. In an earlier article I described an outbound email that addressed a prospect's publicly stated frustration with her current vendor [14]. For an account where a relationship exists, that is right. For a stranger, the better use of that signal is timing: it says now, and the message says something true and general about the problem. The deeper the evidence, the more carefully it should be spent in a first touch.

And the rep-free trend does not mean sellers are obsolete. Gartner's 2026 buyer survey found 67% prefer rep-free buying [15], while a companion survey found 69% turn to sales reps to validate AI-generated insights [16]. Buyers want fewer, better human moments. Automating all of them misreads the data; so does ignoring it.

## How you will know

**The metric is qualified meetings per 1,000 accounts considered, not replies per send.** Sends are a cost. Measuring per account considered rewards the system for deciding well, including deciding not to send. Track the harm side with equal weight: spam complaint rate by domain, unsubscribes, "wrong company" and "remove me" replies per 1,000, and domain reputation in Postmaster Tools.

**The test design.** Randomize at the account level, not the contact level, so two people at one company never see two treatments. Three arms: the best human plain template (the control the pilot never had), researched and tiered outreach, and a no-outbound holdout. The holdout matters because buyers initiate most engagements [1]; some meetings outbound "creates" would have arrived inbound anyway.

**The arithmetic.** Suppose the qualified-meeting rate is 1.0% of accounts and you hope to reach 1.5%. A standard two-sided test at 5% significance and 80% power needs about 7,750 accounts per arm. Larkspur's 300-account pilot could not have detected that lift, which is why it proved nothing except that it was hurting replies. Pipeline and win rate arrive a quarter or two later; reputation damage can arrive faster. Chapter 21 covers holdouts, incrementality, and why reply rate is a proxy that can move in the wrong direction from revenue.

## Reader Q&A

**Should we buy an AI SDR or build one?** Ask any vendor four questions: where does research live and is it shared with the rest of the company; who decides whether to send; what does it write when evidence is thin; and will it run against a holdout on our accounts. Buy or build by the answers, not the demo (Chapter 22).

**Do we still need human SDRs?** Yes, fewer and differently. They work the top accounts from researched briefs, own every positive reply, and label what the system gets wrong.

**Should we disclose that AI drafted the message?** A message a person reviews and sends is theirs. An AI agent conversing with a prospect is different: in the EU, Article 50 of the AI Act has required disclosure of AI interaction since August 2026 [17]. Whatever the law, never let the system imply a human wrote something no human saw.

**How many follow-ups?** As many as you have new, true things to say, capped at two for strangers. "Just bumping this" is a reason to end the sequence.

**Can we use website visits to time outreach?** For timing and routing, carefully and within your consent notices. Never in the message.

## For your AI

```yaml
playbook: S1
title: "Sales I: New Customers"
question: "How do we open a relationship without sounding like every other AI message?"
links: {chapters: [1, 4, 6, 8, 9, 10, 11, 12, 14, 15, 17, 18, 19, 20, 21], playbooks: [M2, P9]}
concepts:
  - name: Reason Now
    definition: "A dated, sourced, account-level trigger that gives a stranger a reason to hear from you; required before any outbound send."
  - name: Decorative Specificity
    definition: "Personal details that prove research was done without changing what is said; the main reason AI outreach reads as fake."
  - name: Entity-Level Research
    definition: "Research cached per company with a freshness window and reused by every record and step, instead of re-researched per lead."
  - name: Signal-Use Table
    definition: "Per signal class, what it may decide (fit, timing, routing) and whether it may be quoted in a message."
  - name: Send Decision
    definition: "Per account and person: do nothing, watch, hand to marketing, send at industry/account/person tier, or route to a human."
  - name: Rep-Internal Boundary
    definition: "Seller-facing intelligence (hypotheses, competitive context) and buyer-facing copy are separate outputs; an explicit serializer decides what leaves the system."
  - name: Hide When Uncertain
    definition: "A firmographic value that fails plausibility, or company data inferred from a freemail address, is marked untrusted and withheld rather than quoted."
decision_rules:
  - if: "no verified trigger event with source and date exists for the account"
    then: "do not send; watch or hand to marketing"
  - if: "a trigger's entity match confidence is below threshold or it has a single source"
    then: "treat it as a hypothesis; it may decide timing but not appear in copy"
  - if: "a signal is engagement with our properties or personal and non-professional"
    then: "use for timing or routing only; never quote it"
  - if: "customer_status at send time is not 'prospect' or an outreach hold exists"
    then: "suppress and route to the account owner"
  - if: "the prospect's address is on a freemail domain, or a firmographic value fails plausibility checks"
    then: "suppress company-specific research or the value; drop a tier; never quote it"
  - if: "the model recommends a capability not on the campaign's capability map, or a figure not on the approved list"
    then: "reject the draft zone and use its fallback"
  - if: "rich enrichment fails near the deadline, or a brief's load-bearing sections are empty"
    then: "retry with bounded backoff, then write a deterministic lower-specificity brief; never invent detail"
  - if: "account value is in the top tier"
    then: "produce a brief and draft for a human to send"
  - if: "a follow-up adds no new verified information"
    then: "end the sequence"
  - if: "reply is classified positive or a question"
    then: "hand to a human the same day; the machine does not negotiate"
  - if: "cold volume would push a domain past bulk-sender limits or spam rate nears 0.3%"
    then: "cut volume; never send cold outreach from the primary transactional domain"
assessment_questions:
  - "What share of your outbound sends has a dated, sourced reason now?"
  - "Is prospect research stored per company and shared, or re-run per lead?"
  - "Where is the send decision made: in code with caps and suppression, or inside the tool?"
  - "Which signals are allowed in copy, and which only for timing?"
  - "Which domain does cold outreach send from, and what is its spam rate?"
  - "Have you tested outbound against a plain-template control and a no-outbound holdout, randomized by account?"
  - "Which jurisdictions are your prospects in, and is that a typed field?"
patterns: [No Reason No Message, Research Once Per Entity, Tiered First Touch, Human Sends the Top Tier, Classify Then Hand Off, Separate Brief From Email, Conditional Best of Two]
anti_patterns: [Decorative Specificity, The Volume Rescue, The Hallucinated Detail, The Creepy Reveal, Customer Prospected as Stranger, Empty Quadrant Pilot]
metrics: [qualified_meetings_per_1000_accounts_considered, spam_complaint_rate_by_domain, remove_me_and_wrong_company_replies_per_1000, tier_distribution]
maturity_dimension: decisioning
```

## References

1. 6sense, "2025 B2B Buyer Experience Report" (nearly 4,000 buyers; first contact at 61% of journey; buyers initiate 79%; day-one shortlist wins 95%). Vendor research. https://6sense.com/science-of-b2b/buyer-experience-report-2025/
2. Gartner, "Gartner Sales Survey Finds 61% of B2B Buyers Prefer a Rep-Free Buying Experience," 2025-06-25 (n = 632, fielded Aug to Sep 2024; 73% avoid suppliers sending irrelevant outreach; 69% report website/seller inconsistencies). https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-sales-survey-finds-61-percent-of-b2b-buyers-prefer-a-rep-free-buying-experience
3. Taheri, H., "Research once" (production pipeline measurements, 2026-08), hamedtaheri.com.
4. Taheri, H., "Enterprise-grade accurate personalization at scale," hamedtaheri.com, 2026-07-13. https://hamedtaheri.com/articles/accurate-personalization-at-scale
5. Google, "Email sender guidelines," Google Workspace Admin Help, accessed 2026-09-26. https://support.google.com/a/answer/81126
6. Google, "Email sender guidelines FAQ," Gmail Help, accessed 2026-09-26. https://support.google.com/mail/answer/14229414
7. Yahoo and Microsoft bulk-sender requirements (Microsoft Outlook from 2025-05-05), per Handbook reference bank, refs-part6 item 25.
8. US Federal Trade Commission, "CAN-SPAM Act: A Compliance Guide for Business." https://www.ftc.gov/business-guidance/resources/can-spam-act-compliance-guide-business
9. CRTC, "Canada's Anti-Spam Legislation." https://crtc.gc.ca/eng/internet/anti.htm
10. Regulation (EU) 2016/679 (GDPR), Art. 21(2) to (3); Directive 2002/58/EC (ePrivacy), Art. 13. https://eur-lex.europa.eu/eli/reg/2016/679/oj
11. FCC Declaratory Ruling, FCC 24-17, 2024-02-08. https://docs.fcc.gov/public/attachments/FCC-24-17A1.pdf
12. LinkedIn User Agreement, Section 8.2, effective 2025-11-03, accessed 2026-09-26. https://www.linkedin.com/legal/user-agreement
13. TechCrunch, "a16z- and Benchmark-backed 11x has been claiming customers it doesn't have," 2025-03-24. Allegations; the company disputes them. https://techcrunch.com/2025/03/24/a16z-and-benchmark-backed-11x-has-been-claiming-customers-it-doesnt-have
14. Taheri, H., "The $450K Email Your AI Sent Wrong," hamedtaheri.com, 2026-03-14. https://hamedtaheri.com/articles/the-450k-email-your-ai-sent-wrong
15. Gartner, "Gartner Sales Survey Finds 67% of B2B Buyers Prefer a Rep-Free Experience," 2026-03-09. https://www.gartner.com/en/newsroom/press-releases/2026-03-09-gartner-sales-survey-finds-67-percent-of-b2b-buyers-prefer-a-rep-free-experience
16. Gartner, "Gartner Survey Finds 69% of B2B Buyers Turn to Sales Reps to Validate AI-Generated Insights," 2026-05-20 (n = 645, fielded Aug to Sep 2025). https://www.gartner.com/en/newsroom/press-releases/2026-05-20-gartner-survey-finds-sixty-nine-percent-of-b-two-b-buyers-turn-to-sales-reps-to-validate-ai-generated-insights
17. EU AI Act (Regulation 2024/1689), Art. 50, applicable from 2026-08-02. https://artificialintelligenceact.eu/article/50/
18. First-party: capabilities reference for a governed personalization engine built by the author (governance layer, known-lead pipeline, output surfaces, and identity and firmographic trust sections; live capabilities, plus one in-progress design direction labeled as such), 2026-08-25. Unpublished. [PRODUCT-NAMING: name the system here if Hamed decides to]

---

# Playbook S2. Growth and Return

*The question: What do we already know, and how do we use it to go deeper or win them back?*

> **Questions this playbook answers**
> - Which existing customers should we offer more to, and which should we leave alone for now?
> - When does cross-selling a second product help an account, and when does it lose us money?
> - How do we win back customers who left without sending everyone the same "we miss you" discount?
> - What does our memory need to hold about a customer who left?
> - Is it true that keeping a customer is five times cheaper than finding one?

## The moment

Larkspur Systems (a fictional company, a composite of many real ones) sells field-service and fleet software. On the first Monday of the quarter, revenue operations hands the sales team two lists. The first is 3,100 current accounts that score high on an expansion model: more seats, the route-optimization add-on, or the new parts-inventory module. The second is 2,300 accounts that cancelled in the last three years. The plan for both is the same: one well-written email per list, a limited-time offer, and a sequence of two follow-ups.

Pick one row from the second list. Ridgeway Pest Services (also fictional) ran 60 technicians on Larkspur for four years and cancelled eighteen months ago. The CRM says "Closed Lost: Churn." The reason field says "Other."

The real reason is in three places nobody joined. The cancellation call recording, where the operations director says the mobile app's offline mode failed at rural job sites and her technicians were writing work orders on paper. Fourteen support tickets about the same failure, two of them escalated. And the product changelog, which shows that Larkspur rebuilt offline mode in its spring release, eleven months after Ridgeway left.

The planned win-back email offers Ridgeway 25 percent off to come back. It does not mention offline mode, because the system that writes it does not know. So it asks a customer who left over a broken feature to return for a discount on the same broken feature, as far as she can tell. The one thing that might bring her back, "the problem you left over is fixed, and here is the evidence," sits unused in a transcript.

Now look at the first list. One of the 3,100 accounts has doubled its fleet and is at its seat limit: a real expansion moment. Another scores high because its usage spiked, and the spike is its dispatchers re-entering routes by hand after a sync failure. Same score, opposite situations.

Both lists are built from prediction. Neither is built from memory.

## The decision

Growth and return are three decisions, not one campaign. **Expand** sells more of what the account already uses (seats, tiers). **Cross-sell** introduces a product it does not use. **Return** tries to reacquire an account that left. For each account, the decision layer (Chapter 14) chooses among the same options: make an offer, route to an account manager, wait for a condition, ask a question, or do nothing. For former customers there is one more: suppress permanently, because some people asked not to hear from you and some relationships ended for reasons no offer fixes.

Here is the sentence this playbook turns on: **a growth or win-back message is only as good as the company's memory of what happened last time.** A propensity model knows who looks like a buyer. Only memory knows that this account already declined the add-on twice, that its champion left in June, or that it cancelled over a defect you have since fixed.

### Start by retiring the arithmetic everyone quotes

Two numbers justify most retention and expansion budgets: "acquiring a customer costs five times more than keeping one," and "a 5 percent increase in retention raises profits by 25 to 95 percent." Neither survives a look at its source.

A widely read 2014 *Harvard Business Review* piece by Amy Gallo states that acquiring a customer is "anywhere from five to 25 times more expensive than retaining an existing one," with no source, and that "increasing customer retention rates by 5% increases profits by 25% to 95%," linking to Bain & Company [1]. The Bain text by Fred Reichheld says something narrower: "In financial services, for example, a 5% increase in customer retention produces more than a 25% increase in profit" [2]. The 95 percent is not there. The older root, Reichheld and Sasser's 1990 HBR article "Zero Defections," reported that cutting defections by five percentage points raised profits by 25 to 85 percent in a handful of service businesses (85 percent in one bank's branch system, 30 percent in an auto-service chain) [3]. That is a real observation about specific firms in 1990, not a law about yours.

The part of the Bain piece nobody quotes is the most useful: "not every customer has potential to be profitable and long-standing," so companies should "segment clients to identify the subset that holds this potential" [2]. The original argument is for selection. Selection is a decision, made per account, which is what this playbook is about.

Cross-selling has the same problem. Shah, Kumar, Qu and Chen studied the customer databases of five firms and found that 10 to 35 percent of customers who cross-buy are unprofitable, and that they account for 39 to 88 percent of the firms' total losses from customers [4]. The unprofitable ones shared persistent traits: limited spending, heavy returns and revenue reversals, excessive service requests, and buying only on promotion. For them, more products meant a deeper loss. "Sell more to existing customers" is not a strategy. It is a hypothesis that holds for some accounts and fails for others.

### Target the change you cause, not the likelihood

The deeper reason to decide per account is the one Chapter 14 develops: a score predicts who will act, not whom your action will move. Accounts at the top of an expansion list are rich in Sure Things, who would have upgraded anyway (the Discount Mistake in Chapter 2 is exactly this). And contact can backfire: Chapter 14 walks through the retention campaign that raised churn and the evidence that sensitivity to the intervention, not risk, picks the right targets [5][6]. An upsell call to an unhappy account can be the prompt it needed to review the whole contract.

### Win-back depends on why they left

For return, the best evidence says the reason for leaving is the variable that matters. Kumar, Bhagwat and Zhang, using eight years of telecom data, found that first-lifetime behavior, the reason for defection, and the type of offer all predict whether a lost customer comes back and how valuable they are afterward [7]. Customers who left over service and were won back with a service upgrade were profitable. Customers who left over price were less profitable once regained but tended to stay longer. Offers combining a discount with a service upgrade performed best, and customers who had referred others or had problems resolved well in their first lifetime were likelier to return [7][8]. One telecom, one market, one period: treat it as a strong prior, not a rule.

That gives a decision table for former customers:

| Why they left (from memory) | Default decision |
|---|---|
| A product gap that is now fixed | Route or send: show exactly what changed, offer a low-risk way to verify it |
| A service failure | Lead with the service remedy (named contact, onboarding, credit), not only price |
| Price | A priced offer from the approved catalog; expect a smaller but steadier second lifetime |
| Acquired, closed, or consolidated vendors | Do nothing; watch for a new decision-maker only where appropriate |
| Champion left and nobody else knew the product | Wait; re-establish a relationship before any offer |
| Left angry, disputed, or asked not to be contacted | Suppress, permanently if they objected |
| Unknown | Ask, or stay honestly general; do not guess a reason |

The last row matters most. A system that invents a reason ("we know budgets are tight") is worse than one that admits it does not know.

## What you need to know

The signals differ by motion, and all of them live in memory before they live in a message (Chapter 10).

**For expansion and cross-sell:** entitlement against actual use (seats bought versus active, features licensed versus adopted); growth events (fleet size, new locations, a hiring wave); support health, including open escalations and any promise made to the account (the Broken Promise in Chapter 18 is an expansion email that ignored one); who the buyer is now, since titles go stale (the Stale Self, Chapter 11); and **the offer history**: what was offered, when, and what the account said. A declined offer with its reason ("not until the integration works") is one of the most valuable memories on the record. It tells you both what not to repeat and what would change the answer. Low seat adoption is a Customer Success play before it is a sales signal: help the unused seats get used first, because a nudge about unused seats can invite a downsizing review (Playbooks CS and SW).

The offer needs a source too. In the governed personalization engine I built, each campaign carries a **capability map**: the products, modules, and offers the model is allowed to recommend (Chapter 18). The model answers "which approved option fits what we know?", not "what would sound good?" Otherwise it will invent a module because it fits the buyer's problem. The same engine refuses relationship claims it cannot back. "Your unused module," "your renewal," and "your current contract" are allowed only when that relationship is on the record, enforced by a guideline and by a check in code; otherwise the copy is reframed as an evaluation. Expansion is where these claims tempt most, because they sound like knowing the customer.

**For return:** a **departure record**, which most companies do not keep. It needs:

- Typed properties (what you filter and decide on): departure date, reason category, reason source (declared, observed, or inferred), whether the reason is resolved, destination if stated, contact permission, prior win-back attempts.
- Evidence memories (what you quote): the exit-call statement, the key tickets, the cancellation email, each with its source and date.
- A synthesis: the relationship in a paragraph, first lifetime through exit, written for the account manager and the agent.
- Graph edges: the account's departure reason linked to the defect or product gap in your own tracker, and people linked to the organizations they now work for.

Apply the Boundary Rule from Chapter 10: "left over offline mode" is a property you will filter on; how the operations director described her technicians writing on paper is a memory you might quote. And apply provenance strictly (Chapter 11): an agent's guess that an account "probably left over price" must stay labeled as an inference, or it will harden into the reason the win-back offer is built on.

The join that makes win-back work is the one Larkspur never built: **customer-side memory of why they left, linked to company-side memory of what has changed.** When the spring release closed the offline-mode defect, every departure record linked to that defect should have changed state from "unresolved" to "resolved, eligible for review." The trigger for a good win-back is usually an event on your side, not a date on theirs.

![A two-sided diagram. On the customer side, Ridgeway Pest Services (fictional) has a departure record: it left because offline mode failed at rural sites, with the exit call and 14 support tickets as evidence, and its status is unresolved. On the company side, the offline-mode defect was rebuilt in the spring release, eleven months after Ridgeway left, and is closed. A graph edge joins the two, so when the defect closes the departure record changes to resolved and eligible for review, which leads to outreach that shows what changed and offers a 30-day pilot; the trigger is an event on the company's side, not a date on the customer's.](/images/handbook/pb-s2-reason-to-resolution.svg)

*Figure S2.1. Link why they left to what you changed, so a fix reopens the record instead of a timer.*

Two cautions. People move (Chapter 11's tenure data) [9], so a two-year-old cancellation record may name someone who is gone. Re-verify the contact first. And the reason for leaving stays true as history forever, while its relevance expires when the cause is fixed. Supersede, do not overwrite. In the self-hosted memory system I built, that is schema, not habit. A replaceable field keeps every prior value with the dates it was valid, so "when did this reason become resolved, and what did the record say before?" is a lookup. An append-only field keeps every value as its own entry, the right shape for offer history and win-back attempts. The system also takes a list of fields its extractor may never infer: put the declared reason on it, and a model reading old tickets can propose an inferred reason in its own field but cannot fill the declared one.

## The action

Most of this is asynchronous work (Chapter 17): a win-back needs to be right, not fast. A nightly or event-triggered job selects the accounts whose state changed (a defect closed, a seat limit reached, a contract anniversary, a new decision-maker detected), re-runs the decision for them, and queues actions. The exception is in-product expansion: a prompt when a user hits a seat or feature limit is a real-time "show" decision, and it should stay small and factual.

**Channel follows value and consequence.** For Larkspur's top 400 accounts, the output is a brief for the account manager, not an email: what changed, what the account said last time, what to offer, what to avoid. For the long tail, email or in-product messages, with human approval on any offer above a discount threshold.

**The message is built from typed zones (Chapter 15), and each zone has a different source:**

1. *What we remember*: one or two grounded facts from memory, each with a receipt. "You left in March last year because offline mode failed at your rural sites."
2. *What changed*: claims only from verified company-side sources (release notes, the closed defect, a named new service). Never generated from general knowledge.
3. *The offer*: selected by code from an approved catalog, with price and terms stated in full, including what happens after any introductory period. The model may describe it; it may not invent or modify it.
4. *The ask*: one low-cost next step, such as a pilot on their own routes or a call with the engineer who rebuilt the feature.

For Ridgeway, that becomes a short note from the account manager: it names the reason they left without excuses, links the release notes for the offline rebuild, and offers a 30-day pilot on three rural routes. No countdown and no manufactured deadline. A former customer is someone who already decided you were not worth it; pressure confirms the decision.

**Close the loop.** Every offer, what was delivered, and what the account did next are written back to the record as typed values (Chapter 10). In the engine I built, that continuity is the point of memory: lead, research, score, content, delivery, engagement, and then future context that sees what already happened. Without the write-back, next quarter's list offers the add-on again to the account that said no in March.

Memory of why someone left is the most useful thing you hold about them, and the easiest to misuse. Use it to show what changed, never to press where it hurt.

## What can go wrong

### Failure story: The Revolving Door

A composite of a pattern that is easy to fall into (illustrative, not a specific deployment). Larkspur's first win-back campaign sent a 25 percent discount to every account that had left in the past two years. It worked on the headline metric: about one in twelve came back, and the campaign was presented as found revenue.

Two renewal cycles later, most reacquired accounts that had left over service problems had left again. Nothing about the service had changed; they had been paid to retry the thing that failed them, and several now asked not to be contacted again. Meanwhile, the accounts that had left over the offline-mode defect, the group most likely to return for the right reason, had received the same generic discount and mostly ignored it.

The campaign measured reactivation. It should have measured second-lifetime survival by reason. **Winning a customer back for the reason they left is a sale. Winning them back despite it is a delay.**

![Two rows compared. The Revolving Door: a 25 percent discount went to every account that left, about one in twelve came back, and the campaign measured reactivation; two renewal cycles later most accounts that had left over service had left again. The alternative: choose the offer by departure reason, randomize within each reason across three arms (reason-specific outreach, the generic offer, and nothing), and measure second-lifetime value, meaning accounts still active and paying at twelve months, net of discounts.](/images/handbook/pb-s2-second-lifetime.svg)

*Figure S2.2. Reactivation rewards the Revolving Door; second-lifetime value by reason shows whether the win-back was a sale.*

### Governance risks

- **Money and commitments are on the irreversible list (Chapter 18).** Discounts, credits, and price terms come from a catalog and pass a gate in code. A generated win-back that improvises "we can match that price" has made a commitment someone will have to honor.
- **Former customer is not a consent status.** An objection to marketing must survive the account closing and even record deletion, as a minimal suppression entry every send path checks (Chapter 19). Rules for contacting former customers differ by jurisdiction and channel, and the existing-customer exceptions some regimes allow are narrow; check them in dated notes, not in this playbook. This is not legal advice.
- **Known, not watched.** Following a former champion to a new employer can feel like service ("you used us at Ridgeway; here is what changed") or like surveillance, depending on what the person expects and how you learned it. Use relationship history the person shared with you; do not assemble a new profile of them from outside sources to reopen a sale.
- **Sleeping Dogs in the expansion list.** Suppress offers during open escalations and unresolved promises, and watch for accounts whose churn rises after contact.
- **Stale inferences.** A reason for leaving that the system guessed is not a reason the customer gave. Never quote an inference back to them as if they said it. The account manager's brief may carry a hypothesis ("likely left over price, unconfirmed"); the customer-facing message may not. Enforce that with an explicit list of fields allowed to reach the customer, not with the model's discretion (Chapter 15).

### What this does not do

Memory does not create a reason to buy. It cannot make an unprofitable account worth expanding, and it cannot bring back a customer whose reason for leaving is still true. What it can do is stop you from asking at the wrong moment, for the wrong reason, with the wrong offer.

## How you will know

Measure the change you caused, per motion and per reason, against a randomized holdout (Chapter 21).

**For win-back:** randomize at the account level within each departure-reason segment, with three arms where volume allows: reason-specific outreach, the generic offer, and nothing. The primary outcome is not reactivation. It is **second-lifetime value**: reactivated accounts still active and paying at twelve months, net of discounts. Guardrails: unsubscribes, complaints, and suppression requests per contact.

**For expansion and cross-sell:** measure incremental expansion revenue over a holdout, and pair it with two guardrails most programs skip. **Expansion that sticks**: seats or modules still in use two renewal cycles later (a sale followed by a downgrade is shelfware with a delay). And **churn among contacted accounts versus holdout**, which is how Sleeping Dogs show up.

Be honest about power. Larkspur's 2,300 lapsed accounts, split by five reasons and three arms, leave small cells. At that size only large effects are detectable within a year. Pool across quarters, measure more frequent outcomes (replies, pilot starts) as leading indicators validated against the slow one, and label underpowered decisions as judgment.

## Reader Q&A

**We never recorded why customers left. Where do we start?**
The reasons usually exist, just not as a field: cancellation calls, the last three months of tickets, CSM notes, the cancellation email itself. Run schema-guided extraction over them (Chapter 8) into a reason category with its source and a confidence, and review a sample by hand before trusting it. From today, capture the reason at exit as both a category and a verbatim quote.

**Should a win-back offer include a discount?**
Match the offer to the reason. The evidence favors combining a service remedy with a price incentive over price alone, and it favors a service remedy for service defectors [7]. A discount answers price. It does not answer "your product failed us."

**How soon after someone leaves should we reach out?**
When something relevant has changed, not on a fixed timer.

**Is it cheaper to keep a customer than to find one?**
Often, for customers worth keeping. But the "five times" and "25 to 95 percent" figures do not trace to evidence that applies to you, and the original source argues for choosing which customers to invest in [2]. Compute your own acquisition and retention costs per segment.

## For your AI

```yaml
playbook: S2
title: "Sales II: Growth and Return"
question: "What do we already know, and how do we use it to go deeper or win them back?"
concepts:
  - name: Growth and return decisions
    definition: "Three per-account decisions (expand, cross-sell, return), each choosing among offer, route, wait, ask, do nothing, and for former customers, permanent suppression."
  - name: Departure record
    definition: "Typed memory of why an account left: date, reason category, reason source (declared/observed/inferred), resolved status, destination, contact permission, prior win-back attempts, plus evidence memories, a synthesis, and graph links to the company-side cause."
  - name: Reason-to-resolution link
    definition: "A graph edge from a customer's departure reason to the defect, gap, or policy on the company side, so a fix changes the departure record's state and triggers review."
  - name: Offer history
    definition: "Every offer made to an account, when, and the account's response and stated reason; used to avoid repeats and to detect when the reason for a no has changed."
  - name: Second-lifetime value
    definition: "Value of a reacquired account over a fixed period after return, net of incentives; the primary win-back outcome instead of reactivation rate."
  - name: Capability map
    definition: "The approved products, modules, and offers a campaign may recommend; expansion and cross-sell suggestions are chosen from it, never generated from general knowledge."
decision_rules:
  - if: "an account's departure reason is unknown"
    then: "ask or stay honestly general; never infer and quote a reason"
  - if: "an account left over a product gap and the linked defect is now resolved"
    then: "mark eligible for review; lead with verified evidence of the change and a low-risk way to test it"
  - if: "an account left over service"
    then: "lead with a service remedy; price incentive only as a complement"
  - if: "an account objected to marketing or left in dispute"
    then: "suppress; the suppression survives account closure and record deletion"
  - if: "an expansion candidate has an open escalation or an unresolved promise"
    then: "suppress the offer and route to the account owner; revisit after resolution"
  - if: "an account shows unprofitable cross-buy traits (heavy returns or reversals, excessive service load, promotion-only buying)"
    then: "do not push additional products; review the account economics first"
  - if: "a win-back or expansion message includes a price, discount, or term"
    then: "select it from the approved catalog in code; require approval above threshold; state full terms"
  - if: "the contact on a departure record is older than its freshness window"
    then: "re-verify the person and role before writing to them"
  - if: "a message would state a relationship fact (your renewal, your unused module, your contract)"
    then: "allow it only when the relationship is on the record; otherwise reframe as an evaluation"
  - if: "an expansion or cross-sell recommendation is generated"
    then: "choose it from the campaign's capability map; write the offer and the response back to the record"
assessment_questions:
  - "Where is the reason each customer left recorded today, and in what form (field, ticket, call, nowhere)?"
  - "Is any product defect or gap linked to the accounts that left over it?"
  - "Do you store offers made to each account and the account's response?"
  - "Do win-back and expansion programs have randomized holdouts, and what outcome do they measure?"
  - "How are marketing objections from former customers enforced across every send path?"
  - "Which expansion offers can an agent send without human approval, and what gate enforces price terms?"
patterns: [Departure Record, Reason-to-Resolution Link, Offer History, Capability Map, Customer-Status Guard, Typed Zones, Do-Nothing Option, Permanent Global Holdout]
anti_patterns: [The Revolving Door, The Discount Mistake, The Stale Self, The Broken Promise, Retention Arithmetic Myth, Reactivation-as-Success]
links: {memory: [10, 11], decision: 14, generation: 15, timing: 17, governance: 18, privacy: 19, measurement: 21}
maturity_dimension: decisioning
```

## References

1. Gallo, A. (2014-10-29). "The Value of Keeping the Right Customers." *Harvard Business Review*. https://hbr.org/2014/10/the-value-of-keeping-the-right-customers
2. Reichheld, F. (2001). "Prescription for cutting costs." Bain & Company. https://media.bain.com/Images/BB_Prescription_cutting_costs.pdf
3. Reichheld, F. F., and Sasser, W. E., Jr. (1990). "Zero Defections: Quality Comes to Services." *Harvard Business Review* 68(5), 105 to 111. https://hbr.org/1990/09/zero-defections-quality-comes-to-services
4. Shah, D., Kumar, V., Qu, Y., and Chen, S. (2012). "Unprofitable Cross-Buying: Evidence from Consumer and Business Markets." *Journal of Marketing* 76(3), 78 to 95. https://journals.sagepub.com/doi/abs/10.1509/jm.10.0445 ; practitioner version: Shah, D., and Kumar, V. (2012-12). "The Dark Side of Cross-Selling." *HBR*. https://hbr.org/2012/12/the-dark-side-of-cross-selling
5. Radcliffe, N. J., and Simpson, R. (2008). "Identifying who can be saved and who will be driven away by retention activity." *Journal of Telecommunications Management* 1(2). https://stochasticsolutions.com/pdf/SavedAndDrivenAway.pdf
6. Ascarza, E. (2018). "Retention Futility: Targeting High-Risk Customers Might Be Ineffective." *Journal of Marketing Research* 55(1). https://journals.sagepub.com/doi/10.1509/jmr.16.0163
7. Kumar, V., Bhagwat, Y., and Zhang, X. (2015). "Regaining 'Lost' Customers: The Predictive Power of First-Lifetime Behavior, the Reason for Defection, and the Nature of the Win-Back Offer." *Journal of Marketing* 79(4). DOI 10.1509/jm.14.0107
8. ScienceDaily (2015-07-07). "Yes, it pays to win back lost customers." https://www.sciencedaily.com/releases/2015/07/150707134215.htm
9. U.S. Bureau of Labor Statistics. "Employee Tenure in 2026." Released 2026-09-24. https://www.bls.gov/news.release/tenure.nr0.htm

---

# Playbook CS. Customer Success

*The question: How does customer success personalize onboarding, adoption and renewal for each account and each user?*

> **Questions this playbook answers**
> - Our onboarding is the same kickoff deck for every new account. What should change by role, goal, and experience?
> - Our health score is green on accounts that leave and red on accounts that stay. Is the score broken, or are we using it wrong?
> - Should we reach out to at-risk accounts before they reach out to us, and when does outreach make things worse?
> - How do we walk into a renewal or a QBR already knowing what happened this year?
> - What can a customer success agent do on its own, and what must stay with a person?

## The moment

Monday, 8:30 a.m. The head of customer success at Larkspur Systems (a fictional company, a composite of many real ones, that sells field-service and fleet software) opens the weekly health review. The dashboard colors every account green, yellow, or red from a blend of logins, seat use, open tickets, and the last survey score. Red accounts get an automated re-engagement sequence and a save call; green accounts get nothing. Six customer success managers (CSMs) carry about 1,200 mid-tier accounts; twelve account managers cover the top 400; everyone else is served by email and in-product messages. Two rows tell the story.

Brookline Landscape Group (also fictional) runs 45 crews. It turned red in December: logins down 60 percent since November. The sequence sends "We miss you! Here are five features you might not know about," and a CSM books a save call. Brookline's owner answers the call from a trade show: they are a landscaping company, it is January, and their crews come back in March. The login drop is the calendar, and it was in the account's own history for four winters running.

Alder Creek Utilities (also fictional) runs 220 vehicles and is deep green: 94 percent of seats active, no open tickets. Its renewal is 90 days out. The executive sponsor who signed the contract left in the summer. The new chief operating officer has asked for a list of every software vendor, with what each one delivered this year. That sentence sits in a CSM's call note. Nothing in the score can read it, and nobody at Larkspur can answer "what did we deliver this year?" without two days of digging.

The score sent a save call to an account that was fine and silence to an account that was deciding. It was asked to do a job a score cannot do.

## The decision

Customer success makes a sequence of decisions per account and per user, from the sales handoff to the renewal, and each includes the option to do nothing (Chapter 14).

1. **Onboard.** Which path does each account and user get, given who they are, why they bought, and what they already know?
2. **Adopt.** Which play runs now, if any: training, an in-product guide, a CSM call, or nothing?
3. **Intervene.** When risk appears, does contact help, hurt, or change nothing?
4. **Renew and review.** What must the renewal and the quarterly business review (QBR) show, and who prepares it?
5. **Hand off.** When does this belong to sales or to support, and what travels with it?

![A left-to-right journey of one account through five stages, each with its decision and the option to do nothing. Handoff from sales: capture the goal and its source. Onboard: choose a path by prior experience first, then role and goal. Adopt: run a play tied to the goal, or leave a progressing account alone. Intervene: contact only where contact is expected to help, and fix the cause where it would hurt. Renew and review: a brief assembled from memory 120 days out. Two exits leave the journey: to support when a severe case opens, which pauses success plays, and to sales when the customer's goal outgrows what it bought. Underneath, one shared memory holds the success plan (goal, criteria, milestones), the relationship map (sponsor, admin, changes), promises as typed constraints, and offer history with value delivered; every stage reads from it and writes back to it.](/images/handbook/pb-cs-account-journey.svg)

*Figure CS.1. One account, five decisions, one memory: every stage reads what the last one learned.*

### Onboarding: personalize by experience first, then by role and goal

The strongest controlled evidence in this function is about onboarding. In a 2011 field experiment at a public cloud infrastructure provider, 366 of 2,673 new customers received proactive guidance on how to use basic features. The treatment halved the number of customers who churned in the first week, cut questions in that week by 19.55 percent, and raised accumulated usage over the next eight months by 46.57 percent. The effect was strongest for customers with less experience with the provider. And the direct effect decayed within one week [1].

The intervention was one kind of guidance, not a personalized program, so the study does not prove that tailoring works. What it shows is heterogeneity: the same help mattered most to people who knew least. That makes **prior experience** the first personalization variable, ahead of job title. A dispatcher moving from whiteboards needs a different first week from an administrator migrating from a competitor, who needs import mapping and the places Larkspur differs from what she knows. And because the direct effect fades in days while the usage gain compounds for months, the first week is the window.

Role and goal come next: the users who do the daily work, the administrator who configures it, and the sponsor who signed because of a goal ("cut overtime on emergency calls"). Each needs a different path, and the sponsor's goal should shape all of them. In-product slots belong to the Product playbook and self-serve activation to the Software Companies playbook; the CS side is the account plan: kickoff, training, milestones, and the definition of success the customer agrees to.

That definition is the most valuable record customer success writes, and it starts at the sales handoff. If the reason they bought is not captured as a typed goal with its source, the CSM re-asks it at kickoff and the renewal team guesses it a year later.

### Adoption plays: a play is a decision with an exit

An adoption play is a decision with five parts: a trigger (a milestone missed, a new user, a goal-linked feature untouched), a target (the user or role who can move it), an action, a success condition, and an exit (including "the account got there without us").

Tie plays to the customer's goal, not to feature breadth. Pendo's often-quoted finding that 80 percent of features are "rarely or never used" comes from its own customers' usage over three months [2]; it says nothing about which features this customer needs.

Usage is still the best signal CS has: in 3,959 subscriptions at a European software provider, adding usage data improved churn prediction over firmographic, transactional, and support features alone [3]. But usage has a shape (seasonal, project-based), and a play that ignores the shape is Brookline's save call.

### Health scores describe; they do not decide

Here is the sentence this playbook turns on: **a health score tells you how an account looks; it does not tell you whether anything you do will change how it ends.**

Start with the evidence. Most health scores are weighted blends designed in a spreadsheet. Gainsight's DEAR framework (deployment, engagement, adoption, ROI) was described by the company's chief customer officer as producing scores with which renewals "aligned almost perfectly," but the published material gives no numbers; its template uses placeholder percentages [4]. I found no peer-reviewed validation of a vendor health-score framework in B2B software. "A model can predict churn" is not "your score does."

Even an accurate score answers the wrong question. A review by twelve retention researchers notes that the customers at highest risk "do not necessarily overlap 100%" with those who should be targeted, that a campaign launched too early can start a customer thinking about churning, and that in a well-known churn-modeling tournament the top decile of predicted risk still contained roughly 96 percent customers who did not churn [5]. Ascarza's field experiments found targeting the highest-risk customers can be ineffective; target on sensitivity to the intervention [6], or, with Lemmens and Gupta, on the incremental profit of intervening, net of its cost [7].

So split what the dashboard blends into three separate things:

1. **State, with receipts.** What is true now: seats active, milestones reached against the success plan, open cases, stakeholder changes, each with a source and a date. No weighting, no color.
2. **Risk.** A calibrated probability of non-renewal, validated against past renewals by cohort, with its reasons.
3. **Expected response.** For each play, including doing nothing, the estimated effect on this account. This is what should choose the action; early on it is judgment backed by experiments, not a model.

Notion's CS team, in a case told on its vendor's blog, redesigned its score per segment, served the long tail with digital CS instead of a CSM, and warned against automated signals without a manual override [8]. The score is an instrument for people, not a trigger for machines.

![A two-by-two grid with renewal risk from the health score on the horizontal axis and the expected effect of contact on the vertical axis. Low risk and contact helps: onboarding and adoption plays. High risk and contact helps: the Persuadables, the save worth a CSM's time. Low risk and contact changes nothing: leave alone, the Sure Things. High risk and contact hurts: the Sleeping Dogs, where a check-in prompts a review, so fix the cause and do not send the nudge. A health score only sorts accounts left to right; the vertical axis, which decides the action, has to be learned from holdouts.](/images/handbook/pb-cs-risk-vs-response.svg)

*Figure CS.2. The health score sorts accounts by risk; only the response to contact tells you what to do.*

### Proactive outreach helps, until it prompts a review

Proactive CS has evidence on both sides. It helped in the cloud onboarding experiment [1]. It hurt at a wireless carrier: Ascarza, Iyengar and Schleicher proactively recommended cost-saving plans to customers, genuinely in their interest, and churn over the next three months rose from 6 percent in the control group to 10 percent among those contacted. The outreach lowered the inertia that kept customers from reconsidering and made them more sensitive to their own usage [9]. Radcliffe and Simpson report a retention campaign that raised churn from 9 to 10 percent for the same reason [10]. And in a telecom field experiment, Becker, Spann and Barrot found proactive post-sales service reduced churn and service calls, until a cross-sell rode along with it and customers began to doubt the motive [11].

The pattern I take from the three: **outreach that helps someone do the job they bought you for tends to help; outreach that invites them to re-evaluate the purchase can wake the dog.** Onboarding guidance, a fix, training tied to their goal: help. "Let's review your plan," "you may be paying for seats you don't use," an empty check-in to a frustrated account: invitations to review. For Sleeping Dogs, the play is not a better email. It is fixing what made them unhappy, and saying so when it is fixed.

## What you need to know

CS needs the relationship over time, not a snapshot. By the Boundary Rule (Chapter 10), what you filter on is a property; what you would quote is evidence:

| What | Form | Why it matters |
|---|---|---|
| Success plan: goal, success criteria, agreed milestones, source (sales call, kickoff) | Typed properties plus the quoted evidence | Onboarding path, adoption plays, QBR, and renewal all read it |
| Onboarding path and prior experience (new to this kind of software, migrating, returning) | Properties | Chooses the first-week path; the heterogeneity in [1] |
| Adoption against the goal, with its seasonal or project shape | Properties computed from telemetry, timestamped | Separates Brookline's winter from a real decline |
| Relationship map: sponsor, admin, champion, users; when each changed | Properties plus graph edges, with validity dates | Alder Creek's sponsor left; the new COO is a new buyer (the Stale Self, Chapter 11) |
| Promises and constraints: "no discount talk," "training before renewal" | Typed constraints (Chapter 18) | Kestrel Haulage's lost promise in Chapter 13 |
| Open and recent cases, linked defects | Properties plus graph edges | Suppresses plays; routes to support |
| Offer history and value delivered, each with a receipt | Append-only entries | Renewal and QBR evidence; stops repeat offers (Playbook S2) |
| Commercial: renewal date, entitlements, tier | Properties | Timing of renewal prep; never changes what the customer is owed |

Stakeholder facts decay fastest: supersede the sponsor when they leave, keeping the old value with its dates (Chapter 11). And a note is not a signal until something reads it: the COO's vendor review in Alder Creek's call note should have become a typed event that moved the account into renewal preparation early.

A person cannot re-read 1,200 accounts' notes every week. A scheduled job can: in the self-hosted memory system I built, recurring prompts run beside the memory, so "each Monday, read last week's notes for every account and propose stakeholder and risk events, citing the sentence behind each" is a configuration, not a project. Proposals stay proposals until a person or a rule accepts them.

## The action

Most CS work is asynchronous (Chapter 17): none of it needs to happen in under a second; all of it needs to be right.

### Renewal preparation starts 120 days out, from memory

The renewal brief is Larkspur's memory-building bet from Chapter 6, where six CSMs were rebuilding account history by hand before every save call. At 120 days, a job assembles a brief per account: the goal from the success plan and the evidence of progress against it; value delivered, each item linked to its source; stakeholder changes; open risks and promises; what was offered before and what the customer said. The account owner reads it and decides.

Three rules from the governed personalization engine I built carry over. **Internal is not customer-facing:** the engine's rep-facing playbooks hold hypotheses that must never be copied into a buyer message; a renewal brief can say "sponsor change, possible vendor consolidation, unconfirmed," and the renewal email cannot. **Technical completion is not semantic completion:** the engine does not accept a playbook because the job reported success; if load-bearing sections are empty, it retries within bounds. A brief with a blank "value delivered" section has failed. **Thin data gets a thin brief, labeled:** when enrichment misses its deadline, the engine produces a deterministic, lower-specificity brief instead of inventing detail. "No success plan on record" tells the CSM what to ask.

For the forecast, start simple: Fader and Hardie's shifted-beta-geometric model projects renewal curves from a few periods of cohort data, in a spreadsheet [12]. Report retention by cohort, or survivor bias will flatter you [5].

For Alder Creek, the brief would have surfaced the sponsor change in the summer. The right action was not a discount. It was a one-page account of what Larkspur delivered this year, in the new COO's terms, and a meeting before her vendor list was drawn up.

### A QBR is the success plan, read back

A good QBR answers three questions: what did you want, what happened, and what next. All three live in memory if the success plan was written at onboarding. Numbers in the deck should be computed by code from telemetry and cited, never generated. In the engine I built, statements like "your renewal is coming up" or "your unused module" are blocked unless that relationship is on the record, by guideline and in code, and otherwise reframed. A QBR draft needs the same guard; it is the document that sounds most like knowing the customer.

The QBR is also where the "ask" action (Chapter 14) belongs: is the goal on record still the goal? Write the answer back to the success plan.

### Handoffs carry state, and keep service clean

**To sales, for expansion.** Hand off when the customer's goal outgrows what they bought (the fleet doubled, a new region), carrying the success plan and evidence, not a lead score (Chapter 13). Keep the conversations separate: proactive service loses its effect when a cross-sell rides along [11]. Expansion decisions belong to Playbook S2.

**To support, and back.** An open severe case pauses every success play for that account until the fix holds (Playbook SP). When it closes, support hands back what happened and what was promised, so the next CS touch does not apologize for a solved problem. Across functions, the coordinator in Playbook P9 decides what reaches the account this week.

### CS agents act inside scoped authority

Agents suit most of the work above: reading notes, assembling briefs, drafting QBR decks, proposing stakeholder events. They should hold no authority over Chapter 18's irreversible list: credits and discounts, renewal pricing and terms, roadmap commitments, contract or consent records, first sends in a new program.

Enforce that in credentials, not prompts. In the memory system I built, each agent can get its own key with a role and a data scope: a reader key's write tools are simply absent, record visibility is enforced inside the database and fails closed, and one account's negotiated exception can be scoped so only those who see that account can read it. Long investigations run as hosted jobs on a credential scoped to the job and revoked when it ends. Governance documents are retrieved on every action, so the pricing rule the agent reads is the one the renewal team signed.

A practical ladder: read and propose from day one; write notes and internal tasks once proposals are mostly accepted; send low-stakes messages (a training reminder, a milestone recap) with sampled review; never commit money or terms. The engine I built treats its own improvement the same way: AI observes, diagnoses, proposes, and validates outside production; production authority stays with people.

## What can go wrong

### Failure story: The Wake-Up Call

A composite of a pattern that is easy to fall into (illustrative, not a specific deployment). Larkspur's first automated success play targeted every yellow and red account with a friendly, well-written email: "We noticed your team is using 60 percent of its seats. Let's make sure you're on the right plan," with a link to book a review. It was meant as service. Over the next quarter, contacted accounts downgraded seats noticeably more than a small group the team had forgotten to include, and some brought a procurement lead to the review.

The email did what Ascarza's plan recommendations did: it made customers look at what they paid for against what they used, at the moment Larkspur chose, not the moment they would have [9]. The play measured bookings. It should have measured seats and renewals against a holdout. **An email that invites a customer to audit you should be sent on purpose, or not at all.**

### Governance risks

- **The Color Trigger.** A health color wired straight to an automated sequence. Scores inform people; actions come from the decision layer, with a do-nothing option and a holdout.
- **The Seasonal Alarm.** Usage compared to last month instead of to the account's own pattern. Brookline's save call.
- **The Lost Why.** The reason they bought is never captured at handoff, so onboarding, the QBR, and the renewal each guess it.
- **Internal in external.** A brief's hypothesis, or a relationship claim with no record behind it, reaches customer copy. Declare internal fields prohibited context and guard relationship claims in code (Chapters 15, 18).
- **Money in an agent's hands.** Discounts come from a catalog, behind an approval, never from an agent's judgment.

### What this does not do

Memory and agents do not make a product worth renewing. If Alder Creek's COO finds Larkspur delivered little, a perfect brief makes that visible sooner; it does not change it. What this playbook can do is stop CS from spending scarce attention on the wrong accounts, at the wrong moment.

The evidence has gaps. Beyond Retana and colleagues' one experiment [1], I found no controlled public evidence that personalized onboarding or in-app guides lift retention. "86 percent stay loyal to brands that invest in onboarding" comes from 216 consumers stating an intention, published by a video vendor [13]. And the discipline's origin story, Salesforce's 8 percent monthly churn in 2005, is told by Gainsight executives in their own book [14]: a useful story, not an audit.

## How you will know

Every play gets a randomized holdout at the account level (Chapter 21), because users in one account influence each other. The outcomes are renewal, cohort gross revenue retention, and adoption against the success plan, not bookings, replies, or health color.

- **For onboarding:** randomize new accounts between the tailored path and the best static one; measure first-month milestones, early support questions, and renewal. Retana and colleagues' design is a usable template [1]; check that the effect is largest among the least experienced.
- **For risk interventions:** three arms where volume allows (the play, a lighter touch, nothing), with downgrades, cancellations, and discount requests among contacted accounts as guardrails. That is where Sleeping Dogs show up.
- **For the health score:** backtest it against past renewals by cohort. Of accounts scored at 20 percent risk, did about 20 percent leave? What share of the top risk decile renewed anyway? If you cannot answer, the score is a mood, not a measure.
- **For renewal preparation:** hours per brief, and the share of briefs that surfaced a risk the owner did not know.

Be honest about power: Larkspur's 6,000 at-risk renewals a year, split across plays and arms, leave small cells for an outcome that arrives once a year. Pool across quarters, validate leading indicators against renewal, and label underpowered calls as judgment. And name the population behind any benchmark: the median private SaaS company in KeyBanc and Sapphire's 2024 survey reported net retention around 101 percent and gross around 90 percent [15], while investor frameworks call 120 percent net "best" [16].

## Reader Q&A

**We have one CSM per 200 accounts. Where do we start?**
With the handoff and the renewal brief. Capture why each account bought, as a typed goal with its source, from today. Build the 120-day brief from what you already have (tickets, calls, telemetry) for next quarter's renewals. Every other play reads from both.

**Should we throw our health score away?**
No. Split it. Keep the state, with receipts; validate the risk part against past renewals; and stop letting either one trigger outreach on its own. The action comes from the expected response to contact, which you learn from holdouts.

**Can the CS agent send the renewal quote?**
It can draft one from the approved catalog for a person to send. Price and terms are commitments (Chapter 18). As drafts prove reliable, lighten the review of the drafting, not the authority over the price.

## For your AI

```yaml
playbook: CS
title: "Customer Success"
question: "How does customer success personalize onboarding, adoption and renewal for each account and each user?"
concepts:
  - name: Success plan
    definition: "Typed record of why the account bought (goal), what success means (criteria, milestones), and the source of each, captured at the sales handoff and read by onboarding, adoption plays, QBRs, and renewal."
  - name: Adoption play
    definition: "A decision with a trigger, a target user or role, an action, a success condition, and an exit, tied to the account's goal rather than to feature breadth."
  - name: Split health
    definition: "Replace one blended health score with three separate things: state with receipts, calibrated risk with reasons, and expected response to each possible action including doing nothing."
  - name: Review invitation
    definition: "Outreach that prompts a customer to re-evaluate the purchase (plan reviews, seat right-sizing, empty check-ins); the class of contact most likely to wake Sleeping Dogs."
  - name: Renewal brief
    definition: "Internal, sourced account summary assembled from memory 120 days before renewal: goal and progress, value delivered with receipts, stakeholder changes, risks, promises, and offer history; labeled thin when data is thin."
  - name: Scoped CS agent
    definition: "An agent whose credentials allow reading and proposing, then writing internal notes and low-stakes sends as trust is earned, and never money, terms, roadmap commitments, or record changes on the irreversible list."
decision_rules:
  - if: "a new account is onboarding"
    then: "choose the path by prior experience first, then by role and goal; deliver it in the first week"
  - if: "the reason an account bought is not recorded at handoff"
    then: "capture it as a typed goal with its source before kickoff; do not re-ask the customer what sales already heard"
  - if: "an account's usage drops"
    then: "compare to the account's own seasonal or project pattern before treating it as risk"
  - if: "a health score or color changes"
    then: "inform the account owner; do not trigger automated outreach from the score alone"
  - if: "an account is high risk and contact is expected to prompt re-evaluation"
    then: "do not send a check-in or plan review; fix the cause and report the fix"
  - if: "a stakeholder (sponsor, admin, champion) changes"
    then: "supersede the relationship record with dates and move the account into renewal preparation"
  - if: "an account has an open severe support case"
    then: "pause success plays until the fix is confirmed"
  - if: "the customer's goal outgrows its entitlement"
    then: "hand off to sales with the success plan and evidence; keep the service conversation free of cross-sell"
  - if: "a renewal brief or QBR draft has empty load-bearing sections or thin data"
    then: "retry within bounds or produce a labeled thin brief; never invent value delivered"
  - if: "customer-facing copy states a relationship fact (your renewal, your plan, your unused seats)"
    then: "allow only when the fact is on the record; otherwise reframe"
  - if: "a CS agent's action involves price, credit, terms, roadmap, or contract records"
    then: "the agent drafts from the approved catalog; a person approves and sends"
assessment_questions:
  - "Where is the reason each customer bought recorded, and who reads it after the sale?"
  - "Has your health score ever been backtested against actual renewals by cohort? What share of the top risk decile renewed anyway?"
  - "Which success plays run automatically from a health color, and do any have a holdout?"
  - "How long does it take to answer 'what did we deliver to this account this year?'"
  - "What credentials do your CS agents hold, and which irreversible actions can they take?"
  - "How does an open support case change what CS sends?"
patterns: [Success Plan at Handoff, Experience-First Onboarding, Adoption Play with Exit, Split Health, Renewal Brief at 120 Days, QBR from the Success Plan, State-Carrying Handoff, Scoped Agent Ladder, Customer-Status Guard]
anti_patterns: [The Wake-Up Call, The Color Trigger, The Seasonal Alarm, The Lost Why, The Stale Self, Bookings as Success]
metrics: [renewal_rate_vs_holdout, gross_revenue_retention_by_cohort, milestone_attainment_vs_success_plan, downgrades_among_contacted_vs_holdout, health_score_calibration, renewal_brief_hours_and_new_risk_surfaced]
links: {memory: [10, 11], long_running: 13, decision: 14, timing: 17, generative_experiences: 16, governance: 18, measurement: 21, playbooks: [PR, SW, S2, SP, P9]}
maturity_dimension: decisioning
```

## References

1. Retana, G. F., Forman, C., Wu, D. J. (2016). "Proactive Customer Education, Customer Retention, and Demand for Technology Support: Evidence from a Field Experiment." *Manufacturing & Service Operations Management* 18(1): 34-50. DOI 10.1287/msom.2015.0547. https://ideas.repec.org/a/inm/ormsom/v18y2016i1p34-50.html
2. Pendo (2019). "The 2019 Feature Adoption Report." Vendor data from 615 subscriptions. https://www.pendo.io/resources/the-2019-feature-adoption-report/
3. Sanchez Ramirez, J., Coussement, K., De Caigny, A., et al. (2024). "Incorporating usage data for B2B churn prediction modeling." *Industrial Marketing Management*. https://www.sciencedirect.com/science/article/abs/pii/S0019850124000865
4. Capote, K. (2022-12-06). "Use customer health data to grow and forecast NRR." TechCrunch. https://techcrunch.com/2022/12/06/use-customer-health-data-to-grow-and-forecast-nrr/ ; Gainsight DEAR ebook https://www.gainsight.com/wp-content/uploads/2023/04/DEAR-Ebook.pdf
5. Ascarza, E., Neslin, S., Netzer, O., et al. (2018). "In Pursuit of Enhanced Customer Retention Management: Review, Key Issues, and Future Directions." *Customer Needs and Solutions* 5(1-2): 65-81. DOI 10.1007/s40547-017-0080-0. https://business.columbia.edu/sites/default/files-efs/pubfiles/25780/ascarza_et_al_choice_symposium.pdf
6. Ascarza, E. (2018). "Retention Futility: Targeting High-Risk Customers Might Be Ineffective." *Journal of Marketing Research* 55(1). https://journals.sagepub.com/doi/10.1509/jmr.16.0163
7. Lemmens, A., Gupta, S. (2020). "Managing Churn to Maximize Profits." *Marketing Science* 39(5): 956-973. DOI 10.1287/mksc.2020.1229
8. Van Lew, T. (2023-06-13). "How Notion Approaches Customer Health Scoring for All Segments." Gainsight blog (vendor-told case). https://www.gainsight.com/blog/how-notion-approaches-customer-health-scoring-for-all-segments/
9. Ascarza, E., Iyengar, R., Schleicher, M. (2016). "The Perils of Proactive Churn Prevention Using Plan Recommendations: Evidence from a Field Experiment." *Journal of Marketing Research* 53(1): 46-60. DOI 10.1509/jmr.13.0483. https://journals.sagepub.com/doi/abs/10.1509/jmr.13.0483
10. Radcliffe, N. J., Simpson, R. (2008). "Identifying who can be saved and who will be driven away by retention activity." *Journal of Telecommunications Management* 1(2). https://stochasticsolutions.com/pdf/SavedAndDrivenAway.pdf
11. Becker, J. U., Spann, M., Barrot, C. (2020). "Impact of Proactive Postsales Service and Cross-Selling Activities on Customer Churn and Service Calls." *Journal of Service Research* 23(1): 53-69. DOI 10.1177/1094670519883347
12. Fader, P. S., Hardie, B. G. S. (2007). "How to Project Customer Retention." *Journal of Interactive Marketing* 21(1): 76-90. DOI 10.1002/dir.20074
13. Wyzowl (2020). "Customer Onboarding Statistics." Survey of 216 consumers. https://wyzowl.com/customer-onboarding-statistics/
14. Mehta, N., Steinman, D., Murphy, L. (2016). *Customer Success: How Innovative Companies Are Reducing Churn and Growing Recurring Revenue.* Wiley. Chapter 1 excerpt: https://catalogimages.wiley.com/images/db/pdf/9781119167969.excerpt.pdf
15. KeyBanc Capital Markets and Sapphire Ventures (2024-10-23). 15th annual Private SaaS Company Survey, press release. https://sapphireventures.com/press/keybanc-capital-markets-and-sapphire-ventures-private-saas-company-survey-reveals-a-continued-focus-on-operational-efficiency-and-profitability/
16. Bessemer Venture Partners (2023-04-11). "State of the Cloud 2023." https://www.bvp.com/atlas/state-of-the-cloud-2023

---

# Playbook SP. Support

*The question: How do we stop asking customers what we already know?*

> **Questions this playbook answers**
> - Our customers keep re-explaining their problem to every bot and every agent. What would it take to stop that?
> - What should a support AI handle on its own, and exactly when should a person take over?
> - How should tone change with the customer's situation without becoming a script?
> - Should support reach out before the customer notices a problem, and when is silence better?
> - How do we prove it worked without being fooled by deflection numbers?

## The moment

Friday, 6:40 a.m. Ines runs dispatch for a 120-vehicle regional delivery company that uses Larkspur Systems' routing software. (Larkspur is a fictional composite used throughout this handbook.) Her drivers leave at seven. For the third time in two weeks, the overnight route sync has failed, and her dispatch board is empty.

She opens the support chat. The assistant asks for her account number. Then which product she uses. Then asks her to describe the problem. She types it out. It suggests clearing her browser cache and links a help article. She asks for a person. The queue says 35 minutes. When the agent arrives, the first message is: "Can you confirm your account number and describe the issue?"

Everything Ines typed was already inside Larkspur. The support desk held two earlier tickets with the same symptom, both closed as "resolved in release 8.2." Engineering had linked the failure to a known defect affecting about 60 accounts. Telemetry showed her sync job failing at 2:14 a.m., four hours before she noticed. Her last ticket ended with "if this happens again I am taking it to our COO." Her renewal is ten weeks out. None of it reached the chat.

She is not unusual. In Zendesk's 2025 survey of 6,182 consumers across 22 countries, 74% said they are frustrated when they have to repeat information, and 81% want agents to continue a conversation without backtracking [1]. Salesforce's 2020 survey of 12,000 consumers and 3,600 business buyers found 65% "often have to repeat or re-explain information to different representatives," and 54% said it "generally feels like I'm communicating with separate departments, not one company" [2].

Here is the sentence this playbook turns on: **the customer should never be the integration layer between your systems.** Every question you ask that your records could have answered is a small bill you hand the customer for your own architecture.

## The decision

Support personalization is not a warmer greeting. For every contact, the system makes four decisions, and each one has a "less" option.

1. **What to ask.** Ask only what memory cannot answer, or what must be confirmed. Everything else is stated back, briefly, so the customer can correct it.
2. **How to respond.** The register (length, warmth, how much to explain, whether to apologize) follows the customer's situation, not a persona. A third recurrence of an outage gets fewer words and more ownership than a first-week how-to question.
3. **Who handles it.** The AI resolves, the AI drafts and a person decides, or a person owns the case. This is the handoff line.
4. **Whether to reach out first.** Proactive help when the system knows before the customer does. Including the decision not to.

The action space is the one from the decision layer (Chapter 14): answer, ask, route, escalate, reach out, suppress other programs, wait, or do nothing. Support adds one discipline the other functions can skip. A support contact is a hard constraint on everyone else. While a customer has an open, severe case, the expansion email, the survey, and the "how are we doing?" check-in should not go out (Chapter 14 walks through that exact upsell).

For Ines, the right decisions were small and specific. Skip intake: her identity, product, and history were known. Recognize the recurrence and the stated escalation, and route straight to a senior engineer-facing agent with the case assembled, instead of the cache article. Tell her what was already known and what happens next. Suppress Larkspur's marketing and expansion programs for her account until the fix holds. And, at 2:30 a.m., before any of the 60 affected accounts woke up, send each dispatch lead a notice: what failed, the manual workaround, and when to expect the next update.

Not every repeat question is waste. Identity verification before account changes is a legitimate ask, and so is confirming a fact past its freshness window ("Are you still the admin on this account?"). The rule is to confirm, not interrogate: one question that checks a stored answer instead of three that rebuild it.

## What you need to know

Continuity is a memory problem before it is a model problem. Two kinds of memory matter here, with different lifetimes (Chapter 13). **Entity memory** is what Larkspur knows about Ines and her company, and it outlives every ticket. **Case memory** is the working file for this problem: symptom, steps tried, promises made, who owns it now. Most support stacks have the second, locked inside the ticketing tool, and none of the first.

The minimum shape, using the Boundary Rule from Chapter 10 (if you would filter on it, it is a property; if you would put it in a paragraph, it is a memory):

| What | Form | Why it matters in support |
|---|---|---|
| Resolved identity and role | Property, from the identity spine (Chapter 9) | Greets the right person with the right account; a wrong merge here shows one customer another's data |
| Open and recent cases: symptom, resolution, whether the fix was confirmed | Properties plus case memory | Detects recurrence; "same issue before?" is the single most valuable field |
| Links to known defects and incidents | Graph edges | Connects one ticket to 60 accounts; enables proactive notice |
| Product state and telemetry (failed jobs, errors, configuration) | Properties, with timestamps | Knows before the customer does |
| Promises made: callbacks, credits, "we will tell you when it is fixed" | Typed constraints (Chapter 18) | A promise in ticket prose is a promise every other agent will miss |
| Stated preferences: channel, hours, "call my deputy" | Properties with source and date | Respects what the customer already said |
| Escalation trajectory and stated frustration | Memory with source; a derived severity property | Changes routing and register |
| Commercial context: renewal date, tier, open deals | Properties | Used to suppress selling and inform routing; never to change what the customer is entitled to |

Two freshness rules do most of the work (Chapter 11). A closed ticket is not a fixed problem: "resolved" should mean the telemetry confirms it, not that an agent clicked a button. And a stated fact about a person decays; the admin who opened last year's ticket may have left.

Some things should stay out of the customer-facing context entirely. Internal notes ("customer is difficult") must never reach generated output, and a prompt instruction will not stop it. In the governed personalization engine I built, the same wall separates what a seller sees from what a buyer reads: internal fields are declared prohibited context, so they never enter the prompt, and an explicit serializer decides what leaves the system, so a newly added internal field cannot leak by default (Chapter 18). Case notes and replies need that wall. Sensitive context a customer shared in a hard moment (an illness, a bereavement) may change whether and when you contact them; it should almost never be quoted back to them (Chapter 19).

Support is also where customers paste what they should not: a password, a card number, an API key inside a log. In a self-hosted memory system I built, secrets, checksum-validated card numbers, and US Social Security numbers are removed before any external model call; emails, phone numbers, and a supplied name list can be swapped for tokens and restored locally afterward; your own patterns (account or case numbers) plug into both; and each call can log a hash of what was sent. Two limits: tokenization governs what the provider sees, not what you store, and an unlisted name in free text is not caught (Chapter 19).

Scope the agents too, as that system allows (opt-in): a key per bot and copilot, so a read-only agent's write tools are simply absent; record visibility enforced inside the database, failing closed; and a per-document audience, so one account's negotiated exception is readable only by those who can see that account.

## The action

Support runs on three clocks, and each needs a different architecture (Chapter 17).

**Real time: chat and voice.** The customer is waiting, so there is no time to research. The case brief (identity, open cases, recurrence, linked incidents, promises, stated preferences) should be computed when the signal arrives: when a ticket opens, a sync job fails, or an incident is declared. At contact, the assistant reads the brief rather than rebuilding it. The first message then states what is known and asks one confirming question: "I can see the overnight sync failed again at 2:14, the third time since release 8.2. Is it the same symptom as last week?"

When the brief is thin or late, the assistant should say less, not guess. In the engine I built, when enrichment misses its deadline the record falls back to a deterministic, lower-specificity brief instead of inventing detail. Specificity can degrade before trust does. The degraded support line is "I can see your account and I am checking the sync status now," never a guessed cause.

![A side-by-side comparison. On the left, re-intake at contact: the bot asks for the account number, the product, and a description of the problem, suggests clearing the browser cache, the queue is 35 minutes, and the human agent asks for the account number again, although all of it was already inside the company. On the right, a case brief is computed when the sync job fails at 2:14 a.m.: identity, account, and product; two earlier tickets with the same symptom; a known defect affecting about 60 accounts; the stated escalation to the COO; and the renewal in ten weeks. The first message then states what is known and asks one confirming question.](/images/handbook/pb-sp-case-brief.svg)

*Figure SP.1. Compute the case brief when the signal arrives, then confirm instead of interrogating.*

**Human-assisted: the copilot.** The strongest field evidence for AI in support is on this side of the line. In a study of 5,172 support agents, access to a generative AI assistant raised issues resolved per hour by 14% on average and by 34% for the least experienced agents, and improved customer sentiment [3]. The mechanism the authors point to is the spread of top performers' practice. For personalization, that means the novice agent who picks up Ines's case sees what the best agent would have looked up.

**Async: email replies and proactive notices.** These can take minutes and should. A proactive notice is a personalized decision: who is affected, what they will see, what to do in the meantime. It is not a broadcast. Promises belong here too: "we will tell you when it is fixed" should be a recurring job that checks open promises against telemetry and drafts the follow-up once the fix holds, not a sentence in a ticket. The memory system I built runs scheduled prompts and queued jobs beside the memory, and can hand a long investigation to a provider's hosted agent on a memory-only credential scoped to that job and revoked when it ends (Chapter 13).

**The handoff.** Most support failures happen at the line between machine and person, so draw it explicitly. Route to a human when:

- the customer asks for one (honored on the first request, never looped back to the bot);
- the issue has recurred, or the customer has stated an escalation;
- the answer would state or change a commitment: a policy, a refund, a credit, an exception (these belong on the irreversible-action list, Chapter 18);
- the assistant's evidence is thin or its sources disagree;
- the conversation involves sensitive context.

A good handoff moves state, not transcripts (Chapter 13). The person receives the case record and the assistant's draft, and the customer is told who is taking over and that they will not need to repeat anything. The test is simple: did the human's first message ask for something the system already had?

![A diagram of the handoff line. Support work spans three modes: the AI resolves, the AI drafts and a person decides, or a person owns the case, with the handoff line drawn after the first. Route to a person when the customer asks (on the first request), the issue has recurred or an escalation was stated, the answer involves a policy, refund, credit, or exception, the evidence is thin or sources disagree, or sensitive context is involved. A transcript or nothing crossing the line makes the person start over; the case record and the assistant's draft crossing it lets the customer be told who is taking over and that they will not repeat anything. The test is whether the person's first message asked for something the system already had.](/images/handbook/pb-sp-handoff-line.svg)

*Figure SP.2. Make the handoff rules explicit and hand over the case record, not the transcript.*

**Tone.** Tone rules are governed centrally, not written into each bot's prompt: own the recurrence, drop the upbeat filler during an outage, never sell inside a support thread. One rule is fixed in code: tone may change how something is said, never what the customer is entitled to. In the memory system I built, guidelines are retrieved for the agent on every action (recall the customer, retrieve the rules, act, store the outcome), and every policy edit keeps the version it replaced, so you can show which refund rule was in force the day a reply cited it.

**Disclosure.** Say it is an AI. As of September 2026, the EU AI Act's Article 50 requires that people be told they are interacting with an AI system, unless that is obvious, from August 2, 2026 [4]. Outside the EU it is still the honest default. (Not legal advice; see the companion regulation note.)

**Proactive help.** The evidence is thinner here than the enthusiasm. The best controlled result I know is from onboarding, not incidents: in a 2011 field experiment at a cloud provider, new customers who received proactive guidance were half as likely to churn in their first week, asked about 20% fewer support questions, and used about 47% more of the service over eight months; effects were strongest for less experienced customers [5]. That is one provider and one kind of outreach. For incident notices, I have not found comparable controlled evidence. The decision rule I use: reach out when the customer is affected in a way they will experience, or that changes a decision they are about to make. Do not turn support signals into engagement nudges; Chapter 14's sleeping dogs apply here too. The Customer Success playbook covers the other side of the line: an open severe case pauses success plays until the fix holds, and support hands back what happened and what was promised.

## What can go wrong

**Failure story: The Invented Policy.** In April 2025, users of the coding tool Cursor were being logged out when they switched machines. They wrote to support and got answers from "Sam," which told them the logouts were expected under a policy limiting use to one device. There was no such policy. Some users, who reported assuming Sam was a person, cancelled their subscriptions. A co-founder apologized for an "incorrect response from a front-line AI support bot" [6][7]. A year earlier, a Canadian tribunal had held Air Canada responsible for a bereavement-fare policy its chatbot described wrongly, rejecting the argument that the bot was "a separate legal entity" [8].

Both failures share a shape. The assistant stated a commitment (a policy) that it had not retrieved from the governed source, nothing checked the claim before it was sent, and in Cursor's case the customer did not know they were talking to a model. The fix is not a better prompt. Policy statements are claims that must cite the governed policy they come from, or the assistant must say it will check and hand off (Chapter 15's "specific and true, or honestly general" applied to terms and conditions).

Other risks to name in your design review:

- **The Re-intake Loop.** Every channel and every tier starts intake from zero. The customer becomes the integration layer.
- **The Handoff Cliff.** The bot hands off a transcript, or nothing, and the human starts over. Worse than no bot: the customer has now explained twice.
- **The Stale Fix.** A closed ticket is read as a solved problem, and the next agent apologizes for the wrong thing or, worse, tells the customer it is fixed.
- **The Wrong-Record Greeting.** A merged identity (Chapter 9) greets one customer with another's cases. Continuity amplifies identity errors, so resolve identity before you personalize support.
- **The Pasted Secret.** A password or card number typed into chat travels unredacted to an external model.
- **The Over-familiar Agent.** The assistant opens with everything it knows. Surface only what bears on this issue. Customers expect you to know their tickets; they did not ask to hear their usage history recited (Chapter 19).

### What this does not do

Klarna is the honest benchmark, because it produced both the claim and the correction. In February 2024 it reported that its AI assistant had handled 2.3 million conversations, two thirds of its customer service chats, doing "the equivalent work of 700 full-time agents," with satisfaction "on par with human agents," a 25% drop in repeat inquiries, and resolution in under 2 minutes instead of 11 [9]. Note the wording: equivalent work, not 700 people replaced. In May 2025 its chief executive told Bloomberg that cost "was a too predominant evaluation factor when organizing this, what you end up having is lower quality," and Klarna began recruiting human agents again; the assistant still handled about two thirds of inquiries [10]. A month later he said human customer service is "always going to be a VIP thing" [11].

I read that as a correction of the boundary, not a verdict on the technology. The repetitive majority stayed automated. What broke first was the complex, emotional minority, where quality mattered and a cost metric could not see it. I part company with the "VIP" framing, though. Route people to humans by need and consequence first, and by account value second. A small customer whose business is down deserves a person more than a large customer asking for an invoice copy.

Memory also does not fix the product. Ines's problem was a sync defect. Continuity makes the failure less insulting; it does not make it less of a failure. And the survey numbers above measure frustration, not causation: they tell you customers hate repeating themselves, not how much retention you will buy by stopping it. Measure that yourself.

## How you will know

Klarna's 2024 announcement led with volume and speed; its 2025 correction was about quality. Measure so you see the second first (method in Chapter 21).

- **Re-asked rate.** Log every question the assistant or agent asks, and whether memory held a fresh answer to it. This is the direct measure of this playbook's question, and almost nobody instruments it.
- **Repeat contact within 7 days for the same issue.** The right family of metric; Klarna reported its own version as "repeat inquiries" [9].
- **Handoff quality.** Share of human handoffs whose first message re-asks a known fact; time from "I want a person" to a person.
- **Outcome by severity.** Satisfaction and sentiment for escalated and recurring cases reported separately from the average, because averages hide exactly the cases that broke Klarna's quality.
- **Downstream retention.** Renewal and churn for accounts that had a severe case, against a holdout, over at least a renewal cycle.

Beware containment and deflection rates. A customer who gives up is "contained." That is Chapter 21's Proxy Trap, and support is where it bites hardest.

**Test design.** Randomize by account or by case, not by message, since one customer's contacts influence each other. For continuity and handoffs, a holdout that keeps the current process for a random share of new cases is fair and clean. For proactive incident notices, do not hold back the notice; test timing, format, and channel instead. For proactive guidance of the onboarding kind, a holdout is reasonable, and the cloud-provider experiment above is a usable template [5]. Record for each case what was known, what was asked, and who handled it, so a bad outcome can be traced to the layer that caused it (Chapter 20).

## Reader Q&A

**We have to ask for account details for security. Is that "asking what we know"?** No. Verification protects the customer and is worth one step. The waste is re-intake: asking the customer to rebuild the case you already hold. Verify once per session, and carry the verified state across the handoff so the human does not verify again.

**Can we start without unified customer memory?** Yes. Start with a case brief built from the ticketing system alone: last cases, recurrence, open promises, linked incidents. Precompute it at ticket open and put it in front of both the bot and the agent. Recurrence detection is the first field to get right. Then add telemetry, then the rest of the customer record.

## For your AI

```yaml
playbook: SP
title: "Support"
question: "How do we stop asking customers what we already know?"
concepts:
  - name: Customer as integration layer
    definition: "The failure mode in which the customer must carry facts between channels, tiers, and systems because the company's records do not."
  - name: Case brief
    definition: "A precomputed summary of identity, open and recent cases, recurrence, linked incidents, promises, and stated preferences, built when a signal arrives and read at contact time."
  - name: Confirm, not interrogate
    definition: "State known facts back and ask one confirming question; ask open questions only for what memory cannot answer or what must be verified."
  - name: Handoff line
    definition: "Explicit rules for when an AI resolves, drafts for a person, or yields the case to a person, and the requirement that state, not a transcript, moves with it."
  - name: Support as a hard constraint
    definition: "An open severe case suppresses marketing, sales, and survey programs for that customer until the fix is confirmed."
decision_rules:
  - if: "memory holds a fresh, sourced answer to an intake question"
    then: "state it back and confirm; do not ask it again"
  - if: "a stored fact about the person is past its freshness window"
    then: "ask one confirming question before relying on it"
  - if: "the customer asks for a human"
    then: "route on the first request, with the case record, and tell the customer they will not need to repeat anything"
  - if: "the issue has recurred or the customer has stated an escalation"
    then: "route to a senior human with the case assembled; skip scripted troubleshooting already tried"
  - if: "a response would state or change a policy, refund, credit, or exception"
    then: "cite the governed policy source or hand off; treat as an irreversible action"
  - if: "a ticket is closed but telemetry does not confirm the fix"
    then: "treat the problem as open for routing, tone, and suppression"
  - if: "a customer has an open severe case"
    then: "suppress expansion, marketing, and survey programs until the fix holds"
  - if: "telemetry detects a failure that customers will experience"
    then: "notify affected customers with status, workaround, and next update time; do not hold back notices for testing"
  - if: "a customer message contains secrets, card numbers, or other listed identifiers"
    then: "redact or tokenize before any external model call; never let internal case notes pass the customer-facing serializer"
  - if: "the case brief is incomplete or late at contact time"
    then: "lower specificity and say what is being checked; never guess a cause or a fix status"
  - if: "the interaction is with an AI system"
    then: "disclose it (required in the EU from 2026-08-02 under AI Act Art. 50, unless obvious)"
assessment_questions:
  - "How many questions does a customer answer at first contact that your systems could have answered?"
  - "When a bot hands off to a person, what does the person receive, and does their first message re-ask anything?"
  - "Can your support tools see product telemetry, known defects, and other open cases for the same account?"
  - "Where are promises made in support (callbacks, credits) stored, and can marketing and sales agents read them?"
  - "Which policy statements can your support AI make, and does each cite a governed source?"
  - "Do you report satisfaction for escalated and recurring cases separately from the average?"
patterns: [Case Brief at Signal Time, Confirm Not Interrogate, State-Carrying Handoff, Consequence-First Routing, Incident-Triggered Proactive Notice, Degraded Not Invented, Promise as Scheduled Check]
anti_patterns: [The Re-intake Loop, The Handoff Cliff, The Invented Policy, The Stale Fix, The Wrong-Record Greeting, The Over-familiar Agent, Containment as Success, The Pasted Secret]
metrics: [re_asked_rate, repeat_contact_7d_same_issue, handoff_reask_rate, time_to_human_after_request, csat_escalated_cases, retention_after_severe_case_vs_holdout]
links: [ch9, ch10, ch11, ch13, ch14, ch17, ch18, ch19, ch20, ch21]
maturity_dimension: identity_and_memory
```

## References

1. Zendesk, "Contextual intelligence becomes the new standard for exceptional customer experience in 2026" (CX Trends 2026), press release, 2025-11-18. Surveys fielded June 2025: 6,182 consumers and 5,115 business respondents, 22 countries. https://www.zendesk.com/newsroom/press-releases/contextual-intelligence-becomes-the-new-standard-for-exceptional-customer-experience-in-2026/
2. Salesforce Research, *State of the Connected Customer*, 4th edition, 2020. Survey July 16 to August 18, 2020; 12,000 consumers and 3,600 business buyers, 27 countries. https://c1.sfdcstatic.com/content/dam/web/en_us/www/documents/research/salesforce-state-of-the-connected-customer-4th-ed.pdf
3. Brynjolfsson, E., Li, D., Raymond, L. (2025). "Generative AI at Work." *Quarterly Journal of Economics* 140(2). https://academic.oup.com/qje/article/140/2/889/7990658 ; working paper NBER w31161. https://www.nber.org/papers/w31161
4. EU AI Act (Regulation (EU) 2024/1689), Article 50; Commission FAQ on transparency obligations. https://artificialintelligenceact.eu/article/50/ ; https://digital-strategy.ec.europa.eu/en/faqs/transparency-obligations-under-article-50-ai-act (status as of September 2026)
5. Retana, G. F., Forman, C., Wu, D. J. (2016). "Proactive Customer Education, Customer Retention, and Demand for Technology Support: Evidence from a Field Experiment." *Manufacturing & Service Operations Management* 18(1): 34-50. https://pubsonline.informs.org/doi/10.1287/msom.2015.0547
6. Fortune, "Cursor's AI glitch triggers viral fallout," April 2025. https://fortune.com/article/customer-support-ai-cursor-went-rogue/
7. AI Incident Database, Incident 1039 (Cursor support bot, April 2025). https://incidentdatabase.ai/cite/1039/
8. *Moffatt v. Air Canada*, 2024 BCCRT 149, British Columbia Civil Resolution Tribunal, 2024-02-14. https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html
9. Klarna, "Klarna AI assistant handles two-thirds of customer service chats in its first month," press release, 2024-02-27. https://www.klarna.com/international/press/klarna-ai-assistant-handles-two-thirds-of-customer-service-chats-in-its-first-month/
10. Customer Experience Dive, "Klarna reinvests in human talent for customer service" (reporting a Bloomberg interview of 2025-05-08), 2025-05-09. https://www.customerexperiencedive.com/news/klarna-reinvests-human-talent-customer-service-AI-chatbot/747586/
11. TechCrunch, "Klarna CEO says company will use humans to offer VIP customer service," 2025-06-04. https://techcrunch.com/2025/06/04/klarna-ceo-says-company-will-use-humans-to-offer-vip-customer-service

---

# Playbook PR. Product

*The question: How does the product adapt to each user, and what does the team learn?*

> **Questions this playbook answers**
> - Our onboarding is one checklist for everyone. Should it be personalized, and how far?
> - Which parts of the interface should change per user, and which should never move?
> - How do we personalize inside the product without adding one more stream of interruptions?
> - How do we prove an adaptive product works better than a well-designed static one?
> - We hold thousands of calls, tickets, and usage traces. How does that become evidence for the roadmap instead of the loudest request?

## The moment

On the same Monday last winter, two new users log in to Larkspur Systems for the first time. (Larkspur is the fictional field-service and fleet software company that runs through this handbook: about 40,000 customer accounts, product telemetry, a support desk, ten years of CRM history.)

Marisol is the only dispatcher at a twelve-van plumbing company that signed on Friday. She has never used dispatch software, and she has eleven jobs to route before 8 a.m.

Tom is a fleet administrator at a 400-seat account migrating from a competitor. His team has already imported 380 routes, and in the sales call his director said, in passing, that the old system's weak point was offline mode at rural job sites.

Both see the same nine-step checklist. Step one is "Create your first route," which assumes Marisol knows what a route template is. Tom needs none of it. He dismisses the checklist, then the tooltip tour, then the "Did you know?" panel, and goes looking for offline settings, which live three menus deep. By Wednesday he has opened a ticket. By Friday, the product team is in its winter planning meeting, months before the spring release that would rebuild offline mode, ranking feature requests by votes on the ideas board. Offline mode has eleven votes. The phrase "offline" appears in 214 call transcripts and support tickets from the last two quarters, across 96 accounts, and nobody in the meeting knows it. (All Larkspur figures here are illustrative.)

Larkspur knew enough to treat the two users differently, and more, in aggregate, about what to build next. It used neither.

## The decision

In marketing and sales, the unit of personalization is a message. In the product, it is a **slot**: a place in the interface where the right content depends on who is looking. The onboarding step, the empty state, the default view, the suggested next action, the help panel, the order of items in a "recommended" row. The decision, per user and per slot, is what fills it now.

The action space is wider than "show a tip," and it follows the menu in Chapter 14:

- **Show or reorder** the contents of a designated slot (the onboarding path, a recommended row).
- **Prefill a default** (a route template sized for a twelve-van fleet; offline sync on for accounts that work rural sites).
- **Defer** a feature until it is useful (route optimization after the first week of routes, not on day one).
- **Ask** one question when the evidence is thin ("How many vehicles do you dispatch?").
- **Route** to a person (the onboarding specialist, when a 400-seat migration stalls).
- **Stay out of the way.** No tip, no panel, no tour. For Tom, most of the week.

Here is the sentence this playbook turns on: **in the product, personalization is judged by the work the user finished, not by the suggestion they clicked.** A product is the one channel where the customer is already doing the work. Every other channel has to earn attention; the product has to avoid spending it. A tooltip is a message too, and it draws on the same per-person attention budget as the nurture email and the account manager's check-in.

The second half of the playbook's question is a different decision, made by the team rather than the system: which problems to solve for everyone. The same memory that tells the product that Tom needs offline settings tells the roadmap that 96 accounts do.

## What you need to know

Product personalization is unusual in one way: its best signal is generated by the product itself. It is also usually B2B-shaped, so memory has two levels.

**The account.** What the customer bought and configured: plan and entitlements, integrations, fleet size, imported data, whether this is a migration. What they have said: the sales call that mentioned rural sites, open tickets, the renewal date. This comes from the CRM, billing, support, and call notes, resolved to one account (Chapter 9).

**The user.** Role (dispatcher, technician, administrator, manager), what they have done, and what they have declared. Telemetry becomes typed properties: `first_route_dispatched_at`, `routes_created_last_7d`, `offline_mode_enabled`, `tour_dismissed`. Following the Boundary Rule in Chapter 10, anything you would filter or count on is a property. "Said on the kickoff call that her technicians lose signal in the valley" is an evidence-level memory, with a source and a date.

Three signals carry most of the weight, in this order.

1. **What the user declared.** For a new user there is no history to model, which is the cold-start problem Chapter 2 describes. The cheapest fix is to ask one or two questions whose answers change the path: role, and fleet size or job type. Declared data is also the most defensible data to use (Chapter 7). Spotify offers a telling example from the other end of a product's life. After ten years of Discover Weekly, a playlist built entirely on inferred taste that had passed 100 billion streams, it added a control letting listeners refresh the playlist by choosing from up to five genres, themselves personalized from listening history [1]. Mature inference did not remove the need to let people say what they want.
2. **What the user did.** Telemetry is the strongest behavioral signal most companies own, and it is first-party and fresh. It is also narrow: it records the click, not the reason.
3. **What the account said.** Calls and tickets hold the reason. "Offline mode fails at rural job sites" explains why Tom goes looking for settings, and warns the product not to promote a feature that is broken for this account (the product-side version of Chapter 14's upsell that should not have gone out).

Freshness matters more here than anywhere, because product state changes by the minute. A checklist that asks a user to do what they finished yesterday reads as a system that is not paying attention. Give each property a freshness policy (Chapter 11), and let the event stream, not a nightly batch, update the ones that drive onboarding.

## The action

### Adapt the contents, never the map

The oldest evidence on adaptive interfaces is a warning. In controlled lab studies at the University of British Columbia, Findlater and McGrenere compared pull-down menus whose top section was static, adaptable (the user chose what went there), or adaptive (the system reordered by frequency and recency). The static menu was significantly faster than the adaptive one, and most participants preferred the adaptable version [2]. People build a map of an interface, and a system that rearranges the map makes them slower even when its predictions are good.

The result is two decades old and about menus, so treat it as a caution, not a law. It points to the design the closing chapter reaches from the other direction: a **stable frame with variable parts** (Closing). Navigation, menu order, and the location of controls stay fixed. Designated slots vary. Users get a visible way to change what the system chose, which is Spotify's lesson again. Chapter 3 gives the fuller research record, including where accurate adaptation wins, and Chapter 16 extends the frame to generated pages, screens, and notifications.

A governed personalization engine I built uses the same shape for pages: a static shell with roughly 8 to 12 text zones, never navigation or legal copy. Each zone has a contract (purpose, allowed and forbidden context, length limit, checks, approved fallback). A product slot needs one too.

Adapt the contents, never the map.

![Two new users see the same product frame: the same navigation (Dispatch, Routes, Jobs, Settings), the same toolbar (Reassign, Hold, Add job), and the same cancel, downgrade, and turn-off controls, fixed for everyone. Only the designated slots differ. Marisol, a sole dispatcher with twelve vans, gets a prefilled route template for 12 vans and route optimization deferred until she has a week of routes. Tom, a fleet administrator at a 400-seat account migrating from a competitor, gets offline sync prefilled on for rural job sites and no tour, tip, or panel. Slots vary per user and can be overridden; the map never moves.](/images/handbook/pb-pr-stable-frame-slots.svg)

*Figure PR.1. Stable frame, variable parts: the same map for everyone, with only designated slots adapting to the person.*

### Personalize the core loop, not the chrome around it

The strongest public examples of product personalization change the work itself, not the tips about it.

Duolingo's Birdbrain model estimates both how much each learner knows and how hard each exercise is, and the lesson generator uses those estimates to build sessions at the right difficulty for that learner. By October 2020, more than 20% of lessons were personalized this way, and the company reported that its A/B tests showed learners who got Birdbrain-built lessons learned more, did more lessons, and were more likely to return day after day [3]. Too hard and the learner quits in frustration; too easy and the lesson teaches nothing. The personalization lives exactly where the product delivers its value.

Netflix personalizes the artwork for each title per member: the same film can show its romantic leads or its comedian, depending on what that member watches. The team treated it as a contextual bandit, evaluated policies offline by replay before exposing members, and looked at "the quality of engagement to avoid learning a model that recommends 'clickbait' images: ones that entice a member to start playing but ultimately result in low-quality engagement" [4]. The artwork is evidence, offered at the point of choice, of why this title might fit this person, and it is held to what happens after the click.

Netflix's 2025 paper on the value of its recommender adds a useful fact about where product personalization pays. Replacing the production system with simpler baselines reduced engagement by roughly 4% (matrix factorization) to 12% (popularity), and most of the gain came from targeting, with the largest effects for titles of middling popularity [5]. Everyone already finds the hits. Personalization earns its keep on the long middle of a product, the features and content that are right for some users and invisible to them.

For Larkspur, the core loop is dispatch. Marisol's first week is not improved by a better tour; it is improved by a route template prefilled for twelve vans and plumbing job types, and by route optimization appearing after she has dispatched a week of routes by hand and has something to optimize.

### Where generation fits, and where it does not

Language models make two product moves newly cheap. Conversational onboarding (the product asks what the user is trying to do and answers in their terms) ships now (Chapter 5). Generated interface does not, at product speed: Google's own generative UI researchers report generation can take more than a minute, with occasional inaccuracies, and raters still preferred pages built by human experts [6]. Against a budget of about 0.1 second to feel instant (Chapter 17), fully generated screens are a lab demo as of September 2026 (Chapter 16's Generative Experience Ladder). An in-app notification is a decision too: whether, what, when, and how loudly, drawn from the same per-person budget as email (Chapter 16).

The workable shape today: precompute each user's slot contents on events (signup, role declared, first route dispatched, ticket opened) and let the live request only select among them (Precompute-then-serve, Chapter 17). Generated text (a help answer, an empty-state explanation) goes into typed zones, and every claim about the user's own data ("your team dispatched 40 routes last week") is grounded in telemetry or left out (Chapter 15). Render only components from a governed catalog. Specific and true, or honestly general.

Generated text is untrusted input to the renderer. In that engine every value is coerced to its zone's type and escaped, and a `javascript:` link is refused however confidently a model returned it. A failed check renders the fallback; thin evidence steps the slot down (user, account, segment, static default), so specificity degrades before trust does.

Expect memory to say no, too. The self-hosted memory system I built refuses work it cannot run, with a retry-after signal, rather than falling behind, and budgets reads apart from writes so a burst of saves does not stop lookups. On the interactive path a refusal is ordinary: render the precomputed default and never block the task.

### Memory to roadmap

The second output of this playbook is not shown to users. It is a quarterly (or continuous) roll-up of what the memory knows about problems, delivered to the product team.

Votes on an ideas board measure who visits the ideas board. Calls, tickets, onboarding answers, and usage traces measure the customer base. The roll-up has four rules:

1. **Count accounts, not mentions.** One angry administrator who writes nine tickets is one account.
2. **Separate the job from the request.** "Add a sync button" is a request; "technicians lose work when they lose signal" is the job. Cluster by job.
3. **Keep the provenance.** Every theme links to the memories behind it (account, date, source, quote), so a product manager can read the twenty best examples before believing the number. Background curation (the dreaming cycle in Chapter 11) can maintain these theme summaries while the system is idle. In my memory system, a new consolidation recipe only proposes until an operator promotes it, every write is stamped and ledgered, corrections overwrite in place with history kept, and curation queues behind live requests, so tidying memory never slows the product.
4. **Read the personalization logs as product data.** If the same adaptation fires for most users (say, 80% of dispatchers hide the analytics widget in week one), it is not personalization. It is a bug report about your default. Chapter 6 makes the same point from the other side: personalization is not a fix for a bad default.

![Calls, tickets, onboarding answers, and usage traces feed one shared memory holding the account (plan, migration, what was said) and the user (role, what they did, what they declared) as typed properties with sourced evidence. The same memory serves one user now, telling the product that Tom needs offline settings so it can prefill offline sync on, and serves the roadmap each quarter. Drawn to scale, offline mode had 11 ideas-board votes but 214 mentions across calls and tickets from 96 accounts. The roll-up counts accounts, not mentions, clusters by the job rather than the request, and links every theme to its evidence.](/images/handbook/pb-pr-memory-to-roadmap.svg)

*Figure PR.2. The memory that adapts the product for one user also counts, by account, what the roadmap should fix for everyone.*

## What can go wrong

### Failure story: The Shifting Map

Larkspur's first adaptive feature was a dispatch-board toolbar that reordered itself by each dispatcher's frequency of use. Offline tests showed it predicted the next click well. In production, experienced dispatchers began mis-clicking during the 7 a.m. rush: "Reassign" had moved to where "Hold" used to be. Tickets about "jobs disappearing" rose, and the support team could not reproduce them, because each dispatcher's toolbar was different and nobody had logged what the user saw. The fix was to freeze the toolbar, move the adaptation into a "suggested for this job" slot beside it, and log every rendered variant with the user and time. Findlater and McGrenere could have predicted the first half. The logging gap was Larkspur's own. (Fictional composite.)

A personalized screen the support team cannot see is a bug nobody can reproduce.

### Governance risks specific to the product

**Upsell inside the workflow.** An in-app prompt for a paid add-on is a sales action in a work surface. It obeys the same hard constraints as an email (suppressed during an open escalation, counted against the contact budget) and never blocks a task (Chapter 14).

**Dark patterns.** The product is where manipulative design is easiest to build and hardest to see. The FTC's 2022 staff report catalogued the common forms: disguised ads, hard-to-cancel subscriptions, buried terms, and tricks that extract data [7]. A personalized product must not learn them. Keep "cancel," "downgrade," and "turn off" out of the adaptive layer entirely: fixed location, fixed wording, for everyone.

**Leakage between users in one account.** B2B memory is shared at the account level; product surfaces are seen by individuals. An administrator may see team activity; a technician should not see a colleague's performance reflected in "people like you" suggestions. Scope what each role's surfaces may read, and test it (Chapter 18). Enforce it below the application where you can. In my memory system, each person and agent gets a scoped key, row-level security in the database fails closed, and an in-product assistant connected over MCP is offered only its role's tools: a read-only assistant has no write tools to misuse.

**Employee data used on employees.** Fleet telemetry includes technician location and timing. Routing jobs with it is the product's purpose; commenting on a technician's pace on their own screen is a different purpose, and it will feel like monitoring. Run the appropriateness check in Chapter 19 before any telemetry field drives a user-facing message. The same purpose test applies to the roadmap roll-up: check contracts and data-processing terms before customer conversations feed product analysis, and aggregate before sharing. (This is design guidance, not legal advice.)

### What this does not do

Evidence on onboarding personalization is thin, and I want to be plain about it. Vendor posts claim large activation lifts from personalized onboarding; in this research pass I could not trace one to a controlled study with a published method. The academic work I found is small: one randomized online experiment with 150 participants found personalization cues amplified the effect of commitment cues on intention to use a mobile app, which is a stated intention, not retention [8]. The public examples with real experiments (Duolingo, Netflix) personalize the core product, not the onboarding tour. The Software Companies and Customer Success playbooks reach the same verdict; the closest controlled evidence they cite, a cloud provider's proactive guidance for new customers, tested help, not personalization.

Personalization also cannot rescue a product that is hard for everyone. If most new users stall at the same step, redesign the step.

## How you will know

**Pick an outcome that is the job, not the click.** For Larkspur: time to first dispatched route; seats active in week four; jobs completed offline at rural accounts. Guardrails: tickets per new account, dismiss and undo rates on adaptive slots, and task time for experienced users, who pay first for a moving interface.

**Randomize at the unit of influence.** Users in one account share configuration and train each other, so account-level assignment is cleaner for onboarding and defaults (Chapter 21). Individual slot content (which recommended item, which help answer) can be tested per user.

**Do the power arithmetic before launch.** Suppose 40% of new accounts dispatch a route in week one and you want to detect a rise to 44%. With 80% power at 5% significance, that needs roughly 2,400 accounts per group. If Larkspur signs about 500 new accounts a quarter (illustrative), a 50/50 account-level test takes more than two years. The honest options from Chapter 21 apply: test a more frequent outcome, test per user where influence does not travel, pool quarters, or label the decision as judgment.

**Do not confuse the design space with the test plan.** Ten slots with five eligible options each allow 5^10, about 9.8 million screens per user context. That number measures what the system can express, not what anyone has tested. The practical program is a few strong hypotheses, tested where the arithmetic above allows.

**Use bandits inside slots, holdouts around the program.** A contextual bandit is well suited to choosing slot contents, as Netflix did with artwork, provided its reward is engagement quality, not the click, and it keeps an exploration floor so it does not become the Feedback Echo of Chapter 21. Whether adaptive onboarding beats the static version is a holdout question: keep a small share of new accounts on the best static flow, permanently.

**Log what each user saw.** Every rendered variant and the decision that chose it goes in the decision trace: support reproduces problems from it, analysts reweight biased data with it, and the roll-up learns from it which defaults are wrong. Review the rendered screen, not only the stored values: a stale template or cache can render correct data wrong.

**Measure the roll-up by what it changed:** how often a theme's evidence changed a priority, and whether the shipped fix moved the job-level outcome for the accounts that raised it.

## Reader Q&A

**Should we ask users questions at signup, or infer everything?**
Ask one or two whose answers change the path, then infer. A question that changes nothing is friction; a question that saves a wrong week is service. Store the answer as declared data with a date, and let behavior revise it.

**Should an LLM generate our interface?**
Not the frame, and not yet at interaction speed. Let models write into typed zones and choose among governed components; precompute what a waiting user needs.

**We are sales-led with few users per account. Does this apply?**
Yes, with the weight shifted to the account: migration status, integrations, and what was said in the sales process are richer signals than a thin usage history.

**Should product memory run in our own infrastructure?**
It is a choice, not a requirement. My memory system runs as one container on your own Postgres with pgvector (any cloud, on-prem, or air-gapped), with a model per function, local ones included. You then own backups, scaling, and a database that is a single point of failure unless made highly available. Self-hosting alone is not compliance (Chapter 19).

**Who owns the roadmap roll-up: product, research, or customer success?**
Product owns the decision; the roll-up is a shared service over the same memory the other functions read. The Research and Strategy playbook covers rolling individual insight into market and ICP strategy; this one covers product priorities.

## For your AI

```yaml
playbook: PR
title: Product
question: "How does the product adapt to each user, and what does the team learn?"
concepts:
  - name: Slot
    definition: "A designated place in the product interface whose contents depend on who is looking; the unit of in-product personalization."
  - name: Stable Frame, Variable Parts
    definition: "Navigation, menu order, and control locations stay fixed; only designated slots adapt, and users can override."
  - name: Core-Loop Personalization
    definition: "Adapting the work the product delivers (difficulty, defaults, templates, ranking) rather than the tips and tours around it."
  - name: Memory-to-Roadmap Roll-up
    definition: "Aggregating account and user memories into job-level themes, counted by account, with provenance to source evidence."
  - name: Slot Contract
    definition: "Per-slot purpose, allowed and prohibited context, length limit, checks, and an approved fallback; generated values are coerced, escaped, and link-checked before render."
  - name: Default Signal
    definition: "When one adaptation fires for most users, it indicates a wrong default for everyone, not a personalization opportunity."
decision_rules:
  - if: "an adaptation would move navigation, menu order, or the location of a control"
    then: "do not adapt it; move the adaptation into a designated slot beside the stable control"
  - if: "a new user has no usage history"
    then: "ask one or two questions whose answers change the path; store answers as declared data with a date"
  - if: "an in-product prompt promotes a paid feature"
    then: "apply the same hard constraints and contact budget as outbound messages; suppress during open high-priority escalations"
  - if: "a feature has an open defect or ticket for this account"
    then: "do not promote it to that account's users"
  - if: "the same adaptation is applied to most users of a role"
    then: "change the default for everyone and report it to the product team"
  - if: "a surface is on the interactive path (about 0.1 s budget)"
    then: "precompute slot contents on events; select live; never generate the frame at request time"
  - if: "a memory read is refused or times out on the interactive path, or a generated value fails its slot checks"
    then: "render the precomputed or approved static default; never block the user's task"
  - if: "generated text makes a claim about the user's own data"
    then: "ground it in telemetry or memory with provenance, or render the honestly general version"
  - if: "a control concerns cancel, downgrade, consent, or turning a feature off"
    then: "exclude it from the adaptive layer; fixed location and wording for all users"
  - if: "customer conversations feed product analysis"
    then: "check contracts and purpose, aggregate by account, keep links to evidence"
assessment_questions:
  - "Which slots in your product could vary per user, and which controls must never move?"
  - "What does your onboarding ask, and does each answer change what the user sees?"
  - "Which telemetry events are stored as typed, fresh properties that surfaces can read?"
  - "Can support see exactly what a given user was shown, and why?"
  - "Is there a static holdout for adaptive onboarding, randomized by account?"
  - "How does evidence from calls, tickets, and usage reach roadmap prioritization today, and is it counted by account?"
  - "Which in-product prompts are commercial, and do they share a contact budget with outbound channels?"
patterns: [Stable Frame Variable Parts, Core-Loop Personalization, Precompute-then-serve, Memory-to-Roadmap Roll-up, Governed Component Catalog, Specificity Ladder, Do-Nothing Option]
anti_patterns: [The Shifting Map, The Tour Nobody Needed, Upsell in the Workflow, Votes as Evidence, Design Space as Test Volume, The Proxy Trap, The Feedback Echo]
metrics: [time to first value by role, week-4 active seats, dismiss and undo rate per slot, task time for experienced users, tickets per new account, share of roadmap priorities changed by evidence]
links: {cold_start: 2, data_types: 7, identity: 9, memory: 10, freshness: 11, decision_layer: 14, generation: 15, latency: 17, governance: 18, privacy: 19, measurement: 21, generative_experiences: 16, forecast: closing}
maturity_dimension: experience_generation
```

## References

1. Spotify Newsroom (2025-06-30). "Discover Weekly Turns 10: Celebrating 100 Billion Tracks Streamed and a Decade of Personalized Discovery." https://newsroom.spotify.com/2025-06-30/discover-weekly-turns-10-celebrating-100-billion-tracks-streamed-and-a-decade-of-personalized-discovery/
2. Findlater, L., and McGrenere, J. (2004). "A Comparison of Static, Adaptive, and Adaptable Menus." CHI 2004. https://dl.acm.org/doi/10.1145/985692.985704 ; thesis version: Findlater, L. (2004), "Comparing Static, Adaptable, and Adaptive Menus," University of British Columbia. https://www.cs.ubc.ca/labs/imager/th/2004/Findlater2004/Findlater2004.pdf
3. Duolingo Blog (2020-10-07). "Learning how to help you learn: Introducing Birdbrain!" https://blog.duolingo.com/learning-how-to-help-you-learn-introducing-birdbrain
4. Chandrashekar, A., Amat, F., Basilico, J., and Jebara, T. (2017-12). "Artwork Personalization at Netflix." Netflix Technology Blog. https://netflixtechblog.com/artwork-personalization-c589f074ad76 ; RecSys 2018 talk: https://dl.acm.org/doi/10.1145/3240323.3241729
5. Zielnicki, K. et al. (2025, rev. 2026). "The Value of Personalized Recommendations: Evidence from Netflix." arXiv:2511.07280. https://arxiv.org/abs/2511.07280
6. Leviathan, Y., Valevski, D., Natchu, V., and Matias, Y. (2025-11-18). "Generative UI: A rich, custom, visual interactive user experience for any prompt." Google Research. https://research.google/blog/generative-ui-a-rich-custom-visual-interactive-user-experience-for-any-prompt/
7. Federal Trade Commission (2022-09). "Bringing Dark Patterns to Light." Staff report. https://www.ftc.gov/system/files/ftc_gov/pdf/P214800+Dark+Patterns+Report+9.14.2022+-+FINAL.pdf ; press release: https://www.ftc.gov/news-events/news/press-releases/2022/09/ftc-report-shows-rise-sophisticated-dark-patterns-designed-trick-trap-consumers
8. Terres, P., Klumpe, J., Jung, D., and Koch, O. (2019). "Digital Nudges for User Onboarding: Turning Visitors into Users." ECIS 2019 Research Papers, 125. https://aisel.aisnet.org/ecis2019_rp/125/

---

# Playbook SW. Software Companies

*The question: How does a software company personalize when its richest signal is how people use the product?*

> **Questions this playbook answers**
> - We are product-led, sales-led, or somewhere in between. What does each motion actually personalize?
> - How do raw product events become something an agent can read and decide on?
> - Our team swears by an activation "magic number." Is it real, and how do we check?
> - Which trials should a salesperson call, which should the product handle, and which should we leave alone?
> - How do we act on low adoption without talking a quiet account into cancelling?

## The moment

*Larkspur Systems is a fictional composite company used throughout this handbook: field-service and fleet software, about 40,000 accounts and 250,000 contacts. Every person and account below is fictional, and all Larkspur figures are illustrative.*

Larkspur sells to mid-market and large fleets through account executives. Alongside that motion, it runs a 14-day free trial, no card required, for fleets under 25 vehicles. About 900 trials start each month. About 5 percent convert to paid within 60 days.

On one Monday, two trials start.

Aisha owns Copperfield Electric, nine vans. She imports 40 jobs on day one, dispatches her first route on day two, and by day four three of her technicians have the mobile app installed. She never opens the help center.

Wes is an IT analyst at Northgate Waste, a 350-truck hauler. He signs up, reads the API documentation twice, opens the integrations page, and invites nobody. He is building a shortlist for a procurement review that his director will run in six weeks.

The trial program treats them identically: the day-three email ("Don't forget to invite your team!") and the day-twelve email ("Your trial ends soon: 20% off if you upgrade today"). The lead score, which runs on company size, flags Northgate, and an SDR calls Wes about routing pain he does not have. His integration question goes unanswered, and the shortlist is drafted without Larkspur. Aisha upgrades on day eleven and takes a 20 percent discount she would not have missed.

Meanwhile, Pinecrest Landscaping, a paying account, uses 23 of the 80 seats it bought, with renewal five months out. Nobody has noticed.

Larkspur had watched every click. It had remembered none of them in a form anyone could act on.

## The decision

A software company's decisions follow the lifecycle: **activate** (reach first value), **convert** (free to paid), **expand** (more seats, usage, products), and **retain** (keep using what was bought). For each person and account, the options are Chapter 14's menu: show something in the product, send something, route to a human, ask one question, wait, or do nothing.

What changes by company is who makes the decision and where it lands. That is the go-to-market motion.

| | Product-led | Hybrid | Sales-led |
|---|---|---|---|
| Unit of personalization | The user, then the workspace | The workspace, then the buying group | The account and its buying group |
| Richest signal | Usage in the first days | Usage plus who the account is | Conversations, contracts, deployment |
| Main surface | In-product and lifecycle email | In-product, plus a human at the right moment | The seller's and CSM's brief |
| Characteristic mistake | Nudging everyone toward one number | Calling the wrong trial | Ignoring usage until renewal |

Product-led growth, a term coined at OpenView around 2016 by Blake Bartlett, names a motion in which the product is "the primary driver of customer acquisition, conversion, and expansion" [1]. Public filings show it working. Slack's 2019 S-1 reported more than 500,000 organizations on its free plan, more than 88,000 paid customers, and adoption "typically initially driven bottoms-up, by end users" [2]. Zoom's said 55 percent of its 344 customers paying more than $100,000 "started with at least one free host" [3]. Atlassian's 2015 F-1 said it had no "traditional, quota-carrying sales personnel" [4].

Most companies end up hybrid: OpenView's 2023 benchmark survey, as summarized by Wing VC (the original is offline), found 47 percent of product-led companies had added sales before $1 million in annual recurring revenue [5]. Larkspur's Monday is the hybrid's typical failure: the product generated the signal, and the sales process ignored it.

Here is the sentence this playbook turns on: **usage is the most honest thing a customer tells you, and the easiest to misread.** It records what people did, never why. Aisha's clicks meant she needed nothing; Wes's meant he was evaluating for someone else and needed a technical answer. The same event stream supported opposite decisions, and the company read neither.

## What you need to know

### Events are not memory until they are typed

A product emits millions of events, and an agent deciding what to do for Northgate should not read them. The step most software companies skip is turning usage into memory, in three forms.

1. **Raw events stay where they are,** in the product database or warehouse. They are the audit trail, not the working record.
2. **Derived, typed properties** land on the user and the account: `first_route_dispatched_at`, `technicians_active_7d`, `seats_paid`, `seats_active_30d`, `integration_docs_viewed`, `trial_day`, `activation_milestone_reached`. By the Boundary Rule in Chapter 10, anything you would filter, count, or trigger on is a property. "Trials on day 5 or later with no dispatched route" is a WHERE clause.
3. **Evidence memories** hold what usage cannot: Wes's reply that the review is about integration with the city's work-order system, or Aisha's note in the in-app survey that her last tool "made me route on paper anyway." Each carries a source and a date (Chapter 11).

Two rules keep this honest. Usage properties are written by code as exact values, never inferred by a model. In the self-hosted memory system I built, structured values are upserted with no extraction step, and an operator can list fields the extractor may never infer, so a model reading a call transcript cannot overwrite a counted value with "probably 20 active users." And each property declares how it changes: `seats_active_30d` replaces, while milestones accumulate with their dates, so "when did this account first reach value?" stays a lookup.

Keep two levels, user and account (Chapter 9). The individual activates; the account buys and renews. Northgate's scoring rule saw only the account, and the SDR saw only the user.

The evidence is real but modest: a 2024 study of 3,959 business-software subscriptions found that usage features improved churn prediction over firmographic, transactional, and support data alone [6]. That says usage belongs in memory, not that any intervention on it works.

![Usage becomes memory before it becomes a decision. Raw product events such as job imported, route dispatched, and API docs viewed stay in the warehouse; code derives typed properties from them as exact values, never inferred, such as first route dispatched at, technicians active in 7 days, and seats active over seats paid. Conversations (calls, replies, surveys) become evidence memories, such as why the user is evaluating, each with a source and date. The decision layer reads both forms and chooses among show in product, send, route to a person, ask one question, wait, or do nothing, and every action and response is written back to memory.](/images/handbook/pb-sw-usage-to-decision.svg)

*Figure SW.1. Events become typed properties and sourced evidence before any agent decides; clicks alone never tell you why.*

### Magic numbers are markers until you test them

Stewart Butterfield told First Round Review in 2015 that after a team had exchanged 2,000 messages, "93% of those customers are still using Slack today" [7]. Facebook's "7 friends in 10 days" is attributed to Chamath Palihapitiya's 2012 growth talk; I could not find a primary transcript [8]. Both are real observations of a threshold that separated retained from churned users. Neither shows that pushing a user across it causes retention. Slack observed; it did not prove.

Benn Stancil, writing at Mode in 2015, argued that the real work is testing whether moving the behavior moves retention [8]. Mixpanel, which sells the analytics used to find these thresholds, now calls magic numbers "an illusion": correlational stories useful for aligning a team [9]. Amplitude's 2025 benchmark of 2,600-plus companies found 69 percent of top-quartile day-7 products were also top performers at three months [10]: vendor data, and a correlation between two measures of retention, not a lever.

So a milestone found this way is a **candidate**. Randomize a nudge toward it against no nudge. If moving the behavior does not move the outcome, you have found a marker of fit, not a cause.

The best experimental evidence I know on early help is not a magic number. In a 2011 field experiment at a public cloud provider, 366 of 2,673 new customers received proactive guidance on basic features; their first-week churn halved and their usage over eight months was 46.57 percent higher, most of all for less experienced customers [11]. Help aimed early at the person who is stuck beats pressure aimed at a number.

### A product-qualified lead is a decision, not a score

Tomasz Tunguz defined the product-qualified lead in January 2013: "potential customers who have used a product and reached pre-defined triggers that signify a strong likelihood to become a paying customer" [12]. OpenView's 2023 survey linked tracking PQLs with a 61 percent higher likelihood of fast growth [5], a correlation that says as much about companies that instrument well as about PQLs.

The trap is Chapter 14's: a likelihood is not a decision. Aisha was the likeliest trial to convert, a Sure Thing, and her discount was pure cost. A useful PQL asks which accounts a human would change the outcome for: high value at stake and a need the product cannot meet alone, such as an integration question, a security review, or a multi-site rollout. Wes was the PQL, and the signal was IT-shaped behavior, not company size.

In the governed personalization engine I built, scoring returns a score, a band, a rationale, a recommended priority, and a suggested approach, and deterministic checks catch contradictions such as a score of 88 labeled "low." The rationale is what lets a seller disagree with the score.

## The action

### Trials: act on the state, not the calendar

Replace the day-3 and day-12 emails with a small decision table, re-evaluated when a trial's state changes:

| Trial state (from memory) | Default decision |
|---|---|
| Activated, self-serving, small team | Stay out of the way. No discount. Offer billing help near the end. |
| Activated, larger fleet or several sites | Route to a person with a brief: what they did, what they declared, what they have not tried |
| Stalled at a known step | Help at that step, in the product; or ask one question |
| Evaluator pattern (docs, API, security pages, no invites) | Offer a technical resource, not a discovery call |
| No meaningful activity | One honestly general note, then silence |

The routed brief is where hybrid companies win or lose. In the engine I built, the seller brief holds an account snapshot, why-now signals, conversation ideas, and a draft opening, and it is rejected if its load-bearing sections come back empty, however "complete" the generation reported. The boundary that matters most: **rep-internal is not customer-facing**. The brief can say "likely evaluating a competitor's API"; the email to Wes cannot.

Whether a discount belongs in the flow is a test, not a habit. Benchmarks need their labels: Poyar and Rachitsky's 2023 survey of 1,000-plus B2B products put good free-trial conversion at 8 to 12 percent within six months [13]; the 2026 report from Poyar with ChartMogul and ProductLed (200 self-selected products) puts it at 4 to 6 percent without a card and 25 to 35 percent with one [14]. The categories differ; do not blend them. And offer design matters too: in a randomized field experiment, adding a second premium version raised sales of the existing one [15].

### Expansion: seats and usage, read both ways

Expansion signals in software are concrete: a seat limit reached, a second site configured, usage nearing a plan boundary. A prompt at the limit is a real-time "show" decision; keep it small and factual (Sales II: Growth and Return covers the account-level motion). For Copperfield, a tenth van is the trigger, and the product already knows.

Consumption pricing tightens the loop both ways. Snowflake reported net revenue retention of 158 percent in its 2020 S-1 and 125 percent for fiscal 2026 (as of January 31, 2026) [16]; customers who expand on usage also optimize on usage. Label benchmarks: the KeyBanc and Sapphire private-SaaS survey put median net retention near 101 percent (October 2024 release) [17], while ICONIQ describes "~110-120%" across its growth-stage set [18]. Slack's 143 percent in its S-1 [2] is an outlier, and definitions differ by company.

One caution belongs on every usage-based nudge. In a wireless-provider field experiment, Ascarza, Iyengar and Schleicher proactively recommended cheaper plans fitted to customers' usage; three-month churn was 10 percent in the treatment group against 6 percent in the control [19]. A genuinely helpful message made customers look at what they paid.

### Low adoption: help first, and beware Sleeping Dogs

The ratio `seats_active_30d / seats_paid` is the most useful retention property most software companies do not store. Low adoption is common: Pendo's 2019 analysis of its customers' products found 80 percent of features "rarely or never used" [20] (vendor data, three-month window).

Acting on it is where the Sleeping Dogs live (Chapter 14). "You're only using 23 of your 80 seats" invites Pinecrest to right-size at renewal, or to review the whole contract; the least happy accounts top the risk score, and contact can crystallize a decision to leave. Help aimed at the reason comes first: which roles never activated, where they stopped, whether the technicians' app was installed. For Pinecrest, an in-product prompt for the administrator to invite the crew leads, and a CSM brief that opens with adoption, not renewal. In a telecom field experiment, proactive service cut churn, but adding a cross-sell to the same contact undermined the effect when customers doubted the motive [21]. Keep adoption free of the sale. The Customer Success playbook covers adoption plays, health scores, and renewals.

### In the product: experiences and notifications

The chapter "Beyond the Message: Generative Experiences" sets out how much of a page, an app, or a notification can be composed per person, and the Product playbook covers stable frames with adaptive slots. Three rules carry most of the weight here. Adapt the slots (empty state, next step, help panel), never the map. Precompute on events so the live request only selects. And treat every in-app notification as a decision: whether, what, when, and through which channel.

Restraint pays. Duolingo's growth team "could not increase the quantity of notifications without strong justification and CEO approval" [22]; within fixed volume, a bandit choosing among templates and penalizing repetition lifted daily active users 0.5 percent and new-user recurring retention 2 percent over a strong baseline [23]. An in-app nudge draws on the same per-person budget as email and the seller's call (One Customer, One Conversation). In my own engine, notification routing is roadmap, not shipped; the design I would pursue decides it from the same memory and rules as every other surface.

### The handoff: one record across product, sales and success

Product, sales, and customer success each own part of the lifecycle, and each has kept its own copy of the customer. The handoff is where value leaks: the seller who calls without knowing what the trial did, the CSM who re-teaches setup the customer finished in the trial, the upgrade banner that appears the week the account executive agreed a price.

The fix is One Customer, One Conversation's: one memory, one ledger of what was said and promised, one set of rules, every surface compiled from the same context. In the engine I built, once a signup provides identity, the same context feeds the score, the seller brief, routing, and follow-up content, so brief and email cannot disagree on facts. Timing can be simple: the memory system I built runs scheduled recurring prompts, so a weekday-morning job can tell each account executive which trials crossed a milestone overnight and which evaluators await an answer. Every touch and response is written back.

A trial is a conversation the customer starts with your product. The handoff decides whether your people join it or start a new one.

![A grid of three go-to-market motions against four decisions: activate, convert, expand, retain. Product-led, where the product acts: help at the stuck step, the product converts with no human, a prompt at the seat limit, in-app adoption help. Hybrid, where the product acts and then a person: activation in the product, high-value trials routed to a person with a brief (highlighted), the usage signal sent to the account owner, adoption help before renewal. Sales-led, where people act with briefs: guided onboarding, the seller's brief, the account plan, the CSM's brief. One shared memory of usage and conversations feeds every cell.](/images/handbook/pb-sw-motion-map.svg)

*Figure SW.2. Every motion personalizes the same four decisions; what differs is who acts on the usage signal.*

## What can go wrong

### Failure story: The Chased Number

Larkspur's analysts found that trials dispatching 25 routes in week one converted about four times as often as the rest. Onboarding was rebuilt around it: a progress bar toward 25, daily emails, a checklist item. Within a month the share of trials reaching 25 nearly tripled, largely because trialists loaded sample routes to fill the bar. Conversion did not move. The 25 routes had marked fleets with real dispatch volume; pushing small fleets through the motion did not give them that volume. A marker had become a target, and stopped measuring (Goodhart's law, in its popular form). The recovery: delete the bar, randomize one nudge toward a real first dispatch against no nudge, and learn that it helped only trials that had imported their own jobs. (Fictional composite.)

A threshold that separates your best customers from the rest describes them. It does not create more of them.

### Governance risks

- **Customer data is not yours in the same way.** In business software, your customers' users generate the telemetry, and your contracts and data-processing terms govern what you may do with it. Check them before usage drives marketing, and prefer uses the customer would expect (Chapter 19). This is not legal advice.
- **Employee monitoring by another name.** Fleet telemetry includes technicians' location and timing. Using it to route jobs is the product; using it in a message about a named technician's pace is surveillance (the Product playbook).
- **One account, many readers.** An administrator can see team activity; a nudge that tells one user what a colleague did has leaked account memory to the wrong person. Scope each surface to its role (Chapter 18).
- **Trial-expiry pressure.** No manufactured countdowns, no hidden auto-renewal, and cancel and downgrade stay outside the adaptive layer (Chapter 18; the FTC's 2022 dark-patterns report catalogues the forms [24]).
- **Inferred reasons quoted as facts.** "Evaluating a competitor" is a hypothesis for the seller, never a sentence to the buyer.

### What this does not do

Usage does not tell you why. It will not fix a product that fits no one, and it cannot tell an evaluator from a buyer without evidence from outside the event stream. The evidence base is thin in places: controlled public evidence that in-app guides or personalized onboarding lift conversion is scarce, mostly vendor case studies without a published method. Activation benchmarks come from analytics vendors' own customers, skewed toward teams that instrument and grow. I found no live primary source comparing net retention across product-led and sales-led companies; the quoted comparisons trace to an OpenView dataset now offline. And the magic-number genre deserves the skepticism its own vendors now apply to it.

## How you will know

**Define every rate with its window and cohort.** Trial-to-paid within 60 days, by signup month; activation as the tested milestone within seven days; time to value as hours to first dispatched route; seat adoption as active over paid in 30 days; net retention with its definition beside it.

**Randomize at the account, not the user,** since people in one workspace influence each other (Chapter 21). Test the discount with a no-discount arm among activated trials. Test PQL routing by leaving a random share of qualifying trials to the product alone, comparing conversion and first-year value. Test adoption outreach against a no-contact holdout and watch churn among the contacted: that is how Sleeping Dogs appear.

**Do the power arithmetic first.** From a 5 percent baseline, detecting a rise to 6 percent at 80 percent power and 5 percent significance needs about 8,100 trials per group: about 18 months at 900 trials a month split evenly. Activation reads faster (30 to 33 percent needs about 3,800 per group, roughly eight months), but use it as a leading indicator only once it predicts conversion in your own data, and label earlier decisions as judgment.

**Log what each account saw and why.** The decision trace (Chapter 21) lets support reproduce an odd prompt and shows the team which "personalization" is really a bad default for everyone.

## Reader Q&A

**We are purely sales-led with a few users per account. Does usage still matter?**
More than you think, and later than you think. The usage signal arrives after the sale, as deployment and adoption, and it belongs in the CSM's and account manager's briefs. Store it the same way.

**Should sales see every product-qualified lead?**
No. Sales should see the trials where a person changes the outcome and the value justifies the cost of their time. Everyone else is better served by a product that helps them and a silence that respects them.

**We already have a magic number. Should we drop it?**
Keep it as a shared description of healthy use. Before you build onboarding around it, run one randomized nudge toward it and see whether retention follows.

## For your AI

```yaml
playbook: SW
title: Software Companies
question: "How does a software company personalize when its richest signal is how people use the product?"
concepts:
  - name: Usage as Memory
    definition: "Raw product events stay in the warehouse; code derives typed properties on the user and account; evidence memories hold the why, with source and date."
  - name: Candidate Milestone
    definition: "An activation threshold found by correlation; treated as a hypothesis until a randomized nudge toward it shows it moves retention or conversion."
  - name: Decision-Grade PQL
    definition: "A product-qualified lead defined by where a human changes the outcome and the value justifies it, not by conversion likelihood alone."
  - name: Motion Map
    definition: "Product-led, hybrid and sales-led companies personalize the same four decisions (activate, convert, expand, retain); they differ in who acts on usage and where."
  - name: Seat Adoption Ratio
    definition: "Active seats over paid seats in a recent window; the most useful retention property most software companies do not store."
decision_rules:
  - if: "a value is counted or derived from product events"
    then: "write it as an exact typed property from code; never let a model infer or overwrite it"
  - if: "a trial is activated, self-serving, and small"
    then: "do not route to sales and do not discount by default; test any discount against a no-discount arm"
  - if: "a trial shows an evaluator pattern (docs, API, security pages, no invites)"
    then: "offer a technical resource and route to a technical person, not a discovery call"
  - if: "an activation threshold is proposed as a target"
    then: "randomize a nudge toward it against no nudge before building onboarding around it"
  - if: "an account has low seat adoption"
    then: "lead with role-specific adoption help; keep sales offers out of the same contact; hold out a no-contact group and watch churn among the contacted"
  - if: "a message recommends a cheaper plan or smaller commitment"
    then: "treat it as a churn-risk action; decide per account and measure against a holdout"
  - if: "an in-app prompt or notification is proposed"
    then: "decide whether, what, when and which channel; count it against the per-person budget shared with email and sellers"
  - if: "a trial or account moves from product to sales or customer success"
    then: "hand over the shared record: what they did, what they declared, what was offered and said; keep rep-internal hypotheses out of customer copy"
assessment_questions:
  - "Is your motion product-led, sales-led or hybrid, and who acts on usage signals today?"
  - "Which product events are stored as typed properties on users and accounts, and which only live in the warehouse?"
  - "What is your activation milestone, and has a randomized test shown that moving it moves conversion or retention?"
  - "How is a PQL defined, and does a holdout show that routing it to sales changes the outcome?"
  - "Do you store seats active over seats paid, and what happens when it falls?"
  - "Do in-app prompts share a contact budget and a ledger with email and sellers?"
  - "What do your contracts and data-processing terms allow you to do with customer usage data?"
patterns: [Usage as Memory, Candidate Milestone Test, Decision-Grade PQL, State-Based Trial Flow, Adoption Before Renewal, Precompute-then-serve, Stable Frame Variable Parts, Propose Do Not Send]
anti_patterns: [The Chased Number, The Discount for the Converted, Size-Scored PQL, Usage Nudge That Invites Downsizing, The Blind Handoff, The Fatigue Spiral]
metrics: [trial-to-paid within window by cohort, tested activation rate, time to first value, seats active over paid, net revenue retention with definition, churn among contacted versus holdout]
links: {identity: 9, memory: 10, freshness: 11, decision_layer: 14, generation: 15, latency: 17, governance: 18, privacy: 19, measurement: 21, generative_experiences: 16, playbooks: [PR, S2, P9, CS]}
maturity_dimension: decisioning
```

## References

1. Bartlett, B. (OpenView), on coining product-led growth; original post (https://openviewpartners.com/blog/inventing-product-led-growth/) offline; definition confirmed via Sacra interview. https://sacra.com/research/blake-bartlett-openview-future-product-led-growth/
2. Slack Technologies, Inc. Form S-1, filed 2019-04-26 (figures as of 2019-01-31). https://www.sec.gov/Archives/edgar/data/1764925/000162828019004786/slacks-1.htm
3. Zoom Video Communications, Inc. Form S-1, filed 2019-03-22. https://www.sec.gov/Archives/edgar/data/1585521/000119312519083351/d642624ds1.htm
4. Atlassian Corporation Plc. Form F-1, filed 2015-11-09. https://www.sec.gov/Archives/edgar/data/0001650372/000104746915008450/a2226437zf-1.htm
5. Wing VC. "OpenView's 2023 Product Benchmarks Report" (summary of OpenView and Pendo survey, ~1,000 companies; original offline). https://www.wing.vc/content/openviews-2023-product-benchmarks-report
6. Sanchez Ramirez, J., Coussement, K., De Caigny, A., et al. (2024). "Incorporating usage data for B2B churn prediction modeling." *Industrial Marketing Management*. https://www.sciencedirect.com/science/article/abs/pii/S0019850124000865
7. First Round Review (2015-02-27). "From 0 to $1B: Slack's Founder Shares Their Epic Launch Strategy." https://review.firstround.com/from-0-to-1b-slacks-founder-shares-their-epic-launch-strategy/
8. Stancil, B. (2015-01-30). "Facebook's 'Aha' Moment Was Simpler Than You Think." Mode. https://mode.com/blog/facebook-aha-moment-simpler-than-you-think/ (the "7 friends in 10 days" figure is attributed to Chamath Palihapitiya, Growth Hackers Conference, 2012; no primary transcript verified)
9. Mixpanel. "Magic numbers are an illusion." (vendor; page dated 2026-06-02) https://mixpanel.com/blog/magic-numbers-are-an-illusion/
10. Amplitude (2025-09-30). "The 7% Retention Rule Explained" (2025 Product Benchmark Report; vendor data, 2,600+ companies). https://amplitude.com/blog/7-percent-retention-rule
11. Retana, G. F., Forman, C., and Wu, D. J. (2016). "Proactive Customer Education, Customer Retention, and Demand for Technology Support: Evidence from a Field Experiment." *Manufacturing & Service Operations Management* 18(1), 34 to 50. DOI 10.1287/msom.2015.0547. https://ideas.repec.org/a/inm/ormsom/v18y2016i1p34-50.html
12. Tunguz, T. (2013-01-15). "The product qualified lead (PQL)." https://tomtunguz.com/the-new-sales-hotness-the-product-qualified-lead-pql/
13. Poyar, K., and Rachitsky, L. (2023-08-01). "What is a good free-to-paid conversion rate." Lenny's Newsletter. https://www.lennysnewsletter.com/p/what-is-a-good-free-to-paid-conversion
14. Poyar, K., with ChartMogul and ProductLed (2026-02-04). "The 2026 free-to-paid conversion report." Growth Unhinged. https://www.growthunhinged.com/p/free-to-paid-conversion-report
15. Gu, X., Kannan, P. K., and Ma, L. (2018). "Selling the Premium in Freemium." *Journal of Marketing* 82(6), 10 to 27. DOI 10.1177/0022242918807170
16. Snowflake Inc. Form S-1 (2020), NRR as of 2020-07-31. https://www.sec.gov/Archives/edgar/data/1640147/000162828020013010/snowflakes-1.htm ; FY2026 results release. https://www.snowflake.com/en/news/press-releases/snowflake-reports-financial-results-for-the-fourth-quarter-and-full-year-of-fiscal-2026/
17. KeyBanc Capital Markets and Sapphire Ventures (2024-10-23). 15th annual Private SaaS Company Survey, press release. https://sapphireventures.com/press/keybanc-capital-markets-and-sapphire-ventures-private-saas-company-survey-reveals-a-continued-focus-on-operational-efficiency-and-profitability/
18. ICONIQ Growth. "State of Software 2025." https://www.iconiq.com/growth/reports/2025-state-of-software
19. Ascarza, E., Iyengar, R., and Schleicher, M. (2016). "The Perils of Proactive Churn Prevention Using Plan Recommendations: Evidence from a Field Experiment." *Journal of Marketing Research* 53(1), 46 to 60. https://journals.sagepub.com/doi/abs/10.1509/jmr.13.0483
20. Pendo. "The 2019 Feature Adoption Report" (vendor; 615 subscriptions, three months of usage). https://www.pendo.io/resources/the-2019-feature-adoption-report/
21. Becker, J. U., Spann, M., and Barrot, C. (2020). "Impact of Proactive Postsales Service and Cross-Selling Activities on Customer Churn and Service Calls." *Journal of Service Research* 23(1), 53 to 69. DOI 10.1177/1094670519883347
22. Mazal, J. (2023-02-28). "How Duolingo reignited user growth." Lenny's Newsletter. https://www.lennysnewsletter.com/p/how-duolingo-reignited-user-growth
23. Yancey, K. P., and Settles, B. (2020). "A Sleeping, Recovering Bandit Algorithm for Optimizing Recurring Notifications." KDD 2020. DOI 10.1145/3394486.3403351. https://research.duolingo.com/papers/yancey.kdd20.pdf
24. Federal Trade Commission (2022-09). "Bringing Dark Patterns to Light." Staff report. https://www.ftc.gov/system/files/ftc_gov/pdf/P214800+Dark+Patterns+Report+9.14.2022+-+FINAL.pdf

---

# Playbook RS. Research and Strategy

*The question: How does understanding individuals roll up into strategy?*

> **Questions this playbook answers**
> - How deep should we research each account, and which accounts deserve it?
> - We hold thousands of account records, call notes, and research briefs. How do we turn them into an ICP and a market view we can defend?
> - Can AI-simulated customers (synthetic panels) replace our surveys and interviews?
> - Where do synthetic panels genuinely help, and where do they mislead?
> - How do we know a strategy built this way is right?

This playbook covers three connected jobs: deep research per account, rolling individual understanding up into an ideal customer profile (ICP) and strategy, and synthetic research panels with their limits. Research for a first message belongs to Sales I; here, account research is raw material for strategy.

## The moment

Larkspur Systems (a fictional composite company used throughout this handbook) is in annual planning, and two documents are on the table.

The first is the ICP slide, unchanged for three years: fleet operators with more than 100 vehicles, in logistics and utilities. Every targeting list is cut from it.

The second is an export nobody asked for. Six months earlier, Larkspur started writing account research into its shared customer memory (Chapter 10) with a fixed schema, including a typed field for what triggered each purchase, and backfilled it from two years of call notes and emails. An analyst in revenue operations counted that field across every won and lost opportunity. Fleet size and industry barely separated the two groups. A situation did: in 61% of wins, the account had opened a second depot or taken over dispatch for another site in the prior year, against 18% of losses (illustrative figures for a fictional company). Larkspur's best customers were not defined by how many vehicles they ran. They were defined by how many places they dispatched from, and by having just added one.

In the same meeting, the product lead proposes testing a new per-depot pricing tier. Recruiting real fleet managers for a survey would take three weeks. A synthetic panel of 2,000 simulated fleet managers could answer overnight.

The CFO asks the useful question: which of these two results would you bet the plan on?

The first came from counting real customers; the second would come from a model's idea of customers. Both have uses. Only one is evidence. Here is the sentence this playbook turns on: **research on one account is a brief; research on every account, written into memory with the same schema, is a market study you did not have to commission.**

## The decision

Three decisions repeat here, and each includes doing nothing.

**1. How deep to research this account, now.** The options are deep research, a standard first pass, a refresh of what is already known, or nothing. The rule: research depth follows a pending decision. A renewal in 120 days or a large expansion justifies depth; an account with no decision waiting gets nothing new. Depth has a real price. Anthropic reported that its multi-agent research system beat a single agent by 90.2% on an internal research evaluation, and that multi-agent systems used about 15 times the tokens of a chat interaction, with token usage alone explaining about 80% of performance variance on the benchmark it analyzed [1]. Deeper research is better research, bought with more reading. Spend it where a decision will change.

**2. Whether a pattern in the roll-up should change strategy.** Adopt it, test it, or leave the strategy alone. A pattern earns the right to change the ICP only after it survives a test on data it was not found in (see "How you will know").

**3. What a synthetic panel may be used for.** The evidence below supports a narrow answer:

| Use | Allowed? | Why |
|---|---|---|
| Generating hypotheses, objections, and questions to ask real customers | Yes | Cheap breadth; errors are caught by the human study that follows |
| Pretesting a survey or interview guide for confusing wording | Yes | The panel stands in for a careful reader, not for the market |
| Shortlisting which variants deserve a human test | Yes, with calibration | Aggregate direction is where panels are strongest; a variant the panel cuts is untested, not refuted |
| Evaluating your own personalization output for likely misreadings | Yes, builder-side | A quality check inside the accuracy stack (Chapter 20) |
| Sizing demand, setting price, or deciding a launch | No | Low variance and drift make wrong answers look certain |
| Writing simulated answers into a customer's record | Never | A simulation is not a fact about a person |

## What you need to know

### Account research has two jobs and one unit

Research has two jobs, and most systems do only the first. The first is to add what you do not have. The second is to challenge what you already believe (Chapter 7 calls this Enrichment as Cross-Examination). A CRM field that says fifty employees, next to current sources describing a national operation, is not a fact. It is a disagreement. A fact from a single source is a hypothesis; the same fact from an independent second source is usable.

In the governed personalization engine I built for go-to-market teams, research draws on several sources and tolerates partial failure: one failed source does not fail the account, and a source that keeps failing is dropped for the rest of the batch. An implausible value (a headcount of 100 beside clear multinational markers) is hidden from the trusted fields. The policy is hide when uncertain: a missing number costs less than a confident wrong one.

Research also has a unit: the entity, not the record. On a production pipeline I run, keying research to the company with a seven-day freshness window raised the share of records with rich research from 60.3% to 93.5% while spend fell (Chapter 5, Research Once per Entity).

Then the step most teams skip: the output of account research is not only a brief for a person. It is a write to memory in three forms (Chapter 10, Memory Layers): typed properties, evidence-level memories with sources, and a synthesis. The brief is read once, by one account manager. The properties are read by every roll-up that follows. **The brief serves a meeting; the properties serve the strategy.**

### You cannot group by a paragraph

The Boundary Rule from Chapter 10 decides whether a roll-up is possible: if you would put it in a WHERE clause or a chart, it is a property. Before any account is researched, define the strategy-grade properties it must fill. A minimal set for Larkspur:

```yaml
purchase_trigger:     # enum: new_site, compliance_deadline, fleet_growth, incumbent_failure, new_leader, cost_mandate, other, unknown
primary_job:          # enum: the job the customer hired the product for
competitor_evaluated: # list of named competitors, or none
stated_objection:     # enum, plus evidence link
outcome:              # won, lost, churned, expanded, open
loss_or_churn_reason: # enum, plus evidence link
evidence_type:        # stated | observed | inferred
```

Each property carries a description written as an instruction to the extractor: what to capture, where to find it, what it is not (Chapter 8, Extract with a Contract). Each value carries its source and whether the customer stated it, you observed it, or a model inferred it (Chapter 7, Inferred Lane). Keep enums stable; when one changes, re-extract history (Chapter 11).

That engine writes research as structured account intelligence, not one long paragraph: buying triggers, leadership changes, acquisitions, competitors, strategic priorities, each a field trust-filtered on its own. The step I am still building attaches source, confidence, and freshness to each field rather than to the record. One rule from the self-hosted memory system I built: when a research job's output is bound straight to a typed property, the binding stays optional, so an empty or invented value is never written into a field a roll-up will count.

### Rolling up: count, then characterize

The roll-up that produced Larkspur's depot finding follows six steps, and the order matters.

1. **Fix the denominator first.** Include lost deals, churned accounts, and accounts that never bought. An ICP built only from winners describes who bought, not what separates buyers from everyone else.
2. **Count deterministically.** Group typed properties by outcome in SQL. No model is involved in the arithmetic. My memory system exposes this as a property filter (purchase trigger is new site, outcome is won) that never calls a model, so the same query returns the same count every time.
3. **Weight by account, not by volume of evidence.** Large accounts generate more calls, tickets, and notes. Counting mentions instead of accounts drifts the strategy toward whoever talks most.
4. **Characterize second.** For each pattern that survives counting, a model reads a stratified sample of the underlying evidence, winners and losers alike, and names what it sees. Every sentence in the resulting memo links to the count and to example evidence. It is the review I run on a campaign's early cohorts: every flagged record plus a stratified sample of clean ones, with questions asked of the population, such as "Which facts appear unreliable?" and "What do the strongest outputs have in common?" For strategy, ask the second of a sample that includes the weakest accounts.
5. **Report stated and inferred separately.** "Customers told us" and "we inferred" are different strengths of evidence, and the strategy memo shows both.
6. **Test out of time** before adopting anything (see "How you will know").

The same machinery produces market insight: competitor mentions, loss reasons, and requested capabilities become quarterly time series instead of anecdotes. Counting is deterministic; naming is generative; never mix them in one step.

![Each account with a pending decision (a renewal, an expansion, a new site) gets research that reads the cache first and fills only the gaps, then writes to memory in three forms: typed properties, evidence, and a synthesis; the brief serves a meeting, the properties serve strategy. Across every account the properties are rolled up in six steps: fix the denominator to include won, lost, and churned; count by outcome in SQL; weight by account; let a model characterize the patterns; separate stated from inferred; and test out of time. The result is a situational ICP (multi-site operators within a year of adding a site) that becomes a versioned guideline every agent reads for scoring, research depth, routing, and framing, and that sets which accounts deserve deep research next.](/images/handbook/pb-rs-research-to-icp.svg)

*Figure RS.1. Research written as typed properties on every account can be counted; count first, name second, test out of time, and the ICP it produces sets the next round of research.*

The ICP that comes out tends to change shape, from a firmographic filter to a situation: multi-site operators within a year of adding a site. That is harder to buy as a list and far more useful for deciding which accounts to research deeply, closing the loop to decision 1.

### Synthetic panels: what the evidence supports

A synthetic panel is a set of simulated respondents produced by a language model, each conditioned on a description of someone, and asked questions as if it were a customer. There are three grades, and the difference between them matters more than the model used.

- **Persona panels** condition the model on demographic or firmographic descriptions. Argyle and colleagues showed in 2023 that GPT-3, conditioned on backstories drawn from real survey participants, reproduced response patterns of US subgroups, a property they named "algorithmic fidelity," producing what they called "silicon samples" [2].
- **Grounded panels** condition each simulated respondent on a real individual's own data. Park and colleagues built agents from two-hour interviews with over a thousand people; on held-out survey items the combined interview-and-survey agents reached 86% of participants' own two-week test-retest consistency, against 74% for agents given demographics only [3].
- **Calibrated hybrids** combine synthetic and human responses statistically. Wang, Zhang, and Zhang showed that in conjoint studies this approach cut the human data needed by 24.9% to 79.8% while reducing estimation error, and that naively substituting synthetic answers for human ones produced no savings, because of systematic bias [4].

The case for is real. Brand, Israeli, and Ngwe found GPT produced realistic willingness-to-pay estimates and textbook patterns such as declining marginal utility, and that fine-tuning on prior survey data from the same category improved alignment for existing and new features [5]. In *Nature* in 2026, Ashokkumar and colleagues had GPT-4 simulate responses to 70 preregistered US survey experiments (469 effects, 119,330 participants); its predicted effects correlated strongly with actual ones and matched pooled human forecasters, including for studies published after its training cutoff, though it tended to overestimate effect sizes [6].

The case against is as well documented. Bisbee and colleagues found synthetic responses had too little variance, that 48% of regression coefficients differed significantly from the human survey, with the sign flipping in 32% of those, and that identical prompts gave different distributions between April and July 2023 [7]. Verasight compared LLM responses with a June 2025 poll of 1,500 US adults: the best model was off by 4 points on average and the worst by 23; on a zoning question one model reported 59% support where the real figure was 28%, and the 29% of real respondents who said "don't know" became 0% [8]. Wang, Morgenstern, and Dickerson argue from how models are trained, and show in human studies, that LLMs standing in for participants misportray and flatten identity groups [9].

Read together, the pattern is consistent. Panels are good at the center of a distribution and poor at its edges and its uncertainty: "don't know" vanishes, minority positions converge on the mean, and low variance makes a wrong answer look significant. Strategy lives at the edges: a new segment, an unmet need, the undecided. **A synthetic panel is a well-read guess about the average customer. Strategy is usually about the customers who are not average.** Chapter 3 reaches the same verdict for UX research: synthetic users draft; people validate.

![Real respondents spread across the range of answers, while a synthetic panel answers in a narrow, tall peak at the center: edges are pulled toward the mean and low variance makes a wrong answer look certain. In the poll the chapter cites, 29% of real respondents said don't know and the model's share was 0%. Panels help with hypotheses and questions to ask, pretesting survey wording, shortlisting variants with calibration, and checking your own output; they mislead on sizing demand, setting price, deciding a launch, and never belong in a customer record. The workable flow is synthetic panel, then shortlist, then a human study, then the decision.](/images/handbook/pb-rs-synthetic-panel-edges.svg)

*Figure RS.2. Panels are good at the center and poor at the edges, so they shortlist and humans decide.*

I build synthetic-panel systems, and the version that behaves best is the grounded one: each simulated respondent is conditioned on one real customer's memory rather than on a persona description. That is the roll-up run in reverse, and it inherits every weakness of the memory beneath it.

## The action

None of this is real-time. All three jobs run asynchronously, precompute their results, and serve them to the systems that act (Chapter 17).

**Account research** is event-triggered: a renewal window, a new opportunity, a leadership change or new site. It reads the entity cache first, researches only the gaps, and writes typed properties, evidence, and an account plan with every inference labeled as a hypothesis. Freshness policies (Chapter 11) decide when a property must be re-verified. In my memory system, a long research run can execute as a background job on the model provider's infrastructure, holding a credential minted for that job alone: limited to saving, retrieving, and searching memory, revoked when the job ends, with every write attributed to the job. Routine refreshes run as scheduled prompts.

**The roll-up** runs monthly for market insight and quarterly for the ICP. Its output is a strategy memo in which every claim carries a count, a denominator, and example evidence. My engine's campaign reports already aggregate stored intelligence this way (population, score distribution, engagement, common signals); the ICP roll-up is that report run over outcomes. When the ICP changes, it does not stay a slide. It becomes a versioned guideline that every agent reads (Chapter 18): lead scoring, research depth, routing, and message framing all consume the same definition. Governance is a code path; so is strategy.

**Synthetic panels** run in two places. Before human research, they pretest instruments and shortlist variants, so the expensive human study asks better questions. After generation, they sit inside the accuracy stack: simulated recipients grounded in memory flag messages a customer would likely find wrong, irrelevant, or confusing, and those go to review. The objective of that check is accuracy and appropriateness, never pressure, and that objective belongs in governance, not in the prompt.

## What can go wrong

### Failure story: The Confident Panel

A year before the planning meeting, Larkspur tested a fuel-analytics add-on on a synthetic panel of 1,500 personas described as fleet managers, with sizes and industries sampled to match the customer base (a fictional composite of a pattern that the studies above predict). Seventy percent said they would pay. The confidence interval was narrow, because the simulated respondents barely disagreed with one another. The add-on launched. Large fleets bought it. Single-site operators, most of the customer base, mostly did not, and the few who did churned within two quarters.

The panel answered as a well-read model expects a fleet manager to answer, and much public writing about fleet management is about large fleets. The panel had no way to say "I don't know what I would use this for," which is what the real small operators would have said. **The Confident Panel** is the anti-pattern: synthetic agreement read as market evidence, with low variance mistaken for certainty. The fix was the rule in the decision table: panels shortlist, humans decide.

### Other risks to govern

- **Survivor ICP.** The profile is built from winners only and describes who bought, not what distinguishes them.
- **The Loudest Account.** Evidence is weighted by mentions, so the accounts with the most calls and tickets write the strategy.
- **Echoed Corroboration** (Chapter 13). Several research runs appear to confirm a fact because they all inherited one unverified write. Count independent sources, not agreeing ones.
- **Synthetic Leak.** A simulated answer is written into a real customer's record and later used to personalize. Tag synthetic output at the source and keep it out of customer memory entirely.
- **Reversing the roll-up onto a person.** A segment trait is not a personal fact. "Multi-site operators tend to struggle with dispatch" may guide research; it may not be asserted to one customer as something you know about them (Chapter 19, not legal advice). Check that using customer data for aggregate analysis is compatible with the purposes it was collected for.

### What is overstated

Pitches for "digital twins of your customers" often cite a high correlation between synthetic and human answers. Ask what was correlated: aggregate means can correlate well while segments are badly wrong. I part company with the popular reading of Argyle and of Brand, Israeli, and Ngwe, which treats their findings as permission to replace respondents. Both papers are more careful than that: Brand and colleagues position models as a supplement to human studies, with the most reliable gains when prior human data from the same category and population is already in hand [5]. I also part company with blanket dismissal. The grounded and calibrated results [3][4][6] are real, and they are improving.

Roll-ups overreach too. The depot finding is a correlation: a second site might be why customers buy, or only when they happen to evaluate. Chapter 21 guards against reading history as cause.

## How you will know

**Account research.** Grade a monthly sample of briefs against sources for unsupported and stale claims. Track the decision-change rate: how often research changed the plan it was commissioned for. Research that never changes a decision is a cost to cut. Track cost per researched decision (Chapter 12).

**The ICP and strategy.** Treat the ICP as a prediction and test it like one. First, out of time: find the pattern on older data, then check it on the most recent two quarters it never saw. Then prospectively: tag new opportunities with fit under both ICPs and compare win rate, time to value, and twelve-month retention, keeping part of prospecting on the old definition as a control (Chapter 21). A new ICP that does not beat the old one on data it was not built from is a story, not a strategy. An out-of-time test is only honest if you can see what was known then. My memory system keeps every superseded property value with the period it held, so the test can read each account as it looked while the deal was open, not as later research rewrote it.

**Synthetic panels.** Keep a calibration ledger per question type. Before trusting a panel on pricing, feature preference, or message comprehension, run it on questions where you already hold human answers, and record four numbers: average error, the ratio of synthetic to human variance, the worst subgroup error, and the "don't know" rate against the human rate. Record which model version produced them. Then track forward hit rate: how many shortlisted variants won the human test. Set thresholds by consequence; wording tolerates error that price cannot.

## Reader Q&A

**Can we stop interviewing customers now?**
No. Panels are cheapest when they make interviews better: better questions, better shortlists, fewer wasted sessions. The strongest results in the literature all depend on real human data, either as the grounding for each simulated respondent or as the calibration sample.

**A vendor offers "digital twins" of our buyers. What should we ask?**
What each twin is grounded in (a persona description or a real person's data); error by subgroup, not only on average; the "don't know" rate; how results changed across model versions; and whether any synthetic output can reach a customer record.

**How many accounts before a roll-up means anything?**
Enough that each pattern holds in both won and lost groups with room to spare, and survives an out-of-time check. A pattern found in forty accounts is a hypothesis for the next research cycle, not a strategy.

## For your AI

```yaml
playbook: RS
title: "Research and Strategy"
question: "How does understanding individuals roll up into strategy?"
concepts:
  - name: Research Depth Follows Decisions
    definition: "Account research depth (deep, standard, refresh, none) is set by the value of a pending decision on that account, not by a blanket policy."
  - name: Two Jobs of Research
    definition: "Research adds what is missing and challenges what is held; a single-source fact is a hypothesis until independently corroborated."
  - name: Strategy-Grade Properties
    definition: "Typed, enumerated properties (purchase trigger, job, objection, competitor, outcome, loss reason, evidence type) filled by every account's research so it can be counted across accounts."
  - name: Count, then Characterize
    definition: "Roll-ups count typed properties deterministically by outcome, weighted by account, then let a model name patterns from a stratified evidence sample; every claim links to a count and evidence."
  - name: Situational ICP
    definition: "An ideal customer profile defined by the situation that separates winners from losers (for example, recently added a site), rather than by firmographics alone."
  - name: Synthetic Panel
    definition: "LLM-simulated respondents conditioned on personas (persona panel), on real individuals' data (grounded panel), or combined statistically with human samples (calibrated hybrid)."
  - name: Calibration Ledger
    definition: "Per question type and model version: average error, synthetic-to-human variance ratio, worst subgroup error, don't-know rate, and forward hit rate against human tests."
decision_rules:
  - if: "an account has no pending decision"
    then: "do not commission new research; refresh only what a freshness policy requires"
  - if: "a roll-up pattern is found"
    then: "test it out of time and prospectively against the current ICP before adopting it"
  - if: "the roll-up counts mentions rather than accounts, or excludes lost and churned accounts"
    then: "rebuild the denominator before drawing conclusions"
  - if: "a synthetic panel result would set price, size demand, or decide a launch"
    then: "treat it as a hypothesis and run a human study"
  - if: "the question concerns a small segment, a new category, or events after the model's training cutoff"
    then: "do not use a synthetic panel as evidence"
  - if: "the panel's model version changed since last calibration"
    then: "re-run the calibration ledger before relying on it"
  - if: "a researched value is implausible against other evidence, or one research source keeps failing"
    then: "hide the value from trusted properties and build from the remaining sources"
  - if: "an out-of-time test is run on properties that research has since rewritten"
    then: "read the property values as they held at the time, from property history"
  - if: "output is synthetic"
    then: "tag it at the source and never write it into a customer record"
  - if: "a segment-level insight is about to be used in a message to one person"
    then: "treat it as inference: guide research with it, do not assert it"
assessment_questions:
  - "Is account research written into shared memory as typed properties, or only as briefs and documents?"
  - "Which properties would you need to count to explain why you win and lose, and are they captured today?"
  - "Does your ICP analysis include lost, churned, and non-buying accounts?"
  - "Where do you use or plan to use synthetic respondents, and what human data calibrates them?"
  - "How would you know if your current ICP is wrong?"
patterns: [Research Depth Follows Decisions, Research Once per Entity, Hide When Uncertain, Count then Characterize, Quote-Backed Insight, Calibration Ledger, Panels Shortlist Humans Decide]
anti_patterns: [The Confident Panel, Survivor ICP, The Loudest Account, Synthetic Leak, Echoed Corroboration]
links: [ch5, ch7, ch8, ch10, ch11, ch13, ch17, ch18, ch19, ch20, ch21]
maturity_dimension: understanding
as_of: 2026-09-26
```

## References

1. Anthropic (2025-06). "How we built our multi-agent research system." https://www.anthropic.com/engineering/multi-agent-research-system
2. Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., Wingate, D. (2023). "Out of One, Many: Using Language Models to Simulate Human Samples." *Political Analysis* 31(3). https://arxiv.org/abs/2209.06899
3. Park, J. S., Zou, C. Q., Kamphorst, J., et al. (2024, rev. 2026). "LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals" (earlier title: "Generative Agent Simulations of 1,000 People"). arXiv:2411.10109. https://arxiv.org/abs/2411.10109
4. Wang, M., Zhang, D. J., Zhang, H. (2024, rev. 2026). "Large Language Models for Market Research: A Data-augmentation Approach." arXiv:2412.19363. https://arxiv.org/abs/2412.19363
5. Brand, J., Israeli, A., Ngwe, D. (2023). "Using GPT for Market Research." Harvard Business School Working Paper 23-062; ACM EC 2024. https://www.msi.org/working-paper/using-gpt-for-market-research/ ; https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4395751
6. Ashokkumar, A., Hewitt, L., Ghezae, I., Willer, R. (2026-07-08). "Large language models can predict the results of social science experiments." *Nature* 656:115-122. https://www.nature.com/articles/s41586-026-10742-x
7. Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., Larson, J. M. (2024). "Synthetic Replacements for Human Survey Data? The Perils of Large Language Models." *Political Analysis* 32(4). https://www.cambridge.org/core/journals/political-analysis/article/synthetic-replacements-for-human-survey-data-the-perils-of-large-language-models/B92267DC26195C7F36E63EA04A47D2FE
8. Morris, G. E., and the Verasight data team (2025-08-18, updated 2026-09-01). "Your Polls on ChatGPT." Verasight. https://www.verasight.io/reports/synthetic-sampling
9. Wang, A., Morgenstern, J., Dickerson, J. P. (2025). "Large language models that replace human participants can harmfully misportray and flatten identity groups." *Nature Machine Intelligence* 7:400-411. https://www.nature.com/articles/s42256-025-00986-z

---

# Playbook P9. One Customer, One Conversation

*The question: How do we stop marketing, sales, support, product, and our agents from contradicting each other, or exhausting the same person?*

> **Questions this playbook answers**
> - Every team's program performs well on its own dashboard. Why do our best customers say we feel like five companies?
> - Who decides what one person hears from us this week, when four teams and six agents all have something to say?
> - How many messages is too many, and should that number be the same for everyone?
> - What has to be shared, beyond customer data, so that two teams never tell one person two different things?
> - How do we prove that coordination is worth the messages it stops?

## The moment

*Larkspur Systems is a fictional composite company used throughout this handbook: field-service and fleet software, about 40,000 accounts and 250,000 contacts. Every person and customer below is fictional.*

Grace Adeyemi joined Harlow Fleet Services as vice president of operations three weeks ago. Harlow is one of Larkspur's larger customers: 400 technician seats, a renewal in the spring, and a route-sync problem last summer that Priya, the account manager, spent a month repairing. Dana, Harlow's fleet operations lead, now reports to Grace.

Here is what Larkspur said to Grace in her first nine days.

An enrichment feed spotted her job change and created a new lead record, because nobody had told it Harlow was a customer. Marketing's new-hire trigger enrolled her in a prospect nurture that opened with "See why fleets like yours are switching to Larkspur." The outbound agent, reading the same lead record, offered her a free fleet assessment. Customer success invited her to administrator training. The support copilot answered her first ticket, about a sync delay, by saying the fix would ship "in the next release, at the end of the month." Two days later an in-product banner told her the fix was already live. The events team invited her to a customer dinner and, separately, to a webinar for prospects. And a promotional email offered 15% off an add-on "for new teams," the same week the renewal agent sent Priya a draft proposal with a price increase.

Fourteen messages, six senders, two direct contradictions. Every one of them was accurate by its sender's lights. Grace forwarded three of them to Priya with one line: "Which of these is Larkspur?"

Every other playbook in Part VIII personalizes one function well. This one makes sure the functions add up to one company. The seams it closes are sharpest in software businesses, where product, sales, and customer success each hold a copy of the same account (Playbooks SW and CS).

## The decision

The decision in this playbook is not what to say. The other playbooks decide that. It is **arbitration**: given everything every team and every agent wants to send to this person in the coming window, which actions go, in what order, merged how, and which wait or die.

Here is the sentence this playbook turns on: **the customer does not experience your org chart; she experiences the sum of it, and somebody has to own the sum.**

Chapter 14 introduced the idea as one line of its checklist (a per-person contact budget enforced at one arbitration point). This playbook is the operating detail. The options for each pending action are:

| Option | What it means |
|---|---|
| Send | The action clears every constraint and wins its slot in the budget |
| Merge | Combine several teams' messages into one, usually from the relationship owner |
| Defer | Hold until a condition clears (a ticket closes, a quiet period ends) or the next window |
| Reroute | Hand it to the person who owns the relationship to deliver or drop |
| Suppress | Block it because a constraint forbids it |
| Do nothing | Nothing beats the value of silence this window; log why |

Arbitration needs a precedence order that everyone agreed to before the argument, not during it. The one I recommend, from strongest to weakest:

1. **Hard constraints.** Consent, opt-outs per purpose and channel, legal holds, and promises the company made (Chapter 18's customer-level governance). These remove actions; they are never weighed.
2. **Service before selling.** An open issue for this person or account defers every promotional and expansion action. Dana's upsell in Chapter 14 dies here.
3. **Operational necessity.** Security notices, billing, contract and renewal paperwork, and service changes the customer needs to run their business. These are exempt from the budget, but they are not exempt from the record, and they still count as contact when the next promotional decision is made.
4. **The relationship owner.** Where a named human owns the account, that person can merge, reroute, or veto anything an agent or a program proposed.
5. **Value against attention.** What remains competes on expected incremental value (Chapter 14's uplift, not propensity) net of the attention it spends.

The fifth rule is where the budget lives. Every person will read only so many of your messages with goodwill, and every team spends from that balance whether it can see it or not.

The strongest recent evidence on what that balance costs comes from a field experiment by Baek, Chen, Ma and Mitrofanov. A major North American online retailer randomly assigned 603,153 customers, for one week in October 2024, either to its usual daily coupon emails or to one email every two days [1]. Fewer emails cut unsubscriptions by 59% and cost 5 to 8% of short-term revenue and purchase rate. Separately, in a year of history for about 200,000 customers, unsubscribing reduced a customer's monthly spending by 36%. When the authors fed both estimates into a lifetime-value model, personalizing email frequency raised average customer lifetime value by 6.8% under a policy that maximized immediate revenue and 8.3% under a full lifetime-value policy, both against the daily baseline [1]. LinkedIn reached the same conclusion from the other direction when it removed four of every ten emails and reported member complaints cut in half (Chapter 14) [2].

Two lessons carry into B2B. The right frequency is **per person**: in Baek's data, cutting frequency reduced churn most among customers who opened many emails but used few coupons, and cost the most revenue among heavy buyers who rarely opened. And the cost of contact **arrives later than the benefit**, which is why every team's weekly metric asks for one more send. Gartner's 2025 survey of 632 B2B buyers found 73% actively avoid suppliers that send irrelevant outreach [3]. Avoidance is not an unsubscribe. It is quieter, and it is what Grace was about to do.

The limits of that evidence matter. Baek et al. is a working paper, from one retailer, testing coupon email over a single week, with a lifetime-value gain that comes from a model rather than observed years. A fleet-software buyer is not a coupon shopper. What transfers is the shape of the tradeoff (short-term lift, deferred churn, heterogeneous people), not the percentages. Anyone quoting you an optimal number of B2B emails per month is quoting an average of people who are not your customer.

## What you need to know

Contradictions have two sources, and teams usually fix only the first. **Different facts**: support knows the fix is live, marketing does not. **Different rules**: the promotion's discount policy lives in one team's prompt and the renewal pricing policy in another's, the fourteen-configurations problem from my own work [4]. When I wrote up the shared memory layer I built for multi-agent workflows, these were the first two structural problems on the list: memory silos across agent workflows, and governance fragmented across teams and tools [8]. One conversation requires four shared things, and each is a different kind of object.

**1. One memory of the person and the account (what we know).** The same record every surface reads (Chapter 10), with identity resolved first (Chapter 9). Grace became a "lead" because a job change created a new record instead of a new role on a known person at a known customer. The first contradiction was an identity failure, and no amount of copy review would have caught it. Freshness matters as much as unity: a role change, a closed ticket, and a shipped fix are exactly the facts that go stale in one system while another has moved on (Chapter 11).

**2. A contact ledger (what we said).** Every customer-facing touch, from every team, tool, agent, and human inbox, written to the person's record: when, channel, sender, topic, and, most importantly, the **claims made**. "Fix ships end of month" is a claim. "15% off for new teams" is a claim. When the next agent drafts, it reads what the company has already told this person, so it can agree with it, correct it deliberately, or stay silent. Most stacks log sends per tool; almost none log claims per person. Stamp the sender on the send path, never take it from the agent: in the memory system I built, the server stamps each automated write with its process and run, where no model can forge it. Zendesk's 2026 trends survey (fielded June 2025, over 11,000 respondents) found 81% of consumers want agents to continue the conversation without backtracking and 74% are frustrated when they must repeat information [5]. The ledger is how a copilot knows what the last conversation was.

**3. Open commitments and constraints (what we promised, what we may not do).** Promises, holds, quiet periods, preferred channel, and the relationship owner are typed fields, not notes (Chapter 18). By the Boundary Rule (Chapter 10), "no outreach before March 1" is a WHERE clause, so it is a property. The gist of last Tuesday's call is a paragraph, so it is a memory.
**4. One set of rules (what anyone may say).** Pricing authority, offer eligibility, roadmap language, and brand voice, owned once and served by reference to every agent (Chapter 18). "By reference" means retrieved at the moment of action, not pasted into fourteen prompts: in the systems I build, one retrieval returns the person's record together with the rules for this action, and each campaign's guideline is addressed by a stable identifier so two similarly named programs cannot borrow each other's offer. If the promotion and the renewal read the same pricing policy, the 15% offer either is not generated for a customer mid-renewal or is generated knowingly.

In B2B there is a fifth consideration: **the account is also a budget.** Grace, Dana, and four other Harlow people forward each other Larkspur's email. Six contacts each at a sensible personal frequency can still add up to a vendor that will not stop talking. Keep a per-person budget and an account-level ceiling, and let the account view show every pending action for every contact.

Each fact in the shared memory also carries its allowed surfaces next to its provenance [6]. Unifying memory does not mean every team may use everything. What Grace told support about a staffing problem may shape the account manager's tone and must never appear in a marketing email (Chapter 19). The boundary I enforce hardest is **rep-internal is not customer-facing**. A seller brief may hold hypotheses and competitive context that must never reach a buyer, so in the engine I built an explicit serializer decides what leaves the system, and seller-only analysis stays behind even when someone adds a new internal field. A prompt instruction to "keep this internal" is not a boundary; serialization is.

## The action

Coordination is an architecture change more than a policy memo. Four moves make it real.

![Every program from the other playbooks (inbound reply M1, accounts and events M2, mailers M3, prospecting S1, expansion S2, in-product PR, proactive support SP) and any new agent can only propose, with no send credential. A daily coordinator applies precedence, strongest rule first: hard constraints remove actions (consent, holds, promises); service before selling defers promotion during an open issue; operational necessity is exempt from the budget but still recorded; the relationship owner merges, reroutes, or vetoes; and value against attention ranks and spends the budget. Each proposal gets a decision and a reason (send, merge, defer, reroute, suppress, or do nothing), the send path re-checks hard constraints under a per-person lock, channel compilers build every surface from one context, and every touch and claim is written back. Proposers, coordinator, and send path share one memory, a contact ledger of claims, commitments, and one set of rules.](/images/handbook/pb-p9-arbitration-architecture.svg)

*Figure P9.1. Propose, do not send: one coordinator applies an agreed precedence, and the send path checks again before anything reaches the customer.*

**Propose, do not send.** Every program, tool, and agent submits a proposed action (who, what, why, which claims, urgency, expiry) to a shared queue instead of sending directly. That includes every other playbook's output: the inbound reply (M1), account and event touches (M2), the mailer (M3), prospecting and expansion (S1, S2), in-product messages (PR), and proactive support (SP). Replies to a customer who wrote in are the exception on timing, not on record: they go immediately and write to the ledger at once.

**Arbitrate on a schedule, re-check at send time.** A coordinator runs daily over the next window for every person with pending proposals: apply the precedence order, spend the budget, merge where three teams want the same person, and emit a decision with a reason for every proposal, including the ones it killed. Planned outreach is asynchronous by nature, so this is a batch job, not a real-time system (Chapter 17). But state changes between decision and delivery, so the send path re-checks hard constraints and takes a short per-person lock, so two messages approved in the same minute cannot both go out. A 6 a.m. decision is not a license to send at 4 p.m. after a noon ticket. The rules themselves are small:

```yaml
arbitration:
  window: 7d
  budget:
    person_promotional_max: 2          # per window; set per segment, then learn per person
    account_promotional_max: 6
  exempt_from_budget: [service_reply, security, billing, contract]
  precedence:
    - suppress_if: [opt_out_for_purpose, legal_hold, active_promise_hold]
    - defer_if: [open_ticket_sev1_or_sev2, onboarding_first_14d]
    - owner_review_if: [account_tier_strategic, claim_conflicts_ledger]
    - rank_by: expected_uplift_minus_attention_cost
  merge:
    when: "3+ proposals from different teams in one window"
    deliver_as: relationship_owner
  always_log: [decision, reason, proposals_considered, rule_version]
```

**Compile every surface from one context.** Once an action wins, the channel that delivers it (email, in-product banner, notification, rep talking point, printed piece; Chapter 16 covers the experiences) compiles from the same governed context, never from its own copy of the customer (Chapter 15's channel compilers) [6]. The banner and the support reply about the sync fix would then have read the same release status. In the engine I built, research and context are assembled once and compiled into the landing page, the email, the seller brief, and the report. Each surface keeps its own job (the brief analytical, the email short) but none has its own facts, which removes contradiction and duplicated integration work together. When programs overlap, scope each page's identity to its campaign: the same company can sit in two programs, and without that scope one program's page overwrites the other's.

**Give the relationship owner the view.** Priya should be able to open Harlow and see the next seven days: every pending proposal for every contact, what the coordinator decided, and why. She can merge the training invite, the dinner, and the renewal preview into one note from her, and drop the prospect nurture outright. For strategic accounts, a human merging three messages into one is often the best-performing "agent" in the building.

Agents deserve one rule of their own: **an agent should not hold its own send credential.** Chapter 18's first control is removing authority, and here it is literal. If every agent can only propose, a new agent added by a new team next quarter joins the conversation instead of starting a second one. The same rule governs memory. In the self-hosted memory system I built, each agent can get its own key scoped to a role and a data slice, and a read-only agent connecting over MCP never sees the write tools. A hosted long-running job gets a credential minted for it alone: three memory tools, expiring within a day, revoked when the job ends. In the multi-agent systems I build, the capability that surprised me most came from specialists writing into shared state rather than reporting to each other: once one agent's finding changes what the next one is allowed to assume, the system stops contradicting itself as a side effect.

For Grace, the coordinated version looks like this. Identity attaches her to Harlow as a new contact. The nurture, the assessment offer, and the prospect webinar are suppressed: prospect programs never reach a customer. The promotion is deferred because Harlow is in the renewal window and the pricing policy says renewal offers come from the owner. The support reply reads release status from the same source as the banner. And Priya sends one message: welcome, what happened with route sync and what was fixed, training for Grace's team, and an invitation to the dinner. Four teams contributed to it. Grace received one.

![Left, the fragmented default: six senders each read their own copy and send directly to Grace. Marketing offers 15% off during a renewal with a price rise, the outbound agent offers a free fleet assessment, customer success invites her to administrator training, the support copilot says the fix ships at the end of the month while the in-product banner says it is already live, and events invites her to a customer dinner and a prospect webinar: 14 messages, 6 senders, 2 contradictions. Right, one conversation: the same teams propose to one coordinator reading one memory, one ledger, and one set of rules. It suppresses the nurture, the assessment, and the prospect webinar, defers the promotion for the renewal window, aligns support with the live release status, and merges the rest, so Grace receives one message from Priya covering the welcome, the sync fix, training, and the dinner.](/images/handbook/pb-p9-fragmented-vs-one-conversation.svg)

*Figure P9.2. The customer experiences the sum of the org chart: fourteen uncoordinated messages with two contradictions, or one message that four teams contributed to.*

## What can go wrong

**Failure story: The Side Door.** Larkspur built its coordinator, wired in marketing automation and the outbound agents, and watched contradictions fall. Six months later the account team at another large customer received a complaint that read exactly like Grace's. The investigation found two senders outside the queue: the events team's registration platform, which sent its own reminders, and an AI sales assistant a regional team had bought on a credit card, sending from reps' own mailboxes. Neither wrote to the ledger. The coordinator's reports looked clean because it could only count what it saw. The fix was organizational as much as technical: a rule that nothing sends to customers without writing to the ledger, enforced where it can be (sending domains, mailbox integrations, vendor contracts), and a metric for the share of customer-facing sends that passed through the coordinator. A contact budget with a side door is a suggestion.

The other risks, in rough order of frequency:

- **Over-suppression.** A budget tuned only against annoyance will silence what the customer needed. Chapter 14's warning applies: a customer in trouble who hears nothing may conclude nobody noticed. Operational messages stay exempt, and the coordinator's suppression log is reviewed for important messages that died.
- **Arbitration by volume.** Without an agreed objective, the team that shouts loudest owns the budget. Precedence and value weights are set by leadership and versioned like any other rule (Chapter 18), not renegotiated per campaign.
- **One budget for everyone.** A flat cap is better than none and worse than a per-person budget learned from each person's own responses. Start with a conservative per-segment cap; do not copy another company's number.
- **The race.** Two teams' messages approved separately arrive in the same hour because the ledger updates after sending. The send-time lock and re-check exist for this.
- **Shared memory as a leak.** Unifying memory makes it easier for a sensitive support disclosure to surface in a sales draft. Allowed-surface metadata and grounding checks on every personal claim (Chapters 15 and 20) are what keep one memory from becoming one megaphone.
- **Coordination used to press harder.** The coordinator sees every touch to one person. The budget exists to spend less attention, not to orchestrate more, and its objective should say so.

## How you will know

Measure the sum, not the parts. Five metrics, most of which no single team tracks today:

1. **Contact distribution per person and per account**, weekly: the median and, more important, the 95th percentile. Averages hide the Graces.
2. **Contradiction rate**: claims in the ledger that conflict with an earlier claim or with current memory (price, dates, product status), per thousand active accounts, from automated checks plus a monthly human sample.
3. **Negative responses per person**: unsubscribes, spam reports, channel disengagement, and "please coordinate" complaints, against contact volume.
4. **Coverage**: the share of customer-facing sends that passed through the coordinator. Below 100% is your Side Door.
5. **Repeat asks**: support conversations where the customer restates something the company already had. In Salesforce's 2020 survey of about 15,600 consumers and business buyers, 54% agreed "it generally feels like I'm communicating with separate departments, not one company," and 65% said they often have to repeat or re-explain information [7]. The data is six years old and vendor-run; your own repeat-ask rate is the current version.

The outcome that justifies the whole thing is slower: retention and expansion at renewal, and the health of the channels themselves.

**Test design.** Randomize at the account level, not the person level, because Harlow's contacts talk to each other and a coordinated Grace next to an uncoordinated Dana contaminates both. Hold out a random share of accounts (Larkspur chose 10%, about 4,000 accounts) that keep today's uncoordinated programs, and run the comparison through at least one full renewal cycle. Expect Baek's pattern: the coordinated arm will look worse on short-term sends-driven metrics in the first weeks, because it sends less. Negative responses and contact distribution will separate within weeks; renewal and expansion need quarters. Write down before the test which metric decides it, or the first month's dip will end it (Chapter 21).

## Reader Q&A

**We have a journey orchestration tool with frequency capping. Is that not this?** Partly. Most caps work per channel inside one tool. The gaps are across tools, across teams, from reps' own mailboxes, and from agents, and none of them log claims. Check your tool's cap against the coverage metric before assuming it covers the conversation.

**Who owns the budget?** Not marketing, sales, or support, since each is a bidder. In most companies it is a revenue or customer operations function with an executive sponsor, and the rules are versioned like any other governance artifact.

**Does a support reply count against the budget?** No: when the customer wrote to you, answer. It does count as context, and it can defer the promotional message scheduled for tomorrow.

**Where do we start?** With the ledger and a single account view for relationship owners. Seeing the sum is what persuades teams to share the rest.

## For your AI

```yaml
playbook: P9
title: "One Customer, One Conversation"
question: "How do we stop marketing, sales, support, product, and agents from contradicting each other or exhausting the same person?"
foundation_links: [6, 9, 10, 11, 13, 14, 15, 17, 18, 19, 20, 21]
playbook_links: [M1, M2, M3, S1, S2, SP, RS, PR, SW, CS]
concepts:
  - name: One customer, one conversation
    definition: "Every team, tool, and agent reads one memory, spends one attention budget, obeys one set of rules, and passes through one arbitration point, so the customer experiences one company."
  - name: Arbitration
    definition: "The decision, per person and window, of which pending actions from all teams send, merge, defer, reroute, are suppressed, or give way to doing nothing."
  - name: Contact ledger
    definition: "A per-person record of every customer-facing touch from every sender, including the claims made, read by every agent before it drafts."
  - name: Shared attention budget
    definition: "A per-person and per-account limit on promotional contact that all teams spend from; learned per person over time; operational messages exempt but recorded."
  - name: Propose, do not send
    definition: "Programs and agents submit proposed actions to a shared queue; only the coordinator's decision, re-checked at send time, reaches the customer."
decision_rules:
  - if: "an action conflicts with consent, a legal hold, or an active customer promise"
    then: "suppress it before ranking"
  - if: "the person or account has an open service issue"
    then: "defer promotional and expansion actions until it closes"
  - if: "three or more teams propose contact with one person in the same window"
    then: "merge into one message from the relationship owner or rank and send the winner only"
  - if: "a draft makes a claim that conflicts with a claim in the contact ledger or with current memory"
    then: "block it and route to the relationship owner"
  - if: "a new tool or agent needs to contact customers"
    then: "grant it propose rights only; no direct send credential; require it to write to the ledger"
  - if: "an agent needs access to shared memory"
    then: "issue it its own scoped, expiring credential; read-only unless it must write; stamp attribution server-side"
  - if: "a surface is customer-facing"
    then: "serialize an explicit allow-list of fields; seller-only analysis never crosses the boundary"
  - if: "an enrichment signal suggests a new lead at an existing customer"
    then: "resolve identity first; attach as a contact on the known account and exclude prospect programs"
  - if: "coordination is being evaluated"
    then: "randomize at the account level, hold out accounts, and pre-register a retention-level success metric"
assessment_questions:
  - "Who can see every message a single customer will receive from you next week, across all teams, tools, and agents?"
  - "Which systems send to customers directly, including reps' mailboxes and team-purchased AI tools?"
  - "Do you record the claims made to each customer (prices, dates, product status), or only that a message was sent?"
  - "Is there a per-person and per-account promotional contact limit shared across teams?"
  - "When sales, marketing, and success want the same person the same week, who decides, by what rule?"
  - "How does a job change at an existing customer appear in your systems: as a new lead or a new role on a known account?"
patterns: [Propose Not Send, Contact Ledger, Shared Attention Budget, Relationship Owner Merge, Send-Time Re-check, Do-Nothing Option, Channel Compilers, Customer-Safe Boundary, Scoped Agent Credentials]
anti_patterns: [The Side Door, The Fatigue Spiral, Two Voices, Siloed Agent Memory, Arbitration by Volume, The Broken Promise]
metrics:
  primary: "retention and expansion at renewal in coordinated vs held-out accounts"
  process: ["contacts per person and per account (median and p95)", "coordinator coverage of customer-facing sends"]
  guardrails: ["contradiction rate per 1,000 active accounts", "negative responses per person", "repeat asks in support", "suppressed operational messages"]
maturity_dimension: decisioning
```

## References

1. Baek, J., Chen, D., Ma, W., and Mitrofanov, D. "Balancing Customer Engagement and Annoyance in Online Retail: Insights from a Field Experiment." Working paper, SSRN 6165626. http://www.columbia.edu/~wm2428/papers/customer_engagement.pdf (accessed 2026-09-26)
2. Awan, A. (2015-07-27). "Less is more: You're about to receive less email from LinkedIn." LinkedIn Blog. https://www.linkedin.com/blog/member/product/less-email-from-linkedin ; engineering work: Gupta, R., Liang, G., Tseng, H.-P., Holur Vijay, R. K., Chen, X., and Rosales, R. (2016). "Email Volume Optimization at LinkedIn." KDD 2016. https://dl.acm.org/doi/10.1145/2939672.2939692
3. Gartner (2025-06-25). "Gartner Sales Survey Finds 61% of B2B Buyers Prefer a Rep-Free Buying Experience." https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-sales-survey-finds-61-percent-of-b2b-buyers-prefer-a-rep-free-buying-experience
4. Taheri, H. (2026-03-14). "14 Agent Configs, 3 Teams, Zero Source of Truth." hamedtaheri.com. https://hamedtaheri.com/articles/fourteen-agent-configs-zero-source-of-truth
5. Zendesk (2025-11-18). "CX Trends 2026" press release. https://www.zendesk.com/newsroom/press-releases/contextual-intelligence-becomes-the-new-standard-for-exceptional-customer-experience-in-2026/
6. Taheri, H. "One brain, many surfaces" and "Building an enterprise personalization engine," sections on channel compilers. hamedtaheri.com. https://hamedtaheri.com/articles/building-enterprise-personalization-engine
7. Salesforce (2020). *State of the Connected Customer*, 4th edition. https://c1.sfdcstatic.com/content/dam/web/en_us/www/documents/research/salesforce-state-of-the-connected-customer-4th-ed.pdf
8. Taheri, H. "Governed Memory: A Production Architecture for Multi-Agent Workflows." arXiv:2603.17787, 2026-03-18. https://arxiv.org/abs/2603.17787