Prove the memory is earning its place

The scenario

Six weeks in, somebody asks whether the memory layer is actually helping. The honest answer is usually “it feels like it” — which is not an answer, and it is the same answer you would give if it had silently broken a month ago. The drop-in proxy makes this measurable in a way most stacks cannot, because memory is one header rather than an architecture.

Step 1 — Turn memory off for one call

off = llm.chat.completions.create(
    model="balanced",
    messages=[{"role": "user", "content": question}],
    extra_headers={"X-Anona-Memory": "false"},
)

X-Anona-Memory: false skips recall and injection for that call only. Everything else — model, prompt, parameters — is identical, which is what makes it a comparison rather than two anecdotes.

Step 2 — Run the same question with it on

on = llm.chat.completions.create(
    model="balanced",
    messages=[{"role": "user", "content": question}],
)
print(on.memories_injected, "memories injected")

Step 3 — Score a real set, not one question

for question, expected in eval_set:
    a = ask(question, memory=False)
    b = ask(question, memory=True)
    record_result(question, a, b, expected)

Twenty to fifty questions drawn from real traffic, where the answer depends on something said in an earlier session. Questions answerable from the current turn alone will score identically and tell you nothing — that is the control.

Step 4 — Keep running it

Put the eval set in CI. A recall regression is otherwise invisible: nothing errors, answers just quietly get worse.

Evals

Build the set from real conversations, including five questions memory cannot help with.
Expect no difference on those five. A difference there means something else is varying, and your measurement is not measuring what you think.
Report per-question, not just an average — memory usually helps a lot on a few and not at all on most, and the average hides both.
Re-run after any change to scoping, extraction settings or recall tuning.

Guardrails

Do not publish a benchmark number without the harness that produced it. This category has already seen a headline score fall by twenty points under independent measurement. An internal number is for deciding; a public number needs a script someone else can run.

Both arms cost credits. An A/B over fifty questions is a hundred calls, and the memory-off arm bills like any other.
memory: false still records the turn unless you also set X-Anona-Auto-Record: false — so a naive A/B slowly teaches the space the answers to its own eval set. Turn both off for the control arm.
Fresh spaces score badly and that is correct. Memory helps in proportion to what it has seen; measure on a space with real history.

Built from

Both are complete, runnable scripts: drop_in_proxy.py, memory_basics.py.

Getting Started

API Essentials

Memory (data plane)

LLM Proxy (drop-in)

Account (control plane)

SDKs & MCP

Integrations

Guides

Use Cases

Changelog

Prove the memory is earning its place

The scenario

Step 1 — Turn memory off for one call

Step 2 — Run the same question with it on

Step 3 — Score a real set, not one question

Step 4 — Keep running it

Evals

Guardrails

Built from

​The scenario

​Step 1 — Turn memory off for one call

​Step 2 — Run the same question with it on

​Step 3 — Score a real set, not one question

​Step 4 — Keep running it

​Evals

​Guardrails

​Built from

The scenario

Step 1 — Turn memory off for one call

Step 2 — Run the same question with it on

Step 3 — Score a real set, not one question

Step 4 — Keep running it

Evals

Guardrails

Built from