← All use cases
The scenario
Memory makes every prompt bigger. That is the deal — and it is fine until the
bill arrives and nobody can say how much of it was memory, or whether the tenth
recalled item was worth anything.
Two levers, and both are measurable rather than vibes.
Step 1 — Put a hard ceiling on the block
get_context returns a prompt-ready string with the budget enforced
server-side. Whole memories are dropped, lowest-ranked first — never text cut
mid-sentence, which would leave the model reading half a fact.
This beats a client-side loop because your loop does not know the ranking.
Step 2 — Make the block cacheable
Providers discount an unchanged prompt prefix. A memory block reordered by
relevance every turn destroys that prefix on every turn.
stable orders by when memories were created rather than by score, so the block
mostly does not change between turns and the discount can apply.
Step 3 — Read what it actually cost
context_tokens is what memory added. cache_prefix_tokens is how much of that
was an unchanged prefix — that is the measured eligibility for a provider
discount, not an estimate of savings.
Evals
- Run twenty representative turns with
max_tokens unset. Record
context_tokens.
- Re-run with a ceiling at roughly half. Score answer quality on the same
questions.
- Find the point where quality drops. That number is your budget; most spaces
sit well below the default.
- Run the same twenty turns in
relevance and then stable order and compare
cache_prefix_tokens. Relevance mode should read near zero — that contrast
is the evidence the setting is doing anything.
Guardrails
cache_prefix_tokens reports eligibility, not money saved. Whether a
discount applies is the provider’s business, and it depends on their cache
state. Do not present this figure to a customer as a saving.
- Stable ordering trades freshness for cacheability. The most relevant
memory is no longer first in the block. If answer quality drops, this is the
first thing to turn off.
- Budget selection happens in relevance order, then emission is reordered —
so a tighter budget drops the weakest memory, not the newest one.
- There is no baseline to compare against. What you would have spent without
a memory layer is unobservable, and any “X% saved” number built on a guessed
baseline will not survive being asked how it was calculated.
Built from
Both are complete, runnable scripts:
drop_in_proxy.py,
memory_basics.py. 
