A support agent I was working on got a redesign, and a week later a dashboard told us it had gotten more expensive. Cost per run had climbed from about 12 cents to about 20 cents.

Most of that jump was a bit of a lie. The redesign merged a collection of small, cheap, single-shot model calls into one larger agent that could now stop early. After it got to prod, around 60% of runs finished having done no LLM work – they short-circuited before spending anything. But the dashboard averaged cost only over the runs that actually spent something.

Measured against every incoming request, the actual growth was a fraction of what the dashboard reported. The rest of the growth traced back, eventually, to a single line of code that had never thrown an error, never changed an answer, and never once stopped charging us full price. Finding it meant following the money.

Following the money

I broke the growth down by token type, and about 87% of it was fresh, uncached input tokens on the expensive model. And input is the kind of tokens you’d expect a cache to swallow.

That pointed at the shape of the redesign. The old pipeline made a handful of one-shot calls and stopped. The new agent loops: it re-reads its entire working context on every internal step, three to four times per request, appending tool results as it goes.

Every extra turn re-sends the whole accumulated prompt at full price. Cost scales with (turns × prompt size), and the redesign had cranked both.

The number that shouldn’t have been possible

Caching was on for this workload, so the provider should recognise the repeated prefix and bill it at the cached rate, which for the model we run is about 90% cheaper ($0.25 versus $2.50 per million tokens). The metric to watch is the cache-hit ratio: cached input tokens divided by total input tokens. For a workload built out of one repeated prompt, it should sit near the ceiling. It was 33%.

The agent’s system prompt was rendered fresh on every turn by a callable, and near the top of it sat the current time. Here’s the shape of the buggy call site – a new prompt object built inside the loop, once per turn:

# inside the agent loop – runs once per turn:
for action in action_list:
    prompt = await render(
        "agent_system_prompt",
        current_datetime=datetime.now(UTC),   # seconds precision, re-evaluated every turn
        ...,
    )
    response = await model.call(prompt, ...)

Prompt caching matches on the longest identical leading run of tokens. The timestamp sat near the front, at seconds precision, and changed on every turn. So on turn two the prefix diverged from turn one within the first few lines, and the cache missed on that turn’s entire prompt. Turn three diverged from turn two, and so on.

One mutable field at the head of the prompt was invalidating the cache for everything behind it. That’s the whole bug, and it’s worth a name: prefix poisoning – a single volatile token at the head spoiling the entire cached tail.

Where this compounds

On a single-shot call, an unstable prefix costs you one cache miss. Annoying, cheap, and easy to ignore. In an agent loop it adds up. A stable prefix gets cached once and re-used for the rest of the run – you pay the write once and read cheap thereafter. An unstable prefix means you pay full freight on every one of N turns. The waste therefore is (turns per run) × (prefix size).

The fix, and the better fix

The obvious fix is to compute the timestamp once per run and reuse it, so the prefix is byte-stable across turns. That works – the ratio recovers and the bill comes down.

But code review pushed it further. Why was the prompt being re-rendered every turn in the first place? To pick up mid-run changes – like newly available tools. But those changes already reached the model another way, appended as tool results. The per-turn re-render wasn’t buying anything. It was pure redundancy that happened to reset the clock.

So the fix wasn’t “freeze the timestamp”. It was “render the prompt once, at the start of the run, and never again”:

async def _load_instructions(self) -> str:
    # rendered a single time, then cached for the whole run – the prefix is now fixed
    if self._instructions is None:
        self._instructions = await render(
            "agent_system_prompt",
            current_datetime=datetime.now(UTC),
            ...,
        )
    return self._instructions

Both versions give the model the same cache benefit. Notice the second still calls datetime.now(UTC) – freezing the clock was never the point. The point is where the render happens.

Measuring it without fooling yourself

The tempting way to report a fix like this is to diff last week’s bill against this week’s and claim the difference. However, dollar totals are confounded by everything at once – traffic moved, the model mix moved.

A cleaner metric is the cache-hit ratio. Freezing the prefix converts fresh tokens into cached tokens. So:

  1. Lock a before-baseline of the cache-hit ratio, per model, over full weeks, before you deploy.
  2. Capture the same ratio over full weeks after.
  3. Price the lift against the actual after-period token volume, at the per-token gap between the fresh and cached rates. Pricing against real traffic means a change in volume can’t inflate your claim.

In one line:

savings ≈ (ratio_a − ratio_b) × input_tokens × (fresh_rate − cached_rate)

_a is for “after”

_b is for “before”

If you want to be precise, converting a miss into a hit still pays a cache-write the first time per cache window, so the very first request in each window isn’t free. Over a long run it rounds away, though.

The before-baseline was 32.9% on that model, over two full weeks of production agent runs. After the fix, it settled at about 53% – a lift of roughly twenty points, about 1.6× the old rate.

Why 53% and not the 80% I’d quietly hoped for? Because the fix didn’t make caching perfect. The agent pulls in new tools as a run goes on, appended as it works – so part of every run is new text the cache has never seen. And the cache doesn’t survive across separate runs, so each fresh request starts cold. The realistic shape of the win is “lifted to about half”, not “eliminated”.

Priced against real traffic, that twenty-point lift is worth about $7,000 a month – roughly $85k a year – in token cost. I trust that figure because I checked it two independent ways: pricing the ratio lift against actual token volume, and watching the model’s real bill fall while that volume was flat.

The checklist

  • Audit the first tokens of every system prompt. Anything that varies per call – timestamps, UUIDs, request IDs, randomised few-shot order, a dict or set serialised in nondeterministic order – poisons the cached prefix behind it.
  • Push per-request variability to the tail, or into a message. Do not bake it into the head. Cacheable content first, volatile content last.
  • Instrument cached-versus-fresh input tokens as a first-class metric, with drift alerting. It is the only signal a prefix-poisoning bug ever emits.
  • Prefer fixes that make the bad state unrepresentable over fixes that merely guard against it. Render-once beats remembering not to re-render.

The clock in the prompt was one line, but it charged us full price on a large workload, for as long as it sat there. That’s the whole danger of a silent-tax bug: the only way it ever shows up is if you go looking.

Thanks for reading!