Observability

A pufferfish with a dark maze-patterned back and yellow belly hanging motionless in green water, blurred mangrove roots behind it

Prefix poisoning: how one line disables prompt caching

A datetime.now() at the head of an agent’s system prompt was re-rendered every turn, so the provider’s prompt cache missed on nearly every request. Prefix caching is exact-match and silent, so prefix poisoning costs money without ever failing a test. Covers the lying cost-per-run metric, the render-once fix that beats freezing the timestamp, and how to measure the cache-hit ratio lift honestly (32.9% to ~53%).

A minimal LLM Ops stack with tracing and model costs

Building a minimal LLM Ops stack: a FastAPI “customer support reply drafter” instrumented with Langfuse for request tracing, grounded retrieval, and per-request model cost tracking, so every LLM call is inspectable.