"Tell me... do you bleed tokens?"
It's the question every CFO should be asking their engineering team right now. Because while the industry headlines celebrate the democratization of AI and the relentless fall of per-token pricing, something quietly alarming is happening inside finance departments across every sector: the AI line item keeps growing. Fast.
This is the Budget Bleed. And most organizations don't even know they have it.
The Paradox Nobody's Talking About
Token costs per million have dropped 90%+ in two years. Your AI bill has doubled. How? Volume and waste. The cheaper compute becomes, the more casually we consume it — and unoptimized pipelines scale that waste with brutal efficiency.
The Price of "Cheap" Is Still Expensive at Scale
Here's the math nobody does before they deploy. GPT-4o costs a fraction of what GPT-4 Turbo did eighteen months ago. Claude 3.5 Sonnet is dramatically cheaper per token than Claude 2. The models keep getting faster, smarter, and less expensive — so teams celebrate and stop watching the meter.
But "cheaper per token" times "exponentially more tokens consumed" still produces an enormous number. Especially when those tokens are doing work they were never meant to do.
A Real Cost Scenario
Imagine a mid-size enterprise running a customer support AI with these (unoptimized) characteristics:
- • 10,000 queries/day routed to a flagship model
- • 4,000 tokens average context per query (most of it unused boilerplate)
- • 15% retry rate due to malformed outputs and timeout errors
- • No caching — identical queries reprocessed daily
That's 46 million tokens per day — before retries. After retries: 53 million. Monthly: ~1.6 billion tokens. The bill isn't about the model being expensive. It's about the architecture being careless.
The Retry Loop Trap
Of all the ways organizations bleed tokens, the retry loop is the most insidious — because it looks like a reliability feature, not a cost center.
The scenario plays out like this: a prompt generates an output in the wrong format. The parsing layer fails. The system retries. The model generates a slightly different malformed output. The system retries again. And again. Each retry burns the full input context plus output tokens, and in some implementations, the failure count itself gets appended to the next attempt — making each retry more expensive than the last.
Why Retry Loops Compound
- The accumulation problem: A 3-retry limit on a 2,000-token prompt costs up to 8,000 tokens for a single failed task. At scale, this isn't an edge case — it's a line item.
- The cascading problem: Agentic workflows retry individual steps, but each downstream agent inherits the inflated context from failed upstream attempts.
- The silence problem: Retry logic is usually invisible in dashboards. Your observability layer shows "requests," not "wasted requests." You're flying blind.
The fix isn't fewer retries — it's prompts that produce reliable, parseable outputs the first time. Structured outputs, constrained generation, and explicit output schemas eliminate most retry scenarios before they happen.
The Context Window Fallacy
Just because your LLM can handle 200,000 tokens of context doesn't mean it should.
The context window is a technical capability, not a prompt strategy. Yet we see it treated as one constantly: dump the entire company knowledge base into the system prompt, throw in every relevant document, include all prior conversation history — and let the model sort it out.
The Real Cost of "Just Add More Context"
A 100-page company wiki in the system prompt adds roughly 70,000–80,000 tokens per query. For a task like "what's the office WiFi password?" — a 3-token answer — you're paying for 80,000 tokens of context you didn't need.
Multiply by 500 queries/day and you've spent more on context than on every actual answer combined. The model performance doesn't even improve — studies consistently show that retrieval-focused tasks degrade with bloated, unfocused context.
The context window is your most expensive real estate. Treat it like prime office space, not a storage unit. Only what's necessary for this specific query, at this specific moment, belongs in it.
The Novelty Trap: When AI Becomes a Toy
There's a deeper organizational problem underneath the technical ones. Many teams are still treating frontier AI models as novelties — impressive demonstrations of capability rather than production infrastructure with real cost structures.
This manifests in choices that would be absurd in any other engineering context:
- Using GPT-4o to classify a customer email into one of five categories (a task any fine-tuned 7B model handles at 1/50th the cost)
- Running a flagship model to extract structured data from a fixed-format invoice (deterministic regex or a small extractor model would do it instantly and for pennies)
- Sending every user interaction through the most capable — and most expensive — model in the portfolio, regardless of task complexity
This isn't an engineering failure. It's a maturity failure. The teams building these pipelines are often genuinely excited about AI capabilities — and excitement tends to default to the most impressive tool, not the most appropriate one.
Three Strategies to Stop the Bleed
1. Tier Your Architecture
Not every task requires a flagship model. Build a tiered system where model selection is a deliberate architectural decision, not a default.
The Model Tiers Framework
Tier 1 — Nano (Haiku, GPT-4o-mini, Gemini Flash)
Classification, routing, intent detection, extraction from structured formats, summarization of templated content. Target: 80% of your query volume.
Tier 2 — Mid (Sonnet, GPT-4o, Gemini Pro)
Complex reasoning, multi-step synthesis, open-ended generation with quality constraints. Target: 15% of query volume.
Tier 3 — Flagship (Opus, o3, Gemini Ultra)
Tasks where output quality directly drives revenue or where errors carry serious consequences. Target: 5% of query volume or less.
The router itself can be lightweight — a small classifier that reads the incoming query and assigns a tier based on complexity signals. The cost of routing is negligible; the savings are not.
// Conceptual routing logic
if (task.type === 'classify' || task.complexity === 'low') {
return routeTo('haiku-3-5') // ~$0.25/M tokens
} else if (task.requiresReasoning && !task.requiresCreativity) {
return routeTo('sonnet-4-6') // ~$3/M tokens
} else {
return routeTo('opus-4-8') // reserved for highest-value work
}
2. Cache Your Prompts
There are two distinct caching layers worth implementing, and most teams are using neither.
Semantic Response Caching
Store previous model responses indexed by embedding vector. When a new query arrives, check if a semantically similar query has already been answered. Cosine similarity above ~0.93 typically indicates the same question asked differently.
- • "What's your return policy?" and "How do I return something?" both resolve to the same cached answer
- • 40–70% cache hit rates are typical in domain-specific applications
- • Cache hits return in under 100ms vs. 2–5 seconds for fresh LLM calls
- • Cost for a cache hit: embedding lookup only — fractions of a cent per thousand queries
Prompt Prefix Caching
Most major LLM providers now support prompt caching at the API level. When a large system prompt is used repeatedly across requests, the provider caches the processed representation of that prefix. Subsequent requests that share the prefix pay dramatically less for those tokens.
- • Anthropic prompt caching: cached tokens cost 90% less than uncached
- • OpenAI: cached input tokens at 50% discount
- • Best for: large system prompts, document contexts, few-shot examples that don't change per request
- • Implementation: structure your prompts so the static prefix comes first, variable content comes last
Together, these two caching layers can eliminate 50–80% of your effective token spend without changing a single line of business logic.
3. Route Simple Tasks to Smaller Models
This deserves its own section because it's the strategy with the largest immediate ROI and the lowest implementation cost — yet it's consistently the last one teams adopt.
The uncomfortable truth is that the majority of production AI tasks are simple. Not in a pejorative sense — they're valuable, they're frequent, they're business-critical. But they're not cognitively complex. And cognitive complexity is what you're paying for when you route to a flagship model.
Tasks That Don't Need Frontier Models
- • Intent classification — "Is this a complaint, a question, or a compliment?" (small fine-tuned model, <1ms)
- • Structured extraction — pulling fields from invoices, receipts, or forms (extractor model or even regex)
- • Sentiment detection — thumbs up / thumbs down signals on short text (7B model or distilled classifier)
- • Translation of known content — product descriptions, FAQ entries (small multilingual model)
- • Simple summarization — "Summarize this 200-word review in one sentence" (Haiku-class model)
- • Moderation — content policy checks (dedicated moderation model)
The Cost Math
Routing 70% of your queries from a flagship model ($15/M output tokens) to a small model ($0.30/M output tokens) represents a 98% cost reduction on those queries. If those queries make up 70% of your volume, your total bill drops by roughly 68% overnight — with zero impact on the quality of work that actually requires the flagship.
What True AI Maturity Looks Like
The immature AI organization measures success in tokens consumed, models deployed, and benchmark scores cited in all-hands meetings. The mature AI organization asks a different question: what did these tokens produce?
True AI maturity is architectural. It's the ability to look at a pipeline and answer, for every single LLM call:
- Is this the right model for this specific task?
- Has this query — or one semantically equivalent — already been answered?
- Is every token in this context window load-bearing, or are we hauling dead weight?
- If this call fails, what's the cost path? Is the retry strategy optimized or naive?
- Can we measure the business value this token spend produced?
The Maturity Checklist
- ✓ Model routing in place — right model for right task
- ✓ Semantic caching layer active — hit rates measured and improving
- ✓ Prompt prefix caching enabled for static system prompts
- ✓ Context trimming in RAG — only relevant chunks, not entire documents
- ✓ Retry policies with structured output schemas, not naive repetition
- ✓ Token spend dashboard with cost-per-outcome, not just cost-per-request
- ✓ Regular prompt audits to remove bloat and stale instructions
Scalable AI Product or Budget Bonfire?
Here's the question every engineering leader and every CFO should be sitting with as 2026 accelerates: are we building a scalable AI product, or are we burning through next year's tech budget on unoptimized infrastructure?
The units economics of AI are actually extraordinary — if you architect for them. The same capabilities that cost hundreds of thousands of dollars two years ago now cost tens of thousands. But that headroom doesn't automatically become savings. It becomes runway for more waste if the underlying architecture stays careless.
Tokens are not a vanity metric. Every token is a decision about where you're spending engineering judgment and financial capital. The teams that build durable AI businesses will be the ones who treat that decision with the same rigor they bring to every other infrastructure cost.
The Real Question
The impressive demo uses the biggest model with the longest context and the most tokens. The impressive business uses the right model with the right context and only as many tokens as the task actually requires. Which one are you building?
Key Takeaways
- The bleed is real: Falling token prices mask rising total spend — volume and waste are the culprits
- Retry loops are silent cost centers: Build structured outputs to eliminate them at the source
- Context windows are expensive real estate: Only load what the specific task requires
- Tier your models: Route by task complexity, not by default or convention
- Cache aggressively: Semantic response caching and prompt prefix caching together can eliminate 50–80% of effective token spend
- Measure outcomes, not tokens: Cost-per-outcome is the metric that maps to business value
Topics