LLM cost optimization: open-weight models run 56% of tokens, 14% of spend

The most useful data on LLM cost optimization this month does not come from a pricing page. It comes from the traffic of thousands of production applications. Vercel's AI Gateway routes tens of trillions of tokens a month between apps and model providers, and its September report shows that open-weight models now handle 56% of those tokens — while taking only 14 cents of every dollar spent. Production teams have already worked out the pattern: send volume to cheap models, pay frontier prices only where the task earns it.

What the gateway data shows

Vercel's Production Index, covering activity through August 2026, reports:

• Open-weight share of tokens: 56% in August, up from 7% in December 2025 — the first month open-weight models ran a majority of gateway volume.
• Price per token: the average fell 23.2% in August, the third consecutive monthly decline. Among teams running more than ten million tokens in both July and August, the median team paid 7.6% less per token.
• Spend concentration: Anthropic's models took 64 cents of every dollar spent in August, and have never taken less than 61 cents in any month since December.
• Stepping down within a lab: Fable 5's share of spend fell from 13.2% in July to 4.9% in August, while Opus 5, at roughly half the price, rose to 22.5%. Vercel says nine in ten teams that ran Fable cut their usage, and more moved to Opus 5 than anywhere else.

As The New Stack notes, token volume and money are different measures. Vercel counts input, output, reasoning, cached-input and cache-creation tokens. A cheap model can process most of the tokens while an expensive model takes most of the budget, and both can be the right choice.

Token share is not cost share

The 56%-versus-14% gap is the whole lesson. It tells you that teams are not replacing frontier models with open-weight ones; they are adding open-weight models for the high-volume work — classification, extraction, summarisation, routing, first drafts — and keeping frontier models for reasoning-heavy steps, where errors are expensive.

It also warns against optimising the wrong number. A 23% drop in average price per token is mostly a mix effect: more of the traffic moved to cheaper models. The median team saw 7.6%. If you are budgeting from headline price cuts, you will over-estimate your savings.

LLM cost optimization in production: five steps

From the gateway data and our own AI engineering work, the pattern that works has five parts.

1. Measure cost per task, not per token

A support-ticket triage, a document extraction, a code review: each is a task with a quality bar and a price. A cheaper model that needs two retries, or a longer prompt, can cost more per completed task than an expensive model that gets it right first time. Log tokens, model and outcome per task, and compare on that. Our guide to agent observability and cost controls covers the instrumentation.

2. Tier models by task

Sort your workloads into those that need frontier reasoning and those that need throughput. Run an evaluation set for each task against two or three candidate models, including at least one open-weight model. Many teams find that a large share of their calls pass their quality bar on a model costing a fraction of their default.

3. Step down before you switch labs

The Fable-to-Opus shift is instructive: when Anthropic offered a cheaper model with a similar profile, teams moved within the lab because behaviour stayed consistent. Vercel's authors put it plainly: "Lab loyalty doesn't follow brand, it follows model profile, and consistency wins." Trying the next tier down from your current model is usually the cheapest experiment you can run, because prompts and output formats tend to carry over.

4. Keep a fallback in another lab

The same report shows the risk of standing still. Google's Gemini 3 Flash lost most of its gateway token share after a successor changed its profile, and more than three-quarters of the lost volume moved to other labs. A routing layer that can send a task to a second provider — tested, not theoretical — protects you from both price changes and model changes. We describe one approach in AI gateway model routing.

5. Cache and trim before you switch

Prompt caching and shorter prompts cut cost on any model. Cached-input tokens are typically billed at a steep discount, so stable system prompts and shared context should be structured to hit the cache. That work pays off whichever model you end up on.

When is a frontier model worth the price?

When errors are costly and hard to detect: multi-step reasoning, code changes across a codebase, decisions a customer sees directly, or tasks where a wrong answer looks plausible. The gateway's spend data suggests production teams agree — they still pay frontier prices for a minority of tokens because those tokens carry most of the risk. The mistake is using that model for everything by default.

Are open-weight models cheaper in production?

Per token through an API, usually yes. Self-hosting them is a different calculation: GPUs, operations and utilisation decide the real price, and at low volume a hosted API is almost always cheaper, as we found when we checked claims about self-hosted LLM cost. Open-weight also brings choices about where the model is served and under which jurisdiction — relevant for regulated data, and for regional pricing quirks like DeepSeek's peak-hour pricing.

What to do this quarter

1. Pick your three most expensive tasks by monthly spend.
2. Build a small evaluation set for each — fifty real examples with known-good outputs is enough to start.
3. Test the next tier down in your current lab and one open-weight model.
4. Route by task, keep the frontier model where the evaluation says it matters, and add a tested fallback.
5. Track cost per completed task monthly, not the provider's headline price.

If your AI bill is growing faster than usage, our AI engineering services team can run that evaluation and build the routing layer, so the saving shows up per task rather than only on the price list.