The Real Cost of Running LLMs in Production (and How We Cut It 40%)

The first cost estimate almost always uses input token price times expected document volume. This ignores output tokens, system prompt tokens repeated on every call, retry costs, and the cost of the reranker and embedding model sitting alongside the main LLM. By the time you account for all of it, the real number is typically three to five times the initial estimate.

Here is the breakdown from a recent deployment: a financial services team processing 50,000 documents per month.

The Real Cost Breakdown

Monthly volume: 50,000 documents
Avg document length: 3,000 tokens input
System prompt: 800 tokens (repeated every call)
Avg output: 600 tokens
Embedding calls: 1 per document chunk (avg 6 chunks per doc)
Reranker calls: 1 per query (avg 8 queries per document processed)

Naive estimate (input only):
  50,000 docs x 3,000 tokens x $0.01/1K = $1,500/month

Actual cost breakdown:
  LLM input (doc + system prompt): $2,300
  LLM output: $1,800
  Embedding (300,000 chunks): $900
  Reranker API calls: $400
  Retries and error handling (est. 3%): $165
  Total: ~$5,565/month

After optimization: ~$3,300/month (41% reduction)

The optimization did not require changing the architecture. It required four specific changes.

Lever 1: Prompt Compression

System prompts grow over time. Engineers add edge case handling, clarifications, and examples. A system prompt that started at 200 tokens tends to be 900 tokens six months later. Every call pays for it.

The fix is periodic prompt auditing. Strip every instruction that does not demonstrably improve output quality. In most cases, half the instructions in a mature system prompt are redundant or counterproductive.

We reduced the system prompt in this deployment from 800 to 310 tokens through a structured audit. That alone saved about $600/month at this volume.

Lever 2: Model Tiering

Not every step in a pipeline needs the most capable model. A typical document processing pipeline has multiple stages:

1. Document classification (is this a contract, invoice, or policy?)
2. Structured extraction (pull specific fields)
3. Validation and summarization
4. Exception handling (flagging anomalies for human review)

Classification and simple extraction are fully solved by GPT-4o-mini or Claude Haiku at a fraction of the cost of the frontier model. We use the flagship model only for steps 3 and 4 where nuanced judgment matters.

This tiering cut LLM costs by about 35% in this deployment, with no measurable reduction in output quality as measured by the client's QA team.

Lever 3: Caching

Semantic caching is underused. When the same or similar query appears multiple times (which it always does in enterprise workflows), you can return the cached response instead of making an LLM call. For a legal team asking the same types of questions about different contracts, the question structure is often identical even when the document changes.

Simple exact-match caching (Redis or a database) handles the easy cases. Semantic caching using embedding similarity handles paraphrased queries. We typically see cache hit rates of 15-25% for enterprise internal tools, which translates directly to cost savings.

Lever 4: Batch Processing

If your workflow is not latency-sensitive (overnight report generation, weekly analysis runs, end-of-month document processing), use batch APIs. OpenAI's Batch API costs 50% less than the real-time API. Claude's batch API offers similar discounts for asynchronous workloads.

The catch is that batch jobs have no latency guarantee. They complete within 24 hours but not in seconds. For any workflow where users are waiting for a result, this is not an option. For scheduled automation workflows, it is a straightforward 50% cost reduction.

When Cheaper Is Actually More Expensive

There is a case where using a cheaper model increases total cost. If the cheaper model has a higher error rate on your task, the downstream cost of handling errors often exceeds the savings on tokens.

A specific example: we tested GPT-4o-mini for contract clause extraction on a dataset where GPT-4o had a 2% error rate. The mini model had an 11% error rate. Each error required a human review step costing $4 in labor. At 50,000 documents per month, the 9% additional error rate cost $18,000 more in human review than the $1,200 saved in API costs. Use the cheaper model for simpler tasks. Validate the error rate before committing.

The Number That Actually Matters

Cost per correct output. Not cost per token. Not cost per API call. If a cheaper model requires a human review step 10% of the time and a more expensive model requires human review 2% of the time, the calculation needs to include the cost of that human time.

In most enterprise automation workflows, the human review cost exceeds the model API cost. Optimize for accuracy first. Optimize for token cost second, once you have established a quality floor you are not willing to go below.

Sources

• OpenAI — API Pricing (June 2026)
• OpenAI — Batch API Guide (50% discount)