How to Reduce LLM API and Token Costs

1. Prompt Engineering Optimization: Pruning System Instructions

The most common cause of token bloat is "lazy prompt engineering." Developers often copy massive, multi-paragraph system prompts containing conversational rules, examples, and markdown schemas, and append them to every single message in a session. Because LLMs are stateless, the provider must process the entire system instructions on every turn.

To optimize prompt size:

• Consolidate Rules: Remove conversational boilerplate (e.g., "You are a helpful assistant. Try to be polite..."). Models are already trained on safety and behavior; prioritize deterministic, functional instructions instead.
• Limit Few-Shot Examples: Providing 5 examples in a prompt might improve accuracy slightly, but it consumes thousands of input tokens. Test if you can get identical accuracy with 1 or 2 high-quality examples, or offload examples to a fine-tuned model.
• Use Short Variables: Replace verbose tags with concise structures.

2. Structuring Payloads: XML vs. JSON

Serialization formats have a high impact on token counts. When communicating structured data to an LLM, developers frequently default to JSON because it maps directly to programming objects. However, JSON is verbose:

A JSON block like:
{"username": "johndoe", "email": "john@example.com", "role": "admin"}
Requires colons, commas, double-quotes, and brackets that tokenizers must parse individually.

XML tags represent a far more token-efficient format:
<user name="johndoe" email="john@example.com" role="admin" />.

Furthermore, models like Claude are pre-trained on XML documentation. They recognize opening and closing XML tags (e.g., <doc> and </doc>) as unified semantic concepts, allowing tokenizers to merge them into fewer token IDs compared to JSON brackets.

3. Implementing Prompt Caching Strategies

For RAG pipelines or complex conversational agents, implementing Context Caching is the highest leverage cost-saving tool available.

To maximize cache-hits, developers must understand how cache keys are calculated. Providers cache context blocks starting from the beginning of the prompt. If any character changes in the middle of a cached segment, the cache invalidates for everything after that character.

Therefore, you must structure prompts with static content first:

// CORRECT STRUCTURE (CACHE FRIENDLY):
1. [STATIC] System Prompt & Instructions
2. [STATIC] RAG Documents / Reference Context
3. [DYNAMIC] User Conversation History
4. [DYNAMIC] New User Query

If you place conversation history (which changes on every turn) before reference documents, the cache will invalidate on every message, rendering prompt caching useless.

4. Model Routing: The Router-Agent Architecture

Not every query requires a $15 / MTok reasoning engine. A robust cost reduction architecture implements a **routing layer**:

When a user query arrives, a cheap classifier model (such as GPT-4o mini or Llama 3.1 8B, costing $0.15 / MTok) evaluates the complexity:

• If the query is a simple greeting or factual question: The classifier responds immediately or routes to the cheap model.
• If the query requires multi-step math or programming logic: The classifier routes the prompt to the premium model (Claude 3.5 Sonnet).

By offloading 70% of low-complexity requests to lightweight models, the blended cost of operation drops dramatically without degrading perceived capability.

5. Summary Checklist for Developers

Keep these items in mind during AI app development:

• Verify token sizes of prompts using our Token Calculator.
• Inject prompt caching headers in API payloads.
• Filter RAG contexts using semantic rerankers (e.g., Cohere Rerank) to restrict retrieval size to under 5 high-relevance chunks.
• Enforce strict output length restrictions via system prompts to limit expensive completion token generation.

Frequently Asked Questions