ROUTE A
Semantic Cache
Stores LLM responses and returns cached answers for semantically similar questions — even if the wording differs.
↓ up to 100% cost on repeated queries
ROUTE B
Prompt Compression
LLMLingua 2 compresses verbose prompts by up to 50% before sending, reducing input token cost immediately. Runs a model to trim the prompt, so it adds about 1–2 seconds — a brief delay traded for large token savings. Disable per-request for latency-critical chat while keeping every other saving.
↓ up to 50% on input tokens · ~1–2s added
ROUTE C
Model Router
Analyses each request and routes simple tasks to cheaper models (gpt-4o-mini, Gemini Flash) in under 1ms.
↓ up to 90% per routed request
ROUTE D
Context Pruning
Summarises old conversation turns when context exceeds 8K tokens, preventing runaway costs in long agent threads.
↓ up to 60% on long conversations
ROUTE E
Thinking Budget
Caps reasoning tokens on thinking models (Claude, o-series, Gemini 2.5/3) based on task complexity. Simple queries get capped budgets — complex tasks get full reasoning.
↓ up to 80% on thinking tokens