The Cost-Quality Trade-off
Not every query needs GPT-4. But some queries absolutely do.
The challenge: route simple queries to cheap models and complex queries to capable models - automatically, in real-time, without hurting user experience.
After optimizing our routing system, we reduced costs by 65% while maintaining quality. Here is how.
•••
The Model Landscape
Cost vs Capability
| Model Tier | Cost (per 1M tokens) | Capability | Best For |
|---|---|---|---|
| Tier 1 (GPT-4) | $30-60 | Highest | Complex reasoning, nuance |
| Tier 2 (GPT-3.5/Claude Instant) | $1-3 | Good | Most tasks |
| Tier 3 (Open source) | $0.10-0.50 | Moderate | Simple tasks |
| Tier 4 (Tiny models) | $0.01-0.05 | Basic | Classification, extraction |
The Opportunity
If 70% of queries can be handled by Tier 2-3 models:
| Scenario | Monthly Cost |
|---|---|
| All Tier 1 | $45,000 |
| Smart routing | $15,750 |
| Savings | $29,250 (65%) |
•••
Routing Strategy 1: Query Classification
Classify queries before routing.
Complexity Dimensions
| Dimension | Simple | Complex |
|---|---|---|
| Reasoning depth | Direct answer | Multi-step logic |
| Domain knowledge | General | Specialized |
| Output format | Short text | Structured/long |
| Ambiguity | Clear intent | Requires interpretation |
| Stakes | Low | High |
Classification Model
We use a fast classifier to predict complexity:
| Feature | Weight |
|---|---|
| Query length | 0.15 |
| Question type | 0.25 |
| Domain keywords | 0.20 |
| Required output format | 0.15 |
| Historical similar queries | 0.25 |
Classification Results
| Predicted | Routed To | Accuracy |
|---|---|---|
| Simple | Tier 3 | 94% appropriate |
| Medium | Tier 2 | 91% appropriate |
| Complex | Tier 1 | 97% appropriate |
•••
Routing Strategy 2: Cascading
Start cheap, escalate if needed.
Cascade Flow
| Step | Model | Action |
|---|---|---|
| 1 | Tier 3 | Attempt response |
| 2 | Check | Confidence above threshold? |
| 3a | Yes | Return response |
| 3b | No | Escalate to Tier 2 |
| 4 | Check | Quality acceptable? |
| 5a | Yes | Return response |
| 5b | No | Escalate to Tier 1 |
Confidence Signals
| Signal | How to Measure |
|---|---|
| Model confidence | Logprobs / perplexity |
| Response completeness | Length and structure |
| Self-consistency | Multiple samples agree |
| Uncertainty markers | "I think", "maybe", hedging |
Cascade Results
| Metric | Value |
|---|---|
| Queries resolved at Tier 3 | 45% |
| Queries resolved at Tier 2 | 38% |
| Queries needing Tier 1 | 17% |
| Average latency overhead | 180ms |
| Quality maintained | 98.5% |
•••
Routing Strategy 3: Task-Based Routing
Different tasks have different requirements.
Task to Model Mapping
| Task | Recommended Tier | Why |
|---|---|---|
| Summarization | Tier 2 | Well-defined task |
| Q&A (factual) | Tier 2-3 | RAG provides context |
| Creative writing | Tier 1-2 | Quality varies by tier |
| Code generation | Tier 1-2 | Correctness critical |
| Classification | Tier 3-4 | Simple output |
| Data extraction | Tier 3-4 | Structured, constrained |
| Complex reasoning | Tier 1 | Capability required |
| Multi-turn dialogue | Tier 1-2 | Context handling |
Task Detection
Detect task type from query:
| Signal | Indicates |
|---|---|
| "Summarize", "TLDR" | Summarization |
| Question words | Q&A |
| "Write", "Create" | Creative |
| Code markers | Code generation |
| "Extract", "Find all" | Data extraction |
| "Why", "How come", "Explain" | Reasoning |
•••
Routing Strategy 4: User-Based Routing
Different users have different needs.
User Tiers
| User Type | Default Model | Upgrade Trigger |
|---|---|---|
| Free | Tier 3 | Complex query detected |
| Pro | Tier 2 | Always available |
| Enterprise | Tier 1 | Guaranteed access |
Dynamic User Routing
| Factor | Routing Impact |
|---|---|
| Query importance | User-flagged priority |
| Session history | Complex session = better model |
| Feedback history | Low satisfaction = upgrade |
| Time sensitivity | Urgent = faster model |
•••
Implementation Architecture
Router Components
| Component | Purpose |
|---|---|
| Query analyzer | Extract features |
| Classifier | Predict complexity |
| Router | Select model |
| Monitor | Track quality |
| Feedback loop | Improve over time |
Real-Time Decision Flow
| Step | Latency | Action |
|---|---|---|
| 1 | 5ms | Parse query |
| 2 | 15ms | Extract features |
| 3 | 10ms | Classify complexity |
| 4 | 5ms | Check user tier |
| 5 | 2ms | Select model |
| Total | 37ms | Route decision |
Fallback Handling
| Scenario | Fallback |
|---|---|
| Classifier fails | Default to Tier 2 |
| Selected model unavailable | Next tier up |
| Quality check fails | Escalate |
| Latency SLA breach | Skip to faster model |
•••
Quality Monitoring
Routing is only valuable if quality is maintained.
Quality Metrics by Route
| Route | Metric | Target | Alert |
|---|---|---|---|
| All | User satisfaction | Over 4.0/5 | Below 3.8 |
| Tier 3 | Escalation rate | Below 15% | Above 20% |
| Tier 2 | Task completion | Over 90% | Below 85% |
| Cascade | Avg escalations | Below 0.5 | Above 0.8 |
A/B Testing Routes
Before changing routing:
| Test | Duration | Sample |
|---|---|---|
| New classifier | 1 week | 10% traffic |
| New model tier | 2 weeks | 5% traffic |
| Threshold change | 3 days | 20% traffic |
Continuous Calibration
| Frequency | Action |
|---|---|
| Daily | Review escalation rates |
| Weekly | Retrain classifier |
| Monthly | Evaluate new models |
| Quarterly | Full routing audit |
•••
Results: Before and After
| Metric | Before | After |
|---|---|---|
| Monthly cost | $45,000 | $15,750 |
| Tier 1 usage | 100% | 17% |
| Quality score | 4.2/5 | 4.1/5 |
| P99 latency | 8.5s | 5.2s |
| User satisfaction | 4.1/5 | 4.2/5 |
•••
Common Pitfalls
| Pitfall | Problem | Solution |
|---|---|---|
| Over-aggressive routing | Quality drops | Conservative thresholds |
| Static routing | Misses optimization | Continuous learning |
| Ignoring latency | Cascade too slow | Parallel evaluation |
| No fallbacks | Single point of failure | Always have backup |
| Poor monitoring | Quality drift undetected | Comprehensive metrics |
•••
Key Takeaways
- 1Most queries do not need the best model - 70%+ can be handled by cheaper alternatives.
- 2Classification is cheap - 37ms overhead for 65% cost savings is excellent ROI.
- 3Cascading catches mistakes - Start cheap, escalate when needed.
- 4Task type matters - Route by what the user is trying to do, not just query text.
- 5Monitor obsessively - Routing only works if quality is maintained. Measure everything.
- 6Iterate continuously - New models change the landscape. Re-evaluate regularly.
Intelligent routing is the difference between AI that is expensive to run and AI that is economically sustainable at scale.