Why LLM Observability Is Different
Traditional observability focuses on latency, errors, and throughput. LLMs add new dimensions: output quality, token usage, model behavior drift, and semantic correctness.
You cannot debug what you cannot see. This post documents our complete observability stack for LLM applications.
•••
The Three Pillars Plus One
| Pillar | Traditional Use | LLM-Specific Use |
|---|---|---|
| Metrics | Request rates, latency | Token usage, costs, quality scores |
| Logs | Error messages | Full prompts and responses |
| Traces | Request flow | Multi-step reasoning chains |
| Evaluations | N/A | Output quality assessment |
•••
Metrics to Track
Operational Metrics
| Metric | Why It Matters | Alert Threshold |
|---|---|---|
| Request latency (p50, p95, p99) | User experience | p99 > 10s |
| Error rate | Reliability | > 1% |
| Throughput | Capacity planning | Sudden changes |
| Queue depth | Backpressure | > 100 requests |
Cost Metrics
| Metric | Why It Matters | Alert Threshold |
|---|---|---|
| Tokens per request | Cost control | > 2x average |
| Cost per request | Budget tracking | > $0.10 per request |
| Daily spend | Budget management | > daily budget |
| Cost by feature | ROI analysis | Feature-specific |
Quality Metrics
| Metric | Why It Matters | Alert Threshold |
|---|---|---|
| User feedback ratio | Satisfaction | Thumbs down > 15% |
| Regeneration rate | First-try quality | > 25% |
| Task completion | Effectiveness | < 80% |
| Hallucination rate | Accuracy | > 5% |
•••
Structured Logging
Every LLM call should log structured data.
Request Log Schema
| Field | Type | Purpose |
|---|---|---|
| request_id | string | Correlation |
| timestamp | datetime | Timeline |
| user_id | string | User context |
| model | string | Which model |
| prompt_tokens | int | Input size |
| completion_tokens | int | Output size |
| latency_ms | int | Performance |
| status | string | Success/failure |
| error_type | string | Failure category |
Response Log Schema
| Field | Type | Purpose |
|---|---|---|
| request_id | string | Correlation |
| prompt_hash | string | Identify prompt version |
| response_hash | string | Identify duplicates |
| finish_reason | string | Why generation stopped |
| tool_calls | array | Functions called |
| quality_score | float | Automated assessment |
Sensitive Data Handling
| Data Type | Handling | Why |
|---|---|---|
| User PII | Redact | Privacy compliance |
| Full prompts | Hash or encrypt | Security |
| Responses | Retention policy | Storage costs |
| API keys | Never log | Security |
•••
Distributed Tracing
LLM applications often involve multiple steps. Trace them all.
Trace Structure
| Span | What It Captures |
|---|---|
| Root | Full request lifecycle |
| Preprocessing | Input validation, prompt building |
| LLM Call | API call to model |
| Tool Execution | Each function call |
| Postprocessing | Parsing, validation |
| Response | Final formatting |
Span Attributes
| Attribute | Value |
|---|---|
| model.name | gpt-4-turbo |
| model.provider | openai |
| tokens.prompt | 1523 |
| tokens.completion | 342 |
| cost.usd | 0.0234 |
| cache.hit | true/false |
Multi-Turn Tracing
For conversations, link related traces:
| Field | Purpose |
|---|---|
| conversation_id | Groups all turns |
| turn_number | Order in conversation |
| parent_trace_id | Previous turn |
| context_tokens | Cumulative context |
•••
The Evaluation Pillar
Automated quality assessment at scale.
Evaluation Types
| Type | Frequency | Method |
|---|---|---|
| Real-time | Every request | Fast heuristics |
| Sampled | 1-5% of requests | LLM-as-judge |
| Batch | Daily | Full evaluation suite |
| Manual | Weekly | Human review sample |
Real-Time Evaluations
Fast checks that run on every request:
| Check | Method | Latency |
|---|---|---|
| Response length | Character count | < 1ms |
| Format validation | Regex/schema | < 5ms |
| Toxicity | Fast classifier | < 50ms |
| Language detection | Fast model | < 20ms |
Sampled Evaluations
Deeper checks on a sample:
| Evaluation | Sample Rate | Method |
|---|---|---|
| Factual accuracy | 2% | LLM comparison |
| Relevance | 5% | Embedding similarity |
| Coherence | 3% | LLM-as-judge |
| Helpfulness | 1% | Human review |
•••
Alerting Strategy
Not every metric needs an alert. Be strategic.
Alert Tiers
| Tier | Response | Examples |
|---|---|---|
| Critical | Immediate page | Service down, data breach |
| High | Within 1 hour | Error rate spike, quality drop |
| Medium | Within 1 day | Cost anomaly, latency trend |
| Low | Weekly review | Minor quality drifts |
Alert Configuration
| Metric | Condition | Tier | Action |
|---|---|---|---|
| Error rate | > 5% for 5 min | Critical | Page on-call |
| Latency p99 | > 15s for 10 min | High | Investigate |
| Daily cost | > 150% budget | Medium | Review usage |
| Quality score | < 0.8 for 1 hour | High | Check model |
| Hallucination rate | > 10% | Critical | Pause feature |
Anomaly Detection
Beyond static thresholds:
| Pattern | Detection Method |
|---|---|
| Sudden spike | Z-score > 3 |
| Gradual drift | Trend analysis |
| Periodic anomaly | Seasonal decomposition |
| User-specific | Per-user baselines |
•••
Debugging Workflows
When something goes wrong, how to investigate.
Issue: Quality Degradation
| Step | What to Check |
|---|---|
| 1 | Recent deployments (prompt changes?) |
| 2 | Model provider status |
| 3 | Input distribution shift |
| 4 | Sample low-quality responses |
| 5 | Compare to baseline prompts |
Issue: Latency Increase
| Step | What to Check |
|---|---|
| 1 | Model provider latency |
| 2 | Input token count trend |
| 3 | Queue depth |
| 4 | Downstream dependencies |
| 5 | Trace analysis of slow requests |
Issue: Cost Spike
| Step | What to Check |
|---|---|
| 1 | Request volume change |
| 2 | Tokens per request trend |
| 3 | Model mix (expensive models?) |
| 4 | Retry rate increase |
| 5 | User/feature breakdown |
•••
Dashboard Design
What to put on your LLM observability dashboard.
Overview Dashboard
| Panel | Visualization | Refresh |
|---|---|---|
| Request rate | Time series | 10s |
| Error rate | Time series + threshold | 10s |
| Latency percentiles | Time series | 10s |
| Cost accumulator | Counter | 1m |
| Quality score | Gauge | 5m |
Cost Dashboard
| Panel | Visualization |
|---|---|
| Daily spend | Bar chart |
| Cost by model | Pie chart |
| Cost by feature | Stacked area |
| Token efficiency | Scatter plot |
| Budget vs actual | Progress bar |
Quality Dashboard
| Panel | Visualization |
|---|---|
| Quality score trend | Time series |
| Feedback ratio | Time series |
| Hallucination rate | Time series |
| Top failure modes | Table |
| Sample bad responses | Log viewer |
•••
Implementation Checklist
| Category | Must Have | Nice to Have |
|---|---|---|
| Metrics | Latency, errors, tokens | Quality scores, costs |
| Logs | Structured request logs | Full prompt/response |
| Traces | LLM call spans | Full conversation traces |
| Evaluations | Real-time checks | Sampled LLM-as-judge |
| Alerts | Error rate, latency | Quality, cost anomalies |
| Dashboards | Overview | Cost, quality, debugging |
•••
Key Takeaways
- 1Add the evaluation pillar - Traditional observability is not enough. Build automated quality assessment.
- 2Log everything structured - Unstructured logs are useless at scale. Define schemas.
- 3Trace multi-step reasoning - LLM apps have complex flows. Distributed tracing is essential.
- 4Be strategic with alerts - Not every metric needs to page someone. Tier appropriately.
- 5Build debugging workflows - When something breaks, have a checklist. Do not improvise.
- 6Dashboards for different audiences - Engineers need details. Leaders need summaries. Build both.
You cannot improve what you cannot measure. LLM observability is the foundation for reliable, cost-effective AI systems in production.