Traditional software has stack traces. LLMs have vibes.
When your AI application misbehaves, there is no error message telling you why. The model just produces bad output, and you are left guessing. This post provides a systematic framework for debugging LLM applications.
| Problem Type | Symptoms | Likely Cause |
|---|
| Wrong answer | Factually incorrect | Hallucination, bad retrieval |
| No answer | Refuses to respond | Safety filters, prompt issue |
| Partial answer | Incomplete response | Token limits, early stop |
| Wrong format | Does not match schema | Prompt clarity, model capability |
| Slow response | High latency | Model choice, prompt length |
| Inconsistent | Different answers each time | Temperature, prompt ambiguity |
| Component | How to Test | Isolation Method |
|---|
| Retrieval | Check retrieved docs | Log and review |
| Prompt | Test prompt directly | Playground testing |
| Model | Same prompt, different model | A/B comparison |
| Post-processing | Check raw output | Bypass formatting |
| Integration | End-to-end trace | Request logging |
| Challenge | Solution |
|---|
| Non-deterministic | Set temperature to 0 |
| Context-dependent | Log full context |
| User-specific | Capture user state |
| Time-sensitive | Record timestamps |
| Symptom | Diagnosis | Solution |
|---|
| Made-up facts | No grounding | Add RAG, cite sources |
| Fake citations | No verification | Verify before display |
| Wrong numbers | No calculation | Use tools for math |
| Invented entities | Context gap | Expand knowledge base |
| Step | Action | If Issue Found |
|---|
| 1 | Check if info exists in context | Expand retrieval |
| 2 | Check if prompt allows uncertainty | Add "say I do not know" |
| 3 | Check temperature | Lower to 0-0.3 |
| 4 | Check model capability | Try stronger model |
| Symptom | Likely Cause | Solution |
|---|
| Missing relevant docs | Bad embedding | Re-embed, fine-tune |
| Too many irrelevant docs | Low threshold | Raise similarity cutoff |
| Wrong chunk retrieved | Bad chunking | Adjust chunk size |
| Outdated information | Stale index | Refresh embeddings |
| Step | Action | Metric |
|---|
| 1 | Log retrieved chunks | Manual review |
| 2 | Check similarity scores | Should be over 0.7 |
| 3 | Test embedding quality | Recall at K |
| 4 | Verify chunk content | Relevance rating |
| Symptom | Cause | Solution |
|---|
| Invalid JSON | Model limitation | Structured output mode |
| Missing fields | Unclear prompt | Explicit field list |
| Wrong types | Ambiguous instruction | Add examples |
| Truncated output | Token limit | Increase max tokens |
| Approach | Success Rate | Trade-off |
|---|
| Better prompt | 70% | Free |
| Few-shot examples | 85% | More tokens |
| Structured output mode | 95% | Model support required |
| Post-processing validation | 99% | Extra latency |
| Cause | Detection | Solution |
|---|
| High temperature | Variance in outputs | Lower temperature |
| Ambiguous prompt | Different interpretations | More specific prompt |
| Context variation | Different retrievals | Stabilize retrieval |
| Model updates | Sudden behavior change | Version pinning |
| Log Level | What to Capture | When |
|---|
| Always | Request ID, latency, tokens | Production |
| Debug | Full prompt, response | Development |
| Trace | Retrieved chunks, scores | Investigation |
| Audit | User input, output | Compliance |
| Field | Purpose | Format |
|---|
| request_id | Trace requests | UUID |
| timestamp | Timeline | ISO 8601 |
| user_id | User context | String |
| prompt_hash | Prompt version | SHA-256 |
| model | Model used | String |
| tokens_in | Input size | Integer |
| tokens_out | Output size | Integer |
| latency_ms | Performance | Integer |
| status | Success/failure | Enum |
| Panel | Metrics | Purpose |
|---|
| Error rate | Failures/total | Health |
| Latency distribution | p50, p95, p99 | Performance |
| Token usage | In/out by model | Cost |
| Quality scores | Thumbs up/down | Satisfaction |
| Retrieval quality | Relevance scores | RAG health |
| Question | Example Investigation |
|---|
| Why wrong output? | Model said Paris is in Germany |
| Why did model say that? | Context mentioned Germany prominently |
| Why was Germany in context? | Retrieved wrong document |
| Why wrong retrieval? | Query embedding matched incorrectly |
| Why bad embedding? | Short query lacked context |
| Category | Frequency | Fix Difficulty |
|---|
| Prompt issues | 40% | Easy |
| Retrieval problems | 25% | Medium |
| Model limitations | 15% | Hard |
| Data quality | 10% | Medium |
| Integration bugs | 10% | Easy |
| Technique | Purpose | Example |
|---|
| Explicit constraints | Limit scope | "Only use provided context" |
| Output validation | Format guarantee | "Respond in valid JSON" |
| Uncertainty acknowledgment | Reduce hallucination | "Say unsure if uncertain" |
| Step-by-step | Improve reasoning | "Think through each step" |
| Gate | When | Action if Failed |
|---|
| Format validation | Before display | Retry or error |
| Fact checking | High-stakes | Human review |
| Toxicity filter | Always | Block response |
| Confidence threshold | Low confidence | Escalate |
| Alert | Threshold | Response |
|---|
| Error rate spike | Over 5% | Investigate immediately |
| Latency increase | Over 2x baseline | Check model/infra |
| Quality drop | Under 80% satisfaction | Review recent changes |
| Cost spike | Over 50% increase | Check usage patterns |
Users reported the chatbot giving wrong answers about product features.
| Step | Finding |
|---|
| 1. Check logs | 15% of queries had low retrieval scores |
| 2. Review retrievals | Wrong product docs retrieved |
| 3. Analyze queries | Short queries like "price" were ambiguous |
| 4. Test embeddings | Generic queries matched many products |
| Change | Impact |
|---|
| Added product context to queries | +25% relevance |
| Implemented query expansion | +15% relevance |
| Added reranking step | +10% relevance |
| Combined | 85% to 97% accuracy |
- 1Classify first - Know whether it is a retrieval, prompt, model, or integration problem before fixing.
- 2Reproduce deterministically - Set temperature to 0 and log everything to reproduce issues.
- 3Log comprehensively - You cannot debug what you did not log.
- 4Most issues are prompt issues - 40% of problems are solved with prompt improvements.
- 5Build quality gates - Validate outputs before showing to users.
- 6Monitor continuously - Catch regressions before users report them.
Debugging LLMs requires a different mindset than traditional software. Embrace uncertainty, instrument everything, and iterate systematically.
