Testing LLM Applications: A Practical Guide

The Testing Paradox

Traditional software testing relies on determinism. Given input X, expect output Y. Run the test 1000 times, get the same result.

LLMs break this assumption. Same prompt, different response. How do you test something that is inherently non-deterministic?

After building test suites for multiple LLM applications, here is what actually works.

•••

The Testing Pyramid for LLM Apps

The traditional testing pyramid needs adaptation:

Layer	Traditional	LLM Application
Unit	Function behavior	Prompt templates, parsing
Integration	API contracts	LLM API calls, tool chains
E2E	User flows	Full conversation flows
Evaluation	N/A	Output quality assessment

The evaluation layer is new and critical for LLM apps.

•••

Layer 1: Unit Tests

Test everything that CAN be deterministic.

What to Unit Test

Component	Testable Aspect	Example
Prompt templates	Variable substitution	Name inserted correctly
Output parsers	Structure extraction	JSON parsed correctly
Validators	Input/output rules	Length limits enforced
Utilities	Helper functions	Token counting accurate

Prompt Template Testing

Test that templates produce expected prompts:

Test Case	Input	Expected in Output
Basic substitution	name="Alice"	"Hello Alice" in prompt
Empty handling	name=""	Graceful handling
Special characters	name="O'Brien"	Properly escaped
Long input	10000 char name	Truncated appropriately

Parser Testing

Test output parsing with known inputs:

Parser	Test Input	Expected Output
JSON extractor	Text with JSON block	Parsed object
List parser	Numbered list	Array of items
Code extractor	Markdown code block	Code string
Entity extractor	Text with entities	Structured entities

•••

Layer 2: Integration Tests

Test LLM interactions with controlled expectations.

Mocking LLM Responses

For fast, deterministic integration tests, mock the LLM:

Scenario	Mock Response	Test Assertion
Happy path	Valid response	Parsed correctly
Malformed output	Invalid JSON	Error handled
Empty response	Empty string	Graceful fallback
Timeout	Delayed response	Timeout triggered

Testing Tool Calls

For function calling, test the full chain:

Stage	What to Test
Tool selection	Correct tool chosen
Argument extraction	Arguments parsed correctly
Tool execution	Function called with args
Result handling	Response incorporates result

Rate Limiting and Retries

Test error handling:

Scenario	Expected Behavior
Rate limit (429)	Retry with backoff
Server error (500)	Retry then fail gracefully
Timeout	Retry once, then fallback
Invalid API key	Immediate failure, clear error

•••

Layer 3: End-to-End Tests

Test complete user flows with real LLM calls.

E2E Test Strategy

Consideration	Approach
Cost	Run subset, not full suite
Flakiness	Broader assertions
Speed	Parallel execution
Reproducibility	Seed when possible

Assertion Strategies

Do not assert exact output. Assert properties:

Bad Assertion	Good Assertion
response === "The capital is Paris"	response.includes("Paris")
response.length === 142	response.length > 50 && response.length < 500
response === expected	containsAllKeyPoints(response, keyPoints)

Conversation Flow Testing

For multi-turn conversations:

Turn	Input	Assert
1	"What is X?"	Explains X
2	"Give an example"	References X, provides example
3	"Compare to Y"	Mentions both X and Y

Test that context is maintained across turns.

•••

Layer 4: Evaluation Framework

The new layer unique to LLM apps.

Evaluation Dimensions

Dimension	What It Measures	Method
Correctness	Factually accurate	Ground truth comparison
Relevance	Answers the question	Semantic similarity
Coherence	Well-structured	LLM-as-judge
Safety	No harmful content	Content classifier
Style	Matches requirements	Pattern matching + LLM

LLM-as-Judge

Use another LLM to evaluate outputs:

Evaluation Task	Judge Prompt Focus
Correctness	Compare to reference answer
Helpfulness	Does it help user achieve goal
Safety	Check for harmful content
Style	Matches specified tone/format

Evaluation Metrics

Track these metrics over time:

Metric	Target	Alert Threshold
Correctness	Over 90%	Below 85%
Relevance	Over 85%	Below 80%
Safety	100%	Any violation
Coherence	Over 80%	Below 75%

•••

Regression Detection

Catch quality regressions before production.

Golden Dataset

Maintain a curated set of test cases:

Dataset Component	Size	Purpose
Core functionality	100 cases	Must-pass scenarios
Edge cases	50 cases	Boundary conditions
Known failures	25 cases	Previous bugs
Adversarial	25 cases	Attack resistance

Regression Test Pipeline

Stage	Action	Pass Criteria
Smoke	10 critical cases	100% pass
Core	100 golden cases	Over 95% pass
Full	All 200 cases	Over 90% pass
Evaluation	Quality metrics	No regression over 2%

Comparing Versions

When updating prompts or models:

Comparison	Method
A/B quality	Run both, compare scores
Regression check	New version >= old version
Cost comparison	Track token usage
Latency comparison	Track response times

•••

CI/CD Integration

Make testing part of every deployment.

Pipeline Stages

Stage	Tests Run	Blocking
PR	Unit + mocked integration	Yes
Merge	Unit + integration + smoke E2E	Yes
Deploy	Full E2E + evaluation	Yes if regression
Post-deploy	Continuous evaluation	Alert only

Cost Management

LLM tests cost money. Manage it:

Strategy	Savings	Trade-off
Mock in PR	90%	Less coverage
Subset in CI	70%	Sample risk
Full on deploy	0%	Necessary cost
Cached responses	50%	Staleness risk

Flaky Test Handling

LLM tests are inherently flaky. Handle it:

Strategy	When to Use
Retry once	Non-deterministic failures
Broader assertions	Output varies but valid
Statistical pass	4/5 runs pass = pass
Skip and alert	Known flaky, monitor manually

•••

Monitoring in Production

Testing does not stop at deployment.

Production Evaluation

Metric	Collection Method	Alert
User feedback	Thumbs up/down	Ratio drops
Task completion	Funnel tracking	Rate drops
Error rate	Log analysis	Rate increases
Latency	APM	p99 increases

Continuous Evaluation

Sample production traffic for ongoing evaluation:

Sample Rate	Evaluation	Action
1%	Automated quality check	Alert on regression
0.1%	Human review	Calibrate automated

•••

Key Takeaways

1Test what you can deterministically - Prompts, parsers, validators, utilities all have deterministic tests.

2Mock for speed, real for confidence - Use mocks in CI, real LLMs before deploy.

3Assert properties, not exact output - Contains key points, within length range, has required structure.

4Build an evaluation layer - LLM-as-judge, golden datasets, and quality metrics are essential.

5Detect regressions statistically - Single test failures mean less. Track trends across runs.

6Budget for test costs - LLM tests cost money. Be strategic about when to run what.

Testing LLM applications is different, not impossible. The key is accepting non-determinism and building systems that catch regressions despite it.