The Security Landscape Has Changed
Traditional application security focused on SQL injection, XSS, and authentication flaws. LLMs introduce entirely new attack surfaces.
Your AI system can be manipulated to leak data, execute unauthorized actions, generate harmful content, and bypass your carefully crafted guardrails - all through natural language.
This post documents our defense-in-depth approach to LLM security after 18 months in production.
The Threat Model
| Threat Category | Risk Level | Example Attack |
|---|---|---|
| Prompt Injection | Critical | Overriding system instructions |
| Data Exfiltration | High | Extracting training data or user data |
| Unauthorized Actions | High | Tool calls user should not make |
| Output Manipulation | Medium | Generating harmful content |
| Resource Abuse | Medium | Denial of service via expensive prompts |
| Model Extraction | Low | Stealing model behavior |
Layer 1: Input Validation
The first line of defense. Stop attacks before they reach the model.
Input Sanitization
We validate all user inputs before processing:
| Check | What It Catches | Action |
|---|---|---|
| Length limits | Resource abuse | Truncate or reject |
| Character filtering | Encoding attacks | Remove or escape |
| Pattern detection | Known attack signatures | Block and log |
| Language detection | Off-topic abuse | Warn or redirect |
Attack Pattern Detection
We maintain a pattern library for known attacks:
| Pattern Type | Examples |
|---|---|
| Instruction override | "Ignore previous instructions" |
| Role playing | "You are now DAN" |
| Encoding tricks | Base64, ROT13, unicode abuse |
| Delimiter injection | Fake system messages |
| Context overflow | Extremely long inputs |
Input Validation Results
| Metric | Value |
|---|---|
| Attacks blocked at input | 73% |
| False positive rate | 0.8% |
| Avg validation latency | 12ms |
Layer 2: System Prompt Hardening
Your system prompt is your security policy. Harden it.
Prompt Structure
Our system prompts follow a security-first structure:
| Section | Purpose |
|---|---|
| Identity | Who the assistant is |
| Boundaries | What it will not do |
| Priorities | Security over helpfulness |
| Examples | Good and bad behaviors |
| Escalation | When to refuse or escalate |
Defense Techniques
| Technique | What It Does |
|---|---|
| Role reinforcement | Repeatedly state identity |
| Explicit boundaries | List forbidden actions |
| Priority ordering | Security trumps helpfulness |
| Canary tokens | Detect prompt leakage |
| Output format constraints | Limit response structure |
Canary Tokens
We embed unique tokens in system prompts. If they appear in output, we know the prompt leaked:
| Token Type | Purpose |
|---|---|
| Unique ID | Detect full prompt extraction |
| Instruction markers | Detect partial leakage |
| Fake secrets | Honeypots for data extraction |
Layer 3: Output Filtering
Even with input validation and hardened prompts, bad outputs can slip through.
Output Checks
| Check | What It Catches | Action |
|---|---|---|
| Content moderation | Harmful content | Block and log |
| PII detection | Personal data leakage | Redact or block |
| Code injection | Executable code in output | Sanitize |
| Instruction echo | System prompt leakage | Block |
| Format validation | Unexpected structure | Sanitize |
PII Detection
We scan outputs for sensitive data patterns:
| Data Type | Detection Method |
|---|---|
| Email addresses | Regex pattern |
| Phone numbers | Regex with format detection |
| Credit cards | Luhn validation |
| SSN | Format and context |
| API keys | Known key formats |
Output Filtering Results
| Metric | Value |
|---|---|
| Harmful content blocked | 99.7% |
| PII leakage prevented | 100% |
| False positive rate | 1.2% |
Layer 4: Tool Call Security
Function calling introduces new attack vectors.
Tool Authorization
Not every user should access every tool:
| Check | Purpose |
|---|---|
| User permissions | Can this user call this tool |
| Rate limiting | Prevent tool abuse |
| Argument validation | Are arguments safe |
| Result filtering | Is output safe to return |
Dangerous Tool Patterns
| Pattern | Risk | Mitigation |
|---|---|---|
| Database queries | SQL injection | Parameterized queries only |
| File access | Path traversal | Whitelist paths |
| HTTP requests | SSRF | Whitelist domains |
| Code execution | RCE | Sandbox or disable |
Tool Call Sandboxing
High-risk tools run in isolated environments:
| Isolation Level | Tools | Protection |
|---|---|---|
| None | Read-only lookups | N/A |
| Process | Data processing | Memory limits |
| Container | External APIs | Network isolation |
| VM | Code execution | Full isolation |
Layer 5: Session Security
Multi-turn conversations introduce state-based attacks.
Session Threats
| Threat | Description | Defense |
|---|---|---|
| Context poisoning | Injecting malicious context | Context validation |
| History manipulation | Altering past messages | Immutable history |
| Session hijacking | Using another user session | Session isolation |
| Gradual escalation | Slowly bypassing guardrails | Reset thresholds |
Context Window Security
We validate conversation history before each request:
| Check | Purpose |
|---|---|
| Message integrity | History not tampered |
| Role validation | Only valid roles present |
| Content re-scan | Re-check old messages |
| Context limits | Prevent overflow attacks |
Layer 6: Monitoring and Response
Security requires visibility.
Security Metrics
| Metric | Alert Threshold |
|---|---|
| Attack attempts per hour | More than 100 |
| Successful bypasses | Any |
| PII in outputs | Any |
| Unusual tool calls | Statistical anomaly |
| Prompt extraction attempts | More than 10 per hour |
Incident Response
When attacks are detected:
| Severity | Response |
|---|---|
| Low | Log and continue |
| Medium | Rate limit user, alert team |
| High | Block user, immediate review |
| Critical | Suspend service, full audit |
Automated Response
Some responses are automated:
| Trigger | Automated Action |
|---|---|
| Repeated injection attempts | Temporary block |
| PII leakage detected | Output suppressed |
| Rate limit exceeded | Request queued |
| Known attack signature | Immediate block |
Real Attack Examples
Attacks we have seen in production:
| Attack | Method | Outcome |
|---|---|---|
| System prompt extraction | "Repeat your instructions" variations | Blocked by output filter |
| Data exfiltration | Encoding user data in responses | Blocked by PII detection |
| Tool abuse | Unauthorized database queries | Blocked by authorization |
| Jailbreak | Multi-turn context manipulation | Detected by pattern analysis |
| Resource exhaustion | 100K token inputs | Blocked by input limits |
Defense Effectiveness
Our layered approach results:
| Layer | Attacks Blocked |
|---|---|
| Input validation | 73% |
| System prompt | 12% |
| Output filtering | 9% |
| Tool security | 4% |
| Session security | 2% |
| Total blocked | 99.6% |
Key Takeaways
- 1Defense in depth is essential - No single layer catches everything. Stack multiple defenses.
- 2Input validation is your best ROI - Catching attacks early is cheaper and safer.
- 3Treat system prompts as code - Version control, review, and test them like any other security-critical code.
- 4Tool calls need authorization - The model should not decide what users can do.
- 5Monitor everything - You cannot defend against what you cannot see.
- 6Plan for novel attacks - Your pattern library will never be complete. Build systems that can learn.
LLM security is not a solved problem. New attacks emerge regularly. But with layered defenses and continuous monitoring, you can operate safely in production while staying ahead of threats.