02 Design Patterns 8 min read 1,644 words
AI Anti-Patterns
Recognizing what NOT to do is as important as knowing best practices. This chapter catalogs common mistakes in AI system design.
Architecture Anti-Patterns
The God Prompt
Problem: Single massive prompt trying to do everything.
# ANTI-PATTERN: God Prompt
SYSTEM_PROMPT = """
You are a helpful assistant. You can:
1. Answer questions about our products
2. Help with technical support
3. Process refunds
4. Schedule appointments
5. Translate languages
6. Write code
7. Analyze data
8. Generate reports
... [continues for 5000 tokens]
"""
Why it fails:
- Context consumed by instructions, not user content
- Model struggles with conflicting instructions
- Impossible to optimize for all cases
- Updates affect everything
Solution:
# PATTERN: Specialized components
class QueryRouter:
async def route(self, query: str) -> str:
intent = await self.classify_intent(query)
handler = self.handlers[intent]
return await handler.process(query)
Single Provider Dependency
Problem: Entire system depends on one LLM provider.
# ANTI-PATTERN: Single provider
async def generate(prompt: str) -> str:
return await openai.chat.completions.create(...)
Why it fails:
- Provider outage = complete system failure
- Rate limits affect all traffic
- No price negotiation leverage
- Locked into one model family
Solution:
# PATTERN: Multi-provider with failover
class LLMClient:
def __init__(self):
self.providers = [OpenAI(), Anthropic(), Google()]
async def generate(self, prompt: str) -> str:
for provider in self.providers:
try:
return await provider.generate(prompt)
except ProviderError:
continue
raise AllProvidersFailedError()
Premature Fine-Tuning
Problem: Fine-tuning before exhausting simpler approaches.
Why it fails:
- Expensive and time-consuming
- Requires quality training data (often unavailable)
- Hard to update and maintain
- Often unnecessary
Decision flow:
Fine-tuning is the fourth thing to try, not the first: each earlier rung is cheaper and only failure justifies the climb
Text version of this diagram
Try prompting first
↓ (not working)
Try few-shot examples
↓ (not working)
Try RAG for knowledge
↓ (not working)
Consider fine-tuning (with 500+ examples)
RAG Anti-Patterns
Retrieve Everything
Problem: Retrieving too many documents regardless of relevance.
# ANTI-PATTERN: Retrieve everything
results = vector_db.search(query, top_k=50)
context = "\n".join([r.text for r in results])
Why it fails:
- Noise drowns out signal
- Exceeds context limits
- Wastes tokens on irrelevant content
- "Lost in the middle" effect
Solution:
# PATTERN: Quality over quantity
results = vector_db.search(query, top_k=20)
reranked = await reranker.rerank(query, results)
context = "\n".join([r.text for r in reranked[:5] if r.score > 0.7])
No Chunking Strategy
Problem: Arbitrary or no chunking of documents.
# ANTI-PATTERN: Fixed-size blind chunking
chunks = [text[i:i+1000] for i in range(0, len(text), 1000)]
Why it fails:
- Breaks mid-sentence, mid-paragraph
- Loses semantic coherence
- Separates related information
- Poor retrieval quality
Solution:
# PATTERN: Semantic-aware chunking
chunks = semantic_chunker.chunk(
text,
chunk_size=500,
overlap=100,
respect_boundaries=["paragraph", "section"]
)
Ignoring Metadata
Problem: Treating all documents as equal text.
# ANTI-PATTERN: Ignore metadata
embedding = embed(document.text)
vector_db.insert(embedding, {"text": document.text})
Why it fails:
- Cannot filter by date, source, type
- No access control per document
- Cannot weight recent vs old
- Loses valuable context
Solution:
# PATTERN: Rich metadata
vector_db.insert(embedding, {
"text": document.text,
"source": document.source,
"date": document.date,
"access_level": document.access_level,
"document_type": document.type,
"section": document.section
})
# Filter query
results = vector_db.search(
query,
filter={"date": {"$gte": "2024-01-01"}, "access_level": user.level}
)
Agent Anti-Patterns
Infinite Loop Risk
Problem: No termination conditions for agents.
# ANTI-PATTERN: No limits
while not done:
action = await agent.decide_action()
result = await execute(action)
done = agent.check_done(result)
Why it fails:
- Agents can loop forever
- Costs spiral out of control
- Never returns to user
- Resource exhaustion
Solution:
# PATTERN: Multiple termination conditions
MAX_STEPS = 20
MAX_COST = 10.0
MAX_TIME = 300 # seconds
for step in range(MAX_STEPS):
if cost_tracker.total > MAX_COST:
return "Cost limit reached"
if time.time() - start > MAX_TIME:
return "Time limit reached"
action = await agent.decide_action()
result = await execute(action)
if agent.check_done(result):
return result
return "Step limit reached"
Unsafe Tool Access
Problem: Giving agents unrestricted tool access.
# ANTI-PATTERN: Full access
tools = [
delete_file,
execute_shell_command,
send_email,
database_query # unrestricted!
]
Why it fails:
- Agent can delete critical files
- Can exfiltrate data
- Can execute malicious commands
- No audit trail
Solution:
# PATTERN: Scoped, validated tools
tools = [
ScopedFileTool(allowed_dirs=["/tmp/agent"]),
RestrictedShellTool(allowed_commands=["ls", "cat"]),
EmailTool(requires_confirmation=True),
ReadOnlyDatabaseTool(allowed_tables=["products"])
]
Agent Without Memory
Problem: Agent restarts from scratch every turn.
# ANTI-PATTERN: Stateless agent
async def handle_message(message: str) -> str:
return await agent.run(message) # No context
Why it fails:
- Cannot do multi-turn tasks
- Repeats same mistakes
- Cannot learn from experience
- Poor user experience
Solution:
# PATTERN: Persistent memory
async def handle_message(session_id: str, message: str) -> str:
memory = await memory_store.get(session_id)
response = await agent.run(message, memory=memory)
await memory_store.update(session_id, memory)
return response
Prompting Anti-Patterns
Vague Instructions
Problem: Ambiguous prompts expecting specific behavior.
# ANTI-PATTERN: Vague
prompt = "Help the user with their request."
Why it fails:
- "Help" is undefined
- No format specified
- No boundaries
- Inconsistent behavior
Solution:
# PATTERN: Specific and structured
prompt = """
You are a customer support agent for TechCorp.
Your role:
- Answer questions about our products
- Help troubleshoot issues
- Escalate to human when unsure
Response format:
1. Acknowledge the issue
2. Provide a solution or ask clarifying questions
3. Offer next steps
Do NOT:
- Make promises about refunds (escalate instead)
- Provide legal or medical advice
- Share internal company information
"""
No Output Format
Problem: Expecting structured output without specifying format.
# ANTI-PATTERN: Hope for structure
prompt = "Extract the person's name, date, and location from this text."
response = await llm.generate(prompt)
# Response: "The person is John, he was there on March 5th in NYC"
# Now try to parse that...
Solution:
# PATTERN: Explicit format
prompt = """
Extract information and return as JSON:
{
"name": "string",
"date": "YYYY-MM-DD",
"location": "string"
}
Text: ...
"""
# Or use structured output APIs
response = await llm.generate(prompt, response_format={"type": "json_object"})
Evaluation Anti-Patterns
Vibes-Based Evaluation
Problem: "It looks good to me" as the evaluation method.
# ANTI-PATTERN: Manual spot-checking
for i in range(5):
response = await generate(test_prompts[i])
print(response) # Developer looks at it
# "Looks good, ship it!"
Why it fails:
- Not reproducible
- Cherry-picked examples
- No baseline comparison
- Misses edge cases
Solution:
# PATTERN: Systematic evaluation
eval_dataset = load_eval_set() # 100+ examples
results = []
for example in eval_dataset:
response = await generate(example["input"])
score = await evaluate(response, example["expected"])
results.append(score)
metrics = {
"accuracy": sum(results) / len(results),
"failures": [e for e, r in zip(eval_dataset, results) if r < 0.5]
}
Training on Test Set
Problem: Using evaluation data for development decisions.
# ANTI-PATTERN: Overfitting to eval
for iteration in range(100):
accuracy = evaluate_on_test_set() # Same set every time
tweak_prompt_based_on_failures(test_set) # Optimizing for test set
Why it fails:
- Overfits to specific examples
- Real-world performance differs
- No true measure of generalization
Solution:
# PATTERN: Proper data splits
dev_set = load_dev_set() # For iteration
test_set = load_test_set() # Final evaluation only
# Iterate on dev set
for iteration in range(100):
accuracy = evaluate(dev_set)
improve_based_on(dev_set)
# Final evaluation on untouched test set
final_accuracy = evaluate(test_set)
Production Anti-Patterns
No Rate Limiting
Problem: Unlimited LLM calls per user.
# ANTI-PATTERN: Open access
@app.route("/generate")
async def generate():
return await llm.generate(request.prompt) # No limits!
Why it fails:
- Single user can exhaust budget
- Denial of service risk
- Cost surprises
- No fair usage
Solution:
# PATTERN: Rate limiting
@app.route("/generate")
@rate_limit(requests_per_minute=10, requests_per_day=100)
@cost_limit(max_cost_per_day=1.0)
async def generate():
return await llm.generate(request.prompt)
No Caching
Problem: Every identical request hits the LLM.
# ANTI-PATTERN: No cache
async def answer_faq(question: str) -> str:
return await llm.generate(question) # Same FAQ, same cost every time
Why it fails:
- Wasted money on identical queries
- Unnecessary latency
- Inconsistent answers to same question
Solution:
# PATTERN: Semantic caching
async def answer_faq(question: str) -> str:
cached = await cache.get_similar(question, threshold=0.95)
if cached:
return cached.response
response = await llm.generate(question)
await cache.set(question, response)
return response
Interview Questions
Q: What is the biggest anti-pattern you see in LLM applications?
Strong answer:
"The most damaging is the 'God Prompt' anti-pattern: a single massive prompt trying to handle every scenario.
Why it is common: It seems simpler to start with one prompt and add instructions as needs arise.
Why it fails:
- Context consumed by instructions, not user content
- Conflicting instructions confuse the model
- Cannot optimize for different use cases
- Changes have unpredictable side effects
The fix: Route to specialized handlers. Each handler has a focused prompt optimized for one task. The router itself can be simple (keyword-based) or smart (LLM-based for complex cases).
This applies beyond prompts. The general principle is: decompose complexity into specialized components rather than cramming everything into one monolith."
Q: How do you avoid agent runaway costs?
Strong answer:
"Multiple limits at different levels:
Per-request limits:
- Maximum steps (e.g., 20)
- Maximum tokens (e.g., 50K)
- Maximum time (e.g., 5 minutes)
Per-session limits:
- Daily token budget
- Daily cost cap
Per-user limits:
- Rate limiting (requests per minute/hour/day)
- Cost attribution and caps
Monitoring:
- Real-time cost tracking
- Alerts for anomalies (single request > $1)
- Circuit breaker if costs spike
Architecture:
- Cascade from cheap to expensive models
- Cache common operations
- Batch similar requests
The key is assuming the agent will try to run forever. Build in hard stops at every level. I have seen agents run up $1000 bills in minutes without proper limits."
Previous: Design Patterns
Key takeaways
01
The God Prompt is the worst offender
One 5000-token system prompt covering every scenario spends context on instructions, creates conflicting rules, and makes any single edit a change to all behaviours at once.
02
Agents need several termination conditions
One done-check is not enough: the corrected loop caps steps at 20, spend at $10 and wall clock at 300 seconds, returning which limit it hit.
03
Retrieving more is retrieving worse
top_k=50 pasted straight into context drowns signal and triggers lost-in-the-middle; the fix retrieves 20, reranks, and keeps only the top 5 scoring above 0.7.
04
Vibes are not evaluation
Printing five outputs and eyeballing them is unreproducible and cherry-picked; the replacement scores 100+ examples and keeps the explicit list of cases below 0.5.
05
Never tune against the test set
Iterating a prompt 100 times on the same held-out examples overfits to them; split a dev set for iteration from a test set touched only at the end.