∞ Production patterns, real tradeoffs, interview-grade depth
AI System Design Guide
One hundred and eight chapters on how production AI systems are actually built: retrieval, agents, inference economics, evaluation, and the failure modes that never make it into a launch post. Written for engineers shipping LLM systems and for the staff-level interviews that interrogate them. It is a repository rather than a book because the model landscape rewrites itself faster than a print run.
The guide
Learning paths
Curated routes through the guide. Each is a reading order, not a syllabus.
Prepare for interviews
5 pages · 127 minThe question bank and the answer frameworks first, then two case studies to practice saying the architecture out loud.
- Interview Question Bank
Eighty numbered questions across RAG, agents, model selection, optimization, evaluation and MLOps, each paired with what interviewers are listening for in the answer
- Answer Frameworks
Five reusable answer structures — SPIDER for system design, ETA for concepts, STAR-L for behavioral — plus what to say when you don't know
- Common Pitfalls
The mistakes that sink otherwise strong candidates: skipping the data pipeline, hand-waving evaluation, over-designing, and talking past the interviewer's actual question
- Multi-Tenant AI SaaS
Coca-Cola and Pepsi on shared infrastructure: hybrid namespace-versus-dedicated-pod isolation, defense in depth on every query, and GDPR deletion that truly removes embeddings.
- Real-Time AI Search Engine
Market intelligence with a five-minute freshness guarantee: Kafka ingestion into paired vector and full-text indexes, RRF fusion, sourced answers inside three seconds.
Learn AI systems fast
5 pages · 49 minThe shortest route from transformer internals to a working mental model of retrieval, model choice and the patterns that recur everywhere.
- LLM Internals
The moving parts inside a transformer LLM — attention, feed-forward blocks, position encodings, MoE routing — and the scaling laws that set what any of it costs
- Embeddings and Vector Spaces
What a vector actually encodes, which distance metric to use, and how Matryoshka truncation and binary quantization cut index size without wrecking recall
- RAG Fundamentals
Grounding versus training as the core choice, the current RAG taxonomy, and where a two-million-token context window makes retrieval unnecessary — and where it does not
- Model Taxonomy
A map of the March 2026 model field — frontier, reasoning, fast, open-weight, specialized — with the capability tiers and residency constraints that narrow your shortlist
- AI Design Patterns
Seventeen named patterns from naive RAG through Corrective RAG, plan-and-execute agents, model cascades and token budgets, each with a when-to-use and a stated cost.
Build production RAG
5 pages · 21 minChunking, vector store selection and reranking, ending in the 500,000-document enterprise build that uses all three.
- RAG Fundamentals
Grounding versus training as the core choice, the current RAG taxonomy, and where a two-million-token context window makes retrieval unnecessary — and where it does not
- Chunking Strategies
Small chunks find, large chunks explain — resolve that tension with recursive structure splitting, semantic boundaries, and parent-child hierarchies tuned per content type
- Vector Database Comparison
A vendor-by-vendor read on Pinecone, Qdrant, Weaviate, Milvus and pgvector, scored on scale, filtering, tenancy and the real cost of managed versus self-hosted
- Reranking Strategies
The bi-encoder to cross-encoder funnel in brief, with listwise LLM reranking and distilled small-model rerankers weighed against the milliseconds they add per query
- Enterprise RAG
A financial firm's 500,000-document search, where daily updates, department permissions and mandatory citations drive the shift from small-chunk RAG to million-token balanced context.
Build agents
6 pages · 27 minLoops, tools and MCP, then LangGraph for state, then sandboxing — because the interesting failures are all about what an agent is allowed to do.
- Agent Fundamentals
What separates an agent from a chatbot — reasoning model, tools, memory, environment feedback — and the autonomy levels you should deliberately choose between
- Tool Use and MCP
Tool schemas, the Model Context Protocol with its Streamable HTTP and auth updates, computer-use tools, and how MCP differs from plain function calling
- LangGraph Orchestration
Agents as graphs: nodes, conditional edges and a typed state schema, with checkpointing that lets a run pause for human approval and resume later.
- Agentic Security and Sandboxing
An injected agent does not leak data, it acts — so isolate execution in sandboxes, scope credentials to minimum agency, proxy every call, and log the trail
- Claude Code
Anthropic's terminal-native coding agent end to end: the tool loop, the CLAUDE.md manifest, sub-agent parallelism, permission model, and headless use inside CI.
- Autonomous Coding Agent
A planner-coder-debugger loop running inside an E2B sandbox on thousand-file repos, iterating until the test suite passes, for under fifty cents a task.
Run it in production
5 pages · 42 minEvaluation before observability, reliability before guardrails, security throughout, and a release pipeline that can catch a bad prompt.
- LLM Evaluation
Building an eval that can actually fail: quality dimensions, calibrated LLM judges, human annotation, RAGAS-style retrieval scores, and monitoring that runs on live traffic.
- LLM Observability
Logs, metrics and traces reshaped for non-deterministic systems, where quality ranks alongside latency as a first-class signal and cost is attributed per pipeline step.
- Reliability Patterns
Retry with jittered backoff, circuit breakers, bulkheads, per-stage timeouts, graceful degradation and multi-provider failover, each sized against a real availability target.
- LLM Security
Prompt injection, indirect injection through retrieved content, data extraction and unsafe output, mapped onto the OWASP LLM Top 10 with layered defenses and red-team tests.
- CI/CD for LLM Applications
Shipping prompt and model changes safely: staged eval suites, quality gates that block a regression, canary rollout, and a rollback that actually works.
Move into an AI role
4 pages · 134 minMap your current job to a target role, close the gaps with vetted courses, then come back through the interview prep section.
- Role Transition Guide
Backend, QA, PM, EM and data engineers mapped onto AI roles: which existing skills transfer, which gaps to close, and which chapters close them.
- Courses & Learning Paths
Verified courses per topic, from Karpathy and Stanford CS324 to the DeepLearning.AI shorts and the Maven evals course, with role-based sequences through them.
- Glossary
Every term the guide uses, defined once: ABAC and BM25 through MCP and SWE-bench Verified to zero-shot, so the chapters can stay dense.
- Interview Question Bank
Eighty numbered questions across RAG, agents, model selection, optimization, evaluation and MLOps, each paired with what interviewers are listening for in the answer
🎯 Why This Guide
Traditional books are outdated before they ship. This is a living document: when new models release, when patterns evolve, this updates.
| This Guide | Printed Books |
|---|---|
| March 2026 models (Claude 3.7 Sonnet, GPT-4.5, o3, Gemini 2.0 Flash, Grok 3) | Stuck on GPT-4 |
| MCP, Claude Code, Agentic RAG, OpenCoder landscape | Does not exist |
| Real pricing with verification dates | Already wrong |
| Staff-level interview Q&A | Generic questions |
📖 Guide Structure
Seventeen numbered sections, a glossary and four root guides — the whole repository fits on one page
Text version of this diagram
├── 00-interview-prep/ # Questions, frameworks, exercises
├── 01-foundations/ # Transformers, attention, embeddings
├── 02-model-landscape/ # Claude 3.7, GPT-4.5, o3, Gemini 2.0, DeepSeek
├── 03-training-and-adaptation/ # Fine-tuning, LoRA, DPO, distillation
├── 04-inference-optimization/ # KV cache, PagedAttention, vLLM
├── 05-prompting-and-context/ # CoT, Extended Thinking, DSPy, prompt injection
├── 06-retrieval-systems/ # RAG, chunking, GraphRAG, Agentic RAG
├── 07-agentic-systems/ # MCP 2.0, multi-agent, swarms, computer-use
├── 08-memory-and-state/ # L1-L3 memory tiers, Mem0, caching
├── 09-frameworks-and-tools/ # LangGraph, DSPy, LlamaIndex, Claude Code, OpenCoder
├── 10-document-processing/ # Vision-LLM OCR, multimodal parsing
├── 11-infrastructure-and-mlops/ # GPU clusters, LLMOps, cost management
├── 12-security-and-access/ # RBAC, ABAC, multi-tenant isolation
├── 13-reliability-and-safety/ # Guardrails, red-teaming
├── 14-evaluation-and-observability/ # RAGAS, LangSmith, drift detection
├── 15-ai-design-patterns/ # Pattern catalog, anti-patterns
├── 16-case-studies/ # Real-world architectures with diagrams
├── GLOSSARY.md # Every term defined
│
├── ai_evals_comprehensive_study_guide.md # 🔬 Deep-dive: AI Evals (Phoenix + Langfuse)
└── ai_evals_complete_guide_langwatch_langfuse.md # 🔬 Deep-dive: AI Evals (LangWatch + Langfuse)
└── COURSES.md # 🎓 Recommended courses & learning paths
└── TRANSITION_GUIDE.md # 🔄 Transition from Backend/QA/PM/EM to AI roles
🔥 Featured Case Studies
Real interview problems with complete solutions and diagrams:
| Case Study | Problem | Key Patterns |
|---|---|---|
| Real-Time Search | 5-minute data freshness at scale | Streaming + Hybrid Search |
| Coding Agent | Autonomous multi-file changes | Sandboxing + Self-Correction |
| Multi-Tenant SaaS | Coca-Cola and Pepsi on same infra | Defense-in-Depth Isolation |
| Customer Support | 60% auto-resolution rate | Tiered Routing + Escalation |
| Document Intelligence | 50K contracts/month extraction | Vision-LLM + Parallel Extractors |
| Recommendation Engine | Personalized explanations at 50M users | ML Ranking + LLM Explanations |
| Compliance Automation | FDA regulation pre-screening | Claim Extraction + Precedent DB |
| Voice Healthcare | Real-time clinical note generation | On-Prem ASR + HIPAA |
| Fraud Detection | 100ms decision with explainability | ML + Rules Hybrid |
| Knowledge Management | 2M docs with access control | Permission-Aware RAG |
🔬 Bonus Deep-Dive Guides
Two companion guides (3,000+ lines each) covering AI evaluation end-to-end — for Engineers, PMs, and QAs:
| Guide | Platforms Covered | What's Inside |
|---|---|---|
| AI Evals: Comprehensive Study Guide | Arize Phoenix + Langfuse | LLM-as-a-Judge, RAG eval, multi-turn eval, production safety, statistical correction with judgy, 30-day learning path |
| AI Evals: LangWatch + Langfuse Guide | LangWatch + Langfuse | Same syllabus with LangWatch's 40+ built-in evaluators, side-by-side platform comparisons, platform choice guidance |
Topics covered across both guides:
- Tracing and observability setup (Phoenix, LangWatch, Langfuse)
- Error analysis: open coding → axial coding → failure mode taxonomy
- Building LLM judges with Train/Dev/Test split and ground truth calibration
- Code-based evaluators (regex, JSON schema, format validators)
- RAG-specific evals: faithfulness, context recall, answer relevance
- Multi-step pipeline evaluation and multi-turn conversation eval
- Production guardrails, safety monitoring, real-time drift detection
- Statistical correction with
judgylibrary - Human annotation best practices and inter-rater reliability
- Cost/latency optimization for eval pipelines at scale
🎓 For Interview Prep
AI system design interviews ask questions like:
This guide gives you concrete patterns, real tradeoffs, and production failure modes: the depth interviewers expect at senior levels.
➡️ Start with Interview Prep
🔄 Living Book
This guide tracks:
- New model releases and real-world performance
- Emerging patterns (MCP, Agentic RAG, Flow Engineering)
- Updated pricing and rate limits
- Deprecations and best practice changes
⭐ Star and Watch to get notified when updates are pushed.
🤝 Contributing
Found outdated info? Have production experience to share? PRs welcome. See Contributing Guide.
📄 License
MIT License. See LICENSE.