How They Will Test You

AI Engineering Design Patterns · 7 Essential Questions

Chapter 4 is all about engineering judgment for production-grade Agents. These questions are appearing more and more often in interviews and design reviews. Answer them yourself first, then check the framework.

How to Use This Page
Each question is labeled with who's asking. This chapter leans technical — tech colleagues have more questions, and they are the most unsparing.
🎙 InterviewerWants to verify you truly understand — not just recite buzzwords
👔 BossWants explanations and commitments
🛠 Tech colleagueTesting whether you're worth trusting
Each question has three layers: What they're assessing → Answer framework → Bonus points. For anything you can't answer, click the linked lesson pages at the end.
Q1Interviewer
"Everyone is talking about context engineering. What exactly is the difference from writing good Prompts? Why did Prompt engineering suddenly become obsolete?"
🎯 What they're assessing
The opening concept question for this chapter, testing whether you've kept up with the paradigm shift from single-turn conversations to multi-step Agents. Anyone who just says "context engineering has a broader scope" is reciting a definition. Someone who truly understands can articulate exactly what's in the window and why it must be managed.
🧭 Answer framework
  1. Give the definition first: Prompt engineering optimizes how instructions are written. Context engineering manages all the Tokens sent to the model at each reasoning step: System Prompt, tool definitions, conversation history, retrieval results, user state — all of it.
  2. Explain the motivation: Context is a scarce resource. Three hard constraints: Context Rot (the longer the context, the lower retrieval accuracy), limited attention budget (irrelevant Tokens dilute useful information), and quadratic complexity (doubling the context quadruples attention computation).
  3. State the goal: Find the minimum high-signal Token set. Every Token must contribute to the reasoning — the mindset of "stuff in as many as possible" doesn't work.
  4. Give three handles: System Prompt at the right altitude (role + principles, don't pile on 50 rules); lean tool set; Few-shot of 2-3 carefully chosen representative examples — don't pad with edge cases to look thorough.
⭐ Bonus points Cite the quadratic cost: expanding context from 50K to 100K quadruples attention computation. Many people know that longer is more expensive; few can say that longer also makes it dumber.
Organize your answer with these lesson pages → From Prompt Engineering to Context Engineering The Three Pillars of Context
Q2Interviewer
"Your Agent needs to run long tasks of dozens of steps. What do you do when the context window is almost full? Can you just open a new session and keep going?"
🎯 What they're assessing
Testing whether you understand the fundamental dilemma of long tasks. Someone who answers "open a new session" doesn't realize the new window remembers nothing. Someone who answers "switch to a larger context model" hasn't calculated the cost of attention dilution. This question separates people who've seen production systems from those who've only played with demos.
🧭 Answer framework
  1. State the dilemma first: Open a new window, and the Agent loses all memory — it will redo work it already completed. Stay in the old window, and Tokens pile up, attention dilutes, and performance keeps dropping. Claude Code, Cursor, and Devin work on this problem every day.
  2. Pillar 1 — Compaction: When the window is near full, use one LLM call to produce a structured summary. Keep architecture decisions and open bugs; discard redundant tool outputs and intermediate steps of completed tasks. Pick the wrong items to discard and the Agent will repeat its mistakes.
  3. Pillar 2 — Structured notes: Proactively write key information to external files; new windows read them back to restore memory. Claude Code's TODO file and the game notes Claude maintains while playing Pokémon are both examples of this.
  4. Pillar 3 — Sub-Agents: Delegate deep exploration. A sub-Agent burns 30K Tokens in its own window reading code and reasoning, then returns only a 1,500-Token conclusion to the main Agent. The main context stays clean at all times.
⭐ Bonus points Articulate each pillar's scope: compact to stay lean within the window; notes to pass memory across windows; sub-Agents to isolate exploration noise. Then add that real products use all three in combination. This shows you grasp the system, not just the terms.
Organize your answer with these lesson pages → The Three Pillars of Context Why Agents Struggle with Long Tasks Session ≠ Context Window
Q3Tech colleague
"The product needs to add codebase Q&A. You're not going to kick off a project to build a vector store, are you? Claude Code does everything with grep on the spot."
🎯 What they're assessing
Testing whether you've defaulted to RAG as the answer. A PM who immediately goes to chunking, vectorization, and building an index looks to the tech colleague like someone who always reaches for a hammer. They want to hear you compare approaches and calculate the cost of maintaining an index.
🧭 Answer framework
  1. Accept the premise: The goal is to put the right information into the context window — RAG is just one means among many. For data that changes frequently like a codebase, just-in-time retrieval is often more appropriate.
  2. Explain JIT retrieval: Use glob/grep to search on demand, keeping the context lean and containing only what's currently needed. The cost is one extra tool-call latency; the gain is eliminating the maintenance burden of building and synchronizing an index.
  3. Give a hybrid strategy: Preload high-frequency information (project conventions, core rules, user preferences); fetch long-tail information on demand. Analogous to browser caching: hot data in memory, cold data fetched on request.
  4. Clarify when RAG is right: RAG suits relatively static knowledge bases, but naive chunking loses context. Contextual Retrieval adds a context prefix to each Chunk, combined with BM25 dual-path retrieval and Reranking, reducing retrieval failure rate by 67%.
⭐ Bonus points Proactively calculate the Contextual Retrieval cost: one extra LLM call per Chunk for the prefix, which can be reduced with Prompt Caching. Only worthwhile for high-accuracy use cases like legal, medical, or financial — not for chat recommendations.
Organize your answer with these lesson pages → JIT Context vs Preloading Contextual Retrieval: Better RAG
Q4Interviewer
"The production Agent keeps selecting the wrong tools and filling in the wrong parameters. The engineers say the model is too dumb — just wait for the next generation. As a PM, what do you think?"
🎯 What they're assessing
Testing whether you know about ACI. Anyone who agrees with "wait for the next model" is immediately out. The interviewer wants to hear: tool definitions are the Agent's user interface, wrong tool selection is most likely a design problem, and PMs have a clear diagnostic checklist.
🧭 Answer framework
  1. Establish the frame: A tool's name, parameters, and description are the Agent's user interface. Traditional APIs are deterministic; Agent tools are non-deterministic — when they're used and how depends entirely on design quality. Tool design deserves the same investment as HCI design.
  2. Give a diagnostic checklist: Go through four principles. Does parameter order give the model thinking space (simple direction first, complex content after)? Does the format align with training data (standard unified diff beats a custom DSL)? Are you forcing the model to count line numbers mechanically? Is there mistake-proofing (Poka-yoke)?
  3. Give a concrete example: On SWE-bench, changing the file path parameter to accept only absolute paths (not relative paths) was a single parameter change that transformed tool calls from frequently erring to nearly perfect.
  4. State the description standard: Write as if documenting for a smart junior developer with no context. Cover the five-pack: example usage, edge cases, input format, how it differs from other tools, and when not to use it.
⭐ Bonus points Add "If humans can't tell which tool to use, AI won't be able to either." Then mention Claude Code's advanced approach: using an Agent to write descriptions for its own tools, run evals, and auto-iterate toward optimization.
Organize your answer with these lesson pages → ACI: Agent-Computer Interface Using an Agent to Optimize Its Own Tools Keep the Tool Set Lean
Q5Boss
"The new model has been out for a week. Competitors announced integration the day after release. We need three weeks to evaluate? Explain where the time is going."
🎯 What they're assessing
On the surface it's pushing for speed; underneath it's asking whether your team has an eval infrastructure. This is the question that turns passive blame into an opportunity to request resources. Answering "that's just how engineering schedules work" is admitting incompetence. Answering "we'll switch tomorrow" is gambling with product quality.
🧭 Answer framework
  1. Lead with the conclusion: Migration speed depends on eval infrastructure. Teams with a solid eval suite run the tests, confirm no regressions, and switch in a few days. Teams without one spend weeks on manual verification. Our slowness is tech debt in infrastructure.
  2. Explain what evals buy you: Change a Prompt, swap a model, tune a parameter — and know in minutes what the overall impact is. Prevent fixing one bug and creating three. Be first in line to benefit every time a new model launches.
  3. Give a launch plan: Start with 20 test cases covering core scenarios. 20 well-designed cases beats 500 that are still in the planning document by an entire generation.
  4. Manage expectations while you're at it: Competitors who integrate fast aren't necessarily testing rigorously. Public benchmark scores are inflated (the model can recognize exams) — use your own business scenario cases. Your eval environment must match production; sandbox configuration differences alone can cause 6 percentage points of error.
⭐ Bonus points Replace adjectives with metric language in your reports: upgrade "feels worse" to "conciseness improved from 72 to 85, but over-engineering degraded from 3% to 7% — needs rollback." Your boss's trust in you will jump a level.
Organize your answer with these lesson pages → Why Evaluation Matters More Than Training Eval Pitfalls: Noise, Cheating, and Regression
Q6Interviewer
"How do you automate scoring of Agent output quality? Is LLM-as-Judge reliable enough on its own?"
🎯 What they're assessing
Testing the depth of your eval toolbox mastery. Anyone who just says "use an LLM to score" has merely heard of it. Someone who can clearly explain the boundaries and combination of the three Grader types sounds like they've actually run evals.
🧭 Answer framework
  1. List all three Grader types: Code Grader (assertions, unit tests, regex — millisecond latency, zero cost, fully reproducible, but too strict on reasonable variants); Model Grader (can evaluate subjective quality, but has cost and bias); Human Grader (highest quality, but doesn't scale).
  2. Answer the LLM-as-Judge question directly: Its reliability depends entirely on the Rubric. "Rate quality from 0 to 1" is nearly useless — you need to be specific at every score level: what does 0 look like, what is 0.3 missing, what conditions must all be met to score 1.
  3. Give the combination: Code Grader as the foundation for deterministic scenarios, Model Grader to extend to subjective quality, humans periodically spot-checking and calibrating for Model Grader drift. All three layers are indispensable.
  4. Add a commonly missed point: What you evaluate should be the Outcome — the final state of the environment. The Agent saying it's done doesn't count; you need to check whether the file was actually changed correctly and whether the API was actually called correctly.
⭐ Bonus points Mention the Trial concept: model output is stochastic — the same Task must be run multiple times to have statistical meaning. Then cite Descript's three-dimension scoring (didn't break anything, did what was required, did it well) to show you've seen real-world cases.
Organize your answer with these lesson pages → Three Grader Types: Code, Model, Human Core Concepts in Evaluation
Q7Tech colleague
"This requirement has the Agent running model-generated code using the user's GitHub Token. If something goes wrong, who's responsible? Adding a line that says 'don't run dangerous operations' to the Prompt isn't going to satisfy me."
🎯 What they're assessing
The classic confrontation in a security review, testing whether you understand structural security. If your answer only contains "add constraints to the Prompt" or "the model will refuse" — this requirement gets rejected on the spot. The interviewer wants to confirm you know the defense must be built into the architecture.
🧭 Answer framework
  1. Agree with the other person's stance first: Prompt-based defenses are unreliable; security must rely on structural design. The goal is that even if the model is completely manipulated by Prompt Injection, the attacker still cannot obtain credentials.
  2. Classify the risks: Three categories, each with separate defenses: intentional user abuse; model-initiated loss of control (over-acting, executing real operations based on hallucinations); external attacks (injection instructions embedded in web pages and documents, without the user's knowledge).
  3. Give the credential solution: First principle: generated code and secrets are always isolated in separate containers. Two modes: Token injected into resource access path (Agent-usable but invisible — e.g., embedded in Git remote URL); Vault proxy forwarding (proxy injects Token per session, Agent never sees a single character).
  4. Give the execution environment solution: OS-level sandbox with triple isolation (file system, network, process), layered with three-level trust control: manual approval for high-risk tools, session-level authorization, global policy as the final backstop (production databases are never reachable).
⭐ Bonus points Proactively say "the stronger the model, the larger the attack surface in old architectures," so security design cannot rely on model upgrades to automatically improve. This statement makes security engineers see you as one of their own.
Organize your answer with these lesson pages → Three Risk Categories: Abuse, Loss of Control, External Attack Sandbox and Credential Isolation
One Final Tip
Most questions in this chapter come from the actual floor of technical design reviews, where what's always wanted is judgment and trade-offs. The right approach is still to speak them out loud — to a colleague, a friend, or a recording. The parts that don't flow smoothly are exactly what you think you understand but don't. Click the linked lesson pages and go fill the gaps.