1.What makes something an agent versus just an LLM
application?
An agent is a system built around three pillars: perception, reasoning and action. It perceives its environment through user input and tool outputs, reasons autonomously about what step to take next-not following a hardcoded script-and acts through tools and APIs. The key differentiator is the autonomous multi-step loop: it breaks down goals, selects tools, observes results, and adapts. A chatbot responds to prompts. A RAG system retrieves and answers. An agent owns the execution loop and adapts when things fail.
2. Can you give me an example of something people call an agent that isn’t really one?
A chatbot with function calling but no autonomous decision loop. If the code hard-codes which tool to call for each intent, it’s a pipeline with an LLM step, not an agent. Apply the autonomy litmus test: can it break down goals, choose tools, adapt on failure, and operate across turns?
3. Where do you draw the line between an agent and an automation pipeline?
The line is autonomous decision-making. If every step is predetermined, it’s a pipeline. If the system decides its next step based on observations from the previous step, it’s agentic. Many systems are hybrids — deterministic structure with agentic decision points.
4. We’re building a system that processes insurance claims. Should it be an agent?
Before deciding, I’d ask: are the processing steps always the same? If every claim follows the same sequence — verify identity, check coverage, calculate amount, approve or deny — that’s a deterministic workflow. I’d use a pipeline with one LLM call for any judgment step like damage assessment. Agents add a complexity tax: harder debugging, higher cost, more failure modes. But if claims vary significantly — some need external appraiser calls, some need fraud investigation, some need multi-party coordination — then the routing logic is context-dependent, and an agent makes sense. For insurance, I’d lean toward a hybrid: deterministic pipeline for the 80% standard claims, with agent routing for the 20% edge cases that need adaptive decision making.
5. What about fraud detection? Would that change your answer?
Fraud investigation is inherently context-dependent — different evidence, different patterns. That’s where an agent adds value: adaptive investigation based on what each step reveals. But standard claim processing remains deterministic. The agent handles the exception path, not the happy path.
6. How would you decide when to move a claim from the pipeline to the agent?
Rules-based triage: claim amount over threshold, flagged keywords, mismatch in documentation, prior fraud history. These triggers route to the agent path. The pipeline handles everything else. Track the routing accuracy — if the pipeline is sending too many to the agent or missing cases, adjust thresholds.
7. How would you handle it when a tool the agent needs is temporarily down?
I would handle it at three layers. Prevention: before calling, check if the service is healthy via a circuit breaker or health endpoint. Recovery: for transient failures like timeouts, retry with exponential backoff – 1 second, 2 seconds, 4 seconds, max 3 attempts. If the service is down, don’t burn retries. Communication: send a structured error back to the LLM: “Payment service is currently uunavailable. Try again in 5 minutes, or use the manual_refund_queue tool to queue for later processing.” The key is the agent adapts — it doesn’t crash. It tries an alternative tool, queues
the action for later, or tells the user what’s happening with a concrete next step.
8. What if 3 out of 5 tools the agent needs are all down?
Graceful degradation. The agent should do what it can with available tools, clearly communicate what it couldn’t do, and queue the remaining actions. “I completed steps 1 and 3. Steps 2, 4, and 5 are queued for when services recover. I’ll notify you when complete.” Partial completion is better than total failure.
9. How would you prevent the agent from retrying a permanently down service?
Circuit breaker pattern. After N consecutive failures (e.g., 5), the circuit opens — all calls fail fast for a cooldown period (e.g., 60 seconds). After cooldown, try one probe request. If it succeeds, close the circuit. This prevents burning resources on a dead service.
10. An agent has access to a database with all customer records. How do you secure it?
The critical principle: the LLM is not an authentication layer. I’d enforce security in code, not prompts. First, user identity propagation — every database query carries the requesting user’s identity. The agent can’t access records the user doesn’t own. Second, read-only by default — the database tool only supports SELECT queries. Any write operation requires a different tool with explicit authorization. Third, parameterized queries only — no raw SQL, ever. The agent generates parameters, my code builds the query. Fourth, scope limiting — queries are automatically filtered to the user’s own data via row-level security. Fifth, audit logging — every query logged with user identity, timestamp, and what was accessed. If a prompt injection tries to “show all customer records,” it hits the permission layer and gets denied in code, regardless of what the LLM was tricked into generating.
11. What about a malicious user who manipulates the agent via prompt injection?
Every control is enforced in code, not in the prompt. Even if the LLM is tricked into generating “SELECT * FROM all_customers,” the row-level security filter automatically scopes it to the user’s records. The parameterized query builder rejects any raw SQL. Defense-in-depth means no single layer’s failure exposes the system.
12. How would you handle a legitimate support agent who needs to access another customer’s records?
Role-based access. Support agents get a different permission set that allows cross-customer access, but with audit logging, access justification (ticket ID required), and time-limited sessions. The agent’s permissions still flow from the authenticated human’s role, not from its own service credentials.
13. Your agent has been running for 6 months. Users are complaining that it ‘doesn’t know them anymore.’ What’s happening?
This is the staleness problem — stored memory that was accurate 6 months ago is now outdated and the agent is acting on it confidently. Three things are likely happening. First, preference drift — users’ tastes and needs have changed but memory hasn’t updated. Second, context shifts —users have changed jobs, relocated, or had life changes the agent doesn’t know about. Third, no decay mechanism — facts stored 6 months ago have the same weight as facts from yesterday. The fix: attach metadata to every stored fact — timestamp, confidence score, source, and expiry
date. Confidence decays over time. When it drops below 70%, the agent verifies: “I remember you prefer X — is that still the case?” Preferences get a 6–12 month TTL. Contextual facts get 1–3 months. Temporal facts get explicit expiry dates. Outdated memory is worse than no memory — it makes the agent confidently wrong.
14. How do you handle it when the agent has conflicting information — old fact says X, new signal says Y?
Contradiction detection. When new input conflicts with stored memory, the agent doesn’t silently overwrite — it asks: “I have on record that you work at X, but you just mentioned Y. Which is current?” User-stated facts override inferred facts. Recency wins if confidence is equal.
15. Wouldn’t all this verification be annoying to users?
Only verify when confidence is below threshold (70%) and the fact is about to be used in a decision. Don’t verify just because it’s old — verify when it matters. And batch verifications naturally: “Last time we talked you were at X and preferred Y — anything changed?” at session start, not mid-task.
16. How would you handle a situation where the context window is 80% full mid-conversation?
At 80% I’m already reactive — ideally I’d trigger at 60%. But here’s my approach: keep the last 10 turns verbatim because the LLM needs recent context for coherent responses. Summarize turns 11 through the oldest into a compressed narrative that preserves key decisions and facts —this reduces tokens by about 80%. Any user preferences or domain knowledge that was mentioned earlier, I’d move to external storage and retrieve on demand instead of keeping it in the window. After this optimization, I’m back to about 40% capacity with the most important
context preserved. Going forward, I’d set the trigger at 60% to stay ahead of this.
17. What do you lose when you summarize? How do you decide what to keep?
You lose nuance and exact wording. Keep: decisions made, action items, specific data points (numbers, dates, IDs), user corrections. Compress: exploratory discussion, repeated back-and-forth, pleasantries. The summarization prompt should explicitly say “preserve all decisions, data points, and corrections.
18. What if the conversation is high-stakes and you can’t afford to lose any detail?
For high-stakes, keep everything in context and accept the cost. But paginate: store the full conversation externally and retrieve relevant sections when needed. Or use a larger context window model for high-stakes tasks specifically. Match the strategy to the stakes.
19. Our RAG system retrieves relevant documents but the agent still gives wrong answers. Why?
If retrieval is returning relevant documents but answers are still wrong, the problem is downstream of retrieval. Three likely culprits. First, bad chunking — the retrieved chunk contains the right topic but the actual answer is split into a different chunk. The LLM has context about the topic but not the specific information it needs. I’d check if procedures or policies are being split mid-content. Second, context injection noise — the retrieved chunks are mixed with too much other context, and the LLM is overwhelmed or confused. I’d check how many tokens of retrieved
content vs conversation history are in the prompt. Third, the LLM is ignoring or misinterpreting the retrieved content — especially if the answer contradicts the LLM’s training data, it may default to its prior knowledge. I’d check the reasoning trace: did the LLM reference the retrieved docs in its thought process, or did it answer from memory? The fix depends on which of these three it is.
20. How would you verify whether the LLM is using the retrieved content or its own training data?
Check the reasoning trace. If the agent’s thought step doesn’t reference the retrieved document, it’s answering from memory. Require source attribution: the agent must cite which chunk it based its answer on. If it can’t cite, flag the response. Also compare: does the answer match the retrieved content or contradict it?
21. What would you change about the chunking if procedures are being split?
Switch from token-limit chunking to semantic chunking at section boundaries. Increase chunk size for procedural docs to 500+ tokens. Add 10–20% overlap between chunks. Test by retrieving for procedural queries and checking if complete instructions are returned.
22. When would you use multi-hop retrieval, and when is it overkill?
Multi-hop is for questions that require synthesizing across multiple sources. Four scenarios: cross-document comparison, entity chaining where the first doc reveals something to look up, iterative query refinement when the first retrieval is too broad, and graph-like knowledge traversal. It’s overkill when the answer exists in a single document, for simple factual lookups, or for linear questions that don’t need synthesis. The cost matters: multi-hop is 2 to 5 times more expensive. I’d default to single-hop and add multi-hop only when I can measure that it materially
improves answer quality. And always: max hops at 3 to 5 with a relevance check after each. Sometimes better chunking eliminates the need for multi-hop entirely — that’s the cheaper fix.
23. How do you prevent multi-hop from becoming an infinite loop?
Three guardrails: hard max at 3–5 hops, relevance check after each hop (did we get useful new info?), and query similarity check (is the new query too similar to a previous one?). If we hit max hops without an answer, fail gracefully: “I couldn’t find that information” is better than burning $5 on empty retrieval.
24. Can you give me a concrete example of entity chaining?
Find suppliers of components used in Product Y.” First hop: retrieve Product Y’s spec → extract component list. Second hop: for each component, look up supplier. Third hop: retrieve supplier details. Each retrieval depends on results from the previous one — that’s genuine multi-hop, not just running the same query multiple times.
25. Your agent is taking 47 steps to complete a task that should take 5. What’s happening and how do you fix it?
47 steps for a 5-step task is almost certainly a loop. I’d check the trace for three things. First, repeated actions — is it calling the same tool with the same parameters over and over? That means no action history — the agent doesn’t know it already tried this. Fix: make action history visible in context and add deduplication that rejects identical tool calls within a 5-step window. Second, hallucinated tools — is it trying to call tools that don’t exist, failing, and retrying? Fix: strict tool validation with clear error messages listing available tools. Third, dead-end cycling — is
it alternating between two approaches that both fail? Fix: after 3 failed attempts at any approach, the agent must try something fundamentally different or escalate. And the immediate fix for all three: hard max step limit at 10 to 15, graceful termination with a summary of what was tried.
26. How would you set the max step limit? Isn’t 10 too low for some tasks?
Calibrate by task type. Simple lookups: max 5. Standard workflows: max 10–15. Complex research: max 25 with explicit checkpoints. The limit should be 2–3x the expected step count. If a task regularly hits the limit, the limit isn’t the problem — the agent design is.
27. What does graceful termination look like to the user?
I attempted to [task] and completed steps 1–3 successfully. I encountered an issue at step 4 and tried 3 alternative approaches. Here’s what I found so far: [partial results]. I’d recommend [next step] to complete this.” Partial progress is always better than a blank failure.
28. Should we use Chain-of-Thought, ReAct, or Tree-of-Thought for our use case?
I’d ask two diagnostic questions. First, does the task require external tools or just reasoning? If all information is in context — analysis, math, planning — Chain-of-Thought is cheapest and fastest: one LLM call, no tool overhead. Second, if tools are needed, does the task have one clear path or multiple valid approaches? One clear path: ReAct — Thought-Action-Observation loop, interpretable, tool-calling built in. Multiple high-stakes paths to explore: Tree-of-Thought — but it’s 5 to 10 times more expensive because it branches. Three branches at 4 levels deep is 81 LLM calls vs 4 for linear. So my default is CoT for in-context reasoning, ReAct for tool-dependent tasks, and ToT only for high-stakes decisions where wrong paths are expensive. Most production agents use linear ReAct.
29. Can you combine these patterns?
Yes. Use ReAct as the outer loop with CoT within the Thought step for deeper reasoning. This gives you tool access plus reasoning depth without the full ToT cost. Only branch at critical decision points, not every step.
30. When is Tree-of-Thought worth the 10x cost?
When the cost of choosing the wrong path exceeds the cost of exploration. Financial planning, legal strategy, treatment options — where wrong decisions have real consequences. Not for customer support, simple lookups, or routine workflows. High stakes + multiple valid paths = ToT. Everything else = linear ReAct or CoT.
31. Your agent shipped last month. How do you know it’s actually working well?
Accuracy alone is a trap — I’d evaluate six dimensions. Correctness: are answers right? Efficiency: how many steps per task — baseline should be under 10. Cost: tokens times price with a per-request budget cap, say $0.50 for support tasks. Latency: P50, P95, P99 — not averages, because averages hide outliers. If P99 is 45 seconds, 1% of users are having a terrible experience. Safety: did the agent follow rules, call real tools, skip any validation steps? Reasoning quality: is the logic sound, or did it get the right answer by accident? For continuous monitoring: automated alerts on all six dimensions, sampled human review of 1 to 5 percent of production traces, and A/B testing when rolling out changes. One-time evaluation at launch is worthless — agent behavior drifts, user queries evolve, and model updates change performance.
32. How do you handle the trade-off between these dimensions? You can’t optimize all six.
Set priorities by domain. Financial: correctness + safety above all. Customer support: latency + cost matter most. Research: reasoning quality + correctness. Explicitly state which dimensions you’re optimizing and which you’re accepting as “good enough.” Track all six but alert only on priority dimensions.
33. What does ‘reasoning quality’ actually mean in practice?
The logic behind the answer is sound, not just the answer itself. An agent that gets the right answer by hallucinating incorrect intermediate steps is a ticking time bomb. Evaluate by sampling traces: does the thought chain logically lead to the conclusion? LLM-as-judge can automate this at scale.
34. We just got a $16,000 cloud bill surprise from our agent system. How do you prevent this?
Agent costs are multiplicative — steps times tokens times price. One stuck loop running 50 steps instead of 10 quintuples the cost for that request. A $16K surprise means no controls existed at any level. I’d implement three layers. Per-request: hard caps of 15 max steps, 50k max tokens, and $1 max cost per request. Exceed any limit and terminate gracefully. Per-user: daily budget of $10, checked before processing each request. System-wide: track daily spend, establish a baseline, and alert at 150% of baseline. The monitoring loop runs continuously: check budget
before each request, track cumulative cost during execution, log actual cost after, and flag anomalies — any request costing 10x average gets reviewed. For optimization: cache repeated queries for 20% savings, route simple tasks to cheaper models at 10x lower cost, and batch operations where possible.
35. What does graceful termination look like when a cost limit is hit?
I’ve reached my processing budget for this request. Here’s what I completed so far: [partial results]. To continue, this task would need to be resumed in a new session or escalated to a human.” Never silently drop the task — always return what was accomplished.
36. How do you balance cost limits with actually completing complex tasks?
Tiered limits by task type. Simple lookups: $0.20 cap. Standard workflows: $1 cap. Complex analysis: $5 cap with explicit user consent. The user or system should pre-classify the task and apply the appropriate tier. Don’t apply the cheapest limit to the hardest tasks.