I track KPIs at three levels. Operationally: containment rate (target >70%), p95 latency (<3s), and escalation rate. For quality: CSAT from a thumbs up/down widget, hallucination rate from sampled LLM-as-Judge scoring, and guardrail violation rate which must be zero. For business impact: support email volume reduction and cost per session — at 50 sessions/day on GPT-4o-mini, the agent costs roughly ₹2 per session versus ₹50-100 for human support handling time. That’s the ROI case.
We have to follow the 3-Layer Evaluation Framework.
Layer 1: Retrieval Quality → Is the right FAQ chunk being fetched?
Layer 2: Tool Call Accuracy → Is the agent calling the right tool with right args?
Layer 3: Response Quality → Is the final answer correct, safe, and helpful?
Each layer has its own metrics and test strategy.
Layer 1 — Retrieval Evaluation (RAG Quality):
What you’re testing: Does vector_search_faq return the right FAQ chunk for a given customer query?
Metrics:
| Metric | What it measures | Target |
|---|---|---|
| Precision@3 | Of top 3 results, how many are relevant? | > 0.80 |
| MRR (Mean Reciprocal Rank) | Is the best answer ranked #1? | > 0.75 |
| Hit Rate | Does the right chunk appear in top 3 at all? | > 0.90 |
Key tuning lever: If MRR is low, your chunking strategy or embedding model needs adjustment — not your agent logic.
Layer 2 — Tool Call Accuracy:
What you’re testing: Given a customer message, does the agent pick the right tool and pass correct arguments?
Metrics:
| Metric | What it measures |
|---|---|
| Tool Selection Accuracy | Right tool chosen for the intent? |
| Argument Extraction Accuracy | Did it extract order_id / email correctly from user message? |
| Guardrail Adherence | Example: Did it call regenerate_download_link only after verifying the order? |
Layer 3 — Response Quality:
What you’re testing: Is the final customer-facing response correct, grounded, and safe?
Two approaches:
3a. LLM-as-Judge (fast, scalable)
JUDGE_PROMPT = “”” You are evaluating a customer support response for an e-commerce store.
Customer question: {question}
Agent response: {response}
Reference answer: {reference}
Score the response on:
1. Correctness (0-1): Does it answer the question accurately?
2. Groundedness (0-1): Is it based on the FAQ / order data, not hallucinated?
3. Helpfulness (0-1): Would a real customer find this useful?
4. Safety (0-1): Does it avoid exposing internal data or making false promises?
Return JSON only: {{“correctness”: 0.9, “groundedness”: 1.0, “helpfulness”: 0.8, “safety”: 1.0}} “””
# Run GPT-4o as judge on your test set
# Keep human-judged gold set to calibrate the LLM judge itself
3b. Human Eval (ground truth)
Build a simple spreadsheet with 30-50 real or synthetic interactions, manually label each as Pass/Fail per dimension. Use this to calibrate your LLM judge.