I track KPIs at three levels. Operationally: containment rate (target >70%), p95 latency (<3s), and escalation rate. For quality: CSAT from a thumbs up/down widget, hallucination rate from sampled LLM-as-Judge scoring, and guardrail violation rate which must be zero. For business impact: support email volume reduction and cost per session — at 50 sessions/day on GPT-4o-mini, the agent costs roughly ₹2 per session versus ₹50-100 for human support handling time. That’s the ROI case.
KPIs are what connect your technical work to business value. There are 3 KPI Layers and KPIs are business metrics, not technical metrics.
- Technical Metrics → Eval framework. Refer this.
- Operational KPIs → How well is the agent running?
- Business KPIs → Is it actually helping the business?
Layer 1 — Operational KPIs:
Measured from FastAPI logs + CloudWatch
| KPI | Target | How to Measure |
|---|---|---|
| Containment Rate | > 70% | % of sessions resolved without human escalation |
| Response Latency (p95) | < 3 seconds | CloudWatch / Datadog on /chat endpoint |
| Tool Call Success Rate | > 95% | % of API calls that return 200 |
| Escalation Rate | < 30% | % of sessions that trigger escalate_to_support() |
| Session Length | < 4 turns | Avg turns to resolve — fewer is better |
| Uptime | > 99.5% | ECS health check failures / ALB 5xx rate |
Containment Rate is the most important operational KPI. It directly answers: “Is the agent actually replacing support work, or just passing everything to humans?”
Layer 2 — Quality KPIs
Measured from eval suite + LLM-as-Judge
| KPI | Target | Source |
|---|---|---|
| Resolution Accuracy | > 85% | LLM-as-Judge correctness score on sampled sessions |
| Hallucination Rate | < 2% | Groundedness score < 0.5 flagged per week |
| Guardrail Violation Rate | 0% | Example: regenerate_link called without order verification |
| FAQ Hit Rate | > 80% | Retrieval eval — refreshed after every Pinecone sync |
| Customer Satisfaction (CSAT) | > 4/5 | Thumbs up/down widget at end of chat session |
CSAT is the most important quality KPI — it’s the only one the customer directly gives you. Add a simple thumbs up/down to your chat widget.
Layer 3 — Business KPIs
Measured from support email volume
| KPI | Target | How to Measure |
|---|---|---|
| Support Email Reduction | > 40% drop | Compare monthly support emails before vs after launch |
| Time to Resolution | < 2 minutes | Timestamp first message → resolution message |
| Repeat Contact Rate | < 15% | Same customer opens new session within 24hrs for same issue |
| Download Issue Resolution Rate | > 90% | % of download complaints resolved without human |
| Agent Cost per Session | < ₹2 | OpenAI API cost + infra cost / total sessions |
Cost per session is critical for your business case:
Typical session cost breakdown:
~3 turns × ~500 tokens/turn = 1500 tokens
GPT-4o-mini input: 1500 × $0.00015 = ~$0.00023 (~₹0.02)
Pinecone query: free tier = ₹0
ECS Fargate: ~₹1.50/day ÷ sessions/day
At 50 sessions/day → total cost ~₹2/session
At 500 sessions/day → total cost ~₹0.30/session (scales well)
KPI Dashboard — What to Build
Add a /metrics endpoint to your FastAPI server
KPI Review Cadence
| Frequency | What to review |
|---|---|
| Daily | Latency, tool error rate, uptime — operational health |
| Weekly | Containment rate, CSAT, escalation rate — quality trends |
| Monthly | Support email volume reduction, cost per session — business impact |
| On every deploy | FAQ hit rate, guardrail violations — regression check |