Case Study · Agentic AI · MCP

Shipping a support agent to production

Aria is a customer-support agent at a SaaS. The demo dazzled — it answered questions, looked up orders, issued refunds. Then came the hard part: making it safe, reliable, and measurable enough to let loose on real customers. This is the gap between a great demo and a production agent, and how it was closed.

A worked application of the Agentic AI & MCP Server playbooks

0The demo-to-production gap

In the demo, Aria was a language model with a few tools and a clever prompt. In production, it faces adversarial users, prompt-injection in the content it reads, tools that can move money, and a business that needs to measure whether it's actually helping. A demo optimizes for "wow"; production optimizes for "won't hurt anyone and can prove it works."

🤖 An agent is a loop that calls tools — treat every tool call as an API request

Strip away the magic and an agent is simple: a model in a think → act → observe loop, where "act" means calling a tool. The danger and the discipline both come from that: a tool call is a real action in the real world — reading private data, sending an email, issuing a refund.

So the entire security model from the rest of the stack applies unchanged: least privilege (scope every tool), authorization (does this user's session permit this action on this object?), input validation (the model's output is untrusted input to your tools), and human-in-the-loop for anything irreversible. The agent isn't a new world — it's your existing API surface, with a language model deciding which endpoints to hit.

1The agent loop & tools via MCP

Aria's tools are exposed through MCP servers — a standard way to give an agent typed tools, data, and prompts behind an authenticated boundary. The model proposes a tool call; a guardrail authorizes it; the MCP server executes it against the real API and returns an observation. Repeat until the model can answer.

Sequence diagram — one turn of the agent loop
sequenceDiagram autonumber participant U as User participant A as Agent (LLM) participant G as Guardrail / authz participant M as MCP server participant T as Order API U->>A: Where is my order 5012? A->>A: think — I need the order status A->>G: proposed tool call get_order(5012) G-->>A: allowed (read-only, user owns 5012) A->>M: call get_order(5012) M->>T: GET /orders/5012 (scoped token) T-->>M: status shipped, ETA Tue M-->>A: observation shipped, ETA Tue A->>A: observe — enough to answer A-->>U: Order 5012 shipped, arrives Tuesday Note over A,G: every tool call is authorized; the model never holds raw admin power
Renders with Mermaid. The loop is think → act (tool) → observe, repeated.
Tools are least-privilege and typed — never a blank "run anything"
// ✅ scoped, typed tools; reads are safe, writes need a human
const tools = [
  { name: 'get_order', scope: 'read',  auth: ownRecordsOnly },   // safe
  { name: 'refund',    scope: 'write', confirm: 'human' }        // requires approval
]

// ❌ NEVER expose a broad tool an injected prompt could hijack:
//   run_sql(query)  ·  shell(cmd)  ·  http_get(anyUrl)  ·  send_email(to, body)
// each of those hands the model (and any attacker who can influence it) real power

2Guardrails — the lethal trifecta & prompt injection

The signature agent vulnerability is prompt injection: untrusted text the agent reads (a customer message, a web page, a product review) contains instructions the model obeys. It becomes catastrophic only when it lines up with the lethal trifecta.

The lethal trifecta — dangerous only when all three combine
   (1) access to PRIVATE data      the user's orders, PII, secrets
   (2) exposure to UNTRUSTED text   a message, a web page, a review, a doc
   (3) ability to EXFILTRATE         send an email, call a URL, post data

        (1) ─────── (2)
          \        /      any TWO of these = fine
           \      /       all THREE = an injected instruction in (2)
            \    /         can use (1) + (3) to steal data
             (3)
   defense: break the triangle. don't grant one agent path all three at once.
Entity diagram — the tool / permission model
erDiagram AGENT ||--o{ TOOL_GRANT : has TOOL_GRANT }o--|| TOOL : "for" TOOL }o--|| MCP_SERVER : "exposed by" USER ||--o{ SESSION : opens SESSION ||--o{ TOOL_CALL : makes TOOL_CALL }o--|| TOOL : invokes TOOL { string name string scope "read-only or write" bool needs_confirmation } TOOL_GRANT { string agent string tool string constraint "own-records only" } MCP_SERVER { string name string auth "OAuth 2.1 scoped token" } TOOL_CALL { string tool datetime at bool allowed }
Renders with Mermaid. Every tool is scoped; every call is authorized and logged.
GuardrailWhat it doesDefends
Least-privilege toolsRead-only by default; writes scoped to the user's own objects; no broad run_sql/shell/http toolsInjection that tries to reach private data or exfiltrate
Human-in-the-loopIrreversible/high-value actions (refunds, cancellations) require explicit human approvalThe agent (or an injection) taking a costly action alone
Authorize every callEach tool call checks the user's session — the model's request is untrusted input, not authorityThe model requesting another user's data
Isolate untrusted contentTreat fetched pages/messages as data, not instructions; don't give a content-reading path exfil toolsThe lethal trifecta lining up
Rate + cost limitsCap tool calls, tokens, and loop iterations per sessionRunaway loops, cost blowups, abuse
Gotcha: you cannot "prompt your way" to safety — no system prompt reliably stops injection, because the attacker's text and yours look identical to the model. Safety comes from what the agent can DO, not what you tell it. Constrain the tools and require human approval for anything irreversible, and an injected instruction hits a wall it can't talk its way past.

3Evals & the staged rollout

"It worked when I tried it" is not a launch criterion. Aria shipped behind evals (does it actually help, and is it safe?) and a gradual rollout, because an agent's behavior is probabilistic and drifts as models and prompts change.

ABuild an eval set before launchthe launch gate

Goal: a repeatable measure of quality and safety, not vibes.

  1. Collect real cases — a labeled set of representative support questions with known-good outcomes, plus known-hard and adversarial ones.
  2. Score automatically — correctness, groundedness (did it use real data, not hallucinate?), tone, and refusal on out-of-scope/injected inputs.
  3. Add red-team cases — prompt-injection payloads, attempts to reach other users' data, attempts to trigger writes. These must fail safely, every run.
  4. Gate releases on the eval — a prompt or model change that regresses the score doesn't ship.
BRoll out gradually, human-in-the-loop first

Goal: limit blast radius while real traffic teaches you what the eval missed.

  1. Suggest-only mode — the agent drafts replies a human agent approves/sends. Zero autonomous actions; you learn its behavior safely.
  2. Canary — enable autonomous read-only answers for a small % of traffic; watch quality and cost.
  3. Expand by capability, not all at once — add write tools (with human approval) only after reads are proven; keep the highest-value actions gated.
  4. Keep an escape hatch — easy hand-off to a human, and a kill switch to disable tools instantly.
CObserve in production

Goal: know what the agent is doing, and catch drift.

  1. Trace every loop — log the full think/act/observe trace, tool calls, and decisions (with a trace id) so you can debug any conversation.
  2. Monitor the right signals — tool-call authz denials, refusal rate, loop length, cost per session, and human-override rate (a proxy for quality).
  3. Sample & re-eval — periodically score live conversations against the eval rubric; alarm on regression as models/prompts change underneath you.

4Scorecard

Nine checks that separate a production agent from a demo.

Every tool is least-privilege and typed — read-only by default, no broad run_sql/shell/http.
Every tool call is authorized against the user's session — the model's request is untrusted input.
Irreversible/high-value actions require human approval (human-in-the-loop).
No single agent path holds the lethal trifecta (private data + untrusted content + exfiltration).
Tools reached via MCP with scoped, authenticated tokens, not ambient credentials.
An eval set (including red-team/injection cases) gates every release.
Rollout is staged — suggest-only → read-only canary → gated writes.
Full traces, cost/loop/refusal metrics, and a kill switch are in place.
There's an easy hand-off to a human and periodic re-eval to catch drift.
The one-line takeaway: a production agent is a probabilistic system wired to real actions, so you engineer it like any other — least-privilege tools, authorization on every call, humans on the irreversible ones, evals as the launch gate, and a staged rollout with observability. The model is the easy part; the guardrails and evals are the product.