0The demo-to-production gap
In the demo, Aria was a language model with a few tools and a clever prompt. In production, it faces adversarial users, prompt-injection in the content it reads, tools that can move money, and a business that needs to measure whether it's actually helping. A demo optimizes for "wow"; production optimizes for "won't hurt anyone and can prove it works."
🤖 An agent is a loop that calls tools — treat every tool call as an API request
Strip away the magic and an agent is simple: a model in a think → act → observe loop, where "act" means calling a tool. The danger and the discipline both come from that: a tool call is a real action in the real world — reading private data, sending an email, issuing a refund.
So the entire security model from the rest of the stack applies unchanged: least privilege (scope every tool), authorization (does this user's session permit this action on this object?), input validation (the model's output is untrusted input to your tools), and human-in-the-loop for anything irreversible. The agent isn't a new world — it's your existing API surface, with a language model deciding which endpoints to hit.
1The agent loop & tools via MCP
Aria's tools are exposed through MCP servers — a standard way to give an agent typed tools, data, and prompts behind an authenticated boundary. The model proposes a tool call; a guardrail authorizes it; the MCP server executes it against the real API and returns an observation. Repeat until the model can answer.
// ✅ scoped, typed tools; reads are safe, writes need a human
const tools = [
{ name: 'get_order', scope: 'read', auth: ownRecordsOnly }, // safe
{ name: 'refund', scope: 'write', confirm: 'human' } // requires approval
]
// ❌ NEVER expose a broad tool an injected prompt could hijack:
// run_sql(query) · shell(cmd) · http_get(anyUrl) · send_email(to, body)
// each of those hands the model (and any attacker who can influence it) real power2Guardrails — the lethal trifecta & prompt injection
The signature agent vulnerability is prompt injection: untrusted text the agent reads (a customer message, a web page, a product review) contains instructions the model obeys. It becomes catastrophic only when it lines up with the lethal trifecta.
(1) access to PRIVATE data the user's orders, PII, secrets
(2) exposure to UNTRUSTED text a message, a web page, a review, a doc
(3) ability to EXFILTRATE send an email, call a URL, post data
(1) ─────── (2)
\ / any TWO of these = fine
\ / all THREE = an injected instruction in (2)
\ / can use (1) + (3) to steal data
(3)
defense: break the triangle. don't grant one agent path all three at once.| Guardrail | What it does | Defends |
|---|---|---|
| Least-privilege tools | Read-only by default; writes scoped to the user's own objects; no broad run_sql/shell/http tools | Injection that tries to reach private data or exfiltrate |
| Human-in-the-loop | Irreversible/high-value actions (refunds, cancellations) require explicit human approval | The agent (or an injection) taking a costly action alone |
| Authorize every call | Each tool call checks the user's session — the model's request is untrusted input, not authority | The model requesting another user's data |
| Isolate untrusted content | Treat fetched pages/messages as data, not instructions; don't give a content-reading path exfil tools | The lethal trifecta lining up |
| Rate + cost limits | Cap tool calls, tokens, and loop iterations per session | Runaway loops, cost blowups, abuse |
3Evals & the staged rollout
"It worked when I tried it" is not a launch criterion. Aria shipped behind evals (does it actually help, and is it safe?) and a gradual rollout, because an agent's behavior is probabilistic and drifts as models and prompts change.
ABuild an eval set before launch▶
Goal: a repeatable measure of quality and safety, not vibes.
- Collect real cases — a labeled set of representative support questions with known-good outcomes, plus known-hard and adversarial ones.
- Score automatically — correctness, groundedness (did it use real data, not hallucinate?), tone, and refusal on out-of-scope/injected inputs.
- Add red-team cases — prompt-injection payloads, attempts to reach other users' data, attempts to trigger writes. These must fail safely, every run.
- Gate releases on the eval — a prompt or model change that regresses the score doesn't ship.
BRoll out gradually, human-in-the-loop first▶
Goal: limit blast radius while real traffic teaches you what the eval missed.
- Suggest-only mode — the agent drafts replies a human agent approves/sends. Zero autonomous actions; you learn its behavior safely.
- Canary — enable autonomous read-only answers for a small % of traffic; watch quality and cost.
- Expand by capability, not all at once — add write tools (with human approval) only after reads are proven; keep the highest-value actions gated.
- Keep an escape hatch — easy hand-off to a human, and a kill switch to disable tools instantly.
CObserve in production▶
Goal: know what the agent is doing, and catch drift.
- Trace every loop — log the full think/act/observe trace, tool calls, and decisions (with a trace id) so you can debug any conversation.
- Monitor the right signals — tool-call authz denials, refusal rate, loop length, cost per session, and human-override rate (a proxy for quality).
- Sample & re-eval — periodically score live conversations against the eval rubric; alarm on regression as models/prompts change underneath you.
4Scorecard
Nine checks that separate a production agent from a demo.