LLM chains and agents are non-deterministic black boxes: a single request fans out across retrieval, model calls, and tools. Tracing captures that execution as a tree of nested spans - with inputs, outputs, tokens, latency, and cost - so tools like LangSmith and Langfuse let you debug, measure, and trust what your app actually did.
Tracing records the full execution of an LLM request as a trace - a tree of nested spans. Each span is one unit of work (a retrieval, an LLM call, a tool invocation, a chain step) with its own inputs, outputs, timing, and metadata. Observability is what you build on top: dashboards, evals, and alerts that turn those traces into insight about correctness, latency, and cost.
The whole request, root to leaf - one user question producing one span tree. It carries a trace ID that ties every step together.
A single operation inside the trace - a retrieval, generation, or tool call - with start/end time, inputs, outputs, and a parent link.
Model & version, prompt template, token counts, cost, user/session IDs, tags, and errors attached to spans for slicing later.
| Field | Why it matters | Example |
|---|---|---|
| Inputs / outputs | The exact prompt sent and text returned - the core of debugging | Rendered prompt, completion, retrieved chunks |
| Tokens | Prompt + completion counts drive cost and context-limit checks | prompt: 1420, completion: 210 |
| Latency | Per-span duration reveals which step is slow | retrieval: 40ms, llm: 2.3s |
| Cost | Derived from tokens × model price for budgeting | $0.0043 / request |
| Model & error | Version pins regressions; errors show failed steps | claude-sonnet-4-5, timeout, 429 |
The same input can produce different outputs, chains hide many steps behind one response, and agents decide their own path at runtime. Without traces you are guessing. Observability turns "it sometimes gives bad answers" into a specific span you can inspect and fix.
Reproduce a bad answer by replaying its exact trace - see the retrieved context, the rendered prompt, and where the model went wrong.
Agents loop, branch, and call tools on their own. The span tree makes their reasoning path visible so you can see why they looped or picked a bad tool.
Every span records tokens and cost, so you can attribute spend per feature, user, or model and catch runaway prompts before the bill does.
Per-span timing exposes the slow step; pinning model & prompt versions lets you catch when a change made quality or latency worse.
Same idea, four ways to picture it, so it clicks whoever you are.
A trace is the worked solution, not just the final answer. Instead of only seeing the grade, you see every step the model took, so you can spot exactly where the reasoning went wrong.
It is a stack trace for probabilistic code. Each span is a frame with its own inputs, outputs, and timing, nested under its parent, so you can drill into the call that actually broke.
Think of an audit trail for every request: who asked what, which sources were pulled, what each step cost, and how long it took, all reconstructable after the fact.
A trace is a flight recorder, the black box on the plane. When something goes wrong, you replay the recording step by step instead of guessing what happened mid-flight.
Your app is instrumented so each step opens a span, records its data, and closes it. Spans are batched and shipped asynchronously to a tracing backend (LangSmith, Langfuse, or any OpenTelemetry collector), which reassembles them into a trace tree and powers dashboards, evals, and alerts.
Wrap functions with a decorator or callback, or auto-instrument the SDK. Each wrapped call emits a span with a parent link, so nesting is captured automatically.
Spans are buffered and flushed in the background to keep the hot path fast. The backend stores them, joins them by trace ID, and indexes by tags, cost, and latency.
flowchart LR
subgraph App["🖥️ Instrumented app"]
Q["❓ Request"] --> RT["🌳 Root span"]
RT --> RET["🔎 Retrieval span"]
RT --> GEN["🧠 LLM span"]
RT --> TOOL["🛠️ Tool span"]
end
subgraph Pipe["🚚 Telemetry pipeline"]
EXP["📤 Span exporter"] --> COL["🧰 OTel collector"]
end
subgraph Backend["📡 Tracing backend"]
ING["📥 Ingest + join by trace ID"] --> STORE[("🗄️ Trace store")]
STORE --> DASH["📊 Dashboards"]
STORE --> EVAL["🧪 Evals + datasets"]
STORE --> ALERT["🚨 Alerts"]
end
RET --> EXP
GEN --> EXP
TOOL --> EXP
COL --> ING
@trace(name="rag_request") # opens the root span
def answer(question, user):
with span("retrieval") as s: # child span
chunks = vector_db.search(question, filter=user.acl)
s.log(inputs=question, outputs=chunks, latency_ms=s.dt)
with span("llm") as s: # sibling child span
out = llm.generate(prompt(question, chunks))
s.log(model="claude-sonnet-4-5",
prompt_tokens=out.usage.input,
completion_tokens=out.usage.output,
cost=price(out.usage))
return out.text
# → backend reconstructs: rag_request → [retrieval, llm]Each with span(...) nests under the active parent, so the backend rebuilds the exact tree. In high-volume systems you sample - keep 100% of errors and a percentage of the rest - to bound storage while still catching problems.
Three views: capturing a trace across a chain, nested spans for a multi-step agent, and traces flowing to dashboards with alerting.
sequenceDiagram
autonumber
participant App as 🖥️ App
participant VDB as 🗄️ Vector DB
participant LLM as 🧠 LLM
participant TB as 📡 Tracing backend
App->>TB: Start root span (trace ID)
App->>VDB: retrieve(question)
VDB-->>App: top-k chunks
App->>TB: Log retrieval span (in, out, latency)
App->>LLM: generate(prompt + chunks)
LLM-->>App: answer + token usage
App->>TB: Log LLM span (tokens, cost, model)
App->>TB: End root span
Note over App,TB: Spans batched then flushed async
sequenceDiagram
autonumber
participant Ag as 🤖 Agent
participant Tool as 🛠️ Tools
participant LLM as 🧠 LLM
participant TB as 📡 Tracing backend
Ag->>TB: Open agent root span
loop reason-act steps
Ag->>LLM: plan next action
LLM-->>Ag: chosen tool + args
Ag->>TB: Log LLM child span
Ag->>Tool: call tool(args)
Tool-->>Ag: observation
Ag->>TB: Log tool child span (nested)
end
alt goal reached
Ag->>TB: End root span (success)
else max steps hit
Ag->>TB: End root span (halted)
end
sequenceDiagram
autonumber
participant App as 🖥️ App fleet
participant TB as 📡 Tracing backend
participant Dash as 📊 Dashboards
participant Ops as 🧑💻 On-call
App->>TB: Stream spans continuously
TB->>TB: Aggregate p95 latency, cost, error rate
TB->>Dash: Update live metrics
alt threshold breached
TB->>Ops: Fire alert (slow / costly / failing)
Ops->>TB: Open offending trace
TB-->>Ops: Root-cause span
else within budget
TB->>Dash: Show healthy trend
end
From an uninstrumented app to a fully observable one - traces feeding dashboards, evals, and alerts, in order.
Add a tracer to your app and decorate the top-level handler so every request opens a root span with a unique trace ID.
Wrap retrieval, LLM calls, and tool calls so each becomes a child span linked to its parent, forming the trace tree.
Log the rendered prompt, retrieved context, and completion on each span - the raw material for every later diagnosis.
Add token counts, per-span duration, model and version, and derived cost so spans are quantitative, not just descriptive.
Attach user, session, tenant, and feature tags plus environment, so traces can be sliced and filtered later.
Buffer and flush spans in the background to LangSmith, Langfuse, or an OpenTelemetry collector without slowing the request.
Keep 100% of errors and a percentage of healthy traffic so storage and cost stay bounded while problems still surface.
The backend stitches spans by trace ID into a navigable tree and indexes them by tag, cost, latency, and error.
Promote real traces into evaluation datasets and score outputs - connecting production behavior to offline quality checks.
Chart p95 latency, cost, and error rate; set thresholds that page on-call when a metric breaches its budget.
Open the offending trace, find the root-cause span, ship a fix, and watch the metric recover - a closed observability loop.
Tracing is easy to bolt on badly. These are the mistakes that make traces leaky, expensive, or useless.
Dumping raw prompts and outputs captures names, emails, and secrets in plaintext. Redact or hash sensitive fields at capture time and set retention rules before you ship.
Persisting 100% of traffic looks fine in dev and explodes storage and cost in production. Keep every error and a sampled fraction of healthy traffic to stay bounded.
Emitting unstructured log lines throws away the parent-child tree. Without nested spans you cannot see which step called which, or where time was actually spent.
Spans with no token counts, model, or cost turn observability into pretty timelines you cannot budget against. Attach usage to every model span.
Capturing spans just when something throws hides the slow-but-successful and the confidently-wrong. Trace the full context, not only the exceptions.
Traces that never feed datasets or scorers stay a debugging toy. Promote real traces into evals so production behavior drives offline quality checks.
Good observability is a handful of numbers you can alert on. Capture these per span and per trace.
| Metric | What it tells you | Good sign |
|---|---|---|
| p50 / p95 latency per span | Which step is slow, and how bad the tail is | Within SLA; small p95-to-p50 gap |
| Tokens & cost per trace | Spend attributed to a request, feature, or user | Stable and within budget |
| Error / exception rate | Share of traces that failed or threw | Low and trending flat |
| Tool-call success rate | How often agent tool calls return usable results | High: agents rarely retry or loop |
| Trace sampling coverage | What fraction of traffic you actually capture | Enough to catch issues without blowing cost |
| Time-to-debug | How fast you go from alert to root-cause span | Minutes, not days |
Three representative patterns showing tracing and observability in production-style use.
An agent occasionally loops forever or answers from the wrong tool, and logs only show the final failure.
An LLM feature's bill is climbing and no one knows which flow or user is driving it.
A RAG endpoint feels slow, but the team can't tell whether retrieval, reranking, or generation is to blame.
LLM observability is converging on open standards and tightening the loop between production traces and quality.
GenAI semantic conventions are standardizing span names and attributes, so traces move between vendors without lock-in.
LLM-as-judge and heuristic scorers run continuously on sampled production traces, flagging quality drops in real time.
Interesting or failing traces flow straight into eval datasets, closing the gap between what shipped and what you test.
Models summarize a failing trace and suggest the offending span and fix, cutting time-to-diagnosis on complex agents.
Purpose-built views for loops, branches, and tool graphs make deep agent traces navigable instead of overwhelming.
Automatic PII redaction and sampling let teams keep rich traces while respecting data-handling and retention rules.